The Experiment movie/Essay

Research for Human Services Michael R. Perkins, MSW, LCSW, Contributing Editor

This edition is adapted from a Psychology research text originally produced in 2010 by a publisher who has requested that they not receive attribution, with some material from Principles of Sociological Inquiry – Qualitative and Quantitative Methods by Amy Blackstone, University of Maine. Both published under this license:

Conditions of Use

FIGURE 1 IS THE SYMBOL FOR CREATIVE COMMONS LICENSE THIS WORK IS RELEASED UNDER.

Attribution-NonCommercial-ShareAlike

CC BY-NC-SA

This work is a derivative that is also published under that license which states:

You are free to:

 Share — copy and redistribute the material in any medium or format

 Adapt — remix, transform, and build upon the material

The licensor cannot revoke these freedoms if you follow the license terms.

The terms of the license are:

 Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

 NonCommercial — You may not use the material for commercial purposes.

 ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.

You may not apply legal terms or technological measures that restrict others from doing anything the license permits.

Notices:

You do not have to comply with the license for elements of the material in the book which are in the public domain or where your use is permitted by an applicable exception or limitation.

No warranties are given.

The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.

Forward About This Book I did not write this book. I did contribute, rewrote parts of it, created some original material, and added sections. My role is rather amorphous. Such is the nature of an open source project like this. An open source project such as this, (when it is done properly) takes on a life of its own - which is exactly what it is supposed to do. I did adapt, edit, and transform the original works (primarily an introductory text on research for Psychology students along with some material from an introductory text on research for Sociology students) into a text for Human Services majors. This book is based on those original works. Most of the material is from the psychology text. The material from the psychology text is by an author who has requested that they not receive attribution. There is also some material from another book. The author of the other book is Amy Blackstone from the University of Maine. I am deeply indebted to both of them for the work that they did, which made this derivative possible. This was all made possible through the Attribution-NonCommercial-ShareAlike 3.0 United States (CC BY-NC-SA 3.0 US) License.

The changes I have made reflect my own view of the research process, and what I think undergraduate students in Human Services need to know. It reflects an approach developed during my first career as a Human Service practitioner, and the twenty years following that teaching an introductory research class in Social Work and Human Services. It also reflects my forays into qualitative and quantitative research over the past three decades. To that end the focus has been shifted to Human Services, and there is much less emphasis on statistics. The emphasis on this text is to assist students into becoming better consumers of quality research, rather than producers of it. I think the latter is more appropriate for graduate work, or a more advanced class specifically designed to accomplish that purpose.

Open Source Earlier I mentioned open source, so what is it exactly? Probably the best place to explore the concept is www.opensource.org . They define open source software (even though in this case it applies to a book) in terms of the license granted:

The license shall not restrict any party from selling or giving away the software as a component of an aggregate software distribution containing programs from several different sources. The license shall not require a royalty or other fee for such sale.

Notice it says software, which is where the open source concept originated, and most notably with the development of the Linux operating system which was developed using that model. However, the same concept has been adapted for other creative endeavors including the written word. This book is open source and released into the wild under a generous creative commons license. For more information on the philosophy and approach of open Source please see The Cathedral and the Bazaar by Eric Steven Raymond (available to read online).

I am a devoted proponent of open source efforts such as Linux, Libre Office, and the growing number of textbooks that are being made available free to the public.

And finally, many thanks to Rebecca Whitworth who has joined me in this project, and who made, and continues to make, many contributions.

Sincerely,

Michael R. Perkins, MSW LCSW

Table of Contents Conditions of Use ......................................................................................................................................... 1

Forward ............................................................................................................................................................ 2

About This Book ........................................................................................................................................... 2

Open Source ............................................................................................................................................. 2

Table of Contents ............................................................................................................................................. 3

Preface .............................................................................................................................................................. I

Relevance, Balance, and Accessibility ......................................................................................................... 1

Introduction: Human Services ......................................................................................................................... 2

What is Human Services? ............................................................................................................................ 2

A Brief History of Human Services ............................................................................................................... 2

Origins of Human Services in Deinstitutionalization .................................................................................... 3

.Chapter 1: Science in the Social Sciences ..................................................................................................... 5

Who talks more? ........................................................................................................................................... 6

1.1 Understanding Science .......................................................................................................................... 6

LEARNING OBJECTIVES ........................................................................................................................ 6

Science & Human Services ...................................................................................................................... 6

Empirically Based Research ..................................................................................................................... 7

The Emergence of Science and the Scientific Method ................................................................................ 7

Features of Science .................................................................................................................................. 7

1. Systematic Empiricism.......................................................................................................................... 7

2. The Scientific Approach ....................................................................................................................... 8

3. Public Knowledge ................................................................................................................................. 8

Publication is an essential feature of science for two reasons:................................................................ 9

Science Versus Pseudoscience ................................................................................................................... 9

Karl Popper ............................................................................................................................................. 10

The Skeptic’s Dictionary ......................................................................................................................... 10

1.2 Scientific Research in the Human Services ......................................................................................... 12

LEARNING OBJECTIVES ...................................................................................................................... 12

A Model of Scientific Research ............................................................................................................... 12

Who Conducts Scientific Research? ...................................................................................................... 13

The Broader Purposes of Scientific Research in the Human Services .................................................. 13

Two Categories: Basic and Applied Research ...................................................................................... 13

Summary ................................................................................................................................................. 14

1.3 Science and Common Sense ............................................................................................................... 15

LEARNING OBJECTIVES .......................................................................................................................... 15

Can We Rely on Common Sense?......................................................................................................... 15

Some Great Myths .................................................................................................................................. 15

How Could We Be So Wrong? ............................................................................................................... 15

Practice: .................................................................................................................................................. 16

1.4 Science and Human Services Practice ................................................................................................ 18

LEARNING OBJECTIVES ...................................................................................................................... 18

Empirically Supported Treatments ......................................................................................................... 18

Discussion: .............................................................................................................................................. 19

Chapter 2: Getting Started in Research ........................................................................................................ 20

2.1 Basic Concepts ..................................................................................................................................... 20

LEARNING OBJECTIVES ...................................................................................................................... 20

Variables ................................................................................................................................................. 20

Sampling and Measurement ...................................................................................................................... 21

Statistical Relationships Between Variables .............................................................................................. 21

Single Variable ........................................................................................................................................ 21

Statistical Relationship Between Multiple Variables ............................................................................... 22

Differences Between Groups .................................................................................................................. 22

Correlations Between Quantitative Variables ......................................................................................... 23

Scatterplots ................................................................................................................................................. 23

Pearson’s r .................................................................................................................................................. 24

Independent and Dependent Variables .................................................................................................. 25

Correlation and Causation .......................................................................................................................... 25

Directionality Problem ............................................................................................................................. 26

Third-Variable Problem ........................................................................................................................... 26

“Lots of Candy Could Lead to Violence” ................................................................................................. 26

Experiment .................................................................................................................................................. 26

Practice ................................................................................................................................................... 27

2.2 Generating Good Research Questions ..................................................................................................... 27

LEARNING OBJECTIVES ...................................................................................................................... 27

Finding Inspiration ...................................................................................................................................... 28

Informal Observations ............................................................................................................................. 28

Practical Problems .................................................................................................................................. 28

Previous Research ................................................................................................................................. 28

Generating Empirically Testable Research Questions .......................................................................... 28

Evaluating Research Questions ................................................................................................................. 29

Interestingness........................................................................................................................................ 29

Does it fill a gap? .................................................................................................................................... 30

Feasibility ................................................................................................................................................ 30

Types of Studies in the Literature............................................................................................................... 30

Summary ................................................................................................................................................. 31

Practice: .................................................................................................................................................. 31

2.3 Reviewing the Research Literature ...................................................................................................... 31

LEARNING OBJECTIVES ...................................................................................................................... 31

What Is the Research Literature?............................................................................................................... 32

Professional Journals ............................................................................................................................. 32

Peer Review ............................................................................................................................................ 33

Scholarly Books ...................................................................................................................................... 33

Literature Search Strategies ................................................................................................................... 33

Using Other Search Techniques ............................................................................................................ 34

What to Search For .................................................................................................................................... 34

Newer Work ............................................................................................................................................ 34

Reviews of the Topic and Meta-analysis ................................................................................................ 34

Practice: .................................................................................................................................................. 35

Chapter 3: Research Ethics .......................................................................................................................... 36

3.1 Moral Foundations of Ethical Research ................................................................................................... 36

LEARNING OBJECTIVES .......................................................................................................................... 36

A Framework for Thinking About Research Ethics ................................................................................ 36

Moral Principles to Consider and Who Research Affects: ..................................................................... 37

Ethical Principles ........................................................................................................................................ 37

Weighing Risks Against Benefits ............................................................................................................ 37

The Milgram Study on Obedience .............................................................................................................. 37

Was It Worth It? ...................................................................................................................................... 38

Acting Responsibly and With Integrity .................................................................................................... 38

Seeking Justice: The Tuskegee Experiment .......................................................................................... 38

“They Were Betrayed” ............................................................................................................................ 39

Respecting People’s Rights and Dignity ................................................................................................ 39

Unavoidable Ethical Conflict ................................................................................................................... 39

Practice: ...................................................................................................................................................... 40

3.2 From Principles to Ethics Codes .............................................................................................................. 40

LEARNING OBJECTIVES .......................................................................................................................... 40

Historical Overview ..................................................................................................................................... 41

Ethics Codes............................................................................................................................................... 41

Research Ethics and Human Services .......................................................................................................... 42

Other Social Science Ethical Codes .......................................................................................................... 42

Research Ethics & Social Work .............................................................................................................. 42

Practice ....................................................................................................................................................... 42

Discussion .................................................................................................................................................. 42

3.3 Putting Ethics into Practice ...................................................................................................................... 43

LEARNING OBJECTIVES .......................................................................................................................... 43

Know and Accept Your Ethical Responsibilities ......................................................................................... 43

Identify and Minimize Risks ........................................................................................................................ 43

Identify and Minimize Deception ................................................................................................................ 44

Weigh the Risks Against the Benefits ........................................................................................................ 45

Create Informed Consent and Debriefing Procedures ............................................................................... 45

Get Approval ............................................................................................................................................... 46

Follow Through ........................................................................................................................................... 46

Discussion .................................................................................................................................................. 47

Practice ....................................................................................................................................................... 47

Chapter 4: Theory .......................................................................................................................................... 48

4.1 Phenomena and Theories ........................................................................................................................ 49

LEARNING OBJECTIVES .......................................................................................................................... 49

Phenomena ................................................................................................................................................ 49

Some Famous Phenomena from Psychology ............................................................................................ 49

What Is a Theory? .......................................................................................................................................... 50

What Are Theories For? ............................................................................................................................. 51

Organization ............................................................................................................................................... 51

Occam’s Razor & Parsimony ..................................................................................................................... 52

Prediction .................................................................................................................................................... 52

Generation of New Research ..................................................................................................................... 53

Multiple Theories: Competing and Complementary .................................................................................. 53

Example of Competing Theories: Where Do Multiple Personalities Come From? .................................... 53

Practice ....................................................................................................................................................... 54

Discussion: ................................................................................................................................................. 54

4.2 The Variety of Theories .............................................................................................................................. 55

LEARNING OBJECTIVES .......................................................................................................................... 55

Formality ..................................................................................................................................................... 55

Formal Theories ......................................................................................................................................... 55

Scope .......................................................................................................................................................... 55

Theoretical Approach ................................................................................................................................. 56

Functional Theories ................................................................................................................................ 56

Mechanistic Theories .............................................................................................................................. 56

Stage Theories........................................................................................................................................ 57

Practice: ...................................................................................................................................................... 58

Discussion: ................................................................................................................................................. 58

4.3 Using Theories in Social Research .......................................................................................................... 58

LEARNING OBJECTIVES .......................................................................................................................... 58

Theory Testing and Revision ...................................................................................................................... 58

Hypothetical-Deductive method .............................................................................................................. 59

Constructing or Choosing a Theory ........................................................................................................... 59

Deriving A Hypotheses ............................................................................................................................... 60

Evaluating and Revising Theories .............................................................................................................. 60

Incorporating Theory into Your Practice ..................................................................................................... 61

Incorporating Theory into Practice ............................................................................................................. 62

Practice: ...................................................................................................................................................... 62

Chapter 5: Measurement............................................................................................................................... 64

Do You Feel You Are a Person of Worth? ................................................................................................. 64

5.1 Understanding Clinical Measurement .................................................................................................. 66

LEARNING OBJECTIVES ...................................................................................................................... 66

What Is Measurement? .............................................................................................................................. 66

Constructs ................................................................................................................................................... 66

Aspects of the Human Personality: The Big Five .................................................................................. 67

Operational Definitions ............................................................................................................................... 68

Converging Operations ........................................................................................................................... 68

Levels of Measurement .............................................................................................................................. 69

Nominal Level ......................................................................................................................................... 69

Ordinal Level ........................................................................................................................................... 69

Interval Level ........................................................................................................................................... 69

Ratio Level .............................................................................................................................................. 69

Practice ....................................................................................................................................................... 71

5.2 Reliability and Validity of Measurement ............................................................................................... 71

LEARNING OBJECTIVES .......................................................................................................................... 71

Reliability .................................................................................................................................................... 71

Test-Retest Reliability ................................................................................................................................. 72

Internal Consistency ................................................................................................................................... 72

Interrater Reliability ..................................................................................................................................... 73

Using Interrater Reliability .......................................................................................................................... 73

Validity ........................................................................................................................................................ 73

Face Validity ............................................................................................................................................... 74

How Prejudiced Are You? .......................................................................................................................... 74

Content Validity .......................................................................................................................................... 74

Criterion Validity ......................................................................................................................................... 74

Discriminant Validity ................................................................................................................................... 75

5.3 Practical Strategies for Clinical Measurement ......................................................................................... 77

LEARNING OBJECTIVES .......................................................................................................................... 77

Conceptually Defining the Construct .......................................................................................................... 77

Deciding on an Operational Definition ........................................................................................................ 77

Using an Existing Measure ..................................................................................................................... 77

Creating Your Own Measure .................................................................................................................. 78

Implementing the Measure ......................................................................................................................... 79

Evaluating the Measure .............................................................................................................................. 79

Practice ....................................................................................................................................................... 80

Chapter 6: Experimental Research ............................................................................................................... 81

The Parable of the 38 Witnesses ............................................................................................................... 81

6.1 Experiment Basics .................................................................................................................................... 82

LEARNING OBJECTIVES .......................................................................................................................... 82

What Is an Experiment? ............................................................................................................................. 82

Internal and External Validity ...................................................................................................................... 82

Internal Validity ....................................................................................................................................... 82

External Validity ...................................................................................................................................... 83

Manipulation of the Independent Variable.................................................................................................. 84

Control of Extraneous Variables................................................................................................................. 84

Extraneous Variables as “Noise” ................................................................................................................ 85

Extraneous Variables as Confounding Variables ....................................................................................... 85

Practice ....................................................................................................................................................... 86

6.2 Experimental Design ................................................................................................................................ 86

LEARNING OBJECTIVES .......................................................................................................................... 87

Between-Subjects Experiments ................................................................................................................. 87

Random Assignment .................................................................................................................................. 87

Block Randomization .................................................................................................................................. 87

Treatment and Control Conditions ............................................................................................................. 88

No-Treatment Control Condition & the Placebo Effect........................................................................... 88

The Powerful Placebo ................................................................................................................................ 90

Within-Subjects Experiments ..................................................................................................................... 90

Carryover Effects and Counterbalancing ................................................................................................... 90

Counterbalancing ....................................................................................................................................... 91

When 9 Is “Larger” Than 221 ..................................................................................................................... 91

Simultaneous Within-Subjects Designs ..................................................................................................... 91

Between-Subjects or Within-Subjects? ...................................................................................................... 92

Summary: ................................................................................................................................................... 92

Discussion .................................................................................................................................................. 93

6.3 Conducting Experiments .......................................................................................................................... 93

LEARNING OBJECTIVES .......................................................................................................................... 93

Recruiting Participants ................................................................................................................................ 93

The Volunteer Subject ................................................................................................................................ 94

Characteristics of Volunteer Participants ............................................................................................... 94

Standardizing the Procedure ...................................................................................................................... 94

Experimenter Gender as an Extraneous Variable ..................................................................................... 95

Experimenter Expectancy Effect ................................................................................................................ 95

Record Keeping .......................................................................................................................................... 96

Pilot Testing ................................................................................................................................................ 96

Practice ....................................................................................................................................................... 97

Discussion .................................................................................................................................................. 97

Chapter 7: Nonexperimental Research ......................................................................................................... 98

7.1 Overview of Nonexperimental Research ............................................................................................. 98

LEARNING OBJECTIVES .......................................................................................................................... 98

What Is Nonexperimental Research? ........................................................................................................ 98

When to Use Nonexperimental Research .................................................................................................. 99

Types of Nonexperimental Research ......................................................................................................... 99

Nonexperiments ....................................................................................................................................... 100

Correlational and Quasi-Experimental Research ................................................................................. 100

Qualitative Research ............................................................................................................................ 100

Internal Validity Revisited ......................................................................................................................... 100

Discussion ................................................................................................................................................ 101

7.2 Correlational Research .......................................................................................................................... 102

LEARNING OBJECTIVES ........................................................................................................................ 102

What Is Correlational Research? ............................................................................................................. 102

Misconceptions about Correlational Research ........................................................................................ 102

Data Collection in Correlational Research ............................................................................................... 103

Naturalistic Observation ........................................................................................................................... 104

The Question of What will be Observed ................................................................................................... 104

Coding Data .......................................................................................................................................... 105

Archival Data ............................................................................................................................................ 105

Content Analysis ................................................................................................................................... 105

Summary: ................................................................................................................................................. 106

Discussion ................................................................................................................................................ 106

7.3 Quasi-Experimental Research ............................................................................................................... 107

LEARNING OBJECTIVES ........................................................................................................................ 107

Nonequivalent Groups Design ................................................................................................................. 107

Pretest-Posttest Design ............................................................................................................................ 108

Regression to the Mean........................................................................................................................ 108

Spontaneous Remission ....................................................................................................................... 108

Does Psychotherapy Work? ..................................................................................................................... 108

Interrupted Time Series Design ................................................................................................................ 109

Combination Designs ............................................................................................................................... 110

Quasi-Experimental Research ................................................................................................................. 111

7.4 Qualitative Research .............................................................................................................................. 112

What Is Qualitative Research? ................................................................................................................. 112

Characteristics of Quantitative Research Methods .............................................................................. 112

Characteristics of Qualitative Methods ................................................................................................. 113

Origins of Qualitative Research ................................................................................................................ 113

The Purpose of Qualitative Research ...................................................................................................... 113

Weaknesses of Quantitative Research are Strengths of Qualitative Research .................................. 114

Thick Description .................................................................................................................................. 114

Data Collection and Analysis in Qualitative Research ............................................................................. 114

Interviews .............................................................................................................................................. 114

Focus Groups ....................................................................................................................................... 114

Participant Observation ........................................................................................................................ 114

Data Analysis in Quantitative Research ................................................................................................... 115

Grounded Theory .................................................................................................................................. 115

Grounded Research in Action .............................................................................................................. 115

Five Themes of Postpartum Depression .............................................................................................. 116

The Quantitative-Qualitative “Debate” ...................................................................................................... 116

Mixed Methods and Triangulation ........................................................................................................ 116

Discussion ................................................................................................................................................ 117

Chapter 8: Survey Research ........................................................................................................................ 118

Why Survey Research? ............................................................................................................................ 118

General Social Survey .............................................................................................................................. 118

8.1 Survey Research: What Is It and When Should It Be Used? ............................................................ 118

LEARNING OBJECTIVES ........................................................................................................................ 118

Surveys are a Quantitative Method .......................................................................................................... 119

Exercise .................................................................................................................................................... 119

8.2 Pros and Cons of Survey Research ....................................................................................................... 119

LEARNING OBJECTIVES ........................................................................................................................ 119

Strengths of Survey Method ................................................................................................................. 119

Versatility .................................................................................................................................................. 120

Weaknesses of Survey Method ................................................................................................................ 120

Validity & Surveys ................................................................................................................................. 121

Exercises .................................................................................................................................................. 121

8.3 Types of Surveys .................................................................................................................................... 121

LEARNING OBJECTIVES ........................................................................................................................ 121

Cross-Sectional and Longitudinal Surveys .............................................................................................. 121

Cross-Sectional Research .................................................................................................................... 122

One Problem with Cross-Sectional Surveys ........................................................................................ 122

Longitudinal Surveys ................................................................................................................................ 122

Trend Survey ........................................................................................................................................ 122

Panel Surveys ....................................................................................................................................... 123

Cohort Survey ........................................................................................................................................... 123

Table 8.1 Types of Longitudinal Surveys ................................................................................................. 124

Retrospective ............................................................................................................................................ 124

Administration ........................................................................................................................................... 124

Example ................................................................................................................................................ 125

Online Delivery ......................................................................................................................................... 126

Summary .................................................................................................................................................. 126

Exercises .................................................................................................................................................. 127

8.4 Designing Effective Questions and Questionnaires............................................................................... 128

LEARNING OBJECTIVES ........................................................................................................................ 128

Asking Effective Questions ....................................................................................................................... 128

Are the questions clear? ....................................................................................................................... 128

Avoiding Confusing Questions ................................................................................................................. 130

Double-Barreled Questions .................................................................................................................. 131

Social Desirability ..................................................................................................................................... 131

Pilot Testing .............................................................................................................................................. 132

Response Options .................................................................................................................................... 132

Closed Ended Questions ...................................................................................................................... 132

Open-Ended Questions ........................................................................................................................ 133

Fence Sitting and Floating ........................................................................................................................ 133

Fence Setting ........................................................................................................................................ 133

Floating ................................................................................................................................................. 133

Using a Matrix ........................................................................................................................................... 133

Designing Questionnaires ........................................................................................................................ 135

Order of Questions: It Does Matter ..................................................................................................... 136

Questions of Time .................................................................................................................................... 136

Pretesting & Time to Complete ................................................................................................................ 137

Appearance .............................................................................................................................................. 137

Summary .................................................................................................................................................. 137

Exercises .................................................................................................................................................. 137

8.5 Analysis of Survey Data ......................................................................................................................... 138

LEARNING OBJECTIVES ........................................................................................................................ 138

From Completed Questionnaires to Analyzable Data .............................................................................. 138

Response Rate ..................................................................................................................................... 138

How to Improve Response Rate ........................................................................................................... 139

Have we been too concerned with high response rates? .................................................................... 139

Managing, Sorting, and Ordering Your Data ........................................................................................ 140

Using Statistical Software ......................................................................................................................... 140

Excel and Open Source Options .......................................................................................................... 140

PSPP..................................................................................................................................................... 140

Libre Office and Calc ............................................................................................................................ 140

R ............................................................................................................................................................ 140

Specialty Commercial Software Options .............................................................................................. 141

Identifying Patterns ................................................................................................................................... 141

Frequency Distribution .............................................................................................................................. 141

Measures of Central Tendency ................................................................................................................ 142

The Three Measures of Central Tendency .............................................................................................. 142

Mode ..................................................................................................................................................... 142

Median .................................................................................................................................................. 142

Mean ..................................................................................................................................................... 142

Bivariate Analysis ..................................................................................................................................... 143

Contingency Tables .............................................................................................................................. 143

Collapsing Categories ........................................................................................................................... 144

Conventions .......................................................................................................................................... 144

Multivariate Analysis ................................................................................................................................. 144

Summary ............................................................................................................................................... 144

Exercises .............................................................................................................................................. 145

Chapter 9 Descriptive Statistics ................................................................................................................... 146

9.1 Describing Single Variables ............................................................................................................... 146

LEARNING OBJECTIVES ........................................................................................................................ 146

Descriptive statistics ................................................................................................................................. 146

The Distribution of a Variable ................................................................................................................... 146

Frequency Tables ..................................................................................................................................... 146

Conventions for Frequency Distribution Tables ................................................................................... 147

Histograms ................................................................................................................................................ 148

Distribution Shapes .................................................................................................................................. 149

Symmetrical or Skewed ............................................................................................................................ 149

Outlier ................................................................................................................................................... 151

Measures of Central Tendency and Variability ........................................................................................ 151

Central Tendency ................................................................................................................................. 151

Mean ..................................................................................................................................................... 151

Median .................................................................................................................................................. 151

MODE ................................................................................................................................................... 151

Bimodal ................................................................................................................................................. 152

Measures of Variability ......................................................................................................................... 152

Standard Deviation ............................................................................................................................... 153

Computing the Standard Deviation ...................................................................................................... 154

N or N-1? .............................................................................................................................................. 154

Simple Standard Deviation Problem: Step by Step ............................................................................. 154

The Standard Deviation in More Detail ................................................................................................ 155

Percentile Ranks and z Scores ................................................................................................................ 155

Percentile Rank .................................................................................................................................... 156

Commonly Used Terms for Specific Percentiles .................................................................................. 156

Some Examples .................................................................................................................................... 156

Z Score ................................................................................................................................................. 157

Online Descriptive Statistics ..................................................................................................................... 157

Use of Spreadsheets for Data Management ............................................................................................ 158

Summary .................................................................................................................................................. 158

Practice ..................................................................................................................................................... 159

9.2 Describing Statistical Relationships ....................................................................................................... 159

LEARNING OBJECTIVES ........................................................................................................................ 159

Differences Between Groups or Conditions ............................................................................................. 159

Cohen’s d .................................................................................................................................................. 160

Sex Differences Expressed as Cohen’s d ............................................................................................ 161

Correlations Between Quantitative Variables .......................................................................................... 161

Nonlinear Relationship .......................................................................................................................... 165

Limited or Restricted Range ................................................................................................................. 165

Chapter 10: Single-Subject Research Designs .......................................................................................... 168

LEARNING OBJECTIVES ........................................................................................................................ 168

The Single-Subject Design ....................................................................................................................... 168

Single-Subject Design Conventions ......................................................................................................... 168

The X and Y Axis .................................................................................................................................. 169

AB Design ............................................................................................................................................. 169

The ABA Design ....................................................................................................................................... 169

ABAB Design ........................................................................................................................................ 170

Multiple-Treatment Reversal Design .................................................................................................... 171

Potential Problems with the Reversal Design .......................................................................................... 171

Multiple Baseline Design ...................................................................................................................... 171

Other Single-Subject Designs .............................................................................................................. 172

Data Analysis in Single-Subject Research ............................................................................................... 172

Visual Inspection ................................................................................................................................... 172

Statistical Analysis .................................................................................................................................... 173

Christopher: A Case History ..................................................................................................................... 173

Summary .................................................................................................................................................. 174

Practice ..................................................................................................................................................... 174

RESEARCH FOR HUMAN SERVICES pg. I

Preface The research methods course, or something very much like it, is inevitably required in the Human Services curriculum. And for good reason. While the importance of understanding research methods is usually clear to students who intend to pursue an advanced degree, I’ve long thought that those of us who teach research methods could do a better job of demonstrating to all of our students the relevance of what it is that we’re teaching. Why should students want what this class has to offer? That question is especially relevant for undergraduate students.

As someone who has used both qualitative and quantitative methods, I appreciate the need not only for students to understand the relevance of research methods for themselves, but also for them to understand the relevance of both qualitative and quantitative techniques. Also, as a teacher I have learned that students will not get much from sources they perceive to be overly boring, too full of jargon, or overly technical. Altogether, my experiences as a student, researcher, and teacher shape the three overriding objectives of this text: relevance, balance, and accessibility.

Relevance, Balance, and Accessibility This text emphasizes the relevance of research methods for the everyday lives of its readers: undergraduate students. The book describes how research methodology is useful for students in the multiple roles they fill:

 (1) As consumers of popular and public information; as citizens in a society where findings from social research shape our laws, policies, and public life; and

 (2) As Human Services professionals. You will find connections to these roles throughout and directly within the main text of the book rather than their being relegated to boxes.

Using a variety of examples, this text also aims to provide balanced coverage of qualitative and quantitative approaches. We’ll also cover some of the debates regarding the values and purposes of qualitative and quantitative research. In addition, we’ll discuss the strengths and weaknesses of both approaches.

One of the most important goals of this text is to introduce you to the core principles of social research in a way that is straightforward and keeps you engaged. As such, the text reflects an emphasis on research being accessible and readable.

In summary? Above all, the purpose of this textbook is to help you to become a more critical thinker, who can identify, implement, evaluate, and communicate best practices (from high quality and reliable sources) in your personal and professional life. One of the key marks of the educated person.

RESEARCH FOR HUMAN SERVICES pg. 2

Introduction: Human Services

It is appropriate for us, no matter which Human Services class we are in at the moment, to review the definition of Human Services and our history as a profession. One of the better definitions of Human Services, can be found on Wikipedia:

What is Human Services? Human services is an interdisciplinary field with the objective of meeting human needs through an applied knowledge base, focusing on prevention as well as remediation of problems, and maintaining a commitment to improving the overall quality of life of service populations. The process involves the study of social technologies (practice methods, models, and theories), service technologies (programs, organizations, and systems), and scientific innovations that are designed to ameliorate problems and enhance the quality of life of individuals, families and communities to improve the delivery of service with better coordination, accessibility and accountability.[1] The mission of human services is to promote a practice that involves simultaneously working at all levels of society (whole-person approach) in the process of promoting the autonomy of individuals or groups, making informal or formal human services systems more efficient and effective, and advocating for positive social change within society.

Human services practitioners strive to advance the autonomy of service users through civic engagement, education, health promotion and social change at all levels of society. Practitioners also engage in advocating so human systems remain accessible, integrated, efficient and effective.

Human services academic programs can be readily found in colleges and universities, which award degrees at the associate, baccalaureate, and graduate levels. Human services programs exist in countries all around the world. Retrieved from: https://en.wikipedia.org/wiki/Human_services on 11 July 2017.

After defining Human Services, the same page provides a brief history:

A Brief History of Human Services Human services has its roots in charitable activities of religious and civic organizations that date back to the Colonial period. However, the academic discipline of human services did not start until the 1960s [during the deinstitutionalization movement]. At that time, a group of college academics started the new human services movement and began to promote the adoption of a new ideology about human service delivery and professionalism among traditional helping disciplines. [2] The movement's major goal was to make service delivery more efficient, effective, and humane. The other goals dealt with the reeducation of traditional helping professionals (interprofessional education), to have a greater appreciation of the individual as a whole person (humanistic psychology) and to be accountable to the communities they serve (postmodernism). Furthermore, professionals would learn to take responsibility at all levels of government, use systems approaches to consider human problems, and be involved in progressive social change.

Traditional academic programs such as education, nursing, social work, law and medicine were resistant to the new human services movement's ideology because it appeared to challenge their professional status. Changing the traditional concept of professionalism involved rethinking consumer control and the distribution of power. The new movement also called on human service professionals to work for social change.[3] It was proposed that the reduction of the monopolistic control of professionals could result in democratization of knowledge and would lead to professionals advocating on behalf of clients and communities against professional establishments.[4] The movement also hoped that human service delivery systems would become integrated, comprehensive, and more accessible, which would make them more humane for

RESEARCH FOR HUMAN SERVICES pg. 3

service users.[5][6] Ultimately, the resistance from traditional helping professions served as the impetus for a group of educators in higher education to start the new academic discipline of human services. Retrieved from: https://en.wikipedia.org/wiki/Human_services on 11 July 2017.

Origins of Human Services in Deinstitutionalization

A glaring omission of the history, as outlined in Wikipedia, is why Human Services began in the first place. Deinstitutionalization was a government policy that moved mental health patients out of state-run psychiatric hospitals into federally funded community mental health centers. At least that was the theory. It began in the 1960s due to a number of coalescing factors, but largely as a cost saving measure. The process of deinstitutionalization was always underfunded and, as we shall see later, understaffed. The result is that many people ended up homeless, or without services that would help them cope in society. Deinstitutionalization can still be seen in the criminal justice system where persons with severe mental health issues who run afoul of the law end up incarcerated. It is a major problem in prisons and jails throughout the country.

As institutions were closed thousands of their inmates were released, many with no place to go, into overwhelming societies ability to care for them [7]. It was supposed to be a way to improve treatment of the mentally ill, and other persons warehoused in total institutions including children, while cutting government expenditures. A total institution is a place where people exist, cut off from mainstream society living together enclosed in a formally administered existence. The movement gained new force after the 1972 exposure of the horrible conditions under which children were warehoused in a now defunct institution called Willowbrook [8], as well as a flurry of court action which has continued over time exemplified by the Olmstead v. L.C. finding [9], which was itself an outgrowth of the Americans with Disabilities Act.

In the mid-1960s a literal flood of persons was released with no place to go. This was a class of people who were ill equipped for a number of reasons to fend for themselves, who quickly overran existing services, and created a severe shortage of personnel trained to work with them using more modern, humane, and effective interventions. The profession was created to fill that gap.

1.Herzberg, Judith T. (2015). Foundations in human services practice: A generalist perspective on individual, agency, and community (1st ed.). Boston: Pearson. ISBN 9780205858255. OCLC 881181908

2. Chenault, Joann; Burnford, Fran (1978). Human services professional education: Future directions. New York: McGraw-Hill. ISBN 9780070107328. OCLC 3650238.

3. Dumont, M (1970). "The changing face of professionalism". Social Policy. 1: 26–31.

4. Reiff, R. (1970). "Community psychology, community mental health and social needs: The need for a body of knowledge in community psychology". In Iscoe, Ira; Spielberger, Charles D. Community psychology: Perspectives in training and research. New York: Appleton. pp. 1–. ISBN 9780390477712. OCLC 92432.

5. Agranoff, R. (1974). "Human services administration: Service delivery, service integration, and training". In Mikulecky, Thomas J. Human services integration: a report of a special project conducted by the American Society for Public Administration. Washington, DC: American Society for Public Administration. pp. 42–51. OCLC 918115.

RESEARCH FOR HUMAN SERVICES pg. 4

6. Baker, F (June 1974). "From community mental health to human service ideology". American Journal of Public Health. 64 (6): 576–581. PMC 1775477 Freely accessible. PMID 4829069. doi:10.2105/ajph.64.6.576.

7. Michael J. Dear, Jennifer R. Wolch (1987) Landscapes of Despair: From Deinstitutionalization to Homelessness (Human Geography) Hardcover, Princeton University Press.

8. Rivera, Geraldo (1972). Willowbrook: A Report on How it is and Why it Doesn’t Have to Be That Way. New York: Random House. ISBN 0-394-71844-5.

9. Olmstead v. L.C., 527 U.S. 581 (1999): a United States Supreme Court case regarding discrimination against people with mental disabilities. The court ruled that under the Americans with Disabilities Act, persons with mental disabilities have a right to live in the community rather than in institutions if "the State's treatment professionals have determined that community placement is appropriate, the transfer from institutional care to a less restrictive setting is not opposed by the affected individual, and the placement can be reasonably accommodated, taking into account the resources available to the State and the needs of others with mental disabilities."

RESEARCH FOR HUMAN SERVICES pg. 5

Chapter 1: Science in the Social Sciences

Science is the attempt to make the chaotic diversity of our sense-experience correspond to a logically uniform system of thought.

— Albert Einstein

Truth is not created. It is discovered. Science is an organized attempt to discover truth.

First of all, which science are we talking about? There are three main categories.

1. Natural sciences: the study of natural phenomena (including cosmological, geological, chemical, and biological factors of the universe)

2. Formal sciences: the study of mathematics and logic, which use an a priori, as opposed to factual, methodology)

3. Social sciences: the study of human behavior and societies. From: https://en.wikipedia.org/wiki/Branches_of_science

Einstein was a theoretical physicist which seems to fall mostly into the formal sciences. He did not like things that were, or at least seemed, unpredictable. Einstein was very much bothered by chaos, and tried to find a way to predict, what might in the end, be unpredictable.

The social sciences are what we are interested in for this class, and in particular how that knowledge can be applied to help systems of all sizes (micro, mezzo, and macro). In other words, Human Services is looking to apply the methods and findings from social science to improve the lives of people as individuals, in groups such as families, and in the larger social context of communities. However, all sciences have much in common. For example, chaos theory is a branch of mathematics, which is itself a science, that deals with conditions where prediction is not possible. [1]

Chaotic diversity not only describes things on the quantum level (which is what Einstein was interested in), but also the human condition. Chaos theory with its origins in mathematics has implications when working with people that has only recently been realized [2] [3]. Like Einstein, we in the Human Services dislike chaos. We want to find ways of explaining, possibly even predicting, and preventing human misery.

So, what is Einstein’s message? Einstein is telling us that we have a rich and imaginative sensory capability that provides us with a vast way of experiencing and interpreting our world. However, from an evolutionary standpoint, it is built for survival in the heat of the moment, rather than the cool calculation of empirical analysis which is by design a longer process. Things like intuition and instinct are certainly useful. But, we

SYSTEMS THEORY – SYSTEMS ARE USUALLY MADE UP OF SMALLER COMPONENTS THAT “FIT” INSIDE THE LARGER ONES.

RESEARCH FOR HUMAN SERVICES pg. 6

can add science to our toolbox which is what this text is about. With the tool of science, we can explore our world, social and physical, in a way which allows us to test our assumptions and theories about the way things are. Science helps us to increase our rational understanding of the totality of the world we live in. We have had some success. Science can not only give us theories about how people act, it can also test our assumptions as we will see in the following example of social science at work.

Who talks more? Many people believe that women tend to talk more than men—with some even suggesting that this difference has a biological basis. One widely cited estimate was that women spoke 20,000 words per day on average and men only 7,000. This claim seemed plausible, but was it true? A group of psychologists led by Matias Mehl decided to find out. How did they check? They used a literature review, a powerful tool we will discuss in depth later on in the text. When they checked the literature to see if anyone had actually tried to count the daily number of words spoken by women and men, no one had. All they found were theories and assumptions, many of them culturally derived based on stereotypical beliefs about gender rather than any real science. The researchers conducted a study of their own in which a sample of 369 female and male college students wore audio recorders while they went about their day. The result? The women spoke an average of 16,215 words per day and the men spoke an average of 15,669 which was a difference of just under one percent. It was an extremely small difference that could easily be explained by chance. In an article in the journal Science, these researchers summed up their findings as follows: “We therefore conclude, on the basis of available empirical evidence, that the widespread and highly publicized stereotype about female talkativeness is unfounded” (Mehl, Vazire, Ramirez-Esparza, Slatcher, & Pennebaker, 2007, p. 82) [4]. Sometimes what we think is true, just isn’t so.

In this case, we had a working assumption about the way things are. However, when that theory was tested it turned out not to be true. What we believe about people, and how they behave, is fundamental to effectively working with our clients. This example shows how what we believe individually or collectively can sometimes be wrong. Sometimes what we believe is inaccurate or incomplete. We should keep in mind that as professionals, whenever possible, we should act according to the best information that we have at hand. This means we need to be informed as to what science has empirically indicated.

[1] Oestreicher C. (2007), Dialogues Clinical Neuroscience. 2007;9(3):279-89.

[2] Chamberlain L. (Editor), Butz, M. (Editor), (1998). Clinical Chaos: A Therapist's Guide To Non-Linear Dynamics And Therapeutic Change 1st Edition, Routeledge.

[3] Eenwyk, J, (2012), Clinical Chaos, Inner City Books.

[4] Mehl, M. R., Vazire, S., Ramirez-Esparza, N., Slatcher, R. B., & Pennebaker, J. W. (2007). Are women really more talkative than men? Science, 317, 82.

1.1 Understanding Science

LEARNING OBJECTIVES  Define science.

 Describe the three fundamental features of science.

 Define pseudoscience and give some examples.

Science & Human Services Human Services is not a science but a profession, and an evolving one at that. As such it is interdisciplinary, drawing not just on our profession’s research, but also that of our good colleagues in such diverse disciplines as social work, anthropology, sociology, education, criminal justice, education, and nursing to name just a few. Some of those disciplines are considered to be social sciences, and some like

RESEARCH FOR HUMAN SERVICES pg. 7

Human Services, are considered to be professions. However, we are all engaged in research to find the best possible methods for helping improve the human condition on practical terms. That is not to say that Human Service professionals do not engage in the research process, because we do. Human Services practice should be based on empirically based information. That means practice should be research based whenever possible.

Empirically Based Research Empirically based practice is considered best practice. Whenever possible, Human Service practitioners should strive to use only those techniques, methods, and approaches in the helping process that are empirically derived.

Empirically based practice means using quality research from peer reviewed sources in your professional practice. Professional journals, online databases, and continuing education, are all ways that we can keep informed as to what is happening in our field. It also means assessing the results of your practice, and doing so in an ethical way. In summary, empirically based research is the application of empirically derived best practices into your everyday practice. That is, we have a moral imperative to use science.

The Emergence of Science and the Scientific Method The Oxford Dictionaries Online defines the scientific method as "a method or procedure that has

characterized natural science since the 17th century, consisting in systematic observation, measurement,

and experiment, and the formulation, testing, and modification of hypotheses1" Since then, the method has

expanded from the natural sciences into the social sciences to including such disciplines as anthropology,

sociology, psychology, psychology and such professions as social work and Human Services.

Francis Bacon (1561-1626) is considered to be the first person to have conceptualized the scientific

method, but he did so under the influence of other great minds such as his contemporary Galileo (1564-

1642), and those who came before him such as Copernicus (1473-1543). But the list goes back even

further to the likes of Plato, Socrates, and Aristotle, and even further back as our ancestors struggled to

make this world (which is such an unpredictable and dangerous place where disasters still seem like

random happenstance) into a more tolerable and predictable place to dwell. This led to the ability to make

fire, create shelter, and of course finding something to eat. The rise of technologies was, in their own time,

no less miraculous to them then than time travel would be to us now. Otherwise we lived brutish and short

lives. Knowledge was gained slowly but accumulated over the years, handed down person-to-person

through an oral tradition. The invention of writing and then printing sped up the process as it became

possible to transmit knowledge across time and space. The scientific method was just one innovation but

we will focus on it now as it forms the foundation for all modern research.

We have a legal, moral, and ethical, responsibility to our profession, ourselves, society, and certainly our clients to do just that. What are the implications? While not all Human Service practitioners are engaged in research, all Human Service practitioners are consumers of research in their personal and professional lives. Research is based on a scientific approach to knowledge. What are the modern features of science?

Features of Science The modern scientific approach has three fundamental features (Stanovich, 2010):

1. Systematic Empiricism The first feature is systematic empiricism. Empiricism refers to learning based on observation. Scientists learn about the natural and social world systematically, by carefully planning, making, recording, and

1 "scientific method", Oxford Dictionaries: British and World English, 2017, retrieved 7 Nov 2017

RESEARCH FOR HUMAN SERVICES pg. 8

analyzing observations of it. A lot of what we learn and what we might know (that may or may not be true) is actually acquired haphazardly. As we will see, logical reasoning, and even creativity, play important roles in science, but scientists are unique in their insistence on checking ideas about the way the world seems to be, against more systematic observations. That is, using the practice of science rather than tradition, superstition, guessing, or any number of other ways we have of knowing. Notice, for example, that Mehl and his colleagues did not trust other people’s stereotypes or even their own informal observations. Instead, they systematically recorded, counted, and compared the number of words spoken by a large sample of women and men. Furthermore, when their systematic observations turned out to conflict with people’s stereotypes, they trusted their systematic observations.

2. The Scientific Approach Following logically from the first feature is the second feature which is the scientific approach. It is concerned with empirical questions. These are questions about the way the world actually is rather than what we think it might be, and are answered by systematic observation. The question of whether women talk more than men was a simple question that could be empirically answered but just counting. Either women really do talk more than men or they do not, and this can be determined by systematically observing how much women and men actually talk.

There are many interesting and extremely important questions that are not empirically testable and that science cannot answer. Among them are questions about values—whether things are good or bad, just or unjust, or beautiful or ugly, and how the world ought to be. Although the question of whether a stereotype is accurate or inaccurate is an empirically testable one that science can answer, the question of whether it is wrong for people to hold to inaccurate stereotypes is not. Similarly, the question of whether criminal behavior has a genetic component is an empirical question, but the question of what should be done with people who commit crimes is not. Questions about permitting the death penalty, abortion, or the legalization of marijuana are ultimately moral rather than scientific issues. It is especially important for researchers in Human Services to be mindful of this distinction.

3. Public Knowledge The third feature of science is that it creates public knowledge. After asking their empirical questions, making their systematic observations, and drawing their conclusions, scientists publish their work. This usually means writing an article for publication in a professional journal, in which they put their research question in the context of previous research, describe in detail the methods they used to answer their question, and clearly present their results and conclusions. These are the kind of articles you use for your

RESEARCH FOR HUMAN SERVICES pg. 9

research papers, and the kinds of articles that should be informing your practice with clients. Earlier we discussed how we progressed from an oral tradition, to the written word, and then to printing. We noted that the venerable art of printing allowed for knowledge to be transferable through time and space. It was more reliable, durable, and transportable. Knowledge was spread far and wide. We have now come to the digital age, maybe even past it, which has exponentially increased the growth of knowledge as well as the speed it is created and disseminated. While this is generally considered progress, we will soon discuss new problems which have arisen because of new communication technologies such as social media, and

massive online databases. One thing that has not changed is that knowledge dies if it is not passed on, and now that is through publication in whatever form be it, print or digital.

Publication is an essential feature of science for two reasons: 1. Science is a social process—a large scale collaboration among many researchers distributed across both time and space. Our current scientific knowledge of most topics is based on many different studies conducted by many different researchers who have shared their work with each other over the years. 2. Publication allows science to be self-correcting. Individual scientists understand that despite their best efforts, their methods can be flawed and their conclusions incorrect.

Publication allows others in the scientific community to detect and correct these errors so that, over time, scientific knowledge increasingly reflects the way the world actually is. If the findings of a research study cannot be reproduced elsewhere, then there is a problem with the original study. The ability of scientist to replicate what others do is how scientific findings are ultimately verified.

Science Versus Pseudoscience Pseudoscience refers to activities and beliefs that are claimed to be scientific by their proponents—and may appear to be scientific at first glance—but are not. Consider the theory of biorhythms (not to be confused with sleep cycles or other biological cycles that do have a scientific basis). The idea is that people’s physical, intellectual, and emotional abilities run in cycles that begin when they are born and continue until they die. The physical cycle has a period of 23 days, the intellectual cycle a period of 33 days, and the emotional cycle a period of 28 days. So, for example, if you had the option of when to schedule an exam, you would want to schedule it for a time when your intellectual cycle will be at a high point. The theory of biorhythms has been around for more than 100 years, and you can find numerous popular books and websites about biorhythms, often containing impressive and scientific-sounding terms like sinusoidal wave and bioelectricity. The problem with biorhythms, however, is that there is no good reason to think they exist (Hines, 1998).

A set of beliefs or activities can be said to be pseudoscientific if adherents claim or imply that it is scientific but it lacks one or more of the three features of science. It might lack systematic empiricism. Either there is no relevant scientific research or, as in the case of biorhythms, there is relevant scientific research but it is ignored. It might also lack public knowledge. People who promote the beliefs or activities might claim to have conducted scientific research but never publish that research in a way that allows others to evaluate it.

WOOD CUT OF AN EARLY PRINTING PRESS. 1

IN THE PUBLIC DOMAIN: HTTPS://COMMONS.WIKIMEDIA.ORG/WIKI/FILE:PRINTER_IN_1568- CE.PNG

RESEARCH FOR HUMAN SERVICES pg. 10

Karl Popper A set of beliefs and activities might also be pseudoscientific because it does not address empirical questions. The philosopher Karl Popper was especially concerned with this idea (Popper, 2002). He argued more specifically that any scientific claim must be expressed in such a way that there are observations that would—if they were made— count as evidence against the claim. In other words, scientific claims must be falsifiable. For example, if a researcher claimed that eating fresh fruit prevents pregnancy that claim is falsifiable, but also on the surface implausible. Some claims, like that one, are patently ridiculous and easily disproved. However, there are other things we believe which may seem plausible, which we have never bothered to question, until they are tested scientifically. The claim that women talk more than men is falsifiable because systematic observations could reveal either that they do talk more than men or that they do not. As an example of an unfalsifiable claim, consider that many people who study extrasensory perception (ESP) and other psychic powers claim that such powers can disappear when they are observed too closely. This makes it so that no possible observation would count as evidence against ESP. If a careful test of a self-proclaimed psychic showed that

she predicted the future at better-than-chance levels, this would be consistent with the claim that she had psychic powers. But if she failed to predict the future at better-than-chance levels, this would also be consistent with the claim because her powers can supposedly disappear when they are observed too closely. Why should we concern ourselves with pseudoscience? There are at least three reasons. One is that learning about pseudoscience helps bring the fundamental features of science—and their importance—into sharper focus. A second is that biorhythms, psychic powers, astrology, and many other pseudoscientific beliefs are widely held and are promoted on the Internet, on television, and in books and magazines. Learning what makes them pseudoscientific can help us to identify and evaluate such beliefs and practices when we encounter them. A third reason is that many pseudosciences purport to explain some aspect of human behavior and mental processes, including biorhythms, astrology, graphology (handwriting analysis), and magnet therapy for pain control. It is important for students of Human Services to distinguish their own field clearly from this “pseudo psychology.”

The Skeptic’s Dictionary An excellent source for information on pseudoscience is The Skeptic’s Dictionary (http://www.skepdic.com). Among the pseudoscientific beliefs and practices you can learn about are the following:

Cryptozoology. The study of “hidden” creatures like Bigfoot, the Loch Ness monster, and the chupacabra.

Pseudoscientific psychotherapies. Past-life regression, rebirthing therapy, and bioscream therapy, among others.

Homeopathy. The treatment of medical conditions using natural substances that have been diluted sometimes to the point of no longer being present.

KARL POPPER, PHILOSOPHER OF SCIENCE, PHOTO BY LSE LIBRARY - HTTP://WWW.FLICKR.COM/PHOTOS/LSELIBRARY/3833724834/I

N/SET-72157623156680255/, NO RESTRICTIONS,

RESEARCH FOR HUMAN SERVICES pg. 11

Pyramidology. Odd theories about the origin and function of the Egyptian pyramids (e.g., that they were built by extraterrestrials) and the idea that pyramids in general have healing and other special powers. T AK E AW AYS

 Science is a general way of understanding the natural world. Its three fundamental features are systematic empiricism, empirical questions, and public knowledge.

 Human Service disciplines, such as anthropology, sociology, and social work conduct research using the scientific approach to understanding human behavior.

 Pseudoscience refers to beliefs and activities that are claimed to be scientific but lack one or more of the three features of science. It is important to distinguish the scientific approach to understanding human behavior from the many pseudoscientific approaches.

Discussion: People sometimes suggest that the social sciences are not “real science” because either (a) human behavior cannot be predicted with perfect accuracy or (b) much of its subject matter (e.g., thoughts and feelings) cannot be observed directly. Do you agree or disagree with each of these ideas? Why?

Practice: List three empirical questions about human behavior. List three non-empirical questions about human behavior.

Discussion: Consider the following psychological claim. “People’s choice of spouse is strongly influenced by their perception of their own parents. Some choose a spouse who is similar in some way to one of their parents. Others choose a spouse who is different from one of their parents.” Is this claim falsifiable? Why or why not?

[1] Stanovich, K. E. (2010). How to think straight about psychology (9th ed.). Boston, MA: Allyn & Bacon.

[2] Hines, T. M. (1998). Comprehensive review of biorhythm theory. Psychological Reports, 83, 19–64.

[3] Popper, K. R. (2002). Conjectures and refutations: The growth of scientific knowledge. New York, NY: Routledge.

RESEARCH FOR HUMAN SERVICES pg. 12

1.2 Scientific Research in the Human Services

LEARNING OBJECTIVES  Describe a general model of scientific research and give specific examples that fit the model.

 Explain who conducts scientific research and why they do it.

 Distinguish between basic research and applied research.

A Model of Scientific Research The researcher (who, more often than not, is actually a small group of researchers) formulates a research question, conducts a study designed to answer the question, analyzes the resulting data, draws conclusions about the answer to the question, and publishes the results so that they become part of the research literature. Because the research literature is one of the primary sources of new research questions, this process can be thought of as a cycle. New research leads to new questions, which lead to new research, and so on. Figure 1.2 "A Simple Model of Scientific Research in Psychology" also indicates that research questions can originate outside of this cycle either with informal observations or with practical problems that need to be solved. But even in these cases, the researcher would start by checking the research literature to see if the question had already been answered and to refine it based on what previous research had already found.

FIGURE 2 THE CIRCLE OF THE SCIENTIFIC METHOD. INFORMAL OBSERVATION LEADS TO A RESEARCH QUESTION, THEN TO EMPIRICAL STUDY, THEN DATA ANALYSIS, THE CONCLUSION, INCLUSION IN THE SCIENTIFIC LITERATURE AND THEN THE CIRCLE STARTS ALL OVER AGAIN.

The research by Mehl and his colleagues is described nicely by this model. Their question—whether women are more talkative than men—was suggested to them both by people’s stereotypes and by published claims about the relative talkativeness of women and men. When they checked the research literature, however, they found that this question had not been adequately addressed in scientific studies. They conducted a careful empirical study, analyzed the results (finding very little difference between women and men), and published their work so that it became part of the research literature. The publication

RESEARCH FOR HUMAN SERVICES pg. 13

of their article is not the end of the story, however, because their work suggests many new questions (about the reliability of the result, about potential cultural differences, etc.) that will likely be taken up by them and by other researchers inspired by their work.

As another example, consider that as cell phones became more widespread during the 1990s, people began to wonder whether, and to what extent, cell phone use had a negative effect on driving. Many psychologists decided to tackle this question scientifically (Collet, Guillot, & Petit, 2010). [1] It was clear from previously published research that engaging in a simple verbal task impairs performance on a perceptual or motor task carried out at the same time, but no one had studied the effect specifically of cell phone use on driving. Under carefully controlled conditions, these researchers compared people’s driving performance while using a cell phone with their performance while not using a cell phone, both in the lab and on the road. They found that people’s ability to detect road hazards, reaction time, and control of the vehicle were all impaired by cell phone use. Each new study was published and became part of the growing research literature on this topic.

Who Conducts Scientific Research? Scientific research is generally conducted by people with doctoral degrees (usually the doctor of philosophy [PhD]) and master’s degrees in the social sciences, often supported by research assistants with bachelor’s degrees or other relevant training. Some of them work for government agencies (e.g., the National Institute of Mental Health), for nonprofit organizations (e.g., the American Cancer Society), or in the private sector (e.g., in product development). However, the majority of them are college and university faculty, who often collaborate with their graduate and undergraduate students. Doctoral-level researchers might be employed to conduct research full-time or, like many college and university faculty members, to conduct research in addition to teaching classes and serving their institution and community in other ways.

Of course, people also conduct research because they enjoy the intellectual and technical challenges involved and the satisfaction of contributing to scientific knowledge of human behavior. You might find that you enjoy the process too. If so, your college or university might offer opportunities to get involved in ongoing research as either a research assistant or a participant. Of course, you might find that you do not enjoy the process of conducting formal research. But at least you will have a better understanding of where scientific knowledge comes from, an appreciation of its strengths and limitations, and an awareness of how it can be applied to solve practical problems in for you as well as those you serve.

The Broader Purposes of Scientific Research in the Human Services People have always been curious about the natural world, including themselves and their behavior. It was also a matter of survival. Knowing the sessions, the rhythm of the weather, navigation by the celestial bodies, learning how to grow crops, or the migration patterns of animals were all key survival skills for our early ancestors. This led to the development of mathematics, astronomy, and the other sciences. Science grew out of necessity and natural curiosity and has become a way to achieve detailed and accurate knowledge. Keep in mind that most of the phenomena and theories that fill social science textbooks are the products of scientific research. In a typical introductory psychology textbook, for example, one can learn about specific cortical areas for language and perception, principles of classical and operant conditioning, biases in reasoning and judgment, and people’s surprising tendency to obey authority. And scientific research continues because what we know right now only scratches the surface of what we can know.

Two Categories: Basic and Applied Research Scientific research is often classified as being either basic or applied:

Basic Research

Basic research is conducted primarily for the sake of achieving a more detailed and accurate understanding, without necessarily trying to address any particular practical problem. The research of Mehl and his colleagues on gender differences in speech falls into this category.

RESEARCH FOR HUMAN SERVICES pg. 14

Applied Research

Applied research is conducted primarily to address some practical problem. Research on the effects of cell phone use on driving, for example, was prompted by safety concerns and has led to the enactment of laws to limit this practice. Although the distinction between basic and applied research is convenient, it is not always clear-cut. For example, basic research on sex differences in talkativeness could eventually have an effect on how marriage therapy is practiced, and applied research on the effect of cell phone use on driving could produce new insights into basic processes of perception, attention, and action. The cell phone research could also eventually be used to inform policy and legislation.

One thing to keep in mind is that while in the beginning there might not be any apparent use for the finding of basic research history, tells us that the probability of a purpose emerging later on is very high.

Summary  Research can be described by a simple cyclical model. A research question based on the research

literature leads to an empirical study, the results of which are published and become part of the research literature.

 Scientific research is conducted mainly by people with doctoral degrees most of whom are college and university faculty members. They do so for professional for personal reasons, as well as to contribute to scientific knowledge about human behavior. Basic research is conducted to learn about human behavior for its own sake, and applied research is conducted to solve some practical problem. Both are valuable, and the distinction between the two is not always clear-cut.

Practice:

Find a description of an empirical study in a professional journal or in a scientific blog from one of the social sciences. Then write a brief description of the research in terms of the cyclical model presented here. One or two sentences for each part of the cycle should suffice.

[1] Collet, C., Guillot, A., & Petit, C. (2010). Phoning while driving I: A review of epidemiological, psychological, behavioural and physiological studies. Ergonomics, 53, 589–601.

RESEARCH FOR HUMAN SERVICES pg. 15

1.3 Science and Common Sense

LEARNING OBJECTIVES  Explain the limitations of common sense when it comes to achieving a detailed and accurate

understanding of human behavior.

 Give several examples of common sense or folk psychology that are incorrect.

 Define skepticism and its role in scientific psychology.

Can We Rely on Common Sense? Some people wonder whether the scientific approach in Human Services is necessary. Can we not reach the same conclusions based on common sense or intuition? Certainly, we all have intuitive beliefs about people’s behavior, thoughts, and feelings—and these beliefs are collectively referred to as folk psychology. Although much of our folk psychology is probably reasonably accurate, it is clear that much of it is not. For example, most people believe that anger can be relieved by “letting it out”—perhaps by punching something or screaming loudly. Scientific research, however, has shown that this approach tends to leave people feeling more angry, not less (Bushman, 2002) [1] . Likewise, most people believe that no one would confess to a crime that he or she had not committed, unless perhaps that person was being physically tortured. But again, extensive empirical research has shown that false confessions are surprisingly common and occur for a variety of reasons (Kassin & Gudjonsson, 2004). [2]

Some Great Myths In 50 Great Myths of Popular Psychology, psychologist Scott Lilienfeld and colleagues discuss several widely held commonsense beliefs about human behavior that scientific research has shown to be incorrect (Lilienfeld, Lynn, Ruscio, & Beyerstein, 2010). [3] Here is a short list.

 “People use only 10% of their brain power.”

 “Most people experience a midlife crisis in their 40’s or 50’s.”

 “Students learn best when teaching styles are matched to their learning styles.”

 “Low self-esteem is a major cause of psychological problems.”

 “Psychiatric admissions and crimes increase during full moons.”

How Could We Be So Wrong? How can so many of our intuitive beliefs about human behavior be so wrong? Notice that this is a psychological question, and it just so happens that psychologists have conducted scientific research on it and identified many contributing factors (Gilovich, 1991). [4] One is that forming detailed and accurate beliefs requires powers of observation, memory, and analysis to an extent that we do not naturally possess. It would be nearly impossible to count the number of words spoken by the women and men we happen to encounter, estimate the number of words they spoke per day, average these numbers for both groups, and compare them—all in our heads. This is why we tend to rely on mental shortcuts in forming and maintaining our beliefs. For example, if a belief is widely shared—especially if it is endorsed by “experts”—and it makes intuitive sense, we tend to assume it is true. This is compounded by the fact that we then tend to focus on cases that confirm our intuitive beliefs and not on cases that disconfirm them.

This is called confirmation bias. Confirmation bias is searching for answers, or interpreting things, in a way that confirms your preexisting beliefs. For example, once we begin to believe that women are more talkative than men, we tend to notice and remember talkative women and silent men but ignore or forget silent women and talkative men. We also hold incorrect beliefs in part because it would be nice if they were true. For example, many people believe that calorie-reducing diets are an effective long-term treatment for obesity, yet a thorough review of the scientific evidence has shown that they are not (Mann et al., 2007). [5] People may continue to believe in the effectiveness of dieting in part because it gives them hope for losing weight if they are obese or makes them feel good about their own “self-control” if they are not.

Scientists understand that they are just as susceptible as anyone else to intuitive but incorrect beliefs. Therefore, they cultivate an attitude of skepticism. Being skeptical does not mean being cynical or

RESEARCH FOR HUMAN SERVICES pg. 16

distrustful, nor does it mean questioning every belief or claim one comes across (which would be impossible anyway). Instead, it means pausing to consider alternatives and to search for evidence— especially systematically collected empirical evidence—when there is enough at stake to justify doing so.

Let’s look at an example. Imagine that you read a magazine article that claims that giving children a weekly allowance is a good way to help them develop financial responsibility. This is an interesting and potentially important claim (especially if you have kids). However, you could take a skeptical attitude toward that claim. Perhaps receiving an allowance just encourages children to spend money on things they don’t need, and become more materialistic in the process? Taking an attitude of skepticism would also mean asking what evidence supports the original claim. Is the author a scientific researcher? Is any scientific evidence cited? If the issue was important enough, it might also mean turning to the research literature to see if anyone else had studied it.

Because there is often not enough evidence to fully evaluate a belief or claim, scientists also cultivate tolerance for uncertainty. They accept that there are many things that they simply do not know. For example, it turns out that there is no scientific evidence that receiving an allowance causes children to be more financially responsible, nor is there any scientific evidence that it causes them to be materialistic. Although this kind of uncertainty can be problematic from a practical perspective—for example, making it difficult to decide what to do when our children ask for an allowance—it is exciting from a scientific perspective. If we do not know the answer to an interesting and empirically testable question, science may be able to provide the answer.

Summary

 People’s intuitions about human behavior, also known as folk psychology which we discussed earlier, often turn out to be wrong. This is one primary reason that psychology relies on science rather than common sense.

 Researchers in the social sciences need to cultivate a critical-thinking attitude. A healthy bit of skepticism. That includes searching for evidence and considering alternatives before accepting a claim about human behavior as true. This should be balanced with some tolerance for uncertainty, and withholding judgment about whether a claim is true or not when there is insufficient evidence to decide.

Practice: For each of the following intuitive beliefs about human behavior, list three reasons that it might be true and three reasons that it might not be true:

 You cannot truly love another person unless you love yourself.

 People who receive “crisis counseling” immediately after experiencing a traumatic event are better able to cope with that trauma in the long term.

 Studying is most effective when it is always done in the same location.

[1] Bushman, B. J. (2002). Does venting anger feed or extinguish the flame? Catharsis, rumination, distraction, anger, and aggressive responding. Personality and Social Psychology Bulletin, 28, 724–731.

[2] Kassin, S. M., & Gudjonsson, G. H. (2004). The psychology of confession evidence: A review of the literature and issues. Psychological Science in the Public Interest, 5, 33–67.

[3] Lilienfeld, S. O., Lynn, S. J., Ruscio, J., & Beyerstein, B. L. (2010). 50 great myths of popular psychology. Malden, MA: Wiley-Blackwell.

[4] Gilovich, T. (1991). How we know what isn’t so: The fallibility of human reason in everyday life. New York, NY: Free Press.

RESEARCH FOR HUMAN SERVICES pg. 17

[5] Mann, T., Tomiyama, A. J., Westling, E., Lew, A., Samuels, B., & Chatman, J. (2007). Medicare’s search for effective obesity treatments: Diets are not the answer. American Psychologist, 62, 220–233.

RESEARCH FOR HUMAN SERVICES pg. 18

1.4 Science and Human Services Practice

LEARNING OBJECTIVES  Explain how science is relevant to clinical practice.

 Define the concept of an empirically supported treatment and give some examples.

A common, and widely known, application of science to Human Services is clinical practice for the diagnosis and treatment of psychological disorders and related problems. Let us use the term clinical practice broadly to refer to the activities of clinical and counseling psychologists, school psychologists, marriage and family therapists, licensed clinical social workers, case managers, and others within the broad spectrum of Human Services who work with people individually or in small groups, families, or communities with helping clients meet their needs. It is important to consider the relationship between scientific research and clinical practice because many students are especially interested in clinical practice, perhaps even as a career.

The main point is that difficulty in functioning occurs at many levels for many different reasons and is a natural part of the human condition. This means that questions about their nature, causes, and consequences are often empirically testable and therefore subject to scientific study. As with other questions about human behavior, we cannot rely on our intuition or common sense for detailed and accurate answers. Consider, for example, that dozens of popular books and thousands of websites claim that adult children of alcoholics have a distinct personality profile, including low self-esteem, feelings of powerlessness, and difficulties with intimacy. Although this sounds plausible, scientific research has demonstrated that adult children of alcoholics are no more likely to have these problems than anybody else (Lilienfeld et al., 2010). [2] Similarly, questions about whether a particular psychotherapy works are empirically testable questions that can be answered by scientific research. If a new psychotherapy is an effective treatment for depression, then systematic observation should reveal that depressed people who receive this psychotherapy improve more than a similar group of depressed people who do not receive this psychotherapy (or who receive some alternative treatment). Treatments that have been shown to work in this way are called empirically supported treatments.

Empirically Supported Treatments An empirically supported treatment is one that has been studied scientifically and shown to result in greater improvement than no treatment, a placebo, or some alternative treatment. These include many forms of psychotherapy, which can be as effective as standard drug therapies. Among the forms of psychotherapy with strong empirical support are the following:

Cognitive behavioral therapy. For depression, panic disorder, bulimia nervosa, and posttraumatic stress disorder.

Exposure therapy. For posttraumatic stress disorder.

Behavioral therapy. For depression.

Behavioral couples therapy. For alcoholism and substance abuse.

Exposure therapy with response prevention. For obsessive-compulsive disorder.

Family therapy. For schizophrenia.

For a more complete list, see the following website, which is maintained by Division 12 of the American Psychological Association, the Society for Clinical Psychology: http://www.psychology.sunysb.edu/eklonsky-/division12.

There is some acknowledgment in the helping professions that we have not historically paid enough attention to scientific research—for example, by failing to use empirically supported treatments.

RESEARCH FOR HUMAN SERVICES pg. 19

Fortunately, that situation is vastly improved. New research in genetics, epigenetics, neuroscience, psychopharmacology, and the traditional social sciences has made great strides in advancing our knowledge of how to help people. Scientific research is relevant to Human Services practice because it provides detailed and accurate knowledge which helps us, help our clients.

Discussion: Some practitioners argue that what they do is an “art form” based on intuition and personal experience and therefore cannot be evaluated scientifically. Write a paragraph about how satisfied you would be with such a worker from each of three perspectives:

 a potential client of the clinician

 a judge who must decide whether to allow the clinician to testify as an expert witness in a child abuse case

 an insurance company representative who must decide whether to reimburse the clinician for his or her services

Practice:

Create a short list of questions that a client could ask a clinician to determine whether he or she pays sufficient attention to scientific research.

[1] American Psychological Association. (2011). About APA. Retrieved fromhttp://www.apa.org/about.

[2] Lilienfeld, S. O., Lynn, S. J., Ruscio, J., & Beyerstein, B. L. (2010). 50 great myths of popular psychology. Malden, MA: Wiley-Blackwell.

[3] Norcross, J. C., Beutler, L. E., & Levant, R. F. (Eds.). (2005). Evidence-based practices in mental health: Debate and dialogue on the fundamental questions. Washington, DC: American Psychological Association.

RESEARCH FOR HUMAN SERVICES pg. 20

Chapter 2: Getting Started in Research If we knew what it was we were doing, it would not be called research, would it? -Albert Einstein

If we knew what it was we were doing, it would not be called research, would it? Albert Einstein

Below is the first paragraph of a 2009 article in the Journal of Experimental Psychology: Applied.

Human figure drawings (HFDs) are commonly used by professionals who interview children about suspected sexual abuse. It is assumed that these drawings will decrease children’s linguistic and emotional or motivational limitations, as well as memory problems, and thus will result in the elicitation of more complete and accurate details of abuse. There is, however, little scientific information to support claims of their benefits. This article presents the results of two studies that examined young children’s ability to use HFDs to report body touches. (Bruck, 2009, p. 361) [1]

In this paragraph, the researcher has identified a research question—about the effect of using human figure drawings on the accuracy of children’s memories of being touched—and has begun to make an argument for why it is interesting. In terms of the general model of scientific research in psychology presented in Figure 1.2 "A Simple Model of Scientific Research in Psychology", these are activities at the “top” of the cycle. In this chapter, we focus on these activities—finding research ideas, turning them into interesting empirical research questions, and reviewing the research literature. We begin, however, with some more basic concepts that are necessary to understand how research questions in psychology are conceptualized.

[1] Bruck, M. (2009). Human figure drawings and children’s recall of touching. Journal of Experimental Psychology: Applied, 15, 361–374.

2.1 Basic Concepts

LEARNING OBJECTIVES Define the concept of a variable, distinguish quantitative from categorical variables, and give examples of variables that might be of interest to psychologists.

 Explain the difference between a population and a sample.

 Describe two basic forms of statistical relationship and give examples of each.

 Interpret basic statistics and graphs used to describe statistical relationships.

 Explain why correlation does not imply causation.

Before we address where research questions come from—and what makes them more or less interesting—it is important to understand the kinds of questions that researchers in Human Services typically ask. This requires a quick introduction to several basic concepts, many of which we will return to in more detail later in the book.

Variables Research questions are about variables. A variable is a quantity or quality that varies across people or situations. For example, the height of the students in a psychology class is a variable because it varies from student to student. The sex of the students is also a variable as long as there are both male and female students in the class. Other examples of variables are weight, income, and grade point average.

RESEARCH FOR HUMAN SERVICES pg. 21

Types of Variables

A quantitative variable is a quantity, such as height, that is typically measured by assigning a number to each individual. Other examples of quantitative variables include people’s level of talkativeness, how depressed they are, and the number of siblings they have. A categorical variable, or nominal variable is a quality, such as sex, and is typically measured by assigning a category label to each individual. Other examples include people’s nationality, their occupation, and whether they are receiving psychotherapy.

Sampling and Measurement

Population versus Sample

Researchers are usually interested in drawing conclusions about some very large group of people. This is called the population. It could be American teenagers, children with autism, professional athletes, or even just human beings—depending on the interests and goals of the researcher. But they usually study only a small subset or sample of the population. We like to be able to generalize from a sample to the general population, but that can be challenging as we will see later. For example, a researcher might measure the talkativeness of a few hundred college students with the intention of drawing conclusions about the talkativeness of men and women in general. It is important, therefore, for researchers to use a representative sample—one that is similar to the population in important respects.

Simple Random Sampling

One method of obtaining a sample is simple random sampling, in which every member of the population has an equal chance of being selected for the sample. For example, a pollster could start with a list of all the registered voters in a city (the population), randomly select 100 of them from the list (the sample), and ask those 100 whom they intended to vote for. Unfortunately, random sampling is difficult or impossible in most social research because the populations are less clearly defined than the registered voters in a city. How could a researcher give all American teenagers or all children with autism an equal chance of being selected for a sample?

Convenience Sampling

The most common alternative to random sampling is convenience sampling, in which the sample consists of individuals who happen to be nearby and willing to participate (such as college students). The obvious problem with convenience sampling is that the sample might not be representative of the population.

Operational Definition

Once the sample is selected, researchers need to measure the variables they are interested in. This requires an operational definition—a definition of the variable in terms of precisely how it is to be measured. Most variables can be operationally defined in many different ways. For example, depression can be operationally defined as people’s scores on a paper-and-pencil depression scale, the number of depressive symptoms they are experiencing, or whether they have been diagnosed with major depressive disorder.

Data

When a variable has been measured for a particular individual, the result is called a score, and a set of scores is called data. Note that data is plural—the singular datum is rarely used—so it is grammatically correct to say, “Those are interesting data” (and incorrect to say, “That is interesting data”).

Statistical Relationships Between Variables

Single Variable Some research questions are about one variable. For example, how accurate are children’s memories for being touched, or is it possible for an interviewer to contaminate, or bias, their memories? How talkative are American college students? How common is it for people to be diagnosed with major depressive disorder? Answering such questions requires operationally defining the variable, measuring it for a sample, analyzing

RESEARCH FOR HUMAN SERVICES pg. 22

the results, and drawing conclusions about the population. This can be difficult, because some variables, such as depression and anxiety, are sometimes surprisingly difficult to differentiate. How do we proceed?

For a quantitative variable, this would typically involve computing the mean and standard deviation of scores on an instrument designed to measure a particular thing such as self-esteem, depression, anxiety, etc. For example, the Rosenberg Anxiety scale, which is in the index, is a commonly used measure of anxiety. It is in the appendix, and in the public domain free for us to use. We might operationalize self- esteem, by defining it according to scores on the Rosenberg Anxiety Scale. For depression, we might operationalize it using the, Beck Depression Inventory-II (BDI-II), the Patient Health Questionnaire (PHQ-9), or any combination of the many scales that have been devised that purport to measure what we generally call depression.

Statistical Relationship Between Multiple Variables However, research questions in the social sciences are more likely to be about statistical relationships between variables.

There is a statistical relationship between two variables when the average score on one differs systematically across the levels of the other. Studying statistical relationships is important because instead of telling us about behaviors and psychological characteristics in isolation, it tells us about the causes, consequences, development, and organization of those behaviors and characteristics. One example would be looking at academic achievement in relation to income

There are two basic forms of statistical relationship: differences between groups and correlations between quantitative variables. Although both are consistent with the general definition of a statistical relationship— the average score on one variable differs across levels of the other—they are usually described and analyzed somewhat differently. For this reason, it is important to distinguish them clearly.

Differences Between Groups One basic form of statistical relationship is a difference between the mean scores of two groups on some variable of interest. A wide variety of research questions take this form. Are women more talkative than men? Do children using human figure drawings recall more touch information than children not using human figure drawings? Do people talking on a cell phone have poorer driving abilities than people not talking on a cell phone? Do people receiving Psychotherapy A tend to have fewer depressive symptoms than people receiving Psychotherapy B? Later, we will also see that such relationships can involve more than two groups and that the groups can consist of the very same individuals tested at different times or under different conditions. For now, however, it is easiest to think in terms of two distinct groups.

Differences between groups are usually described by giving the mean score and standard deviation for each group. This information can also be presented in a bar graph like that in Figure 2.2 "Bar Graph Showing the Very Small Difference in the Mean Number of Words Spoken per Day by Women and Men in a Large Sample", where the heights of the bars represent the group means.

Figure 2.2 Bar Graph Showing the Very Small Difference in the Mean Number of Words Spoken per Day by Women and Men in a Large Sample

RESEARCH FOR HUMAN SERVICES pg. 23

Based on data from “Are Women Really More Talkative Than Men?” by M. R. Mehl, S. Vazire, N. Ramirez- Esparza, R. B. Slatcher, and J. W. Pennebaker, 2007, Science, 317, p. 82.

Correlations Between Quantitative Variables A second basic form of statistical relationship is a correlation between two quantitative variables, where the average score on one variable differs systematically across the levels of the other. Again, a wide variety of research questions in take this form. Is being a happier person associated with being more talkative? Do children’s memories for touch information improve as they get older? Does the effectiveness of psychotherapy depend on how much the patient likes the therapist?

Scatterplots Correlations between quantitative variables are often presented using scatterplots. Figure 2.3 "Scatterplot Showing a Hypothetical Positive Relationship Between Stress and Number of Physical Symptoms" shows some hypothetical data on the relationship between the amount of stress people are under and the number of physical symptoms they have. Each point in the scatterplot represents one person’s score on both variables. For example, the circled point in Figure 2.3 "Scatterplot Showing a Hypothetical Positive Relationship Between Stress and Number of Physical Symptoms" represents a person whose stress score was 10 and who had three physical symptoms. Taking all the points into account, one can see that people under more stress tend to have more physical symptoms.

Positive and Negative Relationships

This is a good example of a positive relationship, in which higher scores on one variable tend to be associated with higher scores on the other. A negative relationship is one in which higher scores on one variable tend to be associated with lower scores on the other. There is a negative relationship between stress and immune system functioning, for example, because higher stress is associated with lower immune system functioning.

Figure 2.3 Scatterplot Showing a Hypothetical Positive Relationship Between Stress and Number of Physical Symptoms

RESEARCH FOR HUMAN SERVICES pg. 24

The circled point represents a person whose stress score was 10 and who had three physical symptoms. Pearson’s r for these data is +.51.

A scatter plot is just a graph between two variables. Figure 2.3 shows the relationship between stress and physical symptoms for twenty-one samples. As you can see, there is a general tendency for stress to be correlated to physical symptoms. The more physical symptoms there are, the more stress. The more stress there is, the more physical symptoms. While a correlation exists, we need to remember that establishing correlation is interesting, and even useful, but does not necessarily posit causation.

Pearson’s r The strength of a correlation between quantitative variables is typically measured using a statistic called Pearson’s r. Sometimes known as correlation coefficient, and sometimes called just r or R. As Figure 2.4 "Range of Pearson’s " shows, Pearson’s r ranges from −1.00 (the strongest possible negative relationship) to +1.00 (the strongest possible positive relationship). A value of 0 means there is no relationship between the two variables. When Pearson’s r is 0, the points on a scatterplot form a shapeless “cloud.” As its value moves toward −1.00 or +1.00, the points come closer and closer to falling on a single straight line.

Figure 2.4 Range of Pearson’s r, From −1.00 (Strongest Possible Negative Relationship), Through 0 (No Relationship), to +1.00 (Strongest Possible Positive Relationship)

Pearson’s r is a good measure only for linear relationships, in which the points are best approximated by a straight line such as in Fig 2.3. It is not a good measure for nonlinear relationships, in which the points are

RESEARCH FOR HUMAN SERVICES pg. 25

better approximated by a curved line. Figure 2.5, for example, shows a hypothetical relationship between the amount of sleep people get per night and their level of depression. In this example, the line that best approximates the points is a curve—a kind of upside-down “U”—because people who get about eight hours of sleep tend to be the least depressed. Those who get too little sleep and those who get too much sleep tend to be more depressed. Nonlinear relationships are fairly common in social science research, but measuring their strength is beyond the scope of this book.

FIGURE 2.5 HYPOTHETICAL NONLINEAR RELATIONSHIP BETWEEN SLEEP AND DEPRESSION. THIS FIGURE DEPICTS HOW TOO LITTLE, AND TOO MUCH SLEEP, IS CORRELATED TO DEPRESSION.

Independent and Dependent Variables Researchers are often interested in a statistical relationship between two variables because they think that one of the variables causes the other. That is, the statistical relationship reflects a causal relationship.

In these situations, the variable that is thought to be the cause is called the independent variable (often referred to as X for short), and the variable that is thought to be the effect is called the dependent variable (often referred to as Y). For example, the statistical relationship between whether or not a depressed person receives psychotherapy and the number of depressive symptoms he or she has, reflects the fact that the psychotherapy (the independent variable) causes the reduction in symptoms (the dependent variable).

Causal Relationships

Understanding causal relationships is important in part because it allows us to change people’s behavior in predictable ways. If we know that psychotherapy causes a reduction in depressive symptoms—and we want people to have fewer depressive symptoms—then we can use psychotherapy to achieve this goal.

One easy way to remember the difference between independent and dependent variable is that the independent variable is the intervention--what we do to something else. The effect of what we do is the dependent variable.

Correlation and Causation But as pointed out earlier, not all statistical relationships reflect causal relationships. This is what psychologists mean when they say, “Correlation does not imply causation.” An obvious example comes from a study in Taiwan showing a positive relationship between the number of electrical appliances that people use and the extent to which they use birth control (Stanovich, 2010). [1] It seems clear, however,

RESEARCH FOR HUMAN SERVICES pg. 26

that this does not mean that owning electrical appliances causes people to use birth control, and it would not make sense to try to increase the use of birth control by giving people toasters and hair dryers.

Directionality Problem There are two reasons that correlation does not imply causation. The first is called the directionality problem. Two variables, X and Y, can be statistically related because X causes Y or because Y causes X. Consider, for example, a study showing that whether or not people exercise is statistically related to how happy they are—such that people who exercise are happier on average than people who do not. This statistical relationship is consistent with the idea that exercising causes happiness, but it is also consistent with the idea that happiness causes exercise. Perhaps being happy gives people more energy or leads them to seek opportunities to socialize with others by going to the gym. Which variable is the prime mover?

Third-Variable Problem The second reason that correlation does not imply causation is called the third-variable problem. Two variables, X and Y, can be statistically related not because X causes Y, or because Y causes X, but because some third variable, Z, causes both X and Y. For example, the fact that people with more electrical appliances are more likely to use birth control probably reflects the fact that having more education or income causes people to own more appliances and causes them to use birth control. Similarly, the statistical relationship between exercise and happiness could mean that some third variable, such as physical health, causes both of the others. Being physically healthy could cause people to exercise and cause them to be happier.

“Lots of Candy Could Lead to Violence” Although researchers know that correlation does not imply causation, many journalists do not. One website about correlation and causation, http://jonathan.mueller.faculty.noctrl.edu/100/correlation_or_causation.htm, links to dozens of media reports about real biomedical and psychological research. Many of the headlines suggest that a causal relationship has been demonstrated, when a careful reading of the articles shows that it has not because of the directionality and third-variable problems.

One article is about a study showing that children who ate candy every day were more likely than other children to be arrested for a violent offense later in life. But could candy really “lead to” violence, as the headline suggests? What alternative explanations can you think of for this statistical relationship? How could the headline be rewritten so that it is not misleading?

Experiment As we will see later in the book, there are various ways that researchers address the directionality and third-variable problems. The most effective, however, is to conduct an experiment. An experiment is a study in which the researcher manipulates the independent variable. For example, instead of simply measuring how much people exercise, a researcher could bring people into a laboratory and randomly assign half of them to run on a treadmill for 15 minutes and the rest to sit on a couch for 15 minutes. Although this seems like a minor addition to the research design, it is extremely important. Now if the exercisers end up in more positive moods than those who did not exercise, it cannot be because their moods affected how much they exercised (because it was the researcher who determined how much they exercised). Likewise, it cannot be because some third variable (e.g., physical health) affected both how much they exercised and what mood they were in (because, again, it was the researcher who determined how much they exercised). Thus, experiments eliminate the directionality and third-variable problems and allow researchers to draw firm conclusions about causal relationships.

We will have much more to say about experimental and nonexperimental research later in the book.

Summary

 Research questions are about variables and relationships between variables.

RESEARCH FOR HUMAN SERVICES pg. 27

 Two basic forms of statistical relationship are differences between group means and correlations between quantitative variables, each of which can be described using a few simple statistical techniques.

 Correlation does not imply causation. A statistical relationship between two variables, X and Y, does not necessarily mean that X causes Y. It is also possible that Y causes X, or that a third variable, Z, causes both X and Y.

Practice

1. List 10 variables that might be of interest to a researcher. For each, specify whether it is quantitative or categorical.

2. Imagine that you categorize people as either introverts (quieter, shyer, more inward looking) or extroverts (louder, more outgoing, more outward looking). Sketch a bar graph showing a hypothetical statistical relationship between this variable and the number of words people speak per day.

3. Now imagine that you measure people’s levels of extroversion as a quantitative variable, with values ranging from 0 (extreme introversion) to 30 (extreme extroversion). Sketch a scatterplot showing a hypothetical statistical relationship between this variable and the number of words people speak per day.

4. For each of the following statistical relationships, decide whether the directionality problem is present and think of at least one plausible third variable:

 People who eat more lobster tend to live longer.

 People who exercise more tend to weigh less.

 College students who drink more alcohol tend to have poorer grades.

[1] Stanovich, K. E. (2010). How to think straight about psychology (9th ed.). Boston, MA: Allyn & Bacon.

2.2 Generating Good Research Questions

LEARNING OBJECTIVES  Describe some common sources of research ideas and generate research ideas using those

sources.

 Describe some techniques for turning research ideas into empirical research questions and use those techniques to generate questions.

 Explain what makes a research question interesting and evaluate research questions in terms of their interestingness.

Good research must begin with a good research question. Yet, coming up with good research questions is something that novice researchers often find difficult and stressful. One reason is that this is a creative process that can appear mysterious—even magical—with experienced researchers seeming to pull interesting research questions out of thin air. However, psychological research on creativity has shown that it is neither as mysterious nor as magical as it appears. It is largely the product of ordinary thinking strategies and persistence (Weisberg, 1993). [1] This section covers some fairly simple strategies for finding general research ideas, turning those ideas into empirically testable research questions, and finally evaluating those questions in terms of how interesting they are and how feasible they would be to answer.

RESEARCH FOR HUMAN SERVICES pg. 28

Finding Inspiration Research questions often begin as more general research ideas—usually focusing on some behavioral, psychological, or social characteristic: talkativeness, memory for touches, depression, bungee jumping, and so on. Before looking at how to turn such ideas into empirically testable research questions, it is worth looking at where such ideas come from in the first place. Three of the most common sources of inspiration are informal observations, practical problems, and previous research.

Informal Observations Informal observations include direct observations of our own and others’ behavior as well as secondhand observations from nonscientific sources such as newspapers, books, and so on. For example, you might notice that you always seem to be in the slowest moving line at the grocery store. Could it be that most people think the same thing? Or you might read in the local newspaper about people donating money and food to a local family whose house has burned down and begin to wonder about who makes such donations and why. Some of the most famous research in psychology has been inspired by informal observations. Stanley Milgram’s famous research on obedience, for example, was inspired in part by journalistic reports of the trials of accused Nazi war criminals—many of whom claimed that they were only obeying orders. This led him to wonder about the extent to which ordinary people will commit immoral acts simply because they are ordered to do so by an authority figure (Milgram, 1963). [2]

Practical Problems Practical problems can also inspire research ideas, leading directly to applied research in such domains as law, health, education, and sports. Can human figure drawings help children remember details about being physically or sexually abused? How effective is psychotherapy for depression compared to drug therapy? To what extent do cell phones impair people’s driving ability? How can we teach children to read more efficiently? What is the best mental preparation for running a marathon?

Previous Research Probably the most common inspiration for new research ideas, however, is previous research. Recall that science is a kind of large-scale collaboration in which many different researchers read and evaluate each other’s work and conduct new studies to build on it. Of course, experienced researchers are familiar with previous research in their area of expertise and probably have a long list of ideas. This suggests that novice researchers can find inspiration by consulting with a more experienced researcher (e.g., students can consult a faculty member). But they can also find inspiration by picking up a copy of almost any professional journal and reading the titles and abstracts. In one typical issue of Psychological Science, for example, you can find articles on the perception of shapes, anti-Semitism, police lineups, the meaning of death, second-language learning, people who seek negative emotional experiences, and many other topics. If you can narrow your interests down to a particular topic (e.g., memory) or domain (e.g., health care), you can also look through more specific journals, such as Memory & Cognition or Health Psychology.

Generating Empirically Testable Research Questions Once you have a research idea, you need to use it to generate one or more empirically testable research questions, that is, questions expressed in terms of a single variable or relationship between variables. One way to do this is to look closely at the discussion section in a recent research article on the topic. This is the last major section of the article, in which the researchers summarize their results, interpret them in the context of past research, and suggest directions for future research. These suggestions often take the form of specific research questions, which you can then try to answer with additional research. This can be a good strategy because it is likely that the suggested questions have already been identified as interesting and important by experienced researchers.

But you may also want to generate your own research questions. How can you do this? First, if you have a particular behavior or psychological characteristic in mind, you can simply conceptualize it as a variable and ask how frequent or intense it is. How many words on average do people speak per day? How accurate are children’s memories of being touched? What percentage of people have sought professional

RESEARCH FOR HUMAN SERVICES pg. 29

help for depression? If the question has never been studied scientifically—which is something that you will learn in your literature review—then it might be interesting and worth pursuing.

If scientific research has already answered the question of how frequent or intense the behavior or characteristic is, then you should consider turning it into a question about a statistical relationship between that behavior or characteristic and some other variable. One way to do this is to ask yourself the following series of more general questions and write down all the answers you can think of.

 What are some possible causes of the behavior or characteristic?

 What are some possible effects of the behavior or characteristic?

 What types of people might exhibit more or less of the behavior or characteristic?

 What types of situations might elicit more or less of the behavior or characteristic?

In general, each answer you write down can be conceptualized as a second variable, suggesting a question about a statistical relationship. If you were interested in talkativeness, for example, it might occur to you that a possible cause of this psychological characteristic is family size. Is there a statistical relationship between family size and talkativeness? Or it might occur to you that people seem to be more talkative in same-sex groups than mixed-sex groups. Is there a difference in the average level of talkativeness of people in same-sex groups and people in mixed-sex groups? This approach should allow you to generate many different empirically testable questions about almost any behavior or psychological characteristic.

If through this process you generate a question that has never been studied scientifically—which again is something that you will learn in your literature review—then it might be interesting and worth pursuing. But what if you find that it has been studied scientifically? Although novice researchers often want to give up and move on to a new question at this point, this is not necessarily a good strategy. For one thing, the fact that the question has been studied scientifically and the research published suggests that it is of interest to the scientific community. For another, the question can almost certainly be refined so that its answer will still contribute something new to the research literature. Again, asking yourself a series of more general questions about the statistical relationship is a good strategy.

 Are there other ways to operationally define the variables?

 Are there types of people for whom the statistical relationship might be stronger or weaker?

 Are there situations in which the statistical relationship might be stronger or weaker—including situations with practical importance?

For example, research has shown that women and men speak about the same number of words per day— but this was when talkativeness was measured in terms of the number of words spoken per day among college students in the United States and Mexico. We can still ask whether other ways of measuring talkativeness—perhaps the number of different people spoken to each day—produce the same result. Or we can ask whether studying elderly people or people from other cultures produces the same result. Again, this approach should help you generate many different research questions about almost any statistical relationship.

Evaluating Research Questions Researchers usually generate many more research questions than they ever attempt to answer. This means they must have some way of evaluating the research questions they generate so that they can choose which ones to pursue. In this section, we consider two criteria for evaluating research questions: the interestingness of the question and the feasibility of answering it.

Interestingness How often do people tie their shoes? Do people feel pain when you punch them in the jaw? Are women more likely to wear makeup than men? Do people prefer vanilla or chocolate ice cream? Although it would be a fairly simple matter to design a study and collect data to answer these questions, you probably would not want to because they might not be interesting. We are not talking here about whether a research

RESEARCH FOR HUMAN SERVICES pg. 30

question is interesting to us personally, or perhaps for marketing purposes, but whether it is interesting to people more generally and, especially, to the scientific community. But what makes a research question interesting in this sense? Here we look at three factors that affect the interestingness of a research question: the answer is in doubt, the answer fills a gap in the research literature, and the answer has important practical implications.

First, a research question is interesting to the extent that its answer is in doubt. Obviously, questions that have been answered by scientific research are no longer interesting as the subject of new empirical research. But the fact that a question has not been answered by scientific research does not necessarily make it interesting. There must be some reasonable chance that the answer to the question will be something that we did not already know. But how can you assess this before actually collecting data? One approach is to try to think of reasons to expect different answers to the question—especially ones that seem to conflict with common sense. If you can think of reasons to expect at least two different answers, then the question might be interesting. If you can think of reasons to expect only one answer, then it probably is not. The question of whether women are more talkative than men is interesting because there are reasons to expect both answers. The existence of the stereotype itself suggests the answer could be yes, but the fact that women’s and men’s verbal abilities are fairly similar suggests the answer could be no. The question of whether people feel pain when you punch them in the jaw is not interesting because there is absolutely no reason to think that the answer could be anything other than a resounding yes.

Does it fill a gap? A second important factor to consider when deciding if a research question is interesting is whether answering it will fill a gap in the research literature. Again, this means in part that the question has not already been answered by scientific research. But, it also means that the question is in some sense a natural one for people who are familiar with the research literature. For example, the question of whether human figure drawings can help children recall touch information would be likely to occur to anyone who was familiar with research on the unreliability of eyewitness memory (especially in children) and the ineffectiveness of some alternative interviewing techniques.

A final factor to consider when deciding whether a research question is interesting is whether its answer has important practical implications. Again, the question of whether human figure drawings help children recall information about being touched has important implications for how children are interviewed in physical and sexual abuse cases. The question of whether cell phone use impairs driving is interesting because it is relevant to the personal safety of everyone who travels by car and to the debate over whether cell phone use should be restricted by law.

Feasibility A third important criterion for evaluating research questions is the feasibility of successfully answering them. There are many factors that affect feasibility, including time, money, equipment and materials, technical knowledge and skill, and access to research participants. Clearly, researchers need to take these factors into account so that they do not waste time and effort pursuing research that they cannot complete successfully. Unfortunately, many research questions that would have a major impact on the wellbeing of humanity go begging for a lack of resources.

Types of Studies in the Literature Looking through a sample of professional journals, especially in such academically oriented fields such as psychology, will reveal many studies that are complicated and difficult to carry out. These include longitudinal designs in which participants are tracked over many years, neuroimaging studies in which participants’ brain activity is measured while they carry out various mental tasks, and complex nonexperimental studies involving several variables and complicated statistical analyses. Keep in mind, though, that such research tends to be carried out by teams of highly trained researchers whose work is often supported in part by government and private grants. Keep in mind also that research does not have to be complicated or difficult to produce interesting and important results. Looking through a sample of professional journals will also reveal studies that are relatively simple and easy to carry out—perhaps involving a convenience sample of college students and a paper-and-pencil task.

RESEARCH FOR HUMAN SERVICES pg. 31

A final point here is that it is generally good practice to use methods that have already been used successfully by other researchers. For example, if you want to manipulate people’s moods to make some of them happy, it would be a good idea to use one of the many approaches that have been used successfully by other researchers (e.g., paying them a compliment). This is good not only for the sake of feasibility—the approach is “tried and true”—but also because it provides greater continuity with previous research. This makes it easier to compare your results with those of other researchers and to understand the implications of their research for yours, and vice versa.

Summary  Research ideas can come from a variety of sources, including informal observations, practical

problems, and previous research.

 Research questions expressed in terms of variables and relationships between variables can be suggested by other researchers or generated by asking a series of more general questions about the behavior or psychological characteristic of interest.

 It is important to evaluate how interesting a research question is before designing a study and collecting data to answer it. Factors that affect interestingness are the extent to which the answer is in doubt, whether it fills a gap in the research literature, and whether it has important practical implications.

 It is also important to evaluate how feasible a research question will be to answer. Factors that affect feasibility include time, money, technical knowledge and skill, and access to special equipment and research participants.

Practice: 1. Generate five research ideas based on each of the following: informal observations, practical

problems, and topics discussed in recent issues of professional journals. 2. Generate five empirical research questions about each of the following behaviors or psychological

characteristics: long-distance running, getting tattooed, social anxiety, bullying, and memory for early childhood events.

3. Evaluate each of the research questions you generated in Exercise 2 in terms of its interestingness based on the criteria discussed in this section.

4. Find an issue of a journal that publishes short empirical research reports (e.g., Psychological Science, Psychonomic Bulletin and Review, Personality and Social Psychology Bulletin). Pick three studies, and rate each one in terms of how feasible it would be for you to replicate it with the resources available to you right now. Use the following rating scale: (1) You could replicate it essentially as reported. (2) You could replicate it with some simplifications. (3) You could not replicate it. Explain each rating.

[1] Weisberg, R. W. (1993). Creativity: Beyond the myth of genius. New York, NY: Freeman.

[2] Milgram, S. (1963). Behavioral study of obedience. Journal of Abnormal and Social Psychology, 67, 371–378.

2.3 Reviewing the Research Literature

LEARNING OBJECTIVES  Define the research literature in Human Services (remember it is a multidisciplinary profession) and

give examples of sources that are part of the research literature and sources that are not.

 Describe and use several methods for finding previous research on a particular research idea or question.

Reviewing the research literature means finding, reading, and summarizing the published research relevant to your question. An empirical research report written in American Psychological Association (APA) style

RESEARCH FOR HUMAN SERVICES pg. 32

always includes a written literature review, but it is important to review the literature early in the research process for several reasons.

 It can help you turn a research idea into an interesting research question.

 It can tell you if a research question has already been answered.

 It can help you evaluate the interestingness of a research question.

 It can give you ideas for how to conduct your own study.

 It can tell you how your study fits into the research literature.

What Is the Research Literature? The research literature in any field is all the published research in that field. The research literature in the social sciences is enormous—including millions of scholarly articles and books dating to the beginning of the field—and it continues to grow. Although its boundaries are somewhat fuzzy, the research literature definitely does not include self-help and other pop psychology books, dictionary and encyclopedia entries, websites (except in rare exceptions such as that of the National Institute of Mental Health), and similar sources that are intended mainly for the general public. These are considered unreliable because they are not reviewed by other researchers and are often based on little more than common sense or personal experience. For our purposes, it helps to define the research literature as consisting almost entirely of two types of sources: articles in professional journals, and scholarly books from the social sciences.

Professional Journals Professional journals are periodicals that publish original research articles. There are thousands of professional journals that publish research in the social science. They are usually published monthly or quarterly in individual issues, each of which contains several articles. The issues are organized into volumes, which usually consist of all the issues for a calendar year. Some journals are published in hard copy only, others in both hard copy and electronic form, and still others in electronic form only.

Most articles in professional journals are one of two basic types: empirical research reports and review articles. Empirical research reports describe one or more new empirical studies conducted by the authors. They introduce a research question, explain why it is interesting, review previous research, describe their method and results, and draw their conclusions. Review articles summarize previously published research on a topic and usually present new ways to organize or explain the results. When a review article is devoted primarily to presenting a new theory, it is often referred to as a theoretical article.

Figure 2.6 This picture depicts just a small Sample of the Thousands of Professional Journals That Publish Research in Psychology and Related Fields.

RESEARCH FOR HUMAN SERVICES pg. 33

Peer Review Most professional journals undergo a process of peer review. Researchers who want to publish their work in the journal submit a manuscript to the editor—who is generally an established researcher too—who, in turn, sends it to two or three experts on the topic. Each reviewer reads the manuscript, writes a critical review, and sends the review back to the editor along with his or her recommendations. The editor then decides whether to accept the article for publication, asks the authors to make changes, and resubmit it for further consideration, or reject it outright. In any case, the editor forwards the reviewers’ written comments to the researchers so that they can revise their manuscript accordingly. Peer review is important because it ensures that the work meets basic standards of the field before it can enter the research literature.

Scholarly Books Scholarly books are books written by researchers and practitioners mainly for use by other researchers and practitioners. A monograph is written by a single author or a small group of authors and usually gives a coherent presentation of a topic much like an extended review article. Edited volumes have an editor or a small group of editors who recruit many authors to write separate chapters on different aspects of the same topic. Although edited volumes can also give a coherent presentation of the topic, it is not unusual for each chapter to take a different perspective or even for the authors of different chapters to openly disagree with each other. In general, scholarly books undergo a peer review process similar to that used by professional journals.

Literature Search Strategies

Using PsycINFO and Other Databases The primary method used to search the research literature involves using one or more electronic databases. These include Academic Search Premier, JSTOR, EPSCOHOST, and ProQuest for all academic disciplines, ERIC for education, and PubMed for medicine and related fields.

One example is PsycINFO, which is produced by the APA. PsycINFO is comprehensive—covering thousands of professional journals and scholarly books going back more than 100 years—that for most purposes its content is synonymous with the research literature in psychology. Like most such databases, PsycINFO is usually available through your college or university library.

PsycINFO consists of individual records for each article, book chapter, or book in the database. Each record includes basic publication information, an abstract or summary of the work, and a list of other works cited by that work. A computer interface allows entering one or more search terms and returns any records that contain those search terms. (These interfaces are provided by different vendors and therefore can look somewhat different depending on the library you use.) Each record also contains lists of keywords that describe the content of the work and also, a list of index terms. The index terms are especially helpful because they are standardized. Research on differences between women and men, for example, is always indexed under “Human Sex Differences.” Research on touching is always indexed under the term “Physical Contact.” If you do not know the appropriate index terms, PsycINFO includes a thesaurus that can help you find them.

Given that there are nearly three million records, and counting, in PsycINFO, you may have to try a variety of search terms in different combinations and at different levels of specificity before you find what you are looking for. Imagine, for example, that you are interested in the question of whether women and men differ in terms of their ability to recall experiences from when they were very young. If you were to enter “memory for early experiences” as your search term, PsycINFO would return only six records, most of which are not particularly relevant to your question. However, if you were to enter the search term “memory,” it would return 149,777 records—far too many to look through individually. This is where the thesaurus helps. Entering “memory” into the thesaurus provides several more specific index terms—one of which is “early memories.” While searching for “early memories” among the index terms returns 1,446 records—still too many too look through individually—combining it with “human sex differences” as a second search term returns 37 articles, many of which are highly relevant to the topic.

RESEARCH FOR HUMAN SERVICES pg. 34

Depending on the vendor that provides the interface to PsycINFO, you may be able to save, print, or e-mail the relevant PsycINFO records. The records might even contain links to full-text copies of the works themselves. (PsycARTICLES is a database that provides full-text access to articles in all journals published by the APA.) If not, and you want a copy of the work, you will have to find out if your library carries the journal or has the book and the hard copy on the library shelves. Be sure to ask a librarian if you need help.

Using Other Search Techniques In addition to entering search terms into PsycINFO and other databases, there are several other techniques you can use to search the research literature. First, if you have one good article or book chapter on your topic—a recent review article is best—you can look through the reference list of that article for other relevant articles, books, and book chapters. In fact, you should do this with any relevant article or book chapter you find. You can also start with a classic article or book chapter on your topic, find its record in PsycINFO (by entering the author’s name or article’s title as a search term), and link from there to a list of other works in PsycINFO that cite that classic article. This works because other researchers working on your topic are likely to be aware of the classic article and cite it in their own work.

You can also do a general Internet search using search terms related to your topic or the name of a researcher who conducts research on your topic. However, this approach should be used with caution since you are likely to discover that the work is unavailable without paying for it yourself, or difficult to obtain. However, this approach might lead you directly to works that are part of the research literature (e.g., articles in open-access journals or posted on researchers’ own websites) which also happen to be readily available. The search engine Google Scholar is especially useful for this purpose. A general Internet search might also lead you to websites that are not part of the research literature but might provide references to works that are. Finally, you can talk to people (e.g., your instructor or other faculty members) who know something about your topic and can suggest relevant articles and book chapters.

What to Search For When you do a literature review, you need to be selective. Not every article, book chapter, and book that relates to your research idea or question will be worth obtaining, reading, and integrating into your review. Instead, you want to focus on sources that help you do four basic things: (a) refine your research question, (b) identify appropriate research methods, (c) place your research in the context of previous research, and (d) write an effective research report. Several basic principles can help you find the most useful sources.

Newer Work It is best to focus on recent research, keeping in mind that what counts as recent depends on the topic. For newer topics that are actively being studied, “recent” might mean published in the past year or two. For older topics that are receiving less attention right now, “recent” might mean within the past 10 years. You will get a feel for what counts as recent for your topic when you start your literature search. A good general rule, however, is to start with sources published in the past five years. The main exception to this rule would be classic articles that turn up in the reference list of nearly every other source. If other researchers think that this work is important, even though it is old, then by all means you should include it in your review.

Reviews of the Topic and Meta-analysis Second, you should look for articles which include a general review of your topic because they can provide a useful overview of it—often discussing important definitions, results, theories, trends, and controversies— giving you a good sense of where your own research fits into the literature. You should also look for empirical research reports addressing your question or similar questions, which can give you ideas about how to operationally define your variables and collect your data. A meta-analysis can be particularly helpful since it will actually be a study of the literature within a given topic.

As a general rule, it is good to use methods that others have already used successfully unless you have good reasons not to. Finally, you should look for sources that provide information that can help you argue for the interestingness of your research question. For a study on the effects of cell phone use on driving

RESEARCH FOR HUMAN SERVICES pg. 35

ability, for example, you might look for information about how widespread cell phone use is, how frequent and costly motor vehicle crashes are, and so on.

How many sources are enough for your literature review? This is a difficult question because it depends on how extensively your topic has been studied and also on your own goals. One study found that across a variety of professional journals in psychology, the average number of sources cited per article was about 50 (Adair & Vohra, 2003). [1] This gives a rough idea of what professional researchers consider to be adequate. As a student, you might be assigned a much lower minimum number of references to use, but the principles for selecting the most useful ones remain the same.

Summary

 The research literature in the social sciences is all the published research in many related areas (such as education, anthropology, sociology, psychology, etc.) consisting primarily of articles in professional journals and scholarly books.

 Early in the research process, it is important to conduct a review of the research literature on your topic to refine your research question, identify appropriate research methods, place your question in the context of other research, and prepare to write an effective research report.

 There are several strategies for finding previous research on your topic. Among the best is using PsycINFO, a computer database that catalogs millions of articles, books, and book chapters in psychology and related fields.

Practice: Use the techniques discussed in this section to find 10 journal articles and book chapters on one of the following research ideas: memory for smells, aggressive driving, the causes of narcissistic personality disorder, the functions of the intraparietal sulcus, or prejudice against the physically handicapped.

[1] Adair, J. G., & Vohra, N. (2003). The explosion of knowledge, references, and citations: Psychology’s unique response to a crisis. American Psychologist, 58, 15–23.

RESEARCH FOR HUMAN SERVICES pg. 36

Chapter 3: Research Ethics It is curious - curious that physical courage should be so common in the world, and moral courage so rare.

- Mark Twain

In 1998, a medical journal called The Lancet published an article of interest to Human Service workers and the general public. It ended up having a global impact, and is still the subject of heated debate even after being completely discredited. The researchers claimed to have shown a statistical relationship between receiving the combined measles, mumps, and rubella (MMR) vaccine and the development of autism— suggesting furthermore that the vaccine might even cause autism. One result of this report was that many parents decided not to have their children vaccinated, which of course put them at higher risk for measles, mumps, and rubella.

However, follow-up studies by other researchers consistently failed to find a statistical relationship between the MMR vaccine and autism—and it is generally accepted now that there is no relationship. In addition, several more serious problems with the original research were uncovered. Among them were that the lead researcher stood to gain financially from his conclusions because he had patented a competing measles vaccine. He had also used biased methods to select and test his research participants and had used unapproved and medically unnecessary procedures on them. In 2010 The Lancet retracted the article, and the lead researcher’s right to practice medicine was revoked (Burns, 2010). [1] Despite this, many parents still cling to a theory that has been completely discredited.

In this chapter, we explore the ethics of scientific research. We begin with a general framework for thinking about the ethics of scientific research. We also need to consider the proliferation of false, and misleading, “news” items that are conveyed through social media and other avenues which for many people are serving instead of news which is curated, and fact checked.

[1] Burns, J. F. (2010, May 24). British medical council bars doctor who linked vaccine to autism. The New York Times. Retrieved fromhttp://www.nytimes.com/2010/05/25/health/policy/25autism.html?ref=andrew_wakefield

3.1 Moral Foundations of Ethical Research LEARNING OBJECTIVES

 Describe a simple framework for thinking about ethical issues regarding research.

 Give examples of several ethical issues that arise in research with human subjects —including ones that affect research participants, the scientific community, and society more generally.

Ethics are the principles of right conduct and is an established field of philosophy. Morality is closely related, and sometimes used interchangeably with ethics, is generally considered to be more personal and subjective. We are concerned here with ethics as a set of principles and practices that provide principles for right conduct in a particular field. There is an ethics of business, medicine, teaching, human services practice, and of course, scientific research. As the opening example illustrates, many kinds of ethical issues can arise in scientific research, especially when it involves human participants. Research with human subjects is held to a higher standard although. The lives and wellbeing of people could be at stake. Not all forms of research with humans is benign or free from risk. For this reason, it is useful to begin with a general framework for thinking through these issues.

A Framework for Thinking About Research Ethics Science does not happen in a social, political, or ethical vacuum. There are important issues to consider and the following principles (are adapted from those in the American Psychological Association [APA]

RESEARCH FOR HUMAN SERVICES pg. 37

Ethics Code but are supported in all of the other social sciences.) provide a place to start when we consider the consequences of research.

Moral Principles to Consider and Who Research Affects:

Moral Principles to consider in research:

1. Weighing risk against benefits. 2. Acting responsibly and with integrity. 3. Seeking justice. 4. Respecting people’s rights and dignity.

People who are affected in research:

a Research participants b The scientific community c Society d The researchers themselves

Ethical Principles Let us look more closely at each of the ethical principles and how they can be applied to each of the three groups.

Weighing Risks Against Benefits Scientific research can be ethical only if its risks are outweighed by its benefits. Among the risks to research participants are that a treatment might fail to help or even be harmful, a procedure might result in physical or psychological harm, and their right to privacy might be violated. Among the potential benefits are receiving a helpful treatment, learning about the human condition, experiencing the satisfaction of contributing to scientific knowledge, and receiving money or course credit for participating. Scientific research can have risks and benefits to the scientific community and to society too (Rosenthal, 1994). [1] A risk to science is that if a research question is uninteresting or a study is poorly designed, then the time, money, and effort spent on that research could have been spent on more productive research. A risk to society is that research results could be misunderstood or misapplied with harmful consequences. The research that mistakenly linked the measles, mumps, and rubella (MMR) vaccine to autism resulted in both of these kinds of harm. Of course, the benefits of scientific research to science and society are that it advances scientific knowledge and can contribute to the welfare of society.

It is not necessarily easy to weigh the risks of research against its benefits because the risks and benefits may not be directly comparable. For example, it is common for the risks of a study to be primarily to the research participants but the benefits primarily for science or society. Consider, for example, Stanley Milgram’s original study on obedience to authority (Milgram, 1963). [2]

The Milgram Study on Obedience In Milgram’s research, the participants were told that they were taking part in a study on the effects of punishment on learning and were instructed to give electric shocks to another participant each time that participant responded incorrectly on a learning task. With each incorrect response, the shock became stronger—eventually causing the other participant (who was in the next room) to protest, complain about his heart, scream in pain, and finally fall silent and stop responding. If the first participant hesitated or expressed concern, the researcher said that he must continue. In reality, the other participant was a confederate of the researcher—a helper who pretended to be a real participant—and the protests, complaints, and screams that the real participant heard were an audio recording that was activated when he flipped the switch to administer the “shocks.” The surprising result of this study was that most of the real participants continued to administer the shocks right through the confederate’s protests, complaints, and screams. Although this is considered one of the most important results in psychology—with implications for understanding events like the Holocaust or the mistreatment of prisoners by US soldiers at Abu Ghraib—it came at the cost of producing severe psychological stress in the research participants.

RESEARCH FOR HUMAN SERVICES pg. 38

Was It Worth It? Much of the debate over the ethics of Milgram’s obedience study concerns the question of whether the resulting scientific knowledge was worth the harm caused to the research participants. To get a better sense of the harm, consider Milgram’s (1963) [3] own description of it.

In a large number of cases, the degree of tension reached extremes that are rarely seen in sociopsychological laboratory studies. Subjects were observed to sweat, tremble, stutter, bite their lips, groan, and dig their fingernails into their flesh. Fourteen of the 40 subjects showed definite signs of nervous laughter and smiling. The laughter seemed entirely out of place, even bizarre. Full blown uncontrollable seizures [of laughter] were observed for three subjects. On one occasion, we observed a seizure so violently convulsive that it was necessary to call a halt to the experiment (p. 375).

Milgram also noted that another observer reported that within 20 minutes one participant “was reduced to a twitching, stuttering wreck, who was rapidly approaching the point of nervous collapse” (p. 377)

To Milgram’s credit, he went to great lengths to debrief his participants—including attempting to return their mental states to normal—and attempted to show that most of them thought the research was valuable and were glad to have participated. Still, this research would be considered unethical by today’s standards. Also, there is some evidence that the participants were still haunted by the experience years later.

An excellent biographical drama, based on the Milgram experiment, is the Experimenter.

Acting Responsibly and With Integrity Researchers must act responsibly and with integrity. This means carrying out their research in a thorough and competent manner, meeting their professional obligations, and being truthful. Acting with integrity is important because it promotes trust, which is an essential element of all effective human relationships. Participants must be able to trust that researchers are being honest with them (e.g., about what the study involves), will keep their promises (e.g., to maintain confidentiality), and will carry out their research in ways that maximize benefits and minimize risk. An important issue here is the use of deception. Some research questions (such as Milgram’s) are difficult or impossible to answer without deceiving research participants. Thus, acting with integrity can conflict with doing research that advances scientific knowledge and benefits society. We will consider how psychologists generally deal with this conflict shortly.

The scientific community and society must also be able to trust that researchers have conducted their research thoroughly and competently and that they have reported on it honestly. Unfortunately, this is not always the case. Again, the example at the beginning of the chapter illustrates what can happen when this trust is violated. In this case, other researchers wasted resources on unnecessary follow-up research and people avoided the MMR vaccine, putting their children at increased risk of measles, mumps, and rubella.

Seeking Justice: The Tuskegee Experiment Researchers must conduct their research in a just manner. They should treat their participants fairly, for example, by giving them adequate compensation for their participation and making sure that benefits and risks are distributed across all participants. For example, in a study of a new and potentially beneficial psychotherapy, some participants might receive the psychotherapy while others serve as a control group that receives no treatment. If the psychotherapy turns out to be effective, it would be fair to offer it to participants in the control group when the study ends.

At a broader societal level, members of some groups have historically faced more than their fair share of the risks of scientific research, including people who are institutionalized, are disabled, or belong to racial or ethnic minorities. A particularly tragic example is the Tuskegee syphilis study conducted by the US Public Health Service from 1932 to 1972 (Reverby, 2009). [4] The participants in this study were poor African American men in the vicinity of Tuskegee, Alabama, who were told that they were being treated for “bad blood.” Although they were given some free medical care, they were not treated for their syphilis. Instead, they were observed to see how the disease developed in untreated patients. Even after the use of penicillin became the standard treatment for syphilis in the 1940s, these men continued to be denied

RESEARCH FOR HUMAN SERVICES pg. 39

treatment without being given an opportunity to leave the study. The study was eventually discontinued only after details were made known to the general public by journalists and activists. It is now widely recognized that researchers need to consider issues of justice and fairness at the societal level.

“They Were Betrayed” In 1997—65 years after the Tuskegee Syphilis Study began and 25 years after it ended—President Bill Clinton formally apologized on behalf of the US government to those who were affected. Here is an excerpt from the apology:

So today America does remember the hundreds of men used in research without their knowledge and consent. We remember them and their family members. Men who were poor and African American, without resources and with few alternatives, they believed they had found hope when they were offered free medical care by the United States Public Health Service. They were betrayed.

Read the full text of the apology at http://www.cdc.gov/tuskegee/clintonp.htm.

Respecting People’s Rights and Dignity Researchers must respect people’s rights and dignity as human beings. One element of this is respecting their autonomy—their right to make their own choices and take their own actions free from coercion. Of fundamental importance here is the concept of informed consent. This means that researchers obtain and document people’s agreement to participate in a study after having informed them of everything that might reasonably be expected to affect their decision. Consider the participants in the Tuskegee study. Although they agreed to participate in the study, they were not told that they had syphilis but would be denied treatment for it. Had they been told this basic fact about the study, it seems likely that they would not have agreed to participate. Likewise, had participants in Milgram’s study been told that they might be “reduced to a twitching, stuttering wreck,” it seems likely that many of them would not have agreed to participate. In neither of these studies did participants give true informed consent.

Another element of respecting people’s rights and dignity is respecting their privacy—their right to decide what information about them is shared with others. This means that researchers must maintain confidentiality, which is essentially an agreement not to disclose participants’ personal information without their consent or some appropriate legal authorization.

Unavoidable Ethical Conflict Ethical questions social research is unavoidable. Research that is beneficial to one group (e.g., the scientific community) can be harmful to another (e.g., the research participants), creating especially difficult tradeoffs. We have also seen that being completely truthful with research participants can make it difficult or impossible to conduct scientifically valid studies on important questions.

Many ethical conflicts are fairly easy to resolve. Nearly everyone would agree that deceiving research participants and then subjecting them to physical harm would not be justified by filling a small gap in the research literature. But many ethical conflicts are not easy to resolve, and competent and well-meaning researchers can disagree about how to resolve them. Consider, for example, an actual study on “personal space” conducted in a public men’s room (Middlemist, Knowles, & Matter, 1976). [5] The researchers secretly observed their participants to see whether it took them longer to begin urinating when there was another man (a confederate of the researchers) at a nearby urinal. While some critics found this to be an unjustified assault on human dignity (Koocher, 1977), [6] the researchers had carefully considered the ethical conflicts, resolved them as best they could, and concluded that the benefits of the research outweighed the risks (Middlemist, Knowles, & Matter, 1977). [7] For example, they had interviewed some preliminary participants and found that none of them was bothered by the fact that they had been observed.

The point here is that although it may not be possible to eliminate ethical conflict completely, it is possible to deal with it in responsible and constructive ways. In general, this means thoroughly and carefully thinking through the ethical issues that are raised, minimizing the risks, and weighing the risks against the benefits.

RESEARCH FOR HUMAN SERVICES pg. 40

It also means being able to explain one’s ethical decisions to others, seeking feedback on them, and ultimately taking responsibility for them.

Summary

 A wide variety of ethical issues arise in research. Thinking them through requires considering how each of four ethical principles (weighing risks against benefits, acting responsibly and with integrity, seeking justice, and respecting people’s rights and dignity) applies to each of three groups of people (research participants, science, and society).

 Ethical conflict in research is unavoidable. Researchers must think through the ethical issues raised by their research, minimize the risks, weigh the risks against the benefits, be able to explain their ethical decisions, seek feedback about these decisions from others, and ultimately take responsibility for them.

Practice: Imagine a study testing the effectiveness of a new drug for treating obsessive-compulsive disorder. Give a hypothetical example of an ethical issue from each cell of Table 3.1 "A Framework for Thinking About Ethical Issues in Scientific Research" that could arise in this research.

Discussion:

It has been argued that researchers are not ethically responsible for the misinterpretation or misuse of their research by others. Do you agree? Why or why not?

[1] Rosenthal, R. M. (1994). Science and ethics in conducting, analyzing, and reporting psychological research. Psychological Science, 5, 127–133.

[2] Milgram, S. (1963). Behavioral study of obedience. Journal of Abnormal and Social Psychology, 67, 371–378.

[3] Milgram, S. (1963). Behavioral study of obedience. Journal of Abnormal and Social Psychology, 67, 371–378.

[4] Reverby, S. M. (2009). Examining Tuskegee: The infamous syphilis study and its legacy. Chapel Hill, NC: University of North Carolina Press.

[5] Middlemist, R. D., Knowles, E. S., & Matter, C. F. (1976). Personal space invasions in the lavatory: Suggestive evidence for arousal. Journal of Personality and Social Psychology, 33, 541–546.

[6] Koocher, G. P. (1977). Bathroom behavior and human dignity. Journal of Personality and Social Psychology, 35, 120–121.

[7] Middlemist, R. D., Knowles, E. S., & Matter, C. F. (1977). What to do and what to report: A reply to Koocher. Journal of Personality and Social Psychology, 35, 122–125.

3.2 From Principles to Ethics Codes LEARNING OBJECTIVES

 Describe the history of ethics codes for scientific research with human participants.

 Summarize the American Psychological Association Ethics Code—especially as it relates to informed consent, deception, debriefing, research with nonhuman animals, and scholarly integrity.

RESEARCH FOR HUMAN SERVICES pg. 41

The general moral principles of weighing risks against benefits, acting with integrity, seeking justice, and respecting people’s rights and dignity provide a useful starting point for thinking about the ethics of psychological research because essentially everyone agrees on them. As we have seen, however, even people who agree on these general principles can disagree about specific ethical issues that arise while conducting research. This is why there also exist more detailed and enforceable ethics codes that provide guidance on important issues that arise frequently. In this section, we begin with a brief historical overview of such ethics codes and then look closely at the one that is most relevant to psychological research—that of the American Psychological Association (APA).

Historical Overview One of the earliest ethics codes was the Nuremberg Code—a set of 10 principles written in 1947 in conjunction with the trials of Nazi physicians accused of shockingly cruel research on concentration camp prisoners during World War II. It provided a standard against which to compare the behavior of the men on trial—many of whom were eventually convicted and either imprisoned or sentenced to death. The Nuremberg Code was particularly clear about the importance of carefully weighing risks against benefits and the need for informed consent.

The Declaration of Helsinki is a similar ethics code that was created by the World Medical Council in 1964. Among the standards that it added to the Nuremberg Code was that research with human participants should be based on a written protocol—a detailed description of the research—that is reviewed by an independent committee. The Declaration of Helsinki has been revised several times, most recently in 2004.

In the United States, concerns about the Tuskegee study and others led to the publication in 1978 of a set of federal guidelines called the Belmont Report. The Belmont Report explicitly recognized the principle of seeking justice, including the importance of conducting research in a way that distributes risks and benefits fairly across different groups at the societal level. The Belmont Report became the basis of a set of laws— the Federal Policy for the Protection of Human Subjects—that apply to research conducted, supported, or regulated by the federal government. An extremely important part of these regulations is that universities, hospitals, and other institutions that receive support from the federal government must establish an institutional review board (IRB)—a committee that is responsible for reviewing research protocols for potential ethical problems. An IRB must consist of at least five people with varying backgrounds, including members of different professions, scientists and nonscientists, men and women, and at least one person not otherwise affiliated with the institution. The IRB helps to make sure that the risks of the proposed research are minimized, the benefits outweigh the risks, the research is carried out in a fair manner, and the informed consent procedure is adequate.

The federal regulations also distinguish research that poses three levels of risk. Exempt research includes research on the effectiveness of normal educational activities, the use of standard psychological measures and surveys of a non-sensitive nature that are administered in a way that maintains confidentiality, and research using existing data from public sources. It is called exempt because the regulations do not apply to it. Minimal risk research exposes participants to risks that are no greater than those encountered by healthy people in daily life or during routine physical or psychological examinations. Minimal risk research can receive an expedited review by one member of the IRB or by a separate committee under the authority of the IRB that can only approve minimal risk research. Some departments have such separate committees but at the very least every college or university has some mechanism for overseeing research. Finally, at- risk research poses greater than minimal risk and must be reviewed by the IRB.

Ethics Codes The link that follows the list—from the Office of Human Subjects Research at the National Institutes of Health—allows you to read the ethics codes discussed in this section in their entirety. They are all highly recommended and, with the exception of the Federal Policy, short and easy to read.

 The Nuremberg Code

 The Declaration of Helsinki

RESEARCH FOR HUMAN SERVICES pg. 42

 The Belmont Report

 Federal Policy for the Protection of Human Subjects http://ohsr.od.nih.gov/guidelines/index.html

Research Ethics and Human Services Human Services, like other professional and academic disciplines, has a code of ethics, and it covers research. As of this writing the NOHS, National Organization for Human Services, has 44 practice standards and four of them cover the topic of research. They are available online at: http://www.nationalhumanservices.org/ethical-standards-for-hs-professionals.

Other Social Science Ethical Codes The code of ethics for Human Services is far less detailed as many others. Especially regarding the issue of research. You can find the code of ethics for Psychology online. The code of ethics for Psychology is quite detailed and is worth reviewing. The code of ethics for social work, which in no way conflicts with that of Human Services, and should be especially helpful. That code strikes a balance between the brevity of the NOHS code of ethics, and the complexity and academic orientation of the American Psychological Association. It is also much more practice, and clinically, oriented.

Research Ethics & Social Work Social Work emphasizes empirically based practice throughout its code of ethics for practitioners as well as academics. There is an emphasis on keeping current with, and applying, current research findings in all aspects of Social Work practice. This emphasis ranges from policy to clinical applications, while encouraging all practitioners to use evaluative methods to measure the effectiveness of practice regardless of the setting. In many ways, the emphasis on research is reversed when comparing Psychology and Social Work. Note how evaluation and research are combined in the approach Social Work takes:

If you need further guidance than what is outlined in the NOHS code of ethics regarding research, you would not be wrong to follow the guidelines outlined by the National Association of Social Workers.

Practice Read the Nuremberg Code, the Belmont Report, and Standard 8 of the APA Ethics Code. List five specific similarities and five specific differences among them.

Discussion In a study on the effects of disgust on moral judgment, participants were asked to judge the morality of disgusting acts, including people eating a dead pet and passionate kissing between a brother and sister (Haidt, Koller, & Dias, 1993). [6] If you were on the IRB that reviewed this protocol, what concerns would you have with it? Refer to the appropriate sections of the APA Ethics Code.

[1] Mann, T. (1994). Informed consent for psychological research: Do subjects comprehend consent forms and understand their legal rights? Psychological Science, 5, 140–143.

[2] Sieber, J. E., Iannuzzo, R., & Rodriguez, B. (1995). Deception methods in psychology: Have they changed in 23 years? Ethics & Behavior, 5, 67–85.

[3] Baumrind, D. (1985). Research using intentional deception: Ethical issues revisited. American Psychologist, 40, 165–174.

[4] Bowd, A. D., & Shapiro, K. J. (1993). The case against animal laboratory research in psychology. Journal of Social Issues, 49, 133–142.

[5] Miller, N. E. (1985). The value of behavioral research on animals. American Psychologist, 40, 423–440.

RESEARCH FOR HUMAN SERVICES pg. 43

[6] Haidt, J., Koller, S. H., & Dias, M. (1993). Affect, culture, and morality, or is it wrong to eat your dog? Journal of Personality and Social Psychology, 65, 613–628.

3.3 Putting Ethics into Practice

LEARNING OBJECTIVES  Describe several strategies for identifying and minimizing risks and deception in psychological

research.

 Create thorough informed consent and debriefing procedures, including a consent form.

In this section, we look at some practical advice for conducting ethical research. Again, it is important to remember that ethical issues arise well before you begin to collect data and continue to arise through publication and beyond.

Know and Accept Your Ethical Responsibilities The emphasis in this section is on research ethics. However, the boundary between clinical, or practice ethics, and research ethics is blurred.

As the American Psychological Association (APA) Ethics Code notes in its introduction, “Lack of awareness or misunderstanding of an ethical standard is not itself a defense to a charge of unethical conduct.” Therefore, the very first thing that you must do as a new researcher is to know and accept your ethical responsibilities. At a minimum, this means reading and understanding the relevant standards, such as those of the American Psychological Association (APA) Ethics Code for research which is available in the appendix), and the standards for outlined by the National Organization for Human Services (NOHS) and the National Organization of Social Workers (NASW). Distinguishing minimal risk from at-risk research, and knowing the specific policies and procedures of your institution—including how to prepare and submit a research protocol for institutional review board (IRB) review.

If you are conducting research as a course requirement, there may be specific course standards, policies, and procedures. If any standard, policy, or procedure is unclear—or you are unsure what to do about an ethical issue that arises—you must seek clarification. You can do this by reviewing the relevant ethics codes, reading about how similar issues have been resolved by others, or consulting with more experienced researchers, your institutional research board (IRB), or your course instructor. Ultimately, you as the researcher must take responsibility for the ethics of the research you conduct. Columbia College maintains an active institutional research board composed of experienced faculty, with recommendations reviewed at the level of the academic deans and the Provost level.

Identify and Minimize Risks As you design your study, you must identify and minimize risks to participants. Start by listing all the risks, including risks of physical and psychological harm and violations of confidentiality. Remember that it is easy for researchers to see risks as less serious than participants do or even to overlook them completely.

RESEARCH FOR HUMAN SERVICES pg. 44

For example, one student researcher wanted to test people’s sensitivity to violent images by showing them gruesome photographs of crime and accident scenes. Because she was an emergency medical technician, however, she greatly underestimated how disturbing these images were to most people. Remember too that some risks might apply only to some participants. For example, while most people would have no problem completing a survey about their fear of various crimes, those who have been a victim of one of those crimes might become upset. Therefore, you should seek input from a variety of people, including your research collaborators, more experienced researchers, and even from non-researchers who might be better able to take the perspective of a participant.

Once you have identified the risks, you can often reduce or eliminate many of them. One way is to modify the research design. For example, you might be able to shorten or simplify the procedure to prevent boredom and frustration. You might be able to replace upsetting or offensive stimulus materials (e.g., graphic accident scene photos) with less upsetting or offensive ones (e.g., milder photos of the sort people are likely to see in the newspaper). A good example of modifying a research design is a 2009 replication of Milgram’s study conducted by Jerry Burger. Instead of allowing his participants to continue administering shocks up to the 450-V maximum, the researcher always stopped the procedure when they were about to administer the 150-V shock (Burger, 2009). [1] This made sense because in Milgram’s study (a) participants’ severe negative reactions occurred after this point and (b) most participants who administered the 150-V shock continued all the way to the 450-V maximum. Thus, the researcher was able to compare his results directly with Milgram’s at every point up to the 150-V shock and also was able to estimate how many of his participants would have continued to the maximum—but without subjecting them to the severe stress that Milgram did. (The results, by the way, were that these contemporary participants were just as obedient as Milgram’s were.)

A second way to minimize risks is to use a prescreening procedure to identify and eliminate participants who are at high risk. You can do this in part through the informed consent process. For example, you can warn participants that a survey includes questions about their fear of crime and remind them that they are free to withdraw if they think this might upset them. Prescreening can also involve collecting data to identify and eliminate participants. For example, Burger used an extensive prescreening procedure involving multiple questionnaires and an interview with a clinical psychologist to identify and eliminate participants with physical or psychological problems that put them at high risk.

A third way to minimize risks is to take active steps to maintain confidentiality. You should keep signed consent forms separately from any data that you collect and in such a way that no individual’s name can be linked to his or her data. In addition, beyond people’s sex and age, you should only collect personal information that you actually need to answer your research question. If people’s sexual orientation or ethnicity is not clearly relevant to your research question, for example, then do not ask them about it. Be aware also that certain data collection procedures can lead to unintentional violations of confidentiality. When participants respond to an oral survey in a shopping mall or complete a questionnaire in a classroom setting, it is possible that their responses will be overheard or seen by others. If the responses are personal, it is better to administer the survey or questionnaire individually in private or to use other techniques to prevent the unintentional sharing of personal information.

Identify and Minimize Deception Remember that deception can take a variety of forms, not all of which involve actively misleading participants. It is also deceptive to allow participants to make incorrect assumptions (e.g., about what will be on a “memory test”) or simply withhold information about the full design or purpose of the study. It is best to identify and minimize all forms of deception.

According to the APA Ethics Code, deception is ethically acceptable only if there is no way to answer your research question without it. In Human Services and Social Work deception is generally not viewed favorably from a philosophical perspective. Therefore, if your research design includes any form of active deception, you should consider whether it is truly necessary. Imagine, for example, that you want to know whether the age of college professors affects students’ expectations about their teaching ability. You could do this by telling participants that you will show them photos of college professors and ask them to rate

RESEARCH FOR HUMAN SERVICES pg. 45

each one’s teaching ability. But if the photos are not really of college professors but of your own family members and friends, then this would be deception. This deception could easily be eliminated, however, by telling participants instead to imagine that the photos are of college professors and to rate them as if they were. If you use deception in your research the IRB will scrutinize it very, very carefully.

In general, it is considered acceptable to wait until debriefing before you reveal your research question as long as you describe the procedure, risks, and benefits during the informed consent process. For example, you would not have to tell participants that you wanted to know whether the age of college professors affects people’s expectations about them until the study was over. Not only is this information unlikely to affect people’s decision about whether or not to participate in the study, but it has the potential to invalidate the results. Participants who know that age is the independent variable might rate the older and younger “professors” differently because they think you want them to. Alternatively, they might be careful to rate them the same so that they do not appear prejudiced. But even this extremely mild form of deception can be minimized by informing participants—orally, in writing, or both—that although you have accurately described the procedure, risks, and benefits, you will wait to reveal the research question until afterward. In essence, participants give their consent to be deceived or to have information withheld from them until later.

Weigh the Risks Against the Benefits Once the risks of the research have been identified and minimized, you need to weigh them against the benefits. This requires identifying all the benefits. Remember to consider benefits to the research participants, to science, and to society. If you are a student researcher, remember that one of the benefits is the knowledge you will gain about how to conduct scientific research that can perhaps increase the knowledge you can then use to complete your studies and succeed in graduate school or in your career.

If the research poses minimal risk—no more than in people’s daily lives or routine physical or psychological examinations—then even a small benefit to participants, science, or society is generally considered enough to justify it. If it poses more than minimal risk, then there should be more benefits. If the research has the potential to upset some participants, for example, then it becomes more important that the study be well designed and answer a scientifically interesting research question or have clear practical implications. It would be unethical to subject people to pain, fear, or embarrassment for no better reason than to satisfy one’s personal curiosity. In general, psychological research that has the potential to cause harm that is more than minor or lasts for more than a short time is rarely considered justified by its benefits. Consider, for example, that Milgram’s study—as interesting and important as the results were—would be considered unethical by today’s standards.

Create Informed Consent and Debriefing Procedures Once you have settled on a research design, you need to create your informed consent and debriefing procedures. Start by deciding whether informed consent is necessary. You will need to consult two authorities. One is your institution and their guidelines for research. The other is using the code of ethics from the NOHS, and perhaps the NASW, for guidance as well.

If informed consent is necessary, there are several things you should do. First, when you recruit participants—whether it is through word of mouth, posted advertisements, or a participant pool—provide them with as much information about the study as you can. This will allow those who might find the study objectionable to avoid it. Second, prepare a script or set of “talking points” to help you explain the study to your participants in simple everyday language. This should include a description of the procedure, the risks and benefits, and their right to withdraw at any time. Third, you must create an informed consent form that participants can read and sign after you have described the study to them. Your university, department, or course instructor may have a sample consent form that you can adapt for your own study. A copy of the forms for Columbia College is included in the appendix. Remember that if appropriate, both the oral and written parts of the informed consent process should include the fact that you are keeping some information about the design or purpose of the study from them but that you will reveal it during debriefing.

RESEARCH FOR HUMAN SERVICES pg. 46

Debriefing is similar to informed consent in that you cannot necessarily expect participants to read and understand written debriefing forms. So, again, it is best to write a script or set of talking points with the goal of being able to explain the study in simple everyday language. During debriefing, you should reveal the research question and full design of the study. For example, if participants are tested under only one condition, then you should explain what happened in the other conditions. If you deceived your participants, you should reveal this as soon as possible, apologize for the deception, explain why it was necessary, and correct any misconceptions that participants might have as a result. Debriefing is also a good time to provide additional benefits to research participants by giving them relevant practical information or referrals to other sources of help. For example, in a study of attitudes toward domestic abuse, you could provide pamphlets about domestic abuse and referral information to the university counseling center for those who might want it.

Remember to schedule plenty of time for the informed consent and debriefing processes. They cannot be effective if you must rush through them.

Get Approval The next step is to get institutional approval for your research based on the specific policies and procedures at your institution or for your course. This will generally require writing a protocol that describes the purpose of the study, the research design and procedure, the risks and benefits, the steps taken to minimize risks, and the informed consent and debriefing procedures. Do not think of the institutional approval process as merely an obstacle to overcome but as an opportunity to think through the ethics of your research and to consult with others who are likely to have more experience or different perspectives than you. If the IRB has questions or concerns about your research, address them promptly and in good faith. This might even mean making further modifications to your research design and procedure before resubmitting your protocol.

Follow Through Your concern with ethics should not end when your study receives institutional approval. It now becomes important to stick to the protocol you submitted or to seek additional approval for anything other than a minor change. During the research, you should monitor your participants for unanticipated reactions and seek feedback from them during debriefing. One criticism of Milgram’s study is that although he did not know ahead of time that his participants would have such severe negative reactions, he certainly knew after he had tested the first several participants and should have made adjustments at that point (Baumrind, 1985). [2] Be alert also for potential violations of confidentiality. Keep the consent forms and the data safe and separate from each other and make sure that no one, intentionally or unintentionally, has access to any participant’s personal information.

Finally, you must maintain your integrity through the publication process and beyond. Address publication credit—who will be authors on the research and the order of authors—with your collaborators early and avoid plagiarism in your writing. Remember that your scientific goal is to learn about the way the world actually is and that your scientific duty is to report on your results honestly and accurately. So, do not be tempted to fabricate data or alter your results in any way. Besides, unexpected results are often as interesting, or more so, than expected ones.

Summary

 It is your responsibility as a researcher to know and accept your ethical responsibilities.

 You can take several concrete steps to minimize risks and deception in your research. These include making changes to your research design, prescreening to identify and eliminate high-risk participants, and providing participants with as much information as possible during informed consent and debriefing.

RESEARCH FOR HUMAN SERVICES pg. 47

 Your ethical responsibilities continue beyond IRB approval. You need to monitor participants’ reactions, be alert for potential violations of confidentiality, and maintain scholarly integrity through the publication process.

EX ER C IS ES

Discussion How could you conduct a study on the extent to which people obey authority in a way that minimizes risks and deception as much as possible? (Note: Such a study would not have to look at all like Milgram’s.)

Practice Find a study in a professional journal and create a consent form for that study. Be sure to include all the information in Standard 8.02.

[1] Burger, J. M. (2009). Replicating Milgram: Would people still obey today? American Psychologist, 64, 1– 11.

[2] Baumrind, D. (1985). Research using intentional deception: Ethical issues revisited. American Psychologist, 40, 165–174.

RESEARCH FOR HUMAN SERVICES pg. 48

Chapter 4: Theory In theory there is no difference between theory and practice. In practice there is.

-Yogi Berra

The birth of science as we know it arguably began with Isaac Newton's formulation of the laws of gravitation and motion. It is no exaggeration to say that physics was reborn in the early 20th-century with the twin revolutions of quantum mechanics and the theory of relativity. -Paul Davies

In the following paragraph, researchers Sherlock Campbell and James Pennebaker describe a remarkable statistical relationship.

Multiple laboratories have demonstrated that people who are asked to write about traumatic experiences subsequently exhibit better physical health than people who are asked to write about superficial topics. In these studies, individuals are randomly assigned to write about either emotional or non-emotional topics for 15 to 20 min per day for 3 to 5 consecutive days. In the past 15 years, dozens of replications have demonstrated that emotional writing can influence frequency of physician visits, immune function, stress hormones, blood pressure, and a host of social, academic, and cognitive variables. These effects hold up across cultures, ages, and diverse samples. (Campbell & Pennebaker, 2003, p. 60) [1]

In other words, researchers have answered the interesting and important question of whether engaging in what has come to be called “expressive writing” improves people’s health. It does. But there is a second question that is equally interesting and important: Why? What psychological and biological variables, structures, and processes are involved, and how do they connect the act of expressive writing to improved health? Several ideas have been proposed. For example, people who write about traumatic experiences might habituate to them. That is, the more they think about them, the less negatively they react both psychologically and physiologically—leading to improvements in mental and physical health (Lepore, Greenberg, Bruno, & Smyth, 2002). [2]

This example illustrates that, like all scientists, researchers in Human Services distinguish between two sorts of knowledge: their systematic observations and their explanations or interpretations of those observations. Typically, the former are called phenomena and the latter are called theories. Up to this point in the book, we have focused on phenomena. In this chapter, however, we focus on the equally important role of theories. We begin by exploring the distinction between phenomena and theories in more detail. We then look at the wide variety of theories that researchers construct. Finally, we consider how researchers use theories, and we present some strategies for incorporating theory into your own research.

[1] Campbell, R. S., & Pennebaker, J. W. (2003). The secret life of pronouns: Flexibility in writing style and physical health. Psychological Science, 14, 60–65.

[2] Lepore, S. J., Greenberg, M. A., Bruno, M., & Smyth, J. M. (2002). Expressive writing and health: Self- regulation of emotion-related experience, physiology, and behavior. In S. J. Lepore & J. M. Smyth (Eds.), The writing cure: How expressive writing promotes health and emotional well-being (pp. 99–117). Washington, DC: American Psychological Association.

RESEARCH FOR HUMAN SERVICES pg. 49

4.1 Phenomena and Theories

LEARNING OBJECTIVES  Define the terms phenomenon and theory and distinguish clearly between them.

 Explain the purposes of scientific theories.

 Explain why there are usually many plausible theories for any set of phenomena.

Phenomena A tornado is a phenomena. The theory behind them is the meeting of warm, moist air from the Gulf of

Mexico and cool, dry air from Canada.

A phenomenon (plural, phenomena) is a general result that has been observed reliably in systematic empirical research. In essence, it is an established answer to a research question. Some phenomena we have encountered in this book are that expressive writing improves health, women do not talk more than men, and cell phone usage impairs driving ability. Some others are that dissociative identity disorder (formerly called multiple personality disorder) increased greatly in prevalence during the late 20th century, people perform better on easy tasks when they are being watched by others (and worse on difficult tasks), and people recall items presented at the beginning and end of a list better than items presented in the middle.

Some Famous Phenomena from Psychology Phenomena are often given names by their discoverers or other researchers, and these names can catch on and become widely known. The following list is a small sample of famous phenomena in psychology.

 Blindsight. People with damage to their visual cortex are often able to respond to visual stimuli

that they do not consciously see.

 Bystander effect. The more people who are present at an emergency situation, the less likely it is

that any one of them will help.

 Fundamental attribution error. People tend to explain others’ behavior in terms of their personal

characteristics as opposed to the situation they are in.

 McGurk effect. When audio of a basic speech sound is combined with video of a person making

mouth movements for a different speech sound, people often perceive a sound that is intermediate

between the two. For a demonstration, see http://www.faculty.ucr.edu/~rosenblu/VSMcGurk.html.

 Own-race effect. People recognize faces of people of their own race more accurately than faces of

people of other races.

 Placebo effect. Placebos (fake psychological or medical treatments) often lead to improvements in

people’s symptoms and functioning.

 Mere exposure effect. The more often people have been exposed to a stimulus, the more they like

it—even when the stimulus is presented subliminally.

 Serial position effect. Stimuli presented near the beginning and end of a list are remembered

better than stimuli presented in the middle. For a demonstration, see

http://cat.xula.edu/thinker/memory/working/serial.

 Spontaneous recovery. A conditioned response that has been extinguished often returns with no

further training after the passage of time.

RESEARCH FOR HUMAN SERVICES pg. 50

Although an empirical result might be referred to as a phenomenon after being observed only once, this term is more likely to be used for results that have been replicated. Replication means conducting a study again—either exactly as it was originally conducted or with modifications—to be sure that it produces the same results. Individual researchers usually replicate their own studies before publishing them. Many empirical research reports include an initial study and then one or more follow-up studies that replicate the initial study with minor modifications. Particularly interesting results come to the attention of other researchers who conduct their own replications. The positive effect of expressive writing on health and the negative effect of cell phone usage on driving ability are examples of phenomena that have been replicated many times by many different researchers.

Sometimes a replication of a study produces results that differ from the results of the initial study. This could mean that the results of the initial study or the results of the replication were a fluke—they occurred by chance and do not reflect something that is generally true. In either case, additional replications would be likely to resolve this. A failure to produce the same results could also mean that the replication differed in some important way from the initial study. For example, early studies showed that people performed a variety of tasks better and faster when they were watched by others than when they were alone. Some later replications, however, showed that people performed worse when they were watched by others. Eventually researcher Robert Zajonc identified a key difference between the two types of studies. People seemed to perform better when being watched on highly practiced tasks but worse when being watched on relatively unpracticed tasks (Zajonc, 1965). [1] These two phenomena have now come to be called social facilitation and social inhibition.

What Is a Theory? A theory is a coherent explanation or interpretation of one or more phenomena. Although theories can take a variety of forms, one thing they have in common is that they go beyond the phenomena they explain by including variables, structures, processes, functions, or organizing principles that have not been observed directly. Consider, for example, Zajonc’s theory of social facilitation and social inhibition. He proposed that being watched by others while performing a task creates a general state of physiological arousal, which increases the likelihood of the dominant (most likely) response. So, for highly practiced tasks, being watched increases the tendency to make correct responses, but for relatively unpracticed tasks, being watched increases the tendency to make incorrect responses. Notice that this theory—which has come to be called drive theory—provides an explanation of both social facilitation and social inhibition that goes beyond the phenomena themselves by including concepts such as “arousal” and “dominant response,” along with processes such as the effect of arousal on the dominant response.

Outside of science, referring to an idea as a theory often implies that it is untested—perhaps no more than a wild guess. In science, however, the term theory has no such implication. A theory is simply an explanation or interpretation of a set of phenomena. It can be untested, but it can also be extensively tested, well supported, and accepted as an accurate description of the world by the scientific community. The theory of evolution by natural selection, for example, is a theory because it is an explanation of the diversity of life on earth—not because it is untested or unsupported by scientific research. On the contrary, the evidence for this theory is overwhelmingly positive and nearly all scientists accept its basic assumptions as accurate. Similarly, the “germ theory” of disease is a theory because it is an explanation of the origin of various diseases, not because there is any doubt that many diseases are caused by microorganisms that infect the body.

In addition to theory, researchers use several related terms to refer to their explanations and interpretations of phenomena. A perspective is a broad approach—more general than a theory—to explaining and interpreting phenomena. For example, researchers who take a biological perspective tend to explain phenomena in terms of genetics or nervous and endocrine system structures and processes, while researchers who take a behavioral perspective tend to explain phenomena in terms of reinforcement, punishment, and other external events. A model is a precise explanation or interpretation of a specific phenomenon—often expressed in terms of equations, computer programs, or biological structures and processes. A hypothesis can be an explanation that relies on just a few key concepts—although this term

RESEARCH FOR HUMAN SERVICES pg. 51

more commonly refers to a prediction about a new phenomenon based on a theory (see Section 4.3 "Using Theories in Psychological Research"). Adding to the confusion is the fact that researchers often use these terms interchangeably. It would not be considered wrong to refer to the drive theory as the drive model or even the drive hypothesis. And the biopsychosocial model of health—the general idea that health is determined by an interaction of biological, psychological, and social factors—is really more like a perspective as defined here. Keep in mind, however, that the most important distinction remains that between observations and interpretations.

What Are Theories For? Of course, scientific theories are meant to provide accurate explanations or interpretations of phenomena. But there must be more to it than this. Consider that a theory can be accurate without being very useful. To say that expressive writing helps people “deal with their emotions” might be accurate as far as it goes, but it seems too vague to be of much use. Consider also that a theory can be useful without being entirely accurate. Figure 4.2 "Representation of the Multistore Model of Human Memory" is a representation of the classic multistore model of human memory, which is still cited by researchers and discussed in textbooks despite the fact that it is now known to be inaccurate in a number of ways (Izawa, 1999). [2] These two examples suggest that theories have purposes other than simply providing accurate explanations or interpretations. Here we look at three additional purposes of theories: the organization of known phenomena, the prediction of outcomes in new situations, and the generation of new research.

FIGURE 4.2 REPRESENTATION OF THE MULTISTORE MODEL OF HUMAN MEMORY. INFORMATION FROM THE ENVIRONMENT GOES INTO THE SENSORY STORE, THEN THE SHORT-TERM STORE, AND THEN THE LONG-TERM STORE. IT CAN BE FORGOTTEN IN ANY OF THOSE THREE STORES. IN ORDER TO BE RETRIEVED, MEMORY IN THE LONG-TERM STORE MUST BE RECALLED BACK IN THE SHORT-TERM STORE FOR IT TO BE “REMEMBERED.”

In the multistore model of human memory, information from the environment passes through a sensory store on its way to a short-term store, where it can be rehearsed, and then to a long-term store, where it can be stored and retrieved much later. This theory has been extremely successful at organizing old phenomena and predicting new ones.

Organization One important purpose of scientific theories is to organize phenomena in ways that help people think about them clearly and efficiently. The drive theory of social facilitation and social inhibition, for example, helps to organize and make sense of a large number of seemingly contradictory results. The multistore model of human memory efficiently summarizes many important phenomena: the limited capacity and short retention time of information that is attended to but not rehearsed, the importance of rehearsing information for long- term retention, the serial-position effect, and so on. Or consider a classic theory of intelligence represented by Figure 4.3 "Representation of One Theory of Intelligence". According to this theory, intelligence consists of a general mental ability, g, plus a small number of more specific abilities that are influenced by g (Neisset et al., 1996). [3] Although there are other theories of intelligence, this one does a good job of summarizing a large number of statistical relationships between tests of various mental abilities. This includes the fact that tests of all basic mental abilities tend to be somewhat positively correlated and the fact that certain subsets of mental abilities (e.g., reading comprehension and analogy completion) are more positively correlated than others (e.g., reading comprehension and arithmetic).

RESEARCH FOR HUMAN SERVICES pg. 52

FIGURE 4.3 REPRESENTATION OF ONE THEORY OF INTELLIGENCE. INTELLIGENCE ABILITY, CALLED “G” IN THIS THEORY IS DERIVED FROM ONE OF THREE ABILITIES: NUMERICAL, SPATIAL, AND VERBAL. THE ABILITY OF ANY ONE PERSON CAN VARY GREATLY FROM ONE NATURAL ABILITY TO ANOTHER, AND CERTAINLY FROM ONE PERSON TO ANOTHER.

In this theory of intelligence, a general mental ability (g) influences each of three more specific mental abilities. Theories of this type help to organize a large number of statistical relationships among tests of various mental abilities.

Thus, theories are good or useful to the extent that they organize more phenomena with greater clarity and efficiency. Scientists generally follow the principle of parsimony, which holds that a theory should include only as many concepts as are necessary to explain or interpret the phenomena of interest. Simpler, more parsimonious theories organize phenomena more efficiently than more complex, less parsimonious theories.

Occam’s Razor & Parsimony Attributed to William of Ockham (c. 1287-1374) is Ocam’s Razor. It is the principle that: Among competing hypotheses, the one with the fewest assumptions should be selected. Scientists generally follow the principle of parsimony, which is in general agreement with the views of William of Ockham. Parsimony holds that a theory should include only as many concepts as are necessary to explain or interpret the phenomena of interest. Simpler, more parsimonious theories organize phenomena more efficiently than more complex, less parsimonious theories.

Prediction A second purpose of theories is to allow researchers and others to make predictions about what will happen in new situations. For example, a gymnastics coach might wonder whether a student’s performance is likely to be better or worse during a competition than when practicing alone. Even if this particular question has never been studied empirically, Zajonc’s drive theory suggests an answer. If the student generally performs with no mistakes, she is likely to perform better during competition. If she generally performs with many mistakes, she is likely to perform worse.

In clinical psychology, treatment decisions are often guided by theories. Consider, for example, dissociative identity disorder (formerly called multiple personality disorder). The prevailing scientific theory of dissociative identity disorder is that people develop multiple personalities (also called alters) because they are familiar with this idea from popular portrayals (e.g., the movie Sybil) and because they are unintentionally encouraged to do so by their clinicians (e.g., by asking to “meet” an alter). This theory

RESEARCH FOR HUMAN SERVICES pg. 53

implies that, rather than encouraging patients to act out multiple personalities, treatment should involve discouraging them from doing this (Lilienfeld & Lynn, 2003). [4]

Generation of New Research A third purpose of theories is to generate new research by raising new questions. Consider, for example, the theory that people engage in self-injurious behavior such as cutting because it reduces negative emotions such as sadness, anxiety, and anger. This theory immediately suggests several new and interesting questions. Is there, in fact, a statistical relationship between cutting and the amount of negative emotions experienced? Is it causal? If so, what is it about cutting that has this effect? Is it the pain, the sight of the injury, or something else? Does cutting affect all negative emotions equally?

Notice that a theory does not have to be accurate to serve this purpose. Even an inaccurate theory can generate new and interesting research questions. Of course, if the theory is inaccurate, the answers to the new questions will tend to be inconsistent with the theory. This will lead researchers to reevaluate the theory and either revise it or abandon it for a new one. And this is how scientific theories become more detailed and accurate over time.

Multiple Theories: Competing and Complementary At any point in time, researchers are usually considering multiple theories for any set of phenomena. One reason is that because human behavior is extremely complex, it is always possible to look at it from different perspectives. For example, a biological theory of sexual orientation might focus on the role of sex hormones during critical periods of brain development, while a sociocultural theory might focus on cultural factors that influence how underlying biological tendencies are expressed. A second reason is that—even from the same perspective—there are usually different ways to “go beyond” the phenomena of interest. For example, in addition to the drive theory of social facilitation and social inhibition, there is another theory that explains them in terms of a construct called “evaluation apprehension”—anxiety about being evaluated by the audience. Both theories go beyond the phenomena to be interpreted, but they do so by proposing somewhat different underlying processes.

Different theories of the same set of phenomena can be complementary—with each one supplying one piece of a larger puzzle. A biological theory of sexual orientation and a sociocultural theory of sexual orientation might accurately describe different aspects of the same complex phenomenon. Similarly, social facilitation could be the result of both general physiological arousal and evaluation apprehension. But different theories of the same phenomena can also be competing in the sense that if one is accurate, the other is probably not. For example, an alternative theory of dissociative identity disorder—the posttraumatic theory—holds that alters are created unconsciously by the patient as a means of coping with sexual abuse or some other traumatic experience. Because the socio-cognitive theory and the posttraumatic theories attribute dissociative identity disorder to fundamentally different processes, it seems unlikely that both can be accurate. See Note 4.10 "Where Do Multiple Personalities Come From?" for more on these competing theories.

The fact that there are multiple theories for any set of phenomena does not mean that any theory is as good as any other or that it is impossible to know whether a theory provides an accurate explanation or interpretation. On the contrary, scientists are continually comparing theories in terms of their ability to organize phenomena, predict outcomes in new situations, and generate research. Those that fare poorly are assumed to be less accurate and are abandoned, while those that fare well are assumed to be more accurate and are retained and compared with newer—and hopefully better—theories. Although scientists generally do not believe that their theories ever provide perfectly accurate descriptions of the world, they do assume that this process produces theories that come closer and closer to that ideal.

Example of Competing Theories: Where Do Multiple Personalities Come From? The literature on dissociative identity disorder (DID) features two competing theories. The socio-cognitive theory is that DID comes about because patients are aware of the disorder, know its characteristic features, and are encouraged to take on multiple personalities by their therapists. The posttraumatic theory is that multiple personalities develop as a way of coping with sexual abuse or some other trauma. There are now

RESEARCH FOR HUMAN SERVICES pg. 54

several lines of evidence that support the socio-cognitive model over the posttraumatic model (Lilienfeld & Lynn, 2003). [5]

 Diagnosis of DID greatly increased after the release of the book and film Sybil—about a woman with DID—in the 1970s.

 DID is extremely rare outside of North America.

 A very small percentage of therapists are responsible for diagnosing the vast majority of cases of DID.

The literature on treating DID includes many practices that encourage patients to act out multiple personalities (e.g., having a bulletin board on which personalities can leave messages for each other).

Normal people can sometimes exhibit the symptoms of DID with minimal suggestion in simulated clinical interviews. Many experienced practitioners consider DID a phenomenon created by therapist who are inclined to see the disorder lurking in many, if not all, of their clients – which is, in itself, yet another theory to consider. AYS

 Scientists distinguish between phenomena, which are their systematic observations, and theories, which are their explanations or interpretations of phenomena.

 In addition to providing accurate explanations or interpretations, scientific theories have three basic purposes. They organize phenomena, allow people to predict what will happen in new situations, and help generate new research.

 Researchers generally consider multiple theories for any set of phenomena. Different theories of the same set of phenomena can be complementary or competing.

Practice 1. Think of at least three different theories to explain the fact that married people tend to report greater

levels of happiness than unmarried people. 2. Find a recent article in a professional journal and do two things: 3. Identify the primary phenomenon of interest. 4. Identify the theory or theories used to explain or interpret that phenomenon.

Discussion: Can a theory be useful even if it is inaccurate? How?

[1] Zajonc, R. B. (1965). Social facilitation. Science, 149, 269–274.

[2] Izawa, C. (Ed.) (1999). On human memory: Evolution, progress, and reflections on the 30th anniversary of the Atkinson-Shiffrin model. Mahwah, NJ: Erlbaum.

[3] Neisser, U., Boodoo, G., Bouchard, T. J., Boykin, A. W., Brody, N., Ceci, Urbina, S. (1996). Intelligence: Knowns and unknowns. American Psychologist, 51, 77–101.

[4] Lilienfeld, S. O., & Lynn, S. J. (2003). Dissociative identity disorder: Multiple personalities, multiple controversies. In S. O. Lilienfeld, S. J. Lynn, & J. M. Lohr (Eds.), Science and pseudoscience in clinical psychology (pp. 109–142). New York, NY: Guilford Press.

[5] Lilienfeld, S. O., & Lynn, S. J. (2003). Dissociative identity disorder: Multiple personalities, multiple controversies. In S. O. Lilienfeld, S. J. Lynn, & J. M. Lohr (Eds.), Science and pseudoscience in clinical psychology (pp. 109–142). New York, NY: Guilford Press.

RESEARCH FOR HUMAN SERVICES pg. 55

4.2 The Variety of Theories

LEARNING OBJECTIVES  Describe three dimensions along which theories vary.

 Give examples of several different types of theories commonly used in Human Services research.

Researchers have found that many different types of theories can help them to organize phenomena, predict what will happen in new situations, and generate new research. It is important for beginning researchers to be aware of the different types so that they recognize theories when they see them in the research literature. (They are not always clearly labeled as “theories.”) It is also important for them to see that some types of theories are well within their ability to understand, use, and even construct. In this section, we look at the variety of psychological theories in terms of three important dimensions: formality, scope, and theoretical approach.

Formality Theories vary widely in their formality—the extent to which the components of the theory and the relationships among them are specified clearly and in detail. At the informal end of this dimension are theories that consist of simple verbal descriptions of a few important components and relationships. The habituation theory of expressive-writing effects on health is relatively informal in this sense. So is the drive theory of social facilitation and inhibition. At the more precise, formal end of this dimension are theories that are expressed in terms of mathematical equations or computer programs.

Formal Theories People who are not familiar with the scientific approach are sometimes surprised to learn that theories can take the form of mathematical equations and computer programs. The following formal theories are among the best known and most successful in the field.

 ACT-R. A comprehensive theory of human cognition that is akin to a programming language, within which more specific models can be created. See http://act-r.psy.cmu.edu.

 Prospect theory. A formal theory of decision making under uncertainty. Psychologist Daniel Kahneman won the Nobel Prize in economics based in part on prospect theory. Read about Kahneman’s Nobel Prize work at http://www.nobelprize.org/nobel_prizes/economics/laureates/2002/kahneman-autobio.html.

 Rescorla-Wagner model. A theory of classical conditioning that features an equation describing how the strength of the association between unconditioned and conditioned stimuli changes when the two are paired. For more on this formal theory—including an interactive version— see http://psych.hanover.edu/javatest/rescrolawagner.

Both informal and formal theories have their place in psychological research. Informal theories tend to be easier to create and to understand but less precise in their predictions, which can make them more difficult to test. They are especially appropriate, however, in the early stages of research when the phenomena of interest has not yet been described in detail. Formal theories tend to be more difficult to create and to understand—sometimes requiring a certain amount of mathematical or computer programming background—but they also tend to be more precise in their predictions and therefore easier to test.

Scope Theories also vary widely in their scope—the number and diversity of the phenomena they explain or interpret. Many early psychological theories were extremely broad in that they attempted to interpret essentially all human behavior. Freud and his followers, for example, applied his theory not only to understanding psychological disorders but also to slips of the tongue and other everyday errors, dreaming, sexuality, art, politics, and even civilization itself (Fine, 1979). [1] Such theories have fallen out of favor in

RESEARCH FOR HUMAN SERVICES pg. 56

scientific psychology, however, because they tend to be imprecise and difficult to test. In addition, they have not been particularly successful at organizing or predicting the range and complexity of human behavior at the level of detail that scientific researchers usually seek.

Still, contemporary theories can vary in their scope. At the broad end of this dimension are theories that apply to many diverse phenomena. Cognitive dissonance theory, for example, assumes that when people hold inconsistent beliefs, this creates mental discomfort that they are motivated to reduce by changing one or both beliefs. This theory has been applied to a wide variety of phenomena, including the persistence of irrational beliefs and behaviors (e.g., smoking), the effectiveness of certain persuasion and sales techniques (e.g., asking for a small favor before asking for a big one), and even placebo effects. At the narrow end of this dimension are theories that apply to a small number of closely related phenomena. Consider, for example, a very specific quantitative ability called subitizing. This refers to people’s ability to quickly and accurately perceive the number of objects in a scene without counting them—as long as the number is four or fewer. Several theories have been proposed to explain subitizing. Among them is the idea that small numbers of objects are associated with easily recognizable patterns. For example, people know immediately that there are three objects in a scene because the three objects tend to form a “triangle” and it is this pattern that is quickly perceived (Logan & Sbrodoff, 2003). [2]

As with informal and formal theories, both broad and narrow theories have their place in psychological research. Broad theories organize more phenomena but tend to be less formal and less precise in their predictions. Narrow theories organize fewer phenomena but tend to be more formal and more precise in their predictions.

Theoretical Approach In addition to varying in formality and scope, the theories we use in Human Services vary widely in the kinds of theoretical ideas they are constructed from as well as from the different disciplines from which they originate. We will refer to this as the theoretical approach.

Functional Theories Functional theories explain psychological phenomena in terms of their function or purpose. For example, one prominent theory of repeated self-injury (e.g., cutting) is that people do it because it produces a short- term reduction in the intensity of negative emotions that they are feeling (Tantam & Huband, 2009). [3] Note that this theory does not focus on how this happens, but on the function of self-injury for the people who engage in it. Theories from the perspective of evolutionary psychology also tend to be functional— assuming that human behavior has evolved to solve specific adaptive problems faced by our distant ancestors. Consider the phenomenon of sex differences in human mating strategies (Buss & Schmitt, 1993). [4] Men are somewhat more likely than women to seek short-term partners and to value physical attractiveness over material resources in a mate. Women are somewhat more likely than men to seek long- term partners and to value material resources over physical attractiveness in a mate. But why? The standard evolutionary theory holds that because the male investment in becoming a parent is relatively small, men reproduce more successfully by seeking several short-term partners who are young and healthy (which is signaled by physical attractiveness). But because the female investment in becoming a parent is quite large, women reproduce more successfully by seeking a long-term partner who has resources to contribute to raising the child.

Mechanistic Theories Mechanistic theories, on the other hand, focus on specific variables, structures, and processes, and how they interact to produce the phenomena. The drive theory of social facilitation and inhibition and the multistore model of human memory are mechanistic theories in this sense. Figure 4.4 "Simplified Representation of One Contemporary Theory of Hypochondriasis" represents another example—a contemporary cognitive theory of hypochondriasis—an extreme form of health anxiety in which people misinterpret ordinary bodily symptoms (e.g., headaches) as signs of a serious illness (e.g., a brain tumor; Williams, 2004). [5] This theory specifies several key variables and the relationships among them. Specifically, people who are high in the personality trait of neuroticism (also called negative emotionality) start to pay excessive attention to negative health information—especially if they have had a significant

RESEARCH FOR HUMAN SERVICES pg. 57

illness experience as a child (e.g., a seriously ill parent). This attention to negative health information then leads to health anxiety and hypochondriasis, especially among people who are low in effortful control, which is the ability to shift attention away from negative thoughts and feelings.

Figure 4.4 Simplified Representation of One Contemporary Theory of Hypochondriasis. In this theory neuroticism can lead one to experience illness, leading one to attention and preoccupation with negative health information which causes one to try to exert control over their health, which leads to health anxiety and the illness we call hypochondriasis. That is, they are a hypochondriac. Remember this is just a theory, and a simplified one at that. A theory does not a fact make.

This theory focuses on key variables and the relationships among them.

Mechanistic theories can also be expressed in terms of biological structures and processes. With advances in genetics and neuroscience, such theories are becoming increasingly common in Human Services. For example, researchers are currently constructing and testing theories that specify the brain structures associated with the storage and rehearsal of information in the short-term store, the transfer of information to the long-term store, and so on. Theories of psychological disorders are also increasingly likely to focus on biological mechanisms. Schizophrenia, for example, has been explained in terms of several biological theories, including theories that focus on genetics, neurotransmitters, brain structures, and even prenatal exposure to infections.

Stage Theories Finally, there are also theoretical approaches that provide organization without necessarily providing a functional or mechanistic explanation. These include stage theories, which specify a series of stages that people pass through as they develop or adapt to their environment. Famous stage theories include Abraham Maslow’s hierarchy of needs and Jean Piaget’s theory of cognitive development.

Typologies provide organization by categorizing people or behavior into distinct types. These include theories that identify several basic emotions (e.g., happiness, sadness, fear, surprise, anger, and disgust), several distinct types of intelligence (e.g., spatial, linguistic, mathematical, kinesthetic, musical, interpersonal, and intrapersonal), and distinct types of personalities (e.g., Type A vs. Type B).

Summary

 Researchers in many fields have found that there is a place for all these theoretical approaches. In fact, multiple approaches are probably necessary to provide a complete understanding of any set of

RESEARCH FOR HUMAN SERVICES pg. 58

phenomena. A complete understanding of emotions, for example, is likely to require identifying the basic emotions that people experience, explaining why we have those emotions, and describing how those emotions work in terms of underlying psychological and biological variables, structures, and processes. M OR E C OM PLEX , LE SS P AR S IM O N IOU S TH EOR IE S

 Theories vary widely in terms of their formality, scope, and theoretical approach. The different types of theories all play important roles in psychological research.

Practice: Find an empirical research report in a professional journal, identify a theory that the researchers present, and then describe the theory in terms of its formality (informal vs. formal), scope (broad vs. narrow), and theoretical approach (functional, mechanistic, etc.).

Discussion: Do you think there will ever be a single theory that explains all psychological disorders? Why or why not?

[1] Fine, R. (1979). A history of psychoanalysis. New York, NY: Columbia University Press.

[2] Logan, G. D., & Sbrodoff, N. J. (2003). Subitizing and similarity: Toward a pattern-matching theory of enumeration. Psychonomic Bulletin & Review, 10, 676–682.

[3] Tantam, D., & Huband, N. (2009). Understanding repeated self-injury: A multidisciplinary approach. New York, NY: Palgrave Macmillan.

[4] Buss, D. M., & Schmitt, D. P. (1993). Sexual strategies theory: A contextual evolutionary analysis of human mating. Psychological Review, 100, 204–232.

[5] Williams, P. G. (2004). The psychopathology of self-assessed health: A cognitive approach to health anxiety and hypochondriasis. Cognitive Therapy and Research, 28, 629–644.

4.3 Using Theories in Social Research

LEARNING OBJECTIVES  Explain how researchers test their theories, and give a concrete example.

 Explain how psychologists reevaluate theories in light of new results, including some of the complications involved.

 Describe several ways to incorporate theory into your own research.

We have now seen what theories are, what they are for, and the variety of forms that they take in psychological research. In this section, we look more closely at how researchers actually use them. We begin with a general description of how researchers test and revise their theories, and we end with some practical advice for beginning researchers who want to incorporate theory into their research.

Theory Testing and Revision

RESEARCH FOR HUMAN SERVICES pg. 59

A theory can be proved by experiment; but no path leads from experiment to the birth of a theory. — Albert Einstein

Hypothetical-Deductive method The primary way that scientific researchers use theories is sometimes called the hypothetico- deductive method (although this term is much more likely to be used by philosophers of science than by scientists themselves). A researcher begins with a set of phenomena and either constructs a theory to explain or interpret them or chooses an existing theory to work with. He or she then makes a prediction about some new phenomenon that should be observed if the theory is correct. Again, this prediction is called a hypothesis. The researcher then conducts an empirical study to test the hypothesis. Finally, he or she reevaluates the theory in light of the new results and revises it if necessary. This process is usually conceptualized as a cycle because the researcher can then derive a new hypothesis from the revised theory, conduct a new empirical study to test the hypothesis, and so on. As Figure 4.5 "Hypothetico- Deductive Method Combined With the General Model of Scientific Research in Psychology" shows, this approach meshes nicely with the model of scientific research in psychology presented earlier in the book— creating a more detailed model of “theoretically motivated” or “theory-driven” research. Together they form a model of theoretically motivated research.

As an example, let us return to Zajonc’s research on social facilitation and inhibition. He started with a somewhat contradictory pattern of results from the research literature. He then constructed his drive theory, according to which being watched by others while performing a task causes physiological arousal, which increases an organism’s tendency to make the dominant response. This leads to social facilitation for well- learned tasks and social inhibition for poorly learned tasks. He now had a theory that organized previous results in a meaningful way—but he still needed to test it. He hypothesized that if his theory was correct, he should observe that the presence of others improves performance in a simple laboratory task but inhibits performance in a difficult version of the very same laboratory task. To test this hypothesis, one of the studies he conducted used cockroaches as subjects (Zajonc, Heingartner, & Herman, 1969). [1] The cockroaches ran either down a straight runway (an easy task for a cockroach) or through a cross-shaped maze (a difficult task for a cockroach) to escape into a dark chamber when a light was shined on them. They did this either while alone or in the presence of other cockroaches in clear plastic “audience boxes.” Zajonc found that cockroaches in the straight runway reached their goal more quickly in the presence of other cockroaches, but cockroaches in the cross-shaped maze reached their goal more slowly when they were in the presence of other cockroaches. Thus, he confirmed his hypothesis and provided support for his drive theory.

Constructing or Choosing a Theory Along with generating research questions, constructing theories is one of the more creative parts of scientific research. But as with all creative activities, success requires preparation and hard work more than anything else. To construct a good theory, a researcher must know in detail about the phenomena of interest and about any existing theories based on a thorough review of the literature. The new theory must provide a coherent explanation or interpretation of the phenomena of interest and have some advantage over existing theories. It could be more formal and therefore more precise, broader in scope, more parsimonious, or it could take a new perspective or theoretical approach. If there is no existing theory, then almost any theory can be a step in the right direction.

As we have seen, formality, scope, and theoretical approach are determined in part by the nature of the phenomena to be interpreted. But the researcher’s interests and abilities play a role too. For example, constructing a theory that specifies the neural structures and processes underlying a set of phenomena requires specialized knowledge and experience in neuroscience (which most professional researchers would acquire in college and then, graduate school). But again, many theories in Human Services are relatively informal, narrow in scope, and expressed in terms that even a beginning researcher can understand and even use to construct his or her own new theory.

RESEARCH FOR HUMAN SERVICES pg. 60

It is probably more common, however, for a researcher to start with a theory that was originally constructed by someone else—giving due credit to the originator of the theory. This is another example of how researchers work collectively to advance scientific knowledge. Once they have identified an existing theory, they might derive a hypothesis from the theory and test it or modify the theory to account for some new phenomenon and then test the modified theory.

Deriving A Hypotheses Again, a hypothesis is a prediction about a new phenomenon that should be observed if a particular theory is accurate. Theories and hypotheses always have this if-then relationship. “If drive theory is correct, then cockroaches should run through a straight runway faster, and a branching runway more slowly, when other cockroaches are present.” Although hypotheses are usually expressed as statements, they can always be rephrased as questions. “Do cockroaches run through a straight runway faster when other cockroaches are present?” Thus, deriving hypotheses from theories is an excellent way of generating interesting research questions.

But how do researchers derive hypotheses from theories? One way is to generate a research question using the techniques discussed in Chapter 2 "Getting Started in Research" and then ask whether any theory implies an answer to that question. For example, you might wonder whether expressive writing about positive experiences improves health as much as expressive writing about traumatic experiences. Although this is an interesting question on its own, you might then ask whether the habituation theory—the idea that expressive writing causes people to habituate to negative thoughts and feelings—implies an answer. In this case, it seems clear that if the habituation theory is correct, then expressive writing about positive experiences should not be effective because it would not cause people to habituate to negative thoughts and feelings. A second way to derive hypotheses from theories is to focus on some component of the theory that has not yet been directly observed. For example, a researcher could focus on the process of habituation—perhaps hypothesizing that people should show fewer signs of emotional distress with each new writing session.

Among the very best hypotheses are those that distinguish between competing theories. For example, Norbert Schwarz and his colleagues considered two theories of how people make judgments about themselves, such as how assertive they are (Schwarz et al., 1991). [2] Both theories held that such judgments are based on relevant examples that people bring to mind. However, one theory was that people base their judgments on the number of examples they bring to mind and the other was that people base their judgments on how easily they bring those examples to mind. To test these theories, the researchers asked people to recall either six times when they were assertive (which is easy for most people) or 12 times (which is difficult for most people). Then they asked them to judge their own assertiveness. Note that the number-of-examples theory implies that people who recalled 12 examples should judge themselves to be more assertive because they recalled more examples, but the ease-of-examples theory implies that participants who recalled six examples should judge themselves as more assertive because recalling the examples was easier. Thus. the two theories made opposite predictions so that only one of the predictions could be confirmed. The surprising result was that participants who recalled fewer examples judged themselves to be more assertive—providing particularly convincing evidence in favor of the ease-of- retrieval theory over the number-of-examples theory.

Evaluating and Revising Theories If a hypothesis is confirmed in a systematic empirical study, then the theory has been strengthened. Not only did the theory make an accurate prediction, but there is now a new phenomenon that the theory accounts for. If a hypothesis is disconfirmed in a systematic empirical study, then the theory has been weakened. It made an inaccurate prediction, and there is now a new phenomenon that it does not account for.

Although this seems straightforward, there are some complications. First, confirming a hypothesis can strengthen a theory but it can never prove a theory. In fact, scientists tend to avoid the word “prove” when talking and writing about theories. One reason for this is that there may be other plausible theories that imply the same hypothesis, which means that confirming the hypothesis strengthens all those theories

RESEARCH FOR HUMAN SERVICES pg. 61

equally. A second reason is that it is always possible that another test of the hypothesis or a test of a new hypothesis derived from the theory will be disconfirmed. This is a version of the famous philosophical “problem of induction.” One cannot definitively prove a general principle (e.g., “All swans are white.”) just by observing confirming cases (e.g., white swans)—no matter how many. It is always possible that a disconfirming case (e.g., a black swan) will eventually come along. For these reasons, scientists tend to think of theories—even highly successful ones—as subject to revision based on new and unexpected observations.

A second complication has to do with what it means when a hypothesis is disconfirmed. According to the strictest version of the hypothetico-deductive method, disconfirming a hypothesis disproves the theory it was derived from. In formal logic, the premises “if A then B” and “not B” necessarily lead to the conclusion “not A.” If A is the theory and B is the hypothesis (“if A then B”), then disconfirming the hypothesis (“not B”) must mean that the theory is incorrect (“not A”). In practice, however, scientists do not give up on their theories so easily. One reason is that one disconfirmed hypothesis could be a fluke or it could be the result of a faulty research design. Perhaps the researcher did not successfully manipulate the independent variable or measure the dependent variable. A disconfirmed hypothesis could also mean that some unstated but relatively minor assumption of the theory was not met. For example, if Zajonc had failed to find social facilitation in cockroaches, he could have concluded that drive theory is still correct but it applies only to animals with sufficiently complex nervous systems.

This does not mean that researchers are free to ignore disconfirmations of their theories. If they cannot improve their research designs or modify their theories to account for repeated disconfirmations, then they eventually abandon their theories and replace them with ones that are more successful.

Incorporating Theory into Your Practice It should be clear from this chapter that theories are a basic ingredient of empirically based practice. Remember, empirical practice is using scientifically validated findings to help guide your work. If you can understand and use them, you will be much more successful at reading and understanding the research literature, generating interesting research questions, and writing and conversing about research. Of course, your ability to understand and use theories will improve with time. But there are several things that you can do to incorporate theory into your research right from the start.

The first thing is to distinguish the phenomena you are interested in from any theories of those phenomena. Beware especially of the tendency to “fuse” a phenomenon to a commonsense theory of it. For example, it might be tempting to describe the negative effect of cell phone usage on driving ability by saying, “Cell phone usage distracts people from driving.” Or it might be tempting to describe the positive effect of expressive writing on health by saying, “Dealing with your emotions through writing makes you healthier.” In both of these examples, however, a vague commonsense explanation (distraction, “dealing with” emotions) has been fused to the phenomenon itself. The problem is that this gives the impression that the phenomenon has already been adequately explained and closes off further inquiry into precisely why or how it happens.

As another example, researcher Jerry Burger and his colleagues were interested in the phenomenon that people are more willing to comply with a simple request from someone with whom they are familiar (Burger, Soroka, Gonzago, Murphy, & Somervell, 1999). [3] A beginning researcher who is asked to explain why this is the case might be at a complete loss or say something like, “Well, because they are familiar with them.” But digging just a bit deeper, Burger and his colleagues realized that there are several possible explanations. Among them are that complying with people we know creates positive feelings, that we anticipate needing something from them in the future, and that we like them more and follow an automatic rule that says to help people we like.

The next thing to do is turn to the research literature to identify existing theories of the phenomena you are interested in. Remember that there will usually be more than one plausible theory. Existing theories may be complementary or competing, but it is essential to know what they are. If there are no existing theories, you should come up with two or three of your own—even if they are informal and limited in scope. Then get in

RESEARCH FOR HUMAN SERVICES pg. 62

the habit of describing the phenomena you are interested in, followed by the two or three best theories of it. Do this whether you are speaking or writing about your research. When asked what their research was about, for example, Burger and his colleagues could have said something like the following:

It’s about the fact that we’re more likely to comply with requests from people we know [the phenomenon]. This is interesting because it could be because it makes us feel good [Theory 1], because we think we might get something in return [Theory 2], or because we like them more and have an automatic tendency to comply with people we like [Theory 3].

At this point, you may be able to derive a hypothesis from one of the theories. At the very least, for each research question you generate, you should ask what each plausible theory implies about the answer to that question. If one of them implies a particular answer, then you may have an interesting hypothesis to test. Burger and colleagues, for example, asked what would happen if a request came from a stranger whom participants had sat next to only briefly, did not interact with, and had no expectation of interacting with in the future. They reasoned that if familiarity created liking, and liking increased people’s tendency to comply (Theory 3), then this situation should still result in increased rates of compliance (which it did). If the question is interesting but no theory implies an answer to it, this might suggest that a new theory needs to be constructed or that existing theories need to be modified in some way. These would make excellent points of discussion in the introduction or discussion of an American Psychological Association (APA) style research report or research presentation.

Be aware that there are two basic ways that researchers usually include theory. The first is to raise a research question, answer that question by conducting a new study, and then offer one or more theories (usually more) to explain or interpret the results. This format works well for applied research questions and for research questions that existing theories do not address. The second way is to describe one or more existing theories, derive a hypothesis from one of those theories, test the hypothesis in a new study, and finally reevaluate the theory. This format works well when there is an existing theory that addresses the research question—especially if the resulting hypothesis is surprising or conflicts with a hypothesis derived from a different theory.

Incorporating Theory into Practice Empirically based practice or sometimes called evidence based practice, mentioned several times previously, is the ability to incorporate the latest relevant developments in the social sciences directly into your practice. One major goal of this text is to prepare you to do just that by finding, evaluating, communicating and applying, empirically based information in your direct practice.

E AW AY S

Summary

 In research, and in our actual practice, we use the hypothetico-deductive method. They construct theories to explain or interpret phenomena (or work with existing theories), derive hypotheses from their theories, test the hypotheses, and then reevaluate the theories in light of the new results.

 There are several things that even beginning researchers can do to incorporate theory into their research. These include clearly distinguishing phenomena from theories, knowing about existing theories, constructing one’s own simple theories, using theories to make predictions about the answers to research questions, and incorporating theories into one’s writing and speaking.

EX ER C IS E

Practice: Find a recent empirical research report in a professional journal. Read the introduction and highlight in different colors descriptions of phenomena, theories, and hypotheses.

RESEARCH FOR HUMAN SERVICES pg. 63

[1] Zajonc, R. B., Heingartner, A., & Herman, E. M. (1969). Social enhancement and impairment of performance in the cockroach. Journal of Personality and Social Psychology, 13, 83–92.

[2] Schwarz, N., Bless, H., Strack, F., Klumpp, G., Rittenauer-Schatka, H., & Simons, A. (1991). Ease of retrieval as information: Another look at the availability heuristic. Journal of Personality and Social Psychology, 61, 195–202.

[3] Burger, J. M., Soroka, S., Gonzago, K., Murphy, E., & Somervell, E. (1999). The effect of fleeting attraction on compliance to requests. Personality and Social Psychology Bulletin, 27, 1578–1586.

RESEARCH FOR HUMAN SERVICES pg. 64

Chapter 5: Measurement The heart of science is measurement. -Erik Brynjolfsson

Researchers Tara MacDonald and Alanna Martineau were interested in the effect of female college students’ moods on their intentions to have unprotected sexual intercourse (MacDonald & Martineau, 2002). [1] In a carefully designed empirical study, they found that being in a negative mood increased intentions to have unprotected sex—but only for students who were low in self-esteem.

Although there are many challenges involved in conducting a study like this, one of the primary ones is the measurement of the relevant variables. In this study, the researchers needed to know whether each of their participants had high or low self-esteem, which of course required measuring their self-esteem. They also needed to be sure that their attempt to put people into a negative mood (by having them think negative thoughts) was successful, which required measuring their moods. Finally, they needed to see whether self- esteem and mood were related to participants’ intentions to have unprotected sexual intercourse, which required measuring these intentions.

Clinicians also use surveys and other measurements, There are thousands available to measure everything from anxiety to marital satisfaction. Hundreds are readily available to the Human Service worker that are in the public domain. One popular one is the Rosenberg Self-Esteem Scale.

Do You Feel You Are a Person of Worth? The Rosenberg Self-Esteem Scale (Rosenberg, 1989) [2] is one of the most common measures of self- esteem and the one that MacDonald and Martineau used in their study.

Participants respond to each of the 10 items that follow with a rating on a 4-point Likert scale: Strongly Agree, Agree, Disagree, Strongly Disagree. Score Items 1, 2, 4, 6, and 7 by assigning 3 points for each Strongly Agree response, 2 for each Agree, 1 for each Disagree, and 0 for each Strongly Disagree. Reverse the scoring for Items 3, 5, 8, 9, and 10 by assigning 0 points for each Strongly Agree, 1 point for each Agree, and so on. The overall score is the total number of points. The actual form is included in the appendix, and the questions used for the scale are:

 I feel that I’m a person of worth, at least on an equal plane with others.

 I feel that I have a number of good qualities.

 All in all, I am inclined to feel that I am a failure.

 I am able to do things as well as most other people.

 I feel I do not have much to be proud of.

 I take a positive attitude toward myself.

 On the whole, I am satisfied with myself.

 I wish I could have more respect for myself.

 I certainly feel useless at times.

 At times, I think I am no good at all.

Likert scales are often used on surveys. Below is a five point Likert scale which allows the respondent to take a neutral stance. There are four point scales that force the respondent to agree or disagree one way or the other. The middle option is missing. Four and five point scales are the most common but occasionally we use surveys with more or less options available.

RESEARCH FOR HUMAN SERVICES pg. 65

FIGURE 3: FOUR POINT LIKERT SCALE FROM: HTTPS://UPLOAD.WIKIMEDIA.ORG/WIKIPEDIA/COMMONS/D/D7/EXAMPLE_LIKERT_SCALE.JPG

[1] MacDonald, T. K., & Martineau, A. M. (2002). Self-esteem, mood, and intentions to use condoms: When does low self-esteem lead to risky health behaviors? Journal of Experimental Social Psychology, 38, 299– 306.

[2] Rosenberg, M. (1989). Society and the adolescent self-image (rev. ed.). Middletown, CT: Wesleyan University Press.

RESEARCH FOR HUMAN SERVICES pg. 66

5.1 Understanding Clinical Measurement

LEARNING OBJECTIVES  Define measurement and give several examples of measurement relevant to Human Services

practice.

 Explain what a construct is and give several examples.

 Distinguish conceptual from operational definitions, give examples of each, and create simple operational definitions.

 Distinguish the four levels of measurement, give examples of each, and explain why this distinction is important.

What Is Measurement? Measurement is applying a score to a characteristic. In psychology, social work, and generally in the social sciences, measurement is very often the assignment of scores to individuals so that the scores represent some characteristic of the individuals.

This very general definition is consistent with the kinds of measurement that everyone is familiar with—for example, weighing oneself by stepping onto a bathroom scale, or checking the internal temperature of a roasting turkey by inserting a meat thermometer. It is also consistent with measurement throughout the sciences. In physics, for example, one might measure the potential energy of an object in Earth’s gravitational field by finding its mass and height (which of course requires measuring those variables) and then multiplying them together along with the gravitational acceleration of Earth (9.8 m/s2). The result of this procedure is a score that represents the object’s potential energy.

Of course, this general definition of measurement is consistent with measurement in the Human Services as well (in psychological measurement is often referred to as psychometrics). Imagine, for example, that a cognitive psychologist wants to measure a person’s working memory capacity—his or her ability to hold in mind and think about several pieces of information all at the same time. To do this, she might use a backward digit span task, where she reads a list of two digits to the person and asks him or her to repeat them in reverse order. She then repeats this several times, increasing the length of the list by one digit each time, until the person makes an error. The length of the longest list for which the person responds correctly is the score and represents his or her working memory capacity. Or imagine a clinical psychologist who is interested in how depressed a person is. He administers the Beck Depression Inventory, which is a 21-item self-report questionnaire in which the person rates the extent to which he or she has felt sad, lost energy, and experienced other symptoms of depression over the past 2 weeks. The sum of these 21 ratings is the score and represents his or her current level of depression.

The important point here is that measurement does not necessarily require any particular instruments or procedures. It does not require placing individuals or objects on bathroom scales, holding rulers up to them, or inserting thermometers into them. What it does require is some systematic procedure for assigning scores to individuals or objects so that those scores represent the characteristic of interest.

Constructs Many variables used by Human Services practitioners, and researchers, are straightforward and simple to measure. These include sex, age, height, weight, and birth order. You can almost always tell whether someone is male or female just by looking. You can ask people how old they are and be reasonably sure that they know and will tell you. Although people might not know or want to tell you how much they weigh, you can have them step onto a bathroom scale. Other variables studied by psychologists—perhaps the majority—are not so straightforward or simple to measure. We cannot accurately assess people’s level of intelligence by looking at them, and we certainly cannot put their self-esteem on a bathroom scale. These kinds of variables are called constructs (pronounced CON-structs) and include personality traits (e.g., extroversion), emotional states (e.g., fear), attitudes (e.g., toward taxes), and abilities (e.g., athleticism).

RESEARCH FOR HUMAN SERVICES pg. 67

Constructs cannot be observed directly. One reason is that they often represent tendencies to think, feel, or act in certain ways. For example, to say that a particular college student is highly extroverted (see Note 5.6 "The Big Five") does not necessarily mean that she is behaving in an extroverted way right now. In fact, she might be sitting quietly by herself, reading a book. Instead, it means that she has a general tendency to behave in extroverted ways (talking, laughing, etc.) across a variety of situations. Another reason constructs cannot be observed directly is that they often involve internal processes. Fear, for example, involves the activation of certain central and peripheral nervous system structures, along with certain kinds of thoughts, feelings, and behaviors—none of which is necessarily obvious to an outside observer. Notice also that neither extroversion nor fear “reduces to” any particular thought, feeling, act, or physiological structure or process. Instead, each is a kind of summary of a complex set of behaviors and internal processes.

Aspects of the Human Personality: The Big Five The Big Five is a conceptual version of five broad dimensions that capture much of the variation in human personality. Each of the Big Five can even be defined in terms of six more specific constructs called “facets” (Costa & McCrae, 1992).[1]

Figure 5.1 The Big Five Personality Dimensions

A conceptual definition is an idea, or the basic concept that defines some phenomenon, but it is amorphous. Not yet well defined. Such things as depression, and anxiety, or concepts. But, concepts are generally complex and multifaceted.

The conceptual definition of a psychological construct describes the behaviors and internal processes that make up that construct, along with how it relates to other variables. For example, a conceptual definition of neuroticism (another one of the Big Five) would be that it is people’s tendency to experience negative emotions such as anxiety, anger, and sadness across a variety of situations. This definition might also include that it has a strong genetic component, remains fairly stable over time, and is positively correlated with the tendency to experience pain and other physical symptoms.

Students sometimes wonder why, when researchers want to understand a construct like self-esteem or neuroticism, they do not simply look it up in the dictionary. One reason is that many scientific constructs do not have counterparts in everyday language (e.g., working memory capacity). More important, researchers are in the business of developing definitions that are more detailed and precise—and that more accurately describe the way the world is—than the informal definitions in the dictionary. As we will see, they do this by proposing conceptual definitions, testing them empirically, and revising them as necessary. Sometimes

RESEARCH FOR HUMAN SERVICES pg. 68

they throw them out altogether. This is why the research literature often includes different conceptual definitions of the same construct. In some cases, an older conceptual definition has been replaced by a newer one that works better. In others, researchers are still in the process of deciding which of various conceptual definitions is the best.

Operational Definitions In operationalizing we define the edges of the concept. It is an attempt to be precise, but often this precession is imposed. For example research in psychopharmacology, neuroscience, and psychotherapy are now strongly suggesting that there is more than one type of depression. So there is a certain amount of fine tuning or operational definitions as we learn more.

An operational definition is an attempt to me more precise with what the variable is in such a way that it can be measured. These measures generally fall into one of three broad categories.

1. Self-report measures are those in which participants report on their own thoughts, feelings, and actions, as with the Rosenberg Self-Esteem Scale.

2. Behavioral measures are those in which some other aspect of participants’ behavior is observed and recorded. This is an extremely broad category that includes the observation of people’s behavior both in highly structured laboratory tasks and in more natural settings. A good example of the former would be measuring working memory capacity using the backward digit span task. A good example of the latter is a famous operational definition of physical aggression from researcher Albert Bandura and his colleagues (Bandura, Ross, & Ross, 1961). [2] They let each of several children play for 20 minutes in a room that contained a clown-shaped punching bag called a Bobo doll. They filmed each child and counted the number of acts of physical aggression he or she committed. These included hitting the doll with a mallet, punching it, and kicking it. Their operational definition, then, was the number of these specifically defined acts that the child committed in the 20-minute period.

3. Physiological measures are those that involve recording any of a wide variety of physiological processes, including heart rate and blood pressure, galvanic skin response, hormone levels, and electrical activity and blood flow in the brain.

For any given variable or construct, there will be multiple operational definitions. Stress is a good example. A rough conceptual definition is that stress is an adaptive response to a perceived danger or threat that involves physiological, cognitive, affective, and behavioral components. But researchers have operationally defined it in several ways. The Social Readjustment Rating Scale is a self-report questionnaire on which people identify stressful events that they have experienced in the past year and assigns points for each one depending on its severity. For example, a man who has been divorced (73 points), changed jobs (36 points), and had a change in sleeping habits (16 points) in the past year would have a total score of 125. The Daily Hassles and Uplifts Scale is similar but focuses on everyday stressors like misplacing things and being concerned about one’s weight. The Perceived Stress Scale is another self-report measure that focuses on people’s feelings of stress (e.g., “How often have you felt nervous and stressed?”). Researchers have also operationally defined stress in terms of several physiological variables including blood pressure and levels of the stress hormone cortisol.

Converging Operations When social scientist use multiple operational definitions of the same construct—either within a study or across studies—they are using converging operations. The idea is that the various operational definitions are “converging” on the same construct. When scores based on several different operational definitions are closely related to each other and produce similar patterns of results, this constitutes good evidence that the construct is being measured effectively and that it is useful. The various measures of stress, for example, are all correlated with each other and have all been shown to be correlated with other variables such as

RESEARCH FOR HUMAN SERVICES pg. 69

immune system functioning (also measured in a variety of ways) (Segerstrom & Miller, 2004). [3] This is what allows researchers eventually to draw useful general conclusions, such as “stress is negatively correlated with immune system functioning,” as opposed to more specific and less useful ones, such as “people’s scores on the Perceived Stress Scale are negatively correlated with their white blood counts.”

Levels of Measurement The psychologist S. S. Stevens suggested that scores can be assigned to individuals so that they communicate more or less quantitative information about the variable of interest (Stevens, 1946). [4] For example, the officials at a 100-m race could simply rank order the runners as they crossed the finish line (first, second, etc.), or they could time each runner to the nearest tenth of a second using a stopwatch (11.5 s, 12.1 s, etc.). In either case, they would be measuring the runners’ times by systematically assigning scores to represent those times. But while the rank ordering procedure communicates the fact that the second-place runner took longer to finish than the first-place finisher, the stopwatch procedure also communicates how much longer the second-place finisher took. Stevens actually suggested four different levels of measurement (which he called “scales of measurement”) that correspond to four different levels of quantitative information that can be communicated by a set of scores.

Nominal Level The nominal level of measurement is used for categorical variables and involves assigning scores that are category labels. Category labels communicate whether any two individuals are the same or different in terms of the variable being measured. For example, if you look at your research participants as they enter the room, decide whether each one is male or female, and type this information into a spreadsheet, you are engaged in nominal-level measurement. Or if you ask your participants to indicate which of several ethnicities they identify themselves with, you are again engaged in nominal-level measurement.

Ordinal Level The remaining three levels of measurement are used for quantitative variables.

The ordinal level of measurement involves assigning scores so that they represent the rank order of the individuals. Ranks communicate not only whether any two individuals are the same or different in terms of the variable being measured but also whether one individual is higher or lower on that variable.

Interval Level The interval level of measurement involves assigning scores so that they represent the precise magnitude of the difference between individuals, but a score of zero does not actually represent the complete absence of the characteristic. A classic example is the measurement of heat using the Celsius or Fahrenheit scale. The difference between temperatures of 20°C and 25°C is precisely 5°, but a temperature of 0°C does not considered to be measured at the interval level does not mean that there is a complete absence of heat.

Ratio Level The ratio level of measurement involves assigning scores in such a way that there is a true zero point that represents the complete absence of the quantity. Height measured in meters and weight measured in kilograms are good examples. So are counts of discrete objects or events such as the number of siblings one has or the number of questions a student answers correctly on an exam. Below is a diagram showing the levels, which figuratively form an ascending staircase, from the most basic nominal level to the highest ratio level. One commonly used variable at the ratio level is money.

RESEARCH FOR HUMAN SERVICES pg. 70

Stevens’s levels of measurement are important for at least two reasons. First, they emphasize the generality of the concept of measurement. Although people do not normally think of categorizing or ranking individuals as measurement, in fact they are as long as they are done so that they represent some characteristic of the individuals. Second, the levels of measurement can serve as a rough guide to the statistical procedures that can be used with the data and the conclusions that can be drawn from them. With nominal-level measurement, for example, the only available measure of central tendency is the mode. Also, ratio-level measurement is the only level that allows meaningful statements about ratios of scores. One cannot say that someone with an IQ of 140 is twice as intelligent as someone with an IQ of 70 because IQ is measured at the interval level, but one can say that someone with six siblings has twice as many as someone with three because number of siblings is measured at the ratio level. Y TSUM M AK E AW AYS

Summary:

 Measurement is the assignment of scores to individuals so that the scores represent some characteristic of the individuals. Psychological measurement can be achieved in a wide variety of ways, including self-report, behavioral, and physiological measures.

 Psychological constructs such as intelligence, self-esteem, and depression are variables that are not directly observable because they represent behavioral tendencies or complex patterns of behavior and internal processes. An important goal of scientific research is to conceptually define psychological constructs in ways that accurately describe them.

 For any conceptual definition of a construct, there will be many different operational definitions or ways of measuring it. The use of multiple operational definitions, or converging operations, is a common strategy in psychological research.

 Variables can be measured at four different levels—nominal, ordinal, interval, and ratio—that communicate increasing amounts of quantitative information. The level of measurement affects the kinds of statistics you can use and conclusions you can draw from your data.

RESEARCH FOR HUMAN SERVICES pg. 71

Practice:

1. Complete the Rosenberg Self-Esteem Scale and compute your overall score. Scoring key is in the appendix.

2. Think of three operational definitions for sexual jealousy, decisiveness, and social anxiety. Consider the possibility of self-report, behavioral, and physiological measures. Be as precise as you can.

3. For each of the following variables, decide which level of measurement is being used. 4. A college instructor measures the time it takes his students to finish an exam by looking through the

stack of exams at the end. He assigns the one on the bottom a score of 1, the one on top of that a 2, and so on.

5. A researcher accesses her participants’ medical records and counts the number of times they have seen a doctor in the past year.

6. Participants in a research study are asked whether they are right-handed or left-handed.

[1] Costa, P. T., Jr., & McCrae, R. R. (1992). Normal personality assessment in clinical practice: The NEO Personality Inventory. Psychological Assessment, 4, 5–13.

[2] Bandura, A., Ross, D., & Ross, S. A. (1961). Transmission of aggression through imitation of aggressive models. Journal of Abnormal and Social Psychology, 63, 575–582.

[3] Segerstrom, S. E., & Miller, G. E. (2004). Psychological stress and the human immune system: A meta- analytic study of 30 years of inquiry. Psychological Bulletin, 130, 601–630.

[4] Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103, 677–680.

5.2 Reliability and Validity of Measurement

LEARNING OBJECTIVES  Define reliability, including the different types and how they are assessed.

 Define validity, including the different types and how they are assessed.

 Describe the kinds of evidence that would be relevant to assessing the reliability and validity of a particular measure.

Measurement involves assigning scores to individuals so that they represent some characteristic of the individuals. But how do researchers know that the scores actually represent the characteristic, especially when it is a construct like intelligence, self-esteem, depression, or working memory capacity? The answer is that they conduct research using the measure to confirm that the scores make sense based on their understanding of the construct being measured. This is an extremely important point. Psychologists do not simply assume that their measures work. Instead, they collect data to demonstrate that they work. If their research does not demonstrate that a measure works, they stop using it.

As an informal example, imagine that you have been dieting for a month. Your clothes seem to be fitting more loosely, and several friends have asked if you have lost weight. If at this point your bathroom scale indicated that you had lost 10 pounds, this would make sense and you would continue to use the scale. But if it indicated that you had gained 10 pounds, you would rightly conclude that it was broken and either fix it or get rid of it. In evaluating a measurement method, psychologists consider two general dimensions: reliability and validity.

Reliability Reliability refers to the consistency of a measure. Psychologists consider three types of consistency: over time (test-retest reliability), across items (internal consistency), and across different researchers (interrater reliability).

RESEARCH FOR HUMAN SERVICES pg. 72

Test-Retest Reliability When researchers measure a construct that they assume to be consistent across time, then the scores they obtain should also be consistent across time. Test-retest reliability is the extent to which this is actually the case. For example, intelligence is generally thought to be consistent across time. A person who is highly intelligent today will be highly intelligent next week. This means that any good measure of intelligence should produce roughly the same scores for this individual next week as it does today. Clearly, a measure that produces highly inconsistent scores over time cannot be a very good measure of a construct that is supposed to be consistent.

Assessing test-retest reliability requires using the measure on a group of people at one time, using it again on the same group of people at a later time, and then looking at test-retest correlation between the two sets of scores. This is typically done by graphing the data in a scatterplot and computing Pearson’s r. Figure 5.3 "Test-Retest Correlation Between Two Sets of Scores of Several College Students on the Rosenberg Self-Esteem Scale, Given Two Times a Week Apart" shows the correlation between two sets of scores of several college students on the Rosenberg Self-Esteem Scale, given two times a week apart. Pearson’s r for these data is +.95. In general, a test-retest correlation of +.80 or greater is considered to indicate good reliability.

High test-retest correlations make sense when the construct being measured is assumed to be consistent over time, which is the case for intelligence, self-esteem, and the Big Five personality dimensions. But other constructs are not assumed to be stable over time. The very nature of mood, for example, is that it changes. So, a measure of mood that produced a low test-retest correlation over a period of a month would not be a cause for concern.

The test-retest correlation for some instrument’s such as the Myers Briggs, has been called into question. See https://www.psychologytoday.com/blog/give-and-take/201309/goodbye-mbti-the-fad-won-t-die

Internal Consistency A second kind of reliability is internal consistency, which is the consistency of people’s responses across the items on a multiple-item measure. In general, all the items on such measures are supposed to reflect the same underlying construct, so people’s scores on those items should be correlated with each other. On the Rosenberg Self-Esteem Scale, people who agree that they are a person of worth should tend to agree that that they have a number of good qualities. If people’s responses to the different items are not correlated with each other, then it would no longer make sense to claim that they are all measuring the same underlying construct. This is as true for behavioral and physiological measures as for self-report measures. For example, people might make a series of bets in a simulated game of roulette as a measure of their level of risk seeking. This measure would be internally consistent to the extent that individual participants’ bets were consistently high or low across trials.

Like test-retest reliability, internal consistency can only be assessed by collecting and analyzing data. One approach is to look at a split-half correlation. This involves splitting the items into two sets, such as the first and second halves of the items or the even- and odd-numbered items. Then a score is computed for each set of items, and the relationship between the two sets of scores is examined. For example, Figure 5.4 "Split-Half Correlation Between Several College Students’ Scores on the Even-Numbered Items and Their Scores on the Odd-Numbered Items of the Rosenberg Self-Esteem Scale" shows the split-half correlation between several college students’ scores on the even-numbered items and their scores on the odd-numbered items of the Rosenberg Self-Esteem Scale. Pearson’s r for these data is +.88. A split-half correlation of +.80 or greater is generally considered good internal consistency.

RESEARCH FOR HUMAN SERVICES pg. 73

Interrater Reliability Many behavioral measures involve significant judgment on the part of an observer or a rater. Interrater reliability is the extent to which different observers are consistent in their judgments. For example, if you were interested in measuring college students’ social skills, you could make video recordings of them as they interacted with another student whom they are meeting for the first time. Then you could have two or more observers watch the videos and rate each student’s level of social skills. To the extent that each participant does in fact have some level of social skills that can be detected by an attentive observer, different observers’ ratings should be highly correlated with each other. If they were not, then those ratings could not be an accurate representation of participants’ social skills. Interrater reliability is often assessed using Cronbach’s α when the judgments are quantitative or an analogous statistic called Cohen’s κ (the Greek letter kappa) when they are categorical. There are also several other ratings besides Cohen’s K.

Using Interrater Reliability Interrater reliability is often used in educational assessment. An example is of a university that wants to assess the quality of their students writing in an English composition class. The scale could have four categories for quality of writing. A score of four would be for exceeding standards, three for meeting standards, two for below standards, and one would be for remedial (meaning that the student is rather far below even the lowest standards). It is best practices to have two people rate each paper. The agreement between the two raters is measured using Cronbach’s alpha or Kohen’s Kappa, or one of the other statistical tests. If the raters generally agree, or their ratings are close, it is considered that there is interrater reliability. For example:

I rate five papers 4,3, 4, 2, 2

In the same order, you rate the same papers as 4, 3, 3, 2, 2

But if you rated the papers 2, 1, 1, 0, 0 we are obviously too far apart.

Validity Validity is the extent to which the scores from a measure represent the variable they are intended to. But how do researchers make this judgment? We have already considered one factor that they take into account—reliability. When a measure has good test-retest reliability and internal consistency, researchers should be more confident that the scores represent what they are supposed to. There has to be more to it, however, because a measure can be extremely reliable but have no validity whatsoever. As an absurd example, imagine someone who believes that people’s index finger length reflects their self-esteem and therefore tries to measure self-esteem by holding a ruler up to people’s index fingers. Although this measure would have extremely good test-retest reliability, it would have absolutely no validity. The fact that one person’s index finger is a centimeter longer than another’s would indicate nothing about which one had higher self-esteem.

Textbook presentations of validity usually divide it into several distinct “types.” But a good way to interpret these types is that they are other kinds of evidence—in addition to reliability—that should be taken into account when judging the validity of a measure. Here we consider four basic kinds: face validity, content validity, criterion validity, and discriminant validity.

RESEARCH FOR HUMAN SERVICES pg. 74

Face Validity Face validity is the extent to which a measurement method appears “on its face” to measure the construct of interest. Most people would expect a self-esteem questionnaire to include items about whether they see themselves as a person of worth and whether they think they have good qualities. So, a questionnaire that included these kinds of items would have good face validity. The finger-length method of measuring self- esteem, on the other hand, seems to have nothing to do with self-esteem and therefore has poor face validity. Although face validity can be assessed quantitatively—for example, by having a large sample of people rate a measure in terms of whether it appears to measure what it is intended to—it is usually assessed informally.

Face validity is at best a very weak kind of evidence that a measurement method is measuring what it is supposed to. One reason is that it is based on people’s intuitions about human behavior, which are frequently wrong. It is also the case that many established measures in psychology work quite well despite lacking face validity. The Minnesota Multiphasic Personality Inventory (MMPI) measures many personality characteristics and disorders by having people decide whether each of over 567 different statements applies to them—where many of the statements do not have any obvious relationship to the construct that they measure. Another example is the Implicit Association Test, which measures prejudice in a way that is nonintuitive to most people (see Note 5.31 "How Prejudiced Are You?").

How Prejudiced Are You? The Implicit Association Test (IAT) is used to measure people’s attitudes toward various social groups. The IAT is a behavioral measure designed to reveal negative attitudes that people might not admit to on a self- report measure. It focuses on how quickly people are able to categorize words and images representing two contrasting groups (e.g., gay and straight) along with other positive and negative stimuli (e.g., the words “wonderful” or “nasty”). The IAT has been used in dozens of published research studies, and there is strong evidence for both its reliability and its validity (Nosek, Greenwald, & Banaji, 2006). [1] You can learn more about the IAT—and take several of them for yourself—at the following website: https://implicit.harvard.edu/implicit.

Content Validity Content validity is the extent to which a measure “covers” the construct of interest. For example, if a researcher conceptually defines test anxiety as involving both sympathetic nervous system activation (leading to nervous feelings) and negative thoughts, then his measure of test anxiety should include items about both nervous feelings and negative thoughts. Or consider that attitudes are usually defined as involving thoughts, feelings, and actions toward something. By this conceptual definition, a person has a positive attitude toward exercise to the extent that he or she thinks positive thoughts about exercising, feels good about exercising, and actually exercises. So, to have good content validity, a measure of people’s attitudes toward exercise would have to reflect all three of these aspects. Like face validity, content validity is not usually assessed quantitatively. Instead, it is assessed by carefully checking the measurement method against the conceptual definition of the construct.

Criterion Validity Criterion validity is the extent to which people’s scores on a measure are correlated with other variables (known as criteria) that one would expect them to be correlated with. For example, people’s scores on a new measure of test anxiety should be negatively correlated with their performance on an important school exam. If it were found that people’s scores were in fact negatively correlated with their exam performance, then this would be a piece of evidence that these scores really represent people’s test anxiety. But if it were found that people scored equally well on the exam regardless of their test anxiety scores, then this would cast doubt on the validity of the measure.

RESEARCH FOR HUMAN SERVICES pg. 75

A criterion can be any variable that one has reason to think should be correlated with the construct being measured, and there will usually be many of them. For example, one would expect test anxiety scores to be negatively correlated with exam performance and course grades and positively correlated with general anxiety and with blood pressure during an exam. Or imagine that a researcher develops a new measure of physical risk taking. People’s scores on this measure should be correlated with their participation in “extreme” activities such as snowboarding and rock climbing, the number of speeding tickets they have received, and even the number of broken bones they have had over the years. Criteria can also include other measures of the same construct. For example, one would expect new measures of test anxiety or physical risk taking to be positively correlated with existing measures of the same constructs. So, the use of converging operations is one way to examine criterion validity.

Assessing criterion validity requires collecting data using the measure. Researchers John Cacioppo and Richard Petty did this when they created their self-report Need for Cognition Scale to measure how much people value and engage in thinking (Cacioppo & Petty, 1982). [2] In a series of studies, they showed that college faculty scored higher than assembly-line workers, that people’s scores were positively correlated with their scores on a standardized academic achievement test, and that their scores were negatively correlated with their scores on a measure of dogmatism (which represents a tendency toward obedience). In the years since it was created, the Need for Cognition Scale has been used in literally hundreds of studies and has been shown to be correlated with a wide variety of other variables, including the effectiveness of an advertisement, interest in politics, and juror decisions (Petty, Briñol, Loersch, & McCaslin, 2009). [3]

Discriminant Validity Discriminant validity is the extent to which scores on a measure are not correlated with measures of variables that are conceptually distinct. For example, self-esteem is a general attitude toward the self that is fairly stable over time. It is not the same as mood, which is how good or bad one happens to be feeling right now. So, people’s scores on a new measure of self-esteem should not be very highly correlated with their moods. If the new measure of self-esteem were highly correlated with a measure of mood, it could be argued that the new measure is not really measuring self-esteem; it is measuring mood instead.

When they created the Need for Cognition Scale, Cacioppo and Petty also provided evidence of discriminant validity by showing that people’s scores were not correlated with certain other variables. For example, they found only a weak correlation between people’s need for cognition and a measure of their cognitive style—the extent to which they tend to think analytically by breaking ideas into smaller parts or holistically in terms of “the big picture.” They also found no correlation between people’s need for cognition and measures of their test anxiety and their tendency to respond in socially desirable ways. All these low correlations provide evidence that the measure is reflecting a conceptually distinct construct.

ST AAW AY

Summary:

 Social scientists do not simply assume that their measures work. Instead, they conduct research to show that they work. If they cannot show that they work, they stop using them.

 There are two distinct criteria by which researchers evaluate their measures: reliability and validity. Reliability is consistency across time (test-retest reliability), across items (internal consistency), and across researchers (interrater reliability). Validity is the extent to which the scores actually represent the variable they are intended to.

 Validity is a judgment based on various types of evidence. The relevant evidence includes the measure’s reliability, whether it covers the construct of interest, and whether the scores it produces are correlated with other variables they are expected to be correlated with and not correlated with variables that are conceptually distinct.

RESEARCH FOR HUMAN SERVICES pg. 76

 The reliability and validity of a measure is not established by any single study but by the pattern of results across multiple studies. The assessment of reliability and validity is an ongoing process.E S

Practice:

1. Ask several friends to complete the Rosenberg Self-Esteem Scale. Then assess its internal consistency by making a scatterplot to show the split-half correlation (even- vs. odd-numbered items). Compute Pearson’s r too if you know how.

2. Take an Implicit Association Test and then list as many ways to assess its criterion validity as you can think of.

Discussion:

Think back to the last college exam you took and think of the exam as a psychological measure. What construct do you think it was intended to measure? Comment on its face and content validity. What data could you collect to assess its reliability, criterion validity, and discriminant validity?

[1] Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2006). The Implicit Association Test at age 7: A methodological and conceptual review. In J. A. Bargh (Ed.), Social psychology and the unconscious: The automaticity of higher mental processes (pp. 265–292). London, England: Psychology Press.

[2] Cacioppo, J. T., & Petty, R. E. (1982). The need for cognition. Journal of Personality and Social Psychology, 42, 116–131.

[3] Petty, R. E, Briñol, P., Loersch, C., & McCaslin, M. J. (2009). The need for cognition. In M. R. Leary & R. H. Hoyle (Eds.), Handbook of individual differences in social behavior (pp. 318–329). New York, NY: Guilford Press.

RESEARCH FOR HUMAN SERVICES pg. 77

5.3 Practical Strategies for Clinical Measurement

LEARNING OBJECTIVES  Specify the four broad steps in the measurement process.

 Explain how you would decide whether to use an existing measure or create your own.

 Describe multiple strategies to identify and locate existing measures of psychological constructs.

 Describe several general principles for creating new measures and for implementing existing and new measures.

 Create a simple plan for assessing the reliability and validity of an existing or new measure.

So far in this chapter, we have considered several basic ideas about the nature of psychological constructs and their measurement. But now imagine that you are in the position of actually having to measure a psychological construct for a research project. How should you proceed? Broadly speaking, there are four steps in the measurement process: (a) conceptually defining the construct, (b) operationally defining the construct, (c) implementing the measure, and (d) evaluating the measure. In this section, we will look at each of these steps in turn.

Conceptually Defining the Construct Having a clear and complete conceptual definition of a construct is a prerequisite for good measurement. For one thing, it allows you to make sound decisions about exactly how to measure the construct. If you had only a vague idea that you wanted to measure people’s “memory,” for example, you would have no way to choose whether you should have them remember a list of vocabulary words, a set of photographs, a newly learned skill, or an experience from long ago. Because psychologists now conceptualize memory as a set of semi-independent systems, you would have to be more precise about what you mean by “memory.” If you are interested in long-term declarative memory (memory for facts), then having participants remember a list of words that they learned last week would make sense, but having them remember and execute a newly learned skill would not. In general, there is no substitute for reading the research literature on a construct and paying close attention to how others have defined it.

Deciding on an Operational Definition

Using an Existing Measure It is usually a good idea to use an existing measure that has been used successfully in previous research. Among the advantages are that (a) you save the time and trouble of creating your own, (b) there is already some evidence that the measure is valid (if it has been used successfully), and (c) your results can more easily be compared with and combined with previous results. In fact, if there already exists a reliable and valid measure of a construct, other researchers might expect you to use it unless you have a good and clearly stated reason for not doing so.

If you choose to use an existing measure, you may still have to choose among several alternatives. You might choose the most common one, the one with the best evidence of reliability and validity, the one that best measures a particular aspect of a construct that you are interested in (e.g., a physiological measure of stress if you are most interested in its underlying physiology), or even the one that would be easiest to use. For example, the Ten-Item Personality Inventory (TIPI) is a self-report questionnaire that measures all the Big Five personality dimensions with just 10 items (Gosling, Rentfrow, & Swann, 2003). [1] It is not as reliable or valid as longer and more comprehensive measures, but a researcher might choose to use it when testing time is severely limited.

When an existing measure was created primarily for use in scientific research, it is usually described in detail in a published research article and is free to use in your own research with a proper citation. You might find that later researchers who use the same measure describe it only briefly but provide a reference to the original article, in which case you would have to get the details from the original article. The American Psychological Association also publishes the Directory of Unpublished Experimental Measures,

RESEARCH FOR HUMAN SERVICES pg. 78

which is an extensive catalog of measures that have been used in previous research. Many existing measures—especially those that have applications in clinical psychology—are proprietary. This means that a publisher owns the rights to them and that you would have to purchase them. These include many standard intelligence tests, the Beck Depression Inventory, and the Minnesota Multiphasic Personality Inventory (MMPI). Details about many of these measures and how to obtain them can be found in other reference books, including Tests in Print and the Mental Measurements Yearbook. There is a good chance you can find these reference books in your college or university library.

Creating Your Own Measure Instead of using an existing measure, you might want to create your own. Perhaps there is no existing measure of the construct you are interested in or existing ones are too difficult or time-consuming to use. Or perhaps you want to use a new measure specifically to see whether it works in the same way as existing measures—that is, to demonstrate converging operations. In this section, we consider some general issues in creating new measures that apply equally to self-report, behavioral, and physiological measures.

First, be aware that most new measures in the social sciences are really variations of existing measures, so you should still look to the research literature for ideas. Perhaps you can modify an existing questionnaire, create a paper-and-pencil version of a measure that is normally computerized (or vice versa), or adapt a measure that has traditionally been used for another purpose. For example, the famous Stroop task (Stroop, 1935) [2]—in which people quickly name the colors that various color words are printed in—has been adapted for the study of social anxiety. Socially anxious people are slower at color naming when the words have negative social connotations such as “stupid” (Amir, Freshman, & Foa, 2002). [3]

When you create a new measure, you should strive for simplicity. Remember that your participants are not as interested in your research as you are and that they will vary widely in their ability to understand and carry out whatever task you give them. You should create a set of clear instructions using simple language that you can present in writing or read aloud (or both). It is also a good idea to include one or more practice items so that participants can become familiar with the task, and to build in an opportunity for them to ask questions before continuing. It is also best to keep the measure brief to avoid boring or frustrating your participants to the point that their responses start to become less reliable and valid.

The need for brevity, however, needs to be weighed against the fact that it is nearly always better for a measure to include multiple items rather than a single item. There are two reasons for this. One is a matter of content validity. Multiple items are often required to cover a construct adequately. The other is a matter of reliability. People’s responses to single items can be influenced by all sorts of irrelevant factors— misunderstanding the particular item, a momentary distraction, or a simple error such as checking the wrong response option. But when several responses are summed or averaged, the effects of these irrelevant factors tend to cancel each other out to produce more reliable scores. Remember, however, that multiple items must be structured in a way that allows them to be combined into a single overall score by summing or averaging. To measure “financial responsibility,” a student might ask people about their annual income, obtain their credit score, and have them rate how “thrifty” they are—but there is no obvious way to combine these responses into an overall score. To create a true multiple-item measure, the student might instead ask people to rate the degree to which 10 statements about financial responsibility describe them on the same five-point scale.

Finally, the very best way to assure yourself that your measure has clear instructions, includes sufficient practice, and is an appropriate length is to test several people. (Family and friends often serve this purpose nicely). Observe them as they complete the task, time them, and ask them afterward to comment on how easy or difficult it was, whether the instructions were clear, and anything else you might be wondering about. Obviously, it is better to discover problems with a measure before beginning any large-scale data collection.

RESEARCH FOR HUMAN SERVICES pg. 79

Implementing the Measure You will want to implement any measure in a way that maximizes its reliability and validity. In most cases, it is best to test everyone under similar conditions that, ideally, are quiet and free of distractions. Testing participants in groups is often done because it is efficient, but be aware that it can create distractions that reduce the reliability and validity of the measure. As always, it is good to use previous research as a guide. If others have successfully tested people in groups using a particular measure, then you should consider doing it too.

Be aware also that people can react in a variety of ways to being measured that reduce the reliability and validity of the scores. Although some disagreeable participants might intentionally respond in ways meant to “mess up” a study, participant reactivity is more likely to take the opposite form. Agreeable participants might respond in ways they believe they are expected to. They might engage in socially desirable responding. For example, people with low self-esteem agree that they feel they are a person of worth not because they really feel this way but because they believe this is the socially appropriate response and do not want to look bad in the eyes of the researcher. Additionally, research studies can have built-in demand characteristics: cues to how the researcher expects participants to behave. For example, a participant whose attitude toward exercise is measured immediately after she is asked to read a passage about the dangers of heart disease might reasonably conclude that the passage was meant to improve her attitude. As a result, she might respond more favorably because she believes she is expected to by the researcher. Finally, your own expectations can bias participants’ behaviors in unintended ways.

There are several precautions you can take to minimize these kinds of reactivity. One is to make the procedure as clear and brief as possible so that participants are not tempted to take out their frustrations on your results. Another is to guarantee participants’ anonymity and make clear to them that you are doing so. If you are testing them in groups, be sure that they are seated far enough apart that they cannot see each other’s responses. Give them all the same type of writing implement so that they cannot be identified by, for example, the pink glitter pen that they used. You can even allow them to seal completed questionnaires into individual envelopes or put them into a drop box where they immediately become mixed with others’ questionnaires. Although informed consent requires telling participants what they will be doing, it does not require revealing your hypothesis or other information that might suggest to participants how you expect them to respond. A questionnaire designed to measure financial responsibility need not be titled “Are You Financially Responsible?” It could be titled “Money Questionnaire” or have no title at all. Finally, the effects of your expectations can be minimized by arranging to have the measure administered by a helper who is unaware of its intent or of any hypothesis being tested. Regardless of whether this is possible, you should standardize all interactions between researchers and participants—for example, by always reading the same set of instructions word for word.

Evaluating the Measure Once you have used your measure on a sample of people and have a set of scores, you are in a position to evaluate it more thoroughly in terms of reliability and validity. Even if the measure has been used extensively by other researchers and has already shown evidence of reliability and validity, you should not assume that it worked as expected for your particular sample and under your particular testing conditions. Regardless, you now have additional evidence bearing on the reliability and validity of the measure, and it would make sense to add that evidence to the research literature.

In most research designs, it is not possible to assess test-retest reliability because participants are tested at only one time. For a new measure, you might design a study specifically to assess its test-retest reliability by testing the same set of participants at two times. In other cases, a study designed to answer a different question still allows for the assessment of test-retest reliability. For example, an instructor might measure his students’ attitude toward critical thinking using the same measure at the beginning and end of the semester to see if there is any change. Even if there is no change, he could still look at the correlation between students’ scores at the two times to assess the measure’s test-retest reliability. It is also customary to assess internal consistency for any multiple-item measure—usually by looking at a split-half correlation or Cronbach’s alpha.

RESEARCH FOR HUMAN SERVICES pg. 80

Criterion and discriminant validity can be assessed in various ways. For example, if your study included more than one measure of the same construct or measures of conceptually distinct constructs, then you should look at the correlations among these measures to be sure that they fit your expectations. Note also that a successful experimental manipulation also provides evidence of criterion validity. Recall that MacDonald and Martineau manipulated participant’s moods by having them think either positive or negative thoughts, and after the manipulation their mood measure showed a distinct difference between the two groups. This simultaneously provided evidence that their mood manipulation worked and that their mood measure was valid.

But what if your newly collected data cast doubt on the reliability or validity of your measure? The short answer is that you have to ask why. It could be that there is something wrong with your measure or how you administered it. It could be that there is something wrong with your conceptual definition. It could be that your experimental manipulation failed. For example, if a mood measure showed no difference between people whom you instructed to think positive versus negative thoughts, maybe it is because the participants did not actually think the thoughts they were supposed to or that the thoughts did not actually affect their moods. In short, it is “back to the drawing board” to revise the measure, revise the conceptual definition, or try a new manipulation.

Summary

 Good measurement begins with a clear conceptual definition of the construct to be measured. This is accomplished both by clear and detailed thinking and by a review of the research literature.

 You often have the option of using an existing measure or creating a new measure. You should make this decision based on the availability of existing measures and their adequacy for your purposes.

 Several simple steps can be taken in creating new measures and in implementing both existing and new measures that can help maximize reliability and validity.

 Once you have used a measure, you should reevaluate its reliability and validity based on your new data. Remember that the assessment of reliability and validity is an ongoing process.

XE R C ISE S

Practice 1. Write your own conceptual definition of self-confidence, irritability, and athleticism. 2. Choose a construct (sexual jealousy, self-confidence, etc.) and find two measures of that construct

in the research literature. If you were conducting your own study, which one (if either) would you use and why?

[1] Gosling, S. D., Rentfrow, P. J., & Swann, W. B., Jr. (2003). A very brief measure of the Big Five personality domains. Journal of Research in Personality, 37, 504–528.

[2] Stroop, J. R. (1935). Studies of interference in serial verbal reactions. Journal of Experimental Psychology, 18, 643–662.

[3] Amir, N., Freshman, M., & Foa, E. (2002). Enhanced Stroop interference for threat in social phobia. Journal of Anxiety Disorders, 16, 1–9.

RESEARCH FOR HUMAN SERVICES pg. 81

Chapter 6: Experimental Research All life is an experiment. The more experiments you make the better. -Ralph Waldo Emerson

In Human Services, and Social Work, most of the research we do is not experimental for reasons that will become clearer as we move through the chapter. The social science which has embraced experimental research the most is without a doubt psychology. However, we do benefit greatly from the insight, theoretical perspectives, and practical knowledge that experimental researchers have provided for us.

In the late 1960s social psychologists John Darley and Bibb Latané proposed a counterintuitive hypothesis. The more witnesses there are to an accident or a crime, the less likely any of them is to help the victim (Darley & Latané, 1968).[1] They also suggested the theory that this happens because each witness feels less responsible for helping—a process referred to as the “diffusion of responsibility.” Darley and Latané noted that their ideas were consistent with many real-world cases. For example, a New York woman named Kitty Genovese was assaulted and murdered while several witnesses failed to help. But Darley and Latané also understood that such isolated cases did not provide convincing evidence for their hypothesized “bystander effect.” There was no way to know, for example, whether any of the witnesses to Kitty Genovese’s murder would have helped had there been fewer of them.

So, to test their hypothesis, Darley and Latané created a simulated emergency situation in a laboratory. Each of their college student participants was isolated in a small room and told that he or she would be having a discussion about college life with other students via an intercom system. Early in the discussion, however, one of the students began having what seemed to be an epileptic seizure. Over the intercom came the following: “I could really-er-use some help so if somebody would-er-give me a little h-help-uh-er- er-er-er-er c-could somebody-er-er-help-er-uh-uh-uh (choking sounds)…I’m gonna die-er-er-I’m…gonna die-er-help-er-er-seizure-er- [chokes, then quiet]” (Darley & Latané, 1968, p. 379). [2]

In actuality, there were no other students. These comments had been prerecorded and were played back to create the appearance of a real emergency. The key to the study was that some participants were told that the discussion involved only one other student (the victim), others were told that it involved two other students, and still others were told that it included five other students. Because this was the only difference between these three groups of participants, any difference in their tendency to help the victim would have to have been caused by it. And sure enough, the likelihood that the participant left the room to seek help for the “victim” decreased from 85% to 62% to 31% as the number of “witnesses” increased.

The Parable of the 38 Witnesses The story of Kitty Genovese has been told and retold in numerous sociology and psychology textbooks. The standard version is that there were 38 witnesses to the crime, that all of them watched (or listened) for an extended period of time, and that none of them did anything to help. However, recent scholarship suggests that the standard story is inaccurate in many ways (Manning, Levine, & Collins, 2007). [3] For example, only six eyewitnesses testified at the trial, none of them were aware that he or she was witnessing a lethal assault, and there have been several reports of witnesses calling the police or even coming to the aid of Kitty Genovese. Although the standard story inspired a long line of research on the bystander effect and the diffusion of responsibility, it may also have directed researchers’ and students’ attention away from other equally interesting and important issues in the psychology of helping—including the conditions in which people do in fact respond collectively to emergency situations.

The study that Darley and Latané conducted was a particular kind of study called an experiment. Experiments are used to determine not only whether there is a statistical relationship between two variables but also whether the relationship is a causal one. For this reason, experiments are one of the most common and useful tools in the psychological researcher’s toolbox. In this chapter, we look at

RESEARCH FOR HUMAN SERVICES pg. 82

experiments in detail. We consider first what sets experiments apart from other kinds of studies and why they support causal conclusions while other kinds of studies do not. We then look at two basic ways of designing an experiment—between-subjects designs and within-subjects designs—and discuss their pros and cons. Finally, we consider several important practical issues that arise when conducting experiments.

[1] Darley, J. M., & Latané, B. (1968). Bystander intervention in emergencies: Diffusion of responsibility. Journal of Personality and Social Psychology, 4, 377–383.

[2] Darley, J. M., & Latané, B. (1968). Bystander intervention in emergencies: Diffusion of responsibility. Journal of Personality and Social Psychology, 4, 377–383.

[3] Manning, R., Levine, M., & Collins, A. (2007). The Kitty Genovese murder and the social psychology of helping: The parable of the 38 witnesses. American Psychologist, 62, 555–562.

6.1 Experiment Basics

LEARNING OBJECTIVES  Explain what an experiment is and recognize examples of studies that are experiments and studies

that are not experiments.

 Explain what internal validity is and why experiments are considered to be high in internal validity.

 Explain what external validity is and evaluate studies in terms of their external validity.

 Distinguish between the manipulation of the independent variable and control of extraneous variables and explain the importance of each.

 Recognize examples of confounding variables and explain how they affect the internal validity of a study.

What Is an Experiment? As we saw earlier in the book, an experiment is a type of study designed specifically to answer the question of whether there is a causal relationship between two variables. Do changes in an independent variable cause changes in a dependent variable? Experiments have two fundamental features. The first is that the researchers manipulate, or systematically vary, the level of the independent variable. The different levels of the independent variable are called conditions. For example, in Darley and Latané’s experiment, the independent variable was the number of witnesses that participants believed to be present. The researchers manipulated this independent variable by telling participants that there were either one, two, or five other students involved in the discussion, thereby creating three conditions. The second fundamental feature of an experiment is that the researcher controls, or minimizes the variability in, variables other than the independent and dependent variable. These other variables are called extraneous variables. Darley and Latané tested all their participants in the same room, exposed them to the same emergency situation, and so on. They also randomly assigned their participants to conditions so that the three groups would be similar to each other to begin with. Notice that although the words manipulation and control have similar meanings in everyday language, researchers make a clear distinction between them. They manipulate the independent variable by systematically changing its levels and control other variables by holding them constant.

Internal and External Validity

Internal Validity Recall that the fact that two variables are statistically related does not necessarily mean that one causes the other. “Correlation does not imply causation.” For example, if it were the case that people who exercise

RESEARCH FOR HUMAN SERVICES pg. 83

regularly are happier than people who do not exercise regularly, this would not necessarily mean that exercising increases people’s happiness. It could mean instead that greater happiness causes people to exercise (the directionality problem) or that something like better physical health causes people to exercise and be happier (the third-variable problem).

The purpose of an experiment, however, is to show that two variables are statistically related and to do so in a way that supports the conclusion that the independent variable caused any observed differences in the dependent variable. The basic logic is this: If the researcher creates two or more highly similar conditions and then manipulates the independent variable to produce just one difference between them, then any later difference between the conditions must have been caused by the independent variable. For example, because the only difference between Darley and Latané’s conditions was the number of students that participants believed to be involved in the discussion, this must have been responsible for differences in helping between the conditions.

An empirical study is said to be high in internal validity if the way it was conducted supports the conclusion that the independent variable caused any observed differences in the dependent variable. Thus, experiments are high in internal validity because the way they are conducted—with the manipulation of the independent variable and the control of extraneous variables—provides strong support for causal conclusions.

External Validity At the same time, the way that experiments are conducted sometimes leads to a different kind of criticism. Specifically, the need to manipulate the independent variable and control extraneous variables means that experiments are often conducted under conditions that seem artificial or unlike “real life” (Stanovich, 2010). [1] In many experiments, the participants are all college undergraduates and come to a classroom or laboratory to fill out a series of paper-and-pencil questionnaires or to perform a carefully designed computerized task. Consider, for example, an experiment in which researcher Barbara Fredrickson and her colleagues had college students come to a laboratory on campus and complete a math test while wearing a swimsuit (Fredrickson, Roberts, Noll, Quinn, & Twenge, 1998). [2] At first, this might seem silly. When will college students ever have to complete math tests in their swimsuits outside of this experiment?

The issue we are confronting is that of external validity. An empirical study is high in external validity if the way it was conducted supports generalizing the results to people and situations beyond those actually studied. As a general rule, studies are higher in external validity when the participants and the situation studied are similar to those that the researchers want to generalize to. Imagine, for example, that a group of researchers is interested in how shoppers in large grocery stores are affected by whether breakfast cereal is packaged in yellow or purple boxes. Their study would be high in external validity if they studied the decisions of ordinary people doing their weekly shopping in a real grocery store. If the shoppers bought much more cereal in purple boxes, the researchers would be fairly confident that this would be true for other shoppers in other stores. Their study would be relatively low in external validity, however, if they studied a sample of college students in a laboratory at a selective college who merely judged the appeal of various colors presented on a computer screen. If the students judged purple to be more appealing than yellow, the researchers would not be very confident that this is relevant to grocery shoppers’ cereal-buying decisions.

We should be careful, however, not to draw the blanket conclusion that experiments are low in external validity. One reason is that experiments need not seem artificial. Consider that Darley and Latané’s experiment provided a reasonably good simulation of a real emergency situation. Or consider field experiments that are conducted entirely outside the laboratory. In one such experiment, Robert Cialdini and his colleagues studied whether hotel guests choose to reuse their towels for a second day as opposed to having them washed as a way of conserving water and energy (Cialdini, 2005). [3] These researchers manipulated the message on a card left in a large sample of hotel rooms. One version of the message emphasized showing respect for the environment, another emphasized that the hotel would donate a portion of their savings to an environmental cause, and a third emphasized that most hotel guests choose to reuse their towels. The result was that guests who received the message that most hotel guests

RESEARCH FOR HUMAN SERVICES pg. 84

choose to reuse their towels reused their own towels substantially more often than guests receiving either of the other two messages. Given the way they conducted their study, it seems very likely that their result would hold true for other guests in other hotels.

A second reason not to draw the blanket conclusion that experiments are low in external validity is that they are often conducted to learn about psychological processes that are likely to operate in a variety of people and situations. Let us return to the experiment by Fredrickson and colleagues. They found that the women in their study, but not the men, performed worse on the math test when they were wearing swimsuits. They argued that this was due to women’s greater tendency to objectify themselves—to think about themselves from the perspective of an outside observer—which diverts their attention away from other tasks. They argued, furthermore, that this process of self-objectification and its effect on attention is likely to operate in a variety of women and situations—even if none of them ever finds herself taking a math test in her swimsuit.

Manipulation of the Independent Variable Again, to manipulate an independent variable means to change its level systematically so that different groups of participants are exposed to different levels of that variable, or the same group of participants is exposed to different levels at different times. For example, to see whether expressive writing affects people’s health, a researcher might instruct some participants to write about traumatic experiences and others to write about neutral experiences. The different levels of the independent variable are referred to as conditions, and researchers often give the conditions short descriptive names to make it easy to talk and write about them. In this case, the conditions might be called the “traumatic condition” and the “neutral condition.”

Notice that the manipulation of an independent variable must involve the active intervention of the researcher. Comparing groups of people who differ on the independent variable before the study begins is not the same as manipulating that variable. For example, a researcher who compares the health of people who already keep a journal with the health of people who do not keep a journal has not manipulated this variable and therefore not conducted an experiment. This is important because groups that already differ in one way at the beginning of a study are likely to differ in other ways too. For example, people who choose to keep journals might also be more conscientious, more introverted, or less stressed than people who do not. Therefore, any observed difference between the two groups in terms of their health might have been caused by whether or not they keep a journal, or it might have been caused by any of the other differences between people who do and do not keep journals. Thus, the active manipulation of the independent variable is crucial for eliminating the third-variable problem.

Of course, there are many situations in which the independent variable cannot be manipulated for practical or ethical reasons and therefore an experiment is not possible. For example, whether or not people have a significant early illness experience cannot be manipulated, making it impossible to do an experiment on the effect of early illness experiences on the development of hypochondriasis. This does not mean it is impossible to study the relationship between early illness experiences and hypochondriasis—only that it must be done using nonexperimental approaches. We will discuss this in detail later in the book.

In many experiments, the independent variable is a construct that can only be manipulated indirectly. For example, a researcher might try to manipulate participants’ stress levels indirectly by telling some of them that they have five minutes to prepare a short speech that they will then have to give to an audience of other participants. In such situations, researchers often include a manipulation check in their procedure. A manipulation check is a separate measure of the construct the researcher is trying to manipulate. For example, researchers trying to manipulate participants’ stress levels might give them a paper-and-pencil stress questionnaire or take their blood pressure—perhaps right after the manipulation or at the end of the procedure—to verify that they successfully manipulated this variable.

Control of Extraneous Variables An extraneous variable is anything that varies in the context of a study other than the independent and dependent variables. In an experiment on the effect of expressive writing on health, for example,

RESEARCH FOR HUMAN SERVICES pg. 85

extraneous variables would include participant variables (individual differences) such as their writing ability, their diet, and their shoe size. They would also include situation or task variables such as the time of day when participants write, whether they write by hand or on a computer, and the weather. Extraneous variables pose a problem because many of them are likely to have some effect on the dependent variable. For example, participants’ health will be affected by many things other than whether or not they engage in expressive writing. This can make it difficult to separate the effect of the independent variable from the effects of the extraneous variables, which is why it is important to control extraneous variables by holding them constant.

Extraneous Variables as “Noise” Extraneous variables make it difficult to detect the effect of the independent variable in two ways. One is by adding variability or “noise” to the data. Imagine a simple experiment on the effect of mood (happy vs. sad) on the number of happy childhood events people are able to recall. Participants are put into a negative or positive mood (by showing them a happy or sad video clip) and then asked to recall as many happy childhood events as they can. The two leftmost columns of Table 6.1 "Hypothetical Noiseless Data and Realistic Noisy Data" show what the data might look like if there were no extraneous variables and the number of happy childhood events participants recalled was affected only by their moods. Every participant in the happy mood condition recalled exactly four happy childhood events, and every participant in the sad mood condition recalled exactly three. The effect of mood here is quite obvious. In reality, however, the data would probably look more like those in the two rightmost columns of Table 6.1 "Hypothetical Noiseless Data and Realistic Noisy Data". Even in the happy mood condition, some participants would recall fewer happy memories because they have fewer to draw on, use less effective strategies, or are less motivated. And even in the sad mood condition, some participants would recall more happy childhood memories because they have more happy memories to draw on, they use more effective recall strategies, or they are more motivated. Although the mean difference between the two groups is the same as in the idealized data, this difference is much less obvious in the context of the greater variability in the data. Thus, one reason researchers try to control extraneous variables is so their data look more like the idealized data in Table 6.1 "Hypothetical Noiseless Data and Realistic Noisy Data", which makes the effect of the independent variable is easier to detect (although real data never look quite that good).

One way to control extraneous variables is to hold them constant. This can mean holding situation or task variables constant by testing all participants in the same location, giving them identical instructions, treating them in the same way, and so on. It can also mean holding participant variables constant. For example, many studies of language limit participants to right-handed people, who generally have their language areas isolated in their left cerebral hemispheres. Left-handed people are more likely to have their language areas isolated in their right cerebral hemispheres or distributed across both hemispheres, which can change the way they process language and thereby add noise to the data.

In principle, researchers can control extraneous variables by limiting participants to one very specific category of person, such as 20-year-old, straight, female, right-handed, sophomore majors in whatever academic field the professor happens to be in. The obvious downside to this approach is that it would lower the external validity of the study—in particular, the extent to which the results can be generalized beyond the people actually studied. For example, it might be unclear whether results obtained with a sample of younger straight women would apply to older gay men. In many situations, the advantages of a diverse sample outweigh the reduction in noise achieved by a homogeneous one.

Extraneous Variables as Confounding Variables The second way that extraneous variables can make it difficult to detect the effect of the independent variable is by becoming confounding variables. A confounding variable is an extraneous variable that differs on average across levels of the independent variable. For example, in almost all experiments, participants’ intelligence quotients (IQs) will be an extraneous variable. But as long as there are participants with lower and higher IQs at each level of the independent variable so that the average IQ is roughly equal, then this variation is probably acceptable (and may even be desirable). What would be bad, however, would be for participants at one level of the independent variable to have substantially lower IQs on

RESEARCH FOR HUMAN SERVICES pg. 86

average and participants at another level to have substantially higher IQs on average. In this case, IQ would be a confounding variable.

To confound means to confuse, and this is exactly what confounding variables do. Because they differ across conditions—just like the independent variable—they provide an alternative explanation for any observed difference in the dependent variable. Figure 6.1 "Hypothetical Results from a Study on the Effect of Mood on Memory" shows the results of a hypothetical study, in which participants in a positive mood condition scored higher on a memory task than participants in a negative mood condition. But if IQ is a confounding variable—with participants in the positive mood condition having higher IQs on average than participants in the negative mood condition—then it is unclear whether it was the positive moods or the higher IQs that caused participants in the first condition to score higher. One way to avoid confounding variables is by holding extraneous variables constant. For example, one could prevent IQ from becoming a confounding variable by limiting participants only to those with IQs of exactly 100. But this approach is not always desirable for reasons we have already discussed. A second and much more general approach— random assignment to conditions—will be discussed in detail shortly. Because IQ also differs across conditions, it is a confounding variable.

Summary:

 An experiment is a type of empirical study that features the manipulation of an independent variable, the measurement of a dependent variable, and control of extraneous variables.

 Studies are high in internal validity to the extent that the way they are conducted supports the conclusion that the independent variable caused any observed differences in the dependent variable. Experiments are generally high in internal validity because of the manipulation of the independent variable and control of extraneous variables.

 Studies are high in external validity to the extent that the result can be generalized to people and situations beyond those actually studied. Although experiments can seem “artificial”—and low in external validity—it is important to consider whether the psychological processes under study are likely to operate in other people and situations.

Practice 1. List five variables that can be manipulated by the researcher in an experiment. List five variables

that cannot be manipulated by the researcher in an experiment. 2. For each of the following topics, decide whether that topic could be studied using an experimental

research design and explain why or why not.

 Effect of parietal lobe damage on people’s ability to do basic arithmetic.

 Effect of being clinically depressed on the number of close friendships people have.

 Effect of group training on the social skills of teenagers with Asperger’s syndrome.

 Effect of paying people to take an IQ test on their performance on that test.

[1] Stanovich, K. E. (2010). How to think straight about psychology (9th ed.). Boston, MA: Allyn & Bacon.

[2] Fredrickson, B. L., Roberts, T.-A., Noll, S. M., Quinn, D. M., & Twenge, J. M. (1998). The swimsuit becomes you: Sex differences in self-objectification, restrained eating, and math performance. Journal of Personality and Social Psychology, 75, 269–284.

[3] Cialdini, R. (2005, April). Don’t throw in the towel: Use social influence research. APS Observer. Retrieved from http://www.psychologicalscience.org/observer/getArticle.cfm?id=1762

6.2 Experimental Design

RESEARCH FOR HUMAN SERVICES pg. 87

LEARNING OBJECTIVES  Explain the difference between between-subjects and within-subjects experiments, list some of the

pros and cons of each approach, and decide which approach to use to answer a particular research question.

 Define random assignment, distinguish it from random sampling, explain its purpose in experimental research, and use some simple strategies to implement it.

 Define what a control condition is, explain its purpose in research on treatment effectiveness, and describe some alternative types of control conditions.

 Define several types of carryover effect, give examples of each, and explain how counterbalancing helps to deal with them.

In this section, we look at some different ways to design an experiment. The primary distinction we will make is between approaches in which each participant experiences one level of the independent variable and approaches in which each participant experiences all levels of the independent variable. The former are called between-subjects experiments and the latter are called within-subjects experiments.

Between-Subjects Experiments In a between-subjects experiment, each participant is tested in only one condition. For example, a researcher with a sample of 100 college students might assign half of them to write about a traumatic event and the other half write about a neutral event. Or a researcher with a sample of 60 people with severe agoraphobia (fear of open spaces) might assign 20 of them to receive each of three different treatments for that disorder. It is essential in a between-subjects experiment that the researcher assign participants to conditions so that the different groups are, on average, highly similar to each other. Those in a trauma condition and a neutral condition, for example, should include a similar proportion of men and women, and they should have similar average intelligence quotients (IQs), similar average levels of motivation, similar average numbers of health problems, and so on. This is a matter of controlling these extraneous participant variables across conditions so that they do not become confounding variables.

Random Assignment The primary way that researchers accomplish this kind of control of extraneous variables across conditions is called random assignment, which means using a random process to decide which participants are tested in which conditions. Do not confuse random assignment with random sampling. Random sampling is a method for selecting a sample from a population, and it is rarely used in psychological research. Random assignment is a method for assigning participants in a sample to the different conditions, and it is an important element of all experimental research in psychology and other fields too.

In its strictest sense, random assignment should meet two criteria. One is that each participant has an equal chance of being assigned to each condition (e.g., a 50% chance of being assigned to each of two conditions). The second is that each participant is assigned to a condition independently of other participants. Thus, one way to assign participants to two conditions would be to flip a coin for each one. If the coin lands heads, the participant is assigned to Condition A, and if it lands tails, the participant is assigned to Condition B. For three conditions, one could use a computer to generate a random integer from 1 to 3 for each participant. If the integer is 1, the participant is assigned to Condition A; if it is 2, the participant is assigned to Condition B; and if it is 3, the participant is assigned to Condition C. In practice, a full sequence of conditions—one for each participant expected to be in the experiment—is usually created ahead of time, and each new participant is assigned to the next condition in the sequence as he or she is tested. When the procedure is computerized, the computer program often handles the random assignment.

Block Randomization One problem with coin flipping and other strict procedures for random assignment is that they are likely to result in unequal sample sizes in the different conditions. Unequal sample sizes are generally not a serious problem, and you should never throw away data you have already collected to achieve equal sample sizes. However, for a fixed number of participants, it is statistically most efficient to divide them into equal-sized groups. It is standard practice, therefore, to use a kind of modified random assignment that keeps the

RESEARCH FOR HUMAN SERVICES pg. 88

number of participants in each group as similar as possible. One approach is block randomization. In block randomization, all the conditions occur once in the sequence before any of them is repeated. Then they all occur again before any of them is repeated again. Within each of these “blocks,” the conditions occur in a random order. Again, the sequence of conditions is usually generated before any participants are tested, and each new participant is assigned to the next condition in the sequence. Table 6.2 "Block Randomization Sequence for Assigning Nine Participants to Three Conditions" shows such a sequence for assigning nine participants to three conditions. The Research Randomizer website (http://www.randomizer.org) will generate block randomization sequences for any number of participants and conditions. Again, when the procedure is computerized, the computer program often handles the block randomization.

Random assignment is not guaranteed to control all extraneous variables across conditions. It is always possible that just by chance, the participants in one condition might turn out to be substantially older, less tired, more motivated, or less depressed on average than the participants in another condition. However, there are some reasons that this is not a major concern. One is that random assignment works better than one might expect, especially for large samples. Another is that the inferential statistics that researchers use to decide whether a difference between groups reflects a difference in the population takes the “fallibility” of random assignment into account. Yet another reason is that even if random assignment does result in a confounding variable and therefore produces misleading results, this is likely to be detected when the experiment is replicated. The upshot is that random assignment to conditions—although not infallible in terms of controlling extraneous variables—is always considered a strength of a research design.

Treatment and Control Conditions Between-subjects experiments are often used to determine whether a treatment works. In psychological research, a treatment is any intervention meant to change people’s behavior for the better. This includes psychotherapies and medical treatments for psychological disorders but also interventions designed to improve learning, promote conservation, reduce prejudice, and so on. To determine whether a treatment works, participants are randomly assigned to either a treatment condition, in which they receive the treatment, or a control condition, in which they do not receive the treatment. If participants in the treatment condition end up better off than participants in the control condition—for example, they are less depressed, learn faster, conserve more, express less prejudice—then the researcher can conclude that the treatment works. In research on the effectiveness of psychotherapies and medical treatments, this type of experiment is often called a randomized clinical trial.

No-Treatment Control Condition & the Placebo Effect There are different types of control conditions. In a no-treatment control condition, participants receive no treatment whatsoever. One problem with this approach, however, is the existence of placebo effects. A placebo is a simulated treatment that lacks any active ingredient or element that should make it effective, and a placebo effect is a positive effect of such a treatment. Many folk remedies that seem to work—such as eating chicken soup for a cold or placing soap under the bedsheets to stop nighttime leg cramps—are probably nothing more than placebos. Although placebo effects are not well understood, they are probably driven primarily by people’s expectations that they will improve. Having the expectation to improve can result in reduced stress, anxiety, and depression, which can alter perceptions and even improve immune system functioning (Price, Finniss, & Benedetti, 2008). [1]

Placebo effects are interesting in their own right (see Note 6.28 "The Powerful Placebo"), but they also pose a serious problem for researchers who want to determine whether a treatment works. Figure 6.2 "Hypothetical Results From a Study Including Treatment, No-Treatment, and Placebo Conditions" shows some hypothetical results in which participants in a treatment condition improved more on average than participants in a no-treatment control condition. If these conditions (the two leftmost bars in Figure 6.2 "Hypothetical Results from a Study Including Treatment, No-Treatment, and Placebo Conditions") were the only conditions in this experiment, however, one could not conclude that the treatment worked. It could be instead that participants in the treatment group improved more because they expected to improve, while those in the no-treatment control condition did not.

RESEARCH FOR HUMAN SERVICES pg. 89

Figure 6.2 Hypothetical Results from a Study Including Treatment, No-Treatment, and Placebo Conditions

Fortunately, there are several solutions to this problem. One is to include a placebo control condition, in which participants receive a placebo that looks much like the treatment but lacks the active ingredient or element thought to be responsible for the treatment’s effectiveness. When participants in a treatment condition take a pill, for example, then those in a placebo control condition would take an identical-looking pill that lacks the active ingredient in the treatment (a “sugar pill”). In research on psychotherapy effectiveness, the placebo might involve going to a psychotherapist and talking in an unstructured way about one’s problems. The idea is that if participants in both the treatment and the placebo control groups expect to improve, then any improvement in the treatment group over and above that in the placebo control group must have been caused by the treatment and not by participants’ expectations. This is what is shown by a comparison of the two outer bars in Figure 6.2 "Hypothetical Results from a Study Including Treatment, No-Treatment, and Placebo Conditions".

Of course, the principle of informed consent requires that participants be told that they will be assigned to either a treatment or a placebo control condition—even though they cannot be told which until the experiment ends. In many cases the participants who had been in the control condition are then offered an opportunity to have the real treatment. An alternative approach is to use a waitlist control condition, in which participants are told that they will receive the treatment but must wait until the participants in the treatment condition have already received it. This allows researchers to compare participants who have

RESEARCH FOR HUMAN SERVICES pg. 90

received the treatment with participants who are not currently receiving it but who still expect to improve (eventually). A final solution to the problem of placebo effects is to leave out the control condition completely and compare any new treatment with the best available alternative treatment. For example, a new treatment for simple phobia could be compared with standard exposure therapy. Because participants in both conditions receive a treatment, their expectations about improvement should be similar. This approach also makes sense because once there is an effective treatment, the interesting question about a new treatment is not simply “Does it work?” but “Does it work better than what is already available?”

The Powerful Placebo Many people are not surprised that placebos can have a positive effect on disorders that seem fundamentally psychological, including depression, anxiety, and insomnia. However, placebos can also have a positive effect on disorders that most people think of as fundamentally physiological. These include asthma, ulcers, and warts (Shapiro & Shapiro, 1999). [2] There is even evidence that placebo surgery—also called “sham surgery”—can be as effective as actual surgery.

Medical researcher J. Bruce Moseley and his colleagues conducted a study on the effectiveness of two arthroscopic surgery procedures for osteoarthritis of the knee (Moseley et al., 2002). [3] The control participants in this study were prepped for surgery, received a tranquilizer, and even received three small incisions in their knees. But they did not receive the actual arthroscopic surgical procedure. The surprising result was that all participants improved in terms of both knee pain and function, and the sham surgery group improved just as much as the treatment groups. According to the researchers, “This study provides strong evidence that arthroscopic lavage with or without débridement [the surgical procedures used] is not better than and appears to be equivalent to a placebo procedure in improving knee pain and self-reported function” (p. 85).

Within-Subjects Experiments In a within-subjects experiment, each participant is tested under all conditions. Consider an experiment on the effect of a defendant’s physical attractiveness on judgments of his guilt. Again, in a between- subjects experiment, one group of participants would be shown an attractive defendant and asked to judge his guilt, and another group of participants would be shown an unattractive defendant and asked to judge his guilt. In a within-subjects experiment, however, the same group of participants would judge the guilt of both an attractive and an unattractive defendant.

The primary advantage of this approach is that it provides maximum control of extraneous participant variables. Participants in all conditions have the same mean IQ, same socioeconomic status, same number of siblings, and so on—because they are the very same people. Within-subjects experiments also make it possible to use statistical procedures that remove the effect of these extraneous participant variables on the dependent variable and therefore make the data less “noisy” and the effect of the independent variable easier to detect. We will look more closely at this idea later in the book.

Carryover Effects and Counterbalancing The primary disadvantage of within-subjects designs is that they can result in carryover effects. A carryover effect is an effect of being tested in one condition on participants’ behavior in later conditions. One type of carryover effect is a practice effect, where participants perform a task better in later conditions because they have had a chance to practice it. Another type is a fatigue effect, where participants perform a task worse in later conditions because they become tired or bored.

Being tested in one condition can also change how participants perceive stimuli or interpret their task in later conditions. This is called a context effect. For example, an average-looking defendant might be judged more harshly when participants have just judged an attractive defendant than when they have just judged an unattractive defendant. Within-subjects experiments also make it easier for participants to guess the hypothesis. For example, a participant who is asked to judge the guilt of an attractive defendant and then is asked to judge the guilt of an unattractive defendant is likely to guess that the hypothesis is that defendant attractiveness affects judgments of guilt. This could lead the participant to judge the unattractive

RESEARCH FOR HUMAN SERVICES pg. 91

defendant more harshly because he thinks this is what he is expected to do. Or it could make participants judge the two defendants similarly in an effort to be “fair.”

Carryover effects can be interesting in their own right. (Does the attractiveness of one person depend on the attractiveness of other people that we have seen recently?) But when they are not the focus of the research, carryover effects can be problematic. Imagine, for example, that participants judge the guilt of an attractive defendant and then judge the guilt of an unattractive defendant. If they judge the unattractive defendant more harshly, this might be because of his unattractiveness. But it could be instead that they judge him more harshly because they are becoming bored or tired. In other words, the order of the conditions is a confounding variable. The attractive condition is always the first condition and the unattractive condition the second. Thus, any difference between the conditions in terms of the dependent variable could be caused by the order of the conditions and not the independent variable itself.

Counterbalancing There is a solution to the problem of order effects, however, that can be used in many situations. It is counterbalancing, which means testing different participants in different orders. For example, some participants would be tested in the attractive defendant condition followed by the unattractive defendant condition, and others would be tested in the unattractive condition followed by the attractive condition. With three conditions, there would be six different orders (ABC, ACB, BAC, BCA, CAB, and CBA), so some participants would be tested in each of the six orders. With counterbalancing, participants are assigned to orders randomly, using the techniques we have already discussed. Thus, random assignment plays an important role in within-subjects designs just as in between-subjects designs. Here, instead of randomly assigning to conditions, they are randomly assigned to different orders of conditions. In fact, it can safely be said that if a study does not involve random assignment in one form or another, it is not an experiment.

There are two ways to think about what counterbalancing accomplishes. One is that it controls the order of conditions so that it is no longer a confounding variable. Instead of the attractive condition always being first and the unattractive condition always being second, the attractive condition comes first for some participants and second for others. Likewise, the unattractive condition comes first for some participants and second for others. Thus, any overall difference in the dependent variable between the two conditions cannot have been caused by the order of conditions. A second way to think about what counterbalancing accomplishes is that if there are carryover effects, it makes it possible to detect them. One can analyze the data separately for each order to see whether it had an effect.

When 9 Is “Larger” Than 221 Researcher Michael Birnbaum has argued that the lack of context provided by between-subjects designs is often a bigger problem than the context effects created by within-subjects designs. To demonstrate this, he asked one group of participants to rate how large the number 9 was on a 1-to-10 rating scale and another group to rate how large the number 221 was on the same 1-to-10 rating scale (Birnbaum, 1999). [4] Participants in this between-subjects design gave the number 9 a mean rating of 5.13 and the number 221 a mean rating of 3.10. In other words, they rated 9 as larger than 221! According to Birnbaum, this is because participants spontaneously compared 9 with other one-digit numbers (in which case it is relatively large) and compared 221 with other three-digit numbers (in which case it is relatively small).

Simultaneous Within-Subjects Designs So far, we have discussed an approach to within-subjects designs in which participants are tested in one condition at a time. There is another approach, however, that is often used when participants make multiple responses in each condition. Imagine, for example, that participants judge the guilt of 10 attractive defendants and 10 unattractive defendants. Instead of having people make judgments about all 10 defendants of one type followed by all 10 defendants of the other type, the researcher could present all 20 defendants in a sequence that mixed the two types. The researcher could then compute each participant’s mean rating for each type of defendant. Or imagine an experiment designed to see whether people with social anxiety disorder remember negative adjectives (e.g., “stupid,” “incompetent”) better than positive

RESEARCH FOR HUMAN SERVICES pg. 92

ones (e.g., “happy,” “productive”). The researcher could have participants study a single list that includes both kinds of words and then have them try to recall as many words as possible. The researcher could then count the number of each type of word that was recalled. There are many ways to determine the order in which the stimuli are presented, but one common way is to generate a different random order for each participant.

Between-Subjects or Within-Subjects? Almost every experiment can be conducted using either a between-subjects design or a within-subjects design. This means that researchers must choose between the two approaches based on their relative merits for the particular situation.

Between-subjects experiments have the advantage of being conceptually simpler and requiring less testing time per participant. They also avoid carryover effects without the need for counterbalancing. Within- subjects experiments have the advantage of controlling extraneous participant variables, which generally reduces noise in the data and makes it easier to detect a relationship between the independent and dependent variables.

A good rule of thumb, then, is that if it is possible to conduct a within-subjects experiment (with proper counterbalancing) in the time that is available per participant—and you have no serious concerns about carryover effects—this is probably the best option. If a within-subjects design would be difficult or impossible to carry out, then you should consider a between-subjects design instead. For example, if you were testing participants in a doctor’s waiting room or shoppers in line at a grocery store, you might not have enough time to test each participant in all conditions and therefore would opt for a between-subjects design. Or imagine you were trying to reduce people’s level of prejudice by having them interact with someone of another race. A within-subjects design with counterbalancing would require testing some participants in the treatment condition first and then in a control condition. But if the treatment works and reduces people’s level of prejudice, then they would no longer be suitable for testing in the control condition. This is true for many designs that involve a treatment meant to produce long-term change in participants’ behavior (e.g., studies testing the effectiveness of psychotherapy). Clearly, a between- subjects design would be necessary here.

Remember also that using one type of design does not preclude using the other type in a different study. There is no reason that a researcher could not use both a between-subjects design and a within-subjects design to answer the same research question. In fact, professional researchers often do exactly this.

E AW AY S

Summary:  Experiments can be conducted using either between-subjects or within-subjects designs. Deciding

which to use in a particular situation requires careful consideration of the pros and cons of each approach.

 Random assignment to conditions in between-subjects experiments or to orders of conditions in within-subjects experiments is a fundamental element of experimental research. Its purpose is to control extraneous variables so that they do not become confounding variables.

 Experimental research on the effectiveness of a treatment requires both a treatment condition and a control condition, which can be a no-treatment control condition, a placebo control condition, or a waitlist control condition. Experimental treatments can also be compared with the best available alternative.

EX ER C IS ES

RESEARCH FOR HUMAN SERVICES pg. 93

Discussion For each of the following topics, list the pros and cons of a between-subjects and within-subjects design and decide which would be better.

 You want to test the relative effectiveness of two training programs for running a marathon.

 Using photographs of people as stimuli, you want to see if smiling people are perceived as more intelligent than people who are not smiling.

 In a field experiment, you want to see if the way a panhandler is dressed (neatly vs. sloppily) affects whether or not passersby give him any money.

 You want to see if concrete nouns (e.g., dog) are recalled better than abstract nouns (e.g., truth).

Imagine that an experiment shows that participants who receive psychodynamic therapy for a dog phobia improve more than participants in a no-treatment control group. Explain a fundamental problem with this research design and at least two ways that it might be corrected.

[1] Price, D. D., Finniss, D. G., & Benedetti, F. (2008). A comprehensive review of the placebo effect: Recent advances and current thought. Annual Review of Psychology, 59, 565–590.

[2] Shapiro, A. K., & Shapiro, E. (1999). The powerful placebo: From ancient priest to modern physician. Baltimore, MD: Johns Hopkins University Press.

[3] Moseley, J. B., O’Malley, K., Petersen, N. J., Menke, T. J., Brody, B. A., Kuykendall, D. H., … Wray, N. P. (2002). A controlled trial of arthroscopic surgery for osteoarthritis of the knee. The New England Journal of Medicine, 347, 81–88.

6.3 Conducting Experiments

LEARNING OBJECTIVES  Describe several strategies for recruiting participants for an experiment.

 Explain why it is important to standardize the procedure of an experiment and several ways to do this.

 Explain what pilot testing is and why it is important.

The information presented so far in this chapter is enough to design a basic experiment. When it comes time to conduct that experiment, however, several additional practical issues arise. In this section, we consider some of these issues and how to deal with them. Much of this information applies to nonexperimental studies as well as experimental ones.

Recruiting Participants Of course, you should be thinking about how you will obtain your participants from the beginning of any research project. Unless you have access to people with schizophrenia or incarcerated juvenile offenders, for example, then there is no point designing a study that focuses on these populations. But even if you plan to use a convenience sample, you will have to recruit participants for your study.

There are several approaches to recruiting participants. One is to use participants from a formal subject pool—an established group of people who have agreed to be contacted about participating in research studies. For example, at many colleges and universities, there is a subject pool consisting of students enrolled in introductory psychology courses who must participate in a certain number of studies to meet a course requirement. This is a common practice in psychology, but very rare elsewhere.

RESEARCH FOR HUMAN SERVICES pg. 94

Researchers post descriptions of their studies and students sign up to participate, usually via an online system. Participants who are not in subject pools can also be recruited by posting or publishing advertisements or making personal appeals to groups that represent the population of interest. For example, a researcher interested in studying older adults could arrange to speak at a meeting of the residents at a retirement community to explain the study and ask for volunteers.

The Volunteer Subject Even if the participants in a study receive compensation in the form of course credit, a small amount of money, or a chance at being treated for a psychological problem, they are still essentially volunteers. This is worth considering because people who volunteer to participate in psychological research have been shown to differ in predictable ways from those who do not volunteer. Specifically, there is good evidence that on average, volunteers have the following characteristics compared with non-volunteers (Rosenthal & Rosnow, 1976): [1]

Characteristics of Volunteer Participants  They are more interested in the topic of the research.

 They are more educated.

 They have a greater need for approval.

 They have higher intelligence quotients (IQs).

 They are more sociable.

 They are higher in social class. This can be an issue of external validity if there is reason to believe that participants with these characteristics are likely to behave differently than the general population. For example, in testing different methods of persuading people, a rational argument might work better on volunteers than it does on the general population because of their generally higher educational level and IQ.

In many field experiments, the task is not recruiting participants but selecting them. For example, researchers Nicolas Guéguen and Marie-Agnès de Gail conducted a field experiment on the effect of being smiled at on helping, in which the participants were shoppers at a supermarket. A confederate walking down a stairway gazed directly at a shopper walking up the stairway and either smiled or did not smile. Shortly afterward, the shopper encountered another confederate, who dropped some computer diskettes on the ground. The dependent variable was whether or not the shopper stopped to help pick up the diskettes (Guéguen & de Gail, 2003). [2] Notice that these participants were not “recruited,” but the researchers still had to select them from among all the shoppers taking the stairs that day. It is extremely important that this kind of selection be done according to a well-defined set of rules that is established before the data collection begins and can be explained clearly afterward. In this case, with each trip down the stairs, the confederate was instructed to gaze at the first person he encountered who appeared to be between the ages of 20 and 50. Only if the person gazed back did he or she become a participant in the study. The point of having a well-defined selection rule is to avoid bias in the selection of participants. For example, if the confederate was free to choose which shoppers he would gaze at, he might choose friendly-looking shoppers when he was set to smile and unfriendly-looking ones when he was not set to smile. As we will see shortly, such biases can be entirely unintentional.

Standardizing the Procedure It is surprisingly easy to introduce extraneous variables during the procedure. For example, the same experimenter might give clear instructions to one participant but vague instructions to another. Or one experimenter might greet participants warmly while another barely makes eye contact with them. To the extent that such variables affect participants’ behavior, they add noise to the data and make the effect of the independent variable more difficult to detect. If they vary across conditions, they become confounding variables and provide alternative explanations for the results. For example, if participants in a treatment group are tested by a warm and friendly experimenter and participants in a control group are tested by a

RESEARCH FOR HUMAN SERVICES pg. 95

cold and unfriendly one, then what appears to be an effect of the treatment might actually be an effect of experimenter demeanor.

Experimenter Gender as an Extraneous Variable It is well known that whether research participants are male or female can affect the results of a study. But what about whether the experimenter is male or female? There is plenty of evidence that this matters too. Male and female experimenters have slightly different ways of interacting with their participants, and of course participants also respond differently to male and female experimenters (Rosenthal, 1976). [3] For example, in a recent study on pain perception, participants immersed their hands in icy water for as long as they could (Ibolya, Brake, & Voss, 2004). [4] Male participants tolerated the pain longer when the experimenter was a woman, and female participants tolerated it longer when the experimenter was a man.

Experimenter Expectancy Effect Researcher Robert Rosenthal has spent much of his career showing that this kind of unintended variation in the procedure does, in fact, affect participants’ behavior. Furthermore, one important source of such variation is the experimenter’s expectations about how participants “should” behave in the experiment. This is referred to as an experimenter expectancy effect (Rosenthal, 1976). [5] For example, if an experimenter expects participants in a treatment group to perform better on a task than participants in a control group, then he or she might unintentionally give the treatment group participants clearer instructions or more encouragement or allow them more time to complete the task. In a striking example, Rosenthal and Kermit Fode had several students in a laboratory course in psychology train rats to run through a maze. Although the rats were genetically similar, some of the students were told that they were working with “maze-bright” rats that had been bred to be good learners, and other students were told that they were working with “maze-dull” rats that had been bred to be poor learners. Sure enough, over five days of training, the “maze-bright” rats made more correct responses, made the correct response more quickly, and improved more steadily than the “maze-dull” rats (Rosenthal & Fode, 1963). [6] Clearly it had to have been the students’ expectations about how the rats would perform that made the difference. But how? Some clues come from data gathered at the end of the study, which showed that students who expected their rats to learn quickly felt more positively about their animals and reported behaving toward them in a more friendly manner (e.g., handling them more).

The way to minimize unintended variation in the procedure is to standardize it as much as possible so that it is carried out in the same way for all participants regardless of the condition they are in. Here are several ways to do this:

 Create a written protocol that specifies everything that the experimenters are to do and say from the time they greet participants to the time they dismiss them.

 Create standard instructions that participants read themselves or that are read to them word for word by the experimenter.

 Automate the rest of the procedure as much as possible by using software packages for this purpose or even simple computer slide shows.

 Anticipate participants’ questions and either raise and answer them in the instructions or develop standard answers for them.

 Train multiple experimenters on the protocol together and have them practice on each other.

 Be sure that each experimenter tests participants in all conditions.

Another good practice is to arrange for the experimenters to be “blind” to the research question or to the condition that each participant is tested in. The idea is to minimize experimenter expectancy effects by minimizing the experimenters’ expectations. For example, in a drug study in which each participant receives the drug or a placebo, it is often the case that neither the participants nor the experimenter who interacts with the participants know which condition he or she has been assigned to. Because both the participants and the experimenters are blind to the condition, this is referred to as a double-blind study. (A single-blind study is one in which the participant, but not the experimenter, is blind to the condition.) Of

RESEARCH FOR HUMAN SERVICES pg. 96

course, there are many times this is not possible. For example, if you are both the investigator and the only experimenter, it is not possible for you to remain blind to the research question. Also, in many studies the experimenter must know the condition because he or she must carry out the procedure in a different way in the different conditions.

Record Keeping It is essential to keep good records when you conduct an experiment. As discussed earlier, it is typical for experimenters to generate a written sequence of conditions before the study begins and then to test each new participant in the next condition in the sequence. As you test them, it is a good idea to add to this list basic demographic information; the date, time, and place of testing; and the name of the experimenter who did the testing. It is also a good idea to have a place for the experimenter to write down comments about unusual occurrences (e.g., a confused or uncooperative participant) or questions that come up. This kind of information can be useful later if you decide to analyze sex differences or effects of different experimenters, or if a question arises about a particular participant or testing session.

It can also be useful to assign an identification number to each participant as you test them. Simply numbering them consecutively beginning with 1 is usually sufficient. This number can then also be written on any response sheets or questionnaires that participants generate, making it easier to keep them together.

Pilot Testing It is always a good idea to conduct a pilot test of your research or experiment. A pilot test is a small-scale study conducted to make sure that a new procedure works as planned. In a pilot test, you can recruit participants formally (e.g., from an established participant pool) or you can recruit them informally from among family, friends, classmates, and so on. The number of participants can be small, but it should be enough to give you confidence that your procedure works as planned. There are several important questions that you can answer by conducting a pilot test:

 Do participants understand the instructions?

 What kind of misunderstandings do participants have, what kind of mistakes do they make, and what kind of questions do they ask?

 Do participants become bored or frustrated?

 Is an indirect manipulation effective? (You will need to include a manipulation check.)

 Can participants guess the research question or hypothesis?

 How long does the procedure take?

 Are computer programs or other automated procedures working properly?

 Are data being recorded correctly?

Of course, to answer some of these questions you will need to observe participants carefully during the procedure and talk with them about it afterward. Participants are often hesitant to criticize a study in front of the researcher, so be sure they understand that this is a pilot test and you are genuinely interested in feedback that will help you improve the procedure. If the procedure works as planned, then you can proceed with the actual study. If there are problems to be solved, you can solve them, pilot test the new procedure, and continue with this process until you are ready to proceed.

T AK EW AYS

Summary:  There are several effective methods you can use to recruit research participants for your

experiment, including through formal subject pools, advertisements, and personal appeals. Field experiments require well-defined participant selection procedures.

 It is important to standardize experimental procedures to minimize extraneous variables, including experimenter expectancy effects.

RESEARCH FOR HUMAN SERVICES pg. 97

 It is important to conduct one or more small-scale pilot tests of an experiment to be sure that the procedure works as planned.

EX R C ISE S

Practice List two ways that you might recruit participants from each of the following populations: (a) elderly adults, (b) unemployed people, (c) regular exercisers, and (d) math majors.

Discussion Imagine a study in which you will visually present participants with a list of 20 words, one at a time, wait for a short time, and then ask them to recall as many of the words as they can. In the stressed condition, they are told that they might also be chosen to give a short speech in front of a small audience. In the unstressed condition, they are not told that they might have to give a speech. What are several specific things that you could do to standardize the procedure?

[1] Rosenthal, R., & Rosnow, R. L. (1976). The volunteer subject. New York, NY: Wiley.

[2] Guéguen, N., & de Gail, Marie-Agnès. (2003). The effect of smiling on helping behavior: Smiling and good Samaritan behavior. Communication Reports, 16, 133–140.

[3] Rosenthal, R. (1976). Experimenter effects in behavioral research (enlarged ed.). New York, NY: Wiley.

[4] Ibolya, K., Brake, A., & Voss, U. (2004). The effect of experimenter characteristics on pain reports in women and men. Pain, 112, 142–147.

[5] Rosenthal, R. (1976). Experimenter effects in behavioral research (enlarged ed.). New York, NY: Wiley.

[6] Rosenthal, R., & Fode, K. (1963). The effect of experimenter bias on performance of the albino rat. Behavioral Science, 8, 183-189.

RESEARCH FOR HUMAN SERVICES pg. 98

Chapter 7: Nonexperimental Research I love science. I hate supposition, superstition, exaggeration and falsified data. Show me the research, show me the results, show me the conclusions - and then show me some qualified peer reviews of all that. -Claire Scovell LaZebnik

What do the following classic studies have in common?

 Stanley Milgram found that about two thirds of his research participants were willing to administer dangerous shocks to another person just because they were told to by an authority figure (Milgram, 1963). [1]

 Elizabeth Loftus and Jacqueline Pickrell showed that it is relatively easy to “implant” false memories in people by repeatedly asking them about childhood events that did not actually happen to them (Loftus & Pickrell, 1995). [2]

 John Cacioppo and Richard Petty evaluated the validity of their Need for Cognition Scale—a measure of the extent to which people like and value thinking—by comparing the scores of college professors with those of factory workers (Cacioppo & Petty, 1982). [3]

 David Rosenhan found that confederates who went to psychiatric hospitals claiming to have heard voices saying things like “empty” and “thud” were labeled as schizophrenic by the hospital staff and kept there even though they behaved normally in all other ways (Rosenhan, 1973). [4]

The answer is that they are not true experiments. In this chapter, we look more closely at nonexperimental research. We begin with a general definition of nonexperimental research, along with a discussion of when and why nonexperimental research is more appropriate than experimental research. We then look separately at three important types of nonexperimental research: correlational research, quasi- experimental research, and qualitative research.

[1] Milgram, S. (1963). Behavioral study of obedience. Journal of Abnormal and Social Psychology, 67, 371–378.

[2] Loftus, E. F., & Pickrell, J. E. (1995). The formation of false memories. Psychiatric Annals, 25, 720–725.

[3] Cacioppo, J. T., & Petty, R. E. (1982). The need for cognition. Journal of Personality and Social Psychology, 42, 116–131.

[4] Rosenhan, D. L. (1973). On being sane in insane places. Science, 179, 250–258.

7.1 Overview of Nonexperimental Research

LEARNING OBJECTIVES  Define nonexperimental research, distinguish it clearly from experimental research, and give

several examples.

 Explain when a researcher might choose to conduct nonexperimental research as opposed to experimental research.

What Is Nonexperimental Research? Nonexperimental research is research that lacks the manipulation of an independent variable, random assignment of participants to conditions or orders of conditions, or both.

RESEARCH FOR HUMAN SERVICES pg. 99

In a sense, it is unfair to define this large and diverse set of approaches collectively by what they are not. But doing so reflects the fact that most researchers in psychology, not so much in the other social sciences, consider the distinction between experimental and nonexperimental research to be an extremely important one. This is because while experimental research can provide strong evidence that changes in an independent variable cause differences in a dependent variable, nonexperimental research generally cannot. As we will see, however, this does not mean that nonexperimental research is less important than experimental research or inferior to it in any general sense.

When to Use Nonexperimental Research As we saw in Chapter 6 "Experimental Research", experimental research is appropriate when the researcher has a specific research question or hypothesis about a causal relationship between two variables—and it is possible, feasible, and ethical to manipulate the independent variable and randomly assign participants to conditions or to orders of conditions. It stands to reason, therefore, that nonexperimental research is appropriate—even necessary—when these conditions are not met. There are many ways in which this can be the case.

 The research question or hypothesis can be about a single variable rather than a statistical relationship between two variables (e.g., How accurate are people’s first impressions?).

 The research question can be about a non-causal statistical relationship between variables (e.g., Is there a correlation between verbal intelligence and mathematical intelligence?).

 The research question can be about a causal relationship, but the independent variable cannot be manipulated or participants cannot be randomly assigned to conditions or orders of conditions (e.g., Does damage to a person’s hippocampus impair the formation of long-term memory traces?).

 The research question can be broad and exploratory, or it can be about what it is like to have a particular experience (e.g., What is it like to be a working mother diagnosed with depression?).

Again, the choice between the experimental and nonexperimental approaches is generally dictated by the nature of the research question. If it is about a causal relationship and involves an independent variable that can be manipulated, the experimental approach is typically preferred. Otherwise, the nonexperimental approach is preferred. But the two approaches can also be used to address the same research question in complementary ways. For example, nonexperimental studies establishing that there is a relationship between watching violent television and aggressive behavior have been complemented by experimental studies confirming that the relationship is a causal one (Bushman & Huesmann, 2001). [1] Similarly, after his original study, Milgram conducted experiments to explore the factors that affect obedience. He manipulated several independent variables, such as the distance between the experimenter and the participant, the participant and the confederate, and the location of the study (Milgram, 1974). [2]

Types of Nonexperimental Research Nonexperimental research falls into three broad categories: single-variable research, correlational and quasi-experimental research, and qualitative research. First, research can be nonexperimental because it focuses on a single variable rather than a statistical relationship between two variables. Although there is no widely shared term for this kind of research, we will call it single-variable research. Milgram’s original obedience study was nonexperimental in this way. He was primarily interested in one variable—the extent to which participants obeyed the researcher when he told them to shock the confederate—and he observed all participants performing the same task under the same conditions. The study by Loftus and Pickrell described at the beginning of this chapter is also a good example of single-variable research. The variable was whether participants “remembered” having experienced mildly traumatic childhood events (e.g., getting lost in a shopping mall) that they had not actually experienced but that the research asked them about repeatedly. In this particular study, nearly a third of the participants “remembered” at least one event. (As with Milgram’s original study, this study inspired several later experiments on the factors that affect false memories.)

As these examples make clear, single-variable research can answer interesting and important questions. What it cannot do, however, is answer questions about statistical relationships between variables. This is a point that beginning researchers sometimes miss. Imagine, for example, a group of research methods

RESEARCH FOR HUMAN SERVICES pg. 100

students interested in the relationship between children’s being the victim of bullying and the children’s self- esteem. The first thing that is likely to occur to these researchers is to obtain a sample of middle-school students who have been bullied and then to measure their self-esteem. But this would be a single-variable study with self-esteem as the only variable. Although it would tell the researchers something about the self- esteem of children who have been bullied, it would not tell them what they really want to know, which is how the self-esteem of children who have been bullied compares with the self-esteem of children who have not. Is it lower? Is it the same? Could it even be higher? To answer this question, their sample would also have to include middle-school students who have not been bullied.

Nonexperiments

Correlational and Quasi-Experimental Research Research can also be nonexperimental because it focuses on a statistical relationship between two variables but does not include the manipulation of an independent variable, random assignment of participants to conditions or orders of conditions, or both. This kind of research takes two basic forms: correlational research and quasi-experimental research. In correlational research, the researcher measures the two variables of interest with little or no attempt to control extraneous variables and then assesses the relationship between them. A research methods student who finds out whether each of several middle-school students has been bullied and then measures each student’s self-esteem is conducting correlational research. In quasi-experimental research, the researcher manipulates an independent variable but does not randomly assign participants to conditions or orders of conditions. For example, a researcher might start an antibullying program (a kind of treatment) at one school and compare the incidence of bullying at that school with the incidence at a similar school that has no antibullying program.

Qualitative Research The final way in which research can be nonexperimental is that it can be qualitative. The types of research we have discussed so far are all quantitative, referring to the fact that the data consist of numbers that are analyzed using statistical techniques. In qualitative research, the data are usually nonnumerical and are analyzed using non-statistical techniques. Rosenhan’s study of the experience of people in a psychiatric ward was primarily qualitative. The data were the notes taken by the “pseudopatients”—the people pretending to have heard voices—along with their hospital records. Rosenhan’s analysis consists mainly of a written description of the experiences of the pseudopatients, supported by several concrete examples. To illustrate the hospital staff’s tendency to “depersonalize” their patients, he noted, “Upon being admitted, I and other pseudopatients took the initial physical examinations in a semipublic room, where staff members went about their own business as if we were not there” (Rosenhan, 1973, p. 256). [3]

Internal Validity Revisited Recall that internal validity is the extent to which the design of a study supports the conclusion that changes in the independent variable caused any observed differences in the dependent variable. Figure 7.1 shows how experimental, quasi-experimental, and correlational research vary in terms of internal validity. Experimental research tends to be highest because it addresses the directionality and third- variable problems through manipulation and the control of extraneous variables through random assignment. If the average score on the dependent variable in an experiment differs across conditions, it is quite likely that the independent variable is responsible for that difference. Correlational research is lowest because it fails to address either problem. If the average score on the dependent variable differs across levels of the independent variable, it could be that the independent variable is responsible, but there are other interpretations. In some situations, the direction of causality could be reversed. In others, there could be a third variable that is causing differences in both the independent and dependent variables. Quasi- experimental research is in the middle because the manipulation of the independent variable addresses some problems, but the lack of random assignment and experimental control fails to address others. Imagine, for example, that a researcher finds two similar schools, starts an antibullying program in one, and then finds fewer bullying incidents in that “treatment school” than in the “control school.” There is no directionality problem because clearly the number of bullying incidents did not determine which school got the program. However, the lack of random assignment of children to schools could still mean that students

RESEARCH FOR HUMAN SERVICES pg. 101

in the treatment school differed from students in the control school in some other way that could explain the difference in bullying.

Figure 7.1

Experiments are generally high in internal validity, quasi-experiments lower, and correlational studies lower still.

Notice also in Figure 7.1 that there is some overlap in the internal validity of experiments, quasi- experiments, and correlational studies. For example, a poorly designed experiment that includes many confounding variables can be lower in internal validity than a well designed quasi-experiment with no obvious confounding variables.

Summary:

 Nonexperimental research is research that lacks the manipulation of an independent variable, control of extraneous variables through random assignment, or both.

 There are three broad types of nonexperimental research. o Single-variable research focuses on a single variable rather than a relationship between

variables. o Correlational and quasi-experimental research focus on a statistical relationship but lack

manipulation or random assignment. o Qualitative research focuses on broader research questions, typically involves collecting

large amounts of data from a small number of participants, and analyzes the data non- statistically.

 In general, experimental research is high in internal validity, correlational research is low in internal validity, and quasi-experimental research is in between.

Discussion For each of the following studies, decide which type of research design it is and explain why.

 A researcher conducts detailed interviews with unmarried teenage fathers to learn about how they feel and what they think about their role as fathers and summarizes their feelings in a written narrative.

 A researcher measures the impulsivity of a large sample of drivers and looks at the statistical relationship between this variable and the number of traffic tickets the drivers have received.

 A researcher randomly assigns patients with low back pain either to a treatment involving hypnosis or to a treatment involving exercise. She then measures their level of low back pain after 3 months.

 A college instructor gives weekly quizzes to students in one section of his course but no weekly quizzes to students in another section to see whether this has an effect on their test performance.

[1] Bushman, B. J., & Huesmann, L. R. (2001). Effects of televised violence on aggression. In D. Singer & J. Singer (Eds.), Handbook of children and the media (pp. 223–254). Thousand Oaks, CA: Sage.

[2] Milgram, S. (1974). Obedience to authority: An experimental view. New York, NY: Harper & Row.

RESEARCH FOR HUMAN SERVICES pg. 102

[3] Rosenhan, D. L. (1973). On being sane in insane places. Science, 179, 250–258.

7.2 Correlational Research

LEARNING OBJECTIVES  Define correlational research and give several examples.

 Explain why a researcher might choose to conduct correlational research rather than experimental research or another type of nonexperimental research.

What Is Correlational Research? Before we go any further it is important to recognize that correlational does not establish causation. Just because A happened, and then B occurred, does not mean that A caused B. Things can be correlated, sometimes just by chance, and even related, but not necessarily achieve cause and effect.

Correlational research is a type of nonexperimental research in which the researcher measures two variables and assesses the statistical relationship (i.e., the correlation) between them with little or no effort to control extraneous variables. There are essentially two reasons that researchers interested in statistical relationships between variables would choose to conduct a correlational study rather than an experiment. The first is that they do not believe that the statistical relationship is a causal one. For example, a researcher might evaluate the validity of a brief extraversion test by administering it to a large group of participants along with a longer extraversion test that has already been shown to be valid. This researcher might then check to see whether participants’ scores on the brief test are strongly correlated with their scores on the longer one. Neither test score is thought to cause the other, so there is no independent variable to manipulate. In fact, the terms independent variable and dependent variable do not apply to this kind of research.

The other reason that researchers would choose to use a correlational study rather than an experiment is that the statistical relationship of interest is thought to be causal, but the researcher cannot manipulate the independent variable because it is impossible, impractical, or unethical. For example, Allen Kanner and his colleagues thought that the number of “daily hassles” (e.g., rude salespeople, heavy traffic) that people experience affects the number of physical and psychological symptoms they have (Kanner, Coyne, Schaefer, & Lazarus, 1981). [1] But because they could not manipulate the number of daily hassles their participants experienced, they had to settle for measuring the number of daily hassles—along with the number of symptoms—using self-report questionnaires. Although the strong positive relationship they found between these two variables is consistent with their idea that hassles cause symptoms, it is also consistent with the idea that symptoms cause hassles or that some third variable (e.g., neuroticism) causes both.

Misconceptions about Correlational Research A common misconception among beginning researchers is that correlational research must involve two quantitative variables, such as scores on two extraversion tests or the number of hassles and number of symptoms people have experienced. However, the defining feature of correlational research is that the two variables are measured—neither one is manipulated—and this is true regardless of whether the variables are quantitative or categorical. Imagine, for example, that a researcher administers the Rosenberg Self- Esteem Scale to 50 American college students and 50 Japanese college students. Although this “feels” like a between-subjects experiment, it is a correlational study because the researcher did not manipulate the students’ nationalities. The same is true of the study by Cacioppo and Petty comparing college faculty and factory workers in terms of their need for cognition. It is a correlational study because the researchers did not manipulate the participants’ occupations.

RESEARCH FOR HUMAN SERVICES pg. 103

Figure 7.2 "Results of a Hypothetical Study on Whether People Who Make Daily To-Do Lists Experience Less Stress Than People Who Do Not Make Such Lists" shows data from a hypothetical study on the relationship between whether people make a daily list of things to do (a “to-do list”) and stress. Notice that it is unclear whether this is an experiment or a correlational study because it is unclear whether the independent variable was manipulated. If the researcher randomly assigned some participants to make daily to-do lists and others not to, then it is an experiment. If the researcher simply asked participants whether they made daily to-do lists, then it is a correlational study. The distinction is important because if the study was an experiment, then it could be concluded that making the daily to-do lists reduced participants’ stress. But if it was a correlational study, it could only be concluded that these variables are statistically related. Perhaps being stressed has a negative effect on people’s ability to plan ahead (the directionality problem). Or perhaps people who are more conscientious are more likely to make to-do lists and less likely to be stressed (the third-variable problem). The crucial point is that what defines a study as experimental or correlational is not the variables being studied, nor whether the variables are quantitative or categorical, nor the type of graph or statistics used to analyze the data. It is how the study is conducted.

FIGURE 7.2 RESULTS OF A HYPOTHETICAL STUDY ON WHETHER PEOPLE WHO MAKE DAILY TO-DO LISTS EXPERIENCE LESS STRESS THAN PEOPLE WHO DO NOT MAKE SUCH LISTS

Data Collection in Correlational Research Again, the defining feature of correlational research is that neither variable is manipulated. It does not matter how or where the variables are measured. A researcher could have participants come to a laboratory to complete a computerized backward digit span task and a computerized risky decision-making task and then assess the relationship between participants’ scores on the two tasks. Or a researcher could go to a shopping mall to ask people about their attitudes toward the environment and their shopping habits

RESEARCH FOR HUMAN SERVICES pg. 104

and then assess the relationship between these two variables. Both of these studies would be correlational because no independent variable is manipulated. However, because some approaches to data collection are strongly associated with correlational research, it makes sense to discuss them here. The two we will focus on are naturalistic observation and archival data. A third, survey research, is discussed in its own chapter.

Naturalistic Observation Naturalistic observation is an approach to data collection that involves observing people’s behavior in the environment in which it typically occurs. Thus, naturalistic observation is a type of field research (as opposed to a type of laboratory research). It could involve observing shoppers in a grocery store, children on a school playground, or psychiatric inpatients in their wards. Researchers engaged in naturalistic observation usually make their observations as unobtrusively as possible so that participants are often not aware that they are being studied. Ethically, this is considered to be acceptable if the participants remain anonymous and the behavior occurs in a public setting where people would not normally have an expectation of privacy. Grocery shoppers putting items into their shopping carts, for example, are engaged in public behavior that is easily observable by store employees and other shoppers. For this reason, most researchers would consider it ethically acceptable to observe them for a study. On the other hand, one of the arguments against the ethicality of the naturalistic observation of “bathroom behavior” discussed earlier in the book is that people have a reasonable expectation of privacy even in a public restroom and that this expectation was violated.

Researchers Robert Levine and Ara Norenzayan used naturalistic observation to study differences in the “pace of life” across countries (Levine & Norenzayan, 1999). [2] One of their measures involved observing pedestrians in a large city to see how long it took them to walk 60 feet. They found that people in some countries walked reliably faster than people in other countries. For example, people in the United States and Japan covered 60 feet in about 12 seconds on average, while people in Brazil and Romania took close to 17 seconds.

Because naturalistic observation takes place in the complex and even chaotic “real world,” there are two closely related issues that researchers must deal with before collecting data. The first is sampling. When, where, and under what conditions will the observations be made, and who exactly will be observed? Levine and Norenzayan described their sampling process as follows:

Male and female walking speed over a distance of 60 feet was measured in at least two locations in main downtown areas in each city. Measurements were taken during main business hours on clear summer days. All locations were flat, unobstructed, had broad sidewalks, and were sufficiently uncrowded to allow pedestrians to move at potentially maximum speeds. To control for the effects of socializing, only pedestrians walking alone were used. Children, individuals with obvious physical handicaps, and window-shoppers were not timed. Thirty-five men and 35 women were timed in most cities. (p. 186)

Precise specification of the sampling process in this way makes data collection manageable for the observers, and it also provides some control over important extraneous variables. For example, by making their observations on clear summer days in all countries, Levine and Norenzayan controlled for effects of the weather on people’s walking speeds.

The Question of What will be Observed The second issue is measurement. What specific behaviors will be observed? In Levine and Norenzayan’s study, measurement was relatively straightforward. They simply measured out a 60-foot distance along a city sidewalk and then used a stopwatch to time participants as they walked over that distance. Often, however, the behaviors of interest are not so obvious or objective. For example, researchers Robert Kraut and Robert Johnston wanted to study bowlers’ reactions to their shots, both when they were facing the pins and then when they turned toward their companions (Kraut & Johnston, 1979). [3] But what “reactions” should they observe? Based on previous research and their own pilot testing, Kraut and Johnston created a list of reactions that included “closed smile,” “open smile,” “laugh,” “neutral face,” “look down,” “look

RESEARCH FOR HUMAN SERVICES pg. 105

away,” and “face cover” (covering one’s face with one’s hands). The observers committed this list to memory and then practiced by coding the reactions of bowlers who had been videotaped. During the actual study, the observers spoke into an audio recorder, describing the reactions they observed. Among the most interesting results of this study was that bowlers rarely smiled while they still faced the pins. They were much more likely to smile after they turned toward their companions, suggesting that smiling is not purely an expression of happiness but also a form of social communication.

Coding Data When the observations require a judgment on the part of the observers—as in Kraut and Johnston’s study—this process is often described as coding. Coding generally requires clearly defining a set of target behaviors. The observers then categorize participants individually in terms of which behavior they have engaged in and the number of times they engaged in each behavior. The observers might even record the duration of each behavior. The target behaviors must be defined in such a way that different observers code them in the same way. This is the issue of interrater reliability. Researchers are expected to demonstrate the interrater reliability of their coding procedure by having multiple raters code the same behaviors independently and then showing that the different observers are in close agreement. Kraut and Johnston, for example, video recorded a subset of their participants’ reactions and had two observers independently code them. The two observers showed that they agreed on the reactions that were exhibited 97% of the time, indicating good interrater reliability.

Archival Data Another approach to correlational research is the use of archival data, which are data that have already been collected for some other purpose. An example is a study by Brett Pelham and his colleagues on “implicit egotism”—the tendency for people to prefer people, places, and things that are similar to themselves (Pelham, Carvallo, & Jones, 2005). [4] In one study, they examined Social Security records to show that women with the names Virginia, Georgia, Louise, and Florence were especially likely to have moved to the states of Virginia, Georgia, Louisiana, and Florida, respectively.

As with naturalistic observation, measurement can be more or less straightforward when working with archival data. For example, counting the number of people named Virginia who live in various states based on Social Security records is relatively straightforward. But consider a study by Christopher Peterson and his colleagues on the relationship between optimism and health using data that had been collected many years before for a study on adult development (Peterson, Seligman, & Vaillant, 1988). [5] In the 1940s, healthy male college students had completed an open-ended questionnaire about difficult wartime experiences. In the late 1980s, Peterson and his colleagues reviewed the men’s questionnaire responses to obtain a measure of explanatory style—their habitual ways of explaining bad events that happen to them. More pessimistic people tend to blame themselves and expect long-term negative consequences that affect many aspects of their lives, while more optimistic people tend to blame outside forces and expect limited negative consequences.

Content Analysis To obtain a measure of explanatory style for each participant, the researchers used a procedure in which all negative events mentioned in the questionnaire responses, and any causal explanations for them, were identified and written on index cards. These were given to a separate group of raters who rated each explanation in terms of three separate dimensions of optimism-pessimism.

These ratings were then averaged to produce an explanatory style score for each participant. The researchers then assessed the statistical relationship between the men’s explanatory style as college students and archival measures of their health at approximately 60 years of age. The primary result was that the more optimistic the men were as college students, the healthier they were as older men. Pearson’s r was +.25.

This is an example of content analysis—a family of systematic approaches to measurement using complex archival data. Just as naturalistic observation requires specifying the behaviors of interest and then noting them as they occur, content analysis requires specifying keywords, phrases, or ideas and then

RESEARCH FOR HUMAN SERVICES pg. 106

finding all occurrences of them in the data. These occurrences can then be counted, timed (e.g., the amount of time devoted to entertainment topics on the nightly news show), or analyzed in a variety of other ways.

TH E Y T AK E AW AYS

Summary:  Correlational research involves measuring two variables and assessing the relationship between

them, with no manipulation of an independent variable.

 Correlational research is not defined by where or how the data are collected. However, some approaches to data collection are strongly associated with correlational research. These include naturalistic observation (in which researchers observe people’s behavior in the context in which it normally occurs) and the use of archival data that were already collected for some other purpose.

EX ER C IS E

Discussion For each of the following, decide whether it is most likely that the study described is experimental or correlational and explain why.

 An educational researcher compares the academic performance of students from the “rich” side of town with that of students from the “poor” side of town.

 A cognitive psychologist compares the ability of people to recall words that they were instructed to “read” with their ability to recall words that they were instructed to “imagine.”

 A manager studies the correlation between new employees’ college grade point averages and their first-year performance reports.

 An automotive engineer installs different stick shifts in a new car prototype, each time asking several people to rate how comfortable the stick shift feels.

 A food scientist studies the relationship between the temperature inside people’s refrigerators and the amount of bacteria on their food.

 A social psychologist tells some research participants that they need to hurry over to the next building to complete a study. She tells others that they can take their time. Then she observes whether they stop to help a research assistant who is pretending to be hurt.

[1] Kanner, A. D., Coyne, J. C., Schaefer, C., & Lazarus, R. S. (1981). Comparison of two modes of stress measurement: Daily hassles and uplifts versus major life events. Journal of Behavioral Medicine, 4, 1–39.

[2] Levine, R. V., & Norenzayan, A. (1999). The pace of life in 31 countries. Journal of Cross-Cultural Psychology, 30, 178–205.

[3] Kraut, R. E., & Johnston, R. E. (1979). Social and emotional messages of smiling: An ethological approach. Journal of Personality and Social Psychology, 37, 1539–1553.

[4] Pelham, B. W., Carvallo, M., & Jones, J. T. (2005). Implicit egotism. Current Directions in Psychological Science, 14, 106–110.

[5] Peterson, C., Seligman, M. E. P., & Vaillant, G. E. (1988). Pessimistic explanatory style is a risk factor for physical illness: A thirty-five year longitudinal study. Journal of Personality and Social Psychology, 55, 23–27.

RESEARCH FOR HUMAN SERVICES pg. 107

7.3 Quasi-Experimental Research

LEARNING OBJECTIVES  Explain what quasi-experimental research is and distinguish it clearly from both experimental and

correlational research.

 Describe three different types of quasi-experimental research designs (nonequivalent groups, pretest-posttest, and interrupted time series) and identify examples of each one.

The prefix quasi means “resembling.” Thus, quasi-experimental research is research that resembles experimental research but is not true experimental research. Although the independent variable is manipulated, participants are not randomly assigned to conditions or orders of conditions (Cook & Campbell, 1979). [1] Because the independent variable is manipulated before the dependent variable is measured, quasi-experimental research eliminates the directionality problem. But because participants are not randomly assigned—making it likely that there are other differences between conditions—quasi- experimental research does not eliminate the problem of confounding variables. In terms of internal validity, therefore, quasi-experiments are generally somewhere between correlational studies and true experiments.

Quasi-experiments are most likely to be conducted in field settings in which random assignment is difficult or impossible. They are often conducted to evaluate the effectiveness of a treatment—perhaps a type of psychotherapy or an educational intervention. There are many different kinds of quasi-experiments, but we will discuss just a few of the most common ones here.

Nonequivalent Groups Design Recall that when participants in a between-subjects experiment are randomly assigned to conditions, the resulting groups are likely to be quite similar. In fact, researchers consider them to be equivalent. When participants are not randomly assigned to conditions, however, the resulting groups are likely to be dissimilar in some ways. For this reason, researchers consider them to be nonequivalent. A nonequivalent groups design, then, is a between-subjects design in which participants have not been randomly assigned to conditions.

Imagine, for example, a researcher who wants to evaluate a new method of teaching fractions to third graders. One way would be to conduct a study with a treatment group consisting of one class of third-grade students and a control group consisting of another class of third-grade students. This would be a nonequivalent groups design because the students are not randomly assigned to classes by the researcher, which means there could be important differences between them. For example, the parents of higher achieving or more motivated students might have been more likely to request that their children be assigned to Ms. Williams’s class. Or the principal might have assigned the “troublemakers” to Mr. Jones’s class because he is a stronger disciplinarian. Of course, the teachers’ styles, and even the classroom environments, might be very different and might cause different levels of achievement or motivation among the students. If at the end of the study there was a difference in the two classes’ knowledge of fractions, it might have been caused by the difference between the teaching methods—but it might have been caused by any of these confounding variables.

Of course, researchers using a nonequivalent groups design can take steps to ensure that their groups are as similar as possible. In the present example, the researcher could try to select two classes at the same school, where the students in the two classes have similar scores on a standardized math test and the teachers are the same sex, are close in age, and have similar teaching styles. Taking such steps would increase the internal validity of the study because it would eliminate some of the most important confounding variables. But without true random assignment of the students to conditions, there remains the possibility of other important confounding variables that the researcher was not able to control.

RESEARCH FOR HUMAN SERVICES pg. 108

Pretest-Posttest Design In a pretest-posttest design, the dependent variable is measured once before the treatment is implemented and once after it is implemented. Imagine, for example, a researcher who is interested in the effectiveness of an antidrug education program on elementary school students’ attitudes toward illegal drugs. The researcher could measure the attitudes of students at a particular elementary school during one week, implement the antidrug program during the next week, and finally, measure their attitudes again the following week. The pretest-posttest design is much like a within-subjects experiment in which each participant is tested first under the control condition and then under the treatment condition. It is unlike a within-subjects experiment, however, in that the order of conditions is not counterbalanced because it typically is not possible for a participant to be tested in the treatment condition first and then in an “untreated” control condition.

If the average posttest score is better than the average pretest score, then it makes sense to conclude that the treatment might be responsible for the improvement. Unfortunately, one often cannot conclude this with a high degree of certainty because there may be other explanations for why the posttest scores are better. One category of alternative explanations goes under the name of history. Other things might have happened between the pretest and the posttest. Perhaps an antidrug program aired on television and many of the students watched it, or perhaps a celebrity died of a drug overdose and many of the students heard about it. Another category of alternative explanations goes under the name of maturation. Participants might have changed between the pretest and the posttest in ways that they were going to anyway because they are growing and learning. If it were a yearlong program, participants might become less impulsive or better reasoners and this might be responsible for the change.

Regression to the Mean Another alternative explanation for a change in the dependent variable in a pretest-posttest design is regression to the mean. This refers to the statistical fact that an individual who scores extremely on a variable on one occasion will tend to score less extremely on the next occasion. For example, a bowler with a long-term average of 150 who suddenly bowls a 220 will almost certainly score lower in the next game. Her score will “regress” toward her mean score of 150. Regression to the mean can be a problem when participants are selected for further study because of their extreme scores. Imagine, for example, that only students who scored especially low on a test of fractions are given a special training program and then retested. Regression to the mean all but guarantees that their scores will be higher even if the training program has no effect.

Spontaneous Remission A closely related concept—and an extremely important one in psychological research— is spontaneous remission. This is the tendency for many medical and psychological problems to improve over time without any form of treatment. The common cold is a good example. If one were to measure symptom severity in 100 common cold sufferers today, give them a bowl of chicken soup every day, and then measure their symptom severity again in a week, they would probably be much improved. This does not mean that the chicken soup was responsible for the improvement, however, because they would have been much improved without any treatment at all. The same is true of many psychological problems. A group of severely depressed people today is likely to be less depressed on average in 6 months. In reviewing the results of several studies of treatments for depression, researchers Michael Posternak and Ivan Miller found that participants in waitlist control conditions improved an average of 10 to 15% before they received any treatment at all (Posternak & Miller, 2001). [2] Thus one must generally be very cautious about inferring causality from pretest-posttest designs.

Does Psychotherapy Work? Early studies on the effectiveness of psychotherapy tended to use pretest-posttest designs. In a classic 1952 article, researcher Hans Eysenck summarized the results of 24 such studies showing that about two thirds of patients improved between the pretest and the posttest (Eysenck, 1952). [3] But Eysenck also compared these results with archival data from state hospital and insurance company records showing that similar patients recovered at about the same rate without receiving psychotherapy. This suggested to

RESEARCH FOR HUMAN SERVICES pg. 109

Eysenck that the improvement that patients showed in the pretest-posttest studies might be no more than spontaneous remission. Note that Eysenck did not conclude that psychotherapy was ineffective. He merely concluded that there was no evidence that it was, and he wrote of “the necessity of properly planned and executed experimental studies into this important field” (p. 323). You can read the entire article here:

http://psychclassics.yorku.ca/Eysenck/psychotherapy.htm

Fortunately, many other researchers took up Eysenck’s challenge, and by 1980 hundreds of experiments had been conducted in which participants were randomly assigned to treatment and control conditions, and the results were summarized in a classic book by Mary Lee Smith, Gene Glass, and Thomas Miller (Smith, Glass, & Miller, 1980). [4] They found that overall psychotherapy was quite effective, with about 80% of treatment participants improving more than the average control participant. Subsequent research has focused more on the conditions under which different types of psychotherapy are more or less effective.

Interrupted Time Series Design A variant of the pretest-posttest design is the interrupted time-series design. A time series is a set of measurements taken at intervals over a period of time. For example, a manufacturing company might measure its workers’ productivity each week for a year. In an interrupted time series-design, a time series like this is “interrupted” by a treatment. In one classic example, the treatment was the reduction of the work shifts in a factory from 10 hours to 8 hours (Cook & Campbell, 1979). [5] Because productivity increased rather quickly after the shortening of the work shifts, and because it remained elevated for many months afterward, the researcher concluded that the shortening of the shifts caused the increase in productivity. Notice that the interrupted time-series design is like a pretest-posttest design in that it includes measurements of the dependent variable both before and after the treatment. It is unlike the pretest- posttest design, however, in that it includes multiple pretest and posttest measurements.

Figure 7.5 "A Hypothetical Interrupted Time-Series Design" shows data from a hypothetical interrupted time-series study. The dependent variable is the number of student absences per week in a research methods course. The treatment is that the instructor begins publicly taking attendance each day so that students know that the instructor is aware of who is present and who is absent. The top panel of Figure 7.5 "A Hypothetical Interrupted Time-Series Design" shows how the data might look if this treatment worked. There is a consistently high number of absences before the treatment, and there is an immediate and sustained drop in absences after the treatment. The bottom panel of Figure 7.5 "A Hypothetical Interrupted Time-Series Design" shows how the data might look if this treatment did not work. On average, the number of absences after the treatment is about the same as the number before. This figure also illustrates an advantage of the interrupted time-series design over a simpler pretest-posttest design. If there had been only one measurement of absences before the treatment at Week 7 and one afterward at Week 8, then it would have looked as though the treatment were responsible for the reduction. The multiple measurements both before and after the treatment suggest that the reduction between Weeks 7 and 8 is nothing more than normal week-to-week variation.

RESEARCH FOR HUMAN SERVICES pg. 110

Figure 7.5 A Hypothetical Interrupted Time-Series Design

The top panel shows data that suggest that the treatment caused a reduction in absences. The bottom panel shows data that suggest that it did not.

Combination Designs A type of quasi-experimental design that is generally better than either the nonequivalent groups design or the pretest-posttest design is one that combines elements of both. There is a treatment group that is given a pretest, receives a treatment, and then is given a posttest. But at the same time there is a control group

RESEARCH FOR HUMAN SERVICES pg. 111

that is given a pretest, does not receive the treatment, and then is given a posttest. The question, then, is not simply whether participants who receive the treatment improve but whether they improve more than participants who do not receive the treatment.

Imagine, for example, that students in one school are given a pretest on their attitudes toward drugs, then are exposed to an antidrug program, and finally are given a posttest. Students in a similar school are given the pretest, not exposed to an antidrug program, and finally are given a posttest. Again, if students in the treatment condition become more negative toward drugs, this could be an effect of the treatment, but it could also be a matter of history or maturation. If it really is an effect of the treatment, then students in the treatment condition should become more negative than students in the control condition. But if it is a matter of history (e.g., news of a celebrity drug overdose) or maturation (e.g., improved reasoning), then students in the two conditions would be likely to show similar amounts of change. This type of design does not completely eliminate the possibility of confounding variables, however. Something could occur at one of the schools but not the other (e.g., a student drug overdose), so students at the first school would be affected by it while students at the other school would not.

Finally, if participants in this kind of design are randomly assigned to conditions, it becomes a true experiment rather than a quasi-experiment. In fact, it is the kind of experiment that Eysenck called for—and that has now been conducted many times—to demonstrate the effectiveness of psychotherapy.

Quasi-Experimental Research Quasi-experimental research involves the manipulation of an independent variable without the random assignment of participants to conditions or orders of conditions. Among the important types are nonequivalent groups designs, pretest-posttest, and interrupted time-series designs.

Quasi-experimental research eliminates the directionality problem because it involves the manipulation of the independent variable. It does not eliminate the problem of confounding variables, however, because it does not involve random assignment to conditions. For these reasons, quasi-experimental research is generally higher in internal validity than correlational studies but lower than true experiments.

EE R C IS E S

Practice:

Imagine that two college professors decide to test the effect of giving daily quizzes on student performance in a statistics course. They decide that Professor A will give quizzes but Professor B will not. They will then compare the performance of students in their two sections on a common final exam. List five other variables that might differ between the two sections that could affect the results.

Discussion:

Imagine that a group of obese children is recruited for a study in which their weight is measured, then they participate for 3 months in a program that encourages them to be more active, and finally their weight is measured again. Explain how each of the following might affect the results:

 regression to the mean

 spontaneous remission

 history

 maturation

[1] Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design & analysis issues in field settings. Boston, MA: Houghton Mifflin.

RESEARCH FOR HUMAN SERVICES pg. 112

[2] Posternak, M. A., & Miller, I. (2001). Untreated short-term course of major depression: A meta-analysis of studies using outcomes from studies using wait-list control groups. Journal of Affective Disorders, 66, 139–146.

[3] Eysenck, H. J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting Psychology, 16, 319–324.

[4] Smith, M. L., Glass, G. V., & Miller, T. I. (1980). The benefits of psychotherapy. Baltimore, MD: Johns Hopkins University Press.

[5] Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design & analysis issues in field settings. Boston, MA: Houghton Mifflin.

7.4 Qualitative Research

Learning Objectives:

 List several ways in which qualitative research differs from quantitative research.

 Describe the strengths and weaknesses of qualitative research compared with quantitative research.

 Give examples of qualitative research from the social sciences or from Human Services.

What Is Qualitative Research? So far, we have focused on quantitative research. It would be useful at this point to compare the two2:

Characteristics of Quantitative Research Methods  Surveys, structured interviews & observations, and reviews of records or documents for numeric

information

 Primarily deductive process used to test pre-specified concepts, constructs, and hypotheses that make up a theory

 More objective: provides observed effects (interpreted by researchers) of a program on a problem or condition

 Number-based

 Less in-depth but more breadth of information across a large number of cases

 Fixed response options

 Statistical tests are used for analysis

 Can be valid and reliable: largely depends on the measurement device or instrument used

 Time expenditure heavier on the planning phase and lighter on the analysis phase

 More generalizable The above list describes the general characteristics of quantitative research methods. We will now look at the characteristics of qualitative methods.

2 These two outstanding lists were obtained from: The U. S. Department of Energy, Office of

Science, on 12/20/2017 from

https://www.orau.gov/cdcynergy/soc2web/Content/phase05/phase05_step03_deeper_qualitative_and_qua

ntitative.htm

RESEARCH FOR HUMAN SERVICES pg. 113

Characteristics of Qualitative Methods

 Methods include focus groups, in-depth interviews, and reviews of documents for types of themes

 Primarily inductive process used to formulate theory or hypotheses

 More subjective: describes a problem or condition from the point of view of those experiencing it

 Text-based

 More in-depth information on a few cases

 Unstructured or semi-structured response options

 No statistical tests

 Can be valid and reliable: largely depends on skill and rigor of the researcher

 Time expenditure lighter on the planning end and heavier during the analysis phase

 Less generalizable

Quantitative researchers typically start with a focused research question or hypothesis, and generally collect a sample from a population, describe the resulting data using statistical techniques, and draw general conclusions about the population. Although this is by far the most common approach to conducting empirical research, there is an important alternative called qualitative research.

Origins of Qualitative Research Qualitative research originated in the disciplines of anthropology and sociology but is now used to study many psychological topics as well. Qualitative researchers generally begin with a less focused research question, collect large amounts of relatively “unfiltered” data from a relatively small number of individuals, and describe their data using non-statistical techniques. They are usually less concerned with drawing general conclusions about human behavior than with understanding in detail the experience of their research participants.

Consider, for example, a study by researcher Per Lindqvist and his colleagues, who wanted to learn how the families of teenage suicide victims cope with their loss (Lindqvist, Johansson, & Karlsson, 2008). [1] They did not have a specific research question or hypothesis, such as, what percentage of family members join suicide support groups? Instead, they wanted to understand the variety of reactions that families had, with a focus on what it is like from their perspectives. To do this, they interviewed the families of 10 teenage suicide victims in their homes in rural Sweden.

The interviews were relatively unstructured, beginning with a general request for the families to talk about the victim and ending with an invitation to talk about anything else that they wanted to tell the interviewer. One of the most important themes that emerged from these interviews was that even as life returned to “normal,” the families continued to struggle with the question of why their loved one committed suicide. This struggle appeared to be especially difficult for families in which the suicide was most unexpected.

The Purpose of Qualitative Research The strength of quantitative research is its ability to provide precise answers to specific research questions and to draw general conclusions about human behavior. This is how we know that people have a strong tendency to obey authority figures, for example, or that female college students are not substantially more talkative than male college students. But while quantitative research is good at providing precise answers to specific research questions, it is not nearly as good at generating novel and interesting research

RESEARCH FOR HUMAN SERVICES pg. 114

questions. Likewise, while quantitative research is good at drawing general conclusions about human behavior, it is not nearly as good at providing detailed descriptions of the behavior of particular groups in particular situations. And it is not very good at all at communicating what it is actually like to be a member of a particular group in a particular situation.

Weaknesses of Quantitative Research are Strengths of Qualitative Research But the relative weaknesses of quantitative research are the relative strengths of qualitative research. Qualitative research can help researchers to generate new and interesting research questions and hypotheses. The research of Lindqvist and colleagues, for example, suggests that there may be a general relationship between how unexpected a suicide is and how consumed the family is with trying to understand why the teen committed suicide. This relationship can now be explored using quantitative research. But it is unclear whether this question would have arisen at all without the researchers sitting down with the families and listening to what they themselves wanted to say about their experience.

Thick Description Qualitative research can also provide rich and detailed descriptions of human behavior in the real-world contexts in which it occurs. Among qualitative researchers, this is often referred to as “thick description” (Geertz, 1973). [2] Similarly, qualitative research can convey a sense of what it is actually like to be a member of a particular group or in a particular situation—what qualitative researchers often refer to as the “lived experience” of the research participants. Lindqvist and colleagues, for example, describe how all the families spontaneously offered to show the interviewer the victim’s bedroom or the place where the suicide occurred—revealing the importance of these physical locations to the families. It seems unlikely that a quantitative study would have discovered this.

Data Collection and Analysis in Qualitative Research

Interviews As with correlational research, data collection approaches in qualitative research are quite varied and can involve naturalistic observation, archival data, artwork, and many other things. But one of the most common approaches, especially for research in the social sciences, is to conduct interviews. Interviews in qualitative research tend to be unstructured—consisting of a small number of general questions or prompts that allow participants to talk about what is of interest to them. The researcher can follow up by asking more detailed questions about the topics that do come up. Such interviews can be lengthy and detailed, but they are usually conducted with a relatively small sample.

Focus Groups This was essentially the approach used by Lindqvist and colleagues in their research on the families of suicide survivors. Small groups of people who participate together in interviews focused on a particular topic or issue are often referred to as focus groups. The interaction among participants in a focus group can sometimes bring out more information than can be learned in a one-on-one interview. The use of focus groups has become a standard technique in business and industry among those who want to understand consumer tastes and preferences. The content of all focus group interviews is usually recorded and transcribed to facilitate later analyses.

Participant Observation Another approach to data collection in qualitative research is participant observation. In participant observation, researchers become active participants in the group or situation they are studying. The data they collect can include interviews (usually unstructured), their own notes based on their observations and interactions, documents, photographs, and other artifacts. The basic rationale for participant observation is that there may be important information that is only accessible to, or can be interpreted only by, someone who is an active participant in the group or situation. An example of participant observation comes from a study by sociologist Amy Wilkins (published in Social Psychology Quarterly) on a college-based religious organization that emphasized how happy its members were (Wilkins, 2008). [3] Wilkins spent 12 months attending and participating in the group’s meetings and social events, and she interviewed several group members. In her study, Wilkins identified several ways in which

RESEARCH FOR HUMAN SERVICES pg. 115

the group “enforced” happiness—for example, by continually talking about happiness, discouraging the expression of negative emotions, and using happiness as a way to distinguish themselves from other groups.

Data Analysis in Quantitative Research Although quantitative and qualitative research generally differ along several important dimensions (e.g., the specificity of the research question, the type of data collected), it is the method of data analysis that distinguishes them more clearly than anything else. To illustrate this idea, imagine a team of researchers that conducts a series of unstructured interviews with recovering alcoholics to learn about the role of their religious faith in their recovery. Although this sounds like qualitative research, imagine further that once they collect the data, they code the data in terms of how often each participant mentions God (or a “higher power”), and they then use descriptive and inferential statistics to find out whether those who mention God more often are more successful in abstaining from alcohol. Now it sounds like quantitative research.

The quantitative-qualitative distinction depends more on what researchers do with the data they have collected than with why or how they collected the data.

Grounded Theory But what does qualitative data analysis look like? Just as there are many ways to collect data in qualitative research, there are many ways to analyze data. Here we focus on one general approach called grounded theory (Glaser & Strauss, 1967). [4]

This approach was developed within the field of sociology in the 1960s and has gradually gained popularity. Remember that in quantitative research, it is typical for the researcher to start with a theory, derive a hypothesis from that theory, and then collect data to test that specific hypothesis. Grounded theory takes another approach.

The Theory Evolves from the Data after the Fact

Using grounded theory, the researchers do not start with a theory. The theory evolves from their analysis of the data. In qualitative research using grounded theory, researchers start with the data and develop a theory or an interpretation that is “grounded in” those data. They do this in stages. First, they identify ideas that are repeated throughout the data. Then they organize these ideas into a smaller number of broader themes. Finally, they write a theoretical narrative—an interpretation—of the data in terms of the themes that they have identified. This theoretical narrative focuses on the subjective experience of the participants and is usually supported by many direct quotations from the participants themselves.

Grounded Research in Action As an example, consider a study by researchers Laura Abrams and Laura Curran, who used the grounded theory approach to study the experience of postpartum depression symptoms among low-income mothers (Abrams & Curran, 2009). [5] Their data were the result of unstructured interviews with 19 participants (remember that one hallmark of qualitative research is the use of a small number of participants, which will generally generate a large amount of data that will have to be analyzed). Often times, the researchers will not use a structured interview because they want to go into the research with a minimum, or no, preconceptions of what they will find. The results were five broad themes the researchers identified and the more specific repeating ideas that made up each of those themes. In their research report, they provide numerous quotations from their participants, such as this direct quote from one participant:

Well, just recently my apartment was broken into and the fact that his Medicaid for some reason was cancelled so a lot of things was happening within the last two weeks all at one time. So that in itself I don’t want to say almost drove me mad but it put me in a funk. Like I really was depressed. (p. 357)

Below are the five themes they found in the interviews along with a representative quote from one of the

participants.

RESEARCH FOR HUMAN SERVICES pg. 116

Five Themes of Postpartum Depression  Ambivalence: “I wasn’t prepared for this baby,” “I didn’t want to have any more children.”

 Caregiving overload: “Please stop crying,” “I need a break,” “I can’t do this anymore.”

 Juggling: “No time to breathe,” “Everyone depends on me,” “Navigating the maze.”

 Mothering alone: “I really don’t have any help,” “My baby has no father.”

 Real-life worry: “I don’t have any money,” “Will my baby be OK?” “It’s not safe here.”

Their theoretical narrative focused on the participants’ experience of their symptoms not as an abstract “affective disorder” but as closely tied to the daily struggle of raising children alone under often difficult circumstances.

The Quantitative-Qualitative “Debate” Given their differences, it may come as no surprise that quantitative and qualitative research in psychology and related fields do not coexist in complete harmony. Some quantitative researchers criticize qualitative methods on the grounds that they lack objectivity, are difficult to evaluate in terms of reliability and validity, and do not allow generalization to people or situations other than those actually studied. At the same time, some qualitative researchers criticize quantitative methods on the grounds that they overlook the richness of human behavior and experience and instead answer simple questions about easily quantifiable variables.

In general, however, qualitative researchers are well aware of the issues of objectivity, reliability, validity, and generalizability. In fact, they have developed a number of frameworks for addressing these issues (which are beyond the scope of our discussion). And in general, quantitative researchers are well aware of the issue of oversimplification. They do not believe that all human behavior and experience can be adequately described in terms of a small number of variables and the statistical relationships among them. Instead, they use simplification as a strategy for uncovering general principles of human behavior.

Mixed Methods and Triangulation

Mixed Methods Research

Many researchers from both the quantitative and qualitative camps now agree that the two approaches can and should be combined into what has come to be called mixed-methods research (Todd, Nerlich, McKeown, & Clarke, 2004).[6] (In fact, the studies by Lindqvist and colleagues and by Abrams and Curran both combined quantitative and qualitative approaches.) One approach to combining quantitative and qualitative research is to use qualitative research for hypothesis generation and quantitative research for hypothesis testing. Again, while a qualitative study might suggest that families who experience an unexpected suicide have more difficulty resolving the question of why, a well-designed quantitative study could test a hypothesis by measuring these specific variables for a large sample.

Triangulation

A second approach to combining quantitative and qualitative research is referred to as triangulation. The idea is to use both quantitative and qualitative methods simultaneously to study the same general questions and to compare the results. If the results of the quantitative and qualitative methods converge on the same general conclusion, they reinforce and enrich each other. If the results diverge, then they suggest an interesting new question: Why do the results diverge and how can they be reconciled?

Summary:

 Qualitative research is an important alternative to quantitative research. It generally involves asking broader research questions, collecting more detailed data (e.g., interviews), and using non- statistical analyses.

 Many researchers conceptualize quantitative and qualitative research as complementary and advocate combining them. For example, qualitative research can be used to generate hypotheses and quantitative research to test them.

RESEARCH FOR HUMAN SERVICES pg. 117

Discussion What are some ways in which a qualitative study of girls who play youth baseball would be likely to differ from a quantitative study on the same topic?

[1] Lindqvist, P., Johansson, L., & Karlsson, U. (2008). In the aftermath of teenage suicide: A qualitative study of the psychosocial consequences for the surviving family members. BMC Psychiatry, 8, 26. Retrieved from http://www.biomedcentral.com/1471-244X/8/26

[2] Geertz, C. (1973). The interpretation of cultures. New York, NY: Basic Books.

[3] Wilkins, A. (2008). “Happier than Non-Christians”: Collective emotions and symbolic boundaries among evangelical Christians. Social Psychology Quarterly, 71, 281–301.

[4] Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago, IL: Aldine.

[5] Abrams, L. S., & Curran, L. (2009). “And you’re telling me not to stress?” A grounded theory study of postpartum depression symptoms among low-income mothers. Psychology of Women Quarterly, 33, 351– 362.

[6] Todd, Z., Nerlich, B., McKeown, S., & Clarke, D. D. (2004) Mixing methods in psychology: The integration of qualitative and quantitative methods in theory and practice. London, UK: Psychology Press.

RESEARCH FOR HUMAN SERVICES pg. 118

Chapter 8: Survey Research The art and science of asking questions is the source of all knowledge.

-Thomas Berger

The art and science of asking questions is the source of all knowledge.

-Thomas Berger

Why Survey Research? Survey research is the art and science of acquiring knowledge by asking questions. Survey research holds a special place in the hierarchy of statistical, and information gathering, techniques. It is the Swiss Army Knife of research, and very well developed.

In 2008, the voters of the United States elected our first African American president, Barack Obama. It may not surprise you to learn that when President Obama was coming of age in the 1970s, one- quarter of Americans reported that they would not vote for a qualified African American presidential nominee. Three decades later, when President Obama ran for the presidency, fewer than 8% of Americans still held that position, and President Obama won the election. [1]

General Social Survey We know about these trends in voter opinion because the General Social Survey (http://www.norc.uchicago.edu/GSS+Website), a nationally representative survey of American adults, included questions about race and voting over the years described here. Without survey research, we may not know how Americans’ perspectives on race and the presidency shifted over these years.

[1] Smith, T. W. (2009). Trends in willingness to vote for a black and woman for president, 1972–2008. GSS Social Change Report No. 55. Chicago, IL: National Opinion Research Center.

8.1 Survey Research: What Is It and When Should It Be Used?

LEARNING OBJECTIVES

 Define survey research.

 Identify when it is appropriate to employ survey research as a data-collection strategy.

Most of you have probably taken a survey at one time or another, so you probably have a pretty good idea of what a survey is. Sometimes students in my research methods classes feel that understanding what a survey is and how to write one is so obvious, there’s no need to dedicate any class time to learning about it. This feeling is understandable—surveys are very much a part of our everyday lives— we’ve probably all taken one, we hear about their results in the news, and perhaps we’ve even administered one ourselves. What students quickly learn is that there is more to constructing a good survey than meets the eye.

Survey design takes a great deal of thoughtful planning and often a great many rounds of revision. But it is worth the effort. As we’ll learn in this chapter, there are many benefits to choosing survey research as one’s method of data collection. We’ll take a look at what a survey is exactly, what some of the benefits and drawbacks of this method are, how to construct a survey, and what to do with survey data once one has it in hand.

RESEARCH FOR HUMAN SERVICES pg. 119

Surveys are a Quantitative Method Survey research is a quantitative method whereby a researcher poses some set of predetermined questions to an entire group, or sample, of individuals. Survey research is an especially useful approach when a researcher aims to describe or explain features of a very large group or groups. This method may also be used as a way of quickly gaining some general details about one’s population of interest to help prepare for a more focused, in-depth study using time-intensive methods such as in- depth interviews or field research. In this case, a survey may help a researcher identify specific individuals or locations from which to collect additional data.

As is true of all methods of data collection, survey research is better suited to answering some kinds of research question more than others. In addition, as you’ll recall from Chapter 6 "Defining and Measuring Concepts", operationalization works differently with different research methods. If your interest is in political activism, for example, you likely operationalize that concept differently in a survey than you would for a field research study of the same topic.

Summary

 Survey research is often used by researchers who wish to explain trends or features of large groups. It may also be used to assist those planning some more focused, in-depth study.

Exercise

1. Recall some of the possible research questions you came up with while reading previous chapters of this text. How might you frame those questions so that they could be answered using survey research?

8.2 Pros and Cons of Survey Research LEARNING OBJECTIVES

 Identify and explain the strengths of survey research.

 Identify and explain the weaknesses of survey research.

Survey research, as with all methods of data collection, comes with both strengths and weaknesses. We’ll examine both in this section.

Strengths of Survey Method Researchers employing survey methods to collect data enjoy a number of benefits. First, surveys are an excellent way to gather lots of information from many people. In my own study of older people’s experiences in the workplace, I was able to mail a written questionnaire to around 500 people who lived throughout the state of Maine at a cost of just over $1,000. This cost included printing copies of my seven-page survey, printing a cover letter, addressing and stuffing envelopes, mailing the survey, and buying return postage for the survey. I realize that $1,000 is nothing to sneeze at. But just imagine what it might have cost to visit each of those people individually to interview them in person. Consider the cost of gas to drive around the state, other travel costs, such as meals and lodging while on the road, and the cost of time to drive to and talk with each person individually. We could double, triple, or even quadruple our costs pretty quickly by opting for an in-person method of data collection over a mailed survey. Thus, surveys are relatively cost effective.

RESEARCH FOR HUMAN SERVICES pg. 120

Related to the benefit of cost effectiveness is a survey’s potential for generalizability. Because surveys allow researchers to collect data from very large samples for a relatively low cost, survey methods lend themselves to probability sampling techniques, which we discussed in Chapter 7 "Sampling". Of all the data-collection methods described in this text, survey research is probably the best method to use when one hopes to gain a representative picture of the attitudes and characteristics of a large group.

Survey research also tends to be a reliable method of inquiry. This is because surveys are standardized in that the same questions, phrased in exactly the same way, are posed to participants. Other methods, such as qualitative interviewing, which we’ll learn about in Chapter 9 "Interviews: Qualitative and Quantitative Approaches", do not offer the same consistency that a quantitative survey offers. This is not to say that all surveys are always reliable. A poorly phrased question can cause respondents to interpret its meaning differently, which can reduce that question’s reliability. Assuming well-constructed question and questionnaire design, one strength of survey methodology is its potential to produce reliable results.

Versatility The versatility of survey research is also an asset. Surveys are used by all kinds of people in all kinds of professions. I repeat, surveys are used by all kinds of people in all kinds of professions. Is there a light bulb switching on in your head? I hope so. The versatility offered by survey research means that understanding how to construct and administer surveys is a useful skill to have for all kinds of jobs. Lawyers might use surveys in their efforts to select juries, social service and other organizations (e.g., churches, clubs, fundraising groups, activist groups) use them to evaluate the effectiveness of their efforts, businesses use them to learn how to market their products, governments use them to understand community opinions and needs, and politicians and media outlets use surveys to understand their constituencies.

In sum, the following are benefits of survey research:

1. Cost-effective 2. Generalizable 3. Reliable 4. Versatile

Should we add survey fatigue? Students frequently report that they feel they are over surveyed. With the advent and proliferation of computer mediated surveys, and the ability to gather and analyze massive amounts of data from our electronic footprint, survey use has grown proportionally.

Weaknesses of Survey Method As with all methods of data collection, survey research also comes with a few drawbacks. First, while one might argue that surveys are flexible in the sense that we can ask any number of questions on any number of topics in them, the fact that the survey researcher is generally stuck with a single instrument for collecting data (the questionnaire), surveys are in many ways rather inflexible. Let’s say you mail a survey out to 1,000 people and then discover, as responses start coming in, that your phrasing on a particular question seems to be confusing a number of respondents. At this stage, it’s too late for a do-over or to change the question for the respondents who haven’t yet returned their surveys. When conducting in-depth interviews, on the other hand, a researcher can provide respondents further explanation if they’re confused by a question and can tweak their questions as they learn more about how respondents seem to understand them.

RESEARCH FOR HUMAN SERVICES pg. 121

Validity & Surveys Validity can also be a problem with surveys. Survey questions are standardized; thus, it can be difficult to ask anything other than very general questions that a broad range of people will understand. Because of this, survey results may not be as valid as results obtained using methods of data collection that allow a researcher to more comprehensively examine whatever topic is being studied. Let’s say, for example, that you want to learn something about voters’ willingness to elect an African American president, as in our opening example in this chapter. General Social Survey respondents were asked, “If your party nominated an African American for president, would you vote for him if he were qualified for the job?” Respondents were then asked to respond either yes or no to the question. But what if someone’s opinion was more complex than could be answered with a simple yes or no? What if, for example, a person was willing to vote for an African American woman but not an African American man?I am not at all suggesting that such a perspective makes any sense, but it is conceivable that an individual might hold such a perspective.

In sum, potential drawbacks to survey research include the following:

1. Inflexibility 2. Validity

Summary

 Strengths of survey research include its cost effectiveness, generalizability, reliability, and versatility.

 Weaknesses of survey research include inflexibility and issues with validity.

Exercises

1. What are some ways that survey researchers might overcome the weaknesses of this method? 2. Find an article reporting results from survey research (remember how to use Sociological

Abstracts?). How do the authors describe the strengths and weaknesses of their study? Are any of the strengths or weaknesses described here mentioned in the article?

8.3 Types of Surveys LEARNING OBJECTIVES

 Define cross-sectional surveys, provide an example of a cross-sectional survey, and outline some of the drawbacks of cross-sectional research.

 Describe the various types of longitudinal surveys.

 Define retrospective surveys, and identify their strengths and weaknesses.

 Discuss some of the benefits and drawbacks of the various methods of delivering self-administered questionnaires.

There is much variety when it comes to surveys. This variety comes both in terms of time—when or with what frequency a survey is administered—and in terms of administration—how a survey is delivered to respondents. In this section, we’ll take a look at what types of surveys exist when it comes to both time and administration.

Cross-Sectional and Longitudinal Surveys In terms of time, there are two main types of surveys: cross-sectional and longitudinal.

RESEARCH FOR HUMAN SERVICES pg. 122

Cross-Sectional Research Cross-sectional surveys are those that are administered at just one point in time. These surveys offer researchers a sort of snapshot in time and give us an idea about how things are for our respondents at the particular point in time that the survey is administered. My own study of older workers mentioned previously is an example of a cross-sectional survey. I administered the survey at just one time.

Another example of a cross-sectional survey comes from Aniko Kezdy and colleagues’ study of the association between religious attitudes, religious beliefs, and mental health among students in Hungary. [1] These researchers administered a single, one-time-only, cross-sectional survey to a convenience sample of 403 high school and college students. The survey focused on how religious attitudes impact various aspects of one’s life and health. The researchers found from analysis of their cross-sectional data that anxiety and depression were highest among those who had both strong religious beliefs and also some doubts about religion.

Yet another recent example of cross-sectional survey research can be seen in Bateman and colleagues’ study of how the perceived publicness of social networking sites influences users’ self- disclosures. [2] These researchers administered an online survey to undergraduate and graduate business students. They found that even though revealing information about oneself is viewed as key to realizing many of the benefits of social networking sites, respondents were less willing to disclose information about themselves as their perceptions of a social networking site’s publicness rose. That is, there was a negative relationship between perceived publicness of a social networking site and plans to self-disclose on the site.

One Problem with Cross-Sectional Surveys One problem with cross-sectional surveys is that the events, opinions, behaviors, and other phenomena that such surveys are designed to assess don’t generally remain stagnant. Thus, generalizing from a cross-sectional survey about the way things are can be tricky; perhaps you can say something about the way things were in the moment that you administered your survey, but it is difficult to know whether things remained that way for long after you administered your survey. Think, for example, about how Americans might have responded if administered a survey asking for their opinions on terrorism on September 10, 2001. Now imagine how responses to the same set of questions might differ were they administered on September 12, 2001. The point is not that cross- sectional surveys are useless; they have many important uses. But researchers must remember what they have captured by administering a cross-sectional survey; that is, as previously noted, a snapshot of life as it was at the time that the survey was administered.

One way to overcome this sometimes problematic aspect of cross-sectional surveys is to administer a longitudinal survey. Longitudinal surveys are those that enable a researcher to make observations over some extended period of time. There are several types of longitudinal surveys, including trend, panel, and cohort surveys. We’ll discuss all three types here, along with another type of survey called retrospective. Retrospective surveys fall somewhere in between cross-sectional and longitudinal surveys.

Longitudinal Surveys

Trend Survey The first type of longitudinal survey is called a trend survey. The main focus of a trend survey is, perhaps not surprisingly, trends. Researchers conducting trend surveys are interested in how people’s inclinations change over time. The Gallup opinion polls are an excellent example of trend surveys. You can read more about Gallup on their website: http://www.gallup.com/Home.aspx. To learn about how public opinion changes over time, Gallup administers the same questions to people at different points

RESEARCH FOR HUMAN SERVICES pg. 123

in time. For example, for several years Gallup has polled Americans to find out what they think about gas prices (something many of us happen to have opinions about). One thing we’ve learned from Gallup’s polling is that price increases in gasoline caused financial hardship for 67% of respondents in 2011, up from 40% in the year 2000. Gallup’s findings about trends in opinions about gas prices have also taught us that whereas just 34% of people in early 2000 thought the current rise in gas prices was permanent, 54% of people in 2011 believed the rise to be permanent. Thus, through Gallup’s use of trend survey methodology, we’ve learned that Americans seem to feel generally less optimistic about the price of gas these days than they did 10 or so years ago. You can read about these and other findings on Gallup’s gasoline questions at http://www.gallup.com/poll/147632/Gas-Prices.aspx#1. It should be noted that in a trend survey, the same people are probably not answering the researcher’s questions each year. Because the interest here is in trends, not specific people, as long as the researcher’s sample is representative of whatever population he or she wishes to describe trends for, it isn’t important that the same people participate each time.

Panel Surveys Next are panel surveys. Unlike in a trend survey, in a panel survey the same people do participate in the survey each time it is administered. As you might imagine, panel studies can be difficult and costly. Imagine trying to administer a survey to the same 100 people every year for, say, 5 years in a row. Keeping track of where people live, when they move, and when they die takes resources that researchers often don’t have. When they do, however, the results can be quite powerful. The Youth Development Study (YDS), administered from the University of Minnesota, offers an excellent example of a panel study. You can read more about the Youth Development Study at its website: http://www.soc.umn.edu/research/yds. Since 1988, YDS researchers have administered an annual survey to the same 1,000 people. Study participants were in ninth grade when the study began, and they are now in their thirties. Several hundred papers, articles, and books have been written using data from the YDS.

One of the major lessons learned from this panel study is that work has a largely positive impact on young people. [3] Contrary to popular beliefs about the impact of work on adolescents’ performance in school and transition to adulthood, work in fact increases confidence, enhances academic success, and prepares students for success in their future careers. Without this panel study, we may not be aware of the positive impact that working can have on young people.

Cohort Survey Another type of longitudinal survey is a cohort survey. In a cohort survey, a researcher identifies some category of people that are of interest and then regularly surveys people who fall into that category. The same people don’t necessarily participate from year to year, but all participants must meet whatever categorical criteria fulfill the researcher’s primary interest. Common cohorts that may be of interest to researchers include people of particular generations or those who were born around the same time period, graduating classes, people who began work in a given industry at the same time, or perhaps people who have some specific life experience in common.

An example of this sort of research can be seen in Christine Percheski’s work on cohort differences in women’s employment. [4] Percheski compared women’s employment rates across seven different generational cohorts, from Progressives born between 1906 and 1915 to Generation Xers born between 1966 and 1975. She found, among other patterns, that professional women’s labor force participation had increased across all cohorts. She also found that professional women with young children from Generation X had higher labor force participation rates than similar women from previous generations, concluding that mothers do not appear to be opting out of the workforce as some journalists have speculated. [5]

RESEARCH FOR HUMAN SERVICES pg. 124

All three types of longitudinal surveys share the strength that they permit a researcher to make observations over time. This means that if whatever behavior or other phenomenon the researcher is interested in changes, either because of some world event or because people age, the researcher will be able to capture those changes. Table 8.1 "Types of Longitudinal Surveys" summarizes each of the three types of longitudinal surveys.

Table 8.1 Types of Longitudinal Surveys

Sample type

Description

Trend Researcher examines changes in trends over time; the same people do not necessarily participate in the survey more than once.

Panel Researcher surveys the exact same sample several times over a period of time.

Cohort Researcher identifies some category of people that are of interest and then regularly surveys people who fall into that category.

Retrospective Finally, retrospective surveys are similar to other longitudinal studies in that they deal with changes over time, but like a cross-sectional study, they are administered only once. In a retrospective survey, participants are asked to report events from the past. By having respondents report past behaviors, beliefs, or experiences, researchers are able to gather longitudinal-like data without actually incurring the time or expense of a longitudinal survey. Of course, this benefit must be weighed against the possibility that people’s recollections of their pasts may be faulty. Imagine, for example, that you’re asked in a survey to respond to questions about where, how, and with whom you spent last Valentine’s Day. As last Valentine’s Day can’t have been more than 12 months ago, chances are good that you might be able to respond accurately to any survey questions about it. But now let’s say the research wants to know how last Valentine’s Day compares to previous Valentine’s Days, so he asks you to report on where, how, and with whom you spent the preceding six Valentine’s Days. How likely is it that you will remember? Will your responses be as accurate as they might have been had you been asked the question each year over the past 6 years rather than asked to report on all years today?

In sum, when or with what frequency a survey is administered will determine whether your survey is cross-sectional or longitudinal. While longitudinal surveys are certainly preferable in terms of their ability to track changes over time, the time and cost required to administer a longitudinal survey can be prohibitive. As you may have guessed, the issues of time described here are not necessarily unique to survey research. Other methods of data collection can be cross-sectional or longitudinal— these are really matters of research design. But we’ve placed our discussion of these terms here because they are most commonly used by survey researchers to describe the type of survey administered. Another aspect of survey administration deals with how surveys are administered. We’ll examine that next.

Administration Surveys vary not just in terms of when they are administered but also in terms of how they are administered. One common way to administer surveys is in the form of self-administered questionnaires. This means that a research participant is given a set of questions, in writing, to which

RESEARCH FOR HUMAN SERVICES pg. 125

he or she is asked to respond. Self-administered questionnaires can be delivered in hard copy format, typically via mail, or increasingly more commonly, online. We’ll consider both modes of delivery here.

Hard copy self-administered questionnaires may be delivered to participants in person or via snail mail. Perhaps you’ve take a survey that was given to you in person; on many college campuses, it is not uncommon for researchers to administer surveys in large social science classes (as you might recall from the discussion in our chapter on sampling). In my own introduction to sociology courses, I’ve welcomed graduate students and professors doing research in areas that are relevant to my students, such as studies of campus life, to administer their surveys to the class. If you are ever asked to complete a survey in a similar setting, it might be interesting to note how your perspective on the survey and its questions could be shaped by the new knowledge you’re gaining about survey research in this chapter.

Researchers may also deliver surveys in person by going door-to-door and either asking people to fill them out right away or making arrangements for the researcher to return to pick up completed surveys. Though the advent of online survey tools has made door-to-door delivery of surveys less common, I still see an occasional survey researcher at my door, especially around election time. This mode of gathering data is apparently still used by political campaign workers, at least in some areas of the country.

If you are not able to visit each member of your sample personally to deliver a survey, you might consider sending your survey through the mail. While this mode of delivery may not be ideal (imagine how much less likely you’d probably be to return a survey that didn’t come with the researcher standing on your doorstep waiting to take it from you), sometimes it is the only available or the most practical option. As I’ve said, this may not be the most ideal way of administering a survey because it can be difficult to convince people to take the time to complete and return your survey.

Often survey researchers who deliver their surveys via snail mail may provide some advance notice to respondents about the survey to get people thinking about and preparing to complete it. They may also follow up with their sample a few weeks after their survey has been sent out. This can be done not only to remind those who have not yet completed the survey to please do so but also to thank those who have already returned the survey. Most survey researchers agree that this sort of follow-up is essential for improving mailed surveys’ return rates. [6]

Example In one study of older workers’ harassment experiences, people in the sample were notified in advance of the survey mailing via an article describing the research in a newsletter they received from the agency with whom I had partnered to conduct the survey. The survey was mailed a $1 bill was included with each in order to provide some incentive and an advance token of thanks to participants for returning the surveys. Two months after the initial mailing went out, those who were sent a survey were contacted by phone. While returned surveys did not contain any identifying information about respondents, research assistants contacted individuals to whom a survey had been mailed to remind them that it was not too late to return their survey and to say thank to those who may have already done so. Four months after the initial mailing went out, everyone on the original mailing list received a letter thanking those who had returned the survey and once again reminding those who had not that it was not too late to do so. The letter included a return postcard for respondents to complete should they wish to receive another copy of the survey. Respondents were also provided a telephone number to call and were provided the option of completing the survey by phone. As you can see, administering a survey by mail typically involves much more than simply arranging a single mailing; participants may be notified in advance of the mailing, they then receive the mailing, and then several follow-up contacts will likely be made after the survey has been mailed.

RESEARCH FOR HUMAN SERVICES pg. 126

Online Delivery Online delivery, referred to earlier, is another way to administer a survey. This delivery mechanism is becoming increasingly common, no doubt because it is easy to use, relatively cheap, and may be quicker than knocking on doors or waiting for mailed surveys to be returned. To deliver a survey online, a researcher may subscribe to a service that offers online delivery or use some delivery mechanism that is available for free. SurveyMonkey offers both free and paid online survey services (http://www.surveymonkey.com). One advantage to using a service like SurveyMonkey, aside from the advantages of online delivery already mentioned, is that results can be provided to you in formats that are readable by data analysis programs such as PSPP, SPSS, Systat, and Excel. This saves you, the researcher, the step of having to manually enter data into your analysis program, as you would if you administered your survey in hard copy format.

Many of the suggestions provided for improving the response rate on a hard copy questionnaire apply to online questionnaires as well. One difference of course is that the sort of incentives one can provide in an online format differ from those that can be given in person or sent through the mail. But this doesn’t mean that online survey researchers cannot offer completion incentives to their respondents. I’ve taken a number of online surveys; many of these did not come with an incentive other than the joy of knowing that I’d helped a fellow social scientist do his or her job, but on one I was given a printable $5 coupon to my university’s campus dining services on completion, and another time I was given a coupon code to use for $10 off any order on Amazon.com. I’ve taken other online surveys where on completion I could provide my name and contact information if I wished to be entered into a drawing together with other study participants to win a larger gift, such as a $50 gift card or an iPad.

Sometimes surveys are administered by having a researcher actually pose questions directly to respondents rather than having respondents read the questions on their own. These types of surveys are a form of interviews. We discuss interviews in Chapter 9 "Interviews: Qualitative and Quantitative Approaches", where we’ll examine interviews of the survey (or quantitative) type and qualitative interviews as well. Interview methodology differs from survey research in that data are collected via a personal interaction. Because asking people questions in person comes with a set of guidelines and concerns that differ from those associated with asking questions on paper or online, we’ll reserve our discussion of those guidelines and concerns for Chapter 9 "Interviews: Qualitative and Quantitative Approaches".

Whatever delivery mechanism you choose, keep in mind that there are pros and cons to each of the options described here. While online surveys may be faster and cheaper than mailed surveys, can you be certain that every person in your sample will have the necessary computer hardware, software, and Internet access in order to complete your online survey? On the other hand, perhaps mailed surveys are more likely to reach your entire sample but also more likely to be lost and not returned. The choice of which delivery mechanism is best, depends on a number of factors including your resources, the resources of your study participants, and the time you have available to distribute surveys and wait for responses. In my own survey of older workers, I would have much preferred to administer my survey online, but because so few people in my sample were likely to have computers, and even fewer would have Internet access, I chose instead to mail paper copies of the survey to respondents’ homes. Understanding the characteristics of your study’s population is key to identifying the appropriate mechanism for delivering your survey.

Summary

 Time is a factor in determining what type of survey researcher administers; cross-sectional surveys are administered at one time, and longitudinal surveys are administered over time.

RESEARCH FOR HUMAN SERVICES pg. 127

 Retrospective surveys offer some of the benefits of longitudinal research but also come with their own drawbacks.

 Self-administered questionnaires may be delivered in hard copy form to participants in person or via snail mail or online.

Exercises

1. If the idea of a panel study piqued your interest, check out the Up series of documentary films. While not a survey, the films offer one example of a panel study. Filmmakers began filming the lives of 14 British children in 1964, when the children were 7 years old. They have since caught up with the children every 7 years. In 2012, the eighth installment of the documentary, 56 Up, will come out. Many clips from the series are available on YouTube.

2. For more information about online delivery of surveys, check out SurveyMonkey’s website: http://www.surveymonkey.com.

[1] Kezdy, A., Martos, T., Boland, V., & Horvath-Szabo, K. (2011). Religious doubts and mental health in adolescence and young adulthood: The association with religious attitudes. Journal of Adolescence, 34, 39–47.

[2] Bateman, P. J., Pike, J. C., & Butler, B. S. (2011). To disclose or not: Publicness in social networking sites. Information Technology & People, 24, 78–100.

[3] Mortimer, J. T. (2003). Working and growing up in America. Cambridge, MA: Harvard University Press.

[4] Percheski, C. (2008). Opting out? Cohort differences in professional women’s employment rates from 1960 to 2005. American Sociological Review, 73, 497–517.

[5] Belkin, L. (2003, October 26). The opt-out revolution. New York Times, pp. 42–47, 58, 85–86.

[6] Babbie, E. (2010). The practice of social research (12th ed.). Belmont, CA: Wadsworth.

RESEARCH FOR HUMAN SERVICES pg. 128

8.4 Designing Effective Questions and Questionnaires

LEARNING OBJECTIVES

 Identify the steps one should take in order to write effective survey questions.

 Describe some of the ways that survey questions might confuse respondents and how to overcome that possibility.

 Recite the two-response option guidelines when writing closed-ended questions.

 Define fence-sitting and floating.

 Describe the steps involved in constructing a well-designed questionnaire.

 Discuss why pretesting is important.

To this point we’ve considered several general points about surveys including when to use them, some of their pros and cons, and how often and in what ways to administer surveys. In this section, we’ll get more specific and take a look at how to pose understandable questions that will yield useable data and how to present those questions on your questionnaire.

Asking Effective Questions What do you want to know?

The first thing you need to do in order to write effective survey questions is identify what exactly it is that you wish to know. As silly as it sounds to state what seems so completely obvious, I can’t stress enough how easy it is to forget to include important questions when designing a survey. Let’s say you want to understand how students at your school made the transition from high school to college. Perhaps you wish to identify which students were comparatively more or less successful in this transition and which factors contributed to students’ success or lack thereof. To understand which factors shaped successful students’ transitions to college, you’ll need to include questions in your survey about all the possible factors that could contribute. Consulting the literature on the topic will certainly help, but you should also take the time to do some brainstorming on your own and to talk with others about what they think may be important in the transition to college. Perhaps time or space limitations won’t allow you to include every single item you’ve come up with, so you’ll also need to think about ranking your questions so that you can be sure to include those that you view as most important.

Although I have stressed the importance of including questions on all topics you view as important to your overall research question, you don’t want to take an everything-but-the-kitchen-sink approach by uncritically including every possible question that occurs to you. Doing so puts an unnecessary burden on your survey respondents. Remember that you have asked your respondents to give you their time and attention and to take care in responding to your questions; show them your respect by only asking questions that you view as important.

Are the questions clear? Once you’ve identified all the topics about which you’d like to ask questions, you’ll need to actually write those questions. Questions should be as clear and to the point as possible. This is not the time to show off your creative writing skills; a survey is a technical instrument and should be written in a way that is as direct and succinct as possible. As I’ve said, your survey respondents have agreed to give their time and attention to your survey. The best way to show your appreciation for their time is to

RESEARCH FOR HUMAN SERVICES pg. 129

not waste it. Ensuring that your questions are clear and not overly wordy will go a long way toward showing your respondents the gratitude they deserve.

Related to the point about not wasting respondents’ time, make sure that every question you pose will be relevant to every person you ask to complete it. This means two things: first, that respondents have knowledge about whatever topic you are asking them about, and second, that respondents have experience with whatever events, behaviors, or feelings you are asking them to report. You probably wouldn’t want to ask a sample of 18-year-old respondents, for example, how they would have advised President Reagan to proceed when news of the United States’ sale of weapons to Iran broke in the mid-1980s. For one thing, few 18-year-olds are likely to have any clue about how to advise a president (nor does this 30-something-year-old). Furthermore, the 18-year-olds of today were not even alive during Reagan’s presidency, so they have had no experience with the event about which they are being questioned. In our example of the transition to college, heeding the criterion of relevance would mean that respondents must understand what exactly you mean by “transition to college” if you are going to use that phrase in your survey and that respondents must have actually experienced the transition to college themselves.

If you decide that you do wish to pose some questions about matters with which only a portion of respondents will have had experience, it may be appropriate to introduce a filter question into your survey. A filter question is designed to identify some subset of survey respondents who are asked additional questions that are not relevant to the entire sample. Perhaps in your survey on the transition to college you want to know whether substance use plays any role in students’ transitions. You may ask students how often they drank during their first semester of college. But this assumes that all students drank. Certainly, some may have abstained, and it wouldn’t make any sense to ask the nondrinkers how often they drank. Nevertheless, it seems reasonable that drinking frequency may have an impact on someone’s transition to college, so it is probably worth asking this question even if doing so violates the rule of relevance for some respondents. This is just the sort of instance when a filter question would be appropriate. You may pose the question as it is presented below:

RESEARCH FOR HUMAN SERVICES pg. 130

FIGURE 8.8 FILTER QUESTIONS

s

Avoiding Confusing Questions There are some ways of asking questions that are bound to confuse a good many survey respondents. [1] Survey researchers should take great care to avoid these kinds of questions. These include questions that pose double negatives, those that use confusing or culturally specific terms, and those that ask more than one question but are posed as a single question. Any time respondents are forced to decipher questions that utilize two forms of negation, confusion is bound to ensue. Taking the previous question about drinking as our example, what if we had instead asked, “Did you

RESEARCH FOR HUMAN SERVICES pg. 131

not drink during your first semester of college?” A response of no would mean that the respondent did actually drink—he or she did not not drink. This example is obvious, but hopefully it drives home the point to be careful about question wording so that respondents are not asked to decipher double negatives. In general, avoiding negative terms in your question wording will help to increase respondent understanding. Though this is generally true, some researchers argue that negatively worded questions should be integrated with positively worded questions in order to ensure that respondents have actually carefully read each question. (See, for example, the following: Vaterlaus, M., & Higgenbotham, B. (2011).)

You should also avoid using terms or phrases that may be regionally or culturally specific (unless you are absolutely certain all your respondents come from the region or culture whose terms you are using). For example, some regions of the country refer to soda as pop, or soda pop. Be sensitive to local uses of language.

Double-Barreled Questions Asking multiple questions as though they are a single question can also be terribly confusing for survey respondents. There’s a specific term for this sort of question; it is called a double-barreled question. Using our example of the transition to college, Figure 8.9 "Double-Barreled Question" shows a double-barreled question.

Figure 8.9 Double-Barreled Question

Do you see what makes this question double-barreled? How would someone respond if they felt their college classes were more demanding but also more boring than their high school classes? Or less demanding but more interesting? Because the question combines “demanding” and “interesting,” there is no way to respond yes to one criterion but no to the other.

Here is a simple definition of a double-barreled question. It is when someone asks a question that touches upon more than one issue, yet allows only one answer. Here is an example of an obvious one: Do you feel like the senate and the president are acting responsibly regarding tax reform? Yes or No.

Social Desirability Another thing to avoid when constructing survey questions is the problem of social desirability. We all want to look good, right? And we all probably know the politically correct response to a variety of questions whether we agree with the politically correct response or not. In survey research, social desirability refers to the idea that respondents will try to answer questions in a way that will present them in a favorable light. Perhaps we decide that to understand the transition to college, we need to know whether respondents ever cheated on an exam in high school or college. We all know that

RESEARCH FOR HUMAN SERVICES pg. 132

cheating on exams is generally frowned upon (at least I hope we all know this). So, it may be difficult to get people to admit to cheating on a survey. But if you can guarantee respondents’ confidentiality, or even better, their anonymity, chances are much better that they will be honest about having engaged in this socially undesirable behavior. Another way to avoid problems of social desirability is to try to phrase difficult questions in the most benign way possible. Earl Babbie offers a useful suggestion for helping you do this—simply imagine how you would feel responding to your survey questions. [2] If you would be uncomfortable, chances are others would as well.

Pilot Testing Earlier we discussed pilot questions. It is always a good idea to do at least a cursory pilot study with your questionnaire. Finally, it is important to get feedback on your survey questions from as many people as possible, especially people who are like those in your sample. Now is not the time to be shy. Ask your friends for help, ask your mentors for feedback, ask your family to take a look at your survey as well. The more feedback you can get on your survey questions, the better the chances that you will come up with a set of questions that are understandable to a wide variety of people and, most importantly, to those in your sample.

In sum, in order to pose effective survey questions, researchers should do the following:

1. Identify what it is they wish to know. 2. Keep questions clear and succinct. 3. Make questions relevant to respondents. 4. Use filter questions when necessary. 5. Avoid questions that are likely to confuse respondents such as those that use double

negatives, use culturally specific terms, or pose more than one question in the form of a single question.

6. Imagine how they would feel responding to questions. 7. Get feedback, especially from people who resemble those in the researcher’s sample.

Response Options While posing clear and understandable questions in your survey is certainly important, so, too, is providing respondents with unambiguous response options. Response options are the answers that you provide to the people taking your survey. Generally, respondents will be asked to choose a single (or best) response to each question you pose, though certainly it makes sense in some cases to instruct respondents to choose multiple response options. One caution to keep in mind when accepting multiple responses to a single question, however, is that doing so may add complexity when it comes to tallying and analyzing your survey results.

Closed Ended Questions Offering response options assumes that your questions will be closed-ended questions. In a quantitative written survey, which is the type of survey we’ve been discussing here, chances are good that most if not all your questions will be closed ended. This means that you, the researcher, will provide respondents with a limited set of options for their responses. To write an effective closed- ended question, there are a couple of guidelines worth following. First, be sure that your response options are mutually exclusive. Look back at Figure 8.8 "Filter Question", which contains questions about how often and how many drinks respondents consumed. Do you notice that there are no overlapping categories in the response options for these questions? This is another one of those points about question construction that seems fairly obvious but that can be easily overlooked. Response options should also be exhaustive. In other words, every possible response should be covered in the set of response options that you provide. For example, note that in question 10a in Figure 8.8 "Filter Question" we have covered all possibilities—those who drank, say, an average of

RESEARCH FOR HUMAN SERVICES pg. 133

once per month can choose the first response option (“less than one time per week”) while those who drank multiple times a day each day of the week can choose the last response option (“7+”). All the possibilities in between these two extremes are covered by the middle three response options.

Open-Ended Questions Surveys need not be limited to closed-ended questions. Sometimes survey researchers include open- ended questions in their survey instruments as a way to gather additional details from respondents. An open-ended question does not include response options; instead, respondents are asked to reply to the question in their own way, using their own words. These questions are generally used to find out more about a survey participant’s experiences or feelings about whatever they are being asked to report in the survey. If, for example, a survey includes closed-ended questions asking respondents to report on their involvement in extracurricular activities during college, an open-ended question could ask respondents why they participated in those activities or what they gained from their participation. While responses to such questions may also be captured using a closed-ended format, allowing participants to share some of their responses in their own words can make the experience of completing the survey more satisfying to respondents and can also reveal new motivations or explanations that had not occurred to the researcher.

In Section 8.4.1 "Asking Effective Questions" we discussed double-barreled questions, but response options can also be double barreled, and this should be avoided. Figure 8.10 "Double-Barreled Response Options" is an example of a question that uses double-barreled response options.

Fence Sitting and Floating Other things to avoid when it comes to response options include fence-sitting and floating.

Fence Setting Fence-sitters are respondents who choose neutral response options, even if they have an opinion. This can occur if respondents are given, say, five rank-ordered response options, such as strongly agree, agree, no opinion, disagree, and strongly disagree. Some people will be drawn to respond “no opinion” even if they have an opinion, particularly if their true opinion is the non-socially desirable opinion.

Floating Floaters, on the other hand, are those that choose a substantive answer to a question when really, they don’t understand the question or don’t have an opinion. If a respondent is only given four rank- ordered response options, such as strongly agree, agree, disagree, and strongly disagree, those who have no opinion have no choice but to select a response that suggests they have an opinion.

As you can see, floating is the flip side of fence-sitting. Thus, the solution to one problem is often the cause of the other.

How you decide which approach to take depends on the goals of your research. Sometimes researchers actually want to learn something about people who claim to have no opinion. In this case, allowing for fence-sitting would be necessary. Other times researchers feel confident their respondents will all be familiar with every topic in their survey. In this case, perhaps it is OK to force respondents to choose an opinion. There is no always-correct solution to either problem.

Using a Matrix Finally, using a matrix is a nice way of streamlining response options. A is a question type that lists a set of questions for which the answer categories are all the same. If you have a set of questions for

RESEARCH FOR HUMAN SERVICES pg. 134

which the response options are the same, it may make sense to create a matrix rather than posing each question and its response options individually. Not only will this save you some space in your survey but it will also help respondents progress through your survey more easily. A sample matrix can be seen in Figure 8.11 "Survey Questions Utilizing Matrix Format".

Figure 8.11 Survey Questions Utilizing Matrix Format

This is also an example of a Likert Scale. The above example is also “forced choice” Likert Scale because the respondent has to take a stand one way or another. Even numbered scales have no middle ground. The next example is an odd numbered scale which allows the respondent the option of essentially having no opinion, or preference, either way.

RESEARCH FOR HUMAN SERVICES pg. 135

Designing Questionnaires In addition to constructing quality questions and posing clear response options, you’ll also need to think about how to present your written questions and response options to survey respondents. Questions are presented on a questionnaire, the document (either hard copy or online) that contains all your survey questions that respondents read and mark their responses on. Designing questionnaires takes some thought, and in this section, we’ll discuss the sorts of things you should think about as you prepare to present your well-constructed survey questions on a questionnaire.

One of the first things to do once you’ve come up with a set of survey questions you feel confident about is to group those questions thematically. In our example of the transition to college, perhaps we’d have a few questions asking about study habits, others focused on friendships, and still others on

RESEARCH FOR HUMAN SERVICES pg. 136

exercise and eating habits. Those may be the themes around which we organize our questions. Or perhaps it would make more sense to present any questions we had about precollege life and habits and then present a series of questions about life after beginning college. The point here is to be deliberate about how you present your questions to respondents.

Order of Questions: It Does Matter Once you have grouped similar questions together, you’ll need to think about the order in which to present those question groups. Most survey researchers agree that it is best to begin a survey with questions that will want to make respondents continue. [3] In other words, don’t bore respondents, but don’t scare them away either. There’s some disagreement over where on a survey to place demographic questions such as those about a person’s age, gender, and race. On the one hand, placing them at the beginning of the questionnaire may lead respondents to think the survey is boring, unimportant, and not something they want to bother completing. On the other hand, if your survey deals with some very sensitive or difficult topic, such as child sexual abuse or other criminal activity, you don’t want to scare respondents away or shock them by beginning with your most intrusive questions.

In truth, the order in which you present questions on a survey is best determined by the unique characteristics of your research—only you, the researcher, hopefully in consultation with people who are willing to provide you with feedback, can determine how best to order your questions. To do so, think about the unique characteristics of your topic, your questions, and most importantly, your sample. Keeping in mind the characteristics and needs of the people you will ask to complete your survey should help guide you as you determine the most appropriate order in which to present your questions.

Questions of Time You’ll also need to consider the time it will take respondents to complete your questionnaire. Surveys vary in length, from just a page or two to a dozen or more pages, which means they also vary in the time it takes to complete them. How long to make your survey depends on several factors. First, what is it that you wish to know? Wanting to understand how grades vary by gender and year in school certainly requires fewer questions than wanting to know how people’s experiences in college are shaped by demographic characteristics, college attended, housing situation, family background, college major, friendship networks, and extracurricular activities. Keep in mind that even if your research question requires a good number of questions be included in your questionnaire, do your best to keep the questionnaire as brief as possible. Any hint that you’ve thrown in a bunch of useless questions just for the sake of throwing them in will turn off respondents and may make them not want to complete your survey.

Second, and perhaps more important, how long are respondents likely to be willing to spend completing your questionnaire? If you are studying college students, asking them to use their precious fun time away from studying to complete your survey may mean they won’t want to spend more than a few minutes on it. But if you have the endorsement of a professor who is willing to allow you to administer your survey in class, students may be willing to give you a little more time (though perhaps the professor will not). The time that survey researchers ask respondents to spend on questionnaires varies greatly. Some advise that surveys should not take longer than about 15 minutes to complete [4] while others suggest that up to 20 minutes is acceptable. [5] As with question order, there is no clear- cut, always-correct answer about questionnaire length. The unique characteristics of your study and your sample should be considered in order to determine how long to make your questionnaire.

RESEARCH FOR HUMAN SERVICES pg. 137

Pretesting & Time to Complete Pretesting is actually another form of pilot testing. We have already discussed pilot testing your questions. Now it is time to take it further. A good way to estimate the time it will take respondents to complete your questionnaire is through pretesting. Pretesting allows you to get feedback on your questionnaire so you can improve it before you actually administer it. Pretesting can be quite expensive and time consuming if you wish to test your questionnaire on a large sample of people who very much resemble the sample to whom you will eventually administer the finalized version of your questionnaire. But you can learn a lot and make great improvements to your questionnaire simply by pretesting with a small number of people to whom you have easy access (perhaps you have a few friends who owe you a favor). By pretesting your questionnaire, you can find out how understandable your questions are, get feedback on question wording and order, find out whether any of your questions are exceptionally boring or offensive, and learn whether there are places where you should have included filter questions, to name just a few of the benefits of pretesting. You can also time pretesters as they take your survey. Ask them to complete the survey as though they were actually members of your sample. This will give you a good idea about what sort of time estimate to provide respondents when it comes time to actually administer your survey, and about whether you have some wiggle room to add additional items or need to cut a few items.

Appearance Perhaps this goes without saying, but your questionnaire should also be attractive. Aesthetics does matter. A messy presentation style can confuse respondents or, at the very least, annoy them. Be brief, to the point, and as clear as possible. If the questionnaire does not convey a sense of competency and professionalism it will almost surely be reflected in how people respond. If you don’t care, why should they?

Avoid cramming too much into a single page, make your font size readable (at least 12 point), leave a reasonable amount of space between items, and make sure all instructions are exceptionally clear. Think about books, documents, articles, or web pages that you have read yourself—which were relatively easy to read and easy on the eyes and why? Try to mimic those features in the presentation of your survey questions.

Summary

 Brainstorming and consulting the literature are two important early steps to take when preparing to write effective survey questions.

 Make sure that your survey questions will be relevant to all respondents and that you use filter questions when necessary.

 Getting feedback on your survey questions is a crucial step in the process of designing a survey.  When it comes to creating response options, the solution to the problem of fence-sitting might

cause floating, whereas the solution to the problem of floating might cause fence sitting.  Pretesting is an important step for improving one’s survey before actually administering it.

Exercises

1. Do a little Internet research to find out what a Likert scale is and when you may use one. 2. Write a closed-ended question that follows the guidelines for good survey question construction.

Have a peer in the class check your work (you can do the same for him or her!).

[1] Writing survey questions for local program evaluations. Retrieved from http://extension.usu.edu/files/publications/publication/FC_Evaluation_2011-02pr.pdf

RESEARCH FOR HUMAN SERVICES pg. 138

[2] Babbie, E. (2010). The practice of social research (12th ed.). Belmont, CA: Wadsworth.

[3] Babbie, E. (2010). The practice of social research (12th ed.). Belmont, CA: Wadsworth

[3] Dillman, D. A. (2000). Mail and Internet surveys: The tailored design method (2nd ed.). New York, NY: Wiley

[3] Neuman, W. L. (2003). Social research methods: Qualitative and quantitative approaches (5th ed.). Boston, MA: Pearson.

[4] As cited in Babbie, E. (2010). The practice of social research (12th ed.). Belmont, CA: Wadsworth. Retrieved from http://www.worldopinion.com/the_frame/frame4.html

[5] Hopper, J. (2010). How long should a survey be? Retrieved from http://www.verstaresearch.com/blog/how-long-should-a-survey-be

8.5 Analysis of Survey Data LEARNING OBJECTIVES

 Define response rate, and discuss some of the current thinking about response rates.

 Describe what a codebook is and what purpose it serves.

 Define univariate, bivariate, and multivariate analysis.

 Describe each of the measures of central tendency.

 Describe what a contingency table displays.

This text is primarily focused on designing research, collecting data, and becoming a knowledgeable and responsible consumer of research. We won’t spend as much time on data analysis, or what to do with our data once we’ve designed a study and collected it, but I will spend some time in each of our data-collection chapters describing some important basics of data analysis that are unique to each method. Entire textbooks could be (and have been) written entirely on data analysis. In fact, if you’ve ever taken a statistics class, you already know much about how to analyze quantitative survey data. Here we’ll go over a few basics that can get you started as you begin to think about turning all those completed questionnaires into findings that you can share.

From Completed Questionnaires to Analyzable Data It can be very exciting to receive those first few completed surveys back from respondents. Hopefully you’ll even get more than a few back, and once you have a handful of completed questionnaires, your feelings may go from initial euphoria to dread. Data are fun and can also be overwhelming. The goal with data analysis is to be able to condense large amounts of information into usable and understandable chunks. Here we’ll describe just how that process works for survey researchers.

Response Rate As mentioned, the hope is that you will receive a good portion of the questionnaires you distributed back in a completed and readable format. The number of completed questionnaires you receive divided by the number of questionnaires you distributed is your response rate.

RESEARCH FOR HUMAN SERVICES pg. 139

Let’s say your sample included 100 people and you sent questionnaires to each of those people. It would be wonderful if all 100 returned completed questionnaires, but the chances of that happening are about zero. If you’re lucky, perhaps 75 or so will return completed questionnaires. In this case, your response rate would be 75% (75 divided by 100). That’s pretty darn good. Though response rates vary, and researchers don’t always agree about what makes a good response rate, having three- quarters of your surveys returned would be considered good, even excellent, by most survey researchers.

How to Improve Response Rate There has been lots of research done on how to improve a survey’s response rate. We covered some of these previously, but suggestions include personalizing questionnaires by, for example, addressing them to specific respondents rather than to some generic recipient such as “madam” or “sir”; enhancing the questionnaire’s credibility by providing details about the study, contact information for the researcher, and perhaps partnering with agencies likely to be respected by respondents such as universities, hospitals, or other relevant organizations; sending out prequestionnaire notices and postquestionnaire reminders; and including some token of appreciation with mailed questionnaires even if small, such as a $1 bill.

The major concern with response rates is that a low rate of response may introduce nonresponse bias into a study’s findings. What if only those who have strong opinions about your study topic return their questionnaires? If that is the case, we may well find that our findings don’t at all represent how things really are or, at the very least, we are limited in the claims we can make about patterns found in our data.

Have we been too concerned with high response rates? While high return rates are certainly ideal, a recent body of research shows that concern over response rates may be overblown. [3] Several studies have shown that low response rates did not make much difference in findings or in sample representativeness. [4] For now, the jury may still be out on what makes an ideal response rate and on whether, or to what extent, researchers should be concerned about response rates. Nevertheless, certainly no harm can come from aiming for as high a response rate as possible.

3 Langer, G. (2003). About response rates: Some unresolved questions. Public Perspective,

May/June, 16–18. Retrieved from

http://www.aapor.org/Content/aapor/Resources/PollampSurveyFAQ1/DoResponseRatesMatter/Respo

nse_Rates_-_Langer.pdf

4 (Curtin, Presser, & Singer, 2000; Keeter, Kennedy, Dimock, Best, & Craighill, 2006; Merkle &

Edelman, 2002).Curtin, R., Presser, S., & Singer, E. (2000). The effects of response rate changes on

the index of consumer sentiment. Public Opinion Quarterly, 64, 413–428; Keeter, S., Kennedy, C.,

Dimock, M., Best, J., & Craighill, P. (2006).

Gauging the impact of growing nonresponse on estimates from a national RDD telephone

survey. Public Opinion Quarterly, 70, 759–779; Merkle, D. M., & Edelman, M. (2002). Nonresponse in

exit polls: A comprehensive analysis. In M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.),

Survey nonresponse (pp. 243–258). New York, NY: John Wiley and Sons.6i

RESEARCH FOR HUMAN SERVICES pg. 140

Whatever your survey’s response rate, the major concern of survey researchers once they have their nice, big stack of completed questionnaires is condensing their data into manageable, and analyzable, bits. One major advantage of quantitative methods such as survey research, as you may recall from, is that they enable researchers to describe large amounts of data because they can be represented by and condensed into numbers. In order to condense your completed surveys into analyzable numbers, you’ll first need to create a codebook. A codebook is a document that outlines how a survey researcher has translated her or his data from words into numbers.

Managing, Sorting, and Ordering Your Data If you’ve administered your questionnaire the old-fashioned way, via snail mail, the next task after creating your codebook is data entry. If you’ve utilized an online tool such as SurveyMonkey to administer your survey, here’s some good news—most online survey tools come with the capability of importing survey results directly into a data analysis program. Trust me—this is indeed most excellent news. (If you don’t believe me, I highly recommend administering hard copies of your questionnaire next time around. You’ll surely then appreciate the wonders of online survey administration.)

For those who will be conducting manual data entry, there probably isn’t much I can say about this task that will make you want to perform it other than pointing out the reward of having a database of your very own analyzable data.

Using Statistical Software

Excel and Open Source Options After graduation, you will probably find yourself sitting at a desk with nothing but Excel available to you for statistical analysis. Most social service agencies will not have the budget for anything else. Excel is powerful, and odds are that it will be capable of doing any statistical analysis your project may require. A whole industry surrounds teaching people how to use it, and you will certainly be able to find tutorials, and even formal classes. Excel tends to be more business oriented though and that can be a factor when dealing with a more science orientation. There are other options. However, you could also avail yourself to some of the open source options which are available free of charge.

PSPP The open source statistical software package we recommend is PSPP. PSPP is a capable open source statistics program and, like this book, is available free for the downloading. It will run on Windows, Linux, and Macs. Like Excel, there are lots of tutorials, and other resources, online to help you learn it. It is not a spreadsheet per se and you can use a spread sheet to enter and organize data and then import it into PSPP for statistical manipulation.

Libre Office and Calc Libre Office is a suite of free open source office productivity programs—very capable, and well developed. It has Calc, a spreadsheet program which is the open source version of ubiquitous Excel. Unfortunately, due to the closed source nature used for proprietary software, it is not fully compatible with Excel. R is another open source statistical package.

R R is a very powerful statistical tool, but it has a steep learning curve. It is an open source programming language and software environment for statistical computing and graphics. It is maintained and supported by the R Foundation for Statistical Computing. It is widely used among scientist, academics, statisticians

and data miners. There are also ancillary programs that work with and enhance R, making it easier to use. In addition to tutorials it too has a wide selection of classes, and publications that you can access. It is robust, and will do anything you need it to do if you only know how. Like PSPP, R will run on any

RESEARCH FOR HUMAN SERVICES pg. 141

of the major operating systems. See the R Project for Statistical computing at https://www.r- project.org/.

Specialty Commercial Software Options Commercial statistical software is available for you to consider. However, commercial statistical packages are expensive, and sometimes subscription based. Unfortunately, many students are taught how to do statistics on this type of software, and will never have access to it again after graduation. A few of the commercial programs some researchers use to analyze data are SPSS, or the Statistical Package for the Social Sciences (http://www.spss.com). SPSS is a statistical analysis computer program designed to analyze just the sort of data quantitative survey researchers collect. It can perform everything from very basic descriptive statistical analysis to more complex inferential statistical analysis. SPSS is touted by many for being highly accessible and relatively easy to navigate (with practice). Other programs that are known for their accessibility include MicroCase (http://www.microcase.com/index.html), which includes many of the same features as SPSS, and Excel), which is far less sophisticated in its statistical capabilities but is relatively easy to use and suits some researchers’ purposes just fine.

What will this software enable you to do? Statistical software will help you to organize and find patterns within your data.

Identifying Patterns Regardless of the software you end up using, it is all about finding patterns and that is done through data analysis. Data analysis is about identifying, describing, and explaining patterns. Univariate analysis is the most basic form of analysis that quantitative researchers conduct. In this form, researchers describe patterns across just one variable. Univariate analysis includes frequency distributions and measures of central tendency. A frequency distribution is a way of summarizing the distribution of responses on a single survey question. Let’s look at the frequency distribution for just one variable.

Frequency Distribution A frequency distribution is a table that displays the frequency of different outcomes in a table. In this case, these questions are from a survey on income security.

RESEARCH FOR HUMAN SERVICES pg. 142

TABLE 8.3 FREQUENCY DISTRIBUTION OF OLDER WORKERS’ FINANCIAL SECURITY

In general, how financially secure would you say you are?

Value Frequency Percentage

Label

Not at all secure 1 46 25.6

Between not at all and moderately secure 2 43 23.9

Moderately secure 3 76 42.2

Between moderately and very secure 4 11 6.1

Very secure 5 4 2.2

Total valid cases = 180; no response = 3

You can tell from the frequency distribution on self-reported financial security that more respondents reported feeling “moderately secure” than any other response category. We also learn from this single frequency distribution that fewer than 10% of respondents reported being in one of the two most secure categories.

Frequency distribution is covered in-depth in the next chapter, along with histograms (a fancy term for a type of bar graph), and there is a separate tutorial in the appendix with four excellent examples of how to create a frequency distribution table.

Measures of Central Tendency Another form of univariate analysis that survey researchers can conduct on single variables is measures of central tendency. Measures of central tendency tell us what the most common, or average, response is on a question. Measures of central tendency can be taken for any level variable of those we learned about in Chapter 6 "Defining and Measuring Concepts", from nominal to ratio.

The Three Measures of Central Tendency There are three kinds of measures of central tendency: modes, medians, and means.

Mode Mode refers to the most common response given to a question. Modes are most appropriate for nominal-level variables.

Median Median is the middle point in a distribution of responses. Median is the appropriate measure of central tendency for ordinal-level variables.

Mean Finally, the measure of central tendency used for interval- and ratio-level variables is the mean. To obtain a mean, one must add the value of all responses on a given variable and then divide that number of the total number of responses.

RESEARCH FOR HUMAN SERVICES pg. 143

In the previous example of older workers’ self-reported levels of financial security, the appropriate measure of central tendency would be the median, as this is an ordinal-level variable. If we were to list all responses to the financial security question in order and then choose the middle point in that list, we’d have our median. In Figure 8.12 "Distribution of Responses and Median Value on Workers’ Financial Security", the value of each response to the financial security question is noted, and the middle point within that range of responses is highlighted. To find the middle point, we simply divide the number of valid cases by two. The number of valid cases, 180, divided by 2 is 90, so we’re looking for the 90th value on our distribution to discover the median. As you’ll see in Figure 8.12 "Distribution of Responses and Median Value on Workers’ Financial Security", that value is 3, thus the median on our financial security question is 3, or “moderately secure.”

Figure 8.12 Distribution of Responses and Median Value on Workers’ Financial Security

As you can see, we can learn a lot about our respondents simply by conducting univariate analysis of measures on our survey. We can learn even more, of course, when we begin to examine relationships among variables. Either we can analyze the relationships between two variables, called bivariate analysis, or we can examine relationships among more than two variables. This latter type of analysis is known as multivariate analysis.

Bivariate Analysis Bivariate analysis allows us to assess covariation among two variables. This means we can find out whether changes in one variable occur together with changes in another.

If two variables do not covary, they are said to have independence. This means simply that there is no relationship between the two variables in question. To learn whether a relationship exists between two variables, a researcher may cross-tabulate the two variables and present their relationship in a contingency table.

Contingency Tables A contingency table shows how variation on one variable may be contingent on variation on the other. Let’s take a look at a contingency table. In Table 8.4 "Financial Security Among Men and Women Workers Age 62 and Up", I have cross-tabulated two questions from my older worker survey: respondents’ reported gender and their self-rated financial security.

TABLE 8.4 FINANCIAL SECURITY AMONG MEN AND WOMEN WORKERS AGE 62 AND UP

Men Women

Not financially secure (%) 44.1 51.8

RESEARCH FOR HUMAN SERVICES pg. 144

Men Women

Moderately financially secure (%) 48.9 39.2

Financially secure (%) 7.0 9.0

Total N = 43 N = 135

Collapsing Categories You’ll see in Table 8.4 "Financial Security Among Men and Women Workers Age 62 and Up" that we collapsed a couple of the financial security response categories (recall that there were five categories presented in Table 8.3 "Frequency Distribution of Older Workers’ Financial Security"; here there are just three). Researchers often collapse response categories on items such as this in order to make it easier to read results in a table. It is a good idea though to collapse data retroactively, and not before the fact – before you have gathered the data. For example, if you ask people their age you would not provide a range of ages, but rather ask them for the year they were born or how old they are. If patterns emerge, as if in this example, you could then collapse the data for clarity.

Conventions Typically, values that are contingent on other values are placed in rows (a.k.a. dependent variables), while independent variables are placed in columns. This makes comparing across categories of our independent variable pretty simple. Reading across the top row of our table, we can see that around 44% of men in the sample reported that they are not financially secure while almost 52% of women reported the same. In other words, more women than men reported that they are not financially secure. You’ll also see in the table that I reported the total number of respondents for each category of the independent variable in the table’s bottom row. This is also standard practice in a bivariate table, as is including a table heading describing what is presented in the table.

Multivariate Analysis Researchers interested in simultaneously analyzing relationships among more than two variables conduct multivariate analysis. If I hypothesized that financial security declines for women as they age but increases for men as they age, I might consider adding age to the preceding analysis. To do so would require multivariate, rather than bivariate, analysis. We won’t go into detail here about how to conduct multivariate analysis of quantitative survey items here. It is beyond the scope of this class.

If you are interested in learning more about the analysis of quantitative survey data, I recommend checking out your campus’s offerings in statistics classes. Kahn Academy and EdX are also some outstanding free resources available online where you can learn more about more advanced methods of statistical analysis – including multivariate analysis. The quantitative data analysis skills you will gain in a statistics class could serve you quite well should you find yourself seeking employment one day. People that have the skills to analyze data are much in demand.

Summary

 While survey researchers should always aim to obtain the highest response rate possible, some recent research argues that high return rates on surveys may be less important than we once thought.

 There are several computer programs designed to assist survey researchers with analyzing their data include SPSS, MicroCase, and Excel.

 Data analysis is about identifying, describing, and explaining patterns.

RESEARCH FOR HUMAN SERVICES pg. 145

 Contingency tables show how, or whether, one variable covaries with another.

Exercises

1. Codebooks can range from relatively simple to quite complex. For an excellent example of a more complex codebook, check out the coding for the General Social Survey (GSS): http://publicdata.norc.org:41000/gss/documents//BOOK/GSS_Codebook.pdf.

2. The GSS allows researchers to cross-tabulate GSS variables directly from its website. Interested? Check out http://www.norc.uchicago.edu/GSS+Website/Data+Analysis.

RESEARCH FOR HUMAN SERVICES pg. 146

Chapter 9 Descriptive Statistics Statistics is the grammar of science. -Karl Pearson

At this point, we need to consider the basics of data analysis in social research in more detail using various methods of statistical analysis. In this chapter, we focus on descriptive statistics—a set of techniques for summarizing and displaying the data from your sample. We look first at some of the most common techniques for describing single variables, followed by some of the most common techniques for describing statistical relationships between variables. We end with some practical advice for organizing and carrying out your analyses.

9.1 Describing Single Variables

LEARNING OBJECTIVES  Use frequency tables and histograms to display and interpret the distribution of a variable.

 Compute and interpret the mean, median, and mode of a distribution and identify situations in which the mean, median, or mode is the most appropriate measure of central tendency.

 Compute and interpret the range and standard deviation of a distribution.

 Compute and interpret percentile ranks and z scores.

Descriptive statistics

Descriptive statistics is a way of summarizing data. Descriptive statistics refers to a set of techniques for summarizing and displaying data. Let us assume here that the data are quantitative and consist of scores on one or more variables for each of several study participants. Although in most cases the primary research question will be about one or more statistical relationships between variables, it is also important to describe each variable individually. For this reason, we begin by looking at some of the most common techniques for describing single variables.

The Distribution of a Variable Every variable has a distribution, which is the way the scores are distributed across the levels of that variable. For example, in a sample of 100 college students, the distribution of the variable “number of siblings” might be such that 10 of them have no siblings, 30 have one sibling, 40 have two siblings, and so on. In the same sample, the distribution of the variable “sex” might be such that 44 have a score of “male” and 56 have a score of “female.”

Frequency Tables One way to display the distribution of a variable is in a frequency table. The concept of frequency tables was introduced in the last chapter. There is an outstanding handout in the appendix that shows you how to create a frequency distribution table, and it is highly recommended that you avail yourself to it.

Let’s look at an example. Table 9.1 "Frequency Table Showing a Hypothetical Distribution of Scores on the Rosenberg Self-Esteem Scale", for example, is a frequency table showing a hypothetical distribution of scores on the Rosenberg Self-Esteem Scale for a sample of 40 college students. The first column lists the values of the variable—the possible scores on the Rosenberg scale—and the second column lists the frequency of each score. This table shows that there were three students who had self-esteem scores of 24, five who had self-esteem scores of 23, and so on. From a frequency table like this, one can quickly see several important aspects of a distribution, including the range of scores (from 15 to 24), the most and least common scores (22 and 17, respectively), and any extreme scores that stand out from the rest.

RESEARCH FOR HUMAN SERVICES pg. 147

TABLE 9.1 FREQUENCY TABLE SHOWING A HYPOTHETICAL DISTRIBUTION OF SCORES ON THE ROSENBERG SELF-ESTEEM SCALE

Self-esteem Score Frequency (# of times this score shows up)

24 3 23 5

22 10

21 8

20 5

19 3 18 3

17 0

16 2

15 1

Conventions for Frequency Distribution Tables There are a few other points worth noting about frequency tables. This includes some conventions which should be followed to avoid confusion. First, the levels listed in the first column usually go from the highest at the top to the lowest at the bottom, and they usually do not extend beyond the highest and lowest scores in the data. For example, although scores on the Rosenberg scale can vary from a high of 30 to a low of 0, Table 12.1 "Frequency Table Showing a Hypothetical Distribution of Scores on the Rosenberg Self-Esteem Scale" only includes levels from 24 to 15 because that range includes all the scores in this particular data set.

Second, when there are many different scores across a wide range of values, it is often better to create a grouped frequency table, in which the first column lists ranges of values and the second column lists the frequency of scores in each range. Table 12.2 "A Grouped Frequency Table Showing a Hypothetical Distribution of Reaction Times", for example, is a grouped frequency table showing a hypothetical distribution of simple reaction times for a sample of 20 participants. In a grouped frequency table, the ranges must all be of equal width, and there are usually between five and 15 of them.

Finally, frequency tables can also be used for categorical variables, in which case the levels are category labels. The order of the category labels is somewhat arbitrary, but they are often listed from the most frequent at the top to the least frequent at the bottom.

RESEARCH FOR HUMAN SERVICES pg. 148

Table 9.2 A Grouped Frequency Table Showing a Hypothetical Distribution of Reaction Times

Reaction time (ms) Frequency

241–260 1

221–240 2

201–220 2 181–200 9

161–180 4

141–160 2

Histograms A histogram is a graphical display of a distribution. It presents the same information as a frequency table but in a way, that is even quicker and easier to grasp. The histogram in Figure 9.1 "Histogram Showing the Distribution of Self-Esteem Scores Presented in " presents the distribution of self-esteem scores in Table 9.1 "Frequency Table Showing a Hypothetical Distribution of Scores on the Rosenberg Self-Esteem Scale". The x-axis of the histogram represents the variable and the y-axis represents frequency. Above each level of the variable on the x-axis is a vertical bar that represents the number of individuals with that score. When the variable is quantitative, as in this example, there is usually no gap between the bars. When the variable is categorical, however, there is usually a small gap between them. (The gap at 17 in this histogram reflects the fact that there were no scores of 17 in this data set.)

FIGURE 9.1 HISTOGRAM SHOWING THE DISTRIBUTION OF SELF-ESTEEM SCORES PRESENTED IN TABLE 9.1 "FREQUENCY TABLE SHOWING A HYPOTHETICAL DISTRIBUTION OF SCORES ON THE ROSENBERG SELF-ESTEEM SCALE"

RESEARCH FOR HUMAN SERVICES pg. 149

Distribution Shapes The shape of a distribution is very important, and when the distribution of a quantitative variable is displayed in a histogram, it has a shape. The shape of the distribution of self-esteem scores in Figure 9.1 "Histogram Showing the Distribution of Self-Esteem Scores Presented in " is typical. There is a peak somewhere near the middle of the distribution and “tails” that taper in either direction from the peak. The distribution of Figure 9.1, is unimodal, meaning it has one distinct peak, but distributions can also be bimodal, meaning they have two distinct peaks.

Figure 9.2 to the left, "Histogram Showing a Hypothetical Bimodal Distribution of Scores on the Beck Depression Inventory", is one example. It shows a hypothetical bimodal distribution of scores on the Beck Depression Inventory. Distributions can also have more than two distinct peaks, but these are relatively rare in most social sciences research.

FIGURE 9.2 HISTOGRAM SHOWING A HYPOTHETICAL BIMODAL DISTRIBUTION OF SCORES ON THE BECK DEPRESSION INVENTORY

An

Symmetrical or Skewed other characteristic of the shape of a distribution is whether it is symmetrical or skewed. The distribution

in the center of Figure 9.3 "Histograms Showing Negatively Skewed, Symmetrical, and Positively Skewed Distributions" is symmetrical. Its left and right halves are mirror images of each other. The distribution on the left is negatively skewed, with its peak shifted toward the upper end of its range and a relatively long negative tail. The distribution on the right is positively skewed, with its peak toward the lower end of its range and a relatively long positive tail.

RESEARCH FOR HUMAN SERVICES pg. 150

FIGURE 9.3 HISTOGRAMS SHOWING THREE KINDS OF DISTRIBUTION: NEGATIVELY SKEWED, SYMMETRICAL, AND POSITIVELY SKEWED DISTRIBUTIONS

Let’s look at a real-world distribution. Below is a chart from the U.S. Census Bureau that shows that the

population is positively skewed toward having a lower income. In other words, wealth in the United States

is unevenly distributed. In this depiction population is on the Y axis, and income is on the X axis. There is

another way we could configure the same data. We could reverse it. We could place income on the Y axis

instead of population, and population on the X axis instead of income.

With a coordinate graph like this there are conventions but no hard and fast rules. However, researchers

like to say that the “independent” variable goes on the x-axis (the bottom, horizontal one) and the

“dependent” variable goes on the y-axis (the left side, vertical one). By the way, the zero point to the far

bottom left where both lines originate is called the origin.

RESEARCH FOR HUMAN SERVICES pg. 151

Outlier An outlier is an extreme score that is much higher or lower than the rest of the scores in the distribution. Sometimes outliers represent truly extreme scores on the variable of interest. In the example above billionaires are extreme outliers.

In the Beck Depression Inventory, a single clinically depressed person might be an outlier in a sample of otherwise happy and high-functioning peers. However, outliers can also represent errors or misunderstandings on the part of the researcher or participant, equipment malfunctions, or similar problems. We will say more about how to interpret outliers and what to do about them later in this chapter.

Measures of Central Tendency and Variability It is also useful to be able to describe the characteristics of a distribution more precisely. Here we look at how to do this in terms of two important characteristics: their central tendency and their variability.

Central Tendency Like the frequency distribution table, the concept of central tendency was introduced in chapter eight. The central tendency of a distribution is its middle—the point around which the scores in the distribution tend to cluster. (Another term for central tendency is average.) Looking back at Figure 12.1 "Histogram Showing the Distribution of Self-Esteem Scores Presented in ", for example, we can see that the self- esteem scores tend to cluster around the values of 20 to 22. Here we will consider the three most common measures of central tendency: the mean, the median, and the mode.

Mean The mean of a distribution (symbolized M) is the sum of the scores divided by the number of scores. As a formula, it looks like this:

M=ΣXN

In this formula, the symbol Σ (the Greek letter sigma) is the summation sign and means to sum across the values of the variable X. N represents the number of scores. The mean is by far the most common measure of central tendency, and there are some good reasons for this. It usually provides a good indication of the central tendency of a distribution, and it is easily understood by most people. In addition, the mean has statistical properties that make it especially useful in doing inferential statistics.

Median An alternative to the mean is the median. The median is the middle score in the sense that half the scores in the distribution are less than it and half are greater than it. The simplest way to find the median is to organize the scores from lowest to highest and locate the score in the middle. Consider, for example, the following set of seven scores:

8 4 12 14 3 2 3

To find the median, simply rearrange the scores from lowest to highest and locate the one in the middle.

2 3 3 4 8 12 14

In this case, the median is 4 because there are three scores lower than 4 and three scores higher than 4. When there is an even number of scores, there are two scores in the middle of the distribution, in which case the median is the value halfway between them. For example, if we were to add a score of 15 to the preceding data set, there would be two scores (both 4 and 8) in the middle of the distribution, and the median would be halfway between them (6).

MODE One final measure of central tendency is the mode. The mode is the most frequent score in a distribution. In the self-esteem distribution presented in Table 12.1 "Frequency Table Showing a Hypothetical Distribution of Scores on the Rosenberg Self-Esteem Scale" and Figure 12.1 "Histogram Showing the

RESEARCH FOR HUMAN SERVICES pg. 152

Distribution of Self-Esteem Scores Presented in ", for example, the mode is 22. More students had that score than any other. The mode is the only measure of central tendency that can also be used for categorical variables.

In a distribution that is both unimodal and symmetrical, the mean, median, and mode will be very close to each other at the peak of the distribution.

Bimodal Bimodal distributions are interesting and have some special characteristics. In a bimodal, or asymmetrical distribution, the mean, median, and mode can be very different. In a bimodal distribution, the mean and median will tend to be between the peaks, while the mode will be at the tallest peak. If your distribution is bimodal there is a good possibility that you are looking at two different groups. In other words, there is some significant factor, which you may or may not be able to ascertain, that separates them.

For example, test scores are normally distributed with a single peak. However, grades sometimes fall into a bimodal distribution with a lot of students getting an A and a lot getting an F. This tells you that you are looking at two different groups that have split in different directions for some reason. For example, it could have been that there was a big campus, or school, even the night before with the preparation and participation of many students distracting them from their studies. Or it could be that the students are divided between those that have taken another class that was excellent preparation for the test while the others did not have that advantage.

In a skewed distribution, the mean will differ from the median in the direction of the skew (i.e., the direction of the longer tail).

For highly skewed distributions, the mean can be pulled so far in the direction of the skew that it is no longer a good measure of the central tendency of that distribution. Imagine, for example, a set of four simple reaction times of 200, 250, 280, and 250 milliseconds (ms). The mean is 245 ms. But the addition of one more score of 5,000 ms—perhaps because the participant was not paying attention—would raise the mean to 1,445 ms. Not only is this measure of central tendency greater than 80% of the scores in the distribution, but it also does not seem to represent the behavior of anyone in the distribution very well. This is why researchers often prefer the median for highly skewed distributions (such as distributions of reaction times).

Keep in mind, though, that you are not required to choose a single measure of central tendency in analyzing your data. Each one provides slightly different information, and all of them can be useful.

Measures of Variability The variability of a distribution is the extent to which the scores vary around their central tendency. Consider the two distributions in Figure 12.4 "Histograms Showing Hypothetical Distributions With the Same Mean, Median, and Mode (10) but With Low Variability (Top) and High Variability (Bottom)", both of which have the same central tendency. The mean, median, and mode of each distribution are 11. Notice, however, that the two distributions differ in terms of their variability. The top one has relatively low variability, with all the scores relatively close to the center. The bottom one has relatively high variability, with the scores are spread across a much greater range.

RESEARCH FOR HUMAN SERVICES pg. 153

FIGURE 9.4 HISTOGRAMS SHOWING HYPOTHETICAL DISTRIBUTIONS WITH THE SAME MEAN, MEDIAN, AND MODE (10) BUT WITH LOW VARIABILITY (TOP) AND HIGH VARIABILITY (BOTTOM)

One simple measure of variability is the range, which is simply the difference between the highest and lowest scores in the distribution. The range of the self-esteem scores in Table 12.1, for example, is the difference between the highest score (24) and the lowest score (15). That is, the range is 24 − 15 = 9. Although the range is easy to compute and understand, it can be misleading when there are outliers. Imagine, for example, an exam on which all the students scored between 90 and 100. It has a range of 10. But if there was a single student who scored 20, the range would increase to 80—giving the impression that the scores were quite variable when in fact only one student differed substantially from the rest.

Standard Deviation A standard deviation is a measure of how spread out numbers are.

We can examine many things like average height, weight, average rainfall, the price of commodities, return on investment, and other social and natural phenomenon with this tool. It is by far the most commonly used measure of variability in the social sciences after the mean. Any person completing an undergraduate degree in research or statistics in this field should thoroughly understand how to us it.

IQ and the Standard Deviation

Even things like intelligence using the IQ, or intelligence quotient, falls along a distribution. The mean, or average, IQ is 100. Normally standard deviations are 15 points. The majority of the population, 68.26%, falls within one standard deviation of the mean (IQ 85-115).

A low standard deviation means that most of the numbers are very close to the average. A high standard deviation means that the numbers are spread out.

RESEARCH FOR HUMAN SERVICES pg. 154

The standard deviation of a distribution is, roughly speaking, the average distance between the scores and the mean. It is a number used to tell how measurements for a group are spread out from the average (mean), or expected value.

Computing the Standard Deviation Here are the steps that you follow to compute the standard deviation. We will look at the formula itself at the end of this section. The steps are:

1. Compute the Mean (the simple average of the numbers) 2. Then for each number: subtract the Mean and square the result. 3. Then work out the mean of those squared differences. 4. Take the square root of that and we are done!

N or N-1? N is the number of participants, or data points. N-1 is that number minus one, and we will explain why it matters in a moment. We are going to work with a small data set so you can become more familiar with the concept. After that we will look at some more examples before moving on. Our first example is a small data set of five test scores:

80 + 70 + 90 +100 + 60

N for this data set = 5. There are five test scores.

N-1 for this data set is 5-1 = 4.

You use N for a population, and N-1 for a sample.

Most calculators and software packages divide the sum of squared differences by N − 1. This is because the standard deviation of a sample tends to be a bit lower than the standard deviation of the population the sample was selected from. Dividing the sum of squares by N − 1 corrects for this tendency and results in a better estimate of the population standard deviation. Because researchers generally think of their data as representing a sample selected from a larger population—and because they are generally interested in drawing conclusions about the population—it makes sense to routinely apply this correction.

What about for our purposes now? For now, we will use N. By definition, the standard deviation is the square root of the mean of the squared differences. This implies dividing the sum of squared differences by N, as in the formula just presented. Computing the standard deviation this way is appropriate when your goal is simply to describe the variability in your sample. And learning it this way emphasizes that the variance is in fact the mean of the squared differences—and the standard deviation is the square root of this mean.

Simple Standard Deviation Problem: Step by Step We will now get started computing the standard deviation for our small data set. First, we will compute the mean. As you will see the computation requires rather simple math with nothing more complicated than squaring numbers and square roots, but there are several steps. We start by calculating the mean from our small data set:

Step 1. To get the mean we add up all the scores: 80 + 70 + 90 +100 + 60 = 400. Next, we divide 400 by the total number of scores: 400/5 = 80. The mean test score, or the “average,” is 80. Now we calculate each score’s difference from the mean. How much is it above, or below, the mean? Step 2: To do that we subtract each number from the mean of 80 and then square the result:

RESEARCH FOR HUMAN SERVICES pg. 155

80-80 = 0 and if we square the result we get: 0 70-80 = -10 and if we square the result we get: 100 90-80 = 10 and if we square the result we get: 100 100-80 = 20 and if we square the result we get: 400 60-80 = -20 and if we square the result we get: 400 Now we get the mean of the squared differences. Step 3. We add them all up: 0+100+100+400+400=1000 Then divide the sum by the total number of scores which is 1000 divided by 5, and that gives us 200. Just one more step left. Step 4. We take the square root of our last number which is 200. The square root of 200 = 14.14, and that is our final number. The standard deviation in our first example is 14.14 For the sake of thoroughness, the actual mathematical formula for the standard deviation pictured below:

The Standard Deviation in More Detail The standard deviations of the distributions below in Figure 12.4 are 1.69 for the top distribution and 4.30 for the bottom one. That is, while the scores in the top distribution differ from the mean by about 1.69 units on average, the scores in the bottom distribution differ from the mean by about 4.30 units on average.

For these examples, the computations for the standard deviation are illustrated for a small set of data in Table 9.3 "Computations for the Standard Deviation". The first column is a set of eight scores that has a mean of 5. The second column is the difference between each score and the mean. The third column is the square of each of these differences. Notice that although the differences can be negative, the squared differences are always positive—meaning that the standard deviation is always positive. At the bottom of the third column is the mean of the squared differences, which is also called the variance (symbolized SD2). Although the variance is itself a measure of variability, it generally plays a larger role in inferential statistics than in descriptive statistics. Finally, below the variance is the square root of the variance, which is the standard deviation.

TABLE 9.3 COMPUTATIONS FOR THE STANDARD DEVIATION

X X – M (X − M)2

3 −2 4

5 0 0

4 −1 1

2 −3 9

7 2 4

6 1 1

5 0 0

8 3 9

M = 5 SD2=288=3.50

SD=3.50−−−√=1.87

Percentile Ranks and z Scores Distribution of data such as where a client fits along a continuum, can be very important. We use a variety of scales and measures for depression, anxiety, symptoms of PTSD, and many other things which are of

RESEARCH FOR HUMAN SERVICES pg. 156

interest to us as professionals working with people. We are going to continue looking at ways of describing distributions.

In many situations, it is useful to have a way to describe the location of an individual score within its distribution. One approach is the percentile rank.

Percentile Rank The percentile rank of a score is the percentage of scores in the distribution that are lower than that score. “The percentile rank of a score is the percentage of scores in its frequency distribution that are equal to or lower than it.”5 For example, a test score that is greater than 75% of the scores of people taking the test is said to be at the 75th percentile, where 75 is the percentile rank. The 75th percentile is also called the third quartile is you will see in the cart of commonly used terms below

Commonly Used Terms for Specific Percentiles The 25th percentile is also called the first quartile. The 50th percentile is generally the median The 75th percentile is also called the third quartile. The difference between the third and first quartiles is the interquartile range.

Some Examples New parents often fret about where their baby is on the height and weight scales. Parents with infants in the lower percentile tend to worry. “If your 3-month-old daughter is in the 40th percentile for weight, that means 40 percent of 3-month-old girls weigh the same as or less than your baby, and 60 percent weigh more. The higher the percentile number, the bigger your baby is compared to other babies her same age.”6 You might also want to look at the distribution in Table 12.1

How to find the Percentile Rank

Notice that five of the students represented by the data in Table 12.1 had self-esteem scores of 23. In this distribution, 32 of the 40 scores (80%) are lower than 23. Thus, each of these students has a percentile rank of 80. (It can also be said that they scored “at the 80th percentile.”) Percentile ranks are often used to report the results of standardized tests of ability or achievement.

In another example, if your percentile rank on a test of verbal ability were 40 this would mean that you scored higher than 40% of the people who took the test. How do we get the percentile rank in the first place?

How to Solve for Percentile Rank

We will bring down the example from earlier when we covered frequency distribution tables:

All the scores, in no particular order, were: 80 + 70 + 90 +100 + 60. For this example, we will say that your score was 80. You want to know how you did compared to the rest of the class.

We have to order the data first. Ordering, organizing, the data is important and should be one of the first steps you take after data has been collected. The convention is to go from lowest to highest.

Step 1: sort from smallest to highest: 60, 70, 80, 90, 100. In order to keep track, and for clarity, your score is now in bold.

5 https://en.wikipedia.org/wiki/Percentile_rank

6 https://www.babycenter.com/0_growth-charts-understanding-the-results_5251.bc

RESEARCH FOR HUMAN SERVICES pg. 157

Step 2: What is your rank out of the whole class? Your number 3 out of 5.

Step 2: We will use the following formula where i is your place, number three, and n is the total number of scores in the data set (100 (i- 0.5)).

Step 3: Plugging in the numbers the formula is (100(3-.0.5)) we solve and get 2.95.

We now divide that by n, which in this case is the total number of scores in the data set which is five. The solved equation is 2.95/5 which equals 0.59, or 59 percent. We can safely round that to .60. You did better than 40% of the class.

So now you wonder how much better your friend, who made 100, did. The data is already ordered. This time your friends score is in bold:

60, 70, 80, 90, 100

i is now their score and they are the 5th person. Of course, n stays the same.

The formula is now (100(5-0.5)). We solve and the answer is 4.5. We divide 4.5 by 5 and get 90. Your friend’s score of 100 is in the 90th percentile. That means that their grade was better than 90% of the other students, making it in the top 10%.

Let’s do it one more time for one more classmate. This time your arch enemy who made 90 on the test.

This time we will put it into the formula from the start. So, we have (100(4-0.5)) /5 which equals .7 or 70 percent.

Z Score Another approach is the z score. The z score for a particular individual is the difference between that individual’s raw score and the mean of the distribution, divided by the standard deviation of the distribution:

z score = raw score - mean / standard deviation

A z score indicates how far above or below the mean a raw score is, but it expresses this in terms of the standard deviation. For example, in a distribution of intelligence quotient (IQ) scores with a mean of 100 and a standard deviation of 15, an IQ score of 110 would have a z score of (110 − 100) / 15 = +0.67. In other words, a score of 110 is 0.67 standard deviations (approximately two thirds of a standard deviation) above the mean. Similarly, a raw score of 85 would have a z score of (85 − 100) / 15 = −1.00. In other words, a score of 85 is one standard deviation below the mean.

There are several reasons that z scores are important. Again, they provide a way of describing where an individual’s score is located within a distribution and are sometimes used to report the results of standardized tests. They also provide one way of defining outliers. For example, outliers are sometimes defined as scores that have z scores less than −3.00 or greater than +3.00. In other words, they are defined as scores that are more than three standard deviations from the mean. Finally, z scores play an important role in understanding and computing other statistics.

Online Descriptive Statistics In the last chapter, we discussed using open source statistical software such as PSPP which is freely available on Linux, Windows machines, and Macs. We also described how many researchers use commercially available software such as SPSS and Excel to analyze their data. There are several free online analysis tools that can also be extremely useful. Many allow you to enter or upload your data and then make one click to conduct several descriptive statistical analyses. Among them are the following.

RESEARCH FOR HUMAN SERVICES pg. 158

 Rice Virtual Lab in Statistics http://onlinestatbook.com/stat_analysis/index.html

 VassarStats http://faculty.vassar.edu/lowry/VassarStats.html

 Bright Stat http://www.brightstat.com For a more complete list, see http://statpages.org/index.html. Y T AK E AW AYS

Use of Spreadsheets for Data Management We have already discussed using spreadsheets. Spreadsheets, data bases, and statistical software serve

different purposes, but there is often some overlap. For example, Excel can be used for all three purposes

– at least on a limited scale. Excel started life as a spreadsheet and has evolved from there. The examples

used in this book are all small. In real life they can become huge, and complex. PSPP, an open source

statistical program mentioned before, is not a spreadsheet, but rather imports data from a spreadsheet for

manipulation. Data management is a science all to itself, and it is recommended that students learn how to

use spreadsheets. Before Excel, and its competitors, there was VisiCalc.

“VisiCalc (for "visible calculator") was the first spreadsheet

computer program for personal computers, originally released

for the Apple II by VisiCorp. It is often considered the

application that turned the microcomputer from a hobby for

computer enthusiasts into a serious business tool, prompting

IBM to introduce the IBM PC two years later. VisiCalc is

considered the Apple II's killer app. It sold over 700,000 copies

in six years, and as many as 1 million copies over its history.”7

SCREEN SHOT OF VISICALC BY USER:GORTU (APPLE2HISTORY.ORG) [PUBLIC DOMAIN], VIA WIKIMEDIA COMMONS

VisCalc was nothing short of revolutionary and it was the first application for home, or personal computers,

that was actually useful. Even in the primitive days of personal computing it made the complex process of

organizing very files of data much easier. Developed and released in the late 1970s it was the precursor to

all modern spreadsheet programs.

Knowledge of how to use spreadsheets extends well beyond research. Any work with data, including

routine lists of clients, or financial information, is done on spreadsheets. Research skills, and spreadsheet

skills, are highly marketable commodities.

Summary  Every variable has a distribution—a way that the scores are distributed across the levels. The

distribution can be described using a frequency table and histogram. It can also be described in words in terms of its shape, including whether it is unimodal or bimodal, and whether it is symmetrical or skewed.

 The central tendency, or middle, of a distribution can be described precisely using three statistics— the mean, median, and mode. The mean is the sum of the scores divided by the number of scores, the median is the middle score, and the mode is the most common score.

 The variability, or spread, of a distribution can be described precisely using the range and standard deviation. The range is the difference between the highest and lowest scores, and the standard deviation is roughly the average amount by which the scores differ from the mean.

7 https://en.wikipedia.org/wiki/VisiCalc

RESEARCH FOR HUMAN SERVICES pg. 159

 The location of a score within its distribution can be described using percentile ranks or z scores. The percentile rank of a score is the percentage of scores below that score, and the z score is the difference between the score and the mean divided by the standard deviation.

OEX R U XWX ER C IS ES

Practice 1. Make a frequency table and histogram for the following data. Then write a short description of the

shape of the distribution in words.

11, 8, 9, 12, 9, 10, 12, 13, 11, 13, 12, 6, 10, 17, 13, 11, 12, 12, 14, 14

2. For the data in Exercise 1, compute the mean, median, mode, standard deviation, and range. 3. Using the data in Exercises 1 and 2, find (a) the percentile ranks for scores of 9 and 14 and (b)

the z scores for scores of 8 and 12.

9.2 Describing Statistical Relationships

LEARNING OBJECTIVES  Describe differences between groups in terms of their means and standard deviations, and in terms

of Cohen’s d.

 Describe correlations between quantitative variables in terms of Pearson’s r.

As we have seen throughout this book, most interesting research questions are about statistical relationships between variables. Recall that there is a statistical relationship between two variables when the average score on one differs systematically across the levels of the other. In this section, we revisit the two basic forms of statistical relationship introduced earlier in the book—differences between groups or conditions and relationships between quantitative variables—and we consider how to describe them in more detail.

Differences Between Groups or Conditions Differences between groups or conditions are usually described in terms of the mean and standard deviation of each group or condition. For example, Thomas Ollendick and his colleagues conducted a study in which they evaluated two one-session treatments for simple phobias in children (Ollendick et al., 2009). [1] They randomly assigned children with an intense fear (e.g., to dogs) to one of three conditions. In the exposure condition, the children actually confronted the object of their fear under the guidance of a trained therapist. In the education condition, they learned about phobias and some strategies for coping with them. In the waitlist control condition, they were waiting to receive a treatment after the study was over.

The severity of each child’s phobia was then rated on a 1-to-8 scale by a clinician who did not know which treatment the child had received. (This was one of several dependent variables.) The mean fear rating in the education condition was 4.83 with a standard deviation of 1.52, while the mean fear rating in the exposure condition was 3.47 with a standard deviation of 1.77. The mean fear rating in the control condition was 5.56 with a standard deviation of 1.21. In other words, both treatments worked, but the exposure treatment worked better than the education treatment.

As we have seen, differences between group or condition means can be presented in a bar graph like that in Figure 9.5, where the heights of the bars represent the group or condition means. We will look more closely at creating American Psychological Association (APA)-style bar graphs shortly.

RESEARCH FOR HUMAN SERVICES pg. 160

FIGURE 9.5 BAR GRAPH SHOWING MEAN CLINICIAN PHOBIA RATINGS FOR CHILDREN IN TWO TREATMENT CONDITIONS

Cohen’s d

It is also important to be able to describe the strength of a statistical relationship, which is often referred to as the effect size. The most widely used measure of effect size for differences between group or condition means is called Cohen’s d, which is the difference between the two means divided by the standard deviation:

d=M1 −M2SD

In this formula, it does not really matter which mean is M1 and which is M2. If there is a treatment group and a control group, the treatment group mean is usually M1 and the control group mean is M2. Otherwise, the larger mean is usually M1 and the smaller mean M2 so that Cohen’s d turns out to be positive. The standard deviation in this formula is usually a kind of average of the two group standard deviations called the pooled- within groups standard deviation. To compute the pooled within-groups standard deviation, add the sum of the squared differences for Group 1 to the sum of squared differences for Group 2, divide this by the sum of the two sample sizes, and then take the square root of that. Informally, however, the standard deviation of either group can be used instead.

Conceptually, Cohen’s d is the difference between the two means expressed in standard deviation units. (Notice its similarity to a z score, which expresses the difference between an individual score and a mean in standard deviation units.) A Cohen’s d of 0.50 means that the two group means differ by 0.50 standard deviations (half a standard deviation).

A Cohen’s d of 1.20 means that they differ by 1.20 standard deviations. But how should we interpret these values in terms of the strength of the relationship or the size of the difference between the means? Table 12.4 "Guidelines for Referring to Cohen’s " presents some guidelines for interpreting Cohen’s d values in psychological research (Cohen, 1992). [2] Values near 0.20 are considered small, values near 0.50 are considered medium, and values near 0.80 are considered large.

RESEARCH FOR HUMAN SERVICES pg. 161

Thus, a Cohen’s d value of 0.50 represents a medium-sized difference between two means, and a Cohen’s d value of 1.20 represents a very large difference in the context of psychological research. In the research by Ollendick and his colleagues, there was a large difference (d = 0.82) between the exposure and education conditions.

Table 9.4 Guidelines for Referring to Cohen’s d and Pearson’s r Values as “Strong,” “Medium,” or “Weak”

Relationship strength Cohen’s d Pearson’s r

Strong/large ± 0.80 ± 0.50

Medium ± 0.50 ± 0.30 Weak/small ± 0.20 ± 0.10

Cohen’s d is useful because it has the same meaning regardless of the variable being compared or the scale it was measured on. A Cohen’s d of 0.20 means that the two group means differ by 0.20 standard deviations whether we are talking about scores on the Rosenberg Self-Esteem scale, reaction time measured in milliseconds, number of siblings, or diastolic blood pressure measured in millimeters of mercury. Not only does this make it easier for researchers to communicate with each other about their results, it also makes it possible to combine and compare results across different studies using different measures.

Be aware that the term effect size can be misleading because it suggests a causal relationship—that the difference between the two means is an “effect” of being in one group or condition as opposed to another. Imagine, for example, a study showing that a group of exercisers is happier on average than a group of nonexercisers, with an “effect size” of d = 0.35. If the study was an experiment—with participants randomly assigned to exercise and no-exercise conditions—then one could conclude that exercising caused a small to medium-sized increase in happiness. If the study was correlational, however, then one could conclude only that the exercisers were happier than the nonexercisers by a small to medium-sized amount. In other words, simply calling the difference an “effect size” does not make the relationship a causal one.

Sex Differences Expressed as Cohen’s d What are the differences between the genders? Researcher Janet Shibley Hyde has looked at the results of numerous studies on psychological sex differences and expressed the results in terms of Cohen’s d (Hyde, 2007). [3] Following are a few of the values she has found, averaging across several studies in each case. (Note that because she always treats the mean for men as M1 and the mean for women as M2, positive values indicate that men score higher and negative values indicate that women score higher.)

Mathematical problem solving +0.08

Reading comprehension −0.09

Smiling −0.40

Aggression +0.50 Attitudes toward casual sex +0.81

Leadership effectiveness −0.02

Hyde points out that although men and women differ by a large amount on some variables (e.g., attitudes toward casual sex), they differ by only a small amount on the vast majority of the other items. In many cases, Cohen’s d is less than 0.10, which she terms a “trivial” difference. (The difference in talkativeness discussed in Chapter 1 was also trivial: d = 0.06.) Although researchers and nonresearchers alike often emphasize sex differences, Hyde has argued that it makes at least as much sense to think of men and women as fundamentally similar. She refers to this as the “gender similarities hypothesis.”

Correlations Between Quantitative Variables As we have seen throughout the book, many interesting statistical relationships take the form of correlations between quantitative variables. For example, researchers Kurt Carlson and Jacqueline Conard conducted a study on the relationship between the alphabetical position of the first letter of people’s last

RESEARCH FOR HUMAN SERVICES pg. 162

names (from A = 1 to Z = 26) and how quickly those people responded to consumer appeals (Carlson & Conard, 2011). [4] In one study, they sent e-mails to a large group of MBA students, offering free basketball tickets from a limited supply. The result was that the further toward the end of the alphabet students’ last names were, the faster they tended to respond. These results are summarized in Figure 12.7.

FIGURE 9.7 LINE GRAPH SHOWING THE RELATIONSHIP BETWEEN THE ALPHABETICAL POSITION OF PEOPLE’S LAST NAMES AND HOW QUICKLY THOSE PEOPLE RESPOND TO OFFERS OF CONSUMER GOODS

Such relationships are often presented using line graphs or scatterplots, which show how the level of one variable differs across the range of the other. In the line graph in Figure 9.7 "Line Graph Showing the Relationship Between the Alphabetical Position of People’s Last Names and How Quickly Those People Respond to Offers of Consumer Goods", for example, each point represents the mean response time for participants with last names in the first, second, third, and fourth quartiles (or quarters) of the name distribution. It clearly shows how response time tends to decline as people’s last names get closer to the end of the alphabet. The scatterplot in Figure 9.8 "Statistical Relationship Between Several College Students’ Scores on the Rosenberg Self-Esteem Scale Given on Two Occasions a Week Apart", which is reproduced from Chapter 5 "Psychological Measurement", shows the relationship between 25 research methods students’ scores on the Rosenberg Self-Esteem Scale given on two occasions a week apart. Here the points represent individuals, and we can see that the higher students scored on the first occasion, the higher they tended to score on the second occasion. In general, line graphs are used when the variable on the x-axis has (or is organized into) a small number of distinct values, such as the four quartiles of the name distribution. Scatterplots are used when the variable on the x-axis has a large number of values, such as the different possible self-esteem scores.

RESEARCH FOR HUMAN SERVICES pg. 163

FIGURE 9.8 STATISTICAL RELATIONSHIP BETWEEN SEVERAL COLLEGE STUDENTS’ SCORES ON THE ROSENBERG SELF-ESTEEM SCALE GIVEN ON TWO OCCASIONS A WEEK APART

The data presented in Figure 9.8 "Statistical Relationship Between Several College Students’ Scores on the Rosenberg Self-Esteem Scale Given on Two Occasions a Week Apart" provide a good example of a positive relationship, in which higher scores on one variable tend to be associated with higher scores on the other (so that the points go from the lower left to the upper right of the graph). The data presented in Figure 9.7 "Line Graph Showing the Relationship Between the Alphabetical Position of People’s Last Names and How Quickly Those People Respond to Offers of Consumer Goods" provide a good example of a negative relationship, in which higher scores on one variable tend to be associated with lower scores on the other (so that the points go from the upper left to the lower right).

Both of these examples are also linear relationships, in which the points are reasonably well fit by a single straight line. Nonlinear relationships are those in which the points are better fit by a curved line. Figure 9.9 "A Hypothetical Nonlinear Relationship Between How Much Sleep People Get per Night and How Depressed They Are", for example, shows a hypothetical relationship between the amount of sleep people get per night and their level of depression. In this example, the line that best fits the points is a curve—a kind of upside down “U”—because people who get about eight hours of sleep tend to be the least depressed, while those who get too little sleep and those who get too much sleep tend to be more depressed. Nonlinear relationships are not uncommon, but a detailed discussion of them is beyond the scope of this book.

RESEARCH FOR HUMAN SERVICES pg. 164

FIGURE 9.9 A HYPOTHETICAL NONLINEAR RELATIONSHIP BETWEEN HOW MUCH SLEEP PEOPLE GET PER NIGHT AND HOW DEPRESSED THEY ARE

As we saw earlier in the book, the strength of a correlation between quantitative variables is typically measured using a statistic called Pearson’s r.

As Figure 9.10 "Pearson’s " shows, its possible values range from −1.00, through zero, to +1.00. A value of 0 means there is no relationship between the two variables. In addition to his guidelines for interpreting Cohen’s d, Cohen offered guidelines for interpreting Pearson’s r in psychological research (see Table 12.4 "Guidelines for Referring to Cohen’s "). Values near ±.10 are considered small, values near ± .30 are considered medium, and values near ±.50 are considered large. Notice that the sign of Pearson’s r is unrelated to its strength. Pearson’s r values of +.30 and −.30, for example, are equally strong; it is just that one represents a moderate positive relationship and the other a moderate negative relationship. Like Cohen’s d, Pearson’s r is also referred to as a measure of “effect size” even though the relationship may not be a causal one.

FIGURE 9.10 PEARSON’S R RANGES FROM −1.00 (REPRESENTING THE STRONGEST POSSIBLE NEGATIVE RELATIONSHIP), THROUGH 0 (REPRESENTING NO RELATIONSHIP), TO +1.00 (REPRESENTING THE STRONGEST POSSIBLE POSITIVE RELATIONSHIP)

The computations for Pearson’s r are more complicated than those for Cohen’s d. Although you may never have to do them by hand, it is still instructive to see how. Computationally, Pearson’s r is the “mean cross- product of z scores.” To compute it, one starts by transforming all the scores to z scores. For the X

RESEARCH FOR HUMAN SERVICES pg. 165

variable, subtract the mean of X from each score and divide each difference by the standard deviation of X. For the Y variable, subtract the mean of Y from each score and divide each difference by the standard deviation of Y. Then, for each individual, multiply the two z scores together to form a cross-product. Finally, take the mean of the cross-products. The formula looks like this:

𝑟 = ∑(𝑧𝑥𝑧𝑦)

𝑁

Table 9.5 "Sample Computations for Pearson’s " illustrates these computations for a small set of data. The first column lists the scores for the X variable, which has a mean of 4.00 and a standard deviation of 1.90. The second column is the z-score for each of these raw scores. The third and fourth columns list the raw scores for the Y variable, which has a mean of 40 and a standard deviation of 11.78, and the corresponding z scores. The fifth column lists the cross-products. For example, the first one is 0.00 multiplied by −0.85, which is equal to 0.00. The second is 1.58 multiplied by 1.19, which is equal to 1.88. The mean of these cross-products, shown at the bottom of that column, is Pearson’s r, which in this case is +.53. There are other formulas for computing Pearson’s r by hand that may be quicker. This approach, however, is much clearer in terms of communicating conceptually what Pearson’s r is.

Table 9.5 Sample Computations for Pearson’s r

X zx Y zy zxzy

4 0.00 30 −0.85 0.00

7 1.58 54 1.19 1.88

2 −1.05 23 −1.44 1.52 5 0.53 43 0.26 0.13

2 −1.05 50 0.85 −0.89

Mx = 4.00 My = 40.00 r = 0.53

SDx = 1.90 SDy = 11.78

There are two common situations in which the value of Pearson’s r can be misleading.

Nonlinear Relationship One is when the relationship under study is nonlinear. Even though Figure 9.9 "A Hypothetical Nonlinear Relationship Between How Much Sleep People Get per Night and How Depressed They Are" shows a fairly strong relationship between depression and sleep, Pearson’s r would be close to zero because the points in the scatterplot are not well fit by a single straight line. This means that it is important to make a scatterplot and confirm that a relationship is approximately linear before using Pearson’s r.

Limited or Restricted Range The other condition is when one or both of the variables have a limited range in the sample relative to the population. This is referred to as restriction of range. Assume, for example, that there is a strong negative correlation between people’s age and their enjoyment of hip hop music as shown by the scatterplot in Figure 9.11 "Hypothetical Data Showing How a Strong Overall Correlation Can Appear to Be Weak When One Variable Has a Restricted Range". Pearson’s r here is −.77. However, if we were to collect data only from 18- to 24-year-olds—represented by the shaded area of Figure 9.11 "Hypothetical Data Showing How a Strong Overall Correlation Can Appear to Be Weak When One Variable Has a Restricted Range"— then the relationship would seem to be quite weak. In fact, Pearson’s r for this restricted range of ages is 0. It is a good idea, therefore, to design studies to avoid restriction of range. For example, if age is one of your primary variables, then you can plan to collect data from people of a wide range of ages. Because restriction of range is not always anticipated or easily avoidable, however, it is good practice to examine your data for possible restriction of range and to interpret Pearson’s r in light of it. (There are also statistical methods to correct Pearson’s r for restriction of range, but they are beyond the scope of this book).

RESEARCH FOR HUMAN SERVICES pg. 166

FIGURE 9.11 HYPOTHETICAL DATA SHOWING HOW A STRONG OVERALL CORRELATION CAN APPEAR TO BE WEAK WHEN ONE VARIABLE HAS A RESTRICTED RANGE

The overall correlation here is −.77, but the correlation for the 18- to 24-year-olds (in the blue box) is 0.E

SUM AR Y T AK E AW AYS

Summary:

 Differences between groups or conditions are typically described in terms of the means and standard deviations of the groups or conditions or in terms of Cohen’s d and are presented in bar graphs.

 Cohen’s d is a measure of relationship strength (or effect size) for differences between two group or condition means. It is the difference of the means divided by the standard deviation. In general, values of ±0.20, ±0.50, and ±0.80 can be considered small, medium, and large, respectively.

 Correlations between quantitative variables are typically described in terms of Pearson’s r and presented in line graphs or scatterplots.

 Pearson’s r is a measure of relationship strength (or effect size) for relationships between quantitative variables. It is the mean cross-product of the two sets of z scores. In general, values of ±.10, ±.30, and ±.50 can be considered small, medium, and large, respectively.

Practice:

1. The following data represent scores on the Rosenberg Self-Esteem Scale for a sample of 10 Japanese college students and 10 American college students. (Although hypothetical, these data are consistent with empirical findings [Schmitt & Allik, 2005]. [5]) Compute the means and standard deviations of the two groups, make a bar graph, compute Cohen’s d, and describe the strength of the relationship in words.

Japan United States

25 27 20 30

24 34

28 37

30 26

32 24

RESEARCH FOR HUMAN SERVICES pg. 167

21 28

24 35

20 33

26 36

2. The hypothetical data that follow are extroversion scores and the number of Facebook friends for 15 college students. Make a scatterplot for these data, compute Pearson’s r, and describe the relationship in words.

Extroversion Facebook Friends 8 75

10 315

4 28

6 214

12 176

14 95

10 120

11 150

4 32

13 250 5 99

7 136

8 185

11 88 10 144

[1] Ollendick, T. H., Öst, L.-G., Reuterskiöld, L., Costa, N., Cederlund, R., Sirbu, C.,…Jarrett, M. A. (2009). One-session treatments of specific phobias in youth: A randomized clinical trial in the United States and Sweden. Journal of Consulting and Clinical Psychology, 77, 504–516. [2] Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159. [3] Hyde, J. S. (2007). New directions in the study of gender similarities and differences. Current Directions in Psychological Science, 16, 259–263. [4] Carlson, K. A., & Conard, J. M. (2011). The last name effect: How last name influences acquisition timing. Journal of Consumer Research. doi: 10.1086/658470 [5] Schmitt, D. P., & Allik, J. (2005). Simultaneous administration of the Rosenberg Self-Esteem Scale in 53 nations: Exploring the universal and culture-specific features of global self-esteem. Journal of Personality and Social Psychology, 89, 623–642.

RESEARCH FOR HUMAN SERVICES pg. 168

Chapter 10: Single-Subject Research Designs

Change is the end result of all true learning.

-Leo Buscaglia

LEARNING OBJECTIVES  Describe the basic elements of a single-subject research design.

 Design simple single-subject studies using reversal and multiple-baseline designs.

 Explain how single-subject research designs address the issue of internal validity.

 Interpret the results of simple single-subject studies based on the visual inspection of graphed data.

The Single-Subject Design The single-subject design is primarily an empirically based clinical tool used with a client system (usually an

individual) to assess the effeteness of an intervention, or treatment, on changing a targeted behavior. It is a

process that we can use to inform us in how to help clients learn more adaptive behavior. The single

subject design, also called single-system or single-case research, is a methodology most often used in the

applied fields of psychology, education, social work, and counseling.

The subject serves as their own control. It is primarily used to track something that can be counted,

measured, and observed such as behaviors. It is useful in facilitating change by replacing supposition, and

guess work, with accurate record keeping that allows for analysis. Its forte is in evaluating and assessing a

specific intervention for a specific person, and adjusting the intervention as needed, rather than for strictly

scientific purposes. Because of its individualistic focus, and lack of a control, it is not considered

generalizable.

Single-Subject Design Conventions Before looking at specific single-subject research designs, it will be helpful to consider some features that are common to most of them.

Best practice is to have some kind of baseline to measure the current level of the target behavior that we either want to increase or decrease. Often times the process of measuring alone can help the client, and others, become more aware of what is actually happening, how often, and when. When focusing on a particularly egregious negative behavior it is common to overestimate how often it occurs, while underestimating how often positive behaviors are occurring. Neurologically our brains are more apt to notice negative interactions and occurrences than positive ones. Therefore the baseline helps provides a more accurate idea of how often the target behavior is occurring.

Some behaviors we want to increase, others to decrease. For example, for a student who is having academic problems we might want to increase time spent studying. Or, for the same student, we might want to decrease the time spent playing video games. Maybe both, but it is not good to make these interventions too complicated. In this scenario the student may underestimate how often, and how long, they are playing video games while the parents overestimate it. The parents may say something like “they are always playing video games.” Obviously nobody is “always” playing video games. The target behavior is plotted over time on a graph. It puts a number on what has otherwise been conjecture.

RESEARCH FOR HUMAN SERVICES pg. 169

The X and Y Axis The dependent variable (represented on the y-axis of the graph) is measured repeatedly over time (represented by the x-axis) at regular intervals. The dependent variable is the target behavior. The y-axis is time.

FIGURE 4. X AND Y AXIS

The single-subject design is divided into distinct phases. The first design we will look at has two phases.

AB Design The simple, and versatile, AB design is the most commonly used variant, and widely used in clinical practice. If you have only one baseline phase, and one treatment phase, it is an AB design.

The Baseline: A

In the first phase, A, the behavior is monitored and recorded, usually for a predetermined amount of time, with no intervention. It is often tempting to skip this step, but unless there are clinical or practical reasons to skip this phase it is best to have an actual count rather than an estimate.

The behavior we are monitoring is the dependent variable. Normally we want one of two things to happen. We want a positive behavior to increase, or a negative one to decrease. In a clinical sitting the A phase usually lasts one week, but as we will see in an example later in the chapter it can be shortened if circumstances warrant. It can also be lengthened, but a baseline of one week is common.

The Treatment Phase: B

Phase B begins with the introduction of the treatment. The application of the independent variable. There may be a period of adjustment to the treatment during which the behavior of interest becomes more erratic, or begins to increase or decrease. Treatment usually consists of some reward, or a combination of a reward and consequence. As you can see, the single subject design is very behaviorally oriented.

The ABA Design The next most commonly used design is the ABA design. It provides us with three distinct phases rather than two. They include the baseline phase, the intervention phase where the behavior is monitored during the intervention, and then the last phase where the behavior is monitored after the intervention is withdrawn.

RESEARCH FOR HUMAN SERVICES pg. 170

Figure 10.3 below shows a behavior that was measured across three discrete phases:

1. In the first phase (A) the behavior was measured but there was no intervention. 2. In the next phase (B) the behavior was measured during the intervention. 3. In the last phase (A), the behavior was measured again after the intervention was stopped.

This is also called a reversal design.

FIGURE 10.3 A GENERIC SINGLE-SUBJECT REVERSAL DESIGN (A, THEN B, AND THEN A AGAIN)

Remember, it has two A phases separated by a single treatment phase B. The last A phase is the reverse

phase where we revert to tracking the behavior after treatment without an intervention.

ABAB Design The study by Hall and his colleagues, shown in Figure 10.4 was an ABAB reversal design. The percentage of time he spent studying (the dependent variable) was low during the first baseline phase, increased during the first treatment phase until it leveled off, decreased during the second baseline phase, and again increased during the second treatment phase.

FIGURE 10.4 AN APPROXIMATION OF THE RESULTS FOR HALL AND COLLEAGUES’ PARTICIPANT ROBBIE IN THEIR ABAB REVERSAL DESIGN

RESEARCH FOR HUMAN SERVICES pg. 171

Multiple-Treatment Reversal Design There are close relatives of the basic reversal design that allow for the evaluation of more than one treatment. In a multiple-treatment reversal design, a baseline phase is followed by separate phases in which different treatments are introduced.

For example, a researcher might establish a baseline of studying behavior for a disruptive student (A), then introduce a treatment involving positive attention from the teacher (B), and then switch to a treatment involving mild punishment for not studying (C). The participant could then be returned to a baseline phase before reintroducing each treatment—perhaps in the reverse order as a way of controlling for carryover effects. This particular multiple-treatment reversal design could also be referred to as an ABCACB design.

Potential Problems with the Reversal Design Reversal designs, such as the ABA design, are frequently used for very practical reasons. We can see if

the intervention helped or not. However, there are two potential problems with the reversal design—both of

which have to do with the removal of the treatment.

Ethical Issues with the Reversal Designs

The first is that if a treatment is working, it may be unethical to remove it. For example, if a treatment seemed to reduce the incidence of self-injury in a developmentally disabled child, it would be unethical to remove that treatment just to show that the incidence of self-injury increases.

What really caused a change in Behavior?

The second potential problem is that the dependent variable may not return to baseline when the treatment is removed. For example, when positive attention for studying is removed, a student might continue to study at an increased rate. This could mean that the positive attention had a lasting effect on the student’s studying, which of course would be good. But it could also mean that the positive attention was not really the cause of the increased studying in the first place. Perhaps something else happened at about the same time as the treatment—for example, the student’s parents might have started rewarding him for good grades. We are never sure, with single-subject designs, that the intervention caused the behavior.

Multiple Baseline Design One solution to these problems is to use a multiple-baseline design, which is represented in Figure 10.5 ". In one version of the design, a baseline is established for each of several participants, and the treatment is then introduced for each one. In essence, each participant is tested in an AB design.

The key to this design is that the treatment is introduced at a different time for each participant. The idea is that if the dependent variable changes when the treatment is introduced for one participant, it might be a coincidence. But if the dependent variable changes when the treatment is introduced for multiple participants—especially when the treatment is introduced at different times for the different participants— then it is extremely unlikely to be a coincidence.

RESEARCH FOR HUMAN SERVICES pg. 172

FIGURE 10.5 RESULTS OF A GENERIC MULTIPLE-BASELINE STUDY

The multiple baselines can be for different participants, dependent variables, or settings. The treatment is introduced at a different time on each baseline.

Other Single-Subject Designs There is a veritable alphabet soup of single-subject designs. Many more than were discussed in this

chapter. However, the simple AB design is the most practical, and one most often used in actual practice.

Data Analysis in Single-Subject Research In addition to its focus on individual participants, single-subject research differs from group research in the way the data are analyzed. Rather can calculations, or sophisticated statistics, the chart will tell the story.

Visual Inspection Single-subject relies heavily on a very different approach from other research methods called visual inspection. This means plotting individual participants’ data as has been illustrated throughout this chapter. A graph, or table, is used to plot the data. We analyze the results by looking carefully at those data, and making judgments about whether and to what extent the independent variable had an effect on the dependent variable. Inferential statistics are typically not used.

Level, Trend, and Latency

In visually inspecting their data, single-subject researchers take several factors into account.

Level

One of them is changes in the level of the dependent variable from condition to condition. How high up, or down, on the Y axis is each data point? If the dependent variable is much higher or much lower in one condition than another, this suggests that the treatment had an effect.

Trend

Is the behavior moving up, down, or staying pretty stable? The trend, refers to gradual increases or decreases in the dependent variable across observations. If the dependent variable begins increasing or decreasing with a change in conditions, then again this suggests that the treatment had an effect. It can be especially telling when a trend changes directions—for example, when an unwanted behavior is increasing during baseline but then begins to decrease with the introduction of the treatment.

RESEARCH FOR HUMAN SERVICES pg. 173

Latency

The third factor is latency. Latency is the time it takes for the dependent variable to begin changing after a change in conditions. In general, if a change in the dependent variable begins shortly after a change in conditions, this suggests that the treatment was responsible.

Statistical Analysis The results of single-subject research can also be analyzed using statistical procedures. This is becoming more common in some settings, but is not terribly useful in clinical situations where visual analysis is usually more than adequate.

Christopher: A Case History The single-subject design is an outstanding tool for clinical practice. The following case illustrate how it can be used to guide and instruct practice when working with one client. This case also shows how flexible the design can be out in the field working with real clients.

Christopher was a 14-year old boy in residential care. While most of the residents were able to attend public school, he was one of the few who had to stay back on the campus for classes because of problems with his behavior. Particularly impulsivity. Unfortunately, his behavior gradually got worse even in that restricted environment, and he was disrupting the whole on campus school routine which only had one teacher. Christopher’s social worker at the agency met with the teacher to talk about what was going on. At first, she spoke in broad generalities about impossible and consistent misbehavior. This is not particularly helpful in a clinical situation because it doesn’t give you any place to start. She was asked to name the top three behaviors that she would like to stop. After a pause, she said that she could live with everything else if only one thing would change. The teacher said that if Christopher would stay in his seat and stop wondering around, and leaving the classroom, it would improve things immensely.

So a specific target behavior was identified. The next thing to do was to determine a schedule of reward and consequences. What would be the consequences for wondering, and what would be a reward for not? Christopher was very social, and liked to come down to the office after school to visit the staff, and the director. It was determined that the reward for staying in his seat would be being able to visit the office right after school if he only got out of his seat twice, or less, during the day. If Christopher got out of his seat three or more times during the day there would be no visit. If he left the classroom altogether there would be no visit. It was as simple as that.

The next step was to establish a baseline, but there was a problem. The social worker could not stay in the classroom over a period of day for hours, and the teacher was too busy to keep track of his behavior. The social worker met with the house parents and secured their agreement to send a house parent to the school to count, and chart, the number of times that Christopher got out of his seat. The following intervention evolved.

The classic baseline for a single-subject design is one week. It was decided to shorten it, and have the house parent sit in the classroom on the upcoming Thursday and Friday to establish the baseline, and then begin the actual intervention the next week on Monday. Christopher would not be told of the reward until Monday. It was important that the extent of his problematic behavior was established first. The house parents were as busy as everyone else, and this truncated plan would minimize their need to have one of their three staff away from the cottage during the day when they would normally have been busy with other routine chores

Per the plan for the first week on Thursday and Friday the house parent sat in the classroom and dutifully charted Christopher’s behavior. Christopher only got out of his seat three times on Thursday, and only twice on Tuesday. It was not a lot of data but it turned out to be enough. Going on only two days’ worth of data, the reward schedule was put into effect and Christopher was told about it. That week he only had two days when he tested the rules, but otherwise he was able to visit the office three out of the five days. The teacher saw this as a vast improvement, and was more than satisfied with the results. The intervention was successful.

RESEARCH FOR HUMAN SERVICES pg. 174

There were two unintended consequence to the intervention. The first was that the teachers and the house parents, who had previously thought Christopher was incorrigible, learned from the actual data that was gathered that his behavior had not been as bad as everyone had thought. The visual of the chart was irrefutable.

Not all interventions using single-subject design work out as well as this one, but this case does illustrate the power of the concept. The extent of Christopher’s behavior before had been amorphous, and was viewed as being totally out of control, when in reality it was not as bad as everyone had thought. Everyone had overestimated how bad the situation was. It was also clear from the data that had been kept on his behavior that things had vastly improved.

Case History Discussion

Our case history was an AB design. It shows the flexibility of this tool, and the importance of actually

recording the target behaviors. In the example, it worked. However, it is not uncommon for an intervention

not to work and to need adjusting. The single-subject design is useful in that it lets us know what works

and what does not. If the intervention does not work we at least know that. That itself is very valuable

information we can use to revise our approach.

Summary

Single-subject design is used in many settings but has its roots in clinical practice. It is not terribly reliable

or valid from a scientific point of view, and not generalizable. It was not designed to be. The situation, and

conditions, outlined in the case history were very specific to the setting, and the client. Instead, it is an

empirically based tool for working with one particular client in a very personalized way. It allows for a more

methodical method of measuring behaviors (the dependent variable) before the intervention (the

independent variable), and then what happens after the intervention is applied. Keep in mind that often the

target behaviors are over, or underestimated. A formal way of monitoring progress, or lack of it, is also

another strong feature.

Practice

1. Design a simple single-subject study (using either a reversal or multiple-baseline design) to answer

the following questions. Be sure to specify the treatment, operationally define the dependent variable, and decide when and where the observations will be made, and so on.

Does positive attention from a parent increase a child’s tooth brushing behavior?

Does self-testing while studying improve a student’s performance on weekly spelling tests?

Does regular exercise help relieve depression?

2. Create a graph that displays the hypothetical results for the study you designed in Exercise 1. Write

a paragraph in which you describe what the results show. Be sure to comment on level, trend, and latency.

[1] Sidman, M. (1960). Tactics of scientific research: Evaluating experimental data in psychology. Boston, MA: Authors Cooperative.

RESEARCH FOR HUMAN SERVICES pg. 175

[2] Ross, S. W., & Horner, R. H. (2009). Bully prevention in positive behavior support. Journal of Applied Behavior Analysis, 42, 747–759.

[3] Fisch, G. S. (2001). Evaluating data from behavioral analysis: Visual inspection or statistical models. Behavioural Processes, 54, 137–154.

[4] Scruggs, T. E., & Mastropieri, M. A. (2001). How to summarize single-participant research: Ideas and applications. Exceptionality, 9, 227–244.