Psychological Constructivism

This page intentionally left blank

This page intentionally left blank

How to Design and Evaluate

Research in Education E I G H T H E D I T I O N

Jack R. Fraenkel San Francisco State University

Norman E. Wallen San Francisco State University

Helen H. Hyun San Francisco State University

TM

fra97851_fm_i-xxviii.indd ifra97851_fm_i-xxviii.indd i 24/12/10 2:14 AM24/12/10 2:14 AM

TM

Published by McGraw-Hill, a business unit of The McGraw-Hill Companies, Inc., 1221 Avenue of the Americas, New York, NY 10020. Copyright © 2012, 2009, 2006, 2003, 2000, 1996, 1993, 1990 by The McGraw-Hill Companies, Inc. All rights reserved. No part of this publication may be reproduced or distributed in any form or by any means, or stored in a database or retrieval system, without the prior written consent of The McGraw-Hill Companies, Inc., including, but not limited to, any network or other electronic storage or transmission, or broadcast for distance learning. Some ancillaries, including electronic and print components, may not be available to customers outside the United States.

This book is printed on acid-free paper.

1 2 3 4 5 6 7 8 9 0 QDQ/QDQ 1 0

ISBN: 978-0-07-809785-0 MHID: 0-07-809785-1

Vice President, Editorial: Michael Ryan Director, Editorial: Beth Mejia Senior Sponsoring Editor: Allison McNamara Director of Development: Dawn Groundwater Development Editor: Vicki Malinee, Van Brien & Associates Editorial Coordinator: Sarah Kiefer Marketing Manager: Julia Flohr Larkin Media Project Manager: Bethuel Jabez Production Editor: Ruth Sakata Corley Interior Designer: Amanda Kavanagh Cover Designer: Allister Fein Buyer II: Tandra Jorgensen Composition: 10/12 Times by MPS Limited, a Macmillan Company Printing: 45# New Era Matte Plus by Quad/Graphics

Cover images: © Jose Luis Pelaez, Inc./Blend Images/Corbis (Female doctor); © Image Source/ Corbis (Science teacher with students)

Library of Congress Cataloging-in-Publication Data

Fraenkel, Jack R., 1932– How to design and evaluate research in education / Jack Fraenkel, Norman Wallen, Helen, Hyun. — 8th ed. p. cm. Includes bibliographical references and index. ISBN-13: 978-0-07-809785-0 (hardback) ISBN-10: 0-07-809785-1 1. Education—Research—Methodology. 2. Education—Research—Evaluation. 3. Proposal writing in educational research. I. Wallen, Norman E. II. Hyun, Helen. III. Title. LB1028.F665 2011 370.72—dc22 2010051892

The Internet addresses listed in the text were accurate at the time of publication. The inclusion of a website does not indicate an endorsement by the authors or McGraw-Hill, and McGraw-Hill does not guarantee the accuracy of the information presented at these sites.

www.mhhe.com

fra97851_fm_i-xxviii.indd iifra97851_fm_i-xxviii.indd ii 12/31/10 4:11 PM12/31/10 4:11 PM

iii

ABOUT THE AUTHORS

Jack R. Fraenkel is currently Professor of Interdisciplinary Studies in Education, and previously Director of the Research and Development Center, College of Education, San Francisco State University. He received his Ph.D. from Stanford Univer- sity and has taught courses in research methodology for more than 30 years. In 1997, he received the James A. Michener prize in writing for his writings about the social studies and the so- cial sciences. His current work centers on advising and assist- ing faculty and students in generating and developing research endeavors.

Norman E. Wallen is Professor Emeritus of Interdisciplinary S tudies in Education at San Francisco State University, where he taught from 1966 to 1992. An experienced researcher, he received his Ph.D. from Syracuse University and taught courses in statistics and research design to master’s and doctoral students for many years. He is a former member of the City Council of Flagstaff, Arizona, and the Executive Committee, Grand Canyon Chapter of the Sierra Club.

Helen H. Hyun is currently Assistant Professor of Interdisciplin- ary Studies in Education at San Francisco State University. She received her Ed.D. from Harvard University and has taught courses in research design and methodology to master’s and doctoral stu- dents at both San Francisco State and the University of California at Berkeley. Her teaching and research interests include higher education policy, mixed methods research, and equity issues in education.

To Marge, Lina, and Jeff for all their support

fra97851_fm_i-xxviii.indd iiifra97851_fm_i-xxviii.indd iii 24/12/10 2:14 AM24/12/10 2:14 AM

iv

CONTENTS IN BRIEF

Preface xix

P A R T 1

Introduction to Research 1 1 The Nature of Research 2

P A R T 2

The Basics of Educational Research 25 2 The Research Problem 26 3 Locating and Reviewing the Literature 37 4 Ethics and Research 60 5 Variables and Hypotheses 75 6 Sampling 90 7 Instrumentation 110 8 Validity and Reliability 146 9 Internal Validity 165

P A R T 3

Data Analysis 185 10 Descriptive Statistics 186 11 Inferential Statistics 219 12 Statistics in Perspective 245

P A R T 4

Quantitative Research Methodologies 263 13 Experimental Research 264 14 Single-Subject Research 301 15 Correlational Research 330 16 Causal-Comparative Research 365 17 Survey Research 392

P A R T 5

Introduction to Qualitative Research 423 18 The Nature of Qualitative Research 424

19 Observation and Interviewing 444 20 Content Analysis 477

P A R T 6

Qualitative Research Methodologies 505 21 Ethnographic Research 506 22 Historical Research 534

P A R T 7

Mixed-Methods Studies 555 23 Mixed-Methods Research 556

P A R T 8

Research by Practitioners 587 24 Action Research 588

P A R T 9

Writing Research Proposals and Reports 615 25 Preparing Research Proposals and Reports 616

Appendixes A-1 APPENDIX A Portion of a Table of Random

Numbers A-2 APPENDIX B Selected Values from a Normal Curve

Table A-3 APPENDIX C Chi-Square Distribution A-4 APPENDIX D Using Microsoft Excel A-5

Glossary G-1 Credits C-1 Index I-1

fra97851_fm_i-xxviii.indd ivfra97851_fm_i-xxviii.indd iv 24/12/10 2:14 AM24/12/10 2:14 AM

v

CONTENTS

P A R T 1 Introduction to Research 1

1 The Nature of Research 2 Interactive and Applied Learning 3

Some Examples of Educational Concerns 3 Why Research Is of Value 4 Ways of Knowing 4 Types of Research 7 General Research Types 15

Critical Analysis of Research 16 A Brief Overview of the Research Process 19 Main Points 21 Key Terms 22 For Discussion 23 Notes 23 Research Exercise 1 24 Problem Sheet 1 24

P A R T 2 The Basics of Educational Research 25

2 The Research Problem 26 Interactive and Applied Learning 27

What Is a Research Problem? 27 Research Questions 27 Characteristics of Good Research Questions 28 Main Points 35 Key Terms 35 For Discussion 35 Research Exercise 2 36 Problem Sheet 2 36

3 Locating and Reviewing the Literature 37

Interactive and Applied Learning 38

The Defi nition and Value of a Literature Review 38 Types of Sources 38 Steps Involved in a Literature Search 39 Doing a Computer Search 45 Writing the Literature Review Report 51 Main Points 56 Key Terms 57 For Discussion 57 Notes 58 Research Exercise 3 59 Problem Sheet 3 59

Preface xix

fra97851_fm_i-xxviii.indd vfra97851_fm_i-xxviii.indd v 24/12/10 2:14 AM24/12/10 2:14 AM

vi C O N T E N T S www.mhhe.com/fraenkel8e

4 Ethics and Research 60 Interactive and Applied Learning 61

Some Examples of Unethical Practice 61 A Statement of Ethical Principles 61 Protecting Participants from Harm 63 Ensuring Confi dentiality of Research Data 64 When (If Ever) Is Deception of Subjects Justifi ed? 64 Three Examples Involving Ethical Concerns 65 Research with Children 67 Regulation of Research 69 Academic Cheating and Plagiarism 71 Main Points 72 Key Terms 73 For Discussion 73 Notes 73 Research Exercise 4 74 Problem Sheet 4 74

5 Variables and Hypotheses 75 Interactive and Applied Learning 76

The Importance of Studying Relationships 76 Variables 77 Hypotheses 83 Main Points 87 Key Terms 88 For Discussion 88 Note 88 Research Exercise 5 89 Problem Sheet 5 89

6 Sampling 90 Interactive and Applied Learning 91

What Is a Sample? 91 Random Sampling Methods 94 Nonrandom Sampling Methods 97 A Review of Sampling Methods 100 Sample Size 102 External Validity: Generalizing from a Sample 103 Main Points 106

Key Terms 107 For Discussion 107 Research Exercise 6 109 Problem Sheet 6 109

7 Instrumentation 110 Interactive and Applied Learning 111

What Are Data? 111 Means of Classifying Data-Collection Instruments 113 Examples of Data-Collection Instruments 116 Norm-Referenced Versus Criterion-Referenced

Instruments 136 Measurement Scales 137 Preparing Data for Analysis 140 Main Points 141 Key Terms 143 For Discussion 143 Notes 143 Research Exercise 7 145 Problem Sheet 7 145

8 Validity and Reliability 146 Interactive and Applied Learning 147

The Importance of Valid Instrumentation 147 Validity 147 Reliability 154 Main Points 162 Key Terms 162 For Discussion 163 Notes 163 Research Exercise 8 164 Problem Sheet 8 164

9 Internal Validity 165 Interactive and Applied Learning 166

What Is Internal Validity? 166 Threats to Internal Validity 167 How Can a Researcher Minimize These Threats

to Internal Validity? 179

fra97851_fm_i-xxviii.indd vifra97851_fm_i-xxviii.indd vi 24/12/10 2:14 AM24/12/10 2:14 AM

C O N T E N T S vii

Main Points 180 Key Terms 181 For Discussion 182

Note 182 Research Exercise 9 183 Problem Sheet 9 183

P A R T 3 Data Analysis 185

10 Descriptive Statistics 186 Interactive and Applied Learning 187

Statistics Versus Parameters 187 Two Fundamental Types of Numerical Data 187 Types of Scores 189 Techniques for Summarizing Quantitative Data 190 Techniques for Summarizing Categorical Data 209 Main Points 215 Key Terms 216 For Discussion 217 Research Exercise 10 218 Problem Sheet 10 218

11 Inferential Statistics 219 Interactive and Applied Learning 220

What Are Inferential Statistics? 220 The Logic of Inferential Statistics 221 Hypothesis Testing 228 Practical Versus Statistical Signifi cance 230 Inference Techniques 233

Main Points 241 Key Terms 243 For Discussion 243 Research Exercise 11 244 Problem Sheet 11 244

12 Statistics in Perspective 245 Interactive and Applied Learning 246

Approaches to Research 246 Comparing Groups: Quantitative Data 247 Relating Variables Within a Group:

Quantitative Data 251 Comparing Groups: Categorical Data 255 Relating Variables Within a Group:

Categorical Data 257 A Recap of Recommendations 259 Main Points 259 Key Terms 260 For Discussion 260 Research Exercise 12 261 Problem Sheet 12 261

P A R T 4 Quantitative Research Methodologies 263

13 Experimental Research 264 Interactive and Applied Learning 265

The Uniqueness of Experimental Research 265 Essential Characteristics of Experimental Research 266 Control of Extraneous Variables 268 Group Designs in Experimental Research 268 Control of Threats to Internal Validity: A Summary 279 Evaluating the Likelihood of a Threat to Internal

Validity in Experimental Studies 281

Control of Experimental Treatments 284 An Example of Experimental Research 285 Analysis of the Study 294 Main Points 296 Key Terms 297 For Discussion 298 Notes 299 Research Exercise 13 300 Problem Sheet 13 300

fra97851_fm_i-xxviii.indd viifra97851_fm_i-xxviii.indd vii 24/12/10 2:14 AM24/12/10 2:14 AM

viii C O N T E N T S www.mhhe.com/fraenkel8e

14 Single-Subject Research 301 Interactive and Applied Learning 302

Essential Characteristics of Single-Subject Research 302

Single-Subject Designs 303 Threats to Internal Validity in Single-Subject

Research 309 An Example of Single-Subject Research 314 Analysis of the Study 326 Main Points 327 Key Terms 329 For Discussion 329 Notes 329

15 Correlational Research 330 Interactive and Applied Learning 331

The Nature of Correlational Research 331 Purposes of Correlational Research 332 Basic Steps in Correlational Research 338 What Do Correlation Coeffi cients Tell Us? 340 Threats to Internal Validity in Correlational

Research 340 Evaluating Threats to Internal Validity in Correlational

Studies 344 An Example of Correlational Research 346 Analysis of the Study 360 Main Points 362 Key Terms 364 For Discussion 364 Notes 364

16 Causal-Comparative Research 365

Interactive and Applied Learning 366

What Is Causal-Comparative Research? 366

Steps Involved in Causal-Comparative Research 369

Threats to Internal Validity in Causal-Comparative Research 370

Evaluating Threats to Internal Validity in Causal- Comparative Studies 372

Data Analysis 373 Associations Between Categorical

Variables 375 An Example of Causal-Comparative

Research 376 Analysis of the Study 388 Main Points 389 For Discussion 391 Note 391

17 Survey Research 392 Interactive and Applied Learning 393

What Is a Survey? 393 Why Are Surveys Conducted? 393 Types of Surveys 394 Survey Research and Correlational Research 395 Steps in Survey Research 395 Nonresponse 405 Problems in the Instrumentation Process in Survey

Research 407 Evaluating Threats to Internal Validity

in Survey Research 407 Data Analysis in Survey Research 407 An Example of Survey Research 407 Analysis of the Study 417 Main Points 419 Key Terms 420 For Discussion 420 Notes 421

fra97851_fm_i-xxviii.indd viiifra97851_fm_i-xxviii.indd viii 24/12/10 2:14 AM24/12/10 2:14 AM

C O N T E N T S ix

18 The Nature of Qualitative Research 424

Interactive and Applied Learning 425

What Is Qualitative Research? 425 General Characteristics of Qualitative Research 426 Philosophical Assumptions Underlying Qualitative as

Opposed to Quantitative Research 427 Postmodernism 429 Steps in Qualitative Research 429 Approaches to Qualitative Research 432 Qualitative Data Analysis 436 Generalization in Qualitative Research 436 Internal Validity in Qualitative Research 438 Ethics and Qualitative Research 438 Qualitative and Quantitative Research

Reconsidered 439 Main Points 440 Key Terms 441 For Discussion 441 Notes 442

19 Observation and Interviewing 444

Interactive and Applied Learning 445

Observation 445

Interviewing 450 Validity and Reliability in Qualitative Research 458 An Example of Qualitative Research 460 Analysis of the Study 471 Main Points 473 Key Terms 474 For Discussion 475 Notes 475

20 Content Analysis 477 Interactive and Applied Learning 478

What Is Content Analysis? 478 Some Applications 479 Categorization in Content Analysis 480 Steps Involved in Content Analysis 480 An Illustration of Content Analysis 486 Using the Computer in Content Analysis 488 Advantages of Content Analysis 489 Disadvantages of Content Analysis 489 An Example of a Content Analysis Study 490 Analysis of the Study 501 Main Points 502 Key Terms 503 For Discussion 504 Notes 504

P A R T 5 Introduction to Qualitative Research 423

P A R T 6 Qualitative Research Methodologies 505

21 Ethnographic Research 506 Interactive and Applied Learning 507

What Is Ethnographic Research? 507 The Unique Value of Ethnographic Research 508 Ethnographic Concepts 509 Sampling in Ethnographic Research 511 Do Ethnographic Researchers Use Hypotheses? 511 Data Collection in Ethnographic Research 512 Data Analysis in Ethnographic Research 516

Roger Harker and His Fifth-Grade Classroom 518 Advantages and Disadvantages of Ethnographic

Research 520 An Example of Ethnographic Research 520 Analysis of the Study 530 Main Points 531 Key Terms 532 For Discussion 532 Notes 533

fra97851_fm_i-xxviii.indd ixfra97851_fm_i-xxviii.indd ix 24/12/10 2:14 AM24/12/10 2:14 AM

x C O N T E N T S www.mhhe.com/fraenkel8e

22 Historical Research 534 Interactive and Applied Learning 535

What Is Historical Research? 535 Steps Involved in Historical Research? 536 Data Analysis in Historical Research 541 Generalization in Historical Research 541 Advantages and Disadvantages of Historical

Research 542

An Example of Historical Research 543 Analysis of the Study 551 Main Points 552 Key Terms 553 For Discussion 554 Notes 554

23 Mixed-Methods Research 556 Interactive and Applied Learning 557

What Is Mixed-Methods Research? 557 Why Do Mixed-Methods Research? 558 Drawbacks of Mixed-Methods Studies 558 A (Very) Brief History 559 Types of Mixed-Methods Designs 560 Other Mixed-Methods Research Design Issues 562 Steps in Conducting a Mixed-Methods Study 563

Evaluating a Mixed-Methods Study 565 Ethics in Mixed-Methods Research 565 Summary 565 An Example of Mixed-Methods Research 565 Analysis of the Study 580 Main Points 583 Key Terms 584 For Discussion 585 Notes 585

P A R T 7 Mixed-Methods Studies 555

24 Action Research 588 Interactive and Applied Learning 589

What Is Action Research? 589 Types of Action Research 590 Steps in Action Research 592 Similarities and Differences Between Action Research

and Formal Quantitative and Qualitative Research 595

The Advantages of Action Research 596

Some Hypothetical Examples of Practical Action Research 597

An Example of Action Research 602 A Published Example of Action Research 603 Analysis of the Study 610 Main Points 611 Key Terms 612 For Discussion 612 Notes 613

P A R T 8 Research by Practitioners 587

fra97851_fm_i-xxviii.indd xfra97851_fm_i-xxviii.indd x 24/12/10 2:14 AM24/12/10 2:14 AM

C O N T E N T S xi

25 Preparing Research Proposals and Reports 616

Interactive and Applied Learning 617

The Research Proposal 617 The Major Sections of a Research Proposal or

Report 617 Sections Unique to Research Reports 624 A Sample Research Proposal 628 Main Points 640 For Review 640 Key Terms 641 For Discussion 641 Notes 641

Appendixes A-1 APPENDIX A Portion of a Table of Random

Numbers A-2 APPENDIX B Selected Values from a Normal Curve

Table A-3 APPENDIX C Chi-Square Distribution A-4 APPENDIX D Using Microsoft Excel A-5

Glossary G-1 Credits C-1 Index I-1

P A R T 9 Writing Research Proposals and Reports 615

fra97851_fm_i-xxviii.indd xifra97851_fm_i-xxviii.indd xi 24/12/10 2:14 AM24/12/10 2:14 AM

xii

ABOUT THE AUTHORS

RESEARCH REPORTS

Cognitive Effects of Chess Instruction on Students at Risk for Academic Failure 286

Progressing from Programmatic to Discovery Research: A Case Example with the Overjustifi cation Effect 315

When Teachers’ and Parents’ Values Differ: Teachers’ Ratings of Academic Competence in Children from Low-Income Families 347

Internet Use, Abuse, and Dependence Among Students at a Southeastern Regional University 376

Russian and American College Students’ Attitudes, Perceptions, and Tendencies Towards Cheating 408

Walk and Talk: An Intervention for Behaviorally Challenged Youths 460

The “Nuts and Dolts” of Teacher Images in Children’s Picture Storybooks: A Content Analysis 490

Lessons on Effective Teaching from Middle School ESL Students 521

Lydia Ann Stow: Self-Actualization in a Period of Transition 544

Perceived Family Support, Acculturation, and Life Satisfaction in Mexican American Youth: A Mixed- Methods Exploration 566

An Action Research Exploration Integrating Student Choice and Arts Activities in a Sixth Grade Social Studies Classroom 603

RESEARCH TIPS

Key Terms to Defi ne in a Research Study 31 What a Good Summary of a Journal Article Should

Contain 50 Some Tips About Developing Your Own

Instrument 114 Sample Size 234 How Not to Interview 457 What to Do About Contradictory Findings 564 Things to Consider When Doing In-School

Research 600 Questions to Ask When Evaluating a Research

Report 624

MORE ABOUT RESEARCH

Chaos Theory 8 The Importance of a Rationale 33 Patients Given Fake Blood Without Their

Knowledge 65 An Example of Unethical Research 67 Department of Health and Human Services Revised

Regulations for Research with Human Subjects 71 Some Important Relationships That Have Been

Clarifi ed by Educational Research 79 The Diffi culty in Generalizing from a Sample 103 Checking Reliability and Validity—An Example 157 Threats to Internal Validity in Everyday Life 175 Some Thoughts About Meta-Analysis 177

LIST OF FEATURES

xii

fra97851_fm_i-xxviii.indd xiifra97851_fm_i-xxviii.indd xii 24/12/10 2:14 AM24/12/10 2:14 AM

L I S T O F F E A T U R E S xiii

Correlation in Everyday Life 213 Interpreting Statistics 258 Signifi cant Findings in Experimental Research 283 Important Findings in Single-Subject Research 303 Examples of Studies Conducted Using Single-Subject

Designs 313 Important Findings in Correlational Research 333 Signifi cant Findings in Causal-Comparative

Research 375 Important Findings in Survey Research 398 Important Findings in Content Analysis

Research 481 Important Findings in Ethnographic Research 508 Important Findings in Historical Research 542 An Important Example of Action Research 597

CONTROVERSIES IN RESEARCH

Should Some Research Methods Be Preferred over Others? 15

Clinical Trials—Desirable or Not? 63 Ethical or Not? 70 Sample or Census? 102 Which Statistical Index Is Valid? 139 High-Stakes Testing 150 Is Consequential Validity a Useful Concept? 160 Can Statistical Power Analysis Be Misleading? 240 Statistical Inference Tests—Good or Bad? 249 Do Placebos Work? 281 How Should Research Methodologies Be

Classifi ed? 368 Is Low Response Rate Necessarily a Bad

Thing? 406 Clarity and Postmodernism 430 Portraiture: Art, Science, or Both? 433 Should Historians Infl uence Policy? 540 Are Some Methods Incompatible with

Others? 560 How Much Should Participants Be Involved in

Research? 593

TABLES

Table 4.1 Criteria for IRB Approval 69 Table 6.1 Part of a Table of Random Numbers 95 Table 7.1 Characteristics of the Four Types of

Measurement Scales 139 Table 7.2 Hypothetical Results of Study Involving a

Comparison of Two Counseling Methods 140 Table 8.1 Example of an Expectancy Table 153 Table 8.2 Methods of Checking Validity and

Reliability 158 Table 9.1 Threats to the Internal Validity of a

Study 179 Table 9.2 General Techniques for Controlling

Threats to Internal Validity 180 Table 10.1 Hypothetical Examples of Raw Scores

and Accompanying Percentile Ranks 190 Table 10.2 Example of a Frequency Distribution 191 Table 10.3 Example of a Grouped Frequency

Distribution 191 Table 10.4 Example of the Mode, Median, and

Mean in a Distribution 196 Table 10.5 Yearly Salaries of Workers in a Small

Business 197 Table 10.6 Calculation of the Standard Deviation

of a Distribution 199 Table 10.7 Comparisons of Raw Scores and z Scores

on Two Tests 202 Table 10.8 Data Used to Construct Scatterplot in

Figure 10.17 206 Table 10.9 Frequency and Percentage of Total of

Responses to Questionnaire 211 Table 10.10 Grade Level and Gender of Teachers

(Hypothetical Data) 212 Table 10.11 Repeat of Table 10.10 with Expected

Frequencies (in Parentheses) 212 Table 10.12 Position, Gender, and Ethnicity of

School Leaders (Hypothetical Data) 212 Table 10.13 Position and Ethnicity of School

Leaders with Expected Frequencies (Derived from Table 10.12) 213

Table 10.14 Position and Gender of School Leaders with Expected Frequencies (Derived from Table 10.12) 213

fra97851_fm_i-xxviii.indd xiiifra97851_fm_i-xxviii.indd xiii 24/12/10 2:14 AM24/12/10 2:14 AM

xiv L I S T O F F E A T U R E S www.mhhe.com/fraenkel8e

Table 10.15 Gender and Ethnicity of School Leaders with Expected Frequencies (Derived from Table 10.12) 213

Table 10.16 Total of Discrepancies Between Expected and Observed Frequencies in Tables 10.13 Through 10.15 214

Table 10.17 Crossbreak Table Showing Relationship Between Self-Esteem and Gender (Hypothetical Data) 214

Table 11.1 Contingency Coeffi cient Values for Different-Sized Crossbreak Tables 238

Table 11.2 Commonly Used Inferential Techniques 239

Table 12.1 Gain Scores on Test of Ability to Explain: Inquiry and Lecture Groups 250

Table 12.2 Calculations from Table 12.1 252 Table 12.3 Interpretation of Correlation

Coeffi cients when Testing Research Hypotheses 253

Table 12.4 Self-Esteem Scores and Gains in Marital Satisfaction 254

Table 12.5 Gender and Political Preference (Percentages) 256

Table 12.6 Gender and Political Preference (Numbers) 256

Table 12.7 Teacher Gender and Grade Level Taught: Case 1 256

Table 12.8 Teacher Gender and Grade Level Taught: Case 2 256

Table 12.9 Crossbreak Table Showing Teacher Gender and Grade Level with Expected Frequencies Added (Data from Table 12.7) 257

Table 12.10 Summary of Commonly Used Statistical Techniques 258

Table 13.1 Effectiveness of Experimental Designs in Controlling Threats to Internal Validity 280

Table 15.1 Three Sets of Data Showing Different Directions and Degrees of Correlation 332

Table 15.2 Teacher Expectation of Failure and Amount of Disruptive Behavior for a Sample of 12 Classes 334

Table 15.3 Correlation Matrix for Variables in Student Alienation Study 337

Table 15.4 Example of Data Obtained in a Correlational Design 339

Table 16.1 Grade Level and Gender of Teachers (Hypothetical Data) 375

Table 17.1 Advantages and Disadvantages of Survey Data Collection Methods 396

Table 17.2 Advantages and Disadvantages of Closed-Ended Versus Open-Ended Questions 400

Table 18.1 Quantitative Versus Qualitative Research 426

Table 18.2 Major Characteristics of Qualitative Research 428

Table 18.3 Differing Philosophical Assumptions of Quantitative and Qualitative Researchers 429

Table 19.1 Interviewing Strategies Used in Educational Research 452

Table 19.2 Qualitative Research Questions, Strategies, and Data Collection Techniques 459

Table 20.1 Coding Categories for Women in Social Studies Textbooks 483

Table 20.2 Sample Tally Sheet (Newspaper Editorials) 486

Table 20.3 Clarity of Studies 486 Table 20.4 Type of Sample 486 Table 20.5 Threats to Internal Validity 488 Table 24.1 Basic Assumptions Underlying Action

Research 590 Table 24.2 Similarities and Differences Between

Action Research and Formal Quantitative and Qualitative Research 596

Table 25.1 References APA Style 625

FIGURES

Figure 1.1 Ways of Knowing 10 Figure 1.2 Example of Results of Experimental

Research: Effect of Method of Instruction on History Test Scores 11

Figure 1.3 Is the Teacher’s Assumption Correct? 18 Figure 1.4 The Research Process 20 Figure 2.1 Researchable Versus Nonresearchable

Questions 29 Figure 2.2 Some Times When Operational

Defi nitions Would Be Helpful 32 Figure 2.3 Illustration of Relationship Between

Voter Gender and Party Affi liation 34

fra97851_fm_i-xxviii.indd xivfra97851_fm_i-xxviii.indd xiv 24/12/10 2:14 AM24/12/10 2:14 AM

L I S T O F F E A T U R E S xv

Figure 3.1 Excerpt from ERIC Journal Article 42 Figure 3.2 Excerpt from ERIC Document 43 Figure 3.3 Excerpt from Proquest Dissertations and

Theses 44 Figure 3.4 Excerpt from Education Full Text 45 Figure 3.5 Venn Diagrams Showing the Boolean

Operators AND and OR 47 Figure 3.6 Summary of Search Results 48 Figure 3.7 Sample Note Card 51 Figure 3.8 The Yahoo! Web Page 53 Figure 4.1 Example of a Consent Form 64 Figure 4.2 Examples of Unethical Research

Practices 66 Figure 4.3 Example of a Consent Form for a Minor

to Participate in a Research Study 68 Figure 5.1 Quantitative Variables Compared with

Categorical Variables 78 Figure 5.2 Relationship Between Instructional

Approach (Independent Variable) and Achievement (Dependent Variable), as Moderated by Gender of Students 81

Figure 5.3 Examples of Extraneous Variables 82 Figure 5.4 A Single Research Problem Can Suggest

Several Hypotheses 84 Figure 5.5 Directional Versus Nondirectional

Hypotheses 86 Figure 6.1 Representative Versus Nonrepresentative

Samples 93 Figure 6.2 Selecting a Stratifi ed Sample 96 Figure 6.3 Cluster Random Sampling 97 Figure 6.4 Random Sampling Methods 98 Figure 6.5 Convenience Sampling 99 Figure 6.6 Nonrandom Sampling Methods 101 Figure 6.7 Population as Opposed to Ecological

Generalizing 105 Figure 7.1 ERIC Database of Tests and

Assessments 115 Figure 7.2 Search Results for Social Studies

Competency-Based Instruments 116 Figure 7.3 Abstract from the ERIC Database 117 Figure 7.4 Excerpt from a Behavior Rating Scale for

Teachers 118 Figure 7.5 Excerpt from a Graphic Rating Scale 118

Figure 7.6 Example of a Product Rating Scale 119 Figure 7.7 Interview Schedule (for Teachers)

Designed to Assess the Effects of a Competency- Based Curriculum in Inner-City Schools 120

Figure 7.8 Semi-structured Interview Protocol 121 Figure 7.9 Sample Observation Form 121 Figure 7.10 Discussion-Analysis Tally Sheet 122 Figure 7.11 Participation Flowchart 123 Figure 7.12 Performance Checklist Noting Student

Actions 124 Figure 7.13 Time-and-Motion Log 125 Figure 7.14 Example of a Self-Checklist 126 Figure 7.15 Examples of Items from a Likert

Scale Measuring Attitude Toward Teacher Empowerment 127

Figure 7.16 Example of the Semantic Differential 128 Figure 7.17 Pictorial Attitude Scale for Use with

Young Children 128 Figure 7.18 Sample Items from a Personality

Inventory 129 Figure 7.19 Sample Items from an Achievement

Test 129 Figure 7.20 Sample Item from an Aptitude Test 130 Figure 7.21 Sample Items from an Intelligence

Test 130 Figure 7.22 Example from the Blum Sewing

Machine Test 131 Figure 7.23 Sample Items from the Picture Situation

Inventory 132 Figure 7.24 Example of a Sociogram 133 Figure 7.25 Example of a Group Play 134 Figure 7.26 Four Types of Measurement Scales 137 Figure 7.27 A Nominal Scale of Measurement 137 Figure 7.28 An Ordinal Scale: The Outcome of a

Horse Race 138 Figure 8.1 Types of Evidence of Validity 149 Figure 8.2 Reliability and Validity 155 Figure 8.3 Reliability of a Measurement 155 Figure 8.4 Standard Error of Measurement 158 Figure 8.5 The “Quick and Easy” Intelligence

Test 159 Figure 8.6 Reliability Worksheet 160

fra97851_fm_i-xxviii.indd xvfra97851_fm_i-xxviii.indd xv 24/12/10 2:14 AM24/12/10 2:14 AM

xvi L I S T O F F E A T U R E S www.mhhe.com/fraenkel8e

Figure 9.1 A Mortality Threat to Internal Validity 168

Figure 9.2 Location Might Make a Difference 169 Figure 9.3 An Example of Instrument Decay 170 Figure 9.4 A Data Collector Characteristics

Threat 170 Figure 9.5 A Testing Threat to Internal Validity 171 Figure 9.6 A History Threat to Internal Validity 172 Figure 9.7 Could Maturation Be at Work

Here? 173 Figure 9.8 The Attitude of Subjects Can Make a

Difference 174 Figure 9.9 Regression Rears Its Head 176 Figure 9.10 Illustration of Threats to Internal

Validity 178 Figure 10.1 Example of a Frequency Polygon 192 Figure 10.2 Example of a Positively Skewed

Polygon 193 Figure 10.3 Example of a Negatively Skewed

Polygon 193 Figure 10.4 Two Frequency Polygons

Compared 193 Figure 10.5 Histogram of Data in Table 10.2 194 Figure 10.6 The Normal Curve 195 Figure 10.7 Averages Can Be Misleading! 197 Figure 10.8 Different Distributions Compared with

Respect to Averages and Spreads 198 Figure 10.9 Boxplots 198 Figure 10.10 Standard Deviations for Boys’ and

Men’s Basketball Teams 200 Figure 10.11 Fifty Percent of All Scores in a Normal

Curve Fall on Each Side of the Mean 200 Figure 10.12 Percentages under the Normal

Curve 200 Figure 10.13 z Scores Associated with the Normal

Curve 201 Figure 10.14 Probabilities under the Normal

Curve 203 Figure 10.15 Table Showing Probability Areas

Between the Mean and Different z Scores 203 Figure 10.16 Examples of Standard Scores 204 Figure 10.17 Scatterplot of Data from

Table 10.8 206

Figure 10.18 Relationship Between Family Cohesiveness and School Achievement in a Hypothetical Group of Students 208

Figure 10.19 Further Examples of Scatterplots 209 Figure 10.20 A Perfect Negative Correlation! 210 Figure 10.21 Positive and Negative Correlations 210 Figure 10.22 Examples of Nonlinear (Curvilinear)

Relationships 210 Figure 10.23 Example of a Bar Graph 211 Figure 10.24 Example of a Pie Chart 212 Figure 11.1 Selection of Two Samples from Two

Distinct Populations 221 Figure 11.2 Sampling Error 222 Figure 11.3 A Sampling Distribution of Means 223 Figure 11.4 Distribution of Sample Means 224 Figure 11.5 The 95 Percent Confi dence

Interval 225 Figure 11.6 The 99 Percent Confi dence

Interval 225 Figure 11.7 We Can Be 99 Percent Confi dent 226 Figure 11.8 Does a Sample Difference Refl ect

a Population Difference? 226 Figure 11.9 Distribution of the Difference Between

Sample Means 227 Figure 11.10 Confi dence Intervals 227 Figure 11.11 Null and Research Hypotheses 229 Figure 11.12 Illustration of When a Researcher

Would Reject the Null Hypothesis 230 Figure 11.13 How Much Is Enough? 231 Figure 11.14 Signifi cance Area for a One-Tailed

Test 231 Figure 11.15 One-Tailed Test Using a Distribution of

Differences Between Sample Means 232 Figure 11.16 Two-Tailed Test Using a Distribution of

Differences Between Sample Means 232 Figure 11.17 A Hypothetical Example of Type I and

Type II Errors 233 Figure 11.18 Rejecting the Null Hypothesis 239 Figure 11.19 An Illustration of Power Under an

Assumed Population Value 239 Figure 11.20 A Power Curve 240 Figure 12.1 Combinations of Data and Approaches

to Research 247

fra97851_fm_i-xxviii.indd xvifra97851_fm_i-xxviii.indd xvi 24/12/10 2:14 AM24/12/10 2:14 AM

L I S T O F F E A T U R E S xvii

Figure 12.2 A Difference That Doesn’t Make a Difference! 250

Figure 12.3 Frequency Polygons of Gain Scores on Test of Ability to Explain: Inquiry and Lecture Groups 251

Figure 12.4 90 Percent Confi dence Interval for a Difference of 1.2 Between Sample Means 251

Figure 12.5 Scatterplots with a Pearson r of .50 253 Figure 12.6 Scatterplot Illustrating the Relationship

Between Initial Self-Esteem and Gain in Marital Satisfaction Among Counseling Clients 255

Figure 12.7 95 Percent Confi dence Interval for r 5 .42 255

Figure 13.1 Example of a One-Shot Case Study Design 269

Figure 13.2 Example of a One-Group Pretest- Posttest Design 269

Figure 13.3 Example of a Static-Group Comparison Design 270

Figure 13.4 Example of a Randomized Posttest- Only Control Group Design 271

Figure 13.5 Example of a Randomized Pretest- Posttest Control Group Design 272

Figure 13.6 Example of a Randomized Solomon Four-Group Design 273

Figure 13.7 A Randomized Posttest-Only Control Group Design, Using Matched Subjects 274

Figure 13.8 Results (Means) from a Study Using a Counterbalanced Design 276

Figure 13.9 Possible Outcome Patterns in a Time- Series Design 277

Figure 13.10 Using a Factorial Design to Study Effects of Method and Class Size on Achievement 278

Figure 13.11 Illustration of Interaction and No Interaction in a 2 by 2 Factorial Design 278

Figure 13.12 Example of a 4 by 2 Factorial Design 279

Figure 13.13 Guidelines for Handling Internal Validity in Comparison Group Studies 282

Figure 14.1 Single-Subject Graph 303 Figure 14.2 A-B Design 304 Figure 14.3 A-B-A Design 305 Figure 14.4 A-B-A-B Design 306

Figure 14.5 B-A-B Design 307 Figure 14.6 A-B-C-B Design 308 Figure 14.7 Multiple-Baseline Design 308 Figure 14.8 Multiple-Baseline Design 309 Figure 14.9 Multiple-Baseline Design Applied to

Different Settings 310 Figure 14.10 Variations in Baseline Stability 311 Figure 14.11 Differences in Degree and Speed

of Change 312 Figure 14.12 Differences in Return to Baseline

Conditions 313 Figure 15.1 Scatterplot Illustrating a Correlation

of 11.00 332 Figure 15.2 Prediction Using a Scatterplot 333 Figure 15.3 Multiple Correlation 335 Figure 15.4 Discriminant Function Analysis 336 Figure 15.5 Path Analysis Diagram 338 Figure 15.6 Scatterplots for Combinations

of Variables 342 Figure 15.7 Eliminating the Effects of Age Through

Partial Correlation 343 Figure 15.8 Scatterplots Illustrating How a Factor

(C) May Not Be a Threat to Internal Validity 346 Figure 15.9 Circle Diagrams Illustrating

Relationships Among Variables 346 Figure 16.1 Examples of the Basic Causal-

Comparative Design 370 Figure 16.2 A Subject Characteristics Threat 371 Figure 16.3 Does a Threat to Internal Validity

Exist? 374 Figure 17.1 Example of an Ideal Versus an

Actual Telephone Sample for a Specifi c Question 397

Figure 17.2 Example of Several Contingency Questions in an Interview Schedule 402

Figure 17.3 Sample Cover Letter for a Mail Survey 403

Figure 17.4 Demographic Data and Representativeness 406

Figure 18.1 How Qualitative and Quantitative Researchers See the World 431

Figure 19.1 Variations in Approaches to Observation 447

fra97851_fm_i-xxviii.indd xviifra97851_fm_i-xxviii.indd xvii 24/12/10 2:14 AM24/12/10 2:14 AM

xviii L I S T O F F E A T U R E S www.mhhe.com/fraenkel8e

Figure 19.2 The Importance of a Second Observer as a Check on One’s Conclusions 449

Figure 19.3 The Amidon/Flanders Scheme for Coding Categories of Interaction in the Classroom 450

Figure 19.4 An Interview of Dubious Validity 455 Figure 19.5 Don’t Ask More Than One Question at

a Time 456 Figure 20.1 TV Violence and Public Viewing

Patterns 479 Figure 20.2 What Categories Should I Use? 483 Figure 20.3 An Example of Coding an

Interview 484 Figure 20.4 Categories Used to Evaluate Social

Studies Research 487 Figure 21.1 Triangulation and Politics 517

Figure 22.1 What Really Happened? 541 Figure 22.2 Historical Research Is Not as Easy as You

May Think! 543 Figure 23.1 Exploratory Design 560 Figure 23.2 Explanatory Design 561 Figure 23.3 Triangulation Design 561 Figure 24.1 Stakeholders 591 Figure 24.2 The Role of the “Expert” in Action

Research 592 Figure 24.3 Levels of Participation in Action

Research 592 Figure 24.4 Participation in Action Research 595 Figure 24.5 Experimental Design for the

DeMaria Study 602 Figure 25.1 Organization of a Research Report 626

fra97851_fm_i-xxviii.indd xviiifra97851_fm_i-xxviii.indd xviii 24/12/10 2:14 AM24/12/10 2:14 AM

xix

PREFACE

How to Design and Evaluate Research in Education is directed to students taking their fi rst course in edu- cational research. Because this fi eld continues to grow so rapidly with regard to both the knowledge it con- tains and the methodologies it employs, the authors of any introductory text are forced to carefully defi ne their goals as a fi rst step in deciding what to include in their book. In our case, we continually kept three main goals in mind. We wanted to produce a text that would:

1. Provide students with the basic information needed to understand the research process, from idea f ormulation through data analysis and interpretation.

2. Enable students to use this knowledge to design their own research investigation on a topic of personal interest.

3. Permit students to read and understand the literature of educational research.

The fi rst two goals are intended to satisfy the needs of those students who must plan and carry out a re- search project as part of their course requirements. The third goal is aimed at students whose course require- ments include learning how to read and understand the research of others. Many instructors, ourselves in- cluded, build all three goals into their courses, since each one seems to reinforce the others. It is hard to read and fully comprehend the research of others if you have not yourself gone through the process of de- signing and evaluating a research project. Similarly, the more you read and evaluate the research of others, the better equipped you will be to design your own meaningful and creative research. In order to achieve the above goals, we have developed a book with the following characteristics.

CONTENT COVERAGE

Goal one, to provide students with the basic information needed to understand the research process, has resulted in a nine-part book plan. Part 1 (Chapter 1) introduces students to the nature of educational research, briefl y overviews each of the seven methodologies discussed later in the text, and presents an overview of the re- search process as well as criticisms of it. Part 2 (Chapters 2 through 9) discusses the basic con- cepts and procedures that must be understood before one can engage in research intelligently or critique it meaning- fully. These chapters explain variables, defi nitions, ethics, sampling, instrumentation, validity, reliability, and internal validity. These and other concepts are covered thoroughly, clearly, and relatively simply. Our emphasis throughout is to show students, by means of clear and appropriate ex- amples, how to set up a research study in an educational setting on a question of interest and importance. Part 3 (Chapters 10 through 12) describes in some detail the processes involved in collecting and analyzing data. Part 4 (Chapters 13 through 17) describes and il- lustrates the methodologies most commonly used in quantitative educational research. Many key concepts presented in Part 2 are considered again in these chap- ters in order to illustrate their application to each meth- odology. Finally, each methodology chapter concludes with a carefully chosen study from the published re- search literature. Each study is analyzed by the authors with regard to both its strengths and weaknesses. Stu- dents are shown how to read and critically analyze a study they might fi nd in the literature. Parts 5 (Chapters 18 through 20) and 6 (Chapters 21 through 22) discuss qualitative research. Part 5 be - g ins the coverage by describing qualitative research, its

fra97851_fm_i-xxviii.indd xixfra97851_fm_i-xxviii.indd xix 24/12/10 2:14 AM24/12/10 2:14 AM

xx P R E F A C E www.mhhe.com/fraenkel8e

philosophy, and essential features. It has been expanded to include various types of qualitative research. This is followed by an expanded treatment of both data collec- tion and analysis methods. Part 6 presents the qualitative methodologies of ethnography and historical research. As with the quantitative methodology chapters, all but one of these is followed by a carefully chosen research report from the published research literature, along with our analysis and critique. Part 7 (Chapter 23) discusses Mixed-Methods Stud- ies, which combine quantitative and qualitative methods. Again, as in other chapters, the discussion is followed by our analysis and critique of a research report we have chosen from the published research literature. Part 8 (Chapter 24) describes the assumptions, charac- teristics, and steps of action research. Classroom exam- ples of action research questions bring the subject to life, as does the addition of a critique of a published study. Part 9 (Chapter 25) shows how to prepare a research proposal or report (involving a methodology of choice) that builds on the concepts and examples developed and illustrated in previous chapters.

RESEARCH EXERCISES

To achieve our second goal of helping students learn to apply their knowledge of basic processes and method- ologies, we organized the fi rst 12 chapters in the same order that students normally follow in developing a re- search proposal or conducting a research project. Then we concluded each of these chapters with a research exercise that includes a fi ll-in problem sheet. These ex- ercises allow students to apply their understanding of the major concepts of each chapter. When completed, these accumulated problem sheets will have led students through the step-by-step processes involved in design- ing their own research projects. Although this step-by- step development requires some revision of their work as they learn more about the research process, the gain in understanding that results as they slowly see their proposal develop “before their eyes” justifi es the extra time and effort involved. Problem Sheet templates are located in the Student Mastery Activities book electronically at the Online Learning Center Web site, www.mhhe.com/fraenkel8e.

ACTUAL RESEARCH STUDIES

Our third goal, to enable students to read and under- stand the literature of educational research, has led us to conclude each of the methodology chapters in

Parts  4, 5, 6, 7, and 8, with an annotated study that illustrates a particular research method. At the end of each study we analyze its strengths and weaknesses and offer suggestions as to how it might be improved. Similarly, at the end of our chapter on writing research proposals and reports, we include a student research proposal that we have critiqued with marginal com- ments. This annotated proposal has proved an effective means of helping students understand both sound and questionable research practices.

STYLE OF PRESENTATION

Because students are typically anxious regarding the content of research courses, we have taken extraor- dinary care not to overwhelm them with dry, abstract discussions, and we have adopted an informal writing style. More than in any text to date, our presentations are laced with clarifying examples and with summa- rizing charts, tables, and diagrams. Our experience in teaching research courses for more than 30 years has convinced us that there is no such thing as having “too many” examples in a basic text. In addition to the many examples and illustra- tions that are embedded in the text, we have built the following pedagogical features into the book: (1) a graphic organizer for each chapter, (2) chapter objectives, (3) chapter-opening examples, (4) end- of- chapter summaries, (5) key terms with page refer- ences, (6) discussion questions, and (7) an extensive end-of-book glossary.

CHANGES IN THE EIGHTH EDITION

A number of key additions, new illustrations, and im- proved or refi ned examples, terminology, and defi ni- tions have been incorporated in this edition to further meet the goals of the text. The Notes (references) have been updated throughout to include the latest research, and the Research Exercises and Problem Sheets have been revised with more effective questions. Following is a sampling of chapter-by-chapter changes:

Chapter 1: The Nature of Educational Research • Moved up discussion of quantitative and qualitative

research to the beginning of the “Types of Research” section

• Added a new section on evaluation research • Added new key terms

Chapter 2: The Research Problem

fra97851_fm_i-xxviii.indd xxfra97851_fm_i-xxviii.indd xx 24/12/10 2:14 AM24/12/10 2:14 AM

P R E F A C E xxi

• Revised the section on research questions, including the distinction between quantitative and qualitative research questions and new examples

• Revised illustrations to include more diversity

Chapter 3: Locating and Reviewing Literature • Originally was Chapter 5; contents moved up to

highlight the importance of fi nding and analyzing research while introducing the many new ideas pre- sented about literature review

• Updated the technological/digital terminology throughout

• Updated the discussion of search engines and other research tools

• Added new key terms

Chapter 4: Ethics and Research • Added a new section on academic cheating and

plagiarism • Revised the opening scenario • Added new information on regulation of research

and human subjects, including the NIH and CITI • Updated the discussion of Institutional Review

Boards and HHS regulations, and added an IRB (Human Subjects) protocol template and consent form

• Added new key terms

Chapter 5: Variables and Hypotheses • Originally was Chapter 3; contents moved back for

more logical fl ow of material • Revised terminology for clarity • Added new discussion of mediator variables • Added new section on hypotheses and qualitative

research • Added new key terms

Chapter 6: Sampling • Added mention of how to use computer-generated

lists • Added a note on qualitative studies and sample

size

Chapter 7: Instrumentation • Revised opening case illustration • Added a note on qualitative researchers and objectivity • Updated the discussion on using the ERIC database

and other testing resources • Updated the screen captures to show recent searches • Added information on using probes (improvised

questions) in interviews

• Added a new example of a qualitative, open-ended interview protocol

• Added new key terms Chapter 8: Validity and Reliability • Updated terminology Chapter 9: Internal Validity • Revised opening illustration to be more realistic • Revised terminology to refl ect more accepted

usage • Added information on qualitative research and inter-

nal validity • Inserted new table on “Threats to the Internal Valid-

ity of a Study”

Chapter 10: Descriptive Statistics • Revised chapter objectives • Revised opening case study • Added and revised Excel boxes throughout, deleting

previous SPSS boxes • Relocated section on “Types of Scores,” originally

in Chapter 7 • Added new key terms Chapter 11: Inferential Statistics • Added new Excel box Chapter 12: Statistics in Perspective • Added Delta to discussion of effect size Chapter 13: Experimental Research • Updated examples of experimental studies con-

ducted by educational researchers • Updated digital terms • Revised discussion of gender and groups • Replaced research report and analysis with a new

study on students at risk

Chapter 14: Single-Subject Research • Replaced opening illustration with case example • Added examples of single-subject studies Chapter 15: Correlational Research • Updated examples of correlational studies Chapter 16: Causal-Comparative Research • Replaced research report and analysis with a new

study on Internet use and abuse by students

Chapter 17: Survey Research • Added examples of surveys • Added a new section on Web-based surveys

Chapter 18: The Nature of Qualitative Research • Added new examples of qualitative studies • Included mention of “theoretical approach” • Revised section on generation of hypotheses

fra97851_fm_i-xxviii.indd xxifra97851_fm_i-xxviii.indd xxi 24/12/10 2:14 AM24/12/10 2:14 AM

xxii P R E F A C E www.mhhe.com/fraenkel8e

• Added discussion of qualitative data analysis • Added new key terms Chapter 19: Observation and Interviewing • Updated discussion of using technology with current

terminology • Added note about attentive listening during

interviewing • Expanded discussion of focus groups and the role of

the moderator

Chapter 20: Content Analysis • Added examples of content analysis studies

Chapter 21: Ethnographic Research • Revised opening case study • Revised examples of studies by ethnographers • Added note regarding theoretical generalizability of

a study • Replaced research report and analysis with a new

study on teaching ESL students

Chapter 22: Historical Research • Revised examples of historical studies Chapter 23: Mixed-Methods Research • Revised examples of mixed-methods studies • Added discussion of multiple samples Chapter 24: Action Research • Added examples of action research • Replaced research report and analysis with a new

study on integrating student choice and arts activities

Chapter 25: Preparing Research Proposals and Reports • Added discussion of qualitative research proposals

offering several propositions • Updated terminology used with organization of a re-

search report

Appendix D: Using Excel • Revised this appendix to include information on using

Excel for various statistical calculations, rather than SPSS, as it is more commonly used today.

New Annotated and Analyzed Research Reports: Four new Research Reports have been added to the text, introducing more research involving diverse populations as well as helping the student apply the text’s concepts and also practice evaluating published studies.

• Cognitive Effects of Chess Instruction on Students at Risk for Academic Failure

• Internet Use, Abuse, and Dependence Among Stu- dents at a Southeastern Regional University

• Lessons on Effective Teaching from Middle School ESL Students

• An Action Research Exploration Integrating Student Choice and Arts Activities in a Sixth-Grade Social Studies Classroom

SPECIAL FEATURES

Support for Student Learning

How to Design and Evaluate Research in Education helps students become critical consumers of research and pre- pares them to conduct and report their own research.

Chapter-opening Features: Each chapter begins with an illustration that visually introduces a topic or issues related to the chapter. This is followed by an outline of chapter content, chapter learning objectives, the Inter- active and Applied Learning feature that lists related supplementary material, and a related vignette.

More About Research, Research Tips, and Controver- sies in Research: These informative sections help stu- dents to think critically about research while illustrating important techniques in educational research.

End-of-Chapter Learning Supports: The chapters conclude with a reminder of the supplementary re- sources available, a detailed Main Points section, a list- ing of Key Terms, and Questions for Discussion. Chapters 1–12 include a Research Exercise and a Problem Sheet to aid students in the construction of a research project. Chapters 13–17 and 19–24 include an actual Re- search Report that has been annotated to highlight concepts discussed in the chapter.

Practical Resources and Examples for Doing and Reading Research

How to Design and Evaluate Research in Education provides a comprehensive introduction to research that is brought to life through practical resources and exam- ples for doing and reading research.

• Research Tips boxes provide practical suggestions for doing research.

• The Annotated Research Reports at the conclusion of Chapters 13–17 and 19–24 present students with re- search reports and author commentary on how the study authors have approached and supported their research.

• Research Exercises and Problems Sheets at the conclusion of Chapters 1–13 are tools for students to use when creating their own research projects.

fra97851_fm_i-xxviii.indd xxiifra97851_fm_i-xxviii.indd xxii 24/12/10 2:14 AM24/12/10 2:14 AM

P R E F A C E xxiii

• Using Excel boxes show how these software pro- grams can be used to calculate various statistics.

• Chapter 24: Action Research details how classroom teachers can and should do research to improve their teaching.

• Chapter 25: Preparing Research Proposals and Reports walks the reader through proposal and report preparation.

• Resources on the Online Learning Center Web site (see listing below) provide students with a place to start when gathering research tools.

SUPPLEMENTS THAT SUPPORT STUDENT LEARNING

Online Learning Center Web Site at www.mhhe.com/fraenkel8e

The Online Learning Center Web site offers tools for study, practice, and application including:

Study Resources

• Multiple quizzes and fl ashcards for testing content knowledge

• Learn More About audio clips that provide additional explanation or examples of key concepts

Practice Resources

• Student Mastery Activities that provide students extra practice with specifi c concepts

• Data Analysis Examples and Exercises

Research Resources

• Statistics Program • Correlation Coeffi cient Applet • Chi Square Applet • Research Wizard, a wizard version of the Problem

Sheets • Forms, including a Research Worksheet, Sample

Consent Forms, Research Checklists, electronic ver- sions of the Problem Sheets

• A Listing of Professional Journals • Bibliography Builder, an electronic reference builder • The McGraw-Hill Guide to Electronic Research

SUPPLEMENTS THAT SUPPORT INSTRUCTORS

Online Learning Center Web Site at www.mhhe.com/fraenkel8e

The Instructor’s portion of the Online Learning Cen- ter offers a number of useful resources for classroom

instruction, including an Instructor’s Manual, Test Bank, Computerized Test Bank, chapter-by-chapter PowerPoint presentations, and additional resources.

CPS by eInstruction

Classroom Performance System is a wireless response system that gives you immediate feedback from every student in the class. These CPS questions are specifi c to How to Design and Evaluate Research in Education, 8/e. Contact your local sales representative for details about this resource.

ACKNOWLEDGMENTS

Directly and indirectly, many people have contributed to the preparation of this text. We will begin by acknowl- edging the students in our research classes, who, over the years, have taught us much. Also, we wish to thank the reviewers of this edition, whose generous comments have guided the preparation of this edition. They include:

Joshua Bagaka, Cleveland State University

Haiyan Bai, University of Central Florida

Patricia Blundell, Chadron State College

Dean Cascadden, Plymouth State University

Joseph Ciechalski, East Carolina University

Noble R. Corey, Indiana State University

Gail Fahoome, Wayne State University

Michael Gardner, University of Utah

Andrea Guillaume, California State University-Fullerton

Deborah Hayes, Lincoln Memorial University

Jeffrey Lorentz, University of Houston-Clear Lake

Jake Marszalek, University of Missouri-Kansas City

Jane Strickland, Idaho State University

James Therrell, Northwestern Oklahoma

Karen Westberg, University of St. Thomas

Guili Zhang, East Carolina University

We would also like to thank the editors and staff at McGraw-Hill for their efforts in turning the manuscript into the fi nished book before you. Finally, we would like to thank our spouses for their unfl agging support during the highs and lows that inevita- bly accompany the preparation of a text of this magnitude.

Jack R. Fraenkel Norman E. Wallen

Helen H. Hyun

fra97851_fm_i-xxviii.indd xxiiifra97851_fm_i-xxviii.indd xxiii 24/12/10 2:14 AM24/12/10 2:14 AM

xxiv

A Guided Tour of How to Design and Evaluate Research in Education

Go to the Online Learning Center at www.mhhe.com/fraenkel8e to:

• Learn More About What Constitutes an Experiment

Go to your online Student Mastery Activities book to do the following activities:

• Activity 13.1: Group Experimental Research Questions • Activity 13.2: Designing an Experiment • Activity 13.3: Characteristics of Experimental Research • Activity 13.4: Random Selections vs. Random Assignment

INTERACTIVE AND APPLIED LEARNING After, or while, reading this chapter:

D oes team-teaching improve the achievement of students in high school social studies classes? Abigail Johnson, the principal of a large high school in Minneapolis, Minnesota, having heard encouraging remarks about the idea at a recent educational conference, wants to find out. Accordingly, she asks some of her eleventh-grade world his-

tory teachers to participate in an experiment. Three teachers are to combine their classes into one large group. These

teachers are to work as a team, sharing the planning, teaching, and evaluation of these students. Three other teachers

are assigned to teach a class in the same subject individually, with the usual arrangement of one teacher per class. The

students selected to participate are similar in ability, and the teachers will teach at the same time, using the same cur-

riculum. All are to use the same standardized tests and other assessment instruments, including written tests prepared

jointly by the six teachers. Periodically during the semester, Mrs. Johnson will compare the scores of the two groups of

students on these tests.

fra97851_ch13_263-300.indd 265 12/21/10 6:58 PM

O B J E C T I V E S Studying this chapter should enable you to:

• Describe briefl y why a literature review is of value.

• Name the steps a researcher goes through in conducting a review of the literature.

• Describe briefl y the kinds of information contained in a general reference and give an example of such a source.

• Explain the difference between a primary and a secondary source and give an example of each type.

• Explain what is meant by the phrase “search term” and how it differs from the

term “descriptor,” and how both terms are used in literature searches.

• Conduct both a manual and electronic search of the literature on a topic of interest to you after a small amount of “hands-on” computer time and a little help from a librarian.

• Write a summary of your literature review. • Explain what a meta-analysis is.

The Defi nition and Value of a Literature Review

Types of Sources

Steps Involved in a Literature Search

Defi ne the Problem as Precisely as Possible

Look through One or Two Secondary Sources

Select the Appropriate General Reference Tools

Formulate Search Terms Search Using General

Reference Tools

Doing a Computer Search

Obtain Primary Sources

Writing the Literature Review Report

Researching the World Wide Web

Locating and Reviewing the Literature 3

“Hey! There’s Joe. Taking it easy,

as usual.” “No, he says he’s reviewing the literature for his Biology class!”

fra97851_ch03_037-059.indd 37 22/12/10 9:36 PM

Welcome to How to Design and Evaluate Research in Education.

This comprehensive introduction to research methods was designed to present the basics of educa- tional research in as interesting and understandable a way as pos- sible. To accomplish this, we’ve created the following features for each chapter.

Opening Illustration

Each chapter opens with an illustrative depiction of a key concept that will be covered in the chapter.

Chapter Outline

Next, a chapter outline lists the topics to follow.

Chapter-Opening Example

The chapter text begins with a practical example— a dialogue between researchers or a peek into a classroom—related to the content to follow.

Objectives

Chapter objectives prepare the student for the chapter ahead.

Interactive and Applied Learning Tools

This special feature lists the practice activities and resources related to the chapter that are available in the student supplements.

fra97851_fm_i-xxviii.indd xxivfra97851_fm_i-xxviii.indd xxiv 24/12/10 2:14 AM24/12/10 2:14 AM

xxv

MORE ABOUT RESEARCH

revolution in science during the twentieth century (the theory of relativity and the discovery of quantum mechanics being the fi rst two), but that it helps to make sense out of what we view as some implications for educational research. What are these implications? *

If chaos theory is correct, the diffi culty in discovering widely generalizable rules or laws in education, let alone the social sciences in general, may not be due to inadequate concepts and theories or to insuffi ciently precise measure- ment and methodology, but may simply be an unavoidable fact about the world. Another implication is that whatever “laws” we do discover may be seriously limited in their applicability—across geography, across individual and/ or group differences, and across time. If this is so, chaos theory provides support for researchers to concentrate on studying topics at the local level—classroom, school, agency—and for repeated studies over time to see if such laws hold up.

Another implication is that educators should pay more at- tention to the intensive study of the exceptional or the unusual, rather than treating such instances as trivial, incidental, or “er- rors.” Yet another implication is that researchers should focus on predictability on a larger scale—that is, looking for pat- terns in individuals or groups over larger units of time. This would suggest a greater emphasis on long-term studies rather than the easier-to-conduct (and cheaper) short-time investiga- tions that are currently the norm.

Not surprisingly, chaos theory has its critics. In education, the criticism is not of the theory itself, but more with misin- terpretations and/or misapplications of it. † Chaos theorists do not say that all is chaos; quite the contrary, they say that we must pay more attention to chaotic phenomena and revise our conceptions of predictability. At the same time, the laws of gravity still hold, as, with less certainty, do many generaliza- tions in education.

Chaos Theory

T he origins of what is now known as chaos theory are usu-ally traced to the 1970s. Since then, it has come to occupy a prominent place in mathematics and the natural sciences and, to a lesser extent, in the social sciences.

Although the physical sciences have primarily been known for their basic laws, or “fi rst principles,” it has long been known by scientists that most of these laws hold precisely only under ideal conditions that are not found in the “real” world. Many phenomena, such as cloud formations, water- fall patterns, and even the weather, elude precise prediction. Chaos theorists argue that the natural laws that are so useful in science may, in themselves, be the exception rather than the rule.

Although precise prediction of such phenomena as the swing of a pendulum or what the weather will be at a particular time is in most cases impossible, repeated patterns, according to a major principle of chaos theory, can be discovered and used, even when the content of the phenomena is chaotic. De- velopments in computer technology, for example, have made it possible to translate an extremely long sequence of “data points,” such as the test scores of a large group of individu- als, into colored visual pictures of fascinating complexity and beauty. Surprisingly, these pictures show distinct patterns that are often quite similar across different content areas, such as physics, biology, economics, astronomy, and geography. Even more surprising is the fi nding that certain patterns recur as these pictures are enlarged. The most famous example is the “Mandlebrot Bug,” shown in Photographs 1.1 and 1.2. Note that Photograph 1.2 is simply a magnifi cation of a portion of Photograph 1.1. The tiny box in the lower left corner of Pho- tograph 1.1 is magnifi ed to produce the box in the upper left- hand corner of Photograph 1.2. The tiny box within this box is then, in turn, magnifi ed to produce the larger portion of Photo- graph 1.2, including the reappearance of the “bug” in the lower right corner. The conclusion is that even with highly complex

fra97851_ch01_001-024.indd 8 12/21/10 6:35 PM

Should Some Research Methods Be Preferred over Others?

R ecently, several researchers * have expressed their concern that the U.S. Department of Education is showing favor- itism toward the narrow view that experimental research is, if

CONTROVERSIES IN RESEARCH

*D. C. Berliner (2002). Educational research: The hardest science of all. Educational Researcher, 31 (8): 18–20; F. E. Erickson and K. Gutierrez (2002). Culture, rigor, and science in educational research. Educational Researcher, 31 (8): 21–24.

†E. A. St. Pierre (2002). Science rejects postmodernism. Educational Researcher, 31 (8): 25.

ALL HAVE VALUE

It must be stressed that each of the research methodolo- gies described so briefl y above has value for us in educa- tion. Each constitutes a different way of inquiring into the realities that exist within our classrooms and schools and into the minds and emotions of teachers, counselors, administrators, parents, and students. Each represents a different tool for trying to understand what goes on, and what works, in schools. It is inappropriate to consider any one or two of these approaches as superior to any of the others. The effectiveness of a particular methodol- ogy depends in large part on the nature of the research question one wants to ask and the specifi c context within which the particular investigation is to take place. We need to gain insights into what goes on in education from as many perspectives as possible, and hence we need to construe research in broad rather than narrow terms.

As far as we are concerned, research in education should ask a variety of questions, move in a variety of directions, encompass a variety of methodologies, and use a variety of tools. Different research orientations,

DESCRIPTIVE STUDIES

Descriptive studies describe a given state of affairs as fully and carefully as possible. One of the best examples of descriptive research is found in botany and zoology, where each variety of plant and animal species is me- ticulously described and information is organized into useful taxonomic categories.

In educational research, the most common descriptive methodology is the survey, as when researchers sum- marize the characteristics (abilities, preferences, behav- iors, and so on) of individuals or groups or (sometimes) physical environments (such as schools). Qualitative ap- proaches, such as ethnographic and historical methodol- ogies are also primarily descriptive in nature. Examples of descriptive studies in education include identifying the achievements of various groups of students; describing the behaviors of teachers, administrators, or counselors; describing the attitudes of parents; and describing the physical capabilities of schools. The description of phe- nomena is the starting point for all research endeavors.

Descriptive research in and of itself, however, is not

not the only, at least the most respectable form of research and the only one worthy of being called scientifi c. Such a prefer- ence has implications for both the funding of school programs and educational research. As one writer commented, “How scared should we be when the federal government endorses a particular view of science and rejects others?” †

fra97851_ch01_001-024.indd 15 12/21/10 6:36 PM

Research Tips

These boxes provide practical pointers for doing research. See a full listing of these boxes on page xii.

More About Research

These boxes take a closer look at important topics in educational research. See a full listing of these boxes, starting on page xii.

Controversies in Research

These boxes highlight a controversy in research to provide you with a greater understanding of the issue. See a full listing of these boxes on page xiii.

Another important point to remember is that often it is a compound term or phrase that needs to be de- fi ned rather than only a single word. For example, the term nondirective therapy will surely not be clarifi ed by precise defi nitions of nondirective and therapy, since it has a more specifi c meaning than the two words de- fi ned separately would convey. Similarly, such terms as learning disability, bilingual education, interactive video, and home-centered health care need to be de- fi ned as linguistic wholes.

Here are three defi nitions of the term motivated to learn. Which do you think is the clearest?

1. Works hard

2. Any classroom judged (by an observer spending at least one day per week for four to fi ve weeks) to pos- sess all the following characteristics: a. No more than three children working with the

same materials at the same time b. The teacher never spending more than 20 minutes

per day addressing the class as a group c. At least half of every class period open for stu-

dents to work on projects of their own choosing at their own pace

d. Several (more than three) sets of different kinds of educational materials available for every stu- dent in the class to use

e Nontraditional seating st dents sit in circles

• Terms that individuals outside the fi eld of study may not understand

• Terms that have multiple meanings • Terms that are essential to understanding what the study

is about • Terms to provide precision in specifi cations for instru-

ments to be developed or located

Key Terms to Defi ne in a Research Study

• Terms necessary to ensure that the research question is sharply focused

RESEARCH TIPS

fra97851_ch02_025-036.indd 31 22/12/10 9:35 PM

fra97851_fm_i-xxviii.indd xxvfra97851_fm_i-xxviii.indd xxv 24/12/10 2:14 AM24/12/10 2:14 AM

xxvi

RESEARCH REPORT

566

From: Journal of Counseling Psychology , 53 , no. 3 (2006): 279–287. Reproduced with permission.

Perceived Family Support, Acculturation, and Life Satisfaction in Mexican American Youth: A Mixed-Methods Exploration L. M. Edwards Marquette University

S. J. Lopez University of Kansas

In this article, the authors describe a mixed-methods study designed to explore per- ceived family support, acculturation, and life satisfaction among 266 Mexican Ameri- can adolescents. Specifi cally, the authors conducted a thematic analysis of open-ended responses to a question about life satisfaction to understand participants’ perceptions of factors that contributed to their overall satisfaction with life. The authors also con- ducted hierarchical regression analyses to investigate the independent and interactive contributions of perceived support from family and Mexican and Anglo acculturation orientations on life satisfaction. Convergence of mixed-methods fi ndings demonstrated that perceived family support and Mexican orientation were signifi cant predictors of life satisfaction in these adolescents. Implications, limitations, and directions for further research are discussed.

Psychologists have identifi ed and studied a number of challenges faced by Latino youth (e.g., juvenile delinquency, gang activity, school dropout, alcohol and drug abuse), yet little scholarly time and energy have been spent on exploring how these adolescents successfully navigate their development into adulthood or how they expe- rience well-being (Rodriguez & Morrobel, 2004). Researchers have yet to understand the personal characteristics that play a role in Latino adolescents’ satisfaction with life or how certain cultural values and/or strengths and resources are related to their well- being. Answers to these questions can begin to provide counseling psychologists with a deeper understanding of how Latino adolescents experience well-being, which can, in turn, hopefully allow researchers to work to improve well-being for those who struggle to fi nd it.

Latino 1 youth are a growing presence in most communities within the United States. The U.S. Census Bureau projects that by the year 2010, 20% of young people be- tween the ages of 10 and 20 years will be of Hispanic origin. Furthermore, it is projected that by the year 2020, one in fi ve children will be Hispanic, and the Hispanic adolescent population will increase by 50% (U.S. Census Bureau, 2000, 2001). Whereas adolescence is a unique developmental period for all youth, Latino adolescents in particular may face additional challenges as a result of their ethnic minority status (Vazquez Garcia, Garcia

Justifi cation

1In this article, the terms Latino and Hispanic have been used interchangeably. Specifically, in cases in which research is summarized, the descriptors used by the authors were retained. The participant sample, however, was restricted to adolescents who self-identified as “Mexican” or “Mexican American” and are thus described as such.

fra97851_ch23_555-586.indd 566 22/12/10 11:00 PM

C H A P T E R 2 3 Mixed-Methods Research 567

Coll, Erkut, Alarcon, & Tropp, 2000). These youth generally have undergone socializa- tion experiences of their Latino culture (known as enculturation ) and also must learn to acculturate to the dominant culture to some degree (Knight, Bernal, Cota, Garza, & Ocampo, 1993). Navigating the demands of these cultural contexts can be challenging, and yet many Latino youth experience well-being and positive outcomes. The increas- ing numbers of Latino youth, along with the counseling psychology fi eld’s imperative to provide culturally competent services, require that professionals continue to understand the full range of psychological functioning for members of this unique population.

Counseling psychologists have continually emphasized the importance of well- being and identifying and developing client strengths in theory, research, and practice (Lopez et al., 2006; Walsh, 2003). This commitment to understanding the whole person, including internal and contextual assets and challenges, has been one hallmark of the fi eld (Super, 1955; Tyler, 1973) and has infl uenced a variety of research about optimal human functioning (see D. W. Sue & Constantine, 2003). More recent discussions in this area have underscored the importance of identifying and nurturing cultural values and strengths in people of color (e.g., family, religious faith, biculturalism), being cautious to acknowledge that strengths are not universal and may differ according to context or cultural background (Lopez et al., 2006; Lopez et al., 2002; D. W. Sue & Constantine, 2003), and may be infl uenced by certain within-group differences such as acculturation level (Marin & Gamba, 2003; Zane & Mak, 2003).

As scholars respond to the emerging need to explore strengths among Latino youth, the importance of investigating these resources and values within a cultural context is evi- dent. Understanding how Latino adolescents experience well-being from their own per- spectives and vantage points is integral, as theories from other cultural worldviews may not be applicable to their lives (Auerbach & Silverstein, 2003; Lopez et al., 2002; D. W. Sue & Constantine, 2003). Furthermore, it is necessary to continue to test propositions about the role of certain Latino cultural values, such as the importance of family, in overall well- being. Given that many Latino adolescents today navigate bicultural contexts and adhere to Latino traditions and customs to differing degrees (Romero & Roberts, 2003), it is likely that the role family plays in adolescent well-being is complex and infl uenced by individual differences such as acculturation. In this study, we sought to explore the relationships be- tween these variables by focusing specifi cally on perceived family support, life satisfaction, and acculturation among Mexican American youth.

PERCEIVED FAMILY SUPPORT, ACCULTURATION, AND LIFE SATISFACTION AMONG LATINO YOUTH

The importance of family has been noted as a core Latino cultural value (Castillo, Conoley, & Brossart, 2004; Marin & Gamba, 2003; Paniagua, 1998; Sabogal, Marin, Otero-Sabogal, Marin, & Perez-Stable, 1987). Familismo (familism) is the term used to describe the im- portance of extended family ties in Latino culture as well as the strong identifi cation and attachment of individuals with their families (Triandis, Marin, Betancourt, Lisansky, & Chang, 1982). Familism is not unique to Latino culture and has been noted as an important value for other ethnic groups such as African Americans, Asian Americans, and American Indians (Cooper, 1999; Marin & Gamba, 2003). Nevertheless, it is considered a central as- pect of Latino culture, and in some studies, it has been shown to be valued by Latino indi- viduals more than by non-Latino Whites (Gaines et al., 1997; Marin, 1993; Mindel, 1980).

In a study of familismo among Latino adolescents, Vazquez Garcia et al. (2000) found that the length of time youth had been in the United States did not affect their ad- herence to the value of familismo . These results demonstrated that the longer adolescents

Defi nitions

Justifi cation

Purpose

Defi nition

Prior research

fra97851_ch23_555-586.indd 567 22/12/10 11:00 PM

Figures and Tables

Numerous fi gures and tables explain or extend concepts presented in the text.

Research Reports

Published research reports are included at the conclusion of methodology chapters. The reports have been annotated to illustrate important points.

Figure 6.6 Nonrandom Sampling Methods

Populations

Z

A B C

D E

FG H

I

J

K

L

M N

O

P Q

R

S T U

V W X

Y

Z

A B

C D

E F

G H IJ

K L

M N

O

P Q

R S

T

U V

W

X

Y Z

U V W X Y P Q R S T K L M N O F G H I J A B C D E

Especially qualified

Easily accessible

Convenience Purposive Systematic

NL

B F

VI

LQ X

Y Q

LB G

V

SAMPLES

fra97851_ch06_090-109.indd 101 12/21/10 6:43 PM

fra97851_fm_i-xxviii.indd xxvifra97851_fm_i-xxviii.indd xxvi 24/12/10 2:14 AM24/12/10 2:14 AM

xxvii

HYPOTHESES

No hypothese are stated. A nondirectional hypothesis is clearly implied—i.e., there will be differences between the two groups.

SAMPLE

The two groups are convenience (and possibly volun- teer) samples from the two nations. Each is described with respect to location, gender, age, and academic class. They consist only of business students, who may not be representative of all college students. Represen- tativeness is further compromised by the unreported number of “unusable” surveys. Sample numbers (443 and 174) are acceptable.

INSTRUMENTATION

The questionnaire consists of yes-no questions (two based on brief scenarios) to measure “tendencies” and seven-point rating scales to assess attitudes and beliefs about cheating, for a total of 29 items, of which 21 are shown in the report. Neither reliability nor validity is discussed. Because the intent was to compare groups on individual items, no summary scores were used. Nevertheless, consistency of response to individual items is essential to meaningful results. Though admit- tedly diffi cult, the procedure followed in the Kinsey study (see page 398) of asking the same question with different wording might have been used with, at least, a subsample of students and items. Similarly, a compari- son of the questionnaire with interview responses to the same content would have provided some evidence of validity.

The question of validity is confused by the lack of clear defi nitions. The items in Table 1 suggest that “ten- dencies to cheat” is taken to mean “having cheated or known of others cheating,” although the two scenario items seem to be asking what is considered to consti- tute cheating. Attitudes and perceptions are combined in Table 2 as “beliefs,” which seem to include both “opinions about the extent of cheating” and “judgments as to what behaviors are acceptable”—as well as what constitutes instructor responsibility. As such, the items appear to have content validity but omit other behaviors, such as destroying required library readings. This does not invalidate the items used unless they are considered to represent all forms of cheating. Finally, the validity of self-report items cannot be assumed, particularly in cross-cultural studies, where meanings may differ.

Analysis of the Study PURPOSE/JUSTIFICATION

The purpose is not explicitly stated. It appears to be to “fi ll in the gap in our knowledge about cross-national differences in attitudes, beliefs, and tendencies towards cheating” and, more specifi cally, to compare college business students in Russia and the United States on these characteristics.

The study is justifi ed by citing both evidence and opinion that cheating is widespread in the United States and, presumably (although with less documentation), worldwide. Additional justifi cation includes the unfair- ness of cheating, the likelihood of cheating carrying into future life, and (in the discussion) the need for teach- ers in multinational classes to understand the issues involved. The importance of attitudes and perceptions seems to be taken for granted; the only justifi cation for studying them is implied in the results of the three stud- ies that found differences between American students and those in other countries. We think a stronger justifi - cation could and should have been made. The fi nal jus- tifi cation is that there have been few such studies, none with business students in Russia and the United States.

The authors’ concern about confi dentiality is important, both with regard to ethics and the validity of information; they appear to have addressed it as effectively as possible. There appear to be no problems of risk or deception.

DEFINITIONS

Defi nitions are not provided and would be very helpful (as discussed below under “Instrumentation”) because the terms attitude, values, and beliefs, especially, have many different meanings. The term tendencies appears to mean (from the example items) actual cheating in various forms. Some clarity is provided by partial op- erational defi nitions in the form of example items. We think a defi nition of cheating should have been provided to readers and to respondents. Based on the items pro- vided, it appears to be something like “receiving credit for work that is not one’s own.”

PRIOR RESEARCH

The authors provide extensive citation of evidence and summaries of studies on the extent of college-level cheating and on cross-national comparisons. They give good brief summaries of what they state are the only three directly related studies.

fra97851_ch17_392-422.indd 417 12/21/10 7:05 PM

Go back to the INTERACTIVE AND APPLIED LEARNING feature at the beginning of the chapter for a listing of interactive and applied activities. Go to the Online Learning Center at www.mhhe.com/fraenkel8e to take quizzes, practice with key terms, and review chapter content.

BASIC ETHICAL PRINCIPLES

• Ethics refers to questions of right and wrong. • There are a number of ethical principles that all researchers should be aware of and

apply to their investigations. • The basic ethical question for all researchers to consider is whether any physical or

psychological harm could come to anyone as a result of the research. • All subjects in a research study should be assured that any data collected from or about

them will be held in confi dence. • The term deception, as used in research, refers to intentionally misinforming the sub-

jects of a study as to some or all aspects of the research topic. • Plagiarism is the act of misrepresenting someone else’s work as one’s own. • Unintentional plagiarism can be avoided through the proper use and citation of pub-

lished and unlisted sources.

RESEARCH WITH CHILDREN

• Children as research subjects present problems for researchers that are different from those of adult subjects. Children are more vulnerable, have fewer legal rights, and often do not understand the meaning of informed consent.

REGULATION OF RESEARCH

• Before any research involving human beings can be conducted at an institution that receives federal funds, it must be reviewed by an institutional review board (IRB) at the institution.

• The federal agency that has the major responsibility for establishing the guidelines for research studies that involve human subjects is the Department of Health and Human Services.

Main Points

fra97851_ch04_060-074.indd 72 22/12/10 11:00 PM

Main Points

Bulleted main points highlight the key concepts of the chapter.

Each research report is critiqued by the authors, with both its strengths and weaknesses discussed.

Chapter Review

The chapter ends with a listing of the review resources available for students on the Online Learning Center Web site at www.mhhe.com/ fraenkel8e.

fra97851_fm_i-xxviii.indd xxviifra97851_fm_i-xxviii.indd xxvii 24/12/10 2:14 AM24/12/10 2:14 AM

xxviii

p p j p y that seem to be increasing in recent years. This is a problem because those who do not respond are very likely to differ from respondents in terms of how they would answer the survey questions.

THREATS TO INTERNAL VALIDITY IN SURVEY RESEARCH

• Threats to the internal validity of survey research include location, instrumentation, instrument decay, and mortality.

DATA ANALYSIS IN SURVEY RESEARCH

• The percentage of the total sample responding for each item on a survey question- naire should be reported, as well as the percentage of the total sample who chose each alternative for each question.

census 394

closed-ended question 399

cohort study 394

contingency question 402

cross-sectional survey 394

interview schedule 399

longitudinal survey 394

nonresponse 405

open-ended question 400

panel study 394

trend study 394

unit of analysis 395

Key Terms

1. For what kinds of topics might a personal interview be superior to a mail or tele- phone survey? Give an example.

2. When might a telephone survey be preferable to a mail survey? to a personal interview? 3. Give an example of a question a researcher might use to assess each of the follow-

ing characteristics of the members of a teacher group: a. Their income b. Their teaching style c. Their biggest worry d. Their knowledge of teaching methods e. Their opinions about homogeneous grouping of students

4. Which mode of data collection—mail, telephone, or personal interview—would be best for each of the following surveys? a. The reasons why some students drop out of college before they graduate b. The feelings of high school teachers about special classes for the gifted c. The attitudes of people about raising taxes to pay for the construction of new

schools d. The duties of secondary school superintendents in a midwestern state e. The reasons why individuals of differing ethnicity did or did not decide to enter

the teaching profession f. The opinions of teachers about the idea of minimum competency testing before

granting permanent tenure g. The opinions of parents of students in a private school about the elimination of

certain subjects from the curriculum

For Discussion

fra97851_ch17_392-422.indd 420 12/21/10 7:05 PM

Q g

Problem Sheet 13

Research Methodology You should complete Problem Sheet 13 once you have decided which of the methodologies described in Chapters 13–17 and 19–24 you plan to use. You might wish to consider, however, whether your research question could be investigated by other methodologies.

1. The question or hypothesis of my study is: _________________________________

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

2. The methodology I intend to use is: ______________________________________

____________________________________________________________________

____________________________________________________________________

3. Describe how you will conduct the study, i.e., the data collection process. When, where, and how will you collect the data? Over what time span will the data be gathe- red, and in what types of situations? Can you foresee any limitations or problems?

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

4. If you are planning an intervention study (e.g., an experiment), please discuss in detail the intervention or treatment planned. ________________________________

____________________________________________________________________

____________________________________________________________________

5. The major problems I foresee at this point include the following: _______________

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

An electronic version of this Problem Sheet that you can fi ll in and print, save, or e-mail is available on the Online Learning Center

Research Exercise 13: Research Methodology Using Problem Sheet 13, describe in as much detail as you can the procedures of your study, in- cluding analysis of results—that is, what you intend to do, when, where, and how. Lastly, indicate any unresolved problems you see at this point in your planning.

fra97851_ch13_263-300.indd 300 12/21/10 6:59 PM

Key Terms

Key terms are listed with page references.

For Discussion

End-of-chapter questions are designed for in-class discussion.

Research Exercises

The research exercise explains how to fi ll in the Problem Sheet that follows.

Problem Sheets

Individually, the problem sheets allow students to apply their understanding of the major concepts of each chapter. As a whole, they walk students through each step of the research process.

fra97851_fm_i-xxviii.indd xxviiifra97851_fm_i-xxviii.indd xxviii 24/12/10 2:14 AM24/12/10 2:14 AM

Research takes many forms. In Part 1, we introduce you to the subject of educational

research and explain why knowledge of the various types of research is of value to

educators. Because research is but one way to obtain knowledge, we also describe

several other ways and compare the strengths and weaknesses of each. We give a brief

overview of educational research methodologies to set the stage for a more extensive

discussion in later chapters. Lastly, we discuss criticisms of the research process.

Introduction to Research

1P A R T

fra97851_ch01_001-024.indd 1fra97851_ch01_001-024.indd 1 12/21/10 6:35 PM12/21/10 6:35 PM

“It seems pretty obvious to me that the more you know about a subject, the better you

can teach it!”

“Right! Just like we know that evidence

is overwhelmingly in favor of phonics as the best way

to teach reading!”

“Well, you both may be in for

a surprise!”

O B J E C T I V E S Studying this chapter should enable you to:

• Explain what is meant by the term “educational research” and give two examples of the kinds of topics educational researchers might investigate.

• Explain why a knowledge of scientifi c research methodology can be of value to educators.

• Name and give an example of four ways of knowing other than the method used by scientists.

• Explain what is meant by the term “scientifi c method.”

• Give an example of six different types of research methodologies used by educational researchers.

• Describe briefl y what is meant by critical research.

• Describe the differences among descriptive, associational, and intervention-type studies.

• Describe briefl y the difference between basic and applied research.

• Describe briefl y the difference between quantitative and qualitative research.

• Describe briefl y what is meant by mixed - methods research.

• Describe briefl y the basic components involved in the research process.

Some Examples of Educational Concerns

Why Research Is of Value

Ways of Knowing

Sensory Experience Agreement with Others Expert Opinion Logic The Scientifi c Method

Types of Research

Quantitative and Qualitative Research

Experimental Research Correlational Research Causal-Comparative Research Survey Research Ethnographic Research Historical Research Action Research Evaluation Research All Have Value

General Research Types

Descriptive Studies Associational Research Intervention Studies Meta-analysis

Critical Analysis of Research

A Brief Overview of the Research Process

1 The Nature of Research

fra97851_ch01_001-024.indd 2fra97851_ch01_001-024.indd 2 12/21/10 6:35 PM12/21/10 6:35 PM

3

• The president of the local PTA in Little Rock, parent of a sixth-grader at Cabrillo School, wonders how he can get more parents involved in school-related activities.

Each of the above examples, although fi ctional, rep- resents a typical sort of question or concern facing many of us in education today. Together, these examples sug- gest that teachers, counselors, administrators, parents, and students continually need information to do their jobs. Teachers need to know what kinds of materials, strategies, and activities best help students learn. Coun- selors need to know what problems hinder or prevent students from learning and how to help them with these problems. Administrators need to know how to provide an environment for happy and productive learning. Par- ents need to know how to help their children succeed in school. Students need to know how to study to learn as much as they can.

Some Examples of Educational Concerns • A high school principal in San Francisco wants to

improve the morale of her faculty. • The director of the gifted student program in Denver

would like to know what happens during a typical week in an English class for advanced placement students.

• An elementary school counselor in Boise wishes he could get more students to open up to him about their worries and problems.

• A tenth-grade biology teacher in Atlanta wonders if discussions are more effective than lectures in moti- vating students to learn biological concepts.

• A physical education teacher in Tulsa wonders if abil- ity in one sport correlates with ability in other sports.

• A seventh-grade student in Philadelphia asks her counselor what she can do to improve her study habits.

Go to the Online Learning Center at www.mhhe.com/fraenkel8e to:

• Learn More About Why Research Is of Value

Go to your online Student Mastery Activities book to do the following activities:

• Activity 1.1: Empirical vs. Nonempirical Research • Activity 1.2: Basic vs. Applied Research • Activity 1.3: Types of Research • Activity 1.4: Assumptions • Activity 1.5: General Research Types

INTERACTIVE AND APPLIED LEARNING After, or while, reading this chapter:

D r. Hunter? I’m Molly Levine. I called you about getting some advice about the master’s degree program in your department.” “Hello, Molly. Pleased to meet you. Come on in. How can I be of help?”

“Well, I’m thinking about enrolling in the master’s degree program in marriage and family counseling, but fi rst I want to know

what the requirements are.”

“I don’t blame you. It’s always wise to know what you are getting into. To obtain the degree, you’ll need to take a number of

courses, and there is also an oral exam once you have completed them. You also will have to complete a small-scale study.”

“What do you mean?”

“You actually will have to do some research.”

“Wow! What does that involve? What do you mean by research, anyway? And how does one do it? What kinds of research

are there?”

To fi nd out the answers to Molly’s questions, as well as a few others, read this chapter.

“

fra97851_ch01_001-024.indd 3fra97851_ch01_001-024.indd 3 12/21/10 6:35 PM12/21/10 6:35 PM

4 P A R T 1 Introduction to Research www.mhhe.com/fraenkel8e

range of human knowing. To obtain reliable knowledge, therefore, we cannot rely on our senses alone but must check what we think we know with other sources.

AGREEMENT WITH OTHERS

One such source is the opinions of others. Not only can we share our sensations with others, we can also check on the accuracy and authenticity of these sensations: Does this soup taste salty to you? Isn’t that John over there? Did you hear someone cry for help? Smells like mustard, doesn’t it?

Obviously, there is a great advantage to checking with others about whether they see or hear what we do. It can help us discard what is untrue and manage our lives more intelligently by focusing on what is true. If, while hiking in the country, I do not hear the sound of an approaching automobile but several of my compan- ions do and alert me to it, I can proceed with caution. All of us frequently discount our own sensations when others report that we are missing something or “seeing” things incorrectly. Using agreement with others as a means of obtaining information, the tenth-grade biology teacher in Atlanta, for example, might check with her colleagues to see if they fi nd discussions more effective than lectures in motivating their students to learn.

The problem with such common knowledge is that it, too, can be wrong. A majority vote of a committee is no guarantee of the truth. My friends might be wrong about the presence of an approaching automobile, or the auto- mobile they hear may be moving away from rather than toward us. Two groups of eyewitnesses to an accident may disagree as to which driver was at fault. Hence, we need to consider some additional ways to obtain reliable knowledge.

EXPERT OPINION

Perhaps there are particular individuals we should consult—experts in their fi eld, people who know a great deal about what we are interested in fi nding out. We are likely to believe a noted heart specialist, for example, if he says that Uncle Charlie has a bad heart. Surely, a person with a PhD in economics knows more than most of us do about what makes the economy tick. And shouldn’t we believe our family dentist if she tells us that back molar has to be pulled? To use expert opin- ion as a means of obtaining information, perhaps the physical education teacher in Tulsa should ask a noted authority in the physical education fi eld whether ability in one sport correlates with ability in another.

How can educators, parents, and students obtain the infor- mation they need? Many ways of obtaining information, of course, exist. One can consult experts, review books and articles, question or observe colleagues with relevant experience, examine one’s own past experience, or even rely on intuition. All these approaches suggest possible ways to proceed, but the answers they provide are not always reliable. Experts may be mistaken; source docu- ments may contain no insights of value; colleagues may have no experience in the matter; and one’s own experi- ence or intuition may be irrelevant or misunderstood.

This is why a knowledge of scientifi c research method- ology can be of value. The scientifi c method provides us with another way of obtaining information—information that is as accurate and reliable as we can get. Let us com- pare it, therefore, with some of the other ways of knowing.

Why Research Is of Value

Ways of Knowing

SENSORY EXPERIENCE

We see, we hear, we smell, we taste, we touch. Most of us have seen fi reworks on the Fourth of July, heard the whine of a jet airplane’s engines overhead, smelled a rose, tasted chocolate ice cream, and felt the wetness of a rainy day. The information we take in from the world through our senses is the most immediate way we have of know- ing something. Using sensory experience as a means of obtaining information, the director of the gifted-student program mentioned above, for example, might visit an advanced placement English class to see and hear what happens during a week or two of the semester.

Sensory data, to be sure, can be refi ned. Seeing the tem- perature on an outdoor thermometer can refi ne our knowl- edge of how cold it is; a top-quality stereo system can help us hear Beethoven’s Fifth Symphony with greater clarity; similarly, smell, taste, and touch can all be enhanced, and usually need to be. Many experiments in sensory percep- tion have revealed that we are not always wise to trust our senses too completely. Our senses can (and often do) de- ceive us: The gunshot we hear becomes a car backfi ring; the water we see in the road ahead is but a mirage; the chicken we thought we tasted turns out to be rabbit.

Sensory knowledge is undependable; it is also incom- plete. The data we take in through our senses do not ac- count for all (or even most) of what we seem to feel is the

fra97851_ch01_001-024.indd 4fra97851_ch01_001-024.indd 4 12/21/10 6:35 PM12/21/10 6:35 PM

C H A P T E R 1 The Nature of Research 5

Well, maybe. It depends on the credentials of the ex- perts and the nature of the question about which they are being consulted. Experts, like all of us, can be mistaken. For all their study and training, what experts know is still based primarily on what they have learned from reading and thinking, from listening to and observing others, and from their own experience. No expert, however, has studied or experienced all there is to know in a given fi eld, and thus even an expert can never be totally sure. All any expert can do is give us an opinion based on what he or she knows, and no matter how much this is, it is never all there is to know. Let us consider, then, another way of knowing: logic.

LOGIC

We also know things logically. Our intellect—our ca- pability to reason things out—allows us to use sensory data to develop a new kind of knowledge. Consider the famous syllogism:

All human beings are mortal. Sally is a human being. Therefore, Sally is mortal.

To assert the fi rst statement (called the major prem- ise ), we need only generalize from our experience about the mortality of individuals. We have never experienced anyone who was not mortal, so we state that all human beings are. The second statement (called the minor premise ) is based entirely on sensory experience. We come in contact with Sally and classify her as a human being. We don’t have to rely on our senses, then, to know that the third statement (called the conclusion ) must be true. Logic tells us it is. As long as the fi rst two statements are true, the third statement must be true.

Take the case of the counselor in Philadelphia who is asked to advise a student on how to improve her study habits. Using logic, she might present the following argu- ment: Students who take notes on a regular basis in class fi nd that their grades improve. If you take notes on a regu- lar basis, then your grades should improve as well.

This is not all there is to logical reasoning, of course, but it is enough to give you an idea of another way of knowing. There is a fundamental danger in logical rea- soning, however: It is only when the major and minor premises of a syllogism are both true that the conclusion is guaranteed to be true. If either of the premises is false, the conclusion may or may not be true. *

There is still another way of knowing to consider: the method of science.

THE SCIENTIFIC METHOD

When many people hear the word science, they think of things like white lab coats, laboratories, test tubes, or space exploration. Scientists are people who know a lot, and the term science suggests a tremendous body of knowledge. What we are interested in here, however, is science as a method of knowing. It is the scientific method that is important to researchers.

What is this method? Essentially it involves testing ideas in the public arena. Almost all of us humans are capable of making connections—of seeing relation- ships and associations—among the sensory information we experience. Most of us then identify these connec- tions as “facts”—items of knowledge about the world in which we live. We may speculate, for example, that our students may be less attentive in class when we lecture than when we engage them in discussion. A physician may guess that people who sleep between six and eight hours each night will be less anxious than those who sleep more or less than that amount. A counselor may feel that students read less than they used to because they spend most of their free time watching television. But in each of these cases, we do not really know if our belief is true. What we are dealing with are only guesses or hunches, or as scientists would say, hypotheses.

What we must do now is put each of these guesses or hunches to a rigorous test to see if it holds up under more controlled conditions. To investigate our specu- lation on attentiveness scientifi cally, we can observe carefully and systematically how attentive our students are when we lecture and when we hold a class discus- sion. The physician can count the number of hours indi- viduals sleep, then measure and compare their anxiety levels. The counselor can compare the reading habits of students who watch different amounts of television.

Such investigations, however, do not constitute sci- ence unless they are made public. This means that all aspects of the investigation are described in suffi cient detail so that the study can be repeated by anyone who questions the results—provided, of course, that those interested possess the necessary competence and re- sources. Private procedures, speculations, and conclu- sions are not scientifi c until they are made public.

There is nothing very mysterious, then, about how scientists work in their quest for reliable knowledge. In reality, many of us proceed this way when we try to

*In the note-taking example, the major premise (all students who take notes on a regular basis in class improve their grades) is probably not true.

fra97851_ch01_001-024.indd 5fra97851_ch01_001-024.indd 5 12/21/10 6:35 PM12/21/10 6:35 PM

6 P A R T 1 Introduction to Research www.mhhe.com/fraenkel8e

reach  an intelligent decision about a problem that is bothering us. These procedures can be boiled down to fi ve distinct steps.

1. First, there is a problem of some sort—some distur- bance in our lives that disrupts the normal or desir- able state of affairs. Something is bothering us. For most of us who are not scientists, it may be a tension of some sort, a disruption in our normal routine. Ex- amples would be if our students are not as attentive as we wish or if we have diffi culty making friends. To the professional scientist, it may be an unex- plained discrepancy in one’s fi eld of knowledge, a gap to be closed. Or it could be that we want to un- derstand the practice of human sacrifi ce in terms of its historical signifi cance.

2. Second, steps are taken to defi ne more precisely the problem or the questions to be answered, to become clearer about exactly what the purpose of the study is. For example, we must think through what we mean by student attentiveness and why we consider it insuffi - cient; the scientist must clarify what is meant by human sacrifi ce (e.g., how does it differ from murder?).

3. Third, we attempt to determine what kinds of infor- mation would solve the problem. Generally speak- ing, there are two possibilities: study what is already known or carry out a piece of research. As you will see, the fi rst is a prerequisite for the second; the sec- ond is a major focus of this text. In preparation, we must be familiar with a wide range of possibilities for obtaining information, so as to get fi rsthand in- formation on the problem. For example, the teacher might consider giving a questionnaire to students or having someone observe during class. The scien- tist might decide to examine historical accounts or spend time in societies where the practice of human sacrifi ce exists (or has until recently). Spelling out the details of information gathering is a major aspect of planning a research study.

4. Fourth, we must decide, as far as it is possible, how we will organize the information that we obtain. It is not uncommon, in both daily life and research, to discover that we cannot make sense of all the infor- mation we possess (sometimes referred to as infor- mation overload ). Anyone attempting to understand another society while living in it has probably ex- perienced this phenomenon. Our scientist will surely encounter this problem, but so will our teacher unless she has fi gured out how to handle the questionnaire and/or observational information that is obtained.

5. Fifth, after the information has been collected and analyzed, it must be interpreted. While this step may seem straightforward at fi rst, this is seldom the case. As you will see, one of the most important parts of research is to avoid kidding ourselves. The teacher may conclude that her students are inattentive be- cause they dislike lectures, but she may be misinter- preting the information. The scientist may conclude that human sacrifi ce is or was a means of trying to control nature, but this also may be incorrect.

In many studies, there are several possible explana- tions for a problem or phenomenon. These are called hy- potheses and may occur at any stage of an investigation. Some researchers state a hypothesis (e.g., “Students are less attentive during lectures than during discussions”) right at the beginning of a study. In other cases, hy- potheses emerge as a study progresses, sometimes even when the information that has been collected is being analyzed and interpreted. The scientist might fi nd that instances of sacrifi ce seemed to be more common after such societies made contact with other cultures, sug- gesting a hypothesis such as: “Sacrifi ce is more likely when traditional practices are threatened.”

We want to stress two crucial features of scientifi c research: freedom of thought and public procedures. At every step, it is crucial that the researcher be as open as humanly possible to alternative ways of focusing and clarifying the problem, collecting and analyzing infor- mation, and interpreting results. Further, the process must be as public as possible. It is not a private game to be played by a group of insiders. The value of scientifi c research is that it can be replicated (i.e., repeated) by anyone interested in doing so. *

The general order of the scientifi c method, then, is as follows:

Identifying a problem or question Clarifying the problem Determining the information needed and how to

obtain it Organizing the information Interpreting the results

In short, the essence of all research originates in curiosity—a desire to fi nd out how and why things

*This is not to imply that replicating a study is a simple matter. It may require resources and training—and it may be impossible to repeat any study in exactly the same way it was done originally. The important principle, however, is that public evidence (as opposed to private experience) is the criterion for belief.

fra97851_ch01_001-024.indd 6fra97851_ch01_001-024.indd 6 12/21/10 6:35 PM12/21/10 6:35 PM

C H A P T E R 1 The Nature of Research 7

happen, including why people do the things they do, as well as whether or not certain ways of doing things work better than others.

A common misperception of science fosters the idea that there are fi xed, once-and-for-all answers to particular questions. This contributes to a common, but unfortunate, tendency to accept, and rigidly adhere to, oversimplifi ed solutions to very complex problems. While certainty is appealing, it is contradictory to a fundamental premise of science: All conclusions are to be viewed as tentative and subject to change, should new ideas and new evidence warrant revision. It is particularly important for educa- tional researchers to keep this in mind, since the demand for fi nal answers from parents, administrators, teachers, and politicians can often be intense. An example of how science changes is shown in the More About Research box on page 8.

For many years, there has been a strong tendency in Western culture to value scientifi c information over all other kinds. In recent years, the limitations of this view have become increasingly recognized and discussed. In education, we would argue that other ways of knowing, in addition to the scientifi c, should at least be considered.

As we have seen, there are many ways to collect information about the world around us. Figure 1.1 on page 10 illustrates some of these ways of knowing.

The term research can mean any sort of “careful, systematic, patient study and investigation in some fi eld of knowledge. 1 Basic research is concerned with clarifying underlying processes, with the hypothesis usually expressed as a theory. Researchers engaged in basic research studies are not particularly interested in examining the effectiveness of specifi c educational practices. An example of basic research might be an attempt to refi ne one or more stages of Erickson’s psy- chological theory of development. Applied research , on the other hand, is interested in examining the effec- tiveness of particular educational practices. Research- ers engaged in applied research studies may or may not want to investigate the degree to which certain theories are useful in practical settings. An example might be an attempt by a researcher to fi nd out whether a particular theory of how children learn to read can be applied to fi rst graders who are non-readers. Many studies com- bine the two types of research. An example would be a study that examines the effects of particular teacher behaviors on students while also testing a theory of personality.

Many methodologies fi t within the framework of research. If we learn how to use more of these meth- odologies where they are appropriate and if we can be- come more knowledgeable in our research efforts, we can obtain more reliable information upon which to base our educational decisions. Let us look, therefore, at some of the research methodologies we might use. We shall return to each of them in greater detail in Parts 4 and 5.

QUANTITATIVE AND QUALITATIVE RESEARCH

Another distinction involves the difference between quantitative and qualitative research . Although we shall discuss the basic differences between these two types of research more fully in Chapter 18, we will provide a brief overview here. In the simplest sense, quantitative data deal primarily with numbers, whereas qualitative data primarily involve words. But this is too simple and too brief. Quantitative and qualitative meth- ods differ in their assumptions about the purpose of re- search itself, methods utilized by researchers, kinds of studies undertaken, the role of the researcher, and the degree to which generalization is possible.

Quantitative researchers usually base their work on the belief that facts and feelings can be separated, that the world is a single reality made up of facts that can be discovered.

Types of Research All of us engage in actions that have some of the char- acteristics of formal research, although perhaps we do not realize this at the time. We try out new methods of teaching, new materials, new textbooks. We com- pare what we did this year with what we did last year. Teachers frequently ask students and colleagues their opinions about school and classroom activities. Coun- selors interview students, faculty, and parents about school activities. Administrators hold regular meet- ings to gauge how faculty members feel about various issues. School boards query administrators, admin- istrators query teachers, teachers query students and each other.

We observe, we analyze, we question, we hypoth- esize, we evaluate. But rarely do we do these things systematically. Rarely do we observe under controlled conditions. Rarely are our instruments as accurate and reliable as they might be. Rarely do we use the vari- ety of research techniques and methodologies at our disposal.

fra97851_ch01_001-024.indd 7fra97851_ch01_001-024.indd 7 12/21/10 6:35 PM12/21/10 6:35 PM

8

MORE ABOUT RESEARCH

revolution in science during the twentieth century (the theory of relativity and the discovery of quantum mechanics being the fi rst two), but that it helps to make sense out of what we view as some implications for educational research. What are these implications? *

If chaos theory is correct, the diffi culty in discovering widely generalizable rules or laws in education, let alone the social sciences in general, may not be due to inadequate concepts and theories or to insuffi ciently precise measure- ment and methodology, but may simply be an unavoidable fact about the world. Another implication is that whatever “laws” we do discover may be seriously limited in their applicability—across geography, across individual and/ or group differences, and across time. If this is so, chaos theory provides support for researchers to concentrate on studying topics at the local level—classroom, school, agency—and for repeated studies over time to see if such laws hold up.

Another implication is that educators should pay more at- tention to the intensive study of the exceptional or the unusual, rather than treating such instances as trivial, incidental, or “er- rors.” Yet another implication is that researchers should focus on predictability on a larger scale—that is, looking for pat- terns in individuals or groups over larger units of time. This would suggest a greater emphasis on long-term studies rather than the easier-to-conduct (and cheaper) short-time investiga- tions that are currently the norm.

Not surprisingly, chaos theory has its critics. In education, the criticism is not of the theory itself, but more with misin- terpretations and/or misapplications of it. † Chaos theorists do not say that all is chaos; quite the contrary, they say that we must pay more attention to chaotic phenomena and revise our conceptions of predictability. At the same time, the laws of gravity still hold, as, with less certainty, do many generaliza- tions in education.

Chaos Theory

T he origins of what is now known as chaos theory are usu-ally traced to the 1970s. Since then, it has come to occupy a prominent place in mathematics and the natural sciences and, to a lesser extent, in the social sciences.

Although the physical sciences have primarily been known for their basic laws, or “fi rst principles,” it has long been known by scientists that most of these laws hold precisely only under ideal conditions that are not found in the “real” world. Many phenomena, such as cloud formations, water- fall patterns, and even the weather, elude precise prediction. Chaos theorists argue that the natural laws that are so useful in science may, in themselves, be the exception rather than the rule.

Although precise prediction of such phenomena as the swing of a pendulum or what the weather will be at a particular time is in most cases impossible, repeated patterns, according to a major principle of chaos theory, can be discovered and used, even when the content of the phenomena is chaotic. De- velopments in computer technology, for example, have made it possible to translate an extremely long sequence of “data points,” such as the test scores of a large group of individu- als, into colored visual pictures of fascinating complexity and beauty. Surprisingly, these pictures show distinct patterns that are often quite similar across different content areas, such as physics, biology, economics, astronomy, and geography. Even more surprising is the fi nding that certain patterns recur as these pictures are enlarged. The most famous example is the “Mandlebrot Bug,” shown in Photographs 1.1 and 1.2. Note that Photograph 1.2 is simply a magnifi cation of a portion of Photograph 1.1. The tiny box in the lower left corner of Pho- tograph 1.1 is magnifi ed to produce the box in the upper left- hand corner of Photograph 1.2. The tiny box within this box is then, in turn, magnifi ed to produce the larger portion of Photo- graph 1.2, including the reappearance of the “bug” in the lower right corner. The conclusion is that even with highly complex data (think of trying to predict the changes that might occur in a cloud formation), predictability exists if patterns can be found across time or when the scale of a phenomenon is increased.

IMPLICATIONS FOR EDUCATIONAL RESEARCH

We hope that this brief introduction has not only stimulated your interest in what has been called, by some, the third

*For more extensive implications in the fi eld of psychology, see M. P. Duke (1994). Chaos theory and psychology: Seven proposi- tions. Genetic, Social and General Psychology Monographs, 120: 267–286.

†See W. Hunter, J. Benson, and D. Garth (1997). Arrows in time: The misapplication of chaos theory to education. Journal of Curricu- lum Studies, 29: 87–100.

fra97851_ch01_001-024.indd 8fra97851_ch01_001-024.indd 8 12/21/10 6:35 PM12/21/10 6:35 PM

9

Photographs 1.1 and 1.2 The Mandlebrot Bug Source: Heinz-Otto Peitgen and Peter H. Richter (1986). The beauty of fractals. Berlin: Springer-Verlag.

fra97851_ch01_001-024.indd 9fra97851_ch01_001-024.indd 9 12/21/10 6:35 PM12/21/10 6:35 PM

10 P A R T 1 Introduction to Research www.mhhe.com/fraenkel8e

Qualitative researchers, on the other hand, assume that the world is made up of multiple realities, socially constructed by different individual views of the same situation.

When it comes to the purpose of research, quantita- tive researchers seek to establish relationships between variables and look for and sometimes explain the causes of such relationships. Qualitative researchers, on the other hand, are more concerned with understanding situ- ations and events from the viewpoint of the participants.

Accordingly, the participants often tend to be directly involved in the research process itself.

Quantitative research has established widely agreed- on general formulations of steps that guide researchers in their work. Quantitative research designs tend to be preestablished. Qualitative researchers have a much greater fl exibility in both the strategies and techniques they use and the overall research process itself. Their designs tend to emerge during the course of the research.

Sensing

Sharing information with others

Being told something by an expert

If the cat is in the basket and the basket is in the box, the cat therefore has to be in the box.

BoxLogical reasoning

Science

“Boy, that’s hot!”

“I’ve concluded that Mr. Johnson has

Lyman disease.”

“Based on what his x-rays and

his blood sample reveal, both of which can be

corroborated by anyone.”

“On what basis?”

Figure 1.1 Ways of Knowing

fra97851_ch01_001-024.indd 10fra97851_ch01_001-024.indd 10 12/21/10 6:36 PM12/21/10 6:36 PM

C H A P T E R 1 The Nature of Research 11

The ideal researcher role in quantitative research is that of a detached observer, whereas qualitative re- searchers tend to become immersed in the situations in which they do their research. The prototypical study in the quantitative tradition is the experiment; for qualita- tive researchers, it is an ethnography.

Lastly, most quantitative researchers want to estab- lish generalizations that transcend the immediate situa- tion or particular setting. Qualitative researchers, on the other hand, often do not even try to generalize beyond the particular situation, but may leave it to the reader to assess applicability. When they do generalize, their generalizations are usually very limited in scope.

Many of the distinctions just described, of course, are not absolute. Sometimes researchers will use both qual- itative and quantitative approaches in the same study. This kind of research is referred to as mixed-methods research . Its advantage is that by using multiple methods, researchers are better able to gather and analyze consid- erably more and different kinds of data than they would be able to using just one approach. Mixed-methods studies can emphasize one approach over the other or give each approach roughly equal weight.

Consider an example. It is often common in surveys to use closed-ended questions that lend themselves to quanti- tative analysis (such as through the calculation of percent- ages of different types of responses), but also open-ended questions that permit qualitative analysis (such as follow- ing up a response that interviewees give to a particular question with further questions by the researcher in order to encourage them to elaborate and explain their thinking).

Studies in which researchers use both quantitative and qualitative methods are becoming more common, as we will see in Chapter 23.

EXPERIMENTAL RESEARCH

Experimental research is the most conclusive of sci- entifi c methods. Because the researcher actually estab- lishes different treatments and then studies their effects, results from this type of research are likely to lead to the most clear-cut interpretations.

Suppose a history teacher is interested in the follow- ing question: How can I most effectively teach impor- tant concepts (such as democracy or colonialism) to my students? The teacher might compare the effectiveness of two or more methods of instruction (usually called the independent variable ) in promoting the learning of historical concepts. After systematically assigning stu- dents to contrasting forms of history instruction (such

as inquiry versus programmed units), the teacher could compare the effects of these contrasting methods by testing students’ conceptual knowledge. Student learn- ing in each group could be assessed by an objective test or some other measuring device. If the average scores on the test (usually called the dependent variable ) dif- fered, they would give some idea of the effectiveness of the various methods. A simple graph could be plotted to show the results, as illustrated in Figure 1.2 .

In the simplest sort of experiment, two contrasting methods are compared and an attempt is made to control for all other (extraneous) variables—such as student abil- ity level, age, grade level, time, materials, and teacher characteristics—that might affect the outcome under in- vestigation. Methods of such control could include hold- ing the classes during the same or closely related periods of time, using the same materials in both groups, compar- ing students of the same age and grade level, and so on.

Of course, we want to have as much control as pos- sible over the assignment of individuals to the various treatment groups, to ensure that the groups are similar. But in most schools, systematic assignment of students to treatment groups is diffi cult, if not impossible, to achieve. Nevertheless, useful comparisons are still pos- sible. You might wish to compare the effect of differ- ent teaching methods (lectures versus discussion, for example) on student achievement or attitudes in two or more intact history classes in the same school. If a

Figure 1.2 Example of Results of Experimental Research: Effect of Method of Instruction on History Test Scores a a Many of the examples of data presented throughout this text, including that shown in Figure 1.2 , are hypothetical. When actual data are shown, the source is indicated.

Lecture

Discussion

Method of instruction (independent variable) x-axis

Te st

s co

re (d

ep en

de nt

v ar

ia bl

e) y-

ax is

fra97851_ch01_001-024.indd 11fra97851_ch01_001-024.indd 11 12/21/10 6:36 PM12/21/10 6:36 PM

12 P A R T 1 Introduction to Research www.mhhe.com/fraenkel8e

difference exists between the classes in terms of what is being measured, this result can suggest how the two methods compare, even though the exact causes of the difference would be somewhat in doubt. We discuss this type of experimental research in Chapter 13.

Another form of experimental research, single-subject research , involves the intensive study of a single indi- vidual (or sometimes a single group) over time. These designs are particularly appropriate when studying indi- viduals with special characteristics by means of direct ob- servation. We discuss this type of research in Chapter 14.

CORRELATIONAL RESEARCH

Another type of research is done to determine relation- ships among two or more variables and to explore their implications for cause and effect; this is called correla- tional research . This type of research can help us make more intelligent predictions.

For instance, could a math teacher predict which sorts of individuals are likely to have trouble learning the subject matter of algebra? If we could make fairly accurate predictions in this regard, then perhaps we could suggest some corrective measures for teachers to use to help such individuals so that large numbers of “algebra-haters” are not produced.

How do we do this? First, we need to collect vari- ous kinds of information on students that we think are related to their achievement in algebra. Such informa- tion might include their performance on a number of tasks logically related to the learning of algebra (such as computational skills, ability to solve word problems, and understanding of math concepts), their verbal abili- ties, their study habits, aspects of their backgrounds, their early experiences with math courses and math teachers, the number and kinds of math courses they’ve taken, and anything else that might conceivably point to how those students who do well in math differ from those who do poorly.

We then examine the data to see if any relationships exist between some or all of these characteristics and subsequent success in algebra. Perhaps those who per- form better in algebra have better computational skills or higher self-esteem or receive more attention from the teacher. Such information can help us predict more ac- curately the likelihood of learning diffi culties for certain types of students in algebra courses. It may even sug- gest some specifi c ways to help students learn better.

In short, correlational research seeks to investigate the extent to which one or more relationships of some

type exist. The approach requires no manipulation or intervention on the part of the researcher other than ad- ministering the instrument(s) necessary to collect the data desired. In general, one would undertake this type of research to look for and describe relationships that may exist among naturally occurring phenomena, with- out trying in any way to alter these phenomena. We talk more about correlational research in Chapter 15.

CAUSAL-COMPARATIVE RESEARCH

Another type of research is intended to determine the cause for or the consequences of differences between groups of people; this is called causal-comparative research . Suppose a teacher wants to determine whether students from single-parent families do more poorly in her course than students from two-parent families. To in- vestigate this question experimentally, the teacher would systematically select two groups of students and then as- sign each to a single- or two-parent family—which is clearly impossible (not to mention unethical!).

To test this question using a causal-comparative de- sign, the teacher might compare two groups of students who already belong to one or the other type of family to see if they differ in their achievement. Suppose the groups do differ. Can the teacher defi nitely conclude that the difference in family situation produced the dif- ference in achievement? Alas, no. The teacher can con- clude that a difference does exist but cannot say for sure what caused the difference.

Interpretations of causal-comparative research are limited, therefore, because the researcher cannot say con- clusively whether a particular factor is a cause or a result of the behavior(s) observed. In the example presented here, the teacher cannot be certain whether (1) any per- ceived difference in achievement between the two groups is due to the difference in home situation, (2) the parent status is due to the difference in achievement between the two groups (although this seems unlikely), or (3) some unidentifi ed factor is at work. Nevertheless, despite prob- lems of interpretation, causal-comparative studies are of value in identifying possible causes of observed varia- tions in the behavior patterns of students. In this respect, they are very similar to correlational studies. We discuss causal-comparative research in Chapter 16.

SURVEY RESEARCH

Another type of research obtains data to determine specifi c characteristics of a group. This is called

fra97851_ch01_001-024.indd 12fra97851_ch01_001-024.indd 12 12/21/10 6:36 PM12/21/10 6:36 PM

C H A P T E R 1 The Nature of Research 13

survey research . Take the case of a high school prin- cipal who wants to fi nd out how his faculty feels about his administrative policies. What do they like about his policies? What do they dislike? Why? Which policies do they like the best or least?

These sorts of questions can best be answered through a variety of survey techniques that measure faculty attitudes toward the policies of the administra- tion. A descriptive survey involves asking the same set of questions (often prepared in the form of a written questionnaire or ability test) of a large number of indi- viduals either by mail, by telephone, or in person. When answers to a set of questions are solicited in person, the research is called an interview. Responses are then tabu- lated and reported, usually in the form of frequencies or percentages of those who answer in a particular way to each of the questions.

The diffi culties involved in survey research are mainly threefold: (1) ensuring that the questions are clear and not misleading, (2) getting respondents to answer questions thoughtfully and honestly, and (3) getting a suffi cient number of the questionnaires completed and returned to enable making meaning- ful analyses. The big advantage of survey research is that it has the potential to provide us with a lot of information obtained from quite a large sample of individuals.

If more details about particular survey questions are desired, the principal (or someone else) can con- duct personal interviews with faculty. The advantages of an interview (over a questionnaire) are that open- ended questions (those requiring a response of some length) can be used with greater confi dence, particular questions of special interest or value can be pursued in depth, follow-up questions can be asked, and items that are unclear can be explained. We discuss survey research in Chapter 17.

ETHNOGRAPHIC RESEARCH

In all the examples presented so far, the questions being asked involve how well, how much, or how effi ciently knowledge, attitudes, or opinions and the like exist or are being developed. Sometimes, however, research- ers may wish to obtain a more complete picture of the educational process than answers to the above ques- tions provide. When they do, some form of qualitative research is called for. Qualitative research differs from the previous (quantitative) methodologies in both its methods and its underlying philosophy. In Chapter 18,

we discuss these differences, along with recent efforts to reconcile the two approaches.

Consider the subject of physical education. Just how do physical education teachers teach their subject? What kinds of things do they do as they go about their daily routine? What sorts of things do students do? In what kinds of activities do they engage? What explicit and implicit rules of games in PE classes seem to help or hinder the process of learning?

To gain some insight into such concerns, an ethno- graphic study can be conducted. The emphasis in this type of research is on documenting or portraying the everyday experiences of individuals by observing and interviewing them and relevant others. An elementary classroom, for example, might be observed on as regular a basis as possible, and the students and teacher involved might be interviewed in an attempt to describe, as fully and as richly as possible, what goes on in that classroom. Descriptions (a better word might be portrayals ) might depict the social atmosphere of the classroom; the in- tellectual and emotional experiences of students; the manner in which the teacher acts toward and reacts to stu- dents of different ethnicities, sexes, or abilities; how the “rules” of the class are learned, modifi ed, and enforced; the kinds of questions asked by the teacher and students; and so forth. The data could include detailed prose de- scriptions by students of classroom activities, audiotapes of teacher-student conferences, videotapes of classroom discussions, examples of teacher lesson plans and student work, sociograms depicting “power” relationships in the classroom, and fl owcharts illustrating the direction and frequency of certain types of comments (for example, the kinds of questions asked by teacher and students of one another and the responses that different kinds produce).

In addition to ethnographic research, qualitative re- search includes historical research (see Chapter 22) and several other, less commonly used approaches. Casey, 2 for example, has identifi ed 18 types of “narra- tive” methods. In Chapter 18, we discuss four of the most distinctive of these. These include biography, where the researcher focuses on important experiences in the life of an individual and interacts with the person to clarify meanings and interpretations (e.g., a study of the career of a high school principal). In phenomenol- ogy, the researcher focuses on a particular phenomenon (such as school board confl ict), collects data through in-depth interviews with participants, and then identi- fi es what is common to their perceptions. A third ap- proach is the case study, in which a single individual, group, or important example is studied extensively and

fra97851_ch01_001-024.indd 13fra97851_ch01_001-024.indd 13 12/21/10 6:36 PM12/21/10 6:36 PM

14 P A R T 1 Introduction to Research www.mhhe.com/fraenkel8e

varied data are collected and used to formulate interpre- tations applicable to the specifi c case (e.g., a particu- lar school board) or to provide useful generalizations. Lastly, grounded theory emphasizes continual interplay between raw data and the researcher’s interpretations that emerge from the data. Its central purpose is to in- ductively develop a theory from data (e.g., a study of teacher morale in a particular school beginning with in- terviews and other types of data).

HISTORICAL RESEARCH

You are probably already familiar with historical research . In this type of research, some aspect of the past is studied, either by perusing documents of the pe- riod or by interviewing individuals who lived during the time. The researcher then attempts to reconstruct as accurately as possible what happened during that time and to explain why it did.

For example, a curriculum coordinator in a large urban school district might want to know what sorts of arguments have been made in the past as to what should be included in the social studies curriculum for grades K–12. She could read what various social studies and other curriculum theorists have written on the topic and then compare their positions. The major problems in his- torical research are making sure that the documents or in- dividuals really did come from (or live during) the period under study and, once this is established, ascertaining that what the documents or individuals say is true. We discuss historical research in more detail in Chapter 22.

ACTION RESEARCH

Action research differs from all the preceding meth- odologies in two fundamental ways. The fi rst is that generalization to other persons, settings, or situations is of minimal importance. Instead of searching for power- ful generalizations, action researchers (often teachers or other education professionals, rather than profes- sional researchers) focus on getting information that will enable them to change conditions in a particular situation in which they are personally involved. Exam- ples would include improving the reading capabilities of students in a specifi c classroom, reducing tensions between ethnic groups in the lunchroom at a particu- lar middle school, or identifying better ways to serve special education students in a specifi ed school district. Accordingly, any of the methodologies discussed ear- lier may be appropriate.

The second difference involves the attention paid to the active involvement of the subjects in a study (i.e., those on whom data is collected), as well as those likely to be affected by the study’s outcomes. Commonly used terms in action research, therefore, are participants or stakeholders, refl ecting an intent to involve them di- rectly in the research process as part of “the research team.” The extent of participation varies from just help- ing to select instruments and/or collect data to helping to formulate the research purpose and question to actu- ally participating in all aspects of the research investiga- tion from start to fi nish. We discuss action research in some detail in Chapter 24.

EVALUATION RESEARCH

There are many different kinds of evaluations depend- ing on the object being evaluated and the purpose of the evaluation. Evaluation research is usually described as ei- ther formative or summative . Formative evaluations are intended to improve the object being evaluated; they help to form or strengthen it by examining the delivery of the program or technology and the quality of its implementa- tion. In contrast, summative evaluations seek to examine the effects or outcomes of an object by describing what happens after the delivery of the program or technology in order to assess whether the object caused the outcome.

An example of a formative evaluation product is a needs assessment report. A needs assessment determines the appropriate audience for the program, as well as the extent of the need and what might work to meet the need. Summative evaluations can be thought of as either (a) outcome evaluations, which investigate whether the program or technology appeared to have caused demon- strable effects on specifi cally defi ned target outcomes, or (b) impact evaluations, which are broader and attempt to assess the overall effects (intended or unintended) of the program or technology as a whole.

Evaluators ask many different kinds of questions and often use a variety of methods to address them. For example, in summative evaluations, evaluators often use quasi-experimental research designs to assess the hypothesized causal effects of a program. Formative evaluations that examine program implementation may also include analysis of existing data sources, surveys, interviews, observational data, and focus groups. *

*For foundational readings on evaluation research, see: M. Q. Patton (1982). Practical evaluation . Beverly Hills, CA: Sage; also, M. Q. Patton (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.

fra97851_ch01_001-024.indd 14fra97851_ch01_001-024.indd 14 12/21/10 6:36 PM12/21/10 6:36 PM

15

General Research Types

Should Some Research Methods Be Preferred over Others?

R ecently, several researchers * have expressed their concern that the U.S. Department of Education is showing favor- itism toward the narrow view that experimental research is, if

CONTROVERSIES IN RESEARCH

*D. C. Berliner (2002). Educational research: The hardest science of all. Educational Researcher, 31 (8): 18–20; F. E. Erickson and K. Gutierrez (2002). Culture, rigor, and science in educational research. Educational Researcher, 31 (8): 21–24.

†E. A. St. Pierre (2002). Science rejects postmodernism. Educational Researcher, 31 (8): 25.

ALL HAVE VALUE

It must be stressed that each of the research methodolo- gies described so briefl y above has value for us in educa- tion. Each constitutes a different way of inquiring into the realities that exist within our classrooms and schools and into the minds and emotions of teachers, counselors, administrators, parents, and students. Each represents a different tool for trying to understand what goes on, and what works, in schools. It is inappropriate to consider any one or two of these approaches as superior to any of the others. The effectiveness of a particular methodol- ogy depends in large part on the nature of the research question one wants to ask and the specifi c context within which the particular investigation is to take place. We need to gain insights into what goes on in education from as many perspectives as possible, and hence we need to construe research in broad rather than narrow terms.

As far as we are concerned, research in education should ask a variety of questions, move in a variety of directions, encompass a variety of methodologies, and use a variety of tools. Different research orientations, perspectives, and goals should be not only allowed but encouraged. The intent of this book is to help you learn how and when to use several of these methodologies.

DESCRIPTIVE STUDIES

Descriptive studies describe a given state of affairs as fully and carefully as possible. One of the best examples of descriptive research is found in botany and zoology, where each variety of plant and animal species is me- ticulously described and information is organized into useful taxonomic categories.

In educational research, the most common descriptive methodology is the survey, as when researchers sum- marize the characteristics (abilities, preferences, behav- iors, and so on) of individuals or groups or (sometimes) physical environments (such as schools). Qualitative ap- proaches, such as ethnographic and historical methodol- ogies are also primarily descriptive in nature. Examples of descriptive studies in education include identifying the achievements of various groups of students; describing the behaviors of teachers, administrators, or counselors; describing the attitudes of parents; and describing the physical capabilities of schools. The description of phe- nomena is the starting point for all research endeavors.

Descriptive research in and of itself, however, is not very satisfying, since most researchers want to have a more complete understanding of people and things. This requires a more detailed analysis of the various aspects of phenomena and their interrelationships. Advances in biol- ogy, for example, have come about, in large part, as a result of the categorization of descriptions and the subsequent determination of relationships among these categories.

ASSOCIATIONAL RESEARCH

Educational researchers also want to do more than sim- ply describe situations or events. They want to know

not the only, at least the most respectable form of research and the only one worthy of being called scientifi c. Such a prefer- ence has implications for both the funding of school programs and educational research. As one writer commented, “How scared should we be when the federal government endorses a particular view of science and rejects others?” †

It is useful to consider the various research methodolo- gies we have described as falling within one or more general research categories: descriptive, associational, or intervention-type studies.

fra97851_ch01_001-024.indd 15fra97851_ch01_001-024.indd 15 12/21/10 6:36 PM12/21/10 6:36 PM

16 P A R T 1 Introduction to Research www.mhhe.com/fraenkel8e

how (or if), for example, differences in achievement are related to such things as teacher behavior, student diet, student interests, or parental attitudes. By investi- gating such possible relationships, researchers are able to understand phenomena more completely. Further- more, the identifi cation of relationships enables one to make predictions. If researchers know that student interest is related to achievement, for example, they can predict that students who are more interested in a subject will demonstrate higher achievement in that subject than students who are less interested. Research that investigates relationships is often referred to as associational research . Correlational and causal- comparative methodologies are the principal examples of associational research. Other examples include studying relationships (1) between achievement and attitude, between childhood experiences and adult characteristics, or between teacher characteristics and student achievement—all of which are correlational studies—and (2) between methods of instruction and achievement (comparing students who have been taught by each method) or between gender and attitude (comparing attitudes of males and females)—both of which are causal-comparative studies.

As useful as associational studies are, they too are ultimately unsatisfying because they do not permit researchers to “do something” to infl uence or change outcomes. Simply determining that student interest is predictive of achievement does not tell us how to change or improve either interest or achievement, although it does suggest that increasing interest would increase achievement. To fi nd out whether one thing will have an effect on something else, researchers need to conduct some form of intervention study.

INTERVENTION STUDIES

In intervention studies , a particular method or treat- ment is expected to infl uence one or more outcomes. Such studies enable researchers to assess, for example, the effectiveness of various teaching methods, curricu- lum models, classroom arrangements, and other efforts to infl uence the characteristics of individuals or groups. Intervention studies can also contribute to general knowledge by confi rming (or failing to confi rm) theo- retical predictions (for instance, that abstract concepts can be taught to young children). The primary method- ology used in intervention research is the experiment.

Some types of educational research may combine these three general approaches. Although historical,

ethnographic, and other qualitative research method- ologies are primarily descriptive in nature, at times they may be associational if the investigator exam- ines relationships. A descriptive historical study of college entrance requirements over time that exam- ines the relationship between those requirements and achievement in mathematics is also associational. An ethnographic study that describes in detail the daily activities of an inner-city high school and also fi nds a relationship between media attention and teacher morale in the school is both descriptive and asso- ciational. An investigation of the effects of different teaching methods on concept learning that also reports the relationship between concept learning and gender is an example of a study that is both an intervention and an associational-type study.

META-ANALYSIS

Meta-analysis is an attempt to reduce the limitations of individual studies by trying to locate all of the stud- ies on a particular topic and then using statistical means to synthesize the results of these studies. In Chapter 3, we discuss meta-analysis in more detail. In subsequent chapters, we examine in detail the limitations that are likely to be found in various types of research. Some apply to all types, while others are more likely to apply to particular types.

Critical Analysis of Research There are some who feel that researchers who engage in the kinds of research we have just described take a bit too much for granted—indeed, that they make a number of unwarranted (and usually unstated) as- sumptions about the nature of the world in which we live. These critics (usually referred to as critical researchers ) raise a number of philosophical, lin- guistic, ethical, and political questions not only about educational research as it is usually conducted but also about all fi elds of inquiry, ranging from the physical sciences to literature.

In an introductory text, we cannot hope to do justice to the many arguments and concerns these critics have raised over the years. What we can do is provide an introduction to some of the major questions they have repeatedly asked.

fra97851_ch01_001-024.indd 16fra97851_ch01_001-024.indd 16 12/21/10 6:36 PM12/21/10 6:36 PM

C H A P T E R 1 The Nature of Research 17

The fi rst issue is the question of reality: As any be- ginning student of philosophy is well aware, there is no way to demonstrate whether anything “really exists.” There is, for example, no way to prove conclusively to others that I am looking at what I call a pencil (e.g., others may not be able to see it; they may not be able to tell where I am looking; I may be dreaming). Fur- ther, it is easily demonstrated that different individuals may describe the same individual, action, or event quite differently—leading some critics to the conclusion that there is no such thing as reality, only individual (and different) perceptions of it. One implication of this view is that any search for knowledge about the “real” world is doomed to failure.

We would acknowledge that what the critics say is correct: We cannot, once and for all, “prove” anything, and there is no denying that perceptions differ. We would argue, however, that our commonsense notion of reality (that what most knowledgeable persons agree exists is what is real) has enabled humankind to solve many problems—even the question of how to put a man on the moon.

The second issue is the question of communication . Let us assume that we can agree that some things are “real.” Even so, the critics argue that it is virtually im- possible to show that we use the same terms to identify these things. For example, it is well known that the Inuit have many different words (and meanings) for the English word snow . To put it differently, no matter how carefully we defi ne even a simple term such as shoe, the possibility always remains that one person’s shoe is not another’s. (Is a slipper a shoe? Is a shower clog a shoe?) If so much of language is imprecise, how then can relationships or laws—which try to indicate how various terms, things, or ideas are connected—be precise?

Again, we would agree. People often do not agree on the meaning of a word or phrase. We would argue, how- ever (as we think would most researchers), that we can defi ne terms clearly enough to enable different people to agree suffi ciently about what words mean that they can communicate and thus get on with the acquisition of useful knowledge.

The third issue is the question of values. Histori- cally, scientists have often claimed to be value free, that is, “objective,” in their conduct of research. Crit- ics have argued, however, that what is studied in the social sciences, including the topics and questions with which educational researchers are concerned, is never objective but rather socially constructed.

Such things as teacher-student interaction in class- rooms, the performance of students on examinations, the questions teachers ask, and a host of other issues and topics of concern to educators do not exist in a vacuum. They are infl uenced by the society and times in which people live. As a result, such topics and con- cerns, as well as how they are defi ned, inevitably re- fl ect the values of that society. Further, even in the physical sciences, the choice of problems to study and the means of doing so refl ect the values of the researchers involved.

Here, too, we would agree. We think that most re- searchers in education would acknowledge the valid- ity of the critics’ position. Many critical researchers charge, however, that such agreement is not suffi ciently refl ected in research reports. They say that many re- searchers fail to admit or identify “where they are com- ing from,” especially in their discussions of the fi ndings of their research.

The fourth issue is the question of unstated assump- tions. An assumption is anything that is taken for granted rather than tested or checked. Although this issue is similar to the previous issue, it is not limited to values but applies to both general and specifi c assump- tions that researchers make with regard to a particular study. Some assumptions are so generally accepted that they are taken for granted by practically all so- cial researchers (e.g., the sun will come out; the earth will continue to rotate). Other assumptions are more questionable. An example given by Krathwohl 3 clari- fi es this. He points out that if researchers change the assumptions under which they operate, this may lead to different consequences. If we assume, for example, that mentally limited students learn in the same way as other students but more slowly, then it follows that given suffi cient time and motivation, they can achieve as well as other students. The consequences of this view are to give these individuals more time, to place them in classes where the competition is less intense, and to motivate them to achieve. If, on the other hand, we as- sume that they use different conceptual structures into which they fi t what they learn, this assumption leads to a search for simplifi ed conceptual structures they can learn that will result in learning that approximates that of other students. Frequently authors do not make such assumptions clear.

In many studies, researchers implicitly assume that the terms they use are clear, that their samples are appropriate, and that their measurements are ac- curate. Designing a good study can be seen as trying

fra97851_ch01_001-024.indd 17fra97851_ch01_001-024.indd 17 12/21/10 6:36 PM12/21/10 6:36 PM

18 P A R T 1 Introduction to Research www.mhhe.com/fraenkel8e

to reduce these kinds of assumptions to a minimum. Readers should always be given enough information so that they do not have to make such assumptions. Figure 1.3 illustrates how an assumption can often be incorrect.

The fi fth issue is the question of societal con- sequences. Critical theorists argue that traditional research efforts (including those in education) pre- dominantly serve political interests that are, at best, conservative or, at worst, oppressive. They point out that such research is almost always focused on im- proving existing practices rather than raising ques- tions about the practices themselves. They argue that, intentional or not, the efforts of most educational researchers have served essentially to reinforce the status quo. A more extreme position alleges that edu- cational institutions (including research), rather than enlightening the citizenry, have served instead to pre- pare them to be uncritical functionaries in an industri- alized society.

We would agree with this general criticism but note that there have been a number of investigations of the status quo itself, followed by suggestions for improve- ment, that have been conducted and offered by research- ers of a variety of political persuasions.

Let us examine each of these issues in relation to a hypothetical example. Suppose a researcher decides to study the effectiveness of a course in formal logic in improving the ability of high school students to analyze arguments and arrive at defensible conclusions from data. The researcher accordingly designs a study that

is sound enough in terms of design to provide at least a partial answer as to the effectiveness of the course. Let us address the fi ve issues presented above in relation to this study.

1. The question of reality: The abilities in question (analyzing arguments and reaching accurate con- clusions) clearly are abstractions. They have no physical reality per se. But does this mean that such abilities do not “exist” in any way whatsoever? Are they nothing more than artifi cial by-products of our conceptual language system? Clearly, this is not the case. Such abilities do indeed exist in a somewhat limited sense, as when we talk about the “ability” of a person to do well on tests. But is test performance indicative of how well a student can perform in real life? If it is not, is the performance of students on such tests important? A critic might allege that the ability to analyze, for example, is situation specifi c: Some people are good analyzers on tests; others, in public forums; others, of written materials; and so forth. If this is so, then the concept of a general abil- ity to “analyze arguments” would be an illusion. We think a good argument can be made that this is not the case—based on commonsense experience and on some research fi ndings. We must admit, how- ever, that the critic has a point (we don’t know for sure how general this ability is), and one that should not be overlooked.

2. The question of communication: Assuming that these abilities do exist, can we defi ne them well enough so

?

“I copied from

Angela.” “I’m a good guesser.”

“I read the right pages in

the book.” “I knew this stuff from last year.”

“You all did well on the test. You must have studied hard!”

Figure 1.3 Is the Teacher’s Assumption Correct?

fra97851_ch01_001-024.indd 18fra97851_ch01_001-024.indd 18 12/21/10 6:36 PM12/21/10 6:36 PM

C H A P T E R 1 The Nature of Research 19

that meaningful communication can result? We think so, but it is true that even the clearest of defi nitions does not always guarantee meaningful communica- tion. This is often revealed when we discover that the way we use a term differs from how someone else uses the same term, despite previous agreement on a defi nition. We may agree, for example, that a “de- fensible conclusion” is one that does not contradict the data and that follows logically from the data, yet still fi nd ourselves disagreeing as to whether or not a particular conclusion is a defensible one. Debates among scientists often boil down to differences as to what constitutes a defensible conclusion from data.

3. The question of values: Researchers who decide to investigate outcomes such as the ones in this study make the assumption that the outcomes are either desirable (and thus to be enhanced) or undesirable (and thus to be diminished), and they usually point out why this is so. Seldom, however, are the values (of the researchers) that led to the study of a par- ticular outcome discussed. Are these outcomes stud- ied because they are considered of highest priority? because they are traditional? socially acceptable? easier to study? fi nancially rewarding?

The researcher’s decision to study whether a course in logic will affect the ability of students to analyze arguments refl ects his or her values. Both the outcomes and the method studied refl ect Eurocentric ideas of value; the Aristotelian notion of the “ratio- nal man” (or woman) is not dominant in all cultures. Might some not claim, in fact, that we need people in our society who will question basic assumptions more than we need people who can argue well from these assumptions? While researchers probably can- not be expected to discuss such complex issues in every study, these critics render a service by urging all of us interested in research to think about how our values may affect our research endeavors.

4. The question of unstated assumptions: In carrying out such a study, the researcher is assuming not only that the outcome is desirable but that the fi ndings of the study will have some infl uence on educational practice. Otherwise, the study is nothing more than an academic exercise. Educational methods research has been often criticized for leading to suggested practices that, for various reasons, are unlikely to be implemented. While we believe that such studies should still be done, researchers have an obligation to make such assumptions clear and to discuss their reasonableness.

5. The question of societal consequences: Finally, let us consider the societal implications of a study such as this. Critics might allege that this study, while per- haps defensible as a scientifi c endeavor, will have a negative overall impact. How so? First by fostering the idea that the outcome being studied (the ability to analyze arguments) is more important than other outcomes (e.g., the ability to see novel or unusual re- lationships). This allegation has, in fact, been made for many years in education—that researchers have overemphasized the study of some outcomes at the expense of others.

A second allegation might be that such research serves to perpetuate discrimination against the less privileged segments of society. If it is true, as some contend, that some cultures are more “linear” and others more “global,” then a course in formal logic (being primarily linear) may increase the advantage already held by students from the dominant linear culture. 4 It can be argued that a fairer approach would teach a variety of argumentative methods, thereby capitalizing on the strengths of all cultural groups.

To summarize, we have attempted to present the major issues raised by an increasingly vocal part of the research community. These issues involve the nature of reality, the diffi culty of communication, the rec- ognition that values always affect research, unstated assumptions, and the consequences of research for so- ciety as a whole. While we do not agree with some of the specifi c criticisms raised by these writers, we believe the research enterprise is the better for their efforts.

A Brief Overview of the Research Process Regardless of methodology, all researchers engage in a number of similar activities. Almost all research plans include, for example, a problem statement, a hypothe- sis, defi nitions, a literature review, a sample of subjects, tests or other measuring instruments, a description of procedures to be followed, including a time schedule, and a description of intended data analyses. We deal with each of these components in some detail through- out this book, but we want to give you a brief overview of them before we proceed.

fra97851_ch01_001-024.indd 19fra97851_ch01_001-024.indd 19 12/21/10 6:36 PM12/21/10 6:36 PM

20 P A R T 1 Introduction to Research www.mhhe.com/fraenkel8e

Figure 1.4 presents a schematic of the research com- ponents. The solid-line arrows indicate the sequence in which the components are usually presented and de- scribed in research proposals and reports. They also in- dicate a useful sequence for planning a study (that is, thinking about the research problem, followed by the hy- pothesis, followed by the defi nitions, and so forth). The broken-line arrows indicate the most likely departures from this sequence (for example, consideration of instru- mentation sometimes results in changes in the sample; clarifying the question may suggest which type of design is most appropriate). The nonlinear pattern is intended to point out that, in practice, the process does not nec- essarily follow a precise sequence. In fact, experienced researchers often consider many of these components simultaneously as they develop their research plan.

Statement of the research problem: The problem of a study sets the stage for everything else. The problem statement should be accompanied by a description of the background of the problem (what factors caused it to be a problem in the fi rst place) and a rationale or justifi cation for studying it. Any legal or ethical ramifi cations related to the problem should be discussed and resolved.

Formulation of an exploratory question or a hypothesis: Research problems are usually stated as questions, and often as hypotheses. A hypothesis is a prediction, a statement of what specifi c results or outcomes are expected to occur. The hypotheses of a study should clearly indicate any relationships expected between the variables (the factors, characteristics, or conditions) being investigated and be so stated that they can be tested

within a reasonable period of time. Not all studies are hypothesis- testing studies, but many are.

Defi nitions: All key terms in the problem statement and hypothesis should be defi ned as clearly as possible.

Review of the related literature: Other studies re- lated to the research problem should be located and their results briefl y summarized. The literature review (of appropriate journals, reports, mono- graphs, etc.) should shed light on what is already known about the problem and should indicate logically why the proposed study would result in an extension of this prior knowledge.

Sample: The subjects * (the sample ) of the study and the larger group, or population (to whom results are to be generalized), should be clearly identifi ed. The sampling plan (the procedures by which the subjects will be selected) should be described.

Instrumentation: Each of the measuring instru- ments that will be used to collect data from the subjects should be described in detail, and a ratio- nale should be given for its use.

Procedures: The actual procedures of the study— what the researcher will do (what, when, where, how, and with whom) from beginning to end, in the order in which they will occur—should be spelled out in detail (although this is not written in stone). This, of course, is much less feasible and appropriate in a qualitative study. A realistic time schedule outlining when various tasks are to

Procedures/ design

Research problem

Hypotheses or

questions

Instrumentation

Data analysis

Sample

Definitions

Literature review

Figure 1.4 The Research Process

*The term subjects is offensive to some because it can imply that those being studied are deprived of dignity. We use it because we know of no other term of comparable clarity in this context.

fra97851_ch01_001-024.indd 20fra97851_ch01_001-024.indd 20 12/21/10 6:36 PM12/21/10 6:36 PM

C H A P T E R 1 The Nature of Research 21

be started, along with expected completion dates, should also be provided. All materials (e.g., text- books) and/or equipment (e.g., computers) that will be used in the study should also be described. The general design or methodology (e.g., an ex- periment or a survey) to be used should be stated. In addition, possible sources of bias should be

identifi ed, and how they will be controlled should be explained.

Data analysis: Any statistical techniques, both de- scriptive and inferential, to be used in the data analysis should be described. The comparisons to be made to answer the research question should be made clear.

Go back to the INTERACTIVE AND APPLIED LEARNING feature at the beginning of the chapter for a listing of interactive and applied activities. Go to the Online Learning Center at www.mhhe.com/fraenkel8e to take quizzes, practice with key terms, and review chapter content.

WHY RESEARCH IS OF VALUE

• The scientifi c method provides an important way to obtain accurate and reliable information.

WAYS OF KNOWING

• There are many ways to obtain information, including sensory experience, agree- ment with others, expert opinion, logic, and the scientifi c method.

• The scientifi c method is considered by researchers the most likely way to produce reliable and accurate knowledge.

• The scientifi c method involves answering questions through systematic and public data collection and analysis.

TYPES OF RESEARCH

• Some of the most commonly used research methodologies in education are experi- mental research, correlational research, causal-comparative research, survey research, ethnographic research, historical research, and action research.

• Experimental research involves manipulating conditions and studying effects. • Correlational research involves studying relationships among variables within a sin-

gle group and frequently suggests the possibility of cause and effect. • Causal-comparative research involves comparing known groups who have had differ-

ent experiences to determine possible causes or consequences of group membership. • Survey research involves describing the characteristics of a group by means of such

instruments as interview questions, questionnaires, and tests. • Ethnographic research concentrates on documenting or portraying the everyday ex-

periences of people, using observation and interviews. • Ethnographic research is one form of qualitative research. Other common forms of

qualitative research include the case study, biography, phenomenology, and grounded theory.

• A case study is a detailed analysis of one or a few individuals. • Historical research involves studying some aspect of the past.

Main Points

fra97851_ch01_001-024.indd 21fra97851_ch01_001-024.indd 21 12/21/10 6:36 PM12/21/10 6:36 PM

22 P A R T 1 Introduction to Research www.mhhe.com/fraenkel8e

• Action research is a type of research by practitioners designed to help improve their practice.

• Evaluation research aims to improve the object or program being evaluated, usually by strengthening its delivery, implementation, and outcomes.

• Each of the research methodologies described constitutes a different way of inquiring into reality and is thus a different tool for understanding what goes on in education.

GENERAL RESEARCH TYPES

• Individual research methodologies can be classifi ed into general research types. De- scriptive studies describe a given state of affairs. Associational studies investigate relationships. Intervention studies assess the effects of a treatment or method on outcomes.

• Quantitative and qualitative research methodologies are based on different assump- tions; they also differ on the purpose of research, the methods used by researchers, the kinds of studies undertaken, the researcher’s role, and the degree to which gen- eralization is possible.

• Mixed-method research incorporates both quantitative and qualitative approaches. • Meta-analysis attempts to synthesize the results of all the individual studies on a

given topic by statistical means.

CRITICAL ANALYSIS OF RESEARCH

• Critical analysis of research raises basic questions about the assumptions and impli- cations of educational research.

THE RESEARCH PROCESS

• Almost all research plans include a problem statement, an exploratory question or hypothesis, defi nitions, a literature review, a sample of subjects, instrumentation, a description of procedures to be followed, a time schedule, and a description of in- tended data analyses.

action research 14

applied research 7

associational research 16

assumption 17

basic research 7

causal-comparative research 12

chaos theory 8

correlational research 12

critical researcher 16

data analysis 21

descriptive studies 15

ethnographic study 13

evaluation research 14

experimental research 11

formative evaluation 14

historical research 14

hypothesis 20

instruments 20

intervention studies 16

literature review 20

meta-analysis 16

mixed-methods research 11

population 20

problem statement 20

procedure 20

qualitative research 7

quantitative research 7

research 7

sample 20

scientifi c method 5

single-subject research 12

subject 20

summative evaluation 14

survey research 13

variables 20

Key Terms

fra97851_ch01_001-024.indd 22fra97851_ch01_001-024.indd 22 12/21/10 6:36 PM12/21/10 6:36 PM

C H A P T E R 1 The Nature of Research 23

1. “Speculation, procedures, and conclusions are not scientifi c unless they are made public.” Is this true? Discuss.

2. Most quantitative researchers believe that the world is a single reality, whereas most qualitative researchers believe that the world is made up of multiple realities. Which position would you support? Why?

3. Can you think of some other ways of knowing besides those mentioned in this chap- ter? What are they? What, if any, are the limitations of these methods?

4. “While certainty is appealing, it is contradictory to a fundamental premise of sci- ence.” What does this mean? Discuss.

5. Is there such a thing as private knowledge? If so, can you give an example? 6. Many people seem to be uneasy about the idea of research, particularly research in

schools. How would you explain this?

1. Webster’s new world dictionary of the American language, 2nd ed. (1984). New York: Simon and Schuster, p. 1208.

2. K. Casey (1995, 1996). The new narrative research in education. Review of Research in Education, 21: 211–253.

3. D. R. Krathwohl (2009). Methods of educational and social science research, 3rd ed. Long Grove, IL: Waveland Press, p. 91.

4. M. Ramirez and A. Casteneda (1974). Cultural democracy, biocognitive development and education. New York: Academic Press.

For Discussion

Notes

fra97851_ch01_001-024.indd 23fra97851_ch01_001-024.indd 23 12/21/10 6:36 PM12/21/10 6:36 PM

24 P A R T 1 Introduction to Research www.mhhe.com/fraenkel8e

Research Exercise 1: What Kind of Research? Think of a research idea or problem you would like to investigate. Using Problem Sheet 1, briefl y describe the problem in a sentence or two. Then indicate the type of research methodology you would use to investigate this problem. Finally, explain briefl y your reasons for choosing this approach.

Problem Sheet 1

Research Method 1. A possible topic or problem I am thinking of researching is: ____________________

____________________________________________________________________

____________________________________________________________________

2. The specifi c method(s) that seem(s) most appropriate for me to use at this time is/are ( circle all you think are appropriate ):

a. an experiment

b. a survey

c. an ethnography

d. a correlational study

e. a causal-comparative study

f. a case study

g. a content analysis

h. a historical study

i. an action research or teacher research study

j. a program evaluation

3. The overall research approach I am planning to use is ( circle one ):

a. Qualitative b. Quantitative c. Mixed Method

4. My reason(s) for using this approach is/are as follows:

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

An electronic version of this Problem Sheet that you can fi ll in and print, save, or e-mail is available on the Online Learning Center at www.mhhe.com/ fraenkel8e.

fra97851_ch01_001-024.indd 24fra97851_ch01_001-024.indd 24 12/21/10 6:36 PM12/21/10 6:36 PM

In Part 2, we introduce or expand on many of the basic ideas involved in educational

research. These include concepts such as hypotheses, variables, sampling, measure-

ment, validity, reliability, and many others. We also begin to supply you with certain

skills that will enhance your ability to understand and master the research process.

These include such things as how to select a research problem, formulate a hypoth-

esis, conduct a literature search, choose a sample, defi ne words and phrases clearly,

develop a valid instrument, plus many others. Regardless of the methodology a re-

searcher uses, all of these skills are important to master. We also discuss the ethical

implications involved in the conduct of research itself.

The Basics of Educational Research

2P A R T

fra97851_ch02_025-036.indd 25fra97851_ch02_025-036.indd 25 22/12/10 9:35 PM22/12/10 9:35 PM

O B J E C T I V E S Studying this chapter should enable you to:

• Give some examples of potential research problems in education.

• Formulate a research question. • Distinguish between researchable and

nonresearchable questions. • Name fi ve characteristics that good

research questions possess. • Describe three ways to clarify unclear

research questions.

• Give an example of an operational defi nition and explain how such defi nitions differ from other kinds of defi nitions.

• Explain what is meant, in research, by the term “relationship” and give an example of a research question that involves a relationship.

What Is a Research Problem?

Research Questions

Characteristics of Good Research Questions

Research Questions Should Be Feasible

Research Questions Should Be Clear

Research Questions Should Be Signifi cant

Research Questions Often Investigate Relationships

2 The Research Problem “My research question is:

‘What’s the absolute best way to teach history to ninth-

graders?’”

“Sorry, Suzy, but that question, as stated,

isn’t researchable.”

fra97851_ch02_025-036.indd 26fra97851_ch02_025-036.indd 26 22/12/10 9:35 PM22/12/10 9:35 PM

27

quantitative, or mixed method) used to conduct the study rather than the other way around, i.e., the meth- odology determining the question. The following ex- amples of initial research questions in education are not suffi ciently developed for actual use in a research project but would be suitable during the early stage of formulating a research question. An appropriate meth- odology and research paradigm (in parentheses) are provided for each question. Although there are other possible methodologies that might be used, we consider those given here as particularly suitable.

• Does client-centered therapy produce more satisfac- tion in clients than traditional therapy? (experimental research; quantitative)

• What goes on in after-school programs during an av- erage week? (ethnographic research; qualitative)

• Does behavior modifi cation reduce aggression in autistic children? (single-subject experimental re- search; quantitative)

• Do teachers behave differently toward students of different genders? (causal-comparative research; quantitative or mixed method)

• How can we predict which students might have trou- ble learning certain kinds of subject matter? (correla- tional research; quantitative)

• How do parents feel about the school counseling pro- gram? (survey research; quantitative)

What Is a Research Problem? A research problem is exactly that—a problem that someone would like to research. A problem can be any- thing that a person fi nds unsatisfactory or unsettling, a diffi culty of some sort, a state of affairs that needs to be changed, anything that is not working as well as it might. Problems involve areas of concern to researchers, con- ditions they want to improve, diffi culties they want to eliminate, questions for which they seek answers.

For researchers and students alike, research problems are usually identifi ed from several sources including (a)  the research literature—see detailed discussion in Chapter 3 on the role of the literature review in formulat- ing research questions; (b) problems in practice or work- related contexts; and (c) personal biography or history (such as current or past personal experiences or iden- tities, race, ethnicity, gender, class background, family customs, religion, and so forth).

Research Questions Usually a research problem is initially posed as a ques- tion, which serves as the focus of the researcher’s in- vestigation. In our view, the research question should dictate the research type and paradigm (qualitative,

Go to the Online Learning Center at www.mhhe.com/fraenkel8e to:

• Learn More About What Makes a Question Researchable

Go to your online Student Mastery Activities book to do the following activities:

• Activity 2.1: Research Questions and Related Designs • Activity 2.2: Changing General Topics into Research Questions • Activity 2.3: Operational Defi nitions • Activity 2.4: Justifi cation • Activity 2.5: Evaluating Research Questions

INTERACTIVE AND APPLIED LEARNING After, or while, reading this chapter:

R obert Adams, a high school teacher in Omaha, Nebraska, wants to investigate whether the inquiry method will increase the interest of his eleventh-grade students in history. Phyllis Gomez, a physical education teacher in an elementary school in Phoenix, Arizona, wants to fi nd out how her sixth-grade students feel about the new exercise program recently mandated

by the school district. Tami Mendoza, a counselor in a large inner-city high school in San Francisco, wonders whether a client-

centered approach might help ease the hostility that many of her students display during counseling sessions. Each of these

examples presents a problem that could serve as a basis for research. Research problems—the focus of a research investigation—

are what this chapter is about.

fra97851_ch02_025-036.indd 27fra97851_ch02_025-036.indd 27 22/12/10 9:35 PM22/12/10 9:35 PM

28 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

3. Does high school achievement infl uence the aca- demic achievement of university students?

4. What is the best way to teach grammar? 5. What would schools be like today if World War II

had not occurred?

We hope you identifi ed questions 2 and 3 as the two that are researchable. Questions 1, 4, and 5, as stated, cannot be researched. Question 1 is another metaphysi- cal question and, as such, does not lend itself to empiri- cal research (we could ask people if they believe God is good, but that would be another question). Ques- tion 4 asks for the “best” way to do something. Think about this one for a moment. Is there any way we can determine the best way to do anything? To be able to determine this, we must examine every possible alterna- tive, and a moment’s refl ection brings us to the realiza- tion that this can never be accomplished. How would we ever be sure that all possible alternatives have been examined? Question 5 requires the creation of impos- sible conditions. We can, of course, investigate what people think schools would be like. Figure 2.1 illustrates the difference between researchable and nonresearch- able questions.

Characteristics of Good Research Questions Once a research question has been formulated, re- searchers want to turn it into as good a question as pos- sible. Good research questions possess four essential characteristics.

1. The question is feasible (i.e., it can be investigated without expending an undue amount of time, energy, or money).

2. The question is clear (i.e., most people would agree as to what the key words in the question mean).

3. The question is signifi cant (i.e., it is worth investi- gating because it will contribute important knowl- edge about the human condition).

4. The question is ethical (i.e., it will not involve phys- ical or psychological harm or damage to human be- ings or to the natural or social environment of which they are a part). We will discuss the subject of ethics in detail in Chapter 4.

Let us discuss some of these characteristics in more detail.

• Why do fi rst-generation college students have a lower graduation rate at San Simeon University? (case study; mixed method)

• How can principals improve faculty morale at low- performing middle schools? (interview research; qualitative)

What all these questions have in common is that we can collect data of some sort to answer them (at least in part). That’s what makes them researchable. For exam- ple, a researcher can measure the satisfaction levels of clients who receive different methods of therapy. Or re- searchers can observe and interview in order to describe the functioning of an elementary school classroom. To repeat, then, what makes these questions researchable is that some sort of information can be collected to answer them.

There are other kinds of questions, however, that cannot be answered by collecting and analyzing data. Here are two examples:

• Should philosophy be included in the high school curriculum?

• What is the meaning of life?

Why can’t these questions be researched? What about them prevents us from collecting information to answer them? The reason is both simple and straightforward: There is no way to collect information to answer either question. Both questions are, in the fi nal analysis, not researchable.

The fi rst question is a question of value —it implies notions of right and wrong, proper and improper—and therefore does not have any empirical (or observable) referents . There is no way to deal, empirically, with the verb should. How can we empirically determine whether or not something “should” be done? What data could we collect? There is no way for us to proceed. However, if the question is changed to “Do people think philosophy should be included in the high school cur- riculum?” it becomes researchable. Why? Because now we can collect data to help us answer the question.

The second question is metaphysical in nature— that is, transcendental, beyond the physical. Answers to this sort of question lie beyond the accumulation of information.

Here are more ideas for research questions. Which ones (if any) do you think are researchable?

1. Is God good? 2. Are children more engaged when taught by a teacher

of the same gender?

fra97851_ch02_025-036.indd 28fra97851_ch02_025-036.indd 28 22/12/10 9:35 PM22/12/10 9:35 PM

C H A P T E R 2 The Research Problem 29

RESEARCH QUESTIONS SHOULD BE FEASIBLE

Feasibility is an important issue in designing research studies. A feasible question is one that can be investigated with available resources. Some questions (such as those involving space exploration, for example, or the study of the long-term effects of special programs, such as Head Start) require a great deal of time and money; others re- quire much less. Unfortunately, the fi eld of education, unlike medicine, business, law, agriculture, pharmacol- ogy, or the military, has never established an ongoing re- search effort tied closely to practice. Most of the research that is done in schools or other educational institutions is likely to be done by “outsiders”—often university profes- sors and their students—and usually is funded by tempo- rary grants. Thus, lack of feasibility often seriously limits research efforts. Following are two examples of research questions, one feasible and one not so feasible.

Feasible: How do the students at Oceana High School feel about the new guidance program re- cently instituted in the district?

Not so feasible: How would achievement be af- fected by giving each student his or her own lap- top computer to use for a semester?

RESEARCH QUESTIONS SHOULD BE CLEAR

Because the research question is the focus of a re- search investigation, it is particularly important that the

question be clear. What exactly is being investigated? Let us consider two examples of research questions that are not clear enough.

Example 1. “Is a humanistically oriented class- room effective?” Although the phrase humanisti- cally oriented classroom may seem quite clear, many people may not be sure exactly what it means. If we ask, What is a humanistically oriented classroom? we begin to discover that it is not as easy as we might have thought to describe its essential characteristics. What happens in such classrooms that is different from what happens in other classrooms? Do teachers use certain kinds of strategies? Do they lecture? In what sorts of activities do students participate? What do such classrooms look like––how is the seating arranged, for example? What kinds of materials are used? Is there much variation to be found from class- room to classroom in the strategies employed by the teacher or in the sorts of activities in which students engage? Do the kinds of materials available and/or used vary?

Another term in this question is also ambiguous. What does the term effective mean? Does it mean “re- sults in increased academic profi ciency,” “results in happier children,” “makes life easier for teachers,” or “costs less money”? Maybe it means all these things and more.

Figure 2.1 Researchable Versus Nonresearchable Questions

No Not researchable

Yes Researchable

Should I put my youngster in preschool?

What is the best way to learn to read?

Are some people born bad?

Do children enrolled in preschool develop better social skills than children

not enrolled?

At which age is it more helpful to introduce phonics

to children—age 5, age 6, or age 7?

Who commits more crimes—poor people

or rich people?

fra97851_ch02_025-036.indd 29fra97851_ch02_025-036.indd 29 22/12/10 9:35 PM22/12/10 9:35 PM

30 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

Example 2. “How do teachers feel about special classes for the educationally handicapped?” The fi rst term that needs clarifi cation is teachers. What age group does this involve? What level of experience (i.e., are probation- ary teachers, for example, included)? Are teachers in both public and private schools included? Are teachers through- out the nation included, or only those in a specifi c locality? Does the term refer to teachers who do not teach special classes as well as those who do?

The phrase feel about is also ambiguous. Does it mean opinions? emotional reactions? Does it suggest actions? or what? The terms special classes and edu- cationally handicapped also need to be clarifi ed. An example of a legal defi nition of an educationally handi- capped student is:

A minor who, by reason of marked learning or behavioral disorders, is unable to adapt to a normal classroom situa- tion. The disorder must be associated with a neurological handicap or an emotional disturbance and must not be due to mental retardation, cultural deprivation, or foreign language problems.

Note that this defi nition itself contains some ambigu- ous words, such as marked learning disorders, which lend themselves to a wide variety of interpretations. This is equally true of the term cultural deprivation, which is not only ambiguous but also often offensive to members of ethnic groups to whom it is frequently applied.

As we begin to think about these (or other) questions, it appears that terms which seemed at fi rst glance to be words or phrases that everyone would easily understand are re- ally quite complex and far more diffi cult to defi ne than we might originally have thought.

This is true of many current educational concepts and methodologies. Consider such terms as core curric- ulum, client-centered counseling, active learning, and quality management. What do such terms mean? If you were to ask a sample of fi ve or six teachers, counselors, or administrators, you probably would get several dif- ferent defi nitions. Although such ambiguity is valuable in some circumstances and for certain purposes, it repre- sents a problem to investigators of a research question. Researchers have no choice but to be specifi c about the terms used in a research question, to defi ne precisely what is to be studied. In making this effort, researchers gain a clearer picture of how to proceed with an investi- gation and, in fact, sometimes decide to change the very nature of the research. How, then, might the clarity of a research question be improved?

Defining Terms. There are essentially three ways to clarify important terms in a research question. The fi rst is to use a constitutive definition —that is, to use what is often referred to as the dictionary approach. Re- searchers simply use other words to say more clearly what is meant. Thus, the term humanistic classroom might be defi ned as

A classroom in which: (1) the needs and interests of students have the highest priority; (2) students work on their own for a considerable amount of time in each class period; and (3) the teacher acts as a guide and a resource person rather than an informant.

Notice, however, that this defi nition is still some- what unclear, since the words being used to explain the term humanistic are themselves ambiguous. What does it mean to say that the “needs and interests of students have the highest priority” or that “students work on their own”? What is a “considerable amount” of each class period? What does a teacher do when acting as a “guide” or a “resource person”? Further clarifi cation is needed.

Students of communication have demonstrated just how diffi cult it is to be sure that the message sent is the message received. It is probably true that no one ever completely understands the meaning of terms that are used to communicate. That is, we can never be certain that the message we receive is the one the sender in- tended. Some years ago, one of the leaders in our fi eld was said to have become so depressed by this idea that he quit talking to his colleagues for several weeks. A more constructive approach is simply to do the best we can. We must try to explain our terms to others. While most researchers try to be clear, there is no question that some do a much better job than others.

© The New Yorker Collection 1998 Tom Cheney from cartoonbank.com. All Rights Reserved.

fra97851_ch02_025-036.indd 30fra97851_ch02_025-036.indd 30 22/12/10 9:35 PM22/12/10 9:35 PM

31

Another important point to remember is that often it is a compound term or phrase that needs to be de- fi ned rather than only a single word. For example, the term nondirective therapy will surely not be clarifi ed by precise defi nitions of nondirective and therapy, since it has a more specifi c meaning than the two words de- fi ned separately would convey. Similarly, such terms as learning disability, bilingual education, interactive video, and home-centered health care need to be de- fi ned as linguistic wholes.

Here are three defi nitions of the term motivated to learn. Which do you think is the clearest?

1. Works hard 2. Is eager and enthusiastic 3. Sustains attention to a task *

As you have seen, the dictionary approach to clari- fying terms has its limitations. A second possibility is clarification by example . Researchers might think of a few humanistic classrooms with which they are familiar and then try to describe as fully as possible what happens in these classrooms. Usually we sug- gest that people observe such classrooms to see for themselves how they differ from other classrooms. This approach also has its problems, however, since our descriptions may still not be as clear to others as they would like.

Thus, a third method of clarifi cation is to defi ne im- portant terms operationally. Operational definitions require that researchers specify the actions or operations necessary to measure or identify the term. For example, here are two possible operational defi nitions of the term humanistic classroom.

1. Any classroom identifi ed by specifi ed experts as constituting an example of a humanistic classroom

2. Any classroom judged (by an observer spending at least one day per week for four to fi ve weeks) to pos- sess all the following characteristics: a. No more than three children working with the

same materials at the same time b. The teacher never spending more than 20 minutes

per day addressing the class as a group c. At least half of every class period open for stu-

dents to work on projects of their own choosing at their own pace

d. Several (more than three) sets of different kinds of educational materials available for every stu- dent in the class to use

e. Nontraditional seating—students sit in circles, small groupings of seats, or even on the fl oor to work on their projects

f. Frequent (at least two per week) discussions in which students are encouraged to give their opin- ions and ideas on topics being read about in their textbooks

The above listing of characteristics and behaviors may be a quite unsatisfactory defi nition of a humanistic classroom to many people (and perhaps to you). But it is considerably more specifi c (and thus clearer) than the defi nition with which we began. † Armed with this defi - nition (and the necessary facilities), researchers could decide quickly whether or not a particular classroom qualifi ed as an example of a humanistic classroom.

Defi ning terms operationally is a helpful way to clarify their meaning. Operational defi nitions are useful tools and should be mastered by all students of research.

*We judge 3 to be the clearest, followed by 1 and then 2.

• Terms that individuals outside the fi eld of study may not understand

• Terms that have multiple meanings • Terms that are essential to understanding what the study

is about • Terms to provide precision in specifi cations for instru-

ments to be developed or located

Key Terms to Defi ne in a Research Study

• Terms necessary to ensure that the research question is sharply focused

RESEARCH TIPS

†This is not to say that this list would not be improved by making the guidelines even more specifi c. These characteristics, however, do meet the criterion for an operational defi nition—they specify the actions researchers need to take to measure or identify the variable being defi ned.

fra97851_ch02_025-036.indd 31fra97851_ch02_025-036.indd 31 22/12/10 9:35 PM22/12/10 9:35 PM

32 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

Remember that the operations or activities necessary to measure or identify the term must be specifi ed. Which of the following possible defi nitions of the term motivated to learn mathematics do you think are operational?

1. As shown by enthusiasm in class 2. As judged by the student’s math teacher using a rat-

ing scale she developed 3. As measured by the “Math Interest” questionnaire 4. As shown by attention to math tasks in class 5. As refl ected by achievement in mathematics 6. As indicated by records showing enrollment in

mathematics electives 7. As shown by effort expended in class 8. As demonstrated by number of optional assignments

completed 9. As demonstrated by reading math books outside class

10. As observed by teacher aides using the “Mathemat- ics Interest” observation record *

In addition to their value in helping readers under- stand how researchers actually obtain the information they need, operational defi nitions are often helpful in clarifying terms. Thinking about how to measure job satisfaction, for example, is likely to force a researcher to clarify, in his or her own mind, what he or she means by the term. (For everyday examples of times when operational defi nitions are needed, see Figure 2.2 .)

Despite their virtues, however, operational defi nitions in and of themselves are often not illuminating. Reading that “language profi ciency is (operationally) defi ned as

*The operational defi nitions are 2, 3, 6, 8, and 10. The nonopera- tional defi nitions are 1, 4, 5, 7, and 9, because the activities or opera- tions necessary for identifying the behavior have not been specifi ed.

Figure 2.2 Some Times When Operational Defi nitions Would Be Helpful

“Margaret really Mrs. Robinson.”

likes

Potential Buyer

“I want to buy an

inexpensive car.”

Car Salesman

“Susan’s teacher says her work isn’t artistic.”

“I disagree! She’s very creative!”

Painting

“Under $10,000.”

“Up to $25,000!”

“Hi”

“Jamie is really smart,

isn’t he?”

“He does get really good

grades.” “No, he has no common sense

at all.”

“No way! He was the only one who got lost on

our camping trip.”

“Nah. He’s no good in woodshop.”

“He knows how to please people, that’s for sure.”

fra97851_ch02_025-036.indd 32fra97851_ch02_025-036.indd 32 22/12/10 9:35 PM22/12/10 9:35 PM

33

the student’s score on the TOLD test” is not very help- ful unless the reader is familiar with this particular test. Even when this is the case, it is more satisfactory to be informed of what the researcher means by the term. For these reasons we believe that an operational defi nition should always be accompanied by a constitutive one.

The importance of researchers being clear about the terms in their research questions cannot be overstated. Researchers will have diffi culty proceeding with plans for the collection and analysis of data if they do not know exactly what kind of data to look for. And they will not know what data to look for if they are unclear about the meaning of the key terms in the research question.

RESEARCH QUESTIONS SHOULD BE SIGNIFICANT

Research questions also should be worth investigat- ing. In essence, we need to consider whether getting the answer to a question is worth the time and energy (and often money). What, we might ask, is the value of investigating a particular question? In what ways will it contribute to our knowledge about education? to our knowledge of human beings? Is such knowledge im- portant in some way? If so, how? These questions ask researchers to think about why a research question is worthwhile—that is, important or signifi cant.

It probably goes without saying that a research ques- tion is of interest to the person who asks it. But is in- terest alone suffi cient justifi cation for an investigation? For some people, the answer is a clear yes. They say that any question that someone sincerely wants an an- swer to is worth investigating. Others, however, say that personal interest, in and of itself, is an insuffi cient reason. Too often, they point out, personal interest can result in the pursuit of trivial or insignifi cant questions. Because most research efforts require some (and often

a considerable) expenditure of time, energy, materials, money, and/or other resources, it is easy to appreciate the point of view that some useful outcome or payoff should result from the research. The investment of one- self and others in a research enterprise should contribute some knowledge of value to the fi eld of education.

Generally speaking, most researchers do not believe that research efforts based primarily on personal interest alone warrant investigation. Furthermore, there is some reason to question a “purely curious” motive on psycho- logical grounds. Most questions probably have some de- gree of hidden motivation behind them, and for the sake of credibility, these reasons should be made explicit.

One of the most important tasks for any researcher, therefore, is to think through the value of the intended research before too much preliminary work is done. Three important questions should be asked:

1. How might answers to this research question ad- vance knowledge in my fi eld?

2. How might answers to this research question im- prove educational practice?

3. How might answers to this research question im- prove the human condition?

As you think about possible research questions, ask yourself: Why would it be important to answer this question? Does the question have implications for the improvement of practice? for administrative decision making? for program planning? Is there an important issue that can be illuminated to some degree by a study of this question? Is it related to a current theory that I have doubts about or would like to substantiate? Thinking through possible answers to these questions can help you judge the signifi cance of a potential research question.

In our experience, student justifi cations for a pro- posed study are likely to have two weaknesses. First, they assume too much—for example, that everyone

trivial. Some recipients complained of “cheap shots,” arguing that their research had not received a complete or fair hear- ing. While it is doubtless true that research is often specialized in nature and not easily communicated to persons outside the fi eld, we believe more attention should be paid to:

• Avoiding esoteric terminology • Defi ning key terms clearly and, when feasible, both consti-

tutively and operationally • Making a clear and persuasive case for the importance of

a study

The Importance of a Rationale

R esearch in education, as in all of social science, has some-times been criticized as trivial. Some years ago, Senator William Proxmire gained considerable publicity for his “golden fl eece” awards, which he bestowed on government- funded studies that he considered particularly worthless or

MORE ABOUT RESEARCH

fra97851_ch02_025-036.indd 33fra97851_ch02_025-036.indd 33 22/12/10 9:35 PM22/12/10 9:35 PM

34 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

would agree with them (i.e., it is self-evident) that it is important to study something like self-esteem or abil- ity to read. In point of fact, not everyone does agree that these are important topics to study; nonetheless, it is still the researcher’s job to make the case that they are important rather than merely assuming that they are.

Second, students often overstate the implications of a study. Evidence of the effectiveness of a particu- lar teaching method does not, for example, imply that the method will be generally adopted or that improve- ment in student achievement will automatically result. It would imply, for example, that more attention should be given to the method in teacher-training programs.

RESEARCH QUESTIONS OFTEN INVESTIGATE RELATIONSHIPS

There is an additional characteristic that good research questions often possess. They frequently (but not always) suggest a relationship of some sort to be investigated. (We discuss the reasons for this in Chapter 5.) A suggested re- lationship means that two qualities or characteristics are tied together or connected in some way. Are motivation and learning related? If so, how? What about age and

attractiveness? speed and weight? height and strength? a principal’s administrative policies and faculty morale?

It is important to understand how the term relation- ship is used in research, since the term has other mean- ings in everyday life. When researchers use the term relationship, they are not referring to the nature or qual- ity of an association between people, for example. What we and other researchers mean is perhaps best clarifi ed visually. Look, for example, at the data for groups A and B in Figure 2.3 . What do you notice?

The hypothetical data for group A show that out of a total of 32 individuals, 16 are Republicans and 16 are Democrats. It also shows that half are male and half are female. Group B shows the same breakdown by party af- fi liation and gender. What is different between the two groups is that there is no association or relationship be- tween gender and political party in group A, whereas there is a very strong relationship between these two fac- tors in group B. We can express the relationship in group B by saying that males tend to be Republicans while fe- males tend to be Democrats. We can also express this re- lationship in terms of a prediction. Should another female join group B, we would predict she would be a Democrat since 14 of the previous 16 females are Democrats.

Figure 2.3 Illustration of Relationship Between Voter Gender and Party Affi liation

Group A: No relationship

Male Female D

em oc

ra t

Re pu

bl ic

an

16 16 16 16

16

16

Group B: Relationship

Male Female

D em

oc ra

t Re

pu bl

ic an

16

16

Go back to the INTERACTIVE AND APPLIED LEARNING feature at the beginning of the chapter for a listing of interactive and applied activities. Go to the Online Learning Center at www.mhhe.com/fraenkel8e to take quizzes, practice with key terms, and review chapter content.

fra97851_ch02_025-036.indd 34fra97851_ch02_025-036.indd 34 22/12/10 9:35 PM22/12/10 9:35 PM

C H A P T E R 2 The Research Problem 35

RESEARCH PROBLEM

• A research problem is the focus of a research investigation.

RESEARCH QUESTIONS

• Many research problems are stated as questions. • The essential characteristic of a researchable question is that there be some sort of

information that can be collected in an attempt to answer the question.

CHARACTERISTICS OF GOOD RESEARCH QUESTIONS

• Research questions should be feasible—that is, capable of being investigated with available resources.

• Research questions should be clear—that is, unambiguous. • Research questions should be signifi cant—that is, worthy of investigation. • Research questions should be ethical—that is, their investigation should not involve

physical or psychological harm or damage to human beings or to the natural or social environment of which they are a part.

• Research questions often (although not always) suggest a relationship to be investi- gated. The term relationship, as used in research, refers to a connection or associa- tion between two or more characteristics or qualities.

DEFINING TERMS IN RESEARCH

• Three common ways to clarify ambiguous or unclear terms in a research question involve the use of constitutive (dictionary-type) defi nitions, defi nition by example, and operational defi nitions.

• A constitutive defi nition uses additional terms to clarify meaning. • An operational defi nition describes how examples of a term are to be measured or

identifi ed.

Main Points

clarifi cation by example 31 Key Terms

1. Here are three examples of research questions. How would you rank them on a scale of 1 to 5 (5 5 highest, 1 5 lowest) for clarity? for signifi cance? Why? a. How many students in the sophomore class signed up for a course in driver train-

ing this semester? b. Why do so many students in the district say they dislike English? c. Is inquiry or lecture more effective in teaching social studies?

2. How would you defi ne humanistically oriented classroom? 3. Some terms used frequently in education, such as motivation, achievement, and even

learning, are very hard to defi ne clearly. Why do you suppose this is so? 4. How might the term excellence be defi ned operationally? Give an example. 5. “Even the clearest of defi nitions does not always guarantee meaningful communica-

tion.” Is this really true? Why or why not? 6. We would argue that operational defi nitions should always be accompanied by con-

stitutive defi nitions. Would you agree? Can you think of an instance when this might not be necessary?

7. Most researchers do not believe that research efforts based primarily on personal interest warrant investigation. Do you agree in all cases? Can you think of a possible exception?

For Discussion

constitutive defi nition 30

empirical referent 28

operational defi nition 31

fra97851_ch02_025-036.indd 35fra97851_ch02_025-036.indd 35 22/12/10 9:35 PM22/12/10 9:35 PM

36 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

Research Exercise 2: The Research Question Using Problem Sheet 2, restate the research problem you listed in Research Exercise 1 in a sen- tence or two, and then formulate a research question related to this problem. Now list all the key terms in the question that you think are not clear and need to be defi ned. Defi ne each of these terms both constitutively and operationally, and then state why you think your question is an important one to study.

Problem Sheet 2

The Research Question 1. My (restated) research problem is: ________________________________________

____________________________________________________________________

2. My research question(s) is/are: ___________________________________________

____________________________________________________________________

3. The following are key terms in the problem or question that are not clear and thus need to be defi ned:

4. Here are my constitutive defi nitions for these terms: __________________________

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

5. Here are my operational defi nitions for these terms: ___________________________

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

6. My rationale for investigating this question/problem ( why I would argue that it is an important question to investigate) is as follows: ______________________________

____________________________________________________________________

____________________________________________________________________

a.

b.

c.

d.

e.

f.

An electronic version of this Problem Sheet that you can fi ll in and print, save, or e-mail is available on the Online Learning Center at www.mhhe.com/ fraenkel8e.

fra97851_ch02_025-036.indd 36fra97851_ch02_025-036.indd 36 22/12/10 9:35 PM22/12/10 9:35 PM

O B J E C T I V E S Studying this chapter should enable you to:

• Describe briefl y why a literature review is of value.

• Name the steps a researcher goes through in conducting a review of the literature.

• Describe briefl y the kinds of information contained in a general reference and give an example of such a source.

• Explain the difference between a primary and a secondary source and give an example of each type.

• Explain what is meant by the phrase “search term” and how it differs from the

term “descriptor,” and how both terms are used in literature searches.

• Conduct both a manual and electronic search of the literature on a topic of interest to you after a small amount of “hands-on” computer time and a little help from a librarian.

• Write a summary of your literature review. • Explain what a meta-analysis is.

The Defi nition and Value of a Literature Review

Types of Sources

Steps Involved in a Literature Search

Defi ne the Problem as Precisely as Possible

Look through One or Two Secondary Sources

Select the Appropriate General Reference Tools

Formulate Search Terms Search Using General

Reference Tools

Doing a Computer Search

Obtain Primary Sources

Writing the Literature Review Report

Researching the World Wide Web

Locating and Reviewing the Literature 3

“Hey! There’s Joe. Taking it easy,

as usual.” “No, he says he’s reviewing the literature for his Biology class!”

fra97851_ch03_037-059.indd 37fra97851_ch03_037-059.indd 37 22/12/10 9:36 PM22/12/10 9:36 PM

38

their intended area of study but also to be able to evalu- ate this work in terms of its relevance to the research question of interest.

Types of Sources A researcher needs to be familiar with three basic types of sources as he or she begins to search for information related to the research question. These terms apply both to computerized searching (online or electronic) as well as manual searching (using print/paper tools to locate print/paper sources). Regardless of the tools involved, the search process is similar.

1. General reference tools are the sources researchers often refer to fi rst. In effect, they tell where to look to locate other sources—such as articles, books, re- ports, and other documents—that deal directly with the research question. General reference tools are usually either indexes, which list the author, title, and place of publication of articles and other mate- rials, or abstracts , which give a brief summary or

The Defi nition and Value of a Literature Review A literature review is an assessment of a body (or bod- ies) of literature that pertains to a specifi c question. A literature review is helpful in several ways. It not only helps researchers glean the ideas of others interested in a particular research question (through important research fi ndings and theories), but it also lets them read about the results of similar or related studies. Lit- erature reviews also give researchers ideas about areas where more research needs to be done. They refer to these as “gaps” in the literature. In fact, a detailed litera- ture review is usually required of master’s and doctoral students when they design a thesis. In some graduate programs, students must propose theses or disserta- tions that address gaps in the existing literature. Thus researchers often weigh information from a literature review in light of their own interests and situation. There are two important points here: Researchers need to be able not only to locate other work dealing with

Go to the Online Learning Center at www.mhhe.com/fraenkel8e to:

• Read the Guide to Electronic Research

Go to your online Student Mastery Activities book to do the following activities:

• Activity 3.1: Library Worksheet • Activity 3.2: Where Would You Look? • Activity 3.3: Do a Computer Search of the Literature

INTERACTIVE AND APPLIED LEARNING After, or while, reading this chapter:

A fter a career in the military, Phil Gomez is in his fi rst year as a teacher at an adult school in Logan, Utah. He teaches United States history to students who did not graduate from high school but who now are trying to obtain a diploma. He has learned the hard way, through trial and error, that there are a number of techniques that simply put students to sleep.

He sincerely wants to be a good teacher, but he is having trouble getting his students interested in the subject. As he is the only

history teacher in the school, the other teachers are not of much help.

He wants to get some ideas, therefore, about other approaches, strategies, and techniques that he might use. He decides to

do an Internet search to see what he can fi nd out about effective strategies for teaching high school history. His fi rst search yields

12,847 hits! Phil is overwhelmed and at a loss for which sources to view. Should he look at books? Journal articles? Web sites?

Government documents? Unpublished reports? Where should he look for the most valid resources? And how could his searching

be done more systematically?

In this chapter, you will learn some answers to these (and related) questions. When you have fi nished reading, you should

have a number of ideas about how to conduct a systematic or “planned” search of the educational literature.

fra97851_ch03_037-059.indd 38fra97851_ch03_037-059.indd 38 22/12/10 9:36 PM22/12/10 9:36 PM

C H A P T E R 3 Locating and Reviewing the Literature 39

Researchers who seek information systematically on a given topic would refer fi rst to one or more general reference tools to locate primary and secondary sources of value. For a quick overview of the problem at hand, secondary sources are probably the best bet. For de- tailed information about the research that others have done, primary sources should be consulted.

Today, most researchers search the literature elec- tronically by means of a personal computer. In the past, before the rise of the Internet and the World Wide Web, most searches were done manually. Man- ual searching (using print/paper tools to locate print/ paper sources) is now used primarily by library users interested in locating rare or historical sources. How- ever, some professors also require students to conduct manual searches because not all sources are available electronically. Although the interface may be differ- ent, both processes are the same in terms of the steps involved.

Steps Involved in a Literature Search The following steps are involved in a literature search:

1. Defi ne the research problem as precisely as possible. 2. Look at relevant secondary sources (these can in-

clude research reviews). 3. Select and peruse one or two appropriate general

reference works. 4. Formulate search terms (key words or phrases) per-

tinent to the problem or question of interest. 5. Search for relevant primary sources using appropri-

ate general reference tools. 6. Obtain and read relevant primary sources, and note

and summarize key points in the sources.

Let us consider each of these steps in some detail.

DEFINE THE PROBLEM AS PRECISELY AS POSSIBLE

The fi rst thing a researcher needs to do is to state the research question as specifi cally as possible. General questions such as “What sorts of teaching methods work well in urban classrooms?” or “How can a principal be a more effective leader?” are too fuzzy to be of much help when looking through a general reference. The question of interest should be narrowed down to a specifi c area of

annotation of various publications, as well as their author, title, and place of publication. Historically, indexes and abstracts were only available in paper format, but since the advent of computers and the Internet, most libraries have access to indexes and abstracts through online databases containing elec- tronic indexes, abstracts, dictionaries, and encyclo- pedias. For example, the Current Index to Journals in Education (CIJE) and Resources in Education (RIE), the indexes most frequently used by re- searchers in education, are no longer available as distinct publications in paper format. Instead, since 2002 the information they contain is now only avail- able electronically in ERIC (Education Resources Information Center), an online database of educa- tion research and information sponsored by the U.S. Department of Education and the Institute of Educa- tion Sciences. (We’ll show you how to do an ERIC search of the literature later in this chapter.) Simi- larly, Psychological Abstracts , the general reference most commonly used by researchers in psychology, is now only available through PsycINFO , a com- puter database compiled by the American Psycho- logical Association (APA) that includes abstracts and bibliographic citations for journal articles, eval- uation reports, conference papers and proceedings, speeches, and the like.

2. Primary sources are publications in which re- searchers report the results of their studies directly to the reader. Most primary sources in education are journals, such as the Journal of Educational Re- search or the Journal of Research in Science Teach- ing . These journals are usually published monthly or quarterly, and the articles in them typically report on a particular research study. Most college librar- ies pay for subscriptions to online collections that provide registered students free access to a wide array of online databases, including electronic jour- nals that allow users to download full text articles on demand.

3. Secondary sources refer to publications in which authors describe the work of others. The most com- mon secondary sources in education are textbooks. A textbook in educational psychology, for example, may describe several studies as a way to illustrate various ideas and concepts in psychology. Other commonly used secondary sources include educa- tional encyclopedias, research reviews (usually peer- reviewed journals that publish literature reviews on specifi c topics), and yearbooks.

fra97851_ch03_037-059.indd 39fra97851_ch03_037-059.indd 39 22/12/10 9:36 PM22/12/10 9:36 PM

40 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

Review of Research in Education: Published yearly, each volume contains surveys and syn- theses of research on important topics written by leading educational researchers. RRE is currently available online in ERIC.

Subject Guide to Books in Print (current edition): Each of the above sources contains reviews of re- search on various topics of importance in educa- tion. There are many topics, however, that have not been treated in a recent review. If a research question deals with such a topic, the best chance for locating information discussing research on the topic lies in recent books on the subject. The best source for identifying books that might dis- cuss research on a topic is the current edition of Books in Print , available in both print and elec- tronic formats.

In addition, many professional associations and or- ganizations have published handbooks of research in their fi elds. These include:

• Handbook of Reading Research • Handbook of Research on Curriculum • Handbook of Research on Educational Administration • Handbook of Research on Mathematics Teaching

and Learning • Handbook of Research on School Supervision • Handbook of Research on Multicultural Education • Handbook of Research on Music Teaching and

Learning • Handbook of Research on Social Studies Teaching

and Learning • Handbook of Research on Teacher Education • Handbook of Research on the Teaching of English • Handbook of Research on the Education of Young

Children

Each of these handbooks includes a current sum- mary of research dealing with important topics related to its particular fi eld of study. To locate a handbook in paper format, use your library catalog; to locate a handbook in electronic format, use your library cata- log, database list, and/or electronic journal list. Other places to look for books on a topic of interest are the li- brary catalog and the curriculum department (for text- books) in the library. Education Index and PsycINFO also list newly published professional books in their fi elds.

concern. More specifi c questions, therefore, might be, “Is discussion more effective than showing a video clip in motivating students to learn social studies concepts?” or “What sorts of strategies do principals at high- performing elementary schools use to improve faculty and staff morale?” A serious effort should be made to state the question so that it focuses on the specifi c issue for investigation.

LOOK THROUGH ONE OR TWO SECONDARY SOURCES

Once the research question has been stated in specifi c terms, it is a good idea to look through one or two sec- ondary sources to get an overview of previous work that has been done on the problem. This needn’t be a monu- mental chore nor take an overly long time to complete. The main intent is to get some idea of what is already known about the problem and of some of the other ques- tions that are being asked. Researchers may also get an idea or two about how to revise or improve the research question. Here are some of the most commonly used secondary sources in educational research:

Encyclopedia of Educational Research (current edition online only): Contains brief summaries of over 300 topics in education. Excellent source for getting a brief overview of the problem. The last print edition was published in 2004.

Handbook of Research on Teaching (latest edi- tion published in 2001) : Contains longer articles on various aspects of teaching. Most are written by educational researchers who specialize in the topic on which they are writing.

National Society for the Study of Education (NSSE) Yearbooks: Published every year, these yearbooks deal with recent research on various topics. Each book usually contains from 10 to 12 chapters dealing with various aspects of the topic. The society, which was founded in 1901 and dissolved in 2008, has continued to pub- lish its yearbooks without interruption as part of the Teachers College Record at Columbia University.

Review of Educational Research: Published four times a year by the American Educational Re- search Association (AERA), this journal contains reviews of research and extensive bibliographies on various topics in education, and is available online through ERIC.

fra97851_ch03_037-059.indd 40fra97851_ch03_037-059.indd 40 22/12/10 9:36 PM22/12/10 9:36 PM

C H A P T E R 3 Locating and Reviewing the Literature 41

in ERIC, the number is ED + a six-digit num- ber (see Figure 3.2). ERIC documents (EDs) are documents produced by state departments of edu- cation, fi nal reports of federally funded research projects, reports from school districts, commis- sioned papers written for government agencies, and other published and unpublished documents. Abstracts and bibliographic information are usu- ally provided on all documents. Many reports that would otherwise never be published are reported in ERIC , which makes this an especially valuable resource to use.

PsycINFO: PsycINFO, the electronic version of the now-ceased Psychological Abstracts , is a database containing summaries and citations of literature in the fi eld of psychology dating back to the 1800s (and even some records from the 1700s and 1600s). Produced by the American Psychological Asso- ciation (APA), the largest and most distinguished professional association of psychologists and sci- entists in the world, PsycINFO contains abstracts and bibliographic data of journal articles, book chapters, books, technical reports, and disserta- tions in the social and behavioral sciences, and is available on the association’s APA PsycNET.

Two additional general reference tools that some- times provide information about educational research are the following:

Exceptional Child Education Resources (ECER) online database: ECER is a bibliographic da- tabase produced by the Council for Exceptional Children (its print publication ceased in 2004). ECER provides information about exceptional children from more than 200 journals. Using a format similar to ERIC , it provides author, sub- ject, and title indexes. It is worth consulting if a research topic deals with exceptional children, since it covers several journals not searched for in ERIC .

Social Science Citation Index (SSCI): Another type of citation and indexing service, SSCI of- fers the forward search, a unique feature that can be helpful to researchers. When a researcher has found an article that contains information of in- terest, he or she can locate the author’s name in the SSCI to fi nd out the names of other authors who have cited this same article and the journals in which their articles appeared. These additional

SELECT THE APPROPRIATE GENERAL REFERENCE TOOLS

After reviewing a secondary source to get a more in- formed overview of the problem, researchers should have a clearer idea of exactly what to investigate. At this point, it is a good idea to look again at the research ques- tion to see if it needs to be rewritten in any way to make it more focused. Once satisfi ed, researchers can select one or two general references to help identify particular journals or other primary sources related to the question. Of the many general reference tools a researcher can consult, here is a list of the ones most commonly used:

Education Index: Since 2004, this online-only publication indexes articles from more than 300 educational publications. This electronic index includes three separate databases: (a) Education Index Retrospective, which covers the period 1929–1982; (b) Education Index, which contains sources from 1983 to the present; and, (c) Educa- tion Full Text, which has abstracts and full-text articles dating back to 1983.

Education Resources Information Center (ERIC): ERIC is an online database of education research and information sponsored by the U.S. Depart- ment of Education and the Institute of Education Sciences. It includes indexes and abstracts, journal articles, reports, and other documents in educa- tion, counseling, and related social science disci- plines. In 2002, after the major education indexes formerly known as the Current Index to Journals in Education (CIJE) and Resources in Education (RIE) merged and ceased print publication, ERIC began to offer access to their content electroni- cally. Today, ERIC provides citations and direct access to more than 1.3 million bibliographic sources, including citations to articles from more than 750 journals, as well as unpublished docu- ments including curriculum guides, conference papers, and research reports. Although full-text access to all current education-related sources is not yet possible, ERIC provides users with ab- stracts and exact citation information about the source. Publication information usually includes the following: article title, author, journal name, page, and volume and issue numbers, as well as an ERIC identifying number. For ERIC journal ar- ticles (EJs), the number is EJ 1 a six-digit number (see Figure 3.1); for nonjournal article documents

fra97851_ch03_037-059.indd 41fra97851_ch03_037-059.indd 41 22/12/10 9:36 PM22/12/10 9:36 PM

42 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

original research and hence are valuable sources for literature reviews.

ProQuest Dissertations and Theses: Proquest maintains a digital library that has more than 1.4 mil- lion titles, including abstracts and full text fi les of doctoral dissertations and masters’ theses sub- mitted by more than 1,000 graduate schools and universities in North America, Europe, and Asia. Coverage includes the complete text of most dis- sertations and theses completed from 1988 to the present, in addition to abstracts of theses and dis- sertations dating back to 1861 (see Figure 3.3).

articles may also be of interest to the researcher, particularly in compiling a references list for an annotated bibliography (a list of sources on a topic with brief summaries) or a literature review. He or she can determine what additional books and articles were cited by these other authors and thus conceivably obtain information that other- wise might be missed. Most libraries offer SSCI online searching since it is available as part of the Web of Science database (currently published by Thomson Scientifi c). Most doctoral dissertations and many masters’ theses in education report on

Figure 3.1 Excerpt from ERIC Journal Article

fra97851_ch03_037-059.indd 42fra97851_ch03_037-059.indd 42 22/12/10 9:36 PM22/12/10 9:36 PM

C H A P T E R 3 Locating and Reviewing the Literature 43

and the like. The researcher would then select the appropriate general reference tool.

Indexes and abstracts, whether in electronic or print/ paper format, are designed to present uniform access to citation information. Each citation—whether in a data- base, index, or abstract—contains information unique to the citation (i.e., author, title, publication date, etc.). In addition, each citation is assigned vocabulary words that help categorize related articles. In most databases the assigned vocabulary words are referred to as subject terms or subject headings , and in ERIC these terms are referred to specifi cally as descriptors . Learning which subject terms, subject headings, or descriptors are used in a specifi c system can help researchers more easily iden- tify all related articles on a particular subject or topic.

If using an online database to search the literature, to retrieve results the researcher would enter search terms in the search boxes provided. (see Figure 3.4). Using a

FORMULATE SEARCH TERMS

Once a general reference work has been selected, re- searchers need to formulate some search terms —words or phrases they can use to locate primary sources. Search terms are the most important words in the prob- lem statement. Take, for example, the research question, “Do students taught by a teaching team learn more than students taught by an individual teacher?” What are the most important words—the key terms—in this ques- tion? Remember that a researcher conducts a literature search to fi nd out what other research has been done with regard to—and what others think about—the re- search question of interest. The key term in this ques- tion, therefore, is teaching team. This term, plus other similar or synonymous terms, should be listed. Possi- bilities here might include team teaching, joint teach- ing, cooperative teaching, collaboration and teaching,

Figure 3.2 Excerpt from ERIC Document

fra97851_ch03_037-059.indd 43fra97851_ch03_037-059.indd 43 22/12/10 9:36 PM22/12/10 9:36 PM

44 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

libraries and most public libraries. Many state depart- ments of education also provide access to online educa- tion databases, as do some county offi ces of education and some large school systems. The database most com- monly used by educational researchers is ERIC, which can be searched electronically back to 1966. Other da- tabases include PsycINFO , Exceptional Child Educa- tion Resources , and ProQuest Dissertations & Theses . More than 200 other specialized databases in other sub- ject areas exist; to fi nd out more about them, contact a nearby college or university library and ask for assis- tance from a reference librarian.

paper/print tool, the user would look at a list of subject terms that match the search terms listed in the resource in order to fi nd a list of relevant citations. He or she would then select the articles that seem to bear on the research topic.

SEARCH USING GENERAL REFERENCE TOOLS

Although there is no magic formula to follow, many researchers in education turn to library resources and other information resources available online. An online search of the literature can be performed in databases available through the Web sites of almost all university

Figure 3.3 Excerpt from ProQuest Dissertations and Theses

fra97851_ch03_037-059.indd 44fra97851_ch03_037-059.indd 44 22/12/10 9:36 PM22/12/10 9:36 PM

C H A P T E R 3 Locating and Reviewing the Literature 45

research question: “What sorts of questioning tech- niques help students understand historical concepts most effectively?”

Decide on the Extent of the Search. The researcher must now decide how many references to obtain. For a review for a journal article, a researcher might decide to review only 20 to 25 fairly recent refer- ences. For a more detailed review, such as a master’s thesis, perhaps 30 or 40 might be reviewed. For a very exhaustive review, as for a doctoral dissertation, as many as 100 or more references might be searched.

Decide on the Database. As we mentioned earlier, many databases are available, but the one most commonly used is ERIC. Subject terms or headings may

Doing a Computer Search To illustrate the steps involved in online searching, we will next describe an actual search conducted using the ERIC database.

Define the Problem as Precisely as Possible. The research problem should be stated as specifi cally as possible so that relevant descriptors can be identifi ed. A broad statement of a problem such as, “How effec- tive are questioning techniques?” is much too general. It is liable to produce an extremely large number of references, many of which probably will be irrelevant to the researcher’s question of interest. For the pur- poses of our search, therefore, we posed the following

Figure 3.4 Excerpt from Education Full Text

fra97851_ch03_037-059.indd 45fra97851_ch03_037-059.indd 45 22/12/10 9:36 PM22/12/10 9:36 PM

46 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

questioning techniques or the descriptor term inquiry, as well as either the descriptor term history instruction or the descriptor term civics instruction.

For our search, we chose the following descriptors: questioning techniques, concept teaching, and history instruction. We also considered using a number of re- lated terms. These included inquiry, teaching methods, and learning processes under questioning techniques, and concept formation and cognitive development under concept teaching. Upon refl ection, however, we decided not to include teaching methods or learning processes , as we felt these terms were too broad to apply specifi - cally to our research question. We also decided not to include cognitive development for the same reason.

Conduct the Search. After determining which descriptors to use, the next step is to enter them into the database and let it do its work. Figure 3.6 presents a summary of the search results. As you can see, we asked the database fi rst to search for questioning techniques (search no. 1), followed by history instruction (search no. 2), followed by a combination (search no. 3) of these two descriptors (note the use of the Boolean operator and ). This resulted in a total of 5,044 references for questioning techniques, 18,447 references for history instruction, and 107 for a combination of these two de- scriptors. We then asked the database to search just for the descriptors concept and teaching (search no. 4). This produced a total of 34,349 references. Because we were particularly interested in concept teaching as applied to questioning techniques and history instruction, how- ever, we asked the database to search for a combination (search no. 5) of these three descriptors (again note the use of the operator and ). This produced 12 references. At this point, the 12 references can be printed, saved to a hard drive or portable device, or e-mailed.

If the initial effort of a search produces too few ref- erences, the search can be broadened by using more general descriptors. Thus, we might have used the term social studies instruction rather than history instruction had we not obtained enough references in our search. Similarly, a search can be narrowed by using more spe- cifi c descriptors. For example, we might have used the specifi c descriptor North American history rather than the inclusive term history.

Documenting Citation Information. Once a researcher has located references, he or she needs to document the information found and prepare it for later use. If any articles are found that deal with some

not be applicable to different databases, although many do overlap. In ERIC, as noted earlier, subject terms are referred to as “descriptors.” We used the ERIC database in this example, as it is still the best for searches involv- ing educational topics.

Select Search Words and Discover Descriptors. Researchers often begin a search in ERIC using keywords they use to describe their topic. The researcher types these keywords in ERIC to tell the computer what to search for. The selection of key- words is somewhat of an art form. If the keyword is too general, too many references may be located, many of which are likely to be irrelevant. If the keyword is too narrow, too few references will be located, and many of those applicable to the research question may be missed. Furthermore, if the keyword used is not the same or similar to the descriptors used by the system to describe the topic, then few or no search results will be found. For ERIC users, the ERIC thesaurus provides a list of descriptors commonly used in their databases. Search results in ERIC also list descriptors and subject terms associated with individual citations.

Keywords and descriptors can be used singly or in various combinations to locate references. Certain key words, called Boolean operators, enable the retrieval of terms in various combinations. The most commonly used Boolean operators are and and or. For example, by asking a computer to search for a single keyword or descriptor such as inquiry, all references containing this term would be selected. By connecting two keywords or descriptors with the word and, however, researchers can narrow the search to locate only the references that contain both of the descriptors. Asking the computer to search for questioning techniques and history instruc- tion would narrow the search because only references containing both keywords or descriptors would be lo- cated. On the other hand, by using the word or, a search can be broadened, since any references with either one of the keywords or descriptors would be located. Thus, asking the computer to search for questioning tech- niques or history instruction would broaden the search because references containing either one of these terms would be located. Figure 3.5 illustrates the results of using these Boolean operators.

All sorts of combinations are possible. For example, a researcher might ask the computer to search for ques- tioning techniques or inquiry and history instruction or civics instruction. For a reference to be selected, it would have to contain either the descriptor term

fra97851_ch03_037-059.indd 46fra97851_ch03_037-059.indd 46 22/12/10 9:36 PM22/12/10 9:36 PM

C H A P T E R 3 Locating and Reviewing the Literature 47

(like index cards) or newer tools (like EndNote, Ref- Works, and Zotero) to take notes and record and orga- nize your citations, the important thing is to take care to record the bibliographic information completely and accurately. Nothing is more annoying than trying to fi nd an incorrect reference listed in a bibliography.

Searching ERIC. Researchers today fi nd it easier and quicker to search reference tools (as well as most other references) online. In addition to compiling abstracts and citations, many articles are now instantly available for downloading as Portable Document Format (PDF) fi les. Another important feature of ERIC is that more than one descriptor can be searched at the same time.

Suppose a researcher were interested in fi nding in- formation on the use of questioning in teaching science. A search of the ERIC database using the descriptors

aspect of the researcher’s topic, the author, title, page, publication date, and publication source should be re- corded. Several output options are available, including saving, e-mailing, printing, and exporting references . Researchers should choose the option that contains the most information, as it might prove useful later. Stan- dard information would include citation information, the abstract, and in some cases the complete full text of the document.

Many databases, including some versions of ERIC provide options to save, e-mail, or print citations in specifi c formats, including APA, MLA (Modern Lan- guage Association), University of Chicago, and other academic citation styles used to cite references in the literature. Researchers must develop their own process for saving and documenting information found while completing their research. Whether using classic tools

Select questioning techniques and history instruction (Combine 1 and 2) Decreases output

Narrow search

Broader search

Select questioning techniques or history instruction (Combine 1 or 2) Increases output

Questioning techniques

(Set 1)

History instruction

(Set 2)

History instruction

(Set 2)

Questioning techniques

(Set 1)

Figure 3.5 Venn Diagrams Showing the Boolean Operators AND and OR

fra97851_ch03_037-059.indd 47fra97851_ch03_037-059.indd 47 22/12/10 9:36 PM22/12/10 9:36 PM

48 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

1. Before 1965: search Education Index. 2. From 1966 to 1968: search ERIC and Education

Index. 3. From 1969 to the present: Search ERIC, Education

Index, and other education databases.

OBTAIN PRIMARY SOURCES

After searching the general references, researchers will have a list of bibliographic citations. The next step is to locate each of the sources listed, then read and take notes on those relevant to the research problem. There are two major types of primary sources to be familiar with in this regard—journals and reports.

questioning techniques and science would reveal the ab- stracts and citations of several articles. Notice that the word source indicates where to fi nd the articles if the researcher wants to read all or part of them—one is in the journal Research in Science and Technological Education and the other in International Journal of Science Education.

Searching PsycINFO. Searching through Psyc- INFO is similar to searching through ERIC . As in ERIC, key words or subject terms and descriptors can be used singularly or in various combinations to locate references. All articles of interest can then be located in the identifi ed journals. The best strategy for a thorough search is probably as follows.

Figure 3.6 Summary of Search Results

fra97851_ch03_037-059.indd 48fra97851_ch03_037-059.indd 48 22/12/10 9:36 PM22/12/10 9:36 PM

C H A P T E R 3 Locating and Reviewing the Literature 49

in education databases . An author’s address can some- times be found in the directory of a professional as- sociation, such as the American Educational Research Association Biographical Membership Directory or Who’s Who in American Education. If a reprint or book cannot be obtained directly from the author, it may be possible to obtain it from another library in the area through interlibrary loan, a service that nearly all libraries provide. By entering information into a database, a library user can fi nd out within seconds which libraries within a designated area have a par- ticular book or journal.

Using Secondary Sources to Locate Primary Sources. Although the principal goal of a literature review is to assess original reports of empirical research that have been published mainly in academic journals, secondary sources can be useful. Locating published review articles—either literature reviews or meta- analyses, which we describe in the next section—can give students a sense of the depth and breadth of the lit- erature related to a topic. They only need to add the key- word “review” to the list of search terms or descriptors used in their initial search of the literature to produce extensive reference lists (or bibliographies) contained in research reviews—a technique called “branching.” In addition, landmark studies are considered highly signifi - cant by experts for understanding a topic. While read- ing a review, a student may come across the name of a researcher repeatedly and/or an explicit statement by the author that a study is especially important. In this case, the student should add the landmark source(s) to his or her reading list for inclusion in the literature review.

Researchers often discuss the fi ndings of their empir- ical studies in terms of the theoretical literature. Under- standing the major theories and theorists— particularly landmark theorists—that have contributed to the litera- ture on a topic is paramount for preparing a compre- hensive review. This can be done by adding the term “theory” as a keyword or descriptor in an electronic database search. In general, major theorists conduct research studies themselves, some of which are con- sidered landmark studies. Including a discussion of the major theories and theorists in a literature review pro- vides important contextual information for the review reader to better understand the empirical research fi nd- ings to be evaluated.

Meta-Analysis. In academic journals, the litera- ture reviews accompanying research reports are usually

Professional Journals. Many journals in educa- tion publish reports of research. Some publish articles on a wide range of educational topics, while others limit what they print to a particular specialization, such as social studies education. Most researchers become familiar with the journals in their fi eld of interest and look them over from time to time. Examples of such journals include the American Educational Research Journal, Child Development, Educational Adminis- tration Quarterly, Journal of Educational Research, Journal of Research in Science Teaching, Reading Re- search Quarterly, and Theory and Research in Social Education.

Reports. Many important research fi ndings are fi rst published as reports. Almost all funded research proj- ects produce a fi nal report of their activities and fi nd- ings when research is completed. In addition, each year many reports on research activities are published by the United States government, by state departments of edu- cation, by private organizations and agencies, by local school districts, and by professional associations. Fur- thermore, many individual researchers report on their recent work at professional meetings and conferences.

Most reports are abstracted in ERIC and are available as PDF fi les. Many papers, such as the reports of presiden- tial task forces, national conferences, or specially called professional meetings, are published only as reports. They are usually far more detailed than journal articles and much more up to date. Also, they are not copyrighted. Reports are a very valuable source of up-to-date informa- tion that could not be obtained anywhere else.

Locating Primary Sources. Most primary source material is located in academic journals, since that is where most of the research fi ndings in education are published. Although more and more journals are avail- able online through library Web sites, to conduct a thorough search of relevant primary sources, it may be necessary to search manually for sources only avail- able in print/paper format. Depending on the layout of the library, users can often go right to the stacks where print/paper journals are shelved alphabetically. In some libraries, however, only the librarian can retrieve the journals.

A problem that every researcher faces at one time or another is that a needed book or journal is not avail- able in the library. When this is the case, it can usu- ally be obtained directly from the author. Addresses of authors (e-mail and conventional) are often listed

fra97851_ch03_037-059.indd 49fra97851_ch03_037-059.indd 49 22/12/10 9:36 PM22/12/10 9:36 PM

Evaluating Primary Sources. When all the de- sired journal articles and documents are gathered to- gether, the review can begin. It is a good idea to begin with the most recent articles and work backward. The reason for this is that most of the more recent articles will cite the earlier articles and thus provide a quick overview of previous work. How should an article be read? While there is no one perfect way to do this, here are some helpful suggestions:

Read the abstract or the summary fi rst. This will tell whether the article is worth reading in its entirety. Record the bibliographic data and take notes on the ar- ticle using your preferred note-keeping tool (electronic, manual, or a hybrid). Almost all research articles follow approximately the same format. They usually include an abstract; an introductory section that presents the research problem or question and reviews other related studies; the objectives of the study or the hypotheses to be tested; a description of the research procedures, including the subjects studied, the research design, and the data collection instruments and tools used; the re- sults or fi ndings of the study; a summary (if there is no abstract); and the researcher’s conclusions. Also, make sure to use the branching technique discussed earlier by perusing the references (or bibliography) listed at the end of the article to help you locate other relevant sources.

Be as brief as possible in taking notes, yet do not ex- clude anything that might be important to describe later in the full review. For each of these steps, the following should be noted.

1. Problem: State it clearly. 2. Hypotheses or objectives: List them exactly as stated

in the article. 3. Procedures: List the research methodology used

(experiment, case study, and so on), the number of subjects and how they were selected, and the

required to be brief. Unfortunately, this largely prevents much in the way of critical analysis of individual stud- ies. Furthermore, traditional literature reviews basically depend on the judgment of the reviewer and hence are prone to subjectivity.

In an effort to moderate this subjective tendency and reduce the time required in reviewing many studies on the same topic, the concept of meta-analysis has been developed. 1 In the simplest terms, when a researcher does a meta-analysis, he or she averages the results of the selected studies to get an overall index of outcome or relationship. The fi rst requirement is that results be described statistically, most commonly through the calculation of effect sizes and correlation coeffi cients (we explain both later in the text). In one of the earli- est studies using meta-analysis, 2 375 studies on the ef- fectiveness of psychotherapy were analyzed, leading to the conclusion that the average client was, after therapy, appreciably better off than the average person not in therapy.

As you might expect, this methodology has had widespread appeal in many disciplines—to date, hun- dreds of meta-analyses have been done. Critics raise a number of objections, some of which have been at least partly remedied by statistical adjustments. We think the most serious criticisms are that a poorly de- signed study counts as much as one that has been care- fully designed and executed and that the evaluation of the meaning of the fi nal index remains a judgment call, although an informed one. The former objection can be remedied by deleting “poor” studies, but this brings back the subjectivity meta-analysis was designed to mitigate. It is clear that meta-analysis is here to stay; we agree with those who argue that it cannot replace an informed, careful review of individual studies, how- ever. In any event, the literature review should include a search for relevant meta-analysis reports, as well as individual studies.

• The hypotheses of the study (if there are any) • The methodology the researcher used • A description of the subjects involved • The results • The conclusions • The particular strengths, weaknesses, and limitations of

the study

What a Good Summary of a Journal Article Should Contain

• The problem being addressed • The purpose of the study

RESEARCH TIPS

50

fra97851_ch03_037-059.indd 50fra97851_ch03_037-059.indd 50 22/12/10 9:36 PM22/12/10 9:36 PM

C H A P T E R 3 Locating and Reviewing the Literature 51

to locating and evaluating your sources, the next steps involve organizing, integrating, and synthesizing these sources. This process is inductive and often leads stu- dents to believe they are regressing, rather than progress- ing. As researchers and professors who have written and supervised many student literature reviews, our advice to you is to be patient and fl exible! The process may appear as if you are going backward (not forward), but this is part of the larger process of discovery involved in doing research. Part of this process involves reformulat- ing the main question that guides your literature review oftentimes in the process of reviewing the research . For example, let’s say you begin your literature search inter- ested in the topic of mixed-ability grouping in elemen- tary school classrooms. Your initial query is “What do we know about heterogeneous grouping in elementary schools?” After reviewing the research and familiar- izing yourself with the vocabulary and background re- lated to the topic, your revised question becomes “What are the effects of inclusion practices on elementary school student achievement?” This restated question is much improved because it (a) includes key terms or vocabulary currently used by researchers in the fi eld; (b) helps to clarify the purpose and scope of the literature

kind of instrument (questionnaire, tally sheet, and so on) used. Make note of any unusual techniques employed.

4. Findings: List the major fi ndings. Indicate whether the objectives of the study were attained and whether the hypotheses were supported. Often the fi ndings are summarized in a table.

5. Conclusions: Record or summarize the author’s conclusions. Note your disagreements with the au- thor and the reasons for such disagreement. Note strengths or weaknesses of the study that make the results particularly applicable or limited with regard to your research question. See Figure 3.7 for an ex- ample of a completed note card.

Writing the Literature Review Report Once you have located and evaluated the sources rel- evant to your topic, you are ready for the fi nal steps in preparing your review of the literature. In addition

Figure 3.7 Sample Note Card

fra97851_ch03_037-059.indd 51fra97851_ch03_037-059.indd 51 22/12/10 9:36 PM22/12/10 9:36 PM

52 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

is essential. There are many ways to format refer- ence lists, but the one outlined in the Publication Manual of the American Psychological Association (2009) is particularly easy to use.

RESEARCHING THE WORLD WIDE WEB

The World Wide Web (WWW) is part of the Inter- net, a vast reservoir of information on all sorts of topics in a wide variety of areas. Prior to 1993, the Internet was barely mentioned in the research literature. Today, it cannot be ignored. Despite the fact that ERIC and (on occasion) PsycINFO remain the databases of choice when it comes to research involving most educational topics, researching the Web should also be considered. Space prevents us from describing the Internet in de- tail, but we do wish to point out some of its important features.

Using a Web browser (the computer program that lets you gain access to the WWW), a researcher can fi nd information on almost any topic with just a few clicks of the mouse button. Some of the information on the Web has been classifi ed into indexes , which can be eas- ily searched by going from one category to another. In addition, several search engines are available that are similar in many respects to those we used in our search of the ERIC database. Let us consider both indexes and search engines in a bit more detail.

Indexes. Indexes group Web sites together under similar categories, such as Australian universities, London art galleries, and science laboratories. This is similar to what libraries do when they group similar kinds of information resources together. The results of an index search will be a list of Web sites related to the topic being searched. Figure 3.8 shows the Yahoo! Web page, a particularly good example of an index. If a re- searcher is interested in fi nding the site for a particular university in Australia, for example, he or she should try using an index.

Indexes often provide an excellent starting point for a review of the literature. This is especially true when a researcher does not have a clear idea for a research question or topic to investigate. Browsing through an index can be a profi table source of ideas. Felden offers an illustration:

For a real-world comparison, suppose I need some house- hold hardware of some sort to perform a repair; I may not always know exactly what is necessary to do the job.

review; and (c) examines the topic more deeply by ex- ploring a possible relationship between inclusion prac- tices (the presumed cause) and student achievement (the presumed outcome).

The process of evaluating, integrating, and synthesiz- ing relevant sources in a literature review involves ana- lyzing and categorizing the literature into major topics and subtopics. There are many strategies for organizing the structure of a review. A common one is to include summary tables to provide readers with an overview of the research related to, for example: (a) defi nitions of key constructs and measures; (b) differing research methods used in studies examining the same research question or phenomenon; and (c) key study charac- teristics and fi ndings. Literature reviews may differ in format, but they typically consist of the following fi ve parts.

1. The introduction briefl y describes the nature of the research problem and states the research question. The researcher also explains in this section what led him or her to investigate the question and why it is an important question to investigate.

2. The body of the review briefl y reports what oth- ers have found or thought about the research prob- lem. Related studies are usually discussed together, grouped under subheadings (to make the review easier to read). Major studies are described in more detail, while less important work can be referred to in just a line or two. Often this is done by referring to several studies that reported similar results in a single sentence, somewhat like this: “Several other small-scale studies reported similar results (Avila, 2009; Brown, 2006; Cartwright, 2009; Davis & Lim, 2008; Martinez, 2007).”

3. The summary of the review ties together the main threads revealed in the literature reviewed and pres- ents a composite picture of what is—and is not— known or thought to date. Findings may be tabulated to give readers some idea of how many other re- searchers have reported identical or similar fi ndings or have similar recommendations.

4. Any conclusions the researcher feels are justifi ed based on the state of knowledge revealed in the lit- erature should be included. What does the literature suggest are appropriate courses of action to take to try to solve the problem? And what are other impor- tant research questions that should be examined?

5. A reference list (or bibliography) with full biblio- graphic data for all sources mentioned in the review

fra97851_ch03_037-059.indd 52fra97851_ch03_037-059.indd 52 22/12/10 9:36 PM22/12/10 9:36 PM

C H A P T E R 3 Locating and Reviewing the Literature 53

of the words on them. The search results obtained are usually ranked in order of relevancy (i.e., the number of times the researcher’s search terms appear in a document or how closely the document appears to match one of the key words submitted as query terms by the researcher).

A search engine like Google will search for and fi nd the individual pages of a Web site that match a research- er’s search, even if the site itself has nothing to do with what the researcher is looking for. As a result, one usu- ally has to wade through an awful lot of irrelevant infor- mation. Felden gives us an example:

Returning to the hardware store analogy, if I went to the store in search of some screws for my household project and employed an automatic robot instead of using my na- tive cunning to browse the (well-arranged) aisles, the robot could conceivably return (after perusing the entire store) with everything that had a screw in it somewhere. The set of things would be a wildly disparate collection. It would include all sorts of boxes of screws, some of them maybe even the kind I was looking for, but also a wide array of

I may have a broken part, which I can diligently carry to a hardware store to try to match. Luckily, most hardware stores are fairly well organized and have an assortment of aisles, some with plumbing supplies, others with nails and other fasteners, and others with rope, twine, and other materials for tying things together. Proceeding by general category (i.e., electrical, plumbing, woodworking, etc.), I can go to approximately the right place and browse the shelves for items that may fi t my repair need. I can ex- amine the materials, think over their potential utility, and make my choice. 3

Search Engines. If one wants more specifi c infor- mation, such as biographical information about George Orwell, however, one should use a search engine, be- cause it will search all of the contents of a Web site. Search engines such as Google or the Librarians’ Index to the Internet use software programs (sometimes called spiders or Web crawlers ) that search the entire Internet, looking at millions of Web pages and then indexing all

Figure 3.8 The Yahoo! Web Page Source: Reproduced with permission of YAHOO! Inc. © 2007 by Yahoo! Inc. YAHOO! and the YAHOO! logo are trademarks of Yahoo! Inc.

fra97851_ch03_037-059.indd 53fra97851_ch03_037-059.indd 53 22/12/10 9:36 PM22/12/10 9:36 PM

54 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

• Lack (sometimes) of credibility: Anyone can pub- lish something on the Internet. As a result, much of the material one fi nds there may have little, if any, credibility.

• Uncertain reliability: It is so easy to publish in- formation on the Internet that it often is diffi cult to judge its worth. One of the most valuable aspects of a library collection is that most of its material has been collected carefully. Librarians make it a point to identify and select important works that will stand the test of time. Much of the information one fi nds on the WWW is ill-conceived or trivial.

• Ethical violations: Because material on the Internet is so easy to obtain, there is a greater temptation for researchers to use the material without citation or permission. Copyright violation is much more likely than with traditional material.

• Undue reliance: The amount of information avail- able on the Internet has grown so rapidly in the last few years that some researchers may be misled to think they can fi nd everything they need on the Internet, thereby causing them to ignore other, more traditional sources of information.

In searching the WWW, then, here are a few tips to get the best search results: 5 Many of these would apply to searching ERIC or PsycINFO as well.

• Use the most specifi c key word you can think of. Take some time to list several of the words that are likely to appear on the kind of Web page you have in mind. Then pick the most unusual word from your list. For example, if you’re looking for information about efforts to save tiger populations in Asia, don’t use tigers as your search term. You’ll be swamped with Web pages about the Detroit Tigers, the Princ- eton Tigers, and every other sports team that uses the word tigers in its name. Instead, try searching for a particular tiger species that you know to be on the endangered list— Bengal tiger or Sumatran tiger or Siberian tiger. 6

• Make it a multistep process. Don’t assume that you will fi nd what you want on the fi rst try. Review the fi rst couple of pages of your results. Look particu- larly at the sites that contain the kind of information you want. What unique words appear on those pages? Now do another search using just those words.

• Narrow the fi eld by using just your previous results. If the keywords you choose return too much infor- mation, try a second search of just the results you ob- tained in your fi rst search. This is sometimes referred

other material, much of it of no use for my project. There might be birdhouses of wood held together with screws, tools assembled with screws, a rake with a screw fasten- ing its handle to its prongs. The robot would have done its job properly. It had been given something to match, in this case a screw, and it went out and did its work effi ciently and thoroughly, although without much intelligence. 4

To be satisfi ed with the results of a search, therefore, one needs to know what to ask for and how to phrase the request to increase the chances of getting what is de- sired. If a researcher wants to fi nd out information about universities, but not English universities, for example, he or she should ask specifi cally in that way.

Thus, although it would be a mistake to search only the Web when doing a literature search (thereby ignor- ing a plethora of other material that often is so much better organized), it has some defi nite advantages for some kinds of research. Unfortunately, it also has some disadvantages. Here are some of each:

Advantages of Searching the World Wide Web

• Currency: Many resources on the Internet are up- dated very rapidly; often they represent the very lat- est information about a given topic.

• Access to a wide variety of materials: Many re- sources, including works of art, manuscripts, even entire library collections, can be reviewed at leisure using a personal computer.

• Varied formats: Material can be sent over the Inter- net in different formats, including text, video, sound, and animation.

• Immediacy: The Internet is “open” 24 hours a day. Information can be viewed on one’s own computer and can be examined as desired or saved to a hard drive or disk for later examination and study.

Disadvantages of Searching the World Wide Web

• Disorganization: Unfortunately, much of the infor- mation on the Web is not well organized. It employs few of the well-developed classifi cation systems used by libraries and archives. This disorganization makes it an absolute necessity for researchers to have good online searching skills.

• Time commitment: There is always a need to search continually for new and more complete information. Doing a search on the WWW often (if not usually) can be quite time-consuming and (regretfully) some- times less productive than doing a search using more traditional sources.

fra97851_ch03_037-059.indd 54fra97851_ch03_037-059.indd 54 22/12/10 9:36 PM22/12/10 9:36 PM

C H A P T E R 3 Locating and Reviewing the Literature 55

accountability, curriculum and instruction, fi - nance and grants, and data and statistics that assess needs and measure performance. ( Note: Also, check the departments of education Web sites of other states.)

RAND Education (www.rand.org/education): RAND Education is an education “think tank,” a nonprofi t organization that conducts policy-based research and analysis to address major problems in the educational system. The Web site provides free access to recent reports and literature re- views, as well as links to other RAND publica- tions and books.

The Urban Institute (www.urban.org) : Is a nonpar- tisan think tank that conducts economic and social policy research affecting urban areas, including an Education Policy Center with a strong emphasis on immigrant children, poverty, and health care. Publishes books, as well as studies and reports that are often available on their Web site.

Google Scholar (http://scholar.google.com): Provides a simple way to do a broad search for scholarly literature, including peer-reviewed pa- pers, theses, books, abstracts and articles. Google Scholar should be used as a supplement to, not as a substitute for, searching through academic databases.

Other education databases, available through most libraries, include:

ProQuest Education Journals: Offers access to more than 745 top educational publications, in- cluding nearly 600 titles in full text.

Education Research Complete: Topics covered in- clude all levels of education from early childhood to higher education, and all educational special- ties, such as multilingual education, health educa- tion, and testing.

Education Full Text: Is a bibliographic database produced by The H. W. Wilson Company that lists indexes, abstracts, and full-text articles from more than 350 journals dating back to 1996.

ERIC (EBSCO): References articles in more than 750 professional journals, thousands of unpub- lished research reports, conference papers, and curriculum guides in all areas of education.

Academic Search Premier: Provides full-text ac- cess to nearly 1,560 academic journals in edu- cation, humanities, and the social and physical sciences.

to as set searching . Here’s a tip we think you’ll fi nd extremely helpful: Simply add another keyword to your search request and submit it again.

• Look for your keyword in the Web page title. Fre- quently, the best strategy is to look for your unique keyword in the title of Web pages. If you are look- ing for information about inquiry teaching in sec- ondary school history classes, for example, begin with a search of Web pages that have inquiry teach- ing in the title. Then do a second search of just those results, looking for secondary school history classes.

• Find out if case counts. Check to fi nd out if the search engine you are using pays any attention to upper- and lowercase letters in your keywords. “Will a search for java, a microsystems program, for example, also fi nd sites that refer to the program as JAVA?” 7

• Check your spelling. If you have used the best key- words that you can think of and the search engine reports “No results found” (or something similar), check your spelling before you do anything else. Usually, the fact that a search engine does not come up with any results is due to a spelling or typing error.

• Assess the credibility and reliability of Internet sources . One quick way to evaluate the accuracy and objectivity of information published on the Internet is to check the URL or domain address extension. Web addresses ending in .gov, .edu, and .org are sponsored, respectively, by the federal government, higher education institutions, and nonprofi t organi- zations. Although these resources are not necessarily free of error and bias, compare them to URL exten- sions ending in .com, which represent commercial vendors that often use Web site advertising to gener- ate revenue for profi t. In addition, scan the Web site for the organization’s purpose statement as well as the author’s credentials (and contact information), then judge for yourself.

Public Internet web sites that provide educational re- sources and information include:

The National Center for Education Statistics (http://nces.ed.gov): NCES is located in the U.S. Department of Education and the Institute of Edu- cation Sciences and serves as the primary federal entity for collecting and analyzing data related to education.

California Department of Education (http:// www.cde.ca.gov): Includes information col- lected by California schools on testing and

fra97851_ch03_037-059.indd 55fra97851_ch03_037-059.indd 55 22/12/10 9:36 PM22/12/10 9:36 PM

56 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

central, authoritative resource for information about doctoral dissertations and master’s theses.

Web of Science: The ISI Web of Science is the Web interface for institutional access to the ISI Cita- tion Databases, which cover over 10,000 leading journals and over 100,000 book-based and journal conference proceedings.

JSTOR: Contains the full text of more than 169 national and international journals available from JSTOR, an organization founded in 1995 to promote global scholarship using its digital archives.

ProQuest Dissertations and Theses: With more than 2 million entries, PQD&T is the single,

Go back to the INTERACTIVE AND APPLIED LEARNING feature at the beginning of the chapter for a listing of interactive and applied activities. Go to the Online Learning Center at www.mhhe.com/fraenkel8e to take quizzes, practice with key terms, and review chapter content.

THE VALUE OF A LITERATURE REVIEW • A literature review helps researchers learn what others have written about a topic. It

also lets researchers see the results of other, related studies. • A detailed literature review is often required of master’s and doctoral students when

they design a thesis.

TYPES OF SOURCES FOR A LITERATURE REVIEW

• Researchers need to be familiar with three basic types of sources (general references, primary sources, and secondary sources) in doing a literature review.

• General reference tools are sources a researcher consults to locate other sources. • Primary sources are publications in which researchers report the results of their in-

vestigations. Most primary source material is located in journal articles. • Secondary sources refer to publications in which authors describe the work of others. • The most common secondary sources in education are textbooks. • Search terms are keywords or phrases researchers use to help locate relevant primary

sources.

STEPS INVOLVED IN A LITERATURE SEARCH

• The essential steps involved in a review of the literature include: (1) defi ning the research problem as precisely as possible; (2) deciding on the extent of the search; (3) deciding on the data base(s) to be searched; (4) formulating search terms; (5) searching general reference tools for relevant primary sources; (6) obtaining and reading the primary sources, and noting and summarizing key points in the sources.

WAYS TO DO A LITERATURE SEARCH

• Today, there are two ways to do a literature search—manually, using print/paper tools to locate print/paper sources; and electronically, by means of a computer. The most common and frequently used method, however, is to search online, via com- puter. Regardless of the tools involved, the search process is similar.

• There are fi ve essential points (problem, hypotheses, procedures, fi ndings, and con- clusions) that researchers should record when taking notes on a study.

Main Points

fra97851_ch03_037-059.indd 56fra97851_ch03_037-059.indd 56 22/12/10 9:36 PM22/12/10 9:36 PM

C H A P T E R 3 Locating and Reviewing the Literature 57

DOING A COMPUTER SEARCH

• Computer searches of the literature have a number of advantages—they are fast, are fairly inexpensive, provide printouts, and enable researchers to search using more than one descriptor at a time.

• The steps in a traditional manual search are similar to those in a computer search, though computer searches are usually the norm.

• Researching the World Wide Web (WWW) should be considered, in addition to ERIC and PsycINFO, in doing a literature search.

• Some of the information on the Web is classifi ed into indexes, which group Web sites together under similar categories. Yahoo! is an example of a directory.

• To obtain more specifi c information, search engines should be used, because they search all of the contents of a Web site.

THE LITERATURE REVIEW REPORT

• The literature review report consists of an introduction, the body of the review, a summary, the researcher’s conclusions, and a bibliography.

• A literature review should include a search for relevant meta-analysis reports, as well as individual studies.

• When a researcher does a meta-analysis, he or she averages the results of a group of selected studies to get an overall index of outcome or relationship.

abstract 38

Boolean operators 46

descriptors 43

general reference tools 41

index 52

interlibrary loan 49

Key Terms literature review 38 meta-analysis 50

primary source 39

search engine 52

search terms 43

secondary source 39

subject heading 43

subject terms 43

Web browser 52

World Wide Web 52

1. Why might it be unwise for a researcher not to do a review of the literature before planning a study?

2. Many published research articles include only a few references to related studies. How would you explain this? Is this justifi ed?

3. Which do you think are more important to emphasize in a literature review—the opinions of experts in the fi eld or related studies? Why?

4. One rarely fi nds books referred to in literature reviews. Why do you suppose this is so? Is it a good idea to refer to books?

5. Can you think of any type of information that should not be included in a literature review? If so, give an example.

6. Professor Jones states that he does not have his students do a literature review before planning their master’s theses because they “take too much time,” and he wants them to get started collecting their data as quickly as possible. In light of the information we have provided in this chapter, what would you say to him? Why?

7. Can you think of any sorts of studies that would not benefi t from having the re- searcher conduct a literature review? If so, what might they be?

For Discussion

fra97851_ch03_037-059.indd 57fra97851_ch03_037-059.indd 57 22/12/10 9:36 PM22/12/10 9:36 PM

58 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

1. M. L. Smith, G. V. Glass, and T. I. Miller (1980). Primary, secondary, and meta-analysis research. Educa- tional Researcher, 5 (10): 3–8.

2. Ibid.

3. N. Felden (2000). Internet research: Theory and practice, 2nd ed. London: McFarland, pp. 124–125.

4. Ibid., p. 127.

5. A. Glossbrenner and E. Glossbrenner (1998). Search engines. San Francisco: San Francisco State Univer- sity Press, pp. 11–13.

6. Ibid., p. 12.

7. Ibid.

Notes

fra97851_ch03_037-059.indd 58fra97851_ch03_037-059.indd 58 22/12/10 9:36 PM22/12/10 9:36 PM

C H A P T E R 3 Locating and Reviewing the Literature 59

Research Exercise 3: Review of the Literature Using Problem Sheet 3, list the specifi c problem(s) and/or question(s) you will address in a brief review of the literature related to your study. Indicate what types of sources you did and did not include and why. Then summarize the conclusions you arrived at based on what you found in your review.

Problem Sheet 3

Review of the Literature 1. What are the specifi c problem(s) or question(s) to be addressed in your literature

review?

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

2. What general reference tools did you use to conduct your search? (List specifi c elec- tronic databases consulted.)

____________________________________________________________________

____________________________________________________________________

3. What search terms did you use?

4. Specify the scope of the review and explain your inclusion/exclusion criteria (i.e., what was and was not included and why?).

____________________________________________________________________

____________________________________________________________________

5. What topics and subtopics emerged about your problem and question as you con- ducted your search?

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

6. What are your conclusions based on the fi ndings of your review?

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

An electronic version of this Problem Sheet that you can fi ll in and print, save, or e-mail is available on the Online Learning Center at www.mhhe.com/ fraenkel8e.

a.

b.

c.

d.

e.

f.

fra97851_ch03_037-059.indd 59fra97851_ch03_037-059.indd 59 22/12/10 9:36 PM22/12/10 9:36 PM

O B J E C T I V E S Studying this chapter should enable you to:

• Describe briefl y what is meant by “ethical” research.

• Describe briefl y three important ethical principles recommended for researchers to follow.

• State the basic question with regard to ethics that researchers need to ask before beginning a study.

• State the three questions researchers need to address in order to protect research participants from harm.

• Describe the procedures researchers must follow in order to ensure confi dentiality of data collected in a research investigation.

• Describe when it might be appropriate to deceive participants in a research investigation and the researcher’s responsibilities in such a case.

• Describe the special considerations involved when doing research with children.

Some Examples of Unethical Practice

A Statement of Ethical Principles

Protecting Participants from Harm

Ensuring Confi dentiality of Research Data

When (If Ever) Is Deception of Subjects Justifi ed?

Three Examples Involving Ethical Concerns

Research with Children

Regulation of Research

Academic Cheating and Plagiarism

Ethics and Research 4 “Now, I can’t require you to participate in this study, but if you want to get a good grade

in this course . . .”

“Hey, wait a minute. That’s unethical!”

fra97851_ch04_060-074.indd 60fra97851_ch04_060-074.indd 60 22/12/10 11:00 PM22/12/10 11:00 PM

M ary Abrams and Lamar Harris, both juniors at a large midwestern university, meet weekly for lunch. “I can’t believe it,” Mary says. “What’s the matter?” replies Lamar.

“Professor Thomas says that we have to participate in one of his research projects if we want to pass his course. He says it is a

course requirement. I don’t think that’s right, and I’m pretty upset about it. Can you believe it?”

“Wow. Can he do that? I mean, is that ethical?”

No, it’s not! Mary has a legitimate (and ethical) complaint here. This issue—whether professors can require students to

participate in research projects in order to pass a course—is one example of an unethical practice that sometimes occurs.

The whole question of what is—and what isn’t—ethical is the focus of this chapter.

61

INTERACTIVE AND APPLIED LEARNING After, or while, reading this chapter:

Go to the Online Learning Center at www.mhhe.com/fraenkel8e to:

• Learn More About What Constitutes Ethical Research

Go to your online Student Mastery Activities book to do the following activities:

• Activity 4.1: Ethical or Not? • Activity 4.2: Some Ethical Dilemmas • Activity 4.3: Violations of Ethical Practice • Activity 4.4: Why Would These Research Practices

Be Unethical?

Some Examples of Unethical Practice The term ethics refers to questions of right and wrong. When researchers think about ethics, they must ask themselves if it is “right” to conduct a particular study or carry out certain procedures—that is, whether they are doing ethical research. Are there some kinds of studies that should not be conducted? You bet! Here are some examples of unethical practice:

A researcher

• requires a group of high school sophomores to sign a form in which they agree to participate in a research study.

• asks fi rst-graders sensitive questions without obtain- ing the consent of their parents to question them.

• deletes data he collects that do not support his hypothesis.

• requires university students to fi ll out a questionnaire about their sexual practices.

• involves a group of eighth-graders in a research study that may harm them psychologically without informing them or their parents of this fact.

Each of the above examples involves one or more violations of ethical practice. When researchers think about ethics, the basic question to ask in this regard is, Will any physical or psychological harm come to any- one as a result of my research? Naturally, no researcher wants this to happen to any of the subjects in a research study. Because this is such an important (and often over- looked) issue, we need to discuss it in some detail.

In a somewhat larger sense, ethics also refers to questions of right and wrong. By behaving ethically, a person is doing what is right. But what does it mean to be “right” as far as research is concerned?

A Statement of Ethical Principles Webster’s New World Dictionary defi nes ethical ( behavior) as “conforming to the standards of con- duct of a given profession or group.” What researchers consider to be ethical, therefore, is largely a matter of agreement among them. Some years ago, the Com- mittee on Scientifi c and Professional Ethics of the American Psychological Association published a list

fra97851_ch04_060-074.indd 61fra97851_ch04_060-074.indd 61 22/12/10 11:00 PM22/12/10 11:00 PM

62 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

of ethical principles for the conduct of research with human subjects. We have adapted many of these prin- ciples so they apply to educational research. Please read the following statement and think carefully about what it means.

The decision to undertake research rests upon a con- sidered judgment by the individual educator about how best to contribute to science and human welfare. Once one decides to conduct research, the educator considers various ways by which he might invest his talents and resources. Keeping this in mind, the educator carries out the research with respect and concern for the dignity and welfare of the people who participate and with cogni- zance of federal and state regulations and professional standards governing the conduct of research with human participants.

a. In planning a study, researchers have the responsi- bility to evaluate carefully any ethical concerns. Should any of the ethical principles listed below be compromised, the educator has a correspondingly serious obligation to observe stringent safeguards to protect the rights of human participants.

b. Considering whether a participant in a planned study will be a “subject at risk” or a “subject at minimal risk,” according to recognized standards, is of primary ethical concern to the researcher.

c. The researcher always retains the responsibil- ity for ensuring that a study is conducted ethically. The researcher is also responsible for the ethical treatment of research participants by collaborators, assistants, stu- dents, and employees, all of whom, however, incur similar obligations.

d. Except in minimal-risk research, the researcher establishes a clear and fair agreement with research participants, before they participate, that clarifi es the ob- ligations and responsibilities of each. The researcher has the obligation to honor all promises and commitments included in that agreement. The researcher informs the participants of all aspects of the research that might reasonably be expected to infl uence their willingness to participate in the study and answers honestly any ques- tions they may have about the research. Failure by the researcher to make full disclosure prior to obtaining informed consent requires additional safeguards to pro- tect the welfare and dignity of the research participants. Furthermore, research with children or with participants who have impairments that would limit understanding and/or communication requires special safeguarding procedures.

e. Sometimes the design of a study makes neces- sary the use of concealment or deception. When this is the case, the researcher has a special responsibility to: (i)  determine whether the use of such techniques is justi- fi ed by the study’s prospective scientifi c or educational value; (ii) determine whether alternative procedures are available that do not use concealment or deception; and (iii) ensure that the participants are provided with suffi - cient explanation as soon as possible.

f. The researcher respects the right of any indi- vidual to refuse to participate in the study or to with- draw from participating at any time. The researcher’s obligation in this regard is especially important when he or she is in a position of authority or infl uence over the participants in a study. Such positions of author- ity include, but are not limited to, situations in which research participation is required as part of employment or in which the participant is a student, client, or em- ployee of the investigator.

g. The researcher protects all participants from physical and mental discomfort, harm, and danger that may arise from participating in a study. If risks of such consequences exist, the investigator informs the partici- pant of that fact. Research procedures likely to cause serious or lasting harm to a participant are not used unless the failure to use these procedures might expose the participant to risk of greater harm, or unless the research has great potential benefi t and fully informed and voluntary consent is obtained from each participant. All participants must be informed as to how they can contact the researcher within a reasonable time period following their participation should stress or potential harm arise.

h. After the data are collected, the researcher provides all participants with information about the nature of the study and does his or her best to clear up any miscon- ceptions that may have developed. Where scientifi c or humane values justify delaying or withholding this infor- mation, the researcher has a special responsibility to care- fully supervise the research and to ensure that there are no damaging consequences for the participant.

i. Where the procedures of a study result in undesir- able consequences for any participant, the researcher has the responsibility to detect and remove or correct these consequences, including long-term effects.

j. Information obtained about a research participant during the course of an investigation is confi dential unless otherwise agreed upon in advance. When the possibility exists that others may obtain access to such information, this possibility, together with the plans

fra97851_ch04_060-074.indd 62fra97851_ch04_060-074.indd 62 22/12/10 11:00 PM22/12/10 11:00 PM

63

involved in such trials. In 1995 about 500,000 volunteers participated; by 1999 the number had jumped to 700,000. † Another concern is that some of the physicians who conduct such trials may have a fi nancial stake in the outcome. No uni- form policy currently exists on the disclosure of investigators’ fi nancial interests to patients who participate in such trials.

Proponents of clinical trials argue that, when properly con- ducted, clinical trials have paved the way for new medicines and procedures that have saved many lives. Volunteers can gain access to promising drugs long before they are available to the general public. And patients usually get excellent care from physicians and nurses while they are undergoing such trials. Last, but not least, such care often is free.

What do you think? Are clinical trials justifi ed?

CONTROVERSIES IN RESEARCH

Clinical Trials—Desirable or Not?

C linical trials are the fi nal test of a new drug. They offer an opportunity for drug companies to prove that new and previously unused medicines are safe and effective to use by giving such medicines to volunteers. Recently, however, there has been an increase in the number of complaints against such trials. The most fl agrant example was recently cited in the San Francisco Chronicle. * A scientist gave a volunteer participant in one such trial what turned out to be a lethal dose of an ex- perimental drug.

There has been an increase in the number of clinical trials, as well as a corresponding increase in the number of volunteers

*T. Abate (2001). Maybe confl icts of interest are scaring clinical trial patients. San Francisco Chronicle, May 28.

†Report issued at the Association of Clinical Research Professionals Convention, San Francisco, California, May 20, 2001.

for protecting confi dentiality, is explained to the par- ticipant as part of the procedure for obtaining informed consent. 1

The above statement of ethical principles suggests three very important issues that every researcher should address: protecting participants from harm, ensuring confi dentiality of research data, and the question of de- ception of subjects. How can these issues be addressed, and how can the interests of the subjects involved in research be protected?

Protecting Participants from Harm It is a fundamental responsibility of every researcher to do all in his or her power to ensure that participants in a research study are protected from physical or psy- chological harm, discomfort, or danger that may arise due to research procedures. This is perhaps the most important ethical decision of all. Any sort of study that is likely to cause lasting, or even serious, harm or dis- comfort to any participant should not be conducted, unless the research has the potential to provide in- formation of extreme benefi t to human beings. Even when this may be the case, participants should be fully

i nformed of the dangers involved and in no way re- quired to participate.

A further responsibility in protecting individu- als from harm is obtaining their informed consent if they may be exposed to any risk. (Figure 4.1 shows an example of a consent form.) Fortunately, almost all educational research involves activities that are within the customary, usual procedures of schools or other agencies and as such involve little or no risk. Legislation recognizes this by specifi cally exempting most categories of educational research from formal review processes. Nevertheless, researchers should carefully consider whether there is any likelihood of risk involved and, if there is, provide full information followed by formal consent by participants (or their guardians). Three important ethical questions to ask about harm in any study are:

1. Could people be harmed (physically or psychologi- cally) during the study?

2. If so, could the study be conducted in another way to fi nd out what the researcher wants to know?

3. Is the information that may be obtained from this study so important that it warrants possible harm to the participants?

These are diffi cult questions, and they deserve dis- cussion and consideration by all researchers.

fra97851_ch04_060-074.indd 63fra97851_ch04_060-074.indd 63 22/12/10 11:00 PM22/12/10 11:00 PM

64 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

Ensuring Confi dentiality of Research Data Once the data in a study have been collected, researchers should make sure that no one else (other than perhaps a few key research assistants) has access to the data. Whenever possible, the names of the subjects should be removed from all data collection forms. This can be done by assigning a number or letter to each form, or subjects can be asked to furnish information anony- mously. When this is done, not even the researcher can link the data to a particular subject. Sometimes, how- ever, it is important in a study to identify individual sub- jects. When this is the case, the linkage system should be carefully guarded.

All subjects should be assured that any data col- lected from or about them will be held in confi dence. The names of individual subjects should never be used in any publications that describe the research. And all participants in a study should always have the right to withdraw from the study or to request that data collected about them not be used.

When (If Ever) Is Deception of Subjects Justifi ed? The issue of deception is particularly troublesome. Many studies cannot be carried out unless some decep- tion of subjects takes place. It is often diffi cult to fi nd naturalistic situations in which certain behaviors occur frequently. For example, a researcher may have to wait a long time for a teacher to reinforce students in a cer- tain way. It may be much easier for the researcher to observe the effects of such reinforcement by employing the teacher as a confederate.

Sometimes it is better to deceive subjects than to cause them pain or trauma, as investigating a particular research question might require. The famous Milgram study of obedience is a good example. 2 In this study, subjects were ordered to give increasingly severe elec- tric shocks to another subject whom they could not see sitting behind a screen. What they did not know was that the individual to whom they thought they were adminis- tering the shocks was a confederate of the experimenter, and no shocks were actually being administered. The

Figure 4.1 Example of a Consent Form CONSENT TO SERVE AS A SUBJECT IN RESEARCH

I consent to serve as a subject in the research investigation entitled: __________ ____________________________________________________________________________ ____________________________________________________________________________

The nature and general purpose of the research procedure and the known risks involved have been explained to me by ________________________________. The investigator is authorized to proceed on the understanding that I may terminate my service as a subject at any time I so desire.

I understand the known risks are: _______________________________________ ____________________________________________________________________________ ____________________________________________________________________________

I understand also that it is not possible to identify all potential risks in an experimental procedure, and I believe that reasonable safeguards have been taken to minimize both the known and the potentially unknown risks.

Witness _______________________________ Signed __________________________ (subject)

Date ____________________________

To be retained by the principal investigator.

fra97851_ch04_060-074.indd 64fra97851_ch04_060-074.indd 64 22/12/10 11:00 PM22/12/10 11:00 PM

65

dependent variable was the level of shock subjects ad- ministered before they refused to administer any more. Out of a total of 40 subjects who participated in the study, 26 followed the “orders” of the experimenter and (so they thought) administered the maximum shock pos- sible of 450 volts! Even though no shocks were actually administered, publication of the study results produced widespread controversy. Many people felt the study was unethical. Others argued that the importance of the study and its results justifi ed the deception. Notice that the study raises questions about not only deception but also harm, since some participants could have suffered emotionally from later consideration of their actions.

Current professional guidelines are as follows:

• Whenever possible, a researcher should conduct the study using methods that do not require deception.

• If alternative methods cannot be devised, the re- searcher should determine whether the use of decep- tion is justifi ed by the prospective study’s scientifi c, educational, or applied value.

• If the participants are deceived, the researcher must ensure that they are provided with suffi cient explana- tion as soon as possible.

Perhaps the most serious problem involving decep- tion is what it has done to the reputation of the scien- tifi c community. In general when people begin to think of scientists and researchers as liars, or as individuals who misrepresent what they are about, the overall image of science suffers. Fewer and fewer people are willing to participate in research investigations today because of this perception. As a result, the search for reliable knowledge about our world may be impeded.

Three Examples Involving Ethical Concerns Here are brief descriptions of three research studies. Let us consider each in terms of (1) presenting possible harm to the participants, (2) ensuring the confi dential- ity of the research data, and (3) knowingly practicing deception. (Figure 4.2 illustrates some examples of un- ethical research practices.)

Study 1. The researcher plans to observe (unobtru- sively) students in each of 40 classrooms—eight visits

“People get involved in something to their detriment without any knowledge of it,” George Annas, a professor of health law at the Boston University School of Public Health, told the Chicago Tribune. “We use people. What’s the justifi cation for that?”

No other company has conducted a no-consent experiment under the rule, FDA offi cials said.

Baxter offi cials halted their clinical trial of HemAssist last spring after reviewing data on the fi rst 100 trauma patients placed in the nationwide study.

Of the 52 critically ill patients given the substitute, 24 died, representing a 46.2 percent mortality rate. The Deerfi eld, Ill.- based company had projected 42.6 percent mortality for criti- cally ill patients seeking emergency treatment.

There has been an intense push to fi nd a blood substitute to ease the effects of whole-blood shortages.

Researchers say artifi cial blood lasts longer than conven- tional blood, eliminates the time-consuming need to match blood types and wipes out the risk of contamination from such viruses as HIV and hepatitis.

The 1996 regulations require a level of community noti- fi cation that is not used in most scientifi c studies, including community meetings, news releases and post-study follow-up.

No lawsuits have arisen from the blood substitute trial, Baxter offi cials said.

Patients Given Fake Blood Without Their Knowledge * Failed Study Used Change in FDA Rules ASSOCIATED PRESS

C hicago—A company conducted an ill-fated blood substi-tute trial without the informed consent of patients in the study—some of whom died, federal offi cials say.

Baxter International Inc. was able to test the substitute, known as HemAssist, without consent because of a 1996 change in federal Food and Drug Administration regulations.

The changes, which broke a 50-year standard to get con- sent for nearly all experiments on humans, were designed to help research in emergency medicine that could not happen if doctors took the time to get consent.

But the problems with the HemAssist trial are prompting some medical ethicists to question the rule change.

MORE ABOUT RESEARCH

*San Francisco Chronicle, January 18, 1999.

fra97851_ch04_060-074.indd 65fra97851_ch04_060-074.indd 65 22/12/10 11:00 PM22/12/10 11:00 PM

66 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

each of 40 minutes’ duration. The purpose of these ob- servations is to look for relationships between the be- havior of students and certain teacher behavior patterns.

Possibility of Harm to the Participants. This study would fall within the exempt category regarding the possibility of harm to the participants. Neither teachers nor students are placed under any risk, and observation is an accepted part of school practice.

Confidentiality of the Research Data. The only issue that is likely to arise in this regard is the possible but unlikely observation of a teacher behaving in an illegal or unethical way (e.g., physically or verbally abusing a student). In the former case, the researcher is legally

required to report the incident. In the latter case, the researcher must weigh the ethical dilemma involved in not reporting the incident against that of violating assur- ances of confi dentiality.

Deception. Although no outright deception is involved, the researcher is going to have to give the teachers a ra- tionale for observing them. If the specifi c teacher charac- teristic being observed (e.g., need to control) is given, the behavior in question is likely to be affected. To avoid this, the researcher might explain that the purpose of the study is to investigate different teaching styles—without divulg- ing the specifi cs. To us, this does not seem to be unethical. An alternative is to tell the teachers that specifi c details cannot be divulged until after data have been collected for fear of changing their behavior. If this alternative is pur- sued, some teachers might refuse to participate.

Study 2. The researcher wishes to study the value of a workshop on suicide prevention for high school stu- dents. The workshop is to consist of three 2-hour meet- ings in which danger signals, causes of suicide, and community resources that provide counseling will be discussed. Students will volunteer, and half will be as- signed to a comparison group that will not participate in the workshop. Outcomes will be assessed by comparing the information learned and attitudes of those attending the meetings with those who do not attend.

Possibility of Harm to the Participants. Whether this study fi ts the exempt category with regard to any possibility of risk for the participants depends on the extent to which it is atypical for the school in question. We think that in most schools, this study would prob- ably be considered atypical. In addition, it is conceiv- able that the material presented could place a student at risk by stirring up emotional reactions. In any case, the researcher should inform parents as to the nature of the study and the possible risks involved and obtain their consent for their children to participate.

Confidentiality of the Research Data. No problems are foreseen in this regard, although confi dentiality as to what will occur during the workshop cannot, of course, be guaranteed.

Deception. No problems are foreseen.

Study 3. The researcher wishes to study the effects of “failure” versus “success” by teaching junior high

Figure 4.2 Examples of Unethical Research Practices

“We are required to ask you to sign this consent form. You needn’t read

it; it’s just routine.”

“A few cases seemed quite different from the rest, so we deleted them.”

“Yes, as a student at this university you are required to participate in this study.”

“There is no need to tell any of the parents that we are

modifying the school lunch diet for this study.”

“Requiring students to participate in class discussions

might be harmful to some, but it is necessary for our

research.”

fra97851_ch04_060-074.indd 66fra97851_ch04_060-074.indd 66 22/12/10 11:00 PM22/12/10 11:00 PM

67

students a motor skill during a series of six 10-minute instructional periods. After each training period, the stu- dents will be given feedback on their performance as compared with that of other students. In order to con- trol extraneous variables (such as coordination), the researcher plans to randomly divide the students into two groups—half will be told that their performance was “relatively poor” and the other half will be told that they are “doing well.” Their actual performance will be ignored.

Possibility of Harm to the Participants. This study presents several problems. Some students in the “fail- ure” group may well suffer emotional distress. Although students are normally given similar feedback on their performance in most schools, feedback in this study (being arbitrary) may confl ict dramatically with their prior experience. The researcher cannot properly in- form students, or their parents, about the deceptive na- ture of the study, since to do so would in effect destroy the study.

Confidentiality of the Research Data. Confi dentiality does not appear to be an issue in this study.

Deception. The deception of participants is clearly an issue. One alternative is to base feedback on actual performance. The diffi culty here is that each student’s extensive prior history will affect both individual perfor- mance and interpretation of feedback, thus confound- ing the results. Some, but not all, of these extraneous variables can be controlled (perhaps by examining school records for data on past history or by pretesting

students). Another alternative is to weaken the experi- mental treatment by trying to lessen the possibility of emotional distress (e.g., by saying to participants in the failure group, “You did not do quite as well as most”) and confi ning the training to one time period. Both of these alternatives, however, would lessen the chances of any relationship emerging.

Research with Children Studies using children as participants present some special issues for researchers. The young are more vul- nerable in some respects, have fewer legal rights, and may not understand the language of informed consent. Therefore, the following specifi c guidelines need to be considered.

• Informed consent of parents or of those legally des- ignated as caretakers is required for participants defi ned as minors. Signers must be provided all nec- essary information in appropriate language and must have the opportunity to refuse. (Figure 4.3 shows an example of a consent form for a minor.)

• Researchers do not present themselves as diagnos- ticians or counselors in reporting results to parents, nor do they report information given by a child in confi dence.

• Children may never be coerced into participation in a study.

• Any form of remuneration for the child’s services does not affect the application of these (and other) ethical principles.

were widely cited to support the conclusion that IQ is about 80 percent hereditary and 20 percent environmental.

Some initial questions were raised when another researcher found a considerably lower hereditary percentage. Subsequent detailed investigation of the initial studies * revealed highly suspicious statistical treatment of data, inadequate specifi ca- tion of procedures, and questionable adjustment of scores, all suggesting unethical massaging of data. Such instances, which are reported occasionally, underscore the importance of repeating studies, as well as the essential requirement that all procedures and data be available for public scrutiny.

An Example of Unethical Research

A series of studies reported in the 1950s and 1960s received widespread attention in psychology and education and earned their author much fame, including a knighthood. They addressed the question of how much of one’s performance on IQ tests was likely to be hereditary and how much was due to environmental factors.

Several groups of children were studied over time, includ- ing identical twins raised together and apart, fraternal twins raised together and apart, and same-family siblings. The results

MORE ABOUT RESEARCH

*L. Kamin (1974). The science and politics of I.Q. New York: John Wiley.

fra97851_ch04_060-074.indd 67fra97851_ch04_060-074.indd 67 22/12/10 11:00 PM22/12/10 11:00 PM

68 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

Figure 4.3 Example of a Consent Form for a Minor to Participate in a Research Study

San Francisco State University Parental Permission for a Minor to Participate in Research

Research Title

A. PURPOSE AND BACKGROUND My name is ________. I am a (graduate student/faculty member) at San Francisco State University and I am conducting a research study about _______. I am inviting your child to take part in the research because he/she________________.

(State the purpose of the research; the purpose must be the same as stated in the protocol. In fact, sections throughout this form should mirror the protocol statement. State why the prospective subject is being invited to participate in this study, e.g. “he/she is in the after school program I am studying.”)

B. PROCEDURES If you agree to let your child participate in this research study, the following will occur:

• Your child will be asked to ( play math games and take a test ) • This will take place in their regular classroom as part of my scheduled curriculum. • Your child will participate in a group discussion in social studies class about their attitudes about extracurricular

activities. The discussions will be audiotaped. (OR///) • Your child will be invited to participate in an after school tutoring project. The tutoring sessions will take place

between 3:45 and 4:45 PM on fi ve Tuesdays and Thursdays during the spring semester.

(State where the research will take place, how long it will take, and at what time of day it will occur. State the time each procedure will take, and also state the total time it will take.)

C. RISKS There is a risk of loss of privacy, which the researcher will reduce by not using any real names or other identifi ers in the written report. The researcher will also keep all data in a locked fi le cabinet in a secure location. Only the researcher will have access to the data. At the end of the study, data will be ______ (see “Guidelines for Data Retention.”)

There may be some discomfort for your child at being asked some of the questions. Your child may answer only those questions he or she wants to, or he or she may stop the entire process at any time, without penalty.

(State the risks involved, and how the researcher will reduce them. If the questions are very sensitive and may cause anxiety or other negative emotions, researcher should include a brief list of counseling contacts they may consult.)

D. CONFIDENTIALITY State how you will protect the confi dentiality of the data collected. Where will you store it, will it be password- protected if stored on a computer, or in a locked offi ce if it’s paper data. How long will the data be kept, what will happen to it when the project is over? (Will it be destroyed, kept for future research—if so the research must be consistent with the original purpose.)

E. DIRECT BENEFITS

F. COSTS

G. COMPENSATION

H. QUESTIONS Questions about your child’s rights as a study participant, or comments or complaints about the study also may be ad- dressed to the Offi ce for the Protection of Human Subjects at Your University.

J. CONSENT You have been given a copy of this consent form to keep. PARTICIPATION IN THIS RESEARCH STUDY IS VOLUNTARY. You are free to decline to have your child participate in this research study. You may withdraw your child’s participation at any point without penalty. Your decision whether or not to participate in this research study will have no infl uence on your or your child’s present or future status at your university

Child’s Name ____________________________________________________________________________

Signature _______________________________________________ Date _____________________

Parent

Signature _______________________________________________ Date _____________________

Researcher

fra97851_ch04_060-074.indd 68fra97851_ch04_060-074.indd 68 22/12/10 11:00 PM22/12/10 11:00 PM

C H A P T E R 4 Ethics and Research 69

Regulation of Research The regulation most directly affecting researchers is the National Research Act of 1974. It requires that all research institutions receiving federal funds establish what are known as institutional review boards (IRBs) to review and approve research projects. Such a review must take place whether the research is to be done by a single researcher or a group of researchers. In the case of federally funded investigations, failure to comply can mean that the entire institution (e.g., a university) will lose all of its federal support (e.g., veterans’ benefi ts, scholarship money). Needless to say, this is a severe pen- alty. The federal agency that has the major responsibil- ity for establishing the guidelines for research involving human subjects is the Department of Health and Human Services (HHS).

At institutions receiving federal funding, any af- fi liated researchers (including co-researchers, research technicians, and student assistants) planning to use human subjects are currently required to pass an online research training course administered by the National Institutes of Health (NIH) or the Collaborative Insti- tutional Training Initiative (CITI). Once the course is completed successfully, a course completion report is issued that is valid for three years. (The NIH course can be found at http://phrp.nihtraining.com/users/login.php and the CITI course at www.citiprogram.org/.) Both courses take approximately two to three hours to com- plete and can be bookmarked so that the course does not have to be taken during one sitting. The CITI course takes a little longer to complete but is recommended for social, behavioral, and educational researchers because of the elective modules that can be tailored to a par- ticular fi eld of study. Researchers and students should check with their own institutions about specifi c policies and procedures regarding the research training course. Usually, the report of completion must be submitted along with any research protocol materials to the IRB for approval.

An IRB must have at least fi ve members, consist of both men and women, and include at least one nonscien- tist. It must include one person not affi liated with the in- stitution. Individuals competent in a particularly relevant area may be invited to assist in a review but may not vote. Furthermore, individuals with a confl ict of interest must be excluded, although they may provide information.

If the IRB regularly reviews research involving a vulnerable category of subjects (e.g., such as studies

involving the developmentally disabled), the board must include one or more individuals who are primarily con- cerned with the welfare of these subjects.

The IRB examines all proposed research with respect to certain basic criteria. Sometimes the criteria used by an IRB to determine whether a study is “exempt,” for example, may differ from those specifi ed by the HHS (see the More About Research box on HHS revised reg- ulations). Oftentimes, the criteria set forth by an institu- tional IRB are more conservative than those stipulated by the federal government because of risk management related to litigation liability and funding withdrawal. Researchers and students are advised to consult with their own institution’s IRB policies and procedures. The IRB board can request that a study be modifi ed to meet their criteria before it will be approved. If a proposed study fails to satisfy any one of these criteria, the study will not be approved (see Table 4.1).

TABLE 4.1 Criteria for IRB Approval

• Minimization of risk to participants (e.g., by using procedures that do not unnecessarily expose subjects to risk).

• Risks that may occur are reasonable in relation to benefi ts that are anticipated.

• Equitable selection—i.e., the proposed research does not discriminate among individuals in the population.

• Protection of vulnerable individuals (e.g., children, pregnant women, prisoners, mentally disabled or economically disadvantaged persons, etc.).

• Informed consent—researchers must provide complete information about all aspects of the proposed study that might be on interest or con- cern to a potential participant, and this must be presented in a form that participants can easily understand.

• Participants have the right to withdraw from the study at any time without penalty.

• Informed consent will be appropriately documented.

• Monitoring of the data being collected to ensure the safety of the participants.

• Privacy and confi dentiality—ensuring that any and all information obtained during a study is not released to outside individuals where it might have embarrass- ing or damaging consequences.

fra97851_ch04_060-074.indd 69fra97851_ch04_060-074.indd 69 22/12/10 11:00 PM22/12/10 11:00 PM

70

IRB Boards classify research proposals in three categories:

Category I (Exempt Review) —the proposed study presents no possible risk to adult participants (e.g., an anonymous mailed survey on innocu- ous topics or an anonymous observation of public behavior). This type of study is exempt from the requirement of informed consent.

Category II (Expedited Review) —the proposed study presents no more than minimal risk to par- ticipants. A typical example would be a study of individual or group behavior of adults where there is no psychological intervention or decep- tion involved. This category of research does not require written documentation of informed consent, although oral consent is required. Most classroom research projects fall in this category.

Category III (Full Review) —the proposed study includes questionable elements, such as research involving special populations, vulnerable individ- uals, unusual equipment or procedures, deception, intervention, or some form of invasive measure- ment. A meeting of all IRB members is required, and the researcher must appear in person to dis- cuss and answer questions about the research.

The question of risk for participants is of particular interest to the IRB. The board may terminate a study if it appears that serious harm to subjects is likely to occur. Any and all potential risk(s) to subjects must be minimized. What this means is that any risk should not

review. A preliminary settlement pledged to locate all of the control subjects by the year 2000, invite them into the Job Corps (if still eligible), and pay each person $1,000. *

In a letter to the editor† of Mother Jones in April 1999, however, Judith M. Gueron, the President of Manpower Dem- onstration Research Corporation ( not the company awarded the evaluation grant) defended the study on two grounds: (1) since there were only limited available openings for the pro- gram, random selection of qualifi ed applicants “is arguably fairer” than fi rst-come, fi rst-served; and (2) the alleged harm to those rejected is unknown, since they were free to seek other employment or training.

What do you think?

Ethical or Not?

I n September 1998, a U.S. District Court judge halted a study begun in 1994 to evaluate the effectiveness of the U.S. Job Corps program. For two years, the researchers had randomly assigned 1 out of every 12 eligible applicants to a control group that was denied service for three years—a total of 6,000 applicants. If applicants refused to sign a waiver agreeing to participate in the study, they were told to reapply two years later. The class action lawsuit alleged psychological, emotional, and economic harm to the control subjects. The basis for the judge’s decision was a failure to follow the fed- eral law that required the methodology to be subject to public

*J. Price (1999). Job Corps lottery. Mother Jones, January/February, pp. 21–22.

†Backtalk (1999). Mother Jones, April, p. 13.

be any greater than those ordinarily encountered in daily life or during the performance of routine physical or psychological examinations or tests.

Some researchers were unhappy with the regulations that were issued in 1974 by HHS because they felt that the rules interfered unnecessarily with risk-free proj- ects. Their opposition resulted in a 1981 set of revised guidelines, as shown in the More About Research box on page 71. These guidelines apply to all research funded by HHS. As mentioned above, Institutional Review Boards determine which studies qualify to be exempt from the guidelines.

Another law affecting research is the Family Privacy Act of 1974, also known as the Buckley Amendment. It is intended to protect the privacy of students’ edu- cational records. One of its provisions is that data that identify students may not, with some exceptions, be made available without permission from the student or, if under legal age, parents or legal guardians. Consent forms must specify what data will be disclosed, for what purposes, and to whom.

The relationship between the current guidelines and qualitative research is not as clear as it is for quantita- tive research. In recent years, therefore, there have been a number of suggestions for a specifi c code of ethics for qualitative research. 3 In quantitative studies, subjects can be told the content and the possible dangers involved in a study. In qualitative studies, however, the relationship between research and participant evolves over time. As Bogdan and Biklen suggest, doing qualitative research with informants can be “more like having a friendship than a contract. The people who are studied have a say

CONTROVERSIES IN RESEARCH

fra97851_ch04_060-074.indd 70fra97851_ch04_060-074.indd 70 22/12/10 11:00 PM22/12/10 11:00 PM

71

in regulating the relationship and they continuously make decisions about their participation.” 4 As a result, Bogdan and Biklen offer the following suggestions for qualitative researchers that might be considered when the criteria used by an IRB may not apply: 5

1. Avoid research sites where informants may feel co- erced to participate in the research.

2. Honor the privacy of informants—fi nd a way to re- cruit informants so that they may choose to partici- pate in the study.

3. Tell participants who are being interviewed how long the interview will take.

4. Unless otherwise agreed to, the identities of in- formants should be protected so that the informa- tion collected does not embarrass or otherwise harm them. Anonymity should extend not only to written reports but also to the verbal reporting of information.

5. Treat informants with respect and seek their coop- eration in the research. Informants should be told of the researcher’s interest and they should give their permission for the researcher to proceed. Written consent should always be obtained.

6. Make it clear to all participants in a study the terms of any agreement negotiated with them.

7. Tell the truth when fi ndings are written up and re- ported. Mail in a separate card indicating that they completed the questionnaire.

One further legal matter should be mentioned. Attorneys, physicians, and members of the clergy are protected by laws concerning privileged communica- tions (i.e., they are protected by law from having to reveal information given to them in confi dence). Researchers do not have this protection. It is possible, therefore, that any subjects who admit, on a questionnaire, to having committed a crime could be arrested and prosecuted. As you can see, it would be a risk therefore for the partici- pants in a research study to admit to a researcher that they had participated in a crime. If such information is required to attain the goals of a study, a researcher can avoid the problem by omitting all forms of identifi cation from the questionnaire. When mailed questionnaires are used, the researcher can keep track of nonrespondents by having each participant mail in a separate card indi- cating that they completed the questionnaire.

Academic Cheating and Plagiarism A chapter on ethics and research would not be complete without some mention of academic dishonesty. Many ed- ucators believe the Internet has facilitated student cheat- ing and plagiarism through easy access to electronic papers and resources. Prior to the Internet, plagiarism— the act of misrepresenting someone else’s work as one’s

3. Survey or interview procedures, except where all of the following conditions prevail: a. Participants could be identifi ed. b. Participants’ responses, if they became public, could

place the subject at risk on criminal or civil charges or could affect the subjects’ fi nancial or occupational standing.

c. Research involves “sensitive aspects” of the partici- pant’s behavior, such as illegal conduct, drug use, sex- ual behavior, or alcohol use.

4. Observation of public behavior (including observation by participants), except where all three of the conditions listed in item 3 above are applicable.

5. The collection or study of documents, records, existing data, pathological specimens, or diagnostic specimens if these sources are available to the public or if the informa- tion obtained from the sources remains anonymous.

Department of Health and Human Services Revised Regulations for Research with Human Subjects

T he guidelines exempt many projects from regulation by HHS. Below is a list of projects now free of the guidelines. 1. Research conducted in educational settings, such as in-

structional strategy research or studies on the effectiveness of educational techniques, curricula, or classroom manage- ment methods.

2. Research using educational tests (cognitive, diagnostic, aptitude, and achievement), provided that subjects remain anonymous.

MORE ABOUT RESEARCH

fra97851_ch04_060-074.indd 71fra97851_ch04_060-074.indd 71 22/12/10 11:00 PM22/12/10 11:00 PM

72 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

own—was more diffi cult to commit and get away with. Most colleges and universities today have academic dis- honesty policies in place and severe consequences for students who get caught, i.e., a failing course grade or even academic dismissal. In our experience of teaching undergraduate and graduate students, we believe a good number of students engage in plagiarism unintentionally. We think many students are unaware of attribution rules related to the proper use and citation of published and unpublished sources. The fi rst place to get clarifi cation

on using sources correctly is a style guide such as those published by the American Psychological Association, Modern Languages Association, or the University of Chicago. In addition, some simple guidelines for avoid- ing plagiarism include the following: (1)  Do not use someone’s words without referencing the source or cit- ing the information as a direct quotation; and (2) Do not use someone’s ideas without citing the source. Finally, in our opinion, it is better to over-cite rather than under-cite words and ideas that are not your own.

Go back to the INTERACTIVE AND APPLIED LEARNING feature at the beginning of the chapter for a listing of interactive and applied activities. Go to the Online Learning Center at www.mhhe.com/fraenkel8e to take quizzes, practice with key terms, and review chapter content.

BASIC ETHICAL PRINCIPLES

• Ethics refers to questions of right and wrong. • There are a number of ethical principles that all researchers should be aware of and

apply to their investigations. • The basic ethical question for all researchers to consider is whether any physical or

psychological harm could come to anyone as a result of the research. • All subjects in a research study should be assured that any data collected from or about

them will be held in confi dence. • The term deception, as used in research, refers to intentionally misinforming the sub-

jects of a study as to some or all aspects of the research topic. • Plagiarism is the act of misrepresenting someone else’s work as one’s own. • Unintentional plagiarism can be avoided through the proper use and citation of pub-

lished and unlisted sources.

RESEARCH WITH CHILDREN

• Children as research subjects present problems for researchers that are different from those of adult subjects. Children are more vulnerable, have fewer legal rights, and often do not understand the meaning of informed consent.

REGULATION OF RESEARCH

• Before any research involving human beings can be conducted at an institution that receives federal funds, it must be reviewed by an institutional review board (IRB) at the institution.

• The federal agency that has the major responsibility for establishing the guidelines for research studies that involve human subjects is the Department of Health and Human Services.

Main Points

fra97851_ch04_060-074.indd 72fra97851_ch04_060-074.indd 72 22/12/10 11:00 PM22/12/10 11:00 PM

C H A P T E R 4 Ethics and Research 73

1. Here are three descriptions of ideas for research. Which (if any) might have some ethical problems? Why? a. A researcher is interested in investigating the effects of diet on physical develop-

ment. He designs a study in which two groups are to be compared. Both groups are composed of 11-year-olds. One group is to be given an enriched diet, high in vitamins, that has been shown to have a strengthening effect on laboratory ani- mals. A second group is not to be given this diet. The groups are to be selected from all the 11-year-olds in an elementary school near the university where the researcher teaches.

b. A researcher is interested in the effects of music on attention span. She designs an experimental study in which two similar high school government classes are to be compared. For a fi ve-week period, one class has classical music played softly in the background as the teacher lectures and holds class discussions on the current unit of study. The other class studies the same material and participates in the same ac- tivities as the fi rst class but does not have any music played during the fi ve weeks.

c. A researcher is interested in the effects of drugs on human beings. He asks the warden of the local penitentiary for subjects to participate in an experiment. The warden assigns several prisoners to participate in the experiment but does not tell them what it is about. The prisoners are injected with a number of drugs whose effects are unknown. Their reactions to the drugs are then described in detail by the researcher.

2. Which, if any, of the above studies would be exempt under the revised guidelines shown in the More About Research box on p. 71?

3. Can you suggest a research study that would present ethical problems if done with children but not if done with adults?

4. Are there any research questions that should not be investigated in schools? If so, why not?

5. “Sometimes the design of a study makes necessary the use of concealment or decep- tion.” Discuss. Can you describe a study in which deception might be justifi ed?

6. “Any sort of study that is likely to cause lasting, or even serious, harm or discomfort to any participant should not be conducted, unless the research has the potential to provide information of extreme benefi t to human beings.” Would you agree? If so, why? What might be an example of such information?

1. Adapted from the Committee on Scientifi c and Professional Ethics and Conduct (1981). Ethical principles of psychologists. American Psychologist, 36: 633–638. Copyright 1981 by the American Psychological As- sociation. Reprinted by permission.

2. S. Milgram (1967). Behavioral study of obedience. Journal of Abnormal and Social Psychology, 67: 371–378.

3. For example, see J. Cassell and M. Wax (Eds.) (1980). Ethical problems in fi eldwork. Social Problems 27 (3); B. K. Curry and J. E. Davis (1995). Representing: The obligations of faculty as researchers . Academe (Sept.–Oct.): 40–43; Y. Lincoln (1995). Emerging criteria for quality in qualitative and interpretive research. Qualitative Inquiry 1 (3): 275–289.

4. R. C. Bogdan and S. K. Biklen (2007). Qualitative research for education: An introduction to theory and methods, 5th ed. Boston: Allyn & Bacon.

5. Op. cit. pp. 49–50.

For Discussion

Notes

ethical research 61

informed consent 63

institutional review boards (IRBs) 69

plagiarism 71 Key Terms

fra97851_ch04_060-074.indd 73fra97851_ch04_060-074.indd 73 22/12/10 11:00 PM22/12/10 11:00 PM

74 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

Research Exercise 4: Ethics and Research Using Problem Sheet 4, restate the research question you developed in Problem Sheet 3. Identify any possible ethical problems in carrying out such a study. How might such problems be remedied?

Problem Sheet 4

Ethics and Research 1. My research question is: _______________________________________________

____________________________________________________________________

2. The possibilities of harm to participants (if any) are as follows: _________________

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

I would handle these problems as follows: __________________________________

____________________________________________________________________

3. The possibilities of problems of confi dentiality (if any) are as follows: ___________

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

I would handle these problems as follows: __________________________________

___________________________________________________________________

4. The possibilities of problems of deception (if any) are as follows: _______________

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

I would handle these problems as follows: _________________________________

____________________________________________________________________

5. In which IRB category (I, II, or III) do you think your proposed study should be con- sidered? State why. ____________________________________________________

____________________________________________________________________

____________________________________________________________________

____________________________________________________________________

An electronic version of this Problem Sheet that you can fi ll in and print, save, or e-mail is available on the Online Learning Center at www.mhhe.com/ fraenkel8e.

fra97851_ch04_060-074.indd 74fra97851_ch04_060-074.indd 74 22/12/10 11:00 PM22/12/10 11:00 PM

O B J E C T I V E S Studying this chapter should enable you to:

• Explain what is meant by the term “variable” and name fi ve variables that might be investigated by educational researchers.

• Explain how a variable differs from a constant.

• Distinguish between a quantitative and a categorical variable.

• Explain how independent and dependent variables are related.

• Give an example of a moderator variable. • Explain what a hypothesis is and formulate

two hypotheses that might be investigated in education.

• Name two advantages and two disadvantages of stating research questions as hypotheses.

• Distinguish between directional and nondirectional hypotheses and give an example of each.

The Importance of Studying Relationships

Variables

What Is a Variable? Quantitative Versus

Categorical Variables Independent Versus

Dependent Variables Moderator Variables Mediator Variables Extraneous Variables

Hypotheses

What Is a Hypothesis? Advantages of Stating

Hypotheses in Addition to Research Questions

Disadvantages of Stating Hypotheses

Important Hypotheses Directional Versus

Nondirectional Hypotheses Hypotheses and Qualitative

Research

Variables and Hypotheses 5

How many variables can you identify?

fra97851_ch05_075-089.indd 75fra97851_ch05_075-089.indd 75 12/21/10 6:42 PM12/21/10 6:42 PM

76

• How does the new reading program differ from the one used in this district in the past?

• What does an inquiry-oriented social studies teacher do?

Notice that no relationship is suggested in these questions. The researcher simply wants to identify char- acteristics, behaviors, feelings, or thoughts. It is often necessary to obtain such information as a fi rst step in designing other research or making educational deci- sions of some sort.

The problem with purely descriptive research ques- tions is that answers to them do not help us understand why people feel or think or behave a certain way, why programs possess certain characteristics, why a par- ticular strategy is to be used at a certain time, and so forth. We may learn what happened, or where or when (and even how) something happened, but not why it happened. As a result, our understanding of a situa- tion, group, or phenomenon is limited. For this reason, scientists highly value research questions that suggest relationships to be investigated, because the answers to them help explain the nature of the world in which we live. We learn to understand the world by learning to

The Importance of Studying Relationships We mentioned in Chapter 2 that an important charac- teristic of many research questions is that they suggest a relationship of some sort to be investigated. Not all research questions, however, suggest relationships. Sometimes researchers are interested only in obtaining descriptive information to fi nd out how people think or feel or to describe how they behave in a particular situ- ation. Other times the intent is to describe a particular program or activity. Such questions also are worthy of investigation. As a result, researchers may ask questions like the following:

• How do the parents of the sophomore class feel about the counseling program?

• What changes would the staff like to see instituted in the curriculum?

• Has the number of students enrolling in college preparatory as compared to noncollege preparatory courses changed over the last four years?

Go to the Online Learning Center at www.mhhe.com/fraenkel8e to:

• Learn More About Hypotheses: To State or Not to State

Go to your online Student Mastery Activities book to do the following activities:

• Activity 5.1: Directional vs. Nondirectional Hypotheses • Activity 5.2: Testing Hypotheses • Activity 5.3: Categorical vs. Quantitative Variables • Activity 5.4: Independent and Dependent Variables • Activity 5.5: Formulating a Hypothesis • Activity 5.6: Moderator Variables

INTERACTIVE AND APPLIED LEARNING After, or while, reading this chapter:

M arge Jenkins and Jenna Rodriguez are having coffee following a meeting of their graduate seminar in educational research. Both are puzzled by some of the ideas that came up in today’s meeting of the class. “I’m not sure I agree with Ms. Naser” (their instructor), says Jenna. “She said that there are a lot of advantages to predicting

how you think a study will come out.”

“Yeah, I know,” replies Marge. “But formulating a hypothesis seems like a good idea to me.”

“Well, perhaps, but there are some disadvantages, too.”

“Really? I can’t think of any.”

“Well, what about . . . ?”

Actually, both Jenna and Marge are correct. There are both advantages and disadvantages to stating a hypothesis in addition

to one’s research question. We’ll discuss examples of both in this chapter.

fra97851_ch05_075-089.indd 76fra97851_ch05_075-089.indd 76 12/21/10 6:42 PM12/21/10 6:42 PM

C H A P T E R 5 Variables and Hypotheses 77

reinforce their students in different ways (one gives ver- bal praise, the second gives monetary rewards, the third gives extra points) for various tasks the students per- form. In this study, reinforcement would be a variable (it contains three variations), while the grade level of the students would be a constant.

Notice that it is easier to see what some of these concepts stand for than others. The concept of chair, for example, stands for the many different objects that we sit on that possess legs, a seat, and a back. Further- more, different observers would probably agree as to how particular chairs differ. It is not so easy, however, to see what a concept like motivation stands for, or to agree on what it means. The researchers must be spe- cifi c here—they must defi ne motivation as clearly as possible. They must do this so that it can be measured or manipulated. We cannot meaningfully measure or manipulate a variable if we cannot defi ne it. As we mentioned above, much educational research involves looking for a relationship among variables. But what variables?

There are many variables “out there” in the world that can be investigated. Obviously, we can’t investi- gate them all, so we must choose. Researchers choose certain variables to investigate because they suspect that these variables are somehow related and believe that discovering the nature of this relationship, if pos- sible, can help us make more sense out of the world in which we live.

QUANTITATIVE VERSUS CATEGORICAL VARIABLES

Variables can be classifi ed in several ways. One way is to distinguish between quantitative and categorical variables. Quantitative variables exist in some degree (rather than all or none) along a continuum from less to more, and we can assign numbers to different in- dividuals or objects to indicate how much of the vari- able they possess. Two obvious examples are height (John is 6 feet tall and Sally is 5 feet 4 inches) and weight (Mr. Adams weighs only 150 pounds and his wife 140 pounds, but their son tips the scales at an even 200 pounds). We can also assign numbers to various individuals to indicate how much “interest” they have in a subject, with a 5 indicating very much interest, a 4 much interest, a 3 some interest, a 2 little interest, a 1 very little interest, down to a 0 indicating no interest. If we can assign numbers in this way, we have the vari- able interest.

explain how parts of it are related. We begin to detect patterns or connections between the parts.

We believe that understanding is generally en- hanced by the demonstration of relationships or connections. It is for this reason that we favor the formation of a hypothesis that predicts the existence of a relationship. There may be times, however, when a researcher wants to hypothesize that a relationship does not exist. Why so? The only persuasive argument we know of is that of contradicting an existing wide- spread (but perhaps erroneous) belief. For example, if it can be shown that a great many people believe, in the absence of adequate evidence, that young boys are less sympathetic than young girls, a study in which a researcher fi nds no difference between boys and girls (i.e., no relationship between gender and sympathy) might be of value (such a study may have been done, although we are not aware of one). Unfortunately, most (but by no means all) of the methodological mistakes made in research (such as using inadequate instruments or too small a sample of participants) in- crease the chance of fi nding no relationship between variables. (We shall discuss several such mistakes in later chapters.)

Variables WHAT IS A VARIABLE?

At this point, it is important to introduce the idea of variables, since a relationship is a statement about vari- ables. What is a variable? A variable is a concept—a noun that stands for variation within a class of objects, such as chair, gender, eye color, achievement, motiva- tion, or running speed. Even spunk, style, and lust for life are variables. Notice that the individual members in the class of objects, however, must differ—or vary—to qualify the class as a variable. If all members of a class are identical, we do not have a variable. Such character- istics are called constants , since the individual mem- bers of the class are not allowed to vary, but rather are held constant. In any study, some characteristics will be variables, while others will be constants.

An example may make this distinction clearer. Sup- pose a researcher is interested in studying the effects of reinforcement on student achievement. The researcher systematically divides a large group of students, all of whom are ninth-graders, into three smaller subgroups. She then trains the teachers of these subgroups to

fra97851_ch05_075-089.indd 77fra97851_ch05_075-089.indd 77 12/21/10 6:42 PM12/21/10 6:42 PM

78 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

Quantitative variables can often (but not always) be subdivided into smaller and smaller units. Length, for example, can be measured in miles, yards, feet, inches, or in whatever subdivision of an inch is needed. By way of contrast, categorical variables do not vary in de- gree, amount, or quantity but are qualitatively different. Examples include eye color, gender, religious prefer- ence, occupation, position on a baseball team, and most kinds of research “treatments” or “methods.” For ex- ample, suppose a researcher wishes to compare certain attitudes in two different groups of voters, one in which each individual is registered as a member of one politi- cal party and the other in which individuals are mem- bers of another party. The variable involved would be political party. This is a categorical variable—a person is either in one or the other category, not somewhere in between being a registered member of one party and

being a registered member of another party. All mem- bers within each category of this variable are consid- ered the same as far as party membership is concerned (see Figure 5.1).

Can teaching method be considered a variable? Yes, it can. Suppose a researcher is interested in studying teachers who use different methods in teaching. The researcher locates one teacher who lectures exclusively, another who buttresses her lec- tures with slides, films, and computer images, and a third who uses the case-study method and lectures not at all. Does the teaching method “vary”? It does. You may need to practice thinking of differences in methods or in groups of people (teachers compared to administrators, for example) as variables, but mas- tering this idea is extremely useful in learning about research.

QUANTITATIVE VARIABLES

Height (inches)

Low High

Science Aptitude

Low High

Price of Houses (Thousands of Dollars)

50 600550500450400350300250200150100

CATEGORICAL VARIABLES

Variable: categories of different types of trees

Variable: categories of different types of animals

Variable: categories of different types of aircraft

Figure 5.1 Quantitative Variables Compared with Categorical Variables

fra97851_ch05_075-089.indd 78fra97851_ch05_075-089.indd 78 12/21/10 6:42 PM12/21/10 6:42 PM

79

Now, here are several variables. Which ones are quantitative variables and which ones are categorical variables?

1. Make of automobile 2. Learning ability 3. Ethnicity 4. Cohesiveness 5. Heartbeat rate 6. Gender *

Researchers in education often study the relationship between (or among) either (1) two (or more) quantita- tive variables; (2) one categorical and one quantitative variable; or (3) two or more categorical variables. Here are some examples of each:

1. Two quantitative variables • Age and amount of interest in school • Reading achievement and mathematics achievement • Classroom humanism and student motivation • Amount of time watching television and aggres-

siveness of behavior

2. One categorical and one quantitative variable • Method used to teach reading and reading

achievement • Counseling approach and level of anxiety • Nationality and liking for school • Student gender and amount of praise given by

teachers 3. Two categorical variables

• Ethnicity and father’s occupation • Gender of teacher and subject taught • Administrative style and college major • Religious affi liation and political party membership

Sometimes researchers have a choice of whether to treat a variable as quantitative or categorical. It is not uncommon, for example, to fi nd studies in which a vari- able such as anxiety is studied by comparing a group of “high-anxiety” students to a group of “low-anxiety” stu- dents. This treats anxiety as though it were a categorical variable. While there is nothing really wrong with doing this, there are three reasons why it is preferable in such situations to treat the variable as quantitative.

1. Conceptually, we consider variables such as anxiety in people to be a matter of degree, not a matter of either-or.

Some Important Relationships That Have Been Clarifi ed by Educational Research

1. “The more time beginning readers spend on phonics, the better readers they become.” (Despite a great deal of re- search on the topic, this statement can neither be clearly supported nor refuted. It is clear that phonic instruction is an important ingredient; what is not clear is how much time should be devoted to it.) *

2. “The use of manipulatives in elementary grades results in better math performance.” (The evidence is quite suppor- tive of this method of teaching mathematics.) †

MORE ABOUT RESEARCH

3. “Behavior modifi cation is an effective way to teach simple skills to very slow learners.” (There is a great deal of evi- dence to support this statement.) ‡

4. “The more teachers know about specifi c subject matter, the better they teach it.” (The evidence is inconclusive de- spite the seemingly obvious fact that teachers must know more than their students.) §

5. “Among children who become deaf before language has developed, those with hearing parents become better readers than those with deaf parents.” (The fi ndings of many studies refute this statement.) ||

‡S. L. Deno (1982). Behavioral treatment methods. In H. E. Mitzel (Ed.), Encyclopedia of educational research, 5th ed. New York: Mac- millan, pp. 199–202.

§L. Shulman (1986). Paradigms and research programs in the study of teaching. In M. C. Wittrock (Ed.), Handbook of research on teach- ing, 3rd ed. New York: Macmillan, pp. 3–36.

||C. M. Kampfe and A. G. Turecheck (1987). Reading achievement of prelingually deaf students and its relationship to parental method of communication: A review of the literature. American Annals of the Deaf, 10 (March): 11–15.

*R. Calfee and P. Drum (1986). Research on teaching reading. In M. C. Wittrock (Ed.), Handbook of research on teaching, 3rd ed. New York: Macmillan, pp. 804–849. †M. N. Suydam (1986). Research report: Manipulative materials and achievement. Arithmetic Teacher, 10 (February): 32.

*1, 3, and 6 represent categorical variables; 2, 4, and 5 represent quantitative variables.

fra97851_ch05_075-089.indd 79fra97851_ch05_075-089.indd 79 12/21/10 6:42 PM12/21/10 6:42 PM

80 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

2. Collapsing the variable into two (or even several) cat- egories eliminates the possibility of using more de- tailed information about the variable, since differences among individuals within a category are ignored.

3. The dividing line between groups (for example, be- tween individuals of high, middle, and low anxiety) is almost always arbitrary (that is, lacking in any de- fensible rationale).

INDEPENDENT VERSUS DEPENDENT VARIABLES

A common and useful way to think about variables is to classify them as independent or dependent. Indepen- dent variables are those that the researcher chooses to study in order to assess their possible effect(s) on one or more other variables. An independent variable is presumed to affect (at least partly cause) or some- how infl uence at least one other variable. The variable that the independent variable is presumed to affect is called a dependent variable . In commonsense terms, the dependent variable “depends on” what the indepen- dent variable does to it, how it affects it. For example, a researcher studying the relationship between childhood success in mathematics and adult career choice is likely to refer to the former as the independent variable and subsequent career choice as the dependent variable.

It is possible to investigate more than one indepen- dent (and also more than one dependent) variable in a study. For simplicity’s sake, however, we present exam- ples in which only one independent and one dependent variable are involved.

The relationship between independent and depen- dent variables can be portrayed graphically as follows:

Independent variable(s)

(presumed or possible cause)

Affects

Dependent variable(s)

(presumed results)

At this point, let’s check your understanding. Sup- pose a researcher plans to investigate the following question: “Will students who are taught by a team of three teachers learn more science than students taught by one individual teacher?” What are the independent and dependent variables in this question? *

Notice that there are two conditions (sometimes called levels ) in the independent variable—“three

teachers” and “one teacher.” Also notice that the depen- dent variable is not “science learning” but “ amount of science learning.” Can you see why?

At this point, things begin to get a bit complicated. Independent variables may be either manipulated or se- lected. A manipulated variable is one that the researcher creates. Such variables are typically found in experimen- tal studies (see Chapter 13). Suppose, for example, that a researcher decides to investigate the effect of different amounts of reinforcement on reading achievement and systematically assigns students to three different groups. One group is praised continuously every day during their reading session; the second group is told simply to “keep up the good work”; the third group is told nothing at all. The researcher, in effect, manipulates the conditions in this experiment, thereby creating the variable amount of reinforcement. Whenever a researcher sets up experi- mental conditions, one or more variables are created. Such variables are called manipulated variables, experi- mental variables , or treatment variables .

Sometimes researchers select an independent variable that already exists. In this case, the researcher must lo- cate and select examples of it, rather than creating it. In our earlier example of reading methods, the researcher would have to locate and select existing examples of each reading method, rather than arranging for them to happen. Selected independent variables are not limited to studies that compare different treatments; they are found in both causal-comparative and correlational studies (see Chapters 15 and 16). They can be either categorical or quantitative. The key idea here, however, is that the inde- pendent variable (either created or selected) is thought to affect the dependent variable. Here are a few examples of some possible relationships between a selected inde- pendent variable and a dependent variable:

Independent Variable Dependent Variable

Gender (categorical)

Mathematical ability (quantitative)

Gang membership (categorical)

Test anxiety (quantitative)

Musical aptitude (quantitative)

Career choice (categorical)

Subsequent marital status (categorical)

Test performance (quantitative)

Notice that none of the independent variables in the  above  pairs could be directly manipulated by the researcher. Notice also that, in some instances, the

*The independent (categorical) variable is the number of teachers, and the dependent (quantitative) variable is the amount of science learning .

fra97851_ch05_075-089.indd 80fra97851_ch05_075-089.indd 80 12/21/10 6:42 PM12/21/10 6:42 PM

C H A P T E R 5 Variables and Hypotheses 81

independent/ dependent relationship might be reversed, depending on which one the researcher thought might be the cause of the other. For example, he or she might think that test performance causes anxiety, not the reverse.

Generally speaking, most studies in education that have one quantitative and one categorical variable are studies comparing different methods or treatments. As we indicated above, the independent variable in such studies (the different methods or treatments) represents a categorical variable. Often the other (dependent) vari- able is quantitative and is referred to as an outcome variable . * The reason is rather clear-cut. The investi- gator, after all, is interested in the effect(s) of the dif- ferences in method on one or more outcomes (student achievement, their motivation, interest, and so on).

Again, let’s check your understanding. Suppose a researcher plans to investigate the following question: “Will students like history more if taught by the inquiry method than if taught by the case-study method?” What is the outcome variable in this question? †

MODERATOR VARIABLES

A moderator variable is a special type of independent variable. It is a secondary independent variable that has been selected for study in order to determine if it affects or modifi es the basic relationship between the primary independent variable and the dependent variable. Thus, if an experimenter thinks that the relationship between variables X and Y might be altered in some way by a third variable Z, then Z could be included in the study as a moderator variable.

Consider an example. Suppose a researcher is inter- ested in comparing the effectiveness of a discussion- ori- ented approach to a more visually oriented approach for teaching a unit in a U.S. History class. Suppose further that the researcher suspects that the discussion approach may be superior for the girls in the class (who appear to be more verbal and to learn better through conversing with others) and that the visual approach may be more ef- fective for boys (who seem to perk up every time a video is shown). When the students are tested together at the end of the unit, the overall results of the two approaches may show no difference, but when the results of the girls are separated from those of the boys, the two approaches

may reveal different results for each subgroup. If so, then the gender variable moderates the relationship between the instructional approach (the independent variable) and effectiveness (the dependent variable). The infl uence of this moderator variable can be seen in Figure 5.2.

Here are two examples of research questions that contain moderator variables.

Research Question 1: “Does anxiety affect test per- formance and, if so, does it depend on test-taking experience?” • Independent variable: anxiety level • Moderator variable: test-taking experience • Dependent variable: test performance

Research Question 2: “Do high school students taught primarily by the inquiry method perform better on tests of critical thinking than high school students taught primarily by the demonstration method and, if so, does it vary with grade level?” • Independent variable: instructional method • Moderator variable: grade level • Dependent variable: performance on critical

thinking tests

As you can see, the inclusion of a moderator variable (or even two or three) in a study can provide consider- ably more information than just studying a single inde- pendent variable alone. We recommend their inclusion whenever appropriate.

MEDIATOR VARIABLES

While a moderator variable can modify or infl uence the strength of a relationship between two other variables, a mediator variable is one that attempts to explain the relationship between the two other variables. Let us reexamine the relationship in Hypothesis 1 above be- tween anxiety level (AL) and test performance (TP) on a

*It is also possible for an outcome variable to be categorical. For example, the variable college completion could be divided into the categories of dropouts and college graduates.

† Liking for history is the outcome variable.

Figure 5.2 Relationship Between Instructional Approach (Independent Variable) and Achievement (Dependent Variable), as Moderated by Gender of Students

Discussion

Boys

Boys Girls

Girls

Instructional Approach

Visual

A ch

ie ve

m en

t

fra97851_ch05_075-089.indd 81fra97851_ch05_075-089.indd 81 12/21/10 6:42 PM12/21/10 6:42 PM

82 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

high-stakes test like the SAT exam. The moderator vari- able in this case is test-taking experience (TTE) because the relationship between AL and TP is stronger for stu- dents with prior experience taking the SAT. A possible mediator variable in this case could be socioeconomic status (SES) because it could explain why there is a re- lationship between AL and TP.

EXTRANEOUS VARIABLES

A basic problem in research is that there are many pos- sible independent variables that could have an effect on the dependent variables. Once researchers have decided which variables to study, they must be concerned about the infl uence or effect of other variables that exist. Such variables are usually called extraneous variables. The

task is to control these extraneous variables somehow to eliminate or minimize their effect.

Extraneous variables are independent variables that have not been controlled. Look again at the research question about team teaching on page 80. What are some other variables that could have an effect on the learning of students in a classroom situation?

There are many possible extraneous variables. The personality of the teachers involved, the experience level of the students, the time of day the classes are taught, the nature of the subject taught, the textbooks used, the type of learning activities the teachers employ, and the teach- ing methods—all are possible extraneous variables that could affect learning in this study. Figure 5.3 illustrates the importance of identifying extraneous variables.

• Size of class • Gender of students • Gender of teacher • Age of teacher • Time of day class meets • Days of week class meets • •

Ethnicity of teacher Length of class

Ms. Brown’s (age 31) history class meets from 9:00 to 9:50 A.M., Tuesdays and Thursdays. The class contains 9 students, all girls.

Mr. Thompson’s (age 54) history class meets from 2:00 to 3:00 P.M. Mondays and Wednesdays. The class contains 16 students, all boys.

Extraneous variables

The principal of a high school compares the final examination scores of two history classes taught by teachers who use different methods, not realizing that they are also different in many other ways because of extraneous variables. The classes differ in:

Figure 5.3 Examples of Extraneous Variables

fra97851_ch05_075-089.indd 82fra97851_ch05_075-089.indd 82 12/21/10 6:42 PM12/21/10 6:42 PM

C H A P T E R 5 Variables and Hypotheses 83

One way to control extraneous variables is to hold them constant. For example, if a researcher includes only boys as the subjects of a study, she is controlling the variable of gender. We would say that the gender of the subjects does not vary; it is a constant in this study.

Researchers must continually think about how they might control the possible effect(s) of extraneous vari- ables. We will discuss how to do this in some detail in Chapter 9, but for now you need to make sure you understand the difference between independent and de- pendent variables and to be aware of extraneous vari- ables. Try your hand at the following question: “Will female students who are taught history by a teacher of the same gender like the subject more than female stu- dents taught by a teacher of a different gender?” What are the variables? *

Hypotheses WHAT IS A HYPOTHESIS?

A hypothesis is, simply put, a prediction of the possible outcomes of a study. For example, here is a research question followed by its restatement in the form of a possible hypothesis:

Question: Will students who are taught history by a teacher of the same gender like the subject more than students taught by a teacher of a different gender?

Hypothesis: Students taught history by a teacher of the same gender will like the subject more than students taught history by a teacher of a different gender.

Here are two more examples of research questions followed by the restatement of each as a possible hypothesis:

Question: Is rapport with clients of counselors using client-centered therapy different from that of counselors using behavior-modifi cation therapy?

Hypothesis: Counselors who use a client-centered therapy approach will have a greater rapport with their clients than counselors who use a behavior- modifi cation approach.

Question: How do teachers feel about special classes for the educationally handicapped?

Hypothesis: Teachers in XYZ School District be- lieve that students attending special classes for the educationally handicapped will be stigmatized.

or Teachers in XYZ School District believe that spe-

cial classes for the educationally handicapped will help such students improve their academic skills.

Many different hypotheses can come from a single research problem, as illustrated in Figure 5.4.

ADVANTAGES OF STATING HYPOTHESES IN ADDITION TO RESEARCH QUESTIONS

Stating hypotheses has both advantages and disadvan- tages. What are some of the advantages? First, a hy- pothesis forces us to think more deeply and specifi cally about the possible outcomes of a study. Elaborating on a question by formulating a hypothesis can lead to a more sophisticated understanding of what the question im- plies and exactly what variables are involved. Often, as in the case of the third example above, when more than one hypothesis seems to suggest itself, we are forced to think more carefully about what we really want to investigate.

A second advantage of restating questions as hy- potheses involves a philosophy of science. The ratio- nale underlying this philosophy is as follows: If one is attempting to build a body of knowledge in addition to answering a specifi c question, then stating hypotheses is a good strategy because it enables one to make specifi c predictions based on prior evidence or theoretical argu- ment. If these predictions are borne out by subsequent re- search, the entire procedure gains both in persuasiveness and effi ciency. A classic example is Albert Einstein’s theory of relativity. Many hypotheses were formulated as a result of Einstein’s theory, which were later verifi ed through research. As more and more of these predictions were shown to be fact, not only did they become useful in their own right, they also provided increasing support for the original ideas in Einstein’s theory, which gener- ated the hypotheses in the fi rst place.

Lastly, stating a hypothesis helps us see if we are, or are not, investigating a relationship. If not, we may be prompted to formulate one.

*The dependent variable is liking for history, the independent variable is the gender of the teacher. Possible extraneous variables include the personality and ability of the teacher(s) involved; the personality and ability level of the students; the materials used, such as textbooks; the style of teaching; ethnicity and/or age of the teacher and students; and others. The researcher would want to control as many of these variables as possible.

fra97851_ch05_075-089.indd 83fra97851_ch05_075-089.indd 83 12/21/10 6:42 PM12/21/10 6:42 PM

84 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

DISADVANTAGES OF STATING HYPOTHESES

Essentially, the disadvantages of stating hypotheses are threefold. First, stating a hypothesis may lead to a bias , either conscious or unconscious, on the part of the researcher. Once investigators state a hypothesis, they may be tempted to arrange the procedures or manipu- late the data in such a way as to bring about a desired outcome.

This is probably more the exception than the rule. Researchers are assumed to be intellectually honest— although there are some famous exceptions. All studies should be subject to peer review; in the past, a review of suspect research has, on occasion, revealed such inad- equacies of method that the reported results were cast into doubt. Furthermore, any particular study can be replicated to verify the fi ndings of the study. Unfortu- nately, few educational research studies are repeated, so this “protection” is somewhat of an illusion. A dishonest investigator stands a fair chance of getting away with falsifying results. Why would a person deliberately dis- tort his or her fi ndings? Probably because professional

recognition and fi nancial reward accrue to those who publish important results.

Even for the great majority of researchers who are honest, however, commitment to a hypothesis may lead to distortions that are unintentional and unconscious. But it is probably unlikely that any researcher in the fi eld of education is ever totally disinterested in the out- comes of a study; therefore, his or her attitudes and/or knowledge may favor a particular result. For this reason, we think it is desirable for researchers to make known their predilections regarding a hypothesis so that they are clear to others interested in their research. This also allows investigators to take steps to guard (as much as possible) against their personal biases.

The second disadvantage of stating hypotheses at the outset is that it may sometimes be unnecessary, or even inappropriate, in research projects of certain types, such as descriptive surveys and ethnographic studies. In many such studies, it would be unduly presumptuous, as well as futile, to predict what the fi ndings of the inquiry will be.

Figure 5.4 A Single Research Problem Can Suggest Several Hypotheses

Principal

“They are sent to us poorly prepared

from the middle schools they have attended.”

“No, it’s resources. We just

don’t have the books and other materials we

need to do an effective job.”

“Many of this generation of students are apathetic—they just don’t care about much

of anything.”

“Why have the test scores of our seniors been going

down?”

“They don’t believe that schooling will pay off for them.”

“I think it is us—the teachers. We’re not teaching as well as we used to because

we’re burned out.”

“They watch too much television!”

“So many of them have to

work at night and don’t have time

to study.”

fra97851_ch05_075-089.indd 84fra97851_ch05_075-089.indd 84 12/21/10 6:42 PM12/21/10 6:42 PM

C H A P T E R 5 Variables and Hypotheses 85

The third disadvantage of stating hypotheses is that focusing attention on a hypothesis may prevent re- searchers from noticing other phenomena that might be important to study. For example, deciding to study the effect of a “humanistic” classroom on student motiva- tion might lead a researcher to overlook its effect on such characteristics as sex-typing or decision making, which would be quite noticeable to another researcher who was not focusing solely on motivation. This seems to be a good argument against all research being di- rected toward hypothesis testing.

Consider the example of a research question presented earlier in this chapter: “How do teachers feel about spe- cial classes for the educationally handicapped?” We of- fered two (of many possible) hypotheses that might arise out of this question: (1) “Teachers believe that students attending special classes for the educationally handi- capped will be stigmatized” and (2) “Teachers believe that special classes for the educationally handicapped will help such students improve their academic skills.” Both of these hypotheses implicitly suggest a compari- son between special classes for the educationally handi- capped and some other kind of arrangement. Thus, the relationship to be investigated is between teacher beliefs and type of class. Notice that it is important to compare what teachers think about special classes with their be- liefs about other kinds of arrangements. If researchers looked only at teacher opinions about special classes without also identifying their views about other kinds of arrangements, they would not know if their beliefs about special classes were in any way unique or different.

IMPORTANT HYPOTHESES

As we think about possible hypotheses suggested by a research question, we begin to see that some of them are more important than others. What do we mean by important? Simply that some may lead to more useful knowledge. Compare, for example, the following pairs of hypotheses. Which hypothesis in each pair would you say is more important?

Pair 1

a. Second-graders like school less than they like watching television.

b. Second-graders like school less than fi rst-graders but more than third-graders.

Pair 2

a. Most students with academic disabilities prefer being in regular classes rather than in special classes.

b. Students with academic disabilities will have more negative attitudes about themselves if they are placed in special classes than if they are placed in regular classes.

Pair 3

a. Counselors who use client-centered therapy pro- cedures get different reactions from counselees than do counselors who use traditional therapy procedures.

b. Counselees who receive client-centered therapy express more satisfaction with the counseling pro- cess than do counselees who receive traditional therapy.

In each of the three pairs, we think that the second hypothesis is more important than the fi rst, since in each case (in our judgment) not only is the relationship to be investigated clearer and more specifi c but also investi- gation of the hypothesis seems more likely to lead to a greater amount of knowledge. It also seems to us that the information to be obtained will be of more use to people interested in the research question.

DIRECTIONAL VERSUS NONDIRECTIONAL HYPOTHESES

Let us make a distinction between directional and non- directional hypotheses. A directional hypothesis indi- cates the specifi c direction (such as higher, lower, more, or less) that a researcher expects to emerge in a rela- tionship. The particular direction expected is based on what the researcher has found in the literature, in theory, or from personal experience. The second hypothesis in each of the three pairs above is a directional hypothesis.

Sometimes it is diffi cult to make specifi c predic- tions. If a researcher suspects that a relationship exists but has no basis for predicting the direction of the re- lationship, she cannot make a directional hypothesis. A nondirectional hypothesis does not make a spe- cifi c prediction about what direction the outcome of a study will take. In nondirectional form, the second hypotheses of the three pairs above would be stated as follows:

1. First-, second-, and third-graders will feel differ- ently toward school.

2. There will be a difference between the scores on an attitude measure of students with academic dis- abilities placed in special classes and such students placed in regular classes.

fra97851_ch05_075-089.indd 85fra97851_ch05_075-089.indd 85 12/21/10 6:42 PM12/21/10 6:42 PM

86 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

3. There will be a difference in expression of satisfac- tion with the counseling process between counselees who receive client-centered therapy and counselees who receive traditional therapy.

Figure 5.5 illustrates the difference between a direc- tional and a nondirectional hypothesis. If the person pic- tured is approaching a street corner, three possibilities exist when he reaches the corner:

• He will continue to look straight ahead. • He will look to his right. • He will look to his left.

A nondirectional hypothesis would predict that he will look one way or the other. A directional hypothesis would predict that he will look in a particular direction (for example, to his right). Since a directional hypoth- esis is riskier (because it is less likely to occur), it is more convincing when confi rmed. *

Both directional and nondirectional hypotheses ap- pear in the literature of research, and you should learn to recognize each.

HYPOTHESES AND QUALITATIVE RESEARCH

What is notable about the formation of hypotheses in qualitative research is that they are typically not stated at the beginning of a study, but rather they emerge as a study progresses. Rather than testing hypotheses as in quantitative studies, qualitative re- searchers are more likely to generate new hypotheses as a result of what they find as they go about their work—as they observe patterns and relationships in the natural setting rather than hypothesizing what such patterns and relationships might be beforehand. Many qualitative researchers do state some of their ideas before they begin a study, but these are usually called propositions rather than hypotheses. 1 Propo- sitions differ from hypotheses in that they are not intended to be tested against the data (as in quantita- tive research) but rather are viewed as flexible tools intended to help guide researchers in their collection and analysis of qualitative data. The reluctance of qualitative researchers to formulate hypotheses at the beginning of a study is based on their conviction that participants and situations often differ widely and must first be understood before any hypotheses can be suggested.

*If he looks straight ahead, neither a directional nor a nondirectional hypothesis is confi rmed.

Nondirectional hypothesis (The man will look either

left or right.)

Directional hypothesis (1) (The man will look only to the right.)

Directional hypothesis (2) (The man will look only to the left.)

Either – Or

Will look only to right

Will look only to left

Figure 5.5 Directional Versus Nondirectional Hypotheses

fra97851_ch05_075-089.indd 86fra97851_ch05_075-089.indd 86 12/21/10 6:42 PM12/21/10 6:42 PM

C H A P T E R 5 Variables and Hypotheses 87

Go back to the INTERACTIVE AND APPLIED LEARNING feature at the beginning of the chapter for a listing of interactive and applied activities. Go to the Online Learning Center at www.mhhe.com/fraenkel8e to take quizzes, practice with key terms, and review chapter content.

THE IMPORTANCE OF STUDYING RELATIONSHIPS

• Identifying relationships among variables enhances understanding. • Understanding relationships helps us to explain the nature of our world.

VARIABLES

• A variable is any characteristic or quality that varies among the members of a par- ticular group.

• A constant is any characteristic or quality that is the same for all members of a par- ticular group.

• A quantitative variable varies in amount or degree, but not in kind. • A categorical variable varies only in kind, not in degree or amount. • Several kinds of variables are studied in educational research, the most common

being independent and dependent variables. • An independent variable is presumed to affect or infl uence other variables. • Independent variables are sometimes called experimental variables or manipulated

variables . • A dependent (or outcome) variable is presumed to be affected by one or more inde-

pendent variables. • Independent variables may be either manipulated or selected. A manipulated variable

is created by the researcher. A selected variable is one that already exists that the researcher locates and then chooses to study.

• A moderator variable is a secondary independent variable that the researcher selects to study because he or she thinks it may affect the basic relationship between the primary independent variable and the dependent variable.

• An extraneous variable is an independent variable that may have unintended effects on a dependent variable in a particular study.

• A proposition is a tentative, fl exible statement used sometimes by qualitative re- searchers to help guide their data collection and analysis.

HYPOTHESES

• The term hypothesis, as used in research, refers to a prediction of results usually made before a study commences.

• Stating a research question as a hypothesis has both advantages and disadvantages. • An important hypothesis is one that is likely to lead, if it is supported, to a greater

amount of important knowledge than a nonimportant hypothesis. • A directional hypothesis is a prediction about the specifi c nature of a relationship—

for example, method A is more effective than method B. • A nondirectional hypothesis is a prediction that a relationship exists without specify-

ing its exact nature—for example, there will be a difference between method A and method B (without saying which will be more effective).

Main Points

fra97851_ch05_075-089.indd 87fra97851_ch05_075-089.indd 87 12/21/10 6:42 PM12/21/10 6:42 PM

88 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

bias 84

categorical variable 78

constant 77

dependent variable 80

directional hypothesis 85

experimental variable 80

extraneous variable 82

hypothesis 83

independent variable 80

manipulated variable 80

mediator variable 81

moderator variable 81

nondirectional hypothesis 85

outcome variable 81

propositions 86

quantitative variable 77

treatment variable 80

variable 77

Key Terms

1. Here are several research questions. Which ones suggest relationships? a. How many students are enrolled in the sophomore class this year? b. As the reading level of a text passage increases, does the number of errors students

make in pronouncing words in the passage increase? c. Do individuals who see themselves as socially “attractive” expect their romantic

partners also to be (as judged by others) socially attractive? d. What does the faculty dislike about the new English curriculum? e. Who is the brightest student in the senior class? f. Will students who score above the 90th percentile on a standardized reading test

also score above the 90th percentile on a standardized writing test? g. Which political party contains the most Protestants—Democratic or Republican?

2. How would you rank the questions in item 1 in terms of signifi cance? Why? 3. What might cause a researcher to state a directional hypothesis rather than a non-

directional hypothesis? What about the reverse? 4. Are there any variables that researchers should not study? Explain. 5. It is often argued that we cannot meaningfully measure a variable if we cannot defi ne

it. Is this true? always? Discuss. 6. “Commitment to a hypothesis may lead to distortions that are unintentional and un-

conscious.” Would you agree? If so, can you give an example of such a hypothesis? 7. Can you think of a possible study for which it would be presumptuous to predict the

outcome?

For Discussion

1. J. A. Maxwell (2005). Qualitative research design: An interactive approach , 2nd ed. Thousand Oaks, CA: Sage, p. 69. Note

fra97851_ch05_075-089.indd 88fra97851_ch05_075-089.indd 88 12/21/10 6:42 PM12/21/10 6:42 PM

C H A P T E R 5 Variables and Hypotheses 89

Research Exercise 5: Variables, Hypotheses, and Propositions If you are planning a quantitative study, try to formulate a testable hypothesis that is related to the research question you developed in Research Exercise 2. Using Problem Sheet 5, state the hypothesis in a sentence or two and check to see if it suggests a relationship between at least two variables. If it does not, revise it so that it does. Now indicate which is the independent and which is the dependent variable. Next, list as many extraneous variables as you can think of that might affect the results of your study. On the other hand, if you are planning a qualitative or mixed-methods study, state your proposition(s).

Problem Sheet 5

Variables, Hypotheses, and Propositions My research question is:

1. For a quantitative study, my hypothesis is:

___________________________________________________________________

___________________________________________________________________

___________________________________________________________________

2. This hypothesis suggests a relationship between at least two variables:

a.

b.

c.

3. More specifi cally, the variables in my study are:

a. Dependent (Is it categorical or quantitative?—circle one.)

b. Independent (Is it categorical or quantitative?—circle one.)

4. Possible extraneous variables that might affect my results include:

a.

b.

c.

5. I am planning a qualitative or mixed-methods study. The proposition(s) is/are:

An electronic version of this Problem Sheet that you can fi ll in and print, save, or e-mail is available on the Online Learning Center at www.mhhe.com/ fraenkel8e.

fra97851_ch05_075-089.indd 89fra97851_ch05_075-089.indd 89 12/21/10 6:42 PM12/21/10 6:42 PM

6

O B J E C T I V E S Studying this chapter should enable you to:

• Distinguish between a sample and a population.

• Explain what is meant by the term “representative sample.”

• Explain how a target population differs from an accessible population.

• Explain what is meant by “random sampling,” and describe briefl y three ways of obtaining a random sample.

• Use a table of random numbers to select a random sample from a population.

• Explain how stratifi ed random sampling differs from cluster random sampling.

• Explain what is meant by “systematic sampling,” “convenience sampling,” and “purposive sampling.”

• Explain how the size of a sample can make a difference in terms of representativeness of the sample.

• Explain what is meant by the term “external validity.”

• Distinguish between population generalizability and ecological generalizability and discuss when it is (and when it is not) appropriate to generalize the results of a study.

What Is a Sample?

Samples and Populations Defi ning the Population Target Versus Accessible

Populations Random Versus Nonrandom

Sampling

Random Sampling Methods

Simple Random Sampling Stratifi ed Random Sampling Cluster Random Sampling Two-Stage Random Sampling

Nonrandom Sampling Methods

Systematic Sampling Convenience Sampling Purposive Sampling

A Review of Sampling Methods

Sample Size

External Validity: Generalizing from a Sample

Population Generalizability When Random Sampling Is

Not Feasible Ecological Generalizability

Sampling

1. 2. 3. 4. 5. 6. 7. 8.

201 Brown Avenue 217 Brown Avenue 219 Brown Avenue 221 Brown Avenue 225 Brown Avenue 231 Green Avenue 239 Green Avenue 245 Green Avenue

9. 10. 11. 12. 13. 14. 15. 16.

1008 4th Street 1014 4th Street 1022 4th Street 1030 4th Street 1034 5th Street 1042 5th Street 1050 5th Street 1056 5th Street

(Example of one listed block.) Every fourth household has been selected, starting with no. 4, which was selected at random.

1st St.

2nd St.

3rd St.

4th St.

5th St.

O ra

ng e

A v.

Ro be

rt s

A v.

Br ow

n A

v.

G re

en A

v.

Sp ru

ce A

v.

Pi ne

A v.

Ta yl

or A

v.

Stage 4: Random sample of blocks

Stage 2: Random sample of wards

Stage 3: Random sample of precincts

Cities of 100,000 people are identified by researchers.

F H

Stage 1: Random sample of cities

A B C D

E

F

G

H

Stage 5: Random sample of households

MULTISTAGE SAMPLING

fra97851_ch06_090-109.indd 90fra97851_ch06_090-109.indd 90 12/21/10 6:43 PM12/21/10 6:43 PM

91

will participate (be observed or questioned). Sampling refers to the process of selecting these individuals.

SAMPLES AND POPULATIONS

A sample in a research study is the group on which information is obtained. The larger group to which one hopes to apply the results is called the population . * All 700 (or whatever total number of) students at State Uni- versity who are majoring in mathematics, for example, constitute a population; 50 of those students constitute a sample. Students who own automobiles make up an- other population, as do students who live in the campus dormitories. Notice that a group may be both a sample in one context and a population in another context. All State University students who own automobiles consti- tute the population of automobile owners at State, yet they also constitute a sample of all automobile owners at state universities across the United States.

When it is possible, researchers would prefer to study the entire population of interest. Usually, however, this is diffi cult to do. Most populations of interest are large, diverse, and scattered over a large geographic area. Find- ing, let alone contacting, all the members can be time- consuming and expensive. For that reason, of necessity,

When we want to know something about a certain group of people, we usually fi nd a few members of the group whom we know—or don’t know—and study them. After we have fi nished “studying” these indi- viduals, we usually come to some conclusions about the larger group of which they are a part. Many “com- monsense” observations, in fact, are based on obser- vations of relatively few people. It is not uncommon, for example, to hear statements such as: “Most female students don’t like math”; “You won’t fi nd very many teachers voting Republican”; and “Most school super- intendents are men.”

What Is a Sample? Most people, we think, base their conclusions about a group of people (students, Republicans, football play- ers, actors, and so on) on the experiences they have with a fairly small number, or sample , of individual mem- bers. Sometimes such conclusions are an accurate rep- resentation of how the larger group of people acts or what they believe, but often they are not. It all depends on how representative (i.e., how similar) the sample is of the larger group.

One of the most important steps in the research pro- cess is the selection of the sample of individuals who

Go to the Online Learning Center at www.mhhe.com/fraenkel8e to:

• Learn More About Sampling and Representativeness

Go to your online Student Mastery Activities book to do the following activities:

• Activity 6.1: Identifying Types of Sampling • Activity 6.2: Drawing a Random Sample • Activity 6.3: When Is It Appropriate to Generalize? • Activity 6.4: True or False? • Activity 6.5: Stratifi ed Sampling • Activity 6.6: Designing a Sampling Plan

INTERACTIVE AND APPLIED LEARNING After, or while, reading this chapter:

R osa Pak, a research professor at a large eastern university, wishes to study the effect of a new mathematics program on the mathematics achievement of students who are performing poorly in math in elementary schools throughout the United States. Because of a number of factors, of which time and money are only two, it is impossible for Rosa and her

colleagues to try out the new program with the entire population of such students. They must select a sample . What is a

sample anyway? Are there different kinds of samples? Are some kinds better than others to study? And just how does one go

about obtaining a sample in the fi rst place? Answers to questions like these are some of the things you will learn about in

this chapter.

*In some instances the sample and population may be identical.

fra97851_ch06_090-109.indd 91fra97851_ch06_090-109.indd 91 12/21/10 6:43 PM12/21/10 6:43 PM

92 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

individuals) who possess certain characteristics. In some cases, however, the population may be defi ned as a group of classrooms, schools, or even facilities. For example,

• All fi fth-grade classrooms in Delaware (the hypoth- esis might be that classrooms in which teachers dis- play a greater number and variety of student products have higher achievement)

• All high school gymnasiums in Nevada (the hypoth- esis might be that schools with “better” physical facilities produce more winning teams)

TARGET VERSUS ACCESSIBLE POPULATIONS

Unfortunately, the actual population (called the target population ) to which a researcher would really like to generalize is rarely available. The population to which a researcher is able to generalize, therefore, is the acces- sible population . The former is the researcher’s ideal choice; the latter, his or her realistic choice. Consider these examples:

Research problem to be investigated: The effects of computer-assisted instruction on the read- ing achievement of fi rst- and second-graders in California.

Target population: All fi rst- and second-grade chil- dren in California.

Accessible population: All fi rst- and second-grade children in the Laguna Salada elementary school district of Pacifi ca, California.

Sample: Ten percent of the fi rst- and second-grade children in the Laguna Salada district in Pacifi ca, California.

Research problem to be investigated: The attitudes of fi fth-year teachers-in-training toward their student- teaching experience.

Target population: All fi fth-year students enrolled in teacher-training programs in the United States.

Accessible population: All fi fth-year students en- rolled in teacher-training programs in the State University of New York.

Sample: Two hundred fi fth-year students selected from those enrolled in the teacher-training pro- grams in the State University of New York.

The more narrowly researchers defi ne the popula- tion, the more they save on time, effort, and (probably) money, but the more they limit generalizability. It is es- sential that researchers describe the population and the sample in suffi cient detail so that interested individuals

researchers often select a sample to study. Some exam- ples of samples selected from populations follow:

• A researcher is interested in studying the effects of diet on the attention span of third-grade students in a large city. There are 1,500 third-graders attending the elementary schools in the city. The researcher selects 150 of these third-graders, 30 each in fi ve different schools, as a sample for study.

• An administrator in a large urban high school is inter- ested in student opinions on a new counseling program in the district. There are six high schools and some 14,000 students in the district. From a master list of all students enrolled in the district schools, the administra- tor selects a sample of 1,400 students (350 from each of the four grades, 9–12) to whom he plans to mail a questionnaire asking their opinion of the program.

• The principal of an elementary school wants to inves- tigate the effectiveness of a new U.S. history textbook used by some of the teachers in the district. Out of a total of 22 teachers who are using the text, she selects a sample of 6. She plans to compare the achievement of the students in these teachers’ classes with those of another 6 teachers who are not using the text.

DEFINING THE POPULATION

The fi rst task in selecting a sample is to defi ne the popu- lation of interest. In what group, exactly, is the researcher interested? To whom does he or she want the results of the study to apply? The population, in other words, is the group of interest to the researcher, the group to whom the researcher would like to generalize the results of the study. Here are some examples of populations:

• All high school principals in the United States • All elementary school counselors in the state of

California • All students attending Central High School in Omaha,

Nebraska, during the academic year 2005–2006 • All students in Ms. Brown’s third-grade class at

Wharton Elementary School

The above examples reveal that a population can be any size and that it will have at least one (and sometimes several) characteristic(s) that sets it off from any other population. Notice that a population is always all of the individuals who possess a certain characteristic (or set of characteristics).

In educational research, the population of interest is usually a group of persons (students, teachers, or other

fra97851_ch06_090-109.indd 92fra97851_ch06_090-109.indd 92 12/21/10 6:43 PM12/21/10 6:43 PM

C H A P T E R 6 Sampling 93

Figure 6.1 Representative Versus Nonrepresentative Samples

A population of 40 men and women

20 men 20 women

A representative sample (n=12) (6 women and 6 men)

A nonrepresentative sample (n=12) (2 women and 10 men)

can determine the applicability of the fi ndings to their own situations. Failure to defi ne in detail the population of interest, and the sample studied, is one of the most common weaknesses of published research reports. It is important to note that the actual sample may be different from the sample originally selected because some sub- jects may refuse to participate, some subjects may drop out, data may be lost, and the like. We repeat, therefore, that it is very important to describe the characteristics of the actual sample studied in some detail.

RANDOM VERSUS NONRANDOM SAMPLING

Following is an example of each of the two main types of sampling.

Random sampling: The dean of a school of edu- cation in a large midwestern university wishes to fi nd out how her faculty feel about the current sabbatical leave requirements at the university. She places all 150 names of the faculty in a hat, mixes them thoroughly, and then draws out the names of 25 individuals to interview. *

Nonrandom sampling: The president of the same university wants to know how his junior faculty feel about a promotion policy that he has recently introduced (with the advice of a faculty commit- tee). He selects a sample of 30 from the total fac- ulty of 1,000 to talk with. Five faculty members from each of the six schools that make up the university are chosen on the basis of the follow- ing criteria: They have taught at the university for less than fi ve years, they are nontenured, they belong to one of the faculty associations on campus, and they have not been a member of the committee that helped the president draft the new policy.

In the fi rst example, 25 names were selected from a hat after all the names had been mixed thoroughly. This is called random sampling because every mem- ber of the population (the 150 faculty members in the school) presumably had an equal chance of being se- lected. There are more sophisticated ways of drawing a random sample, but they all have the same intent— to select a representative sample from the population (Figure 6.1). The basic idea is that the group of individ- uals selected is very much like the entire population.

*A better way to do this will be discussed shortly, but this gives you the idea.

fra97851_ch06_090-109.indd 93fra97851_ch06_090-109.indd 93 12/21/10 6:43 PM12/21/10 6:43 PM

94 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

One can never be sure of this, of course, but if the sample is selected randomly and is suffi ciently large, a researcher should get an accurate view of the larger group. The best way to ensure this is to see that no bias enters the selection process—that the researcher (or other factors) cannot consciously or unconsciously infl uence who gets chosen to be in the sample. We ex- plain more about how to minimize bias later in this chapter.

In the second example, the president wants repre- sentativeness, but not as much as he wants to make sure there are certain kinds of faculty in his sample. Thus, he has stipulated that each of the individuals selected must possess all the criteria mentioned. Each member of the population (the entire faculty of the university) does not have an equal chance of being selected; some, in fact, have no chance. Hence, this is an example of nonrandom sampling , sometimes called purposive sampling (see p. 100). Here is an- other example of a random sample contrasted with a nonrandom sample.

Random: A researcher wishes to conduct a sur- vey of all social studies teachers in a midwest- ern state to determine their attitudes toward the new state guidelines for teaching history in the secondary schools. There are a total of 725 social studies teachers in the state. The names of these teachers are obtained and listed alphabetically. The researcher then numbers the names on the list from 001 to 725. Using a table of random numbers, which he finds in a statistics textbook, he selects 100 teachers for the sample.

Nonrandom: The manager of the campus book- store at a local university wants to fi nd out how students feel about the services the bookstore provides. Every day for two weeks during her lunch hour, she asks every person who enters the bookstore to fi ll out a short questionnaire she has prepared and drop it in a box near the entrance before leaving. At the end of the two- week period, she has a total of 235 completed questionnaires.

In the second example, notice that all bookstore users did not have an equal chance of being included in the sample, which included only those who visited during the lunch hour. That is why the sample is not random. Notice also that some may not have completed the questionnaire.

Random Sampling Methods After making a decision to sample, researchers try hard, in most instances, to obtain a sample that is representa- tive of the population of interest—that means they pre- fer random sampling. The three most common ways of obtaining this type of sample are simple random sam- pling, stratifi ed random sampling, and cluster sampling. A less common method is two-stage random sampling.

SIMPLE RANDOM SAMPLING

A simple random sample is one in which each and every member of the population has an equal and in- dependent chance of being selected. If the sample is large, this method is the best way yet devised to obtain a sample representative of the population of interest. Let’s take an example: Defi ne a population as all eighth-grade students in school district Y. Imagine there are 500 stu- dents. If you were one of these students, your chance of being selected would be 1 in 500, if the sampling procedure were indeed random. Everyone would have the same chance of being selected.

The larger a random sample is in size, the more likely it is to represent the population. Although there is no guarantee of representativeness, of course, the likeli- hood of it is greater with large random samples than with any other method. Any differences between the sample and the population should be small and unsys- tematic. Any differences that do occur are the result of chance, rather than bias on the part of the researcher.

The key to obtaining a random sample is to ensure that each and every member of the population has an equal and independent chance of being selected. This can be done by using what is known as a table of ran- dom numbers —an extremely large list of numbers that has no order or pattern. Such lists can be found in the back of most statistics books. Table 6.1 illustrates part of a typical table of random numbers.

For example, to obtain a sample of 200 from a popu- lation of 2,000 individuals, using such a table, select a column of numbers, start anywhere in the column, and begin reading four-digit numbers. (Why four digits? Because the fi nal number, 2,000, consists of four dig- its, and we must always use the same number of digits for each person. Person 1 would be identifi ed as 0001; person 2, as 0002; person 635, as 0635; and so forth.) Then proceed to write down the fi rst 200 numbers in the column that have a value of 2,000 or less.

fra97851_ch06_090-109.indd 94fra97851_ch06_090-109.indd 94 12/21/10 6:43 PM12/21/10 6:43 PM

C H A P T E R 6 Sampling 95

Let us take the fi rst column of four numbers in Table 6.1 as an example. Reading only the fi rst four dig- its, look at the fi rst number in the column: It is 0117, so number 117 in the list of individuals in the population would be selected for the sample. Look at the second number: It is 9123. There is no 9123 in the population (because there are only 2,000 individuals in the entire population). So go on to the third number: It is 0864, hence number 864 in the list of individuals in the pop- ulation would be chosen. The fourth number is 0593, so number 593 gets selected. The fi fth number is 6662. There is no 6662 in the population, so go on to the next number, and so on, until reaching a total of 200 num- bers, each representing an individual in the population who will be selected for the sample. Most researchers use computer-generated lists to obtain their samples randomly. This can be done quite easily using EXCEL software (see the box entitled “Using EXCEL to Draw a Random Sample” in Chapter 11 on p. 235).

The advantage of random sampling is that, if large enough, it is very likely to produce a representative sample. Its biggest disadvantage is that it is not easy to do. Each and every member of the population must be identifi ed. In most cases, we must be able to contact the individuals selected. In all cases, we must know who 117 (for example) is.

Furthermore, simple random sampling is not used if researchers wish to ensure that certain subgroups are present in the sample in the same proportion as they are in the population. To do this, researchers must engage in what is known as stratifi ed sampling.

STRATIFIED RANDOM SAMPLING

Stratified random sampling is a process in which cer- tain subgroups, or strata, are selected for the sample in the same proportion as they exist in the population. Sup- pose the director of research for a large school district wants to fi nd out student response to a new twelfth-grade American government textbook the district is consider- ing adopting. She intends to compare the achievement of students using the new book with that of students using the more traditional text the district has purchased in the past. Since she has reason to believe that gender is an important variable that may affect the outcomes of her study, she decides to ensure that the proportion of males and females in the study is the same as in the population. The steps in the sampling process would be as follows:

1. She identifi es the target (and accessible) population: all 365 twelfth-grade students enrolled in American government courses in the district.

2. She fi nds that there are 219 females (60 percent) and 146 males (40 percent) in the population. She de- cides to have a sample made up of 30 percent of the target population.

3. Using a table of random numbers, she then ran- domly selects 30 percent from each stratum of the population, which results in 66 female (30 percent of 219) and 44 male (30 percent of 146) students being selected from these subgroups. The proportion of males and females is the same in both the popula- tion and sample—40 and 60 percent (Figure 6.2).

The advantage of stratifi ed random sampling is that it increases the likelihood of representativeness, es- pecially if one’s sample is not very large. It virtually ensures that key characteristics of individuals in the population are included in the same proportions in the sample. The disadvantage is that it requires more effort on the part of the researcher.

CLUSTER RANDOM SAMPLING

In both random and stratifi ed random sampling, research- ers want to make sure that certain kinds of individuals are included in the sample. But there are times when it is not possible to select a sample of individuals from a popula- tion. Sometimes, for example, a list of all members of the population of interest is not available. Obviously, then, simple random or stratifi ed random sampling cannot be used. Frequently, researchers cannot select a sample of individuals due to administrative or other restrictions.

Part of a Table of Random Numbers

011723 223456 222167 032762 062281 565451 912334 379156 233989 109238 934128 987678 086401 016265 411148 251287 602345 659080 059397 022334 080675 454555 011563 237873 666278 106590 879809 899030 909876 198905 051965 004571 036900 037700 500098 046660 063045 786326 098000 510379 024358 145678 560132 345678 356789 033460 050521 342021 727009 344870 889567 324588 400567 989657 000037 121191 258700 088909 015460 223350 667899 234345 076567 090076 345121 121348 042397 045645 030032 657112 675897 079326 987650 568799 070070 143188 198789 097451 091126 021557 102322 209312 909036 342045

TABLE 6.1

fra97851_ch06_090-109.indd 95fra97851_ch06_090-109.indd 95 12/21/10 6:43 PM12/21/10 6:43 PM

96 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

This is especially true in schools. For example, if a tar- get population were all eleventh-grade students within a district enrolled in U.S. history courses, it would be unlikely that the researcher could pull out randomly se- lected students to participate in an experimental curricu- lum. Even if it could be done, the time and effort required would make such selection diffi cult. About the best the researcher could hope for would be to study a number of intact classes—that is, classes already in existence. The selection of groups, or clusters, of subjects rather than individuals is known as cluster random sampling . Just as simple random sampling is more effective with larger numbers of individuals, cluster random sampling is more effective with larger numbers of clusters.

Let us consider another example of cluster random sampling. The superintendent of a large unifi ed school dis- trict in a city on the East Coast wants to obtain some idea of how teachers in the district feel about merit pay. There are 10,000 teachers in all the elementary and secondary schools of the district, and there are 50 schools distributed over a large area. The superintendent does not have the funds to survey all teachers in the district, and he needs the information about merit pay quickly. Instead of randomly selecting a sample of teachers from every school, therefore, he decides to interview all the teachers in selected schools. The teachers in each school, then, constitute a cluster. The superintendent assigns a number to each school and

then uses a table of random numbers to select 10 schools (20 percent of the population). All the teachers in the se- lected schools then constitute the sample. The interviewer questions all the teachers at each of these 10 schools, rather than having to travel to all the schools in the district. If these teachers do represent the remaining teachers in the district, then the superintendent is justifi ed in drawing conclusions about the feelings of the entire population of teachers in his district about merit pay. It is possible that this sample is not representative, of course. Because the teachers to be interviewed all come from a small number of schools in the district, it might be the case that these schools differ in some ways from the other schools in the district, thereby infl uencing the views of the teachers in those schools with regard to merit pay. The more schools selected, the more likely the fi ndings will be applicable to the population of teachers (Figure 6.3).

Cluster random sampling is similar to simple random sampling except that groups rather than individuals are randomly selected (that is, the sampling unit is a group rather than an individual). The advantages of cluster ran- dom sampling are that it can be used when it is diffi cult or impossible to select a random sample of individuals, it is often far easier to implement in schools, and it is frequently less time-consuming. Its disadvantage is that there is a far greater chance of selecting a sample that is not representative of the population.

Figure 6.2 Selecting a Stratifi ed Sample

The researcher identifies two subgroups, or strata:

In a population of 365 12th-grade American government students,

and

219 female students (60%)

66 female students (60%)

146 male students (40%)

44 male students (40%)

From these, she randomly selects a stratified sample of:

fra97851_ch06_090-109.indd 96fra97851_ch06_090-109.indd 96 12/21/10 6:43 PM12/21/10 6:43 PM

C H A P T E R 6 Sampling 97

Many beginning researchers make a common error with regard to cluster random sampling: randomly selecting only one cluster as a sample and then observ- ing or interviewing all individuals within that cluster. Even if there is a large number of individuals within the cluster, it is the cluster that has been randomly selected, rather than individuals; hence the researcher is not en- titled to draw conclusions about a target population of such individuals. Yet some researchers do draw such conclusions. We repeat, they should not.

TWO-STAGE RANDOM SAMPLING

It is often useful to combine cluster random sampling with individual random sampling. This is accomplished by two-stage random sampling . Rather than randomly selecting 100 students from a population of 3,000 ninth- graders located in 100 classes, the researcher might de- cide to select 25 classes randomly from the population of 100 classes and then randomly select 4 students from each class. This is much less time-consuming than visit- ing most of the 100 classes. Why would this be better than

using all the students in four randomly selected classes? Because four classes would be too few to ensure repre- sentativeness, even though they were selected randomly.

Figure 6.4 illustrates the different random sampling methods we have discussed.

Nonrandom Sampling Methods SYSTEMATIC SAMPLING

In systematic sampling , every n th individual in the population list is selected for inclusion in the sample. For example, in a population list of 5,000 names, to se- lect a sample of 500, a researcher would select every tenth name on the list until reaching a total of 500 names. Here is an example of this type of sampling: The principal of a large middle school (grades 6–8) with 1,000 students wants to know how students feel about the new menu in the school cafeteria. She obtains an alphabetical list of all students in the school and se- lects every tenth student on the list to be in the sample.

Figure 6.3 Cluster Random Sampling

3

7

2

21

22

8

20

5

46

All teachers in the selected schools are interviewed

In a school district (population) composed of 50 schools, 10 schools are randomly selected, and then all of the teachers in each school are selected.

13

1 18

1817

27

11

10

23

25

41

42

43

34

33

36

35

44 45

12

29 30 31 32

40

38

39

9

14 15

4

16

19

24

48

49

6

5047

37

26

28

fra97851_ch06_090-109.indd 97fra97851_ch06_090-109.indd 97 12/21/10 6:43 PM12/21/10 6:43 PM

98 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

To guard against bias, she puts the numbers 1 to 10 into a hat and draws one out. It is a 3. So she selects the stu- dents numbered 3, 13, 23, 33, 43, and so on until she has a sample of 100 students to be interviewed.

The above method is technically known as system- atic sampling with a random start . In addition, there are two terms that are frequently used when referring to systematic sampling. The sampling interval is the dis- tance in the list between each of the individuals selected for the sample. In the example given above, it was 10. A simple formula to determine it is:

Population size

_________________ Desired sample size

The sampling ratio is the proportion of individuals in the population that is selected for the sample. In the example above, it was .10, or 10 percent. A simple way to determine the sampling ratio is:

Sample size

_____________ Population size

There is a danger in systematic sampling that is sometimes overlooked. If the population has been ordered systematically—that is, if the arrangement of individuals on the list is in some sort of pattern that accidentally coincides with the sampling interval—a markedly biased sample can result. This is sometimes called periodicity . Suppose that the middle school students in the preceding example had not been listed alphabetically but rather by homeroom and that the homeroom teachers had previously listed the students in their rooms by grade point average, high to low. That would mean that the better students would be at the top of each homeroom list. Suppose also that each homeroom had 30 students. If the principal began her selection of every tenth student with the fi rst or second or third student on the list, her sample would consist of the better students in the school rather than a repre- sentation of the entire student body. (Do you see why? Because in each homeroom, the poorest students would be those who were numbered between 24 and 30, and they would never get chosen.)

Figure 6.4 Random Sampling Methods

A E

B D S

T I UC J H X Q

F Y P G NZ

V K O

L R

M W

D H

Y P N

L

A B C D E

F G H I J K L M N O

P Q R S T

25%

50%

25%

CD NO P

AB

EFG

HI

STU

QR LM

JK

CD NO P

AB

EFG

HI

STU

QR LM

JK

Populations

CD

EFG

QR C, L, T

CD

STU

LM

B D 25%

F M O J 50%

P S 25%

Simple random

Stratified random

Cluster random

Random sample of clusters

Two-stage random

Random sample of individuals

Samples

fra97851_ch06_090-109.indd 98fra97851_ch06_090-109.indd 98 12/21/10 6:43 PM12/21/10 6:43 PM

C H A P T E R 6 Sampling 99

When planning to select a sample from a list of some sort, therefore, researchers should carefully examine the list to make sure there is no cyclical pattern present. If the list has been arranged in a particular order, research- ers should make sure the arrangement will not bias the sample in some way that could distort the results. If such seems to be the case, steps should be taken to ensure representativeness—for example, by randomly selecting individuals from each of the cyclical portions. In fact, if a population list is randomly ordered, a systematic sam- ple drawn from the list is a random sample.

CONVENIENCE SAMPLING

Many times it is extremely diffi cult (sometimes even impossible) to select either a random or a systematic nonrandom sample. At such times, a researcher may use convenience sampling . A convenience sample is a group of individuals who (conveniently) are available for study (Figure 6.5). Thus, a researcher might decide to study two third-grade classes at a nearby elementary

school because the principal asks for help in evaluating the effectiveness of a new spelling textbook. Here are some examples of convenience samples:

• To fi nd out how students feel about food service in the student union at an East Coast university, the manager stands outside the main door of the cafe- teria one Monday morning and interviews the fi rst 50 students who walk out of the cafeteria.

• A high school counselor interviews all the students who come to him for counseling about their career plans.

• A news reporter for a local television station asks passersby on a downtown street corner their opin- ions about plans to build a new baseball stadium in a nearby suburb.

• A university professor compares student reactions to two different textbooks in her statistics classes. In each of the above examples, a certain group of

people was chosen for study because they were avail- able. The obvious advantage of this type of sampling is

Professor

A class of 40 math students. The professor selects a convenience sample of 10 students who are (conveniently) in the front two rows to ask how they like the textbook.

40393837363534333231

30292827262524232221

20191817161514131211

10987654321

1716151413

76543

A convenience sample

Figure 6.5 Convenience Sampling

fra97851_ch06_090-109.indd 99fra97851_ch06_090-109.indd 99 12/21/10 6:43 PM12/21/10 6:43 PM

100 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

convenience. But just as obviously, it has a major dis- advantage in that the sample will quite likely be biased. Take the case of the TV reporter who is interviewing passersby on a downtown street corner. Many possible sources of bias exist. First of all, of course, anyone who is not downtown that day has no chance to be inter- viewed. Second, those individuals who are unwilling to give their views will not be interviewed. Third, those who agree to be interviewed will probably be individu- als who hold strong opinions one way or the other about the stadium. Fourth, depending on the time of day, those who are interviewed quite possibly will be unemployed or have jobs that do not require them to be indoors. And so forth.

In general, convenience samples cannot be consid- ered representative of any population and should be avoided if at all possible. Unfortunately, sometimes they are the only option a researcher has. When such is the case, the researcher should be especially careful to include information on demographic and other char- acteristics of the sample studied. The study should also be replicated, that is, repeated, with a number of simi- lar samples to decrease the likelihood that the results obtained were simply a one-time occurrence. We will discuss replication in more depth later in the chapter.

PURPOSIVE SAMPLING

On occasion, based on previous knowledge of a popu- lation and the specifi c purpose of the research, inves- tigators use personal judgment to select a sample. Researchers assume they can use their knowledge of the population to judge whether or not a particular sample will be representative. Here are some examples:

• An eighth-grade social studies teacher chooses the 2 students with the highest grade point averages in her class, the 2 whose grade point averages fall in the middle of the class, and the 2 with the lowest grade point averages to fi nd out how her class feels about including a discussion of current events as a regular part of classroom activity. Similar samples in the past have represented the viewpoints of the total class quite accurately.

• A graduate student wants to know how retired people age 65 and over feel about their “golden years.” He has been told by one of his professors, an expert on aging and the aged population, that the local Asso- ciation of Retired Workers is a representative cross section of retired people age 65 and over. He decides

to interview a sample of 50 people who are members of the association to get their views.

In both of these examples, previous information led the researcher to believe that the sample selected would be representative of the population. There is a second form of purposive sampling in which it is not expected that the persons chosen are themselves representative of the population, but rather that they possess the neces- sary information about the population. For example:

• A researcher is asked to identify the unoffi cial power hierarchy in a particular high school. She decides to interview the principal, the union representative, the principal’s secretary, and the school custodian because she has prior information that leads her to believe they are the people who possess the informa- tion she needs.

• For the past fi ve years, the leaders of the teachers’ association in a midwestern school district have rep- resented the views of three-fourths of the teachers in the district on most major issues. This year, there- fore, the district administration decides to interview just the leaders of the association rather than select a sample from all the district’s teachers.

Purposive sampling is different from convenience sampling in that researchers do not simply study who- ever is available but rather use their judgment to select a sample that they believe, based on prior information, will provide the data they need. The major disadvantage of purposive sampling is that the researcher’s judgment may be in error—he or she may not be correct in estimat- ing the representativeness of a sample or their expertise regarding the information needed. In the second exam- ple above, this year’s leaders of the teachers’ association may hold views markedly different from those of their members. Figure 6.6 illustrates the methods of conve- nience, purposive, and systematic sampling.

A Review of Sampling Methods Let us illustrate each of the previous sampling methods using the same hypothesis: “Students with low self-es- teem demonstrate lower achievement in school subjects.”

Target population: All eighth-graders in California. Accessible population: All eighth-graders in the

San Francisco Bay Area (seven counties). Feasible sample size: n 5 200 2 250.

fra97851_ch06_090-109.indd 100fra97851_ch06_090-109.indd 100 12/21/10 6:43 PM12/21/10 6:43 PM

C H A P T E R 6 Sampling 101

Simple random sampling: Identify all eighth- graders in all public and private schools in the seven counties (estimated number of eighth-grade students 5 9,000). Assign each student a number, and then use a table of random numbers to select a sample of 200. The diffi culty here is that it is time-consuming to identify every eighth-grader in the Bay Area and to contact (probably) about 200 different schools in order to administer instru- ments to one or two students in those schools.

Cluster random sampling: Identify all public and private schools having an eighth grade in the seven counties. Assign each of the schools a num- ber, and then randomly select four schools and in- clude all eighth-grade classes in each school. (We would estimate 2 classes per school × 30 students per class × 4 schools 5 a total of 240 students.) Cluster random sampling is much more feasible than simple random sampling to implement, but it is limited because of the use of only four schools, even though they are to be selected randomly. For example, the selection of only four schools may exclude the selection of private-school students.

Stratifi ed random sampling: Obtain data on the number of eighth-grade students in public versus private schools and determine the proportion of each type (e.g., 80 percent public, 20 percent pri- vate). Determine the number from each type to be sampled: public 5 80 percent of 200 5 160; private 5 20 percent of 200 5 40. Randomly se- lect samples of 160 and 40 students from respec- tive subpopulations of public and private students. Stratifi cation may be used to ensure that the sam- ple is representative on other variables as well. The diffi culty with this method is that stratifi ca- tion requires that the researcher know the propor- tions in each stratum of the population, and it also becomes increasingly diffi cult as more variables are added. Imagine trying to stratify not only on the public-private variable but also (for example) on student ethnicity, gender, and socioeconomic status, and on teacher gender and experience.

Two-stage random sampling: Randomly select 25 schools from the accessible population of schools, and then randomly select 8 eighth-grade students from each school ( n 5 8 3 25 5 200).  This

Figure 6.6 Nonrandom Sampling Methods

Populations

Z

A B C

D E

FG H

I

J

K

L

M N

O

P Q

R

S T U

V W X

Y

Z

A B

C D

E F

G H IJ

K L

M N

O

P Q

R S

T

U V

W

X

Y Z

U V W X Y P Q R S T K L M N O F G H I J A B C D E

Especially qualified

Easily accessible

Convenience Purposive Systematic

NL

B F

VI

LQ X

Y Q

LB G

V

SAMPLES

fra97851_ch06_090-109.indd 101fra97851_ch06_090-109.indd 101 12/21/10 6:43 PM12/21/10 6:43 PM

102

method is much more feasible than simple random sampling and more representative than cluster sam- pling. It may well be the best choice in this exam- ple, but it still requires permission from 25 schools and the resources to collect data from each.

Convenience sampling: Select all eighth-graders in four schools to which the researcher has access (again, we estimate two classes of 30 students per school, so n 5 30 × 4 × 2 5 240). This method precludes generalizing beyond these four schools, unless a strong argument with supporting data can be made for their similarity to the entire group of accessible schools.

Purposive sampling: Select 8 classes from throughout the seven counties on the basis of demographic data showing that they are representative of all eighth- graders. Particular attention must be paid to self- esteem and achievement scores. The problem is that such data are unlikely to be available and, in any case, cannot eliminate possible differences between the sample and the population on other variables— such as teacher attitude and available resources.

Systematic sampling: Select every 45th student from an alphabetical list for each school.

200 students in sample

_______________________ 9,000 students in population

5 1 ___ 45

This method is almost as inconvenient as simple ran- dom sampling and is likely to result in a biased sample, since the 45th name in each school is apt to be in the last third of the alphabet (remember there are an estimated 60 eighth-graders in each school), introducing probable ethnic or cultural bias.

Sample Size Drawing conclusions about a population after studying a sample is never totally satisfactory, since researchers can never be sure that their sample is perfectly representative of the population. Some differences between the sample and the population are bound to exist, but if the sample is randomly selected and of suffi cient size, these differences are likely to be relatively insignifi cant and incidental. The question remains, therefore, as to what constitutes an adequate, or suffi cient, size for a sample.

Unfortunately, there is no clear-cut answer to this ques- tion. Suppose a target population consists of 1,000 eighth- graders in a given school district. Some sample sizes, of course, are obviously too small. Samples with 1 or 2 or 3 individuals, for example, are so small that they can- not possibly be representative. Probably any sample that has less than 20 to 30 individuals is too small, since that would only be 2 or 3 percent of the population. On the other hand, a sample can be too large, given the amount of time and effort the researcher must put into obtaining it. In this example, a sample of 250 or more individuals would probably be needlessly large, as that would constitute a quarter of the population. But what about samples of 50 or 100? Would these be suffi ciently large? Would a sample of 200 be too large? At what point, exactly, does a sample stop being too small and become suffi ciently large? The best answer is that a sample should be as large as the re- searcher can obtain with a reasonable expenditure of time and energy. This, of course, is not as much help as one would like, but it suggests that researchers should try to obtain as large a sample as they reasonably can.

CONTROVERSIES IN RESEARCH

Sample or Census?

S amples look at only part of the population. A census tries to look at the entire population. The U.S. Census Bureau, charged with conducting the United States Census every 10 years, estimated that the 2000 census missed 1–2 percent of the population—3.4 million people, including 1.2 percent of the African Americans who lived largely in the inner city. The pro- cedure for taking a census consists of sending out mailings and following up with door-to-door canvassing of non-respondents.

Some statisticians have proposed augmenting the head- count by surveying a separate representative sample and using these data to estimate the size and demographics of non-respondents. Supporters of the idea argue that this would provide a better picture of the population; opponents say that the assumptions involved, along with processing errors, would produce more error.

It can be argued that a sizable random sample of the entire population accompanied by more extensive follow-up would provide more accurate data than the current procedure at no greater expense, but this is precluded by the Constitution. (For more on this topic, search the Internet for national census sampling.)

fra97851_ch06_090-109.indd 102fra97851_ch06_090-109.indd 102 12/21/10 6:43 PM12/21/10 6:43 PM

103

There are a few guidelines that we would suggest with regard to the minimum number of subjects needed. For descriptive studies, we think a sample with a minimum number of 100 is essential. For correlational studies, a sample of at least 50 is deemed necessary to establish the existence of a relationship. For experimental and causal- comparative studies, we recommend a minimum of 30 individuals per group, although sometimes experimen- tal studies with only 15 individuals in each group can be defended if they are very tightly controlled; studies using only 15 subjects per group should probably be rep- licated, however, before too much is made of any fi nd- ings. * In qualitative studies, the number of participants in a sample is usually somewhere between 1 and 20.

External Validity: Generalizing from a Sample As indicated earlier in this chapter, researchers general- ize when they apply the fi ndings of a particular study to people or settings that go beyond the particular people or settings used in the study. The whole notion of sci- ence is built on the idea of generalizing . Every science

The Diffi culty in Generalizing from a Sample

I n 1936 the Literary Digest, a popular magazine of the time, se-lected a sample of voters in the United States and asked them for whom they would vote in the upcoming presidential election— Alf Landon (Republican) or Franklin Roosevelt (Democrat). The magazine editors obtained a sample of 2,375,000 individu- als from lists of automobile and telephone owners in the United States (about 20 percent returned the mailed postcards). On the basis of their fi ndings, the editors predicted that Landon would win by a landslide. In fact, it was Roosevelt who won the land- slide victory. What was wrong with the study?

Certainly not the size of the sample. The most frequent ex- planations have been that the data were collected too far ahead

MORE ABOUT RESEARCH

of the election and that a lot of people changed their minds, and/or that the sample of voters was heavily biased in favor of the more affl uent, and/or that the 20 percent return rate intro- duced a major bias. What do you think?

A common misconception among beginning researchers is illustrated by the following statement: “Although I obtained a random sample only from schools in San Francisco, I am entitled to generalize my fi ndings to the entire state of Califor- nia because the San Francisco schools (and hence my sample) refl ect a wide variety of socioeconomic levels, ethnic groups, and teaching styles.” The statement is incorrect because vari- ety is not the same thing as representativeness. In order for the San Francisco schools to be representative of all the schools in California, they must be very similar (ideally, identical) with respect to characteristics such as the ones mentioned. Ask yourself: Are the San Francisco schools representative of the entire state with regard to ethnic composition of students? The answer, of course, is that they are not.

*More specifi c guidelines are provided in the Research Tips box on page 234 in Chapter 11.

seeks to fi nd basic principles or laws that can be applied to a great variety of situations and, in the case of the social sciences, to a great many people. Most research- ers wish to generalize their fi ndings to appropriate pop- ulations. But when is generalizing warranted? When can researchers say with confi dence that what they have learned about a sample is also true of the population? Both the nature of the sample and the environmental conditions—the setting—within which a study takes place must be considered in thinking about generaliz- ability. The extent to which the results of a study can be generalized determines the external validity of the study. In the next two chapters, we also discuss how the concept of validity is applied to instruments (instrument validity) and to the internal design of a study.

POPULATION GENERALIZABILITY

Population generalizability refers to the degree to which a sample represents the population of interest. If the results of a study only apply to the group being studied and if that group is fairly small or is narrowly defi ned, the usefulness of any fi ndings is seriously lim- ited. This is why trying to obtain a representative sam- ple is so important. Because conducting a study takes a considerable amount of time, energy, and (frequently) money, researchers usually want the results of an inves- tigation to be as widely applicable as possible.

fra97851_ch06_090-109.indd 103fra97851_ch06_090-109.indd 103 12/21/10 6:43 PM12/21/10 6:43 PM

104 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

When we speak of representativeness , however, we are referring only to the essential, or relevant, charac- teristics of a population. What do we mean by relevant? Only that the characteristics referred to might possibly be a contributing factor to any results that are obtained. For example, if a researcher wished to select a sample of fi rst- and second-graders to study the effect of read- ing method on pupil achievement, such characteris- tics as height, eye color, or jumping ability would be judged to be irrelevant—that is, we would not expect any variation in them to have an effect on how easily a child learns to read, and hence we would not be overly concerned if those characteristics were not adequately represented in the sample. Other characteristics, such as age, gender, or visual acuity, on the other hand, might (logically) have an effect and hence should be appropri- ately represented in the sample.

Whenever purposive or convenience samples are used, generalization is made more plausible if data are presented to show that the sample is representative of the intended population on at least some relevant variables. This procedure, however, can never guarantee representativeness on all relevant variables.

One aspect of generalizability that is often over- looked in “methods” or “treatment” studies pertains to the teachers, counselors, administrators, or others who administer the various treatments. We must remember that such studies involve not only a sample of students, clients, or other recipients of the treatments but also a sample of those who implement the various treatments. Thus, a study that randomly selects students but not teachers is only entitled to generalize the outcomes to the population of students— if they are taught by the same teachers. To generalize the results to other teach- ers, the sample of teachers must also be selected ran- domly and must be suffi ciently large.

Finally, we must remember that the sample in any study is the group about whom data are actually obtained. The best sampling plan is of no value if in- formation is missing on a sizable portion of the initial sample. Once the sample has been selected, every ef- fort must be made to ensure that the necessary data are obtained on each person in the sample. This is often diffi cult to do, particularly with questionnaire- type survey studies, but the results are well worth the time and energy expended. Unfortunately, there are no clear guidelines as to how many subjects can be lost before representativeness is seriously impaired. Any researchers who lose over 10 percent of the originally selected sample would be well advised to

acknowledge this limitation and qualify their conclu- sions accordingly.

Do researchers always want to generalize? The only time researchers are not interested in generalizing beyond the confi nes of a particular study is when the results of an investigation are of interest only as ap- plied to a particular group of people at a particular time, and where all of the members of the group are included in the study. An example might be the opinions of an elementary school faculty on a specifi c issue such as whether to implement a new math program. This might be of value to that faculty for decision making or program planning, but not to anyone else.

WHEN RANDOM SAMPLING IS NOT FEASIBLE

As we have shown, sometimes it is not feasible or even possible to obtain a random sample. When this is the case, researchers should describe the sample as thor- oughly as possible (detailing, for example, age, gender, ethnicity, and socioeconomic status) so that interested others can judge for themselves the degree to which any fi ndings apply, and to whom and where. This is clearly an inferior procedure compared to random sampling, but sometimes it is the only alternative one has.

There is another possibility when a random sample is impossible to obtain: It is called replication . The re- searcher (or other researchers) repeats the study using different groups of subjects in different situations. If a study is repeated several times, using different groups of subjects and under different conditions of geography, socioeconomic level, ability, and so on, and if the re- sults obtained are essentially the same in each case, a researcher may have additional confi dence about gener- alizing the fi ndings.

In the vast majority of studies that have been done in education, random samples have not been used. There seem to be two reasons for this. First, educational re- searchers may be unaware of the hazards involved in generalizing when one does not have a random sample. Second, in many studies it is simply not feasible for a researcher to invest the time, money, or other resources necessary to obtain a random sample. For the results of a particular study to be applicable to a larger group, then, the researcher must argue convincingly that the sample employed, even though not chosen randomly, is in fact representative of the target population. This is diffi cult, however, and always subject to contrary arguments.

fra97851_ch06_090-109.indd 104fra97851_ch06_090-109.indd 104 12/21/10 6:43 PM12/21/10 6:43 PM

C H A P T E R 6 Sampling 105

ECOLOGICAL GENERALIZABILITY

Ecological generalizability refers to the degree to which the results of a study can be extended to other settings or conditions. Researchers must make clear the nature of the environmental conditions—the setting— under which a study takes place. These conditions must be the same in all important respects in any new situa- tion in which researchers wish to assert that their fi nd- ings apply. For example, it is not justifi able to generalize from studies on the effects of a new reading program on third-graders in a large urban school system to teach- ing mathematics, even to those students in that system. Research results from urban school environments may not apply to suburban or rural school environments; results obtained with transparencies may not apply to those with textbooks. What holds true for one subject, or with certain materials, or under certain conditions, or at certain times may not generalize to other subjects, materials, conditions, or times.

An example of inappropriate ecological generalizing occurred in a study that found that a particular method of instruction applied to map reading resulted in greater transfer to general map interpretation on the part of

fi fth-graders in several schools. The researcher accord- ingly recommended that the method of instruction be used in other content areas, such as mathematics and science, overlooking differences in content, materials, and skills involved, in addition to probable differences in resources, teacher experience, and the like. Improper ecological generalizing such as this remains the bane of much educational research.

Unfortunately, application of the powerful tech- nique of random sampling is virtually never possible with respect to ecological generalizing. While it is conceivable that a researcher could identify “popula- tions” of organization patterns, materials, classroom conditions, and so on, and then randomly select a siz- able number of combinations from all possible com- binations, the logistics of doing so boggle the mind. Therefore, researchers must be cautious about gen- eralizing the results from any one study. Only when outcomes have been shown to be similar through rep- lication across specifi c environmental conditions can we generalize across those conditions. Figure 6.7 illus- trates the difference between population and ecologi- cal generalizing.

Figure 6.7 Population as Opposed to Ecological Generalizing

A

B D

E

F G H IJ

K

M

N

O

P Q

R

S T U

V

W Y

Z

Population

POPULATION GENERALIZING

Sample

D H

NP L Y

Other textbooks

Other methods

Other technology

Other content area(s)

Other aspects

ECOLOGICAL GENERALIZING

Environmental conditions

Textbooks used

in study

Methods used

in study

Audio- visual

materials used

Content area(s)

in study

?

fra97851_ch06_090-109.indd 105fra97851_ch06_090-109.indd 105 12/21/10 6:43 PM12/21/10 6:43 PM

106 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

Go back to the INTERACTIVE AND APPLIED LEARNING feature at the beginning of the chapter for a listing of interactive and applied activities. Go to the Online Learning Center at www.mhhe.com/fraenkel8e to take quizzes, practice with key terms, and review chapter content.

SAMPLES AND SAMPLING

• The term sampling, as used in research, refers to the process of selecting the individuals who will participate (e.g., be observed or questioned) in a research study.

• A sample is any part of a population of individuals on whom information is ob- tained.  It may, for a variety of reasons, be different from the sample originally selected.

SAMPLES AND POPULATIONS

• The term population, as used in research, refers to all the members of a particular group. It is the group of interest to the researcher, the group to whom the researcher would like to generalize the results of a study.

• A target population is the actual population to whom the researcher would like to generalize; the accessible population is the population to whom the researcher is entitled to generalize.

• A representative sample is a sample that is similar to the population on all characteristics.

RANDOM VERSUS NONRANDOM SAMPLING

• Sampling may be either random or nonrandom. Random sampling methods include simple random sampling, stratifi ed random sampling, cluster random sampling, and two-stage random sampling. Nonrandom sampling methods include systematic sam- pling, convenience sampling, and purposive sampling.

RANDOM SAMPLING METHODS

• A simple random sample is a sample selected from a population in such a manner that all members of the population have an equal chance of being selected.

• A stratifi ed random sample is a sample selected so that certain characteris- tics are  represented in the sample in the same proportion as they occur in the population.

• A cluster random sample is one obtained by using groups as the sampling unit rather than individuals.

• A two-stage random sample selects groups randomly and then chooses individuals randomly from these groups.

• A table of random numbers lists and arranges numbers in no particular order and can be used to select a random sample.

Main Points

fra97851_ch06_090-109.indd 106fra97851_ch06_090-109.indd 106 12/21/10 6:43 PM12/21/10 6:43 PM

C H A P T E R 6 Sampling 107

NONRANDOM SAMPLING METHODS

• A systematic sample is obtained by selecting every n th name in a population. • A convenience sample is any group of individuals that is conveniently available to

be studied. • A purposive sample consists of individuals who have special qualifi cations of some

sort or are deemed representative on the basis of prior evidence.

SAMPLE SIZE

• Samples should be as large as a researcher can obtain with a reasonable expenditure of time and energy. A recommended minimum number of subjects is 100 for a de- scriptive study, 50 for a correlational study, and 30 in each group for experimental and causal-comparative studies.

EXTERNAL VALIDITY (GENERALIZABILITY)

• The term external validity, as used in research, refers to the extent that the results of a study can be generalized from a sample to a population.

• The term population generalizability refers to the extent to which the results of a study can be generalized to the intended population.

• The term ecological generalizability refers to the extent to which the results of a study can be generalized to conditions or settings other than those that prevailed in a particular study.

REPLICATION

• When a study is replicated, it is repeated with a new sample and sometimes under new conditions.

accessible population 92 cluster random

sampling 96

convenience sampling 99

ecological generalizability 105

external validity 103

generalizing 103

nonrandom sampling 94

periodicity 98

population 91

population generalizability 103

purposive sampling 100

random sampling 93

random start 98

replication 104

representativeness 104

sample 91

sampling 91

sampling interval 98

sampling ratio 98

simple random sample 94

stratifi ed random sampling 95

systematic sampling 97

table of random numbers 94

target population 92

two-stage random sampling 97

Key Terms

1. A team of researchers wants to determine student attitudes about the recreational ser- vices available in the student union on campus. The team stops the fi rst 100 students it meets on a street in the middle of the campus and asks each of them questions about the union. What are some possible ways that this sample might be biased?

For Discussion

fra97851_ch06_090-109.indd 107fra97851_ch06_090-109.indd 107 12/21/10 6:43 PM12/21/10 6:43 PM

108 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

2. Suppose a researcher is interested in studying the effects of music on learning. He obtains permission from a nearby elementary school principal to use the two third- grade classes in the school. The ability level of the two classes, as shown by stan- dardized tests, grade point averages, and faculty opinion, is quite similar. In one class, the researcher plays classical music softly every day for a semester. In the other class, no music is played. At the end of the semester, he fi nds that the class in which the music was played has a markedly higher average in arithmetic than the other class, although they do not differ in any other respect. To what population (if any) might the results of this study be generalized? What, exactly, could the re- searcher say about the effects of music on learning?

3. When, if ever, might a researcher not be interested in generalizing the results of a study? Explain.

4. “The larger a sample, the more justifi ed a researcher is in generalizing from it to a population.” Is this statement true? Why or why not?

5. Some people have argued that no population can ever be studied in its entirety. Would you agree? Why or why not?

6. “The more narrowly researchers defi ne the population, the more they limit general- izability.” Is this always true? Discuss.

7. “The best sampling plan is of no value if information is missing on a sizable propor- tion of the initial sample.” Why is this so? Discuss.

8. “The use of random sampling is almost never possible with respect to ecological generalizing.” Why is this so? Can you think of a possible study for which ecologi- cal generalizing would be possible? If so, give an example.

fra97851_ch06_090-109.indd 108fra97851_ch06_090-109.indd 108 12/21/10 6:43 PM12/21/10 6:43 PM

C H A P T E R 6 Sampling 109

Research Exercise 6: Sampling Plan Use Problem Sheet 6 to describe, as fully as you can, your sample—that is, the subjects you will include in your study. Describe the type of sample you plan to use and how you will obtain the sample. Indicate whether you expect your study to have population generalizability. If so, to what population? if not, why not? Then indicate whether the study would have ecological generalizability. If so, to what settings? if not, why would it not?

Problem Sheet 6

Sampling Plan 1. My intended sample (participants in your study) consists of ( state who and how many ):

2. Key demographics (characteristics of the sample) are as follows (e.g., age range, sex distribution, ethnic breakdown, socioeconomic status, location [where are these sub- jects located?], etc.):

3. State what type of sample you plan to use (i.e., convenience, purposive, simple random, stratifi ed random, cluster, systematic).

4. I will gain access to and/or get contact information for my sample through the following steps:

5. What, if any, are the inclusion/exclusion criteria for participation in your study?

6. External validity:

a. To whom do you think you can generalize the results of your study? Explain.

b. If applicable, to what settings/conditions could you generalize the results of your study (ecological validity)?

c. If results are not generalizable, why not?

An electronic version of this Problem Sheet that you can fi ll in and print, save, or e-mail is available on the Online Learning Center at www.mhhe.com/ fraenkel8e.

fra97851_ch06_090-109.indd 109fra97851_ch06_090-109.indd 109 12/21/10 6:43 PM12/21/10 6:43 PM

7

O B J E C T I V E S Studying this chapter should enable you to:

• Explain what is meant by the term “data.” • Explain what is meant by the term

“instrumentation.” • Name three ways in which data can be

collected by researchers. • Explain what is meant by the term “data-

collection instrument.” • Describe fi ve types of researcher-

completed instruments used in educational research.

• Describe fi ve types of subject-completed instruments used in educational research.

• Explain what is meant by the term “unobtrusive measures” and give two examples of such measures.

• Name four types of measurement scales and give an example of each.

• Name three different types of scores used in educational research and give an example of each.

• Describe briefl y the difference between norm-referenced and criterion-referenced instruments.

• Describe briefl y how to score, tabulate, and code data for analysis.

What Are Data?

Key Questions Validity, Reliability, and

Objectivity Usability

Means of Classifying Data-Collection Instruments

Who Provides the Information?

Where Did the Instrument Come from?

Written Response Versus Performance

Examples of Data- Collection Instruments

Researcher-Completed Instruments

Subject-Completed Instruments

Unobtrusive Measures

Norm-Referenced Versus Criterion-Referenced Instruments

Norm-Referenced Instruments Criterion-Referenced

Instruments

Measurement Scales

Nominal Scales Ordinal Scales Interval Scales Ratio Scales Measurement Scales

Reconsidered

Preparing Data for Analysis

Scoring the Data Tabulating and Coding

the Data

Instrumentation

“Eternity means forever.”

“Oh yeah, what about eternity?”

“Couldn’t agree more, buddy, but there still ain’t no way to measure it.”

“If something exists, it can

be measured!”

fra97851_ch07_110-145.indd 110fra97851_ch07_110-145.indd 110 22/12/10 9:16 PM22/12/10 9:16 PM

INTERACTIVE AND APPLIED LEARNING After, or while, reading this chapter:

Go to the Online Learning Center at www.mhhe.com/fraenkel8e to:

• Learn More About Developing Instruments

Go to your online Student Mastery Activities book to do the following activities:

• Activity 7.1: Major Categories of Instruments and Their Uses • Activity 7.2: Which Type of Instrument Is Most Appropriate? • Activity 7.3: Types of Scales • Activity 7.4: Norm-Referenced vs. Criterion-Referenced

Instruments • Activity 7.5: Developing a Rating Scale • Activity 7.6: Design an Instrument

M onica Stuart and Ben Colen are discussing last night’s lecture in their educational research class. “I must admit that I was pretty impressed,” says Monica. “How come?”

“That list of measuring instruments Professor Fraenkel described last night. Questionnaires, rating scales, true-false tests,

sociograms, logs, anecdotal records, tally sheets, . . . It just went on and on. I never knew there were so many ways to measure

something—so many measuring devices. And what he said about measurement—that really impressed me! Remember? He said

that if something exists, you can measure it!”

“Yeah, it was quite a feat. I have to admit that. I don’t know about the idea that everything can be measured, though. What

about something that is very abstract?”

“Like what?”

“Well, how about, say, alienation? or motivation? How would you measure them? I’m not sure they even can be measured!”

“Well, here’s what I’d do,” says Monica. “I would . . .”

How would you measure motivation? Do instruments exist that could measure something this abstract? To get some ideas,

read this chapter.

What Are Data? The term data refers to the kinds of information researchers obtain on the subjects of their research. Demographic information, such as age, gender, ethnic- ity, religion, and so on, is one kind of data; scores from a commercially available or researcher-prepared test are another. Responses to the researcher’s questions in an oral interview or written replies to a survey question- naire are other kinds. Essays written by students, grade point averages obtained from school records, perfor- mance logs kept by coaches, anecdotal records main- tained by teachers or counselors—all constitute various kinds of data that researchers might want to collect as part of a research investigation. An important decision

for every researcher to make during the planning phase of an investigation, therefore, is what kind(s) of data he or she intends to collect. The device (such as a pencil- and-paper test, a questionnaire, or a rating scale) the researcher uses to collect data is called an instrument. *

KEY QUESTIONS

Generally, the whole process of preparing to collect data is called instrumentation . It involves not only

*Most, but not all, research requires the use of an instrument. In studies where data are obtained exclusively from existing records (grades, attendance, etc.), no instrument is needed.

111

fra97851_ch07_110-145.indd 111fra97851_ch07_110-145.indd 111 22/12/10 9:16 PM22/12/10 9:16 PM

112 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

individuals. * But to be of any use, these inferences must be correct. All researchers, therefore, want instruments that permit them to draw warranted, or valid, conclu- sions about the characteristics (ability, achievement, attitudes, and so on) of the individuals they study.

To measure math achievement, for example, a re- searcher needs to have some assurance that the instrument she intends to use actually does measure such achieve- ment. Another researcher who wants to know what peo- ple think or how they feel about a particular topic needs assurance that the instrument used will allow him to make accurate inferences. There are various ways to obtain such assurance, and we discuss them in Chapter 8.

A second consideration is reliability . A reliable instrument is one that gives consistent results. If a re- searcher tested the math achievement of a group of indi- viduals at two or more different times, for example, he or she should expect to obtain close to the same results each time. This consistency would give the researcher confi dence that the results actually represented the achievement of the individuals involved. As with valid- ity, a number of procedures can be used to determine the reliability of an instrument. We discuss several of them in Chapter 8.

A fi nal consideration is objectivity. Objectivity refers to the absence of subjective judgments. Whenever possible, researchers should try to eliminate subjectivity from the judgments they make about the achievement, performance, or characteristics of subjects. Unfortu- nately, complete objectivity is probably never attained. Acknowledging this, qualitative researchers openly seek ways to address subjectivity in their research studies in order to minimize bias and maximize validity.

We discuss each of these concepts in much more detail in Chapter 8. In this chapter, we look at some of the vari- ous kinds of instruments that can be (and often are) used in research and discuss how to fi nd and select them.

USABILITY

A number of practical considerations face every re- searcher. One of these is how easy it will be to use any instrument he or she designs or selects. How long will it take to administer? Are the directions clear? Is it appropriate for the ethnic or other groups to whom

the selection or design of the instruments but also the procedures and the conditions under which the instruments will be administered. Several questions arise:

1. Where will the data be collected? This question re- fers to the location of the data collection. Where will it be? in a classroom? a schoolyard? a private home? on the street?

2. When will the data be collected? This question refers to the time of collection. When is it to take place? in the morning? afternoon? evening? over a weekend?

3. How often are the data to be collected? This ques- tion refers to the frequency of collection. How many times are the data to be collected? only once? twice? more than twice?

4. Who is to collect the data? This question refers to the administration of the instruments. Who is to do this? the researcher? someone selected and trained by the researcher?

These questions are important because how researchers answer them may affect the data obtained. It is a mistake to think that researchers need only locate or develop a “good” instrument. The data provided by any instrument may be affected by any or all of the preceding considerations. The most highly regarded of instruments will provide useless data, for instance, if administered incorrectly; by someone disliked by respondents; under noisy, inhospitable conditions; or when subjects are exhausted.

All the above questions are important for research- ers to answer, therefore, before they begin to collect the data they need. A researcher’s decisions about location, time, frequency, and administration are always affected by the kind(s) of instrument to be used. And for it to be of any value, every instrument, no matter what kind, must allow researchers to draw accurate conclusions about the capabilities or other characteristics of the peo- ple being studied.

VALIDITY, RELIABILITY, AND OBJECTIVITY

A frequently used (but somewhat old-fashioned) defi - nition of a valid instrument is that it measures what it is supposed to measure. A more accurate defi nition of validity revolves around the defensibility of the infer- ences researchers make from the data collected through the use of an instrument. An instrument, after all, is a de- vice used to gather data. Researchers then use these data to make inferences about the characteristics of certain

* Sometimes instruments are used to collect data on something other than individuals (such as groups, programs, and environments), but since most of the time we are concerned with individuals in educa- tional research, we use this terminology throughout our discussion.

fra97851_ch07_110-145.indd 112fra97851_ch07_110-145.indd 112 22/12/10 9:16 PM22/12/10 9:16 PM

C H A P T E R 7 Instrumentation 113

• A researcher interested in the concept of mutual attraction describes in ongoing fi eld notes how the behaviors of people who work together in various settings have been observed to differ on this variable.

2. Subject instruments • A researcher in an elementary school adminis-

ters a weekly spelling test that requires students to spell correctly the new words learned in class during the week.

• At a researcher’s request, an administrator passes out a questionnaire during a faculty meeting that asks the faculty’s opinions about the new mathe- matics curriculum recently instituted in the district.

• A researcher asks high school English teachers to have their students keep a daily log in which they record their reactions to the plays they read each week.

3. Informant instruments • A researcher asks teachers to use a rating scale to

rate each of their students on their phonic reading skills.

• A researcher asks parents to keep anecdotal re- cords describing the TV characters their pre- schoolers spontaneously role-play.

• A researcher interviews the president of the stu- dent council about student views on the school’s disciplinary code. Her responses are recorded on an interview schedule .

WHERE DID THE INSTRUMENT COME FROM?

There are essentially two basic ways for a researcher to acquire an instrument: (1) fi nd and administer a previ- ously existing instrument of some sort or (2) administer an instrument the researcher personally developed or had developed by someone else.

Developing an instrument has its problems. Primar- ily, it is not easy to do. Developing a “good” instrument usually takes a fair amount of time and effort, not to mention a considerable amount of skill.

Selecting an already developed instrument when ap- propriate, therefore, is preferred. Such instruments are usually developed by experts who possess the necessary skills. Choosing an instrument that has already been de- veloped takes far less time than developing a new instru- ment to measure the same thing.

Designing one’s own instrument is time-consuming, and we do not recommend it for those without a consid- erable amount of time, energy, and money to invest in

it will be administered? How easy is it to score? Are the results easy to interpret? How much does it cost? Do equivalent forms exist? Have any problems been re- ported by others who used it? Does evidence of its reli- ability and validity exist? Getting satisfactory answers to such questions can save a researcher a lot of time and energy and can prevent a lot of headaches.

Means of Classifying Data-Collection Instruments Instruments can be classifi ed in a number of ways. Here are some of the most useful.

WHO PROVIDES THE INFORMATION?

In educational research, three general methods are avail- able for obtaining information. Researchers can get the information (1) themselves, with little or no involvement of other people; (2) directly from the subjects of the study; or (3) from others, frequently referred to as informants , who are knowledgeable about the subjects. Let us follow a specifi c example. A researcher wishes to test the hy- pothesis that inquiry teaching in history classes results in higher-level thinking than does the lecture method. The researcher may elect option 1, in which case she may observe students in the classroom, noting the frequency of oral statements indicative of higher-level thinking. Or, she may examine existing student records that may in- clude test results and/or anecdotal material she considers indicative of higher-level thinking. If she elects option 2, the researcher is likely to administer tests or request stu- dent products (essays, problem sheets) for evidence. She may also decide to interview students using questions designed to reveal their thinking about history (or other topics). Finally, if the researcher chooses option 3, she is likely to interview persons (teachers, other students) or ask them to fi ll out rating scales in which the interview- ees assess each student’s thinking skills based on their prior experience with the student. Examples of each type of method are given below.

1. Researcher instruments • A researcher interested in learning and memory

development counts the number of times it takes different nursery school children to learn to navi- gate their way correctly through a maze located in their school playground. He records his fi nd- ings on a tally sheet .

fra97851_ch07_110-145.indd 113fra97851_ch07_110-145.indd 113 22/12/10 9:16 PM22/12/10 9:16 PM

description of the intended respondents. Be sure to have them evaluate format as well as content.

6. Revise items based on colleague feedback. At this point, try to have about twice as many items as you intend to use in the fi nal form (generally at least 20). Remember that more items generally provide higher reliability.

7. Locate a group of people with experience appropriate to your study. Have them review your items for logical validity. Make any revisions needed, and complete your items. You should have half again as many items as in- tended in the fi nal form.

8. Try out your instrument with a group of respondents who are as similar as possible to your study respondents. Have them complete the instrument, and then discuss it with them, to the extent that this is feasible, given their age, sophistication, and so forth.

9. If feasible, conduct a statistical item analysis with your tryout data (at least 20 respondents are necessary). Such analyses are not diffi cult to carry out, especially if you have a computer. The information provided on each item indicates how effective it is and sometimes even suggests how to improve it. See, for example, K. R. Murphy and C. O. Davidshofer (1991). Psychological testing: Principles and applications. Englewood Cliffs, NJ: Prentice Hall.

10. Select and revise items as necessary until you have the number you want.

RESEARCH TIPS

Some Tips About Developing Your Own Instrument

1. Be sure you are clear about what variables are to be as- sessed. Much time and effort can be wasted by defi ni- tions that are too ambiguous. If more than one variable is involved, be sure that both the meaning and the items for each variable are kept distinct. In general, a particular item or question should be used for only one variable.

2. Review existing instruments that measure similar vari- ables in order to decide upon a format and to obtain ideas on specifi c items.

3. Decide on a format for each variable. Although it is sometimes appropriate to mix multiple-choice, true-false, matching, rating, and open-ended items, doing so com- plicates scoring and is usually undesirable. Remember: Different variables often require different formats.

4. Begin compiling and/or writing items. Be sure that, in your judgment, each is logically valid—that is, that the item is consistent with the defi nition of the variable. Try to ensure that the vocabulary is appropriate for the in- tended respondents.

5. Have colleagues review the items for logical validity. Supply colleagues with a copy of your defi nitions and a

the endeavor. Fortunately, a number of already devel- oped, useful instruments exist, and they can be located quite easily by means of a computer. A comprehensive listing of testing resources can be found by accessing the ERIC database at the following Web site: http://eric .ed.gov (Figure 7.1).

As an example, we typed the phrase (using quotes) “so- cial studies” and the word instruments in the box labeled “Search Terms.” This produced a list of 765 documents. As this was far too many to peruse, we changed the search terms to “social studies” competency-based instruments. This produced a much more manageable list of 5 refer- ences as shown in Figure 7.2. We clicked #1 “ Social Stud- ies. Competency-Based Education Assessment Series” to obtain a description of the instrument (Figure 7.3), as well as a full-text PDF fi le of the document including the in- strument itself and scoring guide.

Almost any topic can be searched in this way to obtain a list of instruments that measure or assess some aspect

of the topic. In general, an electronic fi le containing the instrument can be downloaded immediately or obtained from the ERIC Document Reproduction Service.

ERIC’s test collection currently includes more than 9,000 instruments of various types. Several years ago ERIC underwent considerable changes. The clearing- houses were closed in early 2004, and in September 2004 a new Web site was introduced that provides users with markedly improved search capabilities that utilize more effi cient retrieval methods to access the ERIC data- base (1966–present). In October 2004 free full-text non- journal ERIC resources were introduced, including over 100,000 full-text documents. Users can now refi ne their search results through the use of the ERIC thesaurus and various ERIC identifi ers. For example, search results in ERIC can now be limited to publications that contain instruments like tests and questionnaires (check it out!).

To get information on a preexisting instrument’s use, validity, and reliability, reviews are available through

114

fra97851_ch07_110-145.indd 114fra97851_ch07_110-145.indd 114 22/12/10 9:16 PM22/12/10 9:16 PM

C H A P T E R 7 Instrumentation 115

Here are some useful Internet resources that include test locator services, advice on how to select and evalu- ate a test, published instrument reviews, information on fair testing practices, and connections to several search- able databases that may be useful to researchers looking for specifi c test information.

The ERIC Clearinghouse on Assessment and Evaluation (http://ericae.net): ERIC/AE is an Internet test locator service provided col- laboratively by ERIC, the Educational Testing Service (ETS), the Buros Institute, and Pro-Ed Publishing.

ERIC/ETS Test Collection Test File (http://www .ets.org/testcoll): The ERIC/ETS Test Collection Test File, a joint project of ERIC and the ETS, contains records on over 9,500 tests and research instruments.

The Mental Measurements Yearbooks and Tests in Print, both produced by the Buros Institute at the University of Nebraska. * The yearbooks are published periodi- cally with supplements produced between issues. Each yearbook provides reviews of the standardized tests that have been published since the last issue. The institute’s Tests in Print is a comprehensive bibliography of com- mercial tests. Unfortunately, only the references to the instruments and reviews of them are available online; the actual instruments themselves are available only in print/paper form from the publisher.

With so many instruments now available to the re- search community, we recommend that, except in un- usual cases, researchers devote their energies to adapting (and/or improving) those that now exist rather than try- ing to start from scratch to develop entire new measures.

Figure 7.1 ERIC Database of Tests and Assessments

*So named for Oscar Buros, who started the yearbooks back in 1938.

fra97851_ch07_110-145.indd 115fra97851_ch07_110-145.indd 115 22/12/10 9:16 PM22/12/10 9:16 PM

116 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

The Buros Test Review Locator (http://buros.unl .edu/buros/jsp/search.jsp): Provides evaluations of tests and over 4,000 commercially available instruments.

WRITTEN RESPONSE VERSUS PERFORMANCE

Another way to classify instruments is in terms of whether they require a written or marked response from subjects or a more general evaluation of subjects’ performance. Written-response instruments include objective (e.g., multiple-choice, true-false, matching, or short-answer) tests, short-essay examinations, questionnaires, interview schedules, rating scales, and checklists. Performance in- struments include any device designed to measure either a procedure or a product. Procedures are ways of doing things, such as mixing a chemical solution, diagnosing a problem in an automobile, writing a letter, solving a puz- zle, or setting the margins on a typewriter. Products are the end results of procedures, such as the correct chemical

solution, the correct diagnosis of auto malfunction, or a properly typed letter. Performance instruments are de- signed to see whether and how well procedures can be followed and to assess the quality of products.

Written-response instruments are generally preferred over performance instruments, since the use of the lat- ter is frequently quite time-consuming and often requires equipment or other resources that are not readily available. A large amount of time would be required to have even a fairly small sample of students (imagine 35!) complete the steps involved in a high school science experiment.

Examples of Data-Collection Instruments When it comes to administering the instruments to be used in a study, either the researchers (or their assistants or other informants) must do it themselves, or they must

Figure 7.2 Search Results for Social Studies Competency-Based Instruments

fra97851_ch07_110-145.indd 116fra97851_ch07_110-145.indd 116 22/12/10 9:16 PM22/12/10 9:16 PM

C H A P T E R 7 Instrumentation 117

ask the subjects of the study to provide the information desired. Therefore, we group the instruments in the fol- lowing discussion according to whether they are com- pleted by researchers or by subjects. Examples of these instruments include the following:

Researcher completes: Subject completes:

Rating scales Questionnaires Interview schedules Self-checklists Observation forms Attitude scales Tally sheets Personality (or character)

inventories Flowcharts Performance

checklists Achievement/aptitude

tests Anecdotal records Performance tests Time-and-motion

logs Projective devices Sociometric devices

This distinction, of course, is by no means absolute. Many of the instruments we list might, on a given oc- casion, be completed by either the researcher(s) or sub- jects in a particular study.

RESEARCHER-COMPLETED INSTRUMENTS

Rating Scales. A rating is a measured judgment of some sort. When we rate people, we make a judgment about their behavior or something they have produced. Thus, both behaviors (such as how well a person gives an oral report) and products (such as a written copy of a report) of individuals can be rated.

Notice that the terms observations and ratings are not synonymous. A rating is intended to convey the rat- er’s judgment about an individual’s behavior or product. An observation is intended merely to indicate whether

Figure 7.3 Abstract from the ERIC Database Source: From ERIC (Educator Resources Information Center). Reprinted by permission of the U.S. Department of Education, operated by Computer Sciences

Corporation. www.eric.ed.gov

fra97851_ch07_110-145.indd 117fra97851_ch07_110-145.indd 117 22/12/10 9:16 PM22/12/10 9:16 PM

118 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

a particular behavior is present or absent (see the time- and-motion log in Figure 7.13 on page 125). Some- times, of course, researchers may do both. The activities of a small group engaging in a discussion, for example, can be both observed and rated.

Behavior Rating Scales. Behavior rating scales ap- pear in several forms, but those most commonly used ask the observer to circle or mark a point on a con- tinuum to indicate the rating. The simplest of these to construct is a numerical rating scale, which provides a series of numbers, each representing a particular rating.

Figure 7.4 shows such a scale designed to rate teach- ers. The problem with this rating scale is that different observers are quite likely to have different ideas about the meaning of the terms that the numbers represent ( excellent, average, etc.). In other words, the differ- ent rating points on the scale are not described fully enough. The same individual, therefore, might be rated quite differently by two different observers. One way to address this problem is to give additional meaning to each number by describing it more fully. For example, in Figure 7.4, the rating 5 could be defi ned as “among the top 5 percent of all teachers you have had.” In the absence of such defi nitions, the researcher must either rely on the training of the respondents or treat the rat- ings as subjective opinions.

The graphic rating scale is an attempt to improve on the vagueness of numerical rating scales. It describes each of the characteristics to be rated and places them on a horizontal line on which the observer is to place a check mark. Figure 7.5 presents an example of a graphic rating scale. Here again, this scale would be improved

by adding defi nitions, such as defi ning always as “95 to 100 percent of the time,” and frequently as “70 to 94 percent of the time.”

Product Rating Scales. As we mentioned earlier, researchers may wish to rate products. Examples of products that are frequently rated in education are book reports, maps and charts, diagrams, drawings, notebooks, essays, and creative endeavors of all sorts. Whereas behavior ratings must be done at a particular time (when the researcher can observe the behavior), a big advantage of product ratings is that they can be

Instructions: For each of the behaviors listed below, circle the appropriate number, using the following key: 5 = Excellent, 4 = Above Average, 3 = Average, 2 = Below Average, 1 = Poor.

A. Explains course material clearly

1 2 3 4 5

B. Establishes rapport with students

1 2 3 4 5

C. Asks high-level questions

1 2 3 4 5

D. Varies class activities

1 2 3 4 5

Figure 7.4 Excerpt from a Behavior Rating Scale for Teachers

Instructions: Indicate the quality of the student’s participation in the following class activities by placing an X anywhere along each line.

1. Listens to teacher’s instructions | | | | |

Always Frequently Occasionally Seldom Never

2. Listens to the opinions of other students | | | | |

Always Frequently Occasionally Seldom Never

3. Offers own opinions in class discussions | | | | |

Always Frequently Occasionally Seldom Never

Figure 7.5 Excerpt from a Graphic Rating Scale

fra97851_ch07_110-145.indd 118fra97851_ch07_110-145.indd 118 22/12/10 9:16 PM22/12/10 9:16 PM

C H A P T E R 7 Instrumentation 119

done at any time. * Figure 7.6 presents an example of a scale rating the product “handwriting.” To use this scale, an actual sample of the student’s handwriting is obtained. It is then moved along the scale until the

quality of the handwriting in the sample is most similar to the example shown on the scale. Although more than 50 years old, it remains a classic example of this type of instrument.

Interview Protocol. Interview protocol involves basically the same kind of instrument as a question- naire—a set of questions to be answered by the subjects

Figure 7.6 Example of a Product Rating Scale Source: Handwriting scale used in the California Achieve- ment Tests, From W (1957), CTB/McGraw-Hill, Monterey, CA. Copyright © 1957 by McGraw-Hill.

*Some behavior rating scales are designed to assess behavior over a period of time—for example, how frequently a teacher asks high- level thought questions.

fra97851_ch07_110-145.indd 119fra97851_ch07_110-145.indd 119 22/12/10 9:16 PM22/12/10 9:16 PM

120 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

of the study. There are some important differences in how interviews and questionnaires are administered, however. Interviews are conducted orally, either in per- son or over the phone, and the answers to the questions are recorded by the researcher (or someone he or she has trained). The advantages of this instrument are that the interviewer can clarify any questions that are obscure and also can ask the respondent to expand on answers that are particularly important or revealing. A big disad- vantage, however, is that it takes much longer than the questionnaire to complete. Furthermore, the presence of the researcher may inhibit respondents from saying what they really think. Establishing a comfortable rap- port with participants is an important fi rst step in the interview process.

Figure 7.7 illustrates a structured interview sched- ule. Notice that this interview schedule requires the interviewers to do considerable writing, unless the interview is taped. Some interview schedules phrase questions so that the responses are likely to fall in certain categories. This is sometimes call precoding. Precoding enables the interviewer to check appropriate items rather than transcribe responses, thus preventing the respondent from having to wait while the inter- viewer records a response.

Figure 7.8 shows an example of a semi-structured, qualitative interview protocol used in a dissertation study of African-American community college students who transferred successfully to a four-year institution. Notice how the interviewer uses “probes” throughout

Figure 7.7 Interview Schedule (for Teachers) Designed to Assess the Effects of a Competency- Based Curriculum in Inner-City Schools

1. Would you rate pupil academic learning as excellent, good, fair, or poor? a. If you were here last year, how would you compare pupil academic learning to previous years? b. Please give specific examples. 2. Would you rate pupil attitude toward school generally as excellent, good, fair, or poor? a. If you were here last year, how would you compare pupil attitude toward school generally to previous years? b. Please give specific examples. 3. Would you rate pupil attitude toward learning as excellent, good, fair, or poor? a. If you were here last year, how would you compare attitude toward learning to previous years? b. Please give specific examples. 4. Would you rate pupil attitude toward self as excellent, good, fair, or poor? a. If you were here last year, how would you compare pupil attitude toward self to previous years? b. Please give specific examples. 5. Would you rate pupil attitude toward other students as excellent, good, fair, or poor? a. If you were here last year, how would you compare attitude toward other students to previous years? b. Please give specific examples. 6. Would you rate pupil attitude toward you as excellent, good, fair, or poor? a. If you were here last year, how would you compare pupil attitude toward you to previous years? b. Please give specific examples. 7. Would you rate pupil creativity–self-expression as excellent, good, fair, or poor? a. If you were here last year, how would you compare pupil creativity–self-expression to previous years? b. Please give specific examples.

fra97851_ch07_110-145.indd 120fra97851_ch07_110-145.indd 120 22/12/10 9:16 PM22/12/10 9:16 PM

C H A P T E R 7 Instrumentation 121

the interview protocol to draw out the participant. Probes are improvised questions that depend on the an- swer given by the interviewee.

Observation Forms. Paper-and-pencil observa- tion forms (sometimes called observation schedules ) are fairly easy to construct. A sample of such a form is shown in Figure 7.9. As you can see, the form requires the observer not only to record certain behaviors but also to evaluate some as they occur.

Initially, observation forms should always be used on a trial basis in situations similar to those to be observed in order to work out any bugs or ambiguities. A frequent

weakness in many observation forms is that they ask the observer to record more behaviors than can be done ac- curately (or to watch too many individuals at the same time). As is frequently the case, the simpler the instru- ment, the better.

Tally Sheets. A tally sheet is a device often used by researchers to record the frequency of student behaviors, activities, or remarks. How many high school students follow instructions during fi re drills, for example? How many instances of aggression or helpfulness do elemen- tary students exhibit on the playground? How often do students in Mr. Jordan’s fi fth-period U.S. history class

1. What people or services were the most beneficial to you for transferring to State University (SU) from your Community College (CC)? 2. Why do you think you were successful? (Probe: Tell me how you achieved your goal.) 3. Did anyone give you personal support or strength at your CC that helped you to transfer to SU? (Probe: Can you tell me about the person you think helped you the most to achieve your goal of transferring to SU? What did that person say or do that helped you?) 4. What did you struggle with as you prepared to transfer? What were some of the major obstacles? (Probe: Thinking back, can you recall a really tough experience or day you had when you thought you weren’t going to succeed? Tell me about it.) 5. If you could talk to an administrator or faculty member at the CC you attended, what would you say to them about your transfer process, and how could it be improved to help other African-American students like you?

Figure 7.8 Semi- structured Interview Protocol

Figure 7.9 Sample Observation Form

Directions:

1. Place a check mark each time the teacher: a. asks individual students a question b. asks questions to the class as a whole c. disciplines students d. asks for quiet e. asks students if they have any questions f. sends students to the chalkboard

2. Place a check mark each time the teacher asks a question that requires: a. memory or recall of information b. comparison c. an inference d. a generalization e. specific application

Frequency 6 2 1 3 1 2

Frequency

5 2 3 0 1

� �

�

�

�

�

�

� �

� �

�

� �

� �

�

�

�

�

� � �

� �

�

fra97851_ch07_110-145.indd 121fra97851_ch07_110-145.indd 121 22/12/10 9:16 PM22/12/10 9:16 PM

122 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

ask questions? How often do they ask inferential ques- tions? Tally sheets can help researchers record answers to these kinds of questions effi ciently.

A tally sheet is simply a listing of various categories of activities or behaviors on a piece of paper. Every time a subject is observed engaging in one of these activities or behaviors, the researcher places a tally in the appro- priate category. The kinds of statements that students make in class, for example, often indicate the degree to which they understand various concepts and ideas. The possible category systems that might be devised are probably endless, but Figure 7.10 presents one example.

Flowcharts. A particular type of tally sheet is the participation fl owchart. Flowcharts are particularly helpful in analyzing class discussions. Both the number and direction of student remarks can be charted to gain some idea of the quantity and focus of students’ verbal participation in class.

One of the easiest ways to do this is to prepare a seat- ing chart on which a box is drawn for each student in

the class being observed. A tally is then placed in the box of a particular student each time he or she makes a verbal comment. To indicate the direction of individual student comments, arrows can be drawn from the box of a student making a comment to the box of the student to whom the comment is directed. Figure 7.11 illustrates what such a fl owchart might look like. This chart sug- gests that Robert, Felix, and Mercedes dominated the discussion, with contributions from Al, Gail, Jack, and Sam. Joe and Nancy said nothing. Note that a subse- quent discussion, or a different topic, however, might reveal a quite different pattern.

Performance Checklists. One of the most frequently used of all measuring instruments is the checklist. A performance checklist consists of a list of behaviors that make up a certain type of performance (using a microscope, typing a letter, solving a math- ematics problem, and so on). It is used to determine whether an individual behaves in a certain (usually desired) way when asked to complete a particular task.

Type of Remark

1. Asks question calling for factual information Related to lesson Not related to lesson

2. Asks question calling for clarification Related to lesson Not related to lesson

3. Asks question calling for explanation Related to lesson Not related to lesson

4. Asks question calling for speculation Related to lesson Not related to lesson

5. Asks question of another student Related to lesson Not related to lesson

6. Gives own opinion on issue Related to lesson Not related to lesson

7. Responds to another student Related to lesson Not related to lesson

8. Summarizes remarks of another student Related to lesson Not related to lesson

9. Does not respond when addressed by teacher Related to lesson Not related to lesson

10. Does not respond when addressed by another student Related to lesson Not related to lesson

Figure 7.10 Discussion-Analysis Tally Sheet

fra97851_ch07_110-145.indd 122fra97851_ch07_110-145.indd 122 22/12/10 9:16 PM22/12/10 9:16 PM

C H A P T E R 7 Instrumentation 123

Jack Al Mercedes

Gail Joe Nancy

Robert Felix Sam

Figure 7.11 Participation Flowchart Source: Adapted from Enoch I. Sawin (1969). Evaluation and the work of the teacher. Belmont, CA: Wadsworth, p. 179. Reprinted by permission of Sage Publishers, Inc.

If a particular behavior is present when an individual is observed, the researcher places a check mark opposite it on the list.

Figure 7.12 presents part of a performance checklist developed more than 70 years ago to assess a student’s skill in using a microscope. Note that the items on this checklist (as any well-constructed checklist should) ask the observer to indicate only if the desired behaviors take place. No subjective judgments are called for on the part of the observer as to how well the individual performs. Items that call for such judgments are best left to rating scales.

Anecdotal Records. Another way of recording the behavior of individuals is the anecdotal record. It is just what its name implies—a record of observed behaviors written down in the form of anecdotes. There is no set format; rather, observers are free to record any behavior they think is important and need not focus on the same behavior for all subjects. To produce the most useful records, however, observers should try to be as specifi c and as factual as possible and to avoid evaluative, interpretive, or overly gen- eralized remarks. The American Council on Educa- tion describes four types of anecdotes, stating that the

fi rst three are to be avoided. Only the fourth type is desired.

1. Anecdotes that evaluate or judge the behavior of the child as good or bad, desirable or undesirable, ac- ceptable or unacceptable .  .  .  evaluative statements (to be avoided).

2. Anecdotes that account for or explain the child’s behavior, usually on the basis of a single fact or thesis . . .  interpretive statements (to be avoided).

3. Anecdotes that describe certain behavior in general terms, as happening frequently, or as characterizing the child . . .  generalized statements (to be avoided).

4. Anecdotes that tell exactly what the child did or said, that describe concretely the situation in which the action or comment occurred, and that tell clearly what other persons also did or said  .  .  .  specifi c or concrete descriptive statements (the type desired). 1

Here are examples of each of the four types.

Evaluative: Julius talked loud and much during po- etry; wanted to do and say just what he wanted and didn’t consider the right working out of things. Had to ask him to sit by me. Showed a bad attitude about it.

fra97851_ch07_110-145.indd 123fra97851_ch07_110-145.indd 123 22/12/10 9:16 PM22/12/10 9:16 PM

124 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

Interpretive: For the last week Sammy has been a perfect wiggle-tail. He is growing so fast he can- not be settled.  .  .  . Of course the inward change that is taking place causes the restlessness.

Generalized: Sammy is awfully restless these days. He is whispering most of the time he is not kept busy. In the circle, during various discussions, even though he is interested, his arms are moving or he is punching the one sitting next to him. He smiles when I speak to him.

Specifi c (the type desired): The weather was so bit- terly cold that we did not go on the playground today. The children played games in the room during the regular recess period. Andrew and

Larry chose sides for a game which is known as stealing the bacon. I was talking to a group of children in the front of the room while the choos- ing was in process and in a moment I heard a loud altercation. Larry said that all the children wanted to be on Andrew’s side rather than on his. Andrew remarked, “I can’t help it if they all want to be on my side.” 2

Time-and-Motion Logs. There are occasions when researchers want to make a very detailed observa- tion of an individual or a group. This is often the case, for example, when trying to identify the reasons under- lying a particular problem or diffi culty that an individual

1. Takes slide ______ 2. Wipes slide with lens paper ______ 3. Wipes slide with cloth ______ 4. Wipes slide with finger ______ 5. Moves bottle of culture along the

table ______ 6. Places drop or two of culture on slide ______ 7. Adds more culture ______ 8. Adds few drops of water ______ 9. Hunts for cover glasses ______

10. Wipes cover glass with lens paper ______ 11. Wipes cover glass with cloth ______ 12. Wipes cover with finger ______ 13. Adjusts cover with finger ______ 14. Wipes off surplus fluid ______ 15. Places slide on stage ______ 16. Looks through eyepiece with

right eye ______ 17. Looks through eyepiece with left eye ______ 18. Turns to objective of lowest power ______ 19. Turns to low-power objective ______ 20. Turns to high-power objective ______ 21. Holds one eye closed ______ 22. Looks for light ______ 23. Adjusts concave mirror ______ 24. Adjusts plane mirror ______ 25. Adjusts diaphragm ______ 26. Does not touch diaphragm ______ 27. With eye at eyepiece, turns down

coarse adjustment screw ______ 28. Breaks cover glass ______ 29. Breaks slide ______ 30. With eye away from eyepiece, turns

down coarse adjustment screw ______

31. Turns up coarse adjustment screw a great distance ______

32. With eye at eyepiece, turns down fine adjustment screw a great distance ______

33. With eye away from eyepiece, turns down fine adjustment screw a great distance ______

34. Turns up fine adjustment screw a great distance ______

35. Turns fine adjustment screw a few turns ______

36. Removes slide from stage ______ 37. Wipes objective with lens paper ______ 38. Wipes objective with cloth ______ 39. Wipes objective with finger ______ 40. Wipes eyepiece with lens paper ______ 41. Wipes eyepiece with cloth ______ 42. Wipes eyepiece with finger ______ 43. Makes another mount ______ 44. Takes another microscope ______ 45. Finds object ______ 46. Pauses for an interval ______ 47. Asks, “What do you want me to do?” ______ 48. Asks whether to use high power ______ 49. Says, “I’m satisfied.” ______ 50. Says that the mount is all right for

his or her eye ______ 51. Says, “I cannot do it.” ______ 52. Told to start a new mount ______ 53. Directed to find object under low

power ______ 54. Directed to find object under high

power ______

Figure 7.12 Performance Checklist Noting Student Actions Source: Educational Research Bulletin (1922–61) by R. W. Tyler. Copyright 1930 by Ohio State University, College of Education. Reproduced with permission of Ohio State University, College of Education in the format Textbook via Copyright Clearance Center.

fra97851_ch07_110-145.indd 124fra97851_ch07_110-145.indd 124 22/12/10 9:16 PM22/12/10 9:16 PM

C H A P T E R 7 Instrumentation 125

or class is having (working very slowly, failing to com- plete assigned tasks, inattentiveness, and so on).

A time-and-motion study is the observation and de- tailed recording over a given period of time of the ac- tivities of one or more individuals (for example, during a 15-minute laboratory demonstration). Observers try to record everything an individual does as objectively as possible and at brief, regular intervals (such as every 3 minutes, with a 1-minute break interspersed between intervals).

The late Hilda Taba, a pioneer in educational evalu- ation, once cited an example of a fourth-grade teacher who believed that her class’s considerable slowness was due to the fact that they were extremely meticulous in their work. To check this out, she decided to conduct a detailed time-and-motion study of one typical student.

The results of her study indicated that this student, rather than being overly meticulous, was actually un- able to focus his attention on a particular task for any concerted period of time. Figure 7.13 illustrates what she observed.

SUBJECT-COMPLETED INSTRUMENTS

Questionnaires. The interview schedule shown in Figure 7.7 on page 120 could be used as a questionnaire. In a questionnaire, the subjects respond to the questions by writing or, more commonly, by marking an answer sheet. Advantages of questionnaires are that they can be mailed or given to large numbers of people at the same time. The disadvantages are that unclear or seem- ingly ambiguous questions cannot be clarifi ed, and the

Figure 7.13 Time-and-Motion Log Source: Hilda Taba (1957). Problem identifi cation. In ASCD 1957 Yearbook: Research for Curriculum Improvement, pp. 60–61. © 1957 ASCD. Used with permission. The Association for Supervision and Curriculum Development is a worldwide community of educators advocating sound policies and sharing best practices to achieve the success of each learner. To learn more, visit ASCD at www.ascd.org.

Time Activity

11:32 Stacked paper Picked up pencil Wrote name Moved paper closer Continued with reading Rubbed nose Looked at Art’s paper Started to work . . .

11:45 Worked and watched Made funny faces Giggled. Looked at Lorrie and smiled Borrowed Art’s paper Erased Stacked paper Read Slid paper around Worked briefly Picked up paper and read Thumb in mouth, watched Miss D

11:47 Worked and watched Made funny face Giggled. Looked and smiled at Lorrie Paper up—read Picked eye Studied bulletin board Paper down—read again Fidgeted with paper Played with pencil and fingers Watched me

Time Activity

Watched L. Laughed at her Erased Hand up Laughed. Watched D. Got help

11:50 Looked at Lorrie Tapped fingers on desk Wrote Slid down in desk Hand to head, listened to D. helping

Lorrie Blew breath out hard Fidgeted with paper Looked at other group Held chin Watched Charles Read, hands holding head Erased Watched other group, chin on hand Made faces—yawned—fidgeted Held head Read, pointing to words Wrote Put head on arm on desk Held chin Read Rubbed eye

11:55 Wrote

fra97851_ch07_110-145.indd 125fra97851_ch07_110-145.indd 125 22/12/10 9:16 PM22/12/10 9:16 PM

126 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

respondent has no chance to expand on or react verbally to a question of particular interest or importance.

Selection items on questionnaires include multiple- choice, true-false, matching, or interpretive-exercise questions. Supply items include short-answer or essay questions. We’ll give some examples of each of these types of items when we discuss achievement tests later in the chapter.

Self-Checklists. A self-checklist is a list of several characteristics or activities presented to the subjects of a study. The individuals are asked to study the list and then to place a mark opposite the characteristics they possess or the activities in which they have engaged for a particular length of time. Self-checklists are often used when researchers want students to diagnose or to appraise their own performance. One example of a self-checklist for use with elementary school students is shown in Figure 7.14.

Attitude Scales. The basic assumption that under- lies all attitude scales is that it is possible to discover attitudes by asking individuals to respond to a series of statements of preference. Thus, if individuals agree with the statement, “A course in philosophy should be required of all candidates for a teaching credential,” researchers infer that these students have a positive at- titude toward such a course (assuming students under- stand the meaning of the statement and are sincere in their responses). An attitude scale, therefore, consists of a set of statements to which an individual responds. The pattern of responses is then viewed as evidence of one or more underlying attitudes.

Attitude scales are often similar to rating scales in form, with words and numbers placed on a continuum. Subjects circle the word or number that best represents how they feel about the topics included in the questions or statements in the scale. A commonly used attitude scale in educational research is the Likert scale , named

Figure 7.14 Example of a Self-Checklist

Date _______________________ Name _________________________________________________________________________________________________________________________________

Instructions: Place a check (✓) in the space provided for those days, during the past week, when you have participated in the activity listed. Circle the activity if you feel you need to participate in it more frequently in the weeks to come.

Mon Tues Wed Thurs Fri

1. I participated in class discussions. ✓ ✓ ✓

2. I did not interrupt others while they were speaking. ✓ ✓ ✓ ✓ ✓

3. I encouraged others to offer their opinions. ✓ ✓

4. I listened to what others had to say. ✓ ✓ ✓ ✓

5. I helped others when asked. ✓

6. I asked questions when I was unclear about what had been said. ✓ ✓

7. I looked up words in the dictionary that I did not know how to spell. ✓

8. I considered the suggestions of others. ✓ ✓ ✓

9. I tried to be helpful in my remarks. ✓ ✓ ✓

10. I praised others when I thought they did a good job. ✓

fra97851_ch07_110-145.indd 126fra97851_ch07_110-145.indd 126 22/12/10 9:16 PM22/12/10 9:16 PM

C H A P T E R 7 Instrumentation 127

after the man who designed it. 3 Figure 7.15 presents a few examples from a Likert scale. On some items, a 5 (strongly agree) will indicate a positive attitude and be scored 5. On other items, a 1 (strongly disagree) will indicate a positive attitude and be scored 5 (thus, the ends of the scale are reversed when scoring), as shown in item 2 in Figure 7.15.

A unique sort of attitude scale that is especially use- ful for classroom research is the semantic differential . 4 It allows a researcher to measure a subject’s attitude to- ward a particular concept. Subjects are presented with a continuum of several pairs of adjectives ( good-bad, cold-hot, priceless-worthless, and so on) and asked to place a check mark between each pair to indicate their attitudes. Figure 7.16 presents an example.

A scale that has particular value for determin- ing the attitudes of young children uses simply drawn faces. When the subjects of an attitude study are pri- mary school children or younger, they can be asked to place an X under a face, such as the ones shown in Fig- ure 7.17, to indicate how they feel about a topic.

The subject of attitude scales is discussed rather exten- sively in the literature on evaluation and test development, and students interested in a more extended treatment should consult a standard textbook on these subjects. 5

Personality (or Character) Inventories. Personality inventories are designed to measure certain traits of individuals or to assess their feelings about

themselves. Examples of such inventories include the Minnesota Multiphasic Personality Inventory, the IPAT Anxiety Scale, the Piers-Harris Children’s Self-Concept Scale (How I Feel About Myself), and the Kuder Prefer- ence Record. Figure 7.18 lists some typical items from this type of test. The specifi c items, of course, refl ect the variable(s) the inventory addresses.

Achievement Tests. Achievement , or ability, tests measure an individual’s knowledge or skill in a given area or subject. They are mostly used in schools to measure learning or the effectiveness of instruction. The California Achievement Test, for example, mea- sures achievement in reading, language, and arithmetic. The Stanford Achievement Test measures a variety of areas, such as language usage, word meaning, spell- ing, arithmetic computation, social studies, and science. Other commonly used achievement tests include the Comprehensive Tests of Basic Skills, the Iowa Tests of Basic Skills, the Metropolitan Achievement Test, and the Sequential Tests of Educational Progress (STEP). In research that involves comparing instructional methods, achievement is frequently the dependent variable.

Achievement tests can be classifi ed in several ways. General achievement tests are usually batteries of tests (such as the STEP tests) that measure such things as vocabulary, reading ability, language usage, math, and social studies. One of the most common general achieve- ment tests is the Graduate Record Examination, which

Instructions: Circle the choice after each statement that indi- cates your opinion.

1. All professors of education should be required to spend at least six months teaching at the elementary or secondary level every five years.

Strongly Strongly agree Agree Undecided Disagree disagree (5) (4) (3) (2) (1)

2. Teachers’ unions should be abolished.

Strongly Strongly agree Agree Undecided Disagree disagree (1) (2) (3) (4) (5)

3. All school administrators should be required by law to teach at least one class in a public school classroom every year.

Strongly Strongly agree Agree Undecided Disagree disagree (5) (4) (3) (2) (1)

Figure 7.15 Examples of Items from a Likert Scale Measuring Attitude Toward Teacher Empowerment

fra97851_ch07_110-145.indd 127fra97851_ch07_110-145.indd 127 22/12/10 9:16 PM22/12/10 9:16 PM

128 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

students must pass before they can be admitted to most graduate programs. Specifi c achievement tests, on the other hand, are tests that measure an individual’s ability in a specifi c subject, such as English, world history, or biology. Figure 7.19 illustrates the kinds of items found on an achievement test.

Aptitude Tests. Another well-known type of abil- ity test is the so-called general aptitude , or intelligence, test , which assesses intellectual abilities that are not, in most cases, specifi cally taught in school. Some mea- sure of general ability is frequently used as either an

independent or a dependent variable in research. In at- tempting to assess the effects of different instructional programs, for example, it is often necessary (and very important) to control this variable so that groups ex- posed to the different programs are not markedly differ- ent in general ability.

Aptitude tests are intended to measure an individ- ual’s potential to achieve; in actuality, they measure present skills or abilities. They differ from achieve- ment tests in their purpose and often in content, usually including a wider variety of skills or knowledge. The same test may be either an aptitude or an achievement

Figure 7.16 Example of the Semantic Differential Instructions: Listed below are several pairs of adjectives. Place a

check mark (�) on the line between each pair to indicate how you feel. Example: Hockey:

exciting :_____:_____:_____:_____:_____:_____:_____:_____: dull

If you feel that hockey is very exciting, you would place a check mark in the first space next to the word exciting. If you feel that hockey is very dull, you would place a check mark in the space nearest the word dull. If you are sort of undecided, you would place a check mark in the middle space between the two words. Now rate each of the activities that follow [only one is listed]:

Working with other students in small groups

friendly :_____:_____:_____:_____:_____:_____:_____:_____: unfriendly

happy :_____:_____:_____:_____:_____:_____:_____:_____: sad

easy :_____:_____:_____:_____:_____:_____:_____:_____: hard

fun :_____:_____:_____:_____:_____:_____:_____:_____: work

hot :_____:_____:_____:_____:_____:_____:_____:_____: cold

good :_____:_____:_____:_____:_____:_____:_____:_____: bad

laugh :_____:_____:_____:_____:_____:_____:_____:_____: cry

beautiful :_____:_____:_____:_____:_____:_____:_____:_____: ugly

How do you feel about arithmetic?

Figure 7.17 Pictorial Attitude Scale for Use with Young Children

fra97851_ch07_110-145.indd 128fra97851_ch07_110-145.indd 128 22/12/10 9:16 PM22/12/10 9:16 PM

C H A P T E R 7 Instrumentation 129

test, depending on the purpose for which it is used. A mathematics achievement test, for example, may also measure aptitude for additional mathematics. Although such tests are used primarily by counselors to help indi- viduals identify areas in which they may have potential, they also can be used in research. In this regard, they are particularly useful for purposes of control. For ex- ample, to measure the effectiveness of an instructional program designed to increase problem-solving ability

in mathematics, a researcher might decide to use an aptitude test to adjust for initial differences in ability. Figure 7.20 presents an example of one kind of item found on an aptitude test.

Aptitude tests may be administered to individuals or groups. Each method has both advantages and disad- vantages. The big advantage of group tests is that they are more convenient to administer and hence save con- siderable time. One disadvantage is that they require a

Instructions: Check the option that most correctly describes you.

Quite Almost SELF-ESTEEM often Sometimes never________ _________ ________

1. Do you think your friends are smarter than you? _____ _____ _____ 2. Do you feel good about your appearance? _____ _____ _____ 3. Do you avoid meeting new people? _____ _____ _____

Quite Almost STRESS often Sometimes never________ _________ ________

1. Do you have trouble sleeping? _______ _______ _______ 2. Do you feel on top of things? _______ _______ _______ 3. Do you feel you have too much to do? _______ _______ _______

Figure 7.18 Sample Items from a Personality Inventory

Figure 7.19 Sample Items from an Achievement Test

Instructions: Use the map to answer questions 1 and 2. Circle the correct answer.

1. Which number shows an area declining in population?

A 1 B 3 C 5 D 7

2. Which number shows an area that once belonged to Mexico?

A 3 B 5 C 6 D None of the above

1 25

7

6

4 3

fra97851_ch07_110-145.indd 129fra97851_ch07_110-145.indd 129 22/12/10 9:16 PM22/12/10 9:16 PM

130 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

great deal of reading, and students who are low in read- ing ability are thus at a disadvantage. Furthermore, it is diffi cult for those taking the test to have test instructions clarifi ed or to have any interaction with the examiner (which sometimes can raise scores). Lastly, the range of possible tasks on which the student can be examined is much less with a group-administered test than with an individually administered test.

The California Test of Mental Maturity (CTMM) and the Otis-Lennon are examples of group tests. The best- known of the individual aptitude tests is the Stanford- Binet Intelligence Scale, although the Wechsler scales are used more widely. Whereas the Stanford-Binet gives only one IQ score, the Wechsler scales also yield a number of subscores. The two Wechsler scales are the Wechsler Intelligence Scale for Children (WISC-III) for ages 5 to 15 and the Wechsler Adult Intelligence Scale (WAIS-III) for older adolescents and adults.

Many intelligence tests provide reliable and valid ev- idence when used with certain kinds of individuals and for certain purposes (for example, predicting the college grades of middle-class Caucasians). On the other hand, they have increasingly come under attack when used with other persons or for other purposes (such as identi- fying members of certain minority groups to be placed in special classes). Furthermore, there is increasing recognition that most intelligence tests fail to measure many important abilities, including the ability to iden- tify or conceptualize unusual sorts of relationships. As a result, the researcher must be especially careful in eval- uating any such test before using it and must determine whether it is appropriate for the purpose of the study. (We discuss some ways to do this when we consider validity in Chapter 8.) Figure 7.21 presents examples of the kinds of items on an intelligence test.

Performance Tests. As we have mentioned, a per- formance test measures an individual’s performance on

a particular task. An example would be a typing test, in which individual scores are determined by how accu- rately and how rapidly people type.

As Sawin has suggested, it is not always easy to determine whether a particular instrument should be called a performance test, a performance checklist, or a performance rating scale. 6 A performance test is the most objective of the three. When a considerable amount of judgment is required to determine if the vari- ous aspects of a performance were done correctly, the device is likely to be classifi ed as either a checklist or rating scale. Figure 7.22 illustrates a performance test developed more than 60 years ago to measure sewing ability. In this test, the individual is requested to sew on the line in part A of the test, and between the lines on part B of the test. 7

Projective Devices. A projective device is any sort of instrument with a vague stimulus that allows in- dividuals to project their interests, preferences, anxiet- ies, prejudices, needs, and so, on through their responses

Figure 7.20 Sample Item from an Aptitude Test

Look at the foldout on the left. Which object on the right can be made from it?

A B C D E

1. How are frog and toy alike and how are they different?

2. Here is a sequence of pictures.

A B C D E

Which of the following would come next in the sequence?

F G H

Figure 7.21 Sample Items from an Intelligence Test

fra97851_ch07_110-145.indd 130fra97851_ch07_110-145.indd 130 22/12/10 9:16 PM22/12/10 9:16 PM

C H A P T E R 7 Instrumentation 131

to it. This kind of device has no “right” answers (or any clear-cut answers of any sort), and its format allows an individual to express something of his or her own per- sonality. There is room for a wide variety of responses.

Perhaps the best-known example of a projective de- vice is the Rorschach Ink Blot Test, in which individu- als are presented with a series of ambiguously shaped ink blots and asked to describe what the blots look like. Another well-known projective test is the Thematic Ap- perception Test (TAT), in which pictures of events are presented and individuals are asked to make up a story about each picture. One application of the projective approach to a classroom setting is the Picture Situation Inventory, one of the few examples especially adapted to classroom situations. This instrument consists of a series of cartoonlike drawings, each portraying a class- room situation in which a child is saying something. Students taking the test are to enter the response of the teacher, thereby presumably indicating something of their own tendencies in the situation. Two of the pictures in this test are reproduced in Figure 7.23.

Sociometric Devices. Sociometric devices ask individuals to rate their peers in some way. Two ex- amples include the sociogram and the “group play.” A sociogram is a visual representation, usually by means of arrows, of the choices people make about other indi- viduals with whom they interact. It is frequently used to assess the climate and structure of interpersonal re- lationships within a classroom, but it is by no means

limited to such an environment. Each student is usually represented by a circle (if female) or a triangle (if male), and arrows are then drawn to indicate different student choices with regard to a particular question. Students may be asked, for example, to list three students whom they consider leaders of the class; admire the most; fi nd especially helpful; would like to have for a friend; would like to have as a partner in a research project; and so forth. The responses students give are then used to construct the sociogram. Figure 7.24 illustrates a sociogram.

Another version of a sociometric device is the group play. Students are asked to cast different members of their group in various roles in a play to illustrate their inter- personal relationships. The roles are listed on a piece of paper, and then the members of the group are asked to write in the name of the student they think each role best describes. Almost any type of role can be suggested. The casting choices that individuals make often shed consider- able light on how some individuals are viewed by others. Figure 7.25 presents an example of this device.

Item Formats. Although the types of items or ques- tions used in different instruments can take many forms, each item can be classifi ed as either a selection item or a supply item. A selection item presents a set of possible responses from which respondents are to select the most appropriate answer. A supply item, on the other hand, asks respondents to formulate and then supply their own answers. Here are some examples of each type.

Directions: Sew on the line in part A and between the lines in part B.

NAME A

NAME B

Figure 7.22 Example from the Blum Sewing Machine Test Source: M. L. Blum. Selection of sewing machine operators. Journal of Applied Psychology, 27 (1): 36. Copyright 1943 by the American Psychological Association. Reproduced with permission.

fra97851_ch07_110-145.indd 131fra97851_ch07_110-145.indd 131 22/12/10 9:16 PM22/12/10 9:16 PM

132 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

Selection Items. True-false items: True-false items present either a true or a false statement, and the respon- dent has to mark either true (T) or false (F). Frequently used variations of the words true and false are yes-no or right-wrong, which often are more useful when attempt- ing to question or interview young children. Here is an example of a true-false item.

T F I get very nervous whenever I have to speak in public.

Multiple-choice items: Multiple-choice items con- sist of two parts: the stem, which contains the question, and several (usually four) possible choices. Here is an example:

Which of the following expresses your opinion on abortion? a. It is immoral and should be prohibited. b. It should be discouraged but permitted under un-

usual circumstances. c. It should be available under a wide range of

conditions. d. It is entirely a matter of individual choice.

Matching items: Matching items are variations of the multiple-choice format. They consist of two groups listed in columns—the left-hand column contain- ing the questions or items to be thought about and the

right-hand column containing the possible responses to the questions. The respondent pairs the choice from the right-hand column with the corresponding question or item in the left-hand column. Here is an example:

Instructions: For each item in the left-hand column, select the item in the right-hand column that represents your fi rst reaction. Place the appropriate letter in the blank. Each lettered item may be used more than once or not at all.

Column A Column B Special classes for the:

___ 1. severely retarded ___ 2. mildly retarded ___ 3. hard of hearing ___ 4. visually impaired ___ 5. learning disabled ___ 6. emotionally disturbed

a. should be increased b. should be maintained c. should be decreased d. should be eliminated

Interpretive exercises: One diffi culty with using true- false, multiple-choice, and matching items to measure achievement is that these items often do not measure complex learning outcomes. One way to get at more

Picture 1 Picture 2

Do I have to do this? You didn’t make Tom.

My mother wants me to read in a harder book.

Figure 7.23 Sample Items from the Picture Situation Inventory Source: N. T. Rowan (1967). The relationship of teacher interaction in classroom situations to teacher personality variables. Unpublished doctoral dissertation. Salt Lake City: University of Utah, p. 68.

fra97851_ch07_110-145.indd 132fra97851_ch07_110-145.indd 132 22/12/10 9:16 PM22/12/10 9:16 PM

C H A P T E R 7 Instrumentation 133

complex learning outcomes is to use what is called an interpretive exercise. An interpretive exercise consists of a selection of introductory material (this may be a paragraph, map, diagram, picture, chart) followed by one or more selection items that ask a respondent to in- terpret this material. Two examples of interpretive exer- cises follow.

Example 1.

Directions: Read the following comments a teacher made about testing. Then answer the question that follows the comments by circling the letter of the best answer.

“Students go to school to learn, not to take tests. In addi- tion, tests cannot be used to indicate a student’s absolute

KEY

Black lines = Choices

= Mutual choice

1, 2, 3, outside circle = order of choice

Jan

4-3

2

1

3

3 1

2

3

2

3

Gloria

2-1

Mary

4-2

Connie

4-2

Pat

0-0

2

3

Irene

4-3

Rose

3-3

1 2

1 1

2

3 3

2

Eve

3-3 Al

2-1

3

1 1 1 Joe

3-2

Doris

2-2

Felix

2-2

June

2-2

3 2

2

2

13 2

3 1

21

3

1

Tom

3-1

Mary Ann

6-3

2

2

3

1

Jean

2-2

Name

1-1

No. of choices received

No. of mutual choices

Absent

Girls Boys

12 4

Date given Grade/class Choice question: seating School Present

Figure 7.24 Example of a Sociogram

fra97851_ch07_110-145.indd 133fra97851_ch07_110-145.indd 133 22/12/10 9:16 PM22/12/10 9:16 PM

134 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

level of learning. All tests can do is rank students in order of achievement, and this relative ranking is infl uenced by guessing, bluffi ng, and the subjective opinions of the teacher doing the scoring. The teaching-learning process would benefi t if we did away with tests and depended on student self-evaluation.”

1. Which one of the following unstated assumptions is this teacher making? a. Students go to school to learn. b. Teachers use essay tests primarily. c. Tests make no contribution to learning. d. Tests do not indicate a student’s absolute level of

learning.

Example 2.

Directions: Paragraph A contains a description of the test- ing practices of Mr. Smith, a high school teacher. Read the description and each of the statements that follow it. Mark each statement to indicate the type of inference that can be drawn about it from the material in the paragraph. Place the appropriate letter in front of each statement using the following key:

T—if the statement may be inferred as true. F—if the statement may be inferred as untrue. N — if no inference may be drawn about it from the

paragraph.

Paragraph A

Approximately one week before a test is to be given, Mr. Smith carefully goes through the textbook and constructs multiple-choice items based on the material in the book. He always uses the exact wording of the textbook for the correct answer so that there will be no question concerning its correctness. He is careful to include some test items from each chapter. After the test is given, he lists the scores from high to low on the blackboard and tells each student his or her score. He does not return the test papers to the students, but he offers to answer any questions they might have about the test. He puts the items from each test into a test fi le, which he is building for future use.

____ 1. Mr. Smith’s tests measure a limited range of learn- ing outcomes.

____ 2. Some of Mr. Smith’s test items measure at the un- derstanding level.

____ 3. Mr. Smith’s tests measure a balanced sample of sub- ject matter.

____ 4. Mr. Smith uses the type of test item that is best for his purpose.

____ 5. Students can determine where they rank in the distri- bution of scores on Mr. Smith’s tests.

____ 6. Mr. Smith’s testing practices are likely to motivate students to overcome their weaknesses. 8

Figure 7.25 Example of a Group Play

Directions: Imagine you are the casting director for a large play. Your job is to choose the individuals who will take the various parts (listed below) in the play. Since some of the parts are rather small, you may select the same individual to play more than one part. Choose individuals you think would be the most natural for the part, that is, those who are most like the role in real life.

1. The parts

Part 1—someone who is well liked by all members of the group ____________________________________

Part 2—someone who is disliked by many people __________________________________________________

Part 3—someone who always gets angry about things of little consequence __________________________

Part 4—someone who has wit and a good sense of humor ___________________________________________

Part 5—someone who is very quiet and rarely says anything ________________________________________

Part 6—someone who does not contribute much to the group ______________________________________

Part 7—someone who is angry a lot of the time _____________________________________________________

2. Your role

Which part do you think you could play best? ________________________________________________________

Which part would other members of the group ask you to play? _____________________________________

(T)

(F)

(N)

(N)

(T)

(F)

fra97851_ch07_110-145.indd 134fra97851_ch07_110-145.indd 134 22/12/10 9:16 PM22/12/10 9:16 PM

C H A P T E R 7 Instrumentation 135

observed, or interviewed—often will, to some degree, affect the nature of the information researchers obtain.

To eliminate this reactive effect, researchers at times attempt to use what are called unobtrusive measures, 10 which are data-collection procedures that involve no intrusion into the naturally occurring course of events. In most instances, no instrument is required, only some form of record keeping. Here are some examples of such procedures:

• The degree of fear induced by a ghost-story-telling session can be measured by noting the shrinking di- ameter of a circle of seated children.

• Library withdrawals could be used to demonstrate the effect of the introduction of a new unit on Chi- nese history in a social studies curriculum.

• The interest of children in Christmas or other holi- days might be demonstrated by the amount of distor- tion in the size of their drawings of Santa Claus or other holiday fi gures.

• Racial attitudes in two elementary schools might be compared by noting the degree of clustering of mem- bers of different ethnic groups in the lunchroom and on the playground.

• The values held by people of different countries might be compared through analyzing different types of published materials, such as textbooks, plays, handbooks for youth organizations, magazine adver- tisements, and newspaper headlines.

• Some idea of the attention paid to patients in a hospi- tal might be determined by observing the frequency of notes, both informal and required, made by attend- ing nurses to patients’ bedside records.

• The degree of stress felt by college students might be assessed by noting the nature and frequency of sick- call visits to the college health center.

• Student attitudes toward, and interest in, various top- ics can be noted by observing the amount of graffi ti about those topics written on school walls.

Many variables of interest can be assessed, at least to some degree, through the use of unobtrusive mea- sures. The reliability and validity of inferences based on such measures will vary depending on the proce- dure used. Nevertheless, unobtrusive measures add an important and useful dimension to the array of pos- sible data sources available to researchers. They are particularly valuable as supplements to interviews and questionnaires, often providing a useful way to cor- roborate (or contradict) what these more traditional data sources reveal. 11

Supply Items. Short-answer items: A short-answer item requires the respondent to supply a word, phrase, number, or symbol that is necessary to complete a state- ment or answer a question. Here is an example:

Directions: In the space provided, write the word that best completes the sentence.

When the number of items in a test is increased, the ____ of the scores on the test is likely to increase. (Answer: reliability.)

Short-answer items have one major disadvantage: It is usually diffi cult to write a short-answer item so only one word completes it correctly. In the question above, for example, many students might argue that the word range would also be correct.

Essay questions: An essay question is one that re- spondents are asked to write about at length. As with short-answer questions, subjects must produce their own answers. Generally, however, they are free to determine how to answer the question, what facts to present, which to emphasize, what interpretations to make, and the like. For these reasons, the essay question is a particularly useful device for assessing an individual’s ability to or- ganize, integrate, analyze, and synthesize information. It is especially useful in measuring the so-called higher- level learning outcomes, such as analysis, synthesis, and evaluation. Here are two examples of essay questions:

Example 1

Mr. Rogers, a ninth-grade science teacher, wants to mea- sure his students’ “ability to interpret scientifi c data” with a paper-and-pencil test. 1. Describe the steps that Mr. Rogers should follow. 2. Give reasons to justify each step.

Example 2

For a course that you are teaching or expect to teach, pre- pare a complete plan for evaluating student achievement. Be sure to include the procedures you would follow, the instruments you would use, and the reasons for your choices. 9

UNOBTRUSIVE MEASURES

Many instruments require the cooperation of the respon- dent in one way or another and involve some kind of intrusion into ongoing activities. On occasion, respon- dents will dislike or even resent being tested, observed, or interviewed. Furthermore, the reaction of respondents to the instrumentation process—that is, to being tested,

fra97851_ch07_110-145.indd 135fra97851_ch07_110-145.indd 135 22/12/10 9:16 PM22/12/10 9:16 PM

136 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

Norm-Referenced Versus Criterion-Referenced Instruments NORM-REFERENCED INSTRUMENTS

All individual scores derive meaning by comparing them to the scores of a particular group. This means that the nature of the group is extremely important. When- ever this is done, researchers must be sure that the refer- ence group makes sense. Comparing a boy’s score on a grammar test to a group of girls’ scores on that test, for example, may be quite misleading since girls usu- ally score higher in grammar. The group to which the comparison is made is called the norm group , and in- struments that provide such information are referred to as norm-referenced instruments .

CRITERION-REFERENCED INSTRUMENTS

An alternative to the use of customary achieve- ment or performance instruments, most of which are norm- referenced, is to use a criterion-referenced instrument —usually a test.

The intent of such tests is somewhat different from that of norm-referenced tests; criterion-referenced tests focus more directly on instruction. Rather than evaluating learner progress through gain in scores (for example, from 40 to 70 on an achievement test), a criterion-referenced test is based on a specifi c goal, or target (called a criterion ), for each learner to achieve. This criterion for mastery, or “pass,” is usually stated as a fairly high percentage of correctly answered ques- tions (such as 80 or 90 percent). Examples of criterion- referenced and norm-referenced evaluation statements are as follows:

Criterion-referenced: A student . . .

• spelled every word in the weekly spelling list correctly. • solved at least 75 percent of the assigned problems. • achieved a score of at least 80 out of 100 on the fi nal

exam. • did at least 25 push-ups within a fi ve-minute period. • read a minimum of one nonfi ction book a week.

Norm-referenced: A student . . .

• scored at the 50th percentile in his group. • scored above 90 percent of all the students in the

class.

• received a higher grade point average in English lit- erature than any other student in the school.

• ran faster than all but one other student on the team. • and one other in the class were the only ones to re-

ceive As on the midterm.

The advantage of a criterion-referenced instrument is that it gives both teacher and students a clear-cut goal to work toward. As a result, it has considerable appeal as a means of improving instruction. In practice, however, several problems arise. First, teachers seldom set or reach the ideal of individualized student goals. Rather, class goals are more the rule, the idea being that all stu- dents will reach the criterion—though, of course, some may not and many will exceed it. The second problem is that it is diffi cult to establish even class criteria that are meaningful. What, precisely, should a class of fi fth- graders be able to do in mathematics? Solve story prob- lems, many would say. We would agree, but of what complexity? and requiring which mathematics sub- skills? In the absence of independent criteria, we have little choice but to fall back on existing expectations, and this is typically (though not necessarily) done by examining existing texts and tests. As a result, the spe- cifi c items in a criterion-referenced test often turn out to be indistinguishable from those in the usual norm-refer- enced test, with one important difference: A criterion- referenced test at any grade level will almost certainly be easier than a norm-referenced test. It must be easier if most students are to get 80 or 90 percent of the items correct. In preparing such tests, researchers must try to write items that will be answered correctly by 80 per- cent of the students—after all, they don’t want 50 per- cent of their students to fail. The desired diffi culty level for norm-referenced items, however, is at or about 50 percent, in order to provide the maximum opportunity for the scores to distinguish the ability of one student from another.

While a criterion-referenced test may be more use- ful at times and in certain circumstances than the more customary norm-referenced test (this issue is still being debated), it is often inferior for research purposes. Why? Because, in general, a criterion-referenced test will provide much less variability of scores, because it is easier. Whereas the usual norm-referenced test will provide a range of scores somewhat less than the possible range (that is, from zero to the total number of items in the test), a criterion-referenced test, if it is true to its rationale, will have most of the students (surely at least half) getting a high score. Because,

fra97851_ch07_110-145.indd 136fra97851_ch07_110-145.indd 136 22/12/10 9:16 PM22/12/10 9:16 PM

C H A P T E R 7 Instrumentation 137

in research, we usually want maximum variability in order to have any hope of fi nding relationships with other variables, the use of criterion-referenced tests is often self-defeating. *

Measurement Scales You will recall from Chapter 3 that there are two basic types of variables—quantitative and categorical. Each uses a different type of analysis and measurement, re- quiring the use of different measurement scales. There are four types of measurement scales: nominal, ordinal, interval, and ratio (Figure 7.26).

NOMINAL SCALES

A nominal scale is the simplest form of measurement researchers can use. When using a nominal scale, re- searchers simply assign numbers to different categories in order to show differences (Figure 7.27). For exam- ple, a researcher concerned with the variable of gender

might group data into two categories, male and female, and assign the number 1 to females and the number 2 to males. Another researcher, interested in studying meth- ods of teaching reading, might assign the number 1 to the whole-word method, the number 2 to the phonics method, and the number 3 to the “mixed” method. In most cases, the advantage to assigning numbers to the categories is to facilitate computer analysis. There is no

* An exception is in program evaluation, where some researchers advocate the use of criterion-referenced tests because they want to determine how many students reach a predetermined standard (criterion).

Nominal Gender

SCALE EXAMPLE

Ordinal Position in a race

4th 3rd 2nd 1st

Interval Temperature (in Fahrenheit)

-20° -10° 0° 10° 20° 30° 40°

20 10

0 –10 –20

Ratio Money

0 $100 $200 $300 $400 $500

$$$$$

Figure 7.26 Four Types of Measurement Scales

“Men are number one!”

“Oh, no! Women are

number one!”

Figure 7.27 A Nominal Scale of Measurement

fra97851_ch07_110-145.indd 137fra97851_ch07_110-145.indd 137 22/12/10 9:16 PM22/12/10 9:16 PM

138 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

implication that the phonics method (assigned number 2) is “more” of anything than the whole-word method (as- signed number 1).

ORDINAL SCALES

An ordinal scale is one in which data may be ordered in some way—high to low or least to most. For example, a researcher might rank-order student scores on a bi- ology test from high to low. Notice, however, that the difference in scores or in actual ability between the fi rst- and second-ranked students and between the fi fth- and sixth-ranked students would not necessarily be the same. Ordinal scales indicate relative standing among individuals, as Figure 7.28 demonstrates.

INTERVAL SCALES

An interval scale possesses all the characteristics of an ordinal scale with one additional feature: The distances between the points on the scale are equal. For example, the distances between scores on most commercially available mathematics achievement tests are usually considered equal. Thus, the distance between scores of 70 and 80 is considered to be the same as the distance between scores of 80 and 90. Notice, however, that the zero point on an interval scale does not indicate a total absence of what is being measured. Thus, 0° (zero de- grees) on the Fahrenheit scale, which measures temper- ature, does not indicate no temperature.

To illustrate further, consider the commonly used IQ score. Is the difference between an IQ of 90 and one of 100 (10 points) the same as the difference between an IQ of 40 and one of 50 (also 10 points)? or between an

IQ of 120 and one of 130? If we believe that the scores constitute an interval scale, we must assume that 10 points has the same meaning at different points on the scale. Do we know whether this is true? No, we do not, as we shall now explain.

With respect to some measurements, we can demon- strate equal intervals. We do so by having an agreed-on standard unit. This is one reason why we have a Bureau of Standards, located in Washington, DC. You could, if you wished to do so, go to the bureau and actually “see” a standard inch, foot, ounce, and so on, that de- fi nes these units. While it might not be easy, you could conceivably check your carpenter’s rule using the “stan- dard inch” to see if an inch is an inch all along your rule. You literally could place the “standard inch” at various points along your rule.

There is no such standard unit for IQ or for virtually any variable commonly used in educational research. Over the years, sophisticated and clever techniques have been developed to create interval scales for use by re- searchers. The details are beyond the scope of this text, but you should know that they all are based on highly questionable assumptions.

In actual practice, most researchers prefer to “act as if” they have an interval scale, because it permits the use of more sensitive data analysis procedures and because, over the years, the results of doing so make sense. Nev- ertheless, acting as if we have interval scales requires an assumption that (at least to date) cannot be proved.

RATIO SCALES

An interval scale that does possess an actual, or true, zero point is called a ratio scale . For example, a scale

Figure 7.28 An Ordinal Scale: The Outcome of a Horse Race

4

Old Paint

3

Slick Sam

2

Starlight

1

Bellybuster

fra97851_ch07_110-145.indd 138fra97851_ch07_110-145.indd 138 22/12/10 9:16 PM22/12/10 9:16 PM

Characteristics of the Four Types of Measurement Scales

Measurement Scale Characteristics

Nominal Groups and labels data only; reports frequencies or percentages.

Ordinal Ranks data; uses numbers only to indicate ranking.

Interval Assumes that equal differences between scores really mean equal differences in the variable measured.

Ratio All of the above, plus a true zero point.

Often researchers have a choice to make. They must decide whether to consider data as ordinal or interval. For example, suppose a researcher uses a self-report questionnaire to measure self-esteem. The question- naire is scored for the number of items answered (yes or no) in the direction indicating high self-esteem. For a given sample of 60, the researcher fi nds that the scores range from 30 to 75.

The researcher may now decide to treat scores as inter- val data, in which case she assumes that equal distances (e.g., 30–34, 35–39, 40–44) in score represent equal dif- ferences in self-esteem. * If the researcher is uncomfort- able with this assumption, she could use the scores to rank the individuals in her sample from highest (rank 1) to low- est (rank 60). If she were then to use only these rankings

designed to measure height would be a ratio scale, be- cause the zero point on the scale represents the absence of height (that is, no height). Similarly, the zero on a bathroom weight scale represents zero, or no, weight. Ratio scales are almost never encountered in educa- tional research, since rarely do researchers engage in measurement involving a true zero point (even on those rare occasions when a student receives a zero on a test of some sort, this does not mean that whatever is being measured is totally absent in the student). Some other variables that do have ratio scales are income, time on task, and age.

MEASUREMENT SCALES RECONSIDERED

At this point, you may be saying, Well, okay, but so what? Why are these distinctions important? There are two reasons why you should have at least a ru- dimentary understanding of the differences between these four types of scales. First, they convey differ- ent amounts of information. Ratio scales provide more information than do interval scales; interval, more than ordinal; and ordinal, more than nominal. Hence, if possible, researchers should use the type of mea- surement that will provide them with the maximum amount of information needed to answer their research question. Second, some types of statistical procedures are inappropriate for the different scales. The way in which the data in a research study are organized dic- tates the use of certain types of statistical analyses, but not others (we shall discuss this point in more detail in Chapter 11). Table 7.1 presents a summary of the four types of measurement scales.

TABLE 7.1

CONTROVERSIES IN RESEARCH

Which Statistical Index Is Valid?

M easurements, of course, are not limited to test scores and the like. For example, a widely used measurement is the “index of unemployment” provided by the Bureau of Labor Statistics. One of its many uses is to study the relation- ship between unemployment and crime. Its validity for this purpose has been questioned by the author of a recent study who found that while this index showed no relationship to

property crimes, two different indexes that refl ected long-term (rather than temporary) unemployment did show substantial relationships. The author concluded, “Clearly, we need to be- come as interested in the selection of appropriate indicators as we are with the specifi cation of appropriate models and the selection of statistical techniques.” *

What do you think? Why did the different indexes give different results?

* M. B. Chamlin (2000). Unemployment, economic theory, and prop- erty crime: A note on measurement. Journal of Quantitative Crimi- nology, 16 (4): 454.

* Notice that she cannot treat the scores as ratio data, since a score of zero cannot be assumed to represent zero (i.e., no) self-esteem.

139

fra97851_ch07_110-145.indd 139fra97851_ch07_110-145.indd 139 22/12/10 9:16 PM22/12/10 9:16 PM

140 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

in subsequent analysis, she would now be assuming that her instrument provides only ordinal data.

Fortunately, researchers can avoid this choice. They have another option—to treat the data separately ac- cording to both assumptions (that is, to treat the scores as ordinal data, and then again as interval data). The im- portant thing to realize is that a researcher must be pre- pared to defend the assumptions underlying her choice of a measurement scale used in the collection and orga- nization of data.

Preparing Data for Analysis Once the instruments being used in a study have been administered, the researcher must score the data that have been collected and then organize it to facilitate analysis.

SCORING THE DATA

Collected data must be scored accurately and consis- tently. If they are not, any conclusions a researcher draws from the data may be erroneous or misleading. Each individual’s test (questionnaire, essay, etc.) should be scored using exactly the same procedures and criteria. When a commercially purchased instrument is used, the scoring procedures are made much easier. Usually the instrument developer will provide a scoring manual that lists the steps to follow in scoring the instrument, along with a scoring key. It is a good idea to double-check one’s scoring to ensure that no mistakes have been made.

The scoring of a self-developed test can produce dif- fi culties, and hence researchers have to take special care to ensure that scoring is accurate and consistent. Essay examinations, in particular, are often very diffi cult to score in a consistent manner. For this reason, it is usu- ally advisable to have a second person also score the results. Researchers should carefully prepare their scor- ing plans, in writing, ahead of time and then try out their instrument by administering and scoring it with a group of individuals similar to the population they intend to sample in their study. Problems with administration and scoring can thus be identifi ed early and corrected before it is too late.

TABULATING AND CODING THE DATA

When the data have been scored, the researcher must tally or tabulate them in some way. Usually this is done by transferring the data to some sort of summary data

sheet or card. The important thing is to record one’s data accurately and systematically. If categorical data are being recorded, the number of individuals scoring in each category are tallied. If quantitative data are being recorded, the data are usually listed in one or more col- umns, depending on the number of groups involved in the study. For example, if the data analysis is to con- sist simply of a comparison of the scores of two groups on a posttest, the data would most likely be placed in two columns, one for each group, in descending order. Table 7.2, for example, presents some hypothetical re- sults of a study involving a comparison of two counsel- ing methods with an instrument measuring rapport. If pre- and post-test scores are to be compared, additional columns could be added. Subgroup scores could also be indicated.

When different kinds of data are collected (i.e., scores on several different instruments) in addition to biographical information (gender, age, ethnicity, etc.), they are usually recorded in a computer or on data cards, one card for each individual from whom data were collected. This facilitates easy comparison and grouping (and regrouping) of data for purposes of anal- ysis. In addition, the data are coded. In other words, some type of code is used to protect the privacy of the

Hypothetical Results of Study Involving a Comparison of Two Counseling Methods

Score for “Rapport” Method A Method B

96–100 0 0

91–95 0 2

86–90 0 3

81–85 2 3

76–80 2 4

71–75 5 3

66–70 6 4

61–65 9 4

56–60 4 5

51–55 5 3

46–50 2 2

41–45 0 1

36–40 0 1

N 5 35 35

TABLE 7.2

fra97851_ch07_110-145.indd 140fra97851_ch07_110-145.indd 140 22/12/10 9:16 PM22/12/10 9:16 PM

C H A P T E R 7 Instrumentation 141

individuals in the study. Thus, the names of males and females might be coded as 1 and 2. Coding of data is es- pecially important when data are analyzed by computer, since any data not in numerical form must be coded in some systematic way before they can be entered into the computer. Thus, categorical data, to be analyzed on a computer, are often coded numerically (e.g., pretest scores 1, and posttest scores 2).

The fi rst step in coding data is often to assign an ID number to every individual from whom data has been collected. If there were 100 individuals in a study, for example, the researcher would number them from 001 to 100. If the highest value for any variable being ana- lyzed involves three digits (e.g., 100), then every indi- vidual code number must have three digits (e.g., the fi rst individual to be numbered must be 001, not 1).

The next step would be to decide how any categori- cal data being analyzed are to be coded. Suppose a researcher wished to analyze certain demographic in- formation obtained from 100 subjects who answered a questionnaire. If his study included juniors and seniors in a high school, he might code the juniors as 11 and the seniors as 12. Or, if respondents were asked to indi- cate which of four choices they preferred (as in certain multiple-choice questions), the researcher might code each of the choices [e.g., ( a ), ( b ), ( c ), ( d ) as 1, 2, 3, or 4, respectively]. The important thing to remember is that the coding must be consistent—that is, once a decision is made about how to code someone, all others must be coded the same way, and this (and any other) cod- ing rule must be communicated to everyone involved in coding the data.

Go back to the INTERACTIVE AND APPLIED LEARNING feature at the beginning of the chapter for a listing of interactive and applied activities. Go to the Online Learning Center at www.mhhe.com/fraenkel8e to take quizzes, practice with key terms, and review chapter content.

WHAT ARE DATA? • The term data refers to the kinds of information researchers obtain on the subjects

of their research.

INSTRUMENTATION • The term instrumentation refers to the entire process of collecting data in a research

investigation.

VALIDITY AND RELIABILITY • An important consideration in the choice of a research instrument is validity: the ex-

tent to which results from it permit researchers to draw warranted conclusions about the characteristics of the individuals studied.

• A reliable instrument is one that gives consistent results.

OBJECTIVITY AND USABILITY • Whenever possible, researchers try to eliminate subjectivity from the judgments they

make about the achievement, performance, or characteristics of subjects. • An important consideration for any researcher in choosing or designing an instru-

ment is its ease of use.

Main Points

fra97851_ch07_110-145.indd 141fra97851_ch07_110-145.indd 141 22/12/10 9:16 PM22/12/10 9:16 PM

142 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

WAYS TO CLASSIFY INSTRUMENTS • Research instruments can be classifi ed in many ways. Some of the more common are

in terms of who provides the data, the method of data collection, who collects the data, and what kind of response they require from the subjects.

• Research data are obtained by directly or indirectly assessing the subjects of a study. • Self-report data are provided by the subjects of a study themselves. • Informant data are provided by other people about the subjects of a study.

TYPES OF INSTRUMENTS • There are many types of researcher-completed instruments. Some of the more com-

monly used are rating scales, interview schedules, observation forms, tally sheets, fl owcharts, performance checklists, anecdotal records, and time-and-motion logs.

• Many types of instruments are completed by the subjects of a study rather than the researcher. Some of the more commonly used of this type are questionnaires; self- checklists; attitude scales; personality inventories; achievement, aptitude, and perfor- mance tests; and projective and sociometric devices.

• The types of items or questions used in subject-completed instruments can take many forms, but they all can be classifi ed as either selection or supply items. Examples of selection items include true-false items, multiple-choice items, matching items, and interpretive exercises. Examples of supply items include short-answer items and essay questions.

• An excellent source for locating already available tests is the ERIC database. • Unobtrusive measures require no intrusion into the normal course of affairs.

NORM-REFERENCED VERSUS CRITERION-REFERENCED INSTRUMENTS • Instruments that provide scores that compare individual scores to the scores of an

appropriate reference group are called norm-referenced instruments. • Instruments that are based on a specifi c target for each learner to achieve are called

criterion-referenced instruments.

MEASUREMENT SCALES • Four types of measurement scales—nominal, ordinal, interval, and ratio—are used

in educational research. • A nominal scale uses numbers to indicate membership in one or more categories. • An ordinal scale uses numbers to rank or order scores from high to low. • An interval scale uses numbers to represent equal intervals in different segments on

a continuum. • A ratio scale uses numbers to represent equal distances from a known zero point.

PREPARING DATA FOR ANALYSIS • Collected data must be scored accurately and consistently. • Once scored, data must be tabulated and coded.

fra97851_ch07_110-145.indd 142fra97851_ch07_110-145.indd 142 22/12/10 9:16 PM22/12/10 9:16 PM

C H A P T E R 7 Instrumentation 143

Key Terms

For Discussion

Notes

achievement test 127

aptitude test 128

criterion-referenced instrument 136

data 111

informants 113

instrumentation 111

interview protocol 119

interval scale 138

Likert scale 126

nominal scale 137

norm group 136

norm-referenced instrument 136

objectivity 112

ordinal scale 138

performance instrument 116

projective device 130

questionnaire 113

ratio scale 138

reliability 112

semantic differential 127

sociometric device 131

tally sheet 113

unobtrusive measure 135

validity 112

written-response instrument 116

1. What type of instrument do you think would be best suited to obtain data about each of the following? a. The free-throw shooting ability of a tenth-grade basketball team b. How nurses feel about a new management policy recently instituted in their

hospital c. Parental reactions to a proposed campaign to raise money for an addition to the

school library d. The “best-liked” boy and girl in the senior class e. The “best” administrator in a particular school district f. How well students in a food management class can prepare a balanced meal g. Characteristics of all students who are biology majors at a midwestern university h. How students at one school compare to students at another school in mathematics

ability i. The potential of various high school seniors for college work j. What the members of a kindergarten class like and dislike about school

2. Which do you think would be the easiest to measure—the attention level of a class, student interest in a poem, or participation in a class discussion? Why? Which would be the hardest to measure?

3. Is it possible to measure a person’s self-concept? If so, how? What about his or her body image?

4. Are there any things (ideas, objects, etc.) that cannot be measured? If so, give an example.

5. Of all the instruments presented in this chapter, which one(s) do you think would be the hardest to use? the easiest? Why? Which one(s) do you think would provide the most dependable information? Why?

6. It sometimes would not be fair to compare an individual’s score on a test to the scores of other individuals taking the same test. Why?

1. American Council on Education (1945). Helping teachers understand children. Washington, DC: American Council on Education, pp. 32–33.

2. Ibid., p. 33.

3. A. Likert (1932). A technique for the measurement of attitudes. Archives de Psychologie, 6 (140): 173–177.

fra97851_ch07_110-145.indd 143fra97851_ch07_110-145.indd 143 22/12/10 9:16 PM22/12/10 9:16 PM

144 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

4. C. Osgood, G. Suci, and P. Tannenbaum (1962). The measurement of meaning. Urbana, IL: University of Illinois Press.

5. See, for example, W. J. Popham (1992). Educational evaluation, 3rd ed. Englewood Cliffs, NJ: Prentice Hall, pp. 150–173.

6. E. I. Sawin (1969). Evaluation and the work of the teacher. Belmont, CA: Wadsworth, p. 176.

7. M. L. Blum (1943). Selection of sewing machine operators. Journal of Applied Psychology, 27 (2): 35–40.

8. N. E. Gronlund (1988). How to construct achievement tests, 4th ed. Englewood Cliffs, NJ: Prentice Hall, pp. 66–67. Reprinted by permission of Prentice Hall, Inc. Also see Gronlund’s (1997) Assessment of student achievement, 6th ed. Boston: Allyn & Bacon.

9. Gronlund (1988), pp. 76–77. Reprinted by permission of Prentice Hall, Inc.

10. E. J. Webb, D. T. Campbell, R. D. Schwartz, and L. Sechrest (1966). Unobtrusive measures: Nonreactive research in the social sciences. Chicago: Rand McNally.

11. The use of unobtrusive measures is an art in itself. We can only scratch the surface of the topic here. For a more extended discussion, along with many interesting examples, the reader is referred to the book by Webb et al., in note 10.

fra97851_ch07_110-145.indd 144fra97851_ch07_110-145.indd 144 22/12/10 9:16 PM22/12/10 9:16 PM

C H A P T E R 7 Instrumentation 145

Research Exercise 7: Instrumentation Name all of the instrument(s) you intend to use in your study, and whether they are preexist- ing or if you will need to develop them yourself. Describe how you will use the instrument(s) (i.e., where, when and how you will collect the data you will need) and how many items each instrument will contain. Then explain how the instrument is to be scored or how it can be interpreted.

Problem Sheet 7

Instrumentation 1. Describe the type(s) of instrument(s) you plan to use for your study (e.g., interview

protocol, attitudinal survey, achievement test, observation scale, questionnaire, focus group protocol, etc.).

____________________________________________________________________

____________________________________________________________________

2. Is it a preexisting instrument or one you plan to develop?

____________________________________________________________________

____________________________________________________________________

3. If preexisting, state the name of the instrument. Also, why did you decide to use this particular instrument?

____________________________________________________________________

____________________________________________________________________

4. What is the instrument supposed to measure or assess?

____________________________________________________________________

____________________________________________________________________

5. How many items will the instrument contain?

____________________________________________________________________

____________________________________________________________________

6. How will the instrument be scored or interpreted?

____________________________________________________________________

____________________________________________________________________

An electronic version of this Problem Sheet that you can fi ll in and print, save, or e-mail is available on the Online Learning Center at www.mhhe.com/ fraenkel8e.

fra97851_ch07_110-145.indd 145fra97851_ch07_110-145.indd 145 22/12/10 9:16 PM22/12/10 9:16 PM

8

O B J E C T I V E S Studying this chapter should enable you to:

• Explain what is meant by the term “validity” as it applies to the use of instruments in educational research.

• Name three types of evidence of validity that can be obtained, and give an example of each type.

• Explain what is meant by the term “correlation coeffi cient” and describe briefl y the difference between positive and negative correlation coeffi cients.

• Explain what is meant by the terms “validity coeffi cient” and “reliability coeffi cient.”

• Explain what is meant by the term “reliability” as it applies to the use of instruments in educational research.

• Explain what is meant by the term “errors of measurement.”

• Explain briefl y the meaning and use of the term “standard error of measurement.”

• Describe briefl y three ways to estimate the reliability of the scores obtained using a particular instrument.

• Describe how to obtain and evaluate scoring agreement.

The Importance of Valid Instrumentation

Validity

Content-Related Evidence Criterion-Related Evidence Construct-Related Evidence

Reliability

Errors of Measurement Test-Retest Method Equivalent-Forms Method Internal-Consistency Methods The Standard Error of

Measurement (SEMeas) Scoring Agreement Validity and Reliability in

Qualitative Research

Validity and Reliability

“Sounds to me like an invalid test.” “But he says

it’s very reliable.”

“I’ve failed Mr. Johnson’s test three times. But he keeps

giving me problems that he has never taught or told

us about.”

“Can a test be reliable but

not valid?”

fra97851_ch08_146-164.indd 146fra97851_ch08_146-164.indd 146 12/21/10 6:46 PM12/21/10 6:46 PM

147

to the selection or design of the instruments a researcher intends to use. In this chapter, therefore, we shall dis- cuss both validity and reliability in some detail.

Validity Validity is the most important idea to consider when preparing or selecting an instrument for use. More than anything else, researchers want the information they obtain through the use of an instrument to serve their purposes. For example, to fi nd out what teachers in a particular school district think about a recent policy passed by the school board, researchers need both an instrument to record the data and some sort of assur- ance that the information obtained will enable them to

The Importance of Valid Instrumentation The quality of the instruments used in research is very important, for the conclusions researchers draw are based on the information they obtain using these instru- ments. Accordingly, researchers use a number of proce- dures to ensure that the inferences they draw, based on the data they collect, are valid and reliable.

Validity refers to the appropriateness, meaningful- ness, correctness, and usefulness of the inferences a re- searcher makes. Reliability refers to the consistency of scores or answers from one administration of an instru- ment to another, and from one set of items to another. Both concepts are important to consider when it comes

Go to the Online Learning Center at www.mhhe.com/fraenkel8e to:

• Learn More About Validity and Reliability

Go to your online Student Mastery Activities book to do the following activities:

• Activity 8.1: Instrument Validity • Activity 8.2: Instrument Reliability (1) • Activity 8.3: Instrument Reliability (2) • Activity 8.4: What Kind of Evidence: Content-Related,

Criterion-Related, or Construct-Related? • Activity 8.5: What Constitutes Construct-Related

Evidence of Validity?

INTERACTIVE AND APPLIED LEARNING After, or while, reading this chapter:

I t isn’t fair, Tony!” “What isn’t, Lily?” “Those tests that Mrs. Leonard gives. Grrr!”

“What about them?”

“Well, take this last test we had on the Civil War. All during her lectures and the class discussions over the last few weeks,

we’ve been talking about the causes and effects of the War.”

“So?”

“Well, then on this test, she asked a lot about battles and generals and other stuff that we didn’t study.”

“Did you ask her how come?”

“Yeah, I did. She said she wanted to test our thinking ability. But she was asking us to think about material she hadn’t even

gone over or discussed in class. That’s why I think she isn’t fair.”

Lily is correct. Her teacher, in this instance, isn’t being fair. Although she isn’t using the term, what Lily is talking about is a

matter of validity . It appears Mrs. Leonard is giving an invalid test. What this means, and why it isn’t a good thing for a teacher

(or any researcher) to do, is largely what this chapter is about.

fra97851_ch08_146-164.indd 147fra97851_ch08_146-164.indd 147 12/21/10 6:46 PM12/21/10 6:46 PM

148 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

low score? And so forth. It is one thing to collect infor- mation from people. We do this all the time—names, addresses, birth dates, shoe sizes, car license numbers, and so on. But unless we can make inferences that mean something from the information we obtain, it is of little use. The purpose of research is not merely to collect data but to use such data to draw warranted conclusions about the people (and others like them) on whom the data were collected.

A useful inference is one that helps researchers make a decision related to what they were trying to fi nd out. Researchers interested in the effects of inquiry- related teaching materials on student achievement, for example, need information that will enable them to infer whether achievement is affected by such materi- als and, if so, how.

Validity, therefore, depends on the amount and type of evidence there is to support the interpretations re- searchers wish to make concerning data they have col- lected. The crucial question is: Do the results of the assessment provide useful information about the topic or variable being measured?

What kinds of evidence might a researcher collect? Essentially, there are three main types.

Content-related evidence of validity refers to the content and format of the instrument. How appro- priate is the content? How comprehensive? Does it logically get at the intended variable? How ad- equately does the sample of items or questions represent the content to be assessed? Is the format appropriate? The content and format must be con- sistent with the defi nition of the variable and the sample of subjects to be measured.

Criterion-related evidence of validity refers to the relationship between scores obtained using the in- strument and scores obtained using one or more other instruments or measures (often called a cri- terion ). How strong is this relationship? How well do such scores estimate, present, or predict future performance of a certain type?

Construct-related evidence of validity refers to the nature of the psychological construct or charac- teristic being measured by the instrument. How well does a measure of the construct explain dif- ferences in the behavior of individuals or their performance on certain tasks? We provide further explanation of this rather complex concept later in the chapter.

Figure 8.1 illustrates these three types of evidence.

draw correct conclusions about teacher opinions. The drawing of correct conclusions based on the data ob- tained from an assessment is what validity is all about. Though not essential, some kind of score that summa- rizes the information for each person greatly simplifi es the comprehension and use of data, and because most instruments provide such scores, we present the follow- ing discussion in this context.

In recent years, validity has been defi ned as referring to the appropriateness, correctness, meaningfulness, and usefulness of the specifi c inferences researchers make based on the data they collect. Validation is the process of collecting and analyzing evidence to sup- port such inferences. There are many ways to collect evidence, and we will discuss some of them shortly. The important point here is to realize that validity refers to the degree to which evidence supports any inferences a researcher makes based on the data he or she collects using a particular instrument. It is the inferences about the specifi c uses of an instrument that are validated, not the instrument itself. * These inferences should be appropriate, meaningful, correct, and useful.

One interpretation of this conceptualization of validity has been that test publishers no longer have a responsibility to provide evidence of validity. We do not agree; publishers have an obligation to state what an instrument is intended to measure and to provide evidence that it does. None- theless, researchers must still give attention to the way in which they intend to interpret the information.

An appropriate inference would be one that is relevant—that is, related—to the purposes of the study. If the purpose of a study were to determine what students know about African culture, for example, it would make no sense to make inferences about this from their scores on a test about the physical geography of Africa.

A meaningful inference is one that says something about the meaning of the information (such as test scores) obtained through the use of an instrument. What exactly does a high score on a particular test mean? What does such a score allow us to say about the individual who received it? In what way is an individual who re- ceives a high score different from one who receives a

*This is somewhat of a change from past interpretations. It is based on the set of standards prepared by a joint committee consisting of mem- bers of the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education. See American Psychological Association (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association, pp. 9–18, 19–23.

fra97851_ch08_146-164.indd 148fra97851_ch08_146-164.indd 148 12/21/10 6:46 PM12/21/10 6:46 PM

C H A P T E R 8 Validity and Reliability 149

CONTENT-RELATED EVIDENCE

Suppose a researcher is interested in the effects of a new math program on the mathematics ability of fi fth- graders. The researcher expects that students who com- plete the program will be able to solve a number of different types of word problems correctly. To assess their mathematics ability, the researcher plans to give them a math test containing about 15 such problems.

Content-related evidence

Criterion-related evidence

Johnny

Maria

Raul

Amy

Yes

No

No

Yes

Yes

No

No

Yes

Instrument A (e.g., test)

Instrument B (e.g., observation scale)

Construct-related evidence

Prediction 1 — confirmed

Prediction 2 — confirmed

Prediction 3 — confirmed

Prediction ? — confirmed

Theory

Definition Sample Content Format

Figure 8.1 Types of Evidence of Validity

The performance of the students on this test is impor- tant only to the degree that it provides evidence of their ability to solve these kinds of problems. Hence, per- formance on the instrument in this case (the math test) will provide valid evidence of the mathematics ability of these students if the instrument provides an adequate sample of the types of word problems covered in the program. If only easy problems are included on the test,

fra97851_ch08_146-164.indd 149fra97851_ch08_146-164.indd 149 12/21/10 6:46 PM12/21/10 6:46 PM

150

or only very diffi cult or lengthy ones, or only problems involving subtraction, the test will be unrepresentative and hence not provide information from which valid in- ferences can be made.

One key element in content-related evidence of validity , then, concerns the adequacy of the sampling. Most instruments (and especially achievement tests) provide only a sample of the kinds of problems that might be solved or questions that might be asked. Con- tent validation, therefore, is partly a matter of deter- mining if the content that the instrument contains is an adequate sample of the domain of content it is supposed to represent.

The other aspect of content validation has to do with the format of the instrument. This includes such things as the clarity of printing, size of type, adequacy of work

space (if needed), appropriateness of language, clarity of directions, and so on. Regardless of the adequacy of the questions in an instrument, if they are presented in an in- appropriate format (such as giving a test written in English to children whose English is minimal), valid results can- not be obtained. For this reason, it is important that the characteristics of the intended sample be kept in mind.

How does one obtain content-related evidence of validity? A common way to do this is to have someone look at the content and format of the instrument and judge whether or not it is appropriate. The “someone,” of course, should not be just anyone, but rather an in- dividual who can be expected to render an intelligent judgment about the adequacy of the instrument—in other words, someone who knows enough about what is to be measured to be a competent judge.

stress on young children. And some argue that they are too long (in Massachusetts they can take up to 13 hours!) or too tough or simply not good enough.” †

In response, the American Educational Research Associa- tion developed a position statement of “conditions essential to sound implementation of high-stakes educational testing pro- grams.” ‡ It contained 14 specifi c points, 4 of the most impor- tant being that (1) such decisions about students should not be based on test scores alone; (2) tests should be made fairer to all students; (3) tests should match the curriculum; and (4) the re- liability and validity of tests should continually be evaluated.

Two examples of responses to the guidelines were the following:

• “In the face of too much testing with far too severe conse- quences, the AERA positions, if implemented, would be a step forward relative to current practice.” §

• “The statement refl ects what is desired for all state tests and assessments. But, just as all students have not yet met the standards, not all state tests and assessments will im- mediately meet the goals contained in this statement.” ||

What do you think? Are the complaints about high-stakes tests warranted?

CONTROVERSIES IN RESEARCH

High-Stakes Testing

H igh-stakes testing refers to the use of tests (often only a single achievement test) as the primary, or only basis for decisions having major consequences. For students, such consequences include retention in grade and/or the denial of diplomas and awards. For schools, they include public praise or condemnation, sanctions, and fi nancial rewards or punish- ments. “In state after state, legislatures, governors, and state boards, supported by business leaders, have imposed tougher requirements in mathematics, English, science, and other fi elds, together with new tests by which the performance of both students and schools is to be judged.” *

For years, tests had been used as one indicator of perfor- mance; what was new was exclusive reliance on them. “The backlash, touching virtually every state that has instituted high-stakes testing, arises from a spectrum of complaints. A major complaint is that the focus on testing and obsessive test preparation, sometimes beginning in kindergarten, is killing innovative teaching and curricula and driving out good teach- ers. Other complaints are that (conversely) the standards on which the tests are based are too vague and that students have not been taught the material on which the tests are based; or that the tests are unfair to poor and minority students or to those who lack test-taking skills; or that the tests put too much

*P. Schrag (2000). High stakes are for tomatoes. Atlantic Monthly, 286 (August): 19.

†Ibid. ‡American Educational Research Association (2000). Position state- ment of the American Educational Research Association concerning high-stakes testing in pre-12 education. Educational Researcher, 29 (11): 24–35. §M. Neill (2000). Quoted in Initial responses to AERA’s position statement concerning high-stakes testing. Educational Researcher, 29 (11): 28. ||W. Martin (2000). Quoted in Ibid., p. 27.

fra97851_ch08_146-164.indd 150fra97851_ch08_146-164.indd 150 12/21/10 6:46 PM12/21/10 6:46 PM

C H A P T E R 8 Validity and Reliability 151

2. Those who believe that increasing consumer ex- penditures would be the best way to stimulate the economy would advocate

a. an increase in interest rates. b. an increase in depletion allowances. c. tax reductions in the lower income brackets. d. a reduction in government expenditures. 3. Compare the dollar amounts spent by the U.S.

government during the past 10 years for ( a ) debt payments, ( b ) defense, and ( c ) social services.

Now, look at each of the questions and the corre- sponding objective they are supposed to measure. Do you think each question measures the objective it was designed for? If not, why not? *

Example 2. Here is what another researcher de- signed as an attempt to measure (at least in part) the ability of students to explain why events occur.

Read the directions that follow, and then answer the question.

Directions: Here are some facts.

Fact W: A camper started a fi re to cook food on a windy day in a forest.

Fact X: A fi re started in some dry grass near a campfi re in a forest.

Here is another fact that happened later the same day in the same forest.

Fact Y: A house in the forest burned down.

You are to explain what might have caused the house to burn down (Fact Y). Would Fact W and X be useful as parts of your explanation?

a. Yes, both W and X and the possible cause-and- effect relationship between them would be useful.

b. Yes, both W and X would be useful, even though neither was likely a cause of the other.

c. No, because only one of Facts W and X was likely a cause of Y.

d. No, because neither W or X was likely a cause of Y. 1

The usual procedure is somewhat as follows. The researcher writes out the defi nition of what he or she wants to measure and then gives this defi nition, along with the instrument and a description of the intended sample, to one or more judges. The judges look at the defi nition, read over the items or questions in the instrument, and place a check mark in front of each question or item that they feel does not measure one or more aspects of the defi nition (objectives, for example) or other criteria. They also place a check mark in front of each aspect not assessed by any of the items. In addition, the judges evaluate the appro- priateness of the instrument format. The researcher then rewrites any item or question so checked and resubmits it to the judges, and/or writes new items for criteria not adequately covered. This continues until the judges approve all the items or questions in the instrument and also indicate that they feel the total number of items is an adequate representation of the total domain of content covered by the variable being measured.

To illustrate how a researcher might go about trying to establish content-related validity, let us consider two examples.

Example 1. Suppose a researcher desires to measure students’ ability to use information that they have pre- viously acquired. When asked what she means by this phrase, she offers the following defi nition.

As evidence that students can use previously acquired in- formation, they should be able to:

1. Draw a correct conclusion (verbally or in writing) that is based on information they are given.

2. Identify one or more logical implications that follow from a given point of view.

3. State (orally or in writing) whether two ideas are iden- tical, similar, unrelated, or contradictory.

How might the researcher obtain such evidence? She decides to prepare a written test that will contain vari- ous questions. Students’ answers will constitute the evi- dence she seeks. Here are three examples of the kinds of questions she has in mind, designed to produce each of the three types of evidence listed above.

1. If A is greater than B, and B is greater than C, then: a. A must be greater than C. b. C must be smaller than A. c. B must be smaller than A. d. All of the above are true.

*We would rate correct answers to questions 1 (choice d ) and 2 (choice c ) as valid evidence, although 1 could be considered ques- tionable, since students might view it as somewhat tricky. We would not rate the answers to 3 as valid, since students are not asked to contrast ideas, only facts.

fra97851_ch08_146-164.indd 151fra97851_ch08_146-164.indd 151 12/21/10 6:46 PM12/21/10 6:46 PM

152 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

obtain on two instruments. A positive relationship is in- dicated when a high score on one of the instruments is accompanied by a high score on the other or when a low score on one is accompanied by a low score on the other. A negative relationship is indicated when a high score on one instrument is accompanied by a low score on the other, and vice versa. All correlation coeffi cients fall somewhere between 11.00 and 21.00. An r of .00 indicates that no relationship exists.

When a correlation coeffi cient is used to describe the relationship between a set of scores obtained by the same group of individuals on a particular instru- ment and their scores on some criterion measure, it is called a validity coefficient . For example, a validity coeffi cient of 11.00 obtained by correlating a set of scores on a mathematics aptitude test (the predictor) and another set of scores, this time on a mathemat- ics achievement test (the criterion), for the same in- dividuals would indicate that each individual in the group had exactly the same relative standing on both measures. Such a correlation, if obtained, would allow the researcher to predict perfectly math achievement based on aptitude test scores. Although this correla- tion coeffi cient would be very unlikely, it illustrates what such coeffi cients mean. The higher the validity coeffi cient obtained, the more accurate a researcher’s predictions are likely to be.

Once again, look at the question and the objective it was designed to measure. Does it measure this objec- tive? If not, why not? *

Attempts like these to obtain evidence of some sort (in the above instances, the support of independent judges that the items measure what they are supposed to measure) typify the process of obtaining content- related evidence of validity. As we mentioned previously, how- ever, the qualifi cations of the judges are always an im- portant consideration, and the judges must keep in mind the characteristics of the intended sample.

CRITERION-RELATED EVIDENCE

To obtain criterion-related evidence of validity , re- searchers usually compare performance on one instru- ment (the one being validated) with performance on some other, independent criterion. A criterion is a second test or other assessment procedure presumed to measure the same variable. For example, if an instrument has been designed to measure academic ability, student scores on the instrument might be compared with their grade-point averages (the external criterion). If the instrument does indeed measure academic ability, then students who score high on the test would also be expected to have high grade-point averages. Can you see why?

There are two forms of criterion-related validity— predictive and concurrent. To obtain evidence of predic- tive validity , researchers allow a time interval to elapse between administration of the instrument and obtaining the criterion scores. For example, a researcher might ad- minister a science aptitude test to a group of high school students and later compare their scores on the test with their end-of-semester grades in science courses.

On the other hand, when instrument data and crite- rion data are gathered at nearly the same time, and the results are compared, this is an attempt by researchers to obtain evidence of concurrent validity . An example is when a researcher administers a self-esteem inventory to a group of eighth-graders and compares their scores on it with their teachers’ ratings of student self-esteem obtained at about the same time.

A key index in both forms of criterion-related valid- ity is the correlation coeffi cient. † A correlation coef- ficient , symbolized by the letter r , indicates the degree of relationship that exists between the scores individuals

*We would rate a correct answer to this question as valid evidence of student ability to explain why events occur. †The correlation coeffi cient, explained in detail in Chapter 10, is an extremely useful statistic. This is one of its many applications or uses.

©The New Yorker Collection 2000 Sidney Harris from cartoonbank.com.

All Rights Reserved.

fra97851_ch08_146-164.indd 152fra97851_ch08_146-164.indd 152 12/21/10 6:46 PM12/21/10 6:46 PM

C H A P T E R 8 Validity and Reliability 153

CONSTRUCT-RELATED EVIDENCE

Construct-related evidence of validity is the broadest of the three categories of evidence for validity that we are considering. There is no single piece of evidence that satisfi es construct-related validity. Rather, researchers at- tempt to collect a variety of different types of evidence (the more and the more varied the better) that will allow them to make warranted inferences—to assert, for example, that the scores obtained from administering a self-esteem inventory permit accurate inferences about the degree of self-esteem that people who receive those scores possess.

Usually, there are three steps involved in obtaining construct-related evidence of validity: (1) the variable being measured is clearly defi ned; (2) hypotheses, based on a theory underlying the variable, are formed about how people who possess a lot versus a little of the vari- able will behave in a particular situation; and (3) the hypotheses are tested both logically and empirically.

To make the process clearer, let us consider an ex- ample. Suppose a researcher interested in developing a pencil-and-paper test to measure honesty wants to use a construct-validity approach. First, he defi nes honesty . Next he formulates a theory about how “honest” people behave as compared to “dishonest” people. For example, he might theorize that honest individuals, if they fi nd an object that does not belong to them, will make a reason- able effort to locate the individual to whom the object belongs. Based on this theory, the researcher might hy- pothesize that individuals who score high on his honesty test will be more likely to attempt to locate the owner of an object they fi nd than individuals who score low on the test. The researcher then administers the honesty test, separates the names of those who score high and those who score low, and gives all of them an opportu- nity to be honest. He might, for example, leave a wal- let with $5 in it lying just outside the test-taking room so that the individuals taking the test can easily see it and pick it up. The wallet displays the name and phone number of the owner in plain view. If the researcher’s hypothesis is substantiated, more of the high scorers than the low scorers on the honesty test will attempt to call the owner of the wallet. (This could be checked by having the number answered by a recording machine asking the caller to leave his or her name and number.) This is one piece of evidence that could be used to sup- port inferences about the honesty of individuals, based on the scores they receive on this test.

We must stress, however, that a researcher must carry out a series of studies to obtain a variety of evidence

Gronlund suggests the use of an expectancy table as another way to depict criterion-related evidence. 2 An expectancy table is nothing more than a two-way chart, with the predictor categories listed down the left-hand side of the chart and the criterion categories listed hori- zontally along the top of the chart. For each category of scores on the predictor, the researcher then indicates the percentage of individuals who fall within each of the categories on the criterion.

Table 8.1 presents an example. As you can see from the table, 51 percent of the students who were classifi ed out- standing by these judges received a grade of A in orches- tra, 35 percent received a B, and 14 percent received a C. Although this table refers only to this particular group, it could be used to predict the scores of other aspiring music students who were evaluated by these same judges. If a student obtained an evaluation of “outstanding,” we might predict (approximately) that he or she would have a 51 percent chance of receiving an A, a 35 percent chance of receiving a B, and a 14 percent chance of receiving a C.

Expectancy tables are particularly useful devices for researchers to use with data collected in schools. They are simple to construct, easily understood, and clearly show the relationship between two measures.

It is important to realize that the nature of the crite- rion is the most important factor in gathering criterion- related evidence. High positive correlations do not mean much if the criterion measure does not make logical sense. For example, a high correlation between scores on an instrument designed to measure aptitude for sci- ence and scores on a physical fi tness test would not be relevant criterion-related evidence for either instrument. Think back to the example we presented earlier of the questions designed to measure student ability to explain why events occur. What sort of criteria could be used to establish criterion-referenced validity for those items?

TABLE 8.1 Example of an Expectancy Table

Judges’ Classifi cation of Music Aptitude

Course Grades in Orchestra

(Percentage Receiving Each Grade)

A B C D

Outstanding 51 35 14 0 Above average 20 43 37 0 Average 0 6 83 11 Below average 0 0 13 87

fra97851_ch08_146-164.indd 153fra97851_ch08_146-164.indd 153 12/21/10 6:46 PM12/21/10 6:46 PM

154 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

Reliability Reliability refers to the consistency of the scores obtained—how consistent they are for each individual from one administration of an instrument to another and from one set of items to another. Consider, for example, a test designed to measure typing ability. If the test is reliable, we would expect a student who receives a high score the fi rst time he takes the test to receive a high score the next time he takes the test. The scores would probably not be identical, but they should be close.

The scores obtained from an instrument can be quite reliable but not valid. Suppose a researcher gave a group of eighth-graders two forms of a test designed to measure their knowledge of the Constitution of the United States and found their scores to be consistent: those who scored high on form A also scored high on form B; those who scored low on A scored low on B; and so on. We would say that the scores were reliable. But if the researcher then used these same test scores to predict the success of these students in their physical education classes, she would probably be looked at in amazement. Any inferences about success in physical education based on scores on a Constitution test would have no validity. Now, what about the reverse? Can an instrument that yields unreliable scores permit valid in- ferences? No! If scores are completely inconsistent for a person, they provide no useful information. We have no way of knowing which score to use to infer an indi- vidual’s ability, attitude, or other characteristic.

The distinction between reliability and validity is shown in Figure 8.2 . Reliability and validity always de- pend on the context in which an instrument is used. De- pending on the context, an instrument may or may not yield reliable (consistent) scores. If the data are unreli- able, they cannot lead to valid (legitimate) inferences— as shown in target ( a ). As reliability improves, validity may improve, as shown in target ( b ), or it may not, as shown in target ( c ). An instrument may have good reli- ability but low validity, as shown in target ( d ). What is desired, of course, is both high reliability and high va- lidity, as target ( e ) shows.

ERRORS OF MEASUREMENT

Whenever people take the same test twice, they will seldom perform exactly the same—that is, their scores or answers will not usually be identical. This may be due to a variety of factors (differences in motivation,

suggesting that the scores from a particular instrument can be used to draw correct inferences about the vari- able that the instrument purports to measure. It is a broad array of evidence, rather than any one particular type of evidence, that is desired.

Consider a second example. Some evidence that might be considered to support a claim for construct validity in connection with a test designed to measure mathematical reasoning ability might be as follows:

• Independent judges all indicate that all items on the test require mathematical reasoning.

• Independent judges all indicate that the features of the test itself (such as test format, directions, scoring, and reading level) would not in any way prevent students from engaging in mathematical reasoning.

• Independent judges all indicate that the sample of tasks included in the test is relevant and representa- tive of mathematical reasoning tasks.

• A high correlation exists between scores on the test and grades in mathematics.

• High scores have been made on the test by students who have had specifi c training in mathematical reasoning.

• Students actually engage in mathematical reason- ing when they are asked to “think aloud” as they go about trying to solve the problems on the test.

• A high correlation exists between scores on the test and teacher ratings of competence in mathematical reasoning.

• Higher scores are obtained on the test by mathemat- ics majors than by general science majors.

Other types of evidence might be listed for the above task (perhaps you can think of some), but we hope this is enough to make clear that it is not just one type, but many types, of evidence that a researcher seeks to ob- tain. Determining whether the scores obtained through the use of a particular instrument measure a particular variable involves a study of how the test was developed, the theory underlying the test, how the test functions with a variety of people and in a variety of situations, and how scores on the test relate to scores on other ap- propriate instruments. Construct validation involves, then, a wide variety of procedures and many different types of evidence, including both content-related and criterion-related evidence. The more evidence research- ers have from many different sources, the more confi - dent they become about interpreting the scores obtained from a particular instrument.

fra97851_ch08_146-164.indd 154fra97851_ch08_146-164.indd 154 12/21/10 6:46 PM12/21/10 6:46 PM

C H A P T E R 8 Validity and Reliability 155

individuals on two different instruments. A reliability coeffi cient also expresses a relationship, but this time it is between scores of the same individuals on the same instrument at two different times, or on two parts of the same instrument. The three best-known ways to obtain a reliability coeffi cient are the test-retest method, the equivalent-forms method; and the internal-consistency methods. Unlike other uses of the correlation coef- fi cient, reliability coeffi cients must range from .00 to 1.00—that is, have no negative values.

TEST-RETEST METHOD

The test-retest method involves administering the same test twice to the same group after a certain time interval has elapsed. A reliability coeffi cient is then cal- culated to indicate the relationship between the two sets of scores obtained.

Reliability coeffi cients will be affected by the length of time that elapses between the two administrations of the test. The longer the time interval, the lower the reliability coeffi cient is likely to be, since there is a greater likelihood of changes in the individuals taking the test. In checking for evidence of test-retest reliability, an appropriate time interval should be selected. This interval should be that during which individuals would be assumed to retain their relative position in a meaningful group.

There is no point in studying, or even conceptual- izing, a variable that fl uctuates wildly in individuals for whom it is measured. When researchers assess someone as academically talented, for example, or skilled in typ- ing or as having a poor self-concept, they assume that this characteristic will continue to differentiate individ- uals for some period of time. It is impossible to study a variable that has no stability in the individual.

energy, anxiety, a different testing situation, and so on), and it is inevitable. Such factors result in errors of measurement ( Figure 8.3 ).

Because errors of measurement are always present to some degree, researchers expect some variation in test scores (in answers or ratings, for example) when an in- strument is administered to the same group more than once, when two different forms of an instrument are used, or even from one part of an instrument to another. Reliability estimates provide researchers with an idea of how much variation to expect. Such estimates are usu- ally expressed as another application of the correlation coeffi cient known as a reliability coefficient .

As we mentioned earlier, a validity coeffi cient ex- presses the relationship between scores of the same

Figure 8.2 Reliability and Validity

The bull’s-eye in each target represents the information that is desired. Each dot represents a separate score obtained with the instrument. A dot in the bull’s-eye indicates that the information obtained (the score) is the information the researcher desires.

(a)

So unreliable as to be invalid

(b)

Fair reliability and

fair validity

(c)

Fair reliability

but invalid

(d)

Good reliability

but invalid

(e)

Good reliability and

good validity

Figure 8.3 Reliability of a Measurement

“I can’t believe that my blood pressure

is 170 over 110!”

“Maybe it’s due to poor reliability of the measurement. Let’s wait a few minutes, and check it again.”

fra97851_ch08_146-164.indd 155fra97851_ch08_146-164.indd 155 12/21/10 6:46 PM12/21/10 6:46 PM

156 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

then calculating a correlation coeffi cient for the two sets of scores. The coeffi cient indicates the degree to which the two halves of the test provide the same re- sults and hence describes the internal consistency of the test.

The reliability coeffi cient is calculated using what is known as the Spearman-Brown prophecy formula. A simplifi ed version of this formula is as follows:

Reliability of

scores on total test

5 2 3 reliability for 1 _ 2 test ____________________ 1 1 reliability for 1 _ 2 test

Thus, if we obtained a correlation coeffi cient of .56 by comparing one half of the test items to the other half, the reliability of scores for the total test would be:

Reliability of

scores on total test

5 2 3 .56 _______ 1 1 .56

5 1.12 ____ 1.56

5 .72

This illustrates an important characteristic of reli- ability. The reliability of a test (or any instrument) can generally be increased by the addition of more items, provided they are similar to the original ones.

Kuder-Richardson Approaches. Perhaps the most frequently employed method for determining in- ternal consistency is the Kuder-Richardson approach , particularly formulas KR20 and KR21. The latter for- mula requires only three pieces of information—the number of items on the test, the mean, and the stan- dard deviation. Note, however, that formula KR21 can be used only if it can be assumed that the items are of equal diffi culty. * A frequently used version of the KR21 formula is the following:

KR21 reliability

coeffi cient

5 K ______ K 2 1

[ 1 2 M(K 2 M _________ K(SD2) ] where K 5 number of items on the test, M 5 mean of the set of test scores, and SD 5 standard deviation of the set of test scores. †

Although this formula may look somewhat intimi- dating, its use is actually quite simple. For example, if

Researchers do not expect all variables to be equally stable. Experience has shown that some abilities (such as writing) are more subject to change than others (such as abstract reasoning). Some personal character- istics (such as self-esteem) are considered to be more stable than others (such as teenage vocational inter- ests). Mood is a variable that, by defi nition, is consid- ered to be stable for short periods of time—a matter of minutes or hours. But even here, unless the instrumen- tation used is reliable, meaningful relationships with other (perhaps causal) variables will not be found. For most educational research, stability of scores over a two- to three-month period is usually viewed as suf- fi cient evidence of test-retest reliability. In reporting test-retest reliability coeffi cients, therefore, the time interval between the two testings should always be reported.

EQUIVALENT-FORMS METHOD

When the equivalent-forms method is used, two dif- ferent but equivalent (also called alternate or parallel ) forms of an instrument are administered to the same group of individuals during the same time period. Al- though the questions are different, they should sample the same content and they should be constructed sepa- rately from each other. A reliability coeffi cient is then calculated between the two sets of scores obtained. A high coeffi cient would indicate strong evidence of reliability—that the two forms are measuring the same thing.

It is possible to combine the test-retest and equivalent- forms methods by giving two different forms of the same test with a time interval between the two administrations. A high reliability coeffi cient would indicate not only that the two forms are measuring the same sort of per- formance but also what we might expect with regard to consistency over time.

INTERNAL-CONSISTENCY METHODS

The methods mentioned so far all require two ad- ministration or testing sessions. There are several internal- consistency methods of estimating reliability, however, that require only a single administration of an instrument.

Split-half Procedure. The split-half procedure involves scoring two halves (usually odd items versus even items) of a test separately for each person and

*Formula KR20 does not require the assumption that all items are of equal diffi culty, although it is harder to calculate. Computer programs for doing so are commonly available, however, and should be used whenever a researcher cannot assume that all items are of equal diffi culty. †See Chapter 10 for an explanation of standard deviation.

fra97851_ch08_146-164.indd 156fra97851_ch08_146-164.indd 156 12/21/10 6:46 PM12/21/10 6:46 PM

157

First, we can compare a given coeffi cient with the ex- tremes that are possible. As you will recall, a coeffi cient of .00 indicates a complete absence of a relationship, hence no reliability at all, whereas 1.00 is the maximum possible coeffi cient that can be obtained. Second, we can compare a given reliability coeffi cient with the sorts of coeffi cients that are usually obtained for measures of the same type. The reported reliability coeffi cients for many commercially available achievement tests, for example, are typically .90 or higher when Kuder-Richardson for- mulas are used. Many classroom tests report reliability coeffi cients of .70 and higher. Compared to these fi gures, our obtained coeffi cient must be judged rather low. For research purposes, a useful rule of thumb is that reliabil- ity should be at least .70 and preferably higher.

Alpha Coefficient. Another check on the in- ternal consistency of an instrument is to calculate an

K 5 50, M 5 40, and SD 5 4, the reliability coeffi cient would be calculated as shown below:

Reliability 5 50 ___ 49

[ 1 2 40(50 2 40) ___________ 50(42) ] 5 1.02 [ 1 2 40(10) ______ 50(16) ] 5 1.02 [ 1 2 400 ____ 800 ] 5 (1.02)(1 2 .50)

5 (1.02)(.50)

5 .51

Thus, the reliability estimate for scores on this test is .51.

Is a reliability estimate of .51 good or bad? high or low? As is frequently the case, there are some bench- marks we can use to evaluate reliability coeffi cients.

She found that teachers scoring high on control need were more likely to (1) be seen by classroom observers as imposing themselves on situations and having a higher content emphasis, (2) be judged by interviewers as having more rigid attitudes of right and wrong, and (3) score higher on a test of authoritarian tendencies.

In a study of ability to predict success in a program pre- paring teachers for inner-city classrooms, evidence was found that the Picture Situation Inventory control score had predictive value. †

Correlations existed between the control score obtained on entrance to the program and a variety of measures subsequently obtained through classroom observation in student teaching and subsequent fi rst-year teaching assignments. The most clear-cut fi nding was that those scoring higher in control need had class- rooms observed as less noisy. The fi nding adds somewhat to the validity of the measurement, since a teacher with higher control need would be expected to have a quieter room.

The reliability of both measures was found to be adequate (.74 and .81) when assessed by the split-half procedure. When assessed by follow-up over a period of eight years, the con- sistency over time was considerably lower (.61 and .53), as would be expected.

Checking Reliability and Validity—An Example

T he projective device (Picture Situation Inventory) de-scribed on pages 130 and 132 consists of 20 pictures, each scored on the variables control need and communication ac- cording to a point system. For example, here are some illustra- tive responses to picture 1 of Figure 7.23. The control need variable, defi ned as “motivated to control moment-to-moment activities of their students,” is scored as follows:

• “I thought you would enjoy something special.” (1 point) • “I’d like to see how well you can do it.” (2 points) • “You and Tom are two different children.” (3 points) • “Yes, I would appreciate it if you would fi nish it.” (4 points) • “Do it quickly please.” (5 points)

In addition to the appeal to content validity, there is some evidence in support of these two measures (control and communication).

Rowan studied relationships between the two scores and sev- eral other measures with a group of elementary school teachers. *

MORE ABOUT RESEARCH

*N. T. Rowan (1967). The relationship of teacher interaction in classroom situations to teacher personality variables. Unpublished doctoral dissertation. Salt Lake City: University of Utah.

†N. E. Wallen (1971). Evaluation report to Step-TTT Project. San Francisco, CA: San Francisco State University.

fra97851_ch08_146-164.indd 157fra97851_ch08_146-164.indd 157 12/21/10 6:46 PM12/21/10 6:46 PM

158 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

TABLE 8.2

more the longer the time between measurements. Thus, a person scoring 110 can expect to have a score between 100 and 120 one year later; fi ve years later, the score can be expected to be between 94 and 126 (see Figure 8.4 ). Note that we doubled the standard errors of measure- ment in computing the ranges within which the second score is expected to fall. This was done so we could be 95 percent sure that our estimates were correct.

alpha  coefficient (frequently called Cronbach alpha after the man who developed it). This coeffi cient (a) is a general form of the KR20 formula to be used in cal- culating the reliability of items that are not scored right versus wrong, as in some essay tests where more than one answer is possible. 3

Table 8.2 summarizes the methods used in checking the validity and reliability of an instrument.

THE STANDARD ERROR OF MEASUREMENT (SEMeas)

The standard error of measurement (SEMeas) is an index that shows the extent to which a measure- ment would vary under changed circumstances (i.e., the amount of measurement error ). Because there are many ways in which circumstances can vary, there are many possible standard errors for a given score. For ex- ample, the standard error will be smaller if it includes only error due to different content (internal-consistency or equivalent-forms reliability) than if it also includes error due to the passage of time (test-retest reliability). Under the assumption that errors of measurement are normally distributed (see p. 195 in Chapter 10), a range of scores can be determined that shows the amount of error to be expected.

For many IQ tests, the standard error of measurement over a one-year period and with different specifi c con- tent is about 5 points. Over a 10-year period, it is about 8 points. This means that a score fl uctuates considerably

Methods of Checking Validity and Reliability

Validity (“Truthfulness”)

Method Procedure

Content-related evidence Criterion-related evidence Construct-related evidence

Obtain expert judgment Relate to another measure of the same variable Assess evidence on predictions made from theory

Reliability (“Consistency”)

Method Content Time Interval Procedure

Test-retest Equivalent forms Equivalent forms/ retest Internal consistency

Scoring observer agreement

Identical Different Different

Different

Identical

Varies None Varies

None

None

Give identical instrument twice Give two forms of instrument Give two forms of instrument, with time interval between Divide instrument into halves and score each or use Kuder-Richardson approach Compare scores obtained by two or more observers or scorers

Figure 8.4 Standard Error of Measurement

120

SE = 5 points

110100

95% Interval

126

SE = 8 points

11094

95% Interval

fra97851_ch08_146-164.indd 158fra97851_ch08_146-164.indd 158 12/21/10 6:46 PM12/21/10 6:46 PM

C H A P T E R 8 Validity and Reliability 159

vulnerable to observer differences. Researchers who use such instruments are obliged to investigate and report the degree of scoring agreement . Such agreement is enhanced by training the observers and by increasing the number of observation periods.

Instruments differ in the amount of training required for their use. In general, observation techniques require considerable training for optimum use. Such training usually consists of explaining and discussing the pro- cedures involved, followed by trainees using the instru- ments as they observe videotapes or live situations. All trainees observe the same behaviors and then discuss any differences in scoring. This process, or some varia- tion thereon, is repeated until independent observers reach an acceptable level of agreement. What is desired is a correlation of at least .90 among scorers or agree- ment of at least 80 percent. Usually, even after such training, 8 to 12 observation periods are required to get evidence of adequate reliability over time.

To further illustrate the concept of reliability, let’s take an actual test and calculate the internal consis- tency of its items. Figure 8.5 presents an example of a

The formula for the standard error of measurement is SD √

______ 1 2 r

11 where SD 5 the standard deviation of

scores and r 11

5 the reliability coeffi cient appropriate to the conditions that vary. In the above example, the standard error (SEMeas) of 5 in the fi rst example was obtained as follows:

SD 5 16, r 11

5 .90

SEM 5 16 √ _______

1 2 .90 5 16 √ ___

.10 5 16(.32) 5 5.1

SCORING AGREEMENT

Most tests and many other instruments are administered with specifi c directions and are scored objectively, that is, with a key that requires no judgment on the part of the scorer. Although differences in the resulting scores with different administrators or scorers are still pos- sible, it is generally considered highly unlikely that they would occur. This is not the case with instruments that are susceptible to differences in administration, scoring, or both, such as essay evaluations. In particu- lar, instruments that use direct observation are highly

Directions: Read each of the following questions and write your answers on a separate sheet of paper. Suggested time to take the test is ten minutes.

1. There are two people in a room. The first is the son of the second person, but the second person is not the first per- son’s father. How are the two people related?

2. Who is buried in Grant’s tomb?

3. Some months have thirty days, some have thirty-one. How many have twenty-eight days?

4. If you had only one match and entered a dark room in which there was an oil lamp, an oil heater, and some fire- wood, which would you light first?

5. If a physician gave you three pills and told you to take one every half hour, how long would they last?

6. A person builds a house with four sides to it, a rectangular structure, with each side having a southern exposure. A big bear comes wandering by. What color is the bear?

7. A farmer has seventeen sheep. All but nine died. How many did he have left?

8. Divide 30 by �12�. Add 10. What is the correct answer?

9. Take two apples from three apples. What do you have?

10. How many animals of each species did Moses take aboard the Ark?

Figure 8.5 The “Quick and Easy” Intelligence Test

fra97851_ch08_146-164.indd 159fra97851_ch08_146-164.indd 159 12/21/10 6:46 PM12/21/10 6:46 PM

160

Ask fi ve other people to take the test. Record their scores on the odd and even sets of items, using the worksheet shown in Figure 8.6 .

Take a look at the scores on each of the fi ve-item sets for each of the fi ve individuals, and compare them with your own. What would you conclude about the re- liability of the scores? What would you say about any

non-typical intelligence test that we have adapted. Fol- low the directions and take the test. Then we will calcu- late the split-half reliability.

Now look at the answer key in the footnote at the bottom of page 161. Give yourself one point for each correct answer. Assume, for the moment, that a score on this test provides an indication of intelligence. If so, each item on the test should be a partial measure of in- telligence. We could, therefore, divide the 10-item test into two 5-item tests. One of these 5-item tests can con- sist of all the odd-numbered items, and the other 5-item test can consist of all the even-numbered items. Now, record your score on the odd-numbered items and also on the even-numbered items.

We now want to see if the odd-numbered items pro- vide a measure of intelligence similar to that provided by the even-numbered items. If they do, your scores on the odd-numbered items and the even-numbered items should be pretty close. If they are not, then the two 5-item tests do not give consistent results. If this is the case, then the total test (the 10 items) probably does not give consistent results either, in which case the score could not be considered a reliable measure.

Disagreement with Messick has been primarily with regard to applying his proposal. Using his experience as a developer of a widely used college admissions test battery (ACT) as an example, Reckase systematically analyzed the feasibility of using this concept. He concluded that, although diffi cult, the critical analysis of value implications is both fea- sible and useful. ‡

However, he argued that assessing the cause-and-effect re- lationships implied in determining potential and actual social consequences of the use of a test is diffi cult or impossible, even with a clear intended use such as determining college ad- missions. Citing the concern of the National Commission on Testing and Public Policy that such tests often undermine vital social policies, § he argues that obtaining the necessary data seems unlikely and that, by defi nition, appraising unintended consequences is not possible ahead of time, because one does not know what they are.

What do you think of Messick’s proposal?

CONTROVERSIES IN RESEARCH

Is Consequential Validity a Useful Concept?

I n recent years, increased attention has been given to a con-cept called consequential validity, originally proposed by Samuel Messick in 1989. * He intended not to change the core meaning of validity, but to expand it to include two new ideas: “value implications” and “social consequences.”

Paying attention to value implications requires the “ap- praisal of the value implications of the construct label, of the theory underlying test interpretation, and the ideologies in which the theory is imbedded.” † This involves expanding the idea of construct-related evidence of validity that we dis- cussed on pages 153–154. Social consequences refers to “the appraisal of both potential and actual social consequences of applied testing.”

*S. Messick (1989). Consequential validity. In R. L. Linn (Ed.). Educational measurement, 3rd ed. New York: American Council on Education, pp. 13–103. †Ibid., p. 20.

‡M. D. Reckase (1998). Consequential validity from the test devel- oper’s perspective. Educational Measurement Issues and Practice, 17 (2): 13–16. §National Commission on Testing and Public Policy. From gate- keeper to gateway: Transforming testing in America (Technical re- port). Chestnut Hill, MA: Boston College.

Figure 8.6 Reliability Worksheet

Score on five- Score on five- item test 1 item test 2

Person (#1, 3, 5, 7, 9) (#2, 4, 6, 8, 10)

You _____________ _____________

#1 _____________ _____________

#2 _____________ _____________

#3 _____________ _____________

#4 _____________ _____________

#5 _____________ _____________

fra97851_ch08_146-164.indd 160fra97851_ch08_146-164.indd 160 12/21/10 6:46 PM12/21/10 6:46 PM

C H A P T E R 8 Validity and Reliability 161

hand and no additional data collection is required. Reli- ability over time does, in most cases, require an additional administration of an instrument, but this can often be done. In considering this option, it should be noted that not all members of the sample need be retested, though this is desirable. It is better to retest a randomly selected subsample, or even a convenience subsample, than to have no evidence of retest reliability at all. Another option is to test and retest a different, though very similar, sample.

Obtaining evidence on validity is more diffi cult but sel- dom prohibitive. Content-related evidence can usually be obtained, since it requires only a few knowledgeable and available judges. It is unreasonable to expect a great deal of construct-related evidence to be obtained, but, in many studies, criterion-related evidence can be obtained. At a minimum, a second instrument should be administered. Locating or developing an additional means of instrumen- tation is sometimes diffi cult and occasionally impossible (for example, there is probably no way to validate a self- report questionnaire on sexual behavior), but the results are well worth the time and energy involved. As with re- test reliability, a subsample can be used, or both instru- ments can be given to a different, but similar, sample.

VALIDITY AND RELIABILITY IN QUALITATIVE RESEARCH

While many qualitative researchers use many of the pro- cedures we have described, some take the position that validity and reliability, as we have discussed them, are either irrelevant or not suited to their research efforts because they are attempting to describe a specifi c situa- tion or event as viewed by a particular individual. They emphasize instead the honesty, believability, expertise, and integrity of the researcher. We maintain that all re- searchers should ensure that any inferences they draw that are based on data obtained through the use of an instrument are appropriate, credible, and backed up by evidence of the sort we have described in this chapter.

Specifi c methods for enhancing the validity and reli- ability of qualitative studies are discussed in Chapters 18, 19, and 21. Moreover, in the next chapter, we dis- cuss the concept of internal validity and how it applies both to quantitative and qualitative research.

inferences about intelligence a researcher might make based on scores on this test? Could they be valid? *

Note that we have examined only one aspect of reliabil- ity (internal consistency) for results of this test. We still do not know how much a person’s score might change if we gave the test at two different times (test-retest reliability). We could get a different indication of reliability if we gave one of the fi ve-item tests at one time and the other 5-item test at another time to the same people (equivalent-forms/ retest reliability). Try to do this with a few individuals, using a worksheet like the one shown in Figure 8.6 .

Researchers typically use the procedures just de- scribed to establish reliability. Normally, however, they test many more people (at least 100). You should also realize that most tests would have many more than 10  items, since longer tests are usually more reliable than short ones, presumably because they provide a larger sampling of a person’s behavior.

In sum, we hope it is clear that a major aspect of research design is the obtaining of reliable and valid information. Because both reliability and validity depend on the way in which instruments are used and on the inferences research- ers wish to make from them, researchers can never simply assume that their instrumentation will provide satisfactory information. They can have more confi dence if they use in- struments on which there is previous evidence of reliability and validity, provided they use the instruments in the same way—that is, under the same conditions as existed previ- ously. Even then, researchers cannot be sure; even when all else remains the same, the mere passage of time may have impaired the instrument in some way.

What this means is that there is no substitute for check- ing reliability and validity as a part of the research pro- cedure. There is seldom any excuse for failing to check internal consistency, since the necessary information is at

*You might want to assess the content validity of this test. How would you defi ne intelligence? As you defi ne the term, how would you evaluate this test as a measure of intelligence? Answer Key for Q-E Intelligence Test on page 159. 1. Mother and son; 2. Ulysses S. Grant; 3. All of them; 4. The match; 5. One hour; 6. White; 7. Nine; 8. 70; 9. Two; 10. None. (It wasn’t Moses, but Noah who took the animals on the Ark.)

Go back to the INTERACTIVE AND APPLIED LEARNING feature at the beginning of the chapter for a listing of interactive and applied activities. Go to the Online Learning Center at www.mhhe.com/fraenkel8e to take quizzes, practice with key terms, and review chapter content.

fra97851_ch08_146-164.indd 161fra97851_ch08_146-164.indd 161 12/21/10 6:46 PM12/21/10 6:46 PM

162 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

VALIDITY

• It is important for researchers to use valid instruments, for the conclusions they draw are based on the information they obtain using these instruments.

• The term validity, as used in research, refers to the appropriateness, meaningful- ness, correctness, and usefulness of any inferences a researcher draws based on data obtained through the use of an instrument.

• Content-related evidence of validity refers to judgments on the content and logical structure of an instrument as it is to be used in a particular study.

• Criterion-related evidence of validity refers to the degree to which information provided by an instrument agrees with information obtained on other, independent instruments.

• A criterion is a standard for judging; with reference to validity, it is a second instru- ment against which scores on an instrument can be checked.

• Construct-related evidence of validity refers to the degree to which the totality of evidence obtained is consistent with theoretical expectations.

• A validity coeffi cient is a numerical index representing the degree of correspondence between scores on an instrument and a criterion measure.

• An expectancy table is a two-way chart used to evaluate criterion-related evidence of validity.

RELIABILITY

• The term reliability, as used in research, refers to the consistency of scores or an- swers provided by an instrument.

• Errors of measurement refer to variations in scores obtained by the same individuals on the same instrument.

• The test-retest method of estimating reliability involves administering the same in- strument twice to the same group of individuals after a certain time interval has elapsed.

• The equivalent-forms method of estimating reliability involves administering two different, but equivalent, forms of an instrument to the same group of individuals at the same time.

• The internal-consistency method of estimating reliability involves comparing re- sponses to different sets of items that are part of an instrument.

• Scoring agreement requires a demonstration that independent scorers can achieve satisfactory agreement in their scoring.

• The standard error of measurement is a numerical index of measurement error.

alpha coeffi cient 158

concurrent validity 152

construct-related evidence of validity 153

content-related evidence of validity 150

correlation coeffi cient 152

criterion 152

criterion-related evidence of validity 152

Cronbach alpha 158

equivalent-forms method 156

errors of measurement 155

expectancy table 153

internal-consistency methods 156

Kuder-Richardson approach 156

predictive validity 152

reliability 154

Main Points

Key Terms

fra97851_ch08_146-164.indd 162fra97851_ch08_146-164.indd 162 12/21/10 6:46 PM12/21/10 6:46 PM

C H A P T E R 8 Validity and Reliability 163

reliability coeffi cient 155

scoring agreement 159

split-half procedure 156

standard error of measurement (SEMeas) 158

test-retest method 155

validity 148

validity coeffi cient 152

1. We point out in the chapter that scores from an instrument may be reliable but not valid, yet not the reverse. Why would this be so?

2. What type of evidence—content-related, criterion-related, or construct-related—do you think is the easiest to obtain? the hardest? Why?

3. In what way(s) might the format of an instrument affect its validity? 4. “There is no single piece of evidence that satisfi es construct-related validity.” Is this

statement true? If so, explain why. 5. Which do you think is harder to obtain, validity or reliability? Why? 6. Might reliability ever be more important than validity? Explain. 7. How would you assess the Q-E Intelligence Test in Figure 8.4 with respect to valid-

ity? Explain. 8. The importance of using valid instruments in research cannot be overstated. Why?

1. N. E. Wallen, M. C. Durkin, J. R. Fraenkel, A. J. McNaughton, and E. I. Sawin (1969). The Taba Cur- riculum Development Project in Social Studies: Development of a comprehensive curriculum model for social studies for grades one through eight, inclusive of procedures for implementation and dissemination. Menlo Park, CA: Addison-Wesley, p. 307.

2. N. E. Gronlund (1988). How to construct achievement tests, 4th ed. Englewood Cliffs, NJ: Prentice Hall, p. 140.

3. See L. J. Cronbach (1951). Coeffi cient alpha and the internal structure of tests. Psychometrika, 16: 297–334.

For Discussion

Notes

fra97851_ch08_146-164.indd 163fra97851_ch08_146-164.indd 163 12/21/10 6:46 PM12/21/10 6:46 PM

164 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

Research Exercise 8: Validity and Reliability Use Problem Sheet 8 to describe how you plan to check on the validity and reliability of scores obtained with your instruments. If you plan to use an existing instrument, summarize what you have been able to learn about the validity and reliability of results obtained with it. If you plan to develop an instrument, explain how you will attempt to ensure validity and reliability. In either case, explain how you will obtain evidence to check validity and reliability.

Problem Sheet 8

Validity and Reliability 1. If you plan to use an existing instrument, describe what you have learned about the

validity and reliability of scores obtained with this instrument.

___________________________________________________________________

___________________________________________________________________

2. If you plan to develop an instrument, explain how you will try to ensure the validity and reliability of results obtained with this instrument by using one or more of the tips described on page 114 ( specify which ).

___________________________________________________________________

___________________________________________________________________

3. If you have not already indicated so above for each instrument that you plan to use, tell specifi cally how you will check for:

a. internal consistency

b. stability (reliability over time)

c. validity

An electronic version of this Problem Sheet that you can fi ll in and print, save, or e-mail is available on the Online Learning Center at www.mhhe.com/ fraenkel8e.

fra97851_ch08_146-164.indd 164fra97851_ch08_146-164.indd 164 12/21/10 6:46 PM12/21/10 6:46 PM

9

O B J E C T I V E S Studying this chapter should enable you to:

• Explain what is meant by the term “internal validity.”

• Explain what is meant by each of the following threats to internal validity and give an example of each:

a “subject characteristics” threat a “mortality” threat a “location” threat an “instrumentation” threat a “testing” threat

a “history” threat a “maturation” threat a “subject attitude” threat a “regression” threat an “implementation” threat

• Identify various threats to internal validity in published research articles.

• Suggest possible remedies for specifi c examples of the various threats to internal validity.

What Is Internal Validity?

Threats to Internal Validity

Subject Characteristics Loss of Subjects (Mortality) Location Instrumentation Testing History Maturation Attitude of Subjects Regression Implementation Factors That Reduce the

Likelihood of Finding a Relationship

How Can a Researcher Minimize These Threats to Internal Validity?

Two Points to Emphasize

Internal Validity

TeacherM.D.

“You think that the increase in your blood pressure is due to the new class you’ve

been assigned. Is anything else different?”

“Well, I have been working longer hours, and I’m on a new diet. Plus I

haven’t been getting much sleep lately because our

new baby is colicky.”

fra97851_ch09_165-184.indd 165fra97851_ch09_165-184.indd 165 12/21/10 6:47 PM12/21/10 6:47 PM

166

Go to the Online Learning Center at www.mhhe.com/fraenkel8e to:

• Learn More About Internal Validity

Go to your online Student Mastery Activities book to do the following activities:

• Activity 9.1: Threats to Internal Validity • Activity 9.2: What Type of Threat? • Activity 9.3: Controlling Threats to Internal Validity

INTERACTIVE AND APPLIED LEARNING After, or while, reading this chapter:

S uppose the results of a study show that high school students taught by the inquiry method score higher on a test of critical thinking, on the average, than do students taught by the lecture method. Is this difference in scores due to the difference in methods—to the fact that the two groups have been taught differently? Surely, the researcher who is conducting the study

would like to conclude this. Your fi rst inclination may be to think the same. This may not be a legitimate interpretation, however.

What if the students who were taught using the inquiry method were better critical thinkers to begin with? What if some of

the students in the inquiry group were also taking a related course during this time at a nearby university? What if the teachers of

the inquiry group were simply better teachers? Any of these (or other) factors might explain why the inquiry group scored higher

on the critical thinking test. Should this be the case, the researcher may be mistaken in concluding that there is a difference in

effectiveness between the two methods, for the obtained difference in results may be due not to the difference in methods but

to something else.

In any study that either describes or tests relationships, there is always the possibility that the relationship shown in the data is,

in fact, due to or explained by something else. If so, the relationship observed is not at all what it seems and it may lose whatever

meaning it appears to have. Many alternative hypotheses may exist, in other words, to explain the outcomes of a study. These

alternative explanations are often referred to as threats to internal validity, and they are what this chapter is about.

What Is Internal Validity? Perhaps unfortunately, the term validity is used in three different ways by researchers. In addition to internal validity, which we discuss in this chapter, you will see reference to instrument (or measurement) validity, as discussed in Chapter 8, and external (or generalization) validity, as discussed in Chapter 6.

When a study has internal validity , it means that any relationship observed between two or more variables should be unambiguous as to what it means rather than being due to “something else.” The “something else” may, as we suggested above, be any one (or more) of a number of factors, such as the age or ability of the sub- jects, the conditions under which the study is conducted, or the type of materials used. If these factors are not in some way or another controlled or accounted for, the re- searcher can never be sure that they are not the reason for any observed results. Stated differently, internal validity means that observed differences on the dependent

variable are directly related to the independent variable, and not due to some other unintended variable.

In qualitative research, a study is said to have good internal validity if alternative explanations (the “some- thing else”) have been systematically ruled out. Toward that goal, qualitative researchers should have a plan for how they treat discrepant or disconfi rming data. Regard- less of whether a study is qualitative or quantitative, if these “rival hypotheses” are not controlled or accounted for in some way, the researcher can never be sure that they are not the reason for any observed results.

Consider this example. Suppose a researcher fi nds a correlation of .80 between height and mathematics test scores for a group of elementary school students (grades 1–5)—that is, the taller students have higher math scores. Such a result is quite misleading. Why? Because it is clearly a by-product of age. Fifth-graders are taller and better in math than fi rst-graders simply because they are older and more developed. To explore this relation- ship further is pointless; to let it affect school practice would be absurd.

fra97851_ch09_165-184.indd 166fra97851_ch09_165-184.indd 166 12/21/10 6:47 PM12/21/10 6:47 PM

C H A P T E R 9 Internal Validity 167

or a subject characteristics threat. In our example of teacher expectations and class disruptive behavior, the ability level of the class fi ts this category. In studies that compare groups, subjects in the groups may differ on such variables as age, gender, ability, socioeconomic background, and the like. If not controlled, these vari- ables may explain away whatever differences between groups are found. The list of such subject characteristics is virtually unlimited, but some examples that might af- fect the results of a study include:

Or consider a study in which the researcher hypoth- esizes that, in classes for learning-disabled students, teacher expectation of student failure is related to amount of disruptive behavior. Suppose the researcher fi nds a high correlation between these two variables. Should he or she conclude that this is a meaningful re- lationship? Perhaps. But the correlation might also be explained by another variable, such as the ability level of the class (classes low in ability might be expected to have more disruptive behavior and higher teacher ex- pectation of failure). *

In our experience, a systematic consideration of possible threats to internal validity receives the least attention of all the aspects of planning a study. Often, the possibility of such threats is not discussed at all. Probably this is because their consideration is not seen as an essential step in carrying out a study. Researchers cannot avoid deciding on what variables to study, or how the sample will be obtained, or how the data will be collected and analyzed. They can, however, ignore or simply not think about possible alternative explana- tions for the outcomes of a study until after the study is completed—at which point it is almost always too late to do anything about them. Identifying possible threats during the planning stage of a study, on the other hand, can often lead researchers to design ways of eliminat- ing or at least minimizing these threats.

In recent years, many useful categories of possible threats to internal validity have been identifi ed. Although most of these categories were originally designed for application to experimental studies, some apply to other types of methodologies as well. We discuss the most im- portant of these possible threats in this chapter.

Various ways of controlling for these threats have also been identifi ed. We discuss some of these in the remain- der of this chapter and others in subsequent chapters.

Threats to Internal Validity SUBJECT CHARACTERISTICS

The selection of people for a study may result in the individuals (or groups) differing from one another in unintended ways that are related to the variables to be studied. This is sometimes referred to as selection bias,

* Can you suggest any other variables that would explain a high cor- relation (should it be found) between a teacher’s expectation of fail- ure and the amount of disruptive behavior that occurs in class?

• Age • Strength • Maturity • Gender • Ethnicity • Coordination • Speed • Intelligence

• Vocabulary • Attitude • Reading ability • Fluency • Manual dexterity • Socioeconomic status • Religious beliefs • Political beliefs

In a particular study, the researcher must decide, based on previous research or experience, which vari- ables are most likely to create problems, and do his or her best to prevent or minimize their effects. In stud- ies comparing groups, there are several methods of equating groups, which we discuss in Chapters 13 and 16. In correlational studies, there are certain statistical techniques that can be used to control such variables, provided information on each variable is obtained. We discuss these techniques in Chapter 15.

LOSS OF SUBJECTS (MORTALITY)

No matter how carefully the subjects of a study are se- lected, it is common to “lose” some as the study progresses ( Figure 9.1 ). This is known as a mortality threat . For one reason or another (for example, illness, family relocation, or the requirements of other activities), some individuals may drop out of the study. This is especially true in most intervention studies, since they take place over time.

Subjects may be absent during the collection of data or fail to complete tests, questionnaires, or other instru- ments. Failure to complete instruments is especially a problem in questionnaire studies. In such studies, it is not uncommon to fi nd that 20 percent or more of the subjects involved do not return their forms. Remember, the actual sample in a study is not the total of those se- lected but only those from whom data are obtained.

Loss of subjects, of course, not only limits generaliz- ability but also can introduce bias— if those subjects who are lost would have responded differently from those

fra97851_ch09_165-184.indd 167fra97851_ch09_165-184.indd 167 12/21/10 6:47 PM12/21/10 6:47 PM

168 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

dropped out would have. It is more likely, in fact, that they will not . Can you see why? *

It is sometimes possible for a researcher to argue that the loss of subjects in a study is not a problem. This is done by exploring the reasons for such loss and then offering an argument as to why these reasons are not rel- evant to the particular study at hand. Absence from class on the day of testing, for example, probably would not in most cases favor a particular group, since it would be incidental rather than intentional—unless the day and time of the testing was announced beforehand.

Another attempt to eliminate the problem of mor- tality is to provide evidence that the subjects lost were similar to those remaining on pertinent characteristics such as age, gender, ethnicity, pretest scores, or other variables that presumably might be related to the study outcomes. While desirable, such evidence can never demonstrate conclusively that those subjects who were lost would not have responded differently from those who remained. When all is said and done, the best solu- tion to the problem of mortality is to do one’s best to prevent or minimize the loss of subjects.

Some examples of a mortality threat include the following:

• A high school teacher decides to teach his two English classes differently. His one o’clock class spends a large amount of time writing analyses of plays, whereas his two o’clock class spends much time acting out and discussing portions of the same plays. Halfway through the semester, several students in the two o’clock class are excused to par- ticipate in the annual school play—thus they are “lost” from the study. If they, as a group, are better students than the rest of their class, their loss will lower the performance of the two o’clock class.

• A researcher wishes to study the effects of a new diet on building endurance in long-distance runners. She receives a grant to study, over a two-year period, a group of such runners who are on the track team at several nearby high schools in a large urban school district. The study is designed to compare runners who are given the new diet with similar runners in the district who are not given the diet. About 5 percent of the runners who receive the diet and about 20 percent of those who do not receive the diet, however, are

from whom data were obtained. Many times this is quite likely, since those who do not respond or who are absent probably act this way for a reason. In the example we presented earlier in which the researcher was studying the possible relationship between amount of disruptive behavior by students in class and teacher expectations of student failure, it is likely that those teachers who failed to describe their expectations to the researcher (and who would therefore be “lost” for the purposes of the study) would differ from those who did provide this informa- tion in ways affecting disruptive behavior.

In studies comparing groups, loss of subjects proba- bly will not be a problem if the loss is about the same in all groups. But if there are sizable differences between groups in terms of the numbers who drop out, this is certainly a conceivable alternative explanation for what- ever fi ndings appear. In comparing students taught by different methods (lecture versus discussion, for exam- ple), one might expect the poorer students in each group to be more likely to drop out. If more of the poorer stu- dents drop out of either group, the other method may appear more effective than it actually is.

Of all the threats to internal validity, mortality is perhaps the most diffi cult to control. A common misconception is that the threat is eliminated simply by replacing the lost subjects. No matter how this is done—even if they are replaced by new subjects selected randomly—researchers can never be sure that the replacement subjects will respond as those who

“I’m afraid my thesis is in big trouble! I lost

25 percent of my experimental group.”

“Good grief! What did you do to them?”

Figure 9.1 A Mortality Threat to Internal Validity

* Since those who drop out have done so for a reason, their replace- ments will be different at least in this respect; thus, they may see things differently or feel differently, and their responses may accord- ingly be different.

fra97851_ch09_165-184.indd 168fra97851_ch09_165-184.indd 168 12/21/10 6:47 PM12/21/10 6:47 PM

C H A P T E R 9 Internal Validity 169

seniors, and they graduate at the end of the fi rst year of the study. Because seniors are probably better run- ners, this loss will cause the remaining no-diet group to appear weaker than the diet group.

LOCATION

The particular locations in which data are collected, or in which an intervention is carried out, may cre- ate alternative explanations for results. This is called a location threat . For example, classrooms in which students are taught by, say, the inquiry method may have more resources (texts and other supplies, equip- ment, parent support, and so on) available to them than classrooms in which students are taught by the lecture method. The classrooms themselves may be larger, have better lighting, or contain better-equipped workstations. Such variables may account for higher performance by students. In our disruptive behavior versus teacher expectations example, the availability of support (resources, aides, and parent assistance) might explain the correlation between the major vari- ables of interest. Classes with fewer resources might be expected to have more disruptive behavior and higher teacher expectations of failure.

The location in which tests, interviews, or other in- struments are administered may affect responses ( Fig- ure 9.2 ). Parent assessments of their children at home may be different from assessments of their children at school. Student performance on tests may be lower if tests are given in noisy or poorly lighted rooms. Ob- servations of student interaction may be affected by the physical arrangement of certain classrooms. Such

differences might provide defensible alternative expla- nations for the results in a particular study.

The best method of control for a location threat is to hold location constant—that is, keep it the same for all participants. When this is not feasible, the researcher should try to ensure that different locations do not sys- tematically favor or jeopardize the hypothesis. This may require the collection of additional descriptions of the various locations.

Here are some examples of a location threat:

• A researcher designs a study to compare the effects of team versus individual teaching of U.S. history on student attitudes toward history. The classrooms in which students are taught by a single teacher have fewer books and materials than the ones in which students are taught by a team of three teachers.

• A researcher decides to interview counseling and spe- cial education majors to compare their attitudes toward their respective master’s degree programs. Over a three- week period, he manages to interview all of the students enrolled in the two programs. Although he is able to interview most of the students in one of the university classrooms, scheduling confl icts prevent this classroom from being available for him to interview the remainder. As a result, he interviews 20 of the counseling students in the coffee shop of the student union.

INSTRUMENTATION

The way in which instruments are used may also con- stitute a threat to the internal validity of a study. As dis- cussed in Chapter 8, scores from the instruments used in a study can lack evidence of validity. Lack of this

“What do you think?” “What do you think?”

Figure 9.2 Location Might Make a Difference

fra97851_ch09_165-184.indd 169fra97851_ch09_165-184.indd 169 12/21/10 6:47 PM12/21/10 6:47 PM

170 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

kind of validity does not necessarily present a threat to internal validity—but it may. *

Instrument Decay. Instrumentation can create problems if the nature of the instrument (including the scoring procedure) is changed in some way or another. This is usually referred to as instrument decay . This is often the case when the instrument permits different in- terpretations of results (as in essay tests) or is especially long or diffi cult to score, thereby resulting in fatigue of the scorer ( Figure 9.3 ). Fatigue often happens when a researcher scores a number of tests one after the other; he or she becomes tired and scores the tests differently (for example, more rigorously at fi rst, more generously later). The principal way to control instrument decay is to schedule data collection and/or scoring so as to minimize changes in any of the instruments or scoring procedures.

Here are some examples of instrument decay:

• A professor grades 100 essay-type fi nal examina- tions over a fi ve-hour period without taking a break. Each essay encompasses between 10 and 12 pages. He grades the papers of each class in turn and then compares the results.

• The administration of a large school district changes its method of reporting absences. Only students who are considered truant (absence is unexcused) are re- ported as absent; students who have a written excuse (from parents or school offi cials) are not reported.

The district reports a 55 percent decrease in absences since the new reporting system has been instituted.

Data Collector Characteristics. The character- istics of the data gatherers—an inevitable part of most instrumentation—can also affect results. Gender, age, ethnicity, language patterns, or other characteristics of the individuals who collect the data in a study may af- fect the nature of the data they obtain ( Figure 9.4 ). If these characteristics are related to the variables being investigated, they may offer an alternative explanation for whatever fi ndings appear. Suppose both male and female data gatherers were used in the prior example of a researcher wishing to study the relationship between disruptive behavior and teacher expectations. It might be that the female data collectors would elicit more confessions of an expectation of student failure on the part of teachers and generate more incidents of disrup- tive behavior on the part of students during classroom observations than would the males. If so, any correlation between teacher expectations of failure and the amount of disruptive behavior by students might be explained (at least partly) as an artifact of who collected the data.

The primary ways to control this threat include using the same data collector(s) throughout, analyzing data separately for each collector, and (in comparison-group studies) ensuring that each collector is used equally with all groups.

Data Collector Bias. There is also the possibil- ity that the data collector(s) and/or scorer(s) may un- consciously distort the data in such a way as to make certain outcomes (such as support for the hypothesis)

* In general, we expect lack of validity of scores to make it less likely that any relationships will be found. There are times, however, when “poor” instrumentation can increase the chances of “phony” or “spu- rious” relationships emerging.

“Boy, I’m bushed! Will I ever get finished with these? Six more

to go!”

“Yep! Bound to affect how he

grades those last few papers!”

“He’s tired.”

“Hello! I’m interested in your

attitudes toward the police. Please complete this

questionnaire.”

Figure 9.3 An Example of Instrument Decay Figure 9.4 A Data Collector Characteristics Threat

fra97851_ch09_165-184.indd 170fra97851_ch09_165-184.indd 170 12/21/10 6:47 PM12/21/10 6:47 PM

C H A P T E R 9 Internal Validity 171

more likely. Examples include some classes being al- lowed more time on tests than other classes; interview- ers asking “leading” questions of some interviewees; observer knowledge of teacher expectations affecting quantity and type of observed behaviors of a class; and judges of student essays favoring (unconsciously) one instructional method over another.

The two principal techniques for handling data col- lector bias are to standardize all procedures, which usually requires some sort of training of the data col- lectors, and to ensure that the data collectors lack the information they would need to distort results—also known as planned ignorance. Data collectors should be either unaware of the hypothesis or unable to iden- tify the particular characteristics of the individuals or groups from whom the data are being collected. Data collectors do not need to be told which method group they are observing or testing or how the individuals they are testing performed on other tests.

Some examples of data collector bias are as follows:

• All teachers in a large school district are interviewed regarding their future goals and their views on fac- ulty organizations. The hypothesis is that those plan- ning a career in administration will be more negative in their views on faculty organizations than those planning to continue teaching. Interviews are con- ducted by the vice principal in each school. Teachers are likely to be infl uenced by the fact that the person interviewing them is the vice principal, and this may account for the hypothesis being supported.

• An interviewer unconsciously smiles at certain an- swers to certain questions during an interview.

• An observer with a preference for inquiry methods observes more “attending behavior” in inquiry- identifi ed than noninquiry-identifi ed classes.

• A researcher is aware, when scoring the end-of-study examinations, which students were exposed to which treatment in an intervention study.

TESTING

In intervention studies, where data are collected over a period of time, it is common to test subjects at the begin- ning of the intervention(s). By testing, we mean the use of any form of instrumentation, not just “tests.” If substantial improvement is found in posttest (compared to pretest) scores, the researcher may conclude that this improve- ment is due to the intervention. An alternative explana- tion, however, may be that the improvement is due to the use of the pretest. Why is this? Let’s look at the reasons.

Suppose the intervention in a particular study in- volves the use of a new textbook. The researcher wants to see if students score higher on an achievement test if they are taught the subject using this new text than did students who have used the regular text in the past. The researcher pretests the students before the new text- book is introduced and then posttests them at the end of a six-week period. The students may be “alerted” to what is being studied by the questions in the pretest, however, and accordingly make a greater effort to learn the material. This increased effort on the part of the stu- dents (rather than the new textbook) could account for the improvement. It may also be that “practice” on the pretest by itself is responsible for the improvement. This is known as a testing threat ( Figure 9.5 ).

Consider another example. Suppose a counselor in a large high school is interested in fi nding out whether student attitudes toward mental health are affected by a special unit on the subject. He decides to administer an attitude questionnaire to the students before the unit is introduced and then administer it again after the unit

“How’d you do on the

final?”

“I killed it! I remembered the questions on the ‘diagnostic’ test we took at the beginning

of the semester.”

Figure 9.5 A Testing Threat to Internal Validity

fra97851_ch09_165-184.indd 171fra97851_ch09_165-184.indd 171 12/21/10 6:47 PM12/21/10 6:47 PM

172 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

is completed. Any change in attitude scores may be due to the students thinking about and discussing their opin- ions as a result of the pretest rather than as a result of the intervention.

Notice that it is not always the administration of a pretest per se that creates a possible testing effect, but rather the “interaction” that occurs between taking the test and the intervention. A pretest sometimes can make students more alert to or aware of what may be about to take place, making them more sensitive to and respon- sive toward the treatment that subsequently occurs. In some studies, the possible effects of pretesting are con- sidered so serious that such testing is eliminated.

A similar problem is created if the instrumentation pro- cess permits subjects to fi gure out the nature of the study. This is most likely to happen in single-group (correla- tional) studies of attitudes, opinions, or similar variables other than ability. Students might be asked their opinions, for example, about teachers and also about different sub- jects to test the hypothesis that student attitude toward teachers is related to student attitude toward the subjects taught. They may see a connection between the two sets of questions, especially if they are both included on the same form, and answer accordingly.

Some examples of testing threats are as follows:

• A researcher uses exactly the same set of problems to measure change over time in student ability to solve mathematics word problems. The fi rst administration of the test is given at the beginning of a unit of in- struction; the second administration is given at the

end of the unit of instruction, three weeks later. If improvement in scores occurs, it may be due to sen- sitization to the problems produced by the fi rst test and the practice effect rather than to any increase in problem-solving ability.

• A researcher incorporates items designed to measure self-esteem and achievement motivation in the same questionnaire. The respondents may fi gure out what the researcher is after and react accordingly.

• A researcher uses pre- and posttests of anxiety level to compare students given relaxation training with students in a control group. Lower scores for the re- laxation group on the posttest may be due to  the training, but they also may be due to sensitivity (created by the pretest) to the training.

HISTORY

On occasion, one or more unanticipated, and unplanned for, events may occur during the course of a study that can affect the responses of subjects ( Figure 9.6 ). Such an event is referred to in educational research as a his- tory threat . In the study we suggested of students being taught by the inquiry versus the lecture method, for ex- ample, a boring visitor who dropped in on and spoke to the lecture class just before an upcoming examination would be an example. If the visitor’s remarks in some way discouraged or turned off students in the lecture class, they might have done less well on the examination than if the visitor had not appeared. Another example involves a personal experience of one of the authors of

Figure 9.6 A History Threat to Internal Validity

“I don’t see how we’re supposed to study with that construction noise outside

all the time!”

fra97851_ch09_165-184.indd 172fra97851_ch09_165-184.indd 172 12/21/10 6:47 PM12/21/10 6:47 PM

C H A P T E R 9 Internal Validity 173

this text. He remembers clearly the day that President John F. Kennedy died, since he had scheduled an ex- amination for that very day. The author’s students at that time, stunned into shock by the announcement of the president’s death, were unable to take the examination. Any comparison of examination results taken on this day with the examination results of other classes taken on other days would have been meaningless.

Researchers can never be certain that one group has not had experiences that differ from those of other groups. As a result, they should continually be alert to any such infl uences that may occur (in schools, for ex- ample) during the course of a study. As you will see in Chapter 13, some research designs handle this threat better than do others.

Two examples of a history threat follow.

• A researcher designs a study to investigate the effects of simulation games on ethnocentrism. She plans to select two high schools to participate in an experi- ment. Students in both schools will be given a pretest designed to measure their attitudes toward minority groups. School A will then be given the simulation games during their social studies classes over a three- day period, and school B will watch travel fi lms. Both schools will then be given the same test to see if their attitude toward minority groups has changed. The researcher conducts the study as planned, but a special documentary on racial prejudice is shown in school A between the pretest and the posttest.

• The achievement scores of fi ve elementary schools whose teachers use a cooperative learning approach are compared with those of fi ve schools whose teach- ers do not use this approach. During the course of the study, the faculty of one of the schools where coop- erative learning is not used is engaged in a disruptive confl ict with the school principal.

MATURATION

Often, change during an intervention may be due to factors associated with the passing of time rather than to the intervention itself ( Figure 9.7 ). This is known as a maturation threat . Over the course of a semester, for example, very young students, in particular, will change in many ways simply because of aging and ex- perience. Suppose, for example, that a researcher is interested in studying the effect of special grasping exercises on the ability of 2-year-olds to manipulate various objects. She fi nds that such exercises are as- sociated with marked increases in the manipulative ability of the children over a six-month period. Two- year-olds mature very rapidly, however, and the im- provement in their manipulative ability may be due simply to this fact rather than the grasping exercises. Maturation is a serious threat only in studies using pre-post data for the intervention group, or in studies that span a number of years. The best way to control for maturation is to include a well-selected compari- son group in the study.

Figure 9.7 Could Maturation Be at Work Here?

“I’m much more confident after a year in that self-esteem class.”

“That’s great!”

fra97851_ch09_165-184.indd 173fra97851_ch09_165-184.indd 173 12/21/10 6:47 PM12/21/10 6:47 PM

174 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

Examples of a maturation threat are as follows:

• A researcher reports that students in liberal arts col- leges become less accepting of authority between their freshman and senior years and attributes this to the many “liberating” experiences they have under- gone in college. This may be the reason, but it also may be because they simply have grown older.

• A researcher tests a group of students enrolled in a special class for “students with artistic potential” every year for six years, beginning when they are age 5. She fi nds that their drawing ability improves markedly over the years.

ATTITUDE OF SUBJECTS

How subjects view a study and participate in it can also threaten internal validity. One example is the well-known Hawthorne effect , fi rst observed in the Hawthorne plant of the Western Electric Company some years ago. 1 It was accidentally discovered that productivity increased not only when improvements were made in physical working conditions (such as an increase in the number of coffee breaks and bet- ter lighting) but also when such conditions were unintentionally made worse (for instance, the num- ber of coffee breaks was reduced and the lighting was dimmed). The usual explanation for this is that

the special attention and recognition received by the workers were responsible; they felt someone cared about them and was trying to help them. This positive effect, resulting from increased attention and recogni- tion of subjects, has subsequently been referred to as the Hawthorne effect.

It has also been suggested that recipients of an ex- perimental treatment may perform better because of the novelty of the treatment rather than the specifi c nature of the treatment. It might be expected, then, that subjects who know they are part of a study may show improve- ment as a result of a feeling that they are receiving some sort of special treatment—no matter what this treatment may be ( Figure 9.8 ).

An opposite effect can occur whenever, in interven- tion studies, the members of the control group receive no treatment at all. As a result, they may become de- moralized or resentful and hence perform more poorly than the treatment group. It may thus appear that the experimental group is performing better as a result of the treatment, when this is not the case.

One remedy for these subject attitude threats is to provide the control or comparison group(s) with a special or novel treatment comparable to that received by the experimental group. While simple in theory, this is not easy to do in most educational settings. Another possibility, in some cases, is to make it easy for stu- dents to believe that the treatment is just a regular part

“OK, Juan and Alicia, I know you’re going to enjoy this. You’re

going to be a part of a very important study with a professor from the

university. Won’t that be exciting?”

Figure 9.8 The Attitude of Subjects Can Make a Difference

fra97851_ch09_165-184.indd 174fra97851_ch09_165-184.indd 174 12/21/10 6:47 PM12/21/10 6:47 PM

of instruction—that is, not part of an experiment. For example, it is sometimes unnecessary to announce that an experiment is being conducted.

Here are examples of a subject attitude threat:

• A researcher decides to investigate the possible re- duction of test anxiety by playing classical music during examinations. She randomly selects 10 fresh- man algebra classes from the fi ve high schools in a large urban school district. In fi ve of these classes, she plays classical music softly in the background during examinations. In the other fi ve (the control group), she plays no music. The students in the con- trol group, however, learn that music is being played in the other classes and express some resentment when their teachers tell them that the music cannot be played in their class. This resentment may actu- ally cause them to be more anxious during exams or intentionally to infl ate their anxiety scores.

• A researcher hypothesizes that critical thinking skill is correlated with attention to detail. He administers a somewhat novel test that provides a separate score for each variable (critical thinking and attention to detail) to a sample of eighth-graders. The novelty of the test may confuse some students, while others may think it is silly. In either case, the scores of these stu- dents are likely to be lower on both variables because of the format of the test, not because of any lack of ability. It may appear, therefore, that the hypothesis is supported. Neither score is a valid indicator of ability for such students, so this particular attitudinal reaction creates a threat to internal validity.

REGRESSION

A regression threat may be present whenever change is studied in a group that is extremely low or high in its pre- intervention performance ( Figure 9.9 ). Studies in special education are particularly vulnerable to this threat, since the students in such studies are frequently selected on the basis of previous low performance. The regression phe- nomenon can be explained statistically, but for our pur- poses it simply describes the fact that a group selected because of unusually low (or high) performance will, on the average, score closer to the mean on subsequent testing, regardless of what transpires in the meantime. Thus, a class of students of markedly low ability may be expected to score higher on posttests regardless of the effect of any intervention to which they are exposed. Like maturation, the use of an equivalent control or com- parison group handles this threat—and this seems to be understood as refl ected in published research.

Some examples of a possible regression threat are as follows:

• An Olympic track coach selects the members of her team from those who have the fastest times during the fi nal trials for various events. She fi nds that their average time increases the next time they run, how- ever, which she perhaps erroneously attributes to poorer track conditions.

• Those students who score in the lowest 20 percent on a math test are given special help. Six months later their average score on a test involving similar prob- lems has improved, but not necessarily because of the special help.

• Minority students are less academically able than students from the dominant culture. (probable subject characteris- tics, subject attitude, location, instrumentation, and history threats)

• People on welfare are lazy. (probable subject characteris- tics, location, and history threats)

• Schooling makes students rebellious. (probable maturation and history threats)

• A policy of expelling students who don’t “behave” im- proves a school’s test scores. (probable mortality threat)

• Indoctrination changes attitude. (probable testing threat) • So-called miracle drugs cure intellectual retardation.

(probable regression threat) • Smoking marijuana leads eventually to using cocaine and

heroin. (probable mortality threat)

Threats to Internal Validity in Everyday Life

C onsider the following commonly held beliefs: • Because failure often precedes suicide, it is therefore the

cause of suicide. (probable history and mortality threats) • Boys are genetically more talented in mathematics than are

girls. (probable subject attitude and history threats) • Girls are genetically more talented in language than are

boys. (probable history and subject attitude threats)

MORE ABOUT RESEARCH

175

fra97851_ch09_165-184.indd 175fra97851_ch09_165-184.indd 175 12/21/10 6:47 PM12/21/10 6:47 PM

176 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

IMPLEMENTATION

The treatment or method in any experimental study must be administered by someone—the researcher, the teach- ers involved in the study, a counselor, or some other per- son. This fact raises the possibility that the experimental group may be treated in ways that are unintended and not necessarily part of the method, yet which give them an advantage of one sort or another. This is known as an implementation threat . It can happen in either of two ways.

First, an implementation threat can occur when dif- ferent individuals are assigned to implement different methods, and these individuals differ in ways related to the outcome. Consider our previous example in which two groups of students are taught by either an inquiry or a lecture method. The inquiry teachers may simply be better teachers than the lecture teachers.

There are a number of ways to control for this pos- sibility. The researcher can attempt to evaluate the individuals who implement each method on pertinent characteristics (such as teaching ability) and then try to equate the treatment groups on these dimensions (for example, by assigning teachers of equivalent abil- ity to each group). Clearly, this is a diffi cult and time- consuming task. Another control is to require that each method be taught by all teachers in the study. Where

feasible, this is a preferable solution, though it also is vulnerable to the possibility that some teachers may have different abilities to implement the different meth- ods. Still another control is to use several different in- dividuals to implement each method, thereby reducing the chances of an advantage to either method.

Second, an implementation threat can occur when some individuals have a personal bias in favor of one method over the other. Their preference for the method, rather than the method itself, may account for the su- perior performance of students taught by that method. This is a good reason why a researcher should, if at all possible, not be one of the individuals who imple- ments a method in an intervention study. It is sometimes possible to keep individuals who are implementers ig- norant of the nature of a study, but it is generally very diffi cult—in part because teachers or others involved in a study will usually need to be given a rationale for their participation. One solution for this is to allow individu- als to choose the method they wish to implement, but this creates the possibility of differences in characteris- tics discussed above. An alternative is to have all meth- ods used by all implementers, but with their preferences known beforehand. Note that preference for a method as a result of using it does not constitute a threat—it is simply one of the by-products of the method itself.

Figure 9.9 Regression Rears Its Head

“Yeah, no doubt. A likely example of a regression

effect, eh?”

“These guys were all selected because they’ve run the mile under 3:58. But you know, I bet that their average time will be slower than

3:58 when they run in tomorrow’s race!”

fra97851_ch09_165-184.indd 176fra97851_ch09_165-184.indd 176 12/21/10 6:47 PM12/21/10 6:47 PM

This is also true of other by-products. If teacher skill or parent involvement, for example, improves as a result of the method, it would not constitute a threat. Finally, the researcher can observe in an attempt to see that the methods are administered as intended.

Examples of an implementation threat are as follows:

• A researcher is interested in studying the effects of a new diet on the physical agility of young children. After obtaining the permission of the parents of the children to be involved, all of whom are fi rst-graders, he randomly assigns the children to an experimental group and a control group. The experimental group is to try the new diet for three months, and the control group is to stay with its regular diet. The researcher overlooks the fact, however, that the teacher of the experimental group is an accomplished instructor of

some fi ve years’ experience, while the instructor of the control group is a fi rst-year teacher, newly appointed.

• A group of clients who stutter is given a relatively new method of therapy called generalization train- ing. Both client and therapist interact with people in the “real world” as part of the therapy. After six months of receiving therapy, the fl uency of these clients is compared with that of a group receiving traditional in-the-offi ce therapy. Speech therapists who use new methods are likely to be more generally competent than those working with the comparison group. If so, greater improvement for the generaliza- tion group may be due not to the new method but rather to the skill of the therapist.

Figure 9.10 illustrates, and Table 9.1 briefl y summa- rizes, each of the threats we have discussed.

As we have mentioned, meta-analysis is a way of quan- tifying replications of a study. It is important to note, how- ever, that the term replication is used rather loosely in this context, since the studies that the researcher(s) has collected may have little in common except that they all have the same independent variable. Our concerns are twofold: Merely ob- taining several studies, even if they all have the same inde- pendent variable, does not mean that they will necessarily balance out each other’s weaknesses—they might all have the same weakness. Secondly, in doing a meta-analysis, equal weight is given to both good and bad studies—that is, no distinction is made between studies that have been well designed and conducted and those that have not been so well designed and/or conducted. Results of a well-designed study in which the researchers used a large random sample, for example, would count the same as results from a poorly controlled study in which researchers used a convenience or purposive sample.

A partial solution to these problems that we support is to combine meta-analysis with judgmental review. This has been done by judging studies as good or bad and comparing the results; sometimes they agree. If, however, there is a suffi cient number of good studies (we would argue for a minimum of seven), we see little to be gained by including poor ones.

Meta-analyses are here to stay, and there is little question that they can provide the research community with valuable information. But we do not think excessive enthusiasm for the technique is warranted. Like many things, it is a tool, not a panacea.

Some Thoughts About Meta-Analysis

A s we mentioned in Chapter 3, the main argument in favor of doing a meta-analysis is that the weaknesses in indi- vidual studies should balance out or be reduced by combining the results of a series of studies. In short, researchers who do a meta-analysis attempt to remedy the shortcomings of any particular study by statistically combining the results of sev- eral (hopefully many) studies that were conducted on the same topic. Thus, the threats to internal validity that we discussed in this chapter should be reduced and generalizability should be enhanced.

How is this done? Essentially by calculating what is called effect size (see Chapter 12). Researchers conducting a meta- analysis do their best to locate all of the studies on a particular topic (i.e., all of the studies having the same independent vari- able). Once located, effect sizes and an overall average effect size for each dependent variable are calculated. * As an ex- ample, Vockell and Asher report an average delta (Δ) of .80 on the effectiveness of cooperative learning. †

MORE ABOUT RESEARCH

177

* This is not always easy to do. Frequently, published reports lack the necessary information, although it can sometimes be deduced from what is reported. † E. L. Vockell and J. W. Asher (1995). Educational research, 2nd ed. Englewood Cliffs, NJ: Prentice Hall, p. 361.

fra97851_ch09_165-184.indd 177fra97851_ch09_165-184.indd 177 12/21/10 6:47 PM12/21/10 6:47 PM

178 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

FACTORS THAT REDUCE THE LIKELIHOOD OF FINDING A RELATIONSHIP

In many studies, the various factors we have discussed could also serve to reduce, or even prevent, the chances of a relationship being found. For example, if the

methods (the treatment) in a study are not adequately implemented—that is, adequately tried—the effect of actual differences between them on outcomes may be obscured. Similarly, if the members of a control or comparison group become “aware” of the experimental

The teachers in this fictitious example are discussing the results

of a study which shows that students who attend private high schools had higher achievement (as shown by test scores) than students who attended

public high schools.

(Instrumentation) (Testing) (History)

(Maturation) (Attitude of subjects)

(Regression) (Implementation)

“Hold on — perhaps private schools are more likely to expel the poorer students. So it’s the policy,

not the nature of the school, that makes the difference.”

“Maybe those attending private schools have

higher academic aptitude — so it is not the type of school that

makes the difference.”

“Wait a minute. Private schools may have more

resources (materials, technologies, parent support); that could account

for the differences instead of the type of school

organization.”

(Subject characteristics)

(Loss of subjects)

(Location)

“Private school students may achieve higher scores, not because

of the type of school, but because they are exposed to a broader range of experiences.

Their parents are more affluent.”

“Maybe private school students have more opportunities to practice taking their tests. This could

account for their higher performance.”

“Is it likely that the tests to assess achievement are

biased in favor of the curricula found in private schools? Could the procedure

used in testing favor the private school students (testing conditions, adherence to instructions)?”

“Perhaps it is the status and self-esteem associated with attending a

private school that motivates these students to achieve at a higher level, rather than the type

of school organization.”

“Maybe there were a lot of students who scored really low on the pretest in the private schools.”

“Maybe private schools have more experienced or dedicated teachers and this is

the reason for a difference.”† “Perhaps private

school students spend more years in high school than those

in public schools.”*

Figure 9.10 Illustration of Threats to Internal Validity Note: We are not implying that any of these statements are necessarily true; our guess is that some are and some are not.

* This seems unlikely.

†If these teacher characteristics are a result of the type of school, then they do not constitute a threat.

fra97851_ch09_165-184.indd 178fra97851_ch09_165-184.indd 178 12/21/10 6:47 PM12/21/10 6:47 PM

C H A P T E R 9 Internal Validity 179

treatment, they may increase their efforts because they feel “left out,” thereby reducing real differences in achievement between treatment groups that otherwise would be seen. Sometimes, teachers of a control group may unwittingly give some sort of “compensation” to motivate the members of their group, thereby lessening the impact of the experimental treatment. Finally, the use of instruments that produce unreliable scores and/or the use of small samples may result in a reduced likeli- hood of a relationship or relationships being observed.

How Can a Researcher Minimize These Threats to Internal Validity? Throughout this chapter, we have suggested a number of techniques or procedures that researchers can employ to control or minimize the possible effects of threats to

internal validity. Essentially, they boil down to four al- ternatives. A researcher can try to do any or all of the following.

1. Standardize the conditions under which the study occurs—such as the way(s) in which the treatment is implemented (in intervention studies), the way(s) in which the data are collected, and so on. This helps control for location, instrumentation, subject atti- tude, and implementation threats.

2. Obtain more information on the subjects of the study—that is, on relevant characteristics of the subjects—and use that information in analyzing and interpreting results. This helps control for a sub- ject characteristics threat and (possibly) a mortality threat, as well as maturation and regression threats.

3. Obtain more information on the details of the study—that is, where and when it takes place, ex- traneous events that occur, and so on. This helps control for location, instrumentation, history, subject attitude, and implementation threats.

TABLE 9.1 Threats to the Internal Validity of a Study

Threat Defi nition

Subject Characteristics The selection of people for a study may result in the individuals or groups differing from one another in unintended ways that are related to the variables being studied. Also called “selection bias.”

Mortality The loss of subjects in a study due to attrition, withdrawal, or low participation rates may introduce bias and affect the outcome of a study.

Location The particular locations in which data are collected, or in which an intervention is carried out, may create alternative explanations for results.

Instrumentation The ways in which instruments are used may constitute an internal validity threat. Possible instrumentation threats include changes in the instrument and how it is scored, characteristics of the data collector, and/or bias on the part of the data collector.

Testing The use of a pretest in intervention studies may create a “practice effect” that can affect the results of a study and/or how participants respond to an intervention.

History A history threat is when an unforeseen or unplanned event occurs during the course of a study.

Maturation Change during an intervention may be due sometimes to factors associated with the passing of time rather than the intervention.

Subject Attitude The way subjects view a study and their participation in it can be considered a threat to internal validity; the positive impact of an intervention is known as the “Hawthorne effect.”

Regression A regression threat is possible when change is studied in a group with extreme low or high performances as determined by a pretest. On average, the group will score closer to the mean on subsequent testing regardless of the treatment or intervention.

Implementation The experimental group may be treated in unintended ways that give them an undue advantage affecting results.

fra97851_ch09_165-184.indd 179fra97851_ch09_165-184.indd 179 12/21/10 6:47 PM12/21/10 6:47 PM

180 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

4. Choose an appropriate design. The proper design can do much to control these threats to internal validity.

Because control by design applies primarily to ex- perimental and causal-comparative studies, we shall discuss it in detail in Chapters 13 and 16. The four alter- natives are summarized in Table 9.2 .

TWO POINTS TO EMPHASIZE

We want to end this chapter by emphasizing two things. First, these various threats to internal validity can be greatly reduced by planning. Second, such planning often requires collecting additional information before a study begins (or while it is taking place). It is often too late to consider how to control these threats once the data have been collected.

TABLE 9.2 General Techniques for Controlling Threats to Internal Validity

Technique

Threat Standardize Conditions

Obtain More Information on

Subjects

Obtain More Information on

Details

Choose an Appropriate

Design

Subject characteristics X X

Mortality X X

Location X X X

Instrumentation X X

Testing X

History X X

Maturation X X

Subject attitude X X X

Regression X X

Implementation X X X

Go back to the INTERACTIVE AND APPLIED LEARNING feature at the beginning of the chapter for a listing of interactive and applied activities. Go to the Online Learning Center at www.mhhe.com/fraenkel8e to take quizzes, practice with key terms, and review chapter content.

THE MEANING OF INTERNAL VALIDITY

• When a study lacks internal validity, one or more alternative hypotheses exist to explain the outcomes. These alternative hypotheses are referred to by researchers as threats to internal validity.

• When a study has internal validity, it means that any relationship observed between two or more variables is unambiguous, rather than being due to something else.

THREATS TO INTERNAL VALIDITY

• Some of the more common threats to internal validity are differences in subject char- acteristics, mortality, location, instrumentation, testing, history, maturation, attitude of subjects, regression, and implementation.

Main Points

fra97851_ch09_165-184.indd 180fra97851_ch09_165-184.indd 180 12/21/10 6:47 PM12/21/10 6:47 PM

C H A P T E R 9 Internal Validity 181

• The selection of people for a study may result in the individuals or groups differing (i.e., the characteristics of the subjects may differ) from one another in unintended ways that are related to the variables to be studied.

• No matter how carefully the subjects of a study (the sample) are selected, it is com- mon to lose some of them as the study progresses. This is known as mortality. Such a loss of subjects may affect the outcomes of a study.

• The particular locations in which data are collected, or in which an intervention is carried out, may create alternative explanations for any results that are obtained.

• The way in which instruments are used may also constitute a threat to the internal va- lidity of a study. Possible instrumentation threats include changes in the instrument, characteristics of the data collector(s), and/or bias on the part of the data collectors.

• The use of a pretest in intervention studies sometimes may create a “practice effect” that can affect the results of a study. A pretest can also sometimes affect the way subjects respond to an intervention.

• On occasion, one or more unanticipated and unplanned for events may occur during the course of a study that can affect the responses of subjects. This is known as a history threat.

• Sometimes change during an intervention study may be due more to factors asso- ciated with the passing of time than to the intervention itself. This is known as a maturation threat.

• The attitude of subjects toward a study (and their participation in it) can create a threat to internal validity. This is known as subject attitude threat .

• When subjects are given increased attention and recognition because they are partici- pating in a study, their responses may be affected. This is known as the Hawthorne effect.

• Whenever a group is selected because of unusually high or low performance on a pretest, it will, on average, score closer to the mean on subsequent testing, regardless of what transpires in the meantime. This is called a regression threat.

• Whenever an experimental group is treated in ways that are unintended and not a necessary part of the method being studied, an implementation threat can occur.

CONTROLLING THREATS TO INTERNAL VALIDITY

• Researchers can use a number of techniques or procedures to control or minimize threats to internal validity. Essentially they boil down to four alternatives: (1) stan- dardizing the conditions under which the study occurs, (2) obtaining and using more information on the subjects of the study, (3) obtaining and using more information on the details of the study, and (4) choosing an appropriate design.

Key Terms data collector bias 171 Hawthorne effect 174

history threat 172

implementation threat 176

instrument decay 170

internal validity 166

location threat 169

maturation threat 173

mortality threat 167

regression threat 175

subject attitude threat 174

subject characteristics threat 167

testing threat 171

threats to internal validity 167

fra97851_ch09_165-184.indd 181fra97851_ch09_165-184.indd 181 12/21/10 6:47 PM12/21/10 6:47 PM

182 P A R T 2 The Basics of Educational Research www.mhhe.com/fraenkel8e

1. Can a researcher prove conclusively that a study has internal validity? Explain. 2. In Chapter 6, we discussed the concept of external validity. In what ways, if any,

are internal and external validity related? Can a study have internal validity but not external validity? If so, how? What about the reverse?

3. Students often confuse the concept of internal validity with the idea of instrument validity. How would you explain the difference between the two?

4. What threat (or threats) to internal validity might exist in each of the following? a. A researcher decides to try out a new mathematics curriculum in a nearby ele-

mentary school and to compare student achievement in math with that of students in another elementary school using the regular curriculum. The researcher is not aware, however, that the students in the new-curriculum school have computers to use in their classrooms.

b. A researcher wishes to compare two different kinds of textbooks in two high school chemistry classes over a semester. She fi nds that 20 percent of one group and 10 percent of the other group are absent during the administration of unit tests.

c. In a study investigating the possible relationship between marital status and per- ceived social changes during the last fi ve years, men and women interviewers get different reactions from female respondents to the same questions.

d. Teachers of an experimental English curriculum as well as teachers of the regular curriculum administer both pre- and posttests to their own students.

e. Eighth-grade students who volunteer to tutor third-graders in reading show greater improvement in their own reading scores than a comparison group that does not participate in tutoring.

f. A researcher compares the effects of weekly individual and group counseling on the improvement of study habits. Each week the students counseled as a group fi ll out questionnaires on their progress at the end of their meetings. The students counseled individually, however, fi ll out the questionnaires at home.

g. Those students who score in the bottom 10 percent academically in a school in an economically depressed area are selected for a special program of enrichment. The program includes special games, extra and specially colored materials, spe- cial snacks, and new books. The students score substantially higher on achieve- ment tests six months after the program is instituted.

h. A group of elderly people are asked to fi ll out a questionnaire designed to investi- gate the possible relationship between activity level and sense of life satisfaction.

5. How could you determine whether the threats you identifi ed in each of the situations in question 4 actually exist?

6. Which threats discussed in this chapter do you think are the most important for a researcher to consider? Why? Which do you think would be the most diffi cult to control? Explain.

For Discussion

Note 1. F. J. Roethlisberger and W. J. Dickson (1939). Management and the worker. Cambridge, MA: Harvard University Press.

fra97851_ch09_165-184.indd 182fra97851_ch09_165-184.indd 182 12/21/10 6:47 PM12/21/10 6:47 PM

C H A P T E R 9 Internal Validity 183

Research Exercise 9: Internal Validity State the question or hypothesis of your study at the top of Problem Sheet 9. In the spaces indi- cated, place an X after each of the threats to internal validity that apply to your study, explain why they are threats, and describe how you intend to control for those most likely to occur (i.e., prevent them from affecting the outcome of your study). Finally, what can you say to convince others that the results of your study are credible and not due merely to coincidence or chance?

Problem Sheet 9

Internal Validity 1. Place an X after any of the threats listed below that you think might apply to your

study:

Subject characteristics _____   Instrumentation _____   Maturation _____

Mortality _____   Testing _____ History _____ Subject attitude _____

Implementation _____ Location _____ Regression _____ Other

2. Please describe how you will attempt to control for those threats that you have marked above:

Threat #1: ___________________________________________________________

Threat #2: ___________________________________________________________

Threat #3: ___________________________________________________________

Threat #4: ___________________________________________________________

3. What assurances can you provide (through your design, sampling procedure, etc.) to support the claims that your study fi ndings are valid? In other words, how will you convince the reader that the fi ndings or relationships resulting from the study are not due to or explained by something other than what you claim?

An electronic version of this Problem Sheet that you can fi ll in and print, save, or e-mail is available on the Online Learning Center at www.mhhe.com/ fraenkel8e.

fra97851_ch09_165-184.indd 183fra97851_ch09_165-184.indd 183 12/21/10 6:47 PM12/21/10 6:47 PM

This page intentionally left blank

Part 3 introduces the subject of statistics—important tools that are frequently used by

researchers in the analysis of their data. Chapter 10 presents a discussion of descrip-

tive statistics and provides a number of techniques for summarizing both quantitative

and categorical data. Chapter 11 deals with inferential statistics—how to determine

whether an outcome can be generalized or not—and briefl y discusses the most com-

monly used inferential statistics. Chapter 12 then places what we have presented in

the previous two chapters in perspective. We provide some examples of comparing

groups and of relating variables within a group. We conclude the chapter with a sum-

mary of recommendations for using both descriptive and inferential statistics.

Data Analysis

3 P A R T

fra97851_ch10_185-218.indd 185fra97851_ch10_185-218.indd 185 22/12/10 9:46 PM22/12/10 9:46 PM

10

OBJECTIVES Reading this chapter should enable you to:

• Explain the difference between a statistic and a parameter.

• Differentiate between categorical and quantitative data and give an example of each.

• Name three different types of scores used in educational research and give an example of each.

• Construct a frequency polygon from data. • Construct a histogram and a stem-leaf plot

from data. • Explain what is meant by the terms

“normal distribution” and “normal curve.” • Calculate the mean, median, and mode

for a frequency distribution of data. • Calculate the range and standard deviation

for a frequency distribution of data.

• Explain what a fi ve-number summary is. • Explain what a boxplot displays. • Explain how any particular score in a

normal distribution can be interpreted in standard deviation units.

• Explain what a “z score” is and tell why it is advantageous to be able to express scores in z score terms.

• Explain how to interpret a normal distribution.

• Construct and interpret a scatterplot. • Explain more fully what a correlation

coeffi cient is. • Calculate a Pearson correlation coeffi cient. • Prepare and interpret a frequency table,

a bar graph, and a pie chart. • Prepare and interpret a crossbreak table.

Statistics Versus Parameters

Two Fundamental Types of Numerical Data

Quantitative Data Categorical Data

Types of Scores

Raw Scores Derived Scores Which Scores to Use?

Techniques for Summarizing Quantitative Data

Frequency Polygons Skewed Polygons Histograms and Stem-Leaf

Plots The Normal Curve Averages Spreads Standard Scores and the

Normal Curve Correlation

Techniques for Summarizing Categorical Data

The Frequency Table Bar Graphs and Pie Charts The Crossbreak Table

Descriptive Statistics

A Normal Distribution?

fra97851_ch10_185-218.indd 186fra97851_ch10_185-218.indd 186 22/12/10 9:46 PM22/12/10 9:46 PM

187

Two Fundamental Types of Numerical Data In Chapter 7, we presented a number of instruments used in educational research. The researcher’s inten- tion in using these instruments is to collect information of some sort—measures of abilities, attitudes, beliefs, reactions, and so forth—that will enable him or her to draw some conclusions about the sample of individuals being studied.

As we have seen, such information can be collected in several ways, but it can be reported in only three ways: through words, through numbers, and sometimes through graphs or charts that show patterns or describe relationships. In certain types of research, such as inter- views, ethnographic studies, or case studies, researchers often try to describe their fi ndings through a narrative description of some sort. Their intent is not to reduce

Statistics Versus Parameters The major advantage of descriptive statistics is that they permit researchers to describe the information contained in many, many scores with just a few indi- ces, such as the mean and median (more about these in a moment). When such indices are calculated for a sample drawn from a population, they are called statistics ; when they are calculated from the entire population, they are called parameters . Because most educational research involves data from samples rather than from populations, we refer mainly to sta- tistics in the remainder of this chapter. We present the most commonly used techniques for summariz- ing such data. Some form of summary is essential to interpret data collected on any variable—a long list of scores or categorical representations is simply unmanageable.

Go to the Online Learning Center at www.mhhe.com/fraenkel8e to:

• Review the Sample Statistics • Learn More About Techniques for Summarizing

Quantitative and Categorical Data

Go to your online Student Mastery Activities book to do the following activities:

• Activity 10.1: Construct a Frequency Polygon • Activity 10.2: Compare Frequency Polygons • Activity 10.3: Calculate Averages • Activity 10.4: Calculate the Standard Deviation • Activity 10.5: Calculate a Correlation Coeffi cient • Activity 10.6: Analyze Crossbreak Tables • Activity 10.7: Compare z Scores • Activity 10.8: Prepare a Five-Number Summary • Activity 10.9: Summarize Salaries • Activity 10.10: Compare Scores • Activity 10.11: Custodial Times • Activity 10.12: Collect Data

INTERACTIVE AND APPLIED LEARNING After, or while, reading this chapter:

T im Williams, Juan Martinez, and Julia McNaughton have just come from their 9:00 A.M. statistics class. They are discussing some ideas over coffee in the student union. Tim: “I just read in today’s paper that the average salary per year for secondary school teachers is $52,000.”

Juan: “Really? Dr. Wallen said its about $43,000.”

Julia: “Well, one of them must be wrong!”

Tim: “Not necessarily, Julia. There are averages and then there are averages . . .”

Yes, Tim is correct. We’ll explain how, and why, later in the chapter, as well as a lot more. Read on.

fra97851_ch10_185-218.indd 187fra97851_ch10_185-218.indd 187 22/12/10 9:46 PM22/12/10 9:46 PM

188 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

• SAT scores (the variable is scholastic aptitude ) • The temperatures recorded each day during the

months of September through December in Omaha, Nebraska, in a given year (the variable is temperature )

• The anxiety scores of all fi rst-year students enrolled at San Francisco State University in 2002 (the vari- able is anxiety )

CATEGORICAL DATA

Categorical data simply indicate the total number of ob- jects, individuals, or events a researcher fi nds in a particu- lar category. Thus, a researcher who reports the number of people for or against a particular government policy, or the number of students completing a program in successive years, is reporting categorical data. Notice that what the researcher is looking for is the frequency of certain char- acteristics, objects, individuals, or events. Many times it is useful, however, to convert these frequencies into percent- ages. Some examples of categorical data follow.

• The representation of each ethnic group in a school (the variable is ethnicity ); for example, Caucasian, 1,462 (41 percent); black, 853 (24 percent); Hispanic, 760 (21 percent); Asian, 530 (15 percent)

• The number of male and female students in a chem- istry class (the variable is gender )

• The number of teachers in a large school district who use (1) the lecture and (2) the discussion method (the variable is teaching method )

• The number of each type of tool found in a work- room (the variable is type of tool )

the information to numerical form but to present it in a descriptive form, and often as richly as possible. We give some examples of this method of reporting infor- mation in Chapters 19, 20, and 21. In this chapter, how- ever, we concentrate on numerical ways of reporting information.

Much of the information reported in educational re- search consists of numbers of some sort—test scores, per- centages, grade point averages, ratings, frequencies, and the like. The reason is an obvious one—numbers are a useful way to simplify information. Numerical informa- tion, usually referred to as data, can be classifi ed in one of two basic ways: as either categorical or quantitative data.

Just as there are categorical and quantitative vari- ables (see Chapter 5), there are two types of numerical data. Categorical data differ in kind, but not in degree or amount. Quantitative data, on the other hand, differ in degree or amount.

QUANTITATIVE DATA

Quantitative data are obtained when the variable being studied is measured along a scale that indicates how much of the variable is present. Quantitative data are reported in terms of scores. Higher scores indicate that more of the variable (such as weight, academic ability, self-esteem, or interest in mathematics) is present than do lower scores. Some examples of quantitative data follow.

• The amount of money spent on sports equipment by various schools in a particular district in a semester (the variable is amount of money spent on sports equipment )

Microsoft Offi ce Excel, more commonly known as Excel, is a software program that can be used to analyze and manipulate data using tables and formulas. We do not have the space to present a complete explanation of how to use the program, but in this and the next chapter, we provide brief, step-by-step instructions on how to use Excel to calculate many of the descriptive statistics in the text including means, standard deviations, and correlations. Excel can also be used to conduct many hypothesis tests, including independent and repeated-measures t-tests, analysis of variance (ANOVA), and chi-square tests.

In Appendix D, we present a more detailed explanation of how to use Excel (for PC) that illustrates many of the steps we describe, including instructions about how to load the analysis ToolPak. (The ToolPak must be loaded in order to perform statistical functions in EXCEL). Note that Macintosh users should consult the Microsoft Web site (www.microsoft.com) for help on how to use EXCEL for Mac computers as the screen interface differs.

Using Excel to Perform Statistical Calculations

fra97851_ch10_185-218.indd 188fra97851_ch10_185-218.indd 188 22/12/10 9:46 PM22/12/10 9:46 PM

C H A P T E R 1 0 Descriptive Statistics 189

• The number of each kind of merchandise found in a large department store (the variable is type of merchandise )

You may fi nd it helpful at this point to refer back to Figure 7.26 in Chapter 7. The ordinal, interval, and ratio scales all pertain to quantitative data; the nominal scale pertains to categorical data.

Types of Scores Quantitative data are usually reported in the form of scores. Scores can be reported in many ways, but an important distinction to understand is the difference between raw scores and derived scores.

RAW SCORES

Almost all measurement begins with what is called a raw score, which is the initial score obtained. It may be the total number of items an individual gets cor- rect or answers in a certain way on a test, the num- ber of times a certain behavior is tallied, the rating given by a teacher, and so forth. Examples include the number of questions answered correctly on a science test, the number of questions answered “positively” on an attitude scale, the number of times “aggressive” behavior is observed, a teacher’s rating on a “self- esteem” measure, or the number of choices received on a sociogram.

Taken by itself, an individual raw score is diffi cult to interpret, since it has little meaning. What, for ex- ample, does it mean to say a student received a score of 62 on a test if that is all the information you have? Even if you know that there were 100 questions on the test, you don’t know whether 62 is an extremely high (or extremely low) score, since the test may have been easy or diffi cult.

We often want to know how one individual’s raw score compares to those of other individuals taking the same test, and (perhaps) how he or she has scored on similar tests taken at other times. This is true when- ever we want to interpret an individual score. Because raw scores by themselves are diffi cult to interpret, they often are converted to what are called derived scores.

DERIVED SCORES

Derived scores are obtained by taking raw scores and converting them into more useful scores on some type

of standardized basis. They indicate where a particular individual’s raw score falls in relation to all other raw scores in the same distribution. They enable a researcher to say how well the individual has performed compared to all others taking the same test. Examples of derived scores are age- and grade-level equivalents, percentile ranks, and standard scores.

Age and Grade-level Equivalents. Age- equivalent scores and grade-equivalent scores tell us of what age or grade an individual score is typical. Sup- pose, for example, that the average score on a begin- ning-of-the-year arithmetic test for all eighth-graders in a certain state is 62 out of a possible 100. Students who score 62 will have a grade equivalent of 8.0 on the test regardless of their actual grade placement— whether in sixth, seventh, eighth, ninth, or tenth grade, the student’s performance is typical of beginning eighth-graders. Similarly, a student who is 10 years and 6 months old may have an age-equivalent score of 12-2, meaning that his or her test performance is typical of students who are 12 years and 2 months old.

Percentile Ranks. A percentile rank refers to the percentage of individuals scoring at or below a given raw score. Percentile ranks are sometimes referred to as percentiles, although this term is not quite correct as a synonym. *

Percentile ranks are easy to calculate. A simple for- mula for converting raw scores to percentile ranks (PRs) is as follows:

PR 5 number of students below score

1 all students all score

____________________________

total number in group 3 100

Suppose a total of 100 students took an examination, and 18 of them received a raw score above 85, while two students received a score of 85. Eighty students, then, scored somewhere below 85. What is the percentile rank of the two students who received the score of 85? Using the formula

PR 5 80 1 2 ______ 100

3 100 5 82

the percentile rank of these two students is 82.

*A percentile is the point below which a certain percentage of scores fall. The 70th percentile, for example, is the point below which 70 percent of the scores in a distribution fall. The 99th percentile is the point below which 99 percent of the scores fall, and so forth. Thus, if 20 percent of the students in a sample score below 40 on a test, then the 20th percentile is a score of 40. A person who obtains a score of 40 has a percentile rank of 20.

fra97851_ch10_185-218.indd 189fra97851_ch10_185-218.indd 189 22/12/10 9:46 PM22/12/10 9:46 PM

190 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

Often percentile ranks are calculated for each of the scores in a group. Table 10.1 presents a group of scores with the percentile rank of each score indicated.

Standard Scores. Standard scores provide an- other means of indicating how one individual com- pares to other individuals in a group. Standard scores indicate how far a given raw score is from a refer- ence point. They are particularly helpful in compar- ing an individual’s relative achievement on different types of instruments (such as comparing a person’s performance on a chemistry achievement test with an instructor’s rating of his work in a laboratory). Many different systems of standard scores exist, but the two most commonly used and reported in educational re- search are z scores and T scores. We shall discuss both types later in the chapter.

WHICH SCORES TO USE?

Given these various types of scores, how do research- ers decide which to use? Recall that the usefulness of derived scores is primarily in making individual raw scores meaningful to students, parents, teachers, and others. Despite their value in this respect, some derived scores should not be used in research. This is the case if the researcher is assuming an interval scale, as is often done. Percentile ranks, for example, should never be used because they, almost certainly, do not constitute an interval scale. Age- and grade-equivalent

scores likewise have serious limitations because of the way they are obtained. Usually the best scores to use are standard scores, which are sometimes pro- vided in instrument manuals and, if not, can easily be calculated. If standard scores are not used, it is far preferable to use raw scores—converting derived scores, for example, back to the original raw scores, if necessary—rather than use percentile ranks or age/ grade equivalents.

Techniques for Summarizing Quantitative Data Note: None of the techniques in this section for sum- marizing quantitative data is appropriate for categorical data; they are for use only with quantitative data.

FREQUENCY POLYGONS

Listed below are the scores of a group of 50 students on a midsemester biology test.

64, 27, 61, 56, 52, 51, 3, 15, 6, 34, 6, 17, 27, 17, 24, 64, 31, 29, 31, 29, 31, 29, 29, 31, 31, 29, 61, 59, 56, 34, 59, 51, 38, 38, 38, 38, 34, 36, 36, 34, 34, 36, 21, 21, 24, 25, 27, 27, 27, 63

How many students received a score of 34? Did most of the students receive a score above 50? How many re- ceived a score below 30? As you can see, when the data are simply listed in no apparent order, as they are here, it is diffi cult to tell.

To make any sense out of these data, we must put the information into some sort of order. One of the most common ways to do this is to prepare a frequency distribution . This is done by listing the scores in rank order from high to low, with tallies to indicate the num- ber of subjects receiving each score ( Table 10.2 ). Often, the scores in a distribution are grouped into intervals. This results in a grouped frequency distribution , as shown in Table 10.3 .

Although frequency distributions like the ones in Tables 10.2 and 10.3 can be quite informative, often the information they contain is hard to visualize. To fur- ther the understanding and interpretation of quantita- tive data, it is helpful to present it in a graph. One such graphical display is known as a frequency polygon . Figure 10.2 presents a frequency polygon of the data in Table 10.3

TABLE 10.1 Hypothetical Examples of Raw Scores and Accompanying Percentile Ranks

Raw Score

Cumulative Frequency

Percentile Frequency Rank

95 1 25 100 93 1 24 96 88 2 23 92 85 3 21 84 79 1 18 72 75 4 17 68 70 6 13 52 65 2 7 28 62 1 5 20 58 1 4 16 54 2 3 12 50 1 1 4

N 5 25

fra97851_ch10_185-218.indd 190fra97851_ch10_185-218.indd 190 22/12/10 9:46 PM22/12/10 9:46 PM

C H A P T E R 1 0 Descriptive Statistics 191

TABLE 10.2 Example of a Frequency Distribution a

Raw Score Frequency

64 2 63 1 61 2 59 2 56 2 52 1 51 2 38 4 36 3 34 5 31 5 29 5 27 5 25 1 24 2 21 2 17 2 15 1 6 2 3 1

n 5 50 a Technically, the table should include all scores, including those for which there are zero frequencies. We have eliminated those to simplify the presentation.

TABLE 10.3 Example of a Grouped Frequency Distribution

Raw Scores (Intervals of Five) Frequency

60–64 5 55–59 4 50–54 3 45–49 0 40–44 0 35–39 7 30–34 10 25–29 11 20–24 4 15–19 3 10–14 0 5–9 2 0–4 1

n 5 50

* Grouping scores into intervals such as fi ve or more is often necessary when there are many scores in the distribution. Generally, 12 to 15 intervals on the x -axis is recommended. † A common mistake of students is to treat the vertical axis as if the numbers represented specifi c individuals. They do not. They repre- sent frequencies. Each number on the vertical axis is used to plot the number of individuals at each score. In Figure 10.1 , the dot above the interval 25–29 shows that 11 persons scored somewhere within the interval 25–29.

The steps involved in constructing a frequency poly- gon are as follows:

1. List all scores in order of size, and tally how many students receive each score. Group scores, if neces- sary, into intervals .*

2. Label the horizontal axis by placing all of the pos- sible scores (or groupings) on that axis, at equal in- tervals, starting with the lowest score on the left.

3. Label the vertical axis by indicating frequencies, at equal intervals, starting with zero.

4. For each score (or grouping of scores), fi nd the point where it intersects with its frequency of occurrence, and place a dot at that point. Remember that each score (or grouping of scores) with zero frequency must still be plotted.

5. Connect all the dots with a straight line.

As you can see by looking at Figure 10.1 , the fact that a large number of the students scored in the middle of this distribution is illustrated quite nicely.†

SKEWED POLYGONS

Data can be distributed in almost any shape. If a re- searcher obtains a set of data in which many indi- viduals received low scores, for example, the shape of the distribution would look something like the fre- quency polygon shown in Figure 10.2 . As you can see, in this particular distribution, only a few individ- uals received the higher scores. The frequency poly- gon in Figure 10.2 is said to be positively skewed because the tail of the distribution trails off to the right, in the direction of the higher (more positive ) score values. Suppose the reverse were true. Imag- ine that a researcher obtained a set of data in which few individuals received relatively low scores. Then the shape of the distribution would look like the fre- quency polygon in Figure 10.3 . This polygon is said to be negatively skewed, since the longer tail of the distribution goes off to the left.

fra97851_ch10_185-218.indd 191fra97851_ch10_185-218.indd 191 22/12/10 9:46 PM22/12/10 9:46 PM

192 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

Frequency polygons are particularly useful in com- paring two (or sometimes more) groups. In Chapter 7, Table 7.2, we presented some hypothetical results of a study involving a comparison of two counseling meth- ods. Figure 10.4 on page 193 shows the polygons con- structed using the data from Table 7.2.

This fi gure reveals several important fi ndings. First, it is evident that method B resulted in higher scores,

overall, than did method A. Second, it is clear that the scores for method B are more spread out. Third, it is clear that the reason for method B being higher overall is not that there are fewer scores at the low end of the scale (although this might have happened, it did not). In fact, the groups are almost identical in the number of scores below 61: A510, B512. The reason method B is higher overall is that there were fewer cases in the

Figure 10.1 Example of a Frequency Polygon 14

15

13

11 12

10 9 8 7

Score

6 5 4 3 2 1 0

0– 4

Fr eq

ue nc

y

5– 9

10 –1

4

15 –1

9

20 –2

4

25 –2

9

30 –3

4

35 –3

9

40 –4

4

45 –4

9

50 –5

4

55 –5

9

60 –6

4

65 –6

9

Using Excel to Construct Frequency Distributions and Histograms or Bar Graphs

In Excel, you can use the Histogram Data Analysis tool to create a frequency distribution table and draw an accompanying histogram or bar graph. First, to construct a frequency distribution table using Excel, choose the Tools Data Analysis command, select Histogram from the Analysis Tools list and click OK. In the Histogram dialog box that appears, use the input range box to identify the data you want to use to construct a frequency distribution and histogram. If your data ranges already include labels, check the Labels box. Next, use the Output Options button to tell Excel where to place you frequency distribution and histogram. If you want to put them in a new workbook, select the New Workbook button. To customize your histogram, make choices using Output Options. For example, checking the Cumulative Percentage creates a line showing cumulative percentages in your histogram. Finally, select the Chart Output check to tell Excel to include a histogram with the frequency distribution, and click OK. (Remember that the Analysis ToolPak must be installed fi rst before any statistical operations can be per- formed. See Appendix D.)

fra97851_ch10_185-218.indd 192fra97851_ch10_185-218.indd 192 22/12/10 9:46 PM22/12/10 9:46 PM

C H A P T E R 1 0 Descriptive Statistics 193

13 12 11 10 9 8 7 6 5 4 3 2 1 0

Score

Fr eq

ue nc

y

200 21 22 23 24 25 26 27 28 29 30 31 32 33 34

Figure 10.2 Example of a Positively Skewed Polygon

13 12 11 10 9 8 7 6 5 4 3 2 1 0

Score

Fr eq

ue nc

y

200 21 22 23 24 25 26 27 28 29 30 31 32 33 34

Figure 10.3 Example of a Negatively Skewed Polygon

9 8 7 6 5 4

2 3

Method A

1 0

Score

Fr eq

ue nc

y

0

36 –4

0

41 –4

5

46 –5

0

51 –5

5

56 –6

0

61 –6

5

66 –7

0

71 –7

5

76 –8

0

81 –8

5

86 –9

0

91 –9

5

96 –1

00

Method B

Figure 10.4 Two Frequency Polygons Compared

fra97851_ch10_185-218.indd 193fra97851_ch10_185-218.indd 193 22/12/10 9:46 PM22/12/10 9:46 PM

194 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

middle range of the scores (between 60 and 75), and more cases above 75. If this is not clear to you, study the shaded areas in the fi gure. Many times we want to know not only which group is higher overall but also where the differences are. In this example we see that method B resulted in more variability and that it re- sulted in a substantial number of scores higher than those in method A.

HISTOGRAMS AND STEM-LEAF PLOTS

A histogram is a bar graph used to display quantita- tive data at the interval or ratio level of measurement. The bars are arranged in order from left to right on the horizontal axis, and the widths of the bars indicate the range of values that fall within each bar. Frequencies are shown on the vertical axis, and the point of intersection of the two axes is always zero. Furthermore, the bars in a histogram (in contrast to those in a bar graph) touch, indicating that they illustrate quantitative rather than

categorical data. Figure 10.5 is a histogram of the data presented in the grouped frequency distribution shown in Table 10.3 .

A stem-leaf plot is a display that organizes a set of data to show both its shape and its distribution. Each data value is split into a “stem” and a “leaf.” The leaf is usually the last digit of the number, and the other digits to the left of the leaf form the stem. For example, the number 149 would be split

stem 14 leaf 9

Let us construct a stem-leaf plot for the following set of scores on a math quiz: 29, 37, 32, 46, 45, 45, 54, 51, 55, 55, 55, 60. First, separate each number into a stem and a leaf. Since these are two-digit numbers, the tens digit is the stem and the units digit is the leaf. Next, group the numbers with the same stems as shown below, listing them in numerical order:

0– 4

5– 9

10 –1

4

15 –1

9

20 –2

4

25 –2

9

30 –3

4

35 –3

9

40 –4

4

45 –4

9

50 –5

4

55 –5

9

60 –6

4

Scores

4

5

6

7

8

9

10

11

12

3

2

1

0

Fr eq

u en

cy

Figure 10.5 Histogram of Data in Table 10.2

fra97851_ch10_185-218.indd 194fra97851_ch10_185-218.indd 194 22/12/10 9:46 PM22/12/10 9:46 PM

C H A P T E R 1 0 Descriptive Statistics 195

Math Quiz Scores Stem Leaf

2 9 3 72 4 655 5 41555 6 0

Finally, reorder the leaf values in sequence:

Math Quiz Scores

Stem Leaf

2 9 3 27 4 556 5 14555 6 0

One advantage that a stem-leaf plot has over a histo- gram is that it shows not only the frequency of values within each interval but also reveals all of the individual values within each interval.

Stem-leaf plots are particularly helpful in compar- ing and contrasting two distributions. For example, fol- lowing is a back-to-back stem-leaf plot comparing the home runs hit by Babe Ruth during his years with the New York Yankees with those hit by Mark McGuire dur- ing his years with the St. Louis Cardinals.

Babe Ruth Mark McGuire

0 9,9 1

5,2 2 2 5,4 3 2,3,9,9

9,7,6,6,6,1,1 4 2,9 9,4,4 5 2,8

0 6 5 7 0

Who do you think was the better home run hitter? How would Barry Bonds of the San Francisco Giants, who hit 73 home runs in 2003, compare overall?

THE NORMAL CURVE

Often researchers draw a smooth curve instead of the series of straight lines in a frequency polygon. The smooth curve suggests that we are not just connecting

a series of dots (that is, the actual frequencies of scores in a particular distribution), but rather showing a gen- eralized distribution of scores that is not limited to one specifi c set of data. These smooth curves are known as distribution curves .

Many distributions of data tend to follow a certain specifi c shape of distribution curve called a normal dis- tribution . When a distribution curve is normal, the large majority of the scores are concentrated in the middle of the distribution, and the scores decrease in frequency the farther away from the middle they are, as shown in Figure 10.6 .

The normal curve is based on a precise mathemati- cal equation. As you can see, it is symmetrical and bell shaped. The distribution of some human characteristics, such as height and weight, approximate such a curve, while many others, such as spatial ability, manual dex- terity, and creativity, are often assumed to do so. The normal curve is very useful to researchers, and we shall discuss it in more detail later in the chapter.

AVERAGES

Averages , or measures of central tendency , enable a researcher to summarize the data in a frequency dis- tribution with a single number. The three most com- monly used averages are the mode, the median, and the mean. Each represents a type of average or typi- cal score attained by a group of individuals on some measure.

The Mode. The mode is the most frequent score in a distribution—that is, the score attained by more stu- dents than any other score. In the following distribution, what is the mode?

25, 20, 19, 17, 16, 16, 16, 14, 14, 11, 10, 9, 9

Figure 10.6 The Normal Curve

fra97851_ch10_185-218.indd 195fra97851_ch10_185-218.indd 195 22/12/10 9:46 PM22/12/10 9:46 PM

196 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

The mode is 16. What about this distribution?

25, 24, 24, 23, 22, 20, 19, 19, 18, 11, 10

This distribution (called a bimodal distribution ) has two modes, 24 and 19. Because the mode really doesn’t tell us very much about a distribution, however, it is not used very often in educational research.

The Median. The median is the point below and above which 50 percent of the scores in a distribution fall—in short, the midpoint. In a distribution that con- tains an uneven number of scores, the median is the middlemost score (provided that the scores are listed in order). Thus, in the distribution 5, 4, 3, 2, 1, the median is 3. In a distribution that contains an even number of scores, the median is the point halfway between the two middlemost scores. Thus, in the distribution 70, 74, 82, 86, 88, 90, the median is 84. Hence, the median is not necessarily one of the actual scores in the distribution being summarized.

Note that two very different distributions might have the same median, as shown below:

Distribution A: 98, 90, 84, 82, 76

Distribution B: 90, 87, 84, 65, 41

In both distributions, the median is 84. It may look like the median is fairly easy to deter-

mine. This is usually the case with ungrouped data. For grouped data, calculating the median requires somewhat more work. It can, however, be estimated by locating the score that has half of the area under the frequency polygon above it and half below it.

The median is the most appropriate average to calcu- late when the data result in skewed distributions.

The Mean. The mean is another average of all the scores in a distribution. * It is determined by adding up all of the scores and then dividing this sum by the total number of scores. The mean of a distribution contain- ing scores of 52, 68, 74, 86, 95, and 105, therefore, is 80. How did we determine this? We simply added up all the scores, which came to 480, and then divided this sum by 6, the total number of scores. In symbolic form, the formula for computing the mean looks like this:

__

X 5 SX ___ n

where S represents “sum of,” X represents any raw score value, n represents the total number of scores, and __

X represents the mean. Table 10.4 presents a frequency distribution of scores

on a test and each of the above measures of central ten- dency. As you can see, each of these indices tells us something a little different. The most frequent score was 62, but would we want to say that this was the most typical score? Probably not. The median of the scores was 64.5. The mean was 66.7. Perhaps the mean is the best description of the distribution of scores, but it, too, is not totally satisfactory because the distribution is skewed. Table 10.4 shows that these indices are only summaries of all the scores in a distribution and often do not have the same value. They are not intended to show variation or spread ( Figure 10.7 ).

Which of the three averages (measures of central ten- dency), then, is best? It depends. The mean is the only one of the three that uses all the information in a distri- bution, since every score is used in calculating it, and it is generally preferred over the other two measures. However, it tends to be unduly infl uenced by extreme scores. (Can you see why?) On occasion, therefore, the median gives a more accurate indication of the typical

*Actually, there are several kinds of means (geometric, harmonic, etc.), but their use is specialized and infrequent. We refer here to the arithmetic mean.

TABLE 10.4 Example of the Mode, Median, and Mean in a Distribution

Raw Score Frequency

98 1 97 1 91 2 85 1 80 5 77 7 72 5 65 3 64 7 62 10 58 3 45 2 33 1 11 1 5 1

n 5 50

Mode 5 62; median 5 64.5; mean 5 66.7

fra97851_ch10_185-218.indd 196fra97851_ch10_185-218.indd 196 22/12/10 9:46 PM22/12/10 9:46 PM

C H A P T E R 1 0 Descriptive Statistics 197

score in a distribution. Suppose, for example, that the yearly salaries earned by various workers in a small business were as shown in Table 10.5 .

The mean of these salaries is $75,000. Would it be correct to say that this is the average yearly salary paid in this company? Obviously it would not. The extremely high salary paid to the owner of the company “infl ates” the mean, so to speak. Using it as a summary fi gure to indicate the average yearly salary would give an errone- ous impression. In this instance, the median would be the more appropriate average to calculate, since it would not be as affected by the owner’s salary. The median is

$27,000, a far more accurate indication of the typical salary for the year.

SPREADS

While measures of central tendency are useful statis- tics for summarizing the scores in a distribution, they are not suffi cient. Two distributions may have identi- cal means and medians, for example, yet be quite dif- ferent in other ways. For example, consider these two distributions:

Distribution A: 19, 20, 25, 32, 39

Distribution B: 2, 3, 25, 30, 75

The mean in both of these distributions is 27, and the median in both is 25. Yet you can see that the distribu- tions differ considerably. In distribution A, the scores are closer together and tend to cluster around the mean. In distribution B, they are much more spread out. Hence the two distributions differ in what statisticians call variability . Figure 10.8 illustrates further examples.

Thus, measures of central tendency, when presented without any accompanying information of how spread out or dispersed the data are, can be misleading. To say that the average annual income of all the players in the National Basketball Association in 1998 was $275,000.00 hides the fact that some players earned far less, whereas someone like Michael Jordan earned more

TABLE 10.5 Yearly Salaries of Workers in a Small Business

Mr. Davis $ 10,500 Mr. Thompson 20,000 Ms. Angelo 22,500 Mr. Schmidt 24,000 Ms. Wills 26,000 Ms. Brown 28,000 Mr. Greene 36,000 Mr. Adams 43,000 Ms. Franklin 65,000 Mr. Payson (owner) 475,000

Figure 10.7 Averages Can Be Misleading!

3 FEET 3 FEET

9 FEET 9 FEET

“Poor Tony. He calculated the average

depth only!”

“Yep.”

fra97851_ch10_185-218.indd 197fra97851_ch10_185-218.indd 197 22/12/10 9:46 PM22/12/10 9:46 PM

198 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

than 5 million. The distribution of players’ salaries was skewed to the right and very spread out. Knowing only the mean gives us an inadequate description of the dis- tribution of salaries for players in the NBA.

There is a need, therefore, for measures research- ers can use to describe the spread, or variability, that exists within a distribution. Let us consider three—the interquartile range, the overall range, and the standard deviation.

Quartiles and the Five-Number Summary. When a distribution is skewed, both the variability and the general shape of the distribution can be described by reporting several percentiles. A percentile in a set of numbers is a value below which a certain percent- age of the numbers fall and above which the rest of the numbers fall.

You may have encountered percentiles if you have ever taken a standardized test such as the SAT and re- ceived a report saying “Raw score 630, percentile 84.” You received a score of 630, but perhaps more useful is the fact that 84 percent of those who took the examina- tion scored lower than you did.

The median is the 50th percentile. Other percentiles that are important are the 25th percentile, also known as the fi rst quartile (Q

1 ), and the 75th percentile, the third

quartile (Q 3 ). A useful way to describe a skewed dis-

tribution, therefore, is to give what is known as a five- number summary , consisting of the lowest score, Q

1 ,

the median, Q 3 , and the highest score. The interquartile

range (IQR) is the difference between the third and fi rst quartiles (Q

3 − Q

1 5 IQR).

Boxplots. The fi ve-number summary of a distribu- tion can be graphically portrayed by means of a box- plot . Boxplots are especially useful in comparing two or more distributions. Figure 10.9 gives boxplots for the distributions of the midterm scores of two classes taking the same biology exam. Each central box has its ends at the quartiles, and the median is marked by the line

within the box. The “whiskers” at either end extend to the lowest and highest scores. *

Figure 10.9 permits an immediate comparison be- tween the two classes. Overall, class B did better, but the upper whiskers illustrate that class A had the student with the highest score. Figure 10.9 is but another exam- ple of how effectively graphs can convey information.

Though the fi ve-number summary is an extremely useful numerical description of a distribution, it is not the most common. That accolade belongs to a combina- tion of the mean (a measure of center) and the standard deviation (a measure of spread). The standard devia- tion and its brother, the variance, measure the spread of scores from the mean. They should only be used in conjunction with the mean.

The Range. The overall range represents the dis- tance between the highest and lowest scores in a distri- bution. Thus, if the highest score in a distribution is 89 and the lowest is 11, the range would be 89–11, or 78. Because it involves only the two most extreme scores in a distribution, the range is but a crude indication of variability. Its main advantage is that it gives a quick (although rough) estimate of variability.

The Standard Deviation. The standard de- viation (SD) is the most useful index of variability. It is a single number that represents the spread of a

Figure 10.8 Different Distributions Compared with Respect to Averages and Spreads

Mean

Same spread, different average

Same average, different spread

MeanMean

*Boxplots are sometimes called box-and-whiskers diagrams.

Figure 10.9 Boxplots

45 50 55 60 65

Class A

70 75 80 85

Class B

fra97851_ch10_185-218.indd 198fra97851_ch10_185-218.indd 198 22/12/10 9:46 PM22/12/10 9:46 PM

C H A P T E R 1 0 Descriptive Statistics 199

distribution. As with the mean, every score in the dis- tribution is used to calculate it. The steps involved in calculating the standard deviation are straightforward.

1. Calculate the mean of the distribution.

__

X 5 SX ___ n

2. Subtract the mean from each score. Each result is symbolized X 2

__ X .

3. Square each of these scores ( X − __

X ) 2 . 4. Add up all the squares of these scores:

S(X 2 __

X )2 .

5. Divide the total by the number of scores. The result is called the variance .

6. Take the square root of the variance. This is the stan- dard deviation.

The above steps can be summarized as follows:

SD 5 √ _________

S(X 2

__ X )2 _________ n

where SD is the symbol for standard deviation, S is the symbol for “sum of,” X is the symbol for a raw score,

__ X

is the symbol for the mean, and n represents the number of scores in the distribution.

This procedure sounds more complicated than it is. It really is not diffi cult to calculate. Table 10.6 illustrates

the calculation of the standard deviation of a distribu- tion of 10 scores.

You will notice that the more spread out scores are, the greater the deviation scores will be and hence the larger the standard deviation. The closer the scores are to the mean, the less spread out they are and hence the smaller the standard deviation. Thus, if we were describ- ing two sets of scores on the same test and we stated that the standard deviation of the scores in set 1 was 2.7 and the standard deviation in set 2 was 8.3, we would know that there was much less variability in set 1—that is, the scores were closer together.

An important point involving the standard deviation is that if a distribution is normal, then the mean plus or minus three standard deviations will encompass about 99 percent of all the scores in the distribution. For exam- ple, if the mean of a distribution is 72 and the standard deviation is 3, then just about 99 percent of the scores in the distribution are somewhere between scores of 63 and 81. Figure 10.10 provides an illustration of standard deviation.

TABLE 10.6 Calculation of the Standard Deviation of a Distribution

Raw Score Mean X −

__ X (X −

__ X ) 2

85 54 31 961 80 54 26 676 70 54 16 256 60 54 6 36 55 54 1 1 50 54 24 16 45 54 29 81 40 54 214 196 30 54 224 576 25 54 229 841

S 5 3640

Variance (SD 2 ) 5 S(X 2 X)2

_________ n

5 3640 _____ 10 5 364 a

Standard deviation (SD) 5 √ _________

S(X 2

__ X )2 _________ n

5 √ ____

364 5 19.08b

a The symbol for the variance of a sample sometimes is shown as s 2 ; the symbol for the variance of a population is s 2 .

b The symbol for the standard deviation of a sample sometimes is shown as s; the symbol for the standard deviation of a population is s.

“I don’t think I can make it to class today, Dr. Roberts. I’m feeling

a couple of standard deviations below the mean.”

fra97851_ch10_185-218.indd 199fra97851_ch10_185-218.indd 199 22/12/10 9:46 PM22/12/10 9:46 PM

200 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

The Standard Deviation of a Normal Distri- bution. The total area under the normal curve repre- sents all of the scores in a normal distribution. In such a curve, the mean, median, and mode are identical, so the mean falls at the exact center of the curve. It thus is also the most frequent score in the distribution. Because the curve is symmetrical, 50 percent of the scores must fall on each side of the mean.

Here are some important facts about the normal distribution:

• Fifty percent of all the observations (e.g., scores) fall on each side of the mean ( Figure 10.11 ).

• In any normal distribution, 68 percent of the scores fall within one standard deviation of the mean. Half of these (34 percent) fall within one standard devia- tion above the mean and the other half within one standard deviation below the mean.

• Another 27 percent of the observations fall between one and two standard deviations away from the mean. Hence 95 percent (68 percent plus 27 percent) fall within two standard deviations of the mean.

• In all, 99.7 percent of the observations fall within three standard deviations of the mean. Figure 10.12 illustrates all three of these facts, often referred to as the 68-95-99.7 rule.

Figure 10.10 Standard Deviations for Boys’ and Men’s Basketball Teams

Boys’

basketball team SD = 5.25

Men’s

basketball team SD = 2.45

54" 57" 61" 64" 69" 72" 73" 75" 76" 79"

Figure 10.11 Fifty Percent of All Scores in a Normal Curve Fall on Each Side of the Mean

0–3 –2 –1

50% 50%

31 2

Figure 10.12 Percentages under the Normal Curve

–3 SD –2 SD –1 SD 1 SD 2 SD 3 SDMean

34%

13.5%

2.15% 0.13%

34%

13.5%

2.15% 0.13%

99.7%

95%

68%

fra97851_ch10_185-218.indd 200fra97851_ch10_185-218.indd 200 22/12/10 9:46 PM22/12/10 9:46 PM

C H A P T E R 1 0 Descriptive Statistics 201

Hence we see that almost all of the scores in a normal distribution lie between the mean and plus or minus three standard deviations. Only .13 percent of all the scores fall above 3 SD, and .13 percent fall below 23 SD.

If a set of scores is normally distributed, we can inter- pret any particular score if we know how far, in standard deviation units, it is from the mean. Suppose, for ex- ample, the mean of a normal distribution is 100 and the standard deviation is 15. A score that lies one standard deviation above the mean, therefore, would equal 115. A score that lies one standard deviation below the mean would equal 85. What would a score that lies 1.5 stan- dard deviations above the mean equal? *

We also can determine how a particular individu- al’s score compares with all the other scores in a nor- mal distribution. For example, if a person’s score lies exactly one standard deviation above the mean, then we know that slightly more than 84 percent of all the other scores in the distribution lie below his or her score. † If a distribution is normal and we know the mean and the standard deviation of the distribution, we can determine the percentage of scores that lie above and below any given score (see Figure 10.12 ). This is one of the most useful characteristics of the normal distribution.

STANDARD SCORES AND THE NORMAL CURVE

Researchers often are interested in seeing how one per- son’s score compares with another’s. To determine this, researchers often convert raw scores to derived scores. Earlier we described two types of derived scores—age/ grade equivalents and percentile ranks—but mentioned another type—standard scores—only briefl y. We dis- cuss them now in somewhat more detail, since they are very useful.

Standard scores use a common scale to indicate how an individual compares to other individuals in a group. These scores are particularly helpful in compar- ing an individual’s relative position on different instru- ments. The two standard scores that are most frequently used in educational research are z scores and T scores.

z Scores. The simplest form of standard score is the z score . It expresses how far a raw score is from the mean in standard deviation units. A raw score that is ex- actly on the mean corresponds to a z score of zero; a raw score that is exactly one standard deviation above the mean equals a z score of 11, while a raw score that is exactly one standard deviation below the mean equals a z score of 21. Similarly, a raw score that is exactly two standard deviations above the mean equals a z score of 12, and so forth. One z, therefore, equals one standard deviation (1 z 5 1 SD), 2 z 5 2 SD, 20.5 z 5 20.5 SD, and so on ( Figure 10.13 ). Thus, if the mean of a distri- bution was 50 and the standard deviation was 2, a raw score of 52 would equal a z score of 11, a raw score of 46 would equal a z score of 22, and so forth.

A big advantage of z scores is that they allow raw scores on different tests to be compared. For example, suppose a student received raw scores of 60 on a biol- ogy test and 80 on a chemistry test. A naive observer might be inclined to infer, at fi rst glance, that the student was doing better in chemistry than in biology. But this might be unwise, for how “well” the student is doing comparatively cannot be determined until we know the mean and standard deviation for each distribution of scores. Let us further suppose that the mean on the biol- ogy test was 50, but on the chemistry test it was 90. Also assume that the standard deviation on the biology test was 5, but on the chemistry test it was 10. What does this tell us? The student’s raw score in biology (60) is actually two standard deviations above the mean (a z score of 12), whereas his raw score in chemistry (80) is one standard deviation below the mean (a z score of −1). Rather than doing better in chemistry, as the raw scores by themselves suggest, the student is actually doing bet- ter in biology. Table 10.7 compares both the raw scores, the z scores, and the percentile rank of the student on both tests.

*122.5 †Fifty percent of the scores in the distribution must lie below the mean; 34 percent must lie between the mean and 11 SD. Therefore 84 percent (50 percent 1 34 percent) of the scores in the distribution must be below 11 SD.

Figure 10.13 z Scores Associated with the Normal Curve

–3 z

–3 SD

–2 z

–2 SD

–1 z

–1 SD

0

Mean

1 z

1 SD

z score 2 z

2 SD

3 z

3 SD

fra97851_ch10_185-218.indd 201fra97851_ch10_185-218.indd 201 22/12/10 9:46 PM22/12/10 9:46 PM

202 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

Excel can be used to calculate many descriptive statistics. Simply click on an empty cell, and then click on Function under the Insert menu. This will open a dialogue box. You can search for a function, such as Average, Median, or Standard Deviation. Enter the function you wish to search for in the Search for a function box, and click Go. Excel will present you with a number of possible matches from which to choose. You may have to access the help fi les to distinguish between similar functions (e.g., STDEV is the sample standard deviation, while STDEVP is the population standard deviation). Click on the function you wish to use, and then click OK. This will bring up the Function Arguments dialog box. In this box you can enter the individual cell numbers you wish to include in the calculation, or you can highlight a group of adjoining cells on the spreadsheet by clicking on the fi rst cell and dragging to the last cell. Finally, click OK on the Function Argument dialogue box. Excel will calculate the result and enter it into the (formerly) empty cell on the spreadsheet that you began with.

As you learn the names of functions you commonly use, you can enter them directly, without resorting to the Insert and Function procedure described above. We did this in the example to the left. First we opened an Excel spreadsheet and then listed the data in cells A1 through A15. We then clicked on the cells where we wanted the statistics to appear and typed in the following commands:

• To obtain the mean: 5AVERAGE(A1:A15)* • To obtain the median: 5MEDIAN(A1:A15) • To obtain the standard deviation: 5STDEV(A1:A15)

We typed in these commands in cells A16, A17, and A18, and hit “Return” each time. The cor- responding values† then appeared in those cells: 71.53 (the mean), 77 (the median), and 17.7 (the standard deviation).

Using Excel to Calculate the Mean, Median, and Standard Deviation of a Distribution

*Note: All commands must always begin with an equal sign (5) before typing in the name of the formula and the list to which the formula is to be applied. You must then hit “Return,” and the corresponding value will appear in the designated cell. See Appendix D. †Shortened to two decimal points.

A B C 1 45 2 56 3 76 4 87 5 88 6 61 7 34 8 67 9 55

10 88 11 92 12 85 13 78 14 84 15 77 16 71.53 17 77 18 17.7

TABLE 10.7 Comparisons of Raw Scores and z Scores on Two Tests

Test Raw Score Mean

SD

z Score

Percentile Rank

Biology 60 50 5 12 98 Chemistry 80 90 10 21 16

Thus, for a raw score of 80, a mean of 65, and a stan- dard deviation of 12, the z score will be:

z 5 x 80 2 65 _______ 12 c 5 1.25 Probability and z Scores. Another important characteristic of the normal distribution is that the per- centages associated with areas under the curve can be thought of as probabilities. A probability is a percent stated in decimal form and refers to the likelihood of an event occurring. For example, if there is a probability that an event will occur 25 percent of the time, this event can be said to have a probability of .25. Similarly, an event that will probably occur 90 percent of the time is

Of course, z scores are not always exactly one or two standard deviations away from the mean. Actually, re- searchers apply the following formula to convert a raw score into a z score.

z score 5 raw score 2 mean _______________ standard deviation

fra97851_ch10_185-218.indd 202fra97851_ch10_185-218.indd 202 22/12/10 9:46 PM22/12/10 9:46 PM

C H A P T E R 1 0 Descriptive Statistics 203

said to have a probability of .90. All of the percentages associated with areas under a normal curve, therefore, can be expressed in decimal form and viewed as proba- bility statements. Some of these probabilities are shown in Figure 10.14 .

Considering the area under the normal curve in terms of probabilities is very helpful to a researcher. Let us consider an example. We have previously shown that approximately 34 percent of the scores in a normal distribution lie between the mean and 1 SD. Because 50 percent of the scores fall above the mean, roughly 16 percent of the scores must therefore lie above 1 SD (50 2 34 5 16). Now, if we express 16 percent in decimal form and interpret it as a prob- ability, we can say that the probability of randomly selecting an individual from the population who has a score of 1 SD or more above the mean is .16. Usually this is written as p 5 16, with the p meaning prob- ability. Similarly, we can determine the probability of randomly selecting an individual who has a score lying at or below 22 SD or lower, or between 11 SD and 21 SD, and so on. Figure 10.14 shows that the probability of selecting an individual who has a score lower than −2 SD is p 5 .0228, or roughly 2 in 100. The probability of randomly selecting an individual who has a score between 21 SD and 11 SD is p 5 .6826, and so forth.

Statistical tables exist (see part of one in Appen- dix B) that give the proportion of scores associated with any particular z score in the normal distribution (e.g., for z   5 1.10, the proportion [i.e., the probability] for

the area between the mean of z 5 0 and z 5 1.10 is .3643, and for the area beyond z , it is .1357). Hence, a researcher can be very precise in describing the position of any particular score relative to other scores in a nor- mal distribution. Figure 10.15 shows a portion of such a table (see Appendix B).

T Scores. Raw scores that are below the mean of a dis- tribution convert to negative z scores. This is somewhat awkward. One way to eliminate negative z scores is to convert them into T scores. T scores are simply z  scores expressed in a different form. To change a z score to a T  score, we simply multiply the z score by 10 and add 50. Thus, a z score of 11 equals a T score of 60 (1 3 10 5 10; 10 1 50 5 60). A z score of −2 equals a T score of 30 (−2 5 10 5 −20; −20 1 50 5 30). A z  score of zero (which is the equivalent of the mean of the raw scores)

–3 SD –2 SD –1 SD

.3413 .3413

.1359 .1359

.0215 .0215

.0013 .0013

0

Mean

1 SD 2 SD 3 SD

Figure 10.14 Probabilities under the Normal Curve

Figure 10.15 Table Showing Probability Areas Between the Mean and Different z Scores

Mean

z score A

Area between mean and z

.5000

.4013

.3085

.1587

.0668

.0250

.0228

.0062

.0049

.0013

.0002

.00003

B Area

beyond z

A B

z

0.00

0.25

0.50

1.00

1.50

1.96

2.00

2.50

2.58

3.00

3.50

4.00

.0000

.0987

.1915

.3413

.4332

.4750

.4772

.4938

.4951

.4987

.4998

.49997

fra97851_ch10_185-218.indd 203fra97851_ch10_185-218.indd 203 22/12/10 9:46 PM22/12/10 9:46 PM

204 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

equals a T score of 50. You should see that a distribution of T scores has a mean of 50 and a standard deviation of 10. If you think about it, you should also see that a T score of 50 equals the 50th percentile.

When a researcher knows, or can assume, that a dis- tribution of scores is normal, both T and z scores can be interpreted in terms of percentile ranks because there is then a direct relationship between the two. Figure 10.16 illustrates this relationship. There are other systems similar to T scores, which differ only in the choice of values for the mean and standard deviation. Two of the most common, those used with the Graduate Record Examination (

__ X 5 500, SD 5 100) and the Wechsler

Intelligence Scales ( __

X 5 100, SD 5 15), are also illus- trated in Figure 10.16 .

The Importance of the Normal Curve and z Scores. You may have noticed that the preced- ing discussion of the use of z scores, percentages, and probabilities in relation to the normal curve was always qualifi ed by the words “ if or when the distribution of scores is normal.” You should recall that z scores can be calculated regardless of the shape of the distribution

of original scores. But it is only when the distribution is normal that the conversion to percentages or probabili- ties as described above is legitimate. Fortunately, many distributions do approximate the normal curve. This is most likely when a sample is chosen randomly from a broadly defi ned population. (It would be very unlikely, for example, with achievement scores in a sample that consisted only of gifted students.)

When actual data do not approximate the normal curve, they can be changed to do so. In other words, any distribution of scores can be “normalized.” The proce- dure for doing so is not complicated, but it makes the assumption that the characteristic is “really” normally distributed. Most published tests that permit use of stan- dard scores have normalized the score distributions in order to permit the translation of z scores to percent- ages. This relationship—between z scores and percent- ages of area under the normal curve—is also basic to many inferential statistics.

CORRELATION

In many places throughout this text we have stated that the most meaningful research is that which seeks to

Figure 10.16 Examples of Standard Scores

Standard deviations

–3 –2 –1 Mean 1 2 3

Percentile ranks

0.1 2 16 50 84 98 99.9

z scores –3 –2 –1 0 +1 +2 +3

T scores 20 30 40 50 60 70 80

GRE scores 200 300 400 500 600 700 800

Wechsler scores

55 70 85 100 115 130 145

fra97851_ch10_185-218.indd 204fra97851_ch10_185-218.indd 204 22/12/10 9:46 PM22/12/10 9:46 PM

C H A P T E R 1 0 Descriptive Statistics 205

fi nd, or verify, relationships among variables. Compar- ing the performance of different groups is, as you have seen, one way to study relationships. In such studies one variable is categorical—the variable that defi nes the groups (for example, method A versus method B). The other variable is most often quantitative, and groups are typically compared using frequency polygons, aver- ages, and spreads.

In correlational research, researchers seek to de- termine whether a relationship exists between two (or more) quantitative variables, such as age and weight or reading and writing ability. Sometimes, such relation- ships are useful in prediction, but most often the even- tual goal is to say something about causation. Although causal relationships cannot be proved through corre- lational studies, researchers hope eventually to make causal statements as an outgrowth of their work. The totality of studies showing a relationship between inci- dence of lung cancer and cigarette use is a current ex- ample. We will discuss correlational research in further detail in Chapter 15.

Scatterplots. What is needed is a means to determine whether relationships exist in data. A useful technique for representing relationships with quantitative data is the scatterplot. A scatterplot is a pictorial representation of the relationship between two quantitative variables.

Scatterplots are easy to construct, provided some common pitfalls are avoided. First, in order to be plot- ted, there must be a score on each variable for each

individual; second, the grouping intervals (if any) within each variable (axis) must be of equal size; third, each individual must be represented by one, and only one, point of intersection. We used the data in Table 10.8 to construct the scatterplot in Figure 10.17 . The steps in- volved are the following:

1. Decide which variable will be represented on each axis. It makes no difference which variable is placed on which axis. We have used the horizontal ( x ) axis for variable 1 and the vertical ( y ) axis for variable 2.

2. Divide each axis into about 12 to 15 sections. Each point on the axis will represent a particular score or group of scores. Be sure all scores can be included.

3. Group scores if desirable. It was not necessary for us to group scores for variable 1, since all of the scores fall within a 15-point range. For variable 2, however, representing each score on the axis would result in a great many points on the vertical axis. Therefore, we grouped them within equal sized intervals of fi ve points each.

4. Plot, for each person, the point where the vertical and horizontal lines for his or her scores on the two variables intersect. For example, Pedro had a score of 12 on variable 1, so we locate 12 on the horizontal axis. He had a score of 41 on variable 2, so we lo- cate that score (in the 40–44 grouping) on the verti- cal axis. We then draw imaginary lines from each of these points until they intersect, and mark an X or a dot at that point.

To conduct correlational analysis using Excel, fi rst enter your data into two columns in a worksheet with the X values in one column and the Y values in another column. Next, select Tools Data Analysis. In the Data Analysis dialog box choose Correlation tool from the Analysis Tools list and click OK. Identify the range of X and Y values you want to analyze by entering the worksheet range. If the input range includes labels, select the Labels in the First Row box. Confi rm that the Grouped By button—either Columns or Rows—refl ects how your data are organized. Next, use the Output Options button to tell Excel where to place your correlation results. For example, to place the results into a range in your existing worksheet, select the Output Range button and then indicate the range address in the Output Range text box. If you want to put them in a new workbook, select the New Workbook button. Then click OK. Excel calculates the correlation coeffi cient and signifi cance results for the data you identifi ed and puts it in the location you speci- fi ed. (For an example, see Appendix D.)

Using Excel to Compute Correlations

fra97851_ch10_185-218.indd 205fra97851_ch10_185-218.indd 205 22/12/10 9:46 PM22/12/10 9:46 PM

206

80–84

75–79

70–74

65–69

60–64

55–59

50–54

45–49

35–39

30–34

25–29

20–24

15–19

10–14

Variable 1

Va ria

bl e

2

7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

40–44

Grouped Scores Variable 2

Frequency of Grouped Scores Variable 2

Scores Variable 1

Scores Variable 2Individual

Pedro 12 21 9 7

13 11 7

18 15 14

41 76 29 21 43 37 14 63 54 39

75–79 70–74 65–69 60–64 55–59 50–54 45–49 40–44 35–39 30–34 25–29 20–24 15–19 10–14

1 0 0 1 0 1 0 2 2 0 1 1 0 1

John Alicia Charles Bonnie Phil Sue Oscar Jean Dave

TABLE 10.8 Data Used to Construct Scatterplot in Figure 10.17 Figure 10.17 Scatterplot of Data from Table 10.8

fra97851_ch10_185-218.indd 206 fra97851_ch10_185-218.indd 206

22/12/10 9:46 PM 22/12/10 9:46 PM

C H A P T E R 1 0 Descriptive Statistics 207

5. In the same way, plot the scores of all 10 students on both variables. The completed result is a scatterplot.

Interpreting a Scatterplot. How do researchers inter- pret scatterplots? What are they intended to reveal? Re- searchers want to know not only if a relationship exists between variables, but also to what degree. The degree of relationship, if one exists, is what a scatterplot illustrates.

Consider Figure 10.17 . What does it tell us about the relationship between variable 1 and variable 2? This question can be answered in several ways.

1. We might say that high scores on variable 1 go with high scores on variable 2 (as in John’s case) and that low scores also tend to go together (as in Sue’s case).

2. We might say that by knowing a student’s score on one variable, we can estimate his or her score on the other variable fairly closely. Suppose, for example, a new student attains a score of 16 on variable 1. What would you predict his or her score would be on vari- able 2? You probably would not predict a score of 75 or one of 25 (we would predict a score somewhere from 45 to 59).

3. The customary interpretation of a scatterplot that looks like this would be that there is a strong or high degree of relationship between the two variables.

Outliers. Outliers are scores or measurements that dif- fer by such large amounts from those of other individuals in a group that they must be given careful consideration as special cases. They indicate an unusual exception to a general pattern. They occur in scatterplots, as well as frequency distribution tables, histograms, and fre- quency polygons. Figure 10.18 shows the relationship between family cohesiveness and school achievement. Notice the lonely individual near the lower right corner with a high score in family cohesiveness but a low score in achievement. Why? The answer should be of interest to the student’s teacher.

Correlation Coefficients and Scatterplots. Figure 10.19 presents several other examples of scat- terplots. Studying them will help you understand the notion of a relationship and also further your under- standing of the correlation coeffi cient. As we mentioned in Chapter  8, a correlation coefficient , designated

Excel can also be used to draw a scatterplot to illustrate the relationship between two quantitative variables. The dataset to be illustrated must be entered into two columns in an Excel worksheet, with the X variable in one column and the Y variable in another. Once

the data are entered, pull down the Insert menu, then click on Chart. When the chart window opens, click on Scatter. Then click on Next at the bottom of the dialogue box. Another dialogue box will open. Click on the fi rst cell in the upper left of the two columns of data, and drag down to the last cell in the lower right. Click on Finish at the bottom of the dialogue box to display the fi nished chart. Shown here is an example of the relationship between two quizzes for a hypothetical class of 15 high school chemistry students. Note that the correlation coeffi cient is shown to the left of the scatterplot. In this instance, r 5 .97, indicating a very strong, positive relationship between the two sets of quiz scores for this group of students.

Using Excel to Draw a Scatterplot

Q ui

z #2

Quiz #1

Scatterplot Showing Relationship Between Two Sets of Quiz Scores

40

40

r = .97

X 27 24 34 36 18 16 13 25 10 15 19 14 27 7 24

Y 32 27 33 38 21 15 17 28 12 16 18 14 29 8 22

35

30

25

20

15

10

5

0 3020100

fra97851_ch10_185-218.indd 207fra97851_ch10_185-218.indd 207 22/12/10 9:46 PM22/12/10 9:46 PM

208 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

by the symbol r , expresses the degree of relationship between two sets of scores. * A positive relationship is indicated when high scores on one variable are accom- panied by high scores on the other, when low scores on one are accompanied by low scores on the other, and so forth. A negative relationship is indicated when high scores on one variable are accompanied by low scores on the other, and vice versa ( Figure 10.20 ).

You should recall that correlation coeffi cients are never more than 11.00, indicating a perfect positive relationship, or 21.00, indicating a perfect negative relationship. Perfect positive or negative correlations, however, are rarely, if ever, achieved ( Figure 10.21 ). If the two variables are highly related, a coeffi cient somewhat close to 11.00 or 21.00 will be obtained (such as .85 or 2.93). The closer the coeffi cient is to either of these extremes, the greater the degree of the relationship. If there is no or hardly any relationship, a coeffi cient of .00 or close to it will be obtained. The coeffi cient is calculated directly from the same scores used to construct the scatterplot.

The scatterplots in Figure 10.19 illustrate different degrees of correlation. Both positive and negative corre- lations are shown. Scatterplots ( a ), ( b ), and ( c ) illustrate different degrees of positive correlation, while scatter- plots ( e ), ( f ), ( g ), and ( h ) illustrate different degrees of negative correlation. Scatterplot ( d ) indicates no rela- tionship between the two variables involved.

In order to better understand the meaning of different values of the correlation coeffi cient, we suggest that you try the following two exercises with Figure 10.19 .

1. Lay a pencil fl at on the paper on scatterplot ( a ) so that the entire length of the pencil is touching the paper. Place it in such a way that it touches or cov- ers as many dots as possible. Take note that there is clearly one “best” placement. You would not, for example, maximize the points covered if you placed the pencil horizontally on the scatterplot. Repeat this procedure for each of the scatterplots, noting what occurs as you move from one scatter- plot to another.

2. Draw a horizontal line on scatterplot ( a ) so that about half of the dots are above the line and half are below it. Next, draw a vertical line so that about half of the dots are to the left of the line and half are to the right. Repeat the procedure for each scatterplot and note what you observe as you move from one scatterplot to another.

The Pearson Product-Moment Coefficient. There are many different correlation coeffi cients, each applying to a particular circumstance and each calcu- lated by means of a different computational formula. The one we have been illustrating is the one most fre- quently used: the Pearson product-moment coefficient of correlation (also known as the Pearson r ). It is sym- bolized by the lowercase letter r . When the data for both variables are expressed in terms of quantitative scores, the Pearson r is the appropriate correlation coeffi cient to use. It assumes that the relationship is best described by a straight line. (It is not diffi cult to calculate, and we’ll show you how in Chapter 12.) It is designed for use with interval or ratio data.

Eta. Another index of correlation that you should be- come familiar with is called eta (symbolized as h). We shall not illustrate how to calculate eta (since it requires computational methods beyond the scope of this text), but you should know that it is used when a scatterplot shows that a straight line is not the best fi t for the plot- ted points. In the examples shown in Figure 10.22 , for example, you can see that a curved line provides a much better fi t to the data than would a straight line.

Eta is interpreted in much the same way as the Pearson r, except that it ranges from .00 to 1.00, rather than from 21.00 to 11.00. Higher values, as with the other correla- tion coeffi cients, indicate higher degrees of relationship.

Figure 10.18 Relationship Between Family Cohesiveness and School Achievement in a Hypothetical Group of Students

Low

A ch

ie ve

m en

t

Family cohesiveness

High

Outlier

Low

High

*In this context, the correlation coeffi cient indicates the degree of re- lationship between two variables. You will recall from Chapter 8 that it is also used to assess the reliability and validity of measurements.

fra97851_ch10_185-218.indd 208fra97851_ch10_185-218.indd 208 22/12/10 9:46 PM22/12/10 9:46 PM

C H A P T E R 1 0 Descriptive Statistics 209

(a)

r = .90

(e)

r = –.90

(b)

r = .65

(f )

r = –.75

(c)

r = .35

(g)

r = –.50

(d)

r = .00

(h)

r = –.10

Figure 10.19 Further Examples of Scatterplots

Techniques for Summarizing Categorical Data THE FREQUENCY TABLE

Suppose a researcher, using a questionnaire, has been collecting data from a random sample of 50 teach- ers in a large urban school district. The questionnaire

covers many variables related to their activities and interests. One of the variables is learning activity I use most frequently in my classroom. The researcher arranges her data on this variable (and others) in the form of a frequency table, which shows the frequency with which each type, or category, of learning ac- tivity is mentioned. The researcher simply places a tally mark for each individual in the sample along- side the activity mentioned. When she has tallied all

fra97851_ch10_185-218.indd 209fra97851_ch10_185-218.indd 209 22/12/10 9:46 PM22/12/10 9:46 PM

210 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

D ep

en de

nc e

on o

th er

s

Age

Ta sk

p er

fo rm

an ce

Anxiety

Figure 10.22 Examples of Nonlinear (Curvilinear) Relationships

“No!” “No!”

“Yes!”

“Can I stay overnight at Susie’s house?”

“Can I have some more ice cream?”

“Do I have to clean my room?”

“The more you like something the less you

get to do it!”

r = –1.00

Figure 10.20 A Perfect Negative Correlation!

Figure 10.21 Positive and Negative Correlations

fra97851_ch10_185-218.indd 210fra97851_ch10_185-218.indd 210 22/12/10 9:46 PM22/12/10 9:46 PM

C H A P T E R 1 0 Descriptive Statistics 211

50  individuals, her results look like the following fre- quency listing.

Response Tally Frequency

Lectures |||| |||| |||| 15 Class discussions |||| |||| 10 Oral reports |||| 4 Library research || 2 Seatwork |||| 5 Demonstrations |||| ||| 8 Audiovisual presentations |||| | 6 n 5 50

The tally marks have been added up at the end of each row to show the total number of individuals who listed

that activity. Often with categorical data, researchers are interested in proportions, because they wish to make an estimate (if their sample is random) about the propor- tions in the total population from which the sample was selected. Thus, the total numbers in each category are often changed to percentages. This has been done in Table 10.9 , with the categories arranged in descending order of frequency.

BAR GRAPHS AND PIE CHARTS

There are two other ways to illustrate a difference in proportions. One is a bar graph portrayal, as shown in Figure 10.23 ; another is a pie chart , as shown in Figure 10.24 .

THE CROSSBREAK TABLE

When a relationship between two categorical vari- ables is of interest, it is usually reported in the form of a crossbreak table (sometimes called a contingency table ). The simplest crossbreak is a 2 by 2 table, as shown in Table 10.10 . Each individual is tallied in one, and only one, cell that corresponds to the combination of gender and grade level. You will notice that the numbers in each of the cells in Table 10.10 represent totals—the total number of individuals who fi t the characteristics of the cell (for example, junior high male teachers). Although percentages and proportions are sometimes calculated for cells, we do not recommend it, as this is often misleading.

TABLE 10.9 Frequency and Percentage of Total of Responses to Questionnaire

Response Frequency Percentage of Total (%)

Lectures 15 30 Class discussions 10 20 Demonstrations 8 16 Audiovisual presentations

6 12

Seatwork 5 10 Oral reports 4 8 Library research 2 4

Total 50 100

Figure 10.23 Example of a Bar Graph

30%

15

Lectures

20%

10

Class discussions

16%

8

Demonstrations

12%

6

Audiovisual presentations

ACTIVITY

Percentages of learning activities used most frequently by 50 teachers

10%

5

Seatwork

8%

4

Oral reports

4%

2

Library research

40

30

20

10

0

PE RC

EN TA

G E

n =

fra97851_ch10_185-218.indd 211fra97851_ch10_185-218.indd 211 22/12/10 9:46 PM22/12/10 9:46 PM

212 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

It probably seems obvious that Table 10.10 reveals a relationship between teacher gender and grade level. A junior high school teacher is more likely to be female; a high school teacher is more likely to be male. Often, however, it is useful to calculate “expected” frequen- cies in order to see results more clearly. What do we mean by “expected”? If there is no relationship between variables, we would “expect” the proportion of cases within each cell of the table corresponding to a category of a variable to be identical to the proportion within that category in the entire group. Look, for example, at Table 10.11. Exactly one-half (50 percent) of the total group of teachers in this table are female. If gender is unrelated to grade level, we would “expect” that the

same proportion (exactly one-half) of the junior high school teachers would be female. Similarly, we would “expect” that one-half of the high school teachers would be female. The “expected” frequencies, in other words, would be 50 female junior high school teachers and 50 female high school teachers, rather than the 60 female junior high school and 40 female high school teachers that were actually obtained. These expected and ac- tual, or “observed,” frequencies are shown in each box (or “cell”) in Table 10.11 . The expected frequencies are shown in parentheses. *

Comparing expected and actual frequencies makes the degree and direction of the relationship clearer. This is particularly helpful with more complex tables. Look, for example, at Table 10.12 . This table contains not two, but three, variables.

The researcher who collected and summarized these data hypothesized that appointment to administrative (or other nonteaching) positions rather than teaching positions is related to (1) gender and (2) ethnicity. While it is possible to examine Table 10.12 in its entirety to evaluate these hypotheses, it is much easier to see the relationships by extracting components of the table. Let us look at the relationship of each variable in the table to the other two variables. By taking two variables

Figure 10.24 Example of a Pie Chart

15 Lectures

10 Class discussions

8 Demonstrations

6 Audiovisual presentations

5 Seatwork

4 Oral reports

2 Library research

Learning activities used most frequently by 50 teachers

TABLE 10.10 Grade Level and Gender of Teachers (Hypothetical Data)

Male Female Total

Junior high school teachers 40 60 100 High school teachers 60 40 100 Total 100 100 200

TABLE 10.11 Repeat of Table 10.10 with Expected Frequencies (in Parentheses)

Male Female Total

Junior high school teachers 40 (50) 60 (50) 100 High school teachers 60 (50) 40 (50) 100 Total 100 100 200

*Expected frequencies can also be provided ahead of time, based on theory or prior experience. In this example, the researcher might have wanted to know whether the characteristics of junior high school teachers in a particular school fi t the national pattern. National percentages would then be used to determine expected frequencies.

TABLE 10.12 Position, Gender, and Ethnicity of School Leaders (Hypothetical Data)

Administrators Teachers

White Nonwhite White Nonwhite Total

Male 50 20 150 80 300

Female 20 10 150 120 300

Total 70 30 300 200 600

fra97851_ch10_185-218.indd 212fra97851_ch10_185-218.indd 212 22/12/10 9:46 PM22/12/10 9:46 PM

213

Correlation in Everyday Life

M any commonplace relationships (true or not) can be ex-pressed as correlations. For example, Boyle’s law states that the volume and pressure of a gas vary inversely if kept at a constant temperature. Another way to express this is that the correlation between volume and pressure is −1.00. This relationship, however, is only theoretically true—that is, it ex- ists only for a perfect gas in a perfect vacuum. In real life, the correlation is lower.

Consider the following sayings:

1. “Spare the rod and spoil the child” implies a negative cor- relation between punishment and spoiled behavior.

MORE ABOUT RESEARCH

2. “Idle hands are the devil’s workplace” implies a positive correlation between idleness and mischief.

3. “There’s no fool like an old fool” suggests a positive cor- relation between foolishness and age.

4. “A stitch in time saves nine” suggests a positive correlation between how long one waits to begin a corrective action and the amount of time (and effort) required to fi x the problem.

5. “The early bird catches the worm” suggests a positive cor- relation between early rising and success.

6. “You can’t teach an old dog new tricks” implies a negative correlation between age of adults and ability to learn.

7. “An apple a day keeps the doctor away” suggests a negative correlation between the consumption of apples and illness.

8. “Faint heart never won fair maiden” suggests a negative correlation between timidity and female receptivity.

TABLE 10.15 Gender and Ethnicity of School Leaders with Expected Frequencies (Derived from Table 10.12 )

White Nonwhite Total

Male 200 (185) 100 (115) 300

Female 170 (185) 130 (115) 300

Total 370 230 600

TABLE 10.14 Position and Gender of School Leaders with Expected Frequencies (Derived from Table 10.12 )

Administrators Teachers Total

Male 70 (50) 230 (250) 300

Female 30 (50) 270 (250) 300

Total 100 500 600

at a time, we can compare (1) position and  ethnicity; (2) position and gender; and (3) gender and ethnicity. Table 10.13 presents  the data for position and ethnic- ity, Table 10.14 presents the data for position and gen- der, and Table 10.15 presents the data for gender and ethnicity.

Let us review the calculation of expected frequen- cies by referring to Table 10.13 . This table shows the relationship between ethnicity and position. Since one- sixth of the total group (100/600) are administrators,

we would expect 62 whites to be administrators ( 1 _ 6 of 370). Likewise, we would expect 38 of the non- whites to be administrators ( 1 _ 6 of 230). Since fi ve-sixths of the total group are teachers, we would expect 308 of the whites ( 5 _ 6 of 370) and 192 of the nonwhites (

5 _ 6 of

230) to be teachers. As you can see, however, the actual frequencies for administrators were 70 (rather than 62) whites, and 30 (rather than 38) nonwhites, and the ac- tual frequencies for teachers were 300 (rather than 308) whites, and 200 (rather than 192) nonwhites. This tells us that there is a discrepancy between what we would expect (if there is no relationship) and what we actu- ally obtained. Discrepancies between the frequencies expected and those actually obtained can also be seen in Tables 10.14 and 10.15.

An index of the strength of the relationships can be obtained by summing the discrepancies in each table. In Table 10.13 , the sum equals 32, in Table 10.14 , it equals 80, and in Table 10.15 it equals 60. The calculation of these sums is shown in Table 10.16 . The discrepancy between expected and observed frequencies is greatest

TABLE 10.13 Position and Ethnicity of School Leaders with Expected Frequencies (Derived from Table 10.12 )

Administrators Teachers Total

White 70 (62) 300 (308) 370

Nonwhite 30 (38) 200 (192) 230

Total 100 500 600

fra97851_ch10_185-218.indd 213fra97851_ch10_185-218.indd 213 22/12/10 9:46 PM22/12/10 9:46 PM

214 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

TABLE 10.16 Total of Discrepancies Between Expected and Observed Frequencies in Tables 10.13 Through 10.15

Table 10.13 Table 10.14 Table 10.15

(70 vs. 62) 5 8 (70 vs. 50) 5 20 (200 vs. 185) 5 15 (30 vs. 38) 5 8 (30 vs. 50) 5 20 (170 vs. 185) 5 15 (300 vs. 308) 5 8 (230 vs. 250) 5 20 (100 vs. 115) 5 15 (200 vs. 192) 5 8 (270 vs. 250) 5 20 (130 vs. 115) 5 15 Total 32 80 60

Go back to the INTERACTIVE AND APPLIED LEARNING feature at the beginning of the chapter for a listing of interactive and applied activities. Go to the Online Learning Center at www.mhhe.com/fraenkel8e to take quizzes, practice with key terms, and review chapter content.

in Table 10.14 , position and gender; less in Table 10.15 , gender and ethnicity; and least in Table 10.13 , position and ethnicity. A numerical index showing degree of relationship—the contingency coeffi cient—will be dis- cussed in Chapter 11.

Thus, the data in the crossbreak tables reveal that there is a slight tendency for there to be more white administrators and more nonwhite teachers than would be expected ( Table 10.13 ). There is a stronger tendency toward more white males and nonwhite females than would be expected ( Table 10.15 ). The strongest rela- tionship indicates more male administrators and more female teachers than would be expected ( Table 10.14 ). In sum, the chances of having an administrative position appear to be considerably greater if one is male, and slightly enhanced if one is white.

In contrast to the preceding example, where each variable (ethnicity, gender, role) is clearly categorical, a researcher sometimes has a choice whether to treat data as quantitative or as categorical. Take the case of a researcher who measures self-esteem by a self-report questionnaire scored for number of items answered (yes or no) in the direction indicating high self-esteem. The researcher might decide to use these scores to di- vide the sample ( n 5 60) into high, middle, and low thirds. He or she might use only this information for each individual and subsequently treat the data as cat- egorical, as is shown, for example, in Table 10.17 . Most researchers would advise against treating the data this

way, however, since it “wastes” so much information— for example, differences in scores within each category are ignored. A quantitative analysis, by way of contrast, would compare the mean self-esteem scores of males and females.

In such situations (i.e., when one variable is quantita- tive and the other is treated as categorical), correlation is another option. When data on a variable assumed to be quantitative have been divided into two categories, a biserial correlation coeffi cient can be calculated and interpreted the same as a Pearson r coeffi cient. * If the categories are assumed to refl ect a true division, calcula- tion of a point biserial coeffi cient is an option, but must be interpreted with caution.

*A detailed explanation of these statistics is beyond the scope of this text. For further details, consult any statistics text.

TABLE 10.17 Crossbreak Table Showing Relationship Between Self-Esteem and Gender (Hypothetical Data)

Self Esteem

Gender Low Middle High

Male 10 15 5 Female 5 10 15

fra97851_ch10_185-218.indd 214fra97851_ch10_185-218.indd 214 22/12/10 9:46 PM22/12/10 9:46 PM

C H A P T E R 1 0 Descriptive Statistics 215

STATISTICS VERSUS PARAMETERS

• A parameter is a characteristic of a population. It is a numerical or graphic way to summarize data obtained from the population.

• A statistic, on the other hand, is a characteristic of a sample. It is a numerical or graphic way to summarize data obtained from a sample.

TYPES OF NUMERICAL DATA

• There are two fundamental types of numerical data a researcher can collect. Quanti- tative data are obtained by determining placement on a scale that indicates amount or degree. Categorical data are obtained by determining the frequency of occurrences in each of several categories.

TYPES OF SCORES

• A raw score is the initial score obtained when using an instrument; a derived score is a raw score that has been translated into a more useful score on some type of stan- dardized basis to aid in interpretation.

• Age/grade equivalents are derived scores that indicate the typical age or grade associ- ated with an individual raw score.

• A percentile rank is the percentage of a specifi c group scoring at or below a given raw score.

• A standard score is a mathematically derived score having comparable meaning on different instruments.

TECHNIQUES FOR SUMMARIZING QUANTITATIVE DATA

• A frequency distribution is a two-column listing, from high to low, of all the scores along with their frequencies. In a grouped frequency distribution, the scores have been grouped into equal intervals.

• A frequency polygon is a graphic display of a frequency distribution. It is a graphic way to summarize quantitative data for one variable.

• A graphic distribution of scores in which only a few individuals receive high scores is called a positively skewed polygon; one in which only a few individuals receive low scores is called a negatively skewed polygon.

• A histogram is a bar graph used to display quantitative data at the interval or ratio level of measurement.

• A stem-leaf plot is similar to a histogram, except it lists specifi c values instead of bars.

• The normal distribution is a theoretical distribution that is symmetrical and in which a large proportion of the scores are concentrated in the middle.

• A distribution curve is a smoothed-out frequency polygon. • The distribution curve of a normal distribution is called a normal curve. It is bell

shaped, and its mean, median, and mode are identical. • There are several measures of central tendency (averages) that are used to summarize

quantitative data. The two most common are the mean and the median. • The mean of a distribution is determined by adding up all of the scores and dividing

this sum by the total number of scores.

Main Points

fra97851_ch10_185-218.indd 215fra97851_ch10_185-218.indd 215 22/12/10 9:46 PM22/12/10 9:46 PM

216 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

• The median of a distribution marks the point above and below which half of the scores in the distribution lie.

• The mode is the most frequent score in a distribution. • The term variability, as used in research, refers to the extent to which the scores on a

quantitative variable in a distribution are spread out. • The most common measure of variability used in educational research is the standard

deviation. • The range, another measure of variability, represents the difference between the

highest and lowest scores in a distribution. • A fi ve-number summary of a distribution reports the lowest score, the fi rst quartile,

the median, the third quartile, and the highest score. • Five-number summaries of distributions are often portrayed graphically by the use

of boxplots.

STANDARD SCORES AND THE NORMAL CURVE

• Standard scores use a common scale to indicate how an individual compares to other individuals in a group. The simplest form of standard score is a z score. A z score expresses how far a raw score is from the mean in standard deviation units.

• The major advantage of standard scores is that they provide a better basis for compar- ing performance on different measures than do raw scores.

• The term probability, as used in research, refers to a prediction of how often a par- ticular event will occur. Probabilities are usually expressed in decimal form.

CORRELATION

• A correlation coeffi cient is a numerical index expressing the degree of relationship between two quantitative variables. The one most commonly used in educational research is the Pearson r .

• A scatterplot is a graphic way to describe a relationship between two quantitative variables.

TECHNIQUES FOR SUMMARIZING CATEGORICAL DATA

• Researchers use various graphic techniques to summarize categorical data, including frequency tables, bar graphs, and pie charts.

• A crossbreak table is a graphic way to report a relationship between two or more categorical variables.

68-95-99.7 rule 200

age-equivalent scores 189

averages 195

bar graph 211

boxplot 198

correlation coeffi cient 207

crossbreak table/ contingency table 211

descriptive statistics 187

derived score 189

distribution curves 195

eta 208

fi ve-number summary 198

frequency distribution 190

frequency polygon 190

grade-equivalent score 189

grouped frequency distribution 190

histogram 194

mean 196

Key Terms

fra97851_ch10_185-218.indd 216fra97851_ch10_185-218.indd 216 22/12/10 9:46 PM22/12/10 9:46 PM

C H A P T E R 1 0 Descriptive Statistics 217

measures of central tendency 195

median 196

mode 195

negatively skewed 191

normal curve 195

normal distribution 195

outlier 207

parameters 187

Pearson product- moment coeffi cient/ Pearson r 208

percentile rank 189

percentile 198

pie chart 211

positively skewed 191

probability 202

quantitative data 188

range 198

raw score 189

scatterplot 205

standard deviation 198

standard score 201

statistics 187

stem-leaf plot 194

T score 203

variability 197

variance 199

z score 201

1. Would you expect the following correlations to be positive or negative? Why? a. Bowling scores and golf scores b. Reading scores and arithmetic scores for sixth-graders c. Age and weight for a group of 5-year-olds; for a group of people over 70 d. Life expectancy at age 40 and frequency of smoking e. Size and strength for junior high students

2. Why do you think so many people mistrust statistics? How might such mistrust be alleviated?

3. Would it be possible for two different distributions to have the same standard devia- tion but different means? What about the reverse? Explain.

4. “The larger the standard deviation of a distribution, the more heterogeneous the scores in that distribution.” Is this statement true? Explain.

5. “The most complete information about a distribution of scores is provided by a fre- quency polygon.” Is this statement true? Explain.

6. Grouping scores in a frequency distribution has its advantages but also its disadvan- tages. What might be some examples of each?

7. “Any single raw score, in and of itself, tells us nothing.” Would you agree? Explain. 8. The relationship between age and strength is said to be curvilinear. What does this

mean? Might there be exceptions to this relationship? What might cause such an exception?

For Discussion

fra97851_ch10_185-218.indd 217fra97851_ch10_185-218.indd 217 22/12/10 9:46 PM22/12/10 9:46 PM

218 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

Research Exercise 10: Descriptive Statistics If you are planning a quantitative study, indicate the descriptive statistics you would use to sum- marize the data you intend to collect. Even if you are planning a qualitative or mixed-methods study, in some cases including descriptive statistics and data analysis strategies may be appropri- ate and useful. Finally, discuss your procedures for dealing with discrepant cases or outliers.

Problem Sheet 10

Descriptive Statistics 1. If you are designing a quantitative study, place an X after each of the descriptive

statistics listed below that you will use to summarize your data.

Frequency polygon _____ Five-number summary _____ Box plot _____ Percentages _____

Mean _____ Median _____ Standard deviation _____ Frequency table _____ Bar graph ___

Pie chart _____ Correlation coeffi cient _____ Scatterplot _____

2. Place an X after the technique(s) you would use to describe any relationships found in your study.

a. Comparison of frequency polygons ______

b. Comparison of averages ______

c. Crossbreak table(s) ______

d. Correlation coeffi cient ______

e. Scatterplot ______

f. Reporting of percentages ______

3. How will you deal with discrepant cases or outliers in your data analysis?

An electronic version of this Problem Sheet that you can fi ll in and print, save, or e-mail is available on the Online Learning Center at www.mhhe.com/ fraenkel8e.

fra97851_ch10_185-218.indd 218fra97851_ch10_185-218.indd 218 22/12/10 9:46 PM22/12/10 9:46 PM

Inferential Statistics 11 “Great! Can we be sure

then that 60 percent of the voters will go for it?”

“No, not exactly. But it is very likely that between 55 percent and

65 percent will!”

“Survey results show that 60 percent

of those we interviewed support our bond issue.”

• Explain what is meant by the term “inferential statistics.”

• Explain the concept of sampling error. • Describe briefl y how to calculate a

confi dence interval. • State the difference between a research

hypothesis and a null hypothesis. • Describe briefl y the logic underlying

hypothesis testing. • State what is meant by the terms

“signifi cance level” and “statistically signifi cant.”

• Explain the difference between a one- and a two-tailed test of signifi cance.

• Explain the difference between parametric and nonparametric tests of signifi cance.

• Name three examples of parametric tests used by educational researchers.

• Name three examples of nonparametric tests used by educational researchers.

• Describe what is meant by the term “power” with regard to statistical tests.

• Explain the importance of random sampling with regard to the use of inferential statistics.

What Are Inferential Statistics?

The Logic of Inferential Statistics

Sampling Error Distribution of Sample Means Standard Error of the Mean Confi dence Intervals Confi dence Intervals and

Probability Comparing More Than One

Sample The Standard Error of the

Difference Between Sample Means

Hypothesis Testing

The Null Hypothesis Hypothesis Testing:

A Review

Practical Versus Statistical Signifi cance

One- and Two-Tailed Tests Use of the Null Hypothesis:

An Evaluation

Inference Techniques

Parametric Techniques for Analyzing Quantitative Data

Nonparametric Techniques for Analyzing Quantitative Data

Parametric Techniques for Analyzing Categorical Data

Nonparametric Techniques for Analyzing Categorical Data

Summary of Techniques Power of a Statistical Test

O B J E C T I V E S Studying this chapter should enable you to:

fra97851_ch11_219-244.indd 219fra97851_ch11_219-244.indd 219 12/21/10 6:49 PM12/21/10 6:49 PM

220 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

What Are Inferential Statistics? Descriptive statistics are but one type of statistic that researchers use to analyze their data. Many times they wish to make inferences about a population based on data they have obtained from a sample. To do this, they use inferential statistics. Let us consider how.

Suppose a researcher administers a commercially available IQ test to a sample of 65 students selected from a particular elementary school district and fi nds their average score is 85. What does this tell her about the IQ scores of the entire population of students in the district? Does the average IQ score of students in the district also equal 85? Or is this sample of students different, on the average, from other students in the

district? If these students are different, how are they dif- ferent? Are their IQ scores higher—or lower?

What the researcher needs is some way to estimate how closely statistics, such as the mean of the IQ scores, obtained on a sample agree with the corresponding pa- rameters of the population without actually obtaining data on the entire population. Inferential statistics pro- vide such a way.

Inferential statistics are certain types of procedures that allow researchers to make inferences about a popu- lation based on fi ndings from a sample. In Chapter 6, we discussed the concept of a random sample and pointed out that obtaining a random sample is desirable because it helps ensure that one’s sample is representative of a larger population. When a sample is representative, all the characteristics of the population are assumed to be

P aulo, I’m worried.” “Why, Julie?” “Well, in my job as director of the new elementary math program for the district, I have to fi nd out how well the students in

the program are doing. This year, we’re testing the fi fth-graders.”

“Yeah, so? What’s to worry about?”

“Well, I gave the end-of-the-semester exam to one of the fi fth-grade classes at Hoover Elementary last week. I got the

results back just today.”

“And?”

“Get this. Their average score was only 65 out of a possible 100 points! Of course this is just one class, but still . . . I’m afraid

that it may be true of all the fi fth-grade classes.”

“Not necessarily, Julie. That would depend on how similar the Hoover kids are to the other fi fth-graders in the district. What

you need is some way to estimate the average score of all of the district’s fi fth-graders—but you can’t do it from that class.”

“I assume you’re thinking about some sort of inferential test, eh?”

“Yep, you got it.”

How might Julie make such an estimate? To fi nd out, read this chapter.

INTERACTIVE AND APPLIED LEARNING After, or while, reading this chapter:

Go to your online Student Mastery Activities book to do the following activities:

• Activity 11.1: Probability • Activity 11.2: Learn to Read a t -Table • Activity 11.3: Calculate a t -Test • Activity 11.4: Perform a Chi-Square Test • Activity 11.5: Conduct a t -Test • Activity 11.6: The Big Game

Go to the Online Learning Center at www.mhhe.com/fraenkel8e to:

• Practice with Sample Graphs • Learn More About the Purpose of Inferential Statistics

220

“

fra97851_ch11_219-244.indd 220fra97851_ch11_219-244.indd 220 12/21/10 6:49 PM12/21/10 6:49 PM

C H A P T E R 1 1 Inferential Statistics 221

present in the sample in the same degree. No sampling procedure, not even random sampling, guarantees a to- tally representative sample, but the chance of obtaining one is greater with random sampling than with any other method. And the more a sample represents a population, the more researchers are entitled to assume that what they fi nd out about the sample will also be true of that population. Making inferences about populations on the basis of random samples is what inferential statistics is all about.

As with descriptive statistics, the techniques of in- ferential statistics differ depending on which type of data—categorical or quantitative—a researcher wishes to analyze. This chapter begins with techniques appli- cable to quantitative data, because they provide the best introduction to the logic behind inference techniques and because most educational research involves such data. Some techniques for the analysis of categorical data are presented at the end of the chapter.

The Logic of Inferential Statistics Suppose a researcher is interested in the difference be- tween males and females with respect to interest in his- tory. He hypothesizes that female students fi nd history more interesting than do male students. To test the hy- pothesis, he decides to perform the following study. He obtains one random sample of 30 male history students from the population of 500 male tenth-grade students taking history in a nearby school district and another random sample of 30 female history students from the female population of 550 female tenth-grade history students in the district. All students are given an at- titude scale to complete. The researcher now has two sets of data: the attitude scores for the male group and the attitude scores for the female group. The design of the study is shown in Figure 11.1 . The researcher wants to know whether the male population is differ- ent from the female population—that is, will the mean score of the male group on the attitude test differ from the mean score of the female group? But the researcher does not know the means of the two populations. All he has are the means of the two samples. He has to rely on the two samples to provide information about the populations.

Is it reasonable to assume that each sample will give a fairly accurate picture of its population? It certainly

is possible, since each sample was randomly selected from its population. On the other hand, the students in each sample are only a small portion of their popu- lation, and only rarely is a sample absolutely identi- cal to its parent population on a given characteristic. The data the researcher obtains from the two samples will depend on the individual students selected to be in each sample. If another two samples were randomly selected, their makeup would differ from the original two, their means on the attitude scale would be differ- ent, and the researcher would end up with a different set of data. How can the researcher be sure that any particular sample he has selected is, indeed, a represen- tative one? He cannot. Maybe another sample would be better.

SAMPLING ERROR

This is the basic diffi culty that confronts us when we work with samples: Samples are not likely to be identi- cal to their parent populations. This difference between a sample and its population is referred to as sampling error ( Figure 11.2 ). Furthermore, no two samples will be the same in all their characteristics. Two different samples from the same population will not be identi- cal: They will be composed of different individuals, they will have different scores on a test (or other mea- sure), and they will probably have different sample means.

Consider the population of high school students in the United States. It would be possible to select liter- ally thousands of different samples from this popula- tion. Suppose we took two samples of 25 students each from this population and measured their heights. What would you estimate our chances would be of fi nding ex- actly the same mean height in both samples? Very, very unlikely. In fact, we could probably take sample after sample and very seldom obtain two sets of people hav- ing exactly the same mean height.

Figure 11.1 Selection of Two Samples from Two Distinct Populations

Population of male history students

n = 500

Sample 1 n = 30

Population of female history students

n = 550

Sample 2 n = 30

fra97851_ch11_219-244.indd 221fra97851_ch11_219-244.indd 221 12/21/10 6:49 PM12/21/10 6:49 PM

222 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

normally distributed. Once sample size reaches 30, however, the distribution of sample means is very nearly normal, even if the population is not normally distributed. (We realize that this is not immediately obvious; should you wish more explanation of why this is true, consult any introductory statistics text.)

Like all normal distributions, a distribution of sample means (called a sampling distribution ) has its own mean and standard deviation. The mean of a sampling distribution (the “mean of the means”) is equal to the mean of the population. In an infi nite number of samples, some will have means larger than the population mean and some will have means smaller than the population mean ( Figure 11.3 ). These data tend to neutralize each other, resulting in an overall average that is equal to the mean of the population. Consider an example. Suppose you have a population of only three scores—1, 2, 3. The mean

DISTRIBUTION OF SAMPLE MEANS

All of this might suggest that it is impossible to for- mulate any rules that researchers can use to determine similarities between samples and populations. Not so. Fortunately, large collections of random samples do pattern themselves in such a way that it is possible for researchers to predict accurately some character- istics of the population from which the sample was selected.

Were we able to select an infi nite number of random samples (all of the same size) from a pop- ulation, calculate the mean of each, and then ar- range these means into a frequency polygon, we would fi nd that they shaped themselves into a fa- miliar pattern. The means of a large number of random samples tend to be normally distributed, unless the size of each of the samples is small (less than 30) and the scores in the population are not

Figure 11.2 Sampling Error

Population of 100 adult women in the United States

Sample Sample Sample

Sample

Notice that none of the samples is exactly like the population. This difference is what is known as sampling error.

fra97851_ch11_219-244.indd 222fra97851_ch11_219-244.indd 222 12/21/10 6:49 PM12/21/10 6:49 PM

C H A P T E R 1 1 Inferential Statistics 223

fall between 62 SEM; and 99.7 percent fall between 63 SEM ( Figure 11.4 ).

Thus, if we know or can accurately estimate the mean and the standard deviation of the sampling distribution, we can determine whether it is likely or unlikely that a particular sample mean could be obtained from that population. Suppose the mean of a population is 100, for example, and the standard error of the mean is 10. A sample mean of 110 would fall at 11 SEM, a sample mean of 120 would fall at 12 SEM, a sample mean of 130 would fall at 13 SEM; and so forth.

It would be very unlikely to draw a sample from the population whose mean fell above 13 SEM. Why? Because, as in all normal distributions (and remember, the sampling distribution is a normal distribution—of means ), only 0.0013 of all values (in this case, sample

of this population is 2. Now, take all of the possible types of samples of size two. How many would there be? Nine—(1, 1); (1, 2); (1, 3); (2, 1); (2, 2); (2, 3); (3, 1); (3, 2); (3, 3). The means of these samples are 1, 1.5, 2, 1.5, 2, 2.5, 2, 2.5, and 3, respectively. Add up all these means and divide by nine (that is, 18 4 9), and you see that the mean of these means equals 2, the same as the population mean.

STANDARD ERROR OF THE MEAN

The standard deviation of a sampling distribution of means is called the standard error of the mean (SEM) . As in all normal distributions, therefore, the 68-95-99.7 rule holds: approximately 68 percent of the sample means fall between 61 SEM; approximately 95 percent

Sample A Mean of sample A = 68

Mean of sample B = 73

Mean of sample C = 82

Mean of sample D = 73

Mean of sample E = 77

Population (mean = 70) Illustrative samples each

of same size (e.g., 60) drawn from this population

Sample B

Sample C

Sample D

Sample E

55

Frequency of

occurrence

Normal curve

60 65 70 75 80 85

Figure 11.3 A Sampling Distribution of Means

fra97851_ch11_219-244.indd 223fra97851_ch11_219-244.indd 223 12/21/10 6:49 PM12/21/10 6:49 PM

224 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

fortunately it can be estimated† using the standard de- viation of the sample. To calculate the SEM, then, sim- ply divide the standard deviation of the sample by the square root of the sample size minus one:

SEM 5 SD ______ √

_____ n 2 1

Let’s review the basic ideas we have presented so far.

1. The sampling distribution of the mean (or any des- criptive statistic) is the distribution of the means (or other statistic) obtained (theoretically) from an infi - nitely large number of samples of the same size.

2. The shape of the sampling distribution in many (but not all) cases is the shape of the normal distribution.

3. The SEM (standard error of the mean)—that is, the standard deviation of a sampling distribution of means—can be estimated by dividing the standard deviation of the sample by the square root of the sample size minus one.

4. The frequency with which a particular sample mean will occur can be estimated by using z scores based on sample data to indicate its position in the sam- pling distribution.

CONFIDENCE INTERVALS

We now can use the SEM to indicate boundaries, or limits, within which the population mean lies. Such boundaries are called confidence intervals . How are they determined?

Let us return to the example of the researcher who administered an IQ test to a sample of 65 elementary

means) fall above 13 SEM. It would not be unusual to select a sample from this population and fi nd that its mean is 105, but selecting a sample with a mean of 130 would be unlikely—very unlikely!

It is possible to use z scores to describe the position of any particular sample mean within a distribution of sample means. We discussed z scores in Chapter 10. We now want to express means as z scores. Remember that a z score simply states how far a score (or mean) differs from the mean of scores (or means) in standard devia- tion units. The z score tells a researcher exactly where a particular sample mean is located relative to all other sample means that could have been obtained. For ex- ample, a z score of 12 would indicate that a particular sample mean is two standard errors above the popula- tion mean. Only about 2 percent of all sample means fall above a z score of 12. Hence, a sample with such a mean would be unusual.

Estimating the Standard Error of the Mean. How do we obtain the standard error of the mean? Clearly, we cannot calculate it directly, since we would need, literally, to obtain a huge number of samples and their means.* Statisticians have shown, however, that the standard error can be calculated using a simple formula requiring the standard deviation of the population and the size of the sample. Although we seldom know the standard deviation of the population,

–3 SEM –3 z

+1 SEM +1 z

–2 SEM –2 z

+2 SEM +2 z

–1 SEM –1 z

+3 SEM +3 z

Population mean

Less likely Less likely

Unlikely Unlikely

.0013 Extremely unlikely

.0013 Extremely unlikely

Likely occurrence

.3413.1359 .1359.3413 .0215 .0215

99+%

95%

68%

Figure 11.4 Distribution of Sample Means

*If we did have these means, we would calculate the standard error just like any other standard deviation, treating each mean as a score. †The fact that the standard error is based on an estimated value rather than a known value does introduce an unknown degree of im- precision into this process.

fra97851_ch11_219-244.indd 224fra97851_ch11_219-244.indd 224 12/21/10 6:49 PM12/21/10 6:49 PM

C H A P T E R 1 1 Inferential Statistics 225

falls.‡ We could be mistaken, of course—the population mean could lie outside these intervals—but it is not very likely.§

school students. You will recall that she obtained a sample mean of 85 and wanted to know how much the population mean might differ from this value. We are now in a position to give her some help in this regard.

Let us assume that we have calculated the estimated standard error of the mean for her sample and found it to equal 2.0. Applying this to a sampling distribution of means, we can say that 95 percent of the time the popula- tion mean will be between 85 6 1.96 (2) 5 85 6 3.92 5 81.08 to 88.92. Why 6 1.96? Because the area between 61.96 z equals 95 percent (.95) of the total area under the normal curve.* This is shown in Figure 11.5.†

Suppose this researcher then wished to establish an interval that would give her more confi dence than p 5 .95 in making a statement about the population mean. This can be done by calculating the 99 percent confi dence interval. The 99 percent confi dence interval is deter- mined in a manner similar to that for determining the 95 percent confi dence interval. Given the characteristics of a normal distribution, we know that 0.5 percent of the sample means will lie below 22.58 SEM and another 0.5 percent will lie above 12.58 SEM (see Figure 10.12 in Chapter 10). Using the previous example, in which the mean of the sample was 85 and the SEM was 2, we calculate the interval as follows: 85 6 2.58 (SEM) 5 85 6 2.58(2.0) 5 85 6 5.16 5 79.84 to 90.16. Thus the 99 confi dence interval lies between 79.84 and 90.16, as shown in Figure 11.6 .

Our researcher can now answer her question about approximately how much the population mean dif- fers from the sample mean. While she cannot know exactly what the population mean is, she can indicate the “boundaries” or limits within which it is likely to fall ( Figure 11.7 ). To repeat, these limits are called confi dence intervals. The 95 percent confi dence inter- val spans a segment on the horizontal axis that we are 95 percent certain contains the population mean. The 99 percent confi dence interval spans a segment on the horizontal axis within which we are even more certain (99 percent certain) that the population mean

Figure 11.5 The 95 Percent Confi dence Interval

81.08 –1.96 SEM

–1.96 z

88.92 +1.96 SEM

+1.96 z 85

Sample mean

95%

79.84 –2.58 SEM

90.16 +2.58 SEM

85 Sample mean

99%

Figure 11.6 The 99 Percent Confi dence Interval

* By looking at the normal curve table in Appendix B, we see that the area between the mean and 1.96 z 5 .4750. Multiplied by 2, this equals .95, or 95 percent, of the total area under the curve. † Strictly speaking, it is not proper to consider a distribution of population means around the sample mean. In practice, we interpret confi dence intervals in this way. The legitimacy of doing so requires a demonstration beyond the level of an introductory text.

‡ Notice that it is not correct to say that the population mean falls within the 95 percent confi dence interval 95 times out of 100. The population mean is a fi xed value, and it either does or does not fall within this interval. The correct way to think of a confi dence inter- val is to view it in terms of replicating the study. Suppose we repli- cated the study with another sample and calculated the 95 percent confi dence interval for that sample. Suppose we then replicated the study once again with a third sample and calculated the 95 percent confi dence interval for this third sample. We would continue until we had drawn 100 samples and calculated the 95 percent confi - dence interval for each of these 100 samples. We would fi nd that the population mean lay within 95 percent of these intervals. §The likelihood of the population mean being outside the 95 percent confi dence interval is only 5 percent, and that of being outside the 99 percent confi dence interval only 1 percent. Analogous reasoning and procedures can be used with sample sizes less than 30.

fra97851_ch11_219-244.indd 225fra97851_ch11_219-244.indd 225 12/21/10 6:49 PM12/21/10 6:49 PM

226 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

difference in attitude between fourth-grade boys and girls in mathematics; whether there is a difference in achievement between students taught by the discussion method as compared to the lecture method; and so forth.

Our previous logic also applies to a difference be- tween means. For example, if a difference between means is found between the test scores of two samples in a study, a researcher wants to know if a difference exists in the populations from which the two samples were selected ( Figure 11.8 ). In essence, we ask the same

CONFIDENCE INTERVALS AND PROBABILITY Let us return to the concept of probability introduced in Chapter 10. As we use the term here, probability is nothing more than predicted relative occurrence, or relative frequency. When we say that something would occur 5 times in 100, we are expressing a probability. We could just as well say the probability is 5 in 100. In our earlier example, we can say, therefore, that the probability of the population mean being outside the 81.08–88.92 limits (the 95 percent confi dence interval) is only 5 in 100. The probability of it being outside the 79.84–90.16 limits (the 99 percent confi dence interval) is even less—only 1 in 100. Remember that it is custom- ary to express probabilities in decimal form, e.g., p 5 .05 or p 5 .01. What would p 5 .10 signify? *

COMPARING MORE THAN ONE SAMPLE

Up to this point, we have been explaining how to make inferences about the population mean using data from just one sample. More typically, however, research- ers want to compare two or more samples. For exam- ple, a researcher might want to determine if there is a

“That’s true, but because our sample was randomly selected, and the

standard error is 0.3, we can be pretty sure— in fact, 99 percent confident—that the average years of schooling for the whole population

of the country is between 12.8 and 14.6 years!”

“You say the average years of schooling for people

over 30 in the U.S. is 13.7 years? But that’s based on a sample. You

don’t know what it would be if you had data

on everyone!”

Figure 11.7 We Can Be 99 Percent Confi dent

Population Mean = ??

Sample A Mean = 25

Sample B Mean = 22

Population Mean = ??

Figure 11.8 Does a Sample Difference Refl ect a Population Difference?a aQuestion: Does the three-point difference between the means of sample A and sample B refl ect a difference between the means of population A and population B?*A probability of 10 in 100.

fra97851_ch11_219-244.indd 226fra97851_ch11_219-244.indd 226 12/21/10 6:49 PM12/21/10 6:49 PM

C H A P T E R 1 1 Inferential Statistics 227

the difference between the means of the two popula- tions is between 8 and 20 (62 SED), and better than a 99 per cent chance of being right if we say the difference between the means of the two populations is between 5 and 23 (63 SED). Figure 11.10 illustrates these confi dence intervals.

Suppose the difference between two other sample means is 12. If we calculated the SED to be 2, would it be likely or unlikely for the difference between popula- tion means to fall between 10 and 14?*

question we asked about one mean, only this time we ask it about a difference between means. Hence we ask, “Is the difference we have found a likely or an unlikely occurrence?” It is possible that the difference can be attributed simply to sampling error—to the fact that certain samples, rather than others, were selected (the “luck of the draw,” so to speak). Once again, inferential statistics help us out.

THE STANDARD ERROR OF THE DIFFERENCE BETWEEN SAMPLE MEANS

Fortunately, differences between sample means are also likely to be normally distributed. The distribution of differences between sample means also has its own mean and standard deviation. The mean of the sampling distribution of differences between sample means is equal to the difference between the means of the two populations. The standard deviation of this distribution is called the standard error of the difference (SED) . The formula for computing the SED is:

SED 5 √ _________________

s SEM1 d 2 1 s SEM2 d 2

where 1 and 2 refer to the respective samples. Because the distribution is normal, slightly more

than 68 percent of the differences between sample means will fall between 61 SED (again, remem- ber that the standard error of the difference is a stan- dard deviation); about 95 percent of the differences between sample means will fall between 62 SED, and 991 percent of these differences will fall between 63 SED ( Figure 11.9 ).

Now we can proceed similarly to the way we did with individual sample means. A researcher estimates the standard error of the difference between means and then uses it, along with the difference between the two sample means and the normal curve, to estimate prob- able limits (confi dence intervals) within which the dif- ference between the means of the two populations is likely to fall.

Let us consider an example. Imagine that the differ- ence between two sample means is 14 raw score points and the calculated SED is 3. Just as we did with one sample population mean, we can now indicate limits within which the difference between the means of the two populations is likely to fall. If we say that the dif- ference between the means of the two populations is be- tween 11 and 17 (61 SED), we have slightly more than a 68 percent chance of being right. We have somewhat more than a 95 percent chance of being right if we say

Figure 11.9 Distribution of the Difference Between Sample Means

–3 SED –2 SED –1 SED +3 SED+2 SED+1 SED

Difference between the means

of the two populations

99+%

95%

68%

5 –3 SED

17 +1 SED

148 –2 SED

20 +2 SED

11 –1 SED

23 +3 SED

Obtained difference between sample means

99+%

95%

68%

Figure 11.10 Confi dence Intervals

*Likely, since 68 percent of the differences between population means fall between these values.

fra97851_ch11_219-244.indd 227fra97851_ch11_219-244.indd 227 12/21/10 6:50 PM12/21/10 6:50 PM

228 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

“There is no difference between the population mean of students using method A and the population mean of students using method B.” (This is the same thing as saying the difference between the means of the two populations is zero.) Figure 11.11 offers a comparison of research and null hypotheses.

The researcher then proceeds to test the null hypoth- esis. The same information is needed as before: the knowledge that the sampling distribution is normal, and the calculated standard error of the difference (SED). What is different in a hypothesis test is that instead of using the obtained sample value (e.g., the obtained dif- ference between sample means) as the mean of the sam- pling distribution (as we did with confi dence intervals), we use zero. *

We then can determine the probability of obtaining a particular sample value (such as an obtained difference between sample means) by seeing where such a value falls on the sampling distribution. If the probability is small, the null hypothesis is rejected, thereby providing support for the research hypothesis. The results are said to be statistically signifi cant.

What counts as “small”? In other words, what consti- tutes an unlikely outcome? Probably you have guessed. It is customary in educational research to view as un- likely any outcome that has a probability of .05 ( p 5 .05) or less. This is referred to as the .05 level of signifi- cance . When we reject a null hypothesis at the .05 level, we are saying that the probability of obtaining such an outcome is only 5 times (or less) in 100. Some research- ers prefer to be even more stringent and choose a .01 level of signifi cance. When a null hypothesis is rejected at the .01 level, it means that the likelihood of obtaining the outcome is only 1 time (or less) in 100.

HYPOTHESIS TESTING: A REVIEW

Let us review what we have said. The logical sequence for a researcher who wishes to engage in hypothesis testing is as follows:

1. State the research hypothesis (e.g., “There is a dif- ference between the population mean of students using method A and the population mean of students using method B”).

2. State the null hypothesis (e.g., “There is no differ- ence between the population mean of students using

Hypothesis Testing How does all this apply to research questions and re- search hypotheses? You will recall that many hypoth- eses predict a relationship. In Chapter 10, we presented techniques for examining data for the existence of rela- tionships. We pointed out in previous chapters that vir- tually all relationships in data can be examined through one (or more) of three procedures: a comparison of means, a correlation, or a crossbreak table. In each in- stance, some degree of relationship may be found. If a relationship is found in the data, is there likely to be a similar relationship in the population, or is it simply due to sampling error—to the fact that a particular sample, rather than another, was selected for study? Once again, inferential statistics can help.

The logic discussed earlier applies to any particular form of a hypothesis and to many procedures used to examine data. Thus, correlation coeffi cients and differ- ences between them can be evaluated in essentially the same way as means and differences between means; we just need to obtain the standard error of the correla- tion coeffi cient(s). The procedure used with crossbreak tables differs in technique, but the logic is the same. We will discuss it later in the chapter.

When testing hypotheses, it is customary to proceed in a slightly different way. Instead of determining the boundaries within which the population mean (or other parameter) can be said to fall, a researcher determines the likelihood of obtaining a sample value (for example, a dif- ference between two sample means) if there is no relation- ship (that is, no difference between the means of the two populations) in the populations from which the samples were drawn. The researcher formulates both a research hypothesis and a null hypothesis. To test the research hy- pothesis, the researcher must formulate a null hypothesis.

THE NULL HYPOTHESIS

As you will recall, the research hypothesis specifi es the predicted outcome of a study. Many research hy- potheses predict the nature of the relationship the re- searcher thinks exists in the population; for example: “The population mean of students using method A is greater than the population mean of students using method B.”

The null hypothesis most commonly used specifi es there is no relationship in the population; for example:

*Actually, any value could be used, but zero is used in virtually all educational research.

fra97851_ch11_219-244.indd 228fra97851_ch11_219-244.indd 228 12/21/10 6:50 PM12/21/10 6:50 PM

C H A P T E R 1 1 Inferential Statistics 229

Figure 11.11 Null and Research Hypotheses

“You know, I read recently that most researchers

believe there really is no difference between the average score of students taught mathematics

by the ’new’ math and the average score of those students taught by the traditional

lecture/discussion approach.”

“I don’t believe it. Let’s compare a sample of our

students this year and see. I bet that the students taught by the new

math will do better.”

(Null Hypothesis) (Research Hypothesis)

method A and the population mean of students using method B,” or “The difference between the two pop- ulation means is zero”).

3. Determine the sample statistics pertinent to the hypothesis (e.g., the mean of sample A and the mean of sample B).

4. Determine the probability of obtaining the sample results (i.e., the difference between the mean of sam- ple A and the mean of sample B) if the null hypoth- esis is true.

5. If the probability is small, reject the null hypothesis, thus affi rming the research hypothesis.

6. If the probability is large, do not reject the null hypothesis, which means you cannot affi rm the re- search hypothesis.

Let us use our previous example in which the dif- ference between sample means was 14 points and the SED was 3 (see Figure 11.10 ). In Figure 11.12 we see that the sample difference of 14 falls far beyond 13 SED; in fact, it exceeds 4 SED. Thus, the prob- ability of obtaining such a sample result is consider- ably less than .01, and as a result, the null hypothesis is rejected. If the difference in sample means had been 4 instead of 14, would the null hypothesis be rejected?*

*No. The probability of a difference of 4 is too high—much larger than .05.

fra97851_ch11_219-244.indd 229fra97851_ch11_219-244.indd 229 12/21/10 6:50 PM12/21/10 6:50 PM

230 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

than the students exposed to method B (the older, more traditional method), and that this difference is statisti- cally signifi cant. We doubt that the mathematics depart- ment would immediately encourage all of its members to adopt method A on the basis of this two-point differ- ence. Would you?

Ironically, the fact that most educational studies in- volve smaller samples may actually be an advantage when it comes to practical signifi cance. Because smaller sample size makes it harder to detect a difference even when there is one in the population, a larger difference in means is therefore required to reject the null hypoth- esis. This is so because a smaller sample results in a larger standard error of the difference in means (SED). Therefore, a larger difference in means is required to reach the signifi cance level (see Figure 11.10 ). It is also possible that relationships of potential practical sig- nifi cance may be overlooked or dismissed because they are not statistically signifi cant (more on this in the next chapter).

One should always take care in interpreting results— just because one brand of radios is signifi cantly more powerful than another brand statistically does not mean that those looking for a radio should rush to buy the fi rst brand.

ONE- AND TWO-TAILED TESTS

In Chapter 5, we made a distinction between directional and nondirectional hypotheses. There is sometimes an advantage to stating hypotheses in directional form that is related to signifi cance testing. We refer again to a hypothetical example of a sampling distribution of

Practical Versus Statistical Signifi cance The fact that a result is statistically signifi cant (not due to chance) does not mean that it has any practical or educational value in the real world in which we all work and live. Statistical significance only means that one’s results are likely to occur by chance less than a certain percentage of the time, say 5 percent. So what? Remem- ber that this only means the observed relationship most likely would not be zero in the population. But it does not mean necessarily that it is important! Whenever we have a large enough random sample, almost any result will turn out to be statistically signifi cant. Thus, a very small correlation coeffi cient, for example, may turn out to be statistically signifi cant but have little (if any) practical significance . In a similar sense, a very small difference in means may yield a statistically signifi cant result but have little educational import.

Consider a few examples. Suppose a random sample of 1,000 high school baseball pitchers on the East Coast reveals an average fastball speed of 75 mph, while a second random sample of 1,000 high school pitch- ers in the Midwest shows an average fastball speed of 71 mph. Now this difference of 4 mph might be sta- tistically signifi cant (due to the large sample size), but we doubt that baseball fans would say that it is of any practical importance (see Figure 11.13 ). Or suppose that a researcher tries out a new method of teaching math- ematics to high school juniors. She fi nds that those stu- dents exposed to method A (the new method) score, on the average, two points higher on the fi nal examination

Figure 11.12 Illustration of When a Researcher Would Reject the Null Hypothesis

–9 –3 SED

3 +1 SED

–6 –2 SED

6 +2 SED

–3 –1 SED

0 9 +3 SED

12 +4 SED

Obtained difference between sample means

Raw scores

14

fra97851_ch11_219-244.indd 230fra97851_ch11_219-244.indd 230 12/21/10 6:50 PM12/21/10 6:50 PM

C H A P T E R 1 1 Inferential Statistics 231

is not. Because the researchers’ hypothesis can be sup- ported only if he or she obtains a positive difference be- tween the sample means, the researcher is justifi ed in using only the positive tail of the sampling distribution to locate the obtained difference. This is referred to as a one-tailed test of statistical signifi cance ( Figure 11.14 ).

At the 5 percent level of signifi cance ( p 5 .05), the null hypothesis may be rejected only if the obtained difference between sample means reaches or exceeds

differences between means in which the calculated SED equals 3. Previously, we interpreted the statistical signif- icance of an obtained difference between sample means of 14 points. The statistical signifi cance of this differ- ence was quite clear-cut, since it was so large. Suppose, however, that the obtained difference was not 14, but 5.5 points. To determine the probability associated with this outcome, we must know whether the researcher’s hypothesis was a directional or a nondirectional one. If the hypothesis was directional, the researcher specifi ed ahead of time (before collecting any data) which group would have the higher mean (for example, “The mean score of students using method A will be higher than the mean score of students using method B.”).

If this had been the case, the researcher’s hypothesis would be supported only if the mean of sample A were higher than the mean of sample B. The researcher must decide beforehand that he or she will subtract the mean of sample B from the mean of sample A. A large differ- ence between sample means in the opposite direction would not support the research hypothesis. A difference between sample means of 12 is in the hypothesized di- rection, therefore, but a difference of 22 (should the mean of sample B be higher than the mean of sample A)

Figure 11.13 How Much Is Enough?

“I think we can say that my guy’s fastball is pretty impressive

—in fact, you might say it is significantly faster than

your guy’s.”

“Are you kidding? Your guy has been clocked at

79 mph, and mine at 77 mph. You call a difference of 2 mph

significant?”

Figure 11.14 Signifi cance Area for a One-Tailed Test

–2 z –1 z 0

Shaded area represents 5% of area

under the curve

+1 z +2 z

1.65 z

fra97851_ch11_219-244.indd 231fra97851_ch11_219-244.indd 231 12/21/10 6:50 PM12/21/10 6:50 PM

232 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

USE OF THE NULL HYPOTHESIS: AN EVALUATION

There appears to be much misunderstanding regarding the use of the null hypothesis. First, it often is stated in place of a research hypothesis. While it is easy to replace a research hypothesis (which predicts a rela- tionship) with a null hypothesis (which predicts no re- lationship), there is no good reason for doing so. As we have seen, the null hypothesis is merely a useful meth- odological device.

Second, there is nothing sacred about the customary .05 and .01 signifi cance levels—they are merely conven- tions. It is a little ridiculous, for example, to fail to reject the null hypothesis with a sample value that has a prob- ability of .06. To do so might very well result in what is known as a Type II error —this error results when a researcher fails to reject a null hypothesis that is false. A Type I error , on the other hand, results when a researcher rejects a null hypothesis that is true. In our example in which there was a 14-point difference between sample means, for example, we rejected the null hypothesis at the .05 level. In doing so, we realized that a 5 percent chance remained of being wrong—that is, a 5  percent

1.65 SED* in the one tail. As shown in Figure 11.15 , this requires a difference between sample means of 5 points or more.† Our previously obtained difference of 5.5 would be signifi cant at this level, therefore, since it not only reaches, but exceeds, 1.65 SED.

What if the hypothesis were nondirectional? If this were the case, the researcher would not have specifi ed beforehand which group would have the higher mean. The hypothesis would then be supported by a suitable difference in either tail. This is called a two-tailed test of statistical signifi cance. If a researcher uses the .05 level of signifi cance, this requires that the 5 percent of the total area must include both tails—that is, there is 2.5 percent in each tail. As a result, a difference in sam- ple means of almost 6 points (either 15.88 or –5.88) is required to reject the null hypothesis ( Figure 11.16 ), since 1.96(3) 5 5.88.

*By looking in the normal curve table in Appendix B, we see that the area beyond 1.65 z equals .05, or 5 percent of the area under the curve. †Since an area of .05 in one tail equals a z of 1.65, and the SED is 3, we multiply 1.65(3) to fi nd the score value at this point: 1.65(3) 5 4.95, or 5 points.

Figure 11.15 One- Tailed Test Using a Distribution of Differences Between Sample Means

0

1 SED

.05

SED = 3

All obtained differences between sample means that fall in this shaded area (i.e., reach 1.65 z

or beyond) would cause a researcher to reject the

null hypothesis.

3 +1 z

5

5.5

1.65 z

Figure 11.16 Two- Tailed Test Using a Distribution of Differences Between Sample Means

–5.88 –1.96 z

–3 –1 z

3 +1 z

5.88 +1.96 z

0

.025

.025

1 SED

SED = 3

All obtained differences between sample means that fall in this shaded area (i.e., reach either

+1.96 z or –1.96 z or beyond) would cause a researcher to reject the null hypothesis.

1 SED

fra97851_ch11_219-244.indd 232fra97851_ch11_219-244.indd 232 12/21/10 6:50 PM12/21/10 6:50 PM

C H A P T E R 1 1 Inferential Statistics 233

kinds of assumptions about the nature of the population from which the sample(s) involved in the research study are drawn. Nonparametric techniques , on the other hand, make few (if any) assumptions about the nature of the population from which the samples are taken. An advantage of parametric techniques is that they are gen- erally more powerful * than nonparametric techniques and hence much more likely to reveal a true difference or relationship if one really exists. Their disadvantage is that often a researcher cannot satisfy the assump- tions they require (for example, that the population is normally distributed on the characteristic of interest). The advantage of nonparametric techniques is that they are safer to use when a researcher cannot satisfy the as- sumptions underlying the use of parametric techniques.

PARAMETRIC TECHNIQUES FOR ANALYZING QUANTITATIVE DATA†

The t -Test for Means. The t -test is a paramet- ric statistical test used to see whether a difference be- tween the means of two samples is signifi cant. The test

chance that the null hypothesis was true. Figure  11.17 provides an example of Type I and Type II errors.

Finally, there is also nothing sacrosanct about testing an obtained result against zero. In our previous example, for instance, why not test the obtained value of 14 (or 5.5, etc.) against a hypothetical population difference of 1 (or 3, etc.)? Testing only against zero can mislead one into exaggerating the importance of the obtained relationship. We believe the reporting of inferential statistics should rely more on confi dence intervals and less on whether a particular level of signifi cance has been attained.

Inference Techniques It is beyond the scope of this text to treat in detail each of the many techniques that exist for answering infer- ence questions about data. We shall, however, present a brief summary of the more commonly used tests of statistical signifi cance that researchers employ and then illustrate how to do one such test.

In Chapter 10, we made a distinction between quanti- tative and categorical data. We pointed out that the type of data a researcher collects often infl uences the type of statistical analysis required. A statistical technique ap- propriate for quantitative data, for example, will gener- ally be inappropriate for categorical data.

There are two basic types of inference techniques that researchers use. Parametric techniques make various

Figure 11.17 A Hypothetical Example of Type I and Type II Errors

Doctor says that symptoms like Susie’s occur only 5 percent of the time in healthy people. To be safe, however, he decides to treat Susie for pneumonia.

Doctor says that symptoms like Susie’s occur 95 percent of the time in healthy people. In his judgment, therefore, her symptoms are a false alarm and do not warrant treatment, and he decides not to treat Susie for pneumonia.

Doctor is correct. Susie does have pneumonia and the treatment cures her.

Susie has pneumonia. Susie does not

have pneumonia.

Doctor is wrong. Susie is not treated and may suffer serious consequences. Type II error (�)

Doctor is wrong. Susie’s treatment was unnecessary and possibly unpleasant and expensive. Type I error (�)

Doctor is correct. Unnecessary treatment is avoided.

*We discuss the power of a statistical test later in this chapter. †Many texts distinguish between techniques appropriate for nominal, ordinal, and interval scales of measurement (see Chapter 7). It turns out that in most cases parametric techniques are most appropriate for interval data, while nonparametric techniques are more appropriate for ordinal and nominal data. Researchers rarely know for certain whether their data justify the assumption that interval scales have actually been used.

fra97851_ch11_219-244.indd 233fra97851_ch11_219-244.indd 233 12/21/10 6:50 PM12/21/10 6:50 PM

She fi nds that the mean score on the achievement test of the students who were taught by method A equals 85 and that the mean score of the students taught by method B equals 80. Obviously, these two sample means are dif- ferent. However, this difference could be due to chance (i.e., sampling error). The key question is whether these means are different enough that our researcher can con- clude that the difference is most likely not due to chance but actually to the difference between the two teaching methods.

To evaluate her research hypothesis, our researcher conducts a one-tailed t -test for independent samples and fi nds that the t -value (the test statistic) is 2.18. To be statistically signifi cant at the .05 level, with 60 de- grees of freedom ( df ), a t -value of at least 1.67 is re- quired, since the test is a one-tailed test. Because the obtained t -value of 2.18 is beyond 1.67, it is an unlikely value, and hence the null hypothesis is rejected. Our researcher concludes that the difference between the two means is statistically signifi cant—that is, it was not merely a chance occurrence but indeed represents a real difference between the achievement scores of the two groups.

Let us say a word about the concept degrees of free- dom ( df ) , which we referred to in the example above.

produces a value for t (called an obtained t ), which the researcher then checks in a statistical table (similar to the one shown in Appendix B) to determine the level of signifi cance that has been reached. As we mentioned earlier, if the .05 level of signifi cance is reached, the researcher customarily rejects the null hypothesis and concludes that a real difference does exist.

There are two forms of this t -test, a t -test for inde- pendent means and a t -test for correlated means. The t -test for independent means is used to compare the mean scores of two different, or independent, groups. For example, if two randomly selected groups of eighth- graders (31 in each group) were exposed to two differ- ent methods of teaching for a semester and then given the same achievement test at the end of the semester, their achievement scores could be compared using a t - test. Let us assume that the researcher suspected that those students exposed to method A would score sig- nifi cantly higher on the end-of-semester achievement test. Accordingly, she formulated the following null and alternative hypotheses:

Null hypothesis : population mean of method A 5 population mean of method B

Research hypothesis : population mean of method A . population mean of method B

Sample Size

Students frequently ask for more specifi c rules on sample size. Unfortunately, there are no simple answers. How- ever, under certain conditions, some guidelines are available. The most important condition is random sampling, but there are other specifi c requirements that are discussed in statistics texts. Assuming these assumptions are met, the following apply:

Top table: Sample size required for concluding that a sample correlation coeffi cient is statistically signifi cant (i.e., different from zero in the population) at the .05 level of confi dence.

Bottom table: Sample size required for concluding that a difference in sample means is statistically signifi cant (i.e., the difference between the means of the two populations is not zero) at the .05 level of confi dence. These calculations require that the population standard deviation be known or estimated from the sample standard deviations. Let us assume, for ex- ample, that the standard deviation in both populations is 15 and each of the samples is the same size.

Value of sample r .05 .10 .15 .20 .25 .30 .40 .50

Sample size required 1,539 400 177 100 64 49 25 16

Difference between 2 5 10 15 the sample means points points points points

Required size 434 71 18 8 of each sample

RESEARCH TIPS

234

fra97851_ch11_219-244.indd 234fra97851_ch11_219-244.indd 234 12/21/10 6:50 PM12/21/10 6:50 PM

C H A P T E R 1 1 Inferential Statistics 235

Thus, should a 5 3 and b 5 5, c must equal 2. Hence, we say that there are two degrees of freedom in this dis- tribution; any two of the values are “free to vary,” so to speak, but once they are set, the third is also fi xed. Degrees of freedom are calculated in an independent samples t- test by subtracting 2 from the total number of values in both groups.* In this example, there are two

This concept is quite important in many inferential sta- tistics tests. In essence, it refers to the number of scores in a frequency distribution that are “free to vary”—that is, that are not fi xed. For example, suppose you had a distribution of only three scores, a , b , and c , that must add up to 10. It is apparent that a , b , and c can have a number of different values (such as 3, 5, and 2; or 1, 6, and 3; or 2, 2, and 6) and still add up to 10. But once any two of these values are fi xed, or set, the third value is also set; it cannot vary.

Using Excel to Draw a Random Sample

Excel can also be used to draw a random sample from a population. Here is how to do it. First, list your dataset in any of the columns on an Excel worksheet. Under the Tools menu, click on Data Analysis, and then on Sampling. In the sampling dialogue box, indicate the array of cells from which you wish to draw the sample. Under “Type of Sample,” click Random, and under “Number of Samples,” type in how many numbers you wish to have in the sample. Then click on the cells where you want the numbers in the random sample to appear.

Here is an example. Following the steps described above, we asked for a random sample of 15 numbers to be selected from the list of 100 numbers we had listed in cells A1:E20. The num- bers shown in cells G1:G15 were selected.

Population of 100 Scores Row A B C D E F G 1 67 65 33 98 9 55 2 74 71 36 87 14 67 3 92 92 69 85 51 85 4 77 80 65 57 32 32 5 88 86 58 59 87 73 6 81 78 57 94 98 9 7 73 70 45 91 68 57 8 72 70 45 28 52 65 9 71 78 41 27 21 11 10 90 89 12 65 30 55 11 89 93 11 38 38 67 12 95 96 10 55 37 92 13 80 82 9 56 59 7 14 70 73 58 66 58 46 15 84 85 57 88 46 37 16 67 65 41 71 60 17 45 73 12 7 80 18 55 26 13 40 11 19 62 33 28 32 12 20 12 11 29 31 29

*In a study with only one group, you would subtract 1 from the total value of the group.

Random sample

fra97851_ch11_219-244.indd 235fra97851_ch11_219-244.indd 235 12/21/10 6:50 PM12/21/10 6:50 PM

236 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

F value. As in a t -test, this F value is then checked in a statistical table to see if it is statistically signifi cant. It is interpreted quite similarly to the t -value, in that the larger the obtained value of F , the greater the likelihood that statistical signifi cance exists.

For example, imagine that a researcher wishes to in- vestigate the effectiveness of three drugs used to relieve headaches. She randomly selects three groups of indi- viduals who routinely suffer from headaches (with 20 in each group) and formulates the following hypotheses:

Null hypothesis: There is no difference between groups.

Research hypothesis: There is a difference between groups.

The researcher obtains an F value of 3.95. The critical region for the corresponding degrees of freedom at the .05 level, for a two-tailed test, is 3.17. Therefore, she would reject the null hypothesis and conclude that the effectiveness of the three drugs is not the same.

When only two groups are being compared, the F test is suffi cient to tell the researcher whether signifi - cance has been achieved. When more than two groups are being compared, the F test will not, by itself, tell us which of the means are different. A further (but quite simple) procedure, called a post hoc analysis, is required to fi nd this out. ANOVA is also used when more than one independent variable is investigated, as in factorial designs, which we discuss in Chapter 13.

Analysis of Covariance. Analysis of covariance (ANCOVA) is a variation of ANOVA used when, for example, groups are given a pretest related in some way to the dependent variable and their mean scores on this pretest are found to differ. ANCOVA enables the researcher to adjust the posttest mean scores on the dependent variable for each group to compen- sate for the initial differences between the groups on the pretest. The pretest is called the covariate. How much the posttest mean scores must be adjusted de- pends on how large the difference between the pre- test means is and the degree of relationship between the covariate and the dependent variable. Several co- variates can be used in an ANCOVA test, so in ad- dition to (or instead of) adjusting for a pretest, the researcher can adjust for the effect of other variables. (We discuss this further in Chapter 13). Like ANOVA, ANCOVA produces an F value, which is then looked up in a statistical table to determine whether it is sta- tistically signifi cant.

groups, with 31 students in each group; hence there are 30 values in each group that are free to vary, resulting in a total of 60 degrees of freedom.

The t- test for correlated means is used to compare the mean scores of the same group before and after a treatment of some sort is given, to see if any observed gain is signifi cant, or when the research design involves two matched groups. It is also used when the same sub- jects receive two different treatments in a study.

Consider an example. Suppose a sports psycholo- gist believes that anxiety often makes basketball play- ers perform poorly when shooting free throws in close games. She decides to investigate the effectiveness of relaxation training for reducing the level of anxiety such athletes experience and thus improving their per- formance at the free throw line. She formulates these hypotheses:

Null hypothesis: There will be no change in perfor- mance at the free throw line.

Research hypothesis: Performance at the free throw line will improve.

She randomly selects a sample of 15 athletes to un- dergo the training. During the week before the training sessions (the treatment), she measures the athletes’ level of anxiety. She then exposes the athletes to the treat- ment. For a week afterward, she again measures the athletes’ level of anxiety.

She fi nds that the mean number of free throws com- pleted equals fi ve per game before the training and seven after the training has been completed. She conducts a t -test for repeated measures and obtains a t - statistic of 2.43. Statistical signifi cance at the .05 level, for a one-tailed test, with 14 degrees of freedom, requires a t - statistic of at least 1.76. Since the obtained t -value is more than that, the researcher concludes that such a result is unusual (i.e., unlikely to be a chance result); therefore, she rejects the null hypothesis. She concludes that relaxation therapy can reduce the anxiety level of these athletes.

Analysis of Variance. When researchers desire to fi nd out whether there are signifi cant differences be- tween the means of more than two groups, they com- monly use a technique called analysis of variance (ANOVA) , which is actually a more general form of the t -test that is appropriate to use with three or more groups. (It can also be used with two groups.) In brief, variation both within and between each of the groups is analyzed statistically, yielding what is known as an

fra97851_ch11_219-244.indd 236fra97851_ch11_219-244.indd 236 12/21/10 6:50 PM12/21/10 6:50 PM

C H A P T E R 1 1 Inferential Statistics 237

parent populations are identical, then the sum of the pooled rankings for each group should be about the same. If the summed ranks are markedly different, on the other hand, then this difference is likely to be statistically signifi cant.

The Kruskal-Wallis One-Way Analysis of Variance. The Kruskal-Wallis one-way analysis of variance is used when researchers have more than two independent groups to compare. The procedure is quite similar to the Mann-Whitney U test. The scores of the individuals in the several groups are pooled and then ranked as though they all came from one group. The sums of the ranks added together for each of the sepa- rate groups are then compared. This analysis produces a value ( H ), whose probability of occurrence is checked by the researcher in the appropriate statistical table.

The Sign Test. The sign test is used when a researcher wants to analyze two related (as opposed to indepen- dent) samples. Related samples are connected in some way. For example, often a researcher will try to equal- ize groups on IQ, gender, age, or some other variable. The groups are matched, so to speak, on these variables. Another example of a related sample is when the same group is both pre- and posttested (that is, tested twice). Each individual, in other words, is tested on two different occasions (as with the t -test for correlated means).

This test is very easy to use. The researcher simply lines up the pairs of related subjects and then determines how many times the paired subjects in one group scored higher than those in the other group. If the groups do not differ signifi cantly, the totals for the two groups should be about equal. If there is a marked difference in scor- ing (such as many more in one group scoring higher), the difference may be statistically signifi cant. Again, the probability of this occurrence can be determined by consulting the appropriate statistical table.

The Friedman Two-Way Analysis of Vari- ance. If more than two related groups are involved, then the Friedman two-way analysis of variance test can be used. For example, if a researcher employs four matched groups, this test would be appropriate.

PARAMETRIC TECHNIQUES FOR ANALYZING CATEGORICAL DATA

t -Test for Proportions. The most commonly used parametric tests for analyzing categorical data are the t -tests for a difference in proportions —that is, whether

Multivariate Analysis of Variance. Multi- variate analysis of variance (MANOVA) differs from ANOVA in only one respect: It incorporates two or more dependent variables in the same analysis, thus permitting a more powerful test of differences among means. It is justifi ed only when the researcher has rea- son to believe correlations exist among the dependent variables. Similarly, multivariate analysis of covari- ance (MANCOVA) extends ANCOVA to include two or more dependent variables in the same analysis. The specifi c value that is calculated is Wilk’s lambda , a number analogous to F in analysis of variance.

The t -Test for r . The t -test for r is used to see whether a correlation coeffi cient calculated on sample data is signifi cant—that is, whether it represents a non- zero correlation in the population from which the sample was drawn. It is similar to the t -test for means, except that here the statistic being dealt with is a correlation coeffi - cient ( r ) rather than a difference between means. The test produces a value for t (again called an obtained t ), which the researcher checks in a statistical probability table to see whether it is statistically signifi cant. As with the other parametric tests, the larger the obtained value for t , the greater the likelihood that signifi cance has been achieved.

For example, a researcher is using a regular two-tailed test, with alpha 5 .05, to determine if a nonzero correla- tion exists in a particular population. He randomly selects a sample of 30 individuals. Checking the appropriate sta- tistical table (critical values for the Pearson correlation), he fi nds that with a 5 .05 and n 5 30 (28 df ), the table lists a critical value of 0.361. The sample correlation, therefore (independent of sign) must have a value equal to or greater than 0.361 for the researcher to reject the null hypothesis and conclude that there is a signifi cant corre- lation in the population. Any sample correlation between 0.361 and –0.361 would be considered likely (that is, due to sampling error) and hence not statistically signifi cant.

NONPARAMETRIC TECHNIQUES FOR ANALYZING QUANTITATIVE DATA

The Mann-Whitney U Test. The Mann- Whitney U test is a nonparametric alternative to the t -test used when a researcher wishes to analyze ranked data. The researcher intermingles the scores of the two groups and then ranks them as if they were all from just one group. The test produces a value ( U ), whose probability of occur- rence is then checked by the researcher in the appropriate statistical table. The logic of the test is as follows: If the

fra97851_ch11_219-244.indd 237fra97851_ch11_219-244.indd 237 12/21/10 6:50 PM12/21/10 6:50 PM

238 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

Thus, in this example, there are two degrees of freedom.

Contingency Coefficient. The fi nal step in the chi- square test process is to calculate the contingency coef- ficient , symbolized by the letter C, to which we referred in Chapter 10. It is a measure of the degree of associa- tion in a contingency table. We show how to calculate both the chi-square test and the contingency coeffi cient in Chapter 12.

The contingency coeffi cient cannot be interpreted in exactly the same way as the correlation coeffi cient. It must be interpreted by using Table 11.1 . This table gives the upper limit for C, depending on the number of cells in the crossbreak table.

SUMMARY OF TECHNIQUES The names of the most commonly used inferential pro- cedures and the data type appropriate to their use are summarized in Table 11.2 .

This summary should be useful to you whenever you encounter these terms in your reading. While the details of both mathematical rationale and calculation differ greatly among these techniques, the most important things to remember are as follows:

1. The end product of all inference procedures is the same: a statement of probability relating the sample data to hypothesized population characteristics.

2. All inference techniques assume random sampling. Without random sampling, the resulting probabili- ties are in error—to an unknown degree.

3. Inference techniques are intended to answer only one question: Given the sample data, what are prob- able population characteristics? These techniques do

the proportion in one category (e.g., males) is different from the proportion in another category (e.g., females). As is the case with the t- tests for means, there are two forms: one t -test for independent proportions and one t -test for correlated proportions . The latter is used pri- marily when the same group is being compared, as in the proportion of individuals agreeing with a statement be- fore and after receiving an intervention of some sort.

NONPARAMETRIC TECHNIQUES FOR ANALYZING CATEGORICAL DATA

The Chi-Square Test. The chi-square test is used to analyze data that are reported in categories. For ex- ample, a researcher might want to compare how many male and female teachers favor a new curriculum to be instituted in a particular school district. He asks a sample of 50 teachers if they favor or oppose the new curriculum. If they do not differ signifi cantly in their responses, he would expect that about the same propor- tion of males and females would be in favor of (or op- posed to) instituting the curriculum.

The chi-square test is based on a comparison between expected frequencies and actual, obtained frequencies. If the obtained frequencies are similar to the expected fre- quencies, then researchers conclude that the groups do not differ (in our example above, they do not differ in their at- titude toward the new curriculum). If there are considerable differences between the expected and obtained frequencies, on the other hand, then researchers conclude that there is a signifi cant difference in attitude between the two groups.

As with all of these inference techniques, the chi- square test yields a value (x 2 ). The chi-square test is not limited to comparing expected and obtained frequencies for only two variables. See Table 10.11 for an example.

After the value for x 2 has been calculated, we want to determine how likely it is that such a result could occur if there were no relationship in the population—that is, whether the obtained pattern of results does not exist in the population but occurred because of the particular sample that was selected. As with all inferential tests, we deter- mine this by consulting a probability table (Appendix C).

You will notice that the chi-square table in Appendix C also has a column headed “degrees of freedom.” Degrees of freedom are calculated in crossbreak tables as follows, using an example of a table with three rows and two columns.

Step 1: Subtract 1 from the number of rows: 3 – 1 5 2 Step 2: Subtract 1 from the number of columns:

2 – 1 5 1 Step 3: Multiply step 1 by step 2: (2)(1) 5 2

aThe upper limits for unequal-sized tables (such as 2 by 3 or 3 by 4) are unknown but can be estimated from the values given. Thus, the upper limit for a 3 by 4 table would approximate .85.

TABLE 11.1 Contingency Coeffi cient Values for Different-Sized Crossbreak Tables

Size of Table (No. of Cells)

Upper Limita for C Calculated

2 by 2 .71

3 by 3 .82

4 by 4 .87

5 by 5 .89

6 by 6 .91

fra97851_ch11_219-244.indd 238fra97851_ch11_219-244.indd 238 12/21/10 6:50 PM12/21/10 6:50 PM

C H A P T E R 1 1 Inferential Statistics 239

not help decide whether the data show results that are meaningful or useful—they indicate only the ex- tent to which they may be generalizable.

POWER OF A STATISTICAL TEST

The power of a statistical test is similar to the power of a telescope. Astronomers looking at Mars or Venus with a low-power telescope probably can see that these planets look like spheres, but it is unlikely that they can see much by way of differences. With high-power telescopes, however, they can see such differences as moons and mountains. When the purpose of a statistical test is to assess differences, power is the probability that the test will correctly lead to the conclusion that there is a difference when, in fact, a difference exists.

Suppose a football coach wants to study a new tech- nique for kicking fi eld goals. He asks a (random) sample of 30 high school players to each kick 30 goals. They take the same “test” after being coached in the new tech- nique. The mean number of goals is 11.2 before coach- ing and 18.8 after coaching—a difference of 7.6. The null hypothesis is that any positive difference (i.e., the number of goals after coaching minus the number of goals before coaching) is a chance difference from the true population difference of zero. A one-tailed t -test is the technique used to test for statistical signifi cance.

In this example, the critical value for rejecting the null hypothesis at the .05 level is calculated. Assume the critical value turns out to be 5.8. Any difference in means that is larger than 15.8 would result in rejection of the null hypothesis. Since our difference is 7.6, we therefore reject the null hypothesis. Figure 11.18 illus- trates this condition.

TABLE 11.2 Commonly Used Inferential Techniques

Parametric Nonparametric

Quantitative t-test for independent means Mann-Whitney U test

t-test for correlated means Kruskal-Wallis one-way analysis of variance

Analysis of variance (ANOVA) Sign test

Analysis of covariance (ANCOVA) Friedman two-way analysis of variance

Multivariate analysis of variance (MANOVA)

t-test for r

Categorical t-test for difference in proportions Chi square

Figure 11.18 Rejecting the Null Hypothesis

Null hypothesis = 0

Critical value = 5.8

Our difference of 7.6

Critical region = 5%

Figure 11.19 An Illustration of Power Under an Assumed Population Value

Critical value = 5.8

True value = 8.0

1–β (will detect)

β (won’t detect)

Now assume that we somehow know that the real dif- ference in the population is actually 8.0. This is shown in Figure 11.19 . The dark-shaded part shows the prob- ability that the null hypothesis (which we now “know”

fra97851_ch11_219-244.indd 239fra97851_ch11_219-244.indd 239 12/21/10 6:50 PM12/21/10 6:50 PM

240 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

There are at least four ways to increase power in ad- dition to the use of parametric tests when appropriate:

1. Decrease sampling error by: a. Increasing sample size. An estimate of the neces-

sary sample size can be obtained by doing a statis- tical power analysis. This requires an estimation

Figure 11.20 A Power Curve

–4

1.0

.9

.8

.7

.6

.5

.4

.3

.2

.1

0 0 4

True value of difference

Po w

er =

1 –

β

8 12

Can Statistical Power Analysis Be Misleading?

Can statistical power analysis mislead researchers? An example is provided by a study which found that in the year following the 1994 federal ban on assault weapons, gun homicides declined 6.7 percent more than would have been expected, based on preexisting trends. This difference was not statistically signifi cant. The authors carried out a statistical power analysis and concluded that a larger sample and lon- ger time period were needed to have a suffi ciently powerful statistical test that could detect an effect if any truly existed.*

Kleck argued that even with a much longer time frame and larger sample (and therefore greater statistical power), the pos- sible impact of the ban was so slight (because assault weapons comprised only two percent of the guns used in crime) that “we could not reliably detect so minuscule an impact.”† He agreed with the study’s authors that the observed decline may well have been due to other factors, such as reduced crack use in high-crime neighborhoods (uncontrolled variables). He asked, in light of such known limitations, whether such studies are worth doing. Koper and Roth argued that they are because of the importance of decisions regarding such social policy issues.‡

What do you think? Are such studies worth doing?

†G. Kleck (2001). Impossible policy evaluations and impossible conclusions: A comment on Koper and Roth. Journal of Quantitative Criminology, 17(1): 80. ‡C. S. Koper and J. A. Roth (2001). A priori assertions versus em- pirical inquiry: A reply to Kleck. Journal of Quantitative Criminol- ogy, 17(1): 81–88.

*C. S. Koper and J. A. Roth (2001). The impact of the 1994 assault weapon ban on gun violence outcomes: An assessment of multiple outcome measures and some lessons for policy evaluation. Journal of Quantitative Criminology, 17(1): 33–74.

to be wrong) will not be rejected by using the critical value of 5.8—that is, it shows how often one would get a value of 5.8 or less, which is not suffi cient to cause the null hypothesis to be rejected. This area is beta, and is determined from a t -table. The power under this par- ticular “known” value of the population difference is the lightly shaded area 1 – b. You can see, then, that in this instance, if the real value is 8, a t -test using p 5 .05 will very often fail to detect it.

The power (1 – b) for a series of assumed “true” values can be obtained in the same way. When power is plotted against assumed “real” values, the result is called a power curve and looks something like Figure 11.20 . Comparing such power curves for differ- ent techniques (e.g., a t -test versus the Mann-Whitney U test) indicates their relative effi ciency for use in a par- ticular circumstance. Parametric tests (e.g., ANOVA, t - tests) are generally, but not always, more powerful than nonparametric tests (e.g., chi square, Mann-Whitney U test).

Clearly, researchers want to use a powerful statistical test—one that can detect a relationship if one exists in the population—if they possibly can. If at all possible, therefore, power should be increased. How might this be done?

240

CONTROVERSIES IN RESEARCH

fra97851_ch11_219-244.indd 240fra97851_ch11_219-244.indd 240 12/21/10 6:50 PM12/21/10 6:50 PM

C H A P T E R 1 1 Inferential Statistics 241

of all of the values (except n 5 sample size) used in calculating the statistic you plan to use and solving for n (see, for example, Table 12.2 on page 252 for the formulas used to calculate the t-test).*

b. Using reliable measures to decrease measure- ment error.

2. Controlling for extraneous variables, as these may obscure the relationship being studied.

*Tables are available for estimating sample size based on a combina- tion of desired effect size (see page 248) and desired level of power. See also M. W. Lipsey (1990). Design sensitivity: Statistical power for experimental research. Newbury Park, CA: Sage.

Go back to the INTERACTIVE AND APPLIED LEARNING feature at the beginning of the chapter for a listing of interactive and applied activities. Go to the Online Learning Center at www.mhhe.com/fraenkel8e to take quizzes, practice with key terms, and review chapter content.

Main Points WHAT ARE INFERENTIAL STATISTICS? • Inferential statistics refer to certain procedures that allow researchers to make infer-

ences about a population based on data obtained from a sample. • The term probability, as used in research, refers to the predicted relative frequency

with which a given event will occur.

SAMPLING ERROR

• The term sampling error refers to the variations in sample statistics that occur as a result of repeated sampling from the same population.

THE DISTRIBUTION OF SAMPLE MEANS

• A sampling distribution of means is a frequency distribution resulting from plotting the means of a very large number of samples from the same population.

• The standard error of the mean is the standard deviation of a sampling distribution of means. The standard error of the difference between means is the standard deviation of a sampling distribution of differences between sample means.

CONFIDENCE INTERVALS

• A confi dence interval is a region extending both above and below a sample statistic (such as a sample mean) within which a population parameter (such as the population mean) may be said to fall with a specifi ed probability of being wrong.

3. Increasing the strength of the treatment (if there is one), perhaps by using a larger time period.

4. Using a one-tailed test, when such is justifi able.

fra97851_ch11_219-244.indd 241fra97851_ch11_219-244.indd 241 12/21/10 6:50 PM12/21/10 6:50 PM

242 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

HYPOTHESIS TESTING

• Statistical hypothesis testing is a way of determining the probability that an obtained sample statistic will occur, given a hypothetical population parameter.

• A research hypothesis specifi es the nature of the relationship the researcher thinks exists in the population.

• The null hypothesis typically specifi es that there is no relationship in the population.

SIGNIFICANCE LEVELS

• The term signifi cance level (or level of signifi cance ), as used in research, refers to the probability of a sample statistic occurring as a result of sampling error.

• The signifi cance levels most commonly used in educational research are the .05 and .01 levels.

• Statistical signifi cance and practical signifi cance are not necessarily the same. Even if a result is statistically signifi cant, it may not be practically (i.e., educationally) signifi cant.

TESTS OF STATISTICAL SIGNIFICANCE

• A one-tailed test of signifi cance involves the use of probabilities based on one-half of a sampling distribution because the research hypothesis is a directional hypothesis.

• A two-tailed test, on the other hand, involves the use of probabilities based on both sides of a sampling distribution because the research hypothesis is a nondirectional hypothesis.

PARAMETRIC TESTS FOR QUANTITATIVE DATA

• A parametric statistical test requires various kinds of assumptions about the nature of the population from which the samples involved in the research study were taken.

• Some of the commonly used parametric techniques for analyzing quantitative data include the t -test for means, ANOVA, ANCOVA, MANOVA, MANCOVA, and the t -test for r .

PARAMETRIC TESTS FOR CATEGORICAL DATA

• The most common parametric technique for analyzing categorical data is the t -test for differences in proportions.

NONPARAMETRIC TESTS FOR QUANTITATIVE DATA

• A nonparametric statistical technique makes few, if any, assumptions about the na- ture of the population from which the samples in the study were taken.

• Some of the commonly used nonparametric techniques for analyzing quantitative data are the Mann-Whitney U test, the Kruskal-Wallis one-way analysis of variance, the sign test, and the Friedman two-way analysis of variance.

NONPARAMETRIC TESTS FOR CATEGORICAL DATA

• The chi-square test is the nonparametric technique most commonly used to analyze categorical data.

• The contingency coeffi cient is a descriptive statistic indicating the degree of relation- ship between two categorical variables.

fra97851_ch11_219-244.indd 242fra97851_ch11_219-244.indd 242 12/21/10 6:50 PM12/21/10 6:50 PM

C H A P T E R 1 1 Inferential Statistics 243

POWER OF A STATISTICAL TEST

• The power of a statistical test for a particular set of data is the likelihood of identifying a difference, when in fact it exists, between population parameters.

• Parametric tests are generally, but not always, more powerful than nonparametric tests.

Key Terms analysis of covariance (ANCOVA) 236 analysis of variance

(ANOVA) 236

chi-square test 238

confi dence interval 224

contingency coeffi cient 238

degrees of freedom (df) 234

Friedman two- way analysis of variance 237

inferential statistics 220

Kruskal-Wallis one-way analysis of variance 237

level of signifi cance 228

Mann-Whitney U test 237

multivariate analysis of covariance (MANCOVA) 237

multivariate analysis of variance (MANOVA) 237

nonparametric technique 233

null hypothesis 228

one-tailed test 231

parametric technique 233

power of a statistical test 239

practical signifi cance 230

probability 226

research hypothesis 228

sampling distribution 222

sampling error 221

sign test 237

standard error of the difference (SED) 227

standard error of the mean (SEM) 223

statistical signifi cance 230

t -test for a difference in proportions 237

t -test for correlated means 236

t -test for correlated proportions 237

t -test for independent means 234

t -test for independent proportions 238

t-test for r 237

two-tailed test 232

Type I error 232

Type II error 232

Wilk’s lambda 237

For Discussion 1. “Hypotheses can never be proven, only supported.” Is this statement true or not? Explain. 2. No two samples will be the same in all of their characteristics. Why won’t they? 3. When might a researcher not need to use inferential statistics to analyze his or her

data? 4. “No sampling procedure, not even random sampling, guarantees a totally representa-

tive sample.” Is this true? Discuss. 5. Could a relationship that is practically signifi cant be ignored because it is not statisti-

cally signifi cant? What about the reverse? Can you suggest an example of each?

fra97851_ch11_219-244.indd 243fra97851_ch11_219-244.indd 243 12/21/10 6:50 PM12/21/10 6:50 PM

244 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

Research Exercise 11: Inferential Statistics Using Problem Sheet 11, indicate which inference technique(s), if any, are appropriate for your study. Indicate whether you would or would not do a signifi cance test and/or calculate a confi - dence interval, and if not, explain why. If you do not intend to use any inference techniques in your study, explain why.

Problem Sheet 11

Inferential Statistics 1. An appropriate inferential technique for my study would be:

____________________________________________________________________

____________________________________________________________________

2. Indicate whether you would use a parametric or nonparametric technique and why.

____________________________________________________________________

____________________________________________________________________

3. Indicate whether you would or would not do a signifi cance test and why.

____________________________________________________________________

____________________________________________________________________

4. Indicate whether you would or would not calculate a confi dence interval because:

____________________________________________________________________

____________________________________________________________________

5. I would not use any inferential techniques in my study because:

____________________________________________________________________

____________________________________________________________________

6. The type of sample used in my sample is:

____________________________________________________________________

____________________________________________________________________

7. The use of this type of sample in my study places the following limitation(s) on my use of inferential statistics:

____________________________________________________________________

____________________________________________________________________

An electronic version of this Problem Sheet that you can fi ll in and print, save, or e-mail is available on the Online Learning Center at www.mhhe.com/ fraenkel8e.

fra97851_ch11_219-244.indd 244fra97851_ch11_219-244.indd 244 12/21/10 6:50 PM12/21/10 6:50 PM

12

O B J E C T I V E S Studying this chapter should enable you to:

• Apply several recommendations when comparing data obtained from two or more groups.

• Apply several recommendations when relating variables within a single group.

• Explain what is meant by the term “effect size.”

• Describe briefl y how to use frequency polygons, scatterplots, and crossbreak tables to interpret data.

• Differentiate between statistically signifi cant and practically signifi cant research results.

Approaches to Research

Comparing Groups: Quantitative Data

Techniques Interpretation

Relating Variables Within a Group: Quantitative Data

Techniques Interpretation

Comparing Groups: Categorical Data

Techniques Interpretation

Relating Variables Within a Group: Categorical Data

A Recap of Recommendations

Statistics in Perspective

Test consultant Principal

“Our method shows significantly higher test results!”

“Hmm. Do you mean statistical significance

or ‘real-world’ significance?”

fra97851_ch12_245-262.indd 245fra97851_ch12_245-262.indd 245 12/21/10 6:57 PM12/21/10 6:57 PM

246

W ell, the results are in,” said Tamara Phillips. “I got the consultant’s report about that study we did last semester.” “What study was that?” asked Felicia Lee, as the two carpooled to Eisenhower Middle School, where they both taught

eighth-grade social studies.

“Don’t you remember? That guy from the university came down and asked some of us who taught social studies to try an

inquiry approach?”

“Oh, yeah, I remember. I was in the experimental group—we used a series of inquiry-oriented lessons that they designed.

They compared the results of our students with those of students who were similar in ability whose teachers did not use those

lessons. What did they fi nd out?”

“Well, the report states that those students whose teachers used the inquiry lessons had signifi cantly higher test scores. But

I’m not quite sure what that suggests.”

“It means that the inquiry method is superior to whatever method the teachers of the other group used, doesn’t it?”

“I’m not sure. It depends on whether the signifi cance they’re talking about refers to practical or only statistical signifi cance.”

“What’s the difference?”

This difference—between statistical and practical signifi cance—is an important one when it comes to talking about the results

of a study. It is one of the things you will learn about in this chapter.

Go to the Online Learning Center at www.mhhe.com/fraenkel8e to:

• Learn More About Statistical Versus Practical Signifi cance

Go to your online Student Mastery Activities book to do the following activities:

• Activity 12.1: Statistical vs. Practical Signifi cance • Activity 12.2: Appropriate Techniques • Activity 12.3: Interpret the Data • Activity 12.4: Collect Some Data

INTERACTIVE AND APPLIED LEARNING After, or while, reading this chapter:

Now that you are somewhat familiar with both descrip- tive and inferential statistics, we want to relate them more specifi cally to practice. What are appropriate uses of these statistics? What are appropriate interpretations of them? What are the common errors or mistakes you should watch out for as either a participant in or con- sumer of research?

There are appropriate uses for both descriptive and inferential statistics. Sometimes, however, either or both types can be used inappropriately. In this chap- ter, therefore, we want to discuss the appropriate use of the descriptive and inferential statistics described in the previous two chapters. We will present a number of recommendations that we believe all researchers should consider when they use either type of statistics. *

Approaches to Research Much research in education is done in one of two ways: either two or more groups are compared or variables within one group are related. Furthermore, as you have seen, the data in a study may be either quantitative or categorical. Thus, four different combinations of research are possible, as shown in Figure 12.1 .

Remember that all groups are made up of individual units. In most cases, the unit is one person and the group is a group of people. Sometimes, however, the unit is itself a group (for example, a class of students). In such cases, the “group” would be a collection of classes. This is illustrated by the following hypothesis: “Teacher friendliness is related to student learning.” This hypoth- esis could be studied with a group of classes and a mea- sure of both teacher “friendliness” and average student learning for each class .

*We acknowledge that not all researchers would agree with these recommendations.

fra97851_ch12_245-262.indd 246fra97851_ch12_245-262.indd 246 12/21/10 6:57 PM12/21/10 6:57 PM

C H A P T E R 1 2 Statistics in Perspective 247

Recommendation 2: Use these polygons to decide which measure of central tendency is appropriate to calculate. If any polygon shows extreme scores at one end, use medians for all groups rather than, or in addition to, means.

INTERPRETATION

Once the descriptive statistics have been calculated, they must be interpreted. At this point, the task is to describe, in words, what the polygons and averages tell researchers about the question or hypothesis being investigated. A key question arises: How large does a difference in means between two groups have to be in order to be important? When will this difference make a difference? How does one decide? You will recall that this is the issue of practical versus statistical signifi - cance that we discussed in Chapter 11.

Use Information About Known Groups. Unfortunately, in most educational research, this in- formation is very diffi cult to obtain. Sometimes, prior experience can be helpful. One of the advantages of IQ scores is that, over the years, many educators have had enough experience with them to make differences between them meaningful. Most experienced counsel- ors, administrators, and teachers realize, for example, that a difference in means of less than 5 points between two groups has little useful meaning, no matter how statistically signifi cant the difference may be. They also know that a difference between means of 10 points is enough to have important implications. At other times, a researcher may have available a frame of reference, or standard, to use in interpreting the magnitude of a difference between means. One such standard consists of the mean scores of known groups . In a study of criti- cal thinking in which one of the present authors par- ticipated, for example, the end-of-year mean score for

Another complication arises in studies in which the same individuals receive two or more different treatments or methods. In comparing treatments, we are not then comparing different groups of people but different groups of scores obtained by the same group at different times. Nevertheless, the statistical analysis fi ts the comparison group model. We discuss this point further in Chapter 13.

Comparing Groups: Quantitative Data TECHNIQUES

Whenever two or more groups are compared using quantitative data, the comparisons can be made in a va- riety of ways: through frequency polygons, calculation of one or more measures of central tendency (averages), and/or calculation of one or more measures of vari- ability (spreads). Frequency polygons provide the most information; averages are useful summaries of each group’s performance; and spreads provide information about the degree of variability in each group.

When analyzing data obtained from two groups, therefore, the fi rst thing researchers should do is con- struct a frequency polygon of each group’s scores. This will show all the information available about each group and also help researchers decide which of the shorter and more convenient indices to calculate. For example, examination of the frequency polygon of a group’s scores can indicate whether the median or the mean is the most appropriate measure of central tendency to use. When comparing quantitative data from two groups, therefore, we recommend the following:

Recommendation 1: As a fi rst step, prepare a fre- quency polygon of each group’s scores.

Figure 12.1 Combinations of Data and Approaches to Research

Two or more groups are compared

Quantitative Categorical

Data

Variables within one group are related

fra97851_ch12_245-262.indd 247fra97851_ch12_245-262.indd 247 12/21/10 6:57 PM12/21/10 6:57 PM

248 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

When pre-to-post gains in the mean scores of two groups are compared, the formula is modifi ed as follows:

D 5 mean experimental gain

2 mean comparison gain

_____________________

standard deviation of gain

  of comparison group

The standard deviation of gain score is obtained by fi rst getting the gain (post-to-pre) score for each individual and then calculating the standard deviation as usual. †

While effect size is a useful tool for assessing the magnitude of a difference between the means of two groups, it does not, in and of itself, answer the ques- tion of how large it must be for researchers to consider an obtained difference important. As is the case with signifi cance levels, this is essentially an arbitrary deci- sion. Most researchers consider that any effect size of .50 (that is, half a standard deviation of the comparison group’s scores) or larger is an important fi nding. If the scores fi t the normal distribution, such a value indicates that the difference in means between the two groups is about one-twelfth the distance between the highest and lowest scores of the comparison group. When assessing the magnitude of a difference between the means of two groups, therefore, we recommend the following:

Recommendation 3: Compare obtained results with data on the means of known groups, if possible.

Recommendation 4: Calculate an effect size. Inter- pret an ES of .50 or larger as important. (Smaller values of ES may have theoretical, as opposed to practical, importance.)

Use Inferential Statistics. A third method for judging the importance of a difference between the means of two groups is by the use of inferential statistics . It is common to fi nd, even before examining polygons or differences in means, that a researcher has applied an inference technique (a t -test, an analysis of variance, and so on) and then used the results as the only a criterion for evaluating the importance of the results. This practice has come under increasing attack for the following reasons:

1. Unless the groups compared are random samples from specifi ed populations (which is unusual), the results (probabilities, signifi cance levels, and confi - dence intervals) are to an unknown degree in error and hence misleading.

a group of eleventh-graders who received a special cur- riculum was higher than is typical of the mean scores of eleventh-graders in general and close to the mean score of a group of college students, whereas a comparison group scored lower than both. Because the special-cur- riculum group also demonstrated a fall-to-spring mean gain that was twice that of the comparison group, the total evidence obtained through comparing their perfor- mance with other groups indicated that the gains made by the special-curriculum group were important.

Calculate the Effect Size. Another technique for assessing the magnitude of a difference between the means of two groups is to calculate what is known as effect size (ES).*

Effect size takes into account the size of the dif- ference between means that is obtained, regardless of whether it is statistically signifi cant. One of the most commonly used indexes of effect size is called Delta (D) and is obtained by dividing the difference between the means of the two groups being compared by the stan- dard deviation of the comparison group. Thus:

D 5 mean of experimental group

2 mean of comparison group

________________________

standard deviation of comparison group

*The term effect size is used to identify a group of statistical indices, all of which have the common purpose of clarifying the magnitude of relationship.

†There are more effective ways to obtain gain scores, but we will delay a discussion until subsequent chapters.

© The New Yorker Collection 1977 Joseph Mirachi from cartoonbank.com. All Rights Reserved.

fra97851_ch12_245-262.indd 248fra97851_ch12_245-262.indd 248 12/21/10 6:57 PM12/21/10 6:57 PM

249

2. The outcome is greatly affected by sample size. With 100 cases in each of two groups, a mean difference in IQ score of 4.2 points is statistically signifi cant at the .05 level (assuming the standard deviation is 15, as is typical with most IQ tests). Although statisti- cally signifi cant, this difference is so small as to be meaningless in any practical sense.

3. The actual magnitude of difference is minimized or sometimes overlooked.

4. The purpose of inferential statistics is to provide in- formation pertinent to generalizing sample results to populations, not to evaluate sample results.

With regard to the use of inferential statistics, there- fore, we recommend the following:

Recommendation 5: Consider using inferential statistics only if you can make a convincing ar- gument that a difference between means of the magnitude obtained is important ( Figure 12.2 ).

Recommendation 6: Do not use tests of statistical signifi cance to evaluate the magnitude of a dif- ference between sample means. Use them only as

they were intended: to judge the generalizability of results.

Recommendation 7: Unless random samples were used, interpret probabilities and/or signifi cance levels as crude indices, not as precise values.

Recommendation 8: Report the results of inference techniques as confi dence intervals rather than (or in addition to) signifi cance levels.

Example. Let us give an example to illustrate this type of analysis. We shall present the appropriate cal- culations in detail and then interpret the results. Imag- ine that we have two groups of eighth-grade students, 60 in each group, who receive different methods of so- cial studies instruction for one semester. The teacher of one group uses an inquiry method of instruction, while the teacher of the other group uses the lecture method. The researcher’s hypothesis is that the in- quiry method will result in greater improvement than the lecture method in explaining skills as measured by the “test of ability to explain” (see page 151) in Chap- ter  8. Each student is tested at the beginning and at

should they indicate how large or important it is (is it a differ- ence that makes a difference).” §

Cahan argued, to the contrary, that the way to avoid mis- leading conclusions regarding effects is not by using signifi - cance tests, but rather using confi dence intervals accompanied by increased sample size. ||

In 1999 the American Psychological Association Task Force on Statistical Inference recommended that inference tests not be banned, but that researchers should “always pro- vide some effect size estimate when reporting a p value,” and further that “reporting and interpreting effect sizes in the context of previously reported research is essential to good research.” #

What do you think? Should signifi cance tests be banned in educational research?

Statistical Inference Tests— Good or Bad?

Our recommendations regarding statistical inference are not free of controversy. At one extreme are the views of Carver * and Schmidt, † who argue that the use of statistical inference tests in educational research should be banned. And in 2000 a survey of AERA members (American Educational Research Association) indicated that 19 percent agreed. ‡

At the other extreme are those who agree with Robinson and Levin that “authors should fi rst indicate whether the ob- served effect is a statistically improbable one, and only if it is

CONTROVERSIES IN RESEARCH

*R. P. Carver (1993). The case against statistical signifi cance testing revisited. Journal of Experimental Education, 61: 287–292.

‡K. C. Mittag and B. Thompson (2000). A national survey of AERA members’ perceptions of statistical signifi cance tests and other statis- tical issues. Educational Researcher, 29 (3): 14–19.

†F. L. Schmidt (1996). Statistical signifi cance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1: 115–129.

§D. H. Robinson and J. R. Levin (1997). Refl ections on statistical and substantive signifi cance, with a slice of replication. Educational Researcher, 26 (January/February): 22. ||S. Cahan (2000). Statistical signifi cance is not a “Kosher Certifi cate” for observed effects: A critical analysis of the two-step approach to the evaluation of empirical results. Educational Researcher, 29 (5): 34. #L. Wilkinson and the APA Task Force on Statistical Inference (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist , 54: 599.

fra97851_ch12_245-262.indd 249fra97851_ch12_245-262.indd 249 12/21/10 6:57 PM12/21/10 6:57 PM

250 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

the end of the semester. The test consists of 40 items; the range of scores on the pretest is from 3 to 32, or 29 points. A gain score (posttest-pretest) is obtained. These gain scores are shown in the frequency distri- butions in Table 12.1 and the frequency polygons in Figure 12.3

These polygons indicate that a comparison of means is appropriate. Why? * The mean of the inquiry group is 5.6 compared to the mean of 4.4 for the lecture group. The difference between means is 1.2. In this instance, a comparison with the means of known groups is not pos- sible, since such data are not available. A calculation of effect size results in an ES of .44, somewhat below the .50 that most researchers recommend for signifi cance. Inspection of Figure 12.3 , however, suggests that the difference between the means of the two groups should not be discounted. Figure 12.4 and Table 12.2 show that the number of students gaining 7 or more points is 25 in the inquiry group and 13 (about half as many) in the lecture group. A gain of 7 points on a 40-item test can be considered substantial, even more so when it is recalled that the range was 29 points (3–32) on the pretest. If a gain of 8 points is used, the numbers are 16 in the inquiry group and 9 in the lecture group. If a gain of 6 points is used, the numbers become 34 and 20. We would argue that these discrepancies are large enough,

in context, to recommend the inquiry method over the lecture method.

The use of an inference technique (a t -test for inde pendent means) indicates that p , .05 in one tail

Figure 12.2 A Difference That Doesn’t Make a Difference!

“Come on. The school average was only

140 out of 200 — and your class average was 136!

Big deal!”

“I’m pretty depressed. My class scored lower than

the overall school average — statistically significant at

the 1% level, too!”

*The polygons are nearly symmetrical without extreme scores at either end.

TABLE 12.1 Gain Scores on Test of Ability to Explain: Inquiry and Lecture Groups

Inquiry Lecture

Gain Scoresa

Frequency

Cumulative Frequency Frequency

Cumulative Frequency

11 1 60 0 60

10 3 59 2 60

9 5 56 3 58

8 7 51 4 55

7 9 44 4 51

6 9 35 7 47

5 6 26 9 40

4 6 20 8 31

3 5 14 7 23

2 4 9 6 16

1 2 5 4 10

0 3 3 5 6

21 0 0 1 1

aA negative score indicates the pretest was higher than the posttest.

fra97851_ch12_245-262.indd 250fra97851_ch12_245-262.indd 250 12/21/10 6:57 PM12/21/10 6:57 PM

C H A P T E R 1 2 Statistics in Perspective 251

(Table 12.2). * This leads the researcher to conclude that the observed difference between means of 1.2 points probably is not due to the particular samples used. Whether this probability can be taken as exact depends primarily on whether the samples were randomly se- lected. The 90 percent confi dence interval is shown in Figure 12.4 . † Notice that a difference of zero between the population means is not within the confi dence interval.

Relating Variables Within a Group: Quantitative Data TECHNIQUES

Whenever a relationship between quantitative vari- ables within a single group is examined, the appropri- ate techniques are the scatterplot and the correlation coefficient. The scatterplot illustrates all the data visu- ally, while the correlation coeffi cient provides a numeri- cal summary of the data. When analyzing data obtained from a single group, therefore, researchers should begin by constructing a scatterplot. Not only will it provide all the information available, but it will help them judge which correlation coeffi cient to calculate (the choice usually will be between the Pearson r , which assumes a linear, or straight-line, relationship, and eta, which describes a curvilinear, or curved, relationship ). ‡

Consider Figure 12.5 . All of the fi ve scatterplots shown represent a Pearson correlation of about .50. Only in ( a ), however, does this coeffi cient (.50) com- pletely convey the nature of the relationship. In ( b ) the relationship is understated, since it is a curvilinear one, and eta would give a higher coeffi cient. In ( c ) the co- effi cient does not refl ect the fan-shaped nature of the relationship. In ( d ) the coeffi cient does not reveal that there are two distinct subgroups. In ( e ) the coeffi cient is greatly infl ated by a few unusual cases. While these illustrations are a bit exaggerated, similar results are often found in real data.

When examining relationships within a single group, therefore, we recommend the following:

Recommendation 9: Begin by constructing a scatterplot.

Recommendation 10: Use the scatterplot to deter- mine which correlation coeffi cient is appropriate to calculate.

Recommendation 11: Use both the scatterplot and the correlation coeffi cient to interpret results.

INTERPRETATION

Interpreting scatterplots and correlations presents pro- blems similar to those we discussed in relation to

Figure 12.3 Frequency Polygons of Gain Scores on Test of Ability to Explain: Inquiry and Lecture Groups *A negative score indicates the pretest was higher than the posttest.

9

8

7

6

5

4

3

2

1

–2 –1 0 1 2 3 4 5 6 7 8 9 10 11 12

Fr eq

ue nc

y

Gain scores on test of ability to explain*

Inquiry

Lecture

Obtained difference between sample means

0.39 1.2 2.01

.05 .05

Figure 12.4 90 Percent Confi dence Interval for a Difference of 1.2 Between Sample Means

*A directional hypothesis indicates use of a one-tailed test (see p. 231). †1.65 SED gives .05 in one tail of the normal curve. 1.65 (SED) 5 1.65(.49) 5 .81. 1.2 6 .81 equals .39 to 2.01. This is the 90 percent confi dence interval. Use of 1.65 rather than 1.96 is justifi ed because the researcher’s hypothesis is concerned only with a positive gain (a one-tailed test). The 95 percent or any other confi dence interval could, of course, have been used.

‡Because both of these correlations describe the magnitude of relationship, they are also examples of effect size (see footnote, page 248).

fra97851_ch12_245-262.indd 251fra97851_ch12_245-262.indd 251 12/21/10 6:57 PM12/21/10 6:57 PM

252 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

differences in means. How large must a correlation co- effi cient be to suggest an important relationship? What does an important relationship look like on a scatterplot?

As you can see, doing or evaluating research is not cut and dried; it is not a matter of following a set of rules, but rather requires informed judgment. In judging correlation coeffi cients, one must fi rst assess their appropriateness, as was done with those in Figure 12.5 . If the Pearson

correlation coeffi cient is an adequate summary (and we have shown in Figure 12.5 that this is not always the case), most researchers would agree to the interpretations shown in Table 12.3 when testing a research hypothesis.

As with a comparison of means, the use of inferential statistics to judge the importance of the magnitude of a relationship is both common and often misleading. With a sample of 100, a correlation of only .20 is statistically

TABLE 12.2 Calculations from Table 12.1

Inquiry Group Lecture Group

Gain Score f a fX b X 2 X 2c (X 2 X2) 2 d f(X 2 X2) 2 e

Gain Score f fX X 2 X2 (X 2 X2)2 f(X 2 X2)2

11 1 11 5.4 29.2 29.2 11 0 0 6.6 43.6 0.0 10 3 30 4.4 19.4 58.2 10 2 20 5.6 31.4 62.8

9 5 45 3.4 11.6 58.0 9 3 27 4.6 21.2 63.6 8 7 56 2.4 5.8 40.6 8 4 32 3.6 13.0 52.0 7 9 63 1.4 2.0 18.0 7 4 28 2.6 6.8 27.2 6 9 54 0.4 0.2 1.8 6 7 42 1.6 2.6 18.2 5 6 30 20.6 0.4 2.4 5 9 45 0.6 0.4 3.6 4 6 24 21.6 2.6 15.6 4 8 32 20.4 0.2 1.6 3 5 15 22.6 6.8 34.0 3 7 21 21.4 2.0 14.0 2 4 8 23.6 13.0 52.0 2 6 12 22.4 5.8 34.8 1 2 2 24.6 21.2 42.4 1 4 4 23.4 11.6 46.4 0 3 0 25.6 31.4 94.2 0 5 0 24.4 19.4 97.0

21 0 0 26.6 43.6 0.0 21 1 21 25.4 29.2 29.2 22 0 0 27.6 57.8 0.0 22 0 0 26.4 41.0 0.0 Total S 5 338 S 5 446.4 S 5 262 S 5 450.4

p .05

ES(Δ) X 1 � X 2

SD2

1.2

2.4 .44

t �

� � �

� � X 1 � X 2

SED

1.2

.49 2.45

SED (SEM1)2 � � �(SEM2)2 .35 2 .35 2 .12 .12 .24 .49

SEM2 SD

n �

�

1

2.7

59

2.7

7.7 .35SEM1

SD

n � 1 2.7

59

2.7

7.7 .35

SD2 f(X X )2

n

450.4

60 7.5 2.7SD1 �

f (X � X )2

n

446.4

60 7.4 2.7

X2 n

262

60 4.4X 1 �

∑ fX n

�

�

�

� � � � �

�

�

� � � �

��

��

�

� � �

� �

338

60 � 5.6

∑ fX

�

af 5 frequency b fX 5 frequency 3 score c X 2

__ X 5 score 2 mean

d (X 2 __

X ) 2 5 (score 2 mean) 2 ef (X 2

__ X ) 2 5 frequency 3 (score 2 mean) 2

fra97851_ch12_245-262.indd 252fra97851_ch12_245-262.indd 252 12/21/10 6:57 PM12/21/10 6:57 PM

C H A P T E R 1 2 Statistics in Perspective 253

signifi cant at the .05 level with a two-tailed test. Ac- cordingly, we recommend the following when interpret- ing scatterplots and correlation coeffi cients:

Recommendation 12: Draw a line that best fi ts all points in a scatterplot, and note the extent of devi- ations from it. The smaller the deviations all along the line, the more useful the relationship. *

Recommendation 13: Consider using inferential statistics only if you can give a convincing argu- ment for the importance of the size of the relation- ship found in the sample.

Recommendation 14: Do not use tests of statisti- cal signifi cance to evaluate the magnitude of a relationship. Use them, as they were intended, to judge generalizability.

Recommendation 15: Unless a random sample was used, interpret probabilities and/or signifi cance levels as crude indices, not as precise values.

Recommendation 16: Report the results of infer- ence techniques as confi dence intervals rather than as signifi cance levels.

Example. Let us now consider an example to illus- trate the analysis of a suspected relationship between variables. Suppose a researcher wishes to test the hy- pothesis that, among counseling clients, improvement in marital satisfaction after six months of counseling is related to self-esteem at the beginning of counseling. In other words, people with higher self-esteem would be expected to show more improvement in marital satisfac- tion after undergoing therapy for a period of six months than people with lower self-esteem. The researcher ob- tains a group of 30 clients, each of whom takes a self- esteem inventory and a marital satisfaction inventory prior to counseling. The marital satisfaction inventory is taken again at the end of six months of counseling. The data are shown in Table 12.4 .

The calculations shown in Table 12.4 are not as hard as they look. Here are the steps that we followed to ob- tain r 5 .42.

1. Multiply n by S XY : 30(7,023) 5 210,690 2. Multiply S X by S Y : (1,007)(192) 5 193,344 3. Subtract step 2 from step 1: 210,690 2 193,344 5

17,346 4. Multiply n by S X 2 : 30(35,507) 5 1,065,210 5. Square S X : (1,007) 2 5 1,014,049 6. Subtract step 5 from step 4: 1,065,210 2 1,014,049

5 51,161 7. Multiply n by S Y 2 : 30(2,354) 5 70,620 8. Square S Y : (192) 2 5 36,864 9. Subtract step 8 from step 7: 70,620 2 36,864 5

33,756 10. Multiply step 6 by step 9: (51,161)(33,756) 5

1,726,990,716 11. Take the square root of step 10: √

____________ 1,726,990,716 5

41,557 12. Divide step 3 by step 11: 17,346/41,557 5 .42

Using the data presented in Table 12.4 , the researcher plots a scatterplot and fi nds that it reveals two things. First, there is a tendency for individuals with higher initial self-esteem scores to show greater improvement

(a) (b) (c) (d) (e)

Figure 12.5 Scatterplots with a Pearson r of .50

TABLE 12.3 Interpretation of Correlation Coeffi cients when Testing Research Hypotheses

Magnitude of r Interpretation

.00 to .40 Of little practical importance except in unusual circumstances; perhaps of theoretical value. *

.41 to .60 Large enough to be of practical as well as theoretical use.

.61 to .80 Very important, but rarely obtained in educational research.

.81 or above Possibly an error in calculation; if not, a very sizable relationship.

* When selecting a very few people from a large group, even correlations this small may have predictive value.

*Try this with Figure 12.5 .

fra97851_ch12_245-262.indd 253fra97851_ch12_245-262.indd 253 12/21/10 6:57 PM12/21/10 6:57 PM

254 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

in marital satisfaction than those with lower initial self- esteem scores. Second, it also shows that the re- lationship is more correctly described as curvilinear— that is, clients with low or high self-esteem show less

improvement than those with a moderate level of self- esteem (remember, these data are fi ctional). Pearson r equals .42. The value of eta obtained for these same data is .82, indicating a substantial degree of relationship

TABLE 12.4 Self-Esteem Scores and Gains in Marital Satisfaction

Client

Self-Esteem Score before

Counseling ( X )

X 2

Gain in Marital Satisfaction after Counseling ( Y ) Y 2

XY

1 20 400 24 16 280 2 21 441 22 4 242 3 22 484 27 49 2 154 4 24 576 1 1 24 5 24 576 4 16 96 6 25 625 5 25 125 7 26 676 21 1 226 8 27 729 8 64 216 9 29 841 2 4 58

10 28 784 5 25 140 11 30 900 5 25 150 12 30 900 14 196 420 13 32 1024 7 49 219 14 33 1089 15 225 495 15 35 1225 6 36 210 16 35 1225 16 256 560 17 36 1269 11 121 396 18 37 1396 14 196 518 19 36 1296 18 324 648 20 38 1444 9 81 342 21 39 1527 14 196 546 22 39 1527 15 225 585 23 40 1600 4 16 160 24 41 1681 8 64 328 25 42 1764 0 0 0 26 43 1849 3 9 129 27 43 1849 5 25 215 28 43 1849 8 64 344 29 44 1936 4 16 176 30 45 2025 5 25 225 Total (S) S 5 1,007 S 5 35,507 S 5 192 S 5 2,354 S 5 7,023

r

nΣXY – ΣX ΣY [nΣX 2 – (ΣX )2][ nΣY 2 – (ΣY 2)]

== 30(7023) – (1007)(192)

[30(35507) – (1007)2][30(2354) – (192)2]

17346

1726990716 = =

17346

41557 .42

=

=

210690 – 193344

(1065210 – 1014049)(70620 – 36864) =

17346

(51161)(33756)

fra97851_ch12_245-262.indd 254fra97851_ch12_245-262.indd 254 12/21/10 6:57 PM12/21/10 6:57 PM

C H A P T E R 1 2 Statistics in Perspective 255

between the two variables. We have not shown the cal- culations for eta since they are somewhat more compli- cated than those for r . The relationship is illustrated by the smoothed curve shown in Figure 12.6 .

The researcher calculates the appropriate inference statistic (a t -test for r ), as shown, to determine whether r 5 .42 is signifi cant.

Standard error of r 5 SE r 5 1 _______

√ _____

n 2 1

5 1 ____ √

___ 29 5 .185

t r 5 r 2 .00 _______

SE r

5 .42 2 .00 ________ .185

5 2.3; p < .01

As you can see, it results in an obtained value of 2.3 and a probability of p , .01, using a one-tailed test. A one-tailed test is appropriate for r if the direction of the relationship was predicted before examining the data. The probability associated with eta would (pre- sumably) be obtained using a two-tailed test (unless the researcher predicted the shape of the curve from Figure 12.6 before examining the data). An eta of .82 is also statistically signifi cant at p 5 .01, indicating that the relationship is unlikely to be due to the par- ticular sample studied. Whether or not these probabili- ties are correct depends on whether or not the sample was randomly selected. The 95 percent confi dence interval around the obtained value for r is shown in Figure 12.7 .

Comparing Groups: Categorical Data TECHNIQUES

When the data involved are categorical data, groups may be compared by reporting either percentages (or proportions) or frequencies in crossbreak tables. Table 12.5 gives a fi ctitious example.

25 to 27 22 to 24 19 to 21 16 to 18 13 to 15 10 to 12

7 to 9 4 to 6 1 to 3

0 to –2 –3 to –5 –6 to –8

–9 to –10

G ai

n in

m ar

ita l s

at is

fa ct

io n

af te

r co

un se

lin g

20 22 24 26 28 30 32 34 36 38 40 42 44

Self-esteem scores before counseling

Figure 12.6 Scatterplot Illustrating the Relationship Between Initial Self-Esteem and Gain in Marital Satisfaction Among Counseling Clients

Obtained sample correlation

.025 .025

.06 .42 .78

95% confidence interval

Figure 12.7 95 Percent Confi dence Interval for r 5 .42

fra97851_ch12_245-262.indd 255fra97851_ch12_245-262.indd 255 12/21/10 6:57 PM12/21/10 6:57 PM

256 P A R T 3 Data Analysis www.mhhe.com/fraenkel8e

INTERPRETATION

Once again, we must look at summary statistics—even percentages—carefully. Percentages can be misleading unless the number of cases is also given. At fi rst glance, Table 12.5 may look impressive—until one discov- ers that the data in it represent 60 females and only 10 males. In crossbreak form, Table 12.6 represents the ac- tual numbers, as opposed to percentages, of individuals.

Table 12.7 illustrates a fi ctitious relationship between teacher gender and grade level taught. As you can see, the largest number of male teachers is to be found in grade 7, and the largest number of female teachers is to be found in grade 4. Here, too, however, we must ask: How much

TABLE 12.7 Teacher Gender and Grade Level Taught: Case 1

Grade

4 Grade

5 Grade

6 Grade

7 Total Male 10 20 20 30 80 Female 40 30 30 20 120

Total 50 50 50 50 200

TABLE 12.8 Teacher Gender and Grade Level Taught: Case 2

Grade

4 Grade

5 Grade

6 Grade

7 Total Male 22 22 25 28 97 Female 28 28 25 22 103

Total 50 50 50 50 200

TABLE 12.5 Gender and Political Preference (Percentages)

Percentage of Males

Percentage of Females

Democrat 20 50 Republican 70 45 Other 10 5

Total 100 100

TABLE 12.6 Gender and Political Preference (Numbers)

Males Females Democrat 2 30 Republican 7 27 Other 1 3

difference must there be between these frequencies for us to consider them important? One of the limitations of categor- ical data is that such evaluations are even harder than with quantitative data. One possible approach is to examine prior experience or knowledge. Table 12.7 does suggest a trend toward an increasingly larger proportion of male teachers in the higher grades—but, again, is the trend substantial enough to be considered important?

The data in Table 12.8 show the same trend, but the pattern is much less striking. Perhaps prior experience or research shows (somehow) that gender differences become important whenever the within-grade difference is more than 10 percent (or a frequency of 5 in these data). Such knowledge is seldom available, however, which leads us to consider the summary statistic (similar to the correla- tion coeffi cient) known as the contingency coeffi cient (see Chapter 11). In order to use it, however, remember that the data must be presented in crossbreak tables. Calculating the contingency coeffi cient is easily done by hand or by computer. You will recall that this statistic is not as straight- forward in interpretation as the correlation coeffi cient, since its interpretation depends on the number of cells in the crossbreak table. Nevertheless, we recommend its use.

Perhaps because of the diffi culties mentioned above, most research reports using percentages or crossbreaks rely on inference techniques to evaluate the magnitude of relationships. In the absence of random sampling, their use suffers from the same liabilities as with quanti- tative data. When analyzing categorical data, therefore, we recommend the following:

Recommendation 17: Whenever possible, place all data into crossbreak tables.

Recommendation 18: To clarify the importance of relationships, patterns, or trends, calculate a con- tingency coeffi cient.

Recommendation 19: Do not use tests of statistical sig- nifi cance to evaluate the magnitude of relationships. Use them, as intended, to judge generalizability.

Recommendation 20: Unless a random sample was used, interpret probabilities and/or signifi cance levels as crude indices, not as precise values.

Example. Once again, let us consider an example to illustrate an analysis, this time involving categorical data when comparing groups. Let us return to Tables 12.7 and 12.8 to illustrate the major recommendations for analyzing categorical data. We shall consider Table 12.7 fi rst. Because there are 50 teachers, or 25 percent, of the total of 200 teachers at each grade level (4–7), we would expect that there would be 25 percent of the

fra97851_ch12_245-262.indd 256fra97851_ch12_245-262.indd 256 12/21/10 6:57 PM12/21/10 6:57 PM