More Research

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 1/35

CHAPTER 11

Sampling

The simplest rationale for sampling is that it may not be feasible because of time or financial constraints, or even physically possible, to collect data from everyone involved in an evaluation.

Sampling strategies provide systematic, transparent processes for choosing who will actually be asked to provide data.

—Mertens and Wilson, 2012, p. 410

Relationships are powerful. Our one-to-one connections with each other are the foundation for change. And building relationships with people from different cultures, often many different cultures, is key in

building diverse communities that are powerful enough to achieve significant goals. —Work Group for Community Health and Development, 2013

In This Chapter • The viewpoints of researchers who work within the postpositivist, constructivist, and transformative

paradigms are contrasted in relation to sampling strategies and generalizability.

• External validity is introduced as a critical concept in sampling decisions.

• Challenges in the definition of specific populations are described in terms of conceptual and operational definitions, identifying a person’s racial or ethnic status, identifying persons with a disability, heterogeneity within populations, and cultural issues.

• Strategies for designing and selecting samples are provided, including probability-based, theoretical- purposive, and convenience sampling. Sampling is also discussed for complex designs such as those using hierarchical linear modeling.

• Sampling bias, access issues, and sample size are discussed.

• Ethical standards for the protection of study participants are described in terms of an institutional review board’s requirements.

• Questions to guide critical analysis of sampling definition, selection, and ethics are provided.

Transformative research implies a philosophy that research should confront and act against the causes of injustice and violence, which can be caused not only by that which is researched but also by the process of research itself. Individuals involved in research can be disenfranchised in a few ways: (1) by the hidden power arrangements uncovered by the research process, (2) by the actions of unscrupulous (and even well-intentioned) researchers, but also (3) by researchers’ failure to expose those arrangements once they become aware of them. Hidden power arrangements are maintained by secrets of those who might be victimized by them (because they fear retaliation). . . . [Researchers] contribute to this disenfranchisement if it prevents the exposure of hidden power arrangements. (Baez, 2002, pp. 51–52)

Definition, Selection, and Ethics p. 319

p. 318

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 2/35

Sampling Strategies: Alternative Paradigms The decisions that a researcher makes regarding from whom data will be collected, who is included, how they are included, and what is done to conceal or reveal identities in research constitute the topics addressed in this chapter on sampling. As can be seen in the opening quotation, these decisions are complex and not unproblematic. In a simple sense, sampling refers to the method used to select a given number of people (or things) from a population. The strategy for selecting your sample influences the quality of your data and the inferences that you can make from it. The issues surrounding from whom you collect data are what sampling is all about. Within all approaches to research, researchers use sampling for very practical reasons. In most research studies, it is simply not feasible to collect data from every individual in a setting or population.

Sampling is one area in which great divergence can be witnessed when comparing the various research paradigms. In general, researchers who function within the postpositivist paradigm see the ideal sampling strategy as some form of probability sampling. Kathleen Collins (2010) describes probability sampling as follows:

A researcher uses probability sampling schemes to select randomly the sampling units that are representative of the population of interest. . . . These methods meet the goal of ensuring that every member of the population of interest has an equal chance of selection. . . . When implementing probabilistic sampling designs, the researcher’s objective is to make external statistical generalizations (i.e., generalizing conclusions for the population from which the sample was drawn). (p. 357)

Researchers within the constructivist paradigm tend to use a theoretical or purposive approach to sampling. Their sampling activities begin with an identification of groups, settings, and individuals where (and for whom) the processes being studied are most likely to occur (K. M. T. Collins, 2010). Collins explains:

When using a purposive sample, the goal is to add to or generate new theories by obtaining new insights or fresh perspectives. . . . Purposive sampling schemes are employed by the researcher to choose strategically elite cases or key informants based on the researcher’s perception that the selected cases will yield a depth of information or a unique perspective. (p. 357)

Researchers within the transformative paradigm could choose either a probability or theoretical- purposive approach to sampling, depending on their choice of quantitative, qualitative, or mixed methods. However, they would function with a distinctive consciousness of representing the populations that have traditionally been underrepresented in research.

Despite the contrasting views of sampling evidenced within the various paradigms, issues of common concern exist. All sampling decisions must be made within the constraints of ethics and feasibility. Although randomized probability samples are set forth as the ideal in the postpositivist paradigm, they are not commonly used in educational and psychological research. Thus, in practice, the postpositivist and constructivist paradigms are more similar than different in that both use nonrandom samples. Sometimes, the use of convenience samples (discussed at greater length later in this chapter) means that less care is taken by those in both of these paradigms. All researchers should make conscious choices in the design of their samples rather than accepting whatever sample presents itself as most convenient.

External Validity (Generalizability) or Transferability As you will recall from Chapter 4, external validity refers to the ability of the researcher (and user of the research results) to extend the findings of a particular study beyond the specific individuals and setting in which that study occurred. Within the postpositivist paradigm, the external validity depends on the design and execution of the sampling strategy. Generalizability is a concept that is linked to the target population—that is, the group to whom we want to generalize findings.

In the constructivist paradigm, every instance of a case or process is viewed as both an exemplar of a l l f h d i l d i i i (D i & Li l 2011 ) Th

p. 320

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 3/35

general class of phenomena and particular and unique in its own way (Denzin & Lincoln, 2011a). The researcher’s task is to provide sufficient thick description about the case so that the readers can understand the contextual variables operating in that setting (Lincoln & Guba, 2000). The burden of generalizability then lies with the readers, who are assumed to be able to generalize subjectively from the case in question to their own personal experiences. Lincoln and Guba label this type of generalizability transferability.

EXTENDING YOUR THINKING

Generalizability or Transferability of Results What is your opinion of a researcher’s ability to generalize results? Is it possible? If so, under what conditions? What do you think of the alternative concept of transferability?

Defining the Population and Sample Research constructs, such as racial or ethnic minority or deaf student, can be defined in two ways. Conceptual definitions are those that use other constructs to explain the meaning, and operational definitions are those that specify how the construct will be measured. Researchers often begin their work with a conceptual idea of the group of people they want to study, such as working mothers, drug abusers, students with disabilities, and so on. Through a review of the literature, they formulate a formal, conceptual definition of the group they want to study. For example, the target population might be first-grade students in the United States.

An operational definition of the population in the postpositivist paradigm is called the experimentally accessible population, defined as the list of people who fit the conceptual definition. For example, the experimentally accessible population might be all the first-grade students in your school district whose names are entered into the district’s database. You would next need to obtain a list of all the students in that school district. This would be called your sampling frame. Examples of sampling frames include (a) the student enrollment, (b) a list of clients who receive services at a clinic, (c) professional association membership directories, or (d) city phone directories. The researcher should ask if the lists are complete and up-to-date and who has been left off the list. For example, lists of clients at a community mental health clinic eliminate those who need services but have not sought them. Telephone directories eliminate people who do not have telephone service, as well as those with unlisted or newly assigned numbers, and most directories do not list people’s cell, or mobile, phone numbers. In the postpositivist view, generalizability is in part a function of the match between the conceptual and operational definitions of the sample. If the lists are not accurate, systematic error can occur because of differences between the true population and the study population. When the accessible population represents the target population, this establishes population validity.

The researcher must also acknowledge that the intended sample might differ from the obtained sample. The issue of response rate was addressed in Chapter 6 on survey research, along with strategies such as follow-up of nonrespondents and comparison of respondents and nonrespondents on key variables. The size and effect of nonresponse or attrition should be reported and explained in all approaches to research to address the effect of people not responding, choosing not to participate, being inaccessible, or dropping out of the study. This effect represents a threat to the internal and external validity (or credibility and transferability) of the study’s findings. You may recall the related discussion of this issue in the section on experimental mortality in Chapter 4 and the discussion of credibility and transferability in Chapter 8. A researcher can use statistical processes (described in Chapter 13) to identify the plausibility of fit between the obtained sample and the group from which it was drawn when the design of the study permits it.

Identification of Sample Members It might seem easy to know who is a member of your sample and who is not; however complexities

p. 321

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 4/35

It might seem easy to know who is a member of your sample and who is not; however, complexities arise because of the ambiguity or inadequacy of the categories typically used by researchers. Examples of errors in identification of sample members can readily be found in research with racial and ethnic minorities and persons with disabilities. Two examples are presented here, and the reader is referred to Chapter 5 on causal comparative and correlational research to review additional complexities associated with this issue.

Identification of Race and Ethnicity in Populations Investigators who examine racial or ethnic groups and differences between such groups frequently do so without a clear sense of what race or ethnicity means in a research context (Blum, 2008). Researchers who use categorization and assume homogeneity of condition are avoiding the complexities of participants’ experiences and social locations. Selection of samples on the basis of race should be done with attention to within-group variation and to the influence of particular contexts. Race as a biogenetic variable should not serve as a proxy variable for actual causal variables, such as poverty, unemployment, or family structure.

Heterogeneity has been recognized as a factor that contributes to difficulty in classifying people as African American or Latino (Stanfield, 2011). In reference to African American populations, Stanfield writes,

The question of what is blackness, which translates into who has black African ancestry and how far back it is in family tree histories, is a subject of empirical analysis and should remain on the forefront in any . . . research project. . . . What is needed . . . is developing theories and methods of data collection and analysis that remind us that whiteness, blackness, and other kinds of racializations are relational phenomena. White people create black people; black people create white people, and people in general create each other and structure each other in hierarchies, communities, movements, and societies, and global spheres. (p. 18)

Thus, Stanfield recognizes that many people are not pure racially, but people are viewed as belonging to specific racial groups in many research studies.

Race is sometimes used as a substitute for ethnicity, which is usually defined in terms of a common origin or culture resulting from shared activities and identity based on some mixture of language, religion, race, and ancestry (C. D. Lee, 2003). Lee suggests that the profoundly contextual nature of race and ethnicity must be taken into account in the study of ethnic and race relations. Blum (2008) makes clear that use of broad categories of race can hide important differences in communities; using labels such as African American and Asian American ignores important differences based on ethnicity. Initial immigration status and social capital among different Asian immigrant groups result in stark differences in terms of advantages and positions in current racial and ethnic stratifications. For example, Hmong and Cambodians are generally less successful in American society than Asians from the southern or eastern parts of Asia. Ethnic plurality is visible in the Black community in terms of people who were brought to America during the times of slavery and those who have come more recently from Africa or the Caribbean.

For instance, the word Latino has been used to categorize people of Mexican, Cuban, Puerto Rican, Dominican, Colombian, Salvadoran, and other extractions. The term Hispanic has been used to include people who trace their origins to an area colonized by Spain. However, both labels obscure important dimensions of diversity within the groups. This has implications for sampling and must be attended to if the results are to be meaningful.

The American Psychological Association Joint Task Force of Divisions 17 and 45’s Guidelines on Multicultural Education Training, Research, Practice, and Organizational Change for Psychologists (American Psychological Association [APA], 2002) and the Council of National Psychological Associations for the Advancement of Ethnic Minority Interests’ (2000) Guidelines for Research in Ethnic Minority Communities, 2000 provide detailed insights into working with four of the major racial/ethnic minority groups in the United States: Asian American/Pacific Islander populations, persons of African descent, Hispanics, and American Indians (see Box 11.1).Although American Indians/Native Americans (AI/NA) make up approximately 1.4% of the national population, there are more than 560 federally recognized American Indian tribes in the United States (J B Unger Soto &

p. 322

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 5/35

more than 560 federally recognized American Indian tribes in the United States (J. B. Unger, Soto, & Thomas, 2008). Each recognized tribe has its own government and court system. The diversity in the AI/NA population is described as follows:

The precise number of AI/ANs in the United States is difficult to quantify because it depends on individuals’ self-reports of their AI/AN ancestry and affiliation. Individuals’ decisions to self- identify as AI/AN are influenced by the wording of race/ethnicity questions on surveys, individuals’ awareness of their ancestry, feelings of identification with AI/AN cultures, and perceptions about the potential benefits and costs of labeling themselves as AI/ANs. (p. 125)

BOX 11.1 Heterogeneity in Racial/Ethnic Minority and Immigrant Communities

The American Psychological Association (APA) developed guidelines for cultural competence in conducting research. Because of the unique salience of race/ethnicity for diversity-related issues in the United States, they developed guidelines for four specific racial ethnic groups: Asian American/Pacific Islander populations, persons of African descent, Hispanics, and American Indian participants (APA, 2002). The APA used race/ethnicity as the organizing framework; however, they also recognized the need to be aware of other dimensions of diversity. They had as a guiding principle the following:

Recognition of the ways in which the intersection of racial and ethnic group membership with other dimensions of identity (e.g., gender, age, sexual orientation, disability, religion/spiritual orientation, educational attainment/experiences, and socioeconomic status) enhances the understanding and treatment of all people. (p. 19)

They included the following narrative in their discussion:

As an agent of prosocial change, the culturally competent psychologist carries the responsibility of combating the damaging effects of racism, prejudice, bias, and oppression in all their forms, including all of the methods we use to understand the populations we serve. . . . A consistent theme . . . relates to the interpretation and dissemination of research findings that are meaningful and relevant to each of the four populations and that reflect an inherent understanding of the racial, cultural, and sociopolitical context within which they exist. (p. 1)

Stake and Rizvi (2009) and Banks (2008) discuss the effects of globalization in terms of complicating our understandings of who belongs in which groups and what the implications are for appropriate inclusion in research for immigrant groups particularly. The majority of immigrants coming to the United States are from Asia, Latin America, the West Indies, and Africa. With national boundaries eroding, people cross boundaries more frequently than ever before, resulting in questions about citizenship and nationality. In addition, political instability and factors such as war, violence, drought, or famine have led to millions of refugees who are essentially stateless. Researchers need to be aware of the status of immigrant and refugee groups in their communities and implications for how they sample in their studies. For example, the University of Michigan’s Center for Arab American Studies (www.casl.umd.umich.edu/caas/) conducts studies that illuminate much of the diversity in that community. The American Psychological Association (APA, 2013) developed a guide that has relevance when working with diverse culture communities called Working With Immigrant-Origin Clients. Kien Lee’s (2004) work in immigrant communities provides guidance in working with immigrants to the United States from a variety of countries, including China, India, El Salvador, and Vietnam. Lee also worked with the Work Group for Community Health and Development (2013) to develop a Community Tool Box, an online resource that contains practical information for working with culturally diverse communities for social change. The tool box is available at http://ctb.ku.edu/en/tablecontents/index.aspx.

People With Disabilities As you will recall from Chapter 6 the federal legislation Individuals with Disabilities Education Act

p. 324

p. 323

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 6/35

As you will recall from Chapter 6, the federal legislation Individuals with Disabilities Education Act (IDEA, 2001; Public Law 108-446, Section 602), reauthorized in 2004, defines the following categories of disabilities:

• Mental retardation • Hearing impairments • Speech or language impairments • Visual impairments • Emotional disturbance • Orthopedic impairments • Other health impairments • Specific learning disabilities • Multiple disabilities • Deaf-blindness • Autism • Traumatic brain injury • Developmental delays

Mertens and McLaughlin (2004) present an operational and conceptual definition for each of these disability categories. The conceptual definitions can be found in the IDEA and a data dictionary that is available at the IDEA website (www.ideadata.org), which includes definitions of key terms in special education legislation (Data Accountability Center, 2012). The translation of these conceptual definitions into operational definitions is fraught with difficulty. You can imagine the diversity of individuals who would be included in a category such as emotional disturbance, which is defined in the federal legislation as individuals who are unable to build or maintain satisfactory interpersonal relationships, exhibit inappropriate types of behaviors or feelings, have a generally pervasive mood of unhappiness or depression, or have been diagnosed with schizophrenia. Psychologists have struggled for years with finding ways to accurately classify people with such characteristics.

A second example of issues that complicate categorizing individuals with disabilities can be seen in the federal definition and procedures for identification for people with learning disabilities displayed in Box 11.2. The definition indicates eight areas in which the learning disability can be manifest. This list alone demonstrates the heterogeneity that is masked when participants in studies are simply labeled “learning disabled.” Even within one skill area, such as reading, there are several potential reasons that a student would display difficulty in that area (e.g., letter identification, word attack, comprehension). Then, there are the complications that arise in moving from this conceptual definition to the operational definition. That is, how are people identified as having a learning disability? And how reliable and valid are the measures used to establish that a student has a learning disability (E. Johnson, Mellard, & Byrd, 2005)? Many researchers in the area of learning disabilities identify their participants through school records of Individualized Education Plans; they do not do independent assessments to determine the validity of those labels. However, Aaron, Malatesha Joshi, Gooden, and Bentum (2008) conclude that many children are not identified as having a learning disability, yet they exhibit similar skill deficits as those who are so labeled, further complicating comparisons between groups. The National Dissemination Center for Children With Disabilities (www.nichcy.org1) published a series of pamphlets on the identification of children with learning disabilities that are geared to professionals and parents (Hozella, 2007).

Cultural issues also come into play in the definition of people with disabilities. For example, people who are deaf use a capital D in writing the word Deaf when a person is considered to be culturally Deaf (Harris, Holmes, & Mertens, 2009). This designation as culturally Deaf is made less on the basis of one’s level of hearing loss and more on the basis of one’s identification with the Deaf community and use of American Sign Language

p. 325

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 7/35

use of American Sign Language.

BOX 11.2 Federal Definition of Specific Learning Disability and Identification Procedures

The following conceptual definition of learning disability is included in the IDEA legislation:

Specific learning disability means a disorder in one or more of the basic psychological processes involved in understanding or in using language, spoken or written, that may manifest itself in an imperfect ability to listen, think, speak, read, write, spell, or to do mathematical calculations, including conditions such as perceptual disabilities, brain injury, minimal brain dysfunction, dyslexia, and developmental aphasia. . . . Specific learning disability does not include learning problems that are primarily the result of visual, hearing, or motor disabilities, of mental retardation, of emotional disturbance, or of environmental, cultural, or economic disadvantage. (34 CFR 300.8[c][10])

The federal government addressed the issue of an operational definition of learning disability as a determination made by the child’s teachers and an individual qualified to do individualized diagnostic assessment such as a school psychologist, based on the following:

• The child does not achieve adequately for the child’s age or to meet State-approved grade-level standards in one or more of the following areas, when provided with learning experiences and instruction appropriate for the child’s age or State-approved grade-level standards:

Oral expression.

Listening comprehension.

Written expression.

Basic reading skills.

Reading fluency skills.

Reading comprehension.

Mathematics calculation.

Mathematics problem solving.

• The child does not make sufficient progress to meet age or State-approved grade-level standards in one or more of the areas identified in 34 CFR 300.309(a)(1) when using a process based on the child’s response to scientific, research-based intervention; or the child exhibits a pattern of strengths and weaknesses in performance, achievement, or both, relative to age, State-approved grade-level standards, or intellectual development, that is determined by the group to be relevant to the identification of a specific learning disability, using appropriate assessments, consistent with 34 CFR 300.304 and 300.305; and the group determines that its findings under 34 CFR 300.309(a)(1) and (2) are not primarily the result of:

A visual, hearing, or motor disability;

Mental retardation;

Emotional disturbance;

Cultural factors;

Environmental or economic disadvantage; or

Limited English proficiency.

To ensure that underachievement in a child suspected of having a specific learning disability is not due to lack of appropriate instruction in reading or math, the group must consider, as part of the evaluation described in 34 CFR 300.304 through 300.306:

• Data that demonstrate that prior to, or as a part of, the referral process, the child was provided appropriate instruction in regular education settings, delivered by qualified personnel; and

D b d d i f d f hi bl i l fl i

p. 326

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 8/35

• Data-based documentation of repeated assessments of achievement at reasonable intervals, reflecting formal assessment of student progress during instruction, which was provided to the child’s parents.

SOURCES: 34 CFR 300.309; 20 U.S.C. 1221e-3, 1401(30), 1414(b)(6).

The American Psychological Association (APA, 2012) developed “Guidelines for Assessment of and Interventions with Persons with Disabilities,” which acknowledge that defining the term disability is difficult. It encourages psychologists to adopt a positive, enablement-focused approach with people with disabilities rather than focusing on what they cannot do. It also provides guidance in how to have a barrier-free physical and communication environment so that people with disabilities can participate in research (and therapy) with dignity.

Sampling Strategies As mentioned previously, the strategy chosen for selecting samples varies based on the logistics, ethics, and paradigm of the researcher. An important strategy for choosing a sample is to determine the dimensions of diversity that are important to that particular study. An example is provided in Box 5.1. Questions for reflection about salient dimensions of diversity in sampling for focus groups are included in Box 11.3.

K. M. T. Collins (2010) divides sampling strategies into probabilistic and purposive. Persons working in the constructivist paradigm prefer the terms theoretical or purposive to describe their sampling. A third category of sampling that is often used, but not endorsed by proponents of any of the major paradigms, is convenience sampling.

BOX 11.3 Dimensions of Diversity: Questions for Reflection on Sampling Strategy in Focus Group Research

Box 5.1 describes the sampling strategy used by Mertens (2000) in her study of deaf and hard-of-hearing people in the court system. The following are questions for reflection about salient aspects of that strategy:

1. What sampling strategies are appropriate to provide a fair picture of the diversity within important target populations? What are the dimensions of diversity that are important in gender groups? How can one address the myth of homogeneity in selected cultural groups—for example, all women are the same, all deaf people are the same, and so on?

2. What is the importance of considering such a concept in the context in which you do research/evaluation?

EXTENDING YOUR THINKING

Dimensions of Diversity How do you think researchers can address the issues of heterogeneity within different populations? Find examples of research studies with women, ethnic minorities, and people with disabilities. How did the researchers address heterogeneity in their studies? What suggestions do you have for improving the way this issue is addressed?

Probability-Based Sampling Probability-based sampling is recommended because it is possible to analyze the possible bias and likely error mathematically (K. M. T. Collins, 2010). Sampling error is defined as the difference between the sample and the population, and can be estimated for random samples. Random samples are those in which every member of the population has a known, nonzero probability of being included in

p. 327

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 9/35

t ose w c eve y e be o t e popu at o as a ow , o e o p obab ty o be g c uded the sample. Random means that the selection of each unit is independent of the selection of any other unit. Random selection can be done in a variety of ways, including using a lottery procedure drawing well-mixed numbers, extracting a set of numbers from a list of random numbers, or producing a computer-generated list of random numbers. If the sample has been drawn in such a way that makes it probable that the sample is approximately the same as the population on the variables to be studied, it is deemed to be representative of the population. Researchers can choose from several strategies for probability-based sampling. K. M. T. Collins (2010) describes probabilistic sampling strategies as follows:

Before the study commences, the researcher establishes a sampling frame and predetermines the number of sampling units, preferably based on a mathematical formula, such as power analysis and selects the units by using simple random sampling or other adaptations of simple random sampling, specifically, stratified, cluster and two-stage or multi-stage random sampling. (p. 357)

Five examples are presented here:

Simple Random Sampling Simple random sampling means that each member of the population has an equal and independent chance of being selected. The researcher can choose a simple random sample by assigning a number to every member of the population, using a table of random numbers, randomly selecting a row or column in that table, and taking all the numbers that correspond to the sampling units in that row or column. Or the researcher could put all the names in a hat and pull them out at random. Computers could also be used to generate a random list of numbers that corresponds to the numbers of the members of the population.

This sampling strategy requires a complete list of the population. Its advantages are the simplicity of the process and its compatibility with the assumptions of many statistical tests (described further in Chapter 13). Disadvantages are that a complete list of the population might not be available or that the subpopulations of interest might not be equally represented in the population. In telephone survey research in which a complete listing of the population is not available, the researcher can use a different type of simple random sampling known as random digit dialing (RDD). RDD involves the generation of random telephone numbers that are then used to contact people for interviews. This eliminates the problems of out-of-date directories and unlisted numbers. If the target population is households in a given geographic area, the researcher can obtain a list of the residential exchanges for that area, thus eliminating wasted calls to business establishments.

Systematic Sampling For systematic sampling, the researcher will take every nth name on the population list. The procedure involves estimating the needed sample size and dividing the number of names on the list by the estimated sample size. For example, if you had a population of 1,000 and you estimated that you needed a sample size of 100, you would divide 1,000 by 100 and determine that you need to choose every 10th name on the population list. You then randomly pick a place to start on the list that is less than n and take every 10th name past your starting point.

The advantage of this sampling strategy is that you do not need to have an exact list of all the sampling units. It is sufficient to have knowledge of how many people (or things) are in the accessible population and to have a physical representation for each person in that group. For example, a researcher could sample files or invoices in this manner. Systematic sampling strategy can be used to accomplish de facto stratified sampling. Stratified sampling is discussed next, but the basic concept is sampling from previously established groups (e.g., different hospitals or schools). If the files or invoices are arranged by group, the systematic sampling strategy can result in de facto stratification by group (i.e., in this example, location of services).

One caution should be noted in the use of systematic sampling. If the files or invoices are arranged in a specific pattern, that could result in choosing a biased sample. For example, if the files are kept in alphabetical order by year and the number n results in choosing only individuals or cases whose last names begin with the letter A, this could be biasing.

p. 329

p. 328

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 10/35

a es beg w t t e ette , t s cou d be b as g.

Stratified Sampling This type of sampling is used when there are subgroups (or strata) of different sizes that you wish to investigate. For example, if you want to study gender differences in a special education population, you need to stratify on the basis of gender, because boys are known to be more frequently represented in special education than girls. The researcher then needs to decide if he or she will sample each subpopulation proportionately or disproportionately to its representation in the population.

• Proportional stratified sampling means that the sampling fraction is the same for each stratum. Thus, the sample size for each stratum will be different when using this strategy. This type of stratification will result in greater precision and reduction of the sampling error, especially when the variance between or among the stratified groups is large. The disadvantage of this approach is that information must be available on the stratifying variable for every member of the accessible population.

• Disproportional stratified sampling is used when there are big differences in the sizes of the subgroups, as mentioned previously in gender differences in special education. Disproportional sampling requires the use of different fractions of each subgroup and thus requires the use of weighting in the analysis of results to adjust for the selection bias. The advantage of disproportional sampling is that the variability is reduced within the smaller subgroup by having a larger number of observations for the group. The major disadvantage of this strategy is that weights must be used in the subsequent analyses; however, most statistical programs are set up to use weights in the calculation of population estimates and standard errors.

Cluster Sampling Cluster sampling is used with naturally occurring groups of individuals—for example, city blocks or classrooms in a school. The researcher would randomly choose the city blocks and then attempt to study all (or a random sample of) the households in those blocks. This approach is useful when a full listing of individuals in the population is not available but a listing of clusters is. For example, individual schools maintain a list of students by grade, but no state or national list is kept. Cluster sampling is also useful when site visits are needed to collect data; the researcher can save time and money by collecting data at a limited number of sites.

The disadvantage of cluster sampling is apparent in the analysis phase of the research. In the calculations of sampling error, the number used for the sample size is the number of clusters, and the mean for each cluster replaces the sample mean. This reduction in sample size results in a larger standard error and thus less precision in estimates of effect.

Multistage Sampling This method consists of a combination of sampling strategies and is described by K. M. T. Collins (2010) as “choosing a sample from the random sampling schemes in multiple states” (p. 358). For example, the researcher could use cluster sampling to randomly select classrooms and then use simple random sampling to select a sample within each classroom. The calculations of statistics for multistage sampling become quite complex; researchers need to aware that too few strata will yield unreliable extremes of the sampling variable. Between roughly 30 and 50 strata work well for multistage samples using regression analysis.

Complex Sampling Designs in Quantitative Research Spybrook, Raudenbush, Liu, Congdon, and Martinez (2008) discuss sampling issues involved in complex designs such as cluster randomized trials, multisite randomized trials, multisite cluster randomized trials, cluster randomized trials with treatment at level three, trials with repeated measures, and cluster randomized trials with repeated measures. The sampling issues arise because these research approaches involve the assignment of groups, rather than individuals, to experimental and control conditions. This complicates sampling issues because the n of the clusters may be quite small and

p. 330

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 11/35

hence limit the ability of the researcher to demonstrate sufficient power in the analysis phase of the study. However, Spybrook and colleagues developed a sophisticated analytic procedure that accommodates the small cluster sizes while still allowing larger sample sizes within the clusters to be tested appropriately. The statistical procedures involved in such designs exceed the scope of this text; hence, readers are referred to Spybrook et al. (2008) and other sources such as Mertler and Vannatta (2005).

Examples of Sampling in Quantitative Studies Researchers in education and psychology face many challenges in trying to use probability-based sampling strategies. Even in G. D. Borman et al.’s (2007) study of the Success for All reading program that is summarized in Chapter 1, they were constrained by the need to obtain agreement from schools to participate. They could not select randomly from the group of schools that agreed to the conditions of the study because it was already a relatively small group. Probability-based sampling is generally easier to do with survey research when a list of people in the population is available. For example, Nardo, Custodero, Persellin, and Fox (2006) used the National Association for the Education of Young Children’s digital database of 8,000 names of programs that had fully accredited centers for their study of the musical practices, musical preparation of teachers, and music education needs of early childhood professionals in the United States. They gave the list to a university-based research center and asked them to prepare a randomized clustered sample of 1,000 early childhood centers. The clusters were based on the state in which the programs were located, and the number of centers chosen was proportional to the number of centers in each state.

Henry, Gordon, and Rickman (2006) conducted an evaluation study of early childhood education in the state of Georgia in which they were able to randomly select 4-year-olds receiving early education services either through Head Start (a federal program) or in a Georgia pre-K program (a state program). They first established strata based on the number of 4-year-olds living in each county. Counties were randomly selected from each stratum. Then, sites within the counties were randomly selected from both Head Start and pre-K programs and five children were randomly selected from each classroom. This resulted in a list of 98 pre-K and Head Start sites, all of which agreed to participate in the study (which the authors acknowledge is “amazing” [p. 83]). The researchers then asked for parental permission; 75% or more of parents in most sites consented, resulting in a Head Start sample size of 134. Data were not collected for 20 of these 134 students because students moved out of state, withdrew from the program, or lacked available baseline data. From the 353 pre-K children, the researchers ended up with 201 students who matched those enrolled in Head Start in terms of eligibility to be considered for that program based on poverty indicators. Clearly, thoughtful strategies are needed in applying random sampling principles in research in education and psychology.

Purposeful or Theoretical Sampling As mentioned previously, researchers working within the constructivist paradigm typically select their samples with the goal of identifying information-rich cases that will allow them to study a case in depth. Although the goal is not generalization from a sample to the population, it is important that the researcher make clear the sampling strategy and its associated logic to the reader. Patton (2002) identifies the following sampling strategies that can be used with qualitative methods:

Extreme or Deviant Cases The criterion for selection of cases might be to choose individuals or sites that are unusual or special in some way. For example, the researcher might choose to study a school with a low record of violence compared with one that has a high record of violence. The researcher might choose to study highly successful programs and compare them with programs that have failed. Study of extreme cases might yield information that would be relevant to improving more “typical” cases. The researcher makes the assumption that studying the unusual will illuminate the ordinary. The criterion for selection then becomes the researcher’s and users’ beliefs about which cases they could learn the most from. Psychologists have used this sampling strategy to study deviant behaviors in specific extreme cases.

p. 331

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 12/35

Intensity Sampling Intensity sampling is somewhat similar to the extreme-case strategy, except there is less emphasis on extreme. The researcher wants to identify sites or individuals in which the phenomenon of interest is strongly represented. Critics of the extreme- or deviant-case strategy might suggest that the cases are so unusual that they distort the situation beyond applicability to typical cases. Thus, the researcher would look for rich cases that are not necessarily extreme. Intensity sampling requires knowledge on the part of the researcher as to which sites or individuals meet the specified criterion. This knowledge can be gained by exploratory fieldwork.

Maximum-Variation Sampling Sites or individuals can be chosen based on the criterion of maximizing variation within the sample. For example, the researcher can identify sites located in isolated rural areas, urban centers, and suburban neighborhoods to study the effect of total inclusion of students with disabilities. The results would indicate what is unique about each situation (e.g., ability to attract and retain qualified personnel) as well as what is common across these diverse settings (e.g., increase in interaction between students with and without disabilities).

Homogeneous Sampling In contrast to maximum variation sampling, homogeneous sampling involves identification of cases or individuals that are strongly homogeneous. In using this strategy, the researcher seeks to describe the experiences of subgroups of people who share similar characteristics. For example, parents of deaf children aged 6 through 7 represent a group of parents who have had similar experiences with preschool services for deaf children. Homogeneous sampling is the recommended strategy for focus group studies. Researchers who use focus groups have found that groups made up of heterogeneous people often result in representatives of the “dominant” group monopolizing the focus group discussion. For example, combining parents of children with disabilities in the same focus group with program administrators could result in the parents’ feeling intimidated.

Typical-Case Sampling If the researcher’s goal is to describe a typical case in which a program has been implemented, this is the sampling strategy of choice. Typical cases can be identified by recommendations of knowledgeable individuals or by review of extant demographic or programmatic data that suggest that this case is indeed average.

Stratified Purposeful Sampling This is a combination of sampling strategies such that subgroups are chosen based on specified criteria, and a sample of cases is then selected within those strata. For example, the cases might be divided into highly successful, average, and failing schools, and the specific cases can be selected from each subgroup.

Critical-Case Sampling Patton (2002) describes critical cases as those that can make a point quite dramatically or are, for some reason, particularly important in the scheme of things. A clue to the existence of a critical case is a statement to the effect that “if it’s true of this one case, it’s likely to be true of all other cases” (p. 243). For example, if total inclusion is planned for children with disabilities, the researcher might identify a community in which the parents are highly satisfied with the education of their children in a separate school for children with disabilities. If a program of inclusion can be deemed to be successful in that community, it suggests that it would be possible to see that program succeed in other communities in which the parents are not so satisfied with the separate education of their children with disabilities.

Snowball or Chain Sampling Snowball sampling is used to help the researcher find out who has the information that is important to

p. 333

p. 332

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 13/35

Snowball sampling is used to help the researcher find out who has the information that is important to the study. The researcher starts with key informants who are viewed as knowledgeable about the program or community. The researcher asks the key informants to recommend other people to whom he or she should talk based on their knowledge of who should know a lot about the program in question. Although the researcher starts with a relatively short list of informants, the list grows (like a snowball) as names are added through the referral of informants.

Criterion Sampling The researcher must set up a criterion and then identify cases that meet that criterion. For example, a huge increase in referrals from a regular elementary school to a special residential school for students with disabilities might lead the researcher to set up a criterion of “cases that have been referred to the special school within the last 6 months.” Thus, the researcher could determine reasons for the sudden increase in referrals (e.g., Did a staff member recently leave the regular elementary school? Did the special school recently obtain staff with expertise that it did not previously have?).

Theory-Based or Operational Construct Sampling Sometimes, a researcher will start a study with the desire to study the meaning of a theoretical construct such as creativity or anxiety. Such a theoretical construct must be operationally defined (as discussed previously in regard to the experimentally accessible population). If a researcher operationalizes the theoretical construct of anxiety in terms of social stresses that create anxiety, sample selection might focus on individuals who “theoretically” should exemplify that construct. This might be a group of people who have recently become unemployed or homeless.

Confirming and Disconfirming Cases You will recall that in the grounded theory approach (discussed in Chapter 8 on qualitative methods), the researcher is interested in emerging theory that is always being tested against data that are systematically collected. The “constant comparative method” requires the researcher to seek verification for hypotheses that emerge throughout the study. The application of the criterion to seek negative cases suggests that the researcher should consciously sample cases that fit (confirming) and do not fit (disconfirming) the theory that is emerging.

Opportunistic Sampling When working within the constructivist paradigm, researchers seldom establish the final definition and selection of sample members prior to the beginning of the study. When opportunities present themselves to the researcher during the course of the study, the researcher should make a decision on the spot as to the relevance of the activity or individual in terms of the emerging theory. Thus, opportunistic sampling involves decisions made regarding sampling during the course of the study.

Purposeful Random Sampling In qualitative research, samples tend to be relatively small because of the depth of information that is sought from each site or individual. Nevertheless, random sampling strategies can be used to choose those who will be included in a very small sample. For example, in a study of sexual abuse at a residential school for deaf students, I randomly selected the students to be interviewed (Mertens, 1996). The result was not a statistically representative sample but a purposeful random sampling that could be defended on the grounds that the cases that were selected were not based on recommendations of administrators at the school who might have handpicked a group of students who would put the school in a “good light.”

Sampling Politically Important Cases The rationale for sampling politically important cases rests on the perceived credibility of the study by the persons expected to use the results. For example, if a program has been implemented in a number of regions, a random sample might (by chance) omit the region in which the legislator who controls funds for the program resides. It would be politically expedient for the legislator to have information

p. 334

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 14/35

that came directly from that region. Therefore, the researcher might choose purposively to include that region in the sample to increase the perceived usefulness of the study results. Henry (1990) agrees that “sampling for studies where the results are to be used in the political environment may require an additional layer of concern for political credibility beyond scientific concerns for validity” (p. 15).

Case Study Sampling Stake (2006) provides direction for choosing the sample for case study research that depends on the purpose of the case study, as well as on logistics, likely receptiveness, and available resources. He identifies three approaches to case studies, each of which calls for different sampling strategies.

1. Intrinsic case studies are conducted when a particular case is of specific interest such that the case is in essence already decided before the research begins. This is often the case in evaluation of specific programs or in biographical research. With this type of study, generalization is less important than achieving a thorough understanding of that particular case.

2. Instrumental case studies are undertaken to gain an understanding of a phenomenon with the goal of enhancing ability to generalize to other cases, for example, improving race relations. Stake (2006) emphasizes that cases should be chosen because they represent an opportunity to learn. That might mean that the case is more difficult to access or may require more time than another case, but the potential to learn is what constitutes the worth of a particular case.

3. Collective case study (also known as multiple case study) is an approach in which several cases are selected to study because of a desire to understand the phenomenon in a broader context. Cases are chosen because the researcher believes that understanding them will lead to better understanding, and perhaps better theorizing, about a still larger collection of cases. It is possible that a researcher will not know all the cases until the study is under way and new issues arise that suggest that other cases are necessary.

For each of these three types of case studies, sampling decisions still need to be made with regard to persons, specific locations, events, timing, subgroups, and dimensions.

Examples of Qualitative Research Sampling Rolon-Dow (2005) specifically chose to study the school experiences of Puerto Rican girls because research about Latino/as does not often distinguish among subgroups by country of origin or gender. She used purposeful sampling to get participants who varied in their academic and social performance at school. She relied on teachers to help her choose 9 second-generation Puerto Rican girls who attended U.S. schools for their early education from kindergarten through middle school. In her published work, she provides a table that lists the characteristics of each girl in terms of her grades at school, socioeconomic level, ethnic identity, language spoken at home, and parents’ level of education and employment. She sought similarities on all the characteristics in the table, except grades in school. For that variable, she sought maximum variation. Remember the qualitative study by Schneider et al. (2004) that involved people with schizophrenia because she believes that they are the experts on their own experiences. Thus, she purposefully sought out support groups for people with schizophrenia and actively involved them in deciding on the focus of the research.

Harry, Klingner, and Hart’s (2005) ethnographic study of special education placement for African American and Hispanic students, also discussed in Chapter 8, described their criteria for purposeful sampling as follows:

Our purpose in this research was to study the processes by which Black and Hispanic children are placed in special education programs. We were concerned with documenting the quality of early instruction, referral decisions and processes, psychological evaluation, and the quality of ultimate placements in special education.

Our research was conducted in a multiethnic, multi-lingual urban school district, which is one of the largest in the nation. Criteria for the selection of 12 elementary schools included race and ethnicity of students, socioeconomic levels as represented by the frequency of free and reduced

p. 335

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 15/35

y , p y q y lunch, and the schools’ rates of special education placement. . . . The research design was funnel- like, moving from the broadest information on the district, to overall rates and placement processes in the 12 schools to 12 targeted classrooms, and finally to 12 selected case study students. (p. 102)

Convenience Sampling Convenience sampling means that the persons participating in the study were chosen because they were readily available (Patton, 2002). Although this might be the least desirable sampling strategy, it is probably the most commonly used. Much psychological research has been conducted using undergraduate students in psychology classes because they are available. When such a convenience sample is used, the researcher must acknowledge the limitations of the sample and not attempt to generalize the results beyond the given population pool. The cost of convenience sampling is illustrated in the following example:

When Ann Landers, the advice columnist, asked people to write in and answer the question, “If you had it to do over again, would you have children?” 70% of the nearly 10,000 respondents said that they would not have children if they could make the choice to be a parent again. However, when a statistically designed opinion poll on the same issue a few months later was conducted, it was reported that 91% of parents said that they would have children again! (D. Moore & McCabe, 2003, p. 248)

It appears that parents who were unhappy about having been parents were much more likely to volunteer their views than parents who were happy being parents. (The “happy” parents were probably either too busy to respond, or they don’t read Ann Landers because they don’t need any advice.)

A qualitative case study of a young woman with multiple disabilities (Fourie & Theron, 2012; described earlier in Chapters 3 and 8) used convenience sampling, but the researchers provided justification for their choice of the particular individual in such a way that they strengthened the transferability of their findings to other young women with this condition. One of the researchers was a live-in caregiver for the young woman; hence, her access to the participant was one criterion for their choice. Another criterion that fit the case study’s purpose, to study resilience in people with this type of disability, supported their choice because the young woman was cheerful and forward-looking and did not portray a deficit perspective that often is associated with this condition. To guard against the researchers’ personal biases and to protect the young woman’s interest, the researchers engaged an advisory panel for the study made up of a psychologist, her teacher, and her mother. This panel served not only to verify the young woman’s willingness to be in the study but also as secondary participants. Here is a partial description of the young woman in the study, whom they called Lucy:

The participant, Lucy, was a White American young woman, 16 years of age, who had been diagnosed with full-mutation FXS. Lucy experience physical, behavioral, emotional, language, sensory integration, and cognitive challenges. For example, she struggled with fine motor skills, low muscle tone, and visual acuity. It was difficult for her to concentrate, and she easily became fretful. When she was anxious, she sometimes threw tantrums, flapped her hands, or behaved inappropriately and impulsively. (p. 1358)

Given this description of Lucy, the reader has a pretty good idea of her characteristics and how she might be similar to or different from other individuals with this syndrome, thus increasing the probability that the findings of this study could be transferable to others who share these characteristics.

It should be noted, however, that because of ethical concerns, all samples are in the end volunteers. In addition, reality constraints such as access and cost must be considered in all sampling decisions. Luanna Ross’s (1992) description of her sampling strategy in a study of Native American women in prison exemplifies the constraints sometimes encountered and how this researcher worked around them (see Box 11.4).

p. 337

p. 336

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 16/35

BOX 11.4 Example of Constraints in Sampling Procedures

A series of constraints were encountered during the initial phase of sample selection. I originally planned on using a snowball technique, starting with my cousin and spinning off from there. After my first interview, which was with my cousin, the treatment specialist presented me with a list of incarcerated mothers, Indian and white, whom she thought I should interview. I wondered who she left off the list and why, and I felt too intimidated to suggest a different technique. Also, she said that I would not be allowed, for my own safety, to interview dangerous offenders with the exception of one woman because the treatment specialist thought she would “be interesting.” I took the list and started interviewing women from my reservation whom I had known for several decades.

Next, feeling constrained by the situation, I followed the treatment specialist’s instructions and went down the list of women. After several weeks, however, I became familiar with women in the halls and would request that the guards send them to me to be interviewed, although they were not on the list. For example, one young pregnant woman seemed appealing given her age and pregnancy; and, from other prisoners, I heard about and then interviewed an Indian woman who was confined in an isolation cell in the general population building. When the staff discovered this, nothing was said and I continued the process. Thus, I purposefully chose women I wanted to interview. I knew that I wanted a sample that would elicit data regarding variations: for instance, different races, mothers with children of various ages, and prisoners confined in different units.

—Ross, 1992, pp. 79–80

Mixed Methods Sampling Mixed methods researchers cannot escape the complexities in sampling for either quantitative or qualitative research; rather, their challenges are magnified by having both sets of issues plus the complexity of mixed methods to deal with. For example, K. M. T. Collins, Onwuegbuzie, and Jiao (2007) raise the question, “Is it appropriate to triangulate, expand, compare, or consolidate quantitative data originating from a large, random sample with qualitative data arising from a small purposive sample?” (p. 269). Mixed methods researchers identify several sampling strategies that are unique to mixed methods research (K. M. T. Collins, 2010; K. M. T. Collins et al., 2007). Recall from Chapter 10 that mixed methods research designs can be parallel (different methods used at the same time) or sequential (different methods used one after the other) or cyclical (multistage). These design options influence the sampling strategies for mixed methods, which include

• identical sampling (same people in qualitative and quantitative samples), • parallel sampling (different people in the quantitative and qualitative samples, but both from the

same population, e.g., school children in a district), • nested sampling (a subset of those in one method of the study are chosen to be in the other part of

the study), and • multilevel sampling (different people from different populations are chosen for the different

approaches of the study).

In a mixed methods study conducted by a team of deaf researchers and myself (Mertens, Holmes, Harris, & Brandt, 2007), we used a sequential nested sampling/multilevel sampling strategy. We started with a qualitative phase in which we had a subsample of graduates from a program to prepare teachers for students who are deaf and have additional disabilities. Following that data collection and analysis, we sent a quantitative survey to all the graduates of the program (nested sampling). Subsequently, we conducted qualitative interviews with the staff and faculty from the university and the cooperating schools (multilevel sampling).

In the pragmatic sequential mixed methods study summarized as Sample Study 1.5, Berliner, Barrat, Fong, and Shirk (2008) used a convenience sample for the quantitative portion of their study that consisted of data from 3,856 students. They then conducted interviews

p. 338

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 17/35

to clarify, affirm, or challenge the study findings and to explore state and district policies and practices that affect reenrollment and students’ experiences dropping out and reenrolling in San Bernardino City Unified School District high schools. In fall 2007 interview data were collected from 20 district contacts during a weeklong, in-person site visit. . . . The assistant superintendent identified seven district administrators to be interviewed based on their professional roles and knowledge of dropout and reenrollment issues. . . . The five principals of the district’s traditional high schools and the two principals of the district’s continuation schools were then interviewed about school perspectives on reenrolling dropouts. . . . The principals then identified dropouts who reenrolled in district schools for the student interview sample. Six students, each from different high schools, were interviewed about their dropout and reenrollment experiences. . . . They then interviewed district administrators, principals, and students. (p. 19)

EXTENDING YOUR THINKING

Sampling Strategies Using some of the research studies you previously identified through literature searches, read through the sections on sample selection and participant characteristics. Try to identify the different types of sampling discussed in this chapter. Justify your choice of a “label” for the type of sampling used in the studies, based on evidence presented in the article. Be sure to look at studies that exemplify the four major paradigms discussed in this book.

Select a research problem of your own (or in a group in your class). Discuss how you could use the different sampling strategies described in this chapter. Provide examples for all of the different types of probability-based, theoretical-purposive, and convenience sampling strategies. What is the effect of the sampling strategy on the way you view your research problem? Do you find yourself modifying the research problem to accommodate the different sampling requirements?

Sampling Bias Fienberg (2003) identifies two types of errors in sampling:

1. Nonsampling error includes systematic errors due to such factors as nonresponse and measurement errors. The bias associated with nonsampling errors is not dependent on the size of the sample; rather, it is associated with differences between those who respond to the questions and those who do not or with differences that arise because of who administers the instrument. Nonsampling errors are associated with the type of data collection and are therefore discussed in the next chapter.

2. Sampling error includes systematic error from sampling approaches that overrepresent a portion of the study population. This type of error is minimized as the size of the sample increases.

Fienberg (2003) notes that probability-based sampling is the tool that allows researchers the ability to quantitatively demonstrate that their sample is representative of the population from which it is drawn by specifying the margin of error (seen commonly in media reports of surveys).

In the constructivist spirit, Guba and Lincoln (1989) reject the notion that it is possible to reach a generalizable conclusion because of a particular sampling strategy. They argue that research and evaluation results are limited by context and time and that “one cannot determine that this curriculum (as an example) will fit into and work in a given setting without trying it in that setting” (p. 61). They continue with their views on sampling within this paradigmatic framework:

First, respondents who will enter into the hermeneutic process must be selected. But such sampling is not carried out for the sake of drawing a group that is representative of some population to which the findings are to be generalized. Nor is the sample selected in ways that satisfy statistical requirements of randomness The sample is selected to serve a different purpose;

p. 339

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 18/35

satisfy statistical requirements of randomness. The sample is selected to serve a different purpose; hence the term “purposive sampling” is used to describe the process. . . . For the constructivist, maximum variation sampling that provides the broadest scope of information (the broadest base for achieving local understanding) is the sampling mode of choice. (pp. 177–178)

Guba and Lincoln (1989) describe a sampling process that is not preordained but allowed to evolve with the emergence of issues in the research.

EXTENDING YOUR THINKING

Sampling Bias Reexamine the research studies that you are using for these exercises. This time, critique studies in terms of their sampling bias. How did the researchers address this issue?

Access Issues Accessibility to a sample or population is an important factor to consider when making decisions about sampling designs. For some populations, such as illegal drug users or homeless people, it might not be possible to obtain a complete listing of the members of the population, thus making it difficult to use the probability-based sampling strategies. The likelihood of a population’s being accessible or willing to respond to mail, phone, or online surveys should also be considered. The populations just mentioned might not be accessible by these means because of residential transience, financial reasons, lack of access to the Internet, or lack of trust of people outside their communities.

Access to different communities can be more or less challenging. Kien Lee (2004) conducted research on civic participation in immigrant communities in the United States. She pursued this work partially because of expressed difficulties on the part of other researchers regarding getting members of these communities to engage in research or evaluation projects, whether this was because of a distrust of government based on past experiences in the home country, wariness of revealing personal information to an outsider, or language barriers that inhibited effective communication. Lee reported that she was able to identify organizations that were viewed as having credibility in the various communities, such as faith-based organizations, regional or hometown associations, professional associations for members of their cultural group, cultural and benevolent associations, advocacy groups, and refugee resettlement services that served as points of entry into the communities.

Access in Educational Settings Accessing samples in educational settings can be a long process to finally reach agreement with the appropriate persons who can authorize the research and who will participate in the research itself as sample members. Identification of the appropriate persons who have the power to grant access is a complex issue in itself. The appropriate point of entry depends partially on the scope of the study that you plan. If you are interested in using only a few classrooms, you might start with teachers to determine their willingness to support your request to administrators. On the other hand, school-level research might start at the principal level, and district-level research with the superintendent. Whichever level you start with should be viewed as only a starting point, because each school district has specific procedures for requesting access to its students and staff for research purposes. Early contacts can be helpful to ascertain procedures, test the waters, and cultivate advocates for your later, more formal requests.

Researchers should be aware of complications from real life that can present obstacles to sampling as it is ideally conceived. For example, some educators might resist participating in research because they view research as a risky operation that can produce results that will damage their reputations. Also, some schools receive requests for a variety of different research projects, and thus the effect of the treatment you design might be compromised because of interactions with another experimental

ff h di i i i l i h l lli f j f

p. 340

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 19/35

treatment. Staff changes or district reorganizations can result in schools pulling out of projects after you think you have your sample defined.

Assuming that you can reach agreement on particular schools or classrooms, you must obtain consent from the individuals (students or parents or staff members) who will be asked to provide data in the study. This issue is discussed further in the section of this chapter on ethics. However, it does have implications for the type of sampling design that is feasible.

School systems will rarely allow a researcher to take all student participants and randomly assign them to conditions. It is not that school systems are mean-spirited; they have the students’ best interests at heart. For example, teachers will often recommend a student with disabilities when they feel that the student is ready to enter a mainstream environment and the environment has been suitably adapted for the student’s special needs. Randomly selecting students for assignment to mainstream classrooms could result in educationally unsound and ethically indefensible decisions.

More frequently, researchers need to adjust their sampling plans to allow for working with intact groups—for example, administering a treatment to an entire class of students. This, of course, presents challenges related to differential selection as a threat to internal validity in quasi-experimental designs, which were discussed in Chapter 4, and has implications for statistical analysis, discussed in Chapter 13.

Researchers who wish to use stratified sampling should also be aware of the complexities of subdividing the sample on the basis of gender, race or ethnicity, disability, or ability levels within a school setting. First, there are the logistical and ethical obstacles to using stratification. It might not be feasible, educationally sound, or ethical to divide students by demographic categories as would be called for in a stratified sampling design. Second, there are definitional problems in that schools might not use the same definition to “label” students as you have in mind. Also, if you are using more than one site, definitions might vary by site.

Mertens and McLaughlin (2004) describe issues that complicate sampling with special education populations. First, the researcher needs to be aware of the heterogeneity within any special education category and variables that might be uniquely associated with any particular group of people with disabilities. For example, in the area of deafness, characteristics that are important include things such as when the person became deaf, whether the parents are hearing or deaf, and what the person’s language is (signed vs. spoken; American Sign Language vs. cued speech). Second, some disability conditions occur with a low frequency, and thus sampling becomes problematic because of the low incidence of individuals within any one setting. This has implications for variability across contexts as well as for adequate sample size. Sample size then has implications for sampling bias and error. Third, researchers commonly seek comparison groups to include in their samples. Selecting an appropriate comparison group for special education populations can be tricky. You can consider the following options:

1. Selecting individuals from the general population. It is important to remember that people with disabilities differ from nondisabled peers in important ways; for example, the disabled population contains more males, African Americans, urban dwellers, people with lower incomes, and single- parent households (U.S. Department of Education, 2011).

2. Comparisons across disability categories. You should be aware of the heterogeneity within categories, as well as variations on variables, such as IQ, between categories.

3. Cross-unit comparisons that involve the comparison of students with disabilities in one school or school district with students in another school or school district. Of course, contextual variation is a problem here.

4. Longitudinal studies that allow comparisons with one group at two or more points in time. Variation in results could be attributable to historical factors (as discussed in Chapter 4 regarding threats to internal validity).

Access to Records Although much of the discussion of sampling has implied that the sample consists of people, I do not want to ignore a source of data that is frequently used in educational and psychological research—

p. 342

p. 341

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 20/35

want to ignore a source of data that is frequently used in educational and psychological research— extant records. The researcher needs to be concerned with the accessibility of the records for research purposes. Researchers might want to sample the desired records on a pilot basis to ensure that the records contain the information required for the study and that they are appropriately organized for the research study. For example, records might be kept only at a local level and not centralized at any one location, making data collection problematic. In addition, the records may or may not contain all the information that the researcher needs for the study.

The researcher must also be able to demonstrate how the confidentiality of the records will be protected. In most organizations, the records contain the names of the clients or students, and thus the organization might not be willing to provide the researcher access to the records themselves. The researcher can then consider the feasibility and appropriateness of alternative means of achieving access to the archival information, such as asking the agency to provide the records with identifying information deleted, asking if an employee of the agency could be paid to code the information from the records without names, or determining if it is possible to obtain a computer printout with codes in place of personal identification information.

Sample Size The optimum sample size is directly related to the type of research you are undertaking. For different types of research, rules of thumb can be used to determine the appropriate sample size. It should be noted that there are some research methodologists who feel that rules of thumb are never appropriate as a basis for making sample size decisions (e.g., Krathwohl, 2009). Nevertheless, I present these as guides to new researchers who need some kind of ballpark feel for sample sizes. In some cases, your sample size will be determined by very practical constraints, such as how many people are participating in a program or are in a classroom. (This is often, but not always, the case in evaluation studies.) When you are conducting quantitative experimental research and you have the freedom to choose a sample size, there are formulas that can guide you in those decisions. So let us take a look at sample size from these different perspectives: rules of thumb and formulaic determination.

Rules of Thumb

Quantitative Research Rules of Thumb Using power analysis formulas, Onwuegbuzie, Jiao, and Bostick (2004) calculated the size of samples needed for correlational, causal comparative, and experimental research in order to find a “medium . . . one-tailed and/or two-tailed statistically significant relationship or difference with .80 power at the 5% level of significance” (K. M. T. Collins et al., 2007, p. 273). The recommended sample sizes for multiple regression and survey research come from Gall, Gall, and Borg (2007).

a. Onwuegbuzie, Jiao, and Bostick (2004). b. Gall, Gall, and Borg (2007).

Qualitative Research Rules of Thumb The sample size decisions are a bit more dynamic in qualitative research than in quantitative research in that the number of observations is not determined in the former type of research prior to data

p. 343

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 21/35

yp p collection. Rather, a researcher makes a decision as to the adequacy of the observations on the basis of having identified the salient issues and finding that the themes and examples are repeating instead of extending. Thus, sample size is integrally related to length of time in the field. Nevertheless, rules of thumb for sample sizes in qualitative research can give you an estimate of the number of observations needed for different kinds of qualitative research. I return to the types of research discussed in Chapter 8 for the suggestions listed in Table 11.1.

Qualitative researchers are often challenged by the question, How do you know you have enough data? How do you know when it is time to stop? Stake (2006) acknowledges that sometimes researchers stop collecting data when their time and money run out, especially in program evaluation studies. Researchers need to plan carefully to ensure that they maximize the time and money available to them in order to do the best study within the constraints of the context. Charmaz’s (2006) advice reflects a perspective that is less bound by these constraints and is more situated in the concept of “saturation.” She suggests that researchers (in grounded theory specifically) call a halt when “gathering fresh data no longer sparks new theoretical insights, nor reveals new properties of your core theoretical categories” (p. 113).

Table 11.1 Rule of Thumb Sample Sizes in Qualitative Research

SOURCE: Adapted from P. H. Collins & Solomos (2010).

Formulaic Determinations of Sample Size Murphy, Myors, and Wolach’s (2009) book describes the logic and procedure for selection of sample size when the researcher is conducting a quantitative study of treatment effectiveness. In other words, how big does your sample have to be to obtain statistically significant results, if the treatment is indeed effective?

So what does sample size have to do with detecting statistically significant differences? Our ability to detect statistically significant differences is determined in part by the amount of variability in our dependent measure within the sample:

Less variability = greater sensitivity More variability = less sensitivity

And sample size has a direct relationship with variability:

Larger sample sizes = less variability Smaller sample sizes = more variability

If you put the logic of these two statements together, you realize that it is easier to obtain statistical significance if you have a larger sample. However, there is one sticking point:

Larger samples = more costly

p. 344

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 22/35

Smaller samples = less costly

So as a researcher, you want to know what is the smallest sample you can use that will take into account the variability in the dependent measure and still be sensitive enough to detect a statistically significant difference, if there is one.

Another important concept that enters the discussion of sample size calculation is the power of a statistical test, defined as the probability that it will lead you to reject the null hypothesis when that hypothesis is in fact wrong (Murphy et al., 2009). Thus, power in one sense is the quantification of the sensitivity. Sensitivity or power in statistical language is described in terms of probability of finding a difference when there really is one there. I digress for a moment to explain a few terms to finish the explanation of statistical power.

If we claim that we do have a real difference, but we really do not, this is called a Type I error. Usually, researchers establish a level of Type I error that they are willing to live with, and this is called an alpha level (α). For example, if researchers are willing to live with a .05 probability that they might say there is a statistically significant difference when there is not, they have established an alpha level of .05. If we claim that we do not have a real difference, but we really do, this is a Type II error (or beta or b). All this explanation was necessary because the power of a statistical test can be defined as 1 – b.

Perhaps a graphic display will help:

Table 11.2 Possible Outcomes of Statistical Tests

SOURCE: Adapted from Murphy, Myors, and Wolach (2009).

There are also tables in Murphy et al. (2009)’s book and many statistical textbooks you can use to estimate sample sizes that are based on the establishment of your acceptable alpha level and the estimated effect size you expect to find between the treatment and comparison groups. (The effect size is the difference between the two group means in terms of their common standard deviation.)

Because this section has the title “Formulaic Determinations of Sample Size,” you are probably wondering where the formulas are. I am going to provide you with one formula to give an idea of how the formulas work. However, you should be aware of a number of caveats in regard to the use of such a formula. D. Moore and McCabe (2003) identify the following assumptions that underlie the use of simplistic formulas such as the one presented here:

1. The data must be from a simple random sample. If data are not from a simple random sample, the researcher must be able to plausibly think of the data as independent observations from a population.

2. The formula is not correct for any sampling designs that are more complex than a simple random sample.

3. There are no correct methods for data that are haphazardly collected with bias of unknown size. 4. Outliers can have a large effect on the confidence interval; therefore, extreme values should be

corrected or removed before conducting the analysis. 5. Small sample sizes and nonnormality in the population can change the confidence level.

Given those caveats, Gall et al. (2007) present the following formula for estimating the size of sample needed:

p. 345

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 23/35

sample needed:

where

N = number of people needed in each group s = standard deviation of your dependent variable t = t test value needed to get your desired alpha level D = estimated difference between experimental and control groups

You may be wondering, “Where do I get the standard deviation or effect size without having already done the research?” Good question. You could rely on estimates of s and D that are available from previous research reported in the literature, or you could conduct pilot work to obtain your own estimates. You could establish an acceptable effect size based on discussions with program leaders to determine what would be a meaningful result. The t test value can be obtained from a table in any statistics book once you have established your desired alpha level.

You may also be wondering what to do in case your sampling strategy does not fit all of the caveats presented previously. In actuality, the preferred method for calculating estimated sample size is to use any of the various statistical programs that are now available for estimating power or sample size. These allow you to personalize the estimation process to your exact needs as well as to play with the estimates of sample size to weigh the differential effects on precision and cost.

You should be aware of the important implications of this discussion of sample size. Much educational and psychological research is doomed to failure before any data are collected because of insufficient power to detect the effects of an intervention. Increasing sample size is not the only route to increasing statistical power. Other alternatives include improving the delivery of the treatment, selecting other statistical tests, or raising the alpha level. Mark and Gamble (2009) discuss ethical implications of power analysis for sample size estimate in that the number of participants needed to observe the effect of interest determines how many people will be exposed to an uncertain treatment or denied access to a beneficial treatment. If power analysis is not conducted, then

more participants than needed may have been exposed to a less effective treatment (or there may have been too few participants to observe the effect of interest, reducing the potential benefit of the study). Power analyses have become relatively commonplace, reducing the magnitude of risk. (p. 205)

Of course, sample size is also influenced by the willingness of the chosen people to participate in the study. The determination of conditions under which people are recruited and informed about the research leads to ethical issues that are discussed in a later section of this chapter.

EXTENDING YOUR THINKING

Sample Size

• Compare a number of studies’ sample sizes. What justification do the researchers provide for their choice of sample size?

• Think of a simplistic experimental design that would have two groups and allow simple random sampling. Using numbers available in published literature, use the preceding formula for calculating sample size. Calculate the sample size needed for different levels of confidence in the results.

• Identify a computer program that allows you to conduct a power analysis of different sample sizes.

• Play with the program with a hypothetical example or use a published research study to determine what

p. 347

p. 346

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 24/35

the effect of having had a different sample size would have been.

Ethics and Protection of Study Participants A useful insight derived from postmodern scholars is encapsulated in Clegg and Slife’s (2009) description of the relationship between research methodology and ethics. They write,

One of the primary lessons of a postmodern approach to research ethics is that every research activity is an exercise in research ethics, every research question is a moral dilemma, and every research decision is an instantiation of values. In short, postmodernism does not permit the distinction between research methods and research ethics. (p. 24)

As Clegg and Slife (2009) make clear, ethics is not something that happens at the stage of sampling; it needs to guide the entire process of planning, conducting, and using research. However, there are specific implications for ethical behavior related to the protection of people who participate in the studies. For that reason, information about ethical review boards is included in this chapter. Most novice researchers encounter serious questions about the ethics of their planned research within the context of their institutions’ institutional review boards (IRBs) or human subjects committees. An IRB is a committee mandated by the National Research Act, Public Law 93-348. Every university or other organization that conducts biomedical or behavioral research involving human subjects is required to have an IRB if federal funding is used for research involving human subjects. The federal regulations are included in Title 45 of the Code of Federal Regulations, Part 46. You can obtain a copy of these regulations from any university’s research office, a reference librarian, or from the Office for Protection from Research Risk in the National Institutes of Health (NIH, located in Bethesda, Maryland). Information is available on the Web on IRB requirements in many locations. You can read The Belmont Report at the NIH website that addresses human subject approval processes and institutional review at http://www.hhs.gov/ohrp/humansubjects/guidance/belmont.xlink.html. It is possible to complete an NIH certificate as a part of your training program via the Web.

You should always contact your own institution’s IRB to find out its policies and procedures early in your research planning process. Sometimes, review by the IRB can occur in stages if you are planning to conduct pilot work or obtain funding from an external agency. In any case, you should be prepared to submit your research proposal in the appropriate format to the IRB well in advance of the time that you actually plan to start data collection. The IRB committee members need time to read and discuss your proposal, and they might have questions that will require some revision of your planned procedures. Lead time, an open mind, and a cooperative attitude help.

IRBs need to approve all research that is conducted that involves human subjects (with a few exceptions that will be discussed soon). If you are planning to conduct pilot testing as a part of your study, you should contact your IRB to determine its policy with regard to that. Sieber and Tolich (2013) found that, generally, IRBs do not review proposals for pilot testing if it just involves fine-tuning an instrument or a research procedure. However, they do review pilot testing that is conducted as an exploratory study to determine if additional research is necessary. Because pilot testing is an important part of many surveys and qualitative studies, it is important that you are aware of this requirement.

Certain exemptions to IRB approval are relevant to research conducted with school children. Even if you think your research might fall into one of the exempt categories, you should contact your IRB because most institutions still require an abbreviated review to establish the legitimacy of the exemption. Possible exemptions include the following:

1. Research that is conducted in established or commonly accepted educational settings, involving normal education practices, such as instructional strategies or classroom management techniques

2. Research that involves the use of educational tests if unique identifiers are not attached to the test results

The Department of Health and Human Services now includes this language that provides expedited

p. 348

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 25/35

The Department of Health and Human Services now includes this language that provides expedited review status for various types of social science research:

Research on individual or group characteristics or behavior (including, but not limited to, research on perception, cognition, motivation, identity, language, communication, cultural beliefs or practices, and social behavior) or research employing survey, interview, oral history, focus group, program evaluation, human factors evaluation, or quality assurance methodologies. (Note: Some research in this category may be exempt from the HHS regulations for the protection of human subjects. 45 CFR 46.101(b)(2) and (b)(3). This listing refers only to research that is not exempt.) (U.S. Department of Health and Human Services, 2007)

You need to give the IRB a research protocol that may be a summary of your actual proposal with specific ethical issues addressed in greater detail. For the exact specifications at your institution, again, check with your IRB. For a general sense of what the IRB might want to see in a research protocol, see Box 11.5 for a checklist. The first three items on the protocol checklist parallel closely what you would put into your research proposal itself, although the IRB usually asks for a shorter version. Items 4 through 10 on the checklist focus on the issues specific to ethical practice in research. You will notice that Item 10 concerns a consent form that you should have your study participants sign. Box 11.6 shows a sample consent form. NOTE: Check with your institution’s ethical review board. Some institutions have a standard form for the informed consent that you are required to use.

BOX 11.5 Checklist for Research Protocol

1. Cover Page, including

Name and department of the principal investigator (PI)

The PI’s faculty rank or student status

The PI’s home and office phone number and address

The project title

The type of research (e.g., faculty research, externally funded project, student-directed research)

Intended project starting and ending dates

The PI’s qualifications in a paragraph or two (or in an attached curriculum vitae or within the description of the methodology)

Signature of the PI

If PI is a student, signature of the adviser

2. A description of the research

The purpose of the research and the hypotheses

The literature review (in a summary form)

The research method, design, and mode of analysis

A realistic statement of the value of the research (specifically addressing what value it will have for the participants and their community, the research institution, the funder, or science)

The location of the research (e.g., the exact laboratory, community, institution), why that setting was chosen, and how the researcher happens to have access to it

Duration of time for the project and how that time frame relates to the project (e.g., in terms of the school calendar, etc.)

3. A description of the research participants

Demographic characteristics: for example, ethnic background, sex, age, and state of health

Rationale for the choice of this population

Source from which the research participants will be obtained

p. 349

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 26/35

A statement of the selection criteria

If vulnerable populations are included (e.g., children, mentally disabled, drug abusers), the rationale for their use should be stated

If the participants are coming from an institutional setting (e.g., school, clinic, hospital, club), written permission of the person in charge must be attached to the protocol

The expected number of participants should be provided

4. A discussion of the possible risks

Inconveniences or discomforts, especially to the participants, and an estimate of the likelihood and magnitude of harm

What will be done to allay each actual risk or unwarranted worry

Any alternative methods that were considered and why they were rejected

A justification if unique identifiers will be collected

5. Discussion of inducements and benefits to the research participants and others

6. Freedom of research participants to withdraw at any time with no adverse consequences

7. Source and amount of compensation in the event of injury (if any, although this is not usually offered to participants in educational and psychological research studies)

8. Analysis of risks and benefits (and a description of how the benefits will substantially outweigh the risks)

9. The informed consent procedure should be described

How, where, and by whom the informed consent will be negotiated

How debriefing will be conducted

Procedures for obtaining children’s consent and parental or guardian permission for the research

The actual consent form should be attached to your protocol

If oral consent is planned, provide a description of the information that will be presented

Attach a copy of the information to be used in debriefing

10. A copy of the consent form itself should be attached. It should include the following:

An explanation of the research purpose, duration, and procedures; if deception is to be used, the researcher should explain that not all of the details can be provided at this time but that a full explanation will be given later

A description of any foreseeable risk or discomfort

A description of alternative ways people can obtain the services (e.g., educational intervention or counseling) if they choose not to participate in the research

A description of how confidentiality or anonymity will be ensured

A statement as to whether compensation or treatment for harm or injury is available (if the research involves more than minimal risk)

The name of a person to contact for answers to additional questions

A statement that participation is voluntary, that refusal will not result in any penalty, and that the person is free to withdraw at any time

11. Any additional attachments (e.g., letters of permission, interview or survey questions, materials to be presented to the participants, tests, or other items connected with the research)

a. If the research is student-directed research, include the name of the faculty adviser and indicate if the research is for a thesis or dissertation or for a course requirement (include course number and faculty name).

b. The person must be given a copy of the consent form.

p. 350

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 27/35

SOURCE: Adapted from Sieber and Tolich (2013).

BOX 11.6 Sample Informed Consent Form

Project Title: Support Services for Parents of Children Who Are Deaf or Hard of Hearing Principal Investigator: Kathryn Meadow-Orlans, Senior Research Scientist, Center for Studies in

Education and Human Development, Gallaudet University, Washington, DC 20002–3695

Phone: (202) 651-XXXX (V or TDD) E-mail: [email protected]

Thank you for volunteering to participate in a group interview as a follow-up to the Gallaudet National Survey for Parents. Your signature on this consent form shows that you have been informed about the conditions, risks, and safeguards of this project.

1. Your participation is voluntary. You can withdraw from the study at any time, for any reason, without penalty.

2. There is no more than minimal risk to individuals who participate in this research, and complete confidentiality is ensured. Your name will not be used. Instead, you will be given a code number in order to guarantee your anonymity. The typed transcript of the interview will show this code number rather than your name. Your comments will be entered on a computer, and any identifying information will be changed for any written reports. Only the project investigators and their research assistants will have access to the transcript.

3. Questions about risk to you because of participation in this study may be addressed to the researcher at the phone number or e-mail listed at the top of this page or to Dr. Carolyn Corbett, Chairperson, Gallaudet University Institutional Review Board for Protection of Human Subjects (IRB) at (202) 651-XXXX (V or TDD).

4. To cover possible incidental expenses that you might have related to your participation, you will be paid an honorarium of $50. In addition, you have our deep appreciation. We believe that this study will help to improve support services for parents and for children who are deaf and hard of hearing.

I have read the information provided and agree to participate in the interview for parents.

Signature Date

Please print name

Marginalized Populations and Informed Consent

Informed Consent and Children. Vargas and Montoya (2009) address ethical concerns of doing research with children. Parents are usually the people who have legal authority to give permission for research participation for their children under the age of 18. However, ethical practice calls for getting “assent” from children by explaining the study to them in language that is understandable to them and getting their agreement to participate. In addition, certain circumstances might arise in which it is potentially not in children’s interest to have their parents’ know about their participation (e.g., if they are lesbian or gay and have not “outed” themselves to their parents). In such a situation, another adult can take responsibility for signing the informed consent as a legal representative for the child. Vargas and Montoya also discuss issues related to cultural complexity and differences in parental expectations that need to be considered when conducting research with children.

Informed Consent and Older People. Szala-Meneok (2009) explains that older people may need special care in terms of ascertaining if they consent to participate in research. Such people may sign a consent f h h l id d i h i b d l f d i I hi

p. 351

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 28/35

form when they are lucid and might experience subsequent development of dementia. In this circumstance, is the informed consent still valid? Szala-Meneok suggests that researchers who work with older people periodically revisit the consent agreement throughout the process of the study. If signs of dementia emerge, then the researcher can ask a significant other to reaffirm the consent in the interest of ethical practice.

Informed Consent and People With Mental Illness. People with mental illness vary in terms of their abilities to provide informed consent. In New York, researchers are barred from conducting experiments with mentally ill patients in the state’s psychiatric hospital without first obtaining their informed consent. This raises a conflict: How can people be declared involuntarily mentally incompetent (which is the basis for placement in the state psychiatric hospital) and execute legal documents subjecting themselves to experiments? Several approaches have been recommended, such as use of advance directives that are signed by the person when his or her symptoms do not impair his or her ability to give consent, and use of monitors such as family members, advocates, or surrogates who can safeguard the person’s interests. Ethical responsibility includes monitoring participants closely to determine any negative effects of the research, such as an increase in symptom severity. The researcher then needs to make a decision about making other treatments available as needed, defining appropriate criteria for withdrawal from the study, and building in “backup” clinical care options for participants who must leave the study (American Psychiatric Association Task Force on Research Ethics, 2006).

Indigenous and Postcolonial Peoples and Research Ethics. Indigenous and postcolonial peoples contribute critically important insights into ethics, not only for research in their own communities but also as a way of understanding broader ethical issues in the surrounding world. LaFrance and Crazy Bull (2009), Cram (2009), Chilisa (2009), and Quigley (2006) provide thoughtful descriptions of the ethical review process in American Indian, Maori, and African communities. Researchers interested in working in such communities will find sophisticated review processes grounded in cultural heritage and values.

Confidentiality and Anonymity Two terms used in the ethical review for the protection of research participants need additional clarification:

• Confidentiality means that the privacy of individuals will be protected in that the data they provide will be handled and reported in such a way that the data cannot be associated with the research participants personally.

• Anonymity means that no uniquely identifying information is attached to the data, and thus no one, not even the researcher, can trace the data back to the individual providing them.

Confidentiality and anonymity promises can sometimes be more problematic than anticipated. See the case described in Box 11.7 for an example of an unanticipated complication. In addition, transformative researchers suggest that the expectation of confidentiality or anonymity that represents the status quo in research may be misguided. Baez (2002) makes the case that confidentiality protects secrecy and thus hinders transformative political action. “Transformative political action requires that researchers and respondents consider themselves involved in a process of exposing and resisting hegemonic power arrangements, but such action is thwarted by secrecy and the methods used to protect it” (p. 35). Baez does not recommend discarding confidentiality entirely; rather, he recommends that researchers critically question what confidentiality allows to be hidden or revealed during the entire process of the research. This illustrates the tensions inherent in the need to protect individuals who have experienced discrimination and oppression by not reporting or altering the details to protect their identity with the need to reveal specific examples of such behaviors in the name of accuracy of results as well as to further social transformation and justice. Researchers need to be cognizant of the repercussions of revealing the identity of persons who provide data.

p. 353

p. 352

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 29/35

BOX 11.7 Protection of Human Participants and the Law

Sheldon Zink is an ethnographer who conducted a study in a hospital on the experiences of a patient who received an experimental heart transplant. After conducting the study for 18 months, the patient died. The family sued the hospital and various others involved in the patient’s care. The lawyers issued a subpoena for Zink’s field notes. Zink refused to turn over her notes because she had promised confidentiality to the participants in the study. Five lawyers told Zink that she would have to choose between turning over her notes and going to jail because the confidentiality of an ethnographer’s notes is not protected by law the way a lawyer’s or physician’s records are. The American Anthropological Association’s code of ethics says that anthropologists must do everything in their power to protect their subjects, including maintaining confidentiality and anonymity. However, it also adds that it should be made clear to informants that such anonymity may be compromised unintentionally. As in Zink’s case, the law, as it is currently written, requires researchers to release their notes or go to jail.

SOURCE: R. Wilson (2003).

EXTENDING YOUR THINKING

Confidentiality Using pseudonyms or fictionalizing details are two ways that researchers deal with confidentiality issues. However, this may not be sufficient protection for the participants as witnessed in an experience I had at a professional meeting in which a researcher reported a study in which the participant was described in an unflattering way (to put it mildly). The researcher said that she had made a previous presentation on this study in another venue, and one person in the audience told her afterward that she knew who the participant was and assured the researcher that she had been very accurate in her portrayal of the woman in question. The researcher had not used the name of the participant; however, she revealed sufficient details about the participant that the audience member was able to identify her. I wondered if the participant knew how she was being portrayed and if she had agreed to the use of her data in that way. The researcher told me that she had not shared her analysis of the data with the participant and that she did not see a need to do so because the participant had signed an informed consent letter. What are your thoughts about the ethics of reporting the information in this study? What responsibility does the researcher have after the informed consent is signed?

Several other scholars write about the complexity of confidentiality, for example:

• Ntseane (2009) conducted a study of poverty reduction in Africa based on data collected from women who owned businesses. The women told Ntseane that they wanted their names used because they were proud of the work they were doing and felt that if she could name researchers in her writing who wrote articles about economic development, surely it would be insulting to them not to use their names as they were the ones who had succeeded in the struggles of business in Africa.

• Dodd (2009) discussed the real dangers of revealing the identities of youth who are lesbian, gay, bisexual, transgender, or queer who had not revealed this to their parents, teachers, or others in the outside community.

• Brabeck and Brabeck (2009) collected data from Spanish-speaking women who were abused by their spouses. Some of the women wanted their names used in the written reports; the researchers counseled them against that based on the need to protect the women from further abuse.

• J. A. King, Nielsen, and Colby (2004) wrestled with the question of revealing identities when the study they conducted indicated that an incompetent manager was responsible for the failure of a program.

h f h l hi hli h h i b h i h id hi l i

p. 354

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 30/35

Each of these examples highlights the tension between the assumptions that guide ethical review boards in terms of the necessity for and feasibility of maintaining confidentiality or anonymity, as well as providing insights into the dynamics of maintaining an ethical stance throughout the research process. Bhattacharya (2007) proposed modifications to IRB guidelines that would acknowledge the fluid nature of consenting and ways the researcher could be responsive to potential departures from the traditional form that is signed at the beginning of a study. She acknowledges that researchers may not be able to anticipate all the ethical dilemmas that will present during a study. If a participant does not fully understand the implications of revealing details about his or her life, alternative means might be used to elicit the degree of revelation that is safe and comfortable. For example, Bhattacharya asked her respondent if all the things revealed in the data were things she would feel comfortable if her mother or grandmother knew about her. Another possibility is to require consent at the time of publication or by asking the participants to coauthor reports of the research. These strategies of course introduce additional challenges for researchers, who may feel that their integrity is being sacrificed to protect the participants.

As discussed in the section on access to records earlier in this chapter, it is sometimes possible to obtain data within the context of anonymity by having someone other than the researcher draw the sample and delete unique identifying information. Sieber and Tolich (2013) also suggest the possibility of having a respondent in a mail survey return the questionnaire and mail a separate postcard with his or her name on it. Thus, the researcher would be able to check off those who had responded and to send a second mailing to those who had not.

However, in many instances, this is not feasible, and the researcher must arrange to respect the privacy and confidentiality of the individuals in the research study. This can be done by coding the data obtained and keeping a separate file with the code linked to unique identifying information. The separate file can then be destroyed once the necessary data collection has been completed.

Federal legal requirements concerning confidentiality include the following:

1. The Buckley Amendment, which prohibits access to children’s school records without parental consent

2. The Hatch Act, which prohibits asking children questions about religion, sex, or family life without parental permission

3. The National Research Act, which requires parental permission for research on children

There are two circumstances in which the IRB can choose not to require parental permission:

1. If the research involves only minimal risk (i.e., no greater risk than in everyday life), parental permission can be waived.

2. If the parent cannot be counted on to act in the best interests of the child, parental permission can be waived. This circumstance usually involves parents who have been abusive or neglectful.

As a part of the confidentiality issue, the research participants should also be informed that researchers are required by law to inform the appropriate authorities if they learn of any behaviors that might be injurious to the participants themselves or that cause reasonable suspicion that a child, elder, or dependent adult has been abused. In Ross’s (1995) study of Native American women in prison, for example, she told her participants that she was required to report to the warden if they revealed involvement in illicit activities, such as drug usage or escape plans.

EXTENDING YOUR THINKING

Ethical Review Boards Contact your institution’s IRB. Review its requirements in terms of review of proposals and format of prospectus. Write a sample research prospectus that you think would satisfy your institution’s IRB. (A smaller bite of the same apple: Write an informed consent form that you could use for a sample study.

p. 355

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 31/35

s a e b te o t e sa e app e: W te a o ed co se t o t at you cou d use o a sa p e study. Variations on a theme: Write the informed consent form for an adult, for a parent of a child, for an adult with limited mental abilities.)

Deception in Research Studies The American Psychological Association (2002) recognizes that deception and invasion of privacy must be given serious consideration in research planning. Deception is an ethical problem that has been debated in the research community for many years. The justification put forward for the use of deception is usually that the results of the study would be compromised without it because people would alter their behavior if they knew what the researcher was really investigating. Most professional associations’ ethical guidelines for psychologists and educators prohibit the use of deception unless it can be justified and the effect of the deception “undone” after the study is completed. The undoing of deception is supposed to be accomplished by the following means:

1. Debriefing the research participants after the research study; this means that the researcher explains the real purpose and use of the research.

2. Dehoaxing the research participants, in which the researcher demonstrates the device that was used to deceive the participants. The researcher’s responsibility is to attempt to allay a sense of generalized mistrust in educational and psychological research.

3. Guarding the privacy and confidentiality of the research participants. 4. Obtaining fully informed consent.

Years ago, Guba and Lincoln (1989) maintained that the allowance of deception in research settings was one of the main failings of the postpositivist paradigm. They point out that the professional associations’ codes of ethics that focus on harm are inadequate to guard against the harm that results from discovering that you have been duped and objectified. Such harm includes “the loss of dignity, the loss of individual agency and autonomy, and the loss of self-esteem” (p. 121). They point out the contradiction in using deception to serve the search for “truth” through science. The requirement for fully informed consent and use of deception also creates a contradiction for the researcher: How can people give their fully informed consent to participate in a research study if they do not know what the real purpose of the research is?

Lincoln (2009) argues that deception cannot be a part of the constructivist paradigm because the goal is to collect and debate the various multiple constructions of the different constituencies affected by an issue. Nevertheless, researchers functioning within the constructivist paradigm are not immune to ethical challenges. The following excerpt provides one example of an ethical dilemma that arose during a study conducted within the parameters of the constructivist paradigm. Gary Fine and Kent Sandstrom (1988) describe the following situation in their study of White, preadolescent boys:

One day I was driving some boys home, we passed some young Blacks riding bicycles in that almost entirely White suburb. One boy leaned out the car window and shouted at the “jungle bunnies” to “go back where you came from.” The ethical problem was what to do or say in reaction to this (and similar) behaviors. In this instance (and others), I offered no direct criticism, although a few times when the situation was appropriate, I reminded the boys of the past prejudices against their own ethnic groups [Irish American]. (pp. 55–56)

Fine comments that he made the judgment not to react to these racist comments because he wanted the children to continue to trust him. This raises other ethical issues in terms of how far researchers should go to engender the trust of their informants. G. Fine and Sandstrom raise the questions: Should you smoke a joint? Join in a gang fight? Commit a crime?

By being present and tolerant of drug use, racist behavior, and so on, is one supporting that behavior? G. Fine and Sandstrom (1988) comment that “one must wonder whether the researcher who ‘enables’ drug dependency or who permits crimes to occur is really acting in accord with the presumption of ‘doing no harm’” (p. 68). For another ethical dilemma, see Box 11.8 on playing the

p. 356

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 32/35

p esu pt o o do g o a (p. 68). o a ot e et ca d e a, see o .8 o p ay g t e card game “Asshole” with abused and at-risk youth.

BOX 11.8 Ethical Dilemma Related to Playing a Card Game With Abused and At-Risk Youth

Donna Podems (2004) designed her dissertation combining action research and feminist approaches to understand a donor-funded program in South Africa. Abused youth participated in wilderness programs designed to build their self-esteem and self-confidence. The program used experiential learning techniques, with a heavy focus on games. She provides the following description of her experience:

I had met with the staff individually and explained the research and each staff signed a consent form promising them confidentiality, among other standard practices. About three months into my research, one male staff member suggested that we play a card game called “Asshole.” Not being a card player, I did not know this game. They explained the game in that the person who got rid of their cards first was the president, then vice-president, then assistant to the asshole, then asshole. The loser has to do what everyone says, which can be getting drinks to something unmentionable.

I asked another card player, who is my best friend, how he felt after this game. He admitted rather sheepishly, that he was “relieved” and felt “glee” that I was the asshole for most of the game, because it meant that he was not. Playing the game with me, he watched me grow more and more upset, yet he, and the others, let the game continue.

Having played this game with the Program staff, I was horrified at their apparent joy at my distress. Even though they saw that I was upset, they wanted to continue the game.

I had a horrible thought: was this game played with emotionally and physically abused kids? I questioned staff members and each staff member that I spoke with said yes, the staff played it with emotionally and physically abused kids, just not officially; they played it “all the time” after dinner in “open” or “rest” hours. Most staff that I interviewed even said that the person in the asshole position usually became upset but explained that was because they were a poor sport. The staff members defended the use of this game and chided me for my feelings.

This was my first of many experiences with a program where what the staff said they believed (we care about people) and were about (growing/healing people) and what they actually believed and did were two different things. I was left with the following dilemma: I accessed the fact that this game was played with emotionally abused and at-risk youth through confidential interviews, yet it greatly disturbed me that this game might be negatively affecting youth. Should I tell the director, or was I bound by confidentiality? (Donna Podems, personal communication, December 2009)

EXTENDING YOUR THINKING

Ethics and Deception in Research The question of whether or not researchers should be allowed to use deception in their research has been hotly debated in the research community and in the wider society. Sieber and Tolich (2013) summarized their ethical stance with regard to deception as follows:

There are four defensible justifications for deception research:

1. To obtain data that would be unobtainable if subjects knew the real purpose of the research.

2. To achieve stimulus control or random assignment of subjects to conditions.

3. To study responses to low-frequency events, such as threat of physical aggression.

4. To avoid serious risk to subjects. For example, in research on aggression, one may employ a confederate who will not escalate the conflict beyond the level needed for the purposes of the research. (p. 149)

Wh t i i i ? Sh ld d ti b ll d i h? If d h t diti ?

p. 358

p. 357

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 33/35

What is your opinion? Should deception be allowed in research? If so, under what conditions? What do you think of the conditions permitted by the various professional associations cited in this chapter (e.g., with debriefing, dehoaxing, informed consent)?

Questions for Critically Analyzing Sampling Strategies

All of these questions might not be equally important for research conducted within the different paradigms. For example, an affirmative answer to the question about generalizability or transferability might not be as important to a researcher in the interpretive or transformative paradigms. Nevertheless, an answer to the question is still possible and informative.

1. What is the population of interest? How was the sample chosen—probability, purposeful, convenience sampling? What are the strengths and weaknesses of the sampling strategy?

2. What are the characteristics of the sample? To whom can you generalize or transfer the results? Is adequate information given about the characteristics of the sample?

3. How large is the population? How large is the sample? What is the effect of the sample size on the interpretation of the data?

4. Is the sample selected related to the target population? 5. Who dropped out during the research? Were they different from those who completed the study? 6. In qualitative research, was thick description used to portray the sample? 7. In qualitative research, what is the effect of using purposive sampling on the transferability to

other situations? 8. Are female participants excluded, even when the research question affects both sexes? Are male

subjects excluded, even when the research affects both sexes? 9. Does the researcher report the sample composition by gender and other background

characteristics, such as race or ethnicity and class? 10. How does the researcher deal with the heterogeneity of the population? Are reified stereotypes

avoided and adequate opportunities provided to differentiate effects within race/gender/disability group by other pertinent characteristics (e.g., economic level)?

11. Did the researcher objectify the human beings who participated in the research study? 12. Did the researcher know the community well enough to make recommendations that will be

found to be truly useful for community members? 13. Did the researcher adequately acknowledge the limitations of the research in terms of contextual

factors that affect its generalizability or transferability? 14. Whose voices were represented in the research study? Who spoke for those who do not have

access to the researchers? Did the researchers seek out those who are silent? To what extent are alternative voices heard?

15. If deception was used in the research, did the researcher consider the following issues (adapted from Sieber & Tolich, 2013): a. Could participant observation, interviews, or a simulation method have been used to produce

valid and informative results? b. Could the people have been told in advance that deception would occur so they could then

consent to waive their right to be informed? c. How are the privacy and confidentiality of the participants ensured?

p. 359

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 34/35

d. If you are studying bad behavior, have the people agreed to participate in the study? Can you run a pilot group in which you honestly inform people of the type of behavior you are studying and determine if they would agree to participate?

e. If studying bad behavior, is the behavior induced? How strongly? f. How will debriefing, dehoaxing, and desensitizing (removing any undesirable emotional

consequences of the research) be handled? g. Is the study important enough and well-designed enough to justify deception?

EXTENDING YOUR THINKING

Critically Analyzing Sampling in Research Identify research studies that exemplify different paradigms and different methods. Use the questions for critical analysis to thoroughly critique their sampling section and ethical integrity.

Summary of Chapter 11: Sampling

Different paradigmatic stances are used to raise questions about appropriate sampling strategies. For example, in postpositivist quantitative research, probability-based sampling strategies raise questions concerning the generalizability of the results. With the constructivist paradigm, purposeful sampling raises questions about the transferability of results. The transformative paradigm provides an opportunity to ask questions about issues of power and respect for those who are included or excluded from participation in research. All researchers share concerns about ethics and have ethical review boards, professional codes of ethics, and cultural awareness to guide them in proper sampling procedures. In the next chapter, we move to the collection of data from the people who have been chosen as participants in our research studies.

Note

1. The National Dissemination Center for Children with Disabilities (NICHCY) is no longer in operation. Its funding from the U.S. Department of Education’s Office of Special Education Programs (OSEP) ended on September 30, 2013. The website and all its free resources will remain available until September 30, 2014.

p. 360

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 35/35

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods