More Research

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 1/39

B

CHAPTER 13

Data Analysis, Interpretation, and Use

Beware of testing too many hypotheses; the more you torture the data, the more likely they are to confess, but confession obtained under duress may not be admissible in the court of scientific opinion.

—Stigler, 1987, p. 148 (cited in Mark & Gamble, 2009, p. 210)

My personal view is that p-values should be relegated to the scrap heap and not considered by those who wish to think and act coherently.

—Lindley, 1999, p. 75

In This Chapter • Common types of statistics used for quantitative data analysis are defined, along with methods for

choosing among them.

• Interpretation issues relevant to quantitative data analysis are discussed, including randomization, sample size, statistical versus practical significance, cultural bias, generalizability, and options for reporting quantitative results, such as effect sizes and variance accounted for, replication, use of nonparametric statistics, exploration of competing explanations, recognition of a study’s limitations, and a principled discovery strategy.

• Statistical synthesis (i.e., meta-analysis) as a literature review method is explained.

• Options for qualitative analysis are described, along with selected computer programs that are available.

• Interpretation issues related to qualitative data analysis are discussed, including use of triangulated data, audits, cultural bias, and generalization of results.

• Mixed methods analysis and interpretation issues are addressed.

• Development of a management plan for conducting a research study is described as a tool to be included in the research proposal.

• Writing research reports is described in terms of dissertation and thesis requirements, alternative reporting formats (including performance), and publication issues.

• Strategies are discussed for improving the probability of the utilization of your research results.

y reading and studying this book, you have moved through the steps of preparing a research proposal or critiquing a research study to the point of data analysis. If you are preparing a research proposal, your next step is to describe the data analysis strategies that you plan to use.

In most research proposals, this section is followed by a management plan that specifies what tasks you will complete within a specified time frame and what resources will be required to complete the research project. Then, you would be in a position to complete the research study itself and to write up the results. Thus, the organizing framework for this chapter is designed to take you through the data analysis decisions, the design of a management plan, and ideas concerning writing research. If your goal is to critique research (rather than conduct it yourself), you will find guidelines that will help you identify the strengths and weaknesses of this portion of a research study.

A final section addresses the utilization of research results Although this section appears at the end

p. 418

p. 417

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 2/39

A final section addresses the utilization of research results. Although this section appears at the end of this text, ideas to enhance utilization have been integrated throughout the descriptions of the research planning process in this text. If you wait until after the research is finished to consider utilization, chances are that your research could become a “dust catcher” on someone’s shelf. That would not be a happy ending after all your work, so it is important to build in strategies for utilization during your planning process.

Quantitative Analysis Strategies Will struggling first-grade language-minority readers who receive tutoring make greater gains in their reading achievement than struggling readers who do not receive tutoring but are enrolled in the same schools (Ehri, Dreyer, Flugman, & Gross, 2007)? How do experiences of discrimination relate to levels of engagement in school for African American youth (Smalls, White, Chavous, & Sellers, 2007)? These are the types of questions for which researchers use quantitative research methods to investigate. Brief descriptions of two studies that explored answers to these questions are provided in Box 13.1. The analytic and interpretive strategies used in these studies are provided as examples of the various concepts described in this section of the chapter.

BOX 13.1 Brief Descriptions of Two Quantitative Studies

Study 1: Reading Rescue: An Effective Tutoring Intervention Model for Language- Minority Students Who Are Struggling in First Grade (Ehri, Dreyer, Flugman, & Gross, 2007) The researchers wanted to test the effectiveness of Reading Rescue, a tutoring program in which school staff provide tutoring to first-grade students in phonological awareness, systematic phonics, vocabulary, fluency, and reading comprehension. They compared students in the Reading Rescue condition with students who had a small-group intervention and with a control group that had neither intervention. The majority of the tutored students reached average reading levels, whereas the majority in both control groups did not.

Study 2: Racial Ideological Beliefs and Racial Discrimination Experiences as Predictors of Academic Engagement Among African American Adolescents (Smalls, White, Chavous, & Sellers, 2007) These researchers studied the relationship between African American adolescents’ experiences with discrimination and their academic engagement outcomes. Experience with discrimination was measured by rating 17 experiences (e.g., having your ideas ignored, being insulted) using a scale from 0 = never to 5 = once a week or more) and how bothered they were by the discrimination experience (with scores ranging from 0 = has never happened to 5 = bothers me extremely). A composite score was created by multiplying respondents’ ratings of the frequency of each event by their ratings of how much the event bothered them and then averaging across the product scores for the 17 events to create a composite racial discrimination score.

Negative school behaviors were measured by a response scale ranging from 1 = never to 6 = more than 20 times with four negative school behaviors: (a) skipped a class without a valid excuse, (b) got into a fight at school, (c) been sent to the principal’s office for doing something wrong, and (d) cheated on tests or exams. Using hierarchical linear regression analysis, the researchers concluded that students who experienced higher levels of racial discrimination also reported more negative school behaviors.

Commonly Used Quantitative Data Analysis Techniques It is not possible to explain all the different types of statistics, the derivation of their formulas, and their appropriate uses in this chapter. The reader is referred to general statistics books for more specific information on this topic (see, e.g., Gelman & Hill, 2007; Holcomb, 2012; Stevens, 2009; Urdan, 2010). First, I define and give examples of some of the more commonly used quantitative data analysis techniques. Then, I provide you with a model to aid you in making decisions about the most appropriate data analysis techniques. Finally, I discuss issues related to the interpretation of quantitative data analysis results.

p. 419

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 3/39

Statistics can be thought of as being descriptive (i.e., they describe characteristics of your sample), correlational (i.e., they describe the strength and direction of relationships), and inferential (i.e., they allow you to make group comparisons). Box 13.2 provides definitions of the most commonly used descriptive, correlational, and inferential statistics.

BOX 13.2 Definitions of Commonly Used Statistics

Descriptive Statistics: Statistics whose function it is to describe or indicate several characteristics common to the entire sample. Descriptive statistics summarize data on a single variable (e.g., mean, median, mode, standard deviation). Measures of Central Tendency

Mean: The mean is a summary of a set of numbers in terms of centrality; it is what we commonly think of as the arithmetic average. In graphic terms, it is the point in a distribution around which the sum of deviations (from the mean point) is zero. It is calculated by adding up all the scores and dividing by the number of scores. It is usually designated by an X with a bar over it (X) or the capital letter M. Median: The median is the midpoint in a distribution of scores. This is a measure of central tendency that is equidistant from low to high; the median is the point at which the same number of scores lies on one side of that point as on the other. Mode: The mode is a measure of central tendency that is the most frequently occurring score in the distribution.

Measures of Variability

Range: The range is a measure of variability that indicates the total extension of the data; for example, the numbers range from 1 to 10. It gives the idea of the outer limits of the distribution and is unstable with extreme scores. Standard Deviation: The standard deviation is the measure of variability—that is, the sum of the deviations from the mean squared. It is a useful statistic for interpreting the meaning of a score and for use in more sophisticated statistical analyses. The standard deviation and mean are often reported together in research tables because the standard deviation is an indication of how adequate the mean is as a summary statistic for a set of data. Variance: The variance is the standard deviation squared and is a statistic used in more sophisticated analyses.

Correlational Statistics: Statistics whose function it is to describe the strength and direction of a relationship between two or more variables.

Simple Correlation Coefficient: The simple correlation coefficient describes the strength and direction of a relationship between two variables. It is designated by the lowercase letter r. Coefficient of Determination: This statistic is the correlation coefficient squared. It depicts the amount of variance that is accounted for by the explanatory variable in the response variable. Multiple Regression: If the researcher has several independent (predictor) variables, multiple regression can be used to indicate the amount of variance that all of the predictor variables explain.1

Inferential Statistics: Statistics that are used to determine whether sample scores differ significantly from each other or from population values. Inferential statistics are used to compare differences between groups.

Parametric Statistics: Statistical techniques used for group comparison when the characteristic of interest (e.g., achievement) is normally distributed in the population, randomization is used i l l i ( Ch 11) d/ i ( Ch 4) d h i l

p. 420

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 4/39

in sample selection (see Chapter 11) and/or assignment (see Chapter 4), and the interval or ratio-level of measurement is used (e.g., many test scores).

t tests: Inferential statistical tests are used when you have two groups to compare. If the groups are independent (i.e., different people are in each group), the t test for independent samples is used. If two sets of scores are available for the same people (or matched groups), the t test for correlated samples is used.

ANOVA: The analysis of variance is used when you have more than two groups to compare or when you have more than one independent variable.

ANCOVA: The analysis of covariance is similar to the ANOVA, except that it allows you to control for the influence of an independent variable (often some background characteristic) that may vary between your groups before the treatment is introduced.

MANOVA: The multivariate analysis of variance is used in the same circumstances as ANOVA, except that you have more than one dependent variable.

Structural Equation Modeling: SEM is used to test complex theoretical models or confirm factor structures of psychological instruments. It can assess relationships among both manifest (observed) and latent (underlying theoretical constructs) variables. For further information, see Chan, Lee, Lee, Kubota, and Allen (2007).

Nonparametric Statistics: Statistical techniques used when the assumption of normality cannot be met, with small samples sizes, and with ordinal (rank) or nominal (categorical) data.

Chi-Square: Used with nominal-level data to test the statistical independence of two variables.

Wilcoxon Matched Pairs Signed-Ranks Test: Used with two related samples and ordinal-level data.

Mann-Whitney U Test: Used with two independent samples and ordinal-level data.

Friedman Two-Way Analysis of Variance: Used with more than two related samples and ordinal-level data.

Kruskal-Wallis One-Way Analysis of Variance: Used with more than two independent samples and ordinal-level data.

Descriptive Statistics Researchers commonly report means and standard deviations for the descriptive statistics portion of their report. The usual format is to first state the mean and then show the standard deviation in parentheses immediately following the mean. Ehri et al. (2007) measured students’ reading comprehension using a scale of the Gates-MacGinitie Reading Tests (4th ed.; MacGinitie, MacGinitie, Maria, & Dreyer, 2002). The results were as follows—experimental group: a mean of 43.5 with a standard deviation of 12.9; for Control Group 1: 37.8 (13.8); and for Control Group 2: 35.3 (11.2). Sample size is usually indicated by the letter n and in this case, the sample sizes for the three groups were 62, 60, and 60, respectively. In the Smalls et al. (2007) study, they reported descriptive statistics as follows:

With regard to primary predictor variables, we found that participants on average reported experiencing racial discrimination infrequently (M = 1.65, SD = 1.05 on a scale of 1 to 5). However, only 3.6% of participants reported experiencing none of the racial discrimination events over the past year. (p. 314)

p. 421

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 5/39

In this study, they use the letter M to indicate the mean and SD for standard deviation.

Correlational Statistics Smalls et al. (2007) wanted to test the strength of the relationship between their predictor variables and school outcomes. They reported simple correlation coefficients between the variables: “Higher racial discrimination scores were related to . . . more [self-reported] negative school behaviors (r = .12, p < .05)” (p. 315). The letter r is used to stand for the correlation coefficient statistic. They also chose to use a hierarchical linear regression technique2 that allowed them to test the relationship of blocks of predictor variables in the same statistical analysis. They used three blocks of predictor variables: background (e.g., gender), racial identity variables, and racial discrimination. They reported that experiencing more racial discrimination related to more self-reported negative behaviors at school (beta = .13, SE = .01, p < .01; F (14, 390) = 2.18, p < .008). The researchers also report that the model accounts for 13% of the variance in negative school behaviors. In English, this parenthetical expression would be read: Beta equals .13, standard error equals .01, and significance level of p is less than .01 for the racial discrimination variable alone. The F value is a test of the statistical significance of the full model of prediction of negative school behaviors. In English, this reads: F equals 2.18 with 14 and 390 degrees of freedom, and a significance level of p less than .008.

Beta is a standardized regression coefficient obtained by multiplying the regression coefficient by the ratio of the standard deviation of the explanatory variable to the standard deviation of the response variable. Thus, a standardized regression coefficient is one that would result if the explanatory and response variables had been converted to standard z scores prior to the regression analysis. This standardization is done to make the size of beta weights from regression analysis easier to compare for the various explanatory variables.

Researchers use the symbol R2 to indicate the proportion of variation in the response variable (in this case, negative school behaviors) explained by the explanatory variable (in this case, experience with racial discrimination) in this multiple regression. F is the statistic used to determine the statistical significance of this result. That is, is the contribution of the explanatory variable to the prediction of the response variable statistically significant? And p is the level of statistical significance associated with F. (Statistical significance is explained in the next section.)

Degrees of freedom indicate the appropriate degrees of freedom for determining the significance of the reported F statistic. F distributions are a family of distributions with two parameters—the degrees of freedom in the numerator of the F statistic (based on the number of predictor variables or groups) and those associated with the denominator (based on the sample size). If you know the number of explanatory variables and the sample size, the computer program will calculate the appropriate degrees of freedom and will use the appropriate sampling distribution to determine the level of statistical significance.

On the basis of these results, Smalls et al. (2007) conclude that experiences of racial discrimination may increase students’ disenfranchisement with school. They suggest a need to make issues of discrimination and oppression more visible so that students can potentially avoid being caught in a negative spiral. They hypothesize that if students are taught to recognize oppression and to understand it in terms of a shared history with other minority groups, then they may be able to develop effective strategies for resistance.

Statistical Significance B. Thompson (2002a) defines statistical significance testing in terms of the calculated probability (p) with possible values between .00 and 1.00 of the sample statistics, given the sample size, and assuming the sample was derived from a population in which the null hypothesis (H0) is exactly true. The null hypothesis is the statement that the groups in the experiment do not differ from one another or that there is no statistically significant relationship between two or more variables. Several important concepts are included in that description: Statistical testing is probability based, sample size influences statistical significance, the sample should be representative of the population, and the probability that is calculated reflects the probability that the null hypothesis can be rejected. A test of statistical significance indicates whether researchers can accept or reject the null hypothesis and the level of

p. 422

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 6/39

significance indicates whether researchers can accept or reject the null hypothesis and the level of confidence they could have in their decision. When you read in a research report that the results were significant at the .05 level (usually depicted as p < .05), the researcher is telling you that there is a 5% chance that he or she rejected a null hypothesis that was true. In other words, there is a 5% chance that the researcher made a mistake and said there is a statistically significant difference (or relationship) when there really is not. This is called a Type I error. (The converse of this is a Type II error; that is, the researcher fails to reject a false hypothesis.) In the Smalls et al. (2007) study, the researchers rejected the null hypothesis that there is no relationship between experiences of discrimination and negative school outcomes. Their hierarchical linear regression results produced a statistical significance level of .008. Thus, the researchers rejected the null hypothesis that no statistically significant relationship existed between experiences of discrimination and self-reported negative school behaviors.

The concept of statistical significance is not unproblematic (B. Thompson, 2002b). The American Psychological Association (APA) actually considered banning the use of statistical significance testing. However, they revised their position to recommend the use of effect sizes, confidence intervals, and meta-analysis to provide a more accurate picture of effects (APA, 2009). Issues associated with the decision to use a test of statistical significance are discussed in two subsequent sections of this chapter: Interpretation Issues in Quantitative Analysis and Options for Reporting Statistical Results.

Inferential Statistics Researchers with two groups or one group with two points of measurement can use a t test to compare scores, if the data are continuous. With two groups, you would use an independent t test; with one group and two points of measurement, you would use a t test for dependent samples. For example, Elbaum (2007) wanted to know if students would improve their performance on mathematics tests if someone read the test to them compared to standard test administration. She also wanted to know if students with and without learning disabilities would differ in their performance with different administration conditions. Before Elbaum proceeded with the test of the main research question, she wanted to confirm that middle school and high school students did not differ from each other, so she conducted a t test that resulted in no statistically significant differences for either students with learning disabilities, t (386) = −1.07, p = .28, or students without disabilities, t (235) = 0.44, p = .66 (p. 224). Therefore, she progressed to testing the effects that were of primary interest.

If researchers want to compare two groups when the data are categorical (e.g., frequency data), then a t test would be inappropriate. However, they can use a chi-square test (χ2) to determine statistically significant differences between groups. Marini, Wang, Etzbach, and Del Castillo (2013) used three categories to investigate undergraduate’s willingness or lack of willingness to be friends with, date, or marry a person who uses a wheelchair. They wanted to compare on the basis of race/ethnicity (White/Hispanic), gender (male/female), and having had a previous relationship with a wheelchair user. The independent variable was categorical (willing compared to less than very willing). The χ2 was significant for race/ethnicity (Whites being more willing), gender (females being more willing), and experience with persons who use wheelchairs (those with experience were more willing).

Researchers have two other nonparametric tests for comparisons of ordinal (rank) data with two groups. In Lindgren, Baigi, Apitzsch, and Berg’s (2011) study of an exercise program for high school girls, they used the Mann-Whitney U-test for comparisons between groups and the Wilcoxon matched- pairs signed-rank test for comparisons within groups.

The definition of ANOVA in Box 13.2 is a bit oversimplified. If researchers have one independent variable with more than two levels (e.g., three approaches to reading instruction), then they would use ANOVA. However, if the researchers had more than one independent variable (e.g., two types of test administration and presence or absence of a learning disability), then they would need to conduct a factorial ANOVA. Elbaum (2007) had two independent variables: type of administration and disability status. Therefore, her study is an example a 2 × 2 factorial design that was analyzed using a factorial ANOVA. The factorial ANOVA allows you to test for main effects (i.e., is there a significant effect for each of the independent variables?) and for interaction effects.

This design can be depicted as follows:

A

p. 424

p. 423

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 7/39

B

A × B

where

A means administration by read-aloud or standard administration procedures B means a student has a learning disability or does not have one A × B is the interaction of the two variables

Elbaum’s (2007) results indicated that math scores were significantly higher by disability status [F (1,623) = 275.56, p < .001], where students without a learning disability scored higher than those with a disability. There was also a statistically significant difference by the type of administration [F(1,623) = 86.21, p < .001]. The read-aloud administration resulted in higher scores than the standard administration condition.

Before drawing conclusions, it was important to also test if there were any interaction effects of the variables (i.e., did the independent variables vary systematically with each other?). In this study, the researcher also reported a significant interaction between disability and test condition [F (1,623) = 13.87, p = < .001]. The researcher concluded, “Overall, students without disabilities benefited more from the read-aloud accommodation than did students with LD” (Elbaum, 2007, p. 224). Interpretation of interaction effects is made far easier by graphing the disaggregated results, as can be seen in Figure 13.1.

The graphical display makes it clear that the people in the read-aloud group uniformly scored higher on their math tests. However, students without disabilities not only started with higher scores, but they also improved their scores more than the students with learning disabilities did under the read-aloud condition.

Two nonparametric analysis of variance tests can also be used: the Friedman two-way analysis of variance for two related samples with ordinal data and the Kruskal-Wallis one-way analysis of variance with more than two independent samples and ordinal data. For example, Gannon, Becker, and Moreno (2012) studied the effect of religiosity on mentions of sexual behavior on Facebook for college freshman. They determined that the characteristics of interest in their sample were not normally distributed; therefore, they used the Kruskal-Wallis test to analyze references to sexual behavior by religious affiliation and mentions of religiosity on their Facebook page. Their findings included (a) number of sexual references was not different for having or not having a religious affiliation and (b) a significant difference did appear between those who made frequent references to religiosity versus those who did not.

Figure 13.1 Graphical Depiction of an Interaction Effect

p. 425

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 8/39

SOURCE: Elbaum (2007).

Variations on ANOVA

ANCOVA. It is also possible to have another type of “independent” variable—one that is not of central interest for the researcher but that needs to be measured and accounted for in the analysis. This type of variable is called a covariate. It might be entry-level test scores or socioeconomic indicators for two different groups. In this case, the researcher would use an analysis of covariance (ANCOVA). In the Ehri et al. (2007) study, the researchers needed to use this statistical technique because they were interested in determining the statistical significance of group differences in reading, while controlling for entry-level reading ability. Therefore, they conducted an ANCOVA that held the pretest scores constant for the three groups, and then they tested for group differences. The results revealed an F value of 7.05, p < .001, indicating that a statistically significant difference was found in comparing the three groups. (To find out which groups had scores that were statistically different from each other, read the section below on post hoc analysis.)

MANOVA. If you have included more than one dependent measure in your design, you may need to use a multivariate analysis of variance (MANOVA). If you are getting into this level of complexity in your analysis, you definitely need to refer to a statistics book.

Post Hoc Analysis Once you have completed an analysis, such as a three-way ANOVA or a MANOVA, you need to determine where the significant effects are. This can be done using multiple-comparison t tests or other post hoc procedures, such as Tukey’s, Scheffe’s, or Bonferroni post hoc tests. Such post hoc tests allow you to focus on which of several variables exhibit the main effect demonstrated by your initial analysis. Ehri et al. (2007) did a post hoc analysis (Bonferroni post hoc pairwise comparisons) that showed that tutored students significantly outperformed both the control groups (small-group intervention and no intervention groups), who did not differ from each other.

Choice of Statistic Before explaining in detail the decision strategy for choosing a statistical procedure, I wish to digress for a moment to describe one concept on which the basis for choosing a statistical procedure rests—the scale of measurement. As a researcher you need to ask, What is the scale of measurement for the data for both the independent and dependent variables?

Scale of Measurement The four scales of measurement are defined and examples of each are provided in Table 13.1. The scale of measurement is important because it determines which type of statistical procedure is appropriate. As you will see later, this has an influence on deciding between parametric or nonparametric statistics as well as on the appropriate choice of correlation coefficient.

The choice of a statistical procedure is outlined in Table 13.2. Your choice will depend on the following factors:

1. Your research question, which can be descriptive, concerns the extent of relationships between variables, determines significance of group differences, makes predictions of group membership, or examines the structure of variables

2. The type of groups that you have (i.e., independent, dependent, repeated measures, matched groups, randomized blocks, or mixed groups)

3. The number of independent and dependent variables you have

p. 426

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 9/39

4. The scale of measurement 5. Your ability to satisfy the assumptions underlying the use of parametric statistics

Each type of research question leads you to a different statistical choice; thus, this is the most important starting point for your decisions.

Before jumping into complex statistical analysis, it is important to really understand what your data look like. Statisticians recommend that you always graph your data before you start conducting analyses. This will help you in several respects. First, you will be closer to your data and know them better in terms of what they are capable of telling you. Second, they will help you determine if you have met the appropriate assumptions required for different types of analyses. Third, you will be able to see if you have any “outliers”—that is, values for variables that are very different from the general group response on your measure.

Table 13.1 Scales of Measurement

Table 13.2 can be used as a flowchart to think logically through your statistical choices. For example, in the Smalls et al. (2007) study, the researchers first wanted to know to what extent the students in their study experienced racial discrimination. This portion of their study is descriptive; therefore, they could go to the first section in Table 13.2 to identify their statistical options. Their scale of measurement is assumed to be interval, so they determined that the mean was the appropriate measure of central tendency and the standard deviation was useful to describe variability in the data.

Table 13.2 Choice of a Statistical Procedure

p. 427

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 10/39

Table 13.2 (Continued)

a. Ordinal and nominal data can be used in multiple regression equations through a process called “dummying-up” a variable. Refer to one of the statistical texts cited at the beginning of this chapter for more details on this procedure. b. All t tests and variations on ANOVA require that the data satisfy the assumptions for parametric statistical procedures. c. Discriminant functions can be one-way, hierarchical, or factorial, depending on the number of independent and dependent variables.

Smalls et al. (2007) had an additional research question: What was the relationship between students’ experiences of discrimination and their reports of negative behaviors? Therefore, they could go to the second section in Table 13.2, because their research question was one of relationships. They have several blocks of predictor variables (background [e.g., gender], racial identity variables, and racial discrimination), with interval data, so they chose to conduct hierarchical linear regression analysis.

Assumptions for Parametric Statistics As mentioned in Table 13.2, it is important for you to be aware of the assumptions that underlie the use of parametric statistics. These include (a) normal distribution of the characteristic of interest in the population, (b) randomization for sample selection or group assignment (experimental vs. control), and (c) an interval or ratio level of measurement. The assumption that the population is normal rules out outliers in your data, so the presence of outliers shows that this assumption is not valid. Also, if the distribution of the characteristic in the population is skewed (i.e., bunched up at one end of the continuum or the other), the assumption of normality is not met. In the case of skewed distribution, it may be possible to transform the data to approximate a normal distribution using a logarithmic transformation (Gay Mills & Airasian 2009) If the assumptions cannot be met you need to consider

p. 429

p. 428

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 11/39

transformation (Gay, Mills, & Airasian, 2009). If the assumptions cannot be met, you need to consider alternative data analysis strategies. That is where the choice of nonparametric statistics becomes attractive (sometimes called distribution-free inference procedures).3

Interpretation Issues in Quantitative Analysis A number of challenges are presented for quantitative researchers for the interpretation of the results of their data analysis:

1. The influence of (or lack of) randomization on statistical choices and interpretation of results 2. The analytic implications of using intact groups 3. The influence of sample size on achieving statistical significance 4. Statistical versus practical significance 5. Issues related to cultural bias 6. Variables related to generalizability

Following a discussion of these challenges, I present options for responding to some of them, such as reporting effect sizes and amount of variance accounted for, the use of confidence intervals, replication, use of nonparametric statistics, exploration of competing explanations, recognition of a study’s limitations, and principled discovery strategies (Mark, 2009).

Randomization Early statisticians (e.g., R. A. Fisher, 1890–1962, and William Sealy Gosset, 1876–1937, inventor of the statistical test called “Student’s t”) based their work on the assumption that randomization is a necessary condition for the use of typical tests of parametric statistics. Randomness can be achieved by either random sampling or random assignment to conditions. Random sampling has to do with how the participants were chosen from the population (see Chapter 11). Random assignment has to do with how participants were assigned to levels of the independent variable so that variability between groups is statistically evened out (see Chapter 4). Random sampling is a very difficult condition to meet in most educational and psychological research, and random assignment is not always possible.

You might recall the difficult time that G. D. Borman et al. (2007) had recruiting schools to be in their study, thus making it impossible to select schools or students at random. They were able to assign schools that volunteered to either the experimental or control conditions, but they could not assign individual students. A second example, from the Ehri et al. (2007) study, illustrates similar challenges. The researchers started with five schools in a district. They could not randomly select the students, as they wanted to select students based on test scores and to match students based on demographic characteristics. They then wanted to assign matched pairs of students randomly to the experimental and control groups. In four of the five schools, the administrators assigned one group of the intended participants to a third group that received a small-group intervention. The researchers were able to randomly assign the remaining students to one of the three groups. In Elbaum’s (2007) study of test administration procedures, she was not able to randomly select the students from the population, but she was able to randomly assign the treatment conditions by classrooms (not by individual student).

In many situations, it is not possible for ethical or practical reasons to assign people randomly, and it may not be possible to randomly select individuals from a larger population. Much research in education and psychology is done with available populations, and therefore the use of parametric statistics is questionable. If intact classes are used, the class becomes the unit of analysis, thus necessitating either having a large number of classes involved in the research to conduct meaningful statistical analysis or the use of more sophisticated statistical strategies (Schochet, 2008). In studies with intact classes or groups, researchers can choose to use a regression analysis rather than ANOVA. In regression analysis, there is no need to create small groups based on collapsing scores on variables. Thus, this approach can provide a more desirable option because it would not require expanding the sample size.

p. 430

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 12/39

Sample Size Sample size is a basic influence on statistical significance (Hubbard & Lindsay, 2008). Virtually any study can have statistically significant results if a large enough sample size is used. For example, with a standard deviation of 10 and a sample size of 20, a difference of 9.4 between two independent means is necessary for statistical significance at the .05 level in a nondirectional test. However, if the sample size is 100, a difference of only 4.0 is required, and with a sample size of 1,000, a difference of only 1.2 is required (Shaver, 1992). An overly large sample size can result in obtaining statistical significance, even though the results may have little practical significance (see the next paragraph for further elaboration of this idea). When researchers are working with low-incidence populations, as commonly happens in special education research, the small sample size itself might prevent the researcher from obtaining statistical significance. Small sample sizes also have implications for the researcher’s ability to disaggregate results by characteristics, such as gender, race or ethnicity, or type of disability. In such cases, the researcher needs to plan a sample of sufficient size to make the disaggregation meaningful. With power analysis (discussed in Chapter 11), a researcher can determine the size of sample needed in order to obtain a statistically significant result.

Statistical Versus Practical Significance The influence of the size of the sample on the ease or difficulty of finding statistical significance brings up the issue of statistical versus practical significance (B. Thompson, 2002a). Simply because it is easier to obtain statistical significance with larger samples, researchers need to be sensitive to the practical significance of their results. For example, statistical significance may be obtained in a study that compares two drug abuse treatment interventions. In examining the size of the difference, the data may indicate that there are only 2 days longer abstinence for the experimental group. Thus, the researcher needs to be aware of the practical significance of the results, particularly if there are big differences in the costs of the two programs. Is it worth changing to a much more expensive program to keep someone off drugs for 2 additional days?

Ziliak and McCloskey (2007) use a simple comparison to illustrate the difference between statistical and practical significance:

Crossing frantically a busy street to save your child from certain death is a good gamble. Crossing frantically to get another mustard packet for your hot dog is not. The size of the potential loss if you don’t hurry to save your child is larger, most will agree, than the potential loss if you don’t get the mustard. [Researchers] look only for a probability of success in the crossing—the existence of a probability of success better than .99 or .95 or .90, and this within the restricted frame of sampling—ignoring in any spiritual or financial currency the value of the prize and the expected cost of pursuing it. In the life and human sciences a majority of scientists look at the world with what we have dubbed “the sizeless stare of statistical significance.” (p. vii)

Cultural Bias As discussed in Chapter 11 on sampling, use of a label to indicate race when investigating the effects of various social programs can do an injustice in terms of who is included in the study as well as how the results are interpreted and used. Ladson-Billings (2006) and Bledsoe (2008) address the injustices when stereotypic beliefs based on skin color or other phenotypic characteristics serve as a basis for cultural bias in the research process, including the analysis, interpretation, and use of data. Rather than relying on an overly simplistic category such as race for an explanatory variable to categorize people uncritically and assume homogeneity in their conditions, they argue that researchers need to address the complexity of participants’ experiences and social locations. Random selection and assignment cannot make up for cultural bias. Bledsoe (2008) cites an example of this lack of cultural sensitivity in her research on social programs designed to provide prenatal care and prevent and treat obesity in African American communities. She discovered that the program providers made assumptions about the reasons African Americans did not get prenatal services or participate in obesity treatment and prevention programs that were in conflict with the pervasive beliefs of members of the community or that were unrealistic given the conditions in their surrounding environments.

p. 431

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 13/39

However, researchers who critically examine race as a distinct cultural pattern realize that interpretations must be tied to the socioeconomic and environmental context. Thus, policy and program decisions based on this perspective can be more responsive to the social and cultural settings of programs. This brings us full circle from the perspectives discussed in Chapter 1. If the researcher starts from a deficit view of the minority population, the interpretation of the data will focus on the dysfunctions found within that community. If the researcher starts with a transformative perspective, the researcher will attempt to identify the broader cultural context within which the population functions. For example, African Americans are at greater risk than other populations for living in poverty. These circumstances are important because of the relationship between poverty and oppressive social experiences. Rather than focusing on the “deficit” of being raised in a single-parent household, researchers need to understand the notion of extended family and its particular importance for African American children. They also need to understand the contextual information about the experiences and characteristics of racial communities concerning the environments in which children live that make them less likely to be in a position to be in a position of privilege with regard to access to social and educational opportunities.

Generalizability As mentioned in previous chapters, external validity is defined in terms of the generalizability of the results of one particular research study to a broader population. Randomized sampling strategies are supposed to ensure that the results can be generalized back to the population from which the sample was drawn. Randomized sampling is not always possible, and therefore researchers need to be careful in the generalizations they make based on their results. When working with racial and ethnic minority groups or people with disabilities, generalizations about group differences are often put forward without much attention to within-group variation and the influence of particular contexts. Bledsoe (2008) describes this problem within the African American community in that many social prevention programs have been conceived from dominant middle-class perspectives, and many of these programs have been implemented in African American communities.

Although researchers sometimes acknowledge that culturally specific approaches are needed, there have been few serious efforts to design and evaluate programs based on culturally diverse perspectives. A notable exception is the work of Hood, Hopson, and Frierson (2005; Hood, Hopson, Obeidat, & Frierson, in press) in culturally responsive evaluations. When researchers use inappropriate measurement strategies with minority groups, they can erroneously reach conclusions that the programs are or are not effective. The people who are hurt by such inappropriate conclusions are those who have the least power. J. E. Davis (1992) instructs researchers about the damage that can be done by relying completely on the results of data analyses without consideration of the contextual knowledge about the community and the participants who live there. He summarizes the problems with reliance on comparative statistical outcomes without proper sensitivity to contextual variables:

An enormous amount of information about the location and contexts of programs is missing from the discussion of programs’ causal claims. Often, knowledge of a program’s clientele and the program’s appropriateness for its environment is needed to advance thinking about the program’s causal assertions. Unfortunately for African Americans and other U.S. racial minorities, this information is, at best, partially known but discarded or, at worst, not known or even cared about. This is not to say that experimental and quasi-experimental studies are not useful for program evaluation; to the contrary. These methods are very powerful in revealing program effects, but results must be examined more carefully, and with sensitivity to diverse populations. (p. 63)

He warns that making the assumption that all African Americans are homogeneous in comparative studies leads to a lack of understanding of program effects for the diverse members of this community. More information is needed about relevant dimensions of diversity within African American communities. He concludes: “African Americans are the largest racial minority in this country, but much within-group variation and in-depth understanding will be completely lost with traditional race- comparative analysis in program evaluation” (p. 63).

Options for Reporting Statistical Results

p. 433

p. 432

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 14/39

Options for Reporting Statistical Results The APA recommendations concerning reform in the use of tests of statistical significance included reporting effect sizes, confidence intervals, and using graphics (APA, 2009; B. Thompson, 2002b). These were not entirely new recommendations, as researchers have offered a number of options in the past for reporting statistical results of quantitative research that include effect sizes, percentage of variance accounted for, and examining within- and between-group differences.

Effect Size. For studies that have experimental and control groups, Schochet (2008, p. 64) provides the following explanation of effect size: Effect size is calculated as a percentage of the standard deviation of the outcome measures (also known as Cohen’s d), to facilitate the comparison of findings across outcomes that are measured on different scales. An effect size is a way of representing the size of a treatment’s effect in standardized units that then allows for comparisons across studies that might use different outcome measures. An effect size can be calculated to capture the difference between the means for experimental and control groups, calculating the distance between the two group means in terms of their common standard deviation. Thus, an effect size of 0.5 means that the two means are separated by one half of a standard deviation. This is a way of describing how well the average student or client who received the treatment performed relative to the average student or client who did not receive the treatment. For example, if an experimental group of persons with behavioral problems received a drug treatment and the control group received a placebo, an effect size of 0.8 would indicate that the experimental group’s mean was 0.8 standard deviation above the control group.

An increasing number of journals in education and psychology are now requiring that researchers include effect size in their submissions. In relation to effect size reporting, APA (2009) recommends that researchers always include an effect size when reporting statistical significance. In Chapter 3, I described how to use effect sizes for meta-analysis—that is, a statistical synthesis of previously conducted research.

B. Thompson (2002a) warns against a blind interpretation of effect size based on magnitude and suggests that the judgment of significance rests with the researcher’s, user’s, and reviewer’s personal value systems; the research questions posed; societal concerns; and the design of a particular study. For more detailed information on effect size, the reader is referred to Hubbard and Lindsay (2008).

Confidence Intervals. Confidence intervals are used to indicate the degree of confidence that the data reflect the population mean or some other population parameter (B. Thompson, 2002b). Confidence intervals are frequently seen in mainstream media in prediction of election outcomes in the form of a percentage of people who agree or disagree on a candidate, plus or minus a certain percentage to indicate the range of values and the level of confidence that the range includes the population parameter. Because of sampling error, researchers expect the mean to vary somewhat from sample to sample. Most commonly used statistical software packages will compute confidence intervals. APA’s manual states, “Because confidence intervals combine information on location and precision and can often be directly used to infer significance levels, they are, in general, the best reporting strategy. The use of confidence intervals is therefore strongly recommended” (APA, 2009, p. 34).

Correlational Research and Variance Accounted For. Correlational results can be interpreted in different ways. For example, a researcher could say that men and women scored significantly differently (p < .01) on a scale. Or they could say that many variables were tested to determine their relationship with performance on a test, gender being one of those variables. If gender accounts for only 3% of the variance and other predictor variables such as training or experience account for 73% of the variance, then you would interpret the results very differently. The amount of variance accounted for by such background characteristics as race and gender is powerful information; additional manipulable variables (such as training and experience) can provide a clearer picture of what accounts for differential performance on dependent measures.

Replication When the data do not meet the assumptions necessary for reaching conclusions about statistical significance, APA (2009) recommends that researchers replicate the study’s results as the best replacement for information about statistical significance Building replication into research helps

p. 434

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 15/39

replacement for information about statistical significance. Building replication into research helps eliminate chance or sampling error as a threat to the internal validity of the results. This also emphasizes the importance of the literature review, discussed in Chapter 3, as a means of providing support for the generalizability of the results. However, Mark and Gamble (2009) note that replication is not always or even often a viable option:

The false confession problem also would be solved if the experimenter could snoop in the data from one study and then, if an important finding emerges (e.g., a program benefits one subgroup and harms another), see if the pattern replicates in another study. Unfortunately, options for replication are limited in many areas of applied social research. For example, some program evaluations take years and cost millions of dollars; so replication is an infeasible strategy relative to the timeline for decision making. (p. 210)

Use of Nonparametric Statistics Nonparametric statistics provide an alternative for researchers when their data do not meet the basic assumptions of normality and randomization, they have small samples, or they use data of an ordinal or nominal nature.

Competing Explanations No matter what design, paradigm, or type of data collected, researchers always need to consider competing explanations. (In the postpositivist paradigm, these are called threats to internal and external validity.) Such competing explanations become critical when it is time to interpret the results. Recall from Chapter 4 that G. D. Borman et al. (2007) discussed numerous possible competing explanations, such as the overall willingness of a school’s leaders and teachers to accept a systemwide change in their approach to reading instruction.

In Elbaum’s (2007) study of test administration conditions with students with and without learning disabilities, she explored a number of competing explanations for why students without learning disabilities benefited more from the read-aloud condition than did the students with learning disabilities. She hypothesized that the read-aloud strategy did not address a specific disability-related characteristic of students with learning disabilities. (There is no “magic bullet” called read-aloud test administration that will bring about equal performance on math tests for these two groups of students.) She explored the possibility that reading aloud benefits both types of students because of several factors: Students who are poor readers will benefit with this approach even if they are not identified with a learning disability. Many of the students in their nondisabled group were poor readers:

95% of students with LD were failing the high-stakes high school reading assessment, so were two-thirds of the students without an identified disability. These students, although not formally identified as having a reading disability, would clearly be defined as poor or very poor readers by the state’s yardstick. (Elbaum, 2007, p. 225)

She offered another competing explanation for the reason that nondisabled students benefited more from the read-aloud condition: “This finding would lend further support to the idea that providing access to test items by removing reading ability as a barrier will differentially improve the performance of students who have a higher level of skill in the content area being tested” (p. 227). And finally, she hypothesized that the read-aloud condition might make it possible for students to stay on task better than standard administrations do.

Recognizing Limitations As should be clear by now, it is not possible to design and conduct the “perfect” research study in education or psychology. Therefore, it is incumbent on the researcher to recognize and discuss the limitations of a study. For example, Elbaum (2007) recognized the limitations of her study because the read-aloud condition could be confounded by the pacing of the reading or the staying-on-task dimension. Also, she did not have individual reading scores for the students that she could use to test the hypotheses about the effect of reading ability on math performance. An important limitation that h did t di i th t d l d t t d t dd ifi di bilit l t d

p. 435

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 16/39

she did not discuss is that a read-aloud strategy does not address a specific disability-related characteristic of students with learning disabilities. What does the read-aloud strategy address? What strategies of teaching, learning, and assessment would address the specific disability-related characteristics of the students?

Principled Discovery Mark and Gamble (2009) propose that a strategy called principled discovery offers promise for addressing many of the challenges associated with statistical significance testing.

In short, the idea is to complement traditional analyses of experimental and quasi-experimental research with iterative analysis procedures, involving both data and conceptual analysis. The goal is to discover complexities, such as unpredicted differential effects across subgroups, while not being misled by chance. (p. 210)

Remember that statistics are probability-based and that conducting multiple analyses with the same data set increases the chance of finding something significant. “Principled discovery has been offered as a way to try to discover more from the data while not being misled by false confessions” (p. 210). Mark and Gamble suggest that principled discovery begins with testing a prior hypothesis, such as the new treatment will result in improved performance compared to the old treatment.

The principled discovery that follows involves two primary stages (which may further iterate). In the first stage, the researcher carries out exploratory analyses. For example, an experimental

program evaluator might examine whether the program has differential effects by looking for interaction effects using one after another of the variables on which participants have been measured (e.g., gender, race, age, family composition). A wide variety of statistical techniques can be used for the exploratory analyses of this first stage of principled discovery (Mark, 2003; Mark et al., 1998).

If the Stage 1, exploratory analyses result in an interesting (and unpredicted) finding (and if replication in another study is infeasible), then in the second stage of principled discovery the researcher would seek one or another form of independent (or quasi-independent) confirmation of the discovery. In many instances, this will involve other tests that can be carried out within the same data set (although data might be drawn from other data sets, or new data might be collected after Stage 1). For example, if a gender effect is discovered in an educational intervention, this might lead to a more specific prediction that boys and girls will differ more after transition to middle school than before. As this example illustrates, Stage 2 of principled discovery includes conceptual analysis as well as data analysis. That is, the second stage of principled discovery will generally require an interpretation of the finding from the Stage 1 exploration. (Mark & Gamble, 2009, pp. 210–211)

Mark and Gamble (2009) explain that this two-stage process protects against tortured data confessing falsely because if the effect observed in Stage 1 is purely due to chance, then there is no expectation that it will occur again in Stage 2.

For example, if a gender effect from Stage 1 had arisen solely due to chance, it would be unlikely that the size of the gender difference would be affected by the transition to middle school. Principled discovery has considerable potential for enhancing emergent discovery in quasi- experimental (and experimental) research, while reducing the likelihood of being misled by chance findings. Despite its potential benefits, important practical limits will often apply to principled discovery. These include the possibility that in some studies data elements will not be available for the second stage of principled discovery, as well as the likelihood that statistical power may be inadequate for some contrasts (Mark, 2009). Despite such limits, principled discovery appears to be an approach that can help address some of the practical and ethical objections to randomized and quasi-experimental studies of the effects of interventions in a complex world. (Mark & Gamble, 2009, p. 211)

p. 436

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 17/39

EXTENDING YOUR THINKING

Statistical Analysis

1. How can sample size influence statistical significance? Why is this particularly important in special education research?

2. Why is randomization an important consideration in the choice of a statistical test? Why is this particularly important for research that uses small, heterogeneous, or culturally diverse samples?

3. What can a researcher do when the basic assumptions for parametric inferential statistics are not met?

Qualitative Analytic Strategies As mentioned before, but repeated here for emphasis, data analysis in qualitative studies is an ongoing process. It does not occur only at the end of the study as is typical in most quantitative studies. The fact that the topic is explored in depth here is simply an artifact of the way the human brain works. It is not possible to learn about everything all at once. So realize that analysis in qualitative studies designed within the ethnographic or phenomenological traditions is recursive; findings are generated and systematically built as successive pieces of data are gathered (Bogdan & Biklen, 2003).

Qualitative data analysis has sometimes been portrayed as a somewhat mysterious process in which the findings gradually “emerge” from the data through some type of mystical relationship between the researcher and the sources of data. Anyone who has conducted in-depth qualitative analysis will testify that a considerable amount of work occurs during the data collection and analysis phases of a qualitative study.

Several analytic strategies can be used in qualitative data analysis, including grounded theory, narrative, and discourse analysis (Wertz et al., 2011). In addition, Miles, Huberman, and Saldana (2014) prepared a sourcebook that can be used to guide the novice researcher through the process of qualitative data analysis that involves data reduction through the use of various kinds of matrices. Researchers can also choose to analyze data the old-fashioned way (cut and paste pieces of paper) or by using one of the many computer-based analysis programs that have grown in popularity over the years. More details about these options are presented later in this chapter.

Steps in Qualitative Data Analysis Bazeley (2013) provides a step-by-step description of data analysis strategies for qualitative data, despite acknowledgment that undertaking such a task sets up an oxymoronic condition in that qualitative data analysis in essence defies a step-by-step approach. Yet as mere mortals, we are faced with the task of starting somewhere; thus, we begin by taking a look at our data as it is collected.

Step 1: Preparing the Data for Analysis This step assumes that the researcher has been reviewing and reflecting on the data as it is collected. How this is done depends to some degree on the type of data collected and the method of collecting and recording the data. For example, if the researcher used video or audio taping, then questions arise about the transcription of the data: Should all the data be transcribed? If not, how are decisions made about which parts should be transcribed and which should not? How will the researcher handle nonverbal behaviors or elements of the interviews such as laughter or pauses, emotions, gestures? Should the researchers themselves do the transcription? I advise researchers to undertake the process of transcription themselves because this is part of the data analysis process engendered by interacting with the data in an intensive and intimate way. As the questions raised in this paragraph illustrate, transcription is not a transparent process. Researchers bring their own point of view to the process, including noting multiple meanings that lie in what might appear to be simple utterances. Hesse-Biber and Leavy (2006) write

p. 438

p. 437

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 18/39

and Leavy (2006) write,

Transcribing research data is interactive and engages the reader in the process of deep listening, analysis, and interpretation. Transcription is not a passive act, but instead provides the researcher with a valuable opportunity to actively engage with his or her research material right from the beginning of data collection. It also ensures that early on, the researcher is aware of his or her own impact on the data gathering process and he or she has an opportunity to connect with this data in a grounded manner that provides for the possibility of enhancing the trustworthiness and validity of his or her data gathering techniques. (p. 347)

While this higher level of thinking is happening, researchers should also take care of practical considerations, such as being sure their notes and files of data are well labeled and organized to facilitate data analysis processes and accurate reporting of the results.

Steps 2 and 3: Data Exploration Phase and Data Reduction Phase These two phases are synergistic: As you explore your data, you will be thinking of ways to reduce it to a manageable size that can be used for reporting. Exploring means reading and thinking and making notes about your thoughts (called “memoing” by the qualitative research community). Memos can take the form of questions about meanings, graphic depictions of how you think data relate to each other, or important quotes that you want to be sure you don’t lose track of during the analysis process. The data reduction occurs as you select parts of the data for coding—that is, assigning a label to excerpts of data that conceptually “hang together.” Box 13.3 contains an example of codes used in a study, along with an excerpt from the codebook. The first few lines of this example provide the information about the source of the data. The comments on the side are the codes used to identify portions of text that exemplify relevant concepts.

BOX 13.3 Codes and Codebook Excerpts

Field Notes by Donna Mertens, evaluator Project Success Teachers Reflective Seminar May 11, 2007, 9AM–noon

Facilitator (F) introduced the evaluation team and their purpose.

F: Divide into pairs and discuss your life experiences. Then we’ll come back together and each of you will describe one “WOW” experience and one challenging experience.

Observation: the 2 current students did not interact; F talked with one teacher. Four African American teachers; 6 white female teachers; 1 white female grad student.

WOW and challenges: T5: My students are under 5 years old and they come with zero language and their behavior is awful. They can’t sit for even a minute. Kids come with temper tantrums and running out of the school building. I have to teach these kids language; I see them start to learn to behave and interact with others. My biggest challenge is seeing three kids run out of school at the same time. Which one do I run after? One kid got into the storm drain. I’m only one teacher and I have an assistant, but that means there is still one kid we can’t chase after at the same time as the other two.

T7: I had my kids write a letter to their teachers to express their appreciation. That was a WOW experience. Challenge—I don’t like to be negative. My real challenge is that I feel kind of isolated because of being in a separate building—other teachers don’t even know who I am.

T6: I had a student from a Spanish speaking country. He struggled to pick up ASL. Once he starting picking it up, he was very quick. I noticed this student is now asking for more challenge; he participated in his IEP meetings. Now he is going to general education classes; I can’t remember any kids from special ed going to general ed before

p. 439

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 19/39

general ed before.

Challenge: I feel teachers in the mainstream resist our students, especially students with multiple disabilities.

T8: I teach students who are mainstreamed for 2 or 3 classes; the school sees my students as being very low, but my students want to know things.

T3: My WOW and challenge are really the same thing. When I graduated, I thought I was ready to teach. Then the principal gave me my list of students and my classroom and just washed his hands of me. You’re on your own. The principal did not require me to submit weekly plans like other teachers because he thought I’d only be teaching sign language. But I told him, I’m here to really teach. We (my students and I) were not invited to field day or assemblies. That first year really hit me—what a challenge and a WOW at the same time. So I changed schools and this one is definitely better. Now I’m in a school where people believe that deaf students can learn.

T4: I have 6 students in my classroom. There are no other teachers there to help me. My class has kids who use 3 different methods of communication: sign, oral, and cued speech. I tried to explain in sign, but the other kids don’t understand. I was always saying: what should I do? I have a co-teacher in the afternoon, but she doesn’t really support me. So I told her I needed her to really help me. So she works with one kid who got a cochlear implant. He can say his name now, but nothing else.

Codebook Excerpts:

ASL—American Sign Language ESL—English as a Second Language ChgScSys—Changed school systems DivBhvr—Diverse Behavior—Teachers’ experience challenges because of diverse student behavior. Includes behavioral issues. DivComm—Diverse communication modes—combination of communication modes in classroom—sign, oral, CI [cochlear implant], cued speech. Iso—Teacher feels isolated. Low—Low expectations for students. NoSup—No support system. WOW—Something wonderful that happened to the teachers.

SOURCE: Field Notes prepared by Mertens for the Project SUCCESS evaluation (Mertens, Holmes, Harris, & Brandt, 2007).

Charmaz (2006) provides a detailed description of coding strategies that she developed within the grounded theory method of data analysis that involves two phases: initial coding and focused coding. (Grounded theory is not the only analytic strategy for qualitative data; however, it does reflect many of the characteristics of qualitative data analysis that are common across other approaches.) Corbin and Strauss (2008) use the terms open coding and axial coding instead of Charmaz’s initial and focused coding. (See Box 13.4 for brief descriptions of two other data analysis approaches: narrative and discourse analysis.) In the initial coding phase, the researcher codes individual words, lines, segments, and incidents. The focused coding phase involves testing the initial codes against the more extensive body of data to determine how resilient the codes are in the bigger picture that emerges from the analysis. The development of codes can be used to form the analytic framework needed for theory construction.

BOX 13.4 Narrative and Discourse Analysis Approaches

Narrative Analysis

p. 440

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 20/39

y Narrative analysis focuses on the stories, whether they are told verbally or in text or performance formats (Bingley, Thomas, Brown, Reeve, & Payne, 2008; Leavy, 2009). The researcher tries to identify the content, structure, and form of life stories based on the available data. This might take the form of a biography or autobiography, a timeline of important life events, or an exploration of the meaning of life events within a broader sociocultural context. Narratives can be analyzed as separate voices to reveal diversity of perspectives of an issue. As a researcher approaches narrative analysis, it is important to consider whose story will be told and how the person’s story will be represented. Arts-based researchers (Leavy, 2009) have contributed to the analytic processes in narrative analysis and extended this to include analysis of such genres as poetry, music, dance, and the visual arts.

Gubrium and Holstein (2009) note that a distinct characteristic of narrative analysis is that it draws on literary devices such as “topics, plots, themes, beginnings, middles, ends, and other border features that are assumed to be the defining characteristics of stories” (p. 226). The stories are of importance; the way the stories are told is of importance, as are the wider societal context and past and future events. They write, “To capture the richness of narrative reality, analysis needs to focus on both the circumstantial variety and the agentic flexibility of stories, not just on the structure of the accounts themselves” (p. 228).

Discourse Analysis Discourse analysis focuses on understanding the meaning of participants’ language. Huygens (2002) offers such questions as the following to guide discourse analysis: “What are the participants actually saying?” and “Why did they choose to say it this way?” The researcher is trying to read between the lines to determine deeper meanings to such qualities of language as colloquialisms, images, rules of turn taking. The researcher is interested in “how the story is told, what identities, activities, relationships, and shared meanings are created through language” (Starks & Trinidad, 2007, p. 1373). The basic analytic process involves examining three dimensions (Fairclough, 2003):

• analysis of the text, which involves the study of language structures such as use of verbs; use of statement, questions, or declarations; and the thematic structure,

• the analysis of discursive practice, which involves how people produce, interpret, disseminate, and consume text, and

• the analysis of sociocultural practice, which involves issues of power in the discourse context and its implications in wider society.

Rogers, Malancharuvil-Berkes, Mosley, Hui, and Joseph (2005) provide a review of critical discourse analysis in educational research based on the major databases and journals in the field. They provide numerous examples of the application of critical discourse analysis in research in education and psychology.

Charmaz (2006) gives this advice for the initial coding phase:

• Remain open • Stay close to the data • Keep your codes simple and precise • Construct short codes • Preserve actions • Compare data with data • Move quickly through the data (p. 49)

From my experience, I would add, involve team and community members in the coding when appropriate; discuss differences in interpretation and use of the codes (Mertens, 2009). Allow the codes to emerge and be revised as necessary, especially in the early stages of coding. Explore differences in interpretations; this can be an opportunity for surprising discoveries. Make a codebook that includes brief descriptions of each code. Having such a codebook facilitates the employment of a constant comparative method of analysis. Corbin and Strauss (2008) provide this definition of constant

p. 441

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 21/39

comparative analysis:

Comparing incident against incident for similarities and differences. Incidents that are found to be conceptually similar to previously coded incidents are given the same conceptual label and put under the same code. Each new incident that is coded under a code adds to the general properties and dimensions of that code, elaborating it and bringing in variation. (p. 195)

This is the bridge to the focused phase of coding that Charmaz (2006) describes: “Focused coding means using the most significant and/or frequent earlier codes to sift through large amounts of data. Focused coding requires decisions about which initial codes make the most analytic sense to categorize your data incisively and completely” (p. 57). If a researcher is using a grounded theory approach, the focused coding will lead to identifying relations among the coding categories and organizing them into a theoretical framework. You validate the hypothesized relationships with the data available to you and fill in categories that need further refinement and development. This step is integrative and relational; however, Corbin and Strauss (2008) note that the analysis that occurs at this stage of the study is done at a higher, more abstract level. During this phase of analysis, the researcher identifies the core category or story line and then relates the subsidiary categories to the core through a model. (Corbin and Strauss use the term paradigm; however, I use the term model, because paradigm has a different meaning in this book.) The model includes an explication of the conditions, context, strategies, and consequences identified in the coding phase. You then validate your theory by grounding it in the data; if necessary, you seek additional data to test the theory.

The coding example shown previously in this chapter of the study of teachers who work with students who are deaf and have an additional disability illustrates how the transformative lens was brought into the analysis to support the focused coding phase (Mertens, Holmes, Harris, & Brandt, 2007). We were able to see connections between social justice issues in teacher preparation and support in early years of teaching based on the challenges that new teachers faced in terms of being marginalized at their schools (manifest by low expectations from administrators and exclusion from mainstream school activities) and a need to fully address language diversity (e.g., home languages other than English; use of a variety of languages and communication modes such as American Sign Language, speaking English while using ASL signs, and cued speech; use of a variety of assistive listening technologies such as cochlear implants and hearing aids). If all students have a right to an education, how can colleges of education prepare teachers who can respond appropriately to these educational challenges?

Theoretical Lenses and Qualitative Data Analysis In Chapter 8, you read about various theoretical lenses that are associated with the transformative paradigm and that have been applied in qualitative research such as feminist, critical race, LatCrit, disability rights, deafness rights, and critical theory. You will recall that use of such a theoretical lens influences the kind of research questions and data that you collect. So it is logical to expect that analytic strategies are also influenced by the use of such theoretical lenses. Milner (2012) describes the use of narrative and counter-narrative analytic tools in order to challenge oppressive belief systems that represent dominant views. In Milner’s study, he was challenging negative stereotypes of Black teachers by analyzing the counter-narratives of the Black teacher in his study. Using critical race theory (CRT), he used counter-narratives

to share a teacher’s experiences in ways that have not necessarily been told because it provides a different picture into the complexities of teaching and learning. . . . Race and racism are placed at the center through the narrative and counter-narrative through a critical race theory framework of analysis. (p. 28)

Using counter-narrative analytic strategies, Milner was able to contrast stereotypes such as Black teachers are too harsh and authoritarian with the teacher’s explanation that she accepted her responsibility for the development of her students, not just academically, but holistically. She kept students engaged throughout class time and accepted no nonsense; she prioritized students’ learning

p. 443

p. 442

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 22/39

that they are responsible members of their community with an obligation to serve and improve their community.

Additional examples of counter-narrative analysis are available in Caton’s (2012) study of Black males’ experiences with zero-tolerance policies that led to their dropping out of high school. Zero- tolerance policies were instituted to ensure a safe environment in schools. However, the policies disproportionately result in Black males being expelled from or dropping out of high school. The use of counter-narratives allowed Caton to “challenge the contemporary ideology of color blindness and the notion of a school-to-prison pipeline. . . . This framework could illuminate what it means to be Black and male in an urban school, pursuing an education under a burden of suspicion” (p. 1063). Rolon- Dow (2005) also provides an example of counter-narrative analysis in her study of the school experiences of Puerto Rican girls in which she used LatCrit theory as a framework.

The use of a feminist theoretical lens provides another example of how theory influences analysis. Sosulski, Buchanan, and Donnell (2010) used feminist theory in a life history narrative study to analyze the experience of mental illness in a Black woman. The feminist lens meant that the participant made decisions about what to reveal about herself and how she interpreted what she said. She talked about her hallucinations in two ways: One type was painful to her and she sought relief from those; another type she considered to be a gift from God that allowed her to predict future events. She wanted to keep the latter ability. Given the multiple demands on this woman, the researchers recommended diversifying treatment and engaging community and family members in supporting her.

Using Computers in Qualitative Analysis Because qualitative studies tend to result in mountains of data (literally), many researchers have turned to computerized systems for storing, analyzing, and retrieving information. Presently, a large number of computer programs are available (e.g., ATLAS/ti, The Ethnograph, HyperRESEARCH, and NVivo), and this is an area in which rapid changes are occurring. Hence, I will not recommend any particular software but instead refer you to a website that I have found to be useful called Computer Assisted Qualitative Data Analysis where you will find analyses of various programs, as well as breaking news in this area (http://caqdas.soc.surrey.ac.uk/). One of the features of computer-based coding is that codes can be generated “in vivo”—that is, codes can be generated by highlighting specific text in the file.

Before making a decision about which software program to use, you should review the resources cited in this chapter (as well as any more recent developments in this rapidly changing field). You need to pick a system that is compatible with your hardware as well as with your research purposes. One caution: No matter how attractive the software, nothing should separate you from active involvement with your data. Qualitative data analysis is really about you thinking about your data and hypothesizing possible relationships and meanings. A computer can be an important aid in this process, but you should not let it become a means of separating you from the process of knowing what your data have to say.

Interpretation Issues in Qualitative Data Analysis

Triangulating Data Triangulation, as it was discussed in Chapter 8, involves the use of multiple methods and multiple data sources to support the strength of interpretations and conclusions in qualitative research. As Guba and Lincoln (1989) note, triangulation should not be used to gloss over legitimate differences in interpretations of data; this is an inaccurate interpretation of the meaning of triangulation. Such diversity should be preserved in the report so that the “voices” of the least empowered are not lost. Richardson and St. Pierre (2005) suggest that a better metaphor for this concept is crystallization; Mertens (2009) suggested the metaphor of a prism. The crystal and the prism metaphors suggest multifaceted sources of data that are brought to bear on the interpretation of findings.

Audits Two types of audits were described in Chapter 8: the dependability audit and the confirmability audit. Through the use of these two strategies, the researcher can document the changes that occurred during

p. 444

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 23/39

oug t e use o t ese two st ateg es, t e esea c e ca docu e t t e c a ges t at occu ed du g the research and the supporting data for interpretations and conclusions. The process of memoing discussed previously in the coding section has been noted to contribute to a researcher’s ability to make visible the decision-making trail that occurred during the course of the study (Birks, Chapman, & Francis, 2008). Memos can serve as an audit trail to document the progression of the study as well as changes that occurred and the context for those changes.

Bazeley (2013) emphasizes the importance of keeping an audit trail that documents your thinking and feelings as you proceed through the data analysis. You can consider such questions as the following when constructing your audit trail:

1. Are findings grounded in the data? (How does the sampling affect interpretation? Is any piece of data given excessive weight compared to others?)

2. Are inferences logical? (Are analytic strategies applied correctly? Are alternative explanations accounted for?)

3. Is the coding structure appropriate? 4. What are the justifications of inquiry decisions and methodological shifts? How did hypotheses

change over the course of the analysis? 5. What is the degree of researcher bias (premature closure, unexplored data in field notes, lack of

search for negative cases, feelings of empathy)? 6. What strategies were used for increasing credibility (community involvement, member checks,

feedback to informants, peer review, adequate time in the field)?

Cultural Bias The comments included in the section on cultural bias for quantitative research are equally appropriate when analyzing and interpreting qualitative research. The opportunity to see things from your own cultural bias is recognized as a potential problem in qualitative research. Many of the safeguards discussed in Chapter 8 are useful for minimizing this source of bias or for recognizing the influence of the researcher’s own framework. You should begin by describing your own values and cultural framework for the reader. Then you should keep a journal or log of how your perspectives change through the study. Discussing your progress with a peer debriefer can enhance your ability to detect when your cultural lens is becoming problematic. Conducting member checks with participants who are members of the culture under study can help you see where divergence in viewpoints may be based on culturally different interpretations.

Generalization/Transferability Differences of opinion exist in the qualitative research community with regard to claims that can be made about the generalizability of the findings. Recall that generalizability is a concept that is rooted in the postpositivist paradigm and technically refers to the ability to generalize results of research conducted with a sample to a population that the sample represents. In qualitative research, Guba and Lincoln (1989) proposed that the concept of transferability would be more appropriate. With this approach, the burden of proof for generalizability lies with the reader, and the researcher is responsible for providing the thick description that allows the reader to make a judgment about the applicability of the research to another setting. Stake (2006) offers the opinion that case studies can be conducted with no intention to generalize to other situations; these cases are studied because of intrinsic interest in that case. He also recognizes that case studies are sometimes undertaken to be able to describe a typical situation, so the case provides an opportunity to learn that may provide insight into other similar situations, or multiple case studies might be undertaken to demonstrate the range of commonality and diversity of a phenomenon. Aggregating across cases must be done cautiously and without loss of the uniqueness in the context of each case.

Ruddin (2006) extends this argument in asserting that it is false to think that a case study cannot grant useful information about the broader class. Rather, the strength of a case study is that it captures “reality” in greater detail and thus allows for both the analysis of a greater number of variables and for generalization from the concrete practical and context dependent knowledge created in the

p. 445

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 24/39

generalization from the concrete, practical, and context-dependent knowledge created in the investigation. He further explains,

We avoid the problem of trying to generalize inductively from single cases by not confusing case inference with statistical inference. Case study reasoning would be seen as a strong form of hypothetico-deductive theorizing, not as a weak form of statistical inference. (p. 800)

Member Checks As mentioned in Chapter 8, member checks can be used during the process of data collection. They have a place at the stages of data analysis and writing as well. Darbyshire, MacDougall, and Schiller (2005) note that repetitive use of member checks at different phases of the study provides a method of increasing validity. They describe a study in which member checking was used in a photo essay project. The children in the study were asked to discuss written and photographic portrayals of the findings. They were able to suggest additions and deletions in order to tell the story the way they thought it should be told.

Analytic and Interpretive Issues in Mixed Methods Analytic and interpretive issues in mixed methods research are influenced by the researcher’s paradigm and the design of the study. If a sequential design is used, it is more likely that the data analysis of one type of data will precede that of the other type, with little consequence for integrating the two types of data. However, it is possible that researchers might want to use the data from both types of data collection to inform their conclusions, and this would be especially true in concurrent mixed methods designs. Hence, discussion of analytic and interpretive issues here focuses on the nexus at which the two types of data actually meet and are intended to have an interactive relationship with the intention of seeing how they might inform each other.4 This is the strategy used in the study of parents and their deaf children (see Chapter 10, Meadow-Orlans, Mertens, & Sass-Lehrer, 2003). A national survey resulted in quantitative data that were disaggregated to determine characteristics of parents associated with different levels of satisfaction with early intervention services. These data were used to identify subgroups of parents from whom qualitative data would be collected by means of in-depth interviews or focus groups. In the final analysis and interpretation phase of the study, the quantitative data were used to provide a picture of each subgroup of parents, while the qualitative data provided a richer explanation of their experiences with their young deaf or hard-of-hearing child.

Jang, McDougall, Pollon, Herbert, and Russell (2008) provide reflections on mixed methods data analysis in their study of successful schools that had high percentages of immigrant students and were in economically distressed areas. They used a concurrent mixed methods design,

composed of qualitative approaches using interviews with teachers and principals and focus groups with students and parents, and a quantitative survey of principals and teachers. This concurrent mixed methods design was to serve the complementarity function in that the general description of school improvement from the survey was enriched, elaborated, and clarified with contextually specific accounts of school success from interviews involving multiple perspectives. (p. 226)

Jang et al. (2008) used traditional quantitative statistical analysis techniques to perform a factor analysis on the quantitative survey data. They also engaged in traditional qualitative analysis strategies of developing codes and checking the coding process with a team of researchers. However, they went much farther with their data analysis to integrate the quantitative and qualitative data. They “qualitized” the factors that were identified by the factor analysis and compared the associated concepts with the themes that had emerged from the qualitative data. They then merged the themes from the two types of data and used the new set of themes as an organizing framework to categorize the items from the survey according to which theme they represented. They then conducted quantitative analyses of the subgroups of survey items to see how the participants responded in this thematic context. They also conducted these analyses for each school to see what differentiated one school from another They were able to see the commonality of themes that emerged across the various schools thatp. 447

p. 446

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 25/39

another. They were able to see the commonality of themes that emerged across the various schools that provide an explanatory basis for the school’s success. They also identified a factor that was different among schools that exhibited higher or lower levels of success—parental involvement. Using the data from the re-analyzed survey data and the comments made by participants about parental involvement, the researchers were able to provide specific recommendations to improve parental involvement in the less-successful schools.

Remember Berliner, Barrat, Fong, and Shirk’s (2008) mixed methods study of policies and practices to improve high school graduation for dropouts? They investigated the quantitative patterns in reenrollment and subsequent graduation from high school. In addition, their qualitative data analysis provided them with insights as to why students reenroll (e.g., it is hard for a high school dropout to get a job). They also identified factors that challenge school districts when trying to reenroll dropouts. Demand at alternative high schools exceeds capacity, yet traditional high schools do not offer the interventions needed to support reenrollees. State funds were tied to enrollment and attendance rates; dropouts even with reenrollment result in fewer state dollars. This is complicated by the requirements for specific tests and other curriculum requirements needed for graduation. The combination of the quantitative results that support the effects of reenrollment in the long term with the identification of challenges from the policies in place at the state and district levels provides a basis for recommendations for improving graduation rates for this population of students.

A Research Plan: The Management Plan and Budget A research plan is needed to outline the steps that must be taken to complete the research study within a specific time frame and to identify the resources needed to accomplish this complex task. The research plan consists of two parts: the management plan and the budget. Together, these can be used to guide you as you conduct your research and to monitor your progress.

The Management Plan To develop a management plan, you need to analyze the major steps in the research process—for example, the literature review, design of the study, implementation of the study, and analysis and report of the study. Then, for each major step, you should list the substeps that you need to accomplish to conduct your research. A sample management plan is presented in Table 13.3. You can use the management plan to monitor your progress and to make any midcourse corrections that might be necessary. In addition, it can serve as a trail that documents your actions, which would allow researchers to audit your work, including any divergences from the original plan and the reasons for those changes. Particularly in qualitative research, the divergences are important because they are expected to occur. Thus, you are able to establish a trail of evidence for decisions that influenced the direction of the project and to support any conclusions you reached based on the research outcomes.

If you are involved in a large-scale research project, it might be helpful to divide the work that will be done by people (defined in terms of positions—e.g., project director, research assistant) and identify the amount of time that each person will devote to the various activities. Such a person-loading chart can help with budgeting and justification for funding requests.

Table 13.3 Sample Management Plan for Research Proposal

p. 448

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 26/39

a. You will be clearer about the other subtasks needed after you complete your literature review.

The Research Budget Many dissertations and theses are written with very little or even no external funding. However, it is common for institutions to offer small grants to support such research or even for advanced students to seek outside funding. A sample budget sheet is presented in Table 13.4. General budget categories include personnel and nonpersonnel items. Under personnel, you should list all the people who will be involved in the research process, the amount of time each will be expected to work, and the rate at which each will be paid. Typical nonpersonnel items include travel, materials, supplies, equipment, communication expenses (telephone, postage), and copying. If you plan to pay people who participate in your research, that should be included in your budget as well. If you are seeking funds from an external source, you will need to include indirect costs. This is typically a percentage of the overall amount requested that your institution will expect to receive to cover such costs as maintaining the building in which you have an office, the heat, the water, and other indirect costs of supporting you as you conduct your research.

The budget can be arranged by research activities, to indicate which categories or activities require varying proportions of the funds. Arranging the budget this way makes it possible to show the impact of budget cuts on the research activities themselves. For example, if funds are budgeted for a follow-up of nonrespondents and this amount is cut from the budget, it would have deleterious effects on the quality of the research.

Writing Research Reports Typically, researchers think of writing up their research in a fairly standard format for reporting results and providing a documentation of their methods and interpretations. This section provides guidance on such writing. Alternatives to presentation of results in a written report are also presented in terms of performance as a means of reporting research results.

Table 13.4 Sample Budget

a. Most institutional small grant programs disallow you paying yourself to complete your dissertation or thesis research. However, you might be able to obtain funds to support a research assistant to help with the copying, mailing, coding, or entry of data. b. Typically, a budget would require specific justification for travel expenditure in terms of the cost of an airline ticket, hotel room, meals (sometimes calculated as a per diem), and so on.

Writing Reports

p. 449

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 27/39

The benchmark style manual for writing educational and psychological research reports is the Publication Manual of the American Psychological Association (APA, 2009). Guidance in writing is available at APA’s website (www.apastyle.org/). APA’s publication manual contains information on citing references and also suggests formats for writing different types of papers (e.g., literature review, empirical study, or meta-analysis). In some disciplines (e.g., history), other style manuals may be recommended, such as The Chicago Manual of Style (2010) or Turabian’s (2007) A Manual for Writers of Term Papers, Theses, and Dissertations. The APA also publishes a number of references on nondiscriminatory use of language, available from their publications office located in Washington, D.C., or online at www.apa.org/ (search: avoiding bias in language).5 They have style tips for removing bias in the use of language for the following topics: people with disabilities, racial or ethnic minority groups, and sexuality. Other professional organizations publish similar sets of guidelines, such as the American Educational Research Association (AERA; www.aera.net), also located in Washington, D.C.

Students should check with their university regarding local style requirements. Individuals who wish to publish their research should check with prospective publishers for their requirements. Journals generally tend to include publication requirements in the front or back of each issue.

Dissertations and Theses Most dissertations and theses consist of the following first three chapters:

Chapter 1: Introduction Chapter 2: Literature Review Chapter 3: Methodology

Quantitative dissertations and theses typically have two additional chapters:

Chapter 4: Results Chapter 5: Discussion and Conclusions

Meloy (2001) found that qualitative dissertations tended to have different formats for the final chapters, such as chapters on emergent analyses, individual case study reports, and conclusions and implications. An outline for writing dissertation and thesis proposals for different paradigms is provided in the appendix.

Before you start writing, you should determine if your institution has a formal style manual. In addition, ask questions to determine the general practices at your school:

1. What is the acceptable number of chapters in a dissertation or thesis? What flexibility exists for alternative formats?

2. How much control do students, faculty members, and the graduate school have in determining the appearance and style of the dissertation or thesis? Which decisions, if any, are negotiable?

3. Do you write in the first-person or third-person voice? Do you use past, present, or future tense? 4. What are the local guidelines and requirements for selecting a committee? How many committee

members must you have? What is the recommended or acceptable role for the committee members?

5. How supportive is the environment at your university or college for postpositivist, constructivist, or transformative research?

Reporting Format Options Although you may have been working in the mind-set of producing a dissertation or thesis, you will want to be cognizant of alternative reporting formats for your academic research, as well as for your future research studies. Researchers have a wide range of options available to them for reporting their

p. 450

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 28/39

g p p g research: for example, presentations at professional meetings, journal articles, and technical reports. Typically, a research report includes an introduction (with a literature review) and sections for method, results, and discussion. The exact structure differs depending on the type of research (quantitative or qualitative), the audience, and purpose of the report. Research reports can have multiple uses for different audiences; thus, alternative reporting formats should be considered for dissemination.

In both quantitative and qualitative studies, the researcher should tie the results back to the purpose for the study and to the literature in the discussion section of the report. Furthermore, findings should be based on data, and caution should be exercised in recommendations for practice. Limitations of the work should be recognized.

Quantitative Reports In the quantitative report format, results are typically reported by the use of tables and graphs. Researchers also tend to write in a detached style, avoiding the use of the first-person pronoun and employing the passive voice. Although qualitative reports can use tables and graphs, they typically present results in a more narrative style and include more personal revelations about the author (Richardson & St. Pierre, 2005).

Some common problems in writing the results of quantitative data analyses include how to use the word significant appropriately. Writers frequently use the term significant without the appropriate accompanying modifier term—for example, statistically—in the description of the results. The reader cannot infer that the term significant implies statistical significance. However, without that modifying term, the intended meaning of significant is not clear in this context. Second, writers sometimes assume that the use of the term statistically significant means that the results are important or meaningful. As mentioned in the section on the difference between statistical and practical significance, you know that having statistical significance does not necessarily equate with being important. Third, writers sometimes talk about approaching significance. This is not appropriate language in a research report. Results are either statistically significant or not; they do not approach (or avoid) significance.

In B. Thompson’s (2007) analysis of common errors found in quantitative dissertations, he also notes this problem with language that describes statistical significance: Writers will sometimes interpret significance tests as if they were effect sizes (e.g., results were highly significant or results approached statistical significance). In addition, he notes the following common errors:

1. When the sample size is small and the effect size is large, the results are often underinterpreted. (As you will recall, it is harder to get statistical significance with a small sample, so if a large effect size is found under these conditions, it should be noted.) If the sample size is large and effect sizes are modest, the results can be overinterpreted.

2. Researchers sometimes use many univariate tests of statistical significance when a multivariate test would be more appropriate. The probability of making a Type I error is increased when multiple univariate tests are applied to one data set.

3. Researchers sometimes convert continuous (ratio or interval) data to categorical (nominal) data to conduct a chi-square or some type of ANOVA. Thompson points out that much variance is lost by this conversion and suggests that researchers ask themselves whether a regression analysis might be more appropriate as a way of preserving the variance in the data.

4. ANCOVA is sometimes employed to provide statistical control when random assignment was not performed with the expectation that the statistical adjustments will magically make groups equivalent. However, when the data cannot satisfy the underlying assumption of homogeneity of regression, it is not appropriate to use ANCOVA. The assumption states that the relationship between the covariate and the dependent variable is equivalent in all experimental groups; for example, children in a treatment group learn at the same rate as children in the control group. If the relationship between the dependent variable and the covariate is different under different treatments, the ANCOVA should not be used. The effect of not satisfying the homogeneity of regression assumption can be to suppress evidence of program effects.

5. Researchers who use regression analysis should apply the regression equation to a fresh sample f d t t h ll it d di t l th it i i bl ( li ti d lid ti )

p. 452

p. 451

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 29/39

of data to see how well it does predict values on the criterion variable (replication and validation). 6. Researchers should always report the psychometric integrity of the measures they use.

Qualitative Reports For qualitative reports, the reader looks for deep and valid descriptions and for well-grounded hypotheses and theories that emerge from a wide variety of data gathered over an extended period of time. The researcher should also seek contextual meaning; that is, he or she should attempt to understand the social and cultural context of the situation in which the statements are made and the behaviors exhibited. This includes a description of relevant contextual variables, such as home, community, history, educational background, physical arrangements, emotional climate, and rules. It should also include a statement that makes clear the positionality of the researcher as the instrument. Borum (2006) positions herself as an African American Womanist and offers this self-description in her writing:

I am an “inside” member of the African American community and have lived in African American communities most of my life. I not only am intellectually aware of cultural nuances but also live the cultural nuances found in African American culture and experiences. Therefore, I will explicate the following: I am single (never married) and have never given birth, thus, I have no biological children—both decisions are by choice. However, I have worked with families and their deaf children in school and home settings for more than 10 years. I received my BA in psychology with minors in biology and philosophy from Mundelein Women’s College in Chicago. I received my MSW with a concentration in deafness from Gallaudet University, and I am fluent in American Sign Language. I received my PhD from Howard University, a Research I Historically Black University where my dissertation topic was African American families with deaf children. (p. 343)

Writing up qualitative research has emerged as an area of controversy partially reflected in the struggle of how to present qualitative data based on whose voice is represented in the written report. Qualitative researchers typically struggle with the implications for representing different voices in research in a way that would be meaningful for those who would be affected by the results. Participants’ constructions of meaning in social context often vary radically from the researcher’s constructions because of different histories or experiences. Such variations should not discourage you from conducting and reporting research of a qualitative nature. Indeed, the variations can be seen as an impetus to seek the most accurate understanding possible of what is happening in contexts from as many perspectives as possible. You need to give thought to which formats for data dissemination can facilitate inclusion of all voices meaningfully in the discourse. Researchers need to establish mechanisms for speaking with the people for whom the research is conducted. For example, in education, researchers need to communicate about the research with students, teachers, and administrators. Currently, there is little value attached to such discourse, because a researcher is rewarded for publication in scholarly journals and presentations at professional meetings that the students, teachers, and administrators typically do not attend.

M. Fine, Weis, Weseen, and Wong (2000) offer the following questions for researchers to ask themselves as a way of gauging their responsiveness to the voices of the participants:

• Have you worked to understand your contribution to the materials/narrations provided and those silenced?

• Have you worked to explain to readers the position from which informants speak? • Have you worked to recast the person(s) whom the informant chooses to “blame” or credit for

social justice or injustice (be it a social worker, the informant’s mother, black men)? (p. 127)

Freeman, deMarrais, Preissle, Roulston, and St. Pierre (2007) also raise questions for qualitative research writers because of the close connection between the researchers and participants, as well as because of concerns about representation. They offer the following guiding questions:

p. 453

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 30/39

• How can we best listen to, work with, and represent the people our work is intended to serve? • Do authors of research reports reveal themselves as their own best critics? • Do they discuss the limits and uncertainties of their work? • How forthright are they about competing interpretations and explanations for the patterns they

claim? (p. 30)

These researchers offer advice on how to answer these questions, such as demonstrating the relationship between the data and claims by giving sufficient data to support each claim. This can be accomplished by inclusion of extensive quotations or by making data available to others. The extent to which a researcher chooses to allow others to have access to the data is based on a complex set of considerations. For example, should a researcher make video data or interview and field notes available on the Web for others to see? Oral historians have tended to make their data available in this way. However, other approaches to research offer confidentiality to their participants, hence suggesting that such a strategy would violate ethical principles. Members of Indigenous groups have expressed concerns about having data about themselves displayed in public forums because of a legacy of distrust with dominant cultures.

Organizing Qualitative Writing. Qualitative researchers often end up with a mountain of data that needs to be reduced to some manageable form so that others will be able and willing to read it. McDuffie and Scruggs (2008) suggest that, at a minimum, the writer include a description of research methods, including how the data were coded, along with a rationale for what was and was not included in the research. Researchers should also include reflections on possible researcher bias and provide references to related research.

Here are some possibilities for organizing the presentation of the results:

• Events can be presented in chronological order, as they occurred in the data collection setting. • Events can be presented in the order in which the narrators revealed them to the researcher. • The researcher can write using progressive focusing; that is, the researcher can describe a broad

context and then progressively focus in on the details of the particular case. • You can report events as a day-in-the-life description of what life is like for people in that

setting. • You can focus on one or two critical or key events. • You can introduce the main characters and then tell the story, revealing the plot as in a stage

play. • The main groups in the setting can be described along with the way they interact with each

other. • You can use an analytic framework (described earlier in the chapter in the section on grounded

theory analysis strategies) to organize the writing. • You can tell the story several different ways, from the viewpoint of different actors in the

setting. Wolf (1992) used such a strategy in her book A Thrice Told Tale, in which she reports the same field study as a work of fiction, in the form of field notes, and as a self-reflexive account.

• You can present the research problem as a mystery to be solved and then bring the pieces of data into the story as a way of solving the mystery.

Performance as Research Reporting. Researchers are increasingly using theater as a means to translate their data into performances with the goal of presenting their results in a more engaging manner and thus encouraging social action to follow the presentation (Beck, Belliveau, Lea, & Wager, 2011). Such an approach means that the researcher needs to write a script and then cast and/or perform the scripted

i l Ch d T (2006) d ib h b fi f h hi f hi

p. 454

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 31/39

materials. Cho and Trent (2006) describe the benefits of ethnographic performance this way: “‘Readers . . . move through the re-created experience with the performer’ (Denzin, 2000, p. 950) to be able to differently perceive the world in which we live and to actively engage themselves in this world” (Cho & Trent, 2006, p. 332). The goal is to bring about transformation by looking at the “taken for granted” in new and unexpected ways.

Mienczakowski (2000) provides an example of the use of performance to report the results of an ethnographic study in the form of theater with transformative intentions. He used the experiences of the health consumer community representing schizophrenic psychosis and institutionalized detoxification processes as a way of communicating to groups of health and student communities. He terms the medium ethnographically derived theater, because the meanings and explanations of the performances are negotiated with audiences in forum discussion at the close of each performance. Thus, the potential is created to share insights and negotiate explanations with an eye to provoking change among those who play an active part in the construction of health services.

Multiple forms of performance have been developed in the qualitative research community. The SAGE Handbook of Qualitative Research (Denzin & Lincoln, 2005) includes chapters on performance in the same genre as Mienczakowski’s (2000) staged production (Finley, 2005), as well as chapters on poetry (Brady, 2005; Stewart, 2005) and visual arts (Harper, 2005). In addition, Leavy’s (2009) book on arts-based methods in research offers a similarly wide range of possible performance genres. Borum (2006) used poetic prose as a method of qualitative reporting in her work with African American women with deaf daughters (see Box 13.5). She interspersed poetic lines with excerpts from her interview data to create a written report that exemplified the strength and resilience she found in these mothers and daughters.

BOX 13.5 Poetic Prose as Qualitative Reporting

She loves struggle—

If I do what God tells me to do, things will work out. Even if I have hard times, I have hard times gloriously. I enjoy my hard times. I can praise Him when I’m doing good, or not doing good. Abundance or nothing, I’m alright with Him. I don’t have any problems saying I don’t have. My Father is rich and I know He won’t let me suffer and I just move on. . . . It gets rough sometimes because it is so hard because I’m in this thing by myself, but I’m not, it just seems like it.

It has been very challenging. I welcome the challenge. I love it.

She loves Spirit—

And I started walking and God told me all the things He would do for her if I stayed with her. It will be rough but He will take care of me and her, and I will be able to handle anything that came against me. God said, “I’ll do anything but fail!”

And, God—definitely. Without Him, where would we be? I give Him all the praise. He has been my strength, and I believe through Him my child will prosper . . .

The first few days were stressful. I had a lot of support, Bible study group, my faith was very strong, and a lot of prayers were all very helpful. I got strength from God.

She loves to love—

I find that this young woman gives me another dimension. I try and I love all these children, but she’s special to me. I actually think she would be special even if she weren’t deaf . . . and I love learning with her. I love helping her. I love the fact that I can see in her eyes that I have improved in my signing and I know that makes her proud of me. I love the fact that I can communicate with her now and that she likes being with me and likes telling me things and comes to me and talks to me.

—Borum, 2006, p. 346

p. 456

p. 455

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 32/39

Web-Based Dissemination Researchers can also use technologically based dissemination strategies such as creating Web pages and dissemination through text or Twitter. When planning for a Web-based dissemination, researchers should be cognizant of accessibility issues for diverse audiences. The Universal Design checklist provides guidance in terms of accessibility for people with disabilities. Web accessibility guidelines have been developed by the Web Accessibility Initiative and can be found at their website: http://www.w3.org/WAI/. Some things to be aware of include the use of color, text, plain language, compliance with federal regulations (such as 508), and optimization for use on mobile devices. High contrast of black and white is good; colors associated with color blindness are not so good (red and green).

Clarity in Writing Clarity in writing seems, at first glance, to be an essential characteristic of a research report, especially one that purports to be directed at political change—a goal toward which transformative researchers strive. As obvious as this criterion might seem, it is not uncontested in the scholarly world. Lather (1995) was criticized because her published writings, which feature complex language and a complicated writing style, are not considered to be easily accessible to many audiences. In her own defense, and as a point to be made more broadly about language, she warns that simple, clear writing might disguise the complexity of an issue. She says, “Sometimes we need a density that fits the thoughts being expressed” (p. 4). She raises some provocative questions:

What would it mean to position language as revealing or productive of new spaces, practices, and values? What might be the value of encouraging a plurality of theoretical discourses and forms of writing in a way that refuses the binary between so-called “plain speaking” and complex writing? What are the power issues involved in assumptions of clear language as a mobilizing strategy? What are the responsibilities of a reader in the face of correspondence theories of truth and transparent theories of language? What is the violence of clarity, its non-innocence? (p. 4)

Lather (1995) contends that writing that the reader is able to understand is accomplished at the cost of filtering the information to minimize demands on the reader. To make use of a text, a reader needs to see it as an opportunity to wrestle with ideas, become reflective, read it again, and come up with a personal understanding.

EXTENDING YOUR THINKING

Clarity in Writing Reread the preceding passage in which Lather (1995) contests the “innocence” of clear, simplistic writing. She further explains her own way of writing as follows:

Across the sweep of post-humanist theory, I find confirmations of and challenges and directions to my efforts. I am on to something, inchoate as it often is, turning to the theory that helps me articulate the investments and effectivities of what I have wrought, reading both the affirmations of my efforts and the critiques of it in a way that lets me keep on keeping on, stubbornly holding on to the rhythms of the unfoldings of a book that is as much writing me as the other way around. This exploration of possibilities in the face of limit questions marks my desire to “trouble” the dualism between calls for accessibility and the assumption that academic “High Theory” is a sort of masturbatory activity aimed at a privileged few that can have no “real” effect in the material world. (p. 12)

What is your view of the dichotomy set up (and rejected) by Lather in this passage: accessibility versus “High Theory”? What do you think Lather means when she says that simple, clear writing is not “innocent”? Can you think of examples of things you have struggled to read and then found that you gained

d d diff t d t di ith di ?

p. 457

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 33/39

new, deeper, and different understandings with rereading?

Representation. Qualitative researchers participated in a panel discussion about issues related to writing their research reports at the International Congress on Qualitative Inquiry in 2007 and published a transcript of their remarks (Ellis et al., 2008). They raised an important ethical issue in the form of a question: “What happens when the people in your study don’t see what you wrote about them until after it is published and they are angry when they see how they are described?” This question goes beyond the ethical principles used by ethical review boards and centers on the more relational aspects of research writing.

Ellis (2007) described the tensions she felt in her work in wanting to be friends with the people in the community yet also wanting to build her career in academia. The people told her things that you tell a friend about relationships, dating, and scandals, but a friend would not reveal such secrets. She admitted that

writing; I failed to consider sufficiently how my blunt disclosures in print might affect the lives of the people about whom I wrote. Instead I cared about how committee members reacted to my dissertation and whether my manuscript would be published as a book. (p. 10)

When she returned to the community, she was faced with people who were angry about what she had written about them, and she said this experience motivated her to conduct her research in a different and more egalitarian and participative way. She gives this sage advice to people who inquire as to how to present results in an ethical manner:

I tell them our studies should lead to positive change and make the world a better place. “Strive to leave the communities, participants, and yourselves better off at the end of the research than they were at the beginning,” I say. “In the best of all worlds, all of those involved in our studies will feel better. But sometimes they won’t; you won’t.” I tell them that most important to me is that they not negatively affect their lives and relationships, hurt themselves, or others in their world. I tell them to hold relational concerns as high as research. I tell them when possible to research from an ethic of care. That’s the best we can do. (p. 26)

Utilization of the Research Results Utilization of research results is more likely to occur when the researcher integrates strategies to enhance utilization into the research proposal. As mentioned in Chapter 2, the Standards for Program Evaluation (Yarbrough, Shulha, Hopson, & Caruthers, 2011) lists utilization as the first, and most important, criterion for judging the quality of an investigative effort. Although their focus was on investigations for evaluation purposes, the importance of utilization of research in education and psychology should not be overlooked. The following strategies have been identified to enhance utilization of research:

1. Identification and involvement of appropriate audiences for the proposed research, including representation of those who would be most likely to benefit from, or be hurt by, the research.

2. Frequent and appropriate methods of communication with the intended users of the research, including targeting reports to appropriate audiences.

3. Provision of reports that clearly describe the theoretical framework for the study, the procedures, and the rules for interpretation of the data.

4. Reaching intended users of the research through a variety of dissemination modes, with presentation of the research results in a timely manner, such that the information can be used for decision making.

Types of utilization of research and evaluation findings range from sharing results with participants to determine next steps to publication in scholarly journals to serving as a basis for social action at the

li l l R i ’ (2000) k l i j i l i A b d J i h l

p. 458

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 34/39

policy level. Rosenstein’s (2000) work on an evaluation project involving Arab and Jewish elementary schools in Jerusalem offers one example of the use of video-based data to stimulate thinking about next steps in this war-torn area (see Box 13.6). The program used folklore in the form of traditional activities, such as doll making, pickle making, and games, as a basis for joint learning experiences in matched Arab and Jewish schools in Israel (Traditional Creativity through the Schools Project operated by The Center for Creativity in Education and Cultural Heritage, Jerusalem6). Later in the project, selections of the video footage were used as a way of reporting back findings and allowing the participants to both interpret what they were seeing and draw inferences about next steps in the program.

BOX 13.6 Utilization of Evaluation Study Results Using Video-Based Images

Before actually viewing the videotape, the stakeholders discussed “what had happened” during the event. The discussion was hypothetical and there was no development. It did not lead to self-generated knowledge, but rather, reiterated preconceived notions. The general feeling as expressed by one of the mothers was “They (the boys) are simply not interested in the program or in communicating.”

In the post viewing session, however, the participants questioned this “given.” “Perhaps, they do want to make friends?” “Look, they are interacting!” This uncertainty concerning the “fact” enabled them to reflect on the event, to examine the issues more deeply, entering into a discussion of kinds of interaction and which kind they wanted as a suitable goal for the program. . . . There was a consensus concerning “what happened” as confirmed by the video, and the interpretations followed. Each interpretation added to the general development of the discussion. From their exclamations during the viewing, “There is a connection. You can see it,” it was clear that there was a discrepancy between what they thought had occurred and what they saw on the screen. Their surprise, “It seems that something is happening” sparked their reflection: “How can we use that connection and build it into a more meaningful relationship?” This reflection in turn generated the ensuing productive discussion. “For my part, if the children learn from this that they are all people, with interests and preferences, then, that’s enough for me.”

—Rosenstein, 2000, pp. 386–387 Used with permission from Elsevier.

Writing for Publication Journals vary in what they are looking for and will find acceptable for publication. Inside the front or back covers and at the websites of most journals, you will find a publication policy statement. Typically, the journal editors prepare a statement that describes the type of articles that they want to publish. Some journals specialize in theoretical work, others focus on empirical research studies, and some publish a combination of both. You generally find a description of the content that the editors view as being appropriate for that journal as well. You can sort of guess what that description will say by the title of the journal; however, it is good to review the statement as well as a sampling of the articles that have been recently published in that source.

When you submit a manuscript for consideration to a journal, if it is a refereed journal, the editors will send the manuscript to several reviewers. The reviewers are given a checklist that allows them to make suggestions about your manuscript:

• That it be published as is (if you receive such a letter, save it, frame it, and hang it on the wall) • That it be published with minor changes • That it be revised and resubmitted for another review • That it be rejected as inappropriate for that journal

Reviewers are typically given a set of criteria for justifying their ratings of various aspects of the

p. 459

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 35/39

manuscript. Usually, they will be asked to rate such things as the following:

• The clarity of the problem • The logical progression of ideas • The significance of the issues raised (for the readers of that journal, specifically) • The appropriateness of the research design for that problem • The appropriateness of the conclusions based on the data analysis • The readability of the text • Appropriateness of tone (not overly emotional) • Need for additional editing • Appropriateness of references (in terms of quantity, quality, inclusion of important studies, and

timeliness)

If you are thinking about publishing your research as a book, you should do some research on publishers to see who is publishing books on similar topics. This is important not just to determine potential interest on the part of the publisher, but also to assess the publisher’s ability to market the book to appropriate audiences for you. When you have identified one or a few potential publishers, it is appropriate to contact them and ask for their prospectus guidelines. Although publishers vary somewhat, they typically have an outline that delineates the type of information they need about your intended book to make a decision about their desire to publish it for you. Some publishers will request a sample chapter, whereas others will be satisfied with a detailed outline. Your prospectus is usually sent out to reviewers by the publisher who then uses their comments as a basis for deciding to accept or reject your book proposal.

Use for Social Change Many researchers believe that their responsibility is limited to the creation of knowledge. Hence, once the data are collected, analyzed, and reported, their job is essentially finished. However, researchers who place themselves in the transformative paradigm hold a fundamental belief that directs them to facilitate the use of their research findings for social action. This, of course, is not an unproblematic stance. For example, how can researchers be held responsible for the use of their findings (Ginsberg & Mertens, 2009)? What happens if the members of the community do not have the means to use the research themselves for social change? Does the responsibility then revert back to the researchers? There are no simple answers to these questions. However, there are resources that have been developed that offer strategies for increasing the probability that research results will be used for social actions focused on increased social justice. In particular, The California Endowment (Guthrie, Louie, David, & Foster, 2005), a foundation that supports equity in access to health services, and a grassroots organization, the Work Group on Health Promotion and Community Development at the University of Kansas in Lawrence, developed community-based toolboxes that lead the reader through a step-by-step process to use research findings for social change. Both organizations have made their resources available for free on the Web (www.calendow.org and http://ctb.ku.edu).

Use of Research in the Courts. Ancheta (2006) reviews the many times that research data have been used in the courts to demonstrate disparities in education and the consequent need for increased constitutional protection for racial and ethnic minorities, women, immigrants, and other subordinated groups. This intersection of research and court personnel is complicated by the dissimilarities between these two cultures in terms of language, expertise, and standards of evidence. Ancheta asks, “How should education researchers move forward in the coming years, particularly when educational inequality remains a pressing problem in American society and educational policies designed to foster racial integration and diversity continue to be challenged in the courts?” (p. 29). He notes that the future will bring additional questions that educational and psychological researchers need to consider, such as how the racial diversity in student bodies teachers and faculty constitutes a compelling state

p. 460

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 36/39

such as how the racial diversity in student bodies, teachers, and faculty constitutes a compelling state interest and whether training minority professionals will lead to improved educational and life experiences for members of their own community. Ancheta recognizes that the courts play a different role than policymakers in education and social policies; however, they do have a part to play in bringing to light the need for revision in public policies that allow problems of segregation and inequality to continue.

Use of Research: Final Reflection. Ferdinand, Pearson, Rowe, and Worthington (2007) revisit the tension created between different purposes of research in their discussion of what the researcher’s responsibility is if corruption is discovered as part of the research findings. They note that for researchers who accept the purpose of research as creating knowledge, there is no need to worry about revealing corruption. Speaking from a transformative stance, they write,

Critical researchers have an ethical responsibility to expose unjust or unethical exploitative, oppressive or illegal practices. The point is not just to understand the world we live in; the point is to change it. (p. 532)

Rapport and ethics: builds up trust and reduces “reactivity,” and thus helps the researchers to capture social reality, in whatever setting, as it really is. But is this not a mild form of deceit and exploitation? We argue that research activism brings about greater awareness of the issues and concerns of people going about their daily lives. The real danger of codes of ethics lies in their potential to silence those voices that do not fit with the current dominant view of ethical research standards and behaviour. If we are complicit in this silencing, as researchers, we are behaving unethically. (p. 540)

EXTENDING YOUR THINKING

Reporting Research Results

1. Elijah Anderson (1990) conducted research in a run-down neighborhood of poor and working-class Blacks. In an article in The Chronicle of Higher Education, Anderson was described as feeling frustration at the way various groups interpret and use his work (Coughlin, 1994). The following passage appeared in the article:

Lingering racism and lack of jobs, he insists repeatedly, almost like a mantra, are at the root of the ghetto’s chaos and despair.

So it is a particular frustration to Mr. Anderson that conservative pundits and others have seized on his work, reading it as evidence for the necessity of cracking down on crime and reforming the welfare system. . . . “Conservatives, liberals, whoever, pick pieces of it to make their points,” he [Anderson] says. “My job is to describe and represent and analyze in such a way that people who have no experience in that setting can learn something.” (p. A9)

Critically analyze Anderson’s description of the role of the researcher in terms of representation, interpretation, and utilization of research.

2. Identify a research article that involves participants or stakeholders in the analysis, interpretation, and use of the data and findings. How did the researcher accomplish this? What kinds of formats were used to give voice to all participants? How was interaction encouraged and valued? Who was included in this process? Who was excluded?

What kinds of changes are necessary to bring about a true discourse between researchers and practitioners? What can be done to make research more “usable” for practitioners? What responsibility do researchers have for interacting with the people who will be affected by the research results?

p. 462

p. 461

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 37/39

Questions for Critically Analyzing Data Analysis and Interpretation

Quantitative Research The reader is referred to the general statistical texts referenced at the beginning of this chapter for further explanations of the statistical terms and concepts used in these questions.

1. What types of statistical analysis were used? Were they appropriate to the level of measurement, hypotheses, and the design of the study? What alpha level was used to determine statistical significance?

2. Is there statistical significance? What was the effect size? 3. Does the researcher interpret significance tests correctly (i.e., avoid saying the results were highly

significant or approached significance)?7 4. When the sample size is small and the effect size large, are the results underinterpreted? Or if the

sample size is large and effect size modest, are the results overinterpreted? 5. Are many univariate tests of significance used when a multivariate test would be more

appropriate? 6. Are basic assumptions for parametric, inferential statistics met (i.e., normal distribution, level of

measurement, and randomization)?

Qualitative Research

1. Did regularities emerge from the data such that addition of new information would not change the results?

2. Was there corroboration between the reported results and people’s perceptions? Was triangulation used? Were differences of opinions made explicit?

3. Was an audit used to determine the fairness of the research process and the accuracy of the product in terms of internal coherence and support by data?

4. Was peer debriefing used? Outside referees? Negative case analysis? Member checks? 5. Is the report long and rambling, thus making the findings unclear to the reader? 6. Was the correct conclusion missed by premature closure, resulting in superficial or wrong

interpretations? 7. Did the researcher provide sufficient description?

Interpretation Issues

1. How do you account for the results? What are the competing explanations and how did the authors deal with them? What competing explanations can you think of other than those the author discussed?

2. How would the results be influenced if applied to different types of people (e.g., rural or urban)? 3. What were the processes that caused the outcomes? 4. What conclusions and interpretations are made? Are they appropriate to the sample, type of study,

duration of the study, and findings? Does the author over- or undergeneralize the results? 5 Is enough information given so that an independent researcher could replicate the study?

p. 463

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 38/39

5. Is enough information given so that an independent researcher could replicate the study? 6. Does the researcher relate the results to the hypotheses, objectives, and other literature? 7. Does the researcher overconclude? Are the conclusions supported by the results? 8. What extraneous variables might have affected the outcomes of this study? Does the author

mention them? What were the controls? Were they sufficient? 9. Did the author acknowledge the limitations of the study?

EXTENDING YOUR THINKING

Critically Analyzing Data Analysis and Interpretation

1. Answer the questions for critically analyzing data analysis and interpretation in quantitative research for a study that you identified in your literature search.

2. What is the basis for judging the quality of data analysis and interpretation in qualitative research?

3. Answer the questions for critically analyzing data analysis and interpretation in qualitative research for a study that you identified in your literature search.

Summary of Chapter 13: Data Analysis, Interpretation, and Use

Data analysis strategies for quantitative data are generally statistical in nature, and the choice of the appropriate statistic is based on the purpose of the research, the design of the study, and the characteristics of the data themselves. Qualitative data analysis can be started even before the interview or observation notes are collected. Researchers can begin writing their thoughts and feelings in a journaling format and use that as part of the data analysis. Qualitative data generally consist of words but can also include visual items such as artifacts, video, and pictures. Interpretation of both types of data requires sensitivity to cultural issues. Strategies for use of research findings range from publication in scholarly formats (including dissertations or theses) to serving as a basis for social action.

Notes

1. More complex correlational statistics are explained in Chapter 5 as part of the causal comparative and correlational approaches to research.

2. The experimental study summarized in Sample Study 1.1 also used a hierarchical linear regression statistical analysis. This allowed G. D. Borman et al. (2007) to test the effects of both school-level and student-level effects. This approach used school-level pretest scores as a covariate before achievement scores were compared.

3. It should be noted that the choice of an analytic strategy is not without controversy. Some researchers claim that you should not use parametric statistics if you cannot satisfy the assumptions. In practice, many researchers assume that the parametric statistics are robust—that is, the assumptions can be violated without serious consequences.

4. Greene (2007) discusses various strategies for importing data of one type into a software program of another type to enable the researcher to conduct analyses that include both types of data. Bazeley (2006) describes software programs that allow for data importation of qualitative or quantitative data to facilitate this type of

p. 464

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 04/29/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

4/29/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252322/View 39/39

software programs that allow for data importation of qualitative or quantitative data to facilitate this type of analysis. Greene (2007) also mentions data conversion (i.e., qualitative data is converted to numbers) as another data analytic strategy used in mixed methods research.

5. More guidance can also be found in the Publication manual of the American Psychological Association (6th ed.; APA, 2009).

6. The Center for Creativity in Education and Cultural Heritage is directed by Dr. Simon Lichman, 20 Koreh HaDorot, Jerusalem, 93387.

7. Items 3 through 6 were adapted from B. Thompson (1988).

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods