More Research

4/29/2019 Evaluating Research

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252321/View 1/16

CHAPTER 4

Conceptual Overview of Statistical Analyses

Aristotle could have avoided the mistake of thinking that women have fewer teeth than men by the simple device of asking Mrs. Aristotle to open her mouth.

—Bertrand Russell (Peter, 1980, p. 457)

Overview

The material presented in this chapter is a follow-up to the research strategies presented in Chapter 1. In addition to learning more about the strategies—exploration, description, prediction, explanation, and action—you will be given an overview of statistical analyses appropriate for each strategy. As I’ve written in previous chapters, research involves asking and attempting to answer questions. How the questions are asked determines the methods used to answer them, which in turn partly determine the answers. Another aspect of research that partly determines the answers is how statistical procedures are used to analyze the data. Therefore, research consumers need to understand the statistics used by the researchers whose work they are consuming. Fortunately, at least for those who shy away from statistics and other mathematical topics, research consumers don’t need to know how to calculate statistics. Instead, consumers only need to know enough about statistics to decide whether or not the researchers chose their statistics correctly and to critically assess the researchers’ interpretations of their results. Thus, this chapter deals with what I refer to as conceptual statistics, which includes basic theory and principles; no calculations or formulas are included. For those of you who have yet to complete a course in statistics, the material will serve as an introduction to the concepts of statistical analyses.

INTRODUCTION

Research design and data analyses are interdependent; one makes little sense without the other. One situation researchers should always avoid is having data without a way to analyze them; researchers who don’t avoid this situation engage in strange, sometimes inappropriate, ways to analyze their data in the hopes of salvaging something of their efforts. As a consumer of research, you need to know enough about statistics to determine that a researcher did not choose just any old analyses. As well, you need to know enough about statistics to interpret the results presented in empirical articles, even those written by researchers who knew exactly how to analyze their data. Thus, even though you don’t have to worry about

p. 80

p. 79

04/29/2019 - RS0000000000000000000001583532 - Evaluating Research

4/29/2019 Evaluating Research

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252321/View 2/16

y y , g y y analyzing data, you do need to be able to figure out whether a researcher analyzed data properly and how the results of those analyses can be used to inform the program, policy, or issue you are addressing. The purpose of this chapter is to help you deal with others’ analyses and results by providing an overview of current research strategies and the analyses most appropriate to them. The chapter is organized around the major research strategies: exploration, description, prediction, and explanation. (Action is not included as a separate research strategy because it almost always incorporates one of the other strategies.)

EXPLORATORY RESEARCH

As defined in Chapter 1, exploratory research is an attempt to determine whether or not a phenomenon exists. That is exactly what Bertrand Russell had in mind when he suggested that Aristotle ought to have looked in Mrs. Aristotle’s mouth instead of writing a philosophical discourse on gender differences.

Appropriate Questions for Exploratory Research

Exploratory research is similar to the insistent question asked by columnist and 60 Minutes commentator Andy Rooney: “Did you ever wonder about. . .?” (see, e.g., Rooney, 2003). Anything is fair game when one tries to discover what is out there, wherever “there” may be. Some examples of exploratory research questions include these: Do people think differently about themselves than about others? What do jury members talk about during deliberations? Has anyone ever done research on this topic before? The last question is one of the questions we ask when conducting a literature review.

Exploratory questions tend to be rather general, but that does not mean they are necessarily frivolous or uninformative. At the very least, exploratory research satisfies personal curiosity, but good exploratory research also has heuristic value in that it stimulates the researcher and others to conduct even more research. For example, the attempt by members of the Chicago Jury Project to answer the question about what jurors discuss during deliberations has been the basis for hundreds, perhaps thousands, of other research projects about the behavior of jurors (Ellison & Buckhout, 1981). Nearly any question about the existence or nature of human behavior, or the lack thereof, is an appropriate question for exploratory research.

Appropriate Statistics for Exploratory Research

Just as the questions addressed through exploratory research are usually general, the data analyses used for exploratory research also tend to be general. Often, analyses for exploratory research are qualitative analyses—nonnumerical analyses concerning quality rather than quantity. For example, early research on conformity as a group process dealt with determining whether or not members of a group who initially disagreed with the majority changed their opinions during group discussions (Levine & Russo, 1987). The quality of the behavior—opinion change—was the focus of analysis; change was either there or it was not. Early researchers on this topic were not particularly interested in how much people’s opinions changed, just whether there was a change.

Even when analyses are not qualitative, they usually include calculating descriptive statistics concerning central tendencies or averages—mean, median, and mode—or concerning dispersion of scores, such as range, interquartile range, and standard deviation. Sometimes, histograms or frequency polygons are used to graphically examine data in more detail or to more clearly present data. Travis (1985), for example, used charts of surgery rates in different areas of the United States to illustrate more clearly the discovery that elective hysterectomies were considerably more prevalent than either elective appendectomies or elective prostate surgery.

Other useful exploratory analyses include stem-and-leaf displays and scattergrams. Stem-and-leaf displays (Tukey, 1977) are an alternative to more traditional histograms or frequency polygons. A stem- and-leaf allows one to visualize the entire data set as a distribution of scores, without having to lose information about what the specific scores are. They also enable one informally to examine differences among groups. Figure 4.1 contains a stem-and-leaf display for the publication dates of the references cited in a social psychology textbook (Dane, 1988a). The stem—the left side of the display—contains the root of the date such as 187 for the 1870s or 196 for the 1960s The leaf the right side of the display contains the

p. 82

p. 81

04/29/2019 - RS0000000000000000000001583532 - Evaluating Research

4/29/2019 Evaluating Research

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252321/View 3/16

the date, such as 187 for the 1870s or 196 for the 1960s. The leaf, the right side of the display, contains the remainder of the number, such as 1 in 1871. By examining the stem and the leaf, one can reconstruct the original score: 187 on the stem together with 1 in the leaf indicates the year 1871.

Traditionally, the digits in the leaf are ordered, increasing from left to right, as in the figure. Although not shown in this figure, one may use the same stem more than once if that makes the display easier to read. However, when doing so, researchers will usually use every stem the same number of times. Longer leaves indicate more data points, so researchers use the same spacing for all of the leaves. That way, the leaves represent the distribution of scores much as a histogram does but without masking the actual numeric values of the data.

Scattergrams, on the other hand, allow one to examine relationships between variables or between two different groups measured on the same variable. However, they are more likely to be used for descriptive and predictive research purposes, so we’ll postpone discussing them until those sections.

Figure 4.1

A Stem-and-Leaf Display of the Publication Years for the Citations Included in Dane (1988a). The stem, three-digit numbers along the left side, is the century and decade of the publication year. The leaf, the single-digit number extending to the right from the stem, is used to complete the year. Note how easy it is to recognize the relative abundance of later references compared to the relatively few citations from earlier years. Stem-and-leaf displays enable one to understand the frequencies included in an entire data set and understand the specific scores that comprise the data set.

In general, data analyses for exploratory research fall under the category of “interocular trauma” analyses: The effects are so apparent that they hit you right between the eyes, or they are not apparent at all. It is clear from Figure 4.1, for example, that there are more references from the 1980s than from any other decade; the length of the 198 leaves hits you between the eyes. Even if the effect in which you are interested is subtle, it becomes apparent because you are looking for it; it is either there or it is not.

The kinds of analyses appropriate for exploratory research are displayed in Figure 4.2. Now we’re ready to turn our attention to descriptive research.

Figure 4.2

Both qualitative and quantitative analyses can be used to examine the data from exploratory research. Which quantitative statistics you will see in articles depends upon the nature of the research question; either central tendency, variability, or both may be included in an article. A variety of graphic presentations are also used to display information about the entire data set.

p. 83

04/29/2019 - RS0000000000000000000001583532 - Evaluating Research

4/29/2019 Evaluating Research

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252321/View 4/16

p y

DESCRIPTIVE RESEARCH

Descriptive research involves attempting to define or measure a particular phenomenon, usually by attempting to estimate the strength or intensity of a behavior or the relationship between two. Rather than assessing whether or not something is going on, descriptive strategies involve assessing exactly (more or less) what is going on.

Appropriate Questions for Descriptive Research

Questions for which a descriptive strategy would be appropriate are those in which the researcher explores the limitations of a phenomenon. These might include its distinctiveness from other phenomena, the extent to which it occurs in various situations, or its strength or quantity. Examples include such questions as these: Under what circumstances do people think differently about themselves than about others? How much time do jurors spend talking about evidence during deliberations? How many different operational definitions of this phenomenon have been used in previous research? These are, of course, descriptive versions of the exploratory questions presented in the previous section. Descriptive research does not necessarily involve different research topics, but it does involve different questions about those topics.

Usually, one of the purposes of descriptive research is to generalize—to relate the findings gathered from the research situation to other situations Typically we want to generalize from the research

p. 85

p. 84 04/29/2019 - RS0000000000000000000001583532 - Evaluating Research

4/29/2019 Evaluating Research

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252321/View 5/16

from the research situation to other situations. Typically we want to generalize from the research participants to another group of people or to people in general. Obtaining an answer to the question about how much time the jurors in our study spend discussing evidence is considerably more useful if that answer can be extended to other jurors as well. Generalization requires external validity—similarity between the physical and social aspects of the research environment and the target environment (Shadish, Cook, & Campbell, 2002). Generalization also requires the use of inferential statistics.

Appropriate Statistics for Descriptive Research

When we attempt to generalize results, we make an inference about the relationship between the research participants and the target of our generalization. Imagine, for example, that we discovered that a particular sample of jurors spent an average of 50% of their deliberation time discussing evidence. If that research sample represents the entire population of jurors, then we might conclude that all jurors spend about 50% of their time in deliberations discussing evidence. That generalization requires the use of inferential statistics.

Inferential statistics are values calculated from a sample and used to estimate the same value for a population. That is, inferential statistics are estimates, based on a given sample, of qualities or quantities existing in a larger group of individuals. The basis for all inferential statistics is a mathematical principle known as the central limit theorem. Fully explaining the central limit theorem is beyond the scope of this book and is not really necessary for our purposes. However, most statistics textbooks include an explanation of it if you are interested (Nowaczyk, 1988; Wike, 1985; Winer, 1971; Witte & Witte, 2006).

One important aspect of the central limit theorem is that it enables us to use a sample to estimate a population if the sample has been obtained by random sampling—a process by which every member of the population had an equal opportunity to be included in the sample. Random sampling, then, does not mean just any old sampling procedure but one with a system that ensures that each person, place, or thing in the population has an equal chance of being included in the sample. Selection of a lottery winner, for example, is a random sampling procedure. The population includes every entry entered into the lottery, and each entry has an equal chance of being selected. If all entries are placed in the equivalent of a large box, mixed, and then chosen one at a time, then every entry in the lottery had an equal chance of being selected. (The fact that some people may purchase more than one entry doesn’t change the fact that each entry has an equal chance of being selected.)

Another important aspect of the central limit theorem is that we can use it, if we have a random sample, to estimate the amount of measurement error associated with any values obtained from the sample. Once we have determined that a sample of jurors spends 50% of their deliberations discussing evidence, we can also determine the accuracy range of that estimate. When reporters announce that Candidate X has obtained 40% of the popular vote in an election, plus or minus 2%, they are reporting an accuracy range (± 2%) for the estimate obtained from their sample.

The types of inferential statistics used to analyze descriptive research data include some of the same calculations used for exploratory research, such as mean, median, mode, standard deviation, and so on. The difference is that these calculations are then used to make inferences, rather than simply describing the data collected from the sample. Other analyses include chi-square (also known as crossbreak analyses), correlation, factor analysis, t test, analysis of variance (also known as ANOVA), and meta-analysis. Although a detailed explanation of each of these procedures is not within the scope of this text, I will provide a brief overview of their use in descriptive research.

Chi-square analyses are statistical techniques used to determine whether the frequencies of scores in the categories defined by the variable match the frequencies one would expect on the basis of chance or on the basis of predictions from a theory. Chi-square analyses are used to make inferences when the data are categorical or nominal—involve measuring participants in terms of categories such as male-female, voter- nonvoter, and so forth. Chi-square procedures can be used to determine whether a relationship exists between two or more categorical variables or whether the categories obtained from one sample of individuals are similar to the categories obtained from another sample. Parish et al. (2000) for example, reported a relationship between type of arrhythmia and survival in their examination of in-hospital resuscitations. Not surprisingly, less severe arrhythmias were more likely to be associated with a successful resuscitation attempt. When the results of chi-square analyses are reported, you are likely to see text that looks something like X2 (2) = 5.7, p < .05, where X2 is the symbol for chi-square, the number in parentheses

p. 87

p. 86

04/29/2019 - RS0000000000000000000001583532 - Evaluating Research

4/29/2019 Evaluating Research

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252321/View 6/16

g ( ) , p , y q , p —in this case 2—reflects the degrees of freedom for the analysis, and the number after the equal sign reflects the actual value for the statistic. This value is then compared against values that should be obtained if the null hypothesis is correct, and this comparison is then used to determine the probability that the data represent the null hypothesis. The p, of course, is the symbol used to indicate the probability that the obtained result is due to chance or random error, which is then followed by the value for that probability.

Similarly, chi-square may be used to determine whether there is a relationship between gender and voting behavior, or it may be used to determine whether psychology majors are composed of a different ratio of men to women than are chemistry majors. Chi-square analyses are frequency analyses; they involve comparing the frequencies of various categories. Haberman’s two volumes (1978, 1979) are excellent sources for more information about chi-square and other categorical analyses.

When research measures involve continuous variables, such as income, grade point average, age, intensity of emotion, and so on, correlational procedures are used to make inferences about relationships between variables. For example, the relationship between the size of a city and its crime rate can be described with correlational analyses. Correlations are statistical procedures used to estimate the extent to which the changes in one variable are associated with changes in the other variable. Essentially, a correlation coefficient is a number summarizing what may be observed from a scatterplot, a graph in which corresponding codes from two variables are displayed on two axes. When data analyses involve correlations, you are likely to see something like this: r (42) = .37, p < .05. The r is the symbol used for the correlation coefficient, the number in parentheses represents the degrees of freedom for the statistic— usually the number of pairs of scores minus one—and the number after the equal sign is the actual value of the correlation coefficient.

Figure 4.3 is a scatterplot representing a positive correlation using data from a study of medical students’ scores on a measure of cynicism at the beginning of their first and second years of medical school (Roche, Scheetz, Dane, Parish, & O’Shea, 2003). Positive correlations reflect a direct relationship—one in which increases in one variable correspond to increases in the other variable. In Figure 4.3, it is clear that students who were more cynical than their peers during their first year of medical school were also more cynical than their peers in their second year of medical school. Notice that there are more overlapping data points in the center of the graph than at the edges of the graph, which also illustrates the general tendency for most of the students to score “average” on the measure of cynicism. If I could make Figure 4.3 a three- dimensional figure, you would see the higher frequency data points appear to come out from the page toward you.

Figure 4.3

A Scatterplot of the Relationship Between Scores on a Measure of Cynicism in the First and Second Years of Medical School. Students who scored high in the first year tended to score high in the second year and vice versa. The darker, denser areas of the plot indicate a greater number of overlapping points or higher frequency in the data set.

p. 88

04/29/2019 - RS0000000000000000000001583532 - Evaluating Research

4/29/2019 Evaluating Research

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252321/View 7/16

Of course, not all variables that are related exhibit a direct relationship. Figure 4.4 contains the scatterplot of two variables one might expect to be indirectly or inversely related, cynicism and altruism. Such a relationship would produce a negative correlation, indicating that increases in one variable are associated with decreases in the other. In Figure 4.4, it is apparent that medical students who are more cynical tend to be less altruistic than their peers (Roche et al., 2003).

Figure 4.4

A Scatterplot of the Relationship Between Beliefs About Cynicism and Beliefs About Altruism. As the strength of the belief about cynicism increases, belief in altruism decreases. The darker, denser areas in the plot indicate a greater number of overlapping points or higher frequency in the data set.

And finally, Figure 4.5 contains the scatterplot of two variables that exhibit no relationship: a random number and first-year cynicism. Again, if I could produce the figure in three dimensions, the scatterplot in Figure 4.5 would appear to be a bell with its base on the page and its top coming out toward you. Notice that both variables, cynicism and the random number, tend to exhibit more data points toward the center of their respective scales, again merely indicating that more people tend to have average cynicism scores and that the random number generator I used has an average of about 0.50. However, for any given random number, there is no particular cynicism score that is more prevalent, except the average score. p. 90

p. 89

04/29/2019 - RS0000000000000000000001583532 - Evaluating Research

4/29/2019 Evaluating Research

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252321/View 8/16

Figure 4.5

A Scatterplot of the (Lack of) Relationship Between Medical Students’ Cynicism and Scores From a Random Number Generator. The circular pattern of the plot indicates that there is no relationship between the two variables. The denser pattern of dots in the center of the plot indicates that more people have average scores.

A very specialized type of correlational analysis is called factor analysis, a statistical technique used to identify groups of variables that share variance in common or measure the same concept. Another statistical procedure, known as principle components analysis, is conceptually similar to factor analysis; the math is somewhat different (Lindeman, Merenda, & Gold, 1980), but interpretation is essentially the same for our purposes. The principle underlying factor analysis is that any given variable probably measures more than one theoretical concept, and that the variance produced by employing more than one such variable can be rearranged so that all of the “pieces” of variance can be rearranged to reflect the different concepts measured by the collection of variables. Typical reports of results from factor analysis include a table in which factor loadings are used to show how strongly each variable is associated with the various dimensions (concepts) represented by the collective responses to the variables. An example of a factor analysis, including results, may help to make this descriptive procedure more understandable.

You are probably familiar with course evaluation forms, sometimes called student opinion forms, through which students have an opportunity to provide evaluative ratings about the course and instructor at the end of the semester. At my university, there are 21 primary feedback items (variables) included on the form, and these items are organized into sections about (1) instructor involvement, (2) student interest, (3) student-instructor interaction, (4) course demands, and (5) course organization. One question that can be asked about student responses to these variables is whether the students in my classes consider the separate sections of the instrument to be, indeed, separate. That is, we can ask whether the five separate sections measure five different aspects of the course experience or whether there is some smaller number of dimensions that can be used to describe students’ reactions to my courses. The results of the factor analysis are displayed in Table 4.1, in which the section numbers and items are listed in the left column and the “rearranged pieces” of the items—the factors—are displayed in the five right columns. Each of the factors represents a concept measured by the pieces of the items. The numbers in those columns—the factor loadings—represent the extent to which the item contributes to the concept measured by the factor. Loadings can range from −1 00 to +1 00; generally a loading of 4 or higher indicates a sizable contribution

p. 91

04/29/2019 - RS0000000000000000000001583532 - Evaluating Research

4/29/2019 Evaluating Research

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252321/View 9/16

Loadings can range from 1.00 to +1.00; generally, a loading of .4 or higher indicates a sizable contribution to that factor.

Table 4.1 Factor Loadings Derived From Students’ Responses to a Course Evaluation Form in Which the Items Were Arranged Into Five Separate Sections. Loadings in bold type indicate a sizable contribution of the item to that factor.

Although the results of the factor analysis contain five different factors, the items are not grouped t ti ti ll tl th th d th l ti f Th f i t t

p. 92

04/29/2019 - RS0000000000000000000001583532 - Evaluating Research

4/29/2019 Evaluating Research

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252321/View 10/16

statistically exactly the same way they were grouped on the course evaluation form. The four instructor items (Section 1) are grouped together in Table 4.1 on Factor 1, but part of the “student has become more competent” item loads on the same factor. Thus, we can conclude that a student’s belief in his or her increase in competence is tied to his or her reaction to the qualities of the instructor. Similarly, from the loadings on Factor 2 we can conclude that students’ opinions about how intellectually challenged they felt are related to the organization of the course. Factor 3 clearly involves the difficulty of the course and is the only factor that corresponds entirely to the section on the questionnaire.

Factor 4 represents the opportunities for participation provided to the students, but interestingly, the instructor’s stimulation of class discussion is not a part of this concept. Factor 5 represents the students’ interest and attentiveness, which is partly related to how much the students enjoyed the course. Interestingly, the student’s overall enjoyment of the course is “pieced” into the student’s interest and the organization of the material, but this is independent of course difficulty, the instructor’s enthusiasm and interest in teaching, and the opportunities for participation. We could, of course, continue discussing these results for a long time, but you should have enough knowledge about factor analysis to understand how it can be used for descriptive research purposes.

More often than not, descriptive research involves trying to determine whether two groups differ according to some quality, such as whether women or men tend to commit more crimes or whether psychology majors or chemistry majors perform better on the Graduate Record Exam. Essentially, such research involves comparing the central tendency of one group with the central tendency of another, and the t test (for two groups) or ANOVA (for more than two groups) are the appropriate statistics. Both statistics enable one to determine whether groups have equivalent or different mean scores.

The principle underlying t tests and ANOVA is the assumption that both (or all) groups, whatever they may be, represent samples from the same population. Men and women, for example, represent two different samples from the same population (humans). If that assumption is correct—that is, if there are no fundamental differences between men and women; they are just humans—then the two samples should have the same central tendency, the same mean. To the extent that the two groups are different, one can conclude that the assumption about them being from the same population is wrong. Of course, that doesn’t mean, in the case of men and women, that one group is not human. Instead, it means that, on whatever variable is being measured, the two groups represent very different populations of humans; they are not the same on whatever the measurement dimension may be. Most any statistical text will include descriptions of t tests and ANOVA. Nowaczyk (1988) and Wike (1985) are among the more readable; Winer (1971) is more advanced and technical.

When researchers report the results of a t test, you will see something like this: t (45) = 3.67, p < .05; “t” is, of course, the symbol for the t test, and the number in parentheses represents the degrees of freedom for the statistic. The number after the equal sign is the calculated value of the statistic. For ANOVA, you will see something like: F (2, 27) = 3.50, p < .05. “F” is the symbol for ANOVA and the number after the equal sign is the actual value of the statistical test. ANOVA has two indicators for degrees of freedom within the parentheses. The first of these represents (but is not equal to) the number of groups compared, while the second indicates the degrees of freedom for the error term.

One of the more complicated forms of ANOVA is multivariate analysis of variance (MANOVA), which is employed whenever two or more dependent variables are analyzed simultaneously, usually because there is reason to believe that the multiple dependent variables are related in some way (Lindeman et al., 1980). Essentially, MANOVA is analogous to employing regular ANOVA except that the “variable” analyzed is the factor, or common variance, of the several dependent variables that actually were measured. Imagine, for example, comparing the course evaluations of several members of a single department. In such a study, MANOVA would be used on each of the factors identified in Table 4.1 above.

In some cases, the data collected in a descriptive research project are the results of other researchers’ studies, in which cases researchers often use meta-analysis to detect trends in published reports. Meta- analysis is the collective name for the various quantitative techniques used to combine the results of empirical studies. The term was coined by Glass (1976) and, since its initial development, has come to include a variety of different statistical techniques. What all of these techniques have in common, however, is that the unit of analysis is the statistical result of data analyses instead of the raw data collected by a researcher (Wachter & Straf, 1990). When reporting the results of a meta-analysis, researchers use effect size, which is a statistical term for the estimate of the magnitude of the difference between groups or the relationship between variables. The effect size from each study is combined with effect sizes from all of the

p. 95

p. 94

p. 93

04/29/2019 - RS0000000000000000000001583532 - Evaluating Research

4/29/2019 Evaluating Research

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252321/View 11/16

elationship between va iables. e e ect s e o eac study s co b ed w t e ect s es o a o t e other studies on the same phenomenon in order to determine, among other things, the average effect observed in the reviewed literature. For example, Kulik, Kulik, and Bangert (1984) used a meta-analysis of research published on the effects of practice tests upon scores on standardized tests such as the SAT. Based on their review of 40 studies, Kulik et al. were able to use effect size to estimate that the average increase in total score on an aptitude test, such as the SAT, was about 20 points for a student’s completing a practice test.

Effect size can also be used in many other kinds of analyses, such as those used for predictive and explanatory research, a point to which we shall turn later in this chapter. The kinds of data and analyses for descriptive research are depicted in Figure 4.6. We next turn to a discussion of predictive research.

Figure 4.6 The kinds of quantitative statistics researchers typically use to analyze data from a descriptive study depends on the types of questions one is asking and the types of data one is using to attempt to answer them.

PREDICTIVE RESEARCH

Predictive research involves any study in which the purpose is to determine whether a relationship between variables exists such that one can use one of the variables in place of another. Such research may be done in order to avoid using one of the variables because it is too expensive or time-consuming or to avoid having to wait for an appropriate situation in which to use one of the variables. College admissions committees use the results of predictive research for accepting new students; it’s too costly to wait an entire year to find out whether or not a student will do well enough to remain in school. Or one might try to predict jurors’ decisions from their attitudes toward the legal system or try to predict whether or not an attempt at cardiopulmonary resuscitation is likely to be successful for a particular type of patient.

In any predictive research, the response variable is the measure one would like to predict, and the predictor variable is the measure one hopes will predict the response variable. These are, respectively, analogous to dependent and independent variables in an experiment. SAT score, for example, would be the predictor variable and first-year grade point average would be the response variable in the research admissions committees use to select students. Graduate admissions committees also make use of the Graduate Record Exam for the same purposes (Educational Testing Service, 1981).

p. 96

04/29/2019 - RS0000000000000000000001583532 - Evaluating Research

4/29/2019 Evaluating Research

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252321/View 12/16

Appropriate Questions for Predictive Research

Predictive research involves measuring relationships between two or more variables, and appropriate questions include any question that fits the general form “Is X related to Y?” or “Can X be used to predict Y?” Predictive versions of the questions used to illustrate exploratory and descriptive strategies would include these: Does someone’s self-esteem predict the impressions formed about others? Is the amount of time jurors spend talking about evidence related to their verdict? Can I predict the outcome of my research from the research of others? Table 4.2 contains a summary of the ways in which these questions have changed as a function of the research strategy. Notice that changes in the questions posed within any given topic are related to the answers one can expect to obtain.

Table 4.2 Exploratory, Descriptive, and Predictive Research Questions Exploratory Do people think differently about themselves than about others?

Descriptive Under what conditions do people think differently about themselves than about others?

Predictive Does someone’s self-esteem predict the impressions formed about others?

Exploratory What do jury members talk about during deliberations?

Descriptive How much time do jurors spend talking about evidence during deliberations?

Predictive Is the amount of time jurors spend talking about evidence related to their verdict?

Exploratory Has anyone ever done research on this topic before?

Descriptive How many different operational definitions of this phenomenon have been used in previous research?

Predictive Can I predict the outcome of my research from the research of others?

Prerequisites for Predictive Research

Because predictive research involves assessing whether or not one variable can be used to predict another, researchers need to be able to determine the extent to which the separate variables actually measure what they are supposed to measure; that is, both variables must be valid. Equally important, however, is that both variables must be as reliable as possible. The requirement for very high reliability results from the fact that the predictive power of the predictor variable is limited to its reliability. If SAT scores, for example, are 80% reliable, then the best they can do is predict about 80% of first-year college grades. Put another way, if the correlation between random halves of the SAT test is only 0.89, then 80% is also the highest amount of variance we can expect to account for when using SAT scores to predict grades.

Statistics for Predictive Research

The types of statistical analyses used for predictive research are all based on correlation—a statistical measure of the degree to which two or more variables are related. In addition to correlation coefficients themselves, the other predominant statistical procedure is regression analysis.

A correlation coefficient generally consists of both an algebraic sign and a number. The sign indicates the direction of the relationship: positive for direct, negative for inverse. In predictive research, the sign is generally not important, except perhaps as a check on calculation accuracy. When our purpose is to predict one variable from another, it makes no difference whether we predict from a direct or inverse relationship. If you look back at Figures 4.3 and 4.4, for example, you should be able to see that one can predict cynicism in the first year of medical school equally well from altruism in the first year or from cynicism in the second year. What matters is the strength of the relationship, not whether the relationship is direct or inverse.

What is important in predictive research is the numeric value of the correlation coefficient. Correlation coefficients can range from −1.00 to +1.00; any coefficient outside this range results from calculation error. A coefficient of −1.00 represents a perfect, inverse relationship and enables prediction with 100% accuracy. Similarly, a coefficient of +1.00 indicates a perfect, direct relationship and 100% predictive power. Such coefficients, however, are extremely rare. A coefficient close to zero indicates no relationship at all and

p. 98

p. 97

04/29/2019 - RS0000000000000000000001583532 - Evaluating Research

4/29/2019 Evaluating Research

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252321/View 13/16

therefore no predictive power. The scatterplots in Figures 4.3 and 4.4 reflect coefficients of about +.70 and −.76, respectively. The square of the correlation coefficient is a measure of the amount of variance the two measures have in common, a measure of the extent to which the two variables measure the same concept. Thus, cynicism in the first and second years of medical school (Figure 4.3) share about 49% of their variance; something happens during the first year to change the students’ cynicism about people (Roche et al., 2003).

More often than not, however, predictive research involves trying to construct a prediction equation, not simply determining the extent to which two variables are related. We might, for example, want to construct an equation to predict the amount of time jurors spend deliberating from the length of the trial. If we could construct such an equation, judges and other court personnel could use the equation to schedule other hearings during the deliberation. If a judge knew the jury was going to deliberate for about two hours, for example, the judge could schedule several short hearings during that time and still be ready to hear the verdict when the jurors have finished deliberating.

Regression analyses are used to construct such prediction equations. A regression equation is a formula for predicting a score on the response variable from the score on the predictor variable. In general, regression equations take the form Y′ = bX + c, where Y′ is the predicted score, b is the regression coefficient, X is the score on the predictor variable, and c is a constant (the point at which the prediction line crosses the ordinate or Y axis on a graph). One example of a regression equation with which you are probably familiar is the expression used to convert Fahrenheit temperatures into Centigrade temperatures. For Centigrade to Fahrenheit conversions, the equation is F = 1.8C + 32. Predictions from a regression equation are approximate and depend on the strength of the relationship between the two variables: The stronger the relationship—the closer the correlation coefficient is to either −1.00 or +1.00—the more accurate the prediction. Fahrenheit and centigrade temperatures are highly correlated simply because they are both highly valid and highly reliable measures of the same thing (heat). The amount of prediction error is very small; however, it is not zero. There is always some error in any measurement, and therefore, there is always some error in any prediction made from a measurement. Estimating the margin of error associated with predictions also falls under the category of regression analysis. Although the mathematics involved are beyond the scope of this text, almost any basic statistics text contains the information you would need.

There are times when more than one predictor variable is used in research, such as when a college admissions board uses SAT scores, high school grades, and high school ranking to predict first-year college grades. In such instances, the analysis known as multiple regression (a statistical technique for estimating simultaneous correlations among any number of predictor variables and a single, continuous response variable) is used. Although considerably more complicated in terms of mathematics, multiple regression relies on the same basic principles as correlation and simple regression. Kerlinger (1979) contains a very readable introduction to multiple regression. When there is more than one predictor variable, the regression equation is expanded to reflect this, and a multiple regression equation might look something like Y′ = b1X1 + b2X2 + b3X3 + c, where b1 refers to the regression coefficient for the first predictor, X1, and so on through c, the constant. Researchers are not likely to report a regression equation in this form, however; the results in Table 4.3 are a more customary presentation of the results from a multiple regression analysis.

The results in Table 4.3 come from a study of medical students’ attitudes about people in general (Roche et al., 2003). The response variable in this analysis is cynicism in the first year of medical school, and the predictor variables include beliefs about people’s trustworthiness, strength of will, and altruism (also measured in the first year of medical school). Notice that all three variables are significant predictors of first-year cynicism, but trustworthiness is the strongest predictor. The column labeled t reflects the fact that a type of the t test is used to determine whether or not the regression coefficient for each predictor is significantly different from zero.

Table 4.3

Results of a multiple regression analysis in which first-year cynicism among medical students is predicted from beliefs about (1) the trustworthiness of people, (2) the strength of will displayed by people, and (3) how altruistic people in general are.

p. 100

p. 99

04/29/2019 - RS0000000000000000000001583532 - Evaluating Research

4/29/2019 Evaluating Research

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252321/View 14/16

Regression analyses, both simple and multiple regression, come in many different forms, but the essence of each form is that one or more predictor variables are tested for its (their) relationship to the response variable. In logistic regression, for example, the response variable is a nominal or categorical measure (determining the presence or absence of a characteristic; naming a quality), which requires different mathematics to calculate the results but produces the same kind of results (regression coefficients). In path analysis, one or more intermediate variables are included in the model such that predictors are used to predict the intermediate variables, which in turn are used to predict other response variables. This enables researchers to understand the sequencing of the predictors; for example, high school grades predict SAT scores, and SAT scores predict first-year college grades. This sequencing enables researchers to understand the process by which one variable influences another, but the results are interpreted the same way one would interpret a simple regression result. Similarly, structural equations analysis is used to examine the predictive power of factors derived from factor analysis, but the results are interpreted much the same way one interprets multiple regression results.

EXPLANATORY RESEARCH

The purpose of explanatory research is to test whether or not one or more independent variables can cause one or more dependent variables. A single explanatory research project, however, will not involve all the potential causes for a given effect, but instead will concentrate on a few. As you learned in Chapter 1, the purpose of explanatory research is to demonstrate that one variable can cause the other, not to demonstrate that the independent variable is necessarily the only cause.

Appropriate Questions for Explanatory Research

Questions for which explanatory strategies are appropriate are those in which a causal relationship is being considered. Using the same topics as in previous sections, these might include these: What causes people to think about the reasons for their behavior? Do different instructions about reasonable doubt affect the length of jury deliberations? Why are my results different from those obtained in previous research? You should also recall from Chapter 1 that explanatory research requires that the researcher be able to manipulate the independent variable and randomly assign participants to the different levels created through manipulation.

Appropriate Statistics for Explanatory Research

You have probably guessed by now that explanatory research involves the use of inferential statistics. If so, you are correct. Because experimental procedures involve creating different groups representing the different levels of the independent variable, inferential statistics are used to determine the extent to which the different groups actually perform differently on the dependent variable. If you can reject the assumption that the groups represent samples from the same population, then you can conclude that the independent variable had some effect.

Any statistical analyses designed to detect differences in central tendencies, such as t test and ANOVA, are appropriate for analyzing explanatory research data. If the dependent variable is a categorical variable, such as verdicts from a jury decision, then chi-square analysis can be used to detect differences between the groups. Contrary to what you might expect, the complicated part of explanatory research is ensuring the internal validity of the research design, not the statistical procedures used to analyze the data obtained from the design. Thus, there are no “special” statistics used for explanatory research. You are likely to encounter any and all of the previously described statistical analyses when reading explanatory research.

p. 102

p. 101

04/29/2019 - RS0000000000000000000001583532 - Evaluating Research

4/29/2019 Evaluating Research

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252321/View 15/16

SUMMARY

Research design and analyses are interdependent. How a project is designed partly determines the statistics the researcher uses to analyze the resulting data. However, lack of familiarity with statistical procedures may limit your ability to interpret a particular set of research results. Designs can be categorized in terms of their purposes: exploration, description, prediction, and explanation. Different purposes involve different designs and, therefore, different statistical analyses. Analyses for exploratory research tend to be qualitative rather than quantitative. They involve determining whether or not something has happened and are rarely complex. Analyses for descriptive research tend to be simple, inferential statistics that enable the researcher to summarize the data he or she has obtained, usually through measures of central tendency, as well as to estimate values for the population to which the researcher wishes to generalize. However, when questions about interrelatedness of variables are addressed, factor analysis may be used to describe those relationships. When describing or summarizing the results of existing research is the purpose of the study, meta-analysis is used. Predictive research analyses are all based on correlational techniques in which analyses are used to determine the extent to which one variable is related to another variable. When specific predictions are required, regression analyses can be used to construct prediction equations. Explanatory research generally involves comparing the various groups created through the manipulation of the independent variable, and appropriate analyses include any that can be used to determine whether or not the groups created can be assumed to belong to the same overall population.

EXERCISES

1. Using your library’s databases, search for and read an article containing a stem-and-leaf display. 2. Using your library’s databases, search for and read an article containing results from a multiple

regression analysis. 3. Using your library’s databases, search for and read an article containing results from an analysis of

variance (ANOVA). 4. Go to the Web site of the American Statistical Association (www.amstat.org), the Statistical Society

of Canada (www.ssc.ca), or the Royal Statistical Society (www.rss.org.uk) and explore the site to find ways to obtain additional information or contacts when you need statistical information in the future.

p. 104

p. 103

04/29/2019 - RS0000000000000000000001583532 - Evaluating Research

4/29/2019 Evaluating Research

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252321/View 16/16

04/29/2019 - RS0000000000000000000001583532 - Evaluating Research