600 MODULE 9

Nonexperimental research: strengths, weaknesses and issues of precision

Thomas G. Reio, Jr Florida International University, Miami, Florida, USA

Abstract Purpose – Nonexperimental research, defined as any kind of quantitative or qualitative research that is not an experiment, is the predominate kind of research design used in the social sciences. How to unambiguously and correctly present the results of nonexperimental research, however, remains decidedly unclear and possibly detrimental to applied disciplines such as human resource development. To clarify issues about the accurate reporting and generalization of nonexperimental research results, this paper aims to present information about the relative strength of research designs, followed by the strengths and weaknesses of nonexperimental research. Further, some possible ways to more precisely report nonexperimental findings without using causal language are explored. Next, the researcher takes the position that the results of nonexperimental research can be used cautiously, yet appropriately, for making practice recommendations. Finally, some closing thoughts about nonexperimental research and the appropriate use of causal language are presented. Design/methodology/approach – A review of the extant social science literature was consulted to inform this paper. Findings – Nonexperimental research, when reported accurately, makes a tremendous contribution because it can be used for conducting research when experimentation is not feasible or desired. It can be used also to make tentative recommendations for practice. Originality/value – This article presents useful means to more accurately report nonexperimental findings through avoiding causal language. Ways to link nonexperimental results to making practice recommendations are explored.

Keywords Research design, Experimental design, Causal inference, Nonexperimental, Social science research, Triangulation

Paper type Conceptual paper

The call for cutting-edge research to meet individual, group and societal needs around the world has never seemed more urgent. As social science researchers, this need seems particularly acute in the field of human resource development (HRD). HRD researchers and practitioners are at the cusp of fostering learning and development in diverse workplace settings that benefit not only individuals and the organization but also society and the common good (Reio, 2007). As applied social scientists, HRD professionals need to better understand how to foster learning and development optimally, as organizational support for such activities can range from being weak or nonexistent (e.g. management not valuing or implementing a formal mentoring program) to strong (e.g. pressing need for cross-cultural training for expatriate managers in an important new geographic region). These better understandings will contribute to organizational efforts to attain and sustain competitive advantage through

The current issue and full text archive of this journal is available on Emerald Insight at: www.emeraldinsight.com/2046-9012.htm

EJTD 40,8/9

676

Received 21 July 2015 Revised 24 January 2016 Accepted 28 January 2016

European Journal of Training and Development Vol. 40 No. 8/9, 2016 pp. 676-690 © Emerald Group Publishing Limited 2046-9012 DOI 10.1108/EJTD-07-2015-0058

a competent, well-trained workforce adept at handling change (Ferguson and Reio, 2010). Yet, to do so, and keep current with these pressing needs, we need the guidance of sound theory and empirical research.

Torraco (2005) and, more recently, Reio (2010a, 2010b) and Reio et al. (2015) have called for additional HRD theory-building methods articles, as well as theory generating and testing articles, to support present and future research needs in the field. Reio et al. (2015) highlighted the particular importance of accurate reporting, as erroneous analysis and subsequent reporting of findings can be highly problematic for theorists and researchers. Reio and Shuck (2015), for instance, noted how inappropriate decision practices in exploratory factor analysis (e.g. using the wrong factor extraction or factor rotation procedure) were all too common, leading to inaccurate interpretations of data when analyzed. The inaccurate data analysis interpretations, in turn, can lead to erroneous reporting that renders otherwise solid research into something that is ambiguous and incorrect. This state of affairs can indeed be detrimental, because future researchers all too often unwittingly use such unsound scholarship as the foundation of their own empirical research (Shaw et al., 2010).

Additionally, of great import is that inaccurate reporting contributes to troubling perceptions that social science research tends to be sloppy, of low quality and, therefore, lacking scientific credibility (Frazier et al., 2004; Levin, 1994). This notion of credibility has been an issue since the early part of the twentieth century, so much so that Campbell and Stanley (1963, p. 2) in one of their seminal works saw experimental research as the:

[…] only means for settling disputes regarding educational practice, as the only way of verifying educational improvements, and as the only way to establish a cumulative tradition in which improvements can be introduced without the danger of a faddish discharge of old wisdom in favor of inferior novelties.

Campbell and Stanley hailed experimental research strongly because they saw it as the only true means to establish causality. Causal inferences and thus causality can be claimed through this type of research, because of its ability through random assignment to handle internal validity issues (i.e. history, maturation, testing, instrumentation, statistical regression, selection, experimental mortality and selection-maturation interaction). Campbell and Stanley presented four threats to external validity that are handled to some lesser degree by experimental research designs as well; that is, reactive effect of testing, interaction effects of selection biases and the experimental variable, reactive effects of experimental arrangements and multiple-treatment interference. Consequently, because of its perceived precision and therefore rigor, they saw experimental research as the one best way to move the field of education and, by extension, social sciences forward.

Some researchers have argued that reporting the results of nonexperimental research should not be used for doing little beyond finding associations between variables and falsifying hypotheses, because it cannot handle internal validity or rival explanation issues sufficiently well (Bullock et al., 2010; Duncan and Gibson-Davis, 2006; Maxk, 2006; Stone-Romero and Rosopa, 2008; White and Pesner, 1983). Indeed, although not limited to the social sciences (basic science results are all-too-often interpreted and generalized beyond what is supportable with the data [Goodman, 2008; Ioannidis, 2005]), there is evidence that social science researchers can sometimes interpret nonexperimental results beyond what they were designed to support. For example,

677

Weaknesses and issues of

precision

Robinson et al. (2007) found that researchers were frequently making “causal” statements and causal conclusions in five of the top-tier teaching-and-learning research journals based on nonexperimental (nonintervention) data. In an interesting extension of Robinson et al.’s research, Shaw et al. (2010) discovered that inappropriate practice recommendations based on nonintervention research were perpetuated by educational researchers themselves (i.e. they cited nonintervention studies when making causal statements). Likewise, using the same journals as in the Robinson et al.’s study, Reinhart et al. (2013) found that the use of prescriptive statements for practice based on nonexperimental data had indeed increased over the 2000-2010 time period, despite a distinct decrease in published intervention studies. Finally, Hsieh et al. (2005) found that over a 21-year period, articles based on randomized experiments had decreased in educational psychology and education, suggesting a lack of attention to answering previous calls (Levin, 1994) for higher-quality educational intervention research. Hsieh et al. lamented that this was unfortunate in that this was the type of research that could bridge the research-to-practice gap.

HRD researchers should be alert to issues associated with the proper reporting of nonexperimental findings. Unmistakably, reporting issues associated with nonexperimental results are salient because equivocal, oblique reporting can be interpreted as being sloppy and unconvincing at best and simply wrong at worst. Imprecise reporting that includes using causal language to describe nonexperimental- generated results can be deleterious to theory building and future research (Bollen and Pearl, 2013; Cook and Campbell, 1979; Reio, 2011). As indicated by the aforementioned discussion, there is a conflicting direction in the literature as to what constitutes the best reporting practices required to guide social science researchers in correcting this situation, as it relates to proper reporting and generalizing. To address this information gap, I will examine the vexing issue associated with the precise reporting of research, such as avoiding causal statements based on nonexperimental data and making tenuous, but supportable, recommendations for practice based upon nonexperimental research designs. First, to undergird discussion in the paper, a brief introduction regarding the relative strength of research designs is presented. The strength of research design section is followed by a section presenting the strengths and weaknesses of nonexperimental research and subsequently by a section highlighting some possible ways to more precisely report nonexperimental findings without using causal language. Next, recommendations are made with regard to how to take nonexperimental results and make them explicitly useful for practice. In the final section, I present some concluding thoughts about nonexperimental research and the appropriate use of causal language.

Relative strength of research designs I must make it clear at this point that there is no such thing as a perfect research design, as each has its strengths and weaknesses (Cook and Campbell, 1979; Coursey, 1989; Howe, 2009; Maxwell, 2004; Newman et al., 2002; Schilling, 2010; Spector and Meier, 2014; Sue, 1999). Thus, in many ways, no particular research design should be in the ascendancy. Rather, the research design that matters the most is the one that will most elegantly, parsimoniously and correctly, within ethical boundaries, support answering the research questions or testing the hypotheses associated with a study (Johnson, 2009; Reis, 2000). Generally speaking, social scientists tend to stress a weakness or strength of

EJTD 40,8/9

678

a research design so much so that they can become dogmatic and narrow in their views (Bredo, 2009). For example, Sue (1999) criticizes experimental designs and their ability to make causal inferences as it relates to cross-cultural research on external validity grounds; Stone-Romero and Rosopa (2008), in contrast, strongly promote the superiority of experimental designs for making causal inferences, not acknowledging the possible external validity issue. Both are quite correct in their assertions, but they are similar to ships passing in the night; their focus is related to the utility of experimental research for their research purposes.

To move past perhaps unnecessarily narrow views, it might be useful to think of the assumptions underlying this research because we must remember that these assumptions can shape interpretation of the findings. When selecting a research method, there are three major assumptions to consider: context, causal inference and generalization (Coursey, 1989). If one selects a phenomenological study, we emphasize context over generalization and causal inference. Alternatively, if we choose a survey study, we favor generalization over the other two assumptions. Further, experimental research would emphasize on the causal inference and not generalization and context. Still, honoring context, causal inference and generalizability can be at odds within the limited confines of a study, and, therefore, the researcher must make tradeoffs. The tradeoffs, in turn, can be compensated for in a research design sense by triangulation; the use of a variety of data collection methods, data, theories or observers to provide convergent evidence and thereby bolster the validity of the findings (Maxwell, 2004). The assumption behind this is that using different, independent measurements of the same phenomenon can provide a means of counterbalancing the weaknesses of one research method with the strengths of another, especially in low “n” studies (Newman et al., 2013).

Being mindful that the decision as to which research design to use is a function of the researcher’s motivation (e.g. Do we want to generalize or make causal inferences? Does the context matter?), available resources and available population samples, the triangulation approach seems tenable because it supports the utility of trade-offs for supporting the researcher’s aims. Indeed, triangulation speaks well to mixed-method designs, but I must acknowledge that such a research design is controversial because of philosophical and methodological issues (Johnson, 2009 for a fuller exploration of this issue). Again, narrow and sometimes unbalanced views dampen the intellectual risk taking required to promote the new thinking consistent with exploring all sides of important issues (Reio, 2007). We cannot move forward in the social sciences if we are bound unrealistically and intransigently to outdated notions as to what constitutes rigorous research (Howe, 2009; Maxwell, 2004). In its place, a means to move forward in the social sciences, and thereby HRD, might be to “advocate thinking in terms of continua on multiple philosophical and methodological dimensions” (Johnson, 2009, p. 451).

Perhaps, a better way to think of research designs then is that their strengths and weaknesses are a matter of degree. Building upon Coursey’s (1989) assumptions underlying selecting a research method, I propose refining thinking about the relative strength of research designs along three continua: causality, generalization and context. Another way might be to examine internal and external validity along continua, or theory-building and theory testing, which have been explored by previous researchers (Cook and Cook, 2008; Maxk, 2006). In the present example, considering causality along

679

Weaknesses and issues of

precision

a continuum from weak to strong, looking at a set of secondary data that are not theoretically or empirically driven, would be an example of no causality evidence. No manner of statistical analysis could make this anything more than just exploring the data; hence, it is arguably almost useless and can even be detrimental. Nonexperimental methods such as case studies, phenomenologies, biographies, focus groups, interviews and surveys would be considered weak on the causality continuum, followed by quasi-experimental (less weak) and finally experimental (strong). Another continuum could be generalization ranging from experimental (weak), quasi-experimental (weak) and nonexperimental (moderate). Last, context would range from experimental (weak) and quasi-experimental (weak) to nonexperimental (moderate). Within the nonexperimental design area, quantitative and qualitative methods (e.g. surveys and focus groups) could be examined along the three continua as well. The point is knowing beforehand the relative strengths and weaknesses associated with a research design along the three continua, balanced against the researcher’s motivation, available resources and an accessible research population, could be a useful tool not only in the research planning process but also for thinking about and accurately interpreting the results with an eye toward creatively and accurately making recommendations for future research and tentatively guiding practice.

Strengths and weaknesses of nonexperimental research Nonexperimental research designs are either quantitative (e.g. surveys), qualitative (e.g. interviews) or mixed-method (e.g. case studies). Nonexperimental research can be distinguished from experimental research in that with experimental research, the researcher manipulates at least one independent variable, controls as many other theoretically relevant variables as feasible and subsequently observes the effect on one or more dependent variables (Campbell and Stanley, 1963; Shadish et al., 2002). Nonexperimental research, on the other hand, does not involve manipulation by the researcher and instead focuses on finding linkages or associations between variables. Researchers tend to see nonexperimental research as being useful at the early stages of a line of research (Cook and Cook, 2008; Johnson, 2001; Maxwell, 2004; Newman et al., 2013) such as preliminarily establishing that a hypothesized relation exists between two variables as predicted by theory (Dannels, 2010). Yet, it is not appropriate for theory validation because such research is not capable of eliminating Campbell and Stanley’s (1963, p. 5) 12 factors that “jeopardize the validity of various experimental designs”.

Nonexperimental design is prevalent in social science research as it often is not feasible or even ethical to manipulate an independent variable (e.g. workplace incivility) for the sake of one’s research (Cook and Cook, 2008; Johnson, 2001; Kerlinger, 1986; Maxk, 2006; Spector and Meier, 2014). Accordingly, because we cannot ethically manipulate independent variables such as workplace incivility, binge drinking, physical disabilities and hours worked per week, we are bound to using nonexperimental methods when researching such variables. Despite this potential limitation, a decided benefit of nonexperimental research is that it is relatively inexpensive and easy to perform, particularly survey research. Surveys are very useful also for measuring perceptions, attitudes and behaviors such that the data generated can be used for correlational analyses to establish the strength and direction of important relations that can guide future experimental study (Cook and Cook, 2008). Finding moderate to strong negative relations between workplace incivility and job

EJTD 40,8/9

680

performance, although not causal, puts forward the notion that managers’ efforts to improve job performance might be focused at least preliminarily on the workplace incivility issue.

Notwithstanding the recognized drawbacks to conducting experimental research (e.g. external validity), nonexperimental research has been much maligned in that it is perceived as not even being true research by some because it is does not well support testing for cause-and-effect relations. Conversely, the well-respected research methodologist Kerlinger (1986) noted that nonexperimental research could be thought of as being more important than experimental research because without it, we might not have even the most rudimentary understanding of links among variables that are not amenable to experimentation. He indicated that in essence that there have been more sound and significant behavioral and education studies than the sum total of experimental studies. It is not hard thinking of nonexperimental research over the years that confirmed causal relations well before the causal mechanisms had been determined. For example, doctors had knowledge about the healing properties of penicillin and aspirin for some time before they discovered why the treatments were beneficial (Gerring, 2010).

Experimental research, despite its ability to support making causal inferences, tends to be costly, time-consuming and difficult to conduct in the context of workplace settings where it can be highly challenging to find organizations willing to participate in such research (Cook and Cook, 2008). Still, the reigning argument is that experimental research supported by randomized control trials (RCTs) is the “gold standard” for social science research, because it is touted as the only true means to infer cause-and-effect relationships (Shadish et al., 2002; Stone-Romero and Rosopa, 2008). This view, however, has been challenged vigorously by researchers from quantitative, qualitative and mixed-method research traditions. Coursey (1989, p. 231) suggests the idea that “nonexperimental designs produce internally invalid results is overstated” because not all internal validity threats (e.g. history, maturation and subject selection) can be completely eliminated and there are limits to the use of random assignment (Cook and Campbell, 1979; Shadish et al., 2002). For instance, random assignment can be problematic when participants believe that they are receiving a less than desirable treatment (e.g. attend a series of lectures from management experts about how to be a motivational manager) as compared to the experimental group in a study (e.g. participate in an engaging program that integrates the latest motivation information from experts along with hands-on activities, cases and application that draw upon the best of adult learning practices), and they alter their behavior (i.e. they half-heartedly participate and, thus, do not learn to be as motivational as they could have) related to the dependent variable (being a more motivational manager).

Maxwell (2004) wrote a compelling challenge to the notion that experimental research was the only way to make causal inferences. Citing a long line of qualitative researchers, Maxwell noted the almost folly of ignoring other plausible means of getting at causality. Instead of the regularity view of causality where causal processes are ignored in favor of causal effects (introduction of x caused y; this is the dominant view of causality), Maxwell advocated a reality approach where causality refers to the actual causal mechanisms and processes involved in events and situations. Qualitative research is uniquely adept at handling the study of causal mechanisms and processes. Thus, taking a reality approach might be an appropriate means to augment

681

Weaknesses and issues of

precision

experimental research seeking causal effects or vice versa. Neither approach, therefore, is superior; both together herald the use of mixed-method approaches to get at causal effects and causal processes in social science research.

The overall contributions of nonexperimental research have been certainly profound and will continue to be as long as we need as social scientists to push the theoretical, conceptual, empirical and practical boundaries of our respective fields. As HRD researchers, we need to know that plausible associations among variables exist through nonexperimental research before we can design experimental studies that would support making causal inferences that one variable (e.g. curiosity) has an effect on another (e.g. learning) after an intervention or manipulation by the researcher.

The importance of precisely reporting nonexperimental findings No matter the type of research, the findings must be reported unambiguously and accurately to allow researchers to understand what was found and afford them the opportunity to falsify the results (Fraenkel and Wallen, 2009; Grigorenko, 2000; Reis, 2000). In the case of nonexperimental research, erroneously utilizing imprecise and mistaken language to present their findings (e.g. causal language; Bullock et al., 2010; Levin, 1994; Maxk, 2006; Reinhart et al., 2013) must be assiduously avoided because the imprecise research, when published, becomes part of the scientific literature and serves as a stepping stone for theory building and research related to the topic area (Reio et al., 2015; Spector and Meier, 2014; Stone-Romero and Rosopa, 2008). To be sure, theory building and research can only proceed with accurately reported empirical studies to build upon. Meta-analyses, for example, are used to support theory building (Callahan and Reio, 2006; Sheskin, 2000). Because meta-analyses allow researchers to support a conclusion about the validity of a hypothesis based upon more than a few studies, they may be subject to misinterpretation when using nonexperimental findings that have been erroneously reported. Thus, meta-analytic results can be only as good as the studies comprising it; that is, they must be accurate and precise (Newman et al., 2002) and not go beyond the data to make untenable, causal claims (Robinson et al., 2007).

As social scientists, we are taught to honor the scientific method and remain skeptical, but not inflexible consumers of research (Bredo, 2009; Howe, 2009; Kerlinger, 1986; Maxk, 2006; Maxwell, 2004, 2013). As part of this teaching, we are introduced to the literature of our respective fields and a number of research methods courses that permit one to not only understand the strengths and weaknesses of each data collection method but also support the critique of published research based on the relative strength of the research design (Reis, 2000). Part of one’s methods exposure is learning things such as “correlation does not imply causation”, “experimental research is the sole means available to researchers to support causal inferences” and “qualitative research is not generalizable”. (Shadish et al., 2002). Thus, by the end of one’s graduate training, one should be well versed in the caveats of conducting research. Unfortunately, often what does not occur is the researcher has been schooled sufficiently in how to avoid making inadvertent, nuanced interpretation and writing errors (that includes using causal language) that feed perceptions of sloppy, unscientific social science research (Sternberg, 2000). This regrettable situation can carry through into one’s academic career until a colleague, reviewer or editor apprises the researcher to the problem. Therefore, it is incumbent upon us as colleagues, mentors, reviewers and editors to do our best to assist moving the field along through helping researchers polish and perfect

EJTD 40,8/9

682

their interpretation and reporting practices (Reio, 2011; Sternberg, 2000). This paper, for example, is one such attempt to augment researchers’ knowledge about the issue of imprecise and mistaken reporting of nonexperimental research and the associated pitfalls.

As such, academic writing is clearly a craft and should be treated that way as in any other genre of writing (Sanders-Reio et al., 2014). Dissertation writing, for instance, is often the first entry into the world of scholarly writing and, not surprisingly, the issues of inaccurate or inappropriate interpretation and reporting are all too real (Clark, 2004). Similarly, when reviewing articles for research journals, the same issue appears, but fortunately somewhat less so (Beins and Beins, 2012; Sternberg, 2000). As Robinson et al. (2007); Reinhart et al. (2013) and Shaw et al. (2010) have found, the imprecise and incorrect writing issue has not disappeared satisfactorily even when published in top-tier teaching-and-learning research journals.

Someone might reasonably ask “Why do these writing errors persist, even in the best social science journals?” First, I do not think it is the case, as some seem to suggest (Stone-Romero and Rosopa, 2008) that authors are trying to make their research appear more important; social science methods training simply does not permit this style of erroneous thinking. More plausibly, what may be missing is that the authors do not realize that they are making such errors in the first place (Sternberg, 2000). For example, in a hypothetical organizational study, a researcher found experiencing workplace incivility correlated negatively with organizational commitment and job satisfaction. When interpreting the results, the researcher is speaking too strongly when he states:

There is a strong negative relationship between experiencing workplace incivility and job performance. This means that the more one experiences workplace incivility, the more likely they will perform poorly on their jobs.

Using the word “means” in this context has causal implications. The research write-up can be polished and made more appropriate by toning down the sentence to read:

Understanding that these results are correlational, preliminary, and that no causality can be implied, the strong association between the variables suggests more research is warranted that would tease out why this might be so.

As an another example based on a phenomenological study of the meaning of experiencing workplace incivility to a group of participants in one organization, if a researcher claimed:

The results of the interviews suggested that experiencing workplace incivility was demeaning, counter-productive, and detrimental to performing well. Organizations, therefore, should find more ways to reduce the likelihood of experiencing incivility if they ever hope to improve job performance.

In this case, the researcher is going beyond the data to generalize to other organizations, and the sentence could be toned down to read:

It seems to be the case that experiencing workplace incivility was a salient issue in this organization for the majority of participants in that they noted that incivility tended to dampen their performance. Future qualitative research might be designed where researchers could look into this issue at other types of organizations to understand how and why job performance suffers when experiencing workplace incivility.

683

Weaknesses and issues of

precision

As another example, in a nonexperimental study where path-analytic procedures were used to test a theoretical model (all observed variables) where negative affect was associated with workplace incivility and then, in turn, job performance, the researcher found a strong, statistically significant negative path coefficient on the incivility to performance path in the model. The researcher reported that “The strong negative path coefficient on the incivility to performance path implies that workplace incivility influences job performance”. The terms “implies” and “influences” are causal terms and, despite the understanding that path models are a form of “causal model” (Bollen and Pearl, 2013, for a fuller exploration of “causal modeling”), it suggests inappropriately by virtue of the language used that incivility in the context of this study had a causal link to performance. Another way to present the results of the path analysis could be to state:

The strong negative path coefficient on the incivility to performance path is a preliminary indication that incivility and performance are strongly associated. Although this model could not be tested experimentally, it could be replicated in a wide variety of workplace contexts to test whether the strength and direction of the relationships represented by this model would be consistent, thereby supporting the model.

Consequently, researchers need to be acutely aware of causal terms such as “effect”, “affect”, “influence”, “imply”, “suggest”, “infer” and “indicate” to avoid biasing reporting of their nonexperimentally based findings. Using terms such as “may”, “perhaps” and the like are more tentative and correct.

I want to caution the reader about the “causal” terms emerging from structural equation modeling (SEM) research (Bollen and Pearl, 2013). Interestingly, Dannels (2010) noted that because SEM as a statistical modeling technique is based on a priori theory and stochastic assumptions, causality claims may be warranted in some cases because the statistical model is in a sense a depiction of the theory being tested. That is, when variable X significantly predicts variable Y as predicted by the theoretical model being tested, the relationship between the two variables could be interpreted as being causal. Thus, variable X is said to have influenced or caused variable Y. Although this “causal modeling” interpretation of what constitutes causality may seem justifiable, it simply cannot be causal in the truest sense of the term with nonexperimental data because of the lack of manipulation of at least one independent variable by the researcher (Bollen and Pearl, 2013). Therefore, causal claims should be avoided even with SEM based on nonexperimental data, despite its statistical sophistication.

Another term to avoid is “causal-comparative research” because it is an outdated and confusing term that some suggest implies cause-and-effect, which simply is not so because it is little more than another form of correlational research (Johnson, 2001). At the very least, when interpreting and writing up nonexperimental quantitative results, the researcher could always use the caveat “With the understanding that these data are correlational and thus no causality can be implied […]”. The researcher must make it clear to the reader that they are being appropriately cautious by not going beyond their data when discussing their work. Similarly, for qualitative findings, the researcher could use the caveat “As this was qualitative research, generalizing the results beyond the study would not be appropriate”.

In this section, I explored the need for precision in the social sciences when interpreting and reporting findings. The use of causal language with nonexperimental research has been troubling because it continues unabated, despite being incorrect and

EJTD 40,8/9

684

misleading. The use of terms such as “causal modeling” when using SEM and “causal-comparative” research was highlighted as being noteworthy because of the confusion they engender. Instead, eschewing terms such as “influence” and “implies” would be positive first steps in staying away from causal language in nonexperimental research.

As the next step in our exploration of using and reporting nonexperimental findings appropriately, in the next section, I take the position that nonexperimental research designs can be used to support making tentative practice recommendations.

Nonexperimental results can be used tentatively for making practice recommendations In applied social science research such as HRD, there is quite a bit of emphasis on taking research and making some sense of its practical implications. If we find evidence, for instance, after testing a hypothesized model where a combination of select demographic variables (age, gender), negative affect and workplace incivility predict job performance, the natural question is “So what?” This question can be handled pretty readily in terms of talking about its possible theoretical and empirical implications, but unless we are using experimental research-generated data, there should be some caution against using it for making practice recommendations (Cook and Cook, 2008). Reinhart et al. (2013) on finding that increasingly nonintervention research was being used in place of experimental research in the educational psychology field, strongly recommended that such data be used strictly for disconfirming and not confirming hypotheses or stated differently, falsifying rather than validating theory. The researchers also stressed that prescriptive statements for practice should be only acceptable based on experimental evidence. Likewise, they specified that recommendations for practice should be based on “evidence-based” practice, which essentially is RCT, experimental research. This in itself seems terribly enigmatic considering that less than 5 per cent of published educational research is experimental (Reinhart et al., 2013). Thus, by extension, we would have very little research on which to base our practice. Undoubtedly, there must be a more productive middle ground if we are to make any meaningful contributions to improving practice.

Acknowledging that RCT experimental research offers the best opportunity for eliminating most sources of bias, we must recall that RCTs merely inform us whether the intervention worked, but not why. Notably, we must also bear in mind that most social science does not lend itself to such designs; they are simply infeasible (Cook and Campbell, 1979; Johnson, 2009; Maxk, 2006; Spector and Meier, 2014). Even Campbell and Stanley (1963, p. 3) strongly advocated that experiments must be replicated and cross-validated at “other times and other conditions before they can become an established part of science”. Thus, by merely conducting one or two experimental studies, it is not sufficient based on the results to make recommendations for either research or practice, unless it is done tentatively.

If using a nonexperimental design, we must accept that we will not be able to eliminate all possible rival explanations (Campbell and Stanley, 1963). However, through our research designs, we can eliminate within reason the most pressing

685

Weaknesses and issues of

precision

theoretical ones. Again, mixed-method designs appear best suited for handling this kind of dilemma because it relies on a number of research methods to answer research questions (Johnson, 2009; Newman et al., 2013). It seems less than productive and almost implausible to dismiss all qualitative research and the vast majority of nonexperimental research for making recommendations to improve practice.

I suggest instead that nonexperimental findings should be used tentatively, rather than definitively, with the clear understanding that whatever the recommendations being made for practice happen to be, that they be preliminary, unambiguous and precise, without the use of causal language. To help move the field forward, practice recommendations should be balanced with recommendations for further research that might more strongly support this practice recommendation. For instance, if an organizational researcher found a moderate to strong positive association between curiosity and training classroom learning, as predicted by well-validated theory and empirical research (Reio and Wiswell, 2000), it would preliminarily suggest that promoting curiosity may be something worthy of consideration in training classrooms. This preliminary finding would support designing experimental research where a curiosity-inducing intervention would be introduced and tested. Another example might be after finding through focus group research that mid-level managers at a large organization did not feel they had the requisite cross-cultural skills to manage supervisees in a foreign country, which is consistent with prior expatriate manager research (Selmer, 2000); one would at least have a sense that additional cross-cultural training might be needed, despite the lack of experimental evidence. Future research could be designed to examine theoretically relevant variables that might shed light on the variables that mattered the most when considering expatriate manager performance.

Accordingly, despite possessing nothing more than nonexperimental data in the two examples, I demonstrated how it would be possible to use such data to tentatively and correctly guide practice. Narrow views of what constitutes acceptable research to support practice recommendations seem unnecessary. Although well intentioned, these narrow views tend to be unproductive in the sense that enriching, interesting and path-breaking nonexperimental research might be overlooked for the sake of the overzealous pursuit of “rigor” (Maxwell, 2004). The gold standard of RCT research is not possible or should not even be desired in every single case because of its inherent flaws as a research design (e.g. generalization).

Conclusions The article was predicated upon the notion that there is a research gap or direction about best reporting practices for social science research such as HRD. Through discussing the relative strength of research designs, highlighting the strengths and weaknesses of nonexperimental designs, presenting how we might improve the precision of nonexperimental research reporting and generalizing and demonstrating how nonexperimental research can be used cautiously and preliminarily to inform practice, I attempted to close the gap in the literature as to what constitutes best reporting practices and why.

I conclude that we cannot continue using causal language in nonexperimental social science research, and this must be taken seriously as rigorous researchers. On the other

EJTD 40,8/9

686

hand, we cannot afford to ignore the vast array of social science research simply because it is not experimental. Because the large majority of published social science research is nonexperimental, it is just too limited to suggest that such research has no scientific merit with regards to making research or practice recommendations. As Kerlinger (1986) noted some time ago, we must not overlook the importance of nonexperimental research because it is the foundation of the social science research we do. Experimental research would be rudderless without being provided possible clues for promising variables generated by nonexperimental research. Nonexperimental research would be useless if pointlessly ignored because of its inherent design weaknesses. Remembering that all research designs have limitations, if used appropriately, tentatively and correctly, the research it generates could be useful for building theory, guiding research and informing practice.

References Beins, B.C. and Beins, A.M. (2012), Effective Writing in Psychology: Papers, Posters and

Presentations, 2nd ed., Wiley-Blackwell, Malden, MA.

Bollen, K.A. and Pearl, J. (2013), “Eight myths about causality and structural equation models”, in Morgan, S.L. (Ed.), Handbook of Causal Analysis for Social Research, Springer, New York, NY, pp. 301-328.

Bredo, E. (2009), “Getting over the methodology wars”, Educational Researcher, Vol. 38 No. 6, pp. 441-448.

Bullock, J.G., Green, D.P. and Ha, S.E. (2010), “Yes, but what’s the mechanism (Don’t expect an easy answer)”, Journal of Personality and Social Psychology, Vol. 98 No. 9, pp. 550-558.

Callahan, J. and Reio, T.G., Jr (2006), “Making subjective judgments in quantitative studies: the importance of effect size indices and confidence intervals”, Human Resource Development Quarterly, Vol. 17, pp. 159-173.

Campbell, D.T. and Stanley, J.C. (1963), Experimental and Quasi-Experimental Designs for Research, Rand McNally College Publishing, Chicago, IL.

Clark, I.L. (2004), Writing the Successful Thesis and Dissertation, Prentice Hall, Upper Saddle River, NJ.

Cook, B.G. and Cook, L. (2008), “Nonexperimental quantitative research and its role in guiding instruction”, Intervention in School and Clinic, Vol. 44 No. 2, pp. 98-104.

Cook, T. and Campbell, D.T. (1979), Quasi-Experimentation: Design and Analysis Issues for Field Settings, Houghton Mifflin, Boston, MA.

Coursey, D.H. (1989), “Using experiments in knowledge utilization research: strengths, weaknesses and potential applications”, Knowledge: Creation, Diffusion, Utilization, Vol. 10 No. 1, pp. 224-238.

Dannels, S.A. (2010), “Research design”, in Hancock, G.R. and Mueller, R.O. (Eds), The Reviewer’s Guide to Quantitative Methods in the Social Sciences, Routledge, New York, NY, pp. 343-355.

Duncan, G.J. and Gibson-Davis, C.M. (2006), “Connecting child care quality to child outcomes”, Evaluation Review, Vol. 30 No. 3, pp. 611-630.

Ferguson, K. and Reio, T.G., Jr (2010), “Human resource management systems and firm performance”, Journal of Management Development, Vol. 29 No. 5, pp. 471-494.

687

Weaknesses and issues of

precision

Fraenkel, J.R. and Wallen, N.E. (2009), How to Design and Evaluate Research in Education, 7th ed., McGraw-Hill, New York, NY.

Frazier, P.A., Tix, A.P. and Barron, K.E. (2004), “Testing moderator and mediator effects in counseling psychology research”, Journal of Counseling Psychology, Vol. 51 No. 1, pp. 115-134.

Gerring, J. (2010), “Causal mechanisms: yes, but […]”, Comparative Political Studies, Vol. 43 No. 11, pp. 1499-1526.

Goodman, S.N. (2008), “A dirty dozen: twelve P-value misconceptions”, Seminars in Hematology Vol. 45 No. 3, pp. 135-140.

Grigorenko, E.L. (2000), “Doing data analyses and writing up their results”, in Sternberg, R.J. (Ed.), Guide to Publishing in Psychology Journals, Cambridge University Press, New York, NY, pp. 98-120.

Howe, K.R. (2009), “Epistemology, methodology, and education sciences: positivist dogmas, rhetoric, and the education science question”, Educational Researcher, Vol. 38 No. 6, pp. 428-440.

Hsieh, P., Acee, T., Chung, W., Hsieh, Y., Kim, H., Thomas, G.D., You, J., Levin, J.R. and Robinson, D.H. (2005), “Is educational research on the decline?”, Journal of Educational Psychology, Vol. 97 No. 4, pp. 523-529, doi: 10.1037/0022-0663.97.4.523.

Ioannidis, J.P.A. (2005), “Why most published research findings are false”, PLoS Medicine, Vol. 2 No. 8, pp. 0101-0106.

Johnson, B. (2001), “Toward a new classification of nonexperimental research”, Educational Researcher, Vol. 30 No. 2, pp. 3-13.

Johnson, R.B. (2009), “Toward a more inclusive ‘Scientific research in education”, Educational Researcher, Vol. 38, pp. 449-457.

Kerlinger, F.N. (1986), Foundations of Behavioral Research, 3rd ed., Holt, Rinehart & Winston, New York, NY.

Levin, J.R. (1994), “Crafting educational intervention research that’s both credible and creditable”, Educational Psychology Review, Vol. 6 No. 3, pp. 231-243.

Maxk, B.A. (2006), “Methodological issues in nurse staffing research”, Western Journal of Nursing Research, Vol. 28 No. 6, pp. 694-709.

Maxwell, J.A. (2004), “Causal explanation, qualitative research, and scientific inquiry in education”, Educational Researcher, Vol. 33 No. 2, pp. 3-11.

Maxwell, J.A. (2013), Qualitative Research Design, 3rd ed., Sage, Los Angeles, CA.

Newman, I., Fraas, J.W., Newman, C. and Brown, R. (2002), “Research practices that produce Type VI errors”, Journal of Research in Education, Vol. 12 No. 1, pp. 138-145.

Newman, I., Ridenour, C.S., Newman, C. and Smith, S. (2013), “Detecting low-incidence effects: the value of mixed method research designs in low-n studies”, Mid-Western Educational Researcher, Vol. 25 No. 4, pp. 31-46.

Reinhart, A.L., Haring, S.H., Levin, J.R., Patall, E.A. and Robinson, D.H. (2013), “Models of not-so-good behavior: yet another way to squeeze causality and recommendations for practice out of correlational data”, Journal of Educational Psychology, Vol. 105 No. 1, pp. 241-247.

Reio, T.G., Jr (2007), “Exploring the links between adult education and human resource development: learning, risk-taking, and democratic discourse”, New Horizons in Adult Education and Human Resource Development, Vol. 21 Nos 1/2, pp. 5-12.

EJTD 40,8/9

688

Reio, T.G., Jr (2010a), “The ongoing quest for theory-building research methods articles [Editorial]”, Human Resource Development Review, Vol. 9 No. 3, pp. 223-225.

Reio, T.G., Jr (2010b), “The threat of common method variance bias in theory building”, Human Resource Development Review, Vol. 9 No. 4, pp. 405-411.

Reio, T.G., Jr (2011), “Toward expert publishing practice [Editorial]”, Human Resource Development Review, Vol. 10 No. 1, pp. 119-122.

Reio, T.G., Jr and Shuck, B. (2015), “Exploratory factor analysis: implications for theory, research, and practice”, Advances in Developing Human Resources, Vol. 17, pp. 12-25.

Reio, T.G., Jr and Wiswell, A. (2000), “Field investigation of the relationship between adult curiosity, workplace learning, and job performance”, Human Resource Development Quarterly, Vol. 11 No. 1, pp. 1-36.

Reio, T.G., Jr, Nimon, K. and Shuck, B. (2015), “Preface: quantitative data-analytic techniques to advance HRD theory and practice”, Advances in Developing Human Resources, Vol. 17 No. 1, pp. 3-11.

Reis, H.T. (2000), “Writing effectively about design”, in Sternberg, R.J. (Ed.), Guide to Publishing in Psychology Journals, Cambridge University Press, New York, NY, pp. 98-120.

Robinson, D.H., Levin, J.R., Thomas, G.D., Pituch, K.A. and Vaughan, S. (2007), “The incidence of ‘causal’ statements in teaching-and-learning research journals”, American Educational Research Journal, Vol. 44 No. 2, pp. 400-413.

Sanders-Reio, J., Alexander, P.A., Reio, T.G., Jr and Newman, I. (2014), “Do students’ beliefs about writing relate to their writing self-efficacy, apprehension, and writing performance?”, Learning and Instruction, Vol. 33 No. 2, pp. 1-11.

Schilling, R. (2010), “Commentary: the challenge of nonexperimental interventions studies in social work”, Research on Social Work Practice, Vol. 20 No. 5, pp. 550-552.

Selmer, J. (2000), “A quantitative needs assessment technique for cross-cultural work adjustment training”, Human Resource Development Quarterly, Vol. 11 No. 3, pp. 269-281.

Shadish, W.R., Cook, T.D. and Campbell, D.T. (2002), Experimental and Quasi- Experimental Designs for Generalized Causal Inference, 2nd ed., Houghton Mifflin, New York, NY.

Shaw, S.M., Walls, S.M., Dacy, S.M., Levin, J.R. and Robinson, D.H. (2010), “A follow-up note on prescriptive statements in nonintervention research studies”, Journal of Educational Psychology, Vol. 102 No. 4, pp. 982-988.

Sheskin, D.J. (2000), Handbook of Parametric and Nonparametric Statistical Procedures, 2nd ed., Chapman & Hall/CRC, New York, NY.

Spector, P.E. and Meier, L.L. (2014), “Methodologies for the study of organizational behavior processes: how to find your keys in the dark”, Journal of Organizational Behavior, Vol. 35 No. 8, pp. 1109-1119.

Sternberg, R.J. (2000), Guide to Publishing in Psychology Journals, Cambridge University Press, New York, NY.

Stone-Romero, E.F. and Rosopa, P.J. (2008), “The relative validity of inferences about mediation as a function of research design characteristics”, Organizational Research Methods, Vol. 11 No. 2, pp. 326-352.

Sue, S. (1999), “Science, ethnicity, and bias: where have we gone wrong?”, American Psychologist, Vol. 54 No. 12, pp. 1070-1077.

689

Weaknesses and issues of

precision

Torraco, R.J. (2005), “Theory development and research methods”, in Swanson, R.A. and Holton, E.F., III (Eds), Research in Organizations: Foundations and Methods of Inquiry, Berrett-Koehler, San Francisco, CA, pp. 351-374.

White, D.R. and Pesner, R. (1983), “Internal replications, the systems concept, and sources of validity in nonexperimental research”, Behavioral Science Research, Vol. 18 No. 1, pp. 26-44.

Further reading Nimon, K., Reio, T.G., Jr and Shuck, B. (2015), “Quantitative data-analytic techniques to advance

HRD theory and practice”, Advances in Developing Human Resources, Vol. 17 No. 1, pp. 1-134.

Corresponding author Thomas G. Reio, Jr can be contacted at: [email protected]

For instructions on how to order reprints of this article, please visit our website: www.emeraldgrouppublishing.com/licensing/reprints.htm Or contact us for further details: [email protected]

EJTD 40,8/9

690

Reproduced with permission of copyright owner. Further reproduction prohibited without permission.