Trends in Evaluation Vocabulary

2

Trends in Evaluation Vocabulary

Redacted

1

Trends in Evaluation Vocabulary

Wheeler, P., Haertel, G. D., & Scriven, M. (1992). Teacher evaluation glossary. Retrieved from http://files.eric.ed.gov/fulltext/ED363977.pdf

Wheeler et al. provide an extensive list of evaluation terms for education professionals, and evaluation language in this work is thoroughly defined. Here, evaluation is defined as a systematic process of determining something’s worth. However, terms like inputs, outputs, and outcomes not provided or defined like they are in newer evaluation glossaries. Validity is described by Wheeler et al. in the same fashion that other organizations define the term. The authors further incorporate the various types of validity typically found in scientific research. Wheeler et al. include verbiage employed in the fields of both evaluation and scientific research. The taxonomy in this work is quite broad, and it provides a good starting point for insight into how earlier evaluation vocabularies might have influenced later or current terminology.

Rockwell, K., & Bennett, C. (2004). Targeting outcomes of programs: A hierarchy for targeting outcomes and evaluating their achievement. Faculty Publications: Agricultural Leadership, Education & Communication Department, 48. Retrieved from http://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1047&context=aglecfacpub

Rockwell and Bennett offer specifically that evaluation is “part of all stages of programming, not just a follow-up to program implementation.” Their characterization of evaluations points to the significance of evaluations timelines suggesting that evaluation is an ongoing process. The authors introduce various types of evaluations: process, implementation, program, outcome, and impact. Outcomes are defined in two ways: as benefits and changes that occur within a program and as behavior changes over time. No other source cited herein provides more than one meaning for the term outcomes. Further, Rockwell and Bennett stipulate that inputs are comprised of resources dedicated to or consumed by programs. Rockwell and Bennett seem to offer inclusive and wide-ranging terminology in that terms appear to combine organizational, research, and evaluation into meanings applicable and workable descriptions characterized for their specific use. However, although the authors’ terminology is collective, the definitions do not conflict with meanings of the same terms elsewhere.

Rabin, B. A., Brownson, R. C., Haire-Joshu, D., Kreuter, M. W., & Weaver, N. L. (2008). A glossary for dissemination and implementation research in health. Journal of Public Health Management & Practice, 14(2), 117-123. Retrieved using the Walden library databases.

Rabin et al. provide explanations of terms frequently cited in research dissemination and application documents and major federal funding agency announcements. Interesting is that the authors differentiate between efficacy and effectiveness research. Efficacy research is defined as the evaluation of initial intervention impacts while effectiveness research is defined as what determines the intervention impact when delivered under real-world circumstances. Rabin et al. also describe measurement and evaluation the same so the terms can be employed interchangeably. Logic models are described in this work, but the term capacity building is used to define logic model activities compared to the term activities or inputs. Capacity building denotes, in this work, any activity that “builds durable resources.” Rabin et al.’s verbiage seems geared specifically toward their purpose, and some defined terms vary in their meaning when compared to other sources herein.

Place-Based Education Evaluation Collaborative. (2012). Glossary of evaluation terminology. Retrieved from http://www.peecworks.org/peec/peec_inst/i01795f61

This source offers an extensive, lengthy list of terminology including both research and evaluation expressions, and terms listed in this work are not provided elsewhere. For instance, goal-free evaluations are described as those that function without any knowledge of purpose or goals. More elements of evaluation are implied by the terminology offered in this list, such as pitfalls, jury trial for projects, and nonreactive measures. Language in this glossary appears to include some basic definitions; ones with which evaluation experts might already be familiar.

Center for Disease Control and Prevention (CDC). (2012). Glossary. Retrieved from https://www.cdc.gov/eval/guide/glossary/

The CDC offers an excellent listing of evaluation terms. In this work, evaluation design, plan, and strategy are defined specifically and explicitly. Various types of evaluations are also defined including process, program, and outcome. Stakeholders and logic model are mentioned and defined by CDC (these words do not appear in any earlier glossaries provided in this paper). CDC incorporates both research and evaluation nomenclature in their glossary, but neither research nor evaluation is defined. Introduced in CDC’s glossary are fresher terms that include different types of program analyses (i.e., ex-ante, ex-post, inferential statistical) not mentioned elsewhere.

Nevada State College. (2016). Glossary of assessment and evaluation terms. Retrieved from https://nsc.edu/provost/initiatives/outcomes-assessment/glossary-of-assessment-and-evaluation-terms.aspx

Nevada State College’s glossary is similar to CDC’s in that it offers definitions of various kinds of evaluations. Unusual in this work is the explanation of the term effectiveness; the word has not been exclusively defined in any earlier sources used. Effectiveness denotes a project’s worth in reaching either formative or summative evaluation objectives, and the word is used interchangeably with success. What is interesting is that evaluation is not, in itself, defined. The terminology used in this glossary is inclusive of scientific research and program evaluation, and it further includes some simple terms like measurement and objective. Meta-evaluation is introduced in this glossary as an evaluation of an evaluation, and Nevada’s glossary is the first of the sources herein to define the term.

University of California Berkeley. (2016). Evaluation terminology tip sheet. Retrieved from http://socrates.berkeley.edu/~pbd/pdfs/Evaluation_Terminology.pdf

This source presents some new terminology, but it also includes similar language to previous references. For instance, continuous quality improvement is used as another term for formative evaluation. Summative evaluation is not used or defined in this glossary; this seems odd as most other sources include it in conjunction with formative. This work, unlike those referenced previously, explains impact evaluations. Ranges of outcome measure are also described and include intermediate and immediate outcomes, but long-term outcomes are not mentioned or defined. Logic model is also defined and provides a more in-depth distinction as to their purpose and use as compared to the definition offered by CDC. Stakeholders are defined in this source’s explanation of participatory evaluation, which assumes the definition of the process of “engaging stakeholders in an evaluation effort.”

Organisation for Economic Co-operation and Development. (2016). Glossary of key terms in evaluation and results based management. Retrieved from http://www.oecd.org/dac/evaluation/

This source provides enhanced and extensive meanings of terms used in evaluation vocabulary. In addition to common evaluation types, other various kinds of evaluations are presented, including cluster, country program, ex-ante, ex-post, independent, joint, meta-, self-, thematic, and utilization-focused evaluations. This organization’s glossary not only provides definitions of frequently used terms in the process of evaluation, but it goes further in detail than other references. Evaluation, however, is defined as “the systematic or objective assessment of an on-going or completed project…” program, or policy. Differing from other explanations in this paper, this source profoundly describes evaluation and what it entails. It is interesting to note the similarities of defined evaluation terms in this glossary as similar to other sources (this glossary is also provided in the Swedish language). Few differences exist between the general overview of evaluation in this resource and other references; all references seem to adhere to comparable standards of evaluation.

American Public University System (APUS). (2016). Glossary of assessment terms. Retrieved from http://www.apus.edu/community-scholars/learning-outcomes-assessment/university-assessment/glossary.htm

Evaluation is defined in this glossary as “the use of assessment findings to judge program effectiveness” as a foundation for decision-making about program changes or augmentations. Like other sources, APUS offers standard evaluation types that include formative, summative, and the like. But, what APUS provides in its term explanations also includes a particular taxonomy of cognitive objectives for education professionals. Specifically, Bloom’s 1956 Taxonomy of Cognitive Objectives is provided in this source to illustrate six levels of increasing difficulty which range from knowledge to evaluation.

Colorado State University Writing Center. (2017). Glossary of key terms. Retrieved from https://writing.colostate.edu/guides/guide.cfm?guideid=90

Terms used in this glossary are reported to be from the 1990s and 2000s. Here, particular two terms, discourse community and empathic neutrality, are mentioned but are not found elsewhere in this paper’s references. The language of Colorado State’s glossary seems to be more scientific research oriented, as most of the verbiage consists of commonly used terms in either quantitative or qualitative examination. Other terms are introduced, including rhetorical inquiry, quixotic reliability, and verisimilitude. This extensive list of terms seems updated and incorporates various terminology that might be used in many different kinds of evaluations.

U.S. Government Accountability Office (GOA). (n.d.). Business Process Reengineering (BPR) glossary of terms. Retrieved from http://www.gao.gov/special.pubs/bprag/bprgloss.htm

This source offers only a limited list of terms with succinct definitions. The explanations are also vaguer than other those provided by other glossaries, and the language used by GOA is very organization-focused. The term inputs is listed and defined, but the term outputs is omitted. Additionally, outcomes are defined only as ultimate or long-term. GOA does, however, introduce new language by providing definitions of terms like risk sensitivity and root-cause analysis; terms not offered in other sources. It is possible that, although the website has no date, the glossary indicates that it was revised in 1998. It is unknown whether GOA still utilizes this glossary 20 years later.

Substance Abuse and Mental Health Services Administration (SAMHSA). (n.d.). SAMHSA’s National Registry of Evidence-Based Programs and Practices glossary. Retrieved from http://nrepp.samhsa.gov/05f_glossary.aspx

SAMHSA’s glossary provides a broad and relevant list of terms to those in the fields of mental health and substance abuse. Further, this source presents a good overview of various aspects of evaluation. Both scientific research and program evaluation/assessment terminology are offered, and updated references are given so for each defined term. What is different about SAMHSA’s glossary, though, is its attention to all aspects of evaluation. Rigor, comparative effectiveness research, and cultural competence are amalgamated in their evaluation terminology to include processes that go beyond typical assessment verbiage. Rossi’s name appears several times in SAMHSA’s glossary as a reference for terminology used.

Davidson Institute. (n.d.). A glossary of terms used in educational assessment. Retrieved from http://www.davidsongifted.org/Search-Database/entry/A10461

The glossary provided by the Davidson Institute is different from others in that assessment, evaluation, and measurement are defined separately and differently from one another. For instance, assessment is described as the process of “taking stock,” evaluation is defined as the “representation or determination of a value,” and measurement is explained, simply, as the “process of assigning a number to something.” Other references in this paper use the terms evaluation, assessment, and measurement interchangeably. This glossary, too, provides definitions of commonly used terms in evaluation such as formative (but not summative) and validity and reliability. It is not, however, an advanced or extensive catalogue of terminology. Although this is a glossary used particularly for educational evaluations, it seems odd that some terms would be omitted (such as scientific research verbiage).

U.S. Department of Energy, Office of Energy Efficiency & Renewable Energy (DOE). (n.d.). Program evaluation: Glossary. Retrieved from https://energy.gov/eere/analysis/program-evaluation-glossary

The DOE’s glossary is geared explicitly toward federal evaluations. The DOE states that the difference between their vocabulary and the private sector’s is that the private sector has not yet espoused the same distinctions between outcome and impact used by the DOE and “logic modelers.” It is significant that this difference is noted; no other glossary listed herein proposes how their evaluation lexicon might employ different terms for similar meanings. However, similar to the references previously listed in this paper, the DOE’s glossary seems to use language that is common in both scientific research and program evaluations. Long-term outcomes, according to the DOE, denote a program’s “impacts,” and all of a program’s outcomes make up its “outcome structure.” I found the latter term to be unusual in that “outcome structure” is not used in any other reference herein (although it seems like a quite a reasonable term!).

Stephen F. Austin State University (SFASU). (n.d.). Glossary of assessment terms. Retrieved from http://www.sfasu.edu/iao/pdf/GlossaryofAssessmentTerms.pdf

Distinct from the college glossaries listed previously, this university’s presents the terms mission statement and curriculum mapping. Commonly used expressions like goals, reliability, and benchmark are included in this lexicon. Also incorporated into this glossary is the familiar terminology of both scientific research and program evaluation. SFASU’s list is not large, but it does consist of conventional and central components of evaluation.

Almost 25 years ago, Nielsen and Turner (1983) suggested that evaluations should be ongoing processes, further highlighting the need for multiple and various approaches to evaluations and conformity of evaluation methods to particular situations (p. 398). Nielsen and Turner’s (1983) description of evaluation is analogous to only one glossary in this paper: Rockwell and Bennett’s (2004). No other sources herein overtly concede that evaluation is an ongoing, continual process, with the exception of the Organisation for Economic Co-operation and Development (2016) who state that evaluations are utilized for either ongoing or completed programs (emphasis added). Thought-provoking is that some glossaries used in this paper do not even define evaluation (i.e., CDC, 2012; Nevada State College, 2016), and some propose numerous types of evaluations (CDC, 2012; Organisation for Economic Co-operation and Development, 2016; Place-Based Education Evaluation Collaborative, 2012; Rockwell & Bennett, 2004; University of California Berkeley, 2016).

In the early 1990s, the meaning of evaluation included “…judging the worth or merit of something or the product of the process” (as cited in Hogan, 2007, p. 3). Hogan’s (2007) meaning of evaluation is similar to some of the glossary definitions listed herein, such as Davidson Institute’s (n.d), Organisation for Economic Co-operation and Development’s (2016), and Wheeler, Haertel, and Scriven’s (1992). Around the year 2000, the definition of evaluation had been modified to “… a systematic process used to determine the merit or worth of a specific program, curriculum, or strategy in a specific context” (as cited in Hogan, 2007, p. 3). Although still ambiguous, the updated version of evaluation included more than just “programs.” Guyadeen and Seasons (2016) propose that earlier evaluation approaches were comprised primarily of ex-ante evaluations (as mentioned by CDC (2012) and the Organisation for Economic Co-operation and Development (2016)) and included organized quantitative goals (p. 4). However, in 2016, the same authors offer a much more comprehensive explanation of evaluation to state that it is “…an empirically oriented discipline that generates information about programs in order to improve the program or guide future decisions” (as cited in Guyadeen & Seasons, 2016, p. 2).

It appears that, over time, the general meaning of evaluation has remained the same. However, the term has also been modified over time to conform to particular circumstances. The evolution of evaluation’s meaning (and its processes) might reflect changes made to assessments and research in various programs or disciplines, but the term’s foundation remains the same today.

References

Guyadeen, D., & Seasons, M. (2016). Evaluation theory and practice: Comparing program evaluation and evaluation in planning. Journal of Planning Education and Research, 1-13. doi: 10.1177/0739456X16675930

Hogan, R. L. (2007). The historical development of program evaluation: Exploring the past and present. Online Journal of Workforce Education and Development, 2(4), 1-14. Retrieved using the Walden library databases.

Nielsen, L., & Turner, S. D. (1983). Program evaluation as an evolutionary process. Evaluation Review, 7(3), 397-405. Retrieved using the Walden library databases.