Annotated Bibliography

05CH_Robinson_Justice.pdf

Home >Law homework help >Criminal homework help >Annotated Bibliography

How to Critique a Journal Article

Learning Objectives

After reading this chapter, you should be able to:

• Critique a journal article, using the holistic critique and the question-and-answer cri- tique methods as possible organizational schemes.

• Explain how journal article critiquing falls within the higher-level inquiry skills of analy- sis, evaluation, and creation.

• Evaluate whether the data collected in a particular study appropriately match with the author’s conclusions.

• Comprehend the basic concepts necessary to be a savvy research consumer, appreciat- ing the role of critical thinking skills and their application to daily life.

Hill Street Studios/Blend Images/Getty Images

CHAPTER 5Section 5.1 Evaluating Journal Articles: The Big Picture

In the previous chapter, we focused on the search for relevant sources related to the topic of study. A student may be searching far and wide for information to include in a research

project, but once a student has located that information and confirmed its validity, what’s next? At this point, the student will begin a careful review of the sources and determine whether the evidence is worth including in the research project; that is, a critique, or criti- cal evaluation, of the retrieved information is conducted. In this chapter, the focus is the criminal justice journal article, simply because journal articles provide students with the most significant material for their research projects. However, many of the critical analysis techniques discussed are also applicable to books, websites, government reports, etc. The ability to analyze information is needed in numerous situations. For example, an indi- vidual wishing to purchase a home or a new car will need to critically evaluate all of the information provided by the bank or lender to make an informed and intelligent decision on which mortgage or loan provides the best options and opportunity for purchase. As such, the ability to critique information has applications well beyond the classroom, mak- ing it a valuable lifelong skill.

5.1 Evaluating Journal Articles: The Big Picture

Students must be able to critically evaluate others’ work for their research projects. In addition, the ability to critique is a useful lifelong skill, as it promotes the ability to

think critically and analytically in everyday situations. Thus, critiquing is a skill that all students should practice regularly.

Critiquing a journal article is different from simply reading a journal article, as described in Chapter 4. A student can read a journal article just to extract ideas or to better under- stand the topic being studied. A journal article critique, however, is a deeper analysis, whereby the student is prompted to think about possible improvements or flaws to the study that must be considered in the context of the research question.

Understanding the anatomy of a journal article can help students organize the critique of the article. Table 5.1 provides a brief description of the typical anatomy of a research- based journal article.

CHAPTER 5Section 5.1 Evaluating Journal Articles: The Big Picture

Table 5.1: Anatomy of the journal article (how the author writes the article)

Title page (take credit)

• Author’s name • Author’s affiliation • Running head information

Abstract (quick summary)

• Usually 120–150 words • May include keywords • Some articles will not include an abstract

Introduction (what was being studied)

• Introduces the problem • Develops the background • States the purpose and rationale for the present

study

Method (how was the study conducted)

• Participants • Apparatus or materials • Procedure • Enough detail for reader to replicate if desired

Results (what happened in the study)

• Tables and/or figures if necessary • Presentation of statistical outcomes • No interpretation, just presentation

Discussion (what do the results mean)

• Was there support for the research idea? • Did the study help resolve the original problem? • What conclusions can be drawn? • Suggests improvements, avenues for further/

new research

Reference page (give credit where credit is due)

• Starts on its own page • Lists authors alphabetically by last name of first

author • No first names used • Shows author’s scholarly ability and how she or

he did her or his homework

Students must also understand the difference between conducting a critique and criticiz- ing another’s work. Criticizing can be relatively easy; simply point out errors and what the author did wrong. However, the goal of a critique is to analyze and evaluate the work on multiple dimensions to determine the impact it has on criminal justice. By conducting a scholarly critique, a student should be able to provide examples of positive contributions and provide critical criticisms that point to gaps in the research or in the author’s conclu- sions of the data (Academic Skills and Learning Centre, 2009). Journal articles, and the connections we make between journal articles, provide the foundation for criminal justice research. Cumulative understanding about criminal justice and crime is developed one study at a time.

CHAPTER 5Section 5.1 Evaluating Journal Articles: The Big Picture

There are several effective methods for critiquing a criminal justice journal article, which are discussed in more detail later in this section. Regardless of which method a student chooses, any criminal justice journal article critique has three main sections:

1. the identification of the author and the formal citation of the source (as it would appear in a references section);

2. a summary of the main points of the article; and 3. the analysis and evaluation (the “critique”) of the article, which is typically the

bulk of the document (Center for Teaching and Learning, 2009).

The identification of these three sections provides a nice organizational structure for a journal article critique, with the major detailed analysis provided in the third section. Students should also note that most of the points raised in a journal article critique center on the keys for evaluating sources discussed in Chapter 4: authorship, expertise, currency, accuracy, and corroboration. Another method that provides a more holistic (general) approach can also be used as a critique strategy.

The Holistic Critique Method An effective approach to conduct- ing a journal article critique, which provides an overall, global assess- ment of the article, is the holistic critique method (Pyrczak, 2008). This method is similar to a global assessment performed when the journal article was selected for possible inclusion in the research paper. In the holistic critique method, the student should focus on the following questions for each journal article (from Pyrczak, 2008, pp. 129–133). (See Writing in Action: Sample of a Holistic Critique for a specific example.)© Getty Images/Jupiterimages/Comstock/Thinkstock

The holistic critique is a method used to globally evaluate the merits of a journal article.

1. Has the researcher(s) selected an important problem? Based upon the student’s learning and knowledge, he or she should be able to identify whether the article discusses an important problem in social and criminal justice, something that impacts a large segment of social and criminal justice.

2. Was the researcher(s) reflective? The student should evaluate whether a compre- hensive critique and literature review of prior studies was provided, to show the researcher(s)’s depth of knowledge.

3. Is the report cohesive? The student should be able to easily understand the mate- rial if it is written in a clear and concise manner. Also, tables and figures should be easy to understand.

CHAPTER 5Section 5.1 Evaluating Journal Articles: The Big Picture

Writing in Action: Sample of a Holistic Critique

Article Critiqued: Sleep Disorders, Health, and Safety in Police Officers

APA Citation/Reference:

Rajaratnam, S. M. W., Barger, L. K., Lockley, S. W., Shea, S. A., Wang, W., Landrigan, C. P., . . . Czeisler, C. A. (2011). Sleep disorders, health, and safety in police officers. Journal of the American Medical Association, 306(23). Retrieved from http://jama.jamanetwork.com/article .aspx?articleid=1104746

Have the Researchers Selected an Important Problem?

Yes. These researchers addressed sleep disorders and the effect they have on police officers’ health and the risk posed to the public. Specifically, the researchers set out to address the associations between sleep disorder risk and self-reported health, safety, and performance outcomes in police officers. Know- ing how sleep disorders affect the job performance of police officers would be valuable to police admin- istrators and officers, as safety on the job is paramount for officers and the communities they serve.

Were the Researchers Reflective?

The researchers were reflective, but with little prior research on the effects of sleep disorders on job performance of police officers, there was not an extensive review of the literature. The researchers noted that “to date, the effect of sleep disorders on police officer health, safety, and performance has not been systematically investigated” (Rajaratnam, et al., 2011, para. 8).

4. Does the report extend the boundaries of the knowledge on a topic, especially for understanding relevant theories? The student should look for information that provides a new assessment or view of the topic, most likely found in the analysis/discussion/conclusion sections of the article.

5. Are any major methodological flaws unavoidable or forgivable? The student should be able to identify flaws or errors in the research methodology and be able to determine whether these flaws or errors have a significantly negative impact on the data.

6. Is the research likely to inspire additional research? The student should deter- mine whether others would be interested in conducting similar research. In other words, is this research, and the information provided in the article, worthy of continued study?

7. Is the research likely to help in decision making? The student should determine whether the research can be used by social and criminal justice personnel in making decisions.

8. All things considered, is the report worthy of publication in an academic journal? After the critique, the student should be able to determine whether this study was worthy of publication, being available to anyone interested in the topic.

9. Would you be proud to have your name on the research article as a coauthor? Based on the overall critique of the article, the student can decide whether he or she would like to have worked on the study and been counted as a coauthor.

(continued)

http://jama.jamanetwork.com/article.aspx?articleid=1104746

CHAPTER 5Section 5.1 Evaluating Journal Articles: The Big Picture

Writing in Action: Sample of a Holistic Critique (continued)

Is the Report Cohesive?

Yes. The report follows the typical anatomy of a research report, with the major sections of introduction, method, results, and discussion. Although there are numerous tables and charts, they are relatively easy to comprehend and provide a good pictorial of the information and support the researchers’ discussion of results.

Does the Report Extend the Boundaries of the Knowledge on a Topic, Especially for Understanding Relevant Theories?

The researchers tackle a little-studied topic and advance the topic through their use of longitudinal research techniques. It was hypothesized that fatigue, related to duration and quality of sleep as well as to untreated sleep disorders, plays a role in unintentional injuries and fatalities of police officers. Understanding the role of sleep disorders for police officers will make a tremendous impact on the field of law enforcement.

Are any Major Methodological Flaws Unavoidable or Forgivable?

There were no major methodological flaws. The researchers used standard methodological tech- niques, which included both online and on-site study components. The substantial case study data available (N = 4,957 sworn police officers) provided for appropriate statistical techniques, and the conclusions drawn represent the data collected.

Is the Research Likely to Inspire Additional Research?

Yes. This is an important criminal justice issue, and understanding the role that sleep disorders have on police job performance is an expanding area in need of future research. Sleep disorders are com- mon but mostly undiagnosed, and if they remain untreated, significant negative consequences in all aspects of police work can result, including adverse health and safety issues. Understanding how sleep affects individuals who work on rotating shifts, which can result in such problems as chronic sleep deficiency, can assist in developing training and education for police officers to identify potential problems before they exist. These results can also prompt further research to help expand the little research that currently exists.

Is the Research Likely to Help in Decision Making?

Yes. The findings in this study are most valuable to police administrators, as they are the ones making decisions on police officer shifts and determining the appropriate work hours to ensure maximum job performance while maintaining the health and safety of officers.

All Things Considered, is the Report Worthy of Publication in an Academic Journal?

Yes, absolutely. This is important research; the methodology presented is important for future researchers, and the results are important for police officers, police administrators, and policy makers on the state, local, and federal levels.

Would You be Proud to Have Your Name on the Research Article as a Coauthor?

Absolutely, yes.

CHAPTER 5Section 5.1 Evaluating Journal Articles: The Big Picture

The Question-and-Answer Critique Method A student seeking advice about how to critique a criminal justice journal article would find many sources that offer long listings of key questions to ask while reviewing and critiquing an article. Using a detailed list of questions for the critique is known as the question-and-answer critique method. Listings from various sources (such as those found in GPRC, n.d.; Salkind, 2009; and Center for Teaching and Learning, 2009) share common characteristics and can be categorized into topic sections based on the focus of each specific question. This method can be considered a more detailed approach than the holistic critique method presented previously. However, there is no one specific set of questions to ask, and as students gain experience in critiquing journal articles, the list of detailed questions can be expanded. Table 5.2 presents a good example of a detailed set of questions students can use to critique a criminal justice journal article.

Table 5.2: Question-and-answer critique questions sorted by topic

Review of the existing literature

1. Is there a theoretical perspective being following here? If so, which perspective?

2. Was the author correct in his or her interpretation of others’ work?

3. Has the author cited the pertinent references necessary?

4. How closely was the literature cited in the journal article actually related to the previous literature? Was it a good match or mismatch?

5. Is it an up-to-date literature review?

6. Are there any classic or seminal research works that should have been included in the literature review but were left out?

Problem and purpose of the study

7. Briefly describe the topic of the journal article, which is usually described in its title.

8. Identify the important concepts the author presents.

9. What are the author’s underlying assumptions that may influence the study’s outcomes?

10. Can you understand the statement of purpose of the study?

11. Is there a rationale that explains why the study’s outcomes would make an important con- tribution to the published literature?

Thesis statement or hypothesis

12. Are the key concepts clearly defined and explained?

13. What is the thesis of the work presented in the journal article?

14. Did the author’s approach to supporting or refuting the thesis make sense?

15. Can the hypothesis or hypotheses presented actually be tested?

16. Do the hypotheses state a clear, expected association between the variables under study?

17. Are the hypotheses grounded in theory or as part of a review of relevant literature?

(continued)

CHAPTER 5Section 5.1 Evaluating Journal Articles: The Big Picture

Methodology, including sample

18. What was the author’s general method for drawing conclusions about this topic? Was this an empirical study?

19. What was the author’s method of collecting data? Qualitative, quantitative, or both?

20. Is there a clear rationale provided for how the key questions of interest can be adequately addressed by the research methodology utilized?

21. Are the research methods adequately described to the extent that another researcher could replicate this research study?

22. How large or representative was the sample that was used?

23. Are independent and dependent variables clearly defined, or at least clearly identifiable?

24. Is it clear how the study was conducted—that is, the participants selected, the materials used, and procedures followed throughout the study?

25. Was the sample selected such that you believe it to be representative of the greater popu- lation of interest?

26. How similar are the participants selected here to participants utilized for past studies?

Analyses and results

27. Are the statistical calculations used appropriate and correct?

28. Did you discover any errors in the analyses selected?

29. Were the results reported fairly and accurately?

30. Are the results reported relatable to the hypotheses being tested?

Discussion, conclusion, and interpretations

31. Do the data analytic outcomes support the assertions made by the author?

32. Which results are the most thought provoking?

33. Are limitations to the study mentioned, and are they reasonable and plausible limitations?

34. Does the author make suggestions for future research? What are the unanswered questions?

35. Did the author draw appropriate generalizations?

36. Did the author present any counter-evidence or alternative explanations for the data col- lected or conclusions drawn from this research?

37. Does the discussion provide closure regarding the initial hypotheses presented earlier in the journal article?

38. What would you do to improve this research?

39. What are the primary implications of these research outcomes?

Table 5.2: Question-and-answer critique questions sorted by topic (continued)

(continued)

CHAPTER 5Section 5.1 Evaluating Journal Articles: The Big Picture

References

40. Are the references recent and relevant?

41. Are the references properly formatted and complete?

42. Does the list of references reflect the most important reference sources available in the field currently?

43. Is there enough information provided in each reference such that the reader can retrace the exploratory steps of the author and locate the original source information?

General comments

44. Provide the full citation of the journal article being critiqued.

45. Is the title of the article understandable?

46. Is the abstract correctly formatted and helpful?

47. Did you come away with at least one big-picture, take-home message?

48. Is the overall writing style clear and understandable? Is the writing style appropriate for the audience?

49. Does the author strive for neutrality and objectivity in word choice, or is the language selected emotionally charged and biased?

50. Do you agree or disagree with the author’s conclusions about the topic? Why?

51. What are the strengths and weaknesses of the research presented in the journal article?

Note that as with the holistic critique approach, students can utilize many of the items presented in Table 5.2 as an evaluative checklist for their own as well as others’ writing.

Table 5.2: Question-and-answer critique questions sorted by topic (continued)

CHAPTER 5Section 5.2 Higher-Order Inquiry Skills

5.2 Higher-Order Inquiry Skills

The ability to critique a journal article is a specialty version of a higher-order skill known as evaluation. The notion of higher-order (and lower-order) skills became popularized

due to Benjamin Bloom’s (1956) work on the taxonomy, or classification, of functions in the cognitive domain (mental skills used to acquire knowledge—intellect). Taxonomy is an organizational scheme arranged in a hierarchy from less to more complex and is based on information and knowledge that educators want students to master. Bloom organized his taxonomy into six levels, which are successive, meaning each level must be mastered before moving to the next highest level. His original six cognitive levels (from lower-order to higher-order) were knowledge, comprehension, application, analysis, synthesis, and evaluation. In a revised taxonomy presented by Anderson & Krathwohl (2001), colleagues of Bloom, the six levels of cognitive processes (from lower-order to higher-order) were amended and termed remembering, understanding, applying, analyzing, evaluating, and creating (see Figure 5.1).

Figure 5.1: Bloom’s revised taxonomy

College students in particular are pressed to obtain the three higher-order skills, which include the process of critiquing journal articles. “Within a critique, students analyze the article’s clarity, organization, purpose, research methodology, findings, and recommen- dations,” while “a literature review demands expansion of the understanding of several articles into knowledge, application, analysis, synthesis, and evaluation of ideas” (Harris, 2006). In other words, understanding and composing an effective critique of a journal article, or other scholarly work, “provides the foundation for improved scholarly writing and enhances students’ critical thinking abilities” (Harris, 2006). The following is a brief description of the three higher-order inquiry skills.

• Generating • Planning • Producing

• Differentiating • Organizing • Attributing

• Checking • Critiquing

Creating

HIGHER ORDER SKILLSEvaluating

Analyzing

Applying

Understanding

Remembering

CHAPTER 5Section 5.2 Higher-Order Inquiry Skills

Analyzing Analyzing “involves breaking material down into its constituent parts and determining how the parts are related to one another and to an overall structure” (Anderson & Krath- wohl, 2001, p. 79). The specific subprocesses include the abilities to differentiate, organize, and attribute. In the revised cognitive taxonomy, analysis is frequently grouped in a trio of processes: understand, analyze, and evaluate. When a student is asked to perform an analy- sis, that task may be to separate fact from fiction, to make an argument and support it with a citation in APA format, to distinguish between evidence that is empirically based ver- sus anecdotal, to uncover the relationship between ideas or behaviors, and so on. Both the holistic and question-and-answer critique methods provide students with analyzing skills.

Evaluating Evaluating “is defined as making judgments based on criteria and standards. The cri- teria most often used are quality, effectiveness, efficiency, and consistency” (Anderson & Krathwohl, 2001, p. 83). The key processes applied with evaluating are checking and critiquing. Checking involves an examination of facts presented so that there are minimal inconsistencies. For example, a statement made in a student’s introduction section/review of the literature does not contradict with a statement made on a similar topic in the discussion/conclusion section of the same paper. Critiquing is one’s ability to make judgments about others’ work based on a specific set of stated criteria, and it takes practice to develop the expertise to effectively evaluate based on those criteria. For example, a correctional officer is promoted to assistant warden, an administrative position in which he is responsible for conducting annual performance evaluations of all correctional officers in the prison. Although the evaluation process may be unclear or confusing at first, the assistant warden’s ability to effectively evaluate officer perfor- mance should improve over time and with practice.

Creating Creating “involves putting elements together to form a coherent or functional whole” (Anderson & Krathwohl, 2001, p. 84). The processes associated within the creation level include generating, planning, and producing. As students write papers of various sorts throughout college, the process of creation is evident. For example, for the capstone crimi- nal justice research paper, each student is generating a list of sources to critique, planning the presentation style of the material to be written, and producing written work that con- forms to APA format. In Bloom’s revised taxonomy, creation is the highest level within the cognitive domain and requires the coordination and competency in all the levels leading to it—remembering, understanding, applying, analyzing, and evaluating. In other words, the different levels within the cognitive domain are different types of tasks and skills that require different levels of competency and practice in order to achieve expertise.

CHAPTER 5Section 5.3 Do the Conclusions Match the Data Collected?

5.3 Do the Conclusions Match the Data Collected?

The focus of a criminal justice journal article critique is to evaluate the merits of the research, which in turn determine the merits of the conclusions and outcomes of that

research. Along with identifying an appropriate critiquing method and using a higher-order level of cognitive reasoning in the critique, students must also determine whether the data collected by the research appropriately match the conclusions drawn by the researcher(s). For example, has the researcher drawn appropriate conclusions from the data obtained, or do the conclusions overreach the study? Is the cause-and-effect relationship language used in the journal article supported by the research design and statistical data results? This level of analysis, like the other evaluative tasks of critiquing a journal article, will require time and practice to acquire and master.

Causality in Research Studies For many criminologists, the most powerful conclusion that can be made in research is a cause-and-effect conclusion. In theory, if we are able to deter- mine causality (one thing causing a change in something else), then we can make positive outcomes occur more often and work to prevent negative outcomes from happening as often. For example, it would be beneficial to know what causes college student suc- cess, what causes happiness, and what causes job satisfaction, so that we could promote those causes and help individuals strive for goals they wish to achieve. In criminal justice, it would be ben- eficial to know what causes rape, what causes underage drinking, what causes armed robbery, what causes domestic vio- lence, and what causes suicide by police officers, so that practitioners and criminal justice personnel could work to prevent antecedent (before the fact) causes that lead to these negative outcomes.

In determining causality, very precise methodology must be used to provide a high level of confidence about the determination, and with this methodology comes different categories of research questions. Meltzoff (1998) provides a description of various types of research ques- tions. Table 5.3 summarizes Meltzoff’s (1998) descriptions, using generic statements, while also adding realistic examples. If the journal article is written with enough clarity, a student should be able to place it in one of the types of research questions identified in Table 5.3.

Martin Barraud/OJO Images/Getty Images

Understanding causality, or how one thing causes a change in something else, is an important part of deciding how to promote or prevent certain outcomes.

CHAPTER 5Section 5.3 Do the Conclusions Match the Data Collected?

Table 5.3: Types of research questions, with examples

Types of research questions Generic example Specific example

Existence questions Does x exist? Can people have a drug addiction? Does pedophilia exist?

Questions of description and classification

What is x like? To what extent does x exist?

What are the best measures to prevent gang violence? What is it like to be a victim of crime? To what extent are inmates controlling prisons?

Questions of composition What are the components that make up x? What are the factors that comprise x?

What variables lead to using illegal drugs? What are the leading indicators that someone is a child molester?

Statistical relationship questions

Is there an association or relationship between x and y?

Is one’s age related to inter- est in pornography? Is there an association between alco- hol and domestic violence?

Descriptive–Comparative questions

Is Group x different from Group y?

Are males or females more likely to engage in underage drinking? Do one- or two- parent homes have more juvenile delinquency?

Causality questions Does x cause, lead to, or prevent changes in y?

Does psychotherapy help individuals with dissocia- tive identity disorder? Does attending DUI school lead to less drunk driving?

Causality–Comparative questions

Does x cause more of a change in y than z does?

Does the death penalty pro- vide more feelings of justice for victims than life in prison? Is medical maintenance better at dealing with drug addiction than detoxification?

Causality–Comparative interaction questions

Does x cause more change in y than does z under certain conditions, but not under other conditions?

Are males raised in urban areas more likely to engage in sexual deviance, but not females raised in urban areas? Are children in elementary school more likely to remain drug-free than middle school children, but only for elementary school children who attended a D.A.R.E. program?

Adapted in part from Meltzoff (1998)

CHAPTER 5Section 5.3 Do the Conclusions Match the Data Collected?

For many criminologists, the ultimate goal is the determination of causality (e.g., under- standing cause-and-effect relationships). However, obtaining data that show a significant positive correlation between two variables does not necessarily mean that causality can be inferred. Burns (1997) identified three criteria for establishing causality. First, there must be clear temporal precedence. That is, for the cause to be the cause, and the effect to be the effect, the cause must come first, and the effect must come second. If both cause and effect occur simultaneously, then we cannot know the cause or the effect. There must be a clear time sequence. Second, measures of cause and effect must covary, or exhibit covariation. That is, if there is a cause-and-effect relationship, the presentation of the cause needs to yield the effect, but if there is no presentation of the cause, then there should be no effect. In other words, if the nature of the cause is changed, then the nature of the effect should also change. Third, there should be no plausible alternative explanation. If we have ade- quately applied our research methods, experimental controls, methodological designs, and so forth, then we need to say, with confidence, that there is no other logical explana- tion for the effect other than the cause. Note that we do not say that we proved that the cause is the reason for the effect, but we infer the relationship when we have (a) temporal precedence, (b) covariation, and (c) no plausible alternative explanations (Burns, 1997). Technically speaking, we do not “prove” anything in criminal justice research.

Concerns About Nonscientific Evidence In drawing conclusions, the quality of evidence and type of data are critical factors. For example, the type or kind of evidence relates to the type or kind of conclusions possible. Evidence gathered through the methods and processes of scientific research often holds special characteristics that nonscientific evidence does not possess. For example, religious or political affiliations, and other nonscientific research, may not go through a rigorous peer-review process, resulting in articles that could include slanted or biased reporting of the evidence. Bell (2005) identified and summarized some of the problems that can occur when nonscientific evidence is used (p. 38):

• An example (one observation) or the observations from a few people may be used as evidence. However, it may not be clear who actually made the observations and under what conditions.

• The observations are not repeated and carefully checked. Thus, a fluke occurrence could prematurely be accepted as fact.

• Observations are not written down as they occur but are recalled at some later date. Given the fallibility of memory, a specified procedure for reliably recording data is highly desired.

• The people being observed, usually those who are available, are rarely representa- tive of a larger population. Representative samples in criminal justice research are critical to be able to generalize results beyond the individuals studied or tested.

• Important terms are not precisely defined and sometimes the definitions of key terms change. Disagreement about the meaning of key terms might mean that researchers who think they are studying the same topics/constructs are not study- ing the same phenomenon at all.

• A comparison group or comparison condition is rarely available. A hallmark of the scientific approach is the comparison of the treatment group to the control group, whenever possible.

CHAPTER 5Section 5.4 Being the Savvy Consumer of Research

• Control over important variables is impossible. It is very difficult to keep con- stant the relevant factors to be able to state a specific cause. Subtle factors can be overlooked.

• All of the evidence is not used. Evidence that is contrary to a statement is over- looked, or only part of the evidence is used.

• Some writers have biases that interfere with objective observation and make it difficult to separate evidence from opinions and value judgments. By relying on scholarly peer review in the publication of journal articles, subjective conclusions and the chances of fraud are minimized.

This is not to say that scientific research and its resulting evidence do not have the poten- tial for flaws, but when developed research methods and peer review procedures are uti- lized, this ensures the conclusions drawn from research data are appropriate and fair. This is important for answering the question of whether the conclusions drawn match the data collected from the research. As noted previously, mastering the skill of determining whether appropriate conclusions were drawn from the research data takes time and prac- tice. However, the benefits from this systematic approach are numerous. Having a specific method for making decisions in scientific research allows individuals to avoid making decisions about people or behaviors based on opinions or biases.

5.4 Being the Savvy Consumer of Research

Criminologists and students alike need to be savvy, or

knowledgeable, consumers of research. As such, they must be aware of potential hazards associ- ated with certain types of research and research findings in order to (a) avoid making fallacious claims based on their own work, and (b) identify false claims made by others to avoid being duped or misled. By developing a honed sense of critiquing, students can expand their evaluations beyond the journal article critique and can utilize them to assess any type of research finding or claim. Some of the potential hazards that should be recognized involve the appear- ance of illusory correlations and the

natural bias to believe what one wants to believe. Being a good researcher (generating research) or a knowledgeable consumer of research (reading research) necessitates a fair measure of quantitative reasoning skills. The following sections provide some specific examples of types of potential hazards.

iStockphoto/Thinkstock

Hazards are associated with certain types of research, so students must be aware of these potential hazards in order to avoid conveying inaccurate information.

CHAPTER 5Section 5.4 Being the Savvy Consumer of Research

Illusory Correlations and Confirmation Biases Illusory correlation and confirmation bias are two ways that common sense can mislead us in evaluations or critiques. An illusory correlation is our tendency to believe that two variables are related when they are not. For example, a student is convinced that her favor- ite sports team wins only when she watches. However, if she were to gather the data and look at her team’s win–loss record when she watches and when she does not watch, she is likely to find that her presence is not a deciding factor. Or perhaps she regularly reads the horoscope and uses this information to guide her behavior. One horoscope on www .astrology.com stated, “A big improvement is in the works, one that you may know noth- ing about, and today is the day for the big unveiling.” If this student is about to graduate and is currently seeking employment, spotting some new listings for interesting positions may convince her that the horoscope was right. When individuals look for an association, they often find it. Yet, over and over again, careful researchers fail to find evidence to sup- port the accuracy of astrology (see Dean & Kelly, 2003). To avoid this, using quantitative reasoning (i.e., statistical numbers) helps individuals to see past illusory correlations.

A confirmation bias is our tendency to “see” evidence that fits with our beliefs and to ignore evidence that does not fit with our beliefs. This tendency is the main reason that we “see” illusory correlations. From the previous example, if the student believes that watching her favorite sports team contributes to that team’s success, then she will become aware of the times that she watched and they won and the times that she did not watch and they lost. However, what happens with her confirmation bias is that she underes- timates the number of times that she watched and they lost or that she did not watch and they won. In the second example, when using horoscopes to guide behavior, many individuals remember when their horoscopes had an uncanny prediction—for example, the prediction of a problem in love on the exact day of the breakup of a romantic relation- ship. To illustrate confirmation bias, Munro and Munro (2000) showed that more students reported accurate predictions for their particular astrological sign when those signs were presented along with the predictions. In the report, 34% of students chose their own horo- scope as the best match for them when the horoscopes were labeled with the signs of the zodiac, whereas only 13% chose their own horoscope when they were labeled only with numbers and in a random order. In another example, it is well documented that driving while trying to operate a cell phone is a dangerous distraction, yet growing numbers of drivers are calling and texting while driving (Richtel, 2009). Confirmation bias can help explain this trend, in that people believe they are such good drivers that they can avoid accidents while using their cell phones. However, this illusion has the potential to create hazardous and perhaps deadly consequences. Oftentimes we know the right thing to do, yet we ignore the information.

CHAPTER 5Section 5.4 Being the Savvy Consumer of Research

Coincidence and Probability When attempting to understand and predict human behavior, it is helpful to know about coin- cidence and probability. When criminologists understand the logic that underlies the occur- rence of behaviors and events, this helps them to make more accurate predictions about future behaviors. Coincidence is when two random events occur close in time to one another, which leads us to think the events are con- nected in some way. A person might observe that every time he attends court as an observer, the judge gives each defendant the maximum sentence. Therefore, he concludes that his presence has an effect on the judge’s sentenc- ing decisions. In fact, our belief in coincidences leads to our beliefs in illusory correlations. If that pairing of events happens enough times (court attendance and maximum sentenc- ing), one may think the events are related, rather than just a coincidence.

Probability is the concept of how frequently an event occurs compared to how often the event could occur. If a person flips a quarter 100 times, he or she is likely to come up with approximately 50 heads and 50 tails. Thus, the expected probability of getting “heads” on the flip of a quarter is 50 out of 100, or 0.50, meaning half the time. Quantitative reason- ing shows that coincidence is without a scientific basis, while probability has a scientific basis. We can statistically determine the rate at which an event will occur, while taking into account the possibility of the event happening by chance alone (e.g., a coincidence).

When looking at a coincidence, students should be sure to examine the full context (e.g., the “background”). In order to use statistics to harness probability, consider the odds that the event will occur compared to the odds that the event could occur. Statistics requires that we look at the background or base rate. It prevents us from declaring something astonishing, perhaps a miracle, when it is really expected, while also preventing us from finding meaning in coincidence.

Critical Thinking and Critiquing for Everyday Life Although quantitative reasoning is a key method in the critical thinking and critiquing processes of journal article evaluations, it can also be used in everyday life when extract- ing information from data. For example, if someone lost weight through a specific diet program, others who want to try the program should first find out if the results were typi- cal. If someone suffering from an anxiety disorder hears about a new medication that is

Michael Blann/Photodisc/Thinkstock

Unlike coincidence, probability has a scientific basis and is determined by quantitative reasoning.

CHAPTER 5Questions for Critical Thinking

reported to be highly effective, he or she should first find statistical data on the medication and its average effectiveness. Individuals buying new cars could seek data from consumer review sites about reliability and performance of the cars in which they are interested.

When data are available, they should always be questioned. If the diet program reports a typical weight loss of 10 pounds in three weeks, an interested dieter should determine whether that is average or atypical and find out whether most people sustain the weight loss. If researchers suggest that cognitive-behavior therapy is better than antianxiety medi- cation for a particular disorder, a potential patient should ask more questions. How big of an effect is this? What was the population that was studied? Do the improvements tend to last? And if the car one is hoping to buy fares well against the competition, follow-up questions should address the comparison group and whether the source is unbiased. By critiquing the data presented and by utilizing critical thinking skills, anyone can learn to become a savvy consumer of research, whether in the journal article context or in daily life.

Chapter Summary

Higher-order skills, such as analysis, evaluation, and creation, must be repeatedly practiced in order to gain both expertise and confidence in the ability to perform

higher-order cognitive tasks, such as critiquing a criminal justice journal article. As stu- dents develop these skills, consistent evaluative criteria typically emerge. Research con- clusions appear within a social context, even when a researcher strives to be as objective and detached as possible. In order to avoid overreaching with the results, researchers need to be careful to match the conclusions drawn with the type of data collected. That is, data collected from experimentation with random assignment may be appropriate for cause-and-effect conclusions, but correlational data are not appropriate for the determi- nation of causality. The ability to critique the work of others is a specialized version of critical thinking, and the savvy consumer of research needs to have the abilities to apply critical thinking principles and quantitative reasoning to research outcomes as well as to everyday life situations. Developing honed critiquing skills, including the careful cri- tique of published work, can lead to the identification of erroneous conclusions as well as educating a researcher in learning what hazards to avoid in future work.

Questions for Critical Thinking • In the process of critiquing a journal article, what is the holistic critique method?

How does this differ from the question-and-answer critique method? • What are the higher-level skills in Bloom’s taxonomy? Why is it important to

acquire the skills of analysis, evaluation, and creation in order to effectively critique a journal article?

• Why is it important to determine whether the conclusions made by the author of a journal article match the statistical data collected in the study? What are some of the concerns about nonscientific evidence?

• What are illusory correlations and confirmation biases? How do these impact a critique of a journal article?

CHAPTER 5Key Terms

Web Links This website gives more examples of analysis, evaluation, and creation skills using Grid Analysis: http://www.mindtools.com/pages/article/newTED_03.htm.

This website provides more details about being a savvy consumer of research—being research-minded: http://www.resmind.swap.ac.uk/content/06_making_sense/making_sense_index.htm.

This website provides more information about illusory correlations: http://changingminds.org/explanations/theories/illusory_correlation.htm.

Key Terms

analyzing The third-highest level of Bloom’s revised taxonomy; involves deconstructing a complex topic or object into its component parts for further study.

cause-and-effect conclusion The strongest type of conclusion that criminologists can reach, in which the antecedents and conse- quences between two actions are explicitly known and connected.

coincidence When two random, unre- lated events occur in close proximity of one another.

confirmation bias A bias in which indi- viduals tend to seek out and believe evidence that is consistent with their already-held beliefs.

covariation A statistical relationship between two or more variables, wherein as one variable changes, other variables change in a predictable and stable fashion.

creating The highest level of Bloom’s revised taxonomy; involves generating, planning, and producing an original work.

critique A critical evaluation of the infor- mation at hand.

evaluating The second-highest level of Bloom’s revised taxonomy; involves deter- mining the worthiness of an idea or object by comparing its characteristics to prede- termined standards or criteria.

holistic critique A broad approach to examining the details of written work.

illusory correlation The tendency to believe that two variables or events are connected in a meaningful fashion when in fact there is no connection (akin to superstition).

plausible alternative explanation In for- mulating hypotheses, this serves as an addi- tional possible description of the relation- ships between variables in addition to the explanation the researcher hopes to support.

probability The frequency of an event occurring in comparison to the frequency that an event could occur.

question-and-answer critique A criticism technique in which the analysis is driven by specific questions to be answered.

temporal precedence The knowledge that one event specifically occurred prior to the onset of the next event.

http://www.mindtools.com/pages/article/newTED_03.htm

http://www.resmind.swap.ac.uk/content/06_making_sense/making_sense_index.htm

http://changingminds.org/explanations/theories/illusory_correlation.htm