i need a experienced project management tutor

lable at ScienceDirect

Learning and Instruction 29 (2014) 31e42

Contents lists avai

Learning and Instruction

journal homepage: www.elsevier.com/locate/learninstruc

Improving critical thinking: Effects of dispositions and instructions on economics students’ reasoning skills

Anita Heijltjes a,*, Tamara van Gog b, Jimmie Leppink c, Fred Paas b,d

a Learning and Innovation Centre, Avans University of Applied Sciences, Hogeschoollaan 1, Postbus 90116, 4800RA Breda, The Netherlands b Institute of Psychology, Erasmus University Rotterdam, The Netherlands c Department of Educational Development and Research, Maastricht University, The Netherlands d Interdisciplinary Educational Research Institute, University of Wollongong, Australia

a r t i c l e i n f o

Article history: Received 15 February 2012 Received in revised form 10 July 2013 Accepted 17 July 2013

Keywords: Critical thinking instructions Dispositions Biased reasoning

* Corresponding author. E-mail address: [email protected] (A. Heijltjes

0959-4752/$ e see front matter � 2013 Elsevier Ltd. http://dx.doi.org/10.1016/j.learninstruc.2013.07.003

a b s t r a c t

This experiment investigated the impact of critical thinking dispositions and instructions on economics students’ performance on reasoning skills. Participants (N ¼ 183) were exposed to one of four conditions: critical thinking instruction, critical thinking instruction with self-explanation prompts during subse- quent practice, critical thinking instruction with activation prompts during subsequent practice, or no critical thinking instruction or prompts (control). In all conditions, practice was a within-subjects factor, some task categories present in the test were practiced on a business case, others were not. Participants in the instruction conditions significantly outperformed participants in the control condition on the immediate and delayed post-test, but only on the practiced task categories e with the exception of the self-explanations condition, which also showed a better performance than the control condition on not- practiced categories, though only on the immediate post-test. Dispositions (i.e., Actively Open-minded Thinking and Need for Cognition) predicted reasoning skills at pre-test but did not interact with in- structions on post-tests performances.

� 2013 Elsevier Ltd. All rights reserved.

1. Introduction

Against the background of complex and rapidly changing busi- ness environments, economics students are expected to become critical thinkers (Klebba & Hamilton, 2007; Smith, 2003). Critical thinking enables students to make sound logical and unbiased de- cisions, and in educational situations it has been shown to lead to better learning and transfer outcomes (e.g., Facione, 2009; Halpern, 1998; Helsdingen, Van Gog, & Van Merriënboer, 2011). Therefore, it is surprising that critical thinking is rarely explicitly taught in eco- nomics curricula (Jones, 2007). This might be due to the prevailing view that immersion in business methods and strategies will lead to the spontaneous development of critical thinking skills (Jones, 2007). However, there is little evidence that such skills develop spontane- ously as a consequence of instruction in a discipline (Halpern, 1999). On the contrary, research has shown that critical thinking seems to increase only if taught explicitly (e.g., Abrami et al., 2008). It is difficult, though, for educators in economics (and other domains for

).

All rights reserved.

that matter) to derive guidelines from existing research regarding when, where and how to foster critical thinking throughout the curriculum (Smith, 2003). According to Abrami et al. (2008), existing studies on critical thinking instruction often lack a powerful empir- ical design (see also McMillan, 1987; Ten Dam & Volman, 2004; Wolcott, Baril, Cunningham, Fordham, & St.Pierre, 2002). Moreover, the definition of critical thinking or the aspects of critical thinking that are investigated tend to differ (Abrami et al., 2008).

The present studyaims to empiricallyexamine the effects of explicit critical thinking instructions on an essential aspect of critical thinking, whichis highly important todecision-making incomplex andhigh-risk situations that often occur in the field of economics: the avoidance of biased reasoning by means of more controlled rational judgment and decision-making (Evans, 2003; Smith, 2003; Stanovich & Stanovich, 2010; Tversky & Kahneman, 1983; West, Toplak, & Stanovich, 2008). Because such biases seem inherent to human reasoning and decision- making, economists are no exception and are also prone to violating the principles of rationality (Sanfey, Loewenstein, McClure, & Cohen, 2006). This study investigates whether declarative critical thinking in- struction followed by practice, either by itself or combined with self- explanation or activation prompts during practice, would enhance

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e4232

economics students’ learning of reasoning skills. Moreover, it is inves- tigated whether individual differences in thinking dispositions are associated with initial performance on those skills and with learning.

Before describing the research on critical thinking instructions and the rationale behind our study, we will first address the prin- ciples of critical thinking (especially in relation to biased reasoning) and its underlying cognitive mechanisms in more detail.

1.1. Critical thinking

The ‘American Philosophical Association Delphi Panel’ (Facione, 1990), characterized critical thinking for educational purposes (i.e., to support the instruction and assessment of critical thinking) as an extensive concept including both cognitive skills and dispositions. A critical thinker must, for example, be skilled at reasoning, which refers to the cognitive process of drawing conclusions from given information (Facione, 1990). Although the critical thinking litera- ture in educational research has focused on a wide variety of topics in the past decades (see e.g. Angeli & Valanides, 2009), little attention has been paid to the avoidance of biased reasoning (West et al., 2008). Critical thinking, in terms of avoiding biased reasoning, may be classified as a subspecies of rational thinking (Facione, 2009; Stanovich, 2011; West et al., 2008). The dual processing framework unraveled the underlying cognitive processes of rational thinking (e.g., Evans, 2003, 2008, 2011). According to the- ories of dual processing, two distinct types of reasoning processes are at work. Type 1 processes have a rapid, automatic nature and involve little reflection. Decision-making is based on past experi- ences and requires little effort, which is useful and efficient in many routine situations. However, because of its automaticity, it might also result in biased thinking in other situations, unless Type 2 processes overrule these automatic responses by explicit reasoning efforts. Type 2 processes are slow, sequential in nature, and require the exclusion of attention to other matters, and therefore draw more heavily on working memory capacity.

Stanovich (2009) distinguishes Type 2 processes into reflective and algorithmic operations. The reflective mind operates at an intentional level based on dispositions such as beliefs, cognitive style, goals, and epistemic values, which affect the algorithmic mind. Research has shown that dispositions such as ‘actively open- minded thinking’ (AOT, i.e., the active search for evidence against one’s own beliefs, plans, or goals and the ability to weigh available evidence fairly; Baron, 2008) and ‘need for cognition’ (NFC, i.e., “the tendency for an individual to engage in and enjoy thinking”; Cacioppo, Petty, & Kao, 1984, p. 306) predict performance on tasks associated with rational thinking such as syllogisms, statistical reasoning, and framing (Stanovich & West, 1997, 2007; for a review on NFC, see Cacioppo, Petty, Feinstein, & Jarvis, 1996). Individuals with higher AOT scores performed better on argument evaluation (Stanovich & West, 1997) and co-variation judgment (Sa, Kelley, Ho, & Stanovich, 2005; West et al., 2008), considered more alternative possibilities (i.e., other possibilities than initially favored), asked themselves more frequently what possibility would produce the best expected outcomes, and showed less overconfidence in hasty conclusions than individuals with lower AOT scores (Baron, 2008). NFC is characterized by cognitive motivation that predicts perfor- mance on cognitive tasks. For example, individuals with higher NFC scores have been found to recall more of the information to which they are exposed, are more focused on substantive merits of the information (e.g., are more influenced by the quality of arguments of persuasive messages), generate more task-relevant thoughts which reflect the quality of arguments, make more thoughtful judgments (i.e., scrutinize and elaborate material more) and put more effort into cognitive tasks than individuals with lower NFC scores.

The algorithmic mind performs analytic and inhibitory processes that enable a person to process information in such a way that the correct actions are taken (Stanovich, 2009). Thus, the algorithmic mind has the ability to override Type 1 processes by applying knowledge of inferential rules and strategies of rational thought (e.g. probabilistic reasoning, causal reasoning and logic). Failures to override Type 1 reasoning often occur on classical heuristics and biases tasks, which tend to evoke an automatic response, while they require causal and probabilistic reasoning, assessment of covaria- tion, a tendency to think statistically, and to think of alternative explanations. Forexample Tverskyand Kahneman (1983) illustrated poor probabilistic reasoning on a classical conjuntion task: “Linda is 31 years old, single, outspoken, and very bright. She majored in phi- losophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations. Which of the two alternatives are more probable: 1. Linda is a bank-teller or 2. Linda is a bank teller and is active in the feminist movement.” (p. 297). Most people are inclined to choose option 2, but indicating that option 2 is more probable than option 1 violates the conjunction rule because a conjunction cannot be more probable than one of its ingredients (P(A&B) � P (B)).

Although heuristics and biases tasks are largely unexploited in the traditional critical thinking literature and measurement (West et al., 2008), these tasks allow for the assessment of the degree of rationality in terms of reflective and algorithmic mechanisms (Stanovich, Toplak, & West, 2008). Failures to override Type 1 pro- cesses, demonstrated on heuristics and biases tasks (Stanovich, 2009), might occur due to lack of declarative knowledge of reasoning skills or insufficient strategies to use available knowledge. Both might presumably be counteracted by instructions.

1.2. Critical thinking instruction: avoiding biased reasoning

Relatively few studies have focused on the questions of how to avoid biased reasoning in favor of more rational thinking (Stanovich & Stanovich, 2010), of whether rational thinking is trainable, of what type of instructions would be most helpful, and of whether instructions have a persistent impact on learners’ thinking beyond the period of instruction (for reviews see Ritchart & Perkins, 2005; Stanovich, 2011).

Research that has been conducted on educational interventions revealed some evidence that explicit rule training on the law of large numbers fostered performance on base-rate problems (Fong, Krantz, & Nisbett, 1986), that explicit debiasing instructions (i.e., instructions to decouple prior beliefs and opinions from evaluation of evidence and arguments such as prompting subjects with ‘give both reasons for and reasons against’) improved performance on argument generation tasks and syllogistic reasoning tasks (Macpherson & Stanovich, 2007). Fong et al. (1986) found that both teaching the rules and guidance to apply these rules in particular domains by examples, were both effective; however, providing examples significantly improved the use of abstract rule systems. According to Nisbett, Fong, Lehman, and Cheng (1987), on some tasks (e.g. causal and conditional tasks) reasoning improvement only had an enduring effect if abstract rule training and example training were both provided.

In the academic domains of psychology, medicine, and law, Lehman, Lempert, and Nisbett (1988) showed that teaching infer- ential and logical rules improved students’ reasoning performances in domain-related tasks as well as on everyday life problems. In the financial domain, Larrick, Morgan, and Nisbett (1990) found that participants who had been instructed on cost-benefit principles, followed by using these principles on examples and indicating whether they agreed or disagreed with the reasoning in the ex- amples, became more rational on reasoning (i.e., responded more

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e42 33

normatively) compared to untrained participants and applied normative principles on a different type of problem, both imme- diately and after a full month, and in different contexts (i.e., transfer took place).

These findings on debiasing instructions are consistent with studies in the broader critical thinking literature (e.g. Abrami et al., 2008; Angeli & Valanides, 2009; Bangert-Drowns & Bankert, 1990), which show that general critical thinking instruction combined with the integration of critical thinking principles into subject matter instruction (e.g., thought provoking activities) works best for improving critical thinking.

Moreover, these findings suggest that two key-factors appear to play a role in critical thinking instruction: explicit teaching of reasoning principles and room for practicing these principles. Ac- cording to Bransford, Sherwood, Vye, and Rieser (1986), combining explicit instructions with practice allows students to integrate general rules into a domain specific context, which helps them to integrate, combine, and cluster information in a meaningful way. This would result in a conceptual and practical organization of knowledge that facilitates future performance on such tasks.

When both explicit instruction and practicing would be required for acquiring reasoning skills, then the question arises whether additional prompts during practice might have added value and if so, which kind of prompts? Two potential prompting methods might be useful and will be elaborated upon below: self- explanation and activation prompts.

1.2.1. Self-explanation prompts Regarding self-explanation, Austin, Gregory, and Chiu (2008)

found that prompting students to self-assess and reflect improved their performances on reasoning tasks. Participants who provided a brief written explanation of how they arrived at the rating and a rationalization of why they selected a particular answer, demonstrated a significantly better performance on critical thinking (i.e., less heuristic and more algorithmic reasoning), than participants who did not receive these prompts. Angeli and Valanides (2009) examined the impact of critical thinking teach- ing methods in performances on an ill-defined problem (e.g. to discuss an issue and to produce an outline for a paper) and found the highest effect size for the condition in which students discussed an issue, and reflected on their thinking combined with a short lecture about critical thinking. These findings are consistent with studies about the effectiveness of self-explanation.

Self-explanation fosters the proper use of available knowledge and skills, which engages students in active and meaningful learning while effectively monitoring their understanding (Roy & Chi, 2005). The underlying cognitive mechanisms of self- explaining have been described as “generating inferences to fill missing information, integrating information within study mate- rials, integrating new information with prior knowledge and monitoring and repairing faulty knowledge” (Roy & Chi, 2005, p. 272). According to Lombrozo (2006), explaining is especially useful for approaches to reasoning as it promotes learning by the inte- gration of new information with prior beliefs. Explanations help to assess the probability of for example claims in light of prior beliefs, evoke beliefs-revision, encourage learners to re-describe materials, and allow them to identify relevant principles (Williams & Lombrozo, 2010).

Moreover, self-explanation fosters generalizations. Aleven and Koedinger (2002) found that students who explained their steps during problem solving learned more effectively and generalize readily to novel situations, than students who did not explain their steps. Hence self-explanation might be a particularly effective strategy to promote both learning and transfer (e.g., Lombrozo, 2006; Renkl, 2005; Rittle-Johnson, 2006), even at a delay: Rittle-

Johnson (2006) found that the effects of direct instruction com- bined with self-explanation facilitated learning and transfer addi- tively and that the benefits persisted over a 2-week delay. Finally, these beneficial effects of self-explanations occur even though participants rarely receive feedback on the quality of their expla- nations (Matthews & Rittle-Johnson, 2009). Based on these studies it can be hypothesized that prompting self-explanation during practice of reasoning tasks would have an additional impact on acquisition of reasoning skills, as it helps not only to integrate ac- quired knowledge and skills (which only practising would also do), but helps to examine gaps, evoke belief-revision, and foster gen- eralizations during practice.

1.2.2. Activation prompts Activation prompts relate to the assumption that reasoning

biases are due to a inhibition failure in working memory and not to a lack of understanding of probabilistic principles or a lack of deductive competence (Moutier, Angeard, & Houdé, 2002; Moutier & Houdé, 2003). Moutier et al. showed that inhibition training led to better performance on Wason selection tasks and conjunction tasks. For instance, they used an adaptation of the classical ‘Linda task’ described above, which requires inhibition of the misleading scheme (the conjunction) and activation of the relevant scheme (the probability). A control group (no training) was compared to a strictly logical training group (in which the experimenter explained the misleading scheme and admonished subjects not to fall in a trap), and an inhibition-training group. The inhibition-training group received the logical training and in addition, participants were encouraged to redirect their attention toward logically rele- vant information. However, even though it can be effective, inhi- bition training may have the unintended side effect of sensitizing the mind to other thoughts that have to be avoided (Wenzlaff & Bates, 2000). Stanovich and Stanovich (2010) also stated that sup- pressing the initial response (Type 1 reasoning) is only helpful when a better response is available to substitute for it. In line with findings on inhibition training, an alternative means of increasing Type 2 reasoning might be the activation of the ‘weaker’ cues instead of the suppression of dominant ones (i.e., evoke an atten- tion shift from irrelevant to relevant information; Miyake, Friedman, Emerson, Witzki, & Howerter, 2000).

Thus, based on previous studies it would be reasonable to expect that activation prompts during practice would encourage an attention shift to relevant tasks aspects, which would have an additional impact on the acquisition of reasoning skills.

In sum, an important question for education, which is addressed in the present study, is whether critical thinking in terms of unbi- ased reasoning can be enhanced by explicit instructions in combi- nation with practice, and whether self-explanation or the activation prompts during practice can further enhance reasoning skills.

1.3. The present study

The present study addresses the following questions: a) What is the impact of individual differences in dispositions on economics students’ critical thinking as measured by their performance on judgment and reasoning tasks? b) What are the effects of in- structions, practice, and prompts during practice on economics students’ critical thinking as measured by their performance on reasoning tasks immediately and after a three-week delay?

Regarding dispositions, we hypothesized in line with previous research that those participants with higher scores on dispositions (i.e., AOT and NFC) would score better on an initial assessment of critical thinking skills than participants with lower scores on these dispositions (Hypothesis 1). An interesting related question is

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e4234

whether students with higher scores on dispositions would also benefit more from the critical instruction than students with lower scores on dispositions, or whether instruction is equally effective for all participants; this question is explored here (Question 1).

Regarding instructions, it is hypothesized that critical thinking instruction enhances performance on critical thinking skills compared to no instruction, both immediately (Hypothesis 2a) and after a three-week delay (Hypothesis 2b); however, based on prior research it can be expected that this beneficial effect would only arise when instruction is combined with practice. Therefore, practice is taken into account as a within-subjects factor, and it is hypothesized that only those tasks which will be instructed and practiced, would enhance reasoning skills (Hypothesis 2c).

If critical thinking instruction has a general effect, then perfor- mance on both practiced and not-practiced tasks should be enhanced compared to the control (i.e., no instruction) condition; however, if instruction only has a beneficial effect when combined with practice, only performance on practiced tasks should be enhanced compared to the control condition. Secondly, it is hy- pothesized that after instruction, combining practice with prompts (either self-explanation or activation prompts) would foster acquisition of critical thinking skills compared to instruction and practice only (i.e., without prompts) both immediately (Hypothesis 3a) and after a three-week delay (Hypothesis 3b).

2. Method

2.1. Participants and design

Participants were 195 part-time Economics students of a Dutch University of Applied Sciences with various educational back- grounds (in the Netherlands, Universities of Applied Sciences can be entered via various secondary education trajectories). Twelve participants had to be excluded because of missing data, leaving a final sample of 183 students (121 men, 62 women; age M ¼ 29.3, SD ¼ 6.5). Participants were randomly assigned to one of four conditions: 1) no critical thinking instruction (control; n ¼ 40), 2) critical thinking instruction (n ¼ 46), 3) critical thinking instruc- tion plus self-explanation prompts (n ¼ 47), and 4) critical thinking instruction plus activation prompts (n ¼ 50). The experiment consisted of 5 phases: 1) pre-test, 2) instructions (critical thinking or unrelated topic depending on condition), 3) practice of reasoning skills on a business case, 4) immediate post-test, 5) delayed post-test. Note regarding phase 3 that in each condition, practice was a within-subjects factor (i.e., some task categories were practiced during the practice phase, others were not). In the prompting conditions, the prompts were provided during practice (phase 3).

2.2. Materials and procedure

2.2.1. Critical thinking disposition tests We used a Dutch translation of the 41-item Actively Open-

minded Thinking test (AOT; Stanovich & West, 2007) and the 18-item (short form) Need for Cognition questionnaire (NFC; Cacioppo et al., 1984) to measure critical thinking dispositions. Both consist of items requiring a response on a 6-point Likert scale (AOT: as in Stanovich & West, 2007; NFC: as in West et al., 2008; note that Cacioppo et al., 1984 used a 9-point scale) ranging from (1) strongly disagree to (6) strongly agree. Scores on the items are summed for AOT and for NFC separately (after reverse scoring items that are formulated negatively). Higher scores on AOT characterize a greater tendency toward open-minded thinking. The reliability of the AOT was good: the internal consistency

(Cronbach’s alpha) was 0.82. Higher scores on the NFC represent a greater tendency to engage in and enjoy effortful cognitive en- deavors. Reliability of the NFC was also good: the internal con- sistency (Cronbach’s alpha) was 0.80.

2.2.2. Critical thinking skills tests The critical thinking skills tests consisted of sixteen tasks (see

Appendix A for an example task in each category). The tasks in the pre-test, immediate, and delayed post-test were structurally equivalent, but surface features (cover stories) differed.

The reasoning tasks examined the heuristic tendency: 1) to be influenced by intense personal and case evidence in favor of more representative statistical evidence (two causal base-rate tasks adapted from Fong et al., 1986), 2) to base judgments on prior belief and intuition without taking sufficient account of the base- rate frequency (two non-causal base-rate tasks adapted from De Neys & Glumicic, 2008), 3) to neglect simple and fundamental qualitative rules of probability in conjunction problems in which a conjunction cannot be more probable than one of its ingredients (two conjunction tasks, adapted from Tversky & Kahneman, 1983), 4) to shift outcomes when the same information is framed in different ways as in cases of gains and risky options (two framing tasks adapted from Tversky & Kahneman, 1981), 5) to evaluate the information given in a 2 � 2 contingency table unequally, that is, to base estimations on already experienced evidence and disre- gard some of the presented evidence (two covariation tasks adapted from Wasserman, Dorner, & Kao, 1990), 6) to examine the tendency to verify rules rather than to falsify them (two Wason selection tasks adapted from Stanovich, 2009, and Wason & Shapiro, 1971), and 7) to examine the tendency to evaluate the logical validity of arguments on the basis of one’s prior beliefs about the truth of conclusions on syllogistic reasoning tasks (Evans, 2003; Markovits & Nantel, 1989; Sa et al., 1999). In syllo- gistic reasoning tasks a conclusion is drawn from two given pre- mises or assumed propositions and participants have to indicate whether or not the conclusion follows logically from the premises. Different types of inconsistent (i.e., the validity of the conclusion is in conflict with its believability) syllogistic tasks were included (one of each type): affirming the antecedent or modus ponens (if p then q, p therefore q; valid), affirmation of the consequent (if p then q, q therefore p; invalid), denial of the antecedent (if p then q, not p therefore not q; invalid) and denial of the consequent, or modus tollens (if p then q, not q therefore not p; valid).

The content of the surface features (cover stories) of the tasks was adapted to the interests of students in the economics domain. The format of the tasks differed; a multiple-choice format with two to five answer options (depending on task characteristics) was used (the correct answer based upon rational reasoning strategies and incorrect answers related to biased reasoning). The internal con- sistency (Cronbach’s alpha) on the three tests was 0.50, 0.70 and 0.73, respectively. Modest reliability for the pre-test composite score could be expected and is in line with previous research (de Bruin, Parker, & Fischhoff, 2007; West et al., 2008). It should be noted that reliability on the post-tests was much higher. It is not uncommon for pre-tests to show a lower reliability as performance prior to instruction is often more random or variable than perfor- mance on post-tests after instruction.

2.2.3. Critical thinking instructions The critical thinking instructions (see Table 1 for an overview of

the study design) in the experimental conditions consisted of computer-based presentation (visual and auditory information) of 15 min. in which the features of critical thinking, its importance, the required reasoning skills, the dispositions, and the risk of biased thinking and fallacies in thinking were explained. Examples and

Table 1 Overview of study design.

Experimental conditions Video-based critical thinking instruction

Practice 4 out of 7 categories of tasks

Additional prompts on practiced tasks

Control No (unrelated video) Yes No Critical thinking instruction Yes Yes No Critical thinking þ self-explanation prompts Yes Yes Yes Critical thinking þ activation prompts Yes Yes Yes

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e42 35

demonstrations of all task categories were provided, referring back to the tasks seen in the pre-test, which could have allowed par- ticipants to mentally correct initially erroneous responses. As an example, the base-rate fallacy was demonstrated by a slide pre- senting an image of a person, not to identify as male or female, with a guitar, beer, and some engineering tools. Next to the image the visual text appears: base-rate fallacy, group: 990 women, 10 men. At the same time the following auditory information was given: “The base-rate fallacy is a thinking failure that occurs when the sta- tistical distribution of a population is ignored. For example when we select a participant randomly out of 1000 participants with 990 women and 10 men, and we will tell you that this person is called Sam, loves drinking beer and listening to hardrock music, and has graduated as mechanical engineer, then most people tend to find it most likely that this person is a man. In that case they ignore that the total group contains of 10 men only” (NB: a different name was used in the video, a Dutch name that can also be given to both men and women, but for the sake of clarity we used an English name here). The computer controlled the sequence and pace of the video, but participants could pause, forward and rewind, although observa- tions during the experiment suggested they hardly made use of these options The control group received a 15 min. digital video on an unrelated topic (i.e., what happens in your brain when you are in love).

2.2.4. Practice phase After the video-instruction all participants were exposed to a

business-case from an economics course containing a description of a coffee manufacturer who had to decide about marketing, quality control, extending the assortment, and the sustainability measures. Four categories of tasks (i.e., a contingency task, a conjunction task, a non-causal base-rate task, and a Wason selec- tion task) were practiced with a similar format as the tasks used in the tests, but the cover stories of the practice tasks were derived from the business case. Participants did not receive feedback on the quality of their performance on the practice tasks.

Participants in the control condition and the critical thinking instruction only condition performed the practice tasks without additional prompts. In the instruction plus self-explanation con- dition, prompts were given after each task to self-explain how the answer was obtained: ‘Explain by using keywords how you’ve come to the answer’. In the instruction plus activation prompts condition, prompts were given prior to the task to focus on the relevant factors. Participants were prompted with hints: ‘search for confirmation and refutation’ on the contingency task, ‘think of the logical probability’ on the conjunction task, ‘think of the statistical distribution’ on the non-causal base-rate task, and ‘violation?’ on the Wason selection task.

To measure whether mental effort invested in practicing tasks differed between conditions, a 9-point subjective rating scale ranging from (1) very, very low effort to (9) very, very high effort (Paas, 1992) was applied after each task in the practice phase. Mental effort is an indicator of actual cognitive load, and this scale

is widely used in educational research (for a review see Van Gog & Paas, 2008).

2.2.5. Procedure The experiment was run in 11 sessions in a computer room at

the university with 15e22 participants per session and all condi- tions represented in each session; participants had been randomly assigned to one of the conditions prior to the experiment. All of the materials were delivered in a computer-based environment that was created for this experiment and participants could work at their own pace. They first completed the pre-test and dispositions tests. Then, participants in the experimental conditions received the critical thinking instruction, while participants in the control condition watched the video on an unrelated topic. Subsequently, all participants read the business case and performed the practice tasks, with or without prompts depending on their assigned con- dition. Finally, they completed the immediate post-test. Three weeks later, all participants were requested via e-mail to complete the delayed post-test online.

2.2.6. Data analysis For each correct answer on the critical thinking skills pre-test,

immediate post-test, and delayed post-test, 1 point was assigned, resulting in a maximum score of 8 points for practiced tasks, and 8 points for not-practiced tasks, on each test.

Ratings on the AOT and the NFC were summed after reverse scoring negatively formulated items, resulting in a maximum score on the AOT of 246 and on the NFC of 108. Partial Eta-squared (hp

2) values were computed to estimate the magnitude of group differences prior to treatment (i.e., pretest) in test performance and critical thinking dispositions, and Cramer’s V was computed for group differences with regard to educational background. Further, Odds Ratios (ORs) were computed to estimate the magnitude of dichotomous variables that might predict dropout from immediate to delayed post-test, and Cohen’s d-values were computed to estimate the standardized difference in average age, AOT, NFC, pre-test performance, immediate post-test performance, and invested mental effort during instruction between dropouts and non-dropouts. R2 values were computed to estimate age as a predictor of post-test scores and AOT and NFC as predictors of pre- test scores. Finally, standardized regression coefficients (betas) were computed for predictor variables in the multilevel model for immediate and delayed post-test performance. To take the hier- archical structure of the data into account, not the raw test per- formance standard deviation (as linked to the standardized beta formula calculated in single-level multiple regression analysis) but the residual (adjusted) standard deviation of test performance was used to calculate the betas. The lowest level variance is then “the amount of variation in the outcome measure attributable to the individual observation after appropriate controls have been made” (Schagen & Elliot, 2004, p. 13). Although effect sizes tend to be slightly larger in the residual standard deviation approach as compared to a raw standard deviation approach, “such

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e4236

calculations are considered appropriate because they explicitly model the extent and impact of clustering in the data” (Schagen & Elliot, 2004, p. 13).

5.00

6.00

7.00

8

3. Results

For all analyzes, a significance level of 0.05 was used, except for post-hoc comparisons between conditions for which a significance level of 0.05/6 was used (note that the uncorrected p-values are reported). Random assignment of participants to conditions had been successful; there were no significant differences between conditions in critical thinking pre-test performance on practiced tasks, F(3, 179) ¼ 1.36, p ¼ 0.258, hp2 ¼ 0.02, not-practiced tasks, F(3, 179) ¼ 0.44, p ¼ 0.723, hp2 ¼ 0.01, critical thinking dispositions (AOT: F(3, 179) ¼ 0.33, p ¼ 0.803, hp2 ¼ 0.01; NFC: F(3, 179) ¼ 1.2, p ¼ 0.311, hp 2 ¼ 0 02), and educational background, c2(6) ¼ 8.38, p ¼ 0.212,

V ¼ 0.15. Regarding the following analyzes, it is worth noting that the response rate on the delayed post-test was approximately 45% (N ¼ 85): control (n ¼ 16), critical thinking instruction (n ¼ 25), critical thinking instruction plus self-explanation prompts (n ¼ 21), and critical thinking instruction plus activation prompts (n ¼ 23). Logistic regression to investigate whether participants who completed the immediate post-test only (i.e., dropouts) differed from participants who completed both the immediate and the delayed post-test (i.e., persisters) reveal that dropouts could not be predicted based on AOT scores (p ¼ 0.284, dropouts M ¼ 179.88, SD ¼ 17.09, persisters M ¼ 177.36, SD 14.45, d ¼ 0.16), NFC scores (p ¼ 0.432, dropouts M ¼ 77.98, SD ¼ 9.11, persisters M ¼ 79.05, SD 9.44, d ¼ �0.12), educational background (p ¼ 0.851, OR ¼ 1.04), gender (p ¼ 0.188, OR ¼ 0.65), pre-test performance (p ¼ 0.513, dropouts M ¼ 7.68, SD ¼ 2.31, persisters M ¼ 7.92, SD ¼ 2.55, d ¼ �0.01), immediate-post-test performance (p ¼ 0.283, dropouts M ¼ 9.11, SD ¼ 3.31, persisters M ¼ 9.60, SD ¼ 2.76, d ¼ �0.16), in- struction conditions (p ¼ 0.862, OR ¼ 0.95) or invested mental effort during instructions (p ¼ 0.342, dropouts M ¼ 3.65, SD ¼ 1.30, persisters M ¼ 3.83, SD ¼ 1.20, d ¼ �0.14). Only age was a significant predictor of dropout (p ¼ 0.004, dropouts M ¼ 30.6, SD ¼ 7.35, persisters M ¼ 27.9, SD ¼ 5.12, d ¼ 0.43), indicating a small to me- dium effect (Cohen, 1992); however this effect seems of minor importance as age was not a predictor of immediate post-test scores (p ¼ 0.089, R2 ¼ 0.02) or delayed post-test scores (p ¼ 0.076, R2 ¼ 0.04). All in all, the p-values in combination with the measures of effect size indicate that the groups who did and did not complete the delayed posttest were comparable on the re- ported variables prior to treatment.

To explore whether invested mental effort during practice differed among instruction conditions an ANOVA was conducted, which showed no significant differences among conditions, F(3, 179) ¼ 1.30, p ¼ 0.276, hp2 ¼ 0.02.

Note. Group A: control, Group B: critical thinking instruction, group C: critical thinking instruction + self- explanation prompts, Group D: critical thinking + activation prompts. Covariates in the model kept constant on their mean value: Scores on pre-test reasoning tasks, invested mental effort during instruction, Actively Open- minded Thinking, and Need for Cognition.

0.00

1.00

2.00

3.00

4.00

Immediate Post-test Delayed Post-test

S co

re s

ra ng

e 0

-

Group A Group B Group C Group D

Fig. 1. Estimated marginal means of practiced tasks on immediate and delayed post- test by instruction conditions.

3.1. Dispositions and pre-test performance

To test the hypothesis that participants with higher scores on dispositions (i.e., AOT and NFC) would score better on the pre-test of critical thinking skills than participants with lower scores on these dispositions (Hypothesis 1), a multiple regression analysis was performed. On the dispositions tests, data from 11 participants were lost due to a technical error. Their scores were replaced with the average sample scores. In line with our hypothesis, the dispo- sitions AOT and NFC significantly predicted pre-test critical thinking skills, F(2, 180) ¼ 11.55, p < 0.001, R2 ¼ 0.11, indicating a medium effect (Cohen, 1992). Regarding the standardized co- efficients the AOT scores showed a slightly higher impact, b ¼ 0.22, t(180) ¼ 2.95, p ¼ 0.004, than the NFC scores, b ¼ 0.20, t(180) ¼

2.64, p ¼ 0.009, indicating a small to medium effect (Lipsey & Wilson, 2001).

3.2. Effects of instructions and dispositions on immediate and delayed post-test performance

To include not only participants who completed the immediate and the delayed test but to use as much information as possible, a multilevel regression model was used. The following fixed factors were included in the model consistently: pre-test scores (as co- variate), dispositions scores (i.e., AOT and NFC), invested mental effort scores during instruction, instruction conditions, practice (i.e., practiced tasks versus not practiced tasks), and test moment (i.e., immediate post-test scores and delayed post-test scores). A student-level random intercept and random slope for practice were included in the model. No abnormal departures from normality or outliers were found. Figs. 1 and 2 display the mean reasoning scores of practiced and not-practiced tasks in each condition on the immediate and delayed post-test. Table 5 pre- sents the adjusted means along with their standard errors and 95% confidence intervals for every combination of condition by prac- tice by test moment, keeping all quantitative covariates constant at their mean value.

Regarding the explorative question of whether students with higher scores on dispositions would also benefit more from the critical instruction than students with lower scores on disposi- tions, or whether instruction would be equally effective for all participants (Question 1), Table 2 shows that the dispositions and critical thinking performance were correlated significantly even after instruction (i.e., mainly between AOT and post-test scores), however the multilevel regression analysis, using Satterthwaite approximation for the degrees of freedom in SPSS version 21 (Table 3), revealed that no interactions were found between scores on dispositions (i.e., AOT and NFC) and instructions, between dispositions and practice, or between dispositions and test moment.

The hypotheses that instruction would have an effect on post- test performance (Hypothesis 2a) when combined with practice (Hypothesis 2c), that prompts during practice would further enhance the effects of instructions and practice (Hypothesis 3a), and that these effects remained the same after a three-week delay (Hypothesis 2b and 3b), were also tested by means of the

Table 3 Fixed effects for predictors of reasoning scores.

Parameter df F p

Intercept 1186.06 43.63 <0.001 Conditions 3289.58 3.94 0.009 Practice 1302.88 3.17 0.076 Test moment 1255.56 51.95 <0.001 Conditions � practice 3302.20 10.85 <0.001 Conditions � test moment 3254.98 1.10 0.351 Conditions � practice � test moment 3259.66 0.28 0.842 Practice � test moment 1260.12 60.93 <0.001 Pre-test reasoning scores 1166.67 82.39 <0.001 Invested mental effort 1173.95 4.62 0.033 NFC 1262.90 0.11 0.747 AOT 1267.42 1.96 0.163 NFC � test moment 1255.12 1.04 0.309 AOT � test moment 1270.45 0.85 0.358 NFC � conditions 3166.92 0.96 0.414 AOT � conditions 3178.35 1.95 0.124 NFC � practice 1217.96 1.50 0.223 AOT � practice 1235.33 2.55 0.112

Note. Intercept based on not-practiced tasks on the immediate post-test; Practice: practiced tasks (code ‘1’) versus not-practiced tasks (code ‘0’); Test moment: im- mediate post-test (code ‘0’) versus delayed post-test (code ‘1’). AOT: Actively Open- minded Thinking, NFC: Need for Cognition, Conditions: control, critical thinking instruction, critical thinking instruction þ self-explanation prompts, critical thin- king þ activation prompts.

Note. Group A: control, Group B: critical thinking instruction, group C: critical thinking instruction + self- explanation prompts, Group D: critical thinking + activation prompts. Covariates in the model kept constant on their mean value: Scores on pre-test reasoning tasks, invested mental effort during instruction, Actively Open- minded Thinking and Need for Cognition.

0.00

1.00

2.00

3.00

4.00

5.00

6.00

Immediate Post-test Delayed Post-test

S co

re s

ra ng

e 0-

8

Group A Group B Group C Group D

Fig. 2. Estimated marginal means of not-practiced tasks on immediate and delayed post-test by instruction conditions.

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e42 37

multilevel regression analysis (Table 3). This analysis showed significant main effects of instruction condition and test moment, and significant interaction effects between instruction condition and practice, and between test moment and practice. Table 4 shows the B-values for the main and interaction effects and the effect sizes (betas). Note that the intercept is based on not- practiced tasks of the immediate post-test in the control condi- tion (i.e., group A). The betas for pre-test reasoning scores and invested mental effort indicate that pre-test reasoning score has a strong positive effect on immediate and delayed post-test per- formance, while invested mental effort during instruction appears to have a small to moderate negative effect on performance. In line with the significance tests presented in Table 3, the signifi- cance tests and betas in Table 4 appear to indicate that NFC and AOT hardly influence post-test performance. Further, the condi- tion by practice interaction appears to be a strong effect; in the control condition, immediate post-test performance is consider- ably lower on practiced tasks (notice the negative beta), while in the other three conditions immediate post-test performance is much higher on the practiced tasks than on the not practiced tasks.

To investigate the interaction effect between condition and practice further, simple effect analyzes taking into account the factors practice (i.e., practiced versus not-practiced tasks) and test moment (i.e., immediate and delayed post-test), revealed that participants in the instruction conditions outperformed the con- trols on practiced tasks on the immediate post-test (all p < 0.001) and on the delayed post-test (all p smaller than 0.002). LSD post-

Table 2 Correlations between critical thinking performance and dispositions (AOT and NFC) on pre

1 2

1. AOT e 2. NFC 0.321*** e 3. Pre-test practiced tasks 0.220** 0.173* 4. Pre-test not-practiced tasks 0.244** 0.263*** 5. Immediate post-test practiced-tasks 0.323*** 0.192** 6. Immediate post-test not-practiced tasks 0.259*** 0.114 7. Delayed post-test practiced tasks 0.330** 0.123 8. Delayed post-test not-practiced tasks 0.222* 0.106

Note. AOT: actively open-minded thinking scores, NFC: need for cognition scores, *** Signi

hoc tests on practiced tasks on the immediate and delayed post-test revealed no significant differences between instruction conditions (immediate test: all p larger than 0.484, delayed test: all p larger than 0.053). On not-practiced tasks, only participants in the instruction condition with self-explanation prompts outperformed those in the control condition on the immediate post-test (p < 0.001) but not on the delayed post-test (p ¼ 0.199). LSD post- hoc tests showed that other comparisons between conditions on not-practiced tasks were not significant on either the immediate post-test (all p larger than 0.020) or the delayed post-test (all p larger than 0.030). Note that the uncorrected p-values are reported, but these are no longer significant after correction for multiple tests (i.e., 0.05/6).

Finally, the practice by test moment interaction (Table 4) in- dicates that there is a decrease in performance on the practiced tasks from immediate to delayed post-test whereas there is no change in performance on not-practiced tasks; the beta of practice by test moment suggests that this is a strong effect. Note though (Table 3), that there was no significant interaction with condition.

4. Discussion

The aim of this study was to examine a) the impact of individual differences in dispositions and b) effects of instructions, practice,

-test, immediate post-test and delayed post-test of practiced and not-practiced tasks.

3 4 5 6 7

e

0.364*** e 0.379*** 0.317*** e 0.341*** 0.502*** 0.440*** e 0.240* 0.198 0.653*** 0.434*** e 0.283** 0.430*** 0.355** 0.655*** 0.428***

ficant at the 0.001 level, ** Significant at the 0.001 level, * Significant at the 0.05 level.

Table 4 Fixed effects estimates and covariance estimates of reasoning scores.

Parameter B (SE) df t p Effect size (beta)

Level 1 Fixed effects Intercept 1.98 (0.43) 208.38 4.62 <0.001 Group B 0.52 (0.29) 289.85 1.81 0.072 0.48 Group C 0.99 (0.29) 289.48 3.44 <0.001 0.91 Group D 0.55 (0.28) 288.78 1.94 0.054 0.51 Practice �1.09 (0.29) 302.01 �3.74 <0.001 �1.01 Test moment 0.92 (0.35) 262.03 2.66 0.008 0.85 Practice � test moment �1.49 (0.48) 267.24 �3.11 0.002 �1.38 Group B � practice 1.86 (0.40) 302.10 4.66 <0.001 1.72 Group C � practice 1.41 (0.40) 301.15 3.53 <0.001 1.30 Group D � practice 2.07 (0.39) 302.64 5.31 <0.001 1.91 Group B � test moment 0.63 (0.45) 256.05 1.40 0.161 0.58 Group C � test moment �0.02 (0.46) 259.51 �0.05 0.962 �0.02 Group D � test moment �0.11 (0.45) 258.31 �0.24 0.807 0.10 Group B � practice � test moment 0.01 (0.62) 259.53 0.02 0.983 0.01 Group C x Practice � Test moment �0.28 (0.64) 264.55 �0.44 0.664 �0.26 Group D � practice � test moment �0.44 (0.63) 264.58 �0.70 0.487 �0.40 Pre-test reasoning scores 0.32 (0.04) 166.67 9.08 <0.001 0.71 Invested mental effort �0.14 (0.07) 173.95 �2.15 0.033 �0.16 NFC 0.01 (0.02) 198.80 0.52 0.602 0.09 AOT �0.01 (0.01) 213.56 �1.18 0.241 �0.18 Group B � AOT 0.03 (0.02) 169.30 1.78 0.076 0.40 Group C � AOT 0.04 (0.01) 186.69 2.07 0.040 0.45 Group D � AOT 0.03 (0.01) 177.66 1.91 0.058 0.40 Group B � NFC �0.02 (0.03) 163.57 �0.60 0.552 �0.14 Group C � NFC �0.04 (0.03) 175.93 �1.46 0.146 �0.33 Group D � NFC �0.00 (0.03) 167.16 �0.09 0.932 �0.02 AOT � practice 0.01 (0.01) 235.33 1.60 0.112 0.19 NFC � practice 0.02 (0.01) 217.96 1.22 0.223 0.15 AOT � test moment 0.01 (0.01) 270.45 0.92 0.358 0.11 NFC � test moment �0.01 (0.01) 255.12 �1.02 0.309 �0.11 Level 2 Random parameters

cov (SE) Wald Z p Intercepts participants 0.53 (0.13) 4.25 <0.001 Practice participants 1.05 (0.26) 4.02 <0.001 Residuals 1.17 (0.11) 10.21 <0.001

Note. Intercept based on not-practiced tasks on the immediate post-test; Practice: practiced tasks (code ‘1’) versus not-practiced tasks (code ‘0’); Test moment: immediate post-test (code ‘0’) versus delayed post-test (code ‘1’). Conditions: group A: control condition, group B: CT instruction, group C: CT instruction þ self-explanation prompts, group D: CT-Instruction þ activation prompts; NFC: Need for Cognition; AOT: Actively Open-minded Thinking. Effect sizes: 0.1 small; 0.25 medium, 0.40 large (Lipsey & Wilson, 2001).

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e4238

and prompts on economics students’ critical thinking as measured by their performance on reasoning tasks on an immediate and delayed post-test.

In line with our first hypothesis (Hypothesis 1), the results clearly indicate that those participants with higher scores on dis- positions (i.e., AOT and NFC) scored significantly better on the

Table 5 Estimated marginal means, standard errors and 95% confidence intervals of prac- ticed and not-practiced task categories on the immediate and delayed post-test by groups.

Group Test-moment

Immediate post-test Delayed post-test

M SE CI M SE CI

Practiced tasks A 4.83 0.34 [4.17, 5.50] 3.98 0.23 [3.53, 4.43] B 5.95 0.27 [5.42, 6.49] 4.64 0.21 [4.04, 4.88] C 5.87 0.29 [5.29, 6.44] 4.94 0.21 [4.52, 5.36] D 5.60 0.28 [5.05, 6.16] 4.50 0.21 [4.09, 4.90]

Not-practiced tasks

A 2.29 0.34 [1.63, 2.96] 2.88 0.23 [2.43, 3.33] B 5.31 0.27 [4.77, 5.84] 5.23 0.21 [4.81, 5.65] C 4.40 0.29 [3.82, 4.98] 5.26 0.21 [4.83, 5.68] D 4.55 0.28 [3.40, 5.11] 5.47 0.21 [5.07, 5.87]

Note. Group A: control, Group B: instruction only, Group C: instruction and self- explanation prompts, Group D: instruction and activation prompts. Covariates in the model kept constant on their mean value: Scores on pre-test reasoning tasks, invested mental effort during instruction, AOT: Actively Open-minded, Thinking NFC: Need for Cognition.

initial assessment of critical thinking skills than participants with lower scores on dispositions. These results converge with findings in other studies on argument evaluation (Stanovich & West, 1997), syllogistic reasoning (Macpherson & Stanovich, 2007), and covari- ation judgment (Sa et al., 2005; West et al., 2008). Regarding the question of whether students with higher scores on dispositions would benefit more from the critical thinking instruction than students with lower scores on dispositions (Question 1), our results showed no significant interaction effects between instruction conditions and dispositions (i.e., AOT and NFC). Thus, it seems that despite the relationship between dispositions and pre-test perfor- mance, all students benefitted equally from instructions in terms of post-test performance; students who score higher on dispositions did not benefit more from instructions than students who score lower on dispositions or vice versa. These results indicate that the active search for evidence against one’s own beliefs, plans, or goals, and the ability to weigh available evidence fairly (i.e., AOT; Baron, 2008), and the intrinsic cognitive motivation (i.e., NFC; Cacioppo et al., 1996), which are both important features of the reflective mind (Stanovich, 2011), appear to regulate reasoning, but not learning to reason. These explorative findings suggest that dispo- sitions might be overruled by extrinsic factors in the educational context; declarative instruction to search for general and under- lying principles and the instruction to give considerable thoughts to the instructions seems to have canceled out any influence of

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e42 39

AOT and NFC on learning. These results indicate that instructions were effective for all participants regardless of their disposition scores.

Our second hypothesis was that critical thinking instruction would enhance performance on critical thinking skills compared to no instruction, both immediately (Hypothesis 2a) and after a three- week delay (Hypothesis 2b), though only when combined with practice (Hypothesis 2c). Indeed, it seems that the combination of instructions with practice is crucial and has a large effect: on both the immediate and the delayed post-test the improvements on reasoning skills compared to the control condition were found on task categories that were practiced with the business case, but not on task categories that had not been practiced. There was one exception: on the immediate test, the self-explanation condition performed better than the control condition on not-practiced tasks. We will return to this finding below, when discussing effects of prompting (Hypothesis 3). Our results are in line with and extend findings from previous studies in which it was found that explicit instructions improved critical thinking (e.g., Abrami et al., 2008; Angeli & Valanides, 2009), that rational thinking to override Type 1 processes is trainable through explicit instruction (e.g., Nisbett et al., 1987) and that even short instructions can have a persistent impact on becoming more rational at reasoning (Larrick et al., 1990). On the other hand, our results also show that critical thinking instruction without practice was not sufficient to lead to sustained effects and that practice had an added value for reasoning instruction. Practice seems to play a key role and presumably leads to better learning by allowing participants to integrate and elabo- rate information from the instructions (e.g., Bransford et al., 1986) with tasks in the economics context. Interestingly, mental effort during practice did not differ between conditions, and combined with the differences in post-test scores on practiced tasks, this suggests that the processes engaged in during practice differ as a function of instructions, but that the cognitive demands imposed by those processes do not differ (i.e., practice becomes more efficient).

The results only lend partial support to our third hypothesis that after instruction, combining practice with prompts, would foster acquisition of critical thinking skills compared to instruction and practice only both immediately (Hypothesis 3a) and after a three-week delay (Hypothesis 3b). Activation prompts, which we expected to be able to affect critical thinking performance by redirecting attention to relevant cues, did not lead to better test performance than instructions only. This finding might indicate that instructions were sufficient to redirect attention and inhibit initial automatic responses during practice. Self-explanation prompts, which we expected to foster the proper use of avail- able knowledge (Roy & Chi, 2005) also did not foster performance on practiced reasoning tasks compared to instructions only. Interestingly, however, participants in this condition performed better on not-practiced reasoning tasks on the immediate post- test than the control condition, suggesting that prompting self- explanations established a kind of transfer from practiced to not-practiced tasks, at least in the short run (this effect was no longer present on the delayed test). This result converges with studies on the benefits of self-explanation of problem solving tasks that showed that self-explanation can foster transfer (e.g., Aleven & Koedinger, 2002; Lombrozo, 2006; Renkl, 2005; Rittle- Johnson, 2006). However, given that the difference with the other instruction conditions was not significant, and that the effect was no longer present on the delayed test, this effect should be interpreted with caution and should be replicated in future research. It might also be worthwhile to attempt to deepen instructional effects of self-explanation in future studies by

explicitly teaching it as a meta-strategy or by providing feedback on self-explanations, as this might further enhance the effects of self-explanation on critical thinking and potentially on transfer to not-practiced tasks.

In sum, based on these findings, we can conclude that dis- positions only have an impact on pre-test reasoning perfor- mance but not on learning of reasoning skills. Secondly we can conclude that explicit reasoning instructions should be com- bined with opportunities for practice to have an effect on crit- ical thinking performance. Thirdly, prompting self-explanation during practice might be an interesting instructional method as we found some indications of transfer in terms of performance benefits on not-practiced reasoning tasks compared to the control condition; however, further research would be necessary to establish the merits of this method for teaching reasoning skills.

This study has some limitations. First, for practical reasons it was not possible to administer the delayed post-test at the institute as the rest of the experiment, however the web-based environment that was used for the delayed post-test was the same as the one used during the session at the institute. Sec- ondly, despite the fact that students could complete the delayed test wherever they wanted, there was a rather high attrition rate on the delayed test. However, it should be noted that participants who did complete the delayed test did not differ from partici- pants who did not on pre-test performance, dispositions, educational background, gender, invested mental effort during instruction, and performance on the immediate test. Third, we did not include conditions that received the prompts during practice but no critical thinking instruction. The reason for that was that it can be questioned whether prompts would have a beneficial effect for students lacking knowledge about the tasks (i.e., without instructions first). As Stanovich and Stanovich (2010) stated, on the tasks we used, suppressing the initial response (Type 1 reasoning) is only helpful when a better response is available to substitute for it. And regarding self- explanations it has been shown that without any knowledge, it would be unlikely that students would be able to provide deep, principled explanations, which have been shown to be the most effective (Renkl, 1997). Under conditions of very low prior knowledge, instructional explanations have been shown to be most effective (Renkl, 2002). The instructions provided students with a basis for responding to the prompts, which they would not have had otherwise. Given the low effectiveness of prompt- ing even after instructions, it seems highly unlikely that prompting without instructions first would have had an effect. Nevertheless, we did not directly test this, and future research might resolve the question of whether self-explanation or acti- vation prompts have an impact on reasoning improvements without prior critical thinking instruction. Finally, it should be kept in mind that this study defined critical thinking in terms of reasoning skills, and that the findings therefore may not gener- alize to other definitions of critical thinking.

Despite these limitations, this study provided promising results for (economics) educators who wish to enhance their students’ critical thinking skills, by showing that such skills can be enhanced through relatively simple explicit in- structions eprovided they are combined with practice. The findings are especially promising given that this experimental study was of relatively short duration; teaching critical thinking throughout the economics curriculum might help students to learn to avoid biased reasoning and better prepare them for decision-making in dynamic and complex business environments.

Appendix A. Example of each category of tasks used on critical thinking tests (* [ right option; 1 [ category practiced; 2 [ category not practiced).

Conjunction task (1)

A kitchen manufacturer wants to reposition itself in the kitchen industry. The goal is to increase the market share to 10%. Based on a strength/weakness analysis, measures have been taken to improve market share, pay more attention to the entrepreneurship of the employees, and conduct a more value-oriented communication campaign towards customers. Which option is most likely? Option 1. The market share of the kitchen manufacturer will increase by 3%.* Option 2: The market share of the kitchen manufacturer will increase by 5%. Option 3: The market share of the kitchen manufacturer will increase by 3%, and the satisfaction of customers and employees will improve. Option 4: The market share of the kitchen manufacturer will increase by 5%, and the satisfaction of customers and employees will improve.

Explanation: Options 2, 3 and 4 violate the conjunction rule as a conjunction cannot be more probable than one of its constituents.

Contingency task (1)

An insurance company claimed that too much time was spent on advising customers without any results. Costs and benefits were not balanced. Applying a new advising system should improve this situation. The new system was used on a part of the customers. A student was assigned to evaluate the effect of the new system to determine the systems’ time efficiency. The student obtained the information below and concludes that the application of the new system resulted in saving of time.

New system used New system not used Saving time 325 260 No time saving 90 55

Table: number of customers on which the system was applied or not and time was saved or not. Do you agree with the student? Option 1. fully agree Option 2. agree Option 3. neither agree nor disagree Option 4. disagree* Option 5. fully disagree*

Explanation: Option 1, 2 and 3 refer to the tendency to evaluate the information given in a 2 x 2 contingency table unequally.

Non-causal

base-rate task (1)

A renowned regional company has two vacancies for junior economists. This company has very good experiences with economics master graduates from a specific University for over 15 years. The company management will continue this policy of hiring graduates from this University. A new personnel manager, however, suggests attracting economics graduates of a Polytechnic University. The manager argued that he gained outstanding experiences in a business where he worked formerly with a polytechnic graduate who functioned at an excellent level from the outset. The personnel manager believes that polytechnic graduates work equally well as master graduates from the specific University. What should the management of the company best decide? Option 1: Definitely chose for master graduates of the specific University.* Option 2: Probably chose for master graduates of the specific University.* Option 3: Probably chose for graduates of the Polytechnic University. Option 4: Definitely chose for graduates of the Polytechnic University.

Explanation: People who choose option 3 or 4 neglect the base-rate, for example motivated by personal and case evidence in favor of more representative statistical evidence.

Causal

base-rate task (2)

A study had 1000 participants. Among the participants there were 25 men and 975 women. Sam is a randomly chosen participant in this study. Sam is 23 years old, graduated as mechanical engineer and enjoys going out with friends, likes drinking beer and is a fan of hard rock music. Which option is most probable? Option 1: Sam is a man. Option 2: Sam is a woman.*

Explanation: Option 1 demonstrates the tendency to base judgments on prior belief and intuition rather than on logical reasoning (i.e., taking into account the prior probability).

Framing task (2)

Imagine that you face the following pair of concurrent decisions: First examine both decisions, then indicate the options you prefer.

A. A sure gain of 480 euro. B. 25% chance to gain 2000 euro and 75% chance to gain nothing.

C. A sure loss of 1500 euro. D. 75% chance to lose 2000 and 25% chance to lose nothing.

Options: AC, BC*, AD, BD

Explanation: The paired choices were presented together but the problem was ‘framed’ as a pair of separate choices. The combination B&C is superior: 25% chance to win 500 and 75% chance to lose 1500. For example, with A&D there is 25% chance to win 480 and 75% chance to lose 1520. In case of gains people tend to chose risk aversive (option A) and in case of losses to chose risk seeking (option D).

Wason selection task (1)

Each of the tickets below has a destination on one side and an airline on the other side. Here is a rule: If ‘Barcelona’ is on one side of the ticket, then ‘Ryanair’ is on the other side of the ticket. Your task is to decide which tickets you would need to turn over in order to find out whether or not the rule is being violated.

Destination Barcelona

Destination Madrid

Airline Ryanair

Airline Transavia

A B C D Options: AC, AD*, BC, BD

Explanation: People who chose other options than AD probably fail to apply logical principles, verify rules rather than to falsify them, or demonstrate matching bias by selecting options explicitly mentioned in the conditional statement.

Syllogistic reasoning tasks (2)

The category syllogistic reasoning tasks exist of 4 types of tasks. In the following assignments, you will be given two premises, which you must assume are true. A conclusion from the premises then follows. You must decide whether the conclusion follows logically from the premises or not.

Premises: All oil companies are quoted companies. Shell is a quoted company. Conclusion: Shell is an oil company. Option 1. Conclusion follows logically from premises. Option 2. Conclusion does not follow logically from premises.*

Explanation:

Syllogism type: Affirmation of consequent, invalid

Premises: All mammals walk. Dolphins are mammals. Conclusion: Dolphins walk. Option 1. Conclusion follows logically from premises.* Option 2. Conclusion does not follow logically from premises.

Explanation:

Syllogism type: Affirmation of antecedent/Modus Ponens, valid

Premises: All things that move love water. Cats do not love water. Conclusion: Cats do not move. Option 1. Conclusion follows logically from premises.* Option 2. Conclusion does not follow logically from premises.

Explanation:

Syllogism Type: Denial of consequent/ Modus Tollens, valid

Premises: All oil countries are rich. Belgium is not an oil country. Conclusion: Belgium is not rich. Option 1. Conclusion follows logically from premises. Option 2. Conclusion does not follow logically from premises.*

Explanation:

Syllogism type: Denial of antecedent, invalid

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e42 41

References

Abrami, P. C., Bernard, R. M., Borokhovski, E., Wade, A., Surkes, M. A., Tamim, R., & Zhang, D. (2008). Instructional interventions affecting critical thinking skills and dispositions: a stage 1 meta-analysis. Review of Educational Research, 78, 1102e1134. http://dx.doi.org/10.3102/0034654308326084.

Aleven, A. W., & Koedinger, K. R. (2002). An effective meta-cognitive strategy: learning by doing and explaining with a computer-based cognitive tutor. Cognitive Science, 26, 147e179. http://dx.doi.org/10.1207/s15516709cog2602_1.

Angeli, C., & Valanides, N. (2009). Instructional effects on critical thinking: perfor- mance on ill-defined issues. Learning and Instruction, 19, 322e334. http:// dx.doi.org/10.1016/j.learninstruc.2008.06.010.

Austin, Z., Gregory, P. A., & Chiu, S. (2008). Use of reflection-in-action and self- assessment to promote critical thinking among pharmacy students. American Journal of Pharmaceutical Education, 72, 1e8.

Bangert-Drowns, R. L., & Bankert, E. (1990). Meta-analysis of explicit instruction for critical thinking. Boston, MA: Paper presented at the annual meeting of the American Educational Research Association.

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e4242

Baron, J. (2008). Actively open-minded thinking. In J. Baron (Ed.), Thinking and deciding (pp. 199e232). New York: Cambridge University Press.

Bransford, J., Sherwood, R., Vye, N., & Rieser, J. (1986). Teaching thinking and problem-solving. American Psychologist, 41, 1078e1089. http://dx.doi.org/ 10.1037/0003-066X.41.10.1078.

de Bruin, W., Parker, A. M., & Fischhoff, B. (2007). Individual differences in adult decion-making competence. Journal of Personality and Social Psychology, 92, 938e956. http://dx.doi.org/10.1037/0022-3514.92.5.938.

De Neys, W., & Glumicic, T. (2008). Conflict monitoring in dual process theories of thinking. Cognition, 106, 1248e1299. http://dx.doi.org/10.1016/j.cognition. 2007.06.002.

Cacioppo, J. T., Petty, R. E., Feinstein, J. A., Blair, G., & Jarvis, W. (1996). Dispositional differences in cognitive motivation: the life and times of individuals varying in need for cognition. Psychological Bulletin, 119, 197e253. http://dx.doi.org/ 10.1037/0033-2909.119.2.197.

Cacioppo, J. T., Petty, R. E., & Kao, C. F. (1984). The efficient assessment of need for cognition. Journal of Personality Assessment, 48, 306e307. http://dx.doi.org/ 10.1207/s15327752jpa4803_13.

Cohen, J. (1992). Quantitative methods in psychology; a power primer. Psychological Bulletin, 112, 155e159. http://dx.doi.org/10.1037/0033-2909.112.1.155.

Evans, J. St. B. T. (2003). In two minds: dual-process accounts of reasoning. Trends in Cognitive Sciences, 7, 454e459. http://dx.doi.org/10.1016/j.tics.2003.08.012.

Evans, J. St. B. T. (2008). Dual processing account of reasoning, judgment and social cognition. Annual Review of Psychology, 59, 255e278. http://dx.doi.org/10.1146/ annurev.psych.59.103006.093629.

Evans, J. St. B. T. (2011). Dual-process theories of reasoning: contemporary issues and developmental applications. Developmental Review, 31, 86e102. http:// dx.doi.org/10.1016/j.dr.2011.07.007.

Facione, P. A. (1990). Critical thinking: a statement of expert consensus for purposes of educational assessment and instruction. Millbrae, CA: The California Academic Press.

Facione, P. A. (2009). Critical thinking: what it is and why it counts. Retrieved from http://www.insightassessment.com/9articles%20WW.html.

Fong, G. T., Krantz, D. H., & Nisbett, R. E. (1986). The effect of statistical training on thinking about everyday problems. Cognitive Psychology, 18, 253e292. http:// dx.doi.org/10.1016/0010-0285(86)90001-0.

Halpern, D. F. (1998). Teaching critical thinking for transfer across domains: dis- positions, skills, structure training, and metacognitive monitoring. American Psychologist, 53, 449e455. http://dx.doi.org/10.1037/0003-066X.53.4.449.

Halpern, D. F. (1999). Teaching for critical thinking: helping college students develop the skills and dispositions of a critical thinker. New Directions for Teaching and Learning, 80, 69e74. http://dx.doi.org/10.1002/tl.8005.

Helsdingen, A., Van Gog, T., & Van Merriënboer, J. J. G. (2011). Effects of practice schedule and critical thinking instruction on learning and transfer of a complex judgment task. Journal of Educational Psychology, 103, 383e398. http:// dx.doi.org/10.1037/a0022370.

Jones, A. (2007). Multiplicities or manna from heaven? Critical thinking and the disciplinary context. Australian Journal of Education, 51, 84e103.

Klebba, J. M., & Hamilton, J. G. (2007). Structured case analysis: developing critical thinking skills in a marketing case course. Journal of Marketing Education, 29, 132e139. http://dx.doi.org/10.1177/0273475307302015.

Larrick, R. P., Morgan, J. N., & Nisbett, R. E. (1990). Teaching the use of cost-benefits reasoning in everyday life. Psychological Science, 1, 362e370. http://dx.doi.org/ 10.1111/j.1467-9280.1990.tb00243.x.

Lehman, D. R., Lempert, R. O., & Nisbett, R. E. (1988). The effects of graduate training on reasoning. American Psychologist, 43, 431e442. http://dx.doi.org/10.1037/ 0003-66X.43.6.431.

Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. London: Sage. Lombrozo, T. (2006). The structure and function of explanations. Trends in Cognitive

Sciences, 10, 464e470. http://dx.doi.org/10.1016/j.tics.2006.08.004. Macpherson, R., & Stanovich, K. E. (2007). Cogitive ability, thinking dispositions, and

instructional set as predictors of critical thinking. Learning and Individual Dif- ferences, 17, 115e127. http://dx.doi.org/10.1016/j.lindif.2007.05.003.

Markovits, H., & Nantel, G. (1989). The belief-bias effect in the production and evaluation of logical conclusions. Memory & Cognition, 17, 11e17. http:// dx.doi.org/10.3758/BF03199552.

Matthews, P., & Rittle-Johnson, B. (2009). In pursuit of knowledge: comparing self- explanations concepts and procedures as pedagogical tools. Journal of Experi- mental Child Psychology, 102, 1e21. http://dx.doi.org/10.1016/j.jecp.2008.08.004.

McMillan, J. H. (1987). Enhancing college students’ critical thinking: a review of studies. Research in Higher Education, 26, 3e29. http://dx.doi.org/10.1007/ BF00991931.

Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., & Howerter, A. (2000). The unity and diversity of executive functions and their contributions to complex ’frontal lobe’ tasks: a latent variable analysis. Cognitive Psychology, 41, 49e100. http://dx.doi.org/10.1006/cogp.1999.0734.

Moutier, S., Angeard, N., & Houdé, O. (2002). Deductive reasoning and matching- bias inhibition training: evidence from a debiasing paradigm. Thinking and Reasoning, 8, 205e224. http://dx.doi.org/10.1080/13546780244000033.

Moutier, S., & Houdé, O. (2003). Judgement under uncertainty and conjunction fallacy inhibition training. Thinking and Reasoning, 9, 185e201. http://dx.doi.org/ 10.1080/13546780343000213.

Nisbett, R. E., Fong, G. T., Lehman, D. R., & Cheng, P. W. (1987). Teaching reasoning. Science, 238, 625e631. http://dx.doi.org/10.1126/science.3672116.

Paas, F. (1992). Training strategies for attaining transfer of problem solving skills in statistics: a cognitive-load approach. Journal of Educational Psychology, 84, 429e 434. http://dx.doi.org/10.1037/0022-0663.84.4.429.

Renkl, A. (1997). Learning from worked-out examples: a study on individual differ- ences. Cognitive Science, 21, 1e29. http://dx.doi.org/10.1207/s15516709cog2101_1.

Renkl, A. (2002). Worked-out examples: instructional explanations support learning by self-explanations. Learning and Instruction, 12, 529e556. http:// dx.doi.org/10.1016/S0959-4752(01)00030-5.

Renkl, A. (2005). The worked-out principle in multi-media learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 229e246). Cam- bridge: Cambridge University Press.

Ritchart, R., & Perkins, D. N. (2005). Learning to think: the challenges of teaching thinking. In K. J. Holyoak, & R. G. Morrison (Eds.), The Cambridge handbook of thinking and reasoning (pp. 775e802). New York: Cambridge University Press.

Rittle-Johnson, B. (2006). Promoting transfer: effects of self-explanation and direct instruction. Child Development, 77, 1e15. http://dx.doi.org/10.1111/j.1467- 8624.2006.00852.x.

Roy, M., & Chi, M. T. (2005). The self-explanation principle in multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 271e 286). Cambridge: Cambridge University Press.

Sa, W., Kelley, C., Ho, C., & Stanovich, K. E. (2005). Thinking about personal theories: individual differences in the coordination of theory and evidence. Personality and Individual Differences, 38, 1149e1161. http://dx.doi.org/10.1016/j.paid.2004. 07.012.

Sa, W. C., West, R. F., & Stanovich, K. E. (1999). The domain specificity and generality of belief bias: searching for a generalizable critical thinking skill. Journal of Educa- tional Psychology, 91, 497e510. http://dx.doi.org/10.1037/0022-0663.91.3.497.

Sanfey, A. G., Loewenstein, G., McClure, S. M., & Cohen, J. D. (2006). Neuro- economics: cross-currents in resaerch on decision-making. Trends in Cognitive Sciences, 10, 108e116. http://dx.doi.org/10.1016/j.tics.2006.01.009.

Schagen, I., & Elliot, K. (2004). But what does it mean? The use of effect sizes in educational research. London: National Foundation for Educational Research.

Smith, G. F. (2003). Beyond critical thinking and decision making: teaching business students how to think. Journal of Management Education, 27, 24e51. http:// dx.doi.org/10.1177/1052562902239247.

Stanovich, K. E. (2009). Distinguishing the reflective algorithmic and autonomous minds: is it time for a tri-process theory? In J. Evans, & K. Frankish (Eds.), In two minds: Dual processess and beyond (pp. 55e88) Oxford: Oxford University Press.

Stanovich, K. E. (2011). Rationality & the reflective mind. New York: Oxford Univer- sity Press.

Stanovich, K. E., & Stanovich, P. J. (2010). A frame-work for critical thinking, rational thinking and intelligence. In D. D. Preiss, & R. J. Sternberg (Eds.), Educational psychology; Perspectives on learning, teaching and human development (pp. 195e 237). New York: Springer Publishing Company.

Stanovich, K. E., Toplak, M. E., & West, R. F. (2008). The development of rational thought: a taxonomy of heuristic and biases. Advances in Child Development and Behavior, 36, 251e285. http://dx.doi.org/10.1016/s0065-2407(08)000062.

Stanovich, K. E., & West, R. F. (1997). Reasoning independently of prior belief and individual differences in actively open-minded thinking. Journal of Educational Psychology, 89, 342e357. http://dx.doi.org/10.1037/0022-0663.89.2.342.

Stanovich, K. E., & West, R. F. (2007). Natural myside bias is independent of cognitive ability. Thinking and Reasoning, 13, 225e247. http://dx.doi.org/ 10.1080/13546780600780796.

Ten Dam, G., & Volman, M. (2004). Critical thinking as a citizenship competence: teaching strategies. Learning and Instruction, 14, 359e379. http://dx.doi.org/ 10.1016/j.learninstruc.2004.01.005.

Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211, 453e458. http://dx.doi.org/10.1126/science.7455683.

Tversky, A., & Kahneman, D. (1983). Extensional versus intuitive reasoning: the conjunction fallacy in probability judgment. Psychological Review, 90, 293e315. http://dx.doi.org/10.1037/0033-295X.90.4.293.

Van Gog, T., & Paas, F. (2008). Instructional efficiency: revisiting the original construct in educational research. Educational Psychologist, 43, 16e26. http:// dx.doi.org/10.1080/00461520701756248.

Wason, P. C., & Shapiro, D. (1971). Natural and contrived experience in a reasoning problem. Quarterly Journal of Experimental Psychology, 23, 63e71. http:// dx.doi.org/10.1080/00335557143000068.

Wasserman, E. A., Dorner, W. W., & Kao, S. F. (1990). Contributions of specific cell information to judgments of interevent contingency. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16, 509e521. http://dx.doi.org/ 10.1037/0278-7393.16.3.509.

Wenzlaff, R. M., & Bates, D. E. (2000). The relative efficacy of concentration and suppression strategies of mental control. Personality and Social Psychology Bulletin, 26, 1200e1212. http://dx.doi.org/10.1177/0146167200262003.

West, R. F., Toplak, M. E., & Stanovich, K. E. (2008). Heuristics and biases as measures of critical thinking: associations with cognitive ability and thinking disposi- tions. Journal of Educational Psychology, 100, 930e941. http://dx.doi.org/10.1037/ a0012842.

Williams, J. J., & Lombrozo, T. (2010). The role of explanation in discovery and generalization: evidence from category learning. Cognitive Science, 34, 776e806. http://dx.doi.org/10.1111/j.1551-6709.2010.01113.x.

Wolcott, S. K., Baril, C. P., Cunningham, B. M., Fordham, D. R., & St.Pierre, K. (2002). Critical thought on critical thinking research. Journal of Accounting Education, 20, 85e103. http://dx.doi.org/10.1016/S0748-5751(01)00029-X.