PY101 Assignment 2

Metacognitive strategies in student learning: Do students practise retrieval when they study

on their own?

Jeffrey D. Karpicke

Purdue University, West Lafayette, IN, USA

Andrew C. Butler and Henry L. Roediger III

Washington University in St. Louis, MO, USA

Basic research on human learning and memory has shown that practising retrieval of information (by testing the information) has powerful effects on learning and long-term retention. Repeated testing enhances learning more than repeated reading, which often confers limited benefit beyond that gained from the initial reading of the material. Laboratory research also suggests that students lack metacognitive awareness of the mnemonic benefits of testing. The implication is that in real-world educational settings students may not engage in retrieval practise to enhance learning. To investigate students’ real-world study behaviours, we surveyed 177 college students and asked them (1) to list strategies they used when studying (an open-ended, free report question) and (2) to choose whether they would reread or practise recall after studying a textbook chapter (a forced report question). The results of both questions point to the same conclusion: A majority of students repeatedly read their notes or textbook (despite the limited benefits of this strategy), but relatively few engage in self-testing or retrieval practise while studying. We propose that many students experience illusions of competence while studying and that these illusions have significant consequences for the strategies students select when they monitor and regulate their own learning.

Keywords: Testing effect; Retrieval; Metacognition; Strategies.

A powerful way to enhance student learning is by

testing information. When students have been

tested on material they remember more in the

long term than if they had repeatedly studied it.

This phenomenon is known as the testing effect

and shows that the act of retrieving information

from memory has a potent effect on learning,

enhancing long-term retention of the tested

information (for review, see Roediger & Kar-

picke, 2006a). The testing effect is especially

striking in light of current findings showing

limited benefits of repeated reading for student

learning (see Callender & McDaniel, 2009;

McDaniel & Callender, 2008). Our recent re-

search has generalised the testing effect to

educational materials (Butler & Roediger, 2007;

Karpicke & Roediger, 2007, 2008; Roediger &

Karpicke, 2006b) and real-world classroom en-

vironments (see McDaniel, Roediger, & McDer-

mott, 2007). Testing enhances learning not only if

instructors give tests and quizzes in the classroom

but also if students practise recall while they study

# 2009 Psychology Press, an imprint of the Taylor & Francis Group, an Informa business

Address correspondence to: Jeffrey D. Karpicke, Department of Psychological Sciences, Purdue University, 703 Third Street,

West Lafayette, IN 47907-2081, USA. E-mail: [email protected]

We thank Stephanie Karpicke and Julie Evans for collecting and scoring the questionnaire data. This research was supported by a

Collaborative Activity Grant from the James S. McDonnell Foundation and a grant from the Institute of Education Science.

MEMORY, 2009, 17 (4), 471�479

http://www.psypress.com/memory DOI:10.1080/09658210802647009

on their own. If students were to practise retrieval of information while studying this strategy would have the potential to greatly improve academic performance. However, we do not know the extent to which students practise recall while they study in real-world educational settings (relative to other less-effective strategies like repeated reading) or whether students who prac- tise recall do so because they are aware of the mnemonic benefits. These are important and practically relevant research questions but few studies have been aimed at answering them (see, e.g., Kornell & Bjork, 2007).

The objective of this research was to determine the extent to which students practise recall relative to other study strategies in real-world educational settings. In addition we wanted to examine whether students who choose to engage in retrieval practice do so because they know that testing promotes long-term retention. Another reason students may use testing during studying is to determine what information is known and what is not known so that future study time can be allocated to the unknown material (see Dunlosky, Hertzog, Kennedy, & Thiede, 2005; Dunlosky, Rawson, & McDonald, 2002). This is a fine justification for testing but it differs from using testing as a learning device in its own right. To accomplish these goals we created a new study strategies questionnaire and surveyed a large sample of undergraduate students. Although there are a variety of study strategy inventories in the education literature (see Entwistle & McCune, 2004; Pintrich, Smith, Garcia, & McKeachie, 1993; Weinstein, Schulte, & Palmer, 1987) these and other inventories do not specifi- cally assess whether students practise retrieval while studying. Our survey included a free report question asking students to list the strategies they use while studying and a forced report question that asked them to choose between repeated reading or repeated testing. The purpose of including both forced and free report question formats was to gain converging evidence aimed at the target issue and to circumvent possible response biases created by using either format alone (see Schuman & Presser, 1996; Schwarz, 1999). We predicted that relatively few students would report self-testing as a study strategy and that the majority of students would report choos- ing to reread or engage in some other non-testing activity when forced to choose a study strategy. We also predicted that most students who

selected self-testing would be unaware of the mnemonic benefits of testing.

In the first section of this paper we provide a brief overview of relevant research on repeated reading, repeated testing, and students’ metacog- nitive awareness of the testing effect. Next we present the results of our survey of study strate- gies. In the final section we interpret the survey results in light of current theories of metacogni- tion and self-regulated learning and then discuss the practical and educational implications of our findings.

MOTIVATION FOR THE SURVEY: PRIOR RESEARCH ON REPEATED READING VS REPEATED TESTING

The testing effect refers to the finding that taking a test enhances long-term retention more than spending an equivalent amount of time repeat- edly studying. There are clear and direct implica- tions of the testing effect for student learning. One way for students to enhance their learning would be to practise recalling information while studying. However, research on the testing effect has also shown that when students are asked to assess their own learning they sometimes fail to predict that testing enhances learning more than repeated reading (e.g., Karpicke & Roediger, 2008). In short, there is a rapidly growing body of research (briefly reviewed below) indicating that testing has powerful effects on learning but students lack metacognitive awareness of the testing effect.

Students often report that they repeatedly read their notes or textbook while studying (Carrier, 2003; Pressley, Van Etten, Yokoi, Freebern, & Van Meter, 1998; Van Etten, Freebern, & Press- ley, 1997). Yet there are several reasons to question the effectiveness of repetitive reading beyond reading a single time. Basic research on memory has shown that spending extra time maintaining or holding items in memory does not by itself promote learning (Craik & Watkins, 1973) and students may spend large amounts of additional time studying despite no gain in later memory for the items, a phenomenon called ‘‘labour-in-vain’’ during learning (Nelson & Leo- nesio, 1988). Recent research with educationally relevant materials has shown that repeatedly reading prose passages produces limited benefits beyond a single reading (Amlund, Kardash, & Kulhavy, 1986; Callender & McDaniel, 2009).

472 KARPICKE, BUTLER, ROEDIGER

This is especially true when repeated readings are massed together in a single learning session, although spaced rereading tends to produce positive effects (Rawson & Kintsch, 2005). In short, memory research has shown many times that repetitive reading by itself is not an effective strategy for promoting learning and long-term retention (for review, see McDaniel & Callender, 2008).

In contrast, several studies have shown that repeated testing is a potent method for producing robust learning. In one of our studies (Karpicke & Roediger, 2008) we had students learn a set of Swahili vocabulary words across alternating study and test periods. In study periods students studied a Swahili word and its English translation (ma- shua � boat) and in test periods they saw the Swahili words as cues to recall the English words (mashua �?). The students learned the words in one of four conditions and students in all condi- tions took a final test 1 week after initial learning. In two learning conditions, once a word was correctly recalled it was dropped from further test periods. The students who recalled each word only once in these two conditions recalled just 35% of the items on the final test a week later. In the other two conditions students continued to repeatedly recall words even after they had recalled them once. Students who repeatedly recalled the words during learning recalled about 80% of the items on the final test. Repeated retrieval practice*even after students were able to successfully recall items in the learning phase* produced large positive effects on long-term retention.

Were students aware of the effect of repeated testing on long-term retention? At the end of the initial learning phase we asked students to predict how many pairs they would recall on the final test a week later. There was no difference in average predictions across the four conditions: All groups predicted they would recall about 50% of the items. Despite the large effect of repeated retrie- val on retention, students were not aware of the mnemonic benefit of testing. Similar findings have occurred in other experiments examining the testing effect and students’ judgements of learning (e.g., Agarwal, Karpicke, Kang, Roedi- ger, & McDermott, 2008; Karpicke, McCabe, & Roediger, 2006; Roediger & Karpicke, 2006b).

In sum, basic laboratory research on human learning and memory has shown that (1) repeated reading by itself is a questionable and often ineffective study strategy, (2) repeated retrieval

practice produces robust learning and long-term retention, but (3) students appear to lack meta- cognitive awareness of the testing effect. The implication of this basic research is that students may not practise retrieval when they study in real- world educational settings. Instead they may spend their time repeatedly reading material when they study. The objective of our survey was to examine the prevalence of retrieval practice, relative to other study strategies, in students’ real-world study behaviours and stu- dents’ metacognitive awareness of the benefits of self-testing.

A SURVEY OF STUDENTS’ LEARNING STRATEGIES

One reaction we have encountered when we present our research on the testing effect goes something like this: ‘‘This is completely obvious. Of course testing enhances learning. We already knew this. None of this is new or surprising.’’ Perhaps the testing effect is obvious to some instructors*but is it obvious to students? If so we would expect students to report that they frequently practise recall while studying. But our basic laboratory research has consistently shown that students lack metacognitive awareness of the testing effect. In fact students sometimes predict that repeated reading will produce better long- term retention than repeated testing (Roediger & Karpicke, 2006b). The intent of our survey was to determine whether students’ self-reported study behaviours would converge with our laboratory findings.

Method

We surveyed 177 undergraduate students at Washington University in St. Louis about the strategies they use to study for exams. The students were participants in various learning and memory experiments in our laboratory and they completed the survey at the very end of their experimental session. Washington University stu- dents are a select group with average SAT scores greater than 1400 (Verbal�Quantitative). Our survey included two questions aimed at identify- ing how often the students practised recalling information while studying. Question 1 was an open-ended free report question in which students listed the strategies they used when

METACOGNITIVE STRATEGIES IN LEARNING 473

studying and then rank ordered the strategies in terms of how frequently they used them. All 177 students answered Question 1. Question 2 was a forced report question that asked students to imagine they were studying a textbook chapter for an exam and to choose one of three alter- natives: (1) repeated reading of the chapter, (2) practising recall of material from the chapter (with or without the opportunity to reread the chapter, in different versions of this question), or (3) engaging in some other study activity. A total of 101 students answered Version 1 of Question 2 (testing without restudy) and the other 76 stu- dents answered Version 2 (testing with restudy). Students completed the entire questionnaire in about 5 to 10 minutes. Our goals were to identify students’ typical study strategies and to assess how frequently they repeatedly read material or engaged in retrieval practice, and our analysis focused on the frequency with which students reported these particular strategies.

Results

Question 1: Students’ free report of study strategies. The first question on the survey asked: ‘‘What kind of strategies do you use when you are studying? List as many strategies as you use and rank-order them from strategies you use most often to strategies you use least often.’’ We initially reviewed all responses from all students. Based on our initial assessment we identified 11 strategies that occurred relatively frequently (more than once across all student responses). Two independent raters then categorised all responses. There was close to 100% agreement between the two raters and the first author resolved any scoring discrepancies.

Figure 1 shows the frequency distribution of the number of strategies listed by students in response to Question 1. The figure shows that most students listed and described three strate- gies (M�2.9). Table 1 shows the 11 strategies and the percent of students who listed each strategy. The table also shows the percent of students who ranked each strategy as their number one strategy and the mean rank of each strategy. Repeated reading was by far the most frequently listed strategy with 84% of students reporting it. Not only did students indicate that they repeatedly read while studying but they also indicated that rereading was a favoured strategy*55% of stu- dents reported that rereading was the number one strategy they used when studying. Table 1 also

shows another key finding: Only 11% of students (19 of 177) reported that they practised retrieval while studying. These students unambiguously indicated in their list of strategies that they practised testing themselves by recalling informa- tion while they studied. Only 1% (2 of 177 students) identified practising recall as their number one strategy. The results in Table 1 clearly show that a large majority of students repeatedly read their textbook or notes but relatively few students engage in self-testing by practicing recall while studying.

Table 1 also shows that students reported other strategies that could be interpreted as forms of self-testing. For example, 43% of students indi- cated that they answer practice problems while studying and 40% reported using flashcards. Each activity could be interpreted as a type of self- testing, but of course there are ways students might use these study methods without engaging in retrieval practice. For example, students may read practice questions and then look up and copy answers from the text. This would qualify as answering practice problems but students who do this would not be practising or even attempting recall of the answers. Likewise, students may write facts on flashcards and repeatedly read them rather than practising recall. A clear limita- tion of the free response question is that our procedure did not prompt each student to

Figure 1. Frequency distribution showing the number of

strategies listed by students.

474 KARPICKE, BUTLER, ROEDIGER

elaborate on potentially ambiguous answers (cf.

the ethnographic interviewing technique of Press-

ley and colleagues; Pressley et al., 1998; Van

Etten et al., 1997). Nevertheless, even if we

considered the 40% of students who use flash-

cards or the 43% who answer practice problems

as students engaging in forms of self-testing, these

percentages are dwarfed by the 84% of students

who repeatedly read while studying. The results of Question 1 indicate that re-

peated reading is the most popular study strategy

among college students (see too Carrier, 2003),

far more popular than practising retrieval, even

though retrieval practice is a more effective study

strategy. Students listed a variety of study strate-

gies but indicated that they use these alternative

study strategies far less frequently than repeated

reading. Question 2 asked students to choose

repeated reading or self-testing and prompted

them to explain the reasoning behind their choice.

By including a second question in forced report

format we hoped to find converging evidence and

to resolve ambiguities inherent in our first open-

ended free report question.

Question 2: Forced report questions about repeated studying vs testing. Question 2 was a forced report question about repeated studying

versus repeated testing. There were two versions

of the question. Version 1 asked students to

consider testing without going back and re-

studying, and Version 2 involved testing followed

by restudying (to get feedback after attempting

recall). The first version was given to 101 students

and the second version was given to 76 students. Version 1 of Question 2 was as follows:

Imagine you are reading a textbook chapter for an upcoming exam. After you have read the chapter one time, would you rather: A. Go back and restudy either the entire

chapter or certain parts of the chapter. B. Try to recall material from the chapter

(without the possibility of restudying the material).

C. Use some other study technique.

The students were asked to select one alter- native and write a brief explanation for their choice. The scenario described in the question was based directly on our research showing that taking a recall test, even without feedback, enhances long-term retention more than spending the same amount of time restudying (Roediger & Karpicke, 2006b).

Table 2 shows the percentage of students who chose to restudy, self-test, or do something else after reading a textbook chapter. Most students unambiguously selected an alternative and ex- plained their choice, but four students gave ambiguous responses that could not be scored. The table shows that 57% of students chose to restudy (option A) and 21% indicated that they would use some other study technique (option C). Thus 78% of students indicated they would not want to test themselves after reading a textbook chapter. Only 18% of the students indicated that they would self-test after studying (option B). To examine students’ metacognitive awareness of the

TABLE 1

Results of Question 1

Strategy Percent who list strategy Percent who rank as #1 strategy Mean rank

1. Rereading notes or textbook 83.6 (148) 54.8 (97) 1.5

2. Do practice problems 42.9 (76) 12.4 (22) 2.1

3. Flashcards 40.1 (71) 6.2 (11) 2.6

4. Rewrite notes 29.9 (53) 12.4 (22) 1.8

5. Study with a group of students 26.5 (47) 0.5 (1) 2.9

6. ‘‘Memorise’’ 18.6 (33) 5.6 (10) 2.0

7. Mnemonics (acronyms, rhymes, etc) 13.5 (24) 2.8 (5) 2.4

8. Make outlines or review sheets 12.9 (23) 3.9 (7) 2.1

9. Practise recall (self-testing) 10.7 (19) 1.1 (2) 2.5

10. Highlight (in notes or book) 6.2 (11) 1.6 (3) 2.3

11. Think of real life examples 4.5 (8) 0.5 (1) 2.8

Percent of students listing different learning strategies, percent who ranked strategies as their #1 strategy, and mean rankings of

strategies. Raw numbers of students are in parentheses.

Mean number of strategies listed was 2.9 (SD�0.96). Percentages of students indicating their #1 strategy do not add to 100%

because some students merged multiple strategies when reporting their #1 strategy (e.g., indicating that rereading and rewriting

notes were their #1 strategy).

METACOGNITIVE STRATEGIES IN LEARNING 475

mnemonic benefits of testing we separated stu- dents’ responses based on their explanation for why they chose self-testing. This analysis showed that 10% of all students (or more than half of those who chose self-testing) reported they would self-test to generate feedback and guide their future studying (even though Version 1 of this question stated that students could not restudy after testing). Only 8% of all students indicated that they would test themselves because practis- ing retrieval would help them do well on the upcoming exam. This pattern of responding suggests that most students were unaware of the mnemonic benefits of self-testing. The results of Version 1 of this forced report question provide converging evidence with our first free report question. Relatively few students reported that they would test themselves after studying a text- book chapter and even fewer indicated they would test themselves because they knew the act of practising recall was valuable for learning.

In Version 2 of Question 2 the scenario and alternatives were identical to Version 1 except that option B read ‘‘Try to recall material from the chapter (with the possibility of restudying afterward).’’ We imagined this would increase the number of students choosing testing perhaps to levels near ceiling if students recognised that testing followed by rereading would produce far superior learning to rereading without testing. Table 3 shows the percent of students who chose each option. The percentage of students choosing self-testing increased when students could reread after the test (42% in Question 2 vs 18% in Question 1) and the percentage was about equal to the percentage of students choosing repeated reading (42% vs 41%). Students’ explanations of their choices indicated that the increased like-

lihood of choosing testing was due to the possi- bility of restudying after the test. Of the 32 students who chose self-testing, 25 provided unambiguous explanations that we categorised as testing for feedback or testing to practise recall. A total of 23 students (30%) indicated that they would test themselves to generate feedback they could use when restudying whereas only two students (3%) chose testing because they believed the act of practising recall would help them remember in the future. The results of Version 2 of Question 2 expand on the results of Version 1 by showing that students were more likely to select self-testing when they could restudy after testing but that very few students are aware that the act of practising recall itself enhances learning. What is perhaps most striking about the data in Table 3 is that even when students had the option of rereading after self- testing, the majority of students (58%) continued to indicate that they would not test themselves.

DISCUSSION

The objective of this research was to collect benchmark data on college students’ real-world study behaviours to assess how often students use retrieval practice relative to other strategies and whether they know about the mnemonic benefits of self-testing. Our basic laboratory studies sug- gested that students are not aware of the testing effect, leading us to predict that they may not practise retrieval while studying in real-world settings. The results of our survey support this prediction. The majority of students indicated that they repeatedly read their notes or textbook while studying. Relatively few reported that they

TABLE 2

Version 1 of Question 2

Imagine you are reading a

textbook chapter for an upcoming

exam. After you have read the

chapter one time, would you rather: Overall

Test for

feedback

Test to

practise

recall

A. Go back and restudy either the entire

chapter or certain parts of the chapter

57.4 (58)

B. Try to recall material from the chapter

(without the possibility of restudying the material) 17.8 (18) 9.9 (10) 7.9 (8)

C. Use some other study technique 20.7 (21)

Percent of students who chose to restudy, self-test (without restudying), or do something else after reading a textbook chapter.

Raw numbers of students are in parentheses (N�101).

We were unable to score ambiguous responses given by four students.

476 KARPICKE, BUTLER, ROEDIGER

tested themselves and of those who engaged in self-testing only a handful reported doing so because they believed the act of practising retrieval would improve their learning. Our survey results point to the conclusion that many students do not view retrieval practice as a strategy that promotes learning. If students do practise recall or test themselves while studying they do it to generate feedback or knowledge about the status of their own learning, not because they believe practising recall itself en- hances learning.

Our results agree with laboratory experiments showing that students lack metacognitive aware- ness of the testing effect when they monitor their own learning. A growing body of research has shown that students sometimes predict that practising retrieval will produce no effect on retention (Karpicke & Roediger, 2008) or that they will remember more in the long term if they repeatedly study material rather than test it (Agarwal et al., 2008; Karpicke et al., 2006; Roediger & Karpicke, 2006b). If we assume that metacognitive monitoring processes guide stu- dents’ decisions to choose different learning strategies*an assumption at the core of the influential monitoring-and-control framework of metacognition (Nelson & Narens, 1990)*then the implication of these laboratory results is that students may not choose to test themselves when they regulate their own learning in real-world educational settings. Our survey data confirm that this lack of awareness of the testing effect has consequences for students’ real-world study behaviours.

In addition to agreeing with basic laboratory findings our survey results also agree to some extent with a recent survey by Kornell and Bjork (2007). They surveyed college students about their study behaviours and asked the students,

‘‘If you quiz yourself while you study . . . why do you do so?’’ The students selected one of four alternatives: 18% selected ‘‘I learn more that way than I would through rereading’’; 68% selected ‘‘To figure out how well I have learned the information I’m studying’’; 4% indicated ‘‘I find quizzing more enjoyable than rereading’’; and 9% said ‘‘I usually do not quiz myself.’’ Kornell and Bjork’s data indicate that the majority of students (91%) do quiz themselves while studying but few do so because they view the act of quizzing itself as a method of enhancing learning (Kornell and Bjork reasoned that the 18% of students who selected ‘‘I learn more that way than I would through rereading’’ believed that quizzing pro- duced a direct mnemonic benefit; cf. Roediger & Karpicke, 2006a). Likewise, our survey data indicate that few students view practising recall as an activity that enhances learning. However, far more students indicated that they tested themselves in the Kornell and Bjork survey than in our study, and this may be due to a difference in survey procedures. Whereas we used a combi- nation of free and forced report questions to gauge how often students practise retrieval, Kornell and Bjork used one question focused on why students might quiz themselves and the framing of this question may have influenced students’ responses (see Schuman & Presser, 1996; Schwarz, 1999). It is well known that a single question can be framed in different ways and alter the choices and decisions people make (Tversky & Kahneman, 1981). Nevertheless our results generally agree with those of Kornell and Bjork in showing that few students view retrieval practice as a method of enhancing learning. Further, the differences between the two sets of results highlight potentially important differences between free and forced report methods of questioning.

TABLE 3

Version 2 of Question 2

Imagine you are reading a textbook

chapter for an upcoming exam. After

you have read the chapter one time,

would you rather: Overall Test for feedback Test to practise recall

A. Go back and restudy either the entire

chapter or certain parts of the chapter

40.8 (31)

B. Try to recall material from the chapter

(with the possibility of restudying afterward) 42.1 (32) 30.3 (23) 2.6 (2)

C. Use some other study technique 17.1 (13)

Percent of students who chose to restudy, self-test followed by restudying, or do something else after reading a textbook chapter.

Raw numbers of students are in parentheses (N�76).

METACOGNITIVE STRATEGIES IN LEARNING 477

Our results fit with the broad theoretical notion that students experience illusions of com- petence when monitoring their own learning (Bjork, 1999; Jacoby, Bjork, & Kelley, 1994; Koriat & Bjork, 2005). Koriat and Bjork (2005) argued that illusions of competence tend to occur when students’ judgements of learning are biased by information available during study but not available during testing (see also Jacoby et al., 1994). Several experimental findings are consis- tent with this view. For example, students’ judge- ments of learning are less accurate when made in study trials than in test trials (Dunlosky & Nelson, 1992). Students are less accurate at judging the difficulty of anagrams when the solution is present than when it is not (Kelley & Jacoby, 1996). We believe repeated reading pro- duces a similar illusion of competence. Specifi- cally, repeatedly reading material like text passages increases the fluency or ease with which students process the text. Students may base their assessments of their learning and comprehension on fluency even though their current processing fluency with the text right in front of them, is not diagnostic of their future retention. Our survey results show that the illusions students experience during learning may have important consequences and implications for the decisions they make and the strategies they choose when studying on their own.

Students generally exhibit little awareness of the fact that practising retrieval enhances learn- ing. A clear practical implication is that instruc- tors should inform students about the benefits of self-testing and explain why testing enhances learning. When students rely purely on their subjective experience while they study (e.g., their fluency of processing during rereading) they may fall prey to illusions of competence and believe they know the material better than they actually do. A challenge for instructional practice is to encourage students to base their study strategies on theories about why a particular strategy*like practising repeated retrieval*promotes learning and long-term retention.

Manuscript received 5 October 2008

Manuscript accepted 20 November 2008

REFERENCES

Agarwal, P. K., Karpicke, J. D., Kang, S. H. K., Roediger, H. L., & McDermott, K. B. (2008). Examining the testing effect with open- and

closed-book tests. Applied Cognitive Psychology, 22, 861�876.

Amlund, J. T., Kardash, C. A. M., & Kulhavy, R. W. (1986). Repetitive reading and recall of expository text. Reading Research Quarterly, 21, 49�58.

Bjork, R. A. (1999). Assessing our own competence: Heuristics and illusions. In D. Gopher & A. Koriat (Eds.), Attention and peformance XVII. Cognitive regulation of performance: Interaction of theory and application (pp. 435�459). Cambridge, MA: MIT Press.

Butler, A. C., & Roediger, H. L. (2007). Testing improves long-term retention in a simulated class- room setting. European Journal of Cognitive Psy- chology, 19, 514�527.

Callender, A. A., & McDaniel, M. A. (2009). The limited benefits of rereading educational texts. Contemporary Educational Psychology, 34, 30�41.

Carrier, L. M. (2003). College students’ choices of study strategies. Perceptual and Motor Skills, 96, 54�56.

Craik, F. I. M., & Watkins, M. J. (1973). The role of rehearsal in short-term memory. Journal of Verbal Learning and Verbal Behavior, 12, 599�607.

Dunlosky, J., Hertzog, C., Kennedy, M. R. F., & Thiede, K. W. (2005). The self-monitoring approach for effective learning. Cognitive Technology, 10, 4�11.

Dunlosky, J., & Nelson, T. O. (1992). Importance of the kind of cue for judgments of learning (JOL) and the delayed-JOL effect. Memory & Cognition, 20, 374� 380.

Dunlosky, J., Rawson, K. A., & McDonald, S. L. (2002). Influence of practice test on the accuracy of predicting memory performance for paired associ- ates, sentences, and text material. In T. J. Perfect & B. L. Schwartz (Eds.), Applied metacognition (pp. 68�92). Cambridge, UK: Cambridge University Press.

Entwistle, N., & McCune, V. (2004). The conceptual bases of study strategy inventories. Educational Psychology Review, 16, 325�345.

Jacoby, L. L., Bjork, R. A., & Kelley, C. M. (1994). Illusions of comprehension, competence, and re- membering. In D. Druckman & R. A. Bjork (Eds.), Learning, remembering, believing: Enhancing hu- man performance (pp. 57�80). Washington, DC: National Academy Press.

Karpicke, J. D., McCabe, D. P., & Roediger, H. L. (2006). Testing enhances recollection: Process dis- sociation estimates and metamemory judgments. Poster presented at the 47th Annual Meeting of the Psychonomic Society, Houston, TX.

Karpicke, J. D., & Roediger, H. L. (2007). Repeated retrieval during learning is the key to long-term retention. Journal of Memory and Language, 57, 151�162.

Karpicke, J. D., & Roediger, H. L. (2008). The critical importance of retrieval for learning. Science, 319, 966�968.

Kelley, C. M., & Jacoby, L. L. (1996). Adult egocentr- ism: Subjective experience versus analytic bases for judgment. Journal of Memory and Language, 35, 157�175.

Koriat, A., & Bjork, R. A. (2005). Illusions of compe- tence in monitoring one’s knowledge during study.

478 KARPICKE, BUTLER, ROEDIGER

Journal of Experimental Psychology: Learning, Memory, and Cognition, 31, 187�194.

Kornell, N., & Bjork, R. A. (2007). The promise and perils of self-regulated study. Psychonomic Bulletin & Review, 14, 219�224.

McDaniel, M. A., & Callender, A. A. (2008). Cogni- tion, memory, and education. In H. L. Roediger (Ed.), Cognitive psychology of memory, Vol. 2 of Learning and memory: A comprehensive reference (pp. 819�843). Oxford, UK: Elsevier.

McDaniel, M. A., Roediger, H. L., & McDermott, K. B. (2007). Generalizing test-enhanced learning from the laboratory to the classroom. Psychonomic Bul- letin & Review, 14, 200�206.

Nelson, T. O., & Leonesio, R. J. (1988). Allocation of self-paced study time and the ‘‘labor-in-vain effect’’. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14, 676�686.

Nelson, T. O., & Narens, L. (1990). Metamemory: A theoretical framework and new findings. In G. H. Bower (Ed.), The psychology of learning and motivation (Vol. 26, (pp. 125�141)). New York: Academic Press.

Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKea- chie, W. J. (1993). Reliability and predictive validity of the Motivated Strategies for Learning Question- naire (MSLQ). Educational and Psychological Mea- surement, 53, 801�813.

Pressley, M., Van Etten, S., Yokoi, L., Freebern, G., & Van Meter, P. (1998). The metacognition of college studentship: A grounded theory approach. In D. J.

Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 347�366). Mahwah, NJ: Lawrence Erlbaum Associates Inc.

Rawson, K. A., & Kintsch, W. (2005). Rereading effects depend on time of test. Journal of Educational Psychology, 97, 70�80.

Roediger, H. L., & Karpicke, J. D. (2006a). The power of testing memory: Basic research and implications for educational practice. Perspectives on Psycholo- gical Science, 1, 181�210.

Roediger, H. L., & Karpicke, J. D. (2006b). Test enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17, 249� 255.

Schuman, H., & Presser, S. (1996). Questions and answers in attitude surveys. Thousand Oaks, CA: Sage.

Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93� 105.

Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211, 453�458.

Van Etten, S., Freebern, G., & Pressley, M. (1997). College students’ beliefs about exam preparation. Contemporary Educational Psychology, 22, 192�212.

Weinstein, C. E., Schulte, A. C., & Palmer, D. R. (1987). Learning and Study Strategies Inventory (LASSI). Clearwater, FL: H&H Publishing.

METACOGNITIVE STRATEGIES IN LEARNING 479