APA Style Psychology Essay

Original Articles and Reviews

The Reliability of Retrieval-Induced Forgetting

Rosalind Potts, Robin Law, John F. Golding, and David Groome

Department of Psychology, University of Westminster, London, UK

Abstract. Retrieval-induced forgetting (RIF) refers to the finding that the retrieval of an item from memory impairs the retrieval of related items. The extent to which this impairment is found in laboratory tests varies between individuals, and recent studies have reported an association between individual differences in the strength of the RIF effect and other cognitive and clinical factors. The present study investigated the reliability of these individual differences in the RIF effect. A RIF task was administered to the same individuals on two occasions (sessions T1 and T2), one week apart. For Experiments 1 and 2 the final retrieval test at each session made use of a category-cue procedure, whereas Experiment 3 employed category-plus-letter cues, and Experiment 4 used a recognition test. In Experiment 2 the same test items that were studied, practiced, and tested at T1 were also studied, practiced, and tested at T2, but for the remaining three experiments two different item sets were used at T1 and T2. A significant RIF effect was found in all four experiments. A significant correlation was found between RIF scores at T1 and T2 in Experiment 2, but for the other three experiments the correlations between RIF scores at T1 and T2 failed to reach significance. This study therefore failed to find clear evidence for reliable individual differences in RIF performance, except where the same test materials were used for both test sessions. These findings have important implications for studies involving individual differences in RIF performance.

Keywords: retrieval-induced forgetting, memory, inhibition, reliability, individual differences

Retrieval-induced forgetting (RIF) refers to the finding that the act of retrieving a memory impairs the subsequent retrie- val of related memories (Anderson, Bjork, & Bjork, 1994). For example, repeated retrieval of ‘‘apple’’ will impair the subsequent retrievability of an unpracticed fruit such as ‘‘orange,’’ but will not affect the retrieval of an unrelated word such as ‘‘armchair.’’ The inhibitory account of RIF (see Anderson, 2003, for a review) attributes this effect to the active inhibition of items related to the target memory as a means of resolving competition from memories which might otherwise interfere with retrieval of the target. A num- ber of studies (e.g., Groome & Grant, 2005; Groome & Sterkaj, 2010), reviewed below, have investigated the possi- bility that individual differences in RIF performance might be associated with a variety of other cognitive and clinical factors. The aim of the current study was to investigate whether individual differences in RIF performance are reli- able across different times and different sets of materials.

In a standard RIF task (Anderson et al., 1994) partici- pants study category-exemplar word pairs and then practice retrieving a subset of these items (the Rp+ items) in response to retrieval cues. When participants are tested on all studied items, recall of the unpracticed items from the practiced categories (Rp� items) is found to be impaired by comparison with recall of the unpracticed items from the unpracticed categories (Nrp items). The RIF effect is measured by calculating the difference between the Nrp and Rp� scores. The occurrence of RIF has now been confirmed by a large number of studies (for reviews, see Anderson, 2003; Anderson & Levy, 2007).

Two main theories have been put forward to explain the occurrence of RIF. The interference theory proposes that the retrieval of one item simply increases the strength of associ- ation between that item and its cue, which reduces the chance of that cue generating a rival item (Camp, Pecher, & Schmidt, 2007; Perfect, Conway, Moulin, & Perry, 2002). The inhibition theory assumes that the retrieval of one item leads to a reduction in the actual memory strength of a rival item, which will apply to any retrieval cue (Anderson, 2003; Anderson & Levy, 2007; Anderson & Spellman, 1995; Levy & Anderson, 2008). One important difference between these two accounts of RIF is that the interference theory predicts that retrieval impairment will be restricted to a specific retrie- val cue, whereas the inhibition theory predicts that impair- ment will generalize to any retrieval cue. Several studies have attempted to establish whether the RIF effect is restricted to the original cue or whether it generalizes to other, independent cues (i.e., where different cues are used for retrieval practice and final test), but at the present time the evidence remains equivocal. Some studies have reported the occurrence of RIF with independent cues (Anderson & Spellman, 1995; Aslan, Bauml, & Pastotter, 2007; Saunders & MacLeod, 2002, 2006) but other studies have not (Camp et al., 2007; Perfect et al., 2002).

It has been proposed that variations in the st rength of the RIF effect in individuals might be associated with a variety of other cognitive and clinical factors. Groome and Grant (2005) reported a significant inverse correlation between RIF and the Cognitive Failures Questionnaire, meaning that individuals showing a weak RIF effect report more memory failures in

� 2011 Hogrefe Publishing European Psychologist 2012; Vol. 17(1):1–10 DOI: 10.1027/1016-9040/a000040

T hi

s do

cu m

en t i

s co

py ri

gh te

d by

th e

A m

er ic

an P

sy ch

ol og

ic al

A ss

oc ia

tio n

or o

ne o

f i ts

a lli

ed p

ub lis

he rs

. T

hi s

ar tic

le is

in te

nd ed

s ol

el y

fo r t

he p

er so

na l u

se o

f t he

in di

vi du

al u

se r a

nd is

n ot

to b

e di

ss em

in at

ed b

ro ad

ly .

everyday life. This findingoffers support for theviewthatRIF may play a part in facilitating selective retrieval in real-life sit- uations. Groome and Sterkaj (2010) found that the RIF effect was significantly reduced in individuals suffering from clini- cal depression, and consistent with this finding Bauml and Kuhbandner (2007) reported that the RIF effect was signifi- cantly reduced in dysphoric members of the normal popula- tion (i.e., normal participants subjected to mood induction techniques). Individual differences in RIF performance have also been investigated in studies of individuals with schizo- phrenia. Normal RIF performance has been observed in schizophrenics when using letter cues in the final RIF recall test (AhnAllen, Nestor, McCarley, & Shenton, 2007; Nestor et al., 2005), but a recent study using recognition as the final RIF retrieval test has reported a significant reduction in RIF compared with normal controls (Soriano, Jimenez, Roman, & Bajo, 2009). These studies suggest that individual differ- ences in RIF are correlated with a number of other factors, such as cognitive failures, depression, and schizophrenia. However, some interpretations of such correlational studies rely on the assumption that there are individual differences in RIF which are reliable over time. In other words, each individual has their own reasonably stable level of RIF, so that an individual producing a high RIF score on one occa- sion will show the same high RIF score on a future occasion. While the finding of significant correlations between RIF strength and other factors does offer some support for the existence of reliable individual differences in RIF, no study to date has provided specific evidence for such individual dif- ferences in terms of test/retest reliability of RIF performance. The aim of the present study was to test the hypothesis that the RIF effect would show individual differences which were reliable and repeatable when tested on two different occa- sions. In a series of four experiments, we investigated the ret- est reliability of individual differences in the RIF effect using different cuing methods and either the same or different materials at each test.

Our first experiment tests reliability at two times using two different sets of materials at the two sessions, while Experiment 2 uses the same set of materials at both sessions. Both of these first two experiments use simple category cuing, since this is the method most commonly used in indi- vidual differences studies of RIF to date. However, use of simple category cuing in RIF experiments has been criti- cized (e.g., Anderson, 2003) on the grounds that part of the effect may be attributable to output interference. Output interference occurs during the final test because the practiced items are usually the first items to be retrieved and may interfere with the subsequent retrieval of the unpracticed items (Anderson & Bell, 2001; Bauml & Hartinger, 2002). The effects of output interference can be reduced by control- ling the order in which items are recalled at the final test, and this is normally achieved by using initial letter cues in addi- tion to the usual category cues (Anderson & Bell, 2001). One drawback of using category-plus-letter cues is that the magnitude of the RIF effect is significantly reduced (Ander- son et al., 1994) and indeed the RIF effect may disappear altogether (Butler, Williams, Maki, & Zacks, 2001). In spite of this reservation, and although there is evidence that out- put interference reflects the same mechanism as that under-

lying RIF (Bauml & Hartinger, 2002), our third experiment used category-plus-letter cues in order to achieve a RIF effect uncontaminated by output interference.

Our final experiment used a recognition test. It has been argued (Anderson & Levy, 2007) that both the category-cued and category-plus-letter cued RIF procedures may be subject to the correlated costs and benefits (CCB) problem whereby the same RIF mechanism impairs retrieval at the retrieval- practice stage but facilitates retrieval in the final retrieval test. According to this account, the purpose of the RIF effect is to enable efficient selection of a target by inhibiting competi- tors. Thus it is argued that an individual with strong inhibi- tory control, who successfully inhibits competitors to the practiced items during retrieval practice, will be able to use their strong inhibitory control at test to overcome competition and retrieve the previously inhibited items, thus canceling out the effect of the original inhibition. The CCB problem is thought to occur when the same cues are employed at both the practice and the test stages, but not when different cues are employed at these stages. Researchers have therefore sought to overcome the CCB problem by using either inde- pendent cues (e.g., Anderson & Spellman, 1995) or a recog- nition procedure (e.g., Hicks & Starns, 2004; Soriano et al., 2009; Veling & Van Knippenberg, 2004) at the final test stage. Given that RIF has not always been found with inde- pendent cues (e.g., Camp et al., 2007), our final experiment employed a recognition test for this purpose.

Experiment 1. Repeated RIF Task Using Category Cues as the Final Retrieval Test and With Different Test Items at Sessions T1 and T2

Experiment 1 investigated the hypothesis that RIF scores would show test/retest reliability when different sets of materials were used at the two sessions. The final test used simple category cuing since this is the method most com- monly used in individual differences studies of RIF to date, and since this method has been shown to produce a strong RIF effect (Anderson et al., 1994). It has been argued (Anderson, 2003) that a RIF effect obtained using simple category cuing will reflect not just inhibition but also output interference. However, our aim in this study was not to determine underlying causes of the RIF effect but to estab- lish whether the effect, however caused, is reliable for an individual over time. This is particularly pertinent to inves- tigate since several studies of individual differences in RIF have used the simple category cuing method (e.g., Groome & Grant, 2005; Groome & Sterkaj, 2010).

Method

Participants

Thirty-seven participants took part in this study, all of whom were students at a British University. The sample comprised

2 R. Potts et al.: The Reliability of RIF

European Psychologist 2012; Vol. 17(1):1–10 � 2011 Hogrefe Publishing

T hi

s do

cu m

en t i

s co

py ri

gh te

d by

th e

A m

er ic

an P

sy ch

ol og

ic al

A ss

oc ia

tio n

or o

ne o

f i ts

a lli

ed p

ub lis

he rs

. T

hi s

ar tic

le is

in te

nd ed

s ol

el y

fo r t

he p

er so

na l u

se o

f t he

in di

vi du

al u

se r a

nd is

n ot

to b

e di

ss em

in at

ed b

ro ad

ly .

28 females and 8 males, with a mean age of 22.6 (SD = 2.83).

Materials

RIF Measurement

Two versions of the RIF task were constructed according to the standard procedure devised by Anderson et al. (1994). These will henceforth be referred to as Set A and Set B. Set A was administered at the first experimental session (T1), and Set B was administered at the second session (T2), which took place seven days later. Set A used com- pletely different items and different categories to those used in Set B. Each set consisted of 36 category-exemplar word pairs (e.g., Fruit-Apple), comprising six exemplars from each of six categories. Items were selected from a recently updated version (Van Overschelde, Rawson, & Dunlosky, 2004) of the traditional Battig and Montague (1969) cate- gory norms. The average taxonomic rank of exemplars in each set was equivalent, at 6.3, and categories within each set were equated for average taxonomic rank of items. Within each category, no two items began with the same let- ter. All participants underwent four main phases at each ses- sion: study, retrieval practice, retention interval, and final retrieval test. Retrieval practice generated three types of items: practiced items (Rp+), unpracticed items from the practiced categories (Rp�), and unpracticed items from the unpracticed categories (Nrp). The RIF effect is defined as the difference between the Nrp� and Rp� scores, both expressed as percentage scores. Across participants, target items served as either Rp+, Rp�, or Nrp an equal number of times. This was done to control for variations in memo- rability of the test items. The item sets were made up into Powerpoint presentations which were presented by project- ing them onto a large wall screen. Participants were given answer sheets on which to write their responses.

Procedure

The RIF task involved four phases at each session:

(i) Study: Participants were shown all 36 category- exemplar word pairs one at a time. Items were randomly presented within six blocks. Each block included one exemplar from each of the six categories. The first item in any block was never from the same category as the last item in the previous block. Each pair was presented for five seconds, and participants were asked to remember as many words as possible.

(ii) Retrieval practice: Participants were required to practice retrieving nine exemplar words in response to a cue consisting of the category word and the first two letters of the exemplar (e.g., Fruit-Ap). These nine practiced items comprised half of the exemplars from half of the category lists. Participants were allowed 7 seconds to retrieve each exemplar and

write it on their answer sheet. Each item appeared three times, at regularly spaced intervals, with the same category never appearing twice in a row.

(iii) Retention interval: This consisted of a 10-min filled time interval, during which participants completed questionnaires.

(iv) Final retrieval test: Participants were shown each of the category names one at a time and were allowed 30 s to list as many exemplars of each category as possible on a sheet of paper.

All participants were administered the two different RIF tasks as described above on two separate occasions (T1 and T2), separated by a period of seven days.

Results and Discussion

The RIF procedure generated percentage recall scores for the following types of test items:

Rp+ = items subjected to retrieval practice; Rp� = unpracticed items from the same category as practiced items; Nrp = unpracticed items from unpracticed categories; RIF score = (Nrp%) � (Rp�%).

Table 1 shows the scores obtained on the RIF test in Experiment 1, each expressed as a percentage of the number of items in that particular retrieval condition. There were no significant effects or interactions in the RIF scores for the variants of the RIF test, so these were amalgamated for the remaining analyses. The average retrieval-practice score in the retrieval-practice phase was 96.6% (SD = 8.6).

A two-way analysis of variance was carried out on the scores in Table 1, with the within-subjects factors of RIF retrieval condition (Rp� and Nrp) and test session (T1 and T2). A significant main effect was found for RIF, F(1, 36) = 60.6, p < .0001, but not for test session, F(1, 36) = 0.04, p = NS, and there was no significant inter- action, F(1, 36) = 0.004, p = NS. Thus Experiment 1 pro- duced a significant RIF effect.

Table 2 shows the correlations between the scores obtained at the two test sessions. There were significant cor- relations between T1 and T2 scores for Rp+, Rp�, and Nrp scores, but the correlation for overall RIF score failed to reach significance. One possible explanation for this low correlation is that individual differences in RIF performance may interact with particular test items, such that variations in the familiarity or memorability of different sets of test items

Table 1. Mean % scores (SDs in brackets) obtained on the RIF test in test sessions T1 and T2 (Experiment 1: category cues with different test items at T1 and T2)

Session Rp+ Rp� Nrp RIF score T1 83.3 (13.8) 31.8 (17.8) 52.3 (18.4) 20.4 (20.9) T2 92.5 (9.3) 31.5 (18.2) 51.7 (19.3) 20.2 (20.9)

R. Potts et al.: The Reliability of RIF 3

� 2011 Hogrefe Publishing European Psychologist 2012; Vol. 17(1):1–10

T hi

s do

cu m

en t i

s co

py ri

gh te

d by

th e

A m

er ic

an P

sy ch

ol og

ic al

A ss

oc ia

tio n

or o

ne o

f i ts

a lli

ed p

ub lis

he rs

. T

hi s

ar tic

le is

in te

nd ed

s ol

el y

fo r t

he p

er so

na l u

se o

f t he

in di

vi du

al u

se r a

nd is

n ot

to b

e di

ss em

in at

ed b

ro ad

ly .

for different participants may have masked a potential corre- lation, although we had attempted to minimize such an effect by matching the items for taxonomic rank as described above. However, in Experiment 2 we sought to eliminate these potential effects altogether by using the same materials at T1 as at T2.

Experiment 2. Repeated RIF Task Using Category Cues and Using the Same Test Items at Sessions T1 and T2

Experiment 1 investigated the hypothesis that individual dif- ferences in RIF scores would show test/retest reliability when different test items were used at both T1 and T2. However, no correlation was found between the RIF scores at T1 and T2. According to the inhibitory account, the RIF effect depends on competition at the retrieval-practice phase from the unpracticed items from practiced categories (the Rp� items) and it is well established that only strong com- petitors are likely to be inhibited (Anderson et al., 1994). Individual differences in the extent to which participants associate items with their categories could therefore lead to variations in RIF scores, despite the fact that the test items were chosen as strong exemplars of their categories based on their rank in published category norms. In Experiment 2 we sought to eliminate such potential effects by using the same test materials at T1 and T2. As in Experiment 1, we used simple category cuing because this method is known to provide a very strong RIF effect (Anderson et al., 1994). We also increased the number of categories from six to eight in order to avoid possible ceiling effects at T2 due to use of the same test items. Thus Experiment 2 was designed to investigate the reliability of RIF perfor- mance under optimal conditions aimed at generating the largest possible RIF effect, by employing category cues, by using the same items at the two sessions, and by increas- ing the number of categories.

Method

Participants

Thirty participants took part in this study, all of whom were students at a British University. All 30 participants completed the study, but one was subsequently excluded as they had not followed the instructions correctly. Of the remaining 29 participants, there were 26 females and 3 males, with a mean age of 20.38 (SD = 4.26).

Materials

RIF Measurement

The RIF task was constructed in the same way as for Exper- iment 1, using 8 of the 12 categories to create one set of materials for use at both T1 and T2.

Procedure

The procedure was identical to that used in Experiment 1 except that participants studied, practiced, and were tested on, the same materials at T2 as at T1.

Results and Discussion

Table 3 shows the scores obtained on the RIF test in Exper- iment 2, each expressed as a percentage of the number of items in that particular retrieval condition. There were no significant effects or interactions in the RIF scores for the variants of the two tests, so these were amalgamated for the table and for the subsequent analyses. The average retrie- val-practice score was 98.5% (SD = 4.82).

A two-way analysis of variance was carried out, with the within-subjects factors of retrieval condition (Rp� and Nrp) and test session (T1 and T2). The analysis revealed signifi- cant main effects for retrieval condition, F(1, 28) = 83.1, p < .001 and for test session, F(1, 28) = 6.40, p < .05, with no significant interaction, F(1, 28) = 0.23, p = NS. Thus there was a significant RIF effect. In addition the scores at session T2 were significantly higher than those at T1, which would be consistent with a residual learning effect since the same test items were used at T1 and T2 in this experiment.

Table 4 shows the correlations between the scores obtained at the two test sessions. There was a significant cor- relation between RIF scores at sessions T1 and T2, indicat- ing that individual differences in RIF performance show significant test/retest reliability provided that the same items are used for both test sessions. It could be argued that, since the same test items are repeated, there is a possibility that the test/retest correlation may at least partially reflect the influ- ence of a residual learning effect, carrying over from one test session to the next. The fact that all scores were higher at T2 than at T1 is suggestive that such an effect may have been present. Residual forgetting effects should also be considered. While a number of studies (e.g., Garcia-Bajos, Migueles, & Anderson, 2008; Storm, Bjork, Bjork, & Nestojko, 2006) have found a RIF effect persisting for at least a week, in these cases the final test alone has been

Table 2. Pearson correlations between scores obtained at test sessions T1 and T2 (Experiment 1: category cues with different test items at T1 and T2)

Rp+ Rp� Nrp RIF score r .539** .382* .501** .148

Note. *p < .05, **p < .01; two-tailed.

Table 3. Mean % scores (SDs in brackets) obtained on the RIF test in test sessions T1 and T2 (Experiment 2: using category cues and with the same test items at T1 and T2)

Session Rp+ Rp� Nrp RIF score T1 87.64 (9.87) 35.35 (16.61) 56.47 (13.29) 21.12 (14.30) T2 88.50 (7.18) 40.23 (21.48) 62.79 (12.83) 22.55 (16.08)

4 R. Potts et al.: The Reliability of RIF

European Psychologist 2012; Vol. 17(1):1–10 � 2011 Hogrefe Publishing

T hi

s do

cu m

en t i

s co

py ri

gh te

d by

th e

A m

er ic

an P

sy ch

ol og

ic al

A ss

oc ia

tio n

or o

ne o

f i ts

a lli

ed p

ub lis

he rs

. T

hi s

ar tic

le is

in te

nd ed

s ol

el y

fo r t

he p

er so

na l u

se o

f t he

in di

vi du

al u

se r a

nd is

n ot

to b

e di

ss em

in at

ed b

ro ad

ly .

administered after the week’s delay, with no restudying or repracticing of the items. To our knowledge, the only study to have examined the effect on RIF of relearning and reprac- ticing the items was conducted by Storm, Bjork, and Bjork (2008) who gave participants several blocks of relearning and repracticing within a single session. They found that Rp� items benefitted more from relearning than did Nrp items, in line with one of the predictions of the new theory of disuse, that the extent to which an item benefits from relearning is a decreasing function of the accessibility or retrieval strength of that item at the time of relearning. Thus, if there had been an enduring effect of suppression of Rp� items over the week’s interval between T1 and T2, we might have expected the restudying of these items at T2 to insulate the Rp� items against further forgetting. However, this was not the case in our study, suggesting that the RIF effects observed at T2 were not simply carry-over effects from T1. The observation of a significant correlation in this exper- iment, therefore, left open the possibility that there may indeed be reliable individual differences in the mechanism underlying the RIF effect. Our next two experiments were designed to explore this possibility further and attempt to demonstrate reliability across different sets of materials. This was considered important since there is no one standard RIF task and studies of individual differences in RIF have used a variety of materials. For correlations between the RIF effect and other measures to be meaningful, the RIF effect should therefore be reliable across these different materials.

Experiments 1 and 2 both made use of a category-cued final retrieval test, but this procedure has been criticized by some recent researchers because category cues offer no con- trol over the order of retrieval, and thus no control over pos- sible output interference effects (Anderson, 2003; Anderson & Bell, 2001; Bauml & Hartinger, 2002). In order to control for output interference, a third experiment was carried out using category-plus-letter cues at final test.

Experiment 3. Repeated RIF Task Using Category-Plus-Letter Cues at the Final Retrieval Test and Different Items at Each Session

Experiment 3 investigated the hypothesis that RIF scores would show test/retest reliability when category-plus-letter cues were used in the final retrieval test. The category- plus-letter-cue procedure was developed as a way of control-

ling retrieval order (Anderson et al., 1994), so as to reduce output interference effects by ensuring that the Rp� and Nrp� items are all tested before commencing the testing of Rp+ and Nrp+ items. As in Experiment 1, different item sets were used for the two sessions.

Method

Participants

Thirty participants took part in this study, all of whom were students at a British University, comprising 21 females and 9 males, with a mean age of 20.9 (SD = 4.6).

Materials

RIF Measurement

Experiment 3 used a category-plus-letter cued procedure for the final retrieval test. A problem with the category-plus-let- ter cuing method is that RIF effects tend to be smaller than with simple category cuing and may disappear altogether (Butler et al., 2001). In order to maximize the possibility of observing a RIF effect, we used materials that have already been shown to produce RIF using the category- plus-letter cuing method. Two sets of RIF materials were constructed (Sets A and B), in the same way as for Experi- ment 1. For Set A we used materials drawn from Anderson et al. (1994) with a few substitutions for a British audience (e.g., ‘‘cockroach’’ for ‘‘roach’’). Exemplars had an average taxonomic rank order of eight according to the Battig and Montague (1969) category norms. For Set B we used items from Anderson and McCulloch (1999), with an average tax- onomic rank of 7.2. No item or category appeared in both test variants. No two items in a category began with the same letter. Each of the RIF tests therefore comprised 48 test items, made up of 12 practiced items (Rp+), 12 unpracticed items from practiced categories (Rp�), and a further 24 unpracticed items from unpracticed categories (Nrp). Since the order of output in this experiment was controlled by the cues provided, the 24 Nrp items were divided into 12 Nrp� items presented in the first half of the retrieval list as controls for the Rp� items and 12 Nrp+ items presented in the second half as controls for the Rp+ items.

Procedure

RIF testing again involved four phases. The first three phases in each session proceeded in exactly the same way as in Experiments 1 and 2. The last phase in each session, the final retrieval test, consisted of a category-plus-letter cued recall test as described above. All participants were administered Set A in the first session (T1) and Set B in the second session (T2). As in the two previous experiments, the two sessions T1 and T2 were separated by a period of 7 days. In the final retrieval test for each session, items were tested one at a time by presenting the category name plus

Table 4. Pearson correlations between scores obtained at test sessions T1 and T2 (Experiment 2: using category cues and with the same test items at T1 and T2)

Rp+ Rp� Nrp RIF score r .234 .634** .621** .440*

Note. *p < .05, **p < .01; two-tailed.

R. Potts et al.: The Reliability of RIF 5

� 2011 Hogrefe Publishing European Psychologist 2012; Vol. 17(1):1–10

T hi

s do

cu m

en t i

s co

py ri

gh te

d by

th e

A m

er ic

an P

sy ch

ol og

ic al

A ss

oc ia

tio n

or o

ne o

f i ts

a lli

ed p

ub lis

he rs

. T

hi s

ar tic

le is

in te

nd ed

s ol

el y

fo r t

he p

er so

na l u

se o

f t he

in di

vi du

al u

se r a

nd is

n ot

to b

e di

ss em

in at

ed b

ro ad

ly .

the initial letter of the word (e.g., Fruit-A) for 5 s each. The Rp� and Nrp� items were tested in random order in the first half of the list, then the Rp+ and Nrp+ items were tested in random order in the second half of the list. This arrange- ment ensured that Rp� items and their controls were pre- sented prior to the presentation of the Rp+ items and their controls, which minimized unwanted output interference effects, as advocated by Anderson (2003).

Results and Discussion

Table 5 shows the scores obtained on the RIF test in Exper- iment 3, each expressed as a percentage of the number of items in each retrieval condition. There were no significant effects or interactions in the RIF scores for the variants of the two tests, so these were amalgamated for the remaining analyses. The average retrieval-practice success rate was 81.1% (SD = 16.5).

A two-way analysis of variance revealed a significant main effect for RIF, F(1, 29) = 7.05, p < .01, but not for test session, F(1, 29) = 2.31, p = NS, and the interaction was not significant, F(1, 29) = 2.55, p = NS. The experi- ment thus produced a significant RIF effect.

Table 6 shows the correlations between the scores obtained at the two test sessions. There were significant test/retest correlations for Rp+ and Nrp�, but not for Rp�, Nrp+, nor the overall RIF scores. Thus Experiment 3 failed to find a significant correlation between RIF scores obtained at the two sessions. However, it is known that the use of category-plus-letter cues substantially reduces the magnitude of the observed RIF effect (Anderson et al., 1994) and may eliminate it altogether (Butler et al., 2001). This may in part be due to the elimination of output interfer- ence, as discussed. However, it has also been suggested (Levy, McVeigh, Marful, & Anderson, 2007) that the use of letter cues may have the effect of undoing suppression, thereby enabling access to a previously inhibited word. This could have resulted in a reduced or eliminated RIF effect which masked the true level of inhibition that occurred dur- ing retrieval practice. RIF scores in Experiment 3 were sub- stantially lower than in Experiments 1 and 2. It is thus possible that the failure to find correlations between RIF scores at T1 and T2 in this experiment may have been a con- sequence of the relatively low RIF scores obtained with cat- egory-plus-letter cues. Furthermore, Anderson and Levy (2007) have argued that RIF procedures making use of cat- egory cues or category-plus-letter cues may be affected by the CCB problem, as explained in the Introduction section. The CCB problem may lead to a reduction in the size of the RIF effect, as well as introducing an additional uncontrolled

variable into the RIF scores obtained. Two main approaches have been used to overcome the CCB problem: the first being the use of independent retrieval cues (Anderson & Levy, 2007) and the second being the use of a recognition procedure (Hicks & Starns, 2004; Soriano et al., 2009; Veling & Van Knippenberg, 2004). Some studies have failed to find a RIF effect when independent cues are employed (Camp et al., 2007; Perfect et al., 2002), so for the present study it was decided to make use of the recognition proce- dure. This was the procedure chosen for Experiment 4.

Experiment 4. Repeated RIF Task Using Recognition as the Final Retrieval Test

Experiment 4 investigated the hypothesis that RIF scores would show test/retest reliability when a recognition proce- dure was used as the final retrieval test. As in Experiments 2 and 3, different item sets were used for the test and retest sessions.

Method

Participants

Thirty-three participants took part in this study, all of whom were students at a British University. The sample comprised 22 females and 11 males, with a mean age of 20.6 (SD = 4.8).

Materials

RIF Measurement

The same test materials were used as in Experiment 3, except for the use of a recognition test at the final retrieval stage.

Table 5. Mean % scores (SDs in brackets) obtained on the RIF test in test sessions T1 and T2 (Experiment 3: category- plus-letter cues)

Session Rp+ Rp� Nrp+ Nrp� RIF score T1 63.89 (17.4) 59.4 (17.2) 27.8 (11.4) 68.9 (12.9) 9.44 (15.7) T2 60.28 (20.7) 58.3 (17.1) 29.2 (14.1) 61.1 (19.4) 2.78 (18.2)

Table 6. Pearson correlations between scores obtained at test sessions T1 and T2 (Experiment 3: category- plus-letter cues)

Rp+ Rp� Nrp+ Nrp� RIF score r .473** .220 .104 .452* .097

Note. *p < .05, **p < .01; two-tailed.

6 R. Potts et al.: The Reliability of RIF

European Psychologist 2012; Vol. 17(1):1–10 � 2011 Hogrefe Publishing

T hi

s do

cu m

en t i

s co

py ri

gh te

d by

th e

A m

er ic

an P

sy ch

ol og

ic al

A ss

oc ia

tio n

or o

ne o

f i ts

a lli

ed p

ub lis

he rs

. T

hi s

ar tic

le is

in te

nd ed

s ol

el y

fo r t

he p

er so

na l u

se o

f t he

in di

vi du

al u

se r a

nd is

n ot

to b

e di

ss em

in at

ed b

ro ad

ly .

The learning and retrieval-practice phases were presented as a Powerpoint presentation, as in the previous experiments. The final test was presented on paper and consisted of a list comprising the 48 test items interspersed with 48 distractor items. As in Experiment 3, output interference was controlled by presenting the Rp� and Nrp� items in the first half of the retrieval checklist, while the Rp+ and Nrp+ items were pre- sented in the second half. The first half of the recognition list thus contained the 12 Rp� items and the 12 Nrp� items, together with 24 distractor items, all mixed together in a ran- dom order. The second half of the recognition list contained the 12 Rp+ items and the 12 Nrp+ items, and a further 24 dis- tractor items, again mixed up in a random order. Participants were required to indicate whether or not they recognized each of the 96 items from the list presented earlier by circling ‘‘yes’’ or ‘‘no’’ against each word on the list. Distractor items were words chosen from the same eight categories as the 48 test items presented previously, and they were again selected from category norm tables (Van Overschelde et al., 2004) to ensure comparable frequencies across the lists. The design of this recognition test was based on the procedure used by Soriano et al. (2009).

Procedure

The procedure used in Experiment 4 was similar to that used in Experiment 3, except for the use of a recognition proce- dure as the final retrieval test at each session and a briefer presentation time (3 s) for each item in the study phase. The final test was given on paper. No time limit was imposed on completion of the final test but participants were instructed to work through the items as quickly as possible and not to return to a previous item. As in the first three experiments, the RIF procedure was presented on two sepa- rate occasions (T1 and T2), separated by a period of 7 days. Participants were administered different versions of the RIF test at each of the test sessions, receiving Set A at session T1 and Set B at session T2.

Results and Discussion

Analysis of Recognition Scores

Table 7 shows the scores obtained on the RIF test in Exper- iment 4, each expressed as a percentage of the number of items in each retrieval condition. There were no significant effects or interactions in the RIF scores for the different vari- ants of the two tests, so these were amalgamated for the remaining analyses. The average retrieval-practice success rate in the retrieval-practice phase was 82.7% (SD = 11.3).

A two-way analysis of variance was carried out on the scores in Table 7. A significant main effect was found for RIF, F(1, 32) = 2.95, p < .05, which confirms the occur- rence of a RIF effect in this experiment. A significant main effect was also found for test session, F(1, 32) = 4.5, p < .05, confirming that scores at time T1 were significantly higher than those at T2, possibly reflecting a difference

between the item sets. There were no significant interac- tions, F(1, 32) = 0.016, p = NS.

Although signal detection methodology such as d-prime is often used for analyzing the results of recognition tests, it is not considered to be suitable for use with the retrieval- practice paradigm, due to the potential for distractor items belonging to the practiced categories to be suppressed along with the Rp� items (see Anderson, 2003, for a detailed dis- cussion of this issue). Several studies have demonstrated RIF for unstudied items (Bauml & Kuhbandner, 2003; Camp et al., 2007; Starns & Hicks, 2004). If the familiarity of both Rp� items and their unstudied distractor counter- parts is reduced by retrieval practice, d-prime will not be able to detect a real impairment to Rp� items caused by suppression during retrieval practice. It was therefore decided not to analyze the present data using d-prime.

Table 8 shows the correlations between the scores obtained at the two test sessions T1 and T2. The correlation did not reach significance for RIF scores, but there were sig- nificant correlations between T1 and T2 scores for both Rp� and Nrp+, though these might possibly reflect the reli- ability of general memory performance.

Experiment 4, using recognition as the final retrieval test, thus failed to produce a significant correlation between RIF scores obtained at times T1 and T2 by the same individuals. It is possible that the failure to find reliable RIF in this experiment was a consequence of the relatively low RIF scores obtained with the recognition procedure, but other possible explanations will be considered in the general dis- cussion which follows.

General Discussion

The aim of this study was to determine the test/retest reli- ability of individual differences in the RIF effect and the conditions under which it may or may not be observed. All four experiments succeeded in demonstrating a signifi- cant RIF effect. However, only Experiment 2, using cate- gory cues at the final test and presenting the same test

Table 7. Mean % scores (SDs in brackets) obtained on the RIF test in test sessions T1 and T2 (Experiment 4: recognition cues)

Session Rp+ Rp� Nrp+ Nrp� RIF score T1 93.9 (9.1) 77.0 (16.0) 66.9 (18.3) 80.8 (12.0) 3.79 (16.1) T2 89.1 (12.0) 71.2 (14.6) 60.6 (12.0) 75.5 (17.8) 4.29 (19.0)

Table 8. Pearson correlations between scores obtained at test sessions T1 and T2 (Experiment 4: recogni- tion cues)

Rp+ Rp� Nrp+ Nrp� RIF score r .310 .359* .417* .128 .179

Note. *p < .05, **p < .01; two-tailed.

R. Potts et al.: The Reliability of RIF 7

� 2011 Hogrefe Publishing European Psychologist 2012; Vol. 17(1):1–10

T hi

s do

cu m

en t i

s co

py ri

gh te

d by

th e

A m

er ic

an P

sy ch

ol og

ic al

A ss

oc ia

tio n

or o

ne o

f i ts

a lli

ed p

ub lis

he rs

. T

hi s

ar tic

le is

in te

nd ed

s ol

el y

fo r t

he p

er so

na l u

se o

f t he

in di

vi du

al u

se r a

nd is

n ot

to b

e di

ss em

in at

ed b

ro ad

ly .

items at sessions T1 and T2, yielded a significant correlation between RIF scores at the two sessions. Experiment 1, which also used category cues but with different test items at the two test sessions, failed to produce a significant cor- relation between RIF scores at the two sessions. Experiment 3, using category-plus-letter cues, and Experiment 4, using a recognition test, also both failed to produce a significant cor- relation between T1 and T2 RIF scores.

Why did we find a correlation between T1 and T2 in Experiment 2 but not in our other three experiments? One possibility is that the mechanism underlying RIF is not reli- able, and this is why no correlation was found in Experi- ments 1, 3, and 4. In this case, the test/retest RIF correlation found in Experiment 2 may be an artifact arising out of the occurrence of residual learning effects when the same test items are used at both sessions. The learning of items at session T1 may have facilitated the relearning of those same items when they were presented again at session T2. However, it is unlikely that residual learning effects would completely explain such a high correlation for the RIF scores, bearing in mind that these RIF scores are calcu- lated by taking the difference between Rp� and Nrp� scores. It is also noteworthy that no corresponding T1/T2 correlation was found for the Rp+ (practiced) items in this experiment, which suggests that residual learning effects were probably not strong enough to provide a complete explanation for the correlation found.

An alternative explanation for the finding of a test/retest correlation in Experiment 2 but not in the other three exper- iments is that there are in fact reliable individual differences in RIF performance, but these are difficult to observe when different test materials are used at both sessions. If RIF per- formance is to some extent item-specific, an individual could exhibit a high amount of RIF when tested on one set of materials and a lower amount when tested on another. The items in the two RIF tests were equated as far as possi- ble for strength of association according to category norm tables, but such tables indicate only the average response and not that of individual participants. The RIF effect is known to depend on the degree of item competition during retrieval practice, and since it is only strong competitors which need to be suppressed (Anderson et al., 1994), varia- tions in the strength of association between the item and its category for different participants will lead to variations in the amount of observed RIF. In this case, a lack of sensitiv- ity of the task could be masking a true reliability effect. This hypothesis is consistent with the findings of the present study, which achieved a significant RIF correlation in Exper- iment 2 but not in the other three experiments. It is also pos- sible that the fairly low correlations observed in these experiments might be a consequence of the relatively small size of the RIF effect obtained, as the resulting low variance in RIF scores would clearly make it more difficult to achieve significance. This is especially likely in Experiments 3 and 4, where the RIF effect is of relatively low magnitude. This is in keeping with the previous findings of Butler et al. (2001), who reported that the RIF effect actually failed to reach significance in an experiment using letter cues, and Levy et al. (2007) have argued that letter cues may have the effect of counteracting inhibitory effects on a test item.

Again, lack of task sensitivity may therefore make it difficult to observe a true effect of reliability.

One consideration in interpreting our findings is the fact that, in the traditional RIF paradigm, participants are not explicitly told that they will be given a final retrieval test. It could thus be argued that participants who perform the RIF procedure twice might not expect the final test at T1 but would expect it at T2 and consequently may have applied different strategies for remembering the items at T2. However, there are several reasons why this is unlikely to have had an effect on our results. Firstly, our participants were told at the start of the study phase that they should try to remember as many of the studied items as they could, so it is likely that even at T1 they expected to be tested at some point for recall of all of the items. Secondly, participants were not explicitly told that they would be undergoing the same procedure at T2 as at T1, so they would have no par- ticular reason to adopt a different strategy in the light of their experience of T1. It is also noteworthy that all four of our experiments achieved a RIF effect, suggesting that at both sessions participants were actively engaged in trying to remember studied items. Previous studies have shown that no RIF effect was found when participants did not try to remember the studied items (Storm, Bjork, & Bjork, 2007). Moreover, Camp, Pecher, and Schmidt (2005) found, on the basis of posttest questionnaires, that the majority of participants in a RIF experiment were aware of the link between the study and the test phase, and it was only these ‘‘aware’’ participants who showed a RIF effect. Finally, our second experiment, using the same materials at T1 and T2, did show a correlation between RIF scores at the two ses- sions. If participants were adopting different strategies at T1 and T2, there is no reason to suppose they would not also have done this in Experiment 2. Thus it seems unlikely that the adoption of different strategies by participants at T2 is responsible for the lack of a correlation in our other experiments.

The overall conclusion to be drawn from the present findings, taking all four experiments together, is that individ- ual differences in observed RIF are reliable only where the same item set is repeated, and there is no clear evidence for reliability of RIF performance across different item sets. However, previous studies have reported correlations between RIF performance and various cognitive and clinical factors. For example, Groome and Sterkaj (2010) found a significant inverse correlation between RIF and depression, though the direction of causality in this relationship remains unknown. The findings of the present study may shed some light on the probable direction of causality in such correla- tions. The hypothesis that RIF might exert a causal influence on depression assumes the existence of reliable individual differences in RIF, an assumption which is not supported by the findings of the present study. The present findings would appear to be more consistent with the hypothesis that variations in depression exert a causal influence on RIF performance, since this hypothesis does not assume the existence of a constant and reliable level of RIF performance in a given individual. Indeed this hypothesis would pre- dict variations in RIF performance for a given individ- ual in response to changes in their level of depression.

8 R. Potts et al.: The Reliability of RIF

European Psychologist 2012; Vol. 17(1):1–10 � 2011 Hogrefe Publishing

T hi

s do

cu m

en t i

s co

py ri

gh te

d by

th e

A m

er ic

an P

sy ch

ol og

ic al

A ss

oc ia

tio n

or o

ne o

f i ts

a lli

ed p

ub lis

he rs

. T

hi s

ar tic

le is

in te

nd ed

s ol

el y

fo r t

he p

er so

na l u

se o

f t he

in di

vi du

al u

se r a

nd is

n ot

to b

e di

ss em

in at

ed b

ro ad

ly .

This conclusion would also be consistent with the finding of a reduction in the RIF effect when a negative mood was induced in healthy participants (Bauml & Kuhbandner, 2007), suggesting that the strength of the RIF effect can vary according to participants’ current state. Thus the findings of the present study may help to resolve the question of causal direction in the relationship between RIF and depression, though of course it is not possible to draw firm conclusions from nonsignificant results.

Our findings have important implications for studies of individual differences in RIF performance, particularly those which rely on correlational designs. In particular they sug- gest that individual differences in RIF performance are not reliable across time. This may reflect the controlling influ- ence upon RIF of a number of other factors, such as depres- sion. Our findings thus highlight the importance of controlling for factors such as negative mood or other tran- sient states which may affect the observed RIF score. It is possible that the reliability of individual differences in RIF may be so small as to be undetectable when different mate- rials are used at the two sessions. Indeed one challenge for researchers investigating individual differences and RIF is to develop a version of the task which is sufficiently sensitive to minimize variations caused by differences in materials.

In summary, the results of the present study indicate that the test/retest reliability of individual differences in RIF did not reach significance, except where the same test items were repeated at both test sessions.

References

AhnAllen, C. G., Nestor, P. G., McCarley, R. W., & Shenton, M. E. (2007). The role of retrieval inhibition in the associative memory impairment of schizophrenia. Psychiatry Research – Neuroimaging, 150, 43–50.

Anderson, M. C. (2003). Rethinking interference theory: Exec- utive control and the mechanisms of forgetting. Journal of Memory and Language, 49, 415–445.

Anderson, M. C., & Bell, T. A. (2001). Forgetting our facts: The role of inhibitory processes in the loss of propositional knowledge. Journal of Experimental Psychology: General, 130, 544–570.

Anderson, M. C., Bjork, R. A., & Bjork, E. L. (1994). Remem- bering can cause forgetting: Retrieval dynamics in long-term memory. Journal of Experimental Psychology: Learning, Memory and Cognition, 20, 1063–1087.

Anderson, M. C., & Levy, B. J. (2007). Theoretical issues in inhibition: Insights from research on human memory. In D. S. Gorfein & C. M. MacLeod (Eds.), Inhibition and Cognition. Washington, DC: American Psychological Association.

Anderson, M. C., & McCulloch, K. C. (1999). Integration as a general boundary condition on retrieval-induced forgetting. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25, 608–629.

Anderson, M. C., & Spellman, B. A. (1995). On the status of inhibitory mechanisms in cognition: Memory retrieval as a model case. Psychological Review, 102, 68–100.

Aslan, A., Bauml, K.-H., & Pastotter, B. (2007). No inhibitory deficit in older adults’ episodic memory. Psychological Science, 18, 72–78.

Battig, W. F., & Montague, W. E. (1969). Category norms for verbal items in 56 categories: A replication and extension of

the Connecticut category norms. Quarterly Journal of Experimental Psychology: Monograph, 80, 1–46.

Bauml,K.-H.,&Hartinger,A.(2002).Ontheroleofitemsimilarityin retrieval-induced forgetting. Memory, 10, 215–224.

Bauml, K.-H., & Kuhbandner, C. (2007). Retrieval-induced forgetting and part-list cuing in associatively structured lists. Memory & Cognition, 31, 1188–1197.

Butler, K., Williams, C., Maki, R., & Zacks, R. (2001). A limit on retrieval-induced forgetting. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27, 1314– 1319.

Camp, G., Pecher, D., & Schmidt, H. G. (2005). Retrieval- induced forgetting in implicit memory tests: The role of test awareness. Psychonomic Bulletin & Review, 12, 490–494.

Camp, G., Pecher, D., & Schmidt, H. G. (2007). No retrieval- induced forgetting using item-specific independent cues: Evidence against an inhibitory account. Journal of Experi- mental Psychology: Learning, Memory, and Cognition, 33, 950–958.

Garcia-Bajos, E., Migueles, M., & Anderson, M. C. (2008). Script knowledge modulates retrieval-induced forgetting for eyewitness events. Memory, 17, 92–103.

Groome, D., & Grant, N. (2005). Retrieval-induced forgetting is inversely related to everyday cognitive failures. British Journal of Psychology, 96, 313–319.

Groome, D., & Sterkaj, F. (2010). Retrieval-induced forgetting and clinical depression. Cognition and Emotion, 24, 63–70.

Hicks, J. L., & Starns, J. L. (2004). Retrieval-induced forgetting occurs in tests of item recognition. Psychonomic Bulletin & Review, 11, 125–130.

Levy, B., & Anderson, M. (2008). Individual differences in the suppression of unwanted memories: The executive deficit hypothesis. Acta Psychologica, 127, 623–635.

Levy, B., McVeigh, N., Marful, A., & Anderson, M. C. (2007). Inhibiting your native language. The role of retrieval-induced forgetting during second-language acquisition. Psychological Science, 18, 29–34.

Nestor, P. G., Piech, R., Allen, C., Nizinikiewicz, M., Shenton, M., & McCarley, R. W. (2005). Retrieval-induced forgetting in schizophrenia. Schizophrenia Research, 75, 199–209.

Perfect, T. J., Conway, M. A., Moulin, C. J. A., & Perry, E. (2002). Assessing the inhibitory account of retrieval-induced forgetting with implicit memory tests. Journal of Experi- mental Psychology: Learning, Memory, and Cognition, 28, 1111–1119.

Saunders, J., & Macleod, M. D. (2002). New evidence on the suggestibility of memory: The role of retrieval-induced forgetting in the misinformation effect. Journal of Experi- mental Psychology: Applied, 8, 127–142.

Saunders, J., & Macleod, M. D. (2006). Can inhibition resolve retrieval competition through the control of spreading activation? Memory & Cognition, 34, 307–322.

Soriano, M. F., Jimenez, J. F., Roman, P., & Bajo, M. T. (2009). Inhibitory processes in memory are impaired in schizophre- nia: Evidence from retrieval induced forgetting. British Journal of Psychology, 100, 661–674.

Starns, J. J., & Hicks, J. L. (2004). Episodic generation can cause semantic forgetting: Retrieval-induced forgetting of false memories. Memory & Cognition, 32, 602–609.

Storm, B. C., Bjork, E. L., & Bjork, R. A. (2007). When intended remembering leads to unintended forgetting. Quar- terly Journal of Experimental Psychology, 60, 909–915.

Storm, B. C., Bjork, E. L., & Bjork, R. A. (2008). Accelerated relearning after retrieval-induced forgetting: The benefit of being forgotten. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34, 230–236.

Storm, B. C., Bjork, E. L., Bjork, R. A., & Nestojko, J. F. (2006). Is retrieval success a necessary condition for retrieval

R. Potts et al.: The Reliability of RIF 9

� 2011 Hogrefe Publishing European Psychologist 2012; Vol. 17(1):1–10

T hi

s do

cu m

en t i

s co

py ri

gh te

d by

th e

A m

er ic

an P

sy ch

ol og

ic al

A ss

oc ia

tio n

or o

ne o

f i ts

a lli

ed p

ub lis

he rs

. T

hi s

ar tic

le is

in te

nd ed

s ol

el y

fo r t

he p

er so

na l u

se o

f t he

in di

vi du

al u

se r a

nd is

n ot

to b

e di

ss em

in at

ed b

ro ad

ly .

induced forgetting? Psychonomic Bulletin & Review, 13, 1023–1027.

Van Overschelde, J. P., Rawson, K. A., & Dunlosky, J. (2004). Category norms: An updated and expanded version of the Battig & Montague (1969) norms. Journal of Memory and Language, 50, 289–335.

Veling, H., & Van Knippenberg, A. (2004). Remembering can cause forgetting: Retrieval-induced forgetting as cue inde- pendent process. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30, 315–318.

Received July 14, 2009 Accepted December 18, 2009 Published online January 26, 2011

About the authors

Rosalind Potts, Robin Law, Professor John Golding, and Dr. David Groome are all members of staff in the Psychology Department at the University of Westminster, and their research interests currently center around memory and cognitive inhibition. John Golding has also published research on cogni- tion and motion sickness. David Groome has published research on memory and mood disorders, and he has also coauthored four text books on various aspects of cognition.

David Groome

Department of Psychology University of Westminster 309 Regent Street London W1B 2UW UK Tel. +44 20 7911 5000 ext. 2042 E-mail [email protected]

10 R. Potts et al.: The Reliability of RIF

European Psychologist 2012; Vol. 17(1):1–10 � 2011 Hogrefe Publishing

T hi

s do

cu m

en t i

s co

py ri

gh te

d by

th e

A m

er ic

an P

sy ch

ol og

ic al

A ss

oc ia

tio n

or o

ne o

f i ts

a lli

ed p

ub lis

he rs

. T

hi s

ar tic

le is

in te

nd ed

s ol

el y

fo r t

he p

er so

na l u

se o

f t he

in di

vi du

al u

se r a

nd is

n ot

to b

e di

ss em

in at

ed b

ro ad

ly .