Discussion 4

Prediction and Control of Operant Behavior: What You See Is Not All There Is

Mark E. Bouton University of Vermont

Bernard W. Balleine University of New South Wales

Prediction and control of operant behavior are major goals of behavior analysis. We suggest that achieving these goals can benefit from doing more than identifying the 3-term contingency among the behavior, its setting stimulus, and its consequences. Basic research now underscores the idea that prediction and control require consider- ation of the behavior’s history. As one example, if an operant is a goal-directed action, it is controlled by the current value of the reinforcer, as illustrated by the so-called reinforcer devaluation effect. In contrast, if the behavior is a habit, it occurs automat- ically, without regard to the reinforcer’s value, as illustrated by its insensitivity to the reinforcer devaluation effect. History variables that distinguish actions and habits include the extent of their prior practice and their schedule of reinforcement. Other operants can appear to have very low or zero strength. However, if the behavior has reached that level through extinction or punishment, it may precipitously increase in strength by changing the context, allowing time to pass, presenting the reinforcer contingently or noncontingently, or extinguishing an alternative behavior. Behaviors that are not suppressed by extinction or punishment are not affected the same way. When predicting the strength of an operant behavior, what you see is not all there is. The behavior’s history counts.

Keywords: action, habit, extinction, punishment, behavioral history

A major goal of behavior analysis is the prediction and control of operant behavior. Accordingly, since at least Skinner (1938, 1969), and extending through today’s use of functional analysis (e.g., Hanley, Iwata, & McCord, 2003), it has been common to em- phasize the three-term contingency in the analysis of an operant response: that is, (1)

the occasion within which the response oc- curs, (2) the response itself, and (3) the rein- forcing consequences of the response. Various terms for the elements of the contingency have been used over the years: for example, the occa- sion has been variously called a discriminative stimulus, state, establishing operation, or context; the response an operant or action; and the rein- forcer a reward, consequence, or outcome. But, whatever terms are used, the approach has implied that to predict and control a response it is suffi- cient to identify its specific setting conditions—its specific occasion—and to develop the means to selectively apply reinforcement. Furthermore, al- though it has been conceded that identifying and applying these key events might prove challeng- ing in practice, from this perspective it is not a problem in principle: An important assumption of the behavior analytic tradition is that each term of the contingency is open to direct observation. Therefore, accurately describing the situation, the response, and its consequences will be sufficient to specify the factors that control any operant.

This article was published Online First June 18, 2018. Mark E. Bouton, Department of Psychological Science,

University of Vermont; Bernard W. Balleine, Decision Neu- roscience Laboratory, School of Psychology, University of New South Wales.

Preparation of this article was supported by National Institutes of Health Grant RO1 DA 033123 to Mark E. Bouton, and by a grant from the Australian Research Coun- cil (DP150104878) and a Senior Principle Research Fellow- ship from the National Health and Medical Research Coun- cil of Australia (GNT1079561) to Bernard W. Balleine. We thank Eric Thrailkill for comments.

Correspondence concerning this article should be ad- dressed to Mark E. Bouton, Department of Psychological Science, University of Vermont, 2 Colchester Avenue, Bur- lington, VT 05405-0134. E-mail: [email protected]

T hi

s do

cu m

en t

is co

py ri

gh te

d by

th e

A m

er ic

an Ps

yc ho

lo gi

ca l

A ss

oc ia

tio n

or on

e of

its al

lie d

pu bl

is he

rs .

T hi

s ar

tic le

is in

te nd

ed so

le ly

fo r

th e

pe rs

on al

us e

of th

e in

di vi

du al

us er

an d

is no

t to

be di

ss em

in at

ed br

oa dl

y.

Behavior Analysis: Research and Practice © 2018 American Psychological Association 2019, Vol. 19, No. 2, 202–212 2372-9414/19/$12.00 http://dx.doi.org/10.1037/bar0000108

202

Fundamentally, this means that if a behavior analyst wants to increase or decrease the strength of some specific operant, she or he needs to find its antecedents or consequences and manipulate them accordingly. But basic research from the animal learning laboratory now provides additional infor- mation that might help predict what will be effec- tive. Two operants that look very much alike may actually have very different properties; depending on their learning histories, they may be controlled by different kinds of event, or influenced quite differently by the same event. In this article, we consider two sets of examples drawn from the current literature. In the first, we show that two seemingly indistinguishable operants that occur at a stable rate might have a different status as either goal-directed actions or habits. This influences the variables or events that control them. In the second, we consider what we call silent operants, which have very low or zero strength. Depending on their actual reinforcement histories, they can respond quite differently to the same event. Both examples violate what Daniel Kahneman (2011) has called the “what you see is all there is” heu- ristic used in human decision making. When it comes to operant behavior, what you see is not all there is. Knowledge of a behavior’s history is useful to predict it and control it accurately.

Actions Versus Habits

Historically, the three-term contingency was developed to describe responses that, at that time, were commonly called habits (e.g., Hull, 1943). In accord with the then-dominant neobe- haviorist position, the performance of any op- erant was thought to reflect a stimulus–response association that was strengthened by the selec- tive application of a reinforcer. In the paradigm case, a hungry rat trained to press a lever for food was argued to do so in the presence of various situational, contextual, or discriminative cues (serving as the S) with which the lever-press re- sponse (the R) became associated due to the de- livery of food (the reinforcer) after the response. Such a description accords with the argument that merely observing the rat lever pressing provides the basis for an adequate description of the re- sponse and of the contingency controlling that response.

Almost immediately, a number of findings began to question the sufficiency of this ac- count. Perhaps the earliest indication that some-

thing more than the simple situation–response association was controlling performance came from incentive contrast studies. Generally, these studies demonstrated that rats were more con- cerned with the relative value of the conse- quences of their actions than they should be if the consequence merely served a reinforcing function. Early studies reported that suddenly changing the value of an earned outcome, for example, changing the food reward from banana to lettuce in monkeys (Tinklepaugh, 1928), caused an immediate and powerful change in behavior from the pursuit of food to frustration and aggres- sion. In one of the more famous studies involving rats running in a straight runway, a sudden reduc- tion in the amount of food earned by running to the goal box caused an immediate reduction in performance of the response on subsequent trials to a level that was below that of animals whose responding was maintained by that reduced amount of food from the outset, the so-called negative contrast effect (Crespi, 1942). Although reinforcement theories could address the overall reduction in performance, the rapidity with which the reduction occurred and the actual depression in responding relative to controls—the contrast effect—was difficult to explain in reinforcement terms. What these results suggested was that rather than merely being controlled by the situa- tional cues—or the occasion—within which the operant was performed, it was also influenced by incentive motivation, which required mediation by some sort of representation or expectation of the consequence that the operant produced (e.g., compare Spence, 1956, and Tolman, 1932).

There have been numerous similar findings in mazes and runways supporting this general claim (see Flaherty, 1996, for a review), as well as studies showing that even lever pressing in rats can be controlled by a representation of the reward value of the food. Adams and Dickinson (1981), for example, reported that hungry rats trained to lever-press for sugar solution were strongly sensitive to changes in the value of the sugar when it was altered offline by conditioned taste aversion. In this study, acquisition of lever pressing for sugar and then taste aversion con- ditioning to sugar each occurred in separate phases. In a final test, the rats were returned to the lever-press apparatus and lever-pressing performance was assessed in extinction. (Note that conducting the test in extinction prevented the opportunity to pair the lever-press response

203PREDICTION AND CONTROL OF OPERANT BEHAVIOR

T hi

s do

cu m

en t

is co

py ri

gh te

d by

th e

A m

er ic

an Ps

yc ho

lo gi

ca l

A ss

oc ia

tio n

or on

e of

its al

lie d

pu bl

is he

rs .

T hi

s ar

tic le

is in

te nd

ed so

le ly

fo r

th e

pe rs

on al

us e

of th

e in

di vi

du al

us er

an d

is no

t to

be di

ss em

in at

ed br

oa dl

y.

directly with the now-devalued sugar in expe- rience.) Importantly, the rats with the new taste aversion to sugar nevertheless reduced their le- ver pressing immediately relative to a group that had received the sugar and the illness unpaired. This is the so-called reinforcer devaluation ef- fect. As a consequence of this finding, the au- thors concluded that lever pressing in rats, as implied by the performance observed in mazes and runways, is goal-directed and influenced by the organism’s knowledge of the current value of the reinforcer. The organism learns about the behavior and its reinforcer, and when the rein- forcer’s value changes, the strength of the op- erant adjusts accordingly.

The benefit of encoding the actual conse- quence of an action is the flexibility it provides when the value of that consequence changes even without any obvious shift in the context or occasion in which the response is produced. Rather than requiring an animal to perform a response to learn that its consequences are no longer valuable (and potentially noxious), inte- grating the relationship between the action and its consequences with its experience of the al- tered value of those consequences is sufficient to alter the performance of the response. This is not always true, however. Although the above studies suggest that operants can be goal di- rected, their performance can also appear to be consistent with simpler reinforcement theory. For example, Adams (1982) trained two groups of rats to lever press on a continuous reinforce- ment schedule for sugar solution. One group was given the opportunity to make 100 rein- forced lever presses, whereas the other was given the opportunity to make 500 reinforced lever presses. After this training, half of each group was given the taste aversion (reinforcer devaluation) treatment in which the sugar was paired with illness, whereas the other half were given the control treatment with the sugar un- paired with illness. Lever pressing was then assessed in all animals in extinction. Despite the fact that the rats were all pressing the lever in a seemingly identical manner before devaluation— suggesting that, from a purely observational per- spective, similar contingencies should be con- trolling their performance—the different amounts of training produced very different forms of behavioral control. The rats given only a moderate amount of training were sensitive to the outcome devaluation treatment and reduced

responding compared to the nondevalued con- trol group. In contrast, rats given extended train- ing did not show this effect; although the rats given the devaluation treatment had a strong taste aversion to the sucrose, they lever-pressed on test in a similar manner to the unpaired control group. As Adams observed, it appeared that with moderate training the rats’ actions were goal- directed, whereas when given extended training, they had become habits.

These early studies suggested that, depending on the amount of training, it is possible for lever pressing to become goal directed or habitual. That is, the contingency controlling an operant can either be driven by a situation-response- reinforcement contingency, in the case of hab- its, or by the association between the action and outcome, in the case of goal-directed actions. Subsequent studies have established a number of other important differences between goal- directed actions and habits. For example, (1) goal-directed actions develop and change very quickly and so can govern choice performance: Colwill and Rescorla (1985) were the first to establish that the devaluation of an action alters the rats’ willingness to choose its associated action without affecting the performance of other, simul- taneously available actions. Furthermore, (2) rather than being controlled by the context or situation, goal-directed actions are controlled by their association with a specific outcome (Colwill & Rescorla, 1986). As a consequence, goal- directed actions are sensitive to degradation of the action-outcome contingency and selectively so; in a situation where two actions are trained with different reinforcers, adding deliveries of one or other reinforcer that are not contiguous with its associated action reduces the performance of that action while leaving other actions unaffected (Bal- leine & Dickinson, 1998; Dickinson & Mulatero, 1989). Indeed, (3) when lengths are taken to make the performance of a goal-directed action condi- tional on a discriminative stimulus (SD), control by the SD is not over the response itself but rather over the response–outcome association (Bradfield & Balleine, 2013; Rescorla, 1991; see also Trask & Bouton, 2014). Other studies have investigated the influence of different forms of devaluation and revaluation of the instrumental reinforcer using shifts in primary motivation (Balleine, 1992; Dickinson & Dawson, 1987) and incentive learn- ing manipulations (Balleine, 2001; Dickinson &

204 BOUTON AND BALLEINE

T hi

s do

cu m

en t

is co

py ri

gh te

d by

th e

A m

er ic

an Ps

yc ho

lo gi

ca l

A ss

oc ia

tio n

or on

e of

its al

lie d

pu bl

is he

rs .

T hi

s ar

tic le

is in

te nd

ed so

le ly

fo r

th e

pe rs

on al

us e

of th

e in

di vi

du al

us er

an d

is no

t to

be di

ss em

in at

ed br

oa dl

y.

Balleine, 1994), including sensory-specific satiety (Balleine & Dickinson, 1998).

In contrast, habits have been found to be difficult to develop in choice situations; despite extensive overtraining, when an explicit choice between two actions is always made available, rats continue to show sensitivity to outcome devaluation (Colwill & Rescorla, 1988; Kosaki & Dickinson, 2010). Furthermore, in contrast to goal-directed actions, habits are strongly and immediately affected by shifts in motivational state (Dickinson, Balleine, Watt, Gonzalez, & Boakes, 1995) and by shifts in context (Thrailkill & Bouton, 2015). And as their strong relationship to the context suggests, habits have been found to be insensitive to treatments that would otherwise degrade the response-outcome contingency (Dez- fouli & Balleine, 2012; Dickinson et al., 1998).

It is important to note that the distinction between goal-directed actions and habits is not unique to rats but is also found in humans. Relatively moderately trained actions in chil- dren above the age of 3 years (but not below) show sensitivity to reinforcer devaluation (Klossek, Russell, & Dickinson, 2008), whereas overtraining can render actions insensitive to out- come devaluation and make them habitual (Tri- comi, Balleine, & O’Doherty, 2009). It should also be noted that actions and habits are not only mediated by distinct contingencies and learning rules but also by distinct neural systems: Balleine and colleagues have established that a cortical basal ganglia circuit involving the medial prefron- tal cortex and caudate/dorsomedial striatum in hu- mans and in rodents mediates goal-directed ac- tions, whereas a parallel circuit involving the sensorimotor cortices and the putamen/dorsolat- eral striatum mediates habits (see Balleine & O’Doherty, 2010, for a review). The dissociation of these circuits provides even more evidence for a division between goal-directed actions and hab- its as fundamentally distinct forms of behavioral control.

Finally, the division between goal-directed actions and habits extends into abnormal behav- ior (Griffiths, Morris, & Balleine, 2014). For example, addiction is often characterized both as a loss of behavioral control and the develop- ment of a drug-taking habit (e.g., Everitt & Robbins, 2005). Consistent with this character- ization, although operant responding for every- day rewards (like chocolate or sugar) in animals

and humans can appear relatively normal com- pared to healthy controls, tests have found that drug exposure attenuates goal-directed action control and causes downregulation of the neural circuitry associated with goal-directed action while causing increases in habitual control and in habit-related circuitry (Furlong et al., 2017; Hogarth, Balleine, Corbit, & Killcross, 2013). Similarly, deficits in goal-directed action con- trol have been found in adolescents with depres- sion, social anxiety, or autism spectrum disor- der and in adults with chronic schizophrenia, based on devaluation tests (Alvares et al., 2014, 2016; Morris, Quail, Griffiths, Green, & Bal- leine, 2015). Conversely, there is some evi- dence to suggest that disorders such as attention deficit/hyperactivity disorder and degenerative conditions, such as Parkinson’s and Hunting- ton’s diseases, produce deficits in the habitual control of operant responding, causing deficits in operant performance due to the demands of multitasking (Griffiths et al., 2014; Redgrave et al., 2010).

In summary, many studies have pointed to the fact that, depending on the training condi- tions, the performance of an operant behavior can be controlled by very different contingen- cies. Goal-directed actions are malleable, flexi- ble, rapidly acquired, and relatively readily sup- pressed or even eliminated from the response repertoire. In contrast, habits are inflexible, au- tomatic, and difficult to change or to suppress. Goal-directed actions are less dependent on the situation or occasion for support than habits, which are highly situationally dependent, and although goal-directed actions depend on the value of their specific consequences, habits are not concerned with the specific properties or values of their consequences at all. Overall, therefore, in operant or instrumental condition- ing, what you see can be ambiguous; an action can appear to be controlled by situational cues, but on closer inspection it may be found that shifts in context have little affect performance; it may appear to be controlled by its conse- quences but subsequent changes in the value of those consequences may demonstrate that it is not. To specify the contingency controlling a response requires, therefore, knowledge of the operant’s history and/or additional testing and assessment.

205PREDICTION AND CONTROL OF OPERANT BEHAVIOR

T hi

s do

cu m

en t

is co

py ri

gh te

d by

th e

A m

er ic

an Ps

yc ho

lo gi

ca l

A ss

oc ia

tio n

or on

e of

its al

lie d

pu bl

is he

rs .

T hi

s ar

tic le

is in

te nd

ed so

le ly

fo r

th e

pe rs

on al

us e

of th

e in

di vi

du al

us er

an d

is no

t to

be di

ss em

in at

ed br

oa dl

y.

Operants With Low or Zero Strength

A related case may be made concerning op- erant behaviors that have very low or even zero rates. How they react to treatment in the future depends crucially on how they achieved those rates. As we describe next, if the response was once reinforced, but was then reduced or elim- inated through extinction or punishment, it may return or relapse after some new, precipitating event occurs. Importantly, the return of the re- sponse cannot be predicted from simple obser- vations taken before the new event. Again, what you see in behavior is not all there is.

Contemporary research has identified a num- ber of triggering or precipitating events. One of the best studied is a change in the background context, typically the Skinner box or operant chamber in which the animal performs, which can be individuated by different floor composi- tion, spatial locations, and scents. If the context is changed after extinction, an extinguished op- erant can return or “renew.” The renewal effect can take different forms. In the most-studied version, a behavior is reinforced in one context (Context A) and then extinguished in another (Context B). If the behavior is then tested in Context A, the response can return—a phe- nomenon known as ABA renewal. Similarly, if the response is tested in a third context (Context C) after reinforcement and extinction in A then B, responding also recovers (ABC renewal). And if responding is reinforced in A and then extin- guished in the same context (Context A), the response also renews when tested in a second context (AAB renewal). All three forms of re- newal have been tested and confirmed with op- erant behavior (e.g., Bouton, Todd, Vurbic, & Winterbauer, 2011; Todd, 2013). ABA renewal also occurs after extinction in children with developmental disabilities (Kelley, Liddon, Ri- beiro, Greif, & Podlesnik, 2015; see Podlesnik, Kelley, Jimenez-Gomez, & Bouton, 2017, for a recent review). And it is also the case that, in animals at least, renewal also occurs if the op- erant has been suppressed by punishment in- stead of extinction. ABA and ABC renewal have both been observed after positive punish- ment, where the response is suppressed by con- tingent footshock (Bouton & Schepers, 2015); and ABA renewal occurs if the response has been suppressed by an omission (differential reinforcement of other behavior [DRO]) contin-

gency, that is, negative punishment (Nakajima, Urushihara, & Masaki, 2002). The renewal ef- fect suggests that neither extinction nor punish- ment destroys the original operant learning. In- stead, the animal learns to refrain from making the response (e.g., Bouton, Trask, & Carranza- Jasso, 2016), and this inhibition is relatively specific to the context in which it is learned (Todd, Vurbic, & Bouton, 2014). Thus, a be- havior that has very low or apparently zero strength can come alive again if the context is changed. A suppressed or inhibited operant can be distinguished from a behavior with true zero strength by testing the effect of changing the context.

There are other events that can cause the return or relapse of extinguished or punished operants. In spontaneous recovery, the mere passage of time after extinction (e.g., Rescorla, 2004) or punish- ment (Estes, 1944) can cause the response to return again. In our view (e.g., Bouton, 1988), spontaneous recovery is another renewal effect in which the context change is provided by the passage of time. Just as extinguished respond- ing renews when it is tested in a new physical context, it recovers when it is tested in a new temporal context. In fact, many different types of stimuli can play the role of context (e.g., Bouton, 2002). In state-dependent learning, drug states provide the context (e.g., Overton, 1985); for example, when extinction occurs when the organism is under the influence of a drug like a benzodiazepine or alcohol, the re- sponse renews when the animal is tested with- out the drug, that is, sober again (e.g., Bouton, Kenney, & Rosengard, 1990; see also Cunning- ham, 1979; Lattal, 2007). And recent experi- ments have suggested that hunger state can be a context: When rats lever pressed for sucrose or sweet-fatty pellets (rodent junk food) while they were satiated and then received extinction while they were hungry, the response recovered when it was tested while the rats were satiated again (Schepers & Bouton, 2017). Such renewal may explain the difficulties faced by dieters who might eat when they do not need food and then starve themselves on a diet—the inhibition of eating when dieting may not transfer too well to the satiated state. Other recent research has shown that when the organism must make a sequence of two separate responses to earn a reinforcer (specifically, in a discriminated het- erogeneous chain), the first response provides a

206 BOUTON AND BALLEINE

T hi

s do

cu m

en t

is co

py ri

gh te

d by

th e

A m

er ic

an Ps

yc ho

lo gi

ca l

A ss

oc ia

tio n

or on

e of

its al

lie d

pu bl

is he

rs .

T hi

s ar

tic le

is in

te nd

ed so

le ly

fo r

th e

pe rs

on al

us e

of th

e in

di vi

du al

us er

an d

is no

t to

be di

ss em

in at

ed br

oa dl

y.

kind of context for the second response (Thrailkill & Bouton, 2016). For example, if the second response is extinguished alone, apart from the chain, it is renewed when it is returned to the chain and tested after the rat has made the first response (Thrailkill et al., 2016). The first response is more affected by changing the phys- ical (Skinner box) context than is the second response. The point is that many kinds of stim- uli or events can play the role of context.

Another precipitating event that causes the return of extinguished or punished behavior is free presentations of the reinforcer. In the rat laboratory, when a few reinforcers are presented freely after an operant response has been extin- guished, the behavior will return (e.g., Ostlund & Balleine, 2007; Reid, 1958; Rescorla & Skucy, 1969). We recently found that even re- inforcers presented contingent on a second re- sponse (instead of freely) were also effective at reinstating an extinguished target response (Win- terbauer & Bouton, 2011). One explanation is that reinforcer presentations are merely part of the background (or context) that sets the occasion for more responding (e.g., Ostlund & Balleine, 2007). Another is that the reinforcer presentations may condition the background context (e.g., the Skin- ner box), which might itself reinvigorate the ex- tinguished behavior (Baker, Steinwald, & Bouton, 1991). Presentations of the reinforcer after extinc- tion are also known to reinstate operant behaviors that are reinforced by drugs (e.g., de Wit & Stew- art, 1981, 1983). And reinstatement effects have been reported with operants that have been sup- pressed by punishment (Panlilio, Thorndike, & Schindler, 2003). Once again, low- or zero-rate operants that have achieved that level through extinction or punishment can return through the occurrence of a precipitating event.

Recent research on extinction has focused on another “relapse” effect that may be especially relevant to behavior analysis. In differential re- inforcement of alternative behavior (DRA), a new operant (R2) is reinforced at the same time a previously reinforced target operant (R1) is extinguished. The DRA procedure is widely used by behavior analysts in treating problem behavior in children with developmental dis- abilities or autism spectrum disorder, and it seems reasonable to think that in nature extin- guished responses are also often replaced by reinforced alternative responses. However, if the alternative behavior is now put on extinc-

tion, the first behavior can return (e.g., Leiten- berg, Rawson, & Bath, 1970). This resurgence effect is now being studied in several laborato- ries. Although several explanations have been suggested (Leitenberg et al., 1970; Shahan & Craig, 2017; Shahan & Sweeney, 2011), most evidence favors a view that is based on the principles described above: Removal of rein- forcement for R2 during testing changes the reinforcer context, so the inhibited R1 renews (e.g., Winterbauer & Bouton, 2010). A number of findings support this hypothesis. Perhaps the most straightforward is that when R1 is rein- forced with one reinforcer and R2 is reinforced with a different one while R1 is extinguished, free (noncontingent) presentations of the second reinforcer during testing eliminates resurgence, whereas presentation of the first reinforcer does not (Bouton & Trask, 2016). Thus, the second reinforcer is demonstrably a cue or context that in this case controls R1’s extinction perfor- mance (see also Trask & Bouton, 2016). When the reinforcer is removed, the response returns; when it is maintained, the response remains inhibited and suppressed. The key to preventing resurgence is thus to encourage generalization between treatment and testing. For a more com- plete discussion of resurgence and the evidence supporting the context explanation, see the re- view by Trask, Schepers, and Bouton (2015).

It is worth mentioning that much of the early research on renewal, reinstatement, and sponta- neous recovery was actually done in Pavlovian (respondent) conditioning (e.g., Bouton, 1988, 2004, 2017). (Resurgence has been studied ex- clusively in the operant domain, although we expect that an analogous effect would occur in respondent conditioning; e.g., see Lindblom & Jenkins, 1981.) The characteristics of extinguished respondents and operants are thus highly similar. We know that a conditional stimulus (CS) that elicits a very weak response or even no response at all can elicit the response again with a change of context (renewal), presentation of the Pavlovian unconditional stimulus or reinforcer (reinstate- ment), or the passage of time (spontaneous recov- ery; see Bouton, 2017, for one recent review). Renewal, reinstatement, and spontaneous recov- ery have also been shown after countercondition- ing, in which, for example, CS–shock pairings are followed by CS–food pairings or CS–food is fol- lowed by CS–shock (e.g., Bouton & Peck, 1992; Brooks, Hale, Nelson, & Bouton, 1995; Holmes,

207PREDICTION AND CONTROL OF OPERANT BEHAVIOR

T hi

s do

cu m

en t

is co

py ri

gh te

d by

th e

A m

er ic

an Ps

yc ho

lo gi

ca l

A ss

oc ia

tio n

or on

e of

its al

lie d

pu bl

is he

rs .

T hi

s ar

tic le

is in

te nd

ed so

le ly

fo r

th e

pe rs

on al

us e

of th

e in

di vi

du al

us er

an d

is no

t to

be di

ss em

in at

ed br

oa dl

y.

Leung, & Westbrook, 2016; Peck & Bouton, 1990). The effects of the precipitating events in turn depend on the CS’s conditioning history.

In a clear illustration of this, Bouton (1984, Experiment 5) paired a CS with a strong foot- shock in an initial phase. He then extinguished the response (“fear”) that the CS elicited by presenting the CS several times without shock. Importantly, extinction was stopped before the response completely disappeared. A second set of rats received only a small number of pairings of the CS with a weak shock—without extinc- tion. They received just enough conditioning trials so that the CS elicited a low level of responding that was indistinguishable from that in the conditioned-then-partially-extinguished group. Then both groups received footshock presentations (a reinstatement treatment). When the CS was then tested, there was a robust increase in responding to the conditioned-then- extinguished CS, but no increase to the condi- tioned-only CS. Thus, the effects of the reinstat- ing shocks (and the conditioning of the context they demonstrably produced) depended cru- cially on the conditioning history of the CS. One could not have predicted the effect of the reinstatement shocks from prior responding to the CS alone. In respondent conditioning, like operant conditioning, behavioral silence can be misleading. What you see is not all there is.

Conclusions

To summarize the argument, it may be easiest to predict what events will control a particular behavior by first considering the behavior’s his- tory. Silent operants (those that have been in- hibited, e.g., by extinction or punishment) may “inexplicably” seem to increase in strength after context change, the passage of time, reinforcer presentation, or the extinction of an alternative behavior that replaced it. Nonextinguished or nonpunished behaviors would not change in response to these events in the same way. To predict or control future behavior, as well as predict the effects of context change, the pas- sage of time, reinforcer presentation, or the ex- tinction of an alternative, one would need to know if the behavior has been inhibited or sup- pressed below some earlier rate. Goal-directed actions, on the other hand, can be strengthened or weakened by degradation of the action– outcome contingency or by offline changes in the

value of the reinforcing outcome—as shown in the reinforcer devaluation effect. In contrast, hab- its are operants that are relatively immune to the immediate effects of reinforcer devaluation and in reducing the action–outcome contingency. Ac- tions and habits (like silent and nonsilent operants) can be distinguished by the kinds of events that control them. But knowing ahead of time whether a behavior is an action or habit will again depend on understanding the behavior’s history: Actions have usually had little practice, have developed in choice situations, or have been reinforced on ratio schedules. Habits have contrastingly had larger amounts of practice, and have often been rein- forced on interval schedules.

We already know that a behavior’s history is important. For example, it is widely understood that a behavior that has been intermittently re- inforced can be more resistant to extinction than one that has been consistently reinforced (e.g., Capaldi, 1967). However, the concepts of ac- tions, habits, and inhibited or silent operants may have unique implications for the control and treat- ment of problem behavior. For one, if a problem behavior is a habit (rather than an action), current thinking suggests that it might be difficult to change. And of course, extinction (and punish- ment) treatments, including DRA or DRO, can be vulnerable to lapse and relapse effects (e.g., Bou- ton, 2014; Podlesnik et al., 2017). On the more positive side, extinction of an undesirable behav- ior might in principle allow a more desirable one (that was previously learned and replaced by an undesirable one) to resurge. And reinforcement of a positive new operant to the point of becoming a habit might make it especially resistant to change. These ideas and applications beg for more re- search to give them more richness and detail.

More generally, it is proposed that the accu- racy and precision of prediction and control, the behavior analyst’s two main desiderata, will be enhanced by considering the basic research and principles just reviewed. Moreover, prediction and control are also crucially enhanced by un- derstanding a behavior’s history. What you see is not necessarily all there is.

References

Adams, C. D. (1982). Variations in the sensitivity of instrumental responding to reinforcer devaluation. Quarterly Journal of Experimental Psychology, 33B, 109–122.

208 BOUTON AND BALLEINE

T hi

s do

cu m

en t

is co

py ri

gh te

d by

th e

A m

er ic

an Ps

yc ho

lo gi

ca l

A ss

oc ia

tio n

or on

e of

its al

lie d

pu bl

is he

rs .

T hi

s ar

tic le

is in

te nd

ed so

le ly

fo r

th e

pe rs

on al

us e

of th

e in

di vi

du al

us er

an d

is no

t to

be di

ss em

in at

ed br

oa dl

y.

Adams, C. D., & Dickinson, A. (1981). Instrumental responding following reinforcer devaluation. Quarterly Journal of Experiment Psychology B: Comparative and Physiological Psychology, 33, 109–121. http://dx.doi.org/10.1080/146407481 08400816

Alvares, G. A., Balleine, B. W., & Guastella, A. J. (2014). Impairments in goal-directed actions pre- dict treatment response to cognitive-behavioral therapy in social anxiety disorder. PLoS ONE, 9, e94778. http://dx.doi.org/10.1371/journal.pone .0094778

Alvares, G. A., Balleine, B. W., Whittle, L., & Guas- tella, A. J. (2016). Reduced goal-directed action con- trol in autism spectrum disorder. Autism Research, 9, 1285–1293. http://dx.doi.org/10.1002/aur.1613

Baker, A. G., Steinwald, H., & Bouton, M. E. (1991). Contextual conditioning and reinstatement of ex- tinguished instrumental responding. Quarterly Journal of Experimental Psychology B: Compar- ative and Physiological Psychology, 43, 199–218.

Balleine, B. W. (1992). Instrumental performance following a shift in primary motivation depends on incentive learning. Journal of Experimental Psy- chology: Animal Behavior Processes, 18, 236– 250.

Balleine, B. W. (2001). Incentive processes in instru- mental conditioning. In R. Mowrer & S. Klein (Eds.), Handbook of contemporary learning theo- ries (pp. 307–366). Hillsdale, NJ: LEA.

Balleine, B. W., & Dickinson, A. (1998). Goal- directed instrumental action: Contingency and in- centive learning and their cortical substrates. Neu- ropharmacology, 37, 407–419. http://dx.doi.org/ 10.1016/S0028-3908(98)00033-1

Balleine, B. W., & O’Doherty, J. P. (2010). Human and rodent homologies in action control: Cortico- striatal determinants of goal-directed and habitual action. Neuropsychopharmacology, 35, 48– 69. http://dx.doi.org/10.1038/npp.2009.131

Bouton, M. E. (1984). Differential control by context in the inflation and reinstatement paradigms. Journal of Experimental Psychology: Animal Behavior Pro- cesses, 10, 56–74. http://dx.doi.org/10.1037/0097- 7403.10.1.56

Bouton, M. E. (1988). Context and ambiguity in the extinction of emotional learning: Implications for exposure therapy. Behaviour Research and Ther- apy, 26, 137–149. http://dx.doi.org/10.1016/0005- 7967(88)90113-1

Bouton, M. E. (2002). Context, ambiguity, and un- learning: Sources of relapse after behavioral ex- tinction. Biological Psychiatry, 52, 976–986. http://dx.doi.org/10.1016/S0006-3223(02)01546-9

Bouton, M. E. (2004). Context and behavioral pro- cesses in extinction. Learning & Memory, 11, 485–494. http://dx.doi.org/10.1101/lm.78804

Bouton, M. E. (2014). Why behavior change is dif- ficult to sustain. Preventive Medicine, 68, 29–36. http://dx.doi.org/10.1016/j.ypmed.2014.06.010

Bouton, M. E. (2017). Extinction: Behavioral mecha- nisms and their implications. In J. H. Byrne (Series Ed.) & R. Menzel (Vol. Ed.), Learning and mem- ory: A comprehensive reference: Learning theory and behavior (Vol. 1, 2nd ed., pp. 61–83). Oxford, UK: Academic Press.

Bouton, M. E., Kenney, F. A., & Rosengard, C. (1990). State-dependent fear extinction with two benzodiaz- epine tranquilizers. Behavioral Neuroscience, 104, 44–55. http://dx.doi.org/10.1037/0735-7044.104 .1.44

Bouton, M. E., & Peck, C. A. (1992). Spontaneous recovery in cross-motivational transfer (counter- conditioning). Animal Learning & Behavior, 20, 313–321. http://dx.doi.org/10.3758/BF03197954

Bouton, M. E., & Schepers, S. T. (2015). Renewal after the punishment of free operant behavior. Journal of Experimental Psychology: Animal Learning and Cognition, 41, 81–90. http://dx.doi .org/10.1037/xan0000051

Bouton, M. E., Todd, T. P., Vurbic, D., & Winter- bauer, N. E. (2011). Renewal after the extinction of free operant behavior. Learning & Behavior, 39, 57– 67. http://dx.doi.org/10.3758/s13420-011- 0018-6

Bouton, M. E., & Trask, S. (2016). Role of the discriminative properties of the reinforcer in resur- gence. Learning & Behavior, 44, 137–150. http:// dx.doi.org/10.3758/s13420-015-0197-7

Bouton, M. E., Trask, S., & Carranza-Jasso, R. (2016). Learning to inhibit the response during instrumental (operant) extinction. Journal of Ex- perimental Psychology: Animal Learning and Cognition, 42, 246–258. http://dx.doi.org/10.1037/ xan0000102

Bradfield, L. A., & Balleine, B. W. (2013). Hierar- chical and binary associations compete for behav- ioral control during instrumental biconditional dis- crimination. Journal of Experimental Psychology: Animal Behavior Processes, 39, 2–13. http://dx.doi .org/10.1037/a0030941

Brooks, D. C., Hale, B., Nelson, J. B., & Bouton, M. E. (1995). Reinstatement after countercondi- tioning. Animal Learning & Behavior, 23, 383– 390. http://dx.doi.org/10.3758/BF03198938

Capaldi, E. J. (1967). A sequential hypothesis of instrumental learning. In K. W. Spence & J. T. Spence (Eds.), Psychology of learning and moti- vation (Vol. 1, pp. 67–156). New York, NY: Ac- ademic Press.

Colwill, R. M., & Rescorla, R. A. (1985). Postcon- ditioning devaluation of a reinforcer affects instru- mental responding. Journal of Experimental Psy- chology: Animal Behavior Processes, 11, 120–

209PREDICTION AND CONTROL OF OPERANT BEHAVIOR

T hi

s do

cu m

en t

is co

py ri

gh te

d by

th e

A m

er ic

an Ps

yc ho

lo gi

ca l

A ss

oc ia

tio n

or on

e of

its al

lie d

pu bl

is he

rs .

T hi

s ar

tic le

is in

te nd

ed so

le ly

fo r

th e

pe rs

on al

us e

of th

e in

di vi

du al

us er

an d

is no

t to

be di

ss em

in at

ed br

oa dl

y.

132. http://dx.doi.org/10.1037/0097-7403.11.1 .120

Colwill, R. C., & Rescorla, R. A. (1986). Associative structures in instrumental learning. In G. H. Bower (Ed.), The psychology of learning and motivation (Vol. 20, pp. 55–104). New York, NY: Academic Press.

Colwill, R. M., & Rescorla, R. A. (1988). The role of response-reinforcer associations increases throughout extended instrumental training. Animal Learning & Behavior, 16, 105–111. http://dx.doi .org/10.3758/BF03209051

Crespi, L. P. (1942). Quantitative variation in incen- tive and performance in the white rat. American Journal of Psychology, 55, 467–517. http://dx.doi .org/10.2307/1417120

Cunningham, C. L. (1979). Alcohol as a cue for extinction: State dependency produced by condi- tioned inhibition. Animal Learning & Behavior, 7, 45–52. http://dx.doi.org/10.3758/BF03209656

de Wit, H., & Stewart, J. (1981). Reinstatement of cocaine-reinforced responding in the rat. Psychop- harmacology, 75, 134–143. http://dx.doi.org/10 .1007/BF00432175

de Wit, H., & Stewart, J. (1983). Drug reinstatement of heroin-reinforced responding in the rat. Psychophar- macology, 79, 29–31. http://dx.doi.org/10.1007/ BF00433012

Dezfouli, A., & Balleine, B. W. (2012). Habits, action sequences and reinforcement learning. European Journal of Neuroscience, 35, 1036–1051. http://dx .doi.org/10.1111/j.1460-9568.2012.08050.x

Dickinson, A., & Balleine, B. W. (1994). Motiva- tional control of goal-directed action. Animal Learning & Behavior, 22, 1–18. http://dx.doi.org/ 10.3758/BF03199951

Dickinson, A., Balleine, B. W., Watt, A., Gonzalez, F., & Boakes, R. A. (1995). Motivational control after extended instrumental training. Animal Learning & Behavior, 23, 197–206. http://dx.doi .org/10.3758/BF03199935

Dickinson, A., & Dawson, G. R. (1987). The role of the instrumental contingency in the motivational control of performance. Quarterly Journal of Ex- perimental Psychology, 398, 77–93.

Dickinson, A., & Mulatero, C. W. (1989). Reinforcer specificity of the suppression of instrumental per- formance on a non-contingent schedule. Behav- ioural Processes, 19, 167–180. http://dx.doi.org/ 10.1016/0376-6357(89)90039-9

Dickinson, A., Squire, S., Varga, Z., & Smith, J. W. (1998). Omission learning after instrumental pre- training. Quarterly Journal of Experimental Psy- chology B: Comparative and Physiological Psy- chology, 51, 271–286.

Estes, W. K. (1944). An experimental study of pun- ishment. Psychological Monographs, 57, i-40. http://dx.doi.org/10.1037/h0093550

Everitt, B. J., & Robbins, T. W. (2005). Neural systems of reinforcement for drug addiction: From actions to habits to compulsion. Nature Neuroscience, 8, 1481– 1489. http://dx.doi.org/10.1038/nn1579

Flaherty, C. F. (1996). Incentive relativity. Cam- bridge, UK: Cambridge University Press.

Furlong, T. M., Supit, A. S. A., Corbit, L. H., Killcross, S., & Balleine, B. W. (2017). Pulling habits out of rats: Adenosine 2A receptor antagonism in dorsome- dial striatum rescues meth-amphetamine-induced deficits in goal-directed action. Addiction Biology, 22, 172–183. http://dx.doi.org/10.1111/adb.12316

Griffiths, K. R., Morris, R. W., & Balleine, B. W. (2014). Translational studies of goal-directed ac- tion as a framework for classifying deficits across psychiatric disorders. Frontiers in Systems Neuro- science, 8, 101.

Hanley, G. P., Iwata, B. A., & McCord, B. E. (2003). Functional analysis of problem behavior: A re- view. Journal of Applied Behavior Analysis, 36, 147–185. http://dx.doi.org/10.1901/jaba.2003.36- 147

Hogarth, L., Balleine, B. W., Corbit, L. H., & Kill- cross, S. (2013). Associative learning mechanisms underpinning the transition from recreational drug use to addiction. Annals of the New York Academy of Sciences, 1282, 12–24. http://dx.doi.org/10 .1111/j.1749-6632.2012.06768.x

Holmes, N. M., Leung, H. T., & Westbrook, R. F. (2016). Counterconditioned fear responses exhibit greater renewal than extinguished fear responses. Learning & Memory, 23, 141–150. http://dx.doi .org/10.1101/lm.040659.115

Hull, C. L. (1943). Principles of behavior: An intro- duction to behavior theory. New York, NY: Ap- pelton-Century-Crofts.

Kahneman, D. (2011). Thinking fast and slow. New York, NY: Farrar, Straus & Groux.

Kelley, M. E., Liddon, C. J., Ribeiro, A., Greif, A. E., & Podlesnik, C. A. (2015). Basic and translational evaluation of renewal of operant responding. Jour- nal of Applied Behavior Analysis, 48, 390–401. http://dx.doi.org/10.1002/jaba.209

Klossek, U. M., Russell, J., & Dickinson, A. (2008). The control of instrumental action following out- come devaluation in young children aged between 1 and 4 years. Journal of Experimental Psychology: General, 137, 39–51. http://dx.doi.org/10.1037/ 0096-3445.137.1.39

Kosaki, Y., & Dickinson, A. (2010). Choice and contingency in the development of behavioral au- tonomy during instrumental conditioning. Journal of Experimental Psychology: Animal Behavior Processes, 36, 334–342. http://dx.doi.org/10.1037/ a0016887

Lattal, K. M. (2007). Effects of ethanol on encoding, consolidation, and expression of extinction following contextual fear conditioning. Behavioral Neurosci-

210 BOUTON AND BALLEINE

T hi

s do

cu m

en t

is co

py ri

gh te

d by

th e

A m

er ic

an Ps

yc ho

lo gi

ca l

A ss

oc ia

tio n

or on

e of

its al

lie d

pu bl

is he

rs .

T hi

s ar

tic le

is in

te nd

ed so

le ly

fo r

th e

pe rs

on al

us e

of th

e in

di vi

du al

us er

an d

is no

t to

be di

ss em

in at

ed br

oa dl

y.

ence, 121, 1280–1292. http://dx.doi.org/10.1037/ 0735-7044.121.6.1280

Leitenberg, H., Rawson, R. A., & Bath, K. (1970). Reinforcement of competing behavior during ex- tinction. Science, 169, 301–303. http://dx.doi.org/ 10.1126/science.169.3942.301

Lindblom, L. L., & Jenkins, H. M. (1981). Responses eliminated by noncontingent or negatively contin- gent reinforcement recover in extinction. Journal of Experimental Psychology: Animal Behavior Processes, 7, 175–190. http://dx.doi.org/10.1037/ 0097-7403.7.2.175

Morris, R. W., Quail, S., Griffiths, K. R., Green, M. J., & Balleine, B. W. (2015). Corticostriatal control of goal-directed action is impaired in schizophrenia. Biological Psychiatry, 77, 187– 195. http://dx.doi.org/10.1016/j.biopsych.2014.06 .005

Nakajima, S., Urushihara, K., & Masaki, T. (2002). Renewal of operant performance formerly elimi- nated by omission or non-contingency training upon return to the acquisition context. Learning and Motivation, 33, 510–525. http://dx.doi.org/10 .1016/S0023-9690(02)00009-7

Ostlund, S. B., & Balleine, B. W. (2007). Selective reinstatement of instrumental performance de- pends on the discriminative stimulus properties of the mediating outcome. Learning & Behavior, 35, 43–52. http://dx.doi.org/10.3758/BF03196073

Overton, D. A. (1985). Contextual stimulus effects of drugs and internal states. In P. D. Balsam & A. Tomie (Eds.), Context and learning (pp. 357–384). Hillsdale, NJ: Erlbaum.

Panlilio, L. V., Thorndike, E. B., & Schindler, C. W. (2003). Reinstatement of punishment-suppressed opioid self-administration in rats: An alternative model of relapse to drug abuse. Psychopharmacol- ogy, 168, 229–235. http://dx.doi.org/10.1007/s00 213-002-1193-0

Peck, C. A., & Bouton, M. E. (1990). Context and performance in aversive-to-appetitive and appeti- tive-to-aversive transfer. Learning and Motivation, 21, 1–31. http://dx.doi.org/10.1016/0023-9690(90) 90002-6

Podlesnik, C. A., Kelley, M. E., Jimenez-Gomez, C., & Bouton, M. E. (2017). Renewed behavior pro- duced by context change and its implications for treatment maintenance: A review. Journal of Ap- plied Behavior Analysis, 50, 675–697. http://dx .doi.org/10.1002/jaba.400

Redgrave, P., Rodriguez, M., Smith, Y., Rodriguez- Oroz, M. C., Lehericy, S., Bergman, H., . . . Obeso, J. A. (2010). Goal-directed and habitual control in the basal ganglia: Implications for Parkinson’s dis- ease. Nature Reviews Neuroscience, 11, 760–772. http://dx.doi.org/10.1038/nrn2915

Reid, R. L. (1958). The role of the reinforcer as a stimulus. British Journal of Psychology, 49, 202–

209. http://dx.doi.org/10.1111/j.2044-8295.1958 .tb00658.x

Rescorla, R. A. (1991). Associative relations in In- strumental learning: The Eighteenth Bartlet Me- morial Lecture. Quarterly Journal of Experimental Psychology B: Comparative and Physiological Psychology, 43, 1–23.

Rescorla, R. A. (2004). Spontaneous recovery. Learning & Memory, 11, 501–509. http://dx.doi .org/10.1101/lm.77504

Rescorla, R. A., & Skucy, J. C. (1969). Effect of re- sponse-independent reinforcers during extinction. Journal of Comparative and Physiological Psychol- ogy, 67, 381–389. http://dx.doi.org/10.1037/h002 6793

Schepers, S. T., & Bouton, M. E. (2017). Hunger as a context: Food seeking that is inhibited during hunger can renew in the context of satiety. Psy- chological Science, 28, 1640–1648. http://dx.doi .org/10.1177/0956797617719084

Shahan, T. A., & Craig, A. R. (2017). Resurgence as choice. Behavioural Processes, 141, 100–127. http://dx.doi.org/10.1016/j.beproc.2016.10.006

Shahan, T. A., & Sweeney, M. M. (2011). A model of resurgence based on behavioral momentum the- ory. Journal of the Experimental Analysis of Be- havior, 95, 91–108. http://dx.doi.org/10.1901/jeab .2011.95-91

Skinner, B. F. (1938). The behavior of organisms. New York, NY: Appleton.

Skinner, B. F. (1969). Contingencies of reinforce- ment. New York, NY: Appleton.

Spence, K. W. (1956). Behavior theory and condi- tioning. New Haven, CT: Yale University Press. http://dx.doi.org/10.1037/10029-000

Thrailkill, E. A., & Bouton, M. E. (2015). Contextual control of instrumental actions and habits. Journal of Experimental Psychology: Animal Learning and Cognition, 41, 69–80. http://dx.doi.org/10.1037/ xan0000045

Thrailkill, E. A., & Bouton, M. E. (2016). Extinction and the associative structure of heterogeneous in- strumental chains. Neurobiology of Learning and Memory, 133, 61–68. http://dx.doi.org/10.1016/j .nlm.2016.06.005

Thrailkill, E. A., Trott, J. M., Zerr, C. L., & Bouton, M. E. (2016). Contextual control of chained instru- mental behaviors. Journal of Experimental Psy- chology: Animal Learning and Cognition, 42, 401–414. http://dx.doi.org/10.1037/xan0000112

Tinklepaugh, O. L. (1928). An experimental study of representative factors in monkeys. Journal of Comparative Psychology, 8, 197–236. http://dx .doi.org/10.1037/h0075798

Todd, T. P. (2013). Mechanisms of renewal after the extinction of instrumental behavior. Journal of Ex- perimental Psychology: Animal Behavior Processes, 39, 193–207. http://dx.doi.org/10.1037/a0032236

211PREDICTION AND CONTROL OF OPERANT BEHAVIOR

T hi

s do

cu m

en t

is co

py ri

gh te

d by

th e

A m

er ic

an Ps

yc ho

lo gi

ca l

A ss

oc ia

tio n

or on

e of

its al

lie d

pu bl

is he

rs .

T hi

s ar

tic le

is in

te nd

ed so

le ly

fo r

th e

pe rs

on al

us e

of th

e in

di vi

du al

us er

an d

is no

t to

be di

ss em

in at

ed br

oa dl

y.

Todd, T. P., Vurbic, D., & Bouton, M. E. (2014). Mechanisms of renewal after the extinction of dis- criminated operant behavior. Journal of Experimen- tal Psychology: Animal Learning and Cognition, 40, 355–368. http://dx.doi.org/10.1037/xan0000021

Tolman, E. C. (1932). Purposive behavior in animals and men. New York, NY: Century.

Trask, S., & Bouton, M. E. (2014). Contextual con- trol of operant behavior: Evidence for hierarchical associations in instrumental learning. Learning & Behavior, 42, 281–288. http://dx.doi.org/10.3758/ s13420-014-0145-y

Trask, S., & Bouton, M. E. (2016). Discriminative properties of the reinforcer can be used to attenuate the renewal of extinguished operant behavior. Learning & Behavior, 44, 151–161. http://dx.doi .org/10.3758/s13420-015-0195-9

Trask, S., Schepers, S. T., & Bouton, M. E. (2015). Context change explains resurgence after the ex- tinction of operant behavior. Mexican Journal of Behavior Analysis, 41, 187–210.

Tricomi, E., Balleine, B. W., & O’Doherty, J. P. (2009). A specific role for posterior dorsolateral striatum in human habit learning. European Jour- nal of Neuroscience, 29, 2225–2232. http://dx.doi .org/10.1111/j.1460-9568.2009.06796.x

Winterbauer, N. E., & Bouton, M. E. (2010). Mech- anisms of resurgence of an extinguished instru- mental behavior. Journal of Experimental Psy- chology: Animal Behavior Processes, 36, 343– 353. http://dx.doi.org/10.1037/a0017365

Winterbauer, N. E., & Bouton, M. E. (2011). Mech- anisms of resurgence: II. Response-contingent re- inforcers can reinstate a second extinguished be- havior. Learning and Motivation, 42, 154–164. http://dx.doi.org/10.1016/j.lmot.2011.01.002

Received August 10, 2017 Accepted October 27, 2017 �

212 BOUTON AND BALLEINE

T hi

s do

cu m

en t

is co

py ri

gh te

d by

th e

A m

er ic

an Ps

yc ho

lo gi

ca l

A ss

oc ia

tio n

or on

e of

its al

lie d

pu bl

is he

rs .

T hi

s ar

tic le

is in

te nd

ed so

le ly

fo r

th e

pe rs

on al

us e

of th

e in

di vi

du al

us er

an d

is no

t to

be di

ss em

in at

ed br

oa dl

y.