Reflection Paper

TECHNICAL AND TUTORIALS

Some Tools for Carrying Out a Proposed Process for Supervising Experience Hours for Aspiring Board Certified Behavior Analysts®

Katie Lynn Garza1 & Heather M. McGee1 & Yannick A. Schenk1 &

Rebecca R. Wiskirchen1

Published online: 14 April 2017 # Association for Behavior Analysis International 2017

Abstract While task clarification, goal setting, feedback, and behavioral skills training (BST) are well-supported methods for performance improvement, there is no standardized ap- proach to supervising aspiring Board Certified Behavior Analysts® (BCBAs®) that specifies how such practices should be used within a comprehensive supervision system, namely for supervising those who are still accruing experience hours for the purpose of becoming board certified. This article outlines a systematic approach to BCBA supervision and pro- vides a set of tools that supervisors can use to ensure that they are engaging in empirically based supervision practices.

Keywords BCBA supervision . Supervision process .

Supervisionmaterials

The Behavior Analyst Certification Board® (BACB®) Supervisor Training Curriculum Outline specifies that BCBAs trained in supervision techniques should be able to describe the purpose and important features of supervision, use behavioral skills training, including the use of effective performance feedback to teach targeted skills, evaluate the effectiveness of their supervision, and proactively engage in professional development activities to improve supervisory performance (BACB, 2012b). Each of these repertoires is es- sential for providing adequate supervision. We believe that of

equal importance is a supervisor’s ability to conduct supervi- sion within a process that accounts for the contingencies act- ing upon his environment and the environment of his supervisees. Insight from the work that has been done in or- ganizational behavior management (OBM) can help us design effective supervision systems.

Behavior analysts should be accustomed to using a behavior-analytic technology to produce behavior change pro- grams for clients. However, manymay not be as accustomed to using the same technology to train and supervise individuals seeking certification, especially those who have limited practi- cal experience. Organizational behavior management is a sub- field of applied behavior analysis that focuses specifically on organizational problems (Bucklin, Alvero, Dickinson, Austin, & Jackson, 2000). Because the same empirical approach that is followed in clinically focused behavior analysis is used in or- ganizational behavior management, much of the research find- ings from the OBM literature can be used to aid in the devel- opment of applied behavior analysis (ABA) practices, includ- ing the supervision of others.

For example, it is clear that ensuring that consequences are contingent on the target performance improves that particular target performance. Komaki (1986) set out to, Bspecifically identify and empirically determine what constitutes effective supervisory behavior^ (p. 270). The author found that effec- tive managers spent significantly more time monitoring per- formance and used work sampling as a specific form of per- formance monitoring more often than marginally effective supervisors. Komaki also found that there was no difference between the groups regarding the number of positive, nega- tive, or neutral consequences between effective and marginal- ly effective managers. In other words, Beffective managers were no more likely than the marginal managers to provide positive consequences, and they were no less likely to provide negative consequences^ (pp. 275–6). The author concluded

Electronic supplementary material The online version of this article (doi:10.1007/s40617-017-0186-8) contains supplementary material, which is available to authorized users.

* Katie Lynn Garza [email protected]

1 Department of Psychology, Western Michigan University, 1903 W. Michigan Avenue, Kalamazoo, MI 49008-5439, USA

Behav Analysis Practice (2018) 11:62–70 DOI 10.1007/s40617-017-0186-8

that because effective managers spent more time monitoring, their consequences were more likely to be contingent on, rath- er than independent of, employees’ performance.

Task clarification, goal setting, and feedback are common practices that have been used in a variety of settings to im- prove on-the-job performance. Tittelbach, DeAngelis, Sturmey, and Alvero (2007) used a task clarification, goal setting, and feedback intervention to improve punctuality, greeting, and correct front-desk behavior of student advisors in a university counseling center. Loewy and Bailey (2007) used graphic feedback, goal setting, and manager praise to improve customer service (i.e., greeting, eye contact, and smiling) in two locations of a home improvement store. Gil and Carter (2016) used graphic feedback, later combined with goal setting and performance feedback to improve data col- lection adherence of direct care staff in a large residential treatment facility. It is reasonable to expect the application of these practices to the supervision of aspiring BCBAs to result in supervisees’ performance improvement in key behavior- analytic skills.

In 2016, Behavior Analysis in Practice released a special section on supervision. Prior to its publication, few articles had been published on recommended practices for the super- vision of aspiring BCBAs. The special section includes arti- cles on ethical considerations in supervision (Sellers, Alai- Rosales, & MacDonald, 2016), recommendations for conducting group supervision (Valentino, LeBlanc, & Sellers, 2016), recommendations for addressing barriers to supervision (Sellers, LeBlanc, & Valentino, 2016), an exami- nation of the relationship between supervision hours, supervi- sor credentials, years of experience, and supervisor caseload on ABA treatment outcomes (Dixon et al., 2016), and recom- mended practice guidelines and approaches to supervision (Sellers, Valentino, & LeBlanc, 2016; Hartley, Courtney, Rosswurm, & LaMarca, 2016; Turner, Fischer, & Luiselli, 2016).

Interestingly, though the supervision model presented by Turner et al. (2016) closely resembles the process that will be presented in this paper, the two systems were established by two separate groups in two separate loca- tions. The current paper further enhances the literature base on supervision by emphasizing that supervision is a process with a beginning, middle, and end and by provid- ing templates to assist supervisors in carrying out that process.

Knowing the general flow of a supervisory relationship from beginning to end helps supervisors to keep their docu- mentation organized and plan for supervisees’ training and assessment needs. This is especially important given that su- pervision may often be a secondary job for BCBAs, resulting in time constraints and competing job responsibilities (Garza, Peterson, McGee, Jackson, & Malott, manuscript in preparation).

In a survey to assess the needs of individuals supervising experience hours for aspiring BCBAs, 18% of participants indicated that lack of available information or materials was a barrier to using assessment in supervision. Furthermore, 30% of participants indicated that time or competing contin- gencies were a barrier to completing assessments in supervi- sion. When asked to indicate barriers to the use of behavioral skills training in supervision practices, 54% of participants indicated that lack of time was a barrier, and 48% of partici- pants indicated that there were too many skills to teach. The majority of participants also indicated that additional materials would be useful for teaching most items in sections I and II of the BACB Task List (Garza et al., manuscript in preparation). In response to these expressed needs, the purpose of this paper is to present a concise process for supervising aspiring BCBAs and to provide materials to assist supervisors in pro- viding high-quality supervision. Supervision for BCBA expe- rience hours can be conceptualized as a process that occurs in five phases: establishing a supervisory relationship, skills as- sessment, training, ongoing performance monitoring, and ending the supervisory relationship. The steps involved in each of these phases will be discussed below. Those phases that are already discussed at length in the existing literature on supervision will be covered in less detail than those for which the existing literature does not address how to complete the phase in the context of BCBA supervision. In addition, strat- egies for ongoing professional development will be discussed.

Establishing a Supervisory Relationship

Individuals may receive supervision as part of a university practicum, from a supervisor within the organization at which he or she works, or from a BCBA who offers contracted su- pervision services. Regardless of the arrangement of supervi- sion, no experience hours can be accrued before both the su- pervisor and supervisee sign a contract outlining the respon- sibilities of each party. The BACB Experience Standards out- line the specific items that must be included in the contract (BACB, 2016), and the BACB® website offers several sam- ple contracts to use as a template. We recommend using these templates as a guide and modifying the contract to describe the specific assessments and experience opportunities that the supervisee can expect to encounter within the organization in which experience hours will be accrued. We also recommend using universities’ or other organizations’ legal or contract departments as a resource for reviewing contracts for proper legalese whenever such resources are available.

In our professional experience, we have heard supervisors express concern that they could get locked into a supervisory relationship with a supervisee who does not make adequate progress or who engages in unprofessional behavior. In their Experience Standards, the BACB specifies that contracts

Behav Analysis Practice (2018) 11:62–70 63

should, Bdelineate the consequences should the parties not adhere to their responsibilities (including proper termination of the relationship)^ (2016, p. 3). If there are circumstances under which a supervisor would be unwilling to continue su- pervising an individual, those circumstances should be clearly outlined in the supervision contract in order to protect both parties.

During an initial meeting, the supervisor should describe the purpose of supervision as described in the BACB’s Supervisor Training Curriculum Outline and go over the su- pervision contract with the supervisee. We recommend read- ing over the terms of the contract aloud during the meeting, pausing after each section to allow for questions, and having the supervisee sign his or her initials after each section of the agreement. This protects both parties by ensuring that there is a legitimate and understood agreement between the supervisor and supervisee. Once both parties sign the contract, the super- visory relationship has been officially established.

Sellers, Valentino, and LeBlanc (2016) provide excellent rationale and guidelines for effective establishment of the su- pervisor–supervisee relationship. We recommend reviewing their practice guidelines for more detailed information on cre- ating supervision contracts, setting clear expectations for the supervisee, specifically setting expectations regarding feed- back in the supervision process, and creating a Bcommitted and positive^ supervisory relationship (p. 276). Turner et al. (2016) also discuss strategies for establishing a supervisory relationship that results in collaborative and ethical practices. In a manner similar to Sellers et al. (2016), Turner et al. (2016) emphasize the importance of setting clear performance expec- tations and clearly defining the supervisory relationship.

Skills Assessment

After establishing the supervisory relationship, a supervisor must determine which performances and corresponding Task List (BACB, 2012a) items the supervisee currently possesses and which will need to be developed over the course of super- vision. This determination is made through a performance/ skills assessment. In assessment, the supervisor is either ob- serving performance or reviewing products of performance without providing prompts, coaching, or instructional feed- back. These components of training and performancemanage- ment could bias the results of the assessment. Assessment should be done before beginning the training process because it helps the supervisor pinpoint which skills the supervisee can successfully practice independently. This allows the supervi- sor to focus training efforts on skills the supervisee has not yet mastered. In our discussion of assessment, we will first present a method for supervisors to define the scope of their assess- ment, followed by a method for conducting the assessment, and, finally, a method for using the results of the assessment.

Job Model

Job models can take different forms and be called by different names (e.g., role/responsibility matrix) but typically include the accomplishments or performance responsibilities associat- ed with the job. Additional information, such as tasks required to complete the performance, measures of performance, goals or standards for performance, inputs to performance, recipi- ents of outputs of performance, and how feedback on perfor- mance will be provided, are sometimes also included in the job model (Gilbert, 1996; Rummler & Brache, 2012).

Because training (discussed in the next section) can be a time-consuming process, we recommend conducting an anal- ysis of the position supervisees will hold in the organization. This analysis helps supervisors identify which skills the supervisee will actually practice throughout the supervisory relationship. If the organization cannot provide specific train- ing on a particular skill, it is categorized as a low priority for assessment and training. If the skill is practiced regularly with- in the organization, it is a high priority for assessment and, if needed, training. This is not to say that the skills not practiced within the organization are not important to the field of behav- ior analysis or would not be worthwhile for the supervisee to learn. However, it is best to match the training setting to the environment in which the performance will occur naturally (Kazbour, McGee, Mooney, Masica, & Brinkerhoff, 2013; Mager, 1997). If there are no natural practice opportunities within the organization, it is best to leave the training to a site in which those opportunities are available.

Job models can be conceptualized as a sort of task clarifi- cation, which is a widely supported performance improve- ment intervention in OBM (Cunningham & Austin, 2007; Gravina, VanWagner, & Austin, 2008; Durgin, Mahoney, Cox, Weetjens, & Poling, 2014; Slowiak, 2014). Job models can be organized in different ways depending on their specific utility within the organization. In the context of supervision for aspiring BCBAs, it is appropriate to examine which Task List (BACB, 2012a) items are practiced when a supervisee engages in each given performance that is part of her position within the organization.

Appendix 1 is an example of a job model for a behavior consulting organization. On the top of the form, there is a space to indicate the name of the position (e.g., graduate ther- apist), the supervisor (e.g., KS), and the job purpose (e.g., to provide clinical services to adults and children with develop- mental disabilities and develop professional skills for provid- ing effective and ethical treatment). Specifying the purpose of the job helps establish the scope of the position and distin- guish the position from others within the organization. This decreases the likelihood of ambiguity as to whose responsibil- ity it is to engage in a given responsibility. Job responsibilities or performances are listed in the left column of the table, and the corresponding BACB Task List items are listed in the right

64 Behav Analysis Practice (2018) 11:62–70

column. For example, a graduate therapist in the behavior consulting organization might conduct a functional behavior assessment, but they might not be responsible for conducting discrete trial training.

We identified 11 different tasks from the Task List that someone might engage in to complete a functional behavior assessment: G-01 Breview records and available data at the outset of the case,^ G-02 Bconsider biological/medical vari- ables that may be affecting the client,^ G-03 Bconduct a pre- liminary assessment of the client in order to identify the referral problem,^ G-04 Bexplain behavioral concepts using nontech- nical language,^ G-06 Bprovide behavior-analytic services in collaboration with others who support and/or provide services to one’s clients,^ G-07 Bpractice within one’s limits of profes- sional competence in applied behavior analysis, and obtain consultation, supervision, and training, or make referrals as necessary,^ G-08 Bidentify and make environmental changes that reduce the need for behavior analysis services,^ I-01 Bdefine behavior in observable and measurable terms,^ I-02 Bdefine environmental variables in observable and measurable terms,^ I-03 Bdesign and implement individualized behavioral assessment procedures,^ and I-07 Bdesign and conduct prefer- ence assessments to identify putative reinforcers^ (BACB, 2012a, pp. 6, 7). Note that job performances could be more narrowly defined if it is likely that supervisees within the or- ganization will complete components of the larger perfor- mance independently. For example, functional assessment in- terview, descriptive assessment, and functional analysis can be listed as separate performances on the job model if they are likely to be practiced as discrete performances that involve the use of unique sets of skills from the Task List.

Once all of the job responsibilities and corresponding Task List items are identified for the job position, a list of all of the Task List items that can be practiced while the supervisee is in that position can be compiled. The Task List items for which there are no natural practice opportunities can be better assessed and trained in another position, perhaps in another organization. While all of the skills on the Task List are rele- vant to behavior analysis as a field, not all of the skills are relevant to every job position. For example, in organizations that do not provide group instruction to clients, there may be few or no opportunities to use Direct Instruction (item F-03 on the Task List). There also may not be an opportunity to use a full range of experimental designs (Section I.B. on the Task List) within many organizations. Although these skills are valuable, eliminating them from a training program allows the supervisor to focus on teaching the skills that are immedi- ately relevant for the supervisee’s role in the organization. This, in turn, eliminates some competition between training supervisees and the supervisor’s other job responsibilities. Once all of Task List items that can be practiced within a supervisee’s job role have been identified, supervisors can conduct a focused assessment to determine training needs.

Assessment of Skills

Assessing supervisees’ ability to perform the Task List items identified as relevant to the supervisee’s job position can also decrease the time supervisees spend on training. Assessment helps determine which skills must be trained and which can be practiced without training, thereby preventing unnecessary training and premature independent practice. Furthermore, initial assessment allows supervisors to obtain a baseline mea- sure of their supervisees’ performance, which allows them to evaluate their own supervisory effectiveness (Kazdin, 1982). There are different methods to assess supervisees’ ability to perform job responsibilities. We recommend the following sequence of assessment: self-assessment, oral and written quizzes, and validation of self-assessment through review of permanent products and observation.

Turner et al. (2016) discuss the use of initial baseline as- sessment, including the use of direct observation; use of infor- mal assessment such as interviews, conversation, and portfo- lio review; review of course syllabi; discussion of progress with previous supervisors; and review of the Task List. Here, we present multiple steps of baseline assessment and discuss a tool that can be used to organize assessment data for ongoing use in the supervision process.

Self-assessment involves having the supervisee rate her skill level for each job responsibility and/or task list skill. While behavior analysts might be hesitant to use self- assessment methods in their supervision practices, asking a supervisee to rate her ability to perform a skill has some ben- efits. Sellers et al. (2016) recommend having supervisees self- evaluate their ability to perform job responsibilities, noting that such an evaluation might help supervisors target their training efforts appropriately. If a supervisee indicates that she has no or limited experience with a task or that she would need coaching to perform the task, she may actually not have the skills to perform that particular task. In this situation, her self-assessment would indicate a training need. However, it is also possible that her rating does not match her actual skill level (i.e., the supervisee might actually perform the task at a mastery level). Even in this situation, it would be worthwhile to provide some training so that the supervisee can learn to tact her own correct performance and better recognize when she has made an error. Turner et al. (2016) point out that the level of rapport between a supervisor and supervisee may affect a supervisee’s tendency to seek her supervisor’s guidance over the guidance of others. Providing training when a supervisee indicates that her ability to perform a skill is below mastery levels could improve rapport and increase the frequency with which a supervisee asks for help when needed.

On the other hand, if a supervisee indicates that she could perform the task independently or even coach others on the performance, her performance could indicate otherwise (i.e., her performance could bewell belowmastery levels). Parsons,

Behav Analysis Practice (2018) 11:62–70 65

Rollyson, and Reid (2012) distinguish between performance skills and verbal skills in their discussion of the behavioral skills training (BST) model (discussed below). Just as it is recommended that training differ depending on the nature of the skills to be trained, we recommend different methods for verifying supervisee’s self-reports depending on the nature of the skills being assessed. Verification of self-assessment can be done via quizzes and interviews for knowledge-based or verbally based skills (e.g., Section III: Foundational Knowledge of the Task List; skills that involve explaining concepts or treatment programs and skills that involve clinical decision-making) and review of permanent products and di- rect observation for performance-based skills. If the supervisee has a portfolio of work she has done in the past, evaluating the items in the portfolio allows supervisors to assess skill levels without scheduling an observation. In fact, we recommend encouraging supervisors to keep a record of work products for this reason. For items that cannot be assessed via quizzes, interviews, and portfolio review, role play or in vivo, direct observations can be used.

The authors have developed a tool for organizing assess- ment data. Readers may access this tool by emailing the first author. The tool presents all items of the BACB Fourth Edition Task List and dropdown menus with self-assessment options for each Task List item. The supervisee can self-assess their ability to perform each skill by selecting an item from the dropdown menu. Once the supervisee completes the self-as- sessment, the assessment results populate other tabs in the spreadsheet, which correspond to each section of the Task List. For each Task List item, there is a space for the self- assessment standing (which is automatically populated from the self-assessment), a broad training goal (train, generalize, monitor, or lead/train others), validation method (portfolio re- view, natural observation, alternative training method, or N/A), and status (validated, in training, or monitoring). There is also a space for the supervisor to make comments regarding the supervisee’s progress on each Task List item.

Goal Setting

A variety of studies have examined the effects of goal setting on performance. Goal setting has been used successfully in interventions to improve the performance of soccer players (Brobst & Ward, 2002), to increase safe methods of passing sharp instruments in a hospital operating room (Cunningham & Austin, 2007), to increase cashiers’ checking of identifica- tion in a grocery store (Downing & Geller, 2012), and to decrease office workers’ bouts of prolonged sitting (Green, Sigurdsson, & Wilder, 2016). Additionally, Turner et al. (2016) discuss goal setting as one method for teaching and promoting skills within the supervisory relationship. Here, we

introduce steps supervisors can take to set appropriate goals for their supervisees.

The results of skills assessment can be used to set long-term and short-term training goals for the supervisee. Long-term goals can be reviewed each quarter or semester, and short- term goals can be reviewed each supervisory period. Other time frames can be used as well, depending on the nature of the supervisory relationship. Goal setting interventions often involve the use of baseline performance data to set interven- tion goals that are achievable and that, if met, would result in meaningful performance improvement (e.g., Downing & Geller, 2012; Green et al., 2016). Additionally, there is some evidence that setting goals based on an individual’s ability results in more pronounced performance improvements over goals that are assigned to an entire group, especially for low performers (Jeffrey, Schulz, & Webb, 2012).

To set appropriate long-term goals, supervisors should con- sider the Task List items that are included in the job model, the supervisee’s overall progress on mastering those Task List items, and the anticipated needs of the organization during the review period. For example, if the supervisee has observed her supervisor collecting interobserver agreement (IOA) data, but has not yet had the opportunity to practice independently, long-term goals for the quarter might include collecting integ- rity data independently with 80% or better (IOA) with her supervisor. This goal could take one or several sessions to achieve. If the supervisee has never observed the process of writing a behavior support plan, one long-term goal for the quarter might be to have the supervisee take part in the treat- ment planning process and read the resulting behavior support plan.

Once goals are determined for the quarter or semester, short-term goals can be established with long-term goals in mind. To set appropriate short-term goals, supervisors should consider the immediate needs of the organization as well as their supervisee’s ability to meet those needs. For example, if the organization is serving a client who needs a functional behavior assessment, the supervisee’s demonstrated abilities related to functional behavior assessment will determine how much independence the supervisee should be allowed when performing the task. If the supervisee indicated on her self- assessment that she has no or limited experience in FBA, training goals for the supervisory period might include the supervisee’s observation of the planning process. If the supervisee has successfully completed a functional behavior assessment, training goals for the supervisory period might include the supervisee writing the protocol without guidance from the supervisor. The supervisor could then evaluate the protocol and provide positive and corrective feedback. A form for setting goals and delivering performance feedback is avail- able in Appendix 5. See the section on BOngoing Performance Monitoring^ below for a more detailed description of the Feedback and Goal Setting Form.

66 Behav Analysis Practice (2018) 11:62–70

Training

The Supervisor Training Curriculum Outline from the BACB lists BBehavioral skills training of supervisee^ as an important feature of supervision (2012, p. 2). In addition, the BACB Experience Standards lists behavioral skills training as a com- ponent of Beffective behavior-analytic supervision^ (2016, p. 2). Therefore, our discussion of training is based on the be- havioral skills training (BST) model.

Behavioral skills training is an approach to training that includes five major components: rationale and instruction, modeling, opportunity to practice, feedback, and repetition of practice and feedback until the learner achieves mastery (Parsons et al., 2012). Some models also include assessment of generalization to other clients, settings, or behaviors (BACB, 2012b, p. 3).

To measure the correctness of a performance, the correct per- formance must be operationally defined (Cooper, Heron, & Heward, 2007). Before implementing BST, one must define the target performance and create the necessary supportingmaterials. We recommend beginning with a task analysis. A task analysis breaks a performance down into its component steps. Behavior analysts often use task analyses to break down target client per- formances into smaller steps and then use chaining or shaping to improve client performance of the target. In training, task analy- ses are used in the same way, but the focus is on supervisee performance rather than client performance. See Appendix 2 for an example of a task analysis template. Our template includes a place to indicate the performance to be taught, the triggering event (i.e., the stimulus that indicates it is time to begin the performance), and the ending event (i.e., permanent products or changes in the environment that define the termination of the performance). Identifying the triggering event and the ending event helps to define the scope of the performance, which later helps to define the scope of the corresponding training. The template also has a place to indicate the tasks involved in the performance. For each task, there is a table to note the materials that are needed to perform the task and the designated location for those materials. Additionally, the template has a place to indicate any steps, sub-steps, or decisions involved in the task. Some performances may have only one task, while others might have multiple tasks, steps, sub-steps, and decisions.

Task analyses make teaching job performances easier be- cause the target skills are broken into smaller, more manage- able, sequential steps (Mager, 1997). Once the performance is broken into observable and measureable sub-steps, the task analysis can be used to create a data sheet. The data sheet can be used to guide an objective observation of the target skill. Data sheets built from a task analysis are more likely to capture the behaviors critical to successful performance. As supervisors build a portfolio of task analyses and data sheets related to the BACB Task List, they can be used to guide initial skills assessments as well.

After completing a task analysis, it is helpful to create a training storyboard to facilitate the training process (Bunch, 1991). A storyboard is a training design document that prompts the trainer to identify the rationale for the perfor- mance, to develop instructions for the trainee, to determine how the performance will be modeled and what he or she will say while modeling the performance, to determine what prac- tice opportunities will look like for the trainee, to determine how feedback will be delivered to the trainee, and to deter- mine how the trainee’s performance will be evaluated to de- termine whether mastery has been achieved. In other words, the storyboard helps ensure that trainers are, in fact, designing BSTwith each of its components and not some other form of training, which may or may not be evidence-based. Appendix 3 is a storyboard template that supervisors can use to develop their trainings.

Once the training storyboard is developed, all materials to be used in the training must also be developed or gathered. Examples of training materials include written instructions, video-based models, practice activities, written tests, and ma- terials that are necessary for running protocols. Once the per- formance to be trained has been analyzed, the training has been designed, and the training materials have been devel- oped, supervisors can implement BST according to the steps outlined in the storyboard.

Ongoing Performance Monitoring

Behavior analysis is not only a science of learning in the sense of skill acquisition. In fact, a significant portion of the field is dedicated to understanding the variables that increase, de- crease, or maintain behavior after it is established (Skinner, 1950). While the BACB Experience Standards state that, Bthe supervisee’s primary focus should be acquiring new behavior- analytic skills related to the BACB Third Edition Task List or the BACB Fourth Edition Task List as appropriate,^ supervisees also must maintain satisfactory performance after new skills are established (2016, p. 5).

The Experience Standards require supervisors to observe their supervisees in the natural environment at least once each supervisory period (2016, p. 2). As Komaki, Zlotnick, and Jensen (1986) point out, Bonly managers who systematically monitor and collect data are likely to be knowledgeable about subordinates’ performance^ (p. 261). After training is com- plete for a given performance, ongoing monitoring serves to continue the development of the supervisee’s skills and to assess for generalization to new clients, targets, and settings. Ongoing performance monitoring involves regularly conducting formal and informal assessments of supervisee’s skills, updating goals according to assessment results, and delivering performance feedback.

Behav Analysis Practice (2018) 11:62–70 67

Because we have already reviewed the process of conducting assessments and establishing goals, this section will focus on the delivery of performance feedback. Feedback is information about performance that allows a per- son to change that performance in the future (Daniels & Bailey, 2014). Effective feedback contains specific informa- tion; pertains to performances under the control of the per- former; is individualized; is self-monitored when possible, if not self-monitored, is delivered by a person in charge; is fo- cused on improvement; is easily understood; is graphed; is used in conjunction with reinforcement; and is provided as often as feasible (Alvero, Bucklin, & Austin, 2001; Balcazar, Hopkins, & Suarez, 1985-86).

Delivering corrective feedback deserves additional discus- sion, as it can easily be done inappropriately or ineffectively. For example, anger is a motivating operation that might increase the value of the reinforcing effects of yelling or other aggressive behaviors. Therefore, feedback should not be delivered when angry. Delivering corrective feedback in public could be per- ceived by others in the organization as unfair, which might in- crease the reinforcing effectiveness of behaviors that are incon- sistent with the instructions given by the supervisor. Feedback should therefore be given privately. Supervisors should also avoid threatening consequences that they cannot deliver, so that their feedback does not lose effectiveness in managing their supervisees’ performance (Daniels & Bailey, 2014). To deliver corrective feedback, supervisors must describe the ineffective performance and the desired performance, provide rationale and instructions for the desired performance, model the correct performance, provide opportunities for the supervisee to practice the desired performance, provide feedback again, and follow-up as necessary (BACB, 2012b, p. 4). Note that this is the same process that is followed for behavioral skills training.

In their Supervisor Training Curriculum Outline, the BACB states that supervisors should be able to describe sev- eral different methods of delivering performance feedback: vocal, written, modeled, video, graphic, self-monitored, for- mal, and informal (2012c, pp. 4–5). Appendix 4 is a table that gives a description of each of these feedback methods and situations in which each method might be appropriate. Appendix 5 is a Supervision Feedback and Goal Setting Form that supervisors can use to guide their written and vocal feedback and establish goals for the next supervisory period. The form prompts the supervisor to identify the job perfor- mances that were practiced during the supervisory period, their supervisee’s current (validated) level of self-assessment, and goals for the upcoming supervisory period (or the next time the performance will be completed). The form also prompts supervisors to comment on their supervisee’s profes- sional behavior during the supervisory period and provides space for the supervisor to write general comments. In addi- tion, the form prompts supervisors to select several areas of focus for the next supervisory period. The Supervision

Feedback and Goal Setting form can be used in conjunction with the tool for organizing assessment data that we described in the assessment section. Whereas assessment data may be collected only several times throughout the supervisory rela- tionship, the Supervision Feedback and Goal Setting form can be used to document progress every supervisory period, allowing for broad analyses of performance change as well as more molecular analyses.

Ending the Supervisory Relationship

There are several steps supervisors should follow to appropri- ately conclude a supervisory relationship: evaluation of the supervisee’s overall skill development, evaluation of the super- visor’s supervision skills, and completion of documentation required by the BACB. See Turner et al. (2016) for additional considerations for unplanned terminations of supervisory relationships.

Evaluation of the Supervisee’s Overall Skill Development

As discussed earlier, evaluation of supervisees’ performance is, ideally, an ongoing process throughout supervision. However, to remain consistent with the Banalytic^ dimension of applied behavior analysis, the supervisee’s performance should be eval- uated before, during, and after supervision to provide a measure of the supervisee’s overall progress and the impact of supervi- sion (Baer, Wolf, & Risley, 1968; Turner et al., 2016). Quizzes on the Foundational Knowledge section of the Task List, self- assessment measures, direct observation, and reviews of perma- nent products—the same assessment tools and methods that are used to conduct an initial assessment of the supervisee’s skills—can be used to obtain an overall measure of the supervisee’s progress throughout the process of supervision (see the BAssessment of Skills^ section above for a more de- tailed discussion of these tools). While the initial assessment gives a baseline fromwhich training and performance goals can be established, the final assessment gives a post-supervision evaluation. Comparing the pre-supervision performance with post-supervision performance can provide a measure of the supervisee’s progress and growth over the course of supervi- sion, which allows the supervisor to evaluate his or her own effectiveness.

During the final supervision meeting, the supervisor should discuss the results of the supervision evaluation. Pre- and post- supervision graphs of supervisee performance, summary statements from pre- and post-assessments, the supervisee’s updated portfolio, and written recommendations for which skills to focus on improving through subsequent supervision

68 Behav Analysis Practice (2018) 11:62–70

experiences should guide discussion during the final supervi- sion meeting.

Evaluation of the Supervisor’s Effectiveness

Comparing pre- and post-assessments of supervisees’ self- assessments and work products not only provides a picture of the supervisees’ progress; it also provides a rough measure of the supervisor’s effectiveness. In addition to evaluating the supervisee’s progress, supervisors can ask supervisees to com- plete an evaluation of their supervisory effectiveness. Consistent with the recommendations of Turner et al. (2016), we also rec- ommend that supervisors self-evaluate their supervisory effec- tiveness. Additionally, we recommend using the BACB (2012b) Supervisor Training Curriculum Outline to guide such an eval- uation, as this document outlines what the BACB considers to be essential features of supervision. Supervisors can self- evaluate their supervisory effectiveness using the same method and compare their self-evaluations to their supervisees’ evalua- tions to help them set appropriate goals for professional devel- opment. When possible, it is a good idea to have peers or men- tors observe and evaluate supervisory performance to obtain a direct measure of supervisory behavior.

Professional Development Planning

Planning to improve supervision practices takes time and ef- fort. Appendix 6 is a template based on the literature on per- formance management (e.g., Malott, 2003, 2004), task clari- fication (e.g., Slowiak, 2014; Wilson, Boni, & Hogg, 1997; Slowiak, Madden, & Mathews, 2006), goal setting (e.g., Slowiak et al., 2006; Anderson, Crowell, Doman, & Howard, 1988), and reinforcement (e.g., Skinner, 1999/ 1961; Skinner, 1999/1968) that supervisors can use to estab- lish goals and deadlines, determine the actions required to achieve the goals, and create a reinforcement plan to follow when those goals have been met. We hope that supervisors will use this tool in their own practices to add to their imple- mentation of effective supervision practices.

Conclusion

BCBA supervision is a process concerned with teaching new behavior-analytic skills and maintaining those skills after they have been mastered. As a field that has developed effective techniques for increasing, decreasing, and maintaining behav- ior, it makes sense that we would use the same technology to teach and maintain the skills of those in training to become professional behavior analysts and to improve the effective- ness of supervision practices. This paper has proposed a

general process to follow while supervising aspiring BCBAs as well as materials to guide the implementation of that pro- cess. We encourage supervisors to use these materials and to share training protocols with others with the goal of maintain- ing the integrity of behavior analysis and keeping an emphasis on data-based decision-making.

Acknowledgements Portions of this project were supported through generous funding from the Michigan Department of Health and Human Services (Grant No. 20152526; Principle Investigators Stephanie Peterson and Wayne Fuqua). The opinions expressed in this paper are those of the authors and do not necessarily reflect those of the funding agency. The authors would like to express appreciation to Dr. Stephanie Peterson and Dr. Wayne Fuqua for making this project possible through their efforts to obtain andmaintain funding and their diligent management and oversight.

Compliance with Ethical Standards This project was funded by the Michigan Department of Health and Human Services (Grant no. 20152526).

Conflict of Interest The authors declare that they have no conflict of interest.

Ethical Approval This article does not contain any studies with human participants or animals performed by any of the authors.

References

Alvero, A.M., Bucklin, B.R., & Austin, J. (2001). An objective review of the effectiveness and essential characteristics of performance feed- back in organizational settings (1985–1998). Journal of Organizational Behavior Management, 21(1), 3-29. doi:10.1300/ J075v21n01_02 .

Anderson, D. C., Crowell, C. R., Doman, M., & Howard, G. S. (1988). Performance posting, goal setting, and activity-contingent praise as applied to a university hockey team. Journal of Applied Psychology, 73(1), 87–95.

Baer, D. M., Wolf, M. M., & Risley, T. R. (1968). Some current dimen- sions of applied behavior analysis. Journal of Applied Behavior Analysis, 1, 91–97.

Balcazar, F. E., Hopkins, B. L., & Suarez, Y. (1985-1986). A critical, ob- jective review of performance feedback. Journal of Organizational Behavior Management, 7(3–4), 65–89.

Behavior Analyst Certification Board®. (2012a). Behavior analyst certi- fication board: fourth-edition task list. Retrieved from http://www. bacb.com/Downloadfiles/TaskList/BACB_Fourth_Edition_Task_ List.pdf

Behavior Analyst Certification Board®. (2012b). Supervisor training curriculum outline. Retrieved from http://bacb.com/wp-content/ uploads/2015/05/supervisor_curriculum.pdf.

Behavior Analyst Certification Board®. (2016). Behavior analyst certifi- cation board: experience standards. Retrieved from http://www. bacb.com/Downloadfiles/ExamApplications/bcba/150706% 20experience%20standards.pdf.

Brobst, B., &Ward, P. (2002). Effects of public posting, goal setting, and oral feedback on the skills of female soccer players. Journal of Applied Behavior Analysis, 35, 247–257.

Bucklin, B. R., Alvero, A. M., Dickinson, A. M., Austin, J., & Jackson, A. K. (2000). Industrial-organizational psychology and

Behav Analysis Practice (2018) 11:62–70 69

organizational behavior management. Journal of Organizational Behavior Management, 20(2), 27–75. doi:10.1300/J075v20n02 03.

Bunch, J. (1991). The storyboard strategy. Training and Development, 45(7), 69–71.

Cooper, J. O., Heron, T. E., & Heward, W. L. (2007). Applied behavior analysis (Second ed.). Columbus: Pearson.

Cunningham, T. R., & Austin, J. (2007). Using goal setting, task clarifi- cation, and feedback to increase the use of the hands-free technique by hospital operating room staff. Journal of Applied Behavior Analysis, 40, 673–677. doi:10.1901/jaba.2007.673-677.

Daniels, A. C., & Bailey, J. S. (2014). Performance management: chang- ing behavior that drives organizational performance (Fifth ed.). Atlanta: Performance Management Publications.

Dixon, D. R., Linstead, E., Granpeesheh, D., Novack, M. N., French, R., Stevens, E., et al. (2016). An evaluation of the impact of supervision intensity, supervisor qualifications, and caseload outcomes in the treatment of autism spectrum disorder. Behavior Analysis in Practice, 9, 339–348. doi:10.1007/s40617-016-0132-1.

Downing, C. O., & Geller, E. S. (2012). A goal-setting and feedback intervention to increase ID-checking behavior: an assessment of social validity and behavioral impact. Journal of Organizational Behavior Management, 32(4), 297–306. doi:10.1080/01608061. 2012.729383.

Durgin, A., Mahoney, A., Cox, C., Weetjens, B. J., & Poling, A. (2014). Using task clarification and feedback training to improve staff per- formance in an East African nongovernmental organization. Journal of Organizational Behavior Management, 34(2), 122–143. doi:10. 1080/0160861.2014.914007.

Gil, P. J., & Carter, S. L. (2016). Graphic feedback, performance feed- back, and goal setting increased staff compliance with a data collec- tion task at a large residential facility. Journal of Organizational Behavior Management, 36(1), 56–70. doi:10.1080/01608061. 2016.1152207.

Gilbert, T. F. (1996). Human competence: engineering worthy performance (Tribute ed.). Washington, DC: The International Society for Performance Improvement.

Gravina, N., VanWagner, M., & Austin, J. (2008). Increasing physical therapy equipment preparation using task clarification, feedback and environmental manipulations. Journal of Organizational Behavior Mangement, 28(2), 110–122. doi:10.1080/0160860802100931.

Green, N., Sigurdsson, S., & Wilder, D. A. (2016). Decreasing bouts of prolonged sitting among office workers. Journal of Applied Behavior Analysis, 49, 717–722. doi:10.1002/jaba.309.

Hartley, B. K., Courtney, W. T., Rosswurm, M., & LaMarca, V. J. (2016). The apprentice: an innovative approach to meet the Behavior Analysis Certification Board’s supervision standards. Behavior Analysis in Practice, 9, 329–338. doi:10.1007/s40617-016-0136-x.

Jeffrey, S. A., Schulz, A., &Webb, A. (2012). The performance effects of an ability-based approach to goal assignment. Journal of Organizational Behavior Management, 32(3), 221–241. doi:10. 1080/01608061.2012.698116.

Kazbour, R. R., McGee, H. M., Mooney, T., Masica, L., & Brinkerhoff, R. O. (2013). Evaluating the impact of a performance-based meth- odology on transfer of training. Performance Improvement Quarterly, 26(1), 5–33. doi:10.1002/piq.21135.

Kazdin, A. (1982). Single-case research designs: methods for clinical and applied settings. New York: Oxford University Press.

Komaki, J. L. (1986). Toward effective supervision: an operant analysis and comparison of managers at work. Journal of Applied Psychology, 71(2), 270–279.

Komaki, J. L., Zlotnick, S., & Jensen, M. (1986). Development of an operant-based taxonomy and observational index of supervisory be- havior. Journal of Applied Psychology, 71(2), 260–269.

Loewy, S., & Bailey, J. (2007). The effects of graphic feedback, goal setting, and manager praise on customer service behaviors.

Journal of Organizational Behavior Mangement, 27(3), 15–26. doi:10.1300/J075v27n03 02.

Mager, R. F. (1997). Making instruction work or Skillbloomers: a step- by-step guide to designing and developing instruction that works (Second ed.). Atlanta: CEP Press.

Malott, R. W. (2003). What OBM needs is more Jewish mothers. Journal of Organizational Behavior Management, 22(2), 71–87. doi:10. 1300/J075v22n02_07.

Malott, R.W. (2004). Notes from an introspective behavior analyst. Journal of Organizational Behavior Management, 24(1–2), 75– 112. doi: 10.1300/J075v24n01_06.

Parsons, M. B., Rollyson, J. H., & Reid, D. H. (2012). Evidence-based staff training: a guide for practitioners. Behavior Analysis in Practice, 5(2), 2–11.

Rummler, G. A., & Brache, A. P. (2012). Improving Performance: How to Manage the White Space on the Organization Chart. Somerset: John Wiley & Sons, Incorporated. Retrieved from https:// ebookcentral-proquest-com.libproxy.library.wmich.edu/lib/ wmichlib-ebooks/detail.action?docID=821830.

Sellers, T. P., Alai-Rosales, S., &MacDonald, R. P. F. (2016a). Taking full responsibility: the ethics of supervision in behavior analytic practice. Behavior Analysis in Practice, 9, 299–308. doi:10.1007/s40617- 016-0144-x.

Sellers, T. P., LeBlanc, L. A., & Valentino, A. L. (2016b). Recommendations for detecting and addressing barriers to success- ful supervision. Behavior Analysis in Practice, 9, 309–319. doi:10. 1007/s40617-016-0142-z.

Sellers, T. P., Valentino, A. L., & LeBlanc, L. A. (2016c). Recommended practices for individual supervision of aspiring behavior analysts. Behavior Analysis in Practice, 9, 274–286. doi:10.1007/s40617- 016-0110-7.

Skinner, B. F. (1950). Are theories of learning necessary? Psychological Review, 57(4), 193–216.

Skinner, B. F. (1999a). The design of cultures. In J. S. Vargas (Ed.), Cumulative Record (Definitive ed.). Cambridge: B.F. Skinner Foundation (Original work published in 1961).

Skinner, B. F. (1999b). The design of experimental communities. In J. S. Vargas (Ed.), Cumulative record (Definitive ed.). Cambridge: B.F. Skinner Foundation (Original work published in 1968).

Slowiak, J. M. (2014). BHow may I help you?^ Improving telephone customer service in a medical clinic setting. Journal of Organizational Behavior Management, 34, 39–51. doi:10.1080/ 01608061.2013.873382.

Slowiak, J. M., Madden, G. J., & Mathews, R. (2006). The effects of a combined task clarification, goal setting, feedback, and performance contingent consequence intervention prackage on telephone custom- er service in a medical clinic environment. Journal of Organizational Behavior Management, 25(4), 15–35. doi:10.1300/ J075v25n04_02.

Tittelbach, D., DeAngelis, M., Sturmey, P., & Alvero, A. M. (2007). The effects of task clarification, feedback, and goal setting on student advisors’ office behaviors and customer service. Journal of Organizational Behavior Management, 27(3), 27–40. doi:10.1300/ J075v27n03 03.

Turner, L. B., Fischer, A. J., & Luiselli, J. K. (2016). Towards a compe- tency-based, ethical, and socially valid approach to the supervision of applied behavior analytic trainees. Behavior Analysis in Practice, 9, 287–298. doi:10.1007/s40617-016-0121-4.

Valentino, A. L., LeBlanc, L. A., & Sellers, T. P. (2016). The benefits of group supervision and a recommended structure for implementation. Behavior Analysis in Practice, 9, 320–328. doi:10.1007/s40617- 016-0138-8.

Wilson, C., Boni, N., & Hogg, A. (1997). The effectiveness of task clar- ification, positive reinforcement, and corrective feedback in chang- ing courtesy among police staff. Journal of Organizational Behavior Management, 17(1), 65–96.

70 Behav Analysis Practice (2018) 11:62–70