Module 3 Reflection Paper

SupervisionCourse-Module3-MeasurementandEval-GuidedNotes1.pdf

Home >Applied Sciences homework help >Module 3 Reflection Paper

Supervision – Measurement and Evaluation Biagi

Supervision Essentials for Behavior Analysts With Shannon Biagi, MS, BCBA (she/her)

Note: This training program is based on the BACB® Supervisor Training Curriculum Outline (2.0) but is offered independent of the BACB®.

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Please sign here upon completion of these guided notes to agree to the statements below: I attest that the content filled in this document is only my own original work. I understand that copying information into this document or representing another's work as my own is academic misconduct and a violation of UWF's Student Code of Conduct. I will not share this document or any answers therein with any other party, as this is also a violation of UWF's Student Code of Conduct.

Printed Name:

Signature:

Supervision – Measurement and Evaluation Biagi

Measurement and Evaluation

in Supervision This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Objectives

The goals of this lesson are to: • Select strategies that may be used to collect baseline

data of supervisee and trainee performance

• Determine the difference between behavior and outcome measures, as well as other factors that are mistaken for behaviors, and determine when to measure each

• Identify several strategies for measuring performance and when best to use each

• Differentiate between formal and informal monitoring, and how to increase staff acceptability of each

• Determine opportunities to collect data to evaluate the effectiveness of the supervision process

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Measurement and Evaluation As supervisors, determining the general, broad content that we should be evaluating is made easier by the task lists provided by the BACB®. However, because of how diverse the practice of behavior analysis can be, the Board does not provide highly precise behaviors to measure - what one task list item may look like

in one setting may look quite different in another.

This leads to some additional work on the part of the supervisor to determine the best behaviors to evaluate, define them clearly, create measurement tools to collect data on them, and evaluate how the

performer is doing.This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

On Pre-Developed Supervision Curricula There have been a number of practitioners and academics who have attempted to alleviate some of this burden, primarily within the clinical ABA space for trainee supervision, by compiling measurement systems for supervision.

Using one of these programs might give supervisors a head start in creating their evaluative tools; however, developing and using measurement tools that are specific to your practice area, organization, population, and setting is a critical skill for a BCBA and part of your _______________ as a supervisor.

Important Note:

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Establishing Supervision

Assessments

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Developing Supervision Assessments

Garza, et al. (2018)

Before beginning a supervision relationship, it is important that we establish a baseline level of skills of the supervisees/trainees that we oversee. Supervisors will need to observe behavior, review permanent products, and the results of this process will influence the trajectory of the

supervision program for each specific supervisee/trainee.

But how to be determine exactly what to assess in a supervision program? This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Conducting a _________ Defines the performance responsibilities and accomplishments associated with a specific job role. This process moves beyond the Task List and establishes priorities in the skills to be taught and evaluated, as the job modelling process requires leaders and supervisors to determine the precise expectations of their supervisees and trainees in the real working context. Creating a _______________ for a behavior analytic role also helps determine if trainees will be able to practice all of the skills required in the Task List, and if those opportunities are not available within a specific role, it may be better for the trainee to accrue hours in a different role or with another organization.

Garza, et al. (2018)

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Garza, et al. (2018) – Initial Skills Assessment in Supervision

Oral and Written Quizzes

Validation through Observation and Product Reviews

Self-Assessment

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Self-assessment involves a trainee rating their own skill level for each job responsibility or task list skill

This allows for trainees to learn how to self-evaluate proficiency and provides a low-effort starting point for supervisors to begin the supervision process

There are caveats, of course, in that folks may over- or under- estimate their abilities, but this is only the ________________ in the supervision evaluation process

Tagg & Biagi (2019), as published in LeBlanc, Sellers, and Ala’I (2020)

Using a Self- Assessment

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Oral and Written Quizzes As a supervisor, you also have the opportunity to ensure that your trainees have gained the necessary book knowledge to be successful on their exam and in practice. Although this is typically the responsibility of their _____________________, it would be in the supervisor’s best interest to help plug any knowledge-based holes with the trainee. You can also help shape their _______________ repertoires through repeated practice of multiple choice questions. Utilizing mock examinations and quizzes can be helpful to establish a baseline of “Learning” of the trainee, but ensure that you vet the source of the exam.

Supervision Assessments

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Example: Bethany, a supervisor, has taken on a brand new trainee who has already started their coursework. The trainee completes a self-assessment rating of their skills with the task list, and reports high competency in the area of consequence interventions.

Bethany decides to confirm this by having the trainee take a brief quiz on consequence strategies, and goes through a few scenarios with the trainee during their early supervision meetings. The trainee scores perfectly on the quiz, and is able to describe the answers to the scenarios fluently. Bethany, based upon this assessment, feels comfortable allowing this trainee to assist with developing consequence-based interventions during supervision, and will continue to monitor the trainee’s performance to ensure competency.

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

How could we assess for initial skills when taking on a supervisee/trainee?

Checking in:

a. Have the supervisee evaluate their skills and comfort level with the items specified on the task list

b. Provide a quiz in which they answer questions related to task list items c. Have them verbally describe how they would conduct specific procedures

required of a behavior analyst d. All of these are acceptable strategies to assess initial skills of

supervisees/trainees

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Observations & Permanent Product

Measurement

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

What Behavior is NOT

Daniels and Bailey (2014)

Commonly mistaken for behavior, the following do not meet the definitions we’ve discussed:

Broad terms like professionalism, communication, or supervising are not precise enough to be able to

accurately describe what someone is doing or measure how someone

is performing.

_____________ Lazy, disengaged, conscientious, or aware… none of these attitude

or personality-related terms describe anything observable in a

useful manner for us to collect data.

_____________ States occur as a result of behavior, and fail the Dead

Person’s test. We must reinforce the behavior (putting on safety

glasses) that resulted in the state (wearing safety glasses), not the

state alone.

___________ Values are umbrella terms for

many different behaviors, and can be interpreted differently across performers. What does respect

look like? What about teamwork? Does everyone agree?

____________

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Getting Precise In OBM, when we __________, we are objectively and precisely describing what a person is

doing or has produced.

The practice of pinpointing is equivalent to developing our _____________________ in ABA, and therefore all of the requirements of a good operational definition apply when developing out

a pinpoint. This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Characteristics of Good Pinpoints (Daniels and Bailey, 2014)

Can be counted or assessed in an objective manner

Measurable Can be seen with the human eye

Observable Multiple observers agree, think Inter-observer agreement (IOA)

Reliable

The performer MUST have a major influence over whatever is being measured

Under the Performer’s Control

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Behaviors and Outputs • The process of pinpointing is as follows:

1. Describe the results, outcomes, or accomplishments that one is looking to impact, then

2. Identify and describe the behaviors that reliably produce the results, outputs, or accomplishments

• It is critical to note that we begin with the defining the _____________, then the _____________.

DiGennaro Reed, et al. (2018)

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation BiagiDaniels and Bailey (2014)

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation BiagiWhen Do I Measure Behaviors or Outcomes? (Rodriguez, Sundberg, and Biagi, 2016)

Measure Behaviors When… Measure Outcomes When… Measure BOTH When…

Employee is learning a new skill and you need to provide frequent, immediate ______________

Behaviors and outcomes are highly correlated – one cannot occur without the other

Outcomes are important, but they must be reached in an __________ or safe manner

Employee needs a great deal of feedback to continue performing the desired behaviors

Leaders are ____________ the employee is engaging in the right behaviors to achieve outcomes

Behavior can be used to provide more meaningful ___________ than results alone

Leaders need to be sure the employees are engaging in the _______ behaviors to achieve the results

Only a quick snapshot is required to determine how well employees are performing

Outcomes of behavior are very __________

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Example: River is interested in targeting the completion of forms of their trainees. They decide to collect data by setting up a camera and having someone monitor as trainees put their completed forms into a box. The trainees feel that this is a rather intrusive strategy for monitoring form submissions, especially since they have been submitting the forms for some time, but River is unsure of a better way to evaluate form submission behavior.

However, rather than measuring the behavior of submitting the form, River could have more easily reviewed the outcome, the permanent product of the submitted form, instead. This would save time, resources, and have likely been more acceptable to the trainees who were being evaluated.

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Is this a behavior or an outcome? Defining a behavior.

Checking in:

a. Behavior

b. Outcome

c. Neither

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Is this a behavior or an outcome? A behavioral definition.

Checking in:

a. Behavior

b. Outcome

c. Neither

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Is this a behavior or an outcome? Implementing a behavior program.

Checking in:

a. Behavior

b. Outcome

c. Neither

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Is this a behavior or an outcome? Client goals met.

Checking in:

a. Behavior

b. Outcome

c. Neither

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Is this a behavior or an outcome? Professionalism.

Checking in:

a. Behavior

b. Outcome

c. Neither

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Which of the following is an acceptable target for evaluation for a

supervisee/trainee?

Checking in:

a. Enthusiasm

b. Teamwork

c. Completed and accurate session notes

d. Respectfulness

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Why Measure? When we don’t measure behaviors, outputs, and business results, we cannot:

Provide targeted ____________,

Improve supervisee/trainee performance,

Or tell if performance is improving, maintaining, or decreasing.

When we measure, we can identify effective reinforcers, communicate more clearly, react less emotionally, and make better decisions for the good of everyone.

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Effective managers spend significantly more time ____________ and work

sampling than marginally effective supervisors.

Komaki, 1986

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Measuring Behavior in OBM DiGennaro Reed, et al. (2018)

Derived Measures

Built upon the data provided in the Fundamental Measures category, allows standardization across an organization.

• Percentage • Trials-to-Criterion

Time Sampling Measures

Useful when continuous measurement isn’t possible, but can vary on accuracy.

• Whole Interval Recording

• Partial Interval Recording

• Momentary Time Sampling

Other Measures Behaviorally-anchored Rating Scales - Numeric scales tied to behavioral definitions

Permanent Product – measuring characteristics of the output of behavior.

Fundamental Measures

Data collected through direct observation that immediately provides useful information.

• Frequency • Rate • Duration • Latency • Inter-response

Time

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Measurement in Human Services Examples:

Fundamental Measures • Number of billable hours delivered

• ___________________

• Number of trials run per hour • ___________________

• Length of each session • ___________________

• Time taken to deliver the reinforcer after a correct client response

• ___________________

• Time between each trial • ___________________

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Advanced Measurement Tools

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

The Power of Checklists Checklists are lists of essential steps required to complete a specific task, or specific

characteristics required of a permanent product produced by a process. They make measuring quality of outputs and completeness of behavior change easy and consistent for those

measuring performance.

In his New York Times best selling book, The Checklist Manifesto, Dr. Atul Gawande takes a deep dive on the use of checklists across industries, including construction, piloting aircrafts, and

in his own industry – surgical medicine.

Creating useful checklists isn’t as easy as simply creating a ___________________ and checking boxes, and his book is HIGHLY recommended reading to explore how to develop

effective checklists and overcome resistance to their use.

According to Gawande, checklists are useful to:

• Ensure that critical steps aren’t accidentally/intentionally skipped

• Provide additional guidance for performers

• Increase communication, efficiency, and consistency

• Decrease expenses and incidentsThis content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation BiagiA Checklist for Checklists (Gawande, 2009)

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Measurement Examples: Behavioral Checklist • Provides a count of the

occurrence of behavior

• Can either be reported as a count (number of items observed) or as a percentage of items met

• Checklist items need to be carefully pinpointed

• Can be weighted or unweighted (items can be worth the same amount, or different amounts based on importance)

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Measurement Examples: Permanent Product Checklist • Provides a count of desired

characteristics present in a permanent product

• Can also be reported as a count (number of items observed) or as a percentage of items met

• Checklist items need to be carefully pinpointed

• Can also be weighted or unweighted (items can be worth the same amount, or different amounts based on importance)

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Behaviorally Anchored Rating Scales (BARS) • Provide a way to measure how an individual's behavior in various performance

categories contributes to achieving the goals of the team or organization of which the individual is a member (Campbell, Dunnette, Arvey, & Hellervik, 1973)

• Provides an “_____________” to numerical values using measurable and observable behavior descriptors

• Ohland, Loughry, & Felder (2012) found BARS to be more predictive of outcomes of teamwork behavior than Likert-style rating scales

• E.g. Individuals who scored high on a teamwork BARS were ______________ to be selected for teams in the future than those who outscored them on Likert scales

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Developing a BARS 1. Define the behavior of interest at a high level

• Example – Receiving Feedback

2. Determine the level of sensitivity necessary for the metric • Is 4-5 levels ideal? Do we need more sensitivity with 10 levels?

Or could we provide only 3 levels and get sufficient information?

3. Determine the _____________________ of the behavior of interest • Example – Preparation, asking follow up questions,

acknowledging mistakes, active listening, etc.

4. Create a table with the same number of columns as your sensitivity level, and a row for each critical component • The levels are often titled things like “Excellent”, “Very Good”,

“Good”, “Needs Improvement”, etc. • These are often also tied to numerical values to allow for ease

of data collection

5. Fill in what each critical component looks like at each level of performance – being as objective and clear as possible

1 2 3 4

Criteria 1

Criteria 2

Criteria 3

Criteria 4

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Measurement Examples: Behaviorally-Anchored Rating Scale (BARS)

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Daniels and Bailey (2014)

Other Considerations • Behavior __________ – multiple behaviors that

produce the same output • Example: Composing an email

• Did you type it? Voice-to-text? Does it matter?

• Behavior _____________ – multiple behaviors, performed in a sequence, that produce a single output

• Example: Parking a car • Pull into the space, press the brake, put

hand on the gear shift, grip, move until the P lights up, release the gear shift and brake…

• The level of analysis depends on how much information is required to produce the valuable behavior change

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Best Practices in Measurement

• Use the ____________________ (“hard data”) whenever possible

• Lean on ________________ when judgement is required

• Don’t shy away from ___________, as it is often impractical to observe each behavior or review every product

• Avoid _________ scales without anchoring whenever possible

• Try to gather at least ____ consistent data points

• Test the tools you develop for inter-observer agreement or inter-rater reliability ASAP

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Considerations When Developing a Measurement System

Who will be collecting the data?

Who will analyze the data?

Who will disseminate the data?

What type of data will we collect?

Leading or lagging indicators?

Continuous or sampling procedures?

When will the data be collected?

How often?

When will we disseminate?

How will we train the data collectors and analyzers?

How will we train the employees on the criteria we’ve defined?

How will we confirm that the measures are getting at the critical factors?

How will we confirm that multiple people using the same tools report the same data?

How will we report data to the various stakeholders?

How will we share the data with the performers themselves?

Graphs!

What? When?Who?

How will we ensure reliability and validity?

How will we share the data?

How will we train others?

DiGennaro Reed, et al. (2018)

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Which is the following is the best strategy for measuring this behavior analytic activity:

Conduct preference assessments.

Checking in:

a. Observe the trainee conducting the preference assessment, and provide a rating on a scale of 1-5 on how well the assessment was conducted.

b. Create a task analysis for the specific preference assessment you are evaluating, observe the trainee, and determine how many steps were performed correctly.

c. Review the data sheet from the preference assessment and evaluate which items act as reinforcers for the client during the session.

d. None of these. This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Performance Monitoring

An Essential Component of Supervision

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Direct Observation of Supervisee/Trainee Performance After the defining the targets and developing the measurement tools, conducting direct

observations of supervisee performance is a large part of evaluating the supervision process!

This is known as “_______________________”, and can be done either formally or informally (Reid, Parsons, and Green, 2021).

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Reid, Parsons and Green (2021)

Formal Monitoring • Structured performance monitoring • Directly observing staff while collecting

specified information on performance • Structured based on objective

performance criteria directly related to client/business outcomes

• Ex.: Regulatory compliance, safety, skill acquisition, etc.

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Formal Monitoring: Proper Use Should be used to either _________ or __________ staff performance

• ___________ • Strengthen proper performance via positive feedback • Identify staff needs (e.g., inadequate resources)

• ___________ • Identify problems in staff performance and provide corrective feedback • Identify training needs and provide necessary additional training

opportunities

Reid, Parsons and Green (2021)This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Formal Monitoring: Misuse • When performance monitoring data are simply stored/ignored

(an institutional pitfall) this becomes a process with no real purpose.

• Monitoring is NOT about demonstrating ____________ with requirements of oversight bodies

• Collection of unused monitoring data can undercut healthy supervisory relationships

• Can promote _____________ about the use and effectiveness of potentially helpful managerial processes

• Can model “data collection” as a _______________________

Reid, Parsons and Green (2021)This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Example: Holly is a new supervisor, with a new caseload and new trainees and supervisees to oversee. When she begins collecting data during supervision, she notices that the staff become very uncomfortable. This makes Holly uncomfortable as well, and she begins to change her behavior. Trying to pair with the team, Holly jokes with the staff about how the evaluations “aren’t that serious” and that it’s not something she really sees value in either, but we have to do. This appears to put the team at ease while Holly completes her formal monitoring tasks. However, Holly notices that the data collected by staff during session begin to decrease in accuracy, and when Holly tries to provide feedback, the staff push back, saying, “you said this didn’t matter anyway”. Moving forward, this issue will be incredibly difficult for Holly to resolve.

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Formal Monitoring: When to Use Should occur most frequently when: 1. Staff are performing ____________

2. Feedback was recently provided to correct performance deficits

3. A supervisor has concerns about the quality of a specific performance

4. Duties directly relate to assisting clients attain outcomes, or engages in activities that interfere with completion of those duties

Reid, Parsons and Green (2021)This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Reid, Parsons and Green (2021)

Informal Monitoring • Requires no ________, can occur at any

point in the day • Consists of quick observations to

determine if performance is adequate • No ______________ • Should occur _____________ the

supervisor is at work!

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Reid, Parsons and Green (2021)

Informal Monitoring: Benefits • Increases opportunity for _________________

• Makes the supervisor visible to staff

• “Wow, he/she must be sincerely interested in my work! My work must be important!”

• Makes it easier for supervisee to ask questions on assignments or work tasks

• Supervisors are more aware of the __________ of the work area

• Establishes ____________ with staff

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Reid, Parsons and Green (2021)

________ to Monitoring • Defined: staff changing their work

behavior when a supervisor is present

• Results in an inaccurate representation of staff performance

• Life example: Police car on highway, under billboard, empty; other car with lights

• Steps should be taken to reduce the impact of ___________

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Reid, Parsons and Green (2021)

Reducing __________ 1. Monitor frequently! 2. Start monitoring as soon as you

enter the workspace 3. Conduct unexpected follow-up

observations shortly after the initial observation

4. Use an ___________ monitoring schedule

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

• Supervisors may be tempted to use covert monitoring (to combat reactivity, for example)

• It can be effective, but… • Rarely remains covert • ___________ must be given eventually! • Attempting to hide a monitoring form will not work

100% • Often reduces supervisee ________ for supervisor • People do not like the sense of being spied upon • Implies a lack of trust, honesty, and

professionalism

• Covert monitoring _____________ be part of the regular supervision process!

Reid, Parsons and Green (2021)

Covert Monitoring

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Covert Monitoring: When to Use -- The Only Exception If something highly unacceptable or illegal is happening:

Client abuse, neglect, sleeping on the job, stealing from agency or clients, consuming illegal substances or alcohol at work…

The aim is not to provide corrective or supportive feedback. Covert monitoring may result in firm disciplinary and/or legal action.

This is ________________, not supervision

Reid, Parsons and Green (2021) This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Increasing Acceptability Staff typically dislike having their performance monitored, especially formal monitoring • Being closely watched can produce anxiety – and sometimes even evoke

______________! • Many people have experience with poor feedback and/or monitoring being

used for punishment • A signal of aversive managerial actions to come – a warning stimulus,

triggering escape or avoidance behavior • Observation onset may function as a form of conditioned Sp+, and may

thus trigger forms of counter-control (resistance)

Reid, Parsons and Green (2021)This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Guidelines for Making Formal Monitoring Acceptable To Staff 1. Inform staff prior to monitoring what will be monitored and why

2. Upon entering the workplace, greet all staff present (before beginning formal monitoring)

3. Discontinue monitoring when a potentially harmful or embarrassing situation occurs

4. Provide __________ to staff soon after monitoring

5. Acknowledge staff upon completion of monitoring prior to departing the workplace

Reid, Parsons and Green (2012)This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Overt Monitoring • Done properly, overt monitoring is the

most effective and ethical professional practice

• One goal is to catch as much good behavior as possible. ____________ works, by definition!

• Once monitoring becomes routine, it produces benefits for everyone: consumers, supervisees, and supervisors

• The ________ must promote the idea that everyone is subject to being openly monitored

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

A manager schedules a specific time to supervise and collects data during the

session. This is what type of monitoring?

Checking in:

A. Formal

B. Informal

C. Neither of these

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

A manager walks through the work area, and stops to provide support to staff who are

handling a tough situation. This is what type of monitoring?

Checking in:

A. Formal

B. Informal

C. Neither of these

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Which is the following is the best strategy for monitoring this behavior

analytic activity: Use Shaping.

Checking in:

a. Observe the session via camera and provide feedback in the next supervision meeting

b. Apologize for conducting the observation and let them know that it’s a regulatory compliance issue that needs to be done

c. Discuss the shaping evaluation with staff prior to observation, and let them know when you are collecting data

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Evaluating Supervision

…not just the supervisees and trainees

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Evaluating the Effectiveness of Supervision During the supervision process, the supervisees and trainees are not the only individuals in the equation

who must have their behaviors and outputs evaluated.

We must also evaluate ourselves as supervisors, and the effectiveness of our supervision systems and practices in order to ensure that our supervisees/trainees and clients are also receiving high-quality

services.

How do we know if our supervision is effective?This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Effective _________ Practices

Effective __________ Behaviors

Increased ________ Outcomes

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Evaluating the Effects of Supervision: Client Factors • Degree of client progress

• Adequacy of operational definitions

• Appropriateness of selected interventions and procedures

• Number and appropriateness of programs

• Type and number of targets in programs

• Appropriately and timely fading of supports when moving into maintenance

• Rate of mastery within and across targets

• Generalization and maintenance of targets

• Quality of therapeutic relationships - satisfaction

LeBlanc, Sellers, and Ala’i (2020)

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

• Post-supervision evaluation compared to ______________

• Complete the self-assessment and quiz/oral exam again

• Collecting ____ on targeted procedures • Evaluate the number of ________ met • Supervisee feedback and social validity

• In-person check-ins • Surveys

• Supervisory Relationship Questionnaire (SRQ) has been validated for this use and is available online

LeBlanc, Sellers, and Ala’i (2020)

Evaluating the Effects of Supervision: Supervisee Factors

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Evaluating the Effects of Supervision: Supervisor Factors • Evaluate own comfort and feelings

around the supervision process

• Audit their own systems and processes

• Collecting data and gathering IOA and feedback on ______________ behaviors

• From supervisees/trainees, other collaborators, and other supervisors or mentors

• Turner, Fischer, & Luiselli (2016) provide a great monitoring form for analysts to have others evaluate supervision behaviors

LeBlanc, Sellers, and Ala’i (2020)

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

In order to evaluate the effects of supervision, a supervisor should measure:

Checking in:

A. Supervisee/Trainee feedback

B. Client outcomes

C. The quality of their systems and processes

D. Post-supervision knowledge and skill compared to baseline

E. All of these

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

Conclusion

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

Supervision – Measurement and Evaluation Biagi

For supervisors, it is an ________________ obligation for us to carefully measure and evaluate

all aspects of the supervisory undertaking.

This means collecting _________, creating evaluation _________, and engaging in

___________________ of supervisees and trainees.

We also must evaluate __________________ and practices to ensure the best outcomes for

everyone involved, from the supervisees and trainees, to each and every client we work with.

This content is protected and may not be shared, uploaded, or distributed other than what has been authorized by the University of West Florida..

as a supervisor: obligation
Conducting a: Job Model
Creating a: job model
in the supervision evaluation: initial starting point
supervisors best interest to help plug any: academic institutions
repertoires through repeated practice of multiple: test-taking
Broad terms like professionalism: Generalities
Lazy disengaged conscientious: Attitudes
States occur as a result of: States
Values are umbrella terms for: Values
we are objectively and precisely describing what a person is: pinpoint
in ABA: operational definitions
then the: outputs and results
undefined: behavior
you need to provide frequent: feedback
Outcomes are important but they: ethical
without the other: confident
Behavior can be used to provide: feedback
employees are engaging in the: right
Outcomes of behavior are very: delayed
undefined_2: reinforcement
undefined_3: count or frequency
undefined_4: rate
undefined_5: duration
undefined_6: Iatency
undefined_7: IRT
checking boxes and his book is HIGHLY recommended reading to explore how to develop: task analysis
Provides an: anchor
Eg Individuals who scored high on a teamwork BARS were: more likely
3 Determine the: critical components
produce the same output: Classes
performed in a sequence that produce a single output: Chains
hard data: Fundamental measures
when judgement is: rating scales
as it is often: sampling
scales without anchoring: Likert
consistent data points: 3
This is known as: Performance Monitoring
Should be used to either: support
or: correct
undefined_8: Support
undefined_9: Correct
Monitoring is NOT about demonstrating: compliance
Can promote: cynicism
Can model data collection as a: useless exercise
Staff are performing: new skills
Requires no: tools
No: documentation
Should occur: every day
Increases opportunity for: positive feedback
Supervisors are more aware of the: realities
Establishes: credibility
Reid Parsons and Green 2021: Reactivity
Steps should be taken to reduce: reactivity
Reducing: Reactivity
Use an: unpredictable
must be given eventually: Feedback
for supervisor: respect
be part of the: should Not
not supervision: investigation
undefined_10: atypical errors
Provide: feedback
behavior as possible: Reinforcement
The: culture
Practices: Supervision
Behaviors: Supervisee
Outcomes: Client
Complete the selfassessment and quizoral: baseline
Evaluate the number of: IOA
Supervisee feedback and social validity: goals
and feedback on: supervision
obligation for us to carefully measure and evaluate: professional and ethical
creating: baseline
and engaging in: tools
of supervisees and trainees: performance monitoring
and: our own behavior
Group1: Choice5
Group2: Choice4
Group3: Choice1
Group4: Choice2
Group5: Choice1
Group6: Choice2
Group7: Choice3
Group8: Choice3
Text2: monitoring
Group9: Choice2
Group10: Choice1
Group11: Choice2
Group12: Choice3
Student Name: