A case study outlining a hypothetical performance issue, including an executive summary, assessments, interventions, and results, using performance management tools and behavioral science principles.

PM – Pinpointing and Measurement Biagi

Performance Management With Shannon Biagi, MS, BCBA (she/her)

PM – Pinpointing and Measurement Biagi

Pinpointing and Measurement: Business IS Behavior

PM – Pinpointing and Measurement Biagi

Objectives

The goals of this lesson are to: • Differentiate between behaviors, generalities,

attitudes, states, and values in the workplace

• Select the appropriate order of steps in the pinpointing process

• Differentiate between behaviors, outputs, and business results

• Determine how detailed a pinpoint must be in context of desired behavior change

• Determine which measurement strategies are fundamental, derived, sampled, or other

• Given a scenario, select the most appropriate measurement system

• Identify the role of interobserver agreement in the measurement development process

• Determine best practices for graphing group performance data

• Identify special considerations for graphing individual data

PM – Pinpointing and Measurement Biagi

What is (or isn’t) Behavior?

PM – Pinpointing and Measurement Biagi

The Differentiator One of the biggest differences between Performance Management (PM) professionals and other similar

disciplines (I/O psychology, HR, traditional business consultants, etc.) is the emphasis on behavior. And not just ANY behavior, but behavior that adds ______________________ to an organization.

Before we start measuring, diagnosing issues, or developing interventions, we must understand what constitutes behavior in business, vs. other things that may be mistaken for behavior.

PM – Pinpointing and Measurement Biagi

Daniels and Bailey (2014)

Important Definitions • Behavior – any observable and

measurable activity of a living person • __________________’s (1965) “Dead

(Person’s) Test” applies!

• _______-added Behavior – behaviors needed by the business that is related in context to the organization’s output

• Performance – the combination of behaviors and what the behaviors produce (output or accomplishment) or how they otherwise impact the environment

PM – Pinpointing and Measurement Biagi

What Behavior is NOT

Daniels and Bailey (2014)

Commonly mistaken for behavior, the following do not meet the definitions we’ve discussed:

Broad terms like professionalism, communication, or supervising are not precise enough to be able to

accurately describe what someone is doing or measure how someone

is performing.

_____________ Lazy, disengaged, conscientious, or aware… none of these attitude

or personality-related terms describe anything observable in a

useful manner for us to collect data.

____________ States occur as a result of behavior, and fail the Dead

Person’s test. We must reinforce the behavior (putting on safety

glasses) that resulted in the state (wearing safety glasses), not the

state alone.

______________ Values are umbrella terms for

many different behaviors, and can be interpreted differently across performers. What does respect

look like? What about teamwork? Does everyone agree?

____________

PM – Pinpointing and Measurement Biagi

What about thoughts and feelings? They are behavior, but “_____________”

are not a focus of PM interventions.

PM – Pinpointing and Measurement Biagi

Which of the following is a behavior? Checking in:

A. Honesty

B. Leadership

C. Walking to one’s assigned work area

D. Remaining in one’s assigned work area

PM – Pinpointing and Measurement Biagi

Pinpointing Developing Operational Definitions in PM Practice

PM – Pinpointing and Measurement Biagi

Getting Precise When we pinpoint, we are objectively and precisely describing what a person is doing or has

produced.

The practice of pinpointing is equivalent to developing our ______________________ in ABA, and therefore all of the requirements of a good operational definition apply when developing out

a pinpoint.

PM – Pinpointing and Measurement Biagi

Behaviors and Outputs • The process of pinpointing is as follows:

1. Describe the results, outcomes, or accomplishments that one is looking to _________, then

2. Identify and describe the behaviors that reliably _________ the results, outputs, or accomplishments

• It is critical to note that we begin with the defining the outputs and results, then the behavior.

• This is a big difference between _____ and PM!

DiGennaro Reed, et al. (2018)

PM – Pinpointing and Measurement Biagi

DiGennaro Reed, et al. (2018)

Starting with the End in Mind • PM practitioners begin with pinpointing outputs and

business results to ensure that resource time is spent dedicated to the critical behaviors that produce the output, rather than an unrelated behavior

• A pinpointed __________________ should include: • The direction of change (to

increase/decrease/maintain) • The unit of measurement (dollars in sales,

number of clients)

• A pinpointed __________ should precisely describe what is left over after behavior has occurred

• Behavior is only valuable in its relationship to the outputs and business results they produce! (Gilbert, 1978)

PM – Pinpointing and Measurement Biagi

Marr (2015)

___________________ (KPIs) • The organizational-level data that help us to:

• Make informed decisions about business practices and learn what is __________ and what isn’t

• Communicate the state of the business to stakeholders and demonstrate compliance to regulatory bodies

• Guide employee behavior and _______

• Includes financial (revenue, profit), customer (satisfaction, market share), operational (efficiency, quality), and employee (“engagement”, retention) factors

• These are your long-term business results

PM – Pinpointing and Measurement Biagi

Work Outputs Binder (2012)

• After determining the KPIs that we’re most interested in, we need to identify the work outputs that create those business results

• Outputs often include: • Permanent products • Transactions • _____________ • Milestones • Relationships • Changes • Solutions • __________ who… • Teams that…

PM – Pinpointing and Measurement Biagi

How We Get There: Behavior

DiGennaro Reed, et al. (2018)

• Once we’ve identified the business results and outputs, we can describe the behaviors presumed to achieve those outcomes

• Behavioral descriptions should precisely describe what the organism does in objective, measurable terms that are not open for inference or _________________

• Behavior analysts are typically very good at this… if they keep their ______________________ on

PM – Pinpointing and Measurement BiagiDaniels and Bailey (2014)

PM – Pinpointing and Measurement Biagi

Behavior Output Business Result

________________ Number of serviced vehicles Increased revenue

Providing feedback to staff

Increased DTT trials run _______________

Meeting to discuss a process ________________ Increased

productivity

PM – Pinpointing and Measurement Biagi Characteristics of Good Pinpoints (Daniels and Bailey, 2014)

Can be counted or assessed in an objective manner

Measurable Can be seen with the human eye Observable

Multiple observers agree, think _______________________(IOA)

Reliable The performer MUST have a major influence over whatever is being measured

Under the _________’s Control

PM – Pinpointing and Measurement Biagi

Daniels and Bailey (2014)

Other Considerations • Behavior ___________ – multiple behaviors that

produce the same output • Example: Composing an email

• Did you type it? Voice-to-text? Does it matter?

• Behavior ____________ – multiple behaviors, performed in a sequence, that produce a single output

• Example: Parking a car • Pull into the space, press the brake, put

hand on the gear shift, grip, move until the P lights up, release the gear shift and brake…

• The level of analysis depends on how much information is required to ______________________________________

PM – Pinpointing and Measurement Biagi

What is the first step when pinpointing?

Checking in:

A. Describe the behavior

B. Describe the outputs and/or business results

C. Determine the measurement system

D. Determine the intervention

PM – Pinpointing and Measurement Biagi

Which of the following is a behavior? Checking in:

A. Delivering a training

B. A training presentation

C. People who can do a new task more efficiently

D. Increased productivity

PM – Pinpointing and Measurement Biagi

Which of the following is a valuable output?

Checking in:

A. Delivering a training

B. A training presentation

C. People who can do a new task more efficiently

D. Increased productivity

PM – Pinpointing and Measurement Biagi

Which of the following is a business result?

Checking in:

A. Delivering a training

B. A training presentation

C. People who can do a new task more efficiently

D. Increased productivity

PM – Pinpointing and Measurement Biagi

What factor determines how detailed a pinpoint needs to be?

Checking in:

A. The value of the pinpoint to the organization

B. The amount of detail necessary to produce the valuable behavior change

C. The leadership’s understanding of the pinpoint

D. The employee’s understanding of the pinpoint

PM – Pinpointing and Measurement Biagi

Meaningful Measurement

How do PM professionals collect data?

PM – Pinpointing and Measurement Biagi

Why Measure? When we don’t measure behaviors, outputs, and business results, we cannot:

Provide targeted ________________ that yields outcomes for the business,

Improve staff __________________,

Or tell if _______________ is improving, maintaining, or decreasing.

When we measure, we can identify effective reinforcers, communicate more clearly, react less emotionally, and make better decisions for the good of everyone.

PM – Pinpointing and Measurement Biagi

Measuring Behavior in PM DiGennaro Reed, et al. (2018)

____________ Measures

Built upon the data provided in the Fundamental Measures category, allows standardization across an organization.

• Percentage • Trials-to-Criterion

Time Sampling Measures

Useful when continuous measurement isn’t possible, but can vary on accuracy.

• Whole Interval Recording

• Partial Interval Recording

• ________________

Other Measures Behaviorally-anchored Rating Scales - Numeric scales tied to behavioral definitions

___________________– measuring characteristics of the output of behavior.

___________ Measures

Data collected through direct observation that immediately provides useful information.

• Frequency • Rate • Duration • Latency • Inter-response

Time

PM – Pinpointing and Measurement Biagi Measurement in Human Services Examples:

___________ Measures • Number of billable hours delivered _______ • Number of trials run per hour ________ • Length of each session ________ • Time taken to deliver the reinforcer after a

correct client response _________ • Time between each trial _______

PM – Pinpointing and Measurement Biagi

Measurement Examples: Behavioral Checklist • Provides a count of the

occurrence of behavior

• Can either be reported as a _______ (number of items observed) or as a percentage of items met

• Checklist items need to be carefully ______________

• Can be weighted or unweighted (items can be worth the same amount, or different amounts based on importance)

PM – Pinpointing and Measurement Biagi

Measurement Examples: Permanent Product Checklist • Provides a count of desired

_____________________ present in a permanent product

• Can also be reported as a count (number of items observed) or as a percentage of items met

• Checklist items need to be carefully pinpointed

• Can also be ___________________ (items can be worth the same amount, or different amounts based on importance)

PM – Pinpointing and Measurement Biagi

Measurement Examples: Behaviorally-Anchored Rating Scale (BARS)

PM – Pinpointing and Measurement BiagiMeasurement Examples: Performance Matrix

PM – Pinpointing and Measurement Biagi

Best Practices in Measurement • Use the _________________ measures (“hard

data”) whenever possible

• Lean on _____________ when judgement is required

• Don’t shy away from ________, as it is often impractical to observe each behavior or review every product

• Avoid Likert scales without anchoring whenever possible

• Try to gather at least ____ consistent data points

• Test the tools you develop for inter-observer agreement or inter-rater reliability ASAP

PM – Pinpointing and Measurement Biagi

Which of the following is NOT a fundamental measure of behavior?

Checking in:

A. Latency

B. Rate

C. BARs

D. IRT

PM – Pinpointing and Measurement Biagi

A construction foreman observes a worker for 5 minutes, and collects data on if the worker wears a hard hat for the entire observation period. This

is an example of what type of data collection?

Checking in:

A. Partial Interval Recording

B. Momentary Time Sampling

C. Duration

D. Whole Interval Recording

PM – Pinpointing and Measurement Biagi

You want to collect data on an experienced staff member submitting accurately

completed forms. What type of data would you most likely collect?

Checking in:

A. Permanent product quality assessment

B. Rate of submission of forms

C. Duration of filling out forms

D. Momentary time sample of whether any forms are outstanding

PM – Pinpointing and Measurement Biagi

When multiple observers use a measurement tool independently, and the

differences between their measurements are analyzed, we are checking for:

Checking in:

A. Social validity

B. Interobserver agreement

C. Consistency

D. Bias

PM – Pinpointing and Measurement Biagi

Performance Monitoring

An Essential Component of OBM

PM – Pinpointing and Measurement Biagi

Direct Observation of Staff Performance

Conducting direct observations of staff performance is a large part of the measurement and evaluation process!

This is known as “______________________”, and can be done either formally or informally (Reid, Parsons, and Green, 2012).

PM – Pinpointing and Measurement Biagi

Reid, Parsons and Green (2012)

Formal Monitoring • Structured performance measurement • Directly observing staff while collecting

specified information on performance • Structured based on objective

performance criteria directly related to client/business outcomes

• Ex.: Regulatory compliance, safety, skill acquisition, etc.

PM – Pinpointing and Measurement Biagi

Formal Monitoring: Misuse • When performance monitoring data are simply ____________

(an institutional pitfall) this becomes a process with no real purpose.

• Monitoring is NOT about demonstrating compliance with requirements of oversight bodies

• Collection of unused monitoring data can undercut healthy supervisory relationships

• Can promote cynicism about the use and effectiveness of potentially helpful managerial processes

• Can model “data collection” as a ________________

Adapted from Reid, Parsons and Green (2012)

PM – Pinpointing and Measurement Biagi

Without ________, _________, and ___________ observations, even carefully collected performance data will most likely provide an inaccurate and often conflicting

picture of staff performance.

Reid, Parsons and Green (2012)

PM – Pinpointing and Measurement Biagi

Formal Monitoring: When to Use Should occur most frequently when: 1. Staff are performing ____ skills

2. Feedback was recently provided to correct performance deficits

3. A supervisor has concerns about the _______ of a specific performance

4. Duties directly relate to assisting _________ attain outcomes, or engages in activities that interfere with completion of those duties

Reid, Parsons and Green (2012)

PM – Pinpointing and Measurement Biagi

Reid, Parsons and Green (2012)

Informal Monitoring • Requires no tools, can occur at any point

in the day • Consists of quick observations to

determine if performance is adequate • No ____________ • Should occur every day the supervisor is at

work!

PM – Pinpointing and Measurement Biagi

Reid, Parsons and Green (2012)

Informal Monitoring: Benefits • Increases opportunity for positive feedback

• Makes the supervisor visible to staff

• “Wow, he/she must be sincerely interested in my work! My work must be _____________!”

• Makes it easier for supervisee to ask questions on assignments or work tasks

• Supervisor is more aware of the _________ in the work area; establishes ________ with staff.

PM – Pinpointing and Measurement Biagi

Reid, Parsons and Green (2012)

________ to Monitoring • Defined: staff changing their work

behavior when a supervisor is present

• Results in an inaccurate representation of staff performance

• Common example: Police car on highway, under billboard, empty; other car with lights

• Steps should be taken to reduce the impact of reactivity

PM – Pinpointing and Measurement Biagi

Reducing Reactivity

1. Monitor __________! Monitoring should be part of the company’s culture, not

something novel With repeated observations, reactivity will fade

Added benefit: Transfer of ____________ from presence of supervisor to the environment itself In areas where police are regularly present, speeding does not occur even when

police are not present. Supervisor presence becomes an ____ (helpful feedback, assistance)

rather than an SdP, or EOfor Sr-

Reid, Parsons and Green (2012)

PM – Pinpointing and Measurement Biagi

Reducing Reactivity

2. Start monitoring _________ the supervisor enters the workplace

Don’t allow time for staff members to change what they are doing

Especially handy for tasks that involve some __________ (having specific materials available)

________ can also be a useful data collection strategy

Reid, Parsons and Green (2012)

PM – Pinpointing and Measurement Biagi

Reducing Reactivity 3. Conduct unexpected follow-up observations

_________ after the initial observations Supervisees should still be behaving as they were during

the initial observation Did they change as soon as the supervisor left?

Reid, Parsons and Green (2012)

PM – Pinpointing and Measurement Biagi

Reducing Reactivity

4. Use an ________________ schedule for formal observations

Inform supervisee(s) that monitoring will be conducted, just do not provide exact ___________

Irregular schedules are vital (to prevent time-based non- discriminated avoidance)

Reid, Parsons and Green (2012)

PM – Pinpointing and Measurement Biagi

• Supervisors may be tempted to use covert monitoring (to combat reactivity, for example)

• It can be effective, but… • Rarely remains covert • __________ must be given eventually! • Attempting to hide a monitoring form will not work

100% • Often reduces supervisee ________ for supervisor • People do not like the sense of being ____ upon • Implies a lack of trust, honesty, and professionalism • Covert monitoring should ____ be part of the regular

supervision process!

Reid, Parsons and Green (2012)

Covert Monitoring

PM – Pinpointing and Measurement Biagi

Covert Monitoring: When to Use -- The Only Exception If something __________________________ is happening:

Client abuse, neglect, sleeping on the job, stealing from agency or clients, consuming illegal substances or alcohol at work…

The aim is not to provide corrective or supportive feedback. Covert monitoring may result in firm disciplinary and/or legal action.

This is ______________, not supervision

Reid, Parsons and Green (2012)

PM – Pinpointing and Measurement Biagi

Overt Monitoring • Done properly, overt monitoring is the

most effective and __________ professional practice

• One goal is to catch as much good behavior as possible. Reinforcement works, by definition!

• Once monitoring becomes __________, it produces benefits for everyone: consumers, supervisees, and supervisors

• The culture must promote the idea that ____________ is subject to being openly monitored

PM – Pinpointing and Measurement Biagi

Overt Monitoring as a Team Process • Overt monitoring can be initially

_______, but if it remains so over time, it is being done incorrectly

• This is not about violating privacy; it is about:

• Protecting client rights and safety • Working together as a team • Helping each other • Promoting open communication • Helping to insure standards of professional

conduct in therapeutic environments

PM – Pinpointing and Measurement Biagi

Increasing Acceptability Staff typically dislike having their performance monitored, especially formal monitoring

• Being closely watched can produce anxiety – and sometimes even evoke _________ errors!

• Many people have experience with poor feedback and/or monitoring being used for punishment • A signal of __________ managerial actions to come – a warning

stimulus, triggering escape or avoidance behavior • Its onset may function as a form of conditioned Sp+, and may thus trigger

forms of _________________ (resistance)

Adapted from Reid, Parsons and Green (2012)

PM – Pinpointing and Measurement Biagi

Monitoring: A Change in Function

Seek to identify the possible __________________ involved in a supervisee’s reaction to overt monitoring. This analysis will help you create successful monitoring processes, and an environment where consistent monitoring signals a higher density of reinforcement for competent performance – and supervisory assistance in obtaining that higher density schedule.

PM – Pinpointing and Measurement Biagi

Guidelines for Making Formal Monitoring Acceptable To Staff 1. Inform staff prior to monitoring what will be monitored and _____

2. Upon entering the workplace, ______ all staff present (before beginning formal monitoring)

3. Discontinue monitoring when a potentially _______ or embarrassing situation occurs

4. Provide feedback to staff ______ after monitoring

5. _________________ staff upon completion of monitoring prior to departing the workplace

Reid, Parsons and Green (2012)

PM – Pinpointing and Measurement Biagi

A manager schedules a specific time to supervise, and collects data during the

session. This is what type of monitoring?

Checking in:

A. Formal

B. Informal

PM – Pinpointing and Measurement Biagi

A manager walks through the work area, and stops to provide support to staff who are

handling a tough situation. This is what type of monitoring?

Checking in:

A. Formal

B. Informal

PM – Pinpointing and Measurement Biagi

How can we reduce reactivity to performance monitoring?

Checking in:

A. Schedule random observations

B. Start collecting data immediately after entering the work area

C. Increase the frequency of monitoring

D. Leave briefly and return to conduct a session very shortly after the initial observation

E. All of these

PM – Pinpointing and Measurement Biagi

Graphing Special Considerations for Performance Management

PM – Pinpointing and Measurement Biagi

Why Graph? Graphing is a critical component of the performance management process - it allows us to visually analyze the data to determine trends, correlations, and the extent of behavior change needed to meet the goals of

the organization.

Graphs have been proven especially effective when ____________________, both individually and to groups of performers. There are, however, special considerations for how to display group and individual

data to ensure we have a positive impact on the performers.

PM – Pinpointing and Measurement BiagiBasic Parts of a Graph

PM – Pinpointing and Measurement Biagi

• Bar graphs • Cumulative records • Scatterplots • Pie Charts • Each have their use, and are more

often seen in _______ than __________

Cooper, Heron, and Heward (2013)

Other Types of Graphic Displays in PM

PM – Pinpointing and Measurement Biagi

General guideline: Individual performance graphs should NOT be ________________, unless the performer chooses to _______

them on their own.

PM – Pinpointing and Measurement Biagi Special Considerations for Group Graphs (Daniels and Bailey, 2014)

In most cases, a line graph is going to be your best option for sharing changes in performance over time. Avoid graphing too many data sets on the same graph.

Easy to Understand

The most effective graphs are large enough for performers to read from a distance.

Easy to See Make your staff WANT to look at the graphs! Ensure that it doesn’t blend in with your other memos. Think about it like you’re trying to advertise your data.

Attractive, Neat, and Colorful

Do not hide data in a back office or conference rooms. People should be able to see them during the course of a regular day without making a special trip.

Accessible and Convenient

Move the graph around, and change its appearance regularly. This will increase the odds that folks will review the data, at least as often as they are varied.

Varied

PM – Pinpointing and Measurement Biagi

Special Considerations when Graphing Staff Behavior • Include goal lines and reinforce when goals are

met!

• Graphs don’t change performance – _____________________ do!

• Ensure that you are graphing, and reinforcing, controllable behavior or outputs, not system outputs

• They should have contributed to the data they’re held accountable for through their behavior

• Employees may resist measurement and graphs at first due to a history of __________, so we must re-_____

PM – Pinpointing and Measurement Biagi

True or False – Employees often have a positive history around data collection and measurement of their performance.

Checking in:

A. True

B. False

PM – Pinpointing and Measurement Biagi

Important characteristics of group PM graphs include that they are:

Checking in:

A. Large enough to see from across the room

B. Colorful and visually appealing

C. Changing in location and appearance often

D. Shared in regularly occupied spaces

E. All of these

PM – Pinpointing and Measurement Biagi

In Closing…

PM – Pinpointing and Measurement Biagi

Thank you For your precious time and your attention.