Prep Guide

145

Chapter 7

Pinpointing and Measuring Employee Behavior

Florence D. DiGennaro Reed Matthew D. Novak

Tyler G. Erath Denys Brand

Amy J. Henley

Have you ever had a job or position where you felt unsure about how to perform your responsibilities, but were too uncomfortable to ask for assistance? Did you receive infrequent performance feedback that lacked transparency about how your supervisor measured your behavior and completed observations? Did your performance appraisals or supervisory observations measure irrelevant results or outcomes over which you had no control? These all-too-common scenarios contribute to decreased job satisfaction (Ray, Rizzacasa, & Levanon, 2013) and employee turnover (Ongori, 2007) or may lead to litigation (Komaki & Minnich, 2008), all of which may be resolved by consistently delivering performance feedback to employees about relevant behaviors and providing support when employees fail to meet expectations. These latter endeavors can only be accomplished if organizations incorporate systems that entail pinpointing and measuring employee behavior.

The value of pinpointing and measuring employee behavior includes facilitating supervisor and employee understanding of employee performance, the delivery of feedback, the delivery of rewards or reinforcers to employees for meeting or surpassing expectations, and the implementation of change efforts to address performance issues (Daniels & Daniels, 2006). Moreover, measurement allows organizational behavior management (OBM) practitioners to evaluate the effects of their

change efforts. Because pinpointing and measuring employee behavior is critical to both employee and organizational success, the purpose of this chapter is to define and provide examples of pinpointing and measuring employee behavior and offer solutions for resolving barriers to measurement in organizations. We also describe features of a good measurement system and how to develop a data collection system. The chapter concludes with an applied example from our organizational consultation experience. Although pinpointing and measuring employee behavior require effort and resources, the return on organizational investment makes completing these activities worthwhile.

Pinpointing

Pinpointing involves two activities in the

following sequence: (1) describing critical results or outcomes in precise terms; and (2) identifying observable and measurable employee behaviors that reliably produce those results (Daniels & Daniels, 2006; Rodriguez, Sundberg, & Biagi, 2016). Because results are products of behavior (Daniels & Daniels, 2006), identifying critical results linked to organizational success is an essential first step in pinpointing. It would be unwise to identify and measure employee behavior that fails to produce desired organizational results. Imagine a situation in which a human service organization must

146 DiGennaro Reed et al.

address staff turnover to remain a viable company, but fails to consider this particular result before designing a measurement system for employee behavior. A myriad of employee behaviors unrelated to desired outcomes could be pinpointed, including correct implementation of treatment plans, offering choices to consumers, clocking in on time, and many others. If an OBM practitioner overlooks the important task of identifying critical results, precious resources could be expended on pinpointing and measuring behaviors that do not produce desired and sometimes necessary outcomes.

A precise results pinpoint will contain three features, including the desired direction of change, the units of measurement that will be used, and a detailed description of the desired outcome (Daniels & Daniels, 2006). First, results pinpoints should specify whether the goal is to increase or decrease the outcome being measured (e.g., employee behavior, work units, sales). The second feature specifies the units of measurement, which can range from dollars (e.g., sales, profits), frequency (e.g., number of customers, number of items sold), pounds (e.g., product weight), and many other units. Finally, results pinpoints should include a detailed description of the desired outcome.

A precise behavior pinpoint entails identifying and defining behaviors presumed to yield those results. The aim is to minimize subjective judgment or guessing in favor of observing and measuring targeted employee behaviors according to a set of rules. Although defining and measuring behavior may appear to be a simple or intuitive task, it entails a complex set of decisions and activities if done effectively. Thus, the majority of this chapter is devoted to considerations for defining and measuring employee behavior.

Several considerations about pinpointing warrant mention. First, when developing pinpoints as part of any measurement system, the OBM practitioner must ensure the measured behavior can be performed and/or is under the control of the employee (Daniels & Daniels, 2006). It would be unfair to hold employees

accountable for organizational results or behavior over which they have no control. Next, practitioners should ensure measured behavior will produce desired results; overlooking this important aspect of pinpointing could waste precious resources and will lead to unsuccessful outcomes. OBM practitioners must also recognize that much of what is performed and accomplished in organizations involves behavior chains or sequences of differing and sometimes complex behaviors, not discrete behaviors. Finally, practitioners can also measure the results of employee behavior much like they measure organizational results. Thus, they must determine when to measure behavior, results, or both, all of which will impact organizational results. Rodriguez et al. (2016) provide helpful guidelines to aid the practitioner in making this important decision, which are detailed in Table 1.

Measuring Behavior

After pinpointing results and behavior, an

OBM practitioner must then identify a system to measure behavior. Measurement is the process of using a standard set of rules to assign qualitative or quantitative labels to events or behaviors; it includes information about what is being measured, the technical skills required of the person measuring behavior, the quality of the measurement tools, and how the measures are used (Bloom, Fischer, & Orme, 2003; Johnston & Pennypacker, 2009). Put simply, measurement provides practitioners with a standardized, objective method through which they may describe and compare behavior.

Why measure behavior? Measurement provides OBM practitioners with a means of comparing behavior and evaluating change. Because measurement systems rely on standardized, objective values to describe a behavior, measurement allows practitioners to compare behavior across different settings, individuals, groups, and over the course of time. The comparison of behavior over time is perhaps the most important feature for OBM

Pinpointing and Measuring Employee Behavior 147

practitioners; these comparisons allow practitioners to evaluate their interventions – in particular, when the comparisons span periods, such as when an intervention is instated or removed. Thus, OBM practitioners measure behavior to detect change, and, if a change has occurred, to determine how quickly and to what extent. Without a carefully designed measurement system, practitioners are unable to evaluate the efficacy of their interventions, and thus susceptible to discontinuing an effective intervention or continuing an ineffective intervention. Behavioral definitions. To ensure proper measurement, practitioners should first establish a behavioral definition. Behavioral definitions are specific statements that describe the target behavior and include descriptions of the behavior to be recorded as well as behaviors that should not be included. Specifically, behavioral definitions should meet three qualities, as described by Hawkins and Dobes (1977): (a)

objective, observable, and free of any inferential terms; (b) clear and unambiguous; and (c) complete, such that they include descriptions of responses to be included and excluded. Many behaviors targeted for intervention in an organization may initially appear obvious and lead to inadequate behavioral definitions. For instance, many readers understand what is meant by the statement “the employee arrived to work on time” without any further explanation. When being measured as part of an intervention, however, this definition is insufficient and leaves numerous questions unanswered. For example, would an employee be considered “on time” if he enter the building at the start of a shift, or only if he clock-ins or starts working by that time? This definition may also lead to employees clocking in well before the start of their shift and lead to costs that exceed the amount organizational leaders budgeted for wages. Although all of the aforementioned criteria should be considered and incorporated, this example highlights the importance of a

Measure behavior(s) when… Measure outcomes when…

Measure behaviors(s) and outcomes when…

Employee is learning a new skill and you want to provide frequent, immediate feedback

Behaviors and outcomes are highly correlated

Outcomes are important, but they need to be reached in a safe and an ethical way

Employee needs a great deal of feedback to continue performing desired behaviors

OBM practitioner is confident the employee is engaging in the right behaviors to achieve outcomes

OBM practitioner needs examples of behavior to provide more meaningful feedback

OBM practitioner needs to be sure employees are engaging in the right behaviors to achieve results

OBM practitioner only needs a quick snapshot of how well employees are performing

Outcomes of the behavior are long delayed

*Adapted and reprinted with permission from Rodriguez et al. (2016)

Table 1. Guidelines for Measuring Behavior and/or Outcomes

148 DiGennaro Reed et al.

developing a complete definition for every behavior assessed. Practitioners should also individualize behavioral definitions to the specific qualities and needs of each context. For example, an appropriate customer greeting for a salesperson at a car dealership will likely not be a suitable consumer greeting for a behavior technician at a residential facility. Behavioral dimensions. Measurements of behavior, much like measurements of physical objects, can take form in different dimensions. Meaningful information is not gathered through measurement alone; practitioners must identify a dimension to measure that adequately reflects the behavior of study. To use a physical object as an example, there are several different ways to measure an orange, such as weight, diameter, color, or sugar content. Any measure of these dimensions is arbitrary unless we have an understanding of the purpose for measurement. Accordingly, behavioral definitions must reflect a dimensional quality that is of interest to the OBM practitioner (Cooper, Heron, & Heward, 2007). Because behavior is an active process, Johnston and Pennypacker (2009) explain that measurement must fall into one of three dimensions: (a) repeatability (i.e., how often the behavior occurs); (b) temporal extent (i.e., duration of the behavior); and (c) temporal locus (i.e., latency, or when behavior occurs with respect to other events). OBM practitioners may measure a single or multiple dimension(s) of a behavior. Fundamental and derived measures of behavior. Table 2 summarizes varied ways to measure behavior, considerations for their use, and examples in the literature. Frequency, rate, duration, latency, and inter-response time are examples of fundamental measures—or data collected through direct observation that can immediately provide useful information. The repeatability dimension incorporates frequency/rate, or the number of occurrences in a given period of time. For example, Thurkow, Bailey, and Stamper (2000) measured the number of surveys completed per hour by

employees at a research firm. Temporal extent includes duration, or the length of time a behavior occurs. Amigo, Smith, and Ludwig (2008) captured this measure by evaluating the effects of task clarification and feedback on the amount of time required for servers to bus a table at a restaurant. Temporal locus entails latency (i.e., length of time between an initial stimulus and its corresponding behavior) and inter-response time (i.e., length of time between two occurrences of the same behavior). Amigo et al. also measured latency by recording the time required by servers to arrive at a table once patrons departed.

Derived measures are those that build upon information obtained by fundamental measures and include percentage and trials to criterion. For example, Williams and Gallinat (2011) examined the effects of different forms of video modeling on paraprofessionals’ teaching skills. The researchers evaluated performance by measuring the percentage of teaching steps correctly completed. They also measured training efficiency by comparing trials to criterion, or how long it took participants in each condition to implement the teaching procedure with 100% accuracy for two consecutive sessions. Derived measures, such as the percentage of employees arriving to work on time, may be useful at the organizational level because they allow OBM practitioners to evaluate a standard metric across an entire organization.

Time sampling measures. A continuous measurement system—similar to those already described—involves an uninterrupted period of time during which an observer collects and records data on all instances of the behavior(s) of interest (Cooper et al., 2007). Referring to a previous example already described, Amigo et al. (2008) directly observed specific behavior by measuring how long it took servers to arrive at and then bus a table after patrons departed. Continuous observation methods are useful because they provide practitioners with an exact measure of how often the behavior of interest occurs, but this type of observation may be

Pinpointing and Measuring Employee Behavior 149

Measure Use Example

Fu nd

am en

ta l M

ea su

re s

Frequency/Rate: number of occurrences in a given time frame

Should be used for behaviors with a consistent duration and a clear beginning and end

Number of surveys completed per hour by employees at a research firm (Thurkow et al., 2008)

Duration: length of time the behavior occurs

Expressed as duration per observation period or duration per bout

Amount of time required for servers to bus a table at a restaurant (Amigo et al., 2008)

Latency: length of time between an initial stimulus and its corresponding behavior

Used when the practitioner is interested in how much time passes between an opportunity to respond and when the behavior occurs

Time it takes for servers to arrive at a table once patrons have left (Amigo et al., 2008)

Inter-response Time (IRT): length of time between two occurrences of the same behavior

Used when the practitioner is interested in time between occurrences of a behavior

Amount of time between two successive scans of an item at a cash register.

D er

iv ed

M ea

su re

s

Percentage: derived measure that is expressed as a proportion

The same dimension is used for both parameters of the proportion (e.g., steps completed divided by total number of steps)

Percentage of task list items completed by servers when greeting customers at a restaurant (Reetz et al., 2016)

Trials to Criterion: number of opportunities required to achieve a desired level of performance

Often used to compare relative efficacy of various programs and/or training methods

Number of trials required for staff at a human service setting to attain two consecutive training sessions with 100% performance accuracy in implementing a skill acquisition program (Williams & Gallinat, 2011)

Table 2. Measures of Behavior, Considerations for Use, and Examples

150 DiGennaro Reed et al.

T im

e Sa

m pl

in g

M ea

su re

s

Time Sampling Procedures: measuring behavior throughout or at certain points in pre- determined intervals (types below)

Whole-Interval Recording (WIR): measurement indicating that behavior occurred through an entire interval

May underestimate actual performance (Alvero et al., 2008)

Safe body posture when lifting objects (Alvero et al., 2008; Taylor et al., 2012)

Partial-Interval Recording (PIR): measurement indicating that behavior occurred at any point in an interval

May overestimate actual behavior (Alvero et al., 2008)

Safe body posture when lifting objects (Alvero et al., 2008)

Momentary Time Sampling (MTS): measurement indicating that behavior occurred at the end of an interval

May over- or under-estimate actual behavior (Alvero et al., 2011) Longer MTS intervals produce less error than relatively shorter PIR and WIR intervals (Alvero et al., 2011)

Safe body posture when lifting objects (Alvero et al., 2008; Alvero et al., 2011; Taylor et al., 2012) Proportion of road construction workers engaged in work-related behavior (Dixon et al., 2014)

O th

er M

ea su

re s

Permanent Product: measurement that takes place after a behavior has occurred by observing the behavior’s effect(s) on the environment

Efficient method of data collection, if behavior does not need to be observed. Requirements to use: (a) occurrences must produce the same product and (b) the product can only be produced by the target behavior.

Completion of data sheets by staff (Gil & Carter, 2016; Szabo et al., 2012; Wine et al., 2014) Service documentation notes (Williams et al., 2003)

Behaviorally Anchored Rating Scales (BARS): performance levels are given a numeric ranking

Number scales should be tied to behavioral definitions.

Team performance in health care tasks (Wright et al., 2009)

Table 2 (continued).

Pinpointing and Measuring Employee Behavior 151

difficult for OBM practitioners with limited time and/or personnel resources.

Three methods of time sampling—whole- interval recording (WIR), partial-interval recording (PIR), and momentary time sampling (MTS)—may be useful options when continuous direct observation is not possible. Time sampling procedures entail dividing an observation period into shorter intervals and recording the occurrence or non-occurrence of a target behavior. For WIR, the OBM practitioner records the occurrence of behavior when it occurs for the entire duration of the interval. Partial-interval recording requires the practitioner to record the occurrence of behavior if it was emitted at any point during the interval. Finally, for MTS the practitioner records behavior occurrence if it takes place at the end of an interval. Time sampling procedures provide practitioners with the opportunity to simultaneously observe a single behavior emitted by several individuals or several behaviors from a single individual, and ultimately produce a representative estimate of how often a behavior is occurring. For example, Dixon, Whiting, Rowsey, Gunnarsson, and Enoch (2014) used MTS to measure the proportion of roadside construction workers who were engaged in work-related motor behavior. Dixon et al. also observed the same construction zone and recorded the total number of workers in the construction zone and the number of those workers who were engaged in a work-related behavior at the end of a 1-min interval for a 16-min observation period.

OBM practitioners should be aware of several considerations for using time sampling measures. Of note, the results of multiple studies have suggested that MTS can either overestimate or underestimate behavior (relative to continuous recording measures), particularly when using intervals longer than 2 min (e.g., Gunter, Venn, Patrick, Miller, & Kelly, 2003). However, Alvero, Rappaport, and Taylor (2011) compared different interval durations in an organizational setting and found minimal estimation errors when using intervals up to 5 min to measure safe working postures. Whole-

and partial-interval recording methods can also produce inaccurate estimates of behavior; whereas PIR may overestimate behavior, WIR underestimates behavior (e.g., Alvero, Struss, & Rappaport, 2008). A final consideration concerns the particular behavior being recorded. It may be inappropriate to use WIR or MTS when measuring behaviors that occur in short durations and/or low frequencies. Similarly, PIR may be most useful for behaviors that occur at a relatively low rate.

Measurement through permanent products. Permanent product recording (i.e., results measures) involves measuring a behavior after it has occurred by observing its effect on the environment at a later point in time (Cooper et al., 2007). For example, Thurkow and colleagues (2000) measured the effects of monetary incentives on the productivity of telephone interviewers by counting the number of surveys that each employee completed. The authors did not observe the employees completing the surveys in real time. One advantage to permanent product recording is that it does not require OBM practitioners to observe the behavior while it occurs, so practitioners are free to do other tasks. Additionally, permanent product recording limits potential reactivity effects (e.g., employee behavior changes due to the presence of an observer) and allows for measurement of behaviors that occur at inconvenient times or in places. For permanent product recording to be appropriate, the behavior of interest must be mutually exclusive with the permanent product being measured. That is, each instance of the target behavior must produce the same permanent product, and the permanent product can only be produced by the target behavior (Cooper et al., 2007). One drawback of permanent product recording entails not knowing which behavior produced the results. Behaviorally anchored rating scales. Behaviorally anchored rating scales (BARS) consist of numeric scales tied to behavioral definitions that act as anchors to define levels of

152 DiGennaro Reed et al.

performance within a particular performance dimension (Smith & Kendall, 1963). For example, an employer might be interested in rating the skill level of employees in the performance dimension of customer service. The employer could develop a five-point rating scale (1 = no customer service skills, 2 = some customer service skills, 3 = adequate customer service skills, 4 = good customer service skills, 5 = excellent customer service skills), accompanied with a behavioral definition of the behavior required for performance to be given a particular rating on the scale. That is, a rating of 1 (indicating no customer service skills) could be defined as, “employee does not make eye contact with customers, employee does not greet customers, employee does not ask customers if they need assistance.” A rating of 5 (indicating excellent customer service skills) could be defined as, “employee always makes eye contact with customers, employee always greets customers while smiling and with a positive tone of voice, and employee always ask customers if they need assistance.”

One of the benefits of using BARS as a system of measurement is that scales are anchored by clear descriptions of behavior (often accompanied by examples), which should result in increased data accuracy and reliability, and decreases in rater biases and subjectivity (Martin-Raugh, Tannenbaum, Tocci, & Reese, 2016). Moreover, reductions in rater biases and subjectivity should increase levels of data accuracy and consistency due to the standardized nature in which data are collected across raters (Tziner, Joanis, & Murphy, 2000). Research (e.g., Blum, Boulet, Cooper, & Muret- Wagstaff, 2014) has shown that BARS produce results that are both reliable (i.e., the ability to produce consistent results across repeated measurements), and valid (i.e., the ability for the measurement to accurately capture what it claims to measure). Behaviorally anchored rating scales are also helpful when the practitioner is interested in measuring how well a task or skill is performed, not just whether or how often it is performed. Conversely, a limitation of BARS as a system of measurement is that it can be time-consuming and expensive

to create (Kell et al., 2017), and often requires subject matter experts to provide information to ensure that the anchors accurately capture the behavior that it intended to be measured (Martin-Raugh et al., 2016).

Resolving Barriers to Measurement in

Organizations

From our experiences, an OBM practitioner may encounter two significant barriers when implementing a behavioral measurement system, including obstacles related to administration and staff and deficits related to time and resources. Implementation of a novel measurement system often disrupts the existing conditions (i.e., status quo) within an organization, allowing for differing types of resistance to materialize across various levels of administration and staff. The OBM practitioner may experience resistance to the measurement system at the employee level, where performance measurement is often perceived as an organizational tool whose singular objective is to correct or enhance performance. Thus, for many of these employees, measurement of workplace performance may function as a negative reinforcer or a punisher (Daniels, 2016) due to repeated pairings with negative implications (e.g., discipline, termination). The practitioner may also experience opposition at the administrative level, where it is commonplace for management to take a reactive approach to measurement, only implementing a measurement system when a deficit emerges within the organizational environment. Thus, the measurement system may function as a negative reinforcer for administration, who sees the measurement system as an organizational tool whose singular objective is to eliminate a performance deficit.

Similarly, implementation of a behavioral measurement system may also lead to undesirable modifications in an employee’s roles, responsibilities, and task priorities. When a new measurement system is being designed, the task of collecting measurement data is frequently assigned to an employee who, in

Pinpointing and Measuring Employee Behavior 153

addition to his typical responsibilities, must now expend both time and energy on data collection. Similarly, the organization must also utilize resources to create data collection tools, assess and pilot measurement materials, and provide training on effective implementation of the data collection system.

Thus, with the goal of increasing buy-in for the measurement system amongst all levels of staff within the organization, the OBM practitioner must be tactful in the mechanisms used to resolve each of these barriers. Two promising techniques to overcome organizational reluctance to employing a measurement system include: (1) increasing the frequency of positive reinforcement given for desirable workplace behavior; and (2) pairing positive reinforcement with measurements already being obtained in the organizational environment (Daniels, 2016). Additionally, it is essential for the OBM practitioner implementing a novel measurement system to: (1) receive support for the measurement system from upper and mid-level administration; (2) clearly articulate to administration, managers, and employees the goals of the measurement system; (3) thoughtfully delineate the roles and responsibilities for each employee within the measurement system; and (4) pinpoint and select behaviors that are central to achieving the desired results and positive behavior change.

Characteristics of an Effective

Measurement System It is beyond the scope of this chapter to

provide a thorough description of all of the necessary features for a behavior measurement system to be successful. Thus, the goal of this section is to outline critical characteristics the OBM practitioner must target to effectively implement a measurement system within organizational settings.

For a measurement system to be most effective, it must establish that the data collected possess validity, accuracy, and reliability. Authentication of each of these criterion within the measurement system is of great importance

to the OBM practitioner, as it ensures the opportunity for the practitioner to better make informed, data-based decisions. Validity refers to the degree to which the measurement tool measures what it claims to be measuring. Accuracy refers to the extent to which the observed data value matches the true (i.e., actual) value. Reliability refers to the extent to which under the same environmental conditions, the measurement will yield the same result (Johnston & Pennypacker, 2009; Cooper et al., 2007).

More broadly, the measurement system should ideally: (1) use a direct form of measurement; (2) measure an appropriate dimension of behavior; and (3) use a continuous form of measurement, when possible (Cooper et al., 2007). Direct measurement refers to the degree of correspondence between the behavior targeted by the measurement system and the behavior that is being measured (Johnston & Pennypacker, 2009). When pinpointing behavior in a measurement system, direct measurement dictates the behavior in question, at a minimum, be measurable, observable, and reliable (Daniels, 2016). Relatedly, the measurement system must also assess the correct dimension of behavior. For instance, if a manager was implementing a measurement system to increase on-task conversations amongst employees working as a group, measuring the intensity (i.e., magnitude) of the conversation would not be an effective dimension of behavior to measure. Instead, the manager should measure the behavior either by the number of on-task group conversations (i.e., frequency) or by the length of on-task group conversations (i.e., duration). Lastly, continuous measurement refers to the measurement of all instances of a response class during an observation period (Cooper et al., 2007).

Individuals implementing behavioral measurement systems must also carefully select the type, format, and frequency with which measurements will be taken. To accomplish this, an OBM practitioner must decide whether he will use leading measures, lagging measures, or some combination. Leading measures refer to measures collected by the organization that use

154 DiGennaro Reed et al.

current behaviors or activities as the main source of measurement data (Rodriguez et al., 2016; Hinze, Thurman, & Wehle, 2012). Leading measures are measures that have the capability of being collected and analyzed quickly, providing the implementer with the opportunity to make data-based modifications to the measurement system as needed. Lagging measures on the other hand, refer to measures collected by the organization that use past behaviors or activities as the main source of measurement data (Rodriguez et al., 2016; Hinze et al., 2012). For example, quarterly performance data collected by the organization would fall into the category of a lagging measure because the data cannot be used to make real- time, data-based decisions or to enact swift changes within the measurement system. From an organizational standpoint, it is essential that the measurement system use leading measures whenever possible. However, this recommendation does not mean a measurement system should not take into account lagging measures; lagging measures should be used as a supplementary, yet vital reference tool in addition to the leading measures being collected.

Development of a Data Collection

System

For a measurement system to be maximally effective, the OBM practitioner must identify and answer a requisite set of topics and questions that are fundamental to successful data collection. These topics include logistics and coordination; training; and evaluation and dissemination.

In regard to logistics and coordination, the OBM practitioner must, at a minimum, identify some questions related to who, what, and when. Who refers to several questions within the data collection system. Of note, three particular questions must be addressed and include: (1) who will be collecting the data; (2) who will be analyzing the data; and (3) who will be disseminating the data? What refers to both the type of data being collected and the type of data collection system used. In regard to the type of

data being collected (e.g., time sampling, permanent product, event recording), the OBM practitioner would benefit from choosing a form of measurement that maximizes resources across all individuals and all levels of the organization (see Table 2). Relatedly, the OBM practitioner must also address whether the system will use a continuous or discontinuous form of measurement. When refers to the time and setting of the data collection procedure. To address this question, the OBM practitioner must carefully ensure the data collected are an accurate representation of the behavior as it naturally occurs in the environment (Cooper et al., 2007).

Training is also an integral component to maximizing the efficacy of a data collection procedure. When measuring and recording behavior within organizational settings, much of the data will be collected by human observers, enabling human error to enter and be an intrinsic component of the data collection procedure. Thus, providing evidence-based training for observers using both performance and competency-based components is of the utmost importance. One type of evidence-based procedure that includes both performance and competency based measures is behavioral skills training (Miltenberger et al., 2004; Parsons, Rollyson, & Reid, 2012). Behavioral skills training includes instructions, modeling or demonstration, rehearsal, and feedback. With respect to its application to data collection training, the OBM practitioner can use behavioral skills training to provide instructions on the data collection procedure, model exemplar observer performance, allow the observer-in-training to practice collecting data, and provide the observer-in-training with feedback on his or her performance.

Lastly, when developing a data collection system, the OBM practitioner must also address how the data will be interpreted, as well as the form in which the data are used. Therefore, the first question that must be addressed is whether the measurement system will present raw or transformed data. Raw data refers to data that are collected and presented as they are recorded.

Pinpointing and Measuring Employee Behavior 155

For example, a data collection sheet using frequency recording would show all occurrences of the behavior for a given session. Transformed data, on the other hand, refers to data that have been manipulated in any manner. For instance, plotting a data point for the frequency of a behavior or creating a histogram would fall into the category of transformed data. If the implementer of the measurement system decides that data should be transformed in some way, shape, or form, then careful consideration must be taken to ensure that: (1) the data that are lost are redundant with the data that are kept; (2) no data are purposefully discarded; and (3) the transformed data do not create an artifact that can be used to influence interpretation (Johnston & Pennypacker, 2009).

Applied Example

This chapter concludes with an example

from our consultation practice to illustrate how OBM practitioners might pinpoint and measure employee behavior. Our team has a consultation contract with a non-profit agency that delivers educational and residential services to children and adults with intellectual and developmental disabilities. The contract includes a myriad of

responsibilities, including overseeing new staff training, tracking monthly and annual quality assurance metrics, coordinating a monetary incentive program, collaborating with the human resources department, and other tasks. In this example, we highlight several of these responsibilities to demonstrate the utility of pinpointing and measuring employee behavior and results for making organizational decisions about change efforts.

Our team tracks 20 quality assurance measures for the residential department that span home stability, home finance, and home key practice and quality metrics. The desired result is for 100% of homes to meet performance expectations for at least 80% of the measures and to consistently maintain this outcome over time. Table 3 summarizes six measures linked to home stability. The table details the measure title, how it is measured, frequency of measurement, thresholds for success, the software containing the data, and the department that provides the data. We receive the data from relevant departments and utilize an individualized software program to create performance scorecards, which contain quarterly data using a stoplight color system (green = performance meets or surpasses

Table 3. Crosswalk of Quality Assurance Measures

156 DiGennaro Reed et al.

threshold; red = performance falls substantially below threshold; yellow = performance does not meet threshold, but is not in the critical red zone). Table 4 contains a sample scorecard for six of the 20 measures for a single residential program.

Note that raw and transformed lagging measures of behavior are captured in Tables 3 and 4, including percentage, frequency, and dichotomous yes/no ratings. The full 20 measures also include BARS scores and rely on permanent product recording. The quality assurance measures do not contain direct measures of employee behavior, which are summarized in a different manner and are not described in detail in this example.

We distribute the performance scorecards to relevant staff (i.e., regional coaches and their supervisors) at a monthly quality assurance meeting. The team reviews the data, celebrates improvements, discusses concerning or alarming trends, and determines how to resolve patterns requiring correction (e.g., performance diagnostics, intervention). On a quarterly basis, we aggregate the data by region and distribute our findings to this same team and the agency’s board of directors. Of particular note, we rely on positive reinforcement and a problem- solving approach when sharing data to enhance buy-in and minimize resistance to behavior measurement.

Figure 1 proposes a model we believe captures the relationships among several of the measures based on two years of data collection, survey results, and performance diagnostics (i.e., Performance Diagnostic Checklist-Human Services; Carr, Wilder, Majdalany, Mathisen, & Strain, 2013). Overall performance of these six measures was well below threshold (i.e., red

zone) after our first year of data collection and analysis and were the target of our subsequent intervention efforts. We hypothesized that home staff experience (i.e., the percentage of home staff employed six months or more) was influenced by the training these staff receive and other position openings in the home. Specifically, if direct service staff do not receive necessary training amidst other open positions in their home, they are likely to experience frustration, burnout, and leave the agency. Data we analyzed on staff turnover supports this assertion. Moreover, the extent to which home supervisors are overworked or experience turnover in their positions was related to position openings and the tenure of staff in their home. Home supervisor measures were also presumed to be related to turnover in the regional coach position.

Figure 1 also captures several system-level interventions our team has introduced in collaboration with agency leadership to address below-threshold performance on these six measures. For example, the agency changed its hiring practices to reduce the length of time to make hiring decisions. Before this change, human resources commonly required weeks and months to fill a position. Their system entailed a face-to-face interview and a follow-up visit to the home; the latter activity sometimes required weeks to schedule and prospective employees accepted offers elsewhere. Interviews now require a single 1-hour visit with a recruiter that includes standard questions, a realistic job preview, role plays, and preference assessments. The recruiter is allowed to make a conditional offer of hire at the conclusion of this visit.

Additionally, we have substantially revised training and performance management efforts.

Table 4. Performance Scorecard

Pinpointing and Measuring Employee Behavior 157

The initial training staff receive during onboarding was updated so it is consistent with best practices (i.e., a behavioral skills training approach) and addresses relevant topics (e.g., behavioral teaching procedures). Moreover, we developed an improved shadow training program requiring employees who serve as home trainers to: (1) complete a performance- and competency-based program on effective staff training practices; and (2) complete and submit a shadow training checklist that details training content and process for new employees. Relatedly, home staff now receive regular observations and feedback on a range of job

skills. Supervisory staff also receive training on effective supervision skills that emphasizes completing observations, delivering feedback, and developing positive relationships with employees. Finally, a monetary incentive program was introduced for regional coaches to incentivize meeting or surpassing the thresholds on the quality assurance measures, retention, and other behaviors and outcomes of relevance to the organization (e.g., submitting billing in a timely manner, completing safety checks).

The effects of the above-described change efforts are presently being evaluated using multi- source data, including the measures we track on

Home staff experience

Home supervisor

overworked

Home supervisor

turnover

Regional coach turnover

Home staff openings too

long

Training requirements

met

Onboarding & shadow training

Feedback checklists

Hiring practices

Supervisory skills

training

Monetary incentive program

Figure 1. Proposed model detailing the relationships among various measures (white boxes) and organizational solutions to address below-threshold performance on those measures (gray diamonds).

158 DiGennaro Reed et al.

the performance scorecards, data from other departments, employee observation results, and satisfaction surveys. Based on these data, our team will work collaboratively with the agency to revise existing or introduce new interventions. By pinpointing and measuring employee behavior and tracking results, we can help the agency achieve its goals and ultimately ensure service recipients receive the quality services to which they are entitled.

References

Alvero, A. M., Rappaport, E., & Taylor, M. A. (2011). A further assessment of momentary time-sampling across extended interval lengths. Journal of Organizational Behavior Management, 31, 117-129. doi: 10.1080/01608061.2011.569203

Alvero, A. M., Struss, K., & Rappaport, E. (2008). Measuring safety performance: A comparison of whole, partial, and momentary time-sampling recording methods. Journal of Organizational Behavior Management, 27, 1-28. doi: 10.1300/J075v27n04_01

Amigo, S., Smith, A., & Ludwig, T. (2008). Using task clarification, goal setting, and feedback to decrease table busing times in a franchise pizza restaurant. Journal of Organizational Behavior Management, 28, 176-187. doi: 10.1080/01608060802251106

Bloom, M., Fischer, J., & Orme, J. G. (2003). Evaluating practice: Guidelines for the accountable professional (4th ed.). Boston, MA: Allyn & Bacon.

Blum, R. H., Boulet, J. R., Cooper, J. B., & Muret-Wagstaff, S. L. (2014). Simulation- based assessment to identify critical gaps in safe anesthesia resident performance. Anesthesiology, 120(1), 129-141.

Carr, J. E., Wilder, D. A., Majdalany, L., Mathisen, D., & Strain, L. A. (2013). An assessment-based solution to a human- service employee performance problem: An initial evaluation of the Performance Diagnostic Checklist-Human Services. Behavior Analysis in Practice, 6, 16-32.

Cooper, J. O., Heron, T. E., & Heward, W. L. (2007). Applied behavior analysis (2nd ed.). Upper Saddle River: NJ.

Daniels, A. C. (2016). Bringing out the best in people: How to apply the astonishing power of positive reinforcement. New York, NY: McGraw-Hill.

Daniels, A. C., & Daniels, J. E. (2006). Performance management: Changing behavior that drives organizational effectiveness. Atlanta, GA: Performance Management Publications.

Dixon, M. R., Whiting, S. W., Rowsey, K., Gunnarsson, K., & Enoch, M. R. (2014). Direct observation of road construction worker behavior. Journal of Organizational Behavior Management, 34, 179-187. doi: 10.1080/01608061.2014.944746

Gil, P. J., & Carter, S. L. (2016). Graphic feedback, performance feedback, and goal setting increased staff compliance with a data collection task at a large residential facility. Journal of Organizational Behavior Management, 36, 56-70. doi: 10.1080/01608061.2016.1152207

Gunter, P. L., Venn, M. L., Patrick, J., Miller, K. A., & Kelly, L. (2003). Efficacy of using momentary time samples to determine on- task behavior of students with emotional/behavioral disorders. Education and Treatment of Children, 26(4), 400-412.

Hawkins, R. P., & Dobes, R. R. (1977). Behavioral definitions in applied behavior analysis: Explicit or implicit? In B.C. Etzel, J.M. LeBlanc, & D.M. Baer (Eds.), New developments in behavioral research: Theory, method, and application (pp. 167-188). Hillsdale, NJ: Erlbaum.

Hinze, J., Thurman, S., & Wehle, A. (2012). Leading indicators of construction safety performance. Safety Science, 51, 23-28.

Johnston, J. M., & Pennypacker, H. S. (2009). Strategies and tactics of behavioral research (3rd ed). New York, NY: Routlege.

Kell, H. J., Martin-Raugh, M. P., Carney, L. M., Inglese, P. A., Chen, L. & Feng, G. (2017). Exploring methods for developing behaviorally anchored rating scales for evaluating structured interview performance. ETS Research Report Series.

Pinpointing and Measuring Employee Behavior 159

Advance online publication. doi: 10.1002/ets2.12152

Komaki, J. C., & Minnich, M. R. (2008). In C.M. Johnson, W.K. Redmon, & T.C. Mawhinney (Eds.), Handbook of organizational performance: Behavior analysis and management (pp. 51-80). Binghamton, NY: Haworth Press

Martin-Raugh, M., Tannenbaum, R. J., Tocci, C. M., & Reese, C. (2016). Behaviorally anchored rating scales: An application for evaluating teaching practice, Teaching and Teacher Education, 59, 414-419.

Miltenberger, R. G., Flessner, C., Gatheridge, B., Johnson, B., Satterlund, M., & Egemo K. (2004). Evaluation of behavioral skills training to prevent gun play in children. Journal of Applied Behavior Analysis, 37, 513–516. doi: 10.1901/jaba.2004.37-513

Ongori, H. (2007). A review of the literature on employee turnover. African Journal of Business Management, 1(3), 49-54.

Parsons, M. B., Rollyson, J. H., & Reid, D. H. (2012). Evidence-based staff training: A guide for practitioners. Behavior Analysis in Practice, 5, 2-11.

Ray, R. L., Rizzacasa, T., & Levanon, G. (2013). Job satisfaction: 2013 edition. (Report No. TCB_R-1524-33-RR). Retrieved from http://www.conferenceboard.org/publicati ons/publicationdetail.cfm?publicationid=25 22

Reetz, N. K., Whiting, S. W., & Dixon, M. R. (2016). The impact of a task clarification and feedback intervention on restaurant service quality. Journal of Organizational Behavior Management, 36, 322-331. doi: 10.1080.01608061.2016.1201035

Rodriguez, M., Sundberg, D., & Biagi, S. (2016). OBM applied: A practical guide to implementing organizational behavior management. Melbourne, FL: ABA Technologies, Inc.

Smith, P.C., & Kendall, L. M. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47(2), 149-155.

Szabo, T. G., Williams, W. L., Rafacz, S. D., Newsome, W., & Lydon, C. A. (2012).

Evaluation of the service review model with performance scorecards. Journal of Organiztional Behavior Management, 32(4), 274- 296. doi: 10.1080/01608061.2012.729408

Taylor, M. A., Skourides, A., & Alvero, A. M. (2012). Observer error when measuring safety-related behavior: Momentary time sampling versus whole-interval recording. Journal of Organizational Behavior Management, 32, 307-319. doi: 10.1080/01608061.2012.729389

Thurkow, N. M., Bailey, J. S., & Stamper, M. R. (2000). The effects of group and individual monetary incentives on productivity of telephone interviewers. Journal of Organizational Behavior Management, 20, 3-25. doi: 10.1300/J075v20n02_02

Tziner, A., Joanis, C., & Murphy, K. R. (2000). A comparison of three methods of performance appraisal with regard to goal properties, goal perception, and rate satisfaction. Group & Organization Management, 25(2), 175-190.

Williams, W. L., & Gallinat, J. (2011). The effects of evaluating video examples of staffs' own versus others' performance on discrete-trial training skills in a human service setting. Journal of Organizational Behavior Management, 31(2), 97-116.

Williams, W. L., Vittorio, T. D., & Hausherr, L. (2003). A description and extension of a human services management model. Journal of Organizational Behavior Management, 22, 47- 71. doi: 10.1080/01608061.2011.570099

Wine, B., Reis, M., & Hantula, D. A. (2014). An evaluation of stimulus preference assessment methodology in organizational behavior management. Journal of Organizational Behavior Management, 34, 7-15. doi: 10.1080/01608061.2013.873379

Wright, M. C., Phillips-Bute, B. G., Petrusa, E. R., Griffin, K. L., Hobbs, G. W., & Taekman, J. M. (2009). Assessing teamwork in medical education and practice: Relating behavioural teamwork ratings and clinical performance. Medical Teacher, 31(1), 30-31.