Evidence-Based Practice Implementation—Anticipating Results
Sparkling Beauty
USW1_SOCW_6311_Week02_evaluatingYourProgram.pdf
Evaluating Your Program
Family Psychoeducation
U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES Substance Abuse and Mental Health Services Administration Center for Mental Health Services www.samhsa.gov
Family Psychoeducation
Evaluating Your Program
U.S. Department of Health and Human Services
Substance Abuse and Mental Health Services Administration
Center for Mental Health Services
Acknowledgments
This document was prepared for the Substance Abuse and Mental Health Services Administration (SAMHSA) by the New Hampshire-Dartmouth Psychiatric Research Center under contract number 280-00-8049 and Westat under contract number 270-03-6005, with SAMHSA, U.S. Department of Health and Human Services (HHS). Neal Brown, M.P.A., and Crystal Blyler, Ph.D., served as SAMHSA Government Project Officers.
Disclaimer
The views, opinions, and content of this publication are those of the authors and contributors and do not necessarily reflect the views, opinions, or policies of the Center for Mental Health Services (CMHS), SAMHSA, or HHS.
Public Domain Notice
All material appearing in this document is in the public domain and may be reproduced or copied without permission from SAMHSA. Citation of the source is appreciated. However, this publication may not be reproduced or distributed for a fee without the specific, written authorization from the Office of Communications, SAMHSA, HHS.
Electronic Access and Copies of Publication
This publication may be downloaded or ordered at http://www.samhsa.gov/shin. Or, please call SAMHSA’s Health Information Network at 1-877-SAMHSA-7 (1-877-726-4727) (English and Español).
Recommended Citation
Substance Abuse and Mental Health Services Administration. Family Psychoeducation: Evaluating Your Program. HHS Pub. No. SMA-09-4422, Rockville, MD: Center for Mental Health Services, Substance Abuse and Mental Health Services Administration, U.S. Department of Health and Human Services, 2009.
Originating Office Center for Mental Health Services Substance Abuse and Mental Health Services Administration 1 Choke Cherry Road Rockville, MD 20857 HHS Publication No. SMA-09-4422 Printed 2009
Family Psychoeducation
Evaluating Your Program
Evaluating Your Program shows quality assurance team members how to evaluate the effectiveness of your Family Psychoeducation program. It includes the following:
n A readiness assessment;
n The Family Psychoeducation Fidelity Scale;
n The General Organizational Index; and
n Outcome measures that are specific to your program.
You will also find instructions for conducting assessments and tips on how to use the data to improve your program.
For references see the booklet, The Evidence.
This KIT is part of a series of Evidence-Based Practices KITs created by the Center for Mental Health Services, Substance Abuse and Mental Health Services Administration, U.S. Department of Health and Human Services.
This booklet is part of the Family Psychoeducation KIT that includes a DVD, CD-ROM, and seven booklets:
How to Use the Evidence-Based Practices KITs
Getting Started with Evidence-Based Practices
Building Your Program
Training Frontline Staff
Evaluating Your Program
The Evidence
Using Multimedia to Introduce Your EBP
What’s in Evaluating Your Program
Why Evaluate Your Family Psychoeducation Program? . . . . . 1
Conduct a Readiness Assessment . . . . . . . . . . . . . . . . . . . . 3
Conduct Process Assessments . . . . . . . . . . . . . . . . . . . . . . . 5
Monitor Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Use Data to Improve Your Program . . . . . . . . . . . . . . . . . . 17
Appendix A: Cover Sheet—Family Psychoeducation Fidelity Scale and General Organizational Index . . . . . 23
Appendix B: Checklist—Observation of Multifamily Group Sessions . . . . . . . . . . . . . . . . . . . . . . . 27
Appendix C: Family Psychoeducation Fidelity Scale and Scoresheet . . . . . . . . . . . . . . . . . . 31
Appendix D: Family Psychoeducation Fidelity Scale Protocol . . . . . . . . . . . . . . . . . . . . . . . 37
Appendix E: General Organizational Index and Scoresheet . . . . . . . . . . . . . . . . . . 49
Appendix F: General Organizational Index Protocol . . . . . . 55
Appendix G: Outcomes Report Form . . . . . . . . . . . . . . . . . 69
Appendix H: Instructions for the Outcomes Report Form . . 73
Appendix I: Assessor Training and Work Performance Checklist . . . . . . . . . . . . . . . . . 79
Family Psychoeducation
Evaluating Your Program 1 Why Evaluate Your Family Psychoeducation Program
Evaluating Your Program
Why Evaluate Your Family Psychoeducation Program?
Key stakeholders who are implementing Family Psychoeducation (FPE) programs may find themselves asking two questions:
n Has the program been implemented as planned?
n Has the program resulted in the expected outcomes?
Asking these two questions and using the answers to help improve your program are critical for ensuring the success of your FPE program.
To answer the first question, collect process measures (by using the FPE
Fidelity Scale and General Organizational Index). Process measures capture how services are provided. To answer the second question, collect outcome measures. Outcome measures capture the results or achievements of your program.
As you prepare to implement your program, we strongly recommend that you develop a quality assurance system using both process and outcome measures to monitor and improve the quality of the program from the startup phase and continuing through the life of the program.
Why Evaluate Your Family Psychoeducation Program 2 Evaluating Your Program
Why you should collect process measures
Process measures give you an objective, structured way to determine if you are delivering services in the way that research has shown will result in desired outcomes. Process measures allow agencies to understand whether they are providing services that are faithful to the evidence-based model. Programs that adhere closely to the FPE model are more effective than those that do not follow the model. Adhering to the model is called fidelity.
Collecting process measures is an excellent method to diagnose program weaknesses while helping to clarify program strengths. Once FPE programs reach high fidelity, ongoing monitoring allows you to test local innovations while ensuring that programs do not drift from the core principles of the evidence-based practice.
Process measures also give mental health authorities a comparative framework to evaluate the quality of FPE programs across the state. They allow mental health authorities to identify statewide trends and exceptions to those trends.
Why you should collect outcome measures
While process measures capture how services are provided, outcome measures capture the program’s results. Every service intervention has both immediate and long-term consumer goals. In addition, consumers have goals for themselves, which they hope to attain with the help of mental health services. These goals translate into outcomes and the outcomes translate into specific measures.
Consumer outcomes are the bottom line for mental health agencies, like profit is in business. No successful businessperson would assume that the business was profitable just because employees work hard.
Why develop a quality assurance system
In your mental health system, you should develop a quality assurance system that collects not only process measures such as those on the FPE Fidelity Scale and General Organizational Index, but also outcome measures such as those specified above to show the effect of FPE. Developing a quality assurance system will help you do the following:
n Diagnose your program’s strengths and weaknesses;
n Formulate action plans for improving your program;
n Help consumers achieve their goals for recovery; and
n Deliver mental health services both efficiently and effectively.
Research Has Shown That You Can Expect These Outcomes from Your FPE Program
n Reduced relapse and hospitalization
n Improved family well-being
n Increased participation in vocational rehabilitation
n Higher rates of employment, when combined with Supported Employment
n Decreased costs of care
Evaluating Your Program 3 Conduct a Readiness Assessment
Evaluating Your Program
Conduct a Readiness Assessment
Let’s assume that administrators and family intervention coordinators have read Building Your Program. Your new FPE practitioners have completed Training Frontline Staff. How do you know if you are ready to begin providing FPE services to consumers?
The Readiness Assessment on the next page will help quality assurance team members, advisory group leaders, and family intervention coordinators track
the processes and administrative tasks required to develop an FPE program.
Answering these questions will help you generate an ongoing to-do list (or implementation plan) to guide your steps in implementing your FPE program. Your answers will also help you understand the components of the FPE model that are already in place in your agency and the work that still remains.
Conduct a Readiness Assessment 4 Evaluating Your Program
Readiness Assessment
Check any areas that you feel you do NOT completely understand.
q Which practitioners will be designated as staff for your FPE program?
q Who will supervise and direct the FPE program (who will be the family intervention coordinator)?
q What are the roles of the family intervention coordinator and practitioners?
q What is the size of the FPE practitioners’ caseloads?
q What is the supervisory structure (how often does the family intervention coordinator meet with FPE practitioners and the agency director)?
q How will you identify and refer consumers to your FPE program?
q How will you inform consumers, families, and others of your FPE program?
q What are your assessment procedures for consumers in your FPE program?
q How will you document the provision of FPE services?
q How often will FPE sessions be offered?
q How long will consumers and families receive FPE?
q What is your planning process for arranging the FPE one-day educational workshop?
q How will FPE consumers and families have access to multimedia educational materials?
q When will you offer FPE multifamily groups?
q How will you measure your program’s fidelity to the evidence-based model and use this information to improve your program?
q How will you collect and use consumer outcomes data?
q How does your FPE staff relate to advisory groups?
Note areas where you still are unclear or have questions. Arrange to speak to an expert consultant or experienced family
intervention coordinator.
Evaluating Your Program 5 Conduct Process Assessments
Evaluating Your Program
Conduct Process Assessments
In addition to the Readiness Assessment, you should conduct your first process assessment before you begin providing any FPE services. By doing so you will determine whether your agency has core components of the evidence-based practice in place. During the first 2 years of implementing your FPE program, plan to assess your program every 6 months.
After your program has matured and achieved high fidelity, you may choose to conduct assessments once a year. Agencies that have successfully implemented FPE programs indicate that you must continue
to evaluate the process to ensure that you do not revert to previous practice patterns.
Once your program has achieved high fidelity to the evidence-based model, FPE practitioners may tailor the program to meet individual needs of the community. If you continue to use process assessments along with outcomes monitoring, you will be able to understand the extent to which your changes result in your program’s departure from model fidelity and whether the changes positively or negatively affect consumers.
Conduct Process Assessments 6 Evaluating Your Program
How to use process measures
Two tools have been developed to monitor how FPE is provided:
n The FPE Fidelity Scale; and
n General Organizational Index.
You may administer both tools at the same time.
The FPE Fidelity Scale has 14 program-specific items. Each item is rated on a 5-point scale, ranging from 1 (not implemented) to 5 (fully implemented). The items assess whether the program is provided as the evidence-based model prescribes.
The General Organizational Index is a second set of process measures that has been developed. In contrast to fidelity scales, which are practice- specific, this assessment can be used when implementing any evidence-based practice. It measures agency-wide operating procedures that have been found to affect agencies’ overall capacity to implement and sustain any evidence- based practice.
For the FPE Fidelity Scale and General Organizational Index, see Appendices C and E. You can also print these forms from the CD-ROM in the KIT.
About the Process Measures that Are Included in the KIT
Quality assurance measures have been developed and are included in all Evidence-Based Practices KITs.
The FPE Fidelity Scale was developed by a group of researchers at Indiana University-Purdue University, Indianapolis, and the developers of the KIT. The standards used for establishing the anchors for the “fully implemented” ratings were determined through a variety of expert sources as well as through empirical research. The scale has undergone numerous drafts and review by
many groups. Revisions were also made based on feedback from a variety of sources during the 3-year pilot testing of the KIT materials.
The General Organizational Index, developed by Robert Drake and Charlie Rapp, is a newly developed scale. This scale has undergone multiple revisions based on feedback gathered during the 3-year pilot testing of the KIT materials.
Evaluating Your Program 7 Conduct Process Assessments
Who can conduct process assessments?
We recommend enlisting two assessors to conduct your process assessment. Data collected by two assessors simultaneously increase the likelihood that information will be reliable and valid.
Agencies that have successfully implemented FPE programs have taken different approaches to identify assessors. Some agencies train FPE Advisory Committee members as assessors and rotate the responsibility of completing assessments. Others have pre-existing quality assurance teams and simply designate members of the team to complete assessments of their FPE program. In other cases, the mental health authorities have designated staff to conduct assessments.
Assessments can be conducted either internally by your agency or program or externally by a review group. External review groups have a distinct advantage because they use assessors who are familiar with FPE but, at the same time, are independent. The goal is to select objective and competent assessors.
Although we recommend using external assessors, agencies can also use internal staff to rate their own programs. The validity of these ratings (or any ratings, for that matter) depends on the following:
n The knowledge of the person making the ratings;
n Access to accurate information pertaining to the ratings; and
n The objectivity of the ratings.
If you do conduct your assessments using internal staff, beware of potential biases of raters who are invested in seeing the program look good or who do not fully understand FPE. It is important for ratings to be made objectively and that they be based on hard evidence.
Circumstances will dictate decisions in this area, but we encourage agencies to choose a review process that fosters objectivity in ratings, for example, by involving a practitioner who is not centrally involved in providing FPE. Only people who have experience and training in interviewing and data collection procedures (including chart reviews) should conduct assessments. Additionally, assessors need to understand the nature and critical ingredients of the evidence-based model.
If your agency chooses to use a consultant or trainer to help implement your FPE program, involving that person in the assessment process will enhance the technical assistance you receive. Whichever approach you choose, we encourage you to make these decisions early in the planning process. For a checklist to help evaluate assessors’ training and work performance, see Appendix I.
How to conduct process assessments
A number of activities take place before, during, and after a process assessment. In general, assessments include the following:
n Interviewing administrators, the family intervention coordinator, FPE practitioners, consumers, and families;
n Interviewing other agency staff (psychiatrists, therapists, or case managers);
n Observing one or more group or individual sessions;
n Observing a planning and supervisory meeting; and
n Conducting a chart review.
Collecting information from multiples sources helps assessors more accurately capture how services are provided. A day-long site visit is the best way to learn this information.
Conduct Process Assessments 8 Evaluating Your Program
To save time, you may interview FPE practitioners in a group. If the FPE program has three or fewer FPE practitioners, you should interview all of them. If the program has more than three FPE practitioners, you should try to interview at least three of them.
For the items that require interviews with consumers and family members, we suggest that you interview at least three (from unique families). Try to interview families who are at different stages of the educational process. Contact the family intervention coordinator to help identify and set up these interviews. The following suggestions outline steps in the assessment process.
Before the process assessment
n n n Prepare your assessment questions
A detailed protocol has been developed to help you understand each item on the FPE Fidelity Scale and General Organizational Index, the rationale for why it was included, guidelines for the types of information to collect, and instructions for completing your ratings. Use the protocols to help prepare the questions that you will ask during your assessment visit. For the FPE Fidelity Scale and General Organizational Index protocols, see Appendices D and F.
While we expect that quality assurance teams will select which outcome measures meet your agency’s needs, you should use the FPE Fidelity Scale and General Organizational Index in full. Collecting data for all the items on these scales will allow your agency to gain a comprehensive understanding of how closely your program resembles the evidence-based model.
n n n Create a timeline for the assessment
List all the necessary activities leading up to and during the visit and create a timeline for completing each task. Carefully coordinating efforts, particularly if you have multiple assessors, will help you complete your assessment in a timely fashion.
n n n Establish a contact person
Have one key person in the FPE program arrange your visit and communicate beforehand the purpose and scope of your assessment to people who will participate in interviews. Typically, this contact person will be the family intervention coordinator.
Exercise common courtesy and show respect for competing time demands by scheduling well in advance and making reminder calls to confirm interview dates and times.
n n n Establish a shared understanding with the staff of the FPE program
The most successful assessments are those in which assessors and the FPE staff share the goal of understanding how the program is progressing according to evidence-based principles. If administrators or FPE practitioners fear that they will lose funding or look bad if they don’t score well, then the accuracy of the data may be compromised. The best assessment is one in which all parties are interested in learning the truth.
n n n Indicate what you will need from respondents during your visit
In addition to the purpose of the assessment, briefly describe what information you need, with whom you must speak, and how long each interview will take to complete.
Evaluating Your Program 9 Conduct Process Assessments
The visit will be most efficient if the family intervention coordinator gathers beforehand as much of the following information as possible:
n Roster of FPE staff (roles and full-time equivalents [FTEs]);
n Number of consumers the agency serves;
n Number of consumers actively receiving FPE;
n Number of consumers and families who have attended:
Three or more joining sessions;
Educational workshop; and
Each multifamily group;
n Number of consumers served through the FPE program in the past 6 months;
n Number of consumers who have dropped out of the FPE program in the past 6 months;
n A copy of the agency’s brochure or mission statement for the FPE program;
n A copy of the policies, procedures, and forms used to identify consumers for FPE;
n A copy of the policies, procedures, and forms used with consumers in the FPE program for assessment and treatment planning;
n A copy of the curriculum used in the educational workshop;
n A copy of the curriculum used to train agency staff on the evidence-based model; and
n A copy of the agency’s quality assurance procedures, specifically a list of process and outcome measures used to evaluate the FPE program.
Reassure the family intervention coordinator that you will be able to conduct the assessment, even if all of the requested information is unavailable. Indicate that some information is more critical (for example, number of FPE practitioners and number of consumers in the FPE program) than other information.
Tell the contact person that you must observe a planning meeting, a group supervision meeting, and a multifamily group session during your visit. These are important factors in determining when you should schedule your visit.
Observing an FPE multifamily group is integral to the assessment process. If observing a multifamily group session is impossible during your visit, arrange to have the sessions videotaped before your site visit.
n n n Alert your contact person that you will need to sample 10 charts
From an efficiency standpoint, it is preferable that the charts be drawn beforehand, using a random selection procedure. There may be a concern that the evaluation may be invalidated if FPE practitioners handpick charts or update them before the visit. If you both understand that the goal is to learn how the program is implementing services, this is less likely to occur.
Additionally, you can further ensure random selection by asking for 20 charts and randomly selecting 10 to review. Other options include asking for a de- identified list (i.e., with names removed) of consumers who receive FPE and using the list to choose 10 charts to review.
Conduct Process Assessments 10 Evaluating Your Program
If the program only has one FPE practitioner with fewer than 10 consumers on its caseload, then review the charts for all consumers in the program.
n n n Clarify reporting procedures
With the appropriate people (agency administrators, the mental health authority, or the family intervention coordinator), clarify who should receive a report of the assessment results. Recipients may include the following:
n Agency administrators;
n Members of the agency’s quality assurance team;
n Members of the FPE Advisory Committee;
n The family intervention coordinator;
n FPE practitioners; and
n Consumers and families.
Assessors should also clarify how the agency would like the report to be distributed. For example, assessors may mail or fax the report and follow up to discuss the results in a meeting or by conference call.
n n n Organize your assessment materials
Four forms have been created to help you conduct your assessment:
n The first form is a cover sheet for the FPE Fidelity Scale and General Organizational Index, which is intended to help you organize your process assessment. It captures general descriptive information about the agency, data collection, and community characteristics.
n The second form is a designed to help you collect data on two FPE fidelity items (Items 11 and 12). Complete this form to record information collected
during your observation of FPE multifamily group sessions.
n The third and fourth forms are scoresheets for the two scales. They help you compare assessment ratings from one time period to the next. They may also be useful if you are interested in graphing results to examine your progress over time.
For the FPE Fidelity Scale and General Organizational Index instruments, cover sheet, checklist, and scoresheets, see Appendices A, B, C and E. You can also print these forms from the CD-ROM in the KIT.
During your assessment visit
n n n Tailor your terminology
To avoid confusion during your interviews, tailor your terminology. For example, an FPE program may use client instead of consumer or it may use clinician instead of practitioner. Every agency has specific job titles for particular staff roles. By adopting the local terminology, you will improve communication.
n n n Conduct your chart review
It is important that you conduct your chart review from a representative sample of charts. When you begin your chart review, note whether your sample reflects families of consumers in different stages of the educational process. You should also note whether your sample includes consumer charts from each FPE practitioners’ caseload. Selecting charts of consumers who have received at least five FPE sessions is preferred. If your random sample is not representative in this manner, consider supplementing your sample with selected charts that will increase its representativeness.
Evaluating Your Program 11 Conduct Process Assessments
Within each chart, examine the screening, referral, assessment, and treatment planning forms. Review recent Progress Notes to understand the amount and type of contact that FPE practitioners have with the consumers on their caseloads and with their treatment team members. If Progress Notes are not integrated into consumer charts, then ask if FPE practitioners have any additional files that you may review.
In some cases, a lag may exist between when a service is given and when it is documented in the consumer’s chart. To get the most accurate representation of services rendered when you sample chart data, try to gather data from the most recent time period in which documentation is completed in full.
To ascertain the most up-to-date time period, ask the family intervention coordinator, FPE practitioners, or administrative staff. Avoid getting an inaccurate sampling of data where office- based services might be charted more quickly than services given in the field.
n n n If discrepancies between sources occur, query the family intervention coordinator
The general strategy in conducting fidelity assessments is to obtain data from as many sources as possible. When all these data sources converge, you can be more confident in the validity of the ratings. However, sometimes sources disagree.
The most common discrepancy is likely to occur when the family intervention coordinator’s interview gives a more idealistic picture of the team’s functioning than the chart and observational data do. For example, on the FPE Fidelity Scale, Assertive engagement and outreach (Item 14) assesses whether FPE practitioners assertively engage all potential consumers and family members in the FPE program.
The chart review may show that consumers who drop out of the program are not contacted, while the family intervention coordinator may indicate that FPE practitioners expend considerable time reaching out to consumers who have disengaged from the program.
To understand and resolve this discrepancy, the assessor should ask the family intervention coordinator the following:
Our chart review shows 10 percent of consumers who disengage are contacted, but your estimate is much higher. Would you help us understand the difference?
Often the family intervention coordinator can provide information that will resolve the discrepancy.
n n n Before you leave, check for missing data
Fidelity scales should be completed in full, with no missing data on any items. Check in with the family intervention coordinator at the end of the visit to collect any additional information you may need.
Conduct Process Assessments 12 Evaluating Your Program
After your assessment visit
n n n Followup
It is important to collect any missing data before completing your rating. If necessary, follow up on any missing data (for example, by calling or sending an e-mail). This would include discussing with the family intervention coordinator any discrepancies between data sources that you notice after you’ve completed the visit.
n n n Score the scales
The purpose of the scale is to assess fidelity to the evidence-based practice at the program level, rather than at the level of a specific practitioner. Ratings are based on current behavior and activities, not on planned or intended behavior. For example, to get full credit (to code the item as “5”) for Family intervention coordinator (Item 1), the program must have a designated staff member fulfilling the tasks of this position. If the agency plans to hire personnel to fill the position, it would not receive credit. If you assess an agency for the first time to determine which components of the evidence-based model the agency already has in place, some items may not apply.
Many agencies that are developing a new FPE program will receive low fidelity ratings on items for which the agency has not yet formulated its policies and procedures. For example, several items are based on evaluating services that are
provided by designated trained FPE practitioners. Agencies that have not yet hired or assigned and trained FPE practitioners, identified consumers and families, offered an FPE 1-day educational workshop, or started an FPE multifamily group cannot be rated for these items. If an item cannot be rated, code the item as “1.”
To receive full credit, many items require that the family intervention coordinator and practitioners both understand and apply the evidence-based practice principle. If FPE practitioners generally do not understand the concepts, then code that item as “1.” If they understand parts of the concept and apply the understanding consistently, code the item as “3.” To receive full credit, there must be evidence that the concepts are applied consistently.
For a complete explanation of how to rate each item, see the FPE Fidelity Scale Protocol and General Organizational Index Protocols in Appendices D and F.
n n n Complete scales independently
If you have two assessors, both should independently review the data collected and rate the scales. They should then compare their ratings, resolve any disagreements, and devise a consensus rating.
n n n Complete the scoresheets
Tally the item scores and determine the level of implementation achieved.
Evaluating Your Program 13 Monitor Outcomes
Evaluating Your Program
Monitor Outcomes
Unlike process measures, which must be used in full to comprehensively understand how services are provided, you must decide which outcome measures will be most informative for your program. Initially, your outcomes monitoring system should be simple to use and maintain. Complexity has doomed many well- intended attempts to collect and use outcomes data.
One way to simplify is to limit the number of outcome measures. Select your outcome measures based on the
type of information that will be most useful to your agency. Based on the research literature, we suggest that you monitor a core set of outcomes such as the following:
n Relapse and hospitalization;
n Family well-being;
n Participation in Supported Employment or vocational rehabilitation;
n Employment rates; and
n Cost of care.
Monitor Outcomes 14 Evaluating Your Program
These few outcomes reflect the primary goals of FPE. Specifically, the goal of FPE is to help consumers move forward in the process of recovering from mental illnesses and pursuing meaningful life goals. For this reason, it is important for you to capture outcomes for recovery in a way that is most useful for your program.
For data to be useful, they must be valid. That is, the data must measure what they are supposed to measure. Thus, the outcomes must be few and concrete for FPE practitioners to focus on key outcomes, to understand them in a similar way, and to make their ratings in a consistent and error- free fashion.
To enhance validity, we recommend using simple ratings initially. Limiting your outcome measures to concrete measures will also allow you to collect data from FPE practitioners.
Develop procedures
Agencies may choose either to develop the outcomes portion of their quality assurance system from scratch or to use existing outcomes monitoring systems. A number of electronic evaluation programs are available to help you develop comprehensive, integrated, user-friendly outcome monitoring systems. Examples include the following:
n Publicly available tools such as the Consumer Outcomes Monitoring Package (see box below), the Decision Support 2000+ Online (http://www.ds2kplus.org); or
n Various commercially available products.
What Is the Consumer Outcomes Monitoring Package?
Sponsored in part by the Substance Abuse and Mental Health Services Administration (SAMHSA), the Consumer Outcomes Monitoring Package (COMP) was designed by a team at the School of Social Welfare, University of Kansas. This computer application allows agencies to choose from a pre- established list of outcomes developed for each evidence-based practice. Data may be entered for the chosen outcomes, and reports can be generated quarterly or monthly. The COMP also allows agencies to view their outcomes data using a variety of tables and graphs.
The designers of COMP tried to make the computer application as easy and as flexible
to use as possible. You may access COMP through the Web. Agencies can download the computer application and print out Installation Instructions and a User Manual, which provides definitions and forms.
To download COMP
n Go to http://research.socwel.ku.edu/ebp.
n Click on the link to the download page.
n Click on the links to download the Installation Instructions and User Manual.
n Follow the instructions to install the application.
Evaluating Your Program 15 Monitor Outcomes
When deciding whether to use an existing outcomes monitoring package or to design your own, it is important to keep your agency’s capabilities in mind. The system must not create undue burden for FPE practitioners, and it must give them information that is useful in their jobs.
The system should fit into the workflow of the agency, whether that means making ratings on paper, using the COMP computer application, or developing your own outcomes monitoring package.
Start with whatever means are available and expand the system from there. In the beginning, you may collect data with a simple report form and you can report hand-tallied summaries to FPE practitioners.
Computer software that allows for data entry and manipulation (for example, Microsoft Access, Excel, and Lotus) makes tabulating data and graphing easier than doing them by hand. A computerized system for data entry and report generation presents a clear advantage and it may be the goal, but do not wait for it. Feedback does not have to come from a sophisticated computer system to be useful. It is more important that it is meaningful and frequent. For a sample Outcomes Report Form, see Appendix G, which is an example of a simple, paper-based way to collect participation and outcomes data regularly. For instructions for using the Outcomes Report Form, see Appendix H.
Expanding Your Outcome Measures
Once you have established your core outcomes monitoring system, have learned how to routinely collect data, and are accustomed to using it to improve your FPE program, you will be ready to expand your outcomes measures.
Consider asking consumers and families for input about how to improve your FPE program, both practically and clinically. Consumers and families are important informants for agencies that are seeking to improve outcomes. Agencies may want to know the following:
n If consumers and families are satisfied with their services;
n How services have affected their quality of life; and
n Whether consumers believe the services are helping them achieve their recovery goals.
While collecting data from consumers and families requires more staff time than the information that may be reported quickly by FPE practitioners, consumers and families can give valuable feedback.
We recommend the following surveys for collecting information from consumers and families:
n The Mental Health Statistics Improvement Program (MHSIP) Consumer Satisfaction Survey at http://www.mhsip.org
n Recovery measurement instruments such as those described in Measuring the Promise: A Compendium of Recovery Measures, Volume II, available through http://www.tecathsri.org
It is difficult to get a representative sample of consumer and family respondents since mailed surveys are often not returned and interviews may only be done with people who are cooperative and easy to reach. Samples that are not representative may be biased.
Avoid bias in your consumer and family data by using a variety of mechanisms to conduct your assessments. For example, consider combining feedback collected through surveys with that obtained from focus groups. Another option is to hire a consultant to conduct qualitative interviews with a small group of consumers or families.
Monitor Outcomes 16 Evaluating Your Program
How often should you collect outcomes data?
Plan to monitor the outcomes for consumers in your FPE program every 3 months and share the data with program staff. Collecting data at regular and short intervals will enhance the reliability of your outcomes data.
While we recommend that you design a system for collecting outcomes early in the implementation process, FPE programs should not expect to see the desired results until the program is fully operational. Depending on resources available to your program, this may take anywhere from 6 to 18 months to accomplish.
How should you identify data collectors?
Agency administrators or mental health authorities may assign the responsibility for collecting outcomes data to the following:
n The family intervention coordinator;
n Members of the FPE Advisory Committee;
n The quality assurance team;
n Independent consultants, including consumers and family members; and
n Other staff.
Unlike collecting process measures, collecting outcome measures does not require a day-long assessment process. Many standard outcome measures will be information that FPE practitioners can report from their daily work with consumers.
It is important to develop a quick, easy, standardized approach to collect outcomes data. For example, create a simple form or computer database that FPE practitioners can routinely update.
Evaluating Your Program 17 Use Data to Improve Your Program
Evaluating Your Program
Use Data to Improve Your Program
As you develop a quality assurance system, family intervention coordinators and FPE practitioners will weave it into the fabric of their daily routines. Process assessments will give you a window into the demanding work done every day. Outcome reports will give you tangible evidence of the use and value of services, and they will become a basis for decisionmaking and supervision.
At some point, your program staff may wonder how they did their jobs without an information system. They will come to view it as an essential ingredient of well- implemented evidence-based practices.
Use Data to Improve Your Program 18 Evaluating Your Program
n n n Create reports from your assessments
For your process data, in addition to completing the FPE Fidelity Scale, General Organizational Index, and scoresheets, assessors should write a report explaining their scores. The report should include the following:
n An interpretation of the results of the assessment;
n Strengths and weaknesses of the FPE program; and
n Clear recommendations to help the program improve.
The report should be informative, factual, and constructive. Since some process measures assess adherence to the evidence-based model at both the agency and program staff levels, remember to target recommendations to administrators, the family intervention coordinator, and FPE practitioners.
When summarizing outcomes data, start with simple, easy-to-read reports. Then let experience determine what additional reports you need. You can design your reports to give information about individual consumers, a single FPE practitioner’s caseload, or the program as a whole. For example, reports generated for individual consumers may track the consumer’s participation in specific stages of treatment and outcomes over time. You could enter these reports in consumers’ charts, and they could be the basis for discussions about consumers’ progress.
n n n Use tables and graphs to understand your outcomes data
After the first process and outcomes assessments, it is often useful to provide a visual representation of a program’s progress over time. We recommend that you use tables and graphs to help understand and report the results.
By graphing your fidelity score, you have a visual representation of how your program has changed over time. For an example, see Figure 1. For your process data, you may simply graph the results using a spreadsheet and include this in your report.
When your program shows greater fidelity over time, the graph will display it and reinforce your efforts. Additionally, as you can see in Figure 1, the graph allows you to quickly compare one team to another. In this example, Team A struggled in the first 6 months. Understanding Team A’s progress compared to Team B’s allowed the teams to partner and share strategies. Consequently, Team A improved dramatically over the next 6-month period.
Another feature of graphing assessment scores is to examine the cut-off scores for fair (52) or good (62) implementation. Your program can use these scores as targets.
Evaluating Your Program 19 Use Data to Improve Your Program
Figure 1. Fidelity Over Time
70
80
90
100
110
120
130
140 ▲
▲
▲
▲
▲
■
■
■
■ ■
■▲
Team A
Team B
Date of Assessment
Feb Apr Jun Aug Oct Dec Feb Apr Jun Aug Oct 2006 2007
Note: 62 – 70 = good implementation
52 – 61 = fair implementation
51 and below = not evidence-based practice
Here are three examples of tables and graphs that can help you understand and use your outcomes data.
Example 1: Periodic summary tables
Periodic summary tables summarize your outcomes data each quarter and address these kinds of questions:
n How many consumers participated in our FPE program during the last quarter?
n What proportion of consumers in our FPE program were hospitalized last quarter?
n How did the hospitalization rate for those participating in FPE compare to the rate for consumers in standard treatment?
Agencies often use this type of table to understand consumer participation or to compare actual results with agency targets or goals. These tables are also frequently used to describe agencies’
services in annual reports or for external community presentations.
Table 1: Sample Periodic Summary Table of Enrollment in Evidence-Based Practices
Not eligible
Eligible but NOT in EBP service
Enrolled Percent of eligible consumers enrolled
Family Psychoeducation
0 30 60 67%
Assertive Community Treatment
30 25 90 78%
This agency provided both Family Psychoeducation (FPE) and Assertive Community Treatment (ACT). The FPE staff identified 90 consumers for the program. Of those, 60 received FPE, while 30 consumers were eligible but received another service. Consequently, 67 percent of consumers who were eligible for the FPE program participated in the program.
Use Data to Improve Your Program 20 Evaluating Your Program
Example 2: Movement tables
Tables that track changes in consumer characteristics (called movement tables) can give you a quick reference for determining service effectiveness. For example, Table 2 compares consumers’ residential status between two quarters.
Table 2: Sample Movement Table
To FY ’06 Qtr 3
From: FY ‘06 Qtr: 2
Institutional Substantial care
Semi- independant
Independant Total
Institutional 2 1 1 3 7
Substantial care 3 8 1 3 15
Semi- independant 1 0 2 4 7
Independant 1 3 2 100 106
Total 7 12 6 110 135
To create this table, the data were collapsed into the four broad categories. The vertical data cells reflect the residential status for consumers for the beginning quarter. The horizontal data cells reflect the most recent quarterly information. The residential status categories are then ordered from the most restrictive setting (institutional) to the least restrictive (independent).
The data in this table are presented in three colors. The purple cells are those above the diagonal, the blue cells are those below the diagonal, and the white cells are those within the diagonal. The data cells above the diagonal represent consumers who moved into a less restrictive environment between quarters. As you can see, one consumer moved from institutional to substantial care, one to semi- independent care, and three to independent living. Furthermore, one consumer moved from
Above the diagonal
Below the diagonal
Within the diagonal
substantial care to semi-independent care, three consumers moved from substantial care to independent care and four consumers moved from semi-independent care to independent care. These 13 consumers (10 percent of the 135 consumers in the program) moved to a more desirable stage of treatment between quarters.
The data reported in the diagonal cells ranging from the upper left quadrant to the lower right reflect consumers who remained in the same residential status between quarters. Two consumers were in an institution for both quarters of this report; eight remained in substantial care, two in semi-independent and 100 in independent living. These 112 consumers (83 percent of the 135 consumers in the program) remained stable between quarters.
Evaluating Your Program 21 Use Data to Improve Your Program
The cells below the diagonal line represent consumers who moved into a more restrictive setting between quarters. Three consumers moved from substantial to institutional care, one consumer moved from semi-independent care to institutional care, one consumer moved from independent living to institutional care, three moved from independent living to substantial care, and two moved from independent living to semi- independent care. These 10 consumers (7 percent of the 135 consumers in the program) experienced some setbacks between quarters. The column totals show the number of consumers in a given residential status for the current quarter, and the row totals show the prior quarter.
You can use movement tables to portray changes in outcomes that are important to consumers, supervisors, and policymakers. The data may stimulate discussion about the progress that consumers are making or the challenges with which they are presented.
Example 3: Longitudinal plots
A longitudinal plot is an efficient and informative way to display participation or outcomes data for more than two successive periods. The goal is to view performance in the long term. You can use a longitudinal plot for a consumer, a caseload, a specific evidence-based practice, or an entire program. A single plot can also contain longitudinal data for multiple consumers, caseloads, or programs for comparison. Figure 2 presents an example of a longitudinal plot comparing critical incidents for one FPE program over an 11-month period.
This plot reveals that with the exception of private psychiatric hospitalizations, all other critical incidents appear to be going in a positive direction (that is, there is a reduction in incidence).
Longitudinal plots are powerful feedback tools because they permit a longer range perspective on participation and outcome, whether for a single
consumer or a group of consumers. They enable a meaningful evaluation of the success of a program, and they provide a basis for setting goals for future performance.
n n n Share your results
The single factor that will most likely determine the success of a quality assurance system is its ability to give useful and timely feedback to key stakeholders. It is fine to worry about what to enter into a system, but ultimately its worth is in converting data into meaningful information. For example, data may show that 20 consumers were homeless during the past quarter, but it is more informative to know that this represents 10 percent of the consumers in the FPE program.
For information to influence practice, it must be understandable and meaningful, and it must be delivered in a timely way.
Figure 2. Sample Longitudinal Plot for Monthly Frequency of Negative Incidents for Consumers
◆
✖
✖
✖
✖
✖
✖
✖
✖ ✖
✖ ✖
✖
▲ ▲▲ ▲
▲
▲
▲
▲ ▲
▲ ▲
◆ ◆
◆ ◆◆
◆
◆ ◆
■
■ ■■
■ ■
■
■ ■
■ ■◆ ◆
◆
Homeless
Incarcerated
In-Patient substance abuse
Priv. Psych. Hosp.
■
▲
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov
2007
16
14
12
10
8
6
4
2
0
Use Data to Improve Your Program 22 Evaluating Your Program
In addition, the quality assurance system must tailor the information to suit the needs of various users and to answer their questions.
Sharing results with FPE practitioners
After each assessment, dedicate time during a supervisory meeting to discuss the results. Numbers that reflect above average or exceptional performance should trigger recognition, compliments, or other rewards. Data that reflect below average performance should evoke a search for underlying reasons and should generate strategies that offer the promise of improvement. By doing this regularly, the family intervention coordinator will create a learning organization characterized by adaptive responses to information that aim to improve consumer outcomes.
Sharing results with your FPE Advisory Committee or quality assurance team
You may also use this information to keep external stakeholders engaged. Sharing information with vested members of the community, staff from your mental health authority, and consumers and family advocates can be valuable. Through these channels, you may develop support for the FPE program, increase consumer participation, and raise private funds for your agency.
Sharing results internally
Agencies may distribute reports during all staff and manager-level meetings to keep staff across the agency informed and engaged in the process of implementing your FPE program. Agencies with successful FPE programs highlight the importance of developing an understanding and support for the evidence-based model across the agency.
Additionally, integrating consumer-specific reports into clinical charts may help you monitor consumers’ progress over time. Reporting consumer-specific outcomes information at the treatment team meetings also helps keep the team focused on consumers’ goals.
Sharing results with consumers and families
Agencies may highlight assessment results in consumer and family meetings. Increasing consumers’ and families’ understanding of the FPE program may motivate them to participate in the treatment process and build trust in the consumer-provider relationship.
Also, sharing results may create hope and enthusiasm for your FPE program. Sharing information motivates people and stimulates changes in behavior. Sharing the results of your assessments with a variety of stakeholders is the key to improving your program.
Evaluating Your Program 23 Appendix A: Cover Sheet
Evaluating Your Program
Appendix A: Cover Sheet— Family Psychoeducation Fidelity Scale and General Organizational Index
Evaluating Your Program 25 Appendix A: Cover Sheet
Cover Sheet: Family Psychoeducation Fidelity Scale and General Organizational Index
Today’s date
Assessors’ names
Program name (or Program code)
Agency name
Agency address Street
City State ZIP code
Family intervention coordinator or contact person
Names of FPE practitioners
Telephone E-mail
Sources used for assessments: q Chart review: Number reviewed q FPE multifamily group observation
q Planning and supervisory meeting observation q Family intervention coordinator interview q FPE practitioner interviews Number interviewed q Consumer interviews Number interviewed q Family member interviews Number interviewed q Other staff interviews Number interviewed q Brochure review q Other
Number of FPE practitioners
Number of consumers/families in the program
Number of consumers/families who left the program in the past 6 months
Number of consumers/families served in the past 6 months
Funding source
Agency location: q Urban q Rural
Date program was started
( ) -
/ /
/ /
Evaluating Your Program 27 Appendix B: Checklist
Evaluating Your Program
Appendix B: Checklist— Observation of Multifamily Group Sessions
Evaluating Your Program 29 Appendix B: Checklist
Checklist—Observation of Multifamily Group Sessions
Today’s date
Assessors’ names
Program name (or Program code)
Agency name
Agency address Street
City State ZIP code
Names of FPE practitioners
Number of consumer participants
Number of family participants
Frequency of sessions
Item 11. Structured Group Sessions
Yes No
1. Beginning socialization q q 2. Review progress from last session’s action plan q q 3. Go-round q q 4. Selection of a single problem q q 5. Structured problem-solving q q 6. End with socialization q q Rating
Item 12. Structured Problem-Solving Technique
Yes No
1. Define the problem
2. Generate solutions
q q q q
3. Discuss advantages and disadvantages of each solution
4. Choose the best solution
q q q q
5. Form an action plan
6. Review the action plan
q q q q
Rating
/ /
Evaluating Your Program 31 Appendix C: Fidelity Scale and Score Sheet
Evaluating Your Program
Appendix C: Family Psychoeducation Fidelity Scale and Score Sheet
Evaluating Your Program 33 Appendix C: Fidelity Scale and Score Sheet
Family Psychoeducation Fidelity Scale
Criteria
Ratings / Anchors
1 2 3 4 5
1. Family intervention coordinator:
Designated clinical administrator who performs the following tasks:
n Establishes, monitors, and automates family intake and engagement procedures
n Assigns potential FPE consumers to FPE practitioners
n Monitors and adjusts FPE practitioner caseloads
n Arranges for training new FPE practitioners and continuing education of existing FPE staff
n Supervises FPE staff
Agency does not have a designated staff member
OR
Cannot rate due to no fit.
Agency has a designated staff member who performs 1 or 2 of the tasks.
Agency has a designated staff member who performs 3 of the tasks.
Agency has a designated staff member who performs 4 of the tasks.
Agency has a designated staff member who performs all tasks.
2. Session frequency:
Families and consumers participate biweekly in FPE sessions.
< Every 3 months
OR
Cannot rate due to no fit.
Every 3 months Every 2 months Monthly At least twice a month
3. Long-term FPE:
Families and consumers are provided with long-term FPE; specifically, at least one family member per consumer participates in FPE sessions for at least 9 months.
Most families and consumers receive less than 6 months of FPE sessions
OR
Cannot rate due to no fit.
Most families and consumers receive 6–7 months of FPE sessions.
Most families and consumers receive 7–8 months of FPE sessions.
Most families and consumers receive 8–9 months of FPE sessions.
More than 90% of families and consumers receive at least 9 months of FPE sessions.
4. Quality of practitioner- consumer-family alliance
FPE practitioners engage family members and consumers with warmth, empathy, acceptance, and attention to each individual’s needs and desires.
High dropout rate
OR
Cannot rate due to no fit.
Sources indicate that alliance is often poor, leading to high dropout rate.
MSources indicate alliance is inconsistent or barely adequate, leading to moderate dropout rate,
OR
Information is inconsistent
Sources indicate a fairly strong alliance.
Sources consistently indicate a strong alliance.
5. Detailed family reaction:
FPE practitioners identify and specify the family’s reaction to their relative’s mental illnesses.
There is consistent evidence for less than 33% of involved families.
There is consistent evidence for 33–49% of involved families.
There is consistent evidence for 50–64% of involved families.
There is consistent evidence for 65–79% of involved families.
There is consistent evidence for 80% or more of involved families.
6. Precipitating factors:
FPE practitioners, consumers, and families identify and specify precipitating factors for the consumers’ mental illnesses.
There is consistent evidence for less than 33% of involved families and consumers.
There is consistent evidence for 33-49% of involved families and consumers.
There is consistent evidence for 50–64% of involved families and consumers.
There is consistent evidence for 65–79% of involved families and consumers.
There is consistent evidence for 80% or more of involved families and consumers.
7. Prodromal signs and symptoms:
FPE practitioners, consumers, and families identify and specify prodromal signs and symptoms of the consumer’s mental illnesses.
There is consistent evidence for less than 33% of involved families and consumers.
There is consistent evidence for 33–49% of involved families and consumers.
There is consistent evidence for 50–64% of involved families and consumers.
There is consistent evidence for 65–79% of involved families and consumers.
There is consistent evidence for 80% or more of involved families and consumers.
Appendix C: Fidelity Scale and Score Sheet 34 Evaluating Your Program
Family Psychoeducation Fidelity Scale
Criteria
Ratings / Anchors
1 2 3 4 5
8. Coping strategies:
FPE practitioners identify, describe, clarify, and teach coping strategies.
There is consistent evidence for less than 33% of involved families and consumers.
There is consistent evidence for 33–49% of involved families and consumers.
There is consistent evidence for 50–64% of involved families and consumers.
There is consistent evidence for 65–79% of involved families and consumers.
There is consistent evidence for 80% or more of involved families and consumers.
9. Educational curriculum:
FPE practitioners use a standardized curriculum to teach families about mental illnesses. The curriculum covers six topics:
n Psychobiology of the specific mental illness;
n Diagnosis;
n Treatment and rehabilitation;
n Impact of mental illness on the family;
n Relapse prevention; and
n Family guidelines.
Less than 33% of involved families receive a standardized educational curriculum, no standardized educational curriculum exists,
OR
Only 1–2 topics are covered
33–49% of involved families receive a standardized educational curriculum covering all 6 topics
OR
Only 3 topics are covered.
50–64% of involved families receive a standardized educational curriculum covering all 6 topics
OR
Only 4–5 topics are covered.
65–79% of involved families receive a standardized educational curriculum covering all 6 topics.
80% or more of involved families receive a standardized educational curriculum covering all 6 topics.
10. Multimedia education:
Consumers and family members are given educational materials about mental illnesses in several formats (for example, paper, video, and Web sites).
Less than 33% of families and consumers receive educational materials
OR
Cannot rate due to no fit.
33–49% of families and consumers receive educational materials
OR
Materials are given in only 1 format.
50–64% of families and consumers receive educational materials
OR
Materials are given in only 2 formats.
65–79% of families and consumers receive educational materials in all 3 formats.
80% or more of families and consumers receive educational materials in all 3 formats.
11 Structured group sessions:
FPE practitioners follow a structured procedure that includes the following:
n Beginning socialization;
n Review progress from last session’s action plan;
n Go-round;
n Selection of a single problem;
n Structured problem solving; and
n Ending with socialization.
Groups include 2 or fewer components.
Groups include 3 of the 6 components.
Groups include 4 of the 6 components.
Groups include 5 of the 6 components.
Groups include all 6 components.
Evaluating Your Program 35 Appendix C: Fidelity Scale and Score Sheet
Family Psychoeducation Fidelity Scale
Criteria
Ratings / Anchors
1 2 3 4 5
12. Structured problem–solving:
FPE practitioners use a standardized approach to help consumers and families with problem solving, which includes the following:
n Define the problem;
n Generate solutions;
n Discuss the advantages and disadvantages of each solution;
n Choose the best solution;
n Form an action plan; and
n Review the action plan.
No more than 2 of 6 components of the structured problem- solving are used.
3 of 6 components of the structured problem-solving are used.
4 of 6 components of the structured problem-solving are used.
5 of 6 components of the structured problem-solving are used.
All 6 components of the structured problem-solving are used.
13. Stage-wise provision of services:
FPE services are provided in the following:
n Engagement;
n 3 or more joining sessions;
n Educational workshop; and
n Multifamily group.
Families and consumers begin multifamily groups with minimal or no engagement, no joining sessions, or no education.
Engagement is minimal and only 1 joining session is completed before entry into the multifamily group. Education is delayed or absent.
Engagement and 2 joining sessions are completed before entry into the multifamily group. Education is delayed or absent.
Most steps are done in order; however, families enter multifamily groups before 3 joining sessions are completed or education is provided.
Engagement, all 3 joining sessions, and education are completed before entry into the multifamily group.
14. Assertive engagement and outreach:
FPE practitioners assertively engage all potential consumers and family members by phone, by mail, or in person (in the agency or in the community) on an ongoing basis.
FPE practitioners do not engage potential consumers and family members.
FPE practitioners engage potential consumers and family members only once as part of initial engagement.
FPE practitioners engage potential consumers and family members 2 times as part of initial engagement.
FPE practitioners assertively engage some potential consumers and family members using all necessary means on a time- limited basis.
FPE practitioners assertively engage all potential consumers and family members using all necessary contact means on an ongoing basis.
FPE practitioners demonstrate tolerance of different levels of readiness using gentle encouragement.
Appendix C: Fidelity Scale and Score Sheet 36 Evaluating Your Program
Score Sheet: Family Psychoeducation Fidelity Scale
Date of visit
Agency name
Assessors’ names
Assessor 1 Assessor 2 Consensus
1 Family intervention coordinator
2 Session frequency
3 Long-term FPE
4 Quality of practitioner-consumer-family alliance
5 Detailed family reaction
6 Precipitating factors
7 Prodromal signs and symptoms
8 Coping strategies
9 Educational curriculum
10 Multimedia education
11 Structured group sessions
12 Structured problem-solving
13 Stage-wise provision of services
14 Assertive engagement and outreach
Total score
Items not rated
62-70 = Good implementation
52-61 = Fair implementation 51 and below = Not evidence-based practice
/ /
Evaluating Your Program 37 Appendix D: Fidelity Scale Protocol
Evaluating Your Program
Appendix D: Family Psychoeducation Fidelity Scale Protocol
Evaluating Your Program 39 Appendix D: Fidelity Scale Protocol
Family Psychoeducation Fidelity Scale Protocol
This protocol explains how to rate each item on the FPE Fidelity Scale. In particular, it provides the following:
n A definition and rationale for each fidelity item. These items have been derived from comprehensive, evidence-based literature.
n A list of data sources most appropriate for each fidelity item (for example, chart review, family intervention coordinator interview, FPE practitioners, consumers, or families). When appropriate, a set of probe questions is provided to help you elicit the critical information needed to code the item. These questions were specifically generated to help you collect information from respondents that is free from bias such as social desirability.
n Decision rules that will help score each item correctly. As you collect information from various sources, these rules will help you determine the specific rating to give for each item.
1. Family intervention coordinator
Definition: One clinical administrator is designated to oversee the FPE program for a substantial portion of the job (time depends on size of program). This person’s role includes activities such as the following:
n Establishing, monitoring, and automating family intake and engagement procedures;
n Assigning potential FPE consumers to FPE practitioners;
n Monitoring and adjusting FPE practitioners’ caseloads;
n Arranging for training of new staff and continuing education of existing FPE staff;
n Supervising FPE practitioners.
Rationale: Delivery of services to families must be subject to accountability and tracking. One effective way for agencies to monitor the delivery of family services is to create a position of family intervention coordinator, who would also serve as the contact person for FPE services, facilitate communication between staff and families, and supervise FPE practitioners.
Sources of information: Before the site visit, determine whether the organization has someone who has a title of family intervention coordinator or its equivalent. During the fidelity visit, interview the agency director, family intervention coordinator, practitioners, consumers, and family members.
Item response coding: The agency director and family intervention coordinator are the primary sources of information for this item. If other sources do not report these responsibilities performed by the coordinator, then fidelity assessors should follow up with the agency director and family intervention coordinator with clarifying questions and documentation (at end of the fidelity visit day or in follow-up call). If the program does not have a designated position of family intervention coordinator (or an equivalent), code the item as “1.” If the program has a designated staff member who performs all five tasks, code the item as “5.”
Probe questions
For family intervention coordinators:
n “What is your role in the FPE program? How much time do you devote to this? What kinds of responsibilities do you have?” [Check who performs the tasks specified above.]
n “Can you explain intake procedures, monitoring, training schedule, and supervision schedule?”
For FPE practitioners:
n “What functions does the family intervention coordinator perform?”
n [Read list of five tasks listed above.] “Is anyone responsible for these tasks?”
For consumers and family members: “What functions does [family intervention coordinator’s name] perform?”
Appendix D: Fidelity Scale Protocol 40 Evaluating Your Program
2. Session frequency
Definition: Families and consumers participate at least in biweekly FPE sessions.
Rationale: It is presumed that families benefit more if sessions are offered regularly and predictably.
Sources of information: Chart review, roster of sessions, and interviews with family intervention coordinator, FPE practitioners, consumers, and family members.
Item response coding: The primary evidence for coding this item would be attendance rosters or a calendar of scheduled events, if such documents exist. The program should have some way of documenting the frequency of FPE sessions. If the documentation suggests that the organization provides at least biweekly FPE sessions, code the item as “5.”
Probe questions
For family intervention coordinators:
n “How often are FPE sessions held for family members?”
n “Do you have attendance rosters, a calendar of events, or other documentation to verify this?”
For FPE practitioners:
n “How often are FPE sessions held for family members?”
n “Do you have attendance rosters, a calendar of events, or other documentation to verify this?”
For consumers and family members: “How often are FPE sessions held for family members?”
3. Long-term FPE
Definition: Families and consumers are provided with long-term FPE; specifically, at least one family member per consumer participates in FPE sessions for at least 9 months.
Rationale: In general, 9 months of biweekly equivalent FPE sessions are required for families and consumers to learn the necessary information and problem-solving skills. After completing the program, families and consumers may also benefit from booster sessions or support groups.
Sources of information: Chart review, roster of sessions, and interviews with the family intervention coordinator, FPE practitioners, consumers, and family members.
Item response coding: The primary evidence for coding this item would be a report containing the number of families and consumers completing FPE and how long they attended, records of duration of FPE groups, or attendance sheets. In the absence of written records, the assessment will depend on interviews. Excluding dropouts, if there is evidence that 90 percent or less of families receive at least 9 months of FPE sessions, code the item as “5.”
Probe questions
For family intervention coordinators or FPE practitioners:
n “How long do family members attend FPE before they graduate?”
n “Do you have attendance rosters, a calendar of events, or other documentation to verify this?”
For consumers and family members:
n “How long have you attended FPE sessions?”
n “How long do you intend to attend?”
Evaluating Your Program 41 Appendix D: Fidelity Scale Protocol
4. Quality of Practitioner-Consumer- Family Alliance
Definition: FPE practitioners engage family members and consumers with warmth, empathy, acceptance, and attention to each individual’s needs and desires.
Rationale: When the alliance between practitioners, consumers, and families is poor, family members and consumers are less likely to participate fully or at all in FPE programs and, as a result, are less likely to benefit from FPE services.
Sources of information: Interviews with FPE practitioners, family members, and consumers. Observations of FPE sessions.
Item response coding: The primary source for rating this item is direct observation. This item requires clinical judgment and is based on the fidelity assessor’s experience. Negative indicators would include comments in interviews, FPE sessions, or charts expressing judgmental or blaming attitudes. If sources consistently indicate a strong alliance for all FPE practitioners, code the item as “5.”
Probe questions
For FPE practitioners:
n “How do you establish rapport or develop an alliance with family members and consumers?”
n “How would you rate or describe your alliance with [family and consumer’s name]?” [Select one family and consumer with whom the practitioner works.]
n “Are there any family members or consumers with whom you feel your relationship is counterproductive or poor?”
For family members and consumers:
n “How would you describe your relationship with [FPE practitioner’s name]?”
n “Do you feel that [FPE practitioner’s name] has worked to establish a good relationship with you? What has he or she done to connect with you? What has he or she done that makes it more difficult for you to work with him or her?”
n “What would you change about your working relationship with [FPE practitioner’s name] to make it better?”
Appendix D: Fidelity Scale Protocol 42 Evaluating Your Program
5. Detailed Family Reaction
Definition: FPE practitioners identify and specify the family’s reaction to their relative’s mental illnesses. Reactions are emotional and behavioral responses (note the distinction from coping strategies in Item 8).
Rationale: A core principle of FPE is to help family members achieve a basic understanding of serious mental illnesses as well as to resolve family conflict by listening and responding sensitively to each family’s emotional distress related to having a relative with serious mental illnesses.
Sources of information: Chart review (especially treatment plan) and interviews with FPE practitioners, consumers, and families.
Item response coding: If documentation in the treatment plan and reports by FPE practitioners, consumers, and families corroborate that family reactions are identified and specified in joining sessions for 80 percent or more of involved families, code the item as “5.”
Probe questions
For practitioners:
n “What sorts of issues do you discuss in joining sessions?”
n “Do you address how families react emotionally or behaviorally to their relatives’ mental illnesses?”
n “What sorts of activities do you engage in to help them deal with their reactions?”
n Using a chart for a family member seen by the practitioner, ask the practitioner to explain the specifics.
For consumers and family members:
n “What sorts of issues did you discuss during the first couple of FPE sessions?”
n “Earlier in the FPE sessions, did you spend time discussing how you felt and reacted about the illness?”
n “Did the practitioner lead you in activities to help you deal with your feelings and reactions?”
6. Precipitating Factors
Definition: FPE practitioners, consumers, and families identify and specify precipitating factors for consumers’ mental illnesses.
Rationale: Exploring factors that have precipitated relapse in the past is a crucial step to developing individualized relapse prevention and illness management strategies. Involving consumers and families as equal partners in planning and delivering treatment is a core principle of FPE.
Sources of information: Chart review (especially treatment plan) and interviews with FPE practitioners, consumers, and families.
Item response coding: If documentation in the treatment plan and reports by FPE practitioners, consumers, and families corroborate that precipitating factors are identified and specified in joining sessions for 80 percent or more of involved families and consumers, code the item as “5.”
Probe questions
For FPE practitioners:
n “In joining sessions, do you discuss the precipitating factors of the illness with families and consumers?” [If yes, “Can you describe the process you use to discuss them? Can you show me examples?”]
n Using a chart, ask the FPE practitioner to explain the specifics.
For consumers and family members:
n “Earlier in the FPE sessions, did the FPE practitioner identify precipitating factors for [your or your relative’s] illness?” [If yes, “Please give examples.”]
n “Did you discuss how to respond to them once you notice these factors? Have you reviewed these strategies in later sessions?”
Evaluating Your Program 43 Appendix D: Fidelity Scale Protocol
7. Prodromal Signs and Symptoms
Definition: FPE practitioners, consumers, and families identify and specify prodromal signs and symptoms of consumers’ mental illnesses.
Rationale: Exploring consumers’ prodromal signs and symptoms is another crucial step to developing individualized relapse prevention and illness management strategies. Involving consumers and families as equal partners in planning and delivering treatment is a core principle of FPE.
Sources of information: Chart review (especially treatment plan) and interviews with FPE practitioners, consumers, and families.
Item response coding: If documentation in the treatment plan and reports by FPE practitioners, consumers, and families corroborate that prodromal signs and symptoms are identified and specified in joining sessions for 80 percent or more of involved families, code the item as “5.”
Probe questions
For FPE practitioners:
n “In joining sessions, do you identify prodromal symptoms with consumers and families?” [If yes, “Can you describe the process you use to identify them? Can you give an example?”]
n Using a chart, ask the practitioner to explain the specifics.
For consumers and family members:
n “Earlier in the FPE sessions, did the FPE practitioner discuss the signs that you (or your family member) may be becoming symptomatic?”
n “What sorts of things were suggested in your sessions for recognizing the early signs and symptoms of the illness? Please give examples. Have you reviewed these suggestions in later sessions?”
8. Coping Strategies
Definition: FPE practitioners identify, describe, clarify, and teach coping strategies. Coping strategies are intentional and thoughtful attempts to change behavior or symptoms related to mental illnesses (note the distinction from family reactions in Item 5).
Rationale: Exploring coping strategies that have and have not worked is a crucial step to developing individualized relapse prevention and illness management strategies. Insight into patterns of ineffective interactions and behaviors is likely to motivate consumers and families toward desired change.
Sources of information: Chart review (especially treatment plan) and interviews with FPE practitioners, consumers, and families.
Item response coding: If documentation in the treatment plan and reports by FPE practitioners, consumers, and families corroborate that practitioners help 80 percent or more of involved families and consumers to identify, describe, clarify, and learn coping strategies in joining sessions, code the item as “5.”
Probe questions
For FPE practitioners:
n “Do you identify coping strategies with consumers and families?” [If yes, “Can you describe the process you use?”]
n Using a chart, ask the FPE practitioner to explain the specifics.
For consumers and family members:
n “Have you discussed coping strategies? What sorts of things did you talk about?”
n “Did you discuss alternative ways of coping with [your or your relative’s] illness?”
Appendix D: Fidelity Scale Protocol 44 Evaluating Your Program
9. Educational Curriculum
Definition: FPE practitioners use a standardized curriculum to teach families about mental illnesses. The curriculum covers six topics:
n Psychobiology of the specific mental illness;
n Diagnosis;
n Treatment and rehabilitation;
n Impact of mental illness on the family;
n Relapse prevention; and
n Family guidelines.
Rationale: Effectively teaching families new information and skills requires structure and systematically using specific evidence- based techniques and strategies. Therefore, it is critical that an FPE program has a standardized educational curriculum that specifies what is taught and how it is taught.
Sources of information: Curriculum review, schedule of completed session, and interviews with family intervention coordinator, FPE practitioners, and families.
Item response coding: If 80 percent or more of involved families receive a standardized educational curriculum covering all six topics, code the item as “5.”
Probe questions
For family intervention coordinators:
n “Does your program have a standardized educational curriculum?” [If yes, “May I have a copy for review? How was it developed?”]
n “How do you ensure that the curriculum is followed? Do you periodically evaluate and update the curriculum? Do you have a schedule of completed sessions and their content?”
n Ask about each area listed above and whether they are included.
For FPE practitioners:
n “Do you use a standardized educational curriculum?” [If yes, “Are there any areas you teach differently from the curriculum?”]
n “Do you have a schedule of completed sessions and their content?”
n Ask about each area listed above and whether they are included.
For family members:
n Have you attended a 1-day educational workshop? [If yes, “What topics were covered?”]
n Ask about each area listed above.
n “Did the FPE practitioners review these educational topic areas with you individually or in a group session?”
Evaluating Your Program 45 Appendix D: Fidelity Scale Protocol
10. Multimedia Education
Definition: Consumers and family members are given educational materials about mental illnesses in several formats (for example, paper, video, and Web sites).
Rationale: Consumers and families benefit from receiving educational materials in a variety of formats. Some people may be more likely to watch a video or search a website than to read the same information in a document.
Sources of information: Review of educational materials and interviews with the family intervention coordinator, FPE practitioners, and families.
Item response coding: If educational materials are provided to families and consumers in all three formats, code the item as “5.”
Probe questions
For family intervention coordinators and FPE
practitioners:
n Ask to see the materials.
n “Do you provide educational materials to families and consumers? How many families and consumers on your caseload or in your FPE program have received educational materials?”
n “Can you give or show me examples or the types of materials that you give to families and consumers?”
For family members and consumers:
n What types of educational materials have you received through the FPE program?” [If they suggest only written materials have been provided, “Have you ever been offered or given videos, Web site addresses, or material in other formats?”
11. Structured Group Sessions
Definition: FPE practitioners adhere to a structured procedure that includes:
n Beginning socialization;
n Review the last session’s action plan;
n Go-round;
n Selection of a single problem;
n Structured problem-solving; and
n Ending with socialization.
Rationale: Families and consumers benefit from structured sessions that follow a predictable pattern. FPE practitioners should establish a clear agenda, goals, and expectations for each FPE session.
Sources of information: Observation of FPE multifamily group sessions and interviews with family intervention coordinator, FPE practitioners, consumers, and families.
Item response coding: If FPE multifamily group sessions include all six components listed above, code the item as “5.”
Probe questions
For family intervention coordinators
and FPE practitioners:
n Can you describe the typical FPE multifamily group session?”
For consumers and family members:
n “Can you describe what you do at the beginning of each multifamily group session? In the middle? At the end?”
n “Does the FPE practitioner seem to have a structured approach to each session?”
n “Is it clear to you what will be accomplished in each session?”
Appendix D: Fidelity Scale Protocol 46 Evaluating Your Program
12. Structured Problem-Solving
Definition: FPE practitioners use a standardized approach to help consumers and families with problem-solving, which includes:
n Define the problem;
n Generate solutions;
n Discuss the advantages and disadvantages of each solution;
n Choose the best solutions;
n Form an action plan; and
n Review the action plan.
Rationale: Studies show that collaborative and structured problem-solving techniques involving setting realistic goals and priorities and breaking goals into small behavioral steps are effective in improving consumers’ functioning and families’ coping.
Sources of information: Observation of FPE multifamily group sessions and interviews with family intervention coordinator, FPE practitioners, consumers, and families.
Item response coding: If all six components of structured problem-solving were used, code the item as “5.”
Probe questions
For family intervention coordinators and FPE
practitioners:
n “Do you focus on problem-solving in multifamily groups?” [If yes, “What strategies do you use? Do you follow the same process during every session?”]
n Listen for the list of six components given above. If a component is omitted, probe for whether it is included.
For the family members and consumers:
n In the multifamily groups, do you discuss how to address problems that may arise?” [If yes, “What sorts of activities do you do in the sessions to work on problems you may be having? Do you ever generate plans of action? Is it a step-by-step procedure? Can you describe the steps?”]
Evaluating Your Program 47 Appendix D: Fidelity Scale Protocol
13. Stage-Wise Provision of Services
Definition: FPE services are provided in the following order:
1. Engagement;
2. Three or more joining sessions;
3. Educational workshop; and
4. Multifamily group.
Rationale: FPE is most effective if all components of the evidence-based model are followed in order. Effective FPE programs ensure that consumers and families are well informed about the practice, establish a strong working alliance, receive a standardized educational curriculum, and develop clear treatment goals before entering into the multifamily group.
Sources of information: Chart review and interviews with family intervention coordinator, FPE practitioners, consumers, and families.
Item response coding: If sources corroborate that engagement, joining sessions, and the educational workshop are completed in a step-wise manner before entering into the multifamily group, code the item as “5.”
Probe questions
For family intervention coordinators and FPE
practitioners:
n “How do you engage consumers and families who would benefit from FPE?”
n “Do you provide joining sessions for consumers and families?” [If yes, “How many joining sessions has each consumer and family on your caseload had? What kind of topics do you cover in your joining sessions?”]
n “Did you offer a 1-day educational workshop? When was it offered? How many consumers and families attended? Did all the attendees complete three or more joining sessions before participating in the workshop?”
n “When did the multifamily group begin? Did all group participants complete three or more joining sessions and participate in the workshop before the group began?”
For consumers and family members:
n Ask if he or she has received each of the four services. Probe further about the timeframe and content of each service.
n “Did you feel that you had a good understanding of FPE before the multifamily group began?”
Appendix D: Fidelity Scale Protocol 48 Evaluating Your Program
14. Assertive Engagement and Outreach
Definition: FPE practitioners assertively engage all potential consumers and family members by phone, by mail, or in person (in the agency or in the community) on an ongoing basis.
Rationale: All consumers and families who may benefit from FPE should be educated about the practice so that they can make informed decisions about participation. Effective FPE programs are flexible in meeting the needs of individual families and consumers and use a variety of means for reaching out to them. Assertive engagement and outreach is also crucial in overcoming barriers to participation such as stigma and hopelessness.
Sources of information: Chart review and interviews with family intervention coordinator, FPE practitioners, consumers, and families.
Item response coding: If FPE practitioners actively engage all potential consumers and family members through all necessary means on an ongoing basis, code the item as “5.”
Probe questions
For family intervention coordinators and FPE
practitioners:
n “How do you engage consumers and families who would benefit from FPE?”
n “How do you engage hard-to-reach consumers and family members? For example, some consumers may not have a phone number to contact. Or, you may not be able to reach some family members during your office hours because they work.”
n “What would you do if a consumer or a family member told you he or she was not ready for FPE?”
n “What do you do with families who don’t show up for treatment? What about families who drop out of treatment? How do you engage or re-engage these families?”
For consumers and family members:
n “How did you come to participate in this FPE program? Did the program do a good job in helping you understand FPE, explore your expectations about the program, and make an informed decision about participating?”
n “Have you ever felt discouraged or ambivalent about participating in FPE or stopped showing up for sessions?” [If yes, “What did the FPE practitioner do to re-engage you in FPE?”]
n “How do you feel about the availability of your FPE practitioner? Do you feel that your practitioner actively reaches out to you?” [If yes, “How does he or she do so?”]
Evaluating Your Program 49 Appendix E: General Organizational Index
Evaluating Your Program
Appendix E: General Organizational Index and Score Sheet
Evaluating Your Program 51 Appendix E: General Organizational Index
General Organizational Index
1 2 3 4 5
G1. Program philosophy
Committed to clearly articulated philosophy consistent with specific evidence-based practice (EBP) model, based on these five sources:
n Program leader
n Senior staff (for example, executive director, psychiatrist)
n Practitioners providing the EBP
n Consumers and families receiving EBP
n Written materials (for example, brochures)
No more than 1 of 5 sources shows clear understanding of program philosophy.
OR
All sources have numerous major areas of discrepancy.
2 of 5 sources show clear understanding of program philosophy.
OR
All sources have several major areas of discrepancy.
3 of 5 sources show clear understanding of program philosophy.
OR
Sources mostly aligned to program philosophy, but have 1 major area of discrepancy.
4 of 5 sources show clear understanding of program philosophy.
OR
Sources mostly aligned to program philosophy, but have 1 or 2 minor areas of discrepancy.
All 5 sources show clear understanding and commitment to program philosophy for specific EBP.
*G2. Eligibility or consumer
identification
All consumers with serious mental illnesses in the community support program, crisis consumers, and institutionalized consumers are screened to determine if they qualify for EBP using standardized tools or admission criteria consistent with EBP. Also, agency systematically tracks number of eligible consumers.
20% of consumers receive standardized screening and/ or agency DOES NOT systematically track eligibility.
21–40% of consumers receive standardized screening and agency systematically tracks eligibility.
41–60% of consumers receive standardized screening and agency systematically tracks eligibility.
61–80% of consumers receive standardized screening and agency systematically tracks eligibility.
>80% of consumers receive standardized screening and agency systematically tracks eligibility.
*G3. Penetration
Maximum number of eligible consumers served by EBP, as defined by the ratio:
Number of consumers receiving EBP Number of consumers eligible for EBP
Ratio .20 Ratio .21 – .40 Ratio .41 – .60 Ratio .61 – .80 Ratio > .80
* These two items coded based on all consumers with serious mental illnesses at the site or sites where EBP is being implemented; all other items refer specifically to those receiving the EBP.
Total number of consumers in target population
Total number of consumers eligible for EBP % % eligible:
Total number of consumers receiving EBP Penetration rate
Appendix E: General Organizational Index 52 Evaluating Your Program
1 2 3 4 5
G4. Assessment Assessments are completely absent or completely non-standardized.
Pervasive deficiencies in 2 of the following:
n Standar di zation;
n Quality of assessments;
n Timeliness; and
n Comprehen- siveness.
Pervasive deficiencies in 1 of the following:
n Standar di zation;
n Quality of assessments;
n Timeliness; and
n Comprehen- siveness.
61%-80% of consumers receive standardized, high-quality assessments at least annually.
OR
Information is deficient for 1 or 2 assessment domains.
More than 80% of consumers receive standardized, high-quality assessments, the information is comprehensive across all assessment domains, and it is updated at least annually.
Full standardized assessment of all consumers who receive EBP services. Assessment includes the following:
n History and treatment of medical, psychiatric, substance use disorders
n Current stages of all existing disorders
n Vocational history
n Any existing support network
n Evaluation of biopsychosocial risk factors
G5. Individualized treatment plan 20% of consumers EBP serves have explicit individualized treatment plan, related to EBP, updated every 3 months.
21–40% of consumers EBP serves have explicit individualized treatment plan, related to EBP, updated every 3 months.
41–60% of consumers EBP serves have explicit individualized treatment plan, related to EBP, updated every 3 months.
OR
Individualized treatment plan updated every 6 months for all consumers.
61–80% of consumers EBP serves have explicit individualized treatment plan, related to EBP, updated every 3 months.
More than 80% of consumers EBP serves have explicit individualized treatment plan related to EBP, updated every 3 months.
For all EBP consumers, an explicit, individualized treatment plan exists related to the EBP that is consistent with assessment and updated every 3 months
G6. Individualized treatment 20% of consumers EBP serves receive individualized services meeting goals of EBP.
21–40% of consumers EBP serves receive individualized services meeting goals of EBP.
41–60% of consumers EBP serves receive individualized services meeting goals of EBP.
61–80% of consumers EBP serves receive individualized services meeting goals of EBP.
More than 80% of consumers EBP serves receive individualized services meeting goals of EBP.
All EBP consumers receive individualized treatment meeting goals of EBP
G7. Training 20% of program staff receive standardized training annually.
21–40% of program staff receive standardized training annually.
41–60% of program staff receive standardized training annually.
61–80% of program staff receive standardized training annually.
More than 80% of program staff receive standardized training annually.
All new program staff receive standardized training in EBP (at least a 2-day workshop or equivalent) within 2 months after hiring. Existing program staff receive annual refresher training (at least 1-day workshop or equivalent).
G8. Supervision 20% of EBP practitioners receive supervision.
21–40% of EBP practitioners receive weekly structured, consumer- centered supervision.
OR
All EBP practitioners receive informal supervision.
41–60% of EBP practitioners receive weekly structured, consumer- centered supervision.
OR
All EBP practitioners receive monthly supervision.
61–80% of EBP practitioners receive weekly structured, consumer- centered supervision.
OR
All EBP practitioners receive supervision 2 times a month.
More than 80% of EBP practitioners receive structured weekly supervision, focusing on specific consumers, in sessions that explicitly address EBP model and its application.
EBP practitioners receive structured, weekly supervision (group or individual format) from a supervisor experienced in particular EBP. Supervision should be consumer-centered and explicitly address EBP model and its application to specific consumer situations.
Evaluating Your Program 53 Appendix E: General Organizational Index
1 2 3 4 5
G9. Process monitoring No attempt at process monitoring is made.
Informal process monitoring is used at least annually.
Process monitor- ing is deficient on 2 of these 3 criteria:
n Comprehen- sive and
standardized;
n Completed every 6 months; and
n Used to guide program improvements.
OR
Standardized monitoring done annually only.
Process monitoring is deficient on 1 of these 3 criteria:
n Comprehen- sive and
standardized;
n Completed every 6 months; and
n Used to guide program improvements.
Standardized comprehensive process monitoring occurs at least every 6 months and is used to guide program improvements.
Program leaders and administrators monitor process of implementing EBP every 6 months and use the data to improve the program. Monitoring involves a standardized approach, for example, using fidelity scale or other comprehensive set of process indicators.
G10. Outcome monitoring No outcome monitoring occurs.
Outcome monitoring occurs at least 1 time a year, but results are not shared with EBP practitioners.
Standardized outcome moni- toring occurs at least 1 time a year. Results are shared with EBP practitioners.
Standardized outcome monitoring occurs at least 2 times a year. Results are shared with EBP practitioners.
Standardized outcome monitoring occurs quarterly. Results are shared with EBP practitioners.
Program leaders and administrators monitor outcomes for EBP consumers every 3 months and share data with EBP practitioners. Monitoring involves standardized approach to assessing a key outcome related to EBP, for example, psychiatric admissions, substance abuse treatment scale, or employment rate.
G11. Quality Assurance (QA) No review or no committee.
QA committee has been formed, but no reviews have been completed.
Explicit QA review occurs less than annually.
OR
QA review is superficial.
Explicit QA review occurs annually
Explicit review occurs every 6 months by QA group or steering committee for EBP
Agency has QA committee or implementation steering committee with an explicit plan to review EBP or components of the program every 6 months.
G12. Consumer choice about service provision
Consumer- centered services are absent (or practitioners make all EBP decisions).
Few sources agree that type and frequency of EBP services reflect consumer choice
Half of the sources agree that type and frequency of EBP services reflect consumer choice.
Most sources agree that type and frequency of EBP services reflect consumer choice.
OR
Agency fully embraces consumer choice with 1 exception.
All sources agree that type and frequency of EBP services reflect consumer choice.
All consumers receiving EBP services are offered choices; EBP practitioners consider and abide by consumer preferences for treatment when offering and providing services. Score Sheet: General Organizational Index
Appendix E: General Organizational Index 54 Evaluating Your Program
Score Sheet: General Organizational Index
Date of visit
Agency name
Assessors’ names
Assessor 1 Assessor 2 Consensus
G1 Program philosophy
G2 Eligibility or consumer identification
G3 Penetration
G4 Assessment
G5 Individualized treatment plan
G6 Individualized treatment
G7 Training
G8 Supervision
G9 Process monitoring
G10 Outcome monitoring
G11 Quality Assurance (QA)
G12 Consumer choice regarding service provision
Total mean score
/ /
Evaluating Your Program 55 Appendix F: GOI Protocol
Evaluating Your Program
Appendix F: General Organizational Index Protocol
Evaluating Your Program 57 Appendix F: GOI Protocol
General Organizational Index Protocol
The General Organizational Index Protocol explains how to rate each item of the index. In particular, it provides the following:
n A definition and rationale for each item; and
n A list of data sources most appropriate for each fidelity item (for example, chart review, program leader, practitioners, consumers, and family interviews).
When appropriate, a set of probe questions is provided to help you elicit the critical information needed to code the fidelity item. These probe questions were specifically generated to help you collect information from respondents that is relatively free from bias, such as social desirability.
Decision rules will help you code each item correctly. As you collect information from various sources, these rules will help you determine the specific rating to give for each item.
G1. Program Philosophy
Definition: The program is committed to a clearly articulated philosophy consistent with the specific evidence-based practice (EBP), based on the following five sources:
n Family intervention coordinator;
n Senior staff (for example, executive director, psychiatrists);
n FPE practitioners;
n Consumers and family members; and
n Written materials (for example, brochures).
Rationale: In agencies that truly endorse EBPs, staff members at all levels embrace the program philosophy and practice it in their daily work.
Sources of information:
Overview: During the site visit, be alert to indicators of program philosophy consistent or inconsistent with the EBP, including observations from casual conversations, staff and consumer activities, etc. Statements that suggest misconceptions or reservations about the practice are negative indicators, while statements that show enthusiasm for and understanding of the practice are positive indicators.
The intent of this item is to gauge the understanding of and commitment toward the practice. It is not necessary that every element of the practice is currently in place (this is gauged by the EBP-specific fidelity scale), but rather whether all those involved are committed to implementing a high-fidelity EBP.
The practitioners rated for this item are limited to those implementing this practice. Similarly, the consumers rated are those receiving the practice.
1. Family intervention coordinator, senior staff,
and practitioner interviews
At the beginning of the interview, have practitioners briefly describe the program.
n “What are the critical ingredients or principles of your services?”
n “What is the goal of your program?”
n “How do you define [EBP area]?”
2. Consumer interview
n “What kind of services do you receive from this program?”
n Using a layperson’s language, describe to the consumer or family the principles of the specific EBP area. [Probe if the program offers services that reflect each principle.]
n “Do you feel the practitioners of this program are competent and help you address your problems?”
3. Written material review (for example, brochure)
n Does the site have written materials on the EBP? If not, then rate item down one scale point (i.e., lower fidelity).
n Does the written material articulate a program philosophy that is consistent with the EBP?
Appendix F: GOI Protocol 58 Evaluating Your Program
Item response coding: The goal of this item is not to quiz every practitioner to determine if each can recite every critical ingredient. Rather, the goal is to gauge whether the understanding is generally accurate and not contrary to the EBP. For example, if a senior staff member says, “We are having trouble identifying consumers for our FPE program since most families are unsupportive,” then that would be a red flag for the practice of FPE.
If all sources show evidence that they clearly understand the program philosophy, code the item as “5.” For a source type that is based on more than one person (for example, practitioner interviews) determine the majority opinion when rating whether that source endorses a clear program philosophy. Note: If no written material exists, then count that source as unsatisfactory.
G2. Eligibility/Consumer Identification
Definition: For EBPs implemented in a mental health center: All consumers in the community support program, consumers in crisis, and those in the hospital are screened using standardized tools or admission criteria that are consistent with the EBP.
For EBPs implemented in a service area: All consumers within the jurisdiction of the service area are screened using standardized tools or admission criteria that are consistent with the EBP. For example, in New York, county mental health administrations are responsible for identifying consumers who will be served by Assertive Community Treatment (ACT) programs.
The target population refers to all adults with serious mental illness (SMI) served by the provider agency or service area. If the agency serves consumers at multiple sites, then assessment is limited to the site or sites that are targeted for the EBP. If the target population is served in discrete programs (for example, case management, residential, day treatment), then ordinarily all adults with serious mental illnesses are included in this definition.
Screening will vary according to the EBP. The intent is to identify all who could benefit from the EBP. In every case, the program should have an explicit, systematic method for identifying the eligibility of every consumer. Screening typically occurs at program admission; programs that are newly adopting an EBP should have a plan for systematically reviewing consumers who are already active in the agency.
Rationale: Accurately identifying consumers who would benefit most from the EBP requires routinely reviewing eligibility, based on criteria that are consistent with the EBP.
Sources of information:
1. Family intervention coordinator, senior staff,
and practitioner interviews
n “Describe the eligibility criteria for your program.”
n “How are consumers referred to your program? How does the agency identify consumers who would benefit from your program? Do all new consumers receive screening for substance abuse or severe mental illness (SMI) diagnosis?”
n “What about consumers who are in crisis (or institutionalized)?”
n Ask for a copy of the screening instrument that the agency uses.
2. Chart review
Review documentation of the screening process and results.
3. County mental health administrators (where applicable)
If eligibility is determined at the service-area level (such as the New York example), then interview the people who are responsible for this screening.
Item response coding: This item refers to all consumers with SMI in the community support program or its equivalent at the sites where the EBP is being implemented; it is not limited to consumers who receive EBP services only. Calculate this percentage and record it on the fidelity scale in the space provided. If 80 percent or more of these consumers receive standardized screening, code the item as “5.”
Evaluating Your Program 59 Appendix F: GOI Protocol
G3. Penetration
Definition: Penetration is defined as the percentage of consumers who have access to an EBP as measured against the total number of consumers who could benefit from the EBP. Numerically, this proportion is defined by––
Number of consumers receiving an EBP
Number of consumers eligible for the EBP
As in the preceding item, the numbers used in this calculation are specific to the site or sites where the EBP is being implemented.
Rationale: Surveys have repeatedly shown that people with SMI often have a limited access to EBPs. The goal of EBP dissemination is not simply to create small exclusive programs, but to make these practices easily accessible within the public mental health system.
Sources of information:
The calculation of the penetration rate depends on the availability of the two statistics defining this rate.
Numerator: The number receiving the service is based on a roster of names that the family intervention coordinator maintains. Ideally, this total should be corroborated with service contact sheets and other supporting evidence that the identified consumers are
actively receiving treatment. As a practical matter, agencies have many conventions for defining active consumers and dropouts, so that it may be difficult to standardize the definition for this item. Use the best estimate of the number actively receiving treatment.
Denominator: If the agency systematically tracks eligibility, then use this number in the denominator. (See the rules listed in G2 to determine the target population before using estimates below.) If the agency doesn’t track eligibility, then estimate the denominator by multiplying the total target population by the corresponding percentage based on the literature for each EBP.
According to the literature, the estimates for EBP KITs available at this writing should be as follows:
n Integrated Treatment for Co-Occurring Disorders––40 percent
n Supported Employment––60 percent
n Illness Management and Recovery––100 percent
n Family Psychoeducation––100 percent (some kind of significant other)
n Assertive Community Treatment––20 percent
Item response coding: Calculate this ratio and record it on the fidelity scale. If the program serves more than 80 percent of eligible consumers, code the item as “5.”
Appendix F: GOI Protocol 60 Evaluating Your Program
G4. Assessment
Definition: All EBP consumers receive standardized, high-quality, comprehensive, and timely assessments.
Standardization refers to a reporting format that is easily interpreted and consistent across consumers.
High quality refers to assessments that provide concrete, specific information that differentiates among consumers. If most consumers are assessed using identical words or if the assessment consists of broad, noninformative checklists, then consider this to be low quality.
Comprehensive assessments include the following:
n History and treatment of medical, psychiatric, and substance use disorders;
n Current stages of all existing disorders;
n Vocational history;
n Any existing support network; and
n Evaluation of biopsychosocial risk factors.
Timely assessments are those updated at least annually.
Rationale: Comprehensive assessment or re- assessment is indispensable in identifying target domains of functioning that may need intervention, in addition to consumers’ progress toward recovery.
Sources of information:
1. Family intervention coordinator, senior staff, and
practitioner interviews
n “Do you give a comprehensive assessment to new consumers? What are the components thatxyou assess?”
n Ask for a copy of the standardized assessment form, if available, and have practitioners go through the form.
n “How often do you re-assess consumers?”
2. Chart review
n Look for comprehensiveness of assessment by looking at multiple completed assessments to see if they address each component of the comprehensive assessment every time an assessment is performed.
n “Is the assessment updated at least yearly?”
Item response coding: If more than 80 percent of consumers receive standardized, high-quality, comprehensive, and timely assessments, code the item as “5.”
Evaluating Your Program 61 Appendix F: GOI Protocol
G5. Individualized Treatment Plan
Definition: For all EBP consumers, an explicit, individualized treatment plan exists (even if it is not called this) related to the EBP that is consistent with assessment and updated every 3 months.
Individualized means that goals, steps to reaching the goals, services and interventions, and intensity of involvement are unique to this consumer. Plans that are the same or similar across consumers are not individualized. One test is to place a treatment plan without identifying information in front of supervisors to see if they can identify the consumer.
Rationale: Core values of EBP include individualizing services and supporting consumers’ pursuit of their goals and progress in their recovery at their own pace. Therefore, treatment plans need ongoing evaluation and modification.
Sources of information:
Note: Assess this item and the next together; that is, ask questions about specific treatment plans along with questions about the treatment.
1. Chart review (treatment plan)
Using the same charts as examined during the EBP-specific fidelity assessment, look for documentation of specific goals and consumer- based, goal-setting process.
n “Are the treatment recommendations consistent with assessment?”
n “What evidence is used for a quarterly review?”
2. Family intervention coordinator interview
“Describe the process of developing a treatment plan. What are the critical components of a typical treatment plan and how are they documented?”
3. Practitioner interview
When feasible, use the specific charts selected above. Ask practitioners to go over a sample treatment plan.
n “How do you come up with consumer goals?” [Listen for consumer involvement and individualization of goals.]
n “How often do you review (or follow up on) the treatment plan?”
4. Consumer interview
n “What are your goals in this program? How did you set these goals?”
n “Do you and your practitioners together review your progress toward achieving your goals?” [If yes, “How often? Please describe the review process.”]
5. Team meeting and supervision observation, if available
Observe how the treatment plan is developed. Listen especially for discussion of assessment, consumer preferences, and individualization of treatment. Do they review treatment plans?
Item response coding: If more than 80 percent of EBP consumers have an explicit, individualized treatment plan that is updated every 3 months, code the item as “5.”
If the treatment plan is individualized but updated only every 6 months, code the item as “3.”
Appendix F: GOI Protocol 62 Evaluating Your Program
G6. Individualized Treatment
Definition: All EBP consumers receive individualized treatment meeting the goals of the EBP.
Individualized treatment means that steps, strategies, services, interventions, and intensity of involvement are focused on specific consumer goals and are unique for each consumer. Progress Notes are often a good source of what really goes on. Treatment could be highly individualized, despite the presence of generic treatment plans.
An example of a low score on this item for Integrated Treatment of Co-Occurring Disorders is the following:
If consumers in the engagement phase of recovery are assigned to a relapse prevention group and are constantly told they need to quit using, rather than using motivational interventions.
Rationale: The key to the success of an EBP is implementing a plan that is individualized and meets the goals for the EBP for each consumer.
Sources of information:
1. Chart review (treatment plan)
Using the same charts as examined during the EBP-specific fidelity assessment, examine the treatment provided. Limit the focus to a recent treatment plan related to the EBP. Judge whether an appropriate treatment occurred during the time frame indicated by the treatment plan.
2. Practitioner interview
When feasible, use the specific charts selected above. Ask practitioners to go over a sample treatment plan and treatment.
3. Consumer interview
“Tell me about how this program is helping you meet your goals.”
Item response coding: If more than 80 percent of EBP consumers receive treatment that is consistent with the goals of the EBP, code the item as “5.”
G7. Training
Definition: All new practitioners receive standardized training in the EBP (at least a 2-day workshop or its equivalent) within 2 months after they are hired. Existing practitioners receive annual refresher training (at least a 1-day workshop or its equivalent).
Rationale: Practitioner training and retraining are warranted to ensure that evidence-based services are provided in a standardized manner, across practitioners and over time.
Sources of information:
1. Family intervention coordinator, senior staff,
and practitioner interviews
n “Do you provide new practitioners with systematic training for [EBP area]?” [If yes, probe for specifics: Mandatory or optional? Length? Frequency? Content? Group or individual format? Who trains? In-house or outside training?]
n “Do practitioners receive refresher trainings?” [If yes, probe for specifics.]
2. Review training curriculum and schedule, if available
Does the curriculum appropriately cover the critical ingredients for [EBP area]?
3. Practitioners interview
n “When you first started in this program, did you receive a systematic and formal training for [EBP area]?” [If yes, probe for specifics: Mandatory or optional? Length? Frequency? Content? Group or individual format? Who trains? In-house or outside training?]
n “Do you receive refresher trainings?” [If yes, probe for specifics.]
Item response coding: If more than 80 percent of practitioners receive at least yearly, standardized training for [EBP area], code the item as “5.”
Evaluating Your Program 63 Appendix F: GOI Protocol
G8. Supervision
Definition: FPE practitioners receive structured, weekly supervision from a supervisor experienced in the particular EBP. The supervision can be either group or individual, but CANNOT be peers-only supervision without a supervisor. The supervision should be consumer-centered and explicitly address the EBP model and how it applies to specific consumer situations. Administrative meetings and meetings that are not specifically devoted to the EBP do not fit the criteria for this item. The consumer-specific EBP supervision should be at least 1 hour each week.
Rationale: Regular supervision is critical not only for individualizing treatment, but also for ensuring the standardized provision of evidence-based services.
Sources of information:
1. Family intervention coordinator, senior staff,
and practitioner interviews
Probe for logistics of supervision: length, frequency, group size, etc.
n “Describe what a typical supervision session looks like.”
n “How does the supervision help your work?”
2. Team meeting and supervision observation, if available
Listen for discussion of [EBP area] in each case reviewed.
3. Supervision logs documenting frequency of meetings
Item response coding: If more than 80 percent of FPE practitioners receive weekly supervision, code the item as “5.”
G9. Process Monitoring
Definition: Family intervention coordinators and administrators monitor the process of implementing the EBP every 6 months and use the data to improve the program. Process monitoring involves a standardized approach, for example, using a fidelity scale or other comprehensive set of process indicators.
An example of a process indicator would be a systematic measurement of how much time case managers spend in the community instead of in the office. Process indicators could include items related to training or supervision. The underlying principle is that whatever is being measured is related to implementing the EBP and is not being measured to track billing or productivity.
Rationale: Systematically and regularly collecting process data is imperative in evaluating program fidelity to EBP.
Sources of information:
1. Family intervention coordinator, senior staff,
and practitioners interviews
n “Does your program collect process data regularly?” [If yes, probe for specifics. Frequency? Who? How (using [EBP area] fidelity scale vs. other scales)? etc.]
n “Does your program collect data on consumer service use and treatment attendance?”
n “Have the process data affected how your services are provided?
2. Review of internal reports and documentation,
if available
Item response coding: If evidence exists that standardized process monitoring occurs at least every 6 months, code the item as “5.”
Appendix F: GOI Protocol 64 Evaluating Your Program
G10. Outcome Monitoring
Definition: Family intervention coordinators and administrators monitor the outcomes of EBP consumers every 3 months and share the data with FPE practitioners in an effort to improve services. Outcome monitoring involves a standardized approach to assessing consumers.
Rationale: Systematically and regularly collecting outcomes data is imperative in evaluating program effectiveness. Effective programs also analyze such data to ascertain what is working and what is not working and use the results to improve the quality of services they provide.
Key outcome indicators for each EBP are discussed in the EBP KITs. A provisional list is as follows:
n Integrated Treatment for Co-Occurring
Disorders—substance use (such as the Stages of Treatment Scale);
n Supported Employment—competitive employment rate;
n Illness Management and Recovery— hospitalization rates, relapse prevention plans, medication compliance rates;
n Family Psychoeducation—hospitalization and family well-being; and
n Assertive Community Treatment— hospitalization and housing.
Sources of information:
1. Family intervention coordinator, senior staff,
and practitioner interviews
n “Does your program have a systematic method for tracking outcomes data?” [If yes, probe for specifics: How (computerized vs. chart only)? How often? Type of outcome variables? Who collects data?]
n “Do you use any checklist or scale to monitor consumer outcome (for example, Substance Abuse Treatment Scale)?”
n “What do you do with the outcomes data? Do your practitioners review the data regularly?” [If yes, “How is the review done (for example, cumulative graph)?”]
n “Have the outcomes data affected how your services are provided?” [If yes, “How?”]
2. Review of internal reports and documentation,
if available
Item response coding: If standardized outcome monitoring occurs quarterly and results are shared with FPE practitioners, code the item as “5.”
Evaluating Your Program 65 Appendix F: GOI Protocol
G11. Quality Assurance
Definition: The agency’s quality assurance (QA) committee has an explicit plan to review the EBP or components of the program every 6 months. The steering committee for the EBP can serve this function.
Good QA committees help the agency in important decisions, such as penetration goals, placement of the EBP within the agency, and hiring and staffing needs. QA committees also help guide and sustain the implementation by doing the following:
n Reviewing fidelity to the EBP model;
n Making recommendations for improvement;
n Advocating and promoting the EBP within the agency and in the community; and
n Deciding on and keeping track of key outcomes relevant to the EBP.
Rationale: Research has shown that programs that most successfully implement EBPs have better outcomes. Again, systematically and regularly collecting process and outcomes data is imperative in evaluating program effectiveness.
Sources of information:
1. Family intervention coordinator interview
“Does your agency have an established team or committee that is in charge of reviewing the components of your [EBP area] program?” [If yes, probe for specifics. “Who? How? When?”]
2. QA committee member interview
n “Please describe the tasks and responsibilities of the QA committee.” [Probe for specifics. “What is the purpose? Who? How? When?”]
n “How do you use your reviews to improve the program’s services?”
Item response coding: If the agency has an established QA or steering committee that reviews the EBP or components of the program every 6 months, code the item as “5.”
Appendix F: GOI Protocol 66 Evaluating Your Program
G12. Consumer Choice About Service Provision
Definition: All consumers who receive EBP services are offered a reasonable range of choices consistent with the EBP; practitioners consider and abide by consumer preferences for treatment when they offer and provide services.
Choice is defined narrowly in this item to refer to services provided. This item does not address broader issues of consumer choice such as choosing to engage in self- destructive behaviors.
To score high on this item, it is not sufficient that a program offers choices. The choices must be consonant with the EBP. So, for example, an agency implementing Integrated Treatment for Co-Occurring Disorders would score low if it only worked with consumers who were abstinent.
A reasonable range of choices means that FPE practitioners offer realistic options to consumers rather than prescribing only one or a couple of choices or dictating a fixed sequence or prescribing conditions that consumers must complete before becoming eligible for a service.
Examples of Relevant Choices by EBPs
Current at this writing
Integrated Treatment for Co-Occurring Disorders
n Group or individual counseling sessions
n Frequency of treatment
n Specific self-management goals
n Selection of other supporters to be involved
Supported Employment
n Type of occupation
n Type of work setting
n Schedules of work and number of hours
n Whether to disclose
n Nature of accommodations
n Type and frequency of followup supports
Family Psychoeducation
n Consumer readiness for involving family
n Who to involve
n Choice of problems and issues to work on
Illness Management and Recovery
n Selection of other supporters to be involved
n Specific self-management goals
n Nature of behavioral tailoring
n Skills to be taught
Assertive Community Treatment
n Type and location of housing
n Nature of health promotion
n Nature of assistance with financial management
n Specific goals
n Daily living skills to be taught
n Nature of medication support
n Nature of substance abuse treatment
Evaluating Your Program 67 Appendix F: GOI Protocol
Rationale: A major premise of EBP is that consumers are capable of playing a vital role in managing their illnesses and in making progress towards achieving their goals. Providers accept the responsibility for getting information to consumers so that they can more effectively participate in treatment.
Sources of information:
1. Family intervention coordinator interview
n “Tell us what your program philosophy is about consumer choice. How do you incorporate consumers’ preferences in the services you provide?”
n “What options exist for your services? Give examples.”
2. Practitioner interview
n “What do you do when a disagreement occurs between what you think is the best treatment for consumers and what they want?”
n “Describe a time when you were unable to abide by a consumer’s preferences.”
3. Consumer interview:
n “Does the program give you options for the services you receive?”
n Are you receiving the services you want?”
4. Team meeting and supervision observation
Look for discussion of service options and consumer preferences.
5. Chart review (especially treatment plan)
Look for documentation of consumer preferences and choices.
Item response coding: If all sources support that type and frequency of EBP services always reflect consumer choice, code the item as “5.”
If the agency embraces consumer choice fully except in one area (for example, requiring the agency to assume representative payeeships for all consumers), then code the item as “4.”
Note: Ratings for both scales are based on current behavior and activities, not planned or intended behavior.
The standards used for establishing the anchors for the fully implemented ratings were determined through a variety of expert sources as well as empirical research.
Evaluating Your Program 69 Appendix G: Outcomes Report Form
Evaluating Your Program
Appendix G: Outcomes Report Form
Evaluating Your Program 71 Appendix G: Outcomes Report Form
Outcomes Report Form
Quarter q January, February, March Year q April, May, June q July, August, September q October, November, December Reported by Agency Team
About the consumer
Consumer ID Discharge date Date of birth
q Male Ethnicity q Female Primary diagnosis
What was the consumer’s evidence-based service status on the last day of the quarter?
Unknown Not Eligible Eligible Enrolled
Integrated Treatment for Co-Occurring Disorders q q q q
Supported Employment q q q q
Assertive Community Treatment q q q q
Illness Management and Recovery q q q q
Family Psychoeducation q q q q
In the past 3 months, how often has the consumer… Number of days Number of incidents
Been homeless?
Been incarcerated?
Been in a State psychiatric hospital?
Been in a private psychiatric hospital?
Been hospitalized for substance abuse reasons?
/ / / /
Appendix G: Outcomes Report Form 72 Evaluating Your Program
In the past 3 months, how many days was the consumer competitively employed? (Use 0 if the consumer has not been competitively employed.)
_____________ Days
Was the consumer competitively employed on the last day
of the reporting period?
q Yes
q No
What was the consumer’s stage of substance abuse treatment
on the last day of the quarter? Check one.
q Not applicable
q Pre-engagement
q Engagement
q Early persuasion
q Late persuasion
q Early active treatment
q Late active treatment
q Relapse prevention
q In remission or recovery
What was the consumer’s living arrangement on the last day
of the quarter? Check one.
q Not applicable or unknown
q Psychiatric hospital
q Substance abuse hospitalization
q General hospital psychiatric ward
q Nursing home
q Family care home
q Living with relatives (heavily dependent for personal care)
q Group home
q Boarding house
q Supervised apartment program
q Living with relatives (but is largely independent)
q Living independently
q Homeless
q Emergency shelter
q Other (specify): ____________________________
What was the consumer’s educational status on the last day
of the quarter? Check one.
q Not applicable or unknown
q No educational participation
q Avocational/Educational involvement
q Pre-educational explorations
q Working on General Educational Development (GED) diploma
q Working on English as Second Language
q Basic educational skills
q Attending vocational school, vocational program, apprenticeship, or high school
q Attending college: 1 to 6 hours
q Attending college: 7 or more hours
q Other (specify): ____________________________
What is the consumer’s highest level of education? Check one.
q No high school
q High school diploma or General Educational Development (GED) diploma
q Some college
q Associate degree
q Vocational training certificate
q Bachelor of Arts or Bachelor of Science
q Master’s degree or Ph.D.
Evaluating Your Program 73 Appendix H: Instructions for the Outcomes Report Form
Evaluating Your Program
Appendix H: Instructions for the Outcomes Report Form
Evaluating Your Program 75 Appendix H: Instructions for the Outcomes Report Form
Instructions for the Outcomes Report Form
Before you fill out the Outcomes Report Form, become familiar with the definitions of the data elements to provide consistency among reporters.
Evidence-based service status
What was the consumer’s evidence-based service
status on the last day of the quarter? Check the appropriate boxes according to these definitions:
Eligible: Does the consumer meet the participation criteria for a specific EBP? Each EBP has criteria for program participation that should be used to determine eligibility.
Enrolled: Is the consumer participating in a particular EBP service or has the consumer participated in the EBP in the past period? Note: Aggregate data about eligibility and enrollment can be used to determine the percent of eligible consumers who received services.
Incident reporting
For the following outcomes, record the number of days and number of incidents that the consumer spent in each category during the reporting period.
Categories:
n Been homeless: Number of days that the consumer was homeless and how many times the consumer was homeless during the reporting period. Homeless refers to consumers who lack a fixed, regular, and adequate nighttime residence.
n Been incarcerated: Number of days and incidents that the consumer spent incarcerated in jails or in other criminal justice lockups.
n Been in a state psychiatric hospital: Number of days and incidents that the consumer spent hospitalized primarily for treatment of psychiatric disorders in a state psychiatric hospital.
General data
Quarter: Check the time frame for the reporting period.
Year: Fill in the current year.
Reported by: Fill in the name and title of the person who completed the form.
Agency: Identify the agency name.
Team: Write the team name or number.
About the consumer
Consumer ID: Write the consumer ID that is used at your agency, usually a name or an identifying number. This information will be accessible only to the agency providing the service.
Discharge date: If the consumer has been discharged during this report period, fill in the discharge date.
Date of birth: Fill in the consumer’s date of birth (Example: 09/22/1950).
Gender: Check the appropriate box.
Ethnicity: Fill in the consumer’s ethnicity.
Primary diagnosis: Write the DSM diagnosis.
Appendix H: Instructions for the Outcomes Report Form 76 Evaluating Your Program
n Been in a private psychiatric hospital: Number of days and incidents that the consumer spent hospitalized primarily for treatment of psychiatric disorders in a private psychiatric hospital.
n Been hospitalized for substance abuse reasons:
Number of days and incidents that the consumer spent hospitalized primarily for treatment of substance use disorders, including both public and private hospitals whose primary function is treating substance use disorders.
Competitive employment
In the past 3 months, how many days was the consumer
competitively employed? Competitive employment means working in a paid position (almost always outside the mental health center) that would be open to all community members to apply. Competitive employment excludes consumers working in sheltered workshops, transitional employment positions, or volunteering. It may include consumers who are self- employed but only if the consumer works regularly and is paid for the work.
Stage of substance abuse treatment
What was the consumer’s stage of substance abuse
treatment on the last day of the quarter? Record the consumer’s stage of substance abuse recovery, according to the following nine categories:
n Not applicable: No history of substance abuse disorder.
n Pre-engagement: No contacts with a case manager, mental health counselor, or integrated treatment specialist.
n Engagement: Contact with an assigned case manager or counselor, but does not have regular contacts. The lack of regular contact implies lack of a working alliance.
n Early persuasion: Regular contacts with a case manager or counselor, but has not reduced substance use for more than a month. Regular contacts imply having a working alliance and a relationship in which substance abuse can be discussed.
n Late persuasion: Engaged in a relationship with a case manager or counselor, is discussing substance use or attending a group, and shows evidence of reducing use for at least 1 month (fewer drugs, smaller quantities, or both). External controls (for example, Antabuse) may be involved in reduction.
n Early active treatment: Engaged in treatment, is discussing substance use or attending a group, has reduced use for at least 1 month, and is working toward abstinence (or controlled use without associated problems) as a goal, even though consumer may still be abusing.
n Late active treatment: Engaged in treatment, has acknowledged that substance abuse is a problem, and has achieved abstinence (or controlled use without associated problems) but for less than 6 months.
n Relapse prevention: Engaged in treatment, has acknowledged that substance abuse is a problem, and has achieved abstinence (or controlled use without associated problems) for at least 6 months. Occasional lapses, not days of problematic use, are allowed.
n In remission or recovery: No problems related to substance use for more than 1 year and is no longer in any type of substance abuse treatment.
Living arrangement
What was the consumer’s living arrangement on the last
day of the quarter? These data give your agency an ongoing record of the consumer’s residential status.
n Not applicable or unknown
n Psychiatric hospital: Those hospitals, both public and private, whose primary function is treating mental disorders. This includes state hospitals and other freestanding psychiatric hospitals.
n Substance use hospitalization: Those hospitals, both public and private, whose primary function is treating substance use disorders.
n General hospital psychiatric ward: Psychiatric wards located in general medical centers that provide short-term, acute crisis care.
Evaluating Your Program 77 Appendix H: Instructions for the Outcomes Report Form
n Nursing home: Facilities that are responsible for the medical and physical care of consumers and have been licensed as such by the state.
n Family care home: Consumers live in single-family dwellings with non-relatives who provide substantial care. Substantial care is determined by the degree to which non-relatives are responsible for the daily care of consumers. Such things as medication management, transportation, cooking, cleaning, restrictions on leaving the home, and money management are considered. Non-relatives may have guardianship responsibilities. If consumers are unable to do most daily living tasks without the aid of caretakers, consider caretakers to be providing substantial care.
n Lives with relatives (heavily dependent for personal care): Consult consumers and relatives about how much family members are responsible for consumers’ daily care. An important distinction between this status and supervised apartment program is to ask, “If the family were not involved, would the consumer be living in a more restrictive setting?” In assessing the extent to which family members provide substantial care, consider such things as taking medication, using transportation, cooking, cleaning, having control of leaving the home, and managing money. If consumers are unable to independently perform most daily living functions, consider family members to be providing substantial care.
n Group home: A residence that is run by staff who provide many functions (shopping, meal preparation, laundry, etc.) that are essential to living independently.
n Boarding house: A facility that provides a place to sleep and meals but it is not seen as an extension of a mental health agency nor is it staffed with mental health personnel. These facilities are largely privately run and consumers have a high degree of autonomy.
n Supervised apartment program: Consumers live (fairly independently) in an apartment sponsored by a mental health agency. In determining whether someone fits this category, look at the extent to which mental health staff have control over key aspects of the living arrangements.
Example characteristics of control include the following:
The mental health agency signs the lease.
The mental health agency has keys to the house or apartment.
Mental health agency staff provides onsite day or evening coverage.
The mental health agency mandates that consumers participate in certain mental health services—medication clinic, day program, etc., to live in the house or apartment.
Note: Consumers who receive only case management support or financial aid are NOT included in this category; they are considered to be living independently.
n Lives with relatives (but is largely independent): An assignment to this category requires having information from consumers and families. The key consideration relates to the degree to which consumers can perform most tasks essential to daily living without being supervised by family members.
n Living independently: Consumers who live independently and are capable of self-care, including those who live independently with case management support. This category also includes consumers who are largely independent and choose to live with others for reasons unrelated to mental illness. They may live with friends, a spouse, or other family members. The reasons for shared housing could include personal choice related to culture or financial considerations.
n Homeless: Consumers who lack a fixed, regular, and adequate nighttime residence.
n Emergency shelter: Temporary arrangements due to a crisis or misfortune that are not specifically related to a recurrence of the consumer’s illness. While many emergency shelters provide emotional support, the need for emergency shelter is due to an immediate crisis unrelated to the consumer’s mental illness.
n Other: Those who complete the form should clearly define this status in the space provided.
Appendix H: Instructions for the Outcomes Report Form 78 Evaluating Your Program
Educational status
What was the consumer’s educational status on the last day
of the quarter? These data give your agency an ongoing record of the consumer’s educational status.
n Not applicable or unknown
n No educational participation: Consumer is not participating in educational activities.
n Avocational/educational involvement: Organized classes in which consumers enroll consistently and expect to take part for the purpose of life enrichment, hobbies, recreation, etc. These classes must be community-based, not run by the mental health center. Classes are those in which anyone could participate, not just consumers. If any of these activities involve college enrollment, use the categories below.
n Pre-educational explorations: Consumers in this status are engaged in educational activities with the specific purpose of working toward an educational goal. This includes consumers who attend a college orientation class with the goal of enrolling, meet with the financial aid office to apply for scholarships, or apply for admission to enroll. This status also includes consumers who attend a mental health center-sponsored activity focusing on an educational goal (for example, campus visits with a case manager to survey the location of classrooms; meetings with the case manager and college staff to secure entitlements).
n Working on General Educational Development (GED)
diploma: Consumers who are taking classes to earn their GED diploma.
n Working on English as Second Language: Consumers who are taking classes in English as a Second Language in a community setting.
n Basic educational skills: Consumers who are taking adult educational classes focused on basic skills, such as math and reading.
n Attending vocational school or apprenticeship,
vocational program, or attending high school:
Consumers who are––
Participating in community-based vocational schools;
Learning skills through an apprenticeship, internship, or in a practicum setting;
Involved in on-the-job training to acquire more advanced skills;
Participating in correspondence courses which lead to job certification; and
Young adults attending high school.
n Attending college: 1 to 6 hours: Consumers who attend college for 6 hours or fewer per term. This status continues over breaks, etc., if consumers plan to continue enrollment. This status suggests that consumers regularly attend college and includes correspondence, TV, or video courses for college credit.
n Attending college: 7 or more hours: Consumers attend college for 7 or more hours per term. This status continues over breaks, etc., if consumers plan to continue enrollment.
Regular attendance with expectations of completing course work is essential for assignment to this status.
n Other: Those who complete the form should clearly define this status in the space provided.
Evaluating Your Program 79 Appendix I: Assessor Training
Evaluating Your Program
Appendix I: Assessor Training and Work Performance Checklist
Evaluating Your Program 81 Appendix I: Assessor Training
Assessor Training and Work Performance Checklist
Assessment date
Assessor’s name ____________________________________________________________________ _________________________________ First Middle Initial Last Title
Agency visited _______________________________________________________________________________________________________
Agency address _______________________________________________________________________________________________________ Street
_______________________________________________________________________________________________________ City State ZIP code
EBP assessed _______________________________________________________________________________________________________
Assessor qualifications
Yes
q 1a. Data collection and skills: Assessor’s skills are evidenced by his or her prior work experience, credentials, or supervisor’s observations.
q 1b. EBP knowledge: Assessor’s knowledge is evidenced by his or her prior work experience, credentials, or passing a knowledge test on a specific EBP.
q 1c. Training: Assessors receive at least 8 hours of systematic training on chart review, interviewing techniques, and process assessment.
q 1d. Shadowing: Assessors complete at least 1 assessment with an experienced assessor before the first official process assessment.
q 1e. Practice rating: Assessors co-rate as practice before being official assessors and agree exactly with an experienced assessor on ratings for at least 80% of items.
/5 Subtotal
Data Collection
/ /
Appendix I: Assessor Training 82 Evaluating Your Program
q 2a. Contact and scheduling: With contact person, assessors identify a date convenient to site, explain purpose of the assessment, identify information to be assembled ahead of time, and develop specific schedule of interviews and assessment activities.
q 2b. Number of assessors: Two or more assessors are present during the assessment visit and independently rate all items. If agency is working with a consultant, assessor may join with consultant to conduct assessments.
q 2c. Time management: Sufficient time is allotted and all necessary materials reviewed (2 days for 2 assessors).
q 2d. Interviewing: Interview all the sources stipulated in the protocol (e.g., for IMR, interviews with the program director, 3 ACT team members, and 3 consumers).
q 2e. Completion of documents: Complete score sheet, cover sheet, and any other supplemental documents relating to the agency.
q 2f. Documentation supporting rating: Each assessor provides written documentation for evidence supporting the rating for each item (e.g., marginal notes).
q 2g. Chart selection and documentation: Chart selection follows guidelines provided in the protocol (e.g., randomized, appropriate type and number of charts). Assessors note discrepancies (e.g., chart unavailability).
q 2h. Chart review: Both assessors review all charts and rate them independently.
q 2i. Resolution of discrepancies: When a discrepancy exists between sources (e.g., charts and ACT team members), assessors make followup probes with an appropriate informant (typically the ACT leader or relevant staff members).
q 2j. Independent ratings: No later than 1 day after the assessment, assessors independently complete scales before discussing ratings.
/10 Subtotal
Post-assessment visit
q 3a. Timely consensus: Within 5 working days after the assessment, assessors discuss their ratings to determine consensus ratings, identifying any followup information needed. A third assessor (e.g., supervisor) may be consulted to resolve difficult ratings.
q 3b. Inter-rater reliability: Raters agree exactly on ratings for at least 80% of the items. Sources of unreliability are discussed with supervisor and strategies developed to reduce future unreliability.
q 3c. Follow up on missing data: If followup calls are needed to complete an item, information obtained within 3 working days.
/3 Subtotal
Comprehensive report writing
Evaluating Your Program 83 Appendix I: Assessor Training
q 4a. Documentation of background information:
n List recipients of report in the header (usually the agency director and ACT leader; add others by mutual agreement).
n Summarize time, place, and method.
n Provide background about scale.
q 4b. Site and normative fidelity data: Provide a table with item-level (consensus) scores, along with normative data (if available). Normative data include both national and State norms. In this table, provide comparative site data from prior assessments. On second and later assessments, provide a graph of global fidelity ratings over time for the site (trend line).
q 4c. Quantitative summary: Provide narrative summary of quantitative data. List strengths and weaknesses.
q 4d. Score interpretations:
n Interpret overall score.
n Include other pertinent observations.
n Provide overall summary.
n Provide opportunity for site to comment and clarify.
q 4e. Report editing: If agency is working with a consultant, consultant may write report. Assessor and supervisor review draft of the report before it is submitted to the agency.
/5 Subtotal
Report submission and followup
q 5a. Timely report: Report sent to agency director within 2 weeks of visit.
q 5b. Follow up on report: If agency is working with a consultant, consultant discusses report with designated agency staff within 1 month of assessment.
/2 Subtotal
Quality control
q 6. Quality control: Supervisor reviews assessments and gives feedback, as necessary, to assessors. Depending on skill level of assessors, supervisor periodically accompanies assessors on assessment for quality assurance purposes.
/1 Subtotal
/27 Total — Add the subtotals.
HHS Publication No. SMA-09-4422 Printed 2009
26173.0709.7765020404
- Cover
- Title Page
- Acknowledgments
- Evaluating Your Program
- What’s in Evaluating Your Program
- Why Evaluate Your Family Psychoeducation Program?
- Why you should collect process measures
- Why you should collect outcome measures
- Why develop a quality assurance system
- Conduct a Readiness Assessment
- Readiness Assessment
- Conduct Process Assessments
- How to use process measures
- Monitor Outcomes
- Develop procedures
- What Is the Consumer Outcomes Monitoring Package?
- Expanding Your Outcome Measures
- Use Data to Improve Your Program
- Appendix A: Cover Sheet- Family Psychoeducation Fidelity Scale and General Organizational Index
- Appendix B: Checklist- Observation of Multifamily Group Sessions
- Appendix C: Family Psychoeducation Fidelity Scale and Score Sheet
- Appendix D: Family Psychoeducation Fidelity Scale Protocol
- Appendix E: General Organizational Index and Score Sheet
- Appendix F: General Organizational Index Protocol
- Appendix G: Outcomes Report Form
- Appendix H: Instructions for the Outcomes Report Form
- Appendix I: Assessor Training and Work Performance Checklist
