Module 5 discussion

271

C H A P T E R 1 7

ASSESSMENT AND EVALUATION Keneca Boyce

Every nonprofit should adopt practices that support and promote assessment and evaluation. Implementing and enhancing systems and protocols that foster assess- ment and evaluation impact everyone: constituents and communities, nonprofit staff and leadership, board members, and funders. Assessment and evaluation are ongoing processes that can be beneficial at all stages of a program. Evaluation is a well-thought-out systematic approach to assess the performance, quality, and ben- efit of a program, service, or the overall functioning of the organization. It is the means by which we are able to establish consistency in exploring, collecting, track- ing, and examining the work we “say” we do. It involves a set of predetermined protocols to assure data are captured in a way that accurately represents the services delivered to determine if the outcomes have been achieved.

Assessment and evaluation provide an opportunity for nonprofits and funders to discuss outcomes and program improvement concretely rather than abstractly. The current funding climate requires nonprofits to embrace evaluation as a required means for achieving sustainability. Although some nonprofits have made great strides in establishing the infrastructure and competencies required to effectively and efficiently assess and evaluate outcomes, it is, in fact, a multifaceted under- taking. This chapter provides an overview of key steps to support assessment and evaluation in addition to building the capacity and competency required to do so successfully.

WHY EVALUATE?

The question should be, why not evaluate? When we consider the scope and the depth of contributions from the nonprofit sector to society, at its core, we find the collective persistence to prioritize the preservation and well-being of human capi- tal. Many nonprofits tout their services, resources, and supports in line with some aspect of social justice and social good, often reflected through their mission. An organizational culture defined by accountability, innovation, and learning, driven by systems and protocols associated with assessment and evaluation, is essential for nonprofit sustainability and social change.

There was a time when “stories of success” were sufficient for the nonprofit sector and funders (Miles, 2006). Although the concept of outcomes and account- ability has been around for some time, there was an evident surge in the demand for

Congress_27372_PTR_17_271-286_07-25-16.indd 271Congress_27372_PTR_17_271-286_07-25-16.indd 271 7/26/2016 4:43:46 PM7/26/2016 4:43:46 PM

C o p y r i g h t 2 0 1 7 . S p r i n g e r P u b l i s h i n g C o m p a n y .

A l l r i g h t s r e s e r v e d . M a y n o t b e r e p r o d u c e d i n a n y f o r m w i t h o u t p e r m i s s i o n f r o m t h e p u b l i s h e r , e x c e p t f a i r u s e s p e r m i t t e d u n d e r U . S . o r a p p l i c a b l e c o p y r i g h t l a w .

EBSCO Publishing : eBook Academic Collection (EBSCOhost) - printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE AN: 1336884 ; Congress, Elaine Piller, Luks, Allan, Petit, Francis.; Nonprofit Management : A Social Justice Approach Account: ecor.main.ehost

272 NONPROFIT MANAGEMENT

outcomes before, during, and certainly after the economic downturn in 2008. The ongoing demand from funders for nonprofits to demonstrate evidence to justify a return on investment created a shift in the culture of nonprofit funding (Morley, Vinson, & Hatry, 2001). The nonprofit leaders and organizations that were attuned to the changing tides in the external environment were much more poised and prepared to respond to the increasing demands for evidence and outcomes from funders. However, many nonprofits were ill equipped, struggling to demonstrate evidence of impact, and, in extreme cases, ultimately phased out. The approach to meeting these demands left many nonprofits floundering to find the appropriate staff, interventions, outcomes, and data tracking systems to remain competitive. This harried approach to collecting and reporting outcomes left many nonprofits scrambling to desperately demonstrate and convince funders that their outcomes were worthy of investment. Assessment and evaluation should be perceived as direct benefits to the organization and not solely to comply with funding require- ments. The following examples highlight the benefits of evaluation to nonprofit organizations.

Example 1: Citizen BUILD is a nonprofit organization with a youth-mentoring workforce development program, designed to increase job placement and reten- tion for formerly incarcerated youth. Findings from the evaluation helped the organization to identify the most effective methods of communication between mentors and mentees, the most effective mentor–mentee engagement trainings, and barriers that compromised mentee job retention.

Example 2: Renewed Reformed Reentry (R3) conducted an evaluation of the alter- native to incarceration treatment program to see if they achieved their goal to reduce recidivism and hospitalization rates. Findings from the evaluation helped program administrators identify the ideal length of time in treatment that resulted in fewer triggers for recidivism. It also revealed the most effective community treatment partnerships and collaborations.

Example 3: Urban Coders is a nonprofit organization that partners with General Education Development (GED) programs to engage youth through technol- ogy to reignite their passion to learn and pursue a college education and major in computer science. Although youth study and prepare for the GED, they receive intensive training in coding, web design, app development, and so forth. Findings from the evaluation revealed a 60%-college engagement and placement rate; however, 15% of students indicated an entrepreneurial career goal. The organization found this information extremely helpful for future fundraising goals in addition to innovative expansion to recruit entrepreneurs as mentors.

BUILDING CAPACITY AND COMPETENCY

Data-Driven Culture

A data-driven agency uses evidence to drive practice and decision making. In order to become “data driven,” “outcomes driven,” or promote a culture of “inquiry” or “continuous improvement,” it is essential that leadership understands assessment

Congress_27372_PTR_17_271-286_07-25-16.indd 272Congress_27372_PTR_17_271-286_07-25-16.indd 272 7/26/2016 4:43:47 PM7/26/2016 4:43:47 PM

EBSCOhost - printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

17: ASSESSMENT AND EVALUATION 273

FIGURE 17.1 Building a culture of evaluation.

and evaluation as both a process and an outcome (Miles & Woodruff-Bolte, 2013). Transitioning or enhancing an organization’s data-driven culture takes time to learn, refine, and master. Figure 17.1, developed by Community Solutions (2001), is an illus- tration of 30 ideas with which to build a culture of evaluation within organizations.

Congress_27372_PTR_17_271-286_07-25-16.indd 273Congress_27372_PTR_17_271-286_07-25-16.indd 273 7/26/2016 4:43:47 PM7/26/2016 4:43:47 PM

EBSCOhost - printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

274 NONPROFIT MANAGEMENT

Nonprofit Leadership

Nonprofit leadership must prioritize assessment and evaluation, as they would any key human resource, budgeting, or fundraising matter. Promoting a data-driven cul- ture requires leadership to be consistent with messaging and communication that rein- forces the value and priority for all staff to adopt and adhere to evaluation systems and protocols (Hernandez & Visher, 2001). At the onset, the leadership and/or the designated evaluation team or staff are tasked with the responsibility of cultivating an outlook of enthusiasm and optimism about assessment and evaluation. Formal and informal meetings with staff should explore and address any discomfort, taboos, or misunderstandings associated with assessment and evaluation. Leadership will need to continually assess and address the impact of increased responsibility of a data-driven culture on the quality of service delivery. This style of open and ongoing communica- tion sets the tone for a data-driven culture of partnership where everyone plays a role in maintaining the integrity of the organization’s assessment and evaluation efforts.

Leadership must remain engaged yet make it clear that the appointed staff, division, or consultant will take the lead in facilitating the organization’s ongoing efforts to be data driven. In the absence of dedicated and qualified personnel with the support of the leadership, the path to become data driven will be more compli- cated and challenging for all (Miles, 2006). Promoting a data-driven culture should be an experience defined by collective learning among leadership, middle manage- ment, and staff.

Characteristics of a Data-Driven Organization

1. Data-informed board, executives, program leadership, and staff 2. Accessible and visible evaluation staff and resources 3. A functional Theory of Change (ToC) 4. Comprehensive evaluation plan 5. Data collection tools align with program strategies and indicators 6. Data collection is integrated and prioritized with existing program practices 7. Effective and efficient data management systems and protocols 8. Collect, store, and analyze intentional data 9. Utilize data to inform quality improvement

10. Data-driven organizational decision making 11. Annual budget allocation to develop and sustain data management system 12. Accessible evaluation and data management staff and resources 13. Prioritize data integrity 14. Effective communication among leadership, management, and staff 15. Functional data reports 16. Formal and informal structures to review and discuss data 17. Quality assurance and evaluation protocols align to monitor compliance, fidel-

ity, and data integrity 18. In-service professional development training on assessment and evaluation 19. Leadership, management, and staff understand their role in assessment and

evaluation

Congress_27372_PTR_17_271-286_07-25-16.indd 274Congress_27372_PTR_17_271-286_07-25-16.indd 274 7/26/2016 4:43:48 PM7/26/2016 4:43:48 PM

EBSCOhost - printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

17: ASSESSMENT AND EVALUATION 275

20. Employee performance evaluation aligns with compliance to data management protocols

THEORY OF CHANGE

Value of a ToC

An often overlooked, yet critical step in assessment and evaluation involves the development of a ToC as a precursor to evaluation. Every nonprofit should have a functional ToC as a blueprint to provide common knowledge for how organiza- tions assess performance, outcomes, and impact (Hunter, 2013). A functional ToC is designed to (a) foster communication, accountability, strategic planning, and decision making and (b) describe how and why a set of interventions and a set of strategies are expected to lead to outcomes. Stakeholders and funders value a ToC because it provides a commonly understood vision of long-term goals, how they will be achieved, and strategies to measure progress along the way (Mackinnon, Ammot, & McGarvey, 2006).

Irrespective of staff roles, discipline, or expertise, a ToC helps all staff under- stand how his or her individual and collective work with the target population will lead to the desired outcomes. It is a visual representation that provides deeper insight to the multidimensional construction of social issues. A ToC illustrates the added value and appreciation for interdisciplinary teams with expertise to address complex social issues. For example, a nonprofit organization worked for years fulfilling its mission to fight hunger by providing meals and building capacity in under-resourced communities. During a strategic planning meeting to review and update their ToC, they ultimately realized the need to rethink their approach to community capacity building. They decided to expand the scope of their services with a nutrition division to educate communities and their existing partners. They determined that more informed communities would be a key strategy to advance their mission to fight hunger. This expansion of a nutrition division deepened the interdisciplinary team expertise and strategy to address hunger as a multidimen- sional social issue.

ToC Development, Implementation, and Utilization

Developing a ToC involves a series of critical thinking steps to produce a compre- hensive picture about the nature of a social problem, and how specific actions will lead to solutions (Hunter, 2013). It is a collaborative process to achieve internal alignment following the exploration of several programmatic building blocks.

Some organizations will have the staff expertise to facilitate the process for developing a ToC. Others will need to hire staff or consultants who are skilled and able to support the organization in developing a ToC and the systems and protocols to support ongoing assessment and evaluation. At the onset, representatives from all levels of staffing (i.e., executive leadership, program manager, case planner, psy- chiatrist, peers, board members, volunteers) should be identified and engaged in the ToC development process. Often, as a result of stretched resources and capacity,

Congress_27372_PTR_17_271-286_07-25-16.indd 275Congress_27372_PTR_17_271-286_07-25-16.indd 275 7/26/2016 4:43:48 PM7/26/2016 4:43:48 PM

EBSCOhost - printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

276 NONPROFIT MANAGEMENT

leadership and management take the lead in developing a ToC. Nevertheless, before a ToC is finalized, organizations should engage representatives of various levels of staff for their input and feedback.

Once finalized, organizations should provide training on the implementation and utilization of a ToC. A fundamental component of training should provide all stakeholders with a deeper understanding of how the organization will meet the mission through the contributions of their roles, expertise, and available resources. ToC training should be ongoing and a frequent point of discussion among staff and departments. Training is particularly essential for direct service delivery profes- sionals who spend a great deal of time with program participants. Leadership and management must help them see the connection in delivering and documenting services as meaningful to their roles and the clients they serve (Miles, 2006). Shifting to a data-driven culture requires continuous communication regarding the added benefits of documenting services with accuracy and consistency. The ToC provides a tangible way for all levels of staff to internalize and actualize the mission. A data- driven organization assures that the ToC is visible and accessible to staff. The ToC

THEORY OF CHANGE

SOCIAL IMPACT

OUTCOMES

INDICATORS

OUTPUTS

STRATEGIES

INPUTS

HIGH PRIORITY SERVICES

TARGET GROUP

SOCIAL ISSUE

FIGURE 17.2 Components of a Theory of Change.

Congress_27372_PTR_17_271-286_07-25-16.indd 276Congress_27372_PTR_17_271-286_07-25-16.indd 276 7/26/2016 4:43:48 PM7/26/2016 4:43:48 PM

EBSCOhost - printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

17: ASSESSMENT AND EVALUATION 277

planning team should convene frequently, to review and determine if the ToC needs to be updated.

If developed, implemented, and used effectively, a ToC allows nonprofits to streamline efforts to assure they collect “intentional data” rather than “dead data.” Intentional data have justification and alignment with the mission and serves the needs of the organization and funders. Although there may be several data points of interest, the goal is to collect data that advances organizational development, organizational impact, and sustainability. The overall goal of the evaluation will determine if nonprofits should develop a programmatic ToC or an organizational ToC. An organizational ToC assesses the quality, cost, and alignment (goodness of fit) among all programs and the mission. A programmatic ToC assesses alignment of the mission, program strategies, and outcomes. A ToC (Figure  17.2) should be carefully developed as an invaluable organizational tool.

Theory of Change

Social Impact: Broad statement about the expected benefi ts or cumulative effect on the general population. Example: More youth will be career or college ready with technical knowledge and skills.

Outcomes: Changes in knowledge, skill, attitude, behavior, condition, or status as a result of program strategies. Example: Greater knowledge of career planning, improved academic performance, effective responses to confl ict, greater fi nancial stability.

• Short-term—knowledge, skills, attitude, motivation, awareness • Mid-term—behaviors, practices, policies, procedures • Long-term—environmental, social, economical, political conditions

Indicators: Specifi c and measurable changes that represent achievement of an outcome. Example: A nonprofi t college prep and placement program for high school seniors would have the following indicators: 80% of high school seniors will apply to a minimum of three universities; 75% of high school students will identify a major; 90% of youth will develop a career plan.

Outputs: Units of service or the amount of work a nonprofi t does. Outputs produce the desired outcomes for program participants. Example: The number of workshops, meals provided, resource packets distributed, participants served, and so forth.

Strategies: What a program does with the inputs. The activities and interventions to achieve the outcomes. Strategies produce outputs. Example: Educate the public about signs of mental illness and depression in youth, provide adult mentors for youth, feed homeless families, and so forth.

Inputs: Resources an organization or program uses to run the program. Example: Staff, volunteers, facilities, clients, equipment, curricula, money, partnerships, and so forth.

High Priority Services: Essential areas of practice supported by literature, research, trends, and practice experience, which drive service delivery.

Target Group: Customers, participants, communities, or population to be served by the program.

Social Issue: Social problem or point of focus identifi ed by the mission.

Congress_27372_PTR_17_271-286_07-25-16.indd 277Congress_27372_PTR_17_271-286_07-25-16.indd 277 7/26/2016 4:43:48 PM7/26/2016 4:43:48 PM

EBSCOhost - printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

278 NONPROFIT MANAGEMENT

Utilizing a ToC to Guide Assessment and Evaluation

Assessment: Provide evidence of target population needs and trends; identify new areas for program support.

Data Collection: Provide a framework for when, how, and what to measure; identify staff responsible for documentation and time frames for documentation.

Management: A centralized guide for defi ning work tasks, roles, and program deliverables; inform hiring, orientation, and professional development training needs; identify areas of emphasis for supervision.

Organizational Functioning: Identify targeted and general fundraising goals; proactive evidence-informed response to new and existing funder request; update the organizational chart.

Quality Improvement: Determine which program and services need to be improved to strengthen outcomes; reveal organizational and programmatic strengths and gaps in services; monitor program fi delity.

Strategic Planning: Identify services and practice models that can be replicated; determine new benchmarks, explore new interventions; reveal critical areas for collaborations and partnerships.

DATA MANAGEMENT

Data Collection

Once a functional ToC is developed, it will inform the evaluation plan to determine the data collection measures, processes, and systems. Data collection requires atten- tion to detail and adherence to strict and predetermined protocols. There are many factors that can compromise the integrity of data, so nonprofits should develop a well-thought-out evaluation plan. Identifying the data collection tools, persons responsible for data collection, and timing for data collection are essential to the quality and integrity of findings. The quality of the data collected will be reflected in the quality of the data reported.

There are several options available for data collection. Nonprofits may decide to administer a survey, have participants complete a questionnaire, extract data from client records, or hold a focus group. An evaluation plan may include a combination of methods. The selected technique will depend on the goal of the evaluation, stage of the program, characteristics of the target population, and available budget and resources to successfully complete the evaluation (W.K. Kellogg Foundation, 2004).

The designated person(s) responsible for collecting data play(s) a critical role in the success of the evaluation. Key staff should be trained on all data collection tools and the data management protocols. Many nonprofits will select a survey or questionnaire and will designate staff with the responsibility to solicit information from participants. All of the staff responsible for interviewing participants should receive training on the role of the interviewer to assure they do not unintentionally compromise the integrity of the data.

Congress_27372_PTR_17_271-286_07-25-16.indd 278Congress_27372_PTR_17_271-286_07-25-16.indd 278 7/26/2016 4:43:48 PM7/26/2016 4:43:48 PM

EBSCOhost - printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

17: ASSESSMENT AND EVALUATION 279

The following tips on interviewing skills should be addressed when training staff.

1. Emphasize overall purpose of the evaluation 2. Discuss confidentiality 3. Ask participants to answer questions as honestly as possible 4. Review response categories clearly and check to make sure the respondent

understands 5. Review response categories (to remind participants) before beginning a new

section 6. Maintain a neutral attitude 7. Refrain from behaviors that could influence how the respondent answers 8. Avoid sharing your personal opinions 9. Never suggest an answer

10. Speak clearly and slowly

It is essential to maintain communication with staff about their experiences collecting data from participants and compliance with the evaluation systems and protocols. These communications will often reveal gaps or challenges in the evalu- ation plan that may not be as transparent to leadership and data management staff. Establishing organizational compliance is fundamental, so systems and protocols must be transparent, effective, and efficient.

Selecting the appropriate data collection measure is as important as the method in which the data is collected. Data collection tools should accurately and efficiently measure the indicators to determine if the outcomes were achieved. More impor- tantly, data collection measures should align with the strategies and services deliv- ered. For example, a nonprofit organization recently received funding to develop and evaluate a new youth entrepreneurship training prevention program. They hired a new program director to develop and implement the program and eval- uation. The program was designed to teach and train youth aged 15 to 17 years entrepreneurship skills and support them in the development and pilot of a small business. On completion of the training program, youth graduated to peer trainers with the responsibility to mentor the next cohort of youth trainees. As a data-driven organization, they identified key indicators and developed a survey designed to measure the outcomes. During the data analysis phase, the data management team notified the program leadership about a gap in the findings when compared with the projected outcomes to the funder. None of the data analyzed measured the par- ticipant’s leadership skills. Although they intended to address leadership skills through the training program, they realized this was an underdeveloped compo- nent of the training curriculum. This evaluation proved to be a valuable lesson to the organization. In retrospect, they acknowledged proceeding with a great deal of haste throughout the program planning and evaluation phase. They also real- ized that they should have allotted more time to assess the new program direc- tor’s evaluation competencies. Three essential lessons emerged from the evaluation: (a) assess program development and evaluation expertise during the interview and hire phase; (b) provide ToC training and supervision for new leadership and staff;

Congress_27372_PTR_17_271-286_07-25-16.indd 279Congress_27372_PTR_17_271-286_07-25-16.indd 279 7/26/2016 4:43:49 PM7/26/2016 4:43:49 PM

EBSCOhost - printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

280 NONPROFIT MANAGEMENT

and (c) confirm alignment throughout the program planning and evaluation plan- ning phase. They agreed to apply these lessons to all future program initiatives and evaluations to avoid this disconnect in services delivered, data collection tools, and outcomes.

Evaluation Planning

1. What are the program indicators? 2. What do you want to learn from the evaluation? 3. What data will you collect? 4. What methods will be used for data collection? 5. Will data be collected from the entire target population? 6. Who will collect the data? 7. How often will data be collected? 8. Who will analyze the data? 9. How frequently will data reports be generated for review?

10. How will the data be used?

Reporting Data

Establishing clarity at the onset regarding the purpose of assessment and evalu- ation will be helpful to inform the organization on how it can leverage the data. Initial decisions about the usage of data will evolve as the organization evolves or as the program continues to demonstrate success or experience new trends and chal- lenges. A nonprofit organization responsible for recruiting and training foster par- ents initially produced evaluation reports that were useful for the board, managers, and funders. As they continued to review and apply the data to strategic decision making for the organization, they determined that the data would be instrumental in local advocacy efforts. It was also determined that the data would be used to inform updates and revisions to the foster parent training curriculum.

The capacity of any nonprofit organization to generate reports will depend on the data management system and staffing capacity. Often times, limitations of an efficient data management system result in a great deal of staff time to gener- ate reports with accuracy. With these considerations in mind, nonprofits should determine the feasibility, frequency, and consistency of generating and dissemi- nating reports. For example, program managers at a nonprofit organization were responsible for providing immigration services and support including English as a second language and computer literacy. At the beginning of the funding cycle, program managers received monthly reports highlighting key performance areas. Throughout the entire third quarter of the funding cycle, program managers did not receive monthly reports, despite their repeated requests. In the absence of current and relevant data, program managers struggled to keep staff engaged in discus- sions specific to assessment and evaluation. Monthly reports resumed during the final quarter of the funding cycle. Program managers reported, “We feel like we’re right where we started. We have to work just as hard as we did in the beginning to reacquaint staff with data and twice as hard to get them engaged to discuss the

Congress_27372_PTR_17_271-286_07-25-16.indd 280Congress_27372_PTR_17_271-286_07-25-16.indd 280 7/26/2016 4:43:49 PM7/26/2016 4:43:49 PM

EBSCOhost - printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

17: ASSESSMENT AND EVALUATION 281

data.” A culture driven by continuous improvement is contingent on staff consis- tently expecting to receive, review, and make decisions based on data.

The ToC can be instrumental in helping staff and leadership identify a dash- board of performance indicators that will communicate progress at different points throughout the year.

Reports should be developed in a way that will translate a clear story about the population, the services, the outcomes, and the overall performance of a program. The goal of disseminating reports is to tell a story in a way that makes it engaging for the audience to connect, reflect, and respond to the data.

Characteristics of a Functional Report

• Features standard template to organize the data • Includes easily interpreted data (graphs, charts, colors, etc.) • Provides relevant data for the intended recipient (board, managers, staff, volun-

teer, clients, etc.) • Highlights performance areas of success and areas of improvement • Provides brief explanatory notes (as needed) for special considerations for inter-

preting the data • Provides comparative or trend data to support interpretation of data and

performance

Review and Discussion of Data

The report has been prepared and disseminated . . . now what? Once reports are disseminated, it can cause a great deal of anxiety for staff and program leaders. Therefore, ongoing support and communication to normalize the experience of discussing performance are key. Forums that are dedicated to the review and discussion of data should be guided and aimed at celebrating the successes and brainstorming a plan of action for areas of improvement. Program leadership should create internal structures to convene key staff and support staff who play a critical role in achieving outcomes. Everyone should be familiar with the contents of the report and prepared to engage in a discussion about the findings.

CASE EXAMPLE

One nonprofit leader decided that it was beneficial for his team to review a dashboard of performance indicators every other month. He took the lead in disseminating reports, convening the team, and leading the discussion. Six months later, he implemented several changes, which proved to be instrumental in supporting staff to build a data-driven culture that values assessment and evaluation. He implemented the following protocol:

• Disseminate reports electronically 1 week in advance of meetings • Instruct program staff to bring one question, comment, or plan of action to the meeting • Rotate meeting facilitators

Congress_27372_PTR_17_271-286_07-25-16.indd 281Congress_27372_PTR_17_271-286_07-25-16.indd 281 7/26/2016 4:43:49 PM7/26/2016 4:43:49 PM

EBSCOhost - printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

282 NONPROFIT MANAGEMENT

Following the general review of the report, staff shared their comment, question, or plan of action. The team used both small and large group discussion formats. Each meeting closed with a brief recap of lessons learned and a plan of action. This new format generated a great deal of discussion and gave the team more ownership of the data. They

FIGURE 17.3 The Innovation Network.

Congress_27372_PTR_17_271-286_07-25-16.indd 282Congress_27372_PTR_17_271-286_07-25-16.indd 282 7/26/2016 4:43:49 PM7/26/2016 4:43:49 PM

EBSCOhost - printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

17: ASSESSMENT AND EVALUATION 283

were more focused on reviewing and using the data to determine the need for course- corrective decisions. Overall, staff developed a deeper appreciation and understanding of nonprofi t assessment and evaluation.

Figure 17.3 illustrates the various ways nonprofi ts use and report data, based on the 2011 research on a national sample of nonprofi ts conducted by the Innovation Network Inc.

Generate Discussion When Reviewing Data

• Do the data reveal any new information about the target population that infl uence program strategies and outcomes?

• Do the data reveal key areas of quality performance that lead to outcomes? • Do the data reveal any challenges or obstacles to achieving outcomes? • Do the data reveal an area of focus for program improvement? • Do the data inform new services and supports for staff or participants? • Should new indicators of success be explored or included in future reports? • Are there any internal or external collaborations and partnerships to be explored?

Data Management System

Technology is a key component for building an infrastructure to support assess- ment and evaluation. The ability to analyze data and maximize on lessons learned will depend on the limitation or sophistication of the data management system. A comprehensive assessment of the needs of the program or organization and the intended purpose of the data management system is the first step to determine the best technology solution. The path to identifying the right data management system requires a great deal of resources and investment to implement and main- tain. Nonprofits must consider these expenses and build it into their budget and fundraising goals (Morariu, Athanasiades, & Emery, 2012).

Increased demands to provide quality services, while maintaining compliance for internal and external purposes, often result in duplicative data entry. Many non- profits are faced with the daunting task of managing several reports for several funders. Reporting requirements vary, with different expectations for program out- comes and how they should be measured and reported (Major, 2011). Several gov- ernment funders require nonprofits to enter data into an online database. However, the data are available for a limited time, so institutional data are lost once the case is closed or when the contract ends. Often times, nonprofits cannot extract data from these online databases to tell their story of success, to inform program improve- ment, or to leverage data for fundraising efforts (Snibbe, 2006). This complicates organizational efforts to motivate staff to enter data that ultimately will have little impact on their day-to-day activities and interventions. The goal of entering data is to use the data. Until these issues are resolved, nonprofits will need to identify the appropriate data management system that will streamline data entry and pro- vide direct access for storing, analyzing, and reporting data. Exploring mechanisms and solutions to leverage data from these portals will ultimately strengthen the sec- tor’s efforts to become more data driven, data informed, and avoid duplicative data entry (Miles & Woodruff-Bolte, 2013).

Congress_27372_PTR_17_271-286_07-25-16.indd 283Congress_27372_PTR_17_271-286_07-25-16.indd 283 7/26/2016 4:43:50 PM7/26/2016 4:43:50 PM

EBSCOhost - printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

284 NONPROFIT MANAGEMENT

Characteristics of a Functional Data Management System

• Systematically store and analyze data • Provide efficiency for staff and program leadership • Generate reports to assess performance at any given time • User friendly • Easily accessible and updated • Minimize duplicity across multiple reporting systems • Interface with other systems in the public domain (i.e., Excel, Access) • Customized to reflect relevant data points specific to the target population, strate-

gies, and indicators

THE ROLE OF FUNDERS

Although funders have increased their demand on nonprofit assessment and evalu- ation with a focus on outcomes, funding for these efforts have not shared the same momentum and fervor. Nonprofits would benefit from a group of funders (govern- ment agencies, private foundations, corporate funders, and individual donors) who partner and invest in the capacity building of data management software, systems, training on assessment and evaluation, and technical assistance (Major, 2011). The technology industry can play a more visible and critical role in building and upgrad- ing the infrastructure of nonprofit data management systems. Capacity is increas- ingly becoming an area of mention and concern across nonprofits. Many nonprofits are taxed with various reporting requirements, frequent audits, new initiatives, and policies. Everyone is required to do more with less, while maintaining quality.

Some funders have made advances by allocating funds for capacity develop- ment, training, and technical assistance. They have redefined a collaborative and supportive relationship with grantees to support mutually beneficial assessment and evaluation (Major, 2011). More funders should broaden the scope of their role in the investment for social justice by acquiring a deeper and realistic understand- ing of the costs and efforts associated with achieving outcomes and creating change. Often times, the approved budget and staffing plan do not account for the required expertise of evaluation staff and the associated technology resources to effectively and efficiently collect and report data. The absence of an effective infrastructure for data management, data integrity, and reporting can compromise the ability of an organization to compete for sustainable funding. It is both logical and timely that funders invest in services and outcomes as well as data-driven capacity- building resources to support assessment and evaluation. It is in the best interest for all invested in social good to assure organizations have the tools to successfully exe- cute their mission.

REFERENCES

Community Solutions. (2011). Building a culture of innovation: 30 ideas to apply to your organiza- tion. Retrieved from http://communitysolutions.ca/web/

Congress_27372_PTR_17_271-286_07-25-16.indd 284Congress_27372_PTR_17_271-286_07-25-16.indd 284 7/26/2016 4:43:51 PM7/26/2016 4:43:51 PM

EBSCOhost - printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

17: ASSESSMENT AND EVALUATION 285

Hernandez, G., & Visher, M. (2001). Creating a culture of inquiry: Changing methods and minds on the use of evaluation in nonprofit organizations (pp. 1–20). San Francisco, CA: The James Irvine Foundation.

Hunter, D. (2013). Working hard—And working well: A practical guide to performance manage- ment. Washington, DC: Venture Philanthropy Partners.

Mackinnon, A., Ammot, N., & McGarvey, C. (2006). Mapping change: Using a theory of change to guide planning and evaluation (pp. 1–11). New York, NY: Grantcraft.

Major, D. (2011). Expanding the impact of grantees: How do we build the capacity of nonprofits to evaluate, learn and improve? Grantmakers for Effective Organizations. Retrieved from http://www.socialimpactexchange.org/sites/www.socialimpactexchange.org/files/ GEO_SWW_BuildCapacityToEvaluateLearnImprove.pdf

Miles, M. (2006). Good stories aren’t enough: Becoming outcomes-driven in workforce development. Working Ventures, Public/Private Ventures. Retrieved from http://www.issuelab.org/ resource/good_stories_arent_enough_becoming_outcomes_driven_in_workforce_ development

Miles, M., & Woodruff-Bolte, S. (2013). Nurturing inquiry and innovation: Lessons from the work- force benchmarking improvement collaborative (pp. 1–22). Ann Arbor, MI: Corporation for a Skilled Workforce.

Morariu, J., Athanasiades, K., & Emery, A. (2012). State of evaluation 2012: Evaluation practice and capacity in the nonprofit sector (pp. 1–20). Washington, DC: Innovation Network.

Morley, E., Vinson, E., & Hatry, H. (2001). Outcome measurement in nonprofit organizations: Current practices and recommendations (pp. 3–10). Washington, DC: Independent Sector.

Snibbe, A. (Fall 2006). Drowning in data. Stanford Social Innovation Review, 4(3), 39–45. W. K. Kellogg Foundation. (2004). Evaluation handbook. Battle Creek, MI: W. K. Kellogg

Foundation. Retrieved from https://www.wkkf.org/resource-directory/resource/2010/ w-k-kellogg-foundation-evaluation-handbook

Congress_27372_PTR_17_271-286_07-25-16.indd 285Congress_27372_PTR_17_271-286_07-25-16.indd 285 7/26/2016 4:43:51 PM7/26/2016 4:43:51 PM

EBSCOhost - printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

Congress_27372_PTR_17_271-286_07-25-16.indd 286Congress_27372_PTR_17_271-286_07-25-16.indd 286 7/26/2016 4:43:51 PM7/26/2016 4:43:51 PM

EBSCOhost - printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use