employing additional staff can improve service delivery?
femme11226
IMPACTOFCAPACITYBUILDINGNONARTICLE1.pdf
Nonprofit and Voluntary Sector Quarterly 2014, Vol. 43(3) 547 –569
© The Author(s) 2013 Reprints and permissions:
sagepub.com/journalsPermissions.nav DOI: 10.1177/0899764013491013
nvsq.sagepub.com
Article
The Impact of Capacity- Building Programs on Nonprofits: A Random Assignment Evaluation
Amy Minzner1, Jacob A. Klerman2, Carrie E. Markovitz3, and Barbara Fink4
Abstract Nonprofit organizations (NPOs) play an important role in delivering social services to those in need. Capacity-building efforts for NPOs derive from funders’ desire to increase NPOs’ effectiveness and redress problems associated with limited administrative and organizational capacity. Through technical assistance, training, and grant funds, funders aim to enhance NPO functioning and ultimately improve client outcomes. Despite a general consensus about the importance of capacity building, little high-quality evidence exists on the impact of capacity-building investments. This article presents the findings from the first random assignment evaluation to be conducted in the field of nonprofit capacity building. The subject of the evaluation was one of country’s largest organizational capacity-building initiatives, the federal government’s Compassion Capital Fund (CCF) Demonstration Program. Findings from the evaluation provide clear evidence that capacity-building efforts increase capacity in each of five critical areas of capacity.
Keywords randomized control trial, nonprofit capacity building, evaluation
1Senior Associate, Abt Associates Inc., Cambridge, MA, USA 2Principal Associate, Abt Associates Inc., Cambridge, MA, USA 3Senior Research Scientist, National Opinion Research Center, Bethesda, MD, USA 4Senior Research Associate, Branch Associates, Inc., Philadelphia, PA, USA
Corresponding Author: Amy Minzner, Senior Associate, Abt Associates Inc., 55 Wheeler Street, Cambridge, MA 02148, USA. Email: [email protected]
491013NVS43310.1177/0899764013491013<italic>Nonprofit and Voluntary Sector Quarterly</italic>Minzner et al. research-article2013
548 Nonprofit and Voluntary Sector Quarterly 43(3)
Overview
Nonprofit organizations (NPOs) play an important role in delivering social services to those in need. Foundations, as well as local, state, and federal governments fund NPOs to deliver a wide range of services, typically in community-based settings accessible to those who need assistance most. While working through NPOs often allows funders to reach deeper into communities and reach more of those in need; some of these same NPOs tend to experience capacity limitations that hinder their growth, sustainability, and effectiveness. These capacity gaps may include incomplete recordkeeping, limited financial monitoring, and/or unstructured volunteer management. Since the mid- 1990s, these concerns have led funders to invest in NPOs, providing technical assis- tance, training, and grant funds to improve the ways organizations function, typically with the goal of ultimately improving client outcomes.
Despite 20 years of investment in organizational capacity building, rigorous empirical evidence of capacity-building programs’ effectiveness is limited. Prior to the study described here there have been no random assignment evaluations. Instead, the evaluations of capacity-building programs in the current literature have used a variety of less rigorous methods such as prepost, cross-sectional analysis (regression adjusted), or quasi-experimental difference in differences (Abt Associates, 2007; Abt Associates & Branch Associates, 2008; Brown, 2008; Doherty & Mayer, 2003; Kapucu, Healy, & Arslan, 2011; Leake et al., 2007; Leviton, Herrera, Pepper, Fishman, & Racine, 2006; Markovitz & Magged, 2008; Millese & Bies, 2007; Patrizi, Gross, & Freedman, 2006; Popescu & Dewan, 2009; Sobeck, 2008; Sobeck & Agius, 2007). Specifically, without random assignment, concern remains that any estimated impacts are due to imperfect controls for preexisting differences between those that did and did not receive capacity building, and are therefore spurious. Random assignment solves the problem of imperfect controls by forming program and control groups by randomization, such that the two groups do not differ system- atically prior to the intervention. Thus, any statistically significant differences between the program and control groups observed at follow-up can be attributed to the intervention or chance (and we can use conventional statistical methods to bound the impact of chance).
The study described in this article was designed to evaluate one of the nation’s largest capacity-building initiatives, the Compassion Capital Fund Demonstration Program (CCF), using a random assignment design. The evaluation’s funder, the Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services, sought answers to two primary research questions: How effective is the capacity-building assistance in increasing organizational capac- ity among NPOs, and in what areas of organizational capacity have NPOs shown the greatest improvement? ACF contracted with Abt Associates to design and complete the evaluation.
The results reported here show that the CCF Demonstration Program had signifi- cant impact in each of the five critical areas of organizational capacity examined. The largest improvements in capacity were also spread across all five capacity areas. Not
Minzner et al. 549
only does this random assignment study provide methodologically rigorous evidence about the effectiveness of capacity-building assistance provided through the CCF Demonstration Program; more broadly, it also provides the first random assignment evidence that capacity-building assistance can be used to improve NPOs’ organiza- tional capacity.
The remainder of this article proceeds in five sections. The first two sections pro- vide additional detail on the context and program design of the CCF Demonstration Program. The third section describes our evaluation design and data methods. The final two sections present the evaluation results and discuss their implications.
Historical Context
Over the past two decades, there has been an increased focus on the role faith-based and community nonprofit organizations (NPOs) play in the country’s social service network (Jolley & Meissen, 2009). NPOs help to fill gaps in the social safety net by providing an array of services such as child care, adult education and job training, mental health counseling, and substance abuse treatment (Allard, 2009a, 2009b; Sobeck, 2008). NPOs are often perceived as more accessible to individuals reluctant to seek help from large, unfamiliar public agencies. NPOs tend to be seen as trusted and respected community institutions, which increases their approachability to indi- viduals with specific needs, such as victims of domestic violence, people with addic- tion issues, and troubled youth. For these reasons, there is an ongoing interest among local, state, and federal governments in partnering with NPOs to deliver services (Sherman & Stanakis, 2002).
Logic of Capacity-Building Assistance
While NPOs have vital attributes that increase their accessibility within and to the community, some NPOs lack the organizational capacity critical to their survival and effectiveness (Blumenthal, 2003; De Vita, Fleming, & Twombly, 2001; Letts, Ryan, & Grossman, 1999; Light, 2004). Organizational deficiencies have been found to include inadequate financial tracking systems, limited fundraising experience, and/or ineffec- tive Boards of Directors. In addition, small, frontline organizations with limited administrative and organizational capacities and resources may have difficulty com- plying with the rigors of state or federal reporting and funds management require- ments. They thus cannot effectively partner with the state or federal government. Because of the unique role that NPOs play and the capacity gaps that many face, funders, including the federal government, believe it is worthwhile to help NPOs increase their potential by assisting them in strengthening their organizational capacity (De Vita et al., 2001; Letts et al., 1999; Light, 2004; McKinsey & Company, 2001).
The capacity-building logic model assumes that areas of limited organizational capacity inhibit the degree to which NPOs can facilitate program and participant outcomes. Technical assistance and supplemental funds provided through capacity- building assistance are expected to lead to improvements in business operations
550 Nonprofit and Voluntary Sector Quarterly 43(3)
(short-term outcomes), more effective service delivery and, ultimately, to better out- comes for service recipients and their communities (a nuanced version of this logic model for the CCF Demonstration Program is presented in Exhibit 1.1).
In the nonprofit field, the specific definition of “organizational capacity” varies. Definitions generally comprise the skills, practices, and systems that allow NPOs to operate more effectively and sustainably. Key components of capacity building often cited in the literature include the implementation of financial controls and clear oper- ating oversight, as well as policies and procedures related to staffing, governance, finances, long-term sustainability, and expansion of services (Brown, 2007; Connolly & York, 2002; De Vita et al., 2001, Glickman & Servon, 1998; Rodriguez & Herzog, 2003).
Capacity-building programs proliferated in the late 1990s and early 2000s, includ- ing the investment of government funds in programs to build the capacity of potential NPO partners. While considerable resources have been devoted to capacity-building programs, we have limited rigorous evidence about the extent to which such programs actually build capacity, and if they do, whether nonprofits with greater capacity actu- ally have larger impacts on clients and constituents (Herman & Renz, 1998; Light, 2004; Linnell, 2003; Sobeck & Agius, 2007; Wing, 2004). While there are a few evalu- ations providing nonexperimental evidence that capacity building does increase NPOs’ capacity, and therefore may make them more effective at serving their clients (Abt Associates, 2007; Abt Associates & Branch Associates, 2008;1 Brown, 2008; Doherty & Mayer, 2003; Elliott, 2002; Glickman & Servon, 2003; Kapucu et al., 2011; Klein, et al., 2009; Leake et al., 2007; Markovitz & Magged, 2008; Millese & Bies, 2007; Patrizi et al., 2006; Popescu & Dewan, 2009; Sobeck, 2008; Sobeck & Agius, 2007), these studies are open to concerns about internal validity. Furthermore, these studies almost exclusively consider the extent to which NPO capacity-building programs actually increase the capacity of NPOs (i.e., columns 1 through 3 on the logic model presented in Exhibit 1). There has been virtually no work on the key follow-on ques- tion: the extent to which NPOs with greater capacity are more effective at helping clients achieve their desired outcomes (i.e., columns 4 and 5 on the logic model pre- sented in Exhibit 1). The current study focused on studying the former question, using rigorous methods to further the field; the latter question was beyond the scope of the study and will require future research.
Capacity-Building Program Description
The federal government has funded several programs designed to increase NPOs’ organizational capacity. Those programs included the CCF, administered by the Administration for Children and Families (ACF) within the Department of Health and Human Services, the Department of Justice’s Prisoner Reentry program and the Department of Labor’s Ready4Work program. Each of these programs provides funds for faith-based and community NPOs to access capacity-building services (e.g., work- shops, one-on-one training).
551
1. S
el ec
tio n
of S
er vi
ce
Re ci
pi en
ts
2. A
ss es
sm en
t o f
O rg
an iz
at io
na l
Ca pa
ci ty
N ee
ds
3. In
te rm
ed ia
ry D
el iv
er s
Se rv
ic es
R el
at ed
to :
4. S
ho rt-
Te rm
N PO
O ut
co m
es
5: M
id -T
er m
O
ut co
m es
6. L
on g-
Te rm
O
ut co
m es
CC F-
fu nd
ed
In te
rm ed
ia rie
s se
le ct
NP
O a
pp lic
an ts
to
re ce
ive c
ap ac
ity -b
ui ld
in g
as sis
ta nc
e.
CC F-
fu nd
ed
In te
rm ed
ia rie
s as
se ss
NP
O s
to d
et er
m in
e wh
er e
or ga
ni za
tio na
l ca
pa cit
y m
ig ht
b e
in hi
bi tin
g in
cr ea
se d
pr og
ra m
o r p
ar tic
ip an
t ou
tc om
es .
O rg
an iza
tio na
l De
ve lo
pm en
t:
St ra
te gi
c Pl
an ni
ng
• W
rit te
n m
iss io
n st
at em
en t
• Re
gu la
r a ss
es sm
en ts
o r o
rg an
iza tio
na l s
tre ng
th s
an d
ne ed
s •
W rit
te n
st ra
te gi
c pl
an d
oc um
en tin
g an
nu al
a nd
lo
ng -ra
ng e
go al
s
O rg
an iza
tio n
de liv
er s
be tte
r/m or
e se
rv ice
s to
it s
cli en
ts
In di
vid ua
ls an
d fa
m ilie
s se
rv ed
b y
NP O
s ac
hi ev
e im
pr ov
ed o
ut co
m es
M an
ag em
en t a
nd
Ad m
in ist
ra tio
n
• An
nu al
p er
fo rm
an ce
re vie
ws fo
r p ai
d an
d un
pa id
st
af f
• W
rit te
n jo
b de
sc rip
tio ns
fo r p
ai d
an d
un pa
id s
ta ff
po sit
io ns
•
Co m
pu te
r s of
tw ar
e to
k ee
p or
ga ni
za tio
n’ s
re co
rd s
an d
fin an
cia l in
fo rm
at io
n
Fi na
nc ia
l M an
ag em
en t
• Bu
dg et
s pr
ep ar
ed o
n a
re gu
la r b
as is
• Sp
ec ific
p er
so n
is de
sig na
te d
to h
an dl
e fin
an cia
l m
an ag
em en
t •
Ex te
rn al
a ud
it on
a re
gu la
r b as
is
Pr og
ra m
D ev
el op
m en
t
• In
cr ea
se in
n um
be r o
f c lie
nt s
se rv
ed
• Tr
ac k
ty pe
s an
d am
ou nt
s of
p ro
gr am
s er
vic es
pr
ov id
ed
• O
bt ai
n cli
en t f
ee db
ac k
• Tr
ac k
cli en
t o ut
co m
e da
ta
Re so
ur ce
D ev
el op
m en
t
• To
ta l r
ev en
ue in
cr ea
se s
• Nu
m be
r o f r
ev en
ue s
ou rc
es in
cr ea
se s
• Hi
gh er
ra tio
o f g
ra nt
a pp
lic at
io ns
s ub
m itt
ed to
ap
pl ica
tio ns
a pp
ro ve
d •
Fu nd
in g
fro m
n ew
s ou
rc es
Co m
m un
ity E
ng ag
em en
t •
In cr
ea se
d ac
tiv itie
s to
e xp
an d
aw ar
en es
s of
th e
or ga
ni za
tio n
• In
cr ea
se d
di ve
rs ity
o f p
ar tn
er sh
ip a
rra ng
em en
ts
Le ad
er sh
ip D
ev el
op m
en t
• Ex
ist en
ce o
f a B
oa rd
o f D
ire ct
or s
• Fo
rm al
o rie
nt at
io n
fo r B
oa rd
m em
be rs
•
In cr
ea se
in a
pp ro
pr ia
te B
oa rd
a ct
ivi tie
s •
Bo ar
d of
D ire
ct or
s is
co m
po se
d of
m em
be rs
w ith
di
ve rs
e ex
pe rti
se a
nd b
ac kg
ro un
ds
C C
F D
em on
st ra
tio n
P ro
gr am
E va
lu at
io n
E x h
ib it
1 : Lo
gi c
M o de
l - C
o m
pa ss
io n
C ap
it al
F un
d D
em o ns
tr at
io n
Pr o gr
am .
552 Nonprofit and Voluntary Sector Quarterly 43(3)
This article reports on the random assignment evaluation of CCF. Specifically, the ACF-funded evaluation focused on CCF’s Demonstration Program, one of CCF’s principal and largest components. The CCF program focused on organizations’ man- agement capacity and was based implicitly on what is perceived to be important for nonprofit organizations’ effectiveness, including such things as a formal mission state- ment, a strategic plan, human resource systems, an independent audit, and an informa- tion technology system or systems (Sowa, Selden, & Sandfort, 2004).
Specifically, the Demonstration Program funded intermediary organizations to deliver capacity-building training, technical assistance, and subawards (financial assistance) to NPOs in five critical areas:
•• Organizational Development: nonprofit incorporation; board governance; existence or adoption of systems related to human resources, finances, informa- tion technology; existence or adoption of specific policies and procedures; and fiscal controls.
•• Program development: expansion or enhancement of program or service delivery; program monitoring; and evaluating program outcomes.
•• Revenue development: diversified funding sources; financial sustainability; and donor development.
•• Leadership development: leadership professional and career development; staff professional and career development; and volunteer training and development.
•• Community engagement: community asset mapping; level of community par- ticipation; use of community needs assessments; and leveraging resources/col- laborative relationships.
Through CCF’scapacity-building assistance in some or all of these areas, NPOs were expected to improve their effectiveness and overall management, for example, their ability to develop and train staff, expand the types and reach of their programs, and improve their access to and management of funds from diverse sources, including federal funds (Administration for Children and Families, 2009). The CCF Logic Model, illustrated in Exhibit 1, further illustrates the connection between the services delivered by intermediaries and the outcomes anticipated by program designers.
Services Delivered
In the target year for the impact evaluation (2006), 10 intermediary grantees were selected, each receiving grants of approximately US$500,000 to deliver capacity- building services to NPOs over a 17-month period. These services included three types of activities: group training, customized technical assistance (TA), and financial sub- awards. At least 40% of the intermediary’s grant funds were to be used for subawards to NPOs that received either TA or group training (i.e., not subawards, but no other capacity-building services). The one-on-one technical assistance was individualized
Minzner et al. 553
based on NPO needs and was provided by either intermediary staff or consultants. The subawards were “working capital” grants and were typically used to purchase equip- ment and software, to update marketing materials such as brochures and websites, and to hire consultants for specific capacity needs. However, program rules stated that subawards could not be used by NPOs to fund direct client services.
Findings from the evaluation’s process study (described below in the Data and Measures section) indicated that topics commonly addressed through both group train- ing and one-on-one technical assistance included: grant writing and fundraising, stra- tegic planning, board development, community outreach, and financial management. Subaward funds were used to purchase equipment, typically but not exclusively com- puters and financial management and database management software, to update mar- keting materials such as brochures and websites, and to hire consultants for capacity building in areas such as board development, strategic planning, and obtaining 501(c) (3) status.
NPO Characteristics
The NPOs included in the CCF impact study were a diverse group. The organizations varied in years of operation and base revenue. Organizations were approximately evenly split between faith-based and secular. Thirty-four percent of the NPOs had existed for less than 5 years, 32% had existed for 5 to 15 years, and 34% had existed for 16 years or more. In terms of their annual revenue, one quarter of the organizations reported revenues under US$25,000 while 40% had annual revenues over US$150,000; the remaining 34% reported revenues between US$25,000 and US$150,000.
At the time they applied for capacity-building assistance, the majority of the NPOs were 501(c)(3)s (68%), had Boards of Directors (80%), and were engaged in community partnerships (88%). However, over half of the NPOs had a part-time or unpaid executive director; most other staff were unpaid and had no job description or performance reviews. Many lacked standard best practices, such as written strate- gic plans, regular organizational assessments, and a system for tracking participant outcomes.
Limitations of Study
Even though the study used a random assignment design, there are some limitations that should be considered when contemplating the results.
•• Results are self-reported. The survey and key informant interview data were entirely self-reported; there was no independent verification.
•• Unvalidated measurement instrument. Because a study of this size and depth had not previously been conducted, it was not possible to use a previously vali- dated instrument for measuring capacity building. Instead, the evaluation
554 Nonprofit and Voluntary Sector Quarterly 43(3)
developed a measurement tool based on the existing literature and validated through an expert review process.
•• Measuring capacity building. Since the study examined changes in capacity over a short timeframe, it was only possible to measure capacity built during that period, but not changes in services delivered and improvements in client outcomes. Given the scale of funders’ and intermediary agencies’ investment, it is possible that observed changes may lead to increases in services delivered and improvements in client outcomes. However, this study was not designed to measure these future outcomes, nor was it designed to identify links between individual capacity-building activities and specific capacity-building outcomes (e.g., how increases in IT capacity alone might lead to more clients served).
•• Limits to generalizability. It is difficult to generalize results about the impact of capacity building because: (a) organizations vary at the outset in their size and sophistication; and (b) they vary in their goals for capacity building. For example, we learned from the process evaluation that some NPOs sought tech- nical assistance to increase their ability to serve more clients. In contrast, other NPOs sought to operate more effectively and efficiently at their current scale. As a result, in our instruments and analysis we have made efforts to detect movement from a number of starting points toward a variety of capacity-build- ing goals, needs, and priorities (e.g., by including two types of outcome ques- tions, as well as measuring capacity changes across a variety of capacity domains). However, results are based on the average change, so they may not generalize to all similar organizations.
Randomization Design
The study sample included the entire population of NPOs that applied for assistance and were determined eligible by the 10 intermediary organizations awarded CCF Demonstration Program grants in the 2006 grant cycle (September, 2006). The NPOs that met the intermediaries’ self-defined eligibility criteria were randomly assigned either to the program group that was to receive capacity-building services (i.e., a sub- award and/or technical assistance) from the intermediary or to a control group that would not receive intensive capacity-building assistance for the duration of the study (an embargo period of 15 months). During the application period, 454 NPOs were enrolled in the Impact Study (237 assigned to the program group and 217 to the control group).
Prior to conducting our analysis, we tested whether program-group members reported receiving significantly more capacity-building assistance than control-group members, a key assumption in the study design. This was particularly important in this study because control-group members were not barred from attending group trainings offered by the intermediaries, nor from receiving intensive services from other capac- ity-building providers. This test was conducted using data from the retrospective fol- low-up survey that all study members were asked to complete (see the Data and Measures section for a description of the follow-up survey). Analysis showed that
Minzner et al. 555
87% of the program-group members received capacity-building assistance during the study period as compared to 52% of the control group, which is a difference significant at the 0.01 level. In addition, Exhibit 2 illustrates that in all topic areas program-group members were significantly more likely to receive assistance than were organizations in the control group.
Data and Measures
As part of the application process and before random assignment, NPOs completed a self-administered baseline survey which was developed for the study (see the follow- ing section for additional detail). This timing (during application, before randomiza- tion) ensured that the baseline survey was completed by organizations in both the program and control groups. The survey was typically completed by the organization’s executive director or a knowledgeable program staff member, who consulted with col- leagues when they needed additional information.
To assess change and improvement in various areas of organizational capacity, 15 months after random assignment the research team administered a follow-up survey to both program and control group NPOs. A hard copy of the follow-up questionnaire was mailed to the individual that completed the baseline survey; after about 4 weeks
Exhibit 2: Assistance Received.
Topics of Assistance
% of NPOs that Received Assistance in Topic Area
Level of SignificanceProgram Group Control Group
Resource Development, Fundraising, & Grant Writing
77% 36% **
Strategic Planning 69% 24% **
Board Development 68% 27% **
Networking, Collaboration, Partnerships
61% 26% **
Human Resources & Volunteer Management
57% 20% **
Evaluation & Outcome Measurement
56% 16% **
Program Design 52% 16% **
Financial Management 49% 19% **
Becoming a 501(c)(3) 23% 7% *
All data presented in this paper have been weighted. * p-value < 0.05; ** p-value <0.01 Source: CCF Demonstration Evaluation NPO Survey
556 Nonprofit and Voluntary Sector Quarterly 43(3)
of follow-up, a shortened version of the questionnaire was sent to the remaining non- respondents. The follow-up survey response rate was 87%—76% of the organizations completed the full survey and 11% completed a shortened version of the survey.
In addition to obtaining information directly from NPOs, intermediaries completed a survey on their technical and financial assistance processes (February, 2008) and participated in in-person, 2- to 3-day site visits. During these site visits, senior evalu- ation team members gathered information about the intermediaries’ missions and his- tories, the services they delivered, and NPOs’ receptiveness to the CCF services provided. This information was collected through semistructured individual and small group interviews with the CCF program director, key intermediary staff, and, when applicable, program volunteers and partners. The evaluation team used data from the surveys together with data collected during site visits to understand and document intermediaries’ approaches to carrying out the CCF-supported activities.
NPO Survey Instrument Characteristics
The research team surveyed the literature and found that funders and capacity-building providers were using a variety of different tools to assess organizational capacity, but there did not appear to be an accepted standard for measuring, or even describing, organizational capacity. Because no existing instrument was found to meet all study needs, the team chose to create a new instrument, drawing relevant questions from a variety of instruments. Knowing that intermediaries were going to customize their capacity-building assistance to meet the individual needs of NPOs across all five domains specified by the CCF funding guidelines—organizational development, pro- gram development, revenue development, leadership development, and community engagement—the survey had to be designed to measure a wide range of capacity- building outcomes, both in terms of content (e.g., board development) and depth of capacity changes.2
To measure the depth of an organization’s change in capacity, the CCF survey asked two different types of questions.3 The first type of question asked about measur- able actions taken by organizations to demonstrate growth in capacity (concrete changes), such as whether the organization implemented financial management proce- dures or whether its revenue increased. Most of these questions were yes/no questions, though there were a few continuous variables as well (e.g., number of staff, total rev- enue). The second type of question measured incremental steps toward capacity (con- tinuum of focus). This second set of questions captured organizations’ perception of their progress in addressing a specific capacity area. NPOs rated their status on a scale ranging from “know we should work on this, but we lack the time or resources” to “not a focus because we are satisfied with our achievement in this area.” NPOs’ progress along this scale was taken as indicating their progress toward building specific capaci- ties. By measuring NPOs’ movement along this continuum it was possible to better understand the CCF Demonstration Program’s success in helping organizations mean- ingfully engage in the capacity-building process.4
Minzner et al. 557
Analysis Methods
Our analysis methods included the assignment of proper weights, joint testing to address the problem of multiple comparisons, and individual item and subgroup analy- sis using a regression model.
To estimate program impacts, we first weighted the sample to adjust for nonre- sponse to the follow-up survey; these weighted data were used to estimate program impacts on individual outcomes as well as to conduct joint tests for significance. Impacts were estimated using regression-adjusted treatment/control differences in each outcome measured at follow-up. The regression models controlled for the base- line value of the outcome being tested and various NPO characteristics.
We also completed an additional level of analysis because there were 241 out- comes, which raises concerns about multiple comparisons; that is, when there are that many measures, a few items will often appear (using uncorrected tests) to be statisti- cally significant merely by chance (Schochet, 2008). Following Schochet, for each of the five critical areas we performed a joint test to determine whether there was any evidence that the intervention collectively affected all of the outcomes in each critical area (e.g., Organizational Development).
Finally, we examined the impacts within subgroups of NPOs to determine whether the services work better for some types of NPOs than others. These subgroups included organization type (faith-based or secular); organization age; paid executive director; organization revenue level; and intermediary’s previous experience providing techni- cal and financial assistance.5 To test for differences in impacts across subgroups, we augmented the basic impact model with an interaction between treatment and a dummy variable for one of the values of the subgroup.
Results
The Impact Evaluation showed that NPOs in the program group—those eligible to receive capacity-building services from CCF intermediary grantees—demonstrated sig- nificantly higher levels of capacity at follow-up in each of the five critical areas (Exhibit 3).6 The program group also had significantly higher capacity growth than the control group on several individual outcome measures in each of the five areas (Exhibit 4). Specifically, NPOs receiving capacity-building services from CCF intermediary grant- ees reported significant (at 5%) improvement on several measures of organizational development (20 of 64), program development (20 of 44), revenue development (21 of 52), leadership development (seven of 17), and community engagement (five of 30).
The next five subsections review significant outcomes in each domain. Full results are included in the online appendix (available at http://nvsq.sagepub.com/supplemental).
Organizational Development
NPOs receiving capacity-building services from CCF intermediary grantees reported significant improvement on several measures of long-term planning, human resources management, technology access and use, and financial management systems.
558 Nonprofit and Voluntary Sector Quarterly 43(3)
Exhibit 3: Joint Tests for No Impact, by Critical Area.
Critical Area^ Mean Treatment
Effect Standard Error of Treatment Effect p-value
Leadership Development (17 measures)
0.2095 0.0408 0.000 **
Concrete Changes (1 measure) NA NA NA NA
Continuum of Focus (4 measures) 0.1465 0.0561 0.026 *
Organizational Development (67 measures)
0.1487 0.0241 0.000 **
Concrete Changes (39 measures) 0.1233 0.0255 0.001 **
Continuum of Focus (24 measure) 0.2273 0.0402 0.000 **
Program Development (44 measures) 0.2335 0.0244 0.000 **
Concrete Changes (19 measures) 0.1180 0.0471 0.031 *
Continuum of Focus (24 measures) 0.3167 0.0290 0.000 **
Revenue Development (50 measures) 0.1399 0.0254 0.000 **
Concrete Changes (32 measures) 0.1191 0.0297 0.002 **
Continuum of Focus (16 measures) 0.1881 0.0452 0.002 **
Community Engagement (30 measures)
0.1686 0.0341 0.001 **
Concrete Changes (26 measures) 0.1292 0.0415 0.011 *
Continuum of Focus (4 measures) 0.0247 0.0584 0.002 **
Notes: ^ Training-related measures were excluded from the second round of analysis. * p-value<.05 ** p-value<.01
Exhibit 4: Summary of Results, by Critical Area.
Capacity Domain
Total Outcomes
Total Number of Indicators
Number of Significant Indicators
Percent Significant
Organizational Development
64 20 31%
Program Development 43 20 47%
Revenue Development 52 21 40%
Leadership Development 17 7 41%
Community Engagement 30 5 17%
Total 206 73 35%
Minzner et al. 559
Specifically, as a result of the CCF services more nonprofits had written strategic plans, completed organizational assessments, written job descriptions for paid staff, and a designated person responsible for financial management. NPOs also increased their level of focus on developing effective financial management systems, recruiting and managing volunteers more effectively, and recruiting and developing board members with a wide range of expertise and community backgrounds. There was no impact on staff size, 501(c)(3) status, board composition, use of staff performance reviews, or the existence of financial management procedures.
Program Development
The CCF services also improved NPOs’ service delivery capacity. In the areas of recordkeeping and evaluation, the number of NPOs in the CCF program that were keeping systematic records on services provided to clients and developing or imple- menting plans to improve outcome measurement systems increased significantly. In the areas of service delivery and program design, significantly more NPOs in the CCF program developed or implemented plans to increase the number of clients served, expanded services to include new types of clients and/or geographic areas, added new types of services, and used new approaches to improve the quality of existing services. There was no impact on the actual number of clients served, the number of NPOs that added or expanded program areas, or the number of NPOs that measured client outcomes.
Revenue Development
NPOs that received CCF services increased their capacity in several areas of revenue development. Specifically, significantly more program-group organizations had writ- ten revenue development plans, hired grant writers to train staff, participated in devel- opment training, and increased their overall organizational focus on revenue development planning. The number of CCF program-group NPOs that applied for or received grants from any source in the prior 12 months increased significantly. Moreover, organizations that had never applied for federal funding were more likely to have applied for and received federal funding as a result of the CCF. While their total number of funding sources remained the same between baseline and follow-up, program-group NPOs were more likely to have obtained funding from new, additional sources during the follow-up period, which indicates that organizations may have been diversifying their funding sources. There was no impact on the total number of grant applications submitted, pending, or approved, or on total grant funds received.
Leadership Development
The CCF program also improved NPOs’ leadership development capacity. In particu- lar, their executive directors completed a greater number of leadership training ses- sions, and they increased their level of focus on providing leadership and job skill
560 Nonprofit and Voluntary Sector Quarterly 43(3)
development opportunities for staff. In the area of board development, CCF services increased the percentage of NPOs that sent board members to training about their roles and responsibilities, and strengthened the overall focus of the organizations on improv- ing board performance. There was no impact on the number of organizations whose directors or staff met regularly with a mentor.
Community Engagement
Within this domain, the percentage of NPOs receiving CCF services that engaged in partnerships in their communities (especially with businesses and schools) increased significantly. There was also a significant increase in the percentage of NPOs that used both paid and unpaid advertisements to expand awareness of their organizations among potential partners and funders. There was no impact on the use of websites, written materials, or presentations to expand awareness of potential partners and funders, or on efforts to expand awareness in the general community.
Areas of Greatest Organizational Capacity Improvement
While NPOs in the program group experienced significant improvements in capacity in each critical area of organizational capacity, certain outcomes showed greater mag- nitudes of impact. Using effect size—that is, impact divided by standard error of out- come in the control group—as the measure of magnitude of improvement, the outcomes with the greatest improvement are presented in two exhibits. As seen in Exhibit 5, relatively large effects (0.40 or larger) were found on several “concrete change” measures. These are important foundational activities for organizations. Within the “continuum of focus” measures (Exhibit 6), there were large effects related to training, expanding program services, increasing evaluation expertise, and identify- ing new funding sources.
In reviewing the results, we noticed that the largest effect sizes were related either to training or to small incremental steps measured by the continuum questions. This obser- vation led us to question whether these more modest capacity changes were driving the overall results of the study, rather than the potentially more meaningful concrete capac- ity outcomes. To answer this question, we performed a second round of joint tests, removing all training-related variables (e.g., “In the past 12 months, number of staff that participated in training related to: service delivery”) and separating the “concrete change” measures and the “continuum of focus” measures into two separate groups. The results of the additional analysis were consistent with the initial findings: the program group organizations experienced significantly larger increases in capacity whether mea- sured on the continuum of focus or in terms of actual capacity outcomes (Exhibit 3).
Subgroup Differences
To determine whether the services work better for some types of NPOs than for oth- ers, outcomes for five subgroups were analyzed, as measured at baseline: (a) organi- zation type: faith-based or secular organization; (b) age of the organization; (c) size
Minzner et al. 561
Exhibit 5: Areas of Greatest Capacity Improvement—Concrete Changes.
Survey Item Effect Size p-value
(Adjusted)
Organizational Development
In the past 12 months, organization conducted or participated in an assessment of organizational strengths/needs
0.54 **
In the past 12 months, number of staff that participated in training related to: Management & Administration
1.79 *
The number of functioning computers that the organization owns is sufficient for organization/staff needs
0.43 *
Leadership and Staff Development
Number of types of training in which head organization participated in the past 12 months
0.59 **
In the past 12 months, any Board member participated in training/learning opportunities to learn more about governance/roles and responsibilities of Board members
0.50 *
Revenue Development
In the past 12 months, the head of the organization participated in training related to fundraising
0.44 **
In the past 12 months, number of staff that participated in training related to: Fundraising
1.05 **
In the past 12 months, number of applications submitted for funding from federal government agencies
0.47 *
Total number of sources of revenue/funding received over the past 12 months
0.40 **
Among organizations that had never applied for or received federal funding at the time of the baseline survey, the number that had received federal funding at the time of the follow-up survey
0.45 *
*p-value < .05; ** p-value < .01
(based on annual expenditures); (d) pay status of the executive director at baseline; and (e) whether the intermediary had prior CCF grant experience.
Our results showed that there was not a consistent pattern of findings to suggest that one type of organization is more receptive to capacity-building assistance. In fact, very few differences in outcomes between faith-based and secular organizations across the five domains of organizational capacity were found. Similarly, there were no differences in the effect of the services based on organization age or prior experience of the intermediary. There were differences on some measures based on the size of the organization and those with paid and unpaid executive directors but no consistent pattern could be identified.
562
E x h
ib it
6 :
A re
as o
f G
re at
es t
C ap
ac it y
Im pr
o ve
m en
t— C
o nt
in uu
m o
f C
ha ng
e.
L e v e l o
f fo
c u
s o
n :
A t
a m
in im
u m
, k n
o w
s
it s
h o
u ld
w o
rk o
n t
h is
, b
u t
la c k s t
h e t
im e o
r re
s o
u rc
e s
A t
a m
in im
u m
, h
a s
d e v e lo
p e d
p la
n s t
o w
o rk
o
n t
h is
, b
u t
h a s n
o t
im p
le m
e n
te d
t h
e m
y e t
A t
a m
in im
u m
, h
a s
im p
le m
e n
te d
s te
p s
to a
d d
re s s f
o c u
s
a re
a
N o
t a f
o c u
s b
e c a u
s e
o
rg a n
iz a ti
o n
i s s
a ti
s fi
e d
w
it h
i ts
a c h
ie v e m
e n
t in
th
is a
re a
O rg
a n iz
a tio
n a l D
e ve
lo p m
e n t
R e cr
u iti
n g , d e ve
lo p in
g , a n d m
a n a g in
g v
o lu
n te
e rs
m o re
e ff e ct
iv e ly
: 0 .4
5 **
D e ve
lo p in
g a
B o a
rd w
ith t ie
s to
d iff
e re
n t co
n st
itu e n ci
e s:
0 .4
8 **
L e a d e rs
h ip
a n d S
ta ff D
e ve
lo p m
e n t
P ro
vi d in
g s
ta ff w
ith p
ro fe
ss io
n a l d
e ve
lo p m
e n t a n d t ra
in in
g t
o
e n h a n ce
s ki
lls in
s e rv
ic e d
e liv
e ry
o r
sk ill
s in
a d m
in is
tr a tio
n
a n d m
a n a g e m
e n t
0 .4
7 **
P ro
g ra
m D
e ve
lo p m
e n t
In cr
e a si
n g t h e n
u m
b e r
o f cl
ie n ts
s e rv
e d b
y th
e o
rg a n iz
a tio
n :
0 .4
3 **
In cr
e a si
n g t h e n
u m
b e r
o r
sc o p e o
f se
rv ic
e s
o ff e re
d t o
cl ie
n ts
: 0 .4
7 **
L e ve
l o f fo
cu s
o n in
co rp
o ra
tin g a
n e w
a p p ro
a ch
t o s
e rv
ic e s
to im
p ro
ve q
u a lit
y/ e
ff e ct
iv e n e ss
: 0 .4
9 **
S tr
e n g th
e n in
g t h e o
rg a n iz
a tio
n 's
a b ili
ty t o e
va lu
a te
it s
o ve
ra ll
e ff e ct
iv e n
e ss
: 0 .5
5 **
0 .4
1 **
D e ve
lo p in
g a
w a y
to c
o lle
ct m
o re
in fo
rm a tio
n a
b o u t cl
ie n ts
, in
cl u d in
g t h e n
u m
b e r
a n d c
h a ra
ct e ri st
ic s
o f cl
ie n ts
a s
w e ll
a s
h o w
t h e y
a re
h e lp
e d b
y th
e p
ro g ra
m s:
0 .4
2 **
R e ve
n u e D
e ve
lo p m
e n t
Id e n tif
yi n g a
n d p
u rs
u in
g n
e w
s o u rc
e s
o f in
-k in
d d
o n a tio
n s
0 .5
6 **
D e ve
lo p in
g a
f u n d -d
e ve
lo p m
e n t p la
n (
in cl
u d in
g s
e tt in
g
fu n d ra
is in
g g
o a ls
0 .4
1 *
*p -v
a lu
e <
. 0 5 ; **
p -v
a lu
e <
. 0 1 .
Minzner et al. 563
These results suggest that the range and types of services provided through CCF interme- diary grantees are relevant and useful to many different types of organizations.
Discussion
Nonprofit organizations play a critical role in addressing the needs of local communities and our nation as a whole. The federal government, through the CCF and other initia- tives, has invested in building the organizational capacity of NPOs with the expectation that this investment will result in improvements in organizational structure and internal operation, more effective and efficient service delivery and, ultimately, better outcomes for service recipients. The results of this random assignment evaluation show that initial expectations were met for the CCF program. To the limited extent that the evaluation was able to measure changes in internal operations and service delivery, positive changes were also observed. Expectations that capacity building would lead to more effective and efficient service delivery and to better outcomes for service recipients were not mea- sured because they were beyond the evaluation’s scope and design.
In the following section we discuss how the CCF results fit into the broader context of the nonprofit capacity-building field, noting areas where additional research might improve knowledge about nonprofit capacity building.
Implications for the Field
Because of its random assignment methods, this study represents a critical advance in the evidence base for the effectiveness of nonprofit capacity building. The results of this CCF evaluation support previous findings form other observational method studies. Like Glickman and Servon (2003), Sobeck and Agius (2007), and Leake et al. (2007), the CCF evaluation found that capacity building increased organizational capacity across the range of capacity areas measured, though not on all capacity indicators.
Having said that, it is important to note that previous studies relied on cross-section or prepost designs. Compared to these studies, this random assignment study provided strong evidence of a causal link between capacity-building activities and the observed changes in capacity. However because these findings cannot be generalized beyond the CCF program and the organizations studied, more rigorous studies are needed to determine whether capacity can be increased through a range of intentional capacity- building activities.7
These findings also raise additional questions for future research.
What is the Relative Effectiveness of Different Types of Capacity-Building Assistance? While the CCF Impact Evaluation found positive impacts on organizational capacity, the study was not designed to provide insights about which types (e.g., training, individual technical assistance, and financial assistance) of capacity-building assistance or com- binations of types contributed to the impacts. Because we did not randomly assign specific treatments we were unable to assess the impact of specific treatments in the random assignment framework. Future studies might adopt this more complicated
564 Nonprofit and Voluntary Sector Quarterly 43(3)
design in order to help the field better understand whether all three types of assistance are critical to producing impacts or whether only one of them or two in combination produce impacts. Knowing this might help funders and others to develop streamlined capacity-building programs focusing on the methods of delivery that provide the greatest contribution to nonprofit capacity change. Similarly, understanding whether building capacity in a given content area (e.g., financial management) leads to broader increases in capacity might also increase the efficiency of capacity-building programs. Future research could address this question through studies that randomly assign NPOs to different models and/or areas of assistance.
What Are the Intermediate and Long-Term Effects of Capacity-Building Assistance on Organi- zational Capacity? The Impact Evaluation was designed to assess short-term (within 15 months) outcomes anticipated as a result of capacity-building efforts. While a number of improvements were reported within the 15-month follow-up period, greater gains across more areas of capacity at later points in time are plausible though not guaranteed. Camp- bell (2011) found similar short-term increases in capacity as a result of California’s Community and Faith-based Initiative; however, few of the impacts remained at the 3-year follow-up. Future studies should include follow-up surveys and interviews with the same organizations 2 to 4 years after receiving services to improve our understanding of how organizational change occurs—for instance, the timing of different outcomes (short-term vs. long-term), whether gains achieved early are sustainable over time, and whether incremental steps measured through the continuum of change measures lead to concrete changes. Related to the timing of different outcomes, it would also be useful to identify which outcomes are achievable in the short term and which can only be expected to be achieved over longer periods of time. By better understanding the progression of capacity building, future researchers can more accurately draw conclusions as to whether organizations are following a trajectory to greater capacity changes later.
What Are the Effects of Capacity Building on Service Delivery and Client Outcomes? The CCF Impact Evaluation was able to document improvements in short-term organiza- tional capacity. It was not, however, able to measure whether these capacity changes resulted in improved client outcomes, the ultimate goal of investments in capacity building. To estimate impact on these ultimate outcomes, longer term random assign- ment studies of programs with shared goals would be needed. In practice, the type of program that might lend itself to this next phase of capacity-building evaluation would be a capacity-building effort with a large number of homogeneous NPOs with existing client outcome data. There would also need to be a sufficient lag between the interven- tion and the outcomes such that it was plausible that capacity building would lead to capacity changes, which would in turn lead to client outcomes (Gan & Klerman, 2011). Finally, future studies would benefit from building upon the literature related to measuring organizational effectiveness. Authors such as Bagnoli and Megali (2011), Sowa, Selden, and Sandfort (2004), Shea (2011) have wrestled with these measure- ment issues and proposed various theoretical and quantitative models for linking
Minzner et al. 565
capacity and effectiveness. A rigorous evaluation could incorporate some combination of these theories and test their validity.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, author- ship, and/or publication of this article: This research was funded by the Administration for Children and Families, U.S. Department of Health and Human Services, under Contract No. 233-02-0088, Task Order No. 12, “Compassion Capital Fund Evaluation.” An extended version of this article is available from the authors upon request. This research does not necessarily represent the official positions of Abt Associates Inc. or its research sponsors.
Notes
1. Eisinger (2002) reports opposite findings and claims that increased capacity does little to improve organizational effectiveness.
2. Because of the wide range of topics covered by the CCF capacity building assistance, the survey instrument was very lengthy. It included items intended to capture changes in each of the targeted capacity areas. The downside to this exhaustive approach to change measurement was the fact that each capacity element was treated as discreet. When in fact, organizational capacity is a complicated and dynamic system in which the various elements of capacity interact. Future capacity building evaluations should consider how to use systems dynamic theories as a lens for capturing the interrelated nature of the capacity elements (Williams & Imam, 2007).
3. CCF’s five critical areas include organizational development, program development, rev- enue development, leadership development, and community engagement.
4. We chose to include two different types of questions in our evaluation in order to assess measurable external changes as well as participants’ behavior changes. Wing (2004) advo- cates this approach. We developed the actual continuum of focus scale in collaboration with Bill Ryan. The scale was inspired by conventional behavior-change literature, broadly speaking, and by Connolly and York’s (2002) conceptualization of how to measure short- term outcomes of capacity-building interventions.
5. These subgroups are consistent with suggestions in the earlier literature including Allard (2008); Brown (2008); Cnaan and Boddie (2006); Cornforth and Simpson (2003); Dart, Bradshaw, Murray, and Wolpin (1996); Diiulio (2004); Leake et al., (2007); Millese and Bies (2007); Rochester (2003); Schuh and Leviton (2006); Smith, Bartkowski, and Grettenberger (2005); and Sobeck (2008).
6. Exhibit 2 includes the results of the overall joint test for no impact as well as results from the tests that analyzed concrete changes and continuum of focus items separately, which will be discussed later in the report.
7. The results of the Impact Study only represent the 454 NPOs that applied for capac- ity-building assistance from the 10 CCF intermediary grantees funded in 2006. However, to the extent to which these NPOs are similar to other NPOS, the results
566 Nonprofit and Voluntary Sector Quarterly 43(3)
are suggestive of how a broader population might benefit from similar capacity- building assistance.
References
Abt Associates, Inc. (2007). Findings from a retrospective survey of faith-based and commu- nity organizations (FBCOs) (ACF-OPRE Report). Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research and Evaluation.
Abt Associates, Inc., & Branch Associates, Inc. (2008). An assessment of the Compassion Capital Fund Demonstration Program: Outcome study final report (ACF-OPRE Report). Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research and Evaluation.
Abt Associates, Inc., & Branch Associates, Inc. (2010). Improving Capacity among nonprof- its—Random assignment impact study of the Compassion Capital Fund Demonstration Program (ACF-OPRE Report). Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research and Evaluation.
Administration for Children and Families. (2009). Compassion capital fund fact sheet. Retrieved from http://www.acf.hhs.gov/programs/ocs/ccf/about_ccf/facts.html
Allard, S. (2008). Accessibility and stability of nonprofit service providers: Faith-based and community-based organizations in urban and rural America. In P. Joshi, S. Hawkins & J. Novey (Eds.), Compendium of research papers presented at the faith-based and community initiatives conference on research, outcomes, and evaluation. Washington, DC. Retrieved from http://aspe.hhs.gov/fbci/comp08/
Allard, S. (2009a). Out of reach: Place, poverty, and the new American welfare state. New Haven, CT: Yale University Press.
Allard, S. (2009b, November). The overlooked role of faith-based organizations and secular nonprofits in the modern American welfare state. Paper presented at the Annual Meetings of the Association for Public Policy Analysis and Management, Washington, DC.
Bagnoli, L., & Megali, C. (2011). Measuring performance in social enterprises. Nonprofit and Voluntary Sector Quarterly, 40, 149-165.
Blumenthal, B. (2003). Investing in capacity building: A guide to high-impact approaches. New York, NY: The Foundation Center.
Brown, M. (2008). Improving organizational capacity of faith-based and community- based domestic violence service providers. In P. Joshi, S. Hawkins & J. Novey (Eds.), Compendium of research papers presented at the faith-based and community initiatives conference on research, outcomes, and evaluation. Washington, DC. Retrieved from http:// aspe.hhs.gov/fbci/comp08/
Brown, W. (2007). Board development practices and competent board members: Implication for performance. Nonprofit Management & Leadership, 17, 301-317.
Campbell, D. (2011). Reconsidering the implementation strategy in faith-based policy initia- tives. Nonprofit and Voluntary Sector Quarterly, 40, 130-148.
Cnaan, R., & Boddie, S. (2006). Setting the context. Journal of Religion & Spirituality in Social Work: Social Thought, 25, 5-18.
Connolly, P., & York, P. (2002). Evaluating capacity building efforts for nonprofit organiza- tions. OD Practitioner, 34, 33-39.
Minzner et al. 567
Cornforth, C., & Simpson, C. (2003). The changing face of charity governance—The impact of organizational size. In C. Cornforth (Ed.), The governance of public and non-profit organi- zations (pp. 187-206). New York, NY: Routledge.
Dart, R., Bradshaw, P., Murray, V., & Wolpin, J. (1996). Boards of directors in nonprofit orga- nizations: Do they follow a life-cycle model? Nonprofit Management and Leadership, 6, 367-379.
De Vita, C. J., Fleming, C., & Twombly, E. C. (2001). Building nonprofit capacity: A frame- work for addressing the problem. In C. J. De Vita & C. Fleming (Eds.), Building capacity in nonprofit organizations (pp. 5-32). Washington, DC: Urban Institute.
Diiulio, J. (2004). Getting faith-based programs right. Public Interest, 155, 75-88. Doherty, S., & Mayer, S. (2003). Results of an inquiry into capacity building programs for
nonprofit programs. Minneapolis, MN: Effective Communities. Eisinger, P. (2002). Organizational capacity and organizational effectiveness among street-level
food assistance programs. Nonprofit and Voluntary Sector Quarterly, 31, 115-130. Elliott, G. L. (2002). Capacity as a fundamental objective: Definition and measurement in
the AmeriCorps*VISTA—Habitat for Humanity International Affiliate Partnerships. Washington, DC: Corporation for National & Community Service, National & Community Service Fellowship Program.
Gan, K., & Klerman, J.A. (2011). Designing the next generation of capacity building evalua- tions. Cambridge, MA: Abt Associates.
Glickman, N. J., & Servon, L. J. (1998). More than bricks: Five components of community development corporation capacity. Housing Policy Debate, 9, 497-539.
Glickman, N. J., & Servon, L. J. (2003). By the numbers: Measuring community development corporations’ capacity. Journal of Planning Education and Research, 22, 240-256.
Herman, R. D., & Renz, D. O. (1998). Nonprofit organizational effectiveness: Contrasts between especially effective and less effective organizations. Nonprofit Management & Leadership, 9, 23-38.
Jolley, S., & Meissen, G. (2009). Capacity building with faith-based and community organiza- tions. Paper presented at the biannual meeting of the Society for Community Research and Action, Montclair State University, Montclair, NJ. Retrieved from http://www.allacademic. com/meta/p301770_index.html
Kapucu, N., Healy, B., & Arslan, T. (2011). Survival of the fittest: Capacity building for small nonprofit organizations. Evaluation and Program Planning, 34, 236-245.
Klein, A., Brown, M., Small, M., Tucker, D., & Fischer, R. (2009). Evaluation of the rural domestic violence and child victimization grant program special initiative: Faith-based and community organization pilot program. Sudbury, MA: Advocates for Human Potential.
Leake, R., Green, S., Marquez, C., Vanderburg, J., Guillaume, S., & Gardner, V. A. (2007). Evaluating the capacity of faith-based programs in Colorado. Research on Social Work Practice, 17, 216-228.
Letts, C. W., Ryan, W. P., & Grossman, A. (1999). High performance nonprofit organizations: Managing upstream for greater impact. New York, NY: Wiley.
Leviton, L. C., Herrera, C., Pepper, S. K., Fishman, N., & Racine, D. (2006). Faith in action: Capacity and sustainability of volunteer organizations. Evaluation and Program Planning, 29, 201-207.
Light, P. C. (2004). Sustaining nonprofit performance: The case for capacity building and the evidence to support it. Washington, DC: Brookings Institution Press.
568 Nonprofit and Voluntary Sector Quarterly 43(3)
Linnell, D. (2003). Evaluation of capacity building: Lessons from the field. Washington, DC: Alliance for Nonprofit Management.
Markovitz, C., & Magged, L. (2008). HOPE II: Faith-based and community organization pro- gram evaluation study—Summary report. Cambridge, MA: Abt Associates Inc.
McKinsey & Company. (2001). Effective capacity building in nonprofit organizations. Reston, VA: Venture Philanthropy Partners.
Millese, J., & Bies, L. (2007). Nonprofit capacity building: Who is doing what for whom and to what end? Journal of Nonprofit Management, 11, 1-10.
Patrizi, P. A., Gross, E. K., & Freedman, S. (2006). Strength in flexibility: Lessons from a cluster of capacity building grants in the juvenile justice field. Evaluation and Program Planning, 29, 162-170.
Popescu, M., & Dewan, S. (2009). Esperanza Trabajando gains, losses, lessons learned and steps forward: An evaluation report. International NGO Journal, 4, 403-420.
Rochester, C. (2003). The role of boards in small voluntary organisations. In C. Cornforth (Ed.), The governance of public and non-profit organizations (pp. 115-130). New York, NY: Routledge.
Rodriguez, A., & Herzog, N. (2003). Replacing passionate leaders. Paper presented at the U.S. Department of Housing and Urban Development Capacity-Building Conference, Milano Graduate School of Management and Urban Policy, New School University, New York, NY.
Schochet, P. Z. (2008). Technical methods report: Guidelines for multiple testing in impact evaluations. Washington, DC: Institute for Education Sciences.
Schuh, R., & Leviton, L. (2006). A framework to assess the development and capacity of non- profit agencies. Evaluation and Program Planning, 29, 171-179.
Shea, J. (2011). Taking nonprofit intermediaries seriously: A middle-range theory for imple- mentation research. Public Administration Review, 71, 57-66.
Sherman, A. L., & Stanakis, M. (2002). Building fruitful collaboration between Florida’s faith communities and one stop career centers. Washington, DC: Hudson Institute.
Smith, S. R., Bartkowski, J. P., & Grettenberger, S. (2005). Comparative views on the role and effect of faith in social services. Albany: SUNY, Albany Rockefeller Institute of Government, The Roundtable on Religion and Social Welfare Policy.
Sobeck, J. (2008). How cost-effective is capacity building in grassroots organizations? Administration in Social Work, 32, 49-68.
Sobeck, J., & Agius, E. (2007). Organizational capacity building: Addressing a research and practice gap. Evaluation and Program Planning, 30, 237-246.
Sowa, J., Coleman, S., & Sandfort, J. (2004). No longer unmeasurable? A multidimensional integrated model of nonprofit organizational effectiveness. Nonprofit and Voluntary Sector Quarterly, 33, 711-728.
Williams, B., & Imam, I. (2007). Systems concepts in evaluation: An expert reader. Point Reyes, CA: EdgePress/ American Evaluation Association.
Wing, K. (2004). Assessing the effectiveness of capacity-building Initiatives: Seven issues for the field. Nonprofit and Voluntary Sector Quarterly, 33, 153-160.
Author Biographies
Amy Minzner is a senior associate with Abt Associates Inc. in Cambridge, Massachusetts. Her research addresses urban development, workforce policy, nonprofit capacity building, and com- munity revitalization.
Minzner et al. 569
Jacob A. Klerman is a principal associate and Abt Fellow with Abt Associates in Cambridge, Massachusetts. His research expertise includes evaluation methodology, social welfare policy, and American labor markets.
Carrie E. Markovitz is a statistician at NORC at the University of Chicago who specializes in both qualitative and quantitative evaluations of youth and community programs. Dr. Markovitz has contributed to several evaluations of small faith-based and community organizations that work with disadvantaged youth, minority populations, and victims of crime.
Barbara Fink is a senior research associate at Branch Associates, Inc. Her research addresses a variety of topics relating to low-income families and communities, including studies of capac- ity building in the nonprofit sector.
