Reflection Paper
Jonnes
Article1.pdf
Community development as health promotion: evaluating a complex locality-based project in New Zealand
Jeffery Adams, Karen Witten and Kim Conway
Abstract This article examines the evaluation of a complex public health
intervention – the Ranui Action Project (RAP). The RAP utilized a
community development approach to address the social determinants
of health inequalities in a high need, ethnically diverse, urban locality.
The rationale and impetus for the project was emerging evidence in
the public health literature on the significance of working
intersectorally with an emphasis on social capital and social cohesion
as neighbourhood-level determinants of health. This article describes
RAP and examines the strengths, limitations and challenges of the
formative, process and impact evaluation model used to evaluate the
programme’s effectiveness. Responding to diverse evaluative
expectations, while sustaining research integrity and rigour, requires a
pragmatic multi-methods approach, responsiveness to local context,
regular communication between funders, community stakeholders and
evaluators, and flexible, reflective practice.
Introduction
Since the early 1990s in New Zealand, and elsewhere, health policy has
recognized that reducing health inequalities will require action on the
social, economic and cultural determinants of health. The impact of inequi-
table access to income, housing, education and employment on the health of
specific population groups, and in the New Zealand context, Māori (indi-
genous people of New Zealand), Pacific peoples1 and low socioeconomic
140 Community Development Journal Vol 44 No 2 April 2009 pp. 140 – 157
1 Includes migrants from Pacific countries and their families and people of Pacific descent born in
New Zealand.
& Oxford University Press and Community Development Journal. 2007 All rights reserved. For permissions, please email: [email protected]
doi:10.1093/cdj/bsm049
Advance Access publication 16 November 2007
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
groups in particular, is acknowledged (Howden-Chapman and Tobias,
2000; Ministry of Health and University of Otago, 2006). In keeping with
the experiences of indigenous people elsewhere, colonization and dispos-
session and alienation from land and resources have had negative conse-
quences for Māori health (Durie, 1997; Voyle and Simmons, 1999). In
recent decades, along with demands for greater self-determination, main-
stream health and social policy and programme initiatives have increas-
ingly targeted the needs and aspirations of Māori.
Within public health discourse, there has also been a growing recognition
that the social structures and ecology of neighbourhoods impact on health
and wellbeing of residents (Baum and Palmer, 2002; Popay et al., 2003). The
development of the New Zealand Deprivation (NZDep) Index2 (Salmond
and Crampton, 2001) has facilitated investigations of area-level variations
in health and social outcomes. For example, it has been used to show that
mortality rates in New Zealand’s most deprived areas can be up to twice
as high as those found in the least deprived areas (Blakely, 2002). While
the characteristics of people living in different neighbourhoods explain
much of the observed differences in health, a number of studies internation-
ally have found that places also matter: contextual characteristics of neigh-
bourhoods also contribute to health disparities between areas (Ellen,
Mijanovich and Dillman, 2001; Pickett and Pearl, 2001).
The health-enhancing and health-inhibiting effects of various locality
characteristics have been investigated, and access to services and amenities,
opportunities for social and community participation, social capital and
social cohesion are among the neighbourhood characteristics identified as
influencing health and wellbeing (Woolever, 1992; Lynch, 2000; Ellaway,
Macintyre and Kearns, 2001). Such findings provide a rationale and evi-
dence base for health sector investment in locality-based interventions to
improve the wellbeing of residents and to reduce health disparities.
Social capital and social cohesion as important determinants of health
have entered the health inequalities debate through the work of Wilkinson
and others (Kawachi and Berkman, 2000; Wilkinson, 2000). Various mean-
ings, distinct and overlapping, are given for the terms, and debates continue
to rage in the public health literature about the utility of the concept to
health and health inequalities.3 Social capital has been theorized to arise
out of the social practices of durable social networks, generated through
exchanges of words, gifts and favours between individuals and groups
(Bourdieu, 1986). Although the conceptual development of social capital
2 The NZDep Index is a meshblock-level index derived from nine census variables: car access,
telephone access, proportion of population on a means-tested benefit, single parent households,
educational qualifications, income, housing tenure, employment status and household crowding.
3 See International Journal of Epidemiology, 33 (1), 2004.
Community development as health promotion 141
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
lies with Bourdieu (1986) and Coleman (1990), it is Putnam’s definition of
social capital as social networks and the associated norms of mutuality
and reciprocity (Putnam, 1993; Putnam, 2000; Putnam, 2004) that has
gained currency within public health practice. All the three authors
portray neighbourhood groups and organizations, and local venues that
bring people together for a common purpose, as significant sites for
forming social ties and building social capital (Bourdieu, 1986; Coleman,
1990; Putnam, 2000).
In New Zealand, locality-based community action and community devel-
opment projects in sectors such as health, welfare, housing and the environ-
ment have burgeoned during the term of the current Labour Government
(Casswell, 2001; Allen, Kilvington and Horn, 2002; Greenaway and
Witten, 2006). The theorizing that has underpinned government funding
of many of these programmes has identified social capital and social cohe-
sion as pathways to more inclusive communities and a reduction in social
and health inequalities (Huckle, Witten and Adams, 2002; Taylor, 2004).
The geographic community is conceptualized as the chief arena for tackling
such health inequalities (Billings, 2000).
Although community development has often been linked to health pro-
motion, historically there have been different views about its utility
(Epstein, Tripodi and Fellin, 1973) which have in turn led to calls for effec-
tive evaluation of such initiatives (Craig, 2002; Barr, 2005). The diversity of
the programmes that are being evaluated means that there is no one best
evaluation approach, but there are calls to ensure that the evaluation
design fits with community realities (Judd, Frankish and Moulton, 2001;
Becker, 2006). Community-driven health promotion interventions are
often complex and the time frames required for social change offer particu-
lar challenges to their evaluation (Susser, 1995; Judge and Bauld, 2001).
Although the randomized control trial and other experimental approaches
are often positioned as the most desirable evaluation designs to determine
the effectiveness of an intervention, it is not necessarily a feasible or appro-
priate option for complex community-driven programmes (Gillies, 1998;
Oakley, 1998; Judge and Bauld, 2001; Rychetnik et al., 2002). Alternative,
non-experimental evaluation models have evolved in response to the con-
tingencies of complex health promotion programmes (Pawson and Tilly,
1997; Patton, 2002). Alternative approaches to evaluation in community pro-
jects in New Zealand have ensured that formative evaluation is a critical
component of the evaluation mix (Greenaway and Witten, 2006).
This paper describes the evaluation of a complex, locality-based health
promotion programme, the Ranui Action Project (RAP). The following
section describes the RAP project – its funding, objectives and key features
of its implementation – along with the methodology used for its evaluation.
142 Jeffery Adams et al.
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
Key findings of the evaluation and reflections on the strengths and limit-
ations of the evaluation follow. The discussion section questions the scope
of the evaluation in the light of the complexity of the RAP intervention,
and the adequacy of the evaluative findings as evidence on which to base
funding decisions for similar future projects.
Background: Ranui Action Project
The RAP was set up with a very broad aim of enhancing the health and
wellbeing of Ranui people, with a particular focus on the wellbeing of chil-
dren, young people and their families.
Ranui is a community of approximately 8000 people in west Auckland,
located within the Waitakere City Council (WCC) local government area.
Housing development over recent decades has transformed Ranui from a
rural horticultural area to a fringe city suburb. The population is diverse
in terms of ethnicity with large populations of Māori, Pacific peoples and
new settler groups. Ranui has a young age profile with 52 percent of the
population aged less than 15 years. The area is predominantly low
income and characterized by associated concerns including housing
issues, unemployment, transient school rolls and poor use of and access
to health and social services. Although Ranui was assessed by the WCC
as being a community of high social and health need, the Council also ident-
ified good levels of social community organizing giving it confidence that a
community-based strategy would be appropriate in this community.
Funding
RAP was initially funded by the Ministry of Health (MOH)4 as an Intersec-
toral Health Initiative5 and approximately six months later, Ranui was also
confirmed and funded as a Stronger Community Action Fund (SCAF)6
project by the Department of Child, Youth and Family (CYF). The agencies
agreed to support RAP as a ‘seamless’ jointly funded community develop-
ment project hosted by local government (the WCC).
Objectives
The two funding agencies did however have distinct, although complemen-
tary, goals for the project. As detailed in Table 1, the MOH’s goals were
4 The project was funded by the Health Funding Authority and following health sector restructuring
by the MOH.
5 RAP is one of three intersectoral health initiatives. It continues to be funded by the MOH.
6 RAP was one of the seven initial SCAF projects. The SCAF programme has been discontinued.
Community development as health promotion 143
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
broad and oriented to health outcomes. CYF’s goals focused on partici-
pation and community development. The discourses of social capital and
social cohesion were significant to the project rationale and programme
logic underpinning RAP activities for both agencies. As Taylor (2004,
p. 69) points out, SCAF ‘draws on social capital theory and on research
that links improved outcomes for children and families to an increase in
social capital in communities’. Funding for the evaluation of RAP was pro-
vided by the MOH and CYF for the initial three and three and a half years,
respectively.
Implementation
Community development practice is well documented in health promotion
literature. At its core, it involves communities ‘identifying their health
issues, planning and acting upon their strategies for social action/change,
and gaining increased self-reliance and decision-making power as a result
of their activities’ (Labonte, 1993, p. 237). It is a foundation strategy of the
Ottawa Charter for health promotion (Germann and Wilson, 2004).
Community-level strategies are supported as a potential way to enhance
health (Hawe, 1994; Billings, 2000). Recently, many of these community
development initiatives have featured partnerships between community
and local/central government, sometimes referred to as Third Way
approaches, and have arisen in various guises in a number of countries
including Ireland (Loughry, 2002; Murphy, 2002), the UK (Blaxter, Farnell
Table 1. Goals of RAP funders
Ministry of Health To address the determinants of health in an area of high health
need To initiate actions that will reduce health and other disparities for
local residents To provide and implement a sector-wide approach to population
health Department of Child Youth and Family
To develop policies that build stronger communities, that increase people’s capacity to participate in their communities and that reduce social exclusion
To contribute towards strengthening communities through providing funding for devolved decision-making relating to the funding of locally based social services and community initiatives
To provide for a pilot to devolve funding to communities through involving the community in the assessment of social need and the decision making about how available funding will be used to address those needs
144 Jeffery Adams et al.
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
and Watts, 2003), Australia (Simpson, Wood and Daws, 2003) and New
Zealand (Craig, 2004).
RAP was successfully established as a community development project in
terms of being a bottom-up, self-governing entity focused on identifying
local community issues and needs and addressing these with their own
community solutions. Local community members were early, active partici-
pants in the project, having been involved in the negotiations with the
funders and fund holders to locate this project in Ranui. Early on, a steering
group of local residents was established, and this body ensured that from
inception, RAP adopted inclusive ways of working to enhance community
engagement. Approaches were specifically tailored to the needs of Māori
and Pacific residents and young people.
Rychetnik et al. (2002, p. 119) refer to the tendency for public health inter-
ventions to be ‘complex, programmatic and context dependent’. RAP fits
this description having facilitated a very extensive, wide ranging pro-
gramme of activity. The action plan focused on seven key areas: young
people’s development; health and social services; public safety; employ-
ment and business; environment; education; community pride/vitality/
identity. In addition, RAP hosted major community events and funded
over 60 small-scale projects initiated and run by local community members.
Evaluating RAP
The overarching purpose of the evaluation was to determine the effective-
ness of the intervention to improve the health and wellbeing of children
and families in Ranui via the critical pathways of strengthened communities
and enhanced social capital. The funders had some specific expectations of
the project’s evaluation. The MOH sought a formative and impact/outcome
evaluation including a before and after survey of residents’ perceptions of
the social relations of place. CYF’s evaluation requirements focused on a
process evaluation of the organizational structures and decision-making
processes set up by the community to meet the project’s goals.
Community representatives were initially suspicious of the funders’
evaluation requirements and challenged the evaluation team to demon-
strate what value it could bring to the project. In doing so, a common
language for evaluative practices was developed, an evaluation plan was
negotiated and an evaluation group comprising community representatives
was established to guide and critique the evaluation in its initial stages.
Debate between the evaluation group and the evaluators aimed to foster
an appreciation of evaluation within the project (among some members at
least), both at the project-wide level and within the individual projects
funded by RAP.
Community development as health promotion 145
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
The evaluation plan was grounded on an action research cycle of plan-
ning, implementing and reviewing, which sets up a recurrent feedback
loop between evaluators, community representatives and programme
implementers (Casswell, 2000, 2001; Conway et al., 2000; Conway and Cass-
well, 2003). This approach to evaluation acknowledges that the process of
evaluation itself is useful, irrespective of project outcomes (Patton, 1997,
1998). We were mindful that it was not possible for effective evaluation to
be undertaken unless there was a shared understanding among all parties
(Oliver, Spee and Wolfgramm, 2003). The approach incorporated some col-
laborative principles (Davidson, 2005) including involvement of pro-
gramme staff, community member and project partners in the
development of the evaluation plan and the monitoring of evaluation activi-
ties. It included formative, process and impact evaluation.
Formative
Formative evaluation, also referred to as developmental assistance and
front end (Owen, 1993), typically includes activities such as needs assess-
ment, review of best practice, research synthesis (Smallwood and Hur-
worth, 1998) and guidance with developing a programme logic and
programme planning (Conway et al., 2000). Evaluation input at the early
stage in a project’s life cycle assists in successful programme implemen-
tation (Dehar, Casswell and Duignan, 1993) through ensuring that effective
strategies to meet project objectives are adopted. Formative evaluation
maintains a focus on capacity building, working where possible with the
communities involved in the evaluation to provide training opportunities
to improve project sustainability once the evaluators exit (Duignan, 2002).
In the RAP evaluation, the formative evaluator operated as a ‘critical friend’
and mentor walking alongside the project workers asking key reflective ques-
tions and providing guidance where required (McClintock, 1986; Conway
et al., 2000). She completed a needs assessment, developed a community
profile with project staff and facilitated project planning by checking out
with project workers and committee members that there were clear links
between objectives, strategies and activities. Monitoring mechanisms were
developed and subsequently used to review progress with project staff and
community members. The formative evaluator assisted with the provision
of external leadership and facilitation training and provided additional plan-
ning workshops for the convenors of small-scale community initiated projects.
Process
Process evaluation is typically undertaken from the initial stages of a pro-
gramme and continues throughout the life-cycle of the programme. The
emphasis is on ‘documenting and analysing the way the programme
146 Jeffery Adams et al.
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
works in practice, to identify and understand important influences on its
operation and achievements’ (Dehar, Casswell and Duignan, 1993, p. 217).
Process evaluation is useful to help provide the context for understanding
and interpreting how a programme achieves what it does (Waa, Holibar
and Spinola, 1998).
Process evaluation for RAP consisted of observation at meetings and
events, a review of documents generated within the project, and monitoring
media coverage of the project. Key informant interviews with project staff
and community stakeholders were undertaken throughout the project to
elicit views and understanding of the project’s structures, decision-making
processes, progress, successes and difficulties.
Impact/outcome
Impact/outcome evaluation measures programme effects, rather than pro-
gramme implementation (Dehar, Casswell and Duignan, 1993). It answers
the question ‘did the programme meet it’s objectives?’ and is based on
specific and measurable objectives developed through the formative evalu-
ation (Duignan and Casswell, 1989).
The RAP impact/outcome evaluation involved qualitative interviews
with key informants directly and indirectly involved with RAP and quanti-
tative social surveys of residents. The qualitative component was based
around a naturalistic approach, with purposeful sampling (Patton, 1990)
used to identify key informants who participated in semi-structured
face-to-face interviews. A core group were re-interviewed three times.
Each round of interview data was transcribed and analysed thematically.
Two community surveys were undertaken during the course of the
project to assess community perceptions, participation and connectedness.
The telephone-based surveys undertaken in 2001 and 2004 asked residents
about their perceptions and experiences of the social and physical environ-
ment in Ranui and their knowledge of RAP-related activities. The surveys
provided a measure of change over time on variables relevant to neighbour-
hood social capital and specific RAP objectives and strategies.
Applying the SHORE/Whariki evaluation model outlined earlier
requires more than one programme evaluator. The formative evaluation
role of working closely with the project implementers is complemented
by the more distanced and objective positions adopted by the process and
impact evaluators. Implicit in this evaluation model are iterative feedback
loops in which data gathered by the process and impact evaluators are
fed back into the project and used by the formative evaluators as they
work with the project to ask the critical questions designed to reflect on
and enhance strategies and activities for project effectiveness. The indicators
of project success identified in the project planning processes became the
Community development as health promotion 147
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
measures around which the process and impact evaluation were structured.
Many of the change strategies and mechanisms that were articulated as
linking RAP activities to its overall programme goals were common com-
munity development principles and approaches.
The nature of the RAP project with its multiple and changing foci had
implications for the allocation of evaluator time. Although there was a
basic focus on core project aims, and the links between objectives, strategies,
activities and intended outcomes were made explicit, the emerging direc-
tions and new project foci were constantly being introduced. This
dilemma was not as acute for the process evaluator as documenting, and
analysing the twists and turns of community decision making was central
to the task. The impact evaluation again had to remain responsive to emer-
ging project directions and their community impacts while simultaneously
following progress towards outcome measures of central project objectives.
Key evaluation findings
RAP established an identity locally and nationally as a dynamic
community-based project. It achieved wide community reach and orga-
nized a number of very well attended community events and celebrations.
It also succeeded in engaging a diverse and committed core of community
members in its governance and activities. It funded a considerable number
of innovative, small-scale Go Now7 initiatives. Some projects had an ethnic
focus (e.g. Pacific Island gardening projects, kapa haka uniforms8), whereas
others were youth focused (e.g. youth development camp). In many
instances, the projects enhanced the skills of the community members
involved in their planning and delivery, contributed to building social
capital and offered Ranui people new learning opportunities (e.g. computer
skills, driver license training) that would not have existed otherwise.
RAP initially found it difficult to engage youth with the project. However,
building on a successful Go Now project (school examination coaching),
young people became engaged and subsequently organized a number of
very successful community events, including a touch rugby tournament,
local talent quests, a hip-hop national event, kids’ fun days as well as a
regular youth group and youth development initiatives.
RAP took positive steps towards ensuring sustainability beyond govern-
ment funding, an implicit goal of the project. Additional funding was
7 Go Now projects were small budget initiatives developed by community representatives and
funded by RAP. They were designed to give the project an activity profile in Ranui and provide
opportunities for community participation while the projects’ longer term consultation and planning
processes were undertaken.
8 Māori performing arts, in Ranui this focused on young people’s involvement.
148 Jeffery Adams et al.
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
secured from other agencies for particular projects, for purchasing premises
(and a much-needed health service tenant) and initiating a local business
association. Various approaches were used to communicate with and
engage local residents, such as Radio Ranui (a radio programme on com-
munity access radio), community events, a newsletter and regular publicity
in local media. Some were short-term one-off strategies, whereas others
have been sustained over a longer period – such as Radio Ranui and
monthly market days.
RAP has faced a number of challenges, particularly in the areas of govern-
ance, programme prioritization and staffing. Concern over governance
issues emerged early on and remained unresolved after four years. The con-
sultation process used to decide upon the structures was extensive and
inclusive, but the level of Māori representation agreed upon was not seen
as satisfactory by Māori. Allied to this were debates over how to operatio-
nalize the Treaty of Waitangi9 within RAP. There were varying understand-
ings among project members of this issue ( just as there are in New Zealand
society as a whole).
Prioritizing programme objectives and activities was a slow and time-
consuming process. A comprehensive community visioning and consul-
tation process was undertaken and an action plan developed early in the
project (2002), but the plan was far-ranging. Needs, goals, objectives, activi-
ties and change measures were well documented but it was not strategically
focused. A prioritized workplan was not agreed until December 2003.
Staffing was problematic. The importance of having skilled staff in such
projects has been noted as a key success factor in community projects
(Greenaway and Witten, 2006). The governance group was slow to staff
the project adequately despite early recognition that a core group of pro-
fessional staff was needed to manage and implement the ambitious range
of project activities in the action plan. Staff were initially employed by the
WCC, with the intention that RAP would assume legal responsibility for
the employment of staff after the establishment phase. This was not
achieved in the period of the evaluation. Tensions over the respective work-
9 The Treaty of Waitangi is the founding document of New Zealand. It is an agreement entered into
by representatives of the Crown and representatives of some Māori chiefs on behalf of iwi (tribes)
and hapū (subtribes). In relation to health policy, the Public Health Association of New Zealand
acknowledges the importance of the Treaty and advocates that health policy should support health
gains for Māori (policy outcomes); be responsive to Māori needs and expectations (policy outcomes
and policy development process) and be analytically sound (policy development process) (Waldon,
with the Maori Caucus, 2002). Although references to the principles of the Treaty are included in
many health-related legislation and policies, this inclusion is not universally supported and the removal
of such references is currently being debated by the New Zealand Parliament (Principles of the
Treaty of Waitangi Deletion Bill).
Community development as health promotion 149
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
loads of paid project staff and community volunteers underpinned the
reluctance of RAP, staff and the WCC to affect the transfer of responsibility.
The key issues raised in this section are related to political and other dif-
ficulties in implementing a participatory community project. Taken as a
whole, there were clearly issues which restricted the building of relation-
ships within the organization that could be considered transformative
(Ledwith, 1997). Although there was mixed evidence of success for
factors such as the development of relevant skills and experiences, shared
values, shared sense of purpose and focus on learning and accepting
risks (Blaxter, Farnell and Watts, 2003), overall there were a number of
key tensions that were not resolved adequately. Interestingly, the political
issues of the participatory approach were mostly at the local level, as the
strategy of the funders remaining largely hands-off meant they were not
involved in the day-to-day realities of the programme.
Intersectoral engagement
The social cohesion surveys identified areas of positive social change in
Ranui between 2001 and 2004 (Adams et al., 2005). Residents reported stat-
istically significant increases in the amount of activity for youth in the area,
levels of involvement in skill-building activities and participation in com-
munity events, all of which were key foci of RAP. Perceptions also improved
significantly of Ranui as a safe place for children to play, a good place to
bring up children and to buy a home. Change in social cohesion ratings
was in a positive direction but the level of change was not statistically sig-
nificant. Although it is not possible given the cross-sectional nature of the
surveys to attribute causality to the presence and activities of RAP, the find-
ings are consistent with those of the qualitative impact evaluation, namely
that RAP provided opportunities for engagement, participation and skill
building for residents, especially young people, and brokered consultation
and positive dialogue between the community and local government to aid
the upgrading of the built environment.
Discussion
In broad terms, RAP aimed to enhance the health and wellbeing of Ranui
people with a particular focus on the wellbeing of children, young people
and their families. Building social capital and social cohesion through com-
munity engagement and participation were theorized by funders and the
project members as primary change mechanisms which would contribute
to improving health in the community. The purpose of the evaluation was
to assist RAP to achieve its aim as well as to determine its relative
150 Jeffery Adams et al.
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
success at doing so. In this section, we address the effectiveness of the evalu-
ation of RAP.
Ranui is a low-income community that faces a complex array of social and
health concerns. Community-led social change to reduce health inequal-
ities, and improve the wellbeing of Ranui children and families, was
never going to be a simple or fast process. Not surprisingly, RAP’s response
to Ranui’s plethora of needs has been a complex, multi-faceted range of
interventions. Within this jigsaw of programme components, effort and pro-
gress at any point in time reflected a blend of planned process and the avail-
ability of voluntary local champions. The broad programme brief, the dual
expectations of the funders and the complexity of the evolving programme
posed a myriad of evaluation needs and opportunities. A decision was
made to allocate significant evaluation resources to formative evaluation
to ensure the project had access to planning assistance, research-based
knowledge of effective community strategies and critical questioning of
project direction and focus, and also to ensure action planning that was con-
gruent with the programme logic. These steps were seen as essential to
meeting project outcomes and the longer-term effectiveness and sustain-
ability of the project.
Formative
From the outset, there was a demand from the community and funders for
tangible and immediate evidence that progress was being made on the
broad indicators such as the participation and upskilling of local people
and increasing youth involvement. This meant the project decided to
‘pick the low hanging fruit’ in the early years, although moving in direc-
tions that were consistent with the longer term objectives. The formative
evaluation role of making the explicit links between objectives and activities
was essential to achieving the longer term aims of the project. A fundamen-
tal tension for the formative evaluator was to maintain a focus on planning
to achieve RAP’s longer term aspirations, while also working on immediate
demands.
Confronted with a regular flow of new Go Now projects, it was apparent
that evaluation input could not extend to assisting the development of all
projects and was better directed at supporting the project to set up
generic planning templates, monitoring and accountability processes.
Although this was a sound approach from an evaluation perspective, it
proved to be beyond RAP’s capacity to monitor the growing number of pro-
jects effectively over time. This realization contributed to a scaling back of
Go Now projects with the recognition that the project and the evaluation
(both of which were restricted by limited staff resource) were not able to
Community development as health promotion 151
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
meet all of the competing demands of this project. Over time, the practical-
ities of workload tempered the pace of programme activity.
Skills transfer, including nurturing the development of capability and
capacity to undertake strategic planning and evaluation for project activi-
ties, was a formative evaluation task. However, tensions in project dynamics
between different parties meant that strategic planning for project develop-
ment proceeded more slowly than hoped. The formative evaluation did
provide insights into programme effectiveness through reviewing pro-
gramme strengths as well as barriers to more effective programme planning
and implementation.
Process
To identify and understand important influences on RAP’s operation and
achievements, the process evaluation relied on accessing all the documen-
tation produced within the project (e.g. minutes of meetings, planning
documents, etc) in addition to key informant interview data. Staff were
not always able to compile a full set of project documents, with documents
relating to volunteer-run Go Now projects being particularly difficult to
acquire. Case studies of the implementation of selected projects had been
planned as part of the process evaluation but these were abandoned
when it became apparent that completing them would place an unreason-
able burden on volunteers’ time. Although individual Go Now projects
were clearly effective in contributing to measurable programme outcomes
such as community participation, increased youth-initiated community
events and skill acquisition by local people, the overall effectiveness of
the Go Now project scheme was not adequately established. The adminis-
trative cost of running what was in effect a community funding scheme
was too high for RAP staff. Tempering evaluative expectations in the light
of project and evaluation resources needed regular monitoring.
Despite some limitations in access to information, the project was docu-
mented at a level that enabled other related projects to assess the merits
of adopting aspects of RAP’s operation. For example, the Go Now project
scheme was initiated in two other SCAF sites. The process evaluation also
assisted in understanding the social and multi-ethnic context of the
project, including the tensions that underpinned some of the difficulties
as well as achievements of RAP’s implementation.
Impact
The quantitative component of the impact evaluation was a before and after
survey of neighbourhood social cohesion. The funder argued that a baseline
measure should be taken before the project got underway but as evaluators
we were cautious about designing a survey without having a clear idea of
152 Jeffery Adams et al.
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
the project’s direction and in particular the areas in which people and finan-
cial resources were to be invested. The limitations of attributing change over
time to RAP interventions were also emphasized from the outset. Once
initial strategic planning had been completed, the evaluators were able to
design a survey that was aligned to the targeted activities of RAP,
arguing that this was more likely to provide meaningful insights into the
project impacts. Although this did not resolve the difficulties of attribution,
it meant that the findings of the qualitative key informant interviews could
be considered alongside the survey results as project effectiveness in
specific areas was reviewed. For example, an increased proportion of 2004
survey respondents reported that young people were valued in Ranui,
and that there was more going on for the young.
In many areas, key informant interview data complemented the survey
results. In other areas, they provided interpretations of events such as
where conflicts and tensions between factions were undermining social
change, data that could not be collected by a survey. The use of multiple
data sources increased the robustness of the evaluation and its value in
the eyes of stakeholders. Evaluative findings informed the on-going devel-
opment of the programme through the reflective practices of the evaluation
team and via regular meetings between funders, community stakeholders
and evaluators. The latter was an enlightened requirement of the WCC,
as the local government project host. These open meetings were attended
by all stakeholders and the struggles and triumphs of the project and its
evaluation were shared. This process served to maintain a level of personal
and intersectoral commitment to the project as well as reaffirming the pro-
ject’s directions, and its evaluation.
Conclusion
Evaluating complex community projects is difficult and inevitably choices
are made as to where scarce resources are best directed. Pragmatic
choices were made in the evaluation of RAP that took into account the
need to be responsive to local context while maintaining robust research
practice, and at the same time balancing the evaluative requirements of
dual funders. Our evaluation activity placed a heavy emphasis on formative
evaluation in line with our model of evaluation based on the premise that a
well-planned programme enhances the likelihood of good outcomes. Cap-
turing the complexity of RAP was aided by triangulation of data through a
multi-methods approach as well as a triangulation of understandings of
project planning and implementation from the ‘insider ’ perspective of the
formative evaluator and the ‘outsider ’ perspectives of the process and
impact evaluators. Working simultaneously across several SCAF and inter-
Community development as health promotion 153
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
sectoral initiatives further enabled the evaluation team to examine the
efficacy of specific strategies as they were applied in different settings.
Funding
The evaluation of the Ranui Action Project was funded by the New Zealand
Ministry of Health and the Department of Child, Youth and Family.
Jeffery Adams and Karen Witten both work at the Centre for Social and Health Outcomes
Research and Evaluation (SHORE), Massey University, Auckland, New Zealand. Kim Conway
works at the Waitakere Wellbeing Collaboration Project, Waitakere City Council, Waitakere,
New Zealand.
Address for correspondence: Jeffery Adams, Centre for Social and Health Outcomes
Research and Evaluation (SHORE), Massey University, PO Box 6137, Wellesley Street,
Auckland, New Zealand; email: [email protected]
References
Adams, J., Witten, K., Woodson, T., Bala, M. and Huckle, T. (2005) A Report on the
Findings of the Ranui Social Cohesion Surveys, Centre for Social and Health Outcomes
Research and Evaluation (SHORE) and Te Ropu Whariki, Massey University,
Auckland.
Allen, W., Kilvington, M. and Horn, C. (2002) Using Participatory and Learning-Based
Approaches for Environmental Management to Help Achieve Constructive Behaviour
Change, Landcare Research Contract Report LC102/057 prepared for the Ministry of
the Environment, Landcare, Lincoln.
Barr, A. (2005) The contribution of research to community development: a reflection on
types, methods and experience, Community Development Journal, 40 (4), 143 – 159.
Baum, F. and Palmer, C. (2002) ‘Opportunity structure’: urban landscape, social capital
and health promotion in Australia, Health Promotion International, 17, 351 – 361.
Becker, J. (2006) Improving community health through evaluations, Community
Development Journal, 42 (3), 348 – 364.
Billings, J. (2000) Community development: a critical review of approaches to
evaluation, Journal of Advanced Nursing, 31 (2), 472 – 480.
Blakely, T. (2002) The New Zealand Census – Mortality Study: Socioeconomic Inequalities and
Adult Mortality 1991 – 94, Ministry of Health, Wellington.
Blaxter, L., Farnell, R. and Watts, J. (2003) Difference, ambiguity and the potential for
learning – local communities working in partnership with local government,
Community Development Journal, 38 (2), 130 – 139.
Bourdieu, P. (1986) The forms of capital, in J. Richardson, ed., Handbook of Theory and
Research for the Sociology of Education, Greenwood Press, New York.
Casswell, S. (2000) A decade of community action research, Substance Use & Misuse, 35
(1 – 2), 55 – 74.
154 Jeffery Adams et al.
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
Casswell, S. (2001) Community capacity building and social policy – what can be
achieved?, Social Policy Journal of New Zealand, 17, 22 – 35.
Coleman, J.S. (1990) Foundations of Social Theory, Harvard University Press, Cambridge,
MA.
Conway, K. and Casswell, S. (2003) Riding the waves: the politics and funding context
of twenty five years of community action on alcohol research in New Zealand, Nordic
Journal of Alcohol Studies, 20 (English Supplement), 13 – 24.
Conway, K., Tunks, M., Henwood, W. and Casswell, S. (2000) Te Whanau Cadillac – a
waka for change, Health Education and Behavior, 27 (3), 339 – 350.
Craig, D. (2004) Building a better context for partnership: a review of core issues, with
on-the-ground examples from the ‘Waitakere Way’, Social Policy Journal of New
Zealand, 23, 45 – 64.
Craig, G. (2002) Towards the measurement of empowerment: the evaluation of
community development, Community Development Journal, 33 (1), 124 – 146.
Davidson, E. (2005) Evaluation Methodology Basics: The Nuts and Bolts of Sound
Evaluation, Sage, Thousand Oaks.
Dehar, M., Casswell, S. and Duignan, P. (1993) Formative and process evaluation of
health promotion and disease prevention programs, Evaluation Review, 17 (2),
204 – 220.
Duignan, P. (2002) Building social policy evaluation capacity, Social Policy Journal of New
Zealand, 19, 179 – 194.
Duignan, P. and Casswell, S. (1989) Evaluating community development programs for
health promotion: problems illustrated by a New Zealand example. Community
Health Studies, XIII (1), 74 – 81.
Durie, A. (1997) Te Aka Matua: keeping a Maori identity, in P. Te Whaiti, M. McCarthy,
A. Durie, eds, Mai i Rangiatea: Maori Wellbeing and Development, Auckland University
Press, Auckland.
Ellaway, A., Macintyre, S. and Kearns, A. (2001) Perceptions of place and health in
socially contrasting neighbourhoods, Urban Studies, 38 (12), 2299 – 2316.
Ellen, I., Mijanovich, T. and Dillman, K.-N. (2001) Neighbourhood effects on health:
exploring the links and assessing the evidence, Journal of Urban Affairs, 23, 391 – 408.
Epstein, I., Tripodi, T. and Fellin, P. (1973) Community development programmes and
their evaluation, Community Development Journal, 8, 28 – 36.
Germann, K. and Wilson, D. (2004) Organisational capacity for community
development in regional health authorities: a conceptual model, Health Promotion
International, 19, 289 – 298.
Gillies, P. (1998) Effectiveness of alliances and partnerships for health promotion,
Health Promotion International, 13 (2), 99 – 120.
Greenaway, A. and Witten, K. (2006) Meta-analysing community action projects in
Aotearoa New Zealand, Community Development Journal, 41, 143 – 159.
Hawe, P. (1994) Capturing the meaning of ‘community’ in community intervention
evaluation: some contributions from community psychology, Health Promotion
International, 9, 199 – 210.
Howden-Chapman, P. and Tobias, M. (eds) (2000) Social Inequalities in Health: New
Zealand 1999, Ministry of Health, Wellington.
Community development as health promotion 155
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
Huckle, T., Witten, K. and Adams, J. (2002) A Report on the Findings of the Ranui Social
Cohesion Survey. Evaluation of the Ranui Action Project, Alcohol and Public Health
Research Unit, University of Auckland, Auckland.
Judd, J., Frankish, C. and Moulton, G. (2001) Setting standards in the evaluation of
community-based health promotion programmes – a unifying approach, Health
Promotion International, 16 (4), 367 – 380.
Judge, K. and Bauld, L. (2001) Strong theory, flexible methods: evaluating complex
community-based initiatives, Critical Public Health, 11 (1), 19 – 38.
Kawachi, I. and Berkman, L. (2000) Social cohesion, social capital, and health, in
L. Berkman and I. Kawachi, eds, Social Epidemiology, Oxford University Press,
New York, pp. 174 – 190.
Labonte, R. (1993) Community development and partnerships, Canadian Journal of
Public Health, 84, 237 – 240.
Ledwith, M. (1997) Participating in Transformation: Towards a Working Model of
Community Empowerment, Ventura Press, Birmingham.
Loughry, R. (2002) Partnering the state at the local level: the experience of one
community worker, Community Development Journal, 37, 61 – 68.
Lynch, J. (2000) Income inequality and health: expanding the debate, Social Science and
Medicine, 51, 1001 – 1005.
McClintock, C. (1986) Towards a theory of formative programme evaluation, Evaluation
Studies Review Annual, 11, 205 – 223.
Ministry of Health and University of Otago (2006) Decades of Disparity III: Ethnic and
Socioeconomic Inequalities in Mortality, New Zealand 1981 – 1999, Public Health
Intelligence Occasional Bulletin Number 31, Ministry of Health, Wellington.
Murphy, M. (2002) Social partnership – is it ‘the only game in town’?, Community
Development Journal, 37, 80 – 90.
Oakley, A. (1998) Experimentation and social interventions: a forgotten but important
history, BMJ, 317, 1239 – 1242.
Oliver, P., Spee, K. and Wolfgramm, T. (2003) A partnership approach to evaluation
communities, in N. Lunt, C. Davidson and K. Mckegg, eds, Evaluating Policy and
Practice: A New Zealand Reader, Pearson Education, Auckland, pp. 163 – 172.
Owen, J. (1993) Program Evaluation: Forms and Approaches, Allen & Unwin, St Leonards
(NSW).
Patton, M. (1997) Utilization-Focused Evaluation: the New Century Text, 3rd edition, Sage,
Thousand Oaks, CA.
Patton, M. (1998) Discovering process use, Evaluation, 4, 225 – 233.
Patton, M. (2002) Qualitative Research and Evaluation Methods, Sage, London.
Patton, M.Q. (1990) Qualitative Evaluation and Research Methods, 2nd edition, Sage,
Newbury Park.
Pawson, R. and Tilly, N. (1997) Realistic Evaluation, Sage Publications Ltd, London.
Pickett, K. and Pearl, M. (2001) Multilevel analyses of neighbourhood socioeconomic
context and health outcomes: a critical review, Journal of Epidemiology and Community
Health, 55, 111 – 122.
156 Jeffery Adams et al.
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
Popay, J., Thomas, C., Williams, G., Bennett, S., Gatrell, A. and Bostock, L. (2003) A
proper place to live: health inequalities, agency and the normative dimensions of
space, Social Science & Medicine, 57, 55 – 69.
Putnam, R. (2000) Bowling Alone: The Collapse and Revival of American Community, Simon
and Schuster, New York.
Putnam, R. (2004) ‘Health by association’: some comments [Commentary], International
Journal of Epidemiology, 33, 1 – 4.
Putnam, R.D. (1993) The prosperous community: social capital and public life, The
American Prospect, 13 (Spring).
Rychetnik, L., Frommer, M., Hawe, P. and Shiell, A. (2002) Criteria for evaluating
evidence on public health interventions, Journal of Epidemiology and Community
Health, 56, 119 – 127.
Salmond, C. and Crampton, P. (2001) NZDep96 – what does it measure? Social Policy
Journal of New Zealand, 17, 82 – 100.
Simpson, L., Wood, L. and Daws, L. (2003) Community capacity building: starting with
people not projects, Community Development Journal, 38 (4), 277 – 286.
Smallwood, H. and Hurworth, R. (1998) Literature syntheses for program planning
and policy development: a guide, Evaluation News and Comment, 7 (1), 37 – 44.
Susser, M. (1995) The tribulations of trials. Interventions in communities [Editorial],
American Journal of Public Health, 85, 156 – 158.
Taylor, L. (2004) Building social capital through devolved decision making: the
Stronger Communities Action Fund, Social Policy Journal of New Zealand, 21, 67 – 82.
Voyle, J.A. and Simmons, D. (1999) Community development through partnership:
promoting health in an urban indigenous community in New Zealand, Social Science
& Medicine, 49, 1035 – 1050.
Waa, A., Holibar, F. and Spinola, C. (1998) Programme Evaluation: An Introductory Guide
for Health Promotion, The University of Auckland, Auckland.
Waldon, J., with the Maori Caucus (2002) Advocating Public Health Policy for Maori:
Checklist for the Public Health Association of New Zealand Incorporated. Trial version –
for PHA Council decision, Public Health Association of New Zealand.
Wilkinson, R.G. (2000) Inequality and the social environment: a reply to Lynch et al.,
Journal of Epidemiology and Community Health, 54, 411 – 413.
Woolever, C. (1992) A contextual approach to neighbourhood attachment, Urban
Studies, 29 (1), 99 – 121.
Community development as health promotion 157
D ow
nloaded from https://academ
ic.oup.com /cdj/article-abstract/44/2/140/316445 by R
yerson U niversity Library user on 11 S
eptem ber 2018
