Discussion 4

Evaluating children's rehabilitation services: an application of a programme logic model.

To apply a programme logic model to evaluate the effectiveness of a new therapy service for children with special needs who were in transition from pre-school to kindergarten. A children's outpatient rehabilitation centre in Ontario, Canada. The short-term outcomes included parents’ perceptions of the transition process itself and the information they required, the children's skill development for the transition to kindergarten, and parents’ perceptions of services and satisfaction with resources. A combination of quantitative methods [Goal Attainment Scaling (GAS), Measure of Processes of Care (MPOC), Client Satisfaction Questionnaire (CSQ)] and qualitative interviews were used to evaluate both the process (‘ Outputs’) and outcomes (‘ Short- term objectives’) of the new therapy service. The children involved in the evaluation met or exceeded goals that were set by therapists and parents. Parents’ perceptions of, and satisfaction with, the new service were higher than the provincial average. Qualitative data from interviews with parents and service providers supported the findings from standardized measures, and provided suggestions for future service delivery. The programme logic model provided researchers and service providers a collaborative and systematic approach to conducting programme evaluation in a relatively short-time frame. It appears to be a useful option for evaluation of other children's services. [ABSTRACT FROM AUTHOR]

Copyright of Child: Care, Health & Development is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

A I S N

Evaluating children's rehabilitation services: an application of a programme logic model.

Full Text

Objectives To apply a programme logic model to evaluate the effectiveness of a new therapy service

for children with special needs who were in transition from pre‐school to kindergarten. Setting A children's outpatient rehabilitation centre in Ontario, Canada. Main outcomes The short‐term outcomes included parents' perceptions of the transition process itself and the information they required, the children's skill development for the transition to kindergarten, and parents' perceptions of services and satisfaction with resources. Methods A combination of quantitative methods [Goal Attainment Scaling (GAS), Measure of Processes of Care (MPOC), Client Satisfaction Questionnaire (CSQ)] and qualitative

interviews were used to evaluate both the process ('Outputs') and outcomes ('Short‐term objectives') of the new therapy service. Results The children involved in the evaluation met or exceeded goals that were set by therapists and parents. Parents' perceptions of, and satisfaction with, the new service were higher than the provincial average. Qualitative data from interviews with parents and service providers supported the findings from standardized measures, and provided suggestions for future service delivery. Conclusions The programme logic model provided researchers and service providers a collaborative

and systematic approach to conducting programme evaluation in a relatively short‐time frame. It appears to be a useful option for evaluation of other children's services.

Keywords: rehabilitation; children; service delivery; programme evaluation In an effort to bring evidence into practice, many agencies and services have identified programme evaluation as an essential part of health care. Service providers and managers recognize the importance of evaluating their programmes to ensure that they are effective. Unfortunately, many who work in child health are unsure where to start in evaluating their programmes. Numerous textbooks and models are available, but they may appear to be overwhelming to health care providers who do not have any training or experience in this area.

This article describes an innovative, easy‐to‐use approach to programme evaluation that involves the use of a programme logic model ([24]). An application of this approach in a children's outpatient rehabilitation centre is presented. Background information Programme evaluation A programme evaluation is a systematic application of research procedures that investigates the impact and effectiveness of a programme/service ([ 7]; [28]; [17]). A programme is defined as 'an organized set of activities that are managed towards a particular set of goals . . .' ([25], p. 42). The evaluation of programmes has been in existence for many years in all sectors of human services, including health care. Programme evaluation is recognized as a type of inquiry that is different from basic academic research ([20]). It is viewed as useful, or 'applied' research that informs decision making and promotes action being taken to make services more effective ([20], [21]). Programme evaluation has emerged as a unique professional field of practice, with associations, journals and guiding principles directed towards evaluators ([18]). There is a broad range of frameworks and methodologies available to evaluate programmes. Experts agree that the choice depends on multiple factors such as the purpose of the evaluation, the information needs, resources and values of the stakeholders, and the desired outcomes ([21]; [10]; [ 5]). Recent literature has supported the benefits of involving stakeholders directly in the evaluation process to contribute to the effectiveness of programmes ([23]; [ 3]; [ 8]; [29]). Programme evaluation is recognized as an essential part of evidence‐based practice in today's health care climate ([17]). It is also connected to quality assurance and improvement practices ([19]). Health care literature provides numerous examples from around the world of the application of programme evaluation in public health, health promotion and disease prevention (e.g. Centers for Disease Control and Prevention; [ 9]) and national or state/regional programmes ([ 6]; [15]; [22]; [27]). Most of the literature on programme evaluation at a local, individual service level focuses on the effectiveness of the service itself, and does not emphasize the framework or methods used for the evaluation. This article places emphasis on the description and application of a specific evaluation model in health care, in an effort to increase service providers' and administrators' awareness of a useful approach to evaluate their programmes. The programme logic model The programme logic model ([24]) involves a systematic approach that is applicable to most settings and programmes, and has been used successfully in a variety of settings ([24]; [17]). Users have found it to be clear and helpful in guiding their programme evaluation activities. Recent published examples of the application of the programme logic model are in fields of community rehabilitation ([16]), vocational rehabilitation ([13]), and case management ([ 1]). It was also used as the basis for a workbook on programme evaluation for occupational therapists in Canada ([17]). Some services have used the programme logic model to assist in the development of new programmes, to be ready for evaluation ([ 4]; [26]). A programme logic model is a chart that provides basic information about a programme, its components and the linkages between what a programme does and what it is expected to achieve ([24]). A key characteristic of this model is that the 'means' (what you do) and the 'ends' (the results or outcome of what you do) are separated. 'Means' refers to the process of the programme, and 'ends' are the outcomes ([17]). These two important aspects of a programme are clearly distinguished for an evaluation. There are five main elements in a programme logic model. The primary activities of the programme are identified as the Main Components. The Implementation Objectives then translate the main components into objectives about what the programme is expected to do. Outputs are the indicators or process components of the programme's implementation, and are comprised of the data about service delivery

and the people who are served. Short‐term and Long‐term Outcome Objectives refer to what you expect

to change or occur as a result of the programme over shorter or longer‐time frames. Finally, the arrows link the various elements of the model to show the main 'logic' of the programme. The programme: transition services at a children's rehabilitation centre A pilot programme at an outpatient children's rehabilitation centre in the province of Ontario, Canada was implemented in 1999, with funding from the provincial Ministry of Health. The programme (called

'transition services') focused on providing extra therapy services to children in transition from pre‐school to the elementary school system. Additional services covered under this new programme initiative were

occupational therapy and physiotherapy, both at a 0.5 full‐time equivalent (FTE) level. In addition to the extra therapy services provided individually to children and their families, Centre staff participated in four evening parent education sessions. Parents who attended the education sessions received a binder with written information and resources. The evaluation Part of the funding for this new initiative was directed towards an evaluation, which was conducted by CanChild Centre for Childhood Disability Research, at McMaster University. CanChild is a Ministry of Health funded research unit with extensive experience in design and evaluation. It is formally partnered with children's rehabilitation centres across Ontario, and this close relationship prompted the staff at the Children's Centre to approach and work with CanChild. The programme evaluation began with preliminary meetings to discuss the purpose and outcomes of the programme evaluation, and to clearly identify the people involved and their roles and responsibilities. This early work ensured a smooth process during the 10 months of evaluation. The budget for the evaluation

was negotiated, and a research co‐ordinator was hired part‐time. An evaluation committee at CanChild was formed to oversee the process and provide input to data collection, analysis and interpretation. Ethics approval for the study was received from the university's research ethics committee. Evaluation methods The design of the programme evaluation was driven by a programme logic model, which was developed collaboratively by staff at the Children's Centre and CanChild (Fig. 1). A combination of quantitative and

qualitative methods was used to measure the Outputs and Short‐term objectives. The administration of the Children's Centre collected service statistics as part of their regular operations, and this data was used to measure most of the Outputs of the programme.

Graph: 1 Programme logic model for transition services. FTE, full‐time equivalent; OT, occupational therapy; PT, physiotherapy. The measures used by the evaluation team were administered after the children had made the transition to kindergarten. They included the following: • 1

 A chart audit tool, developed specifically for this evaluation, was used to collect data about the process of service delivery. Occupational therapy and physiotherapy students were hired and trained

to conduct the chart audits. Inter‐rater reliability of 0.70 or greater overall was established before chart abstraction. The chart audit tool was a checklist that allowed the extractors to record the number of occurrences of different types of service each child received.

• 2

 The Measure of Processes of Care (MPOC), 20‐item version ([11]) is a questionnaire about parents' perceptions of the services they and their children received. It uses a seven‐point Likert rating scale.

• 3

 The Client Satisfaction Questionnaire (CSQ), adapted from [14]), is completed by parents about their satisfaction with the services they and their child received. It contains nine questions about services

using a four‐point scale, plus two open‐ended questions. • 4

 Goal Attainment Scaling (GAS) ([12]) involves a therapist who is working with each child to set a goal in consultation with parents, and then identify five possible outcomes for the goal. The five levels

include a baseline level (what the child is doing now), an expected or hoped‐for outcome level, and outcomes that are above expectations. See Fig. 2 for an example of a GAS form completed for this evaluation. All goals were evaluated by an independent researcher to ensure that they were measurable, achievable and relevant. At the end of the evaluation period, an independent evaluator observed the child or interviewed the parent(s) to determine the level achieved for each goal.

• 5

 Qualitative interviews were conducted with parents and service providers who were purposefully selected by the research team to represent a broad range of experiences and perceptions of the services they and their child received. Qualitative data were gathered to supplement the quantitative data, and to explore parents' and service providers' perceptions of the new services in greater depth. The evaluators were interested in exploring the factors that made a difference for some families –

what they liked, what they didn't like – and their recommendations for the future. Interviews were tape‐ recorded and transcribed to text for analysis. The research team read the transcripts and developed a coding scheme of the key themes that emerged. One researcher used the coding scheme to analyse all transcripts after coding agreement of 75% was achieved.

Graph: 2 Sample goal attainment scaling form for transition services programme evaluation. Sample selection The parents of all clients involved in the new transition services during the evaluation period (1 January 2000 to 31 October 2000) were approached to participate in this study. Parents were recruited by phone for their verbal consent to participate before providing informed written consent. Parents provided demographic information about their child, their family and the services they were receiving. Results Sample demographics All clients born in 1995 or 1996 were eligible to start junior or senior kindergarten in the public school system in the year 2000/01. There were 81 clients in the Centre's catchment region who were eligible, and only 21 clients chose to start junior or senior kindergarten in this school year. Reasons parents chose to hold their children back from starting school included a lack of support for their child in kindergarten or parents feeling their child was not ready for the community school system. Sixteen clients consented to participate, however, one client dropped out from the study for medical reasons, and two families did not return their questionnaires. Thirteen clients were included in the final analysis. The mean age of the 13 children involved in the evaluation of the transition services was 5.0 years (range 3.10–6.4 years). The primary diagnoses were: cerebral palsy (37.5%), spina bifida (12.5%), developmental delay (6.3%), syndromes (6.3%) and 'other' (18.8%), such as behavioural or communication disorders. The results of this evaluation are reported using the structure of the programme logic model. Outputs The service statistics collected by the administration of the Children's Centre demonstrated that additional occupational therapy and physiotherapy services were provided over the course of the 10 months of the

new initiative. The results of the chart audit indicated that 90% of the time, therapists used non‐ standardized assessment methods when assessing clients. These methods included clinical observation, interview with parents and others, and checklists developed by the therapists themselves. For intervention, therapists spent approximately 45% of their time on direct services with the child, 30% in consultation, education and mediator training, and 17% in conferences and meetings (8% of time was not clearly documented in charts). Short‐term objectives Quantitative results Goal attainment Goal Attainment Scaling was used to evaluate the achievement of goals set for each of the 13 children in the study. At least one goal was set for each child, and a rating scale was established with five levels (−2 to +2) representing five possible outcomes for each goal (see Fig. 2 for sample GAS). Raw scores were converted to T‐scores to account for some clients having more than one GAS goal. A T‐score of 30 equals the baseline score of −2; 40 equals a score of −1 which is considered slightly less goal achievement than expected; 50 equals a level of 0 or expected outcome; 60 equals a score of +1, which is consider somewhat better than expected; and 70 equals +2 or much better than expected). The mean T‐score was for the transition services was 52.64, indicating achievement of a functional outcome slightly above what was expected by the therapists and parents who set the goals for each child. Parents' perceptions of services The MPOC was used to determine the parents' perceptions of the services they and their child received. Five subdomain scores are calculated for the MPOC: enabling and

partnership, providing general information, providing specific information, co‐ordinated and comprehensive care, and respectful and supportive care. Thirteen parents (one for each child) completed the MPOC questionnaire at the end of the evaluation period.

The MPOC scores were compared to data from the Family Centred Service (FCS‐II) study conducted in the province of Ontario in 1999 ([ 2]). This study surveyed 494 parents from across Ontario. Comparisons

are graphically represented in Fig. 3. This comparative analysis demonstrates that the MPOC scores for the transition services are significantly higher in all five domains when compared to the data from the

FCS‐II study. Graph: 3 Measure of Processes of Care (MPOC) scores – comparison between transition services and

Family Centred Service (FCS‐II) (1999) study provincial data. (Error bars indicate 95% confidence interval around mean). ep, enabling and partnership; pgi, providing general information; psi, providing specific

information about the child; ccc, co‐ordinated and comprehensive care for the child and family; rsc, respectful and supportive care. Client satisfaction: general Thirteen parents also completed the CSQ at the end of the evaluation

period. Nine items use a four‐point rating scale, with a score of 4 representing a high level of satisfaction (total possible score is 36). The mean raw score for the transition services was 30.72, with a range from

27.5 to 32.0. This score was compared to CSQ data from the FCS‐II (1999) study in which the mean was 25.89. This demonstrates a higher‐than‐average level of parent satisfaction for this service compared to the Ontario‐wide study (see Table 1). 1 Client Satisfaction Questionnaire (CSQ) scores for transition services

Mean CSQ score 95% Confidence interval

Lower Upper

Transition services 30.72 29.43 32.01

FCS‐II (1999) study 25.89 25.45 26.33 Qualitative results Qualitative data were gathered from parents and service providers about the factors that made a difference, what they liked and didn't like about the new services, and their ideas for improvement. Results are reported under the themes that emerged from the analysis. Access to services Parents reported that location of the programme and therapists' flexibility in scheduling appointments promoted access to services. Barriers to access included busy schedules, and lack of babysitting services. Some parents reported a desire for increased access to specialists and equipment. Service co‐ordination Parents reported satisfaction with the continuity and co‐ordination between their therapists and external agencies, including the schools. In this newly established service, parents and therapists found the joint visits with the 'old' and 'new' therapists in the public schools, together with the

parents and teachers, very helpful for co‐ordination. Communication Parents were typically pleased with their ability to communicate with therapists, and openly express their concerns. They valued the personal relationships they had with their therapists. Progress and growth Parents consistently commented on how they felt the services had improved their child's abilities in everyday activities, in particular, their preparation for kindergarten. Their children also developed increased confidence, initiative and independence as a result of the services. Family participation Parents frequently cited their appreciation of being involved in therapy sessions, and being given suggestions for activities to do at home. They identified their roles as experts, advocates and active team members, as well as their interest in defining the types of service delivery models they want their child to receive. Empowerment Parents suggested that being educated and informed about their child's needs and services allowed them to be more effective parents. Some parents who felt they lacked knowledge in this area, particularly as they started the transition services, expressed feelings of being lost, afraid and uncertain. Parents felt that the new services provided them with insights about their child and about the school system that they would not have received otherwise. Change as the transition services were new, this resulted in changes for the families, in addition to the added stress of a major life change for their child entering the publicly funded school system. Parents recognized that change was always difficult, but they felt for the most part that the Centre staff were trying to be flexible and accommodating. Service providers reported that they also felt the stress of change, but

felt that the earlier transition to the school‐based team was worthwhile as it helped to ease some of the anxieties associated with starting school for some of the families. These themes highlight that parents felt the new therapy services were valuable in helping their children develop and grow, and were delivered using a family‐centred approach. Communication and information helped parents to feel knowledgeable about their child's needs and the services available. The parents offered suggestions for future services, including more information about their child's development, and other possible services available within and outside of the Children's Centre.

Discussion This evaluation of a new service at an outpatient children's rehabilitation centre provided valuable information about the process and outcomes of the services. The programme logic model served as a useful framework to guide the evaluation, and to meet the needs of both evaluators and Centre staff. The Outputs, or process of service delivery were evaluated through service statistics and a chart audit

tool. Short‐term objectives, or outcomes of the services were evaluated using a combination of quantitative and qualitative methods. The Outputs of this new service indicated that additional therapy services were provided to the study

participants during the 10‐month evaluation period, as planned. The audits of therapy charts for the children involved in this evaluation revealed that assessments used primarily non‐standardized methods such as observations, interviews and checklists. These methods provided therapists with qualitative information about the children they were working with, and helped them to plan intervention, however,

non‐standardized measures make it difficult to measure change in children over time. Intervention approaches were varied, with both direct service (with the child present) and indirect service (consultation, mediator training, etc.). This indicates that the therapists selected and adapted their service approaches to meet their clients' needs. The nature and frequency of services identified through the chart audit were similar to other services at the children's rehabilitation centre, except for the higher percentage of conferences and meetings. Given the nature of this new service, it was not surprising to find that 17% of the therapists' time involved conferences and meetings to ensure a smooth transition to a new team and environment. The Short‐term Objectives, or outcomes, of the new  transition  services  were  measured  by  GAS, the MPOC the CSQ and qualitative interviews. The mix of quantitative and qualitative measures ensured that breadth and depth of outcomes were evaluated. The results of the GAS overall indicate that the children involved in this evaluation met or exceeded the expected level of goal attainment for the study period. The GAS scores for the transition services were close to the expected mean outcome of 50, which indicates that therapists in the transition services set goals that were realistic and achievable for the time period. The MPOC and the CSQ results indicated that parents' overall satisfaction with services was higher than the provincial average. On the MPOC, the scores in all five domains of parents' perceptions of services

were significantly higher than the Ontario‐wide study data. The small sample size means that these findings should be interpreted with some caution. Some of the higher scores, particularly those related to information, may be partly because of the extra education sessions and binder of information parents received about transition to school. The higher scores on both measures could also be an indication that specific 'block' periods of therapy with a focused purpose, such as the transition from one developmental stage to another for children, are viewed positively by parents. Parents' and service providers' perceptions of services were explored through qualitative interviews. This information supported the findings from the standardized measures and questionnaires. The parents who

were interviewed provided many examples of strong service co‐ordination, opportunities for partnership, and participation. They felt that their children were making progress as a result of these services. Information about their child and the services available was identified as a key theme that influenced the parents' feelings of empowerment, and the parents who were interviewed suggested that more information is always welcome about their children's development. Service providers also reported positive perceptions of the new service, and focused primarily on continuing to improve communication between the 'old' team that knew the family well, and 'new' team that started to work with the child and family within the public school system. Overall, it can be concluded that the Long‐term Objective of this new Transition Service: 'To improve the transition process of clients and families, from pre‐school to school entry' was achieved for the clients and families who participated in this evaluation. The small number of participants limits generalizability of results to other settings, but the positive results support a continuation of this new service and ongoing programme evaluation using a programme logic model. Future evaluations could include a cost–benefit analysis of the new programme. Summary and conclusions This evaluation of a new service initiative at an outpatient children's rehabilitation centre was guided by a programme logic model. Process and outcome elements for evaluation were clearly identified in the model that was developed by Centre staff and researchers before the evaluation began. A combination of

quantitative and qualitative methods of evaluation was then used to measure the identified Outputs and Short‐term objectives. The results of this evaluation suggest that the programme logic model was a useful way to organize this evaluation. The model helped to guide the researchers and the Centre staff in using a systematic

approach to conducting a programme evaluation in a relatively short‐time frame. The model promoted ongoing communication between the evaluation team and Centre staff, who met regularly to review it and make any adjustments. Feedback from Centre staff at the end of the evaluation indicated that the programme logic model helped them to feel comfortable with and engaged in the programme evaluation. Based on the experience and results of this pilot study, this type of systematic approach to programme evaluation appears to be a viable and useful option for other children's services. Acknowledgements The authors wish to acknowledge the work of the Centre staff, parents and children who participated in this study. Their interest and efforts contributed greatly to the success of this collaborative evaluation. The students who completed the Chart Audits were valuable additions to the research team. Finally, we acknowledge the work of Nancy Pollock as an early member of the research team and an evaluator for the GAS part of this evaluation. References

1 Aubry, T., Farrell, S., O'Connor, B. V., Kerr, P., Weston, J. & Elliott, D. (2000) Family‐focused case management: a case study of an innovative demonstration program. Canadian Journal of Community Mental Health, 19, 63 – 78. 2 Canchild Centre for Childhood Disability Research (1999) Children with Disabilities in Ontario: A Profile

of Children's Services (FCS‐II). McMaster University, Hamilton, ON, Canada. 3 Centers for Disease Control and Prevention (1999) Framework for Program Evaluation in Public Health. Centers for Disease Control and Prevention, 48 (no. RR‐11), Atlanta, GA, USA. Retrieved 7 May 2004 from: http://www.cdc.gov/eval/ 4 Dwyer, J. J. & Makin, S. (1997) Using a program logic model that focuses on performance measurement to develop a program. Canadian Journal of Public Health, 88, 421 – 425. 5 Eccles, M., Grimshaw, J., Campbell, M. & Ramsay, C. (2003) Research designs for studies evaluating the effectiveness of change and improvement strategies. Quality and Safety in Health Care, 12, 47 – 52. 6 Evans, P. J., Zinkin, P., Harpham, T. & Chaudury, G. (2001) Evaluation of medical rehabilitation in community based rehabilitation. Social Science and Medicine, 53, 333 – 348. 7 Favaro, P. & Ferris, L. (1988) Increasing the relevancy and usefulness of programme evaluation: a comprehensive evaluation. International Journal of Health Education, 7, 28 – 31. 8 Gilliam, A., Davis, D., Barrington, T., Lacson, R., Uhl, G. & Phoenix, U. (2002) The value of engaging stakeholders in planning and implementing evaluations. AIDS Education and Prevention, 14, 5 – 17. 9 Goodstadt, M. S., Hyndman, B., McQueen, D. V., Potvin, L., Rootman, I. & Springett, J. (2001) Evaluation in health promotion: synthesis and recommendations. WHO Regional Publications. European Series, 92, 517 – 533. Greene, J. C. (2000) Understanding social programs through evaluation. In: Handbook of Qualitative Research, 2nd edn (eds N.K. Denzin & Y.S. Lincoln), pp. 981 – 1000. Sage Publications Inc., Thousand Oaks, CA, USA.

King, S., Rosenbaum, P. & King, G. (1998) Measure of Processes of Care. 20‐Item Version. CanChild Centre for Childhood Disability Research, Hamilton, ON, Canada. Kiresuk, T. J., Smith, A. & Cardillo, J. P. (1994) Goal Attainment Scaling: Application, Theory and Measurement. Lawrence Erlbaum Associates, Hillsdale, NJ, USA. Lal, S. & Mercier, C. (2002) Thinking out of the box: an intersectoral model for vocational rehabilitation. Psychiatric Rehabilitation Journal, 26, 145 – 153. Larsen, D. L., Attkisson, C. C., Hargreaves, W. A. & Nguyen, T. D. (1979) Assessment of client/patient satisfaction: development of a general scale. Evaluation and Program Planning, 2, 197 – 207. Leese, B., Baxter, K., Goodwin, N., Scott, J. & Mahon, A. (2001) Measuring the success of primary care organizations: is it possible? Journal of Management in Medicine, 15, 172 – 180. Letts, L. & Dunal, L. (1995) Tackling evaluation: applying a programme logic model to community rehabilitation for adults with brain injury. Canadian Journal of Occupational Therapy, 62, 268 – 277. Letts, L., Law, M., Pollock, N., Stewart, D., Westmorland, M., Philpot, A. & Bosch, J. (1999) A Programme

Evaluation Workbook for Occupational Therapists: An Evidence‐Based Practice Tool. CAOT Publications ACE, Ottawa, ON, Canada.

Newman, D., Scheirer, M. A., Shadish, W. & Wye, C. (1995) Guiding Principles for Evaluators. A report from the AEA Task Force on guiding principles for evaluators [www.document]. American Evaluation Association, Fairhaven, MA, USA. URL http://www.eval.org/Publications/publications.html#Guiding%20Prin [accessed on 6 May 2004] Ovretveit, J. & Gustafson, D. (2002) Evaluation of quality improvement programmes. Quality and Safety in Health Care, 11, 270 – 275.

Patton, M. W. (1986) Utilization‐Focused Evaluation, 2nd edn. Sage Publications Inc., Beverly Hills, CA, USA. Patton, M. Q. (1990) Qualitative Evaluation and Research Methods, 2nd edn. Sage Publications Inc., Beverly Hills, CA, USA. Pirkis, J., Herrman, H., Schweitzer, I., Yung, A., Grigg, M. & Burgess, P. (2001) Evaluating complex, collaborative programmes: the partnership project as a case study. Australian and New Zealand Journal of Psychiatry, 35, 639 – 646. Population Health Directorate (1996) Guide to Project Evaluation: A Participatory Approach. Health Canada, Ottawa, ON, Canada. Rush, B. & Ogborne, A. (1991) Program logic models: expanding their role and structure for program planning and evaluation. The Canadian Journal of Program Evaluation, 6, 95 – 106. Rutman, L. (1980) Planning Useful Evaluations: Evaluability Assessment. Sage, Newbury Park, CA, USA. Steinwender, S., Law, M. & Harms, S. (1998) Programme Development and Evaluation of an Early Intervention Programme. Unpublished document. McMaster University, Hamilton, ON, Canada. Stoltzfus, R. J. & Pillai, G. (2002) Measuring performance: a strategy to improve programs. Journal of Nutrition, 132, 845S– 848S. Taylor, A. R. & Botschner, J. V. (1998) Evaluation Handbook. Ontario Community Support Association, Toronto, ON, Canada. Zukoski, A. & Luluquisen, M. (2002) Participatory evaluation. What is it? Why do it? What are the challenges? Community-Based Public Health Policy and Practice, 5, 1 – 6.