Design and Planning of PHCs

Conducting the Evaluation

Anbrasi Edward, PhD Johns Hopkins University

The material in this video is subject to the copyright of the owners of the material and is being provided for educational purposes under rules of fair use for registered students in this course only. No additional copies of the copyrighted work may be made or distributed.

Planning the Evaluation

3

Planning for the Evaluation—1

► Design instruments, pretest, translate, maps, population data for selection of clusters, identify trainers and supervisors, develop guidelines for training, invitation list for dissemination

► Survey dates: KPC surveys, participatory evaluation, dissemination meetings

4

Planning for the Evaluation—2

► Select evaluation task team—invitation to participants; logistics for evaluation; organize additional data to be reviewed during evaluation exercise

► Select external consultant to serve as team leader; obtain donor approval

► Analysis

► Dissemination plan

5

Define the Purpose of the Evaluation

Summarize four to five objectives

► If the program met the stated goals and objectives

► The effectiveness of the technical approach (soundness of the program design and valid indicators to measure results; use of data for decision making)

► Capacity-building efforts within the project and among partners (specifically those enhancing sustainability)

► Lessons learned from the program (emphasis is on innovative activities—those which are transferable and contribute to sustainability of achievements)

► A strategy for application and communication of these lessons both within the organization and to partners

► Provide recommendations for follow-up or future program implementation

6

Evaluation Methodology

► Team leader ► Facilitation skills, familiar with context, program (HSS, RBM, HIV, CHIS, etc.), culture,

language, advocacy or marketing skills, credibility with scientific community ► Team members (title and role)

► MOH, donor representatives: USAID, UNICEF, community leaders, other NGOs ► Data collection and analysis; observation checklist; interview guide ► Site visits (purposeful sampling) ► KPC survey ► Other surveys or questionnaires ► Feedback session and report

7

Selecting Sites for Evaluation

8

Sample Evaluation Schedule

► Day 1: team leader meets with stakeholders to discuss objectives for program evaluation

► Day 2: project site—debrief project staff, plan evaluation, logistics, review data and project records

► Day 3–4: design KII/FGD guidelines, translate

► Day 5–6: teams conduct site visits

► Day 7–8: analyze information, make recommendations, document lessons learned

► Day 9–10: report summary findings and key recommendations, local and national stakeholders

9

Evaluation Costs

► Data collection (KPC survey or qualitative studies)

► Field visits to conduct observations, interviews

► Training venue, meals, accommodation for evaluation team, per diem (if MOH)

► Stakeholder dissemination meetings: AV, accommodation, meal arrangements

► External evaluator

► Report publication

10

Analyzing Information—Drawing Conclusions

► Are the objectives relevant in the program context?

► Were program objectives and targets achieved?

► If yes, what strategies were employed? At what cost?

► Constraints and successes—promising practices are identified

► Conclusions and recommendations

► Develop action plan to improve performance

11

Evaluation Report Outline—1

► Summary ► Evaluation methods, achievements, constraints, conclusions, and recommendations

► Results ► Comparison of baseline and final data, achievements, constraints ► Community awareness and demand for services ► Service utilization, coverage, quality (provider performance, essential drug availability,

etc.) ► Effectiveness of BCC strategies, sustainability, scale up

12

Evaluation Report Outline—2

► Program management ► Planning; protocols, manuals ► HR: supervision, training, satisfaction, turnover ► Technical support from HQ, MOH, academia, other ► Financial management—variance, cash flow ► Logistics management: supplies, transport ► Information management ► Stakeholder management and capacity building ► Lessons learned, sustainability, phase out plan, community

empowerment

► Conclusions and recommendations

► Appendices: team members, sites, survey reports, data forms

13

Writing and Reviewing SOW

Source: IEG. (2011). Writing terms of reference for an evaluation: a how-to guide. Retrieved September 10, 2018, from https://siteresources.worldbank.org/EXTEVACAPDEV/Resources/ecd_writing_TORs.pdf

14

USAID Final Evaluator Requirements

► Proven expertise and leadership in integrated community- oriented reproductive, MNCH projects

► Conduct of evaluations using mixed methods

► Experience with design, collection, and analysis using applied research methods

► Familiarity with public health system in Rwanda

► Demonstrated ability to communicate with and lead a team of stakeholders, staff, and national experts in participatory evaluation

► Familiarity with USAID programming

► Skill or familiarity with cost analysis methods

► Excellent analytical and writing skills (English)

15

Key Tasks of Evaluator

► Review project documents and resources

► Refine the evaluation objectives and key questions

► Develop the field evaluation schedule and assessment tools

► Train enumerators and team members on objectives and process of the evaluation including evaluation tools

► Lead the team to complete the collection, analysis, and synthesis of information on program performance

► Interpret results and draw conclusions, lessons learned, and recommendations regarding project outcome

► Lead an in-country debriefing meeting with stakeholders

► Prepare and submit report before September 1

16

USAID Guidelines for Final Evaluation

Source: USAID. (2013). Guidelines for final evaluation.

The material in this video is subject to the copyright of the owners of the material and is being provided for educational purposes under rules of fair use for registered students in this course only. No additional copies of the copyrighted work may be made or distributed.

Example Projects

2

Kilifi Project—Mid-Term Evaluation

Indicators Baseline KPC LQAS

2004 2005 2006 2007

Immunization - - - -

Proportion of children fully immunized 62 65 55 68

Pneumonia - - - - Child with cough or difficult breathing accessed care from HF in 24 hour

87 90 88.5 88

Malaria control - - - -

Child slept under an ITN previous night 21 39 83 76

Women received IPT for previous pregnancy 39 37 61 54

Child receiving correct treatment in 24 hours 18 55 64 NA

3

Project Performance

- Indicators Baseline KPC LQAS

2004 2005 200 6

2007

HIV / AIDS

Mothers aware of how to prevent HIV 41 NA NA 48

Mothers utilized VCT services 18 NA NA 57

Nutrition Child received vitamin A 61 NA NA 86

Diarrhea

HW before feeding/after defecation 4 39 39 27

Households using chlorine 1 11 16 13

Children with diarrhea received CRS 31 49 48 38

Children received increased fluids 48 64 63 75

4

Quality of Care: HFA

Case Management Observations Baseline HFA %

Mid-term HFA %

Assessment of sick child - -

All danger signs 21% 6%

All major symptoms 26% 33%

All assessment tasks for ARI 26% 43%

All assessment tasks for diarrhea 2% 21%

Correctly assessed for nutrition 5% 9%

All assessment tasks for fever 12% 8%

Temperature was checked before Rx NA 80%

Weighed before clinical observation 71% 90%

Immunization cards checked 53% 61%

Assessed correctly by health worker 41% 27%

5

Sustainability of C-HIS—1

Mozambique—Community Vital Registration and HIS

6

Sustainability of C-HIS—2

Mozambique—Community Vital Registration and HIS

7

Pioneering the Care Group Model in Rwanda

8

Rwanda Strategic Stakeholder Engagement—1

9

Rwanda Strategic Stakeholder Engagement—2

10

Rwanda Integrated CG Model

11

Supervision of Community- Based Health Interventions

12

Integrating Community HIS with District Statistics

13

World Vision Multi-Country Research Study

► Five-year multi-country multi-site mixed methods research study on impact of integrated package of community-based interventions for MNCH

► 2012–2017, $3.3 million in partnership with four country research institutions

Images: World Vision International. (n.d.). Child health and nutrition impact study: an overview. Retrieved September 11, 2018, from https://www.wvi.org/sites/default/files/CHNIS%202%20page%20study%20overview.pdf

14

Sustainable Development Goals and CHNIS

15

Disease Burden for IMCI Conditions, Immunization, Nutrition Outcomes Reported (past 2 weeks) Cambodia (n=3813) % Guatemala (n=4096) % Kenya (n=6407) % Zambia (n=2678) %

Diarrhea 23.3 17.8 8.0 16.1

Cough or difficult breathing 19.9 15.7 5.4 9.7

Fever 46.8 22.1 15.1 22.2

Full immunization 56.1 66.5 39.3 63.1

Measles vaccination 62.9 70.8 54.6 70.8

Underweight (<-2SD) 39.9 26.9 14.8 11.1

Severe underweight (<-3SD) 19.6 11.5 5.8 2.6

Stunted (<-2SD) 47.5 48.3 40.3 37.4

Severe stunting (<-3SD) 28.6 20.7 22.8 14.7

Wasted (<-2SD) 6.0 10.6 11.3 3.8

Severe wasting (<-3SD) 1.4 5.1 5.8 1.3

Anemic (<11g/dl) 74.8 - 57.2 -

Severe anemia (<7g/dl) 1.5 - 4.0 -

16

Quality of Screening and Assessment, IMCI

17

Child Caretaker Counseling

18

Stakeholder Perspectives—1

► KII and FGD ► Mothers and caretakers ► Volunteers ► Heath center staff ► Hospital staff ► Community development council ► TBA, midwives, other community providers ► Pastors, teachers, drugs sellers, shop keepers ► Representatives of clubs: women, youth, anti-SIDA ► Program staff ► District director ► Other partners and NGOs

19

Stakeholder Perspectives—2

20

Kilifi, Kenya— Formative Assessments

Community stakeholders FGD/KII (n) Participants (n)

Mothers, caretakers 11 107

Fathers 10 92

Care groups 12 114

CHW and CHW TOT 12 83

VHC 11 70

Head teacher 5 5

Village chief 6 6

CTC 3 31

DHMT 1 4

Health workers and in charge 4 4

PHO/PHT 4 4

DHC 5 35

AKS/PSI/Ministry of Agriculture 3 3

21

Kilifi, Kenya—Mothers, Caretakers

► Fathers play a minimal role in child care, not interested in clinic visits

► Very few access village bank services, not aware of puppet shows

► Financial constraints in purchasing food supplies for PD hearth

► Distance to HF is a major barrier to service utilization

► “Before the project started, we were in the ‘dark’ regarding health issues but now we are in the ‘light’! We no longer go to consult the traditional healers.” –Judith, Bundacho Village

22

Rwanda Fathers FGD

► Majority mentioned benefits of nutrition weeks and balanced food

► Most utilize health services as a result of CHW and CG advocacy and follow medical advice and prescriptions

► Claimed diseases decreased since the initiation of the program

► Many appreciated the strategies to understand health issues and even resolve disagreements at home

► Interesting accounts of improved relationships with their wives due to: ► Improved hygiene, reduced complaints in the home ► Provision of health insurance, give women security and

lessened pressure on men, hence less conflict in the home ► FP lessons have decreased the burden of raising a large

family they cannot

23

Kilifi, Kenya—Village Health Committee

► Equal representation of both male and females in VHC

► Willing to sustain project activities even after phase out

► Provide bicycles, badges, uniforms and stationery

► More outreach visits for remote areas

► “Before the project many children died of diarrhea and we did not know the cause. After we were trained we realized it was caused by drinking dirty water. The project provided water guard and Pur which we now use. We can now see that diarrhea has reduced. We are also trying to build toilets as a result of the project.” –Fatumah Ngao, Mlango Village

24

Kilifi, Kenya—Head Teacher

► Most schools have regular health activities

► Reduced illness, improved hygiene, less absenteeism among school children

► Teachers mobilize community for health promotion meetings

25

Kilifi, Kenya—Health Providers and In Charge

“The increased participation of the CHWs and trainers has led to a reduction in the work load at the health facility” Wahome, Clinical officer in charge Vitengeni Health Centre

“The transport provided by the project during the health outreaches has enabled us to surpass our set immunization targets” Nzioka, PHO

26

Kilifi, Kenya—DHMT

► “Before the project started we had frequent stock outs of Ab, now after the EPI logistic management support we no longer have stock outs in the dispensaries.”

► “Issues like the ‘talking nets’ can be effectively addressed by opinion leaders like the CORPS who can change the perception of the community.”

► “The 10 households/homesteads model is an excellent strategy to sustain health promotion activities which is now included in the MOH policy. Want to expand this concept to other districts.”

27

Rwanda—Community Voices

► “After attending NW, our husbands are now helping us getting ingredients for household food preparation and even in preparing them. Their attendance has even helped to improve our relationships at home.” A Mother in NW, Mugano Sector

► “Before I knew that a pregnant woman eats once a day. But today, I know that she has to eat three times per day. I was also not used to drinking clean water but I now drink boiled water.” Clementine, 22, Mushubi Sector

► “I have learnt how to breastfeed my baby. Before, I would wait for a long time and breastfeed when I had time, or when she cried, I would breastfeed her while doing other things. I now enjoy breastfeeding often and my baby enjoy it as well.” Said enthusiastically by Jacqueline, 24, from Mushubi Sector

28

Rwanda: Nyamagabe Vice Mayor and Health Director—1

► CG model’s significant contribution in district’s goal to eradicate malnutrition and stunting (was at 53% but has reduced to 41% at last recording)

► Project has helped the district to reduce infant and child mortality ► Women appreciate iCCM and there is a great level

of acceptability ► Even when they are referred to a health facility,

they go reluctantly, as they prefer to be treated by CHWs in their neighborhood

29

Rwanda: Nyamagabe Vice Mayor and Health Director—2

► Significant progress has been observed by the district in key indicators such as ANC, facility delivery, indicators on nutrition status of children under 2, etc.

► Improved community hygiene—people now understand the value of using and maintaining a clean pit latrine

► The CG infrastructure was used for advocacy against GBV, 100 percent coverage, reached all households

30

Afghanistan— RBF Evaluation

31

Mozambique Final Evaluation

Performance indicator Baseline Final

Full immunization 74% 91%

U5 who slept under an ITN previous night 0.3% 85%

Rx for fever < 24h in HC/HF 28% 90%

Child treated with ORT for diarrhea 53% 94%

Child received extra food for 2w following diarrhea

19% 87%

Rx for fast/difficult breathing < 24h in HC/HF

2% 87%

Severe MN 13% 7.2%

Trained providers in district MOH < 1% 100%

Delivery by SBA 66% 88%

Latrine coverage 28% 75%

32

Mortality Research Studies Assessments

- All household census data (HIS for 2002–2003)

- 3/2000-2/2001 3/2001-2/2002 3/2002-2/2003 3/2000-2/2003

Total births 4557 6244 6410 17211

Total child deaths

543 408 286 1237

Total infant deaths

318 231 153 702

Child death rate

119 65 45 71.9

Infant death rate

70 37 23.9 40.8

33

Comparison of Mortality Estimates

Source: Edward, A., Ernst, P., Taylor, C., Becker, S., Mazive, E., & Perry, H. (2007). Examining the evidence of under-five mortality reduction in a community-based programme in Gaza, Mozambique. Trans R Soc Trop Med Hyg, 101(8), 814–822. Retrieved from https://doi.org/10.1016/j.trstmh.2007.02.025

34

Cambodia—Full Immunization (12–23m)

35

Cambodia Final Evaluation Use of Iodized Salt

36

Cambodia—Verbal Autopsy 0–59m

37

Final Evaluation Promising Practices

► Cross cutting: “Traditional healers or drug vendors interested in participating in health promotion”

► Enhancing motivation: “Community micro-projects and incentive mechanisms enhance sustainability and ownership of project activities and outcomes”

► Community training: “Regular supervision of CHWs was instrumental in CHW motivation”

► Organizational learning: “Collaborations with AKF and Malaria Research Institute strengthened district health management and prevention of malaria”

► CG infrastructure; integration for emergency preparedness strategies

38

Theory of Change Concepts

► Process of desired change by making explicit the way we think about a current situation or problem, its underlying causes, the long-term change we seek, and what needs to happen in order for that change to come about

► Product that contains a set of hypotheses, outcomes, assumptions, and indicators that make up causal pathways of change needed to bring about a desired long-term goal

► Causal loop analysis

Adapted from Food Security and Nutrition Network. (2017). Theory of change training curriculum. Retrieved September 12, 2018, from https://www.fsnnetwork.org/theory-change-training-curriculum

39

Theory of Change

Source: Mobile Alliance for Maternal Action (MAMA). (2012). Global monitoring and evaluation network [PDF]. Retrieved September 12, 2018.

The material in this video is subject to the copyright of the owners of the material and is being provided for educational purposes under rules of fair use for registered students in this course only. No additional copies of the copyrighted work may be made or distributed.

Personal Reflections

2

Evidence of Local Innovations…

Source: Litsios, S. (2002). The long and difficult road to Alma-Ata: a personal reflection. International Journal of Health Services, 32(4), 709–732. Retrieved from https://doi.org/10.2190/rp8c-l5ub-4raf-nrh2

3

Read…the Fine Print

4

Read…the Fine Print—1

5

Read…the Fine Print—2

6

Addressing Food Security and AIDS Prevention through Youth Clubs

7

Youth Club Evaluation: CDD and Malaria

8

Knowledge on MN and HIV/AIDS

9

Disseminating Results

► Multi-stakeholder

► Content and delivery designed for strategic audience ► Donor (UNICEF, USAID, taskforce, other) ► MOH: district, national ► PVO staff, other NGOs ► VHC, VDC, CHW, etc. ► Community

10

Stakeholder Dissemination

11

Policy Makers

► Project HQ offices ► Results oriented: lessons learned and best practices ► Problems to be addressed

► Health system ► Coordination of local health activities ► Give credit to their contributions ► National—presentation if results have major implications

12

Kibogora District Hospital Director, Rwanda

13

Kibogora District Hospital

14

Scale Up—Care Group Model

► Dissemination of results: local, conferences

► Research publication—role of academic stakeholders

► Advocacy with donors for extension, scale up, and integration into national policy ► CG Model ► Mozambique: national scale up (USAID/UNICEF) ► Cambodia: USAID Mission, CS taskforce ► Rwanda: scale up, selected Provinces ► USAID: Coregroup consortium > 40 NGO’s ► CHIS; Kenya: national scale up ► CSC; Afghanistan, national scale up

15

Monitoring and Evaluation’s Raison d'Être—Grun

Source: Grun, R. E. (2006). Monitoring and evaluating projects: a step-by-step primer on monitoring, benchmarking, and impact evaluation. World Bank, Washington, DC. Retrieved September 12, 2018, from https://openknowledge.worldbank.org/handle/10986/13640

16

Performance Metrics

“You cannot manage what you cannot measure” –Attributed to Lord Kelvin

“Not everything that can be counted counts, and not everything that counts can be counted” –Attributed to Albert Einstein

17

Evidence from PHC—Informing Policy

“I am like this lamp, lighting the lamp of better health. Workers like me can light another and another and thus encircle the whole earth. This is ‘Health for All’” –Muktabai Pol, village health worker Jamkhed, India, GHC conference, DC