Design and Planning of PHCs

Elements of Project Evaluation

Anbrasi Edward, PhD Johns Hopkins University

The material in this video is subject to the copyright of the owners of the material and is being provided for educational purposes under rules of fair use for registered students in this course only. No additional copies of the copyrighted work may be made or distributed.

What Are the Elements of Project Evaluation?

3

Power of Measuring Results

 If you do not measure results, you cannot tell success from failure

 If you cannot see success, you cannot reward it

 If you cannot reward success, you are probably rewarding failure

 If you cannot see success, you cannot learn from it

 If you cannot recognize failure, you cannot correct it

 If you can demonstrate results, you can win public support

Source: Osborne, D., & Gaebler, T. (1992). Reinventing government. Penguin Press: New York, NY.

4

Purpose of M&E Systems

► Track progress, results, and effectiveness of programs

► Provide government officials, development managers, and civil society evidence from the investments, improving service delivery, planning and allocating resources, and informing management decision-making

► Accountability of resources

► Systems reengineering

► Reward successes and recognize improved performance

► Organizational learning and growth, opportunity for future funding

5

USAID CLA ► Collaborating, learning, adapting ► Facilitating collaboration internally and with external

stakeholders ► Integrate new learning, innovations, and performance

information into the current strategy to inform funding allocations, program design, and project management

► Translate new learning and information about changing conditions into iterative strategic and programmatic adjustments

► Catalyze collaborative learning, systemic analysis, and problem solving among communities and institutions to develop and implement programs that are more effective at achieving results

6

World Bank Monitoring and Evaluation (M&E) Are Synergistic—1

► Monitoring ► “A continuing function that uses systematic collection of data on specified indicators to

provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds”

► Embodies the regular tracking of inputs, activities, outputs, outcomes, and impacts

Source: World Bank. (n.d.).

7

World Bank Monitoring and Evaluation (M&E) Are Synergistic—2

► Evaluation (more detailed, time consuming, resource intensive) ► Process of determining the worth or significance of a project to determine the

relevance of objectives, the efficacy of design and implementation, the efficiency of resource use, and the sustainability of results ● An evaluation should enable the incorporation of lessons learned into the decision-

making process of both partner and donor

Source: World Bank. (n.d.).

8

Defining Project Evaluation

► “The systematic collection of information about the activities, characteristics, and outcomes of programs, for use by people to reduce uncertainties, improve effectiveness, and make decisions” (Patton, 2008)

► “Systematic collection and analysis of information about the characteristics and outcomes of strategies, projects, and activities as a basis for judgments to improve effectiveness, and/or to inform decisions about current and future programming” (USAID)

“If you can’t measure it, you can’t manage it” –W. Edwards Deming

“If you can’t measure it, you can’t improve it” –Peter Drucker

9

Rationale for M&E

► Well-designed and high-quality evaluation is important to track progress, results, and effectiveness of programs

► Appreciate why programs are succeeding or failing, and can provide insight to adapt and improve performance

► M&E contribute data-based evidence to facilitate strategic planning, project design, allow mid-term project remedies and resource decisions, and provide a forum for learning and growth

► Improve the quality of evaluation metrics, determine appropriate study designs, answer research questions, and accommodate external third party evaluations for objective inference on project performance

10

Evaluation Elements

► Systemic collection and analysis of data to assess the conceptualization, design, implementation, and utility of programs

► Requires data collection at multiple points in time (baseline and end of project) in order to demonstrate change

► Longer time frame, 3–5 years

► By examining longer-term results, identifying how and why activities succeeded, failed, or changed, evaluation informs the design of future projects

► Outcome or impact evaluations, which assess program achievements and effects

► Evaluation data may include some routine monitoring data but require a higher degree of rigor to collect and are gathered by an external evaluator to the implementing partnership

Source: Rossi, P. H., and Freeman, H. E. (1993). Evaluation: a systematic approach (5th ed.). Newbury Park, CA: Sage Publications, Inc.

11

“Drive out Fear” Edward Deming

► FEAR, ANTICIPATION, EXCITEMENT

► Evaluation outcomes ► End of project, second funding cycle, cost extension ► Promotion, performance incentives

► Nature of evaluation ► Internal

● Participatory—team leader engages program stakeholders in all aspects

● Opportunity for problem solving, less threatening ● Bias, validity, generalization

► External, third party ● Academic institutions, independent contractors ● Perform assessments and present to stakeholders

► Accreditation

The material in this video is subject to the copyright of the owners of the material and is being provided for educational purposes under rules of fair use for registered students in this course only. No additional copies of the copyrighted work may be made or distributed.

Evaluation Principles

2

Evaluation Principles: USAID—1

► Integrated into the design of strategies, projects, and activities ► Planning for and identifying key evaluation questions during the planning stage will

improve the quality of strategy development and project design and guide data collection during implementation

► Unbiased in measurement and reporting ► Not subject to the perception or reality of biased measurement or reporting due to

conflicts of interest or other factors

► Relevant ► Address the most important and relevant questions about strategies, projects, or

activities

3

Evaluation Principles: USAID—2

► Based on evidence-based methods that generate the highest quality and most credible evidence that corresponds to the questions being asked, taking into consideration time, budget, and other practical considerations

► Foster local capacity ► Consistent with aims of local ownership through strategic engagement of all

stakeholders, including community participants

► Transparent ► Cascade results downstream with a commitment to full and active disclosure of results

4

Evaluation Considerations—1

► Program context, objectives, intervention strategies

► External versus participatory, quantitative or formative

► Process or outcome measures

► Data sources: clinical observations, record review, participant interviews, formative assessment, focus groups and key informant interviews

5

Evaluation Considerations—2

► Enumeration, sample size, design effect, power calculations

► Sampling: simple random, systematic, stratified, cluster (multistage), non-probability, convenience, LQAS

► Clinical, managerial, cost effectiveness measures

► Other considerations ► Primary sampling unit, unit of analysis ► Informed consent procedures, ethical considerations ► Resources: time, personnel, finance

6

Hierarchy of Study Designs

Study design Definition

Randomized control trial (RCT)

Involves random allocation of the intervention and comparison (e.g., usual care) to different study groups, including measurement of the outcome before and after the intervention has been made.

Non-RCT Observations are made before and after the implementation of an intervention, both in a group that receives the intervention and in a control group that does not.

Case-control Compares people with a specific outcome of interest (“cases”) with people from the same source population but without that outcome (“controls”) to examine the association between the outcome and prior exposure (e.g., having an intervention).

Cohort Participants who do not have the outcome of interest are allocated (but not randomly) to receive alternative treatment, therapy, or interventions (experimental group) or the standard of care or conventional treatment (comparison group) or no intervention (control group) and then followed up to determine whether they experience the outcome of interest.

Cohort One group (pre- and post-): single selected group under observation, with a careful measurement being done before applying the experimental treatment and then measuring after. Two groups (pre- and post-): group of people with a common set of characteristics or a set of characteristics that are followed up for a period of time to determine the incidence of an outcome.

Interrupted time-series Uses observations at multiple time points before and after an intervention (the “interruption”) in an effort to detect whether the intervention has had an effect significantly greater than any underlying trend over time.

7

Study Designs Continued

Study design Definition

Case or time series Observations are made on a series of individuals, usually all receiving the same intervention, before and after an intervention but with no control group.

Cross-sectional Both exposures and outcomes are measured at a single point in time, and the prevalence of the outcome is compared among those with and without exposure. BEFORE/AFTER MEASURES

Quantitative, other Systematic empirical investigation of social phenomena via statistical, mathematical, or numerical data or computational techniques other than those listed above.

Cost-effectiveness Cost effectiveness analysis (CEA) is a type of economic evaluation that examines both the costs and health outcomes of alternative intervention strategies.

Mixed methods Combines data collection approaches, sometimes both qualitative and quantitative, into the study methodology. Some studies combine study designs, whereas others may have a single overarching research design but use mixed methods for data collection.

Implementation science

Scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice and to improve the quality (effectiveness, reliability, safety, appropriateness, equity, efficiency) of care. It includes the study of factors impacting behavior at the individual (i.e., provider, patient) and organizational levels.

Case study The study of selected current phenomenon over time to provide an in-depth description of essential dimensions and processes of the phenomenon. Case studies may involve both qualitative and quantitative data and may focus on single or multiple cases.

Participatory action research (PAR)

Systematic inquiry, with the collaboration of those affected by the issue being studied, for purposes of education and taking action or effecting change.

Qualitative Used to generate an understanding of complex, unquantifiable phenomena, such as people’s experiences or perceptions.

8

Hierarchy of Evaluation Methods?

► Robust experimental designs to weaker qualitative techniques?

► Contextual application; no right and wrong approach ► Appropriateness of the strategy, knowledge base, resource availability, time,

environment, level of analysis

9

Sample Size and Power Determinations

► An important consideration for any evaluation study is sample size ► Power analysis should be performed to estimate the sample size needed to detect

program effects with a reasonable level of certainty ► The power analysis can take into account various kinds of outcome indicators (such as

continuous, binary, and count measures) and the consequent statistical requirements for rejecting the null hypothesis assuming small, moderate, or large effect sizes, as appropriate

► Standard power calculations can be done using widely available software ► The longitudinal nature of the measures proposed may require large samples to

accommodate more independent and dependent variables and to account for anticipated attrition from the study

► In impact evaluations, timing is essential—many outcomes of interest may take months to demonstrate change (maternal mortality, illness incidence, etc.)

10

Crafting Evaluation Frameworks—1

► Performance indicators ► The logical framework approach ► Formal surveys ► Rapid appraisal methods ► Participatory methods ► Public expenditure tracking surveys ► Cost-benefit and cost-effectiveness analysis ► Impact evaluation ► Not a comprehensive list! Some are complimentary ► Broad and narrow applicability ► Depends on the need of users; the speed for

information and cost

11

Crafting Evaluation Frameworks—2

► Performance indicators ► The logical framework approach ► Formal surveys ► Rapid appraisal methods ► Participatory methods ► Public expenditure tracking surveys ► Cost-benefit and cost-effectiveness analysis ► Impact evaluation ► Not a comprehensive list! Some are complimentary ► Broad and narrow applicability ► Depends on the need of users; the speed for

information and cost

12

Crafting Evaluation Frameworks—3

► Performance indicators ► The logical framework approach ► Formal surveys ► Rapid appraisal methods ► Participatory methods ► Public expenditure tracking surveys ► Cost-benefit and cost-effectiveness analysis ► Impact evaluation ► Not a comprehensive list! Some are complimentary ► Broad and narrow applicability ► Depends on the need of users; the speed for

information and cost

13

Performance Indicators

► Measures of inputs, processes, outputs, outcomes, and impacts for development projects, programs, or strategies

► Appropriate study design with formal surveys, analysis and reporting, indicators enable managers to track progress, demonstrate results, and take corrective action to improve service delivery

► Setting performance targets and assessing progress toward achieving them

► Provides an early warning system to allow corrective action to be taken

► Effective means to measure progress toward objectives, facilitates benchmarking comparisons between different organizational units, districts, and over time

► Tendency to include too many indicators, or those without adequate data sources resulting in expensive, impractical, and underutilized data

14

Configuring Performance Frameworks

► Country demand driven and country led

► Build on existing in-country processes and experience

► Audience: inclusive, involves civil society and other stakeholders

► Align with leading global innovations

► System framework with a value chain and validated indicators (avoid collinearity)

► Address epidemiological transition

► Measurement within the scope of the healthcare organization ► Maturity and sophistication ► Mandate, skills, resources and independence to succeed ► Measures, methods, frequency, sampling, weighting ► Financing and cost structure

15

Common Project Errors

Source: The Project Cartoon. (n.d.). [Cartoon]. CC BY 3.0. Retrieved September 25, 2018.

The material in this video is subject to the copyright of the owners of the material and is being provided for educational purposes under rules of fair use for registered students in this course only. No additional copies of the copyrighted work may be made or distributed.

Evaluation Components

2

Evaluation Components—1

► Log frame with monitoring and evaluation (M&E) plan (goal, objectives, indicators, means of verification, activities, assumptions)

► Baseline, midterm, final, post-project (qualitative and quantitative)

► Performance monitoring (HMIS, supervision)

► Annual self-evaluations—informal, opportunity to redirect activity plan and strategy as well as build consensus

► Interim and formative evaluation, midterm review

► Special studies or evaluations

3

Evaluation Components—2

► Health facility assessments: utilization, quality of care, capacity, referral ► Household surveys: perceptions, knowledge, care-seeking behavior ► Community capacity assessment: health councils, traditional healers, religious leaders ► Organizational learning: cost/unit output, cost effectiveness analysis ► Stakeholder capacity building: partners, drug sellers ► Research: verbal autopsy, operations research: new products or drugs, delivery systems

(CHW blood pressure monitoring) ► Process and impact evaluation: public health significance ► Equity, gender, country ownership, “smart” integration, etc. ► Scale: can the project be scaled up geographically to other regions? ► Sustainability: does the project have strong partnerships and financing for sustainability?

4

Facility Performance Measures

Method Description

Service provision assessment, DHS measure

National sample, 3–5 years

Rapid—SPA, CSTS Provincial/district, 3–5 years

Service availability mapping, WHO Mapping of service and resource availability, national and district level decision making, annual

Health facility census, JICA Health assets mapping, national, 5 years

FASQ, measure Facility audit of service quality

QIQ, measure Quick investigation of quality

LQAS, (EPI, HIV/AIDS) Lot quality assurance sampling

IMCI—MCE, WHO, Rapid HFA Focus on child health

GFATM, RBM, EPI, RBF, etc. Global initiatives

Modules include system readiness, QoC, provider and client interviews, donor assistance, social marketing, record reviews, costing, etc. Technical areas: MNCH, STI, TB, HIV/AIDS, FP, etc. Management, financing, motivation, design, sampling, duration and types of data collection methods vary

5

Community and Household Surveys

Method –

UNICEF MICS Multi-indicator cluster surveys

USAID, DHS Demographic health survey

KPC 30 cluster surveys Knowledge practice coverage

KAP 30 cluster surveys Knowledge attitude practice

PEPFAR, PMI President’s initiatives for AIDS, malaria

WHO verbal autopsy Verbal autopsy for non-clinicians

CS-KPC rapid catch Child survival key performance indicators

PETS, WB Public expenditure tracking surveys

LSMS Living standards measurement surveys

Indonesia FLS Indonesia family life surveys

HEIDE Household expenditure and income data for transitional economies

IHSN International household survey network

DDP WB development data platform

6

Multidimensional Evaluation Measures—1

► Health system and district team ► Capacity building: training—technical,

managerial, information systems ► Utilization patterns, quality of care,

equity (OOP), efficiency ► Supply chain management; HR,

essential commodities, facility preparedness

► Community linkages; CHW training and oversight

► Cost recovery ► Scope, cost, data propriety

► PVO/NGO ► Capacity building; training, field visits,

etc. ► Communication, annual and quarterly

reports ► Staff competency, performance,

satisfaction; oversight, personnel development, compensation systems, etc.

► Financial management ► Organizational learning and leadership

7

Multidimensional Evaluation Measures—2

► Program effectiveness ► Achievement of goals and objectives

● Quantitative: MICS/KPC ● Qualitative: KII, FGD, etc.

► Community capacity and health system capacity

► Sustainability strategy, scale up

► Community ► Participation, empowerment ► Functional community structures:

CHW, VHC, health facility councils and other interest groups, etc.

8

Global Reference List

► Guide monitoring of health results nationally and globally

► Reduce excessive and duplicative reporting requirements

► Enhance efficiency of data collection investments in countries

► Enhance availability and quality of data on results

► Improve transparency and accountability

► Disaggregations that include equity stratifiers (e.g., age and sex, geography, socioeconomic status, place of residence)

► Data sources: civil registration, vital statistics systems, population-based health surveys, facility-generated data, routine facility information systems, health facility assessments and surveys, administrative data sources such as financial and human resources information systems

Source: WHO. (2018). Global reference list of 100 core health indicators (plus health-related SDGs), 2018. Retrieved September 26, 2018, from http://www.who.int/healthinfo/indicators/2018/en/

9

WHO 100 Core Health Indicators

Source: WHO. (2018). Global reference list of 100 core health indicators (plus health-related SDGs), 2018. Retrieved September 26, 2018, from http://www.who.int/healthinfo/indicators/2018/en/

10

Impact Evaluation

► Systematic identification of the effects; positive/negative, intended/not, on individual households, institutions, and the environment caused by a program or project

► Helps determine equity, effectiveness, coverage, and outcomes ► Range from large scale sample surveys and control groups are

compared before and after, to small-scale rapid assessment ► Measuring outcomes and impacts of an activity and

distinguishing these from the influence of other, external factors

► Accountability of investments, informing decisions on whether to expand, modify or eliminate projects, programs or policies

► Drawing lessons for improving the design and management of future activities

► Comparing the effectiveness of alternative interventions ► Expensive, time consuming

11

Cost-Benefit and Cost-Effectiveness Analysis

► Cost-benefit analysis measures both inputs and outputs in monetary terms ► Cost-effectiveness analysis estimates inputs in monetary terms and outcomes in non-

monetary quantitative terms

► Informing decisions about the most efficient allocation of resources

► Identifying projects that offer the highest rate of return on investment

12

Rapid Appraisal ► Rapid low-cost ways to obtain feedback of participants and other key stakeholders

► Provides real time information for management decision- making, especially at the project or program level

► Provides qualitative understanding of complex socioeconomic changes, highly interactive social situations, or people’s values, motivations, and reactions

► Providing context and interpretation for quantitative data collected by more formal methods

► Findings usually relate to specific communities or localities— thus difficult to generalize from findings

13

Rapid Appraisal Methods—1

► Key informant interviews: a series of open-ended questions posed to individuals selected for their knowledge and experience in a topic of interest ► Interviews are qualitative, in-depth, and semi-structured

and use guides with a list of topics or questions

► Focus group discussions: a facilitated discussion among 8–12 carefully selected participants with similar backgrounds ► Participants might be in the study, program staff, or

stakeholders ► The facilitator uses a discussion guide, note-takers record

comments and observations

14

Rapid Appraisal Methods—2

► Direct Observation: use of a detailed observation form to record what is seen and heard at a program site

► Mini-survey: structured questionnaire with a limited number of closed-ended questions that is administered to 50–75 people ► Selection of respondents may be random or “purposive”

15

Participatory Evaluation

► Collaborative problem solving

► Contributes to learning and leads to strengthened program design and implementation

► External expert plays a facilitating role with stakeholders

► Share ideas and encourage team to consider options

16

Participatory Methods

► Learning about local conditions and local people’s perspectives and priorities to design more responsive and sustainable interventions

► Identifying problems and troubleshooting problems during implementation

► Evaluating a project, program, or policy—mixed methods studies

► Providing knowledge and skills to empower poor people

► Stakeholder analysis: develop an understanding of the power relationships, influence, and interests of the various people involved in an activity and determine who should participate and when

17

Constraints of Participatory Evaluation

► Considered less objective because of stakeholder involvement

► Difficulty in addressing highly technical aspects

► Time and resources for conduct and consensus of an array of stakeholders

► Risk of domination and misuse by some stakeholders to further their own interests

However, the benefits of fully engaging stakeholders throughout the evaluation outweigh these concerns (Fetterman et al., 1996)

18

Mixed Methods Evaluation

► In addition to determining impact, it is equally important to understand the implementation process of the program, the successes and challenges to bring about improvements in quality of the program, and learning for program improvement and expansion

► Therefore, it is critical that the impact evaluation and formative research are systematically aligned

19

Data Quality, Storage Security

► Validity—data clearly and adequately represent the intended result

► Integrity—data have safeguards to minimize the risk of transcription error or data manipulation

► Precision—data have sufficient level of detail to permit management decision making

► Reliability—data reflect consistent collection processes and analysis methods over time

► Timeliness—data are available at a useful frequency, are current, and timely enough to influence management decision making

Proper data storage and security are critical to protecting data integrity, optimizing data usability, and safeguarding potentially sensitive or personally identifiable information.

20

Data Analysis—1

► Data may require some validation and cleaning before analysis to ensure they are accurate and make sense

► Data may require substantial analysis before it is included in a report

► Type of analysis depends on the kind of data collected and intended use

21

Data Analysis— 2

► Qualitative data will undergo content or pattern analyses to see trends

► Quantitative data may be simple analyses to generate sums or averages, disaggregated, or they require complex approaches such as regression analyses or multilevel modeling (logistic, Poisson, etc.)

► Uses dependent variables designed to reflect the appropriate level of measurement for each outcome of interest—binary events, such as birth in a clinic, or other events, such as number of antenatal clinic visits and number of birth complications, may be measured with count variables

► Include data analysis plan, software, etc.

22

Presentation and Use of Data

► Data can presented using a variety of visuals: tables, graphs, and charts as appropriate

► Key findings will be summarized in PowerPoint presentations

► The data can presented at the monthly and quarterly meetings within the project; annually with partners; and end of project with donors and key stakeholders

► The primary use of the data will be to inform the program management if changes are required for program activities

23

Road Map ► Formulate research and program questions, theory of change

► Configure evaluation framework, determine indicators, data sources

► Study design, sampling, data collection strategies

► Conduct and manage the evaluation

► Data management

► Results sharing and reporting—use, learning, action plans

24

References ► Monitoring Toolkit (USAID) ► Performance Indicator Reference Sheet (USAID)

► Ten Steps to a Results-Based Monitoring and Evaluation System (World Bank)

► Monitoring and Evaluation: Some tools, methods and approaches (World Bank)

► The Monitoring and Evaluation Handbook For Business Environment Reform (World Bank)

► Impact Evaluation Toolkit (World Bank)

► Global Reference List of 100 Core Indicators (WHO)

► MAMA Global Monitoring and Evaluation Framework (USAID)

References retrieved September 26, 2018.