Design and Planning of PHCs

Collecting and Managing Information for Project Management

Gilbert Burnham, MD Johns Hopkins University

The material in this video is subject to the copyright of the owners of the material and is being provided for educational purposes under rules of fair use for registered students in this course only. No additional copies of the copyrighted work may be made or distributed.

Part 1: Basic Concepts of Monitoring and Evaluation

3

Introductory Thought

“You manage what you can measure,” said Lord Kelvin in a roundabout way

► In your PHC project, you are promising what you will achieve for a population with someone’s resources

► In this session, we will discuss: ► How to collect data that will help you manage the

project you are now designing (mostly) ► How to determine what your project will have

achieved for resources used (a bit)

Image: Messrs. (1910). Portrait of William Thomson (Lord Kelvin) [Photo]. Via Wikimedia Commons. Retrieved June 19, 2018.

4

Overview

We will talk about: ► The purpose of collecting project information

► Definitions of monitoring and evaluation

► What information should be collected

► When and from where it should be collected

► Who is responsible for doing this

► Once the data have been collected, how it is used

► How this all fits together in a monitoring and evaluation (M&E) plan

5

Commonly: Monitoring and Evaluation (M&E)

6

Monitoring and Evaluation—1

7

Monitoring and Evaluation—2

8

Monitoring and Evaluation—3

9

Why Monitor the Project?

► While the project is running ► Are the objectives still the right objectives? ► Are the methods being used still the appropriate ones? ► Is the project making adequate progress in achieving

these objectives? ► Is it on track? ► Are unexpected issues emerging?

► What are we learning about the community? ► Fine tuning our implementation ► Building ownership by the community members

► How are we using project resources?

10

Why Evaluate the Project

Do this periodically—annually or end of project. These data help you understand: ► Has the project made the change that it promised in the

proposal? ► Were there major outcome changes to health status?

● Beware of trying to assess mortality shifts ● Beware of trying to assign causality

► Were there unintended findings? ► Positive and negative

► Were the methods used appropriate? ► Were the costs to achieve the outcomes realistic? ► The evaluation relies on much the same data as the monitoring

plan—so there are no shortcuts ► We will discuss evaluation in detail later

11

The Monitoring Plan—1

► Your project needs a monitoring plan—a priority in the review of proposals for funding

► This needs an adequate budget—and usually will amount to 5–10% of the total project budget

► One person has overall charge of the process

► Monitoring plan starts clearly with the indicators and log frame

12

The Monitoring Plan—2

► Indicates the nature of data collected and its frequency

► Specifies the sources of data and how they flow into the monitoring and analytic process

► Details the reports to be generated, their circulation, and how the information will be used

► This information is accountability information to stakeholders

13

Monitoring Plan Based on the Study Design

Monitoring plan is derived from the project logic framework

14

Monitoring Starts with the Indicators—1

► What objectives were selected? ► For these, what were the indicators? ► Think again—do they measure the

things that will be the most helpful? ► Too many indicators bog down

monitoring and too few make the project blind

► The log frame is the source of indicator information

15

Monitoring Starts with the Indicators—2

► Data management ► Who collects the data and how often? ► What are the forms? ► How do the forms reach the

monitoring office? ► Data entry, analysis, and report

creation

16

Collecting the Data

► Data collection—how hard will this be? ► If it is too difficult, it will be faked or omitted ► As you put together the monitoring plan, it is a good

time to think what data you really need ► Which indicators will be most helpful for decision-

making?

► The question always arises—is this monitoring system a health information system or a management information system? ► There may be some health data recorded ► The majority will be management-related information

17

Health Information System (HIS) or Health Management Information System (HMIS)

18

Not All Data Needs to be Centrally Analyzed

► Level 1 analysis ► Data collected from community, field and project offices ► Sent to project level for analysis and feedback and often

for action ► Level 2 analysis

► Data collected at community level ► Analyzed at field level ► Acted on at field and community level, without recourse

to level 1 ► Level 3 analysis

► Data collected, analyzed, and acted on at community level

The material in this video is subject to the copyright of the owners of the material and is being provided for educational purposes under rules of fair use for registered students in this course only. No additional copies of the copyrighted work may be made or distributed.

Part 2: Designing Monitoring to Manage the PHC Project

2

Managing the Project

Main reason for collecting data is for project CONTROL ► Control is the process by which projects deliver the desired outcomes, and this is based

on information or feedback ► Control is the sum of all activities that help a project to achieve its objectives and

outcomes ► Control is based on collection, analysis, and use of information to make appropriate

decisions

3

Why Collect Information? Project Control Mechanisms

4

Information Needed for Project Control Varies with Project Stage

5

Establishing Monitoring Plan

6

Using the Information Generated

► Information generated has important uses ► Monitoring has a major steering function ► Trends can be measured (Cluster or LQAS) ► Efficiency of implementation can be tracked ► Operational research can take place as project unfolds ► There is support for a resetting of mid-course targets ► Information can help recast objectives being implemented ► Focus is kept on project priorities

7

Data for Decision- Making—1

8

Data for Decision- Making—2

9

Data for Decision- Making—3

10

Data for Decision- Making—4

11

Establishing Monitoring—1

12

Establishing Monitoring—2

13

Establishing Monitoring—3

14

Establishing Monitoring—4

15

Establishing Monitoring—5

16

Establishing Monitoring—6

The material in this video is subject to the copyright of the owners of the material and is being provided for educational purposes under rules of fair use for registered students in this course only. No additional copies of the copyrighted work may be made or distributed.

Part 3: Making the Monitoring System Work

2

PHC Projects Commonly Collect…

Information about: ► Implementation process ► Outputs (persons trained, health talks delivered) ► Indicators of progress toward meeting project objectives ► Changes in perceptions (outcomes) ► Changes in behavior or knowledge or practices ► Trend is to target individuals or households rather than

communities

Looking for: ► Changes in health status ► Changes in household factors ► Changes in practices

3

How to Collect This?

► Continuous input (project activity records)

► Routine reports summarizing activities in the work plan

► Regular surveys (monthly, quarterly, half-yearly, etc.) ► Quantitative

● Exit interviews ● Drugs given versus drugs

prescribed ► Qualitative

● Market surveys ● User, non-user surveys

► Special surveys (midterm, final surveys) or OR project such as PRA, KAP/KPC, anthropometry, LQAS, anemia

► OR: operational research

► PRA: participatory rural appraisal

► KAP: knowledge attitudes and practice

► KPC: knowledge practice and coverage

► LQAS: lot quality assurance sampling

4

When to Collect?

► Daily or weekly or monthly? Ask: ► How much accumulation to allow? ► How quickly is action needed?

► Daily—tick  register or tally sheet can record actions as they occur ► Sorting through this volume may be massive work

► Monthly or quarterly data summary ► Data is usually much more complex ► There may be problems in transfer of data

► Annual survey data ► Data collection for midterm and final evaluation

5

Who Collects Information?

► Health workers may report on own work ► Illiterate or minimally

educated CHWs ● Problems:

complexity—growth monitoring-trends

► Or professionals? ● Problems: busy with

clinical work, record keeping is a low priority

► Monitoring component must have its own team and its own data flow system

6

Who Manages? And Where?

► One person responsible for data management in the project ► From central level through to the field and community

level

► Person has skills in: ► Managing the monitoring team ► Data management ► Data entry, cleaning, and analysis ► Writing reports, making presentations, and

interpretation

► Managers must protect this person’s time

7

Managing the Data—1

► Protocols must be established from the beginning for:

► Entry of data—software, entry format, timeliness

► Storage of data files, backup of files, and storing paper forms

► Aggregation of data, and format for the reports

► Handling missing data

► Data quality and logic checking

8

Managing the Data—2

9

Managing the Data—3

10

Managing the Data

Protocols must be established from the beginning for: ► Ensuring data collected matches indicators for

objectives

► Ensuring data match narrative reports

► Any change in data collection methods

► Changes to the type of data being collected

► New variables being added—which will occur

► What to do when disaster strikes—as it will

11

What to Do with the Data?

► The purpose of data is to help make stronger decisions ► And to verify that activities are on

target

► There is a critical need to provide rapid feedback to decision makers and stakeholders

► Remember decision makers at village and field level

12

Management Uses Data for Decision-Making

The NGO board of directors is receiving information to plan the winter household assistance program—Georgia/Russia border area

13

Data to the PHC Clinic Level

14

Feedback to the Community

15

Reporting Results

Who needs this information to make decisions? ► Local political leadership

► Health workers and field managers who are project partners

► HQ management

► Local health system

► Ministry of Health or national health system

► Donors—country office as well as source for the funding

16

Monitoring Reports for the Project Staff

► For project staff, think of a standard format ► Weekly, monthly, quarterly, annual? ► Product: often a workshop and a written

report ► Photos of project activities easy to do ► Display of charts and graphs

► Interpretation

► Recommendations

17

Public Launch or Reports

► Public meetings can be helpful to announce the launch of a new initiative or at the conclusion

► Also good to announce particularly noteworthy achievements

18

Overview of the Monitoring Process—1

19

Overview of the Monitoring Process—2

20

Overview of the Monitoring Process—3

21

Overview of the Monitoring Process—4

22

Looking for New Approaches

► M&E long static—now new efforts that emphasize learning ► Long felt monitoring was rigid and provided historical information ► There is a need to change from “what has happened” to “what is happening now” ► A new term is emerging: “Developmental evaluation”—evaluation as you implement

► Rapid feedback  rapid course correction ► Learning collaboratives have been able to rapidly respond to monitoring data ► Narrative-based approaches is another approach ► Measurement in complex, rapidly shifting environments is hard

► Numerators and denominators are sometimes hard to measure ► Implementation science is setting out to change the way monitoring data are being

utilized

23

Summary ► Monitoring produces the project vital signs ► The monitoring plan is based on indicators developed in

conjunction with the objectives ► The management of monitoring data is based on the log

frame ► Specific plans for this made at the beginning of the project ► Separate team manages the monitoring process ► In all, this will consume 5–10% of the budget ► Main goal—giving project leadership the data needed to make

the right decisions to keep the project on track ► Monitoring officer needs to know data management and

analysis ► But also have the presentation, interpretation skills, and be

able to make recommendations to project leadership ► The information generated guides decision-making, which

keeps the project focused on objectives