write a 2-3 page (that means a minimum of 2 full pages and no longer than 3 full pages) paper (in APA format)

RESEARCH ARTICLE Open Access

Using routine health information data for research in low- and middle-income countries: a systematic review Yuen W. Hung1, Klesta Hoxha1, Bridget R. Irwin2, Michael R. Law3 and Karen A. Grépin4*

Abstract

Background: Routine health information systems (RHISs) support resource allocation and management decisions at all levels of the health system, as well as strategy development and policy-making in many low- and middle- income countries (LMICs). Although RHIS data represent a rich source of information, such data are currently underused for research purposes, largely due to concerns over data quality. Given that substantial investments have been made in strengthening RHISs in LMICs in recent years, and that there is a growing demand for more real-time data from researchers, this systematic review builds upon the existing literature to summarize the extent to which RHIS data have been used in peer-reviewed research publications.

Methods: Using terms ‘routine health information system’, ‘health information system’, or ‘health management information system’ and a list of LMICs, four electronic peer-review literature databases were searched from inception to February 202,019: PubMed, Scopus, EMBASE, and EconLit. Articles were assessed for inclusion based on pre-determined eligibility criteria and study characteristics were extracted from included articles using a piloted data extraction form.

Results: We identified 132 studies that met our inclusion criteria, originating in 37 different countries. Overall, the majority of the studies identified were from Sub-Saharan Africa and were published within the last 5 years. Malaria and maternal health were the most commonly studied health conditions, although a number of other health conditions and health services were also explored.

Conclusions: Our study identified an increasing use of RHIS data for research purposes, with many studies applying rigorous study designs and analytic methods to advance program evaluation, monitoring and assessing services, and epidemiological studies in LMICs. RHIS data represent an underused source of data and should be made more available and further embraced by the research community in LMIC health systems.

Keywords: Routine health information systems, Low- and middle-income countries, Systematic review

Background Routine health information systems (RHISs) collect and provide information at regular intervals on services and activities delivered in health facilities [1]. RHISs have been implemented in many low and middle-income

country (LMIC) health systems to support resource allo- cation and day-to-day management decisions at facility, district, provincial, and national levels, as well as to fa- cilitate strategy development and policy-making [2, 3]. Despite the fact that RHISs are being implemented at scale in many LMICs, and that they have been widely recognized as an important component of health sys- tems strengthening [4, 5], prior studies have suggested

© The Author(s). 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

* Correspondence: [email protected] 4School of Public Health, Hong Kong University, Pok Fu Lam, Hong Kong Full list of author information is available at the end of the article

Hung et al. BMC Health Services Research (2020) 20:790 https://doi.org/10.1186/s12913-020-05660-1

that researchers continue to prefer using intermittent cross-sectional population-based surveys rather than RHISs data to conduct studies, including the monitoring of health programs and policy evaluations [6–8]. In order to improve health system performance, reli-

able, timely, and transparent data on health services are crucial [9, 10]. RHISs collect such data and thus could provide important insights into health system perform- ance [4]. Substantial investments have been made in the development and strengthening of RHISs in many LMICs over the past two decades [5, 11], and interventions tar- geting data collection, processing, analysis, and dissemin- ation have increased the accessibility of RHIS data [5, 12]. While early RHISs were established using paper- based health facility reports, newer web-based systems have been adopted in many LMICs over the last decade [13, 14]. The most common of these is the District Health Information System 2 (DHIS 2) platform, which is used as the foundation for the national health manage- ment information systems (HMIS) in at least 46 countries and has been piloted in at least another 21 countries [15]. Studies have shown that the implementation of newer in- formation and communication technology systems, along with supportive feedback mechanisms to encourage their use in routine practice, can lead to substantial improve- ments in RHIS reporting and data quality [5, 13, 16, 17]. Despite the documented improvements in data quality,

recent studies have shown a persistent underuse of RHIS data for research purposes in LMICs [8, 18]. A number of factors may contribute to the underuse of RHIS data. Numerous studies and commentators have questioned the usefulness of RHIS-sourced data to monitor and evaluate health services due to data quality concerns, such as incompleteness and inaccuracy [19–23]. Add- itionally, RHIS data are often not publicly available for secondary analyses, which further limits their use [24]. Due to these concerns, the research community has shown a persistent preference to use data sourced from intermittent cross-sectional population-based surveys ra- ther than data sourced from RHISs to conduct research on health service utilization and policy evaluation in LMICs [8, 18, 25, 26]. However, population-based sur- veys also have drawbacks, including the fact that they may be costly [26] and are often unable to generate suf- ficient data at the district or other subnational-levels [27]. In addition, reliance on such data may encourage the use of potentially weak evaluation designs [8] and may make establishing an appropriate baseline challen- ging when trying to evaluate specific programs [28]. Intermittent cross-sectional population-based surveys themselves also suffer from a number of quality concerns and thus should not be considered the gold standard for estimating service coverage or other population-based estimates [29].

Given the potential of RHISs to play a greater role in the evaluation of health policy and programs and to monitor the performance of health systems, it is import- ant to better understand the extent to which such data are currently being used in research studies. To date, there have been no systematic reviews of the use of RHIS data for research purposes beyond studies that were specific to malaria control [18], a gap this paper seeks to address. Specifically, we systematically reviewed the published literature to identify and describe the dif- ferent ways in which RHIS data have been used in peer- reviewed research, including the types of health conditions studied. We also summarized the different methodologies that have been used to analyze RHIS data in research and the types of strategies that were applied to circumvent common RHIS data issues, such as in- complete or inaccurate data. It is our goal to provide guidance to other researchers who may be interested in using such data for research purposes by helping them to gain a better understanding on how such data have been successfully used in other contexts.

Methods This systematic literature review followed the Preferred Reporting Items for Systematic Reviews and Meta- Analyses (PRISMA) guidelines. Peer-reviewed published studies that used data from RHISs in LMICs were in- cluded in this study, where RHISs were defined as data systems designed to collect and generate information on services provided by health facilities at regular intervals of a year or less [1]. This included data systems that col- lect information on clinical service delivery, pharmaceu- ticals, or diagnostic service provision, as well as routine administrative management. Conversely, systems that collect individual-level data for clinical decision-making purposes and pilot systems to test the implementation of a new data collection component or method were not considered to be RHISs.

Search strategy Relevant studies were identified through an electronic search of four databases of peer-reviewed literature: PubMed, Scopus, EMBASE, and EconLit -- from incep- tion through February 20, 2019, the date we launched the search. For each database, we identified studies that contained any of the following free text terms in their ti- tles or abstracts: ‘routine health information system’, ‘health information system’, or ‘health management in- formation system’, and any LMIC, as defined by the World Bank’s 2019 classifications (Appendix 1). Articles were included in the study if they met the following cri- teria: a) full-text article available in English, b) original research, and c) used data from a RHIS in at least one LMIC for research purposes. In order to be considered

Hung et al. BMC Health Services Research (2020) 20:790 Page 2 of 15

as having used data from a RHIS for research purposes, studies had to involve an analysis, either descriptive or analytical, of RHIS data, or applied RHIS data to inform their study design. We excluded studies that: a) only de- scribed RHISs, b) only described the administrative decision-making process, c) only focused on RHIS data collection issues, or d) only assessed RHIS data quality.

Selection of studies Figure 1 shows the number of articles identified and retained at each stage of the systematic review process. After removing duplicates from the various database searches, we identified 1459 potential articles. Two re- viewers independently screened the search results by title and abstract for inclusion eligibility. When there was insufficient information to determine eligibility at the title and abstract screening stage, the article was in- cluded for full-text screening. Full texts of the potentially eligible articles were then obtained and further screened

for inclusion eligibility. At both stages, the reasons for excluding individual articles were recorded. The full- texts for all but one article were found. Disagreements at each stage were resolved through discussion. Where an agreement could not be reached, a third reviewer made the final determination.

Data extraction and analysis Two authors extracted data from all included studies using a piloted data extraction form. For each included article, data were extracted on study design, study ob- jective, disease or health condition categories, study sample, description of RHIS data used, use of other data sources, analytic methods of RHIS data, strategies ap- plied to circumvent data quality issues, and study find- ings. Due to the heterogeneity of the studies in terms of study design, study purpose, health conditions, and ana- lysis methods, we thematically analyzed the studies ac- cording to research purpose, types of diseases studied,

Fig. 1 PRISMA flowchart of study identification and screening process of publications use RHIS data

Hung et al. BMC Health Services Research (2020) 20:790 Page 3 of 15

analytic methods applied, impact factor of journals in which the articles were published, and types of strategies used to circumvent RHIS data quality issues.

Results Of the 1459 unique articles retrieved from the database search, 132 studies met the inclusion criteria after full- text screening and were thus included in the review. The characteristics of these studies are presented in Table 1. Our review identified studies from 37 different countries. Three quarters of the studies were from Sub-Saharan African countries (74%), followed by South Asia (11%). The vast majority of the studies were published in the last decade, and more than half were published after 2014 (55%), suggesting an increase in the use of RHIS data for research purposes over time. Most of the studies included an analysis of RHIS data (97%), and a few used RHIS data to inform the study but did not describe ana- lysis of RHIS data. One study, for example, used infor- mation from RHISs to justify for the selection of the indicators to be used at the individual-level in their study. Among the studies that analyzed RHIS data, most utilized an ecological study design (79%). Of those, more than half included statistical inferences (61%), while the remaining studies only used RHIS data for descriptive purposes (39%). Nearly a fifth of the studies were mixed methods or case studies (18%), a third of which included statistical analyses of RHIS data (33%). A quarter of articles included a description of how they managed missing data (25%), while only a small number of studies described how they detected and dealt with extreme values (14%).

Types of disease and research purpose Figure 2 shows the different research purposes for which RHIS data were used, along with the health topics inves- tigated. The most common purpose of the studies was program evaluation (51%). RHIS data have been used to evaluate a wide range of interventions, ranging from programs that targeted specific diseases to interventions or policies that affected multiple types of diseases or health services. These included: the effect of malaria control strategies [30–36], user fee exemption policies [37–40], health financing schemes [41–44], interventions on health governance [45–53], the administration of new vaccines and vaccination campaigns [54–56], as well as community-level interventions such as approaches to enhance community participation and improve referrals from traditional birth attendants in increasing the de- mand for maternal and child care [57–59]. Additionally, RHIS data were used to monitor or as-

sess service provision (23%) and to describe disease epi- demiology (17%). Similar to the program evaluation studies, these studies also investigated a diverse set of

health services and the allocation of healthcare re- sources. Some of these studies found large discrepancies between RHIS data and an estimated disease burden in populations or highlighted the lack of service provision. A few studies also used RHIS data to describe specific programs [60–64], conduct impact evaluations (non- programmatic) [65–68], and estimate costs [69, 70]. Most of the studies investigated a communicable dis- ease (95%), of which malaria was most studied health condition (24%). A few studies focused on mental health (2%), diabetes mellitus (1%), and permanent tooth extraction (1%). Only two studies used RHIS data to research the health workforce or the equity of funding allocations [71, 72].

Analytic methods using RHIS data Among articles that conducted statistical analyses using RHIS data (n = 68), time series analyses to test or ac- count for trends were most commonly performed (25%), followed by geostatistical analyses (16%), pre-post com- parisons (15%), interrupted time series (ITS) (10%), and difference-in-difference analyses (7%). Other longitudinal analyses (13%), other cross-sectional analyses (12%), and scenario analysis on cost effectiveness (2%) were also conducted. Table 2 presents the range of methodologies identified across studies using RHIS data, as well as the corresponding articles.

Time series analysis Time series analysis using RHIS data was most often ap- plied to evaluate programs and identify disease epidemi- ology, with one study assessing the impact of an infectious disease outbreak on primary health service utilization [82]. Studies analyzed indicators using large quantities of monthly or yearly data to estimate change (range of time units: 5–168). For instance, two-thirds of the studies analyzed three or more years of monthly data. Many of the studies utilized the highly disaggre- gated nature of the data by using either facility or district level data, with the exception of two studies which mod- elled national trends [33, 116]. Studies commonly ap- plied strategies to account for temporal autocorrelation and the correlation between geographical units, includ- ing generalized linear models [58], multi-level analysis [77, 78], and ordinary least-squares regression with ad- justment for seasonality and lag [34, 37, 117]. Among studies that modelled multiple facilities or administrative regions, random effects were commonly applied to ac- count for heterogeneity. In addition to RHIS data, a number of included studies

incorporated data from external sources in their models based on geographical location such as district or region. Studies of malaria, for example, commonly included cli- mate data from satellites in their models to control for

Hung et al. BMC Health Services Research (2020) 20:790 Page 4 of 15

important temporal factors, for example precipitation, humidity, and temperature [73, 117]. Other studies in- corporated information from other national community surveys, health facility surveys, and program data as co- variates [34, 77]. While most studies controlled for po- tential confounders by including covariates in analytic models, one study on maternal health service applied propensity score matching to further remove biases from differences in covariate distribution [37].

Geostatistical analysis Geostatistical analyses using RHIS data were predomin- antly conducted for epidemiological purposes and the monitoring and assessment of service provision by exploiting geospatial information included in the RHIS at the facility or district level. Three of the studies that applied geostatistical analysis were cross-sectional, while the remainder were spatial-temporal. About half of the studies focused on malaria, of which three compared and illustrated various kriging methods to provide a reli- able estimate of malaria burden amid missing reporting [105–107], and one study applied geostatistical modeling to select the most relevant health facility indicators for severe malaria outcomes [108]. Studies on other topics investigated the spatial or spatial-temporal dynamics of malaria in pregnancy [100], childhood diarrhea [101], clustering of malaria and HIV [102], and meningitis [118]. About half of the studies did not include data from external sources, and others triangulated data sourced from satellite data, Demographic and Health Surveys, national Malaria Indicator Surveys, and Service Delivery Indicator Surveys in their analyses. Studies that included covariates in the geostatistical analysis applied Bayesian hierarchical Poisson models or Bayesian geosta- tistical negative binomial models [103, 108, 110].

Pre-post comparison analysis Pre-post comparison was commonly applied among studies that used RHIS data for program evaluation, and several studies used simple descriptive statistics to

Table 1 Characteristics of research studies that used RHIS data

n Percent

Geographical region

East Asia and Pacific 8 6.1

Latin America and the Caribbean 9 6.8

Middle East and North Africa 2 1.5

South Asia 15 11.4

Sub-Saharan Africa 98 74.2

Year of publication

< 2000 3 2.3

2000–2004 7 5.3

2005–2009 10 7.6

2010–2014 40 30.3

2015–2019 72 54.5

RHIS data as source or to inform study

Data source 128 97.0

Inform study 4 3.0

Types of study design

Ecological study - cross-sectional 13 9.8

Ecological study - longitudinal 51 38.6

Ecological study - descriptive 41 31.1

Case study 11 8.3

Mixed methods study 13 9.8

Cross-sectional study 1 0.8

Pre- and post-intervention study 1 0.8

Nested clustered randomized controlled trial 1 0.8

Data use purpose

Program evaluation 67 50.8

Epidemiology 23 17.4

Monitoring and assessment of service provisions 30 22.7

Program description 6 4.5

Impact evaluation 4 3.0

Cost estimation 2 1.5

Health conditions/service type

General (multiple aspects) 21 15.9

Secondary health utilization 2 1.5

General causes of death 1 0.8

Maternal and Child health/healthcare 12 9.1

Maternal health/healthcare 24 18.2

Child health/healthcare 11 8.3

Vaccine prevented childhood illnesses 10 7.6

Malaria 30 22.7

Malaria & HIV/AIDS 1 0.8

Malaria & other parasitic diseases 1 0.8

HIV and related diseases 8 6.1

Mental health/healthcare 3 2.3

Table 1 Characteristics of research studies that used RHIS data (Continued)

n Percent

Other diseases 5 3.8

Healthcare workforce and other resources 2 1.5

Data issue of RHIS: missingness

Described how missing data was managed 33 25.0

No description of how missing data was managed 99 75.0

Data issue of RHIS: outlier

Described how outlier was detected 19 14.4

No description of how outlier was detected 113 85.6

Hung et al. BMC Health Services Research (2020) 20:790 Page 5 of 15

compare the periods before and after interventions. As pre-post comparison is subject to the limitation of temporal confounders and secular trends, two of the studies included contextual factors in regression mod- elling [35, 119].

Interrupted time series analysis Most of the studies that conducted ITS analysis used it to evaluate interventions, and one assessed the impact of an infectious disease outbreak on maternal and child health service use [68]. The studies used large quantities of monthly data to model trend and level change (range of time unit: 44–132). RHIS data were minimally aggre- gated in these studies, which mostly analyzed facility or district level data, and similar to studies using time series analysis, accounted for autocorrelation through incorp- orating autoregressive structures or clustered standard errors in their modelling. As ITS analyses are generally unaffected by confound-

ing variables that do not change over time by design

[120], baseline characteristics were typically not included in these models. Nonetheless, ITS analyses can be af- fected by time-varying confounders that rapidly change and some models included contextual factors from other data sources, such as climate and program data. To strengthen the quasi-experimental design, two studies also included a contrast group of time series to control for contextual changes that occurred at the same time as the interventions [38, 45].

Difference-in-difference analysis Five studies applied difference-in-difference techniques using a wide range of time periods (range of time units: 4–48) and levels of geographical units (facility, district, provincial). Only one study included contextual charac- teristics from other data sources in its analysis. Analytic methods varied from descriptive comparison between and within intervention and control groups [41, 59, 87, 88], to ordinary least square regression with propensity score matching [42].

Fig. 2 Types of service and research purpose of RHIS data use (n = 132)

Hung et al. BMC Health Services Research (2020) 20:790 Page 6 of 15

Table 2 Types of analytic methods applied among studies that analyzed RHIS data

Data use purpose

Type of disease/service studied

Range of data (unit)

Level of aggregation

Analytic methods Other information sources included

Reference

Time series analysis

Epidemiology Child health, malaria, tooth extraction

15 (year) - 120 (month)

Ward, municipal, district

Time series correlograms; ordinary least-squares regres- sions adjusted for seasonality and lag; non-linear time series correlation and regressions

GPS coordinates, Climate Hazards Group Infrared Precipitation with Station Data, satellite data, meteorological department data, program data

[73–76]

Program evaluation

General, maternal and child health, maternal health, vaccine prevented childhood illnesses, malaria

5 (year) - 168 (month)

Facility, district, region, nation

Ordinary least squares regression; negative binomial generalized linear model; random effects negative binomial regressions; switching regression methods weighted by propensity scores

Program data, program reports, data from Bureau of Statistics and Ministry of Health, Malaria Indicator Survey, Demographic Health Survey, Health Facility Survey, community survey, satellite data, sentinel site case- investigations/surveillance, ab- straction from hospital registries

[33, 34, 37, 40, 54, 55, 58, 77– 81]

Impact evaluation (non- program)

General 84 months (month)

Facility Linear mixed-effect time-series analysis with a segmented re- gression parameterization

None [82]

Interrupted time series analysis

Program evaluation

General, maternal and child health, maternal health, malaria

53 (month) - 132 (month)

Facility, intervention vs. control groups, district

Generalized least square model with autoregressive structure; generalized least square model with controls, with autoregressive process and moving average process; segmented linear regression

Meteorology Department data, program data, facility survey

[38, 45, 83–86]

Impact evaluation (non- program)

Maternal and child health 44 (month)

District Segmented linear regression with district fixed effect and clustered standard error at district level

Demographic Health Survey [68]

Difference-in-difference analysis

Program evaluation

General, child health, maternal health

4 (year) - 48 (month)

Facility, district, province

Ordinary least squares regression with and without propensity score matching; Wilcoxon rank-sum test on me- dian difference-in-differences between facilities; descriptive comparison of means

Verified data from Performance-Based Financing system

[41, 42, 59, 87, 88]

Pre-post comparison analysis

Program evaluation

Child health, maternal health, maternal and child health, vaccine prevented childhood illnesses, malaria, HIV or related diseases

2 (year) - 48 (month)

Facility, district

Chi-square test; Pearson correlation; Wilcoxon signed- rank test; paired sample t-test; linear regressions; Poisson re- gression; negative binomial re- gression; logistic regression

Bureau of Statistics data, program reports, Meteorological Department data, entomological sentinel surveys, Demographic and Health Survey, UN Interagency Group for Childhood Mortality Estimation(CME Info) database, abstraction from facility registers, community surveys, vital registry, provincial maternal death notification register

[35, 39, 48, 57, 89–93]

Impact evaluation (non- program)

Child health 26 (month)

District Pearson chi-square test District hospital registers, Safe and dignified burials for all deaths database

[67]

Other longitudinal analysis

Hung et al. BMC Health Services Research (2020) 20:790 Page 7 of 15

Impact of research using RHIS data Most of the studies that conducted statistical analyses using RHIS data were published in journals with impact factors (88%, Fig. 3), two-thirds of which were two or higher, and more than a fifth of which were greater than three. Among those studies published in journals with

the highest impact factors, most of them focused on program evaluation (53%), followed by monitoring and assessment of service provision (20%), epidemi- ology (20%) and impact evaluation (7%). These studies encompassed a range of health topics commonly stud- ied using RHIS data.

Table 2 Types of analytic methods applied among studies that analyzed RHIS data (Continued)

Data use purpose

Type of disease/service studied

Range of data (unit)

Level of aggregation

Analytic methods Other information sources included

Reference

Epidemiology Maternal health, malaria 12 (year)

- 16 (year)

District Chi-square test; negative binomial regression

Review of hospital death records

[94, 95]

Monitoring and assessment of service provision

HIV or related diseases 3 (year) District Descriptive comparison over time

Surveys with health facility managers

[96]

Program evaluation

Genera, child health, malaria, malaria and other parasitic diseases

3 (year) - 24 (month)

Facility, district, nation

Poisson regression to explore association between intervention coverage and disease burden; Mann–Whitney U Test to compare prevalence in intervention and non- intervention area; linear regres- sion model; student t-test

Sentinel surveillance data, program reports, national facility and community survey, Bureau of Statistics data, program data

[47, 52, 66, 97– 99]

Geostatistical analysis

Epidemiology Child health, malaria, malaria and HIV/AIDS, meningococcal meningitis

1 (year) - 520 (week)

District Cluster analysis; cross- correlations of different spatial scales between time series of cases; Bayesian hierarchical Poisson model and smoothed model estimates plotted on dis- trict maps

Malaria Indicator Survey, Demographic Health Survey, program data

[100–104]

Monitoring and assessment of service provision

Malaria, maternal health 1 (year) - 57 (month)

Facility, district

Kriging (ordinary kriging, space- time ordinary kriging, local space-time ordinary kriging); Bayesian geostatistical negative binomial model

Service Delivery Indicator Survey

[105–109]

Program evaluation

Malaria 36 (month)

District Bayesian geostatistical models and Bayesian generalized linear models

Malaria Indicator Survey, malaria control program data, satellite data, Demographic Health Survey, ACTWatch household surveys

[110]

Other cross-sectional analysis

Epidemiology Maternal health Median

of 24 months

Province Linear regression model None [111]

Monitoring and assessment of service provision

General, child health, maternal health, mental health

1 (year) Facility, district, municipality, state

Descriptive statistics, Tobit regression model, bivariate and multivariate linear regression models,

Nutrition Service Delivery Assessment, abstraction from Integrated Nutrition Register, structured questionnaire with district health officers, District- level household and facility surveys, National Register of Health Service Providers, data from Institute of Geography and Statistics

[112–115]

Program evaluation

HIV and related diseases 1 (year) District Mixed-methods Register reviews and a series of patient folder (health record) reviews

[51]

Hung et al. BMC Health Services Research (2020) 20:790 Page 8 of 15

Strategies to circumvent RHIS data quality issues Data quality is commonly cited as a barrier to using RHIS data in research, and slightly more than a quarter of the included studies described the strategies that they used to handle missing data and/or identify extreme values (Table 3). These strategies consisted of exclusion, imputation, interpolation, verification, and accounting for missing data in modeling. Exclusion of missing data was the most common practice, and among studies that used this technique, they excluded facilities from the analytic samples [38, 41, 45, 52, 65, 79, 83, 84, 87, 94, 96, 121], restricted the study period based on explicit cri- teria [54, 122], or applied sensitivity analysis to compare various exclusion criteria [41, 89, 90]. Imputation methods varied from assigning specific values to the missing observation [42, 87, 118, 123–125], to various modeling strategies such as conditional autoregressive model [110], generalized linear regression [124], and it- erative singular value decomposition [124]. A sensitivity analysis was also conducted to select a specific imput- ation strategy [124]. Interpolation involves predicting values at unsampled locations. Methods described in- cluded the use of space-time kriging [105–107], and the adjustment of results by calibrating with other relevant information [52, 53, 55]. Some studies assumed data were missing at random, which was accounted for in specific modeling methods such as mixed-effect models [65, 124]. When the source of data could be reached, some studies also described verifying the missing infor- mation using registries where the original data were re- corded [39, 73, 97, 111, 122]. Slightly fewer articles described methods to identify

and handle extreme values in the RHIS data, of which three types of strategies emerged: setting specific

thresholds, visual inspection, and analytic assessment. Thresholds were set based on the distribution of the data, such as proportions or standard deviations from univariate regression. Several studies used visual inspec- tion of outliers [38, 107], while the use of jackknifing analysis and the identification of influential points through Cook’s distance statistics were also applied [112, 126]. Upon identification of extreme values, several strategies were utilized: exclusion, replacement with the average value, replacement with the missing value, verifi- cation with a data source, or discounting the observation in statistical estimation. However, studies that replaced the extreme value with an explicit value potentially in- troduced bias into their estimates. A few studies also de- scribed the strategies applied to assess the reliability of the RHIS data, some of which were routine processes administered in the health systems [39, 97].

Discussion In recent years, there have been increased investments made to improve the quality of RHIS data in many LMICs. Over the same time period, we found an in- crease in published research using RHIS sourced data, especially over the past 5 years, likely due to the in- creased availability, accessibility, and quality of RHIS data [18]. While these studies have made contributions to the literature, we also found that the total number of studies conducted (n = 132) remains a small part of the overall literature base on health system evaluation and performance in LMICs. Malaria and maternal health conditions were the most

commonly studied health conditions, despite the fact that RHISs collect data on a wide range of other diseases and conditions. In particular, the use of RHIS data for non-communicable diseases (NCDs) research was very limited. As LMICs are undergoing an epidemiologic transition and the importance of NCDs is increasing [127], LMIC health systems face the increasing chal- lenges of addressing the dual burden of communicable and non-communicable diseases [128, 129]. In spite of the limited implementation of non-communicable disea- seinterventions [129], the few studies that used RHIS data for non-communicable disease research mainly ana- lyzed the gap in service provision and estimated disease burden, highlighting the large unmet need for health care in affected populations. A couple of the studies de- scribed how their research was limited by data availabil- ity and quality, such as the lack of diagnostic categories of the investigated health conditions in the RHIS. Future research should investigate how RHIS data on non- communicable diseases could better help to provide insights on its epidemiology and service provision to ad- dress these health conditions.

Fig. 3 Distribution of impact factor of journals that published research studies that conducted statistical analysis of RHIS data (n = 68)

Hung et al. BMC Health Services Research (2020) 20:790 Page 9 of 15

Our systematic review found that many of the studies took advantage of some of the features of RHIS data, in particular by exploiting the high frequency nature of these data at the level of health facilities, as well as com- bining external information to enhance estimations and enable assessing new research questions. The triangula- tion of populational health characteristics,

environmental factors, and service coverage strengthens the analysis and the understanding of their influence [130]. In addition, the overlay of different information in analyses of RHIS data allows for the advancement of re- search methods. For instance, a recent study demon- strated how to assess the effects of facility readiness on severe malaria outcomes through constructing a

Table 3 Strategies applied in research articles to counter issues of RHIS data

Type of strategy Description of strategy

Missing data

Exclusion Exclude facility data if a certain threshold was reached (e.g. more than two-thirds of months in a year; more than a sixth of baseline data; facilities with any missing data)

Restrict analysis to a period with a low level of missing data

Sensitivity analysis to compare analysis of restricted period and full period

Imputation Assign missing observations with mean-value for the year

Assign missing observations with the average of precedent and subsequent data

Imputation using conditional autoregressive model

Missing value was replaced as positive (binary form) to prevent exaggeration of the fade-out effect

Sensitivity analysis of imputation strategies: 1) single imputation using means, trimmed means, and median, 2) Poisson generalized linear modeling, 3) iterative singular value decomposition method

Interpolation Interpolation using space-time kriging

Adjust results by dividing each indicator by the percentage of reports submitted

Adjust the data by calibrating to the total population using proportion reported in a household survey to have occurred in health facilities

Verification Account in the modeling method

Manual verification of the missing data with register at the health facility

Missing data was assumed missing at random and accounted for in the mixed-effect models using standard maximum likelihood estimation

Identifying extreme values

Specific threshold Establishing a lower and upper limit based on proportion of the annual average or feasible value

Univariate regression on individual facility-level to identify deviation from the mean time trend (e.g. if exceed 8 standard deviations)

Visual Visual inspection of outliers

Analytic assessment Jackknifing analysis to assess influence

Student residual higher than an absolute value of 2 and influence on the estimated coefficients determined by high Cook’s distance statistics

Handling of extreme values

Exclusion Extreme values were excluded from analyses

Replacing extreme value with average Extreme values were assigned the average value of the year; with exceptions of low average values

Replacing extreme value with missing Outliers set to missing

Verification with data source Any drastic change in monthly data reported electronically were manually verified with register at the health facility. Discrepancies were replaced with data in the register

Discount observation in estimation Outliers were allocated a dummy coding to discount the observation in the calculation of coefficients

Assess reliability

Data validation process Randomly selected 10% of the total sample to check accuracy and reliability of data with reports and registers

Verify data with another source (e.g. payroll)

Established routine data validation process by health information and records officer (e.g. monthly data review meetings)

Hung et al. BMC Health Services Research (2020) 20:790 Page 10 of 15

composite facility readiness index based on health facil- ity characteristics and spatial data, and using RHIS data as the outcome variable [108]. The detailed routine nature of RHIS data and the ability to link with other geographically based information, including data on population, environmental, health behavior, and facility characteristics, can generate high impact research and advance our understanding of disease epidemiology and health improvement efforts in LMICs. Despite the increasing use of RHIS data for research

purposes, the quality of these data remains imperfect and such issues should be identified and addressed in order to limit estimation error and bias. RHIS data qual- ity issues remain a particular concern in some settings [131–133], however, other studies have shown that strat- egies that have been implemented to improve RHIS data across different international contexts can be successful [5, 134]. Multiple strategies were discussed in the articles we reviewed in our paper, including strategies to address common data quality issues such as missingness and data validity, for example the simple exclusion of miss- ing data and various imputation and interpolation methods. However, the majority of the studies that used RHIS data did not describe the extent of the quality is- sues or the steps they took to overcome them. The use of sensitivity analyses in assessing the effect of specific cut-offs or methods was scarce. Explicit descriptions of the extent of the data quality issues and the reasons for selecting a particular approach should be encouraged in future research. While our review used major databases and systematic

methods, it nevertheless has some limitations that are worth noting. First, we included only peer-reviewed studies that were published in English, and therefore may have overlooked potentially relevant studies published in the grey literature or written in other languages. Additionally, given our focus on original re- search, we did not search the broader body of literature for books, reports, or grey literature. Our literature search also identified phrases that described health infor- mation systems in title and abstracts only, possibly resulting in the exclusion of studies that only mentioned RHIS data use in the full text. Finally, additional variants on these search terms may have generated more articles or a slightly different set of articles.

Conclusions In this systematic review we summarized the use of data collected from RHISs in LMICs. Overall, we found that researchers are increasingly using data sourced from RHISs to conduct health system planning and evaluation studies in LMIC health systems, however these data likely remain underutilized by the broader research com- munity. As many of the studies included in this review

were published in prominent journals and were able to use strong quasi-experimental or geo-spatial methods, we believe this makes the case for greater use of these data for research purposes in the future, which will likely happen as RHIS data become more openly available to the research community. However, there is a need to help build the case to use these data for a broader range of health conditions and to develop more of a consensus on methods to deal with data imperfections, given that our findings underlined the limited use and comparison of these methods. That said, our review clearly demon- strates the feasibility of use RHIS data in conjunction with rigorous study designs and analytic methods in LMICs. We suggest that future program evaluations should consider their use more broadly, to assess an in- creased variety of health conditions in conjunction with, or as a replacement for, household or facility survey methods.

Supplementary information Supplementary information accompanies this paper at https://doi.org/10. 1186/s12913-020-05660-1.

Additional file 1.

Abbreviations DHIS 2: District health information system 2; HMIS: Health management information system; ITS: Interrupted time series; LMIC: Low- and middle- income country; NCD: Non-communicable disease; PRISMA: Preferred reporting items for systematic reviews and meta-analyses; RHIS: Routine health information system

Acknowledgements We acknowledge that this work was conducted on the Haldimand Tract, traditional territory of the Neutral, Anishinaabe and Haudenosaunee peoples.

Authors’ contributions YWH, KH, and KAG identified the search strategy. YWH and KH conducted the literature screening, and KAG was consulted when required. YWH, KH, and BRI extracted data from the included studies. YWH wrote the manuscript. KAG and MRL provided feedback and edits on the manuscript. The authors read and approved the final manuscript.

Funding This work was supported by AXA Research Fund. The foundation had no role in the design and implementation of the study, writing of the findings, or decision regarding any of these aspects of the study.

Availability of data and materials Not applicable.

Ethics approval and consent to participate Not applicable.

Consent for publication Not applicable.

Competing interests The authors declare that they have no competing interests.

Author details 1University of Waterloo, School of Public Health and Health Systems, Waterloo, Canada. 2Department of Health Sciences, Wilfrid Laurier University, Waterloo, Canada. 3Centre for Health Services and Policy Research, The

Hung et al. BMC Health Services Research (2020) 20:790 Page 11 of 15

University of British Columbia, Vancouver, Canada. 4School of Public Health, Hong Kong University, Pok Fu Lam, Hong Kong.

Received: 18 April 2020 Accepted: 16 August 2020

References 1. Hotchkiss DR, Diana ML, Foreit KGF. How can routine health information

systems improve health systems functioning in lowand middle-income countries? Assessing the evidence base. Adv Health Care Manag. 2012. https://doi.org/10.1108/S1474-8231(2012)0000012006.

2. AbouZahr C, Boerma T. Health information systems: the foundations of public health. Bull World Health Organ. 2005;83:578–83.

3. World Health Organization. Everybody’s business--strengthening health systems to improve health outcomes: WHO’s framework for action; 2007.

4. Lippeveld T, Sauerborn R, Bodart C, Organization WH. Design and implementation of health information systems; 2000.

5. Mutale W, Chintu N, Amoroso C, et al. Improving health information systems for decision making across five sub-Saharan African countries: implementation strategies from the African health initiative. BMC Health Serv Res. 2013;13(Suppl 2):1–12.

6. Hogan DR, Stevens GA, Hosseinpoor AR, Boerma T. Monitoring universal health coverage within the sustainable development goals: development and baseline data for an index of essential health services. Lancet Glob Health. 2018;6(2):e152–68. https://doi.org/10.1016/S2214-109X(17)30472-2.

7. Cutts FT, Claquin P, Danovaro-Holliday MC, Rhoda DA. Monitoring vaccination coverage: defining the role of surveys. Vaccine. 2016;34(35): 4103–9. https://doi.org/10.1016/j.vaccine.2016.06.053.

8. Wagenaar BH, Sherr K, Fernandes Q, Wagenaar AC. Using routine health information systems for well-designed health evaluations in low- and middle-income countries. Health Policy Plan. 2016;31(1):129–35. https://doi. org/10.1093/heapol/czv029.

9. World Health Organization. Monitoring the building blocks of health systems: a handbook of indicators and their measurement strategies. Geneva; 2010. http://www.who.int/healthinfo/systems/WHO_MBHSS_2010_ full_web.pdf?ua=1.

10. World Health Organization. Framework and standards for country health information systems (second edition). Geneva; 2012. http://www.who.int/ healthinfo/country_monitoring_evaluation/who-hmn-framework-standards- chi.pdf.

11. Warren AE, Wyss K, Shakarishvili G, Atun R, de Savigny D. Global health initiative investments and health systems strengthening: a content analysis of global fund investments. Glob Health. 2013;9(1):30. https://doi.org/10. 1186/1744-8603-9-30.

12. Gimbel S, Mwanza M, Nisingizwe MP, et al. Improving data quality across 3 sub-Saharan African countries using the consolidated framework for implementation research (CFIR): results from the African health initiative. BMC Health Serv Res. 2017. https://doi.org/10.1186/s12913-017-2660-y.

13. Kiberu VM, Matovu JKB, Makumbi F, Kyozira C, Mukooyo E, Wanyenze RK. Strengthening district-based health reporting through the district health management information software system: the Ugandan experience. BMC Med Inform Decis Mak. 2014;14:40. https://doi.org/10.1186/1472-6947-14-40.

14. Hazel E, Wilson E, Anifalaje A, Sawadogo-Lewis T, Heidkamp R. Building integrated data systems for health and nutrition program evaluations: lessons learned from a multi-country implementation of a DHIS 2-based system. J Glob Health. 2018;8(2):20307. https://doi.org/10.7189/jogh.08. 020307.

15. Dehnavieh R, Haghdoost A, Khosravi A, et al. The district health information system (DHIS2): a literature review and meta-synthesis of its strengths and operational challenges based on the experiences of 11 countries. Heal Inf Manag J. June 2018:1833358318777713. https://doi.org/10.1177/ 1833358318777713.

16. Githinji S, Oyando R, Malinga J, et al. Completeness of malaria indicator data reporting via the district health information software 2 in Kenya, 2011–2015. Malar J. 2017;16:344. https://doi.org/10.1186/s12936-017-1973-y.

17. MEASURE Evaluation. Lessons in health information system strengthening: what worked in the Democratic Republic of the Congo. Chapel Hill; 2019. https://www.measureevaluation.org/resources/publications/wp-19-229.

18. Ashton RA, Bennett A, Yukich J, Bhattarai A, Keating J, Eisele TP. Methodological considerations for use of routine health information system data to evaluate malaria program impact in an era of declining

malaria transmission. Am J Trop Med Hyg. 2017. https://doi.org/10.4269/ ajtmh.16-0734.

19. Rowe AK, Kachur SP, Yoon SS, Lynch M, Slutsker L, Steketee RW. Caution is required when using health facility-based data to evaluate the health impact of malaria control efforts in Africa. Malar J. 2009;8(1):209.

20. Hazel E, Requejo J, David J, Bryce J. Measuring coverage in MNCH: evaluation of community-based treatment of childhood illnesses through household surveys. PLoS Med. 2013. https://doi.org/10.1371/ journal.pmed.1001384.

21. Lim SS, Stein DB, Charrow A, Murray CJL. Tracking progress towards universal childhood immunisation and the impact of global initiatives: a systematic analysis of three-dose diphtheria, tetanus, and pertussis immunisation coverage. Lancet. 2008;372(9655):2031–46. https://doi.org/10. 1016/S0140-6736(08)61869-3.

22. Cutts FT, Izurieta HS, Rhoda DA. Measuring coverage in MNCH: design, implementation, and interpretation challenges associated with tracking vaccination coverage using household surveys. PLoS Med. 2013;10(5): e1001404. https://doi.org/10.1371/journal.pmed.1001404.

23. Sychareun V, Hansana V, Phengsavanh A, Chaleunvong K, Eunyoung K, Durham J. Data verification at health centers and district health offices in Xiengkhouang and Houaphanh provinces, Lao PDR. BMC Health Serv Res. 2014;14(1):255. https://doi.org/10.1186/1472-6963-14-255.

24. Gloyd S, Wagenaar BH, Woelk GB, Kalibala S. Opportunities and challenges in conducting secondary analysis of HIV programmes using data from routine health information systems and personal health information. J Int AIDS Soc. 2016. https://doi.org/10.7448/IAS.19.5.20847.

25. Aqil A, Lippeveld T, Hozumi D. PRISM framework: a paradigm shift for designing, strengthening and evaluating routine health information systems. Health Policy Plan. 2009;24(3):217–28.

26. Short Fabic M, Choi Y, Bird S. A systematic review of demographic and health surveys: data availability and utilization for research. Bull World Health Organ. 2012. https://doi.org/10.2471/BLT.11.095513.

27. Fernandes QF, Wagenaar BH, Anselmi L, Pfeiffer J, Gloyd S, Sherr K. Effects of health-system strengthening on under-5, infant, and neonatal mortality: 11- year provincial-level time-series analyses in Mozambique. Lancet Glob Health. 2014. https://doi.org/10.1016/S2214-109X(14)70276-1.

28. Ratnayake R, Ratto J, Hardy C, et al. The effects of an integrated community case management strategy on the appropriate treatment of children and child mortality in Kono District, Sierra Leone: a program evaluation. Am J Trop Med Hyg. 2017. https://doi.org/10.4269/ajtmh.17-0040.

29. Bishai DM. Global initiatives in universal childhood immunisation. Lancet. 2008;372(9655):2004–5. https://doi.org/10.1016/S0140-6736(08)61853-X.

30. Seitio-Kgokgwe O, Mashalla Y, Seloilwe E, et al. Developing block wise composite health index in Yavatmal district, Maharashtra state, India: an analysis of available data. Malar J. 2017;16(1 PG):35. https://doi.org/10.4103/ 2008-7802.160056.

31. Pagnoni F, Convelbo N, Tiendrebeogo J, Cousens S, Esposito F. A community-based programme to provide prompt and adequate treatment of presumptive malaria in children. Trans R Soc Trop Med Hyg NS. 1997;91(5 PG-512-7):512–7.

32. Rehman A, Mann A, Schwabe C, et al. Five years of malaria control in the continental region, Equatorial Guinea. Malar J. 2013;12(PG-154):154. https:// doi.org/10.1186/1475-2875-12-154.

33. Nyarango P, Gebremeskel T, Mebrahtu G, et al. A steep decline of malaria morbidity and mortality trends in Eritrea between 2000 and 2004: the effect of combination of control methods. Malar J. 2006;5(PG-33):33. https://doi. org/10.1186/1475-2875-5-33.

34. Over M, Bakote’e B, Velayudhan R, Wilikai P, Graves PM. Impregnated nets cannot fully substitute for DDT: the Field effectiveness of alternative methods of malaria prevention in Solomon Islands, 1993–99. World Bank, Policy Res Work Pap Ser 3044, 2003. 2003;(PG) https://libproxy.wlu.ca/login?url=https:// search.proquest.com/docview/56217804?accountid=15090NS-.

35. Okullo AE, Matovu JKB, Ario AR, et al. Malaria incidence among children less than 5 years during and after cessation of indoor residual spraying in northern Uganda. Malar J. 2017;16(1 PG-319):319. https://doi.org/10.1186/ s12936-017-1966-x.

36. Sande S, Zimba M, Mberikunashe J, Tangwena A, Chimusoro A. Progress towards malaria elimination in Zimbabwe with special reference to the period 2003-2015. Malar J. 2017. https://doi.org/10.1186/s12936-017-1939-0.

37. Manthalu G. User fee exemption and maternal health care utilisation at mission health facilities in Malawi: an application of disequilibrium

Hung et al. BMC Health Services Research (2020) 20:790 Page 12 of 15

theory of demand and supply. Health Econ. 2019;(PG). https://doi.org/ 10.1002/hec.3856.

38. Nguyen H, Zombré D, Ridde V, De Allegri M. The impact of reducing and eliminating user fees on facility-based delivery: a controlled interrupted time series in Burkina Faso. Health Policy Plan. 2018;33(8 PG-948-956):948–56. https://doi.org/10.1093/heapol/czy077.

39. Njuguna J, Kamau N, Muruka C. Impact of free delivery policy on utilization of maternal health services in county referral hospitals in Kenya. BMC Health Serv Res. 2017. https://doi.org/10.1186/s12913-017-2376-z.

40. Ridde V, Richard F, Bicaba A, Queuille L, Conombo G. The national subsidy for deliveries and emergency obstetric care in Burkina Faso. Health Policy Plan. 2011. https://doi.org/10.1093/heapol/czr060.

41. Steenland M, Robyn P, Compaore P, et al. Performance-based financing to increase utilization of maternal health services: evidence from Burkina Faso. SSM - Popul Heal. 2017;3(PG-179-184):179–84. https://doi.org/10.1016/j. ssmph.2017.01.001.

42. Falisse JB, Ndayishimiye J, Kamenyero V, Bossuyt M. Performance-based financing in the context of selective free health-care: an evaluation of its effects on the use of primary health-care services in Burundi using routine data. Health Policy Plan. 2015. https://doi.org/10.1093/heapol/ czu132.

43. Witter S, Zulfiqur T, Javeed S, Khan A, Bari A. Paying health workers for performance in Battagram district, Pakistan. Hum Resour Health. 2011. https://doi.org/10.1186/1478-4491-9-23.

44. Ir P, Horemans D, Souk N, Van Damme W. Using targeted vouchers and health equity funds to improve access to skilled birth attendants for poor women: a case study in three rural health districts in Cambodia. BMC Pregnancy Childbirth. 2010. https://doi.org/10.1186/1471-2393-10-1.

45. Iyer HS, Hirschhorn LR, Nisingizwe MP, et al. Impact of a district-wide health center strengthening intervention on healthcare utilization in rural Rwanda: use of interrupted time series analysis. PLoS One. 2017;12(8 PG-e0182418): e0182418. https://doi.org/10.1371/journal.pone.0182418.

46. de Lima RT, Fernandes TG, da Balieiro AA, et al. Primary health Care in Brazil and the Mais Médicos (more doctors) program: an analysis of production indicators. Cien Saude Colet. 2016. https://doi.org/10.1590/1413- 81232015219.15412016.

47. Mikkelsen-Lopez I, Cowley P, Kasale H, Mbuya C, Reid G, de Savigny D. Essential medicines in Tanzania: does the new delivery system improve supply and accountability? Heal Syst. 2014. https://doi.org/10.1057/hs.2013. 14.

48. Mphatswe W, Mate K, Bennett B, et al. Improving public health information: a data quality intervention in KwaZulu-Natal, South Africa. Bull World Health Organ. 2012. https://doi.org/10.2471/blt.11.092759.

49. Newbrander W, Ickx P, Feroz F, Stanekzai H. Afghanistan’s basic package of health services: its development and effects on rebuilding the health system. Glob Public Health. 2014;9(SUPPL.1 PG-S6-S28):S6-S28. https://doi. org/10.1080/17441692.2014.916735.

50. Santos LMP, Oliveira A, Trindade JS, et al. Implementation research: towards universal health coverage with more doctors in Brazil. Bull World Health Organ. 2017. https://doi.org/10.2471/BLT.16.178236.

51. Scott VE, Sanders D. Evaluation of how integrated HIV and TB programs are implemented in South Africa and the implications for rural-urban equity. Rural Remote Health. 2013;13(2):2165.

52. Vindigni SM, Riley PL, Kimani F, et al. Kenya’s emergency-hire nursing programme: a pilot evaluation of health service delivery in two districts. Hum Resour Health. 2014;12(1 PG). https://doi.org/10.1186/1478-4491-12-16.

53. Field E, Usurup J, Nathan S, Rosewell A. Contextual factors and health service performance from the perspective of the provincial health administrators in Papua New Guinea. Rural Remote Health. 2018. https://doi. org/10.22605/RRH4484.

54. Inchauste L, Patzi M, Halvorsen K, Solano S, Montesano R, Iñiguez V. Impact of rotavirus vaccination on child mortality, morbidity, and rotavirus-related hospitalizations in Bolivia. Int J Infect Dis. 2017. https://doi.org/10.1016/j.ijid. 2017.06.006.

55. Nanyunja M, Lewis RF, Makumbi I, et al. Impact of mass measles campaigns among children less than 5 years old in Uganda. J Infect Dis. 2003. https:// doi.org/10.1086/368026.

56. Tafesse B, Tekle E, Wondwossen L, et al. Effects of polio eradication activities on routine immunization: lessons from the 2013 outbreak response in Somali region of Ethiopia. Pan Afr Med J. 2017;27(Suppl 2 PG-4):4. https:// doi.org/10.11604/pamj.supp.2017.27.2.10569.

57. Mochache V, Irungu E, El-Busaidy H, Temmerman M, Gichangi P. “Our voices matter”: a before-after assessment of the effect of a community- participatory intervention to promote uptake of maternal and child health services in Kwale, Kenya 11 medical and health sciences 1117 public health and health services. BMC Health Serv Res. 2018;18(1 PG). https://doi.org/10. 1186/s12913-018-3739-9.

58. Kitui J, Dutton V, Bester D, Ndirangu R, Wangai S, Ngugi S. Traditional birth attendant reorientation and Motherpacks incentive’s effect on health facility delivery uptake in Narok County, Kenya: an impact analysis. BMC Pregnancy Childbirth. 2017;17(1 PG-125):125. https://doi.org/10.1186/s12884-017-1307-7.

59. Paudel D, Shrestha I, Siebeck M, Rehfuess E. Impact of the community- based newborn care package in Nepal: a quasi-experimental evaluation. BMJ Open. 2017;7(10 PG-e015285):e015285. https://doi.org/10.1136/ bmjopen-2016-015285.

60. Khurmi M, Sayinzoga F, Berhe A, et al. Newborn survival case study in Rwanda - bottleneck analysis and projections in key maternal and child mortality rates using lives saved tool (LiST). Int J MCH AIDS. 2017;6(2 PG-93- 108):93–108. https://doi.org/10.21106/ijma.214.

61. Lindgren TG, Deutsch K, Schell E, et al. Using mobile clinics to deliver HIV testing and other basic health services in rural Malawi. Rural Remote Health. 2011;11(2): 1682.

62. Mapira P, Morgan C. The contribution of church health services to maternal health care provision in Papua New Guinea. P N G Med J. 2011;54(3-4):139–146.

63. Robertson LJ, Szabo CP. Community mental health services in southern Gauteng: an audit using Gauteng district health information systems data. South African J Psychiatry. 2017;23(1 PG-a1055):a1055. https://doi.org/10. 4102/sajpsychiatry.v23i0.1055.

64. Yellapurkar MV. Health management information system in leprosy control programme. Indian J Lepr. 1984;56(1 PG-86-97):86–97 https://www.scopus. com/inward/record.uri?eid=2-s2.0-0021634802&partnerID=40&md5=9c9d755 790bfa4154b2372c12bdeeb37NS-.

65. Wagenaar BH, Augusto O, Beste J, et al. The 2014–2015 Ebola virus disease outbreak and primary healthcare delivery in Liberia: time-series analyses for 2010–2016. PLoS Med. 2018;15(2):e1002508. https://doi.org/10.1371/journal. pmed.1002508.

66. Quaglio G, Pizzol D, Bome D, et al. Maintaining maternal and child health services during the Ebola outbreak: experience from Pujehun, Sierra Leone. PLoS Curr. 2016;8. https://doi.org/10.1371/CURRENTS.OUTBREAKS. D67AEA257F572201F835772D7F188BA5.

67. Sesay T, Denisiuk O, Shringarpure KK, et al. Paediatric care in relation to the 2014–2015 Ebola outbreak and general reporting of deaths in Sierra Leone. Public Heal Action. 2017. https://doi.org/10.5588/pha.16.0088.

68. Sochas L, Channon AA, Nam S. Counting indirect crisis-related deaths in the context of a low-resilience health system: the case of maternal and neonatal health during the Ebola epidemic in Sierra Leone. Health Policy Plan. 2017;32(suppl_3):iii32–9. https://doi.org/10.1093/heapol/czx108.

69. Cassim N, Coetzee LM, Schnippel K, Glencross DK. Estimating the cost-per- result of a national reflexed Cryptococcal antigenaemia screening program: forecasting the impact of potential HIV guideline changes and treatment goals. PLoS One. 2017;12(8 PG-e0182154):e0182154. https://doi.org/10.1371/ journal.pone.0182154.

70. Novaes H, Itria A, Silva G, Sartori A, Rama C, Soárez P. Annual national direct and indirect cost estimates of the prevention and treatment of cervical cancer in Brazil. Clinics (Sao Paulo). 2015;70(4 PG-289-95):289–95. https://doi. org/10.6061/clinics/2015(04)12.

71. Namaganda G, Oketcho V, Maniple E, Viadro C. Making the transition to workload-based staffing: using the workload indicators of staffing need method in Uganda. Hum Resour Health. 2015;13(PG-89):89. https://doi.org/ 10.1186/s12960-015-0066-7.

72. Kusemererwa D, Alban A, Obua OT, Trap B. An exploratory study on equity in funding allocation for essential medicines and health supplies in Uganda’s public sector. BMC Health Serv Res. 2016;16:453((Obua) Ministry of Health, Pharmacy Division, Lourdel Road, Wandegeya, Kampala, Uganda PG- 453). https://doi.org/10.1186/s12913-016-1698-6.

73. Hajison PL, Mwakikunga BW, Mathanga DP, Feresu SA. Seasonal variation of malaria cases in children aged less than 5 years old following weather change in Zomba district, Malawi. Malar J. 2017. https://doi.org/10.1186/ s12936-017-1913-x.

74. Wright J, Vazé P, Russell G, et al. Seasonal aspects of weight-for-age in young children in Zimbabwe. Public Health Nutr. 2001. https://doi.org/10. 1079/phn2000100.

Hung et al. BMC Health Services Research (2020) 20:790 Page 13 of 15

75. Gunda R, Chimbari M, Shamu S, Sartorius B, Mukaratirwa S. Malaria incidence trends and their association with climatic variables in rural Gwanda, Zimbabwe, 2005–2015. Malar J. 2017;16(1 PG-393):393. https://doi. org/10.1186/s12936-017-2036-0.

76. De Melo Cunha MAG, Lino PA, Dos Santos TR, Vasconcelos M, Lucas SD, De Abreu MHNG. A 15-year time-series study of tooth extraction in Brazil. Med (United States). 2015;94(47 PG-e1924):e1924. https://doi.org/10.1097/MD. 0000000000001924.

77. Luboga SA, Stover B, Lim T, et al. Did PEPFAR investments result in health system strengthening? A retrospective longitudinal study measuring non- HIV health service utilization at the district level. Health Policy Plan. 2016; 31(7 PG-897-909):897–909. https://doi.org/10.1093/heapol/czw009.

78. Wagenaar B, Gimbel S, Hoek R, et al. Effects of a health information system data quality intervention on concordance in Mozambique: time-series analyses from 2009–2012. Popul Health Metr. 2015;13(PG-9):9. https://doi. org/10.1186/s12963-015-0043-3.

79. Sibomana H, Rugambwa C, Mwenda JM, et al. Impact of routine rotavirus vaccination on all-cause and rotavirus hospitalizations during the first four years following vaccine introduction in Rwanda. Vaccine. 2018. https://doi. org/10.1016/j.vaccine.2018.01.072.

80. Larsen DA, Winters A, Cheelo S, et al. Shifting the burden or expanding access to care? Assessing malaria trends following scale-up of community health worker malaria case management and reactive case detection. Malar J. 2017;16(1 PG-441):441. https://doi.org/10.1186/s12936-017-2088-1.

81. Ngabo F, Tate J, Gatera M, et al. Effect of pentavalent rotavirus vaccine introduction on hospital admissions for diarrhoea and rotavirus in children in Rwanda: a time-series analysis. Lancet Glob Health. 2016;4(2 PG-e129–36): e129–36. https://doi.org/10.1016/S2214-109X(15)00270-3.

82. Wagenaar BH, Augusto O, Beste J, et al. The 2014–2015 Ebola virus disease outbreak and primary healthcare delivery in Liberia: time-series analyses for 2010–2016. PLoS Med. 2018;15(2 PG-e1002508):e1002508. https://doi.org/10. 1371/journal.pmed.1002508.

83. Ruton H, Musabyimana A, Gaju E, et al. The impact of an mHealth monitoring system on health care utilization by mothers and children: an evaluation using routine health information in Rwanda. Health Policy Plan. 2018. https://doi.org/10.1093/heapol/czy066.

84. Gebrehiwot TG, Sebastian MS, Edin K, Goicolea I. The health extension program and its association with change in utilization of selected maternal health services in Tigray region, Ethiopia: a segmented linear regression analysis. PLoS One. 2015. https://doi.org/10.1371/journal.pone.0131195.

85. Ir P, Korachais C, Chheng K, Horemans D, Van Damme W, Meessen B. Boosting facility deliveries with results-based financing: a mixed-methods evaluation of the government midwifery incentive scheme in Cambodia. BMC Pregnancy Childbirth. 2015. https://doi.org/10.1186/s12884-015-0589-x.

86. Landoh E, Tchamdja P, Saka B, et al. Morbidity and mortality due to malaria in Est mono district, Togo, from 2005 to 2010: a times series analysis. Malar J. 2012;11(PG-389):389. https://doi.org/10.1186/1475-2875-11-389.

87. Assegaai T, Reagon G, Schneider H. Evaluating the effect of ward-based outreach teams on primary healthcare performance in north West Province, South Africa: a plausibility design using routine data. S Afr Med J. 2018; 108(4 PG-329-335):329–35. https://doi.org/10.7196/SAMJ.2017.v108i4.12755.

88. Anwari Z, Shukla M, Maseed B, et al. Implementing people-centred health systems governance in 3 provinces and 11 districts of Afghanistan: a case study. Confl Health. 2015;9(PG-2):2. https://doi.org/10.1186/1752-1505-9-2.

89. Mugeni C, Levine AC, Munyaneza RM, et al. Nationwide implementation of integrated community case management of childhood illness in Rwanda. Glob Heal Sci Pract. 2014. https://doi.org/10.9745/ghsp-d-14-00080.

90. phane VS, Jassat W, Bertram MY, et al. Supplementary immunization activities (SIAs) in South Africa: comprehensive economic evaluation of an integrated child health delivery platform. Glob Health Action. 2013. https:// doi.org/10.3402/gha.v6i0.20056.

91. Björkman A, Shakely D, Ali AS, et al. From high to low malaria transmission in Zanzibar - challenges and opportunities to achieve elimination. BMC Med. 2019. https://doi.org/10.1186/s12916-018-1243-z.

92. Oguttu DW, Matovu JKB, Okumu DC, et al. Rapid reduction of malaria following introduction of vector control interventions in Tororo District, Uganda: a descriptive study. Malar J. 2017. https://doi.org/10.1186/s12936- 017-1871-3.

93. Schoon MG. Impact of inter-facility transport on maternal mortality in the free state province. South African Med J. 2013;103(8 PG-534-537):534–7. https://doi.org/10.7196/SAMJ.6828.

94. Buchmann EJ, Mnyani CN, Frank KA, Chersich MF, McIntyre JA. Declining maternal mortality in the face of persistently high HIV prevalence in a middle-income country. BJOG An Int J Obstet Gynaecol. 2015. https://doi. org/10.1111/1471-0528.13064.

95. Kakchapati S, Ardkaew J. Modeling of malaria incidence in Nepal. J Res Health Sci. 2011;1(1 PG-7-13):7–13 https://www.scopus.com/inward/record. uri?eid=2-s2.0-79959717379&partnerID=40&md5=259481ca13eda8f1ef5 9d22356034ffd NS.

96. Walsh A, Ndubani P, Simbaya J, Dicker P, Brugha R. Task sharing in Zambia: HIV service scale-up compounds the human resource crisis. BMC Health Serv Res. 2010. https://doi.org/10.1186/1472-6963-10-272.

97. Njuguna J. Effect of eliminating open defecation on diarrhoeal morbidity: an ecological study of Nyando and Nambale sub-counties, Kenya. BMC Public Health. 2016. https://doi.org/10.1186/s12889-016-3421-2.

98. Mufunda J, Nyarango P, Usman A, et al. Roll back malaria - an African success story in Eritrea. South African Med J. 2007;97(1):46–50.

99. Kamuliwo M, Chanda E, Haque U, et al. The changing burden of malaria and association with vector control interventions in Zambia using district- level surveillance data, 2006-2011. Malar J. 2013. https://doi.org/10.1186/ 1475-2875-12-437.

100. Kamuliwo M, Kirk K, Chanda E, et al. Spatial patterns and determinants of malaria infection during pregnancy in Zambia. Trans R Soc Trop Med Hyg. 2015;109(8 PG-514-21):514–21. https://doi.org/10.1093/trstmh/trv049.

101. Beyene H, Deressa W, Kumie A, Grace D. Spatial, temporal, and spatiotemporal analysis of under-five diarrhea in southern Ethiopia. Trop Med Health. 2018;46(PG-18):18. https://doi.org/10.1186/s41182-018-0101-1.

102. Gwitira I, Murwira A, Mberikunashe J, Masocha M. Spatial overlaps in the distribution of HIV/AIDS and malaria in Zimbabwe. BMC Infect Dis. 2018;18(1 PG-598):598. https://doi.org/10.1186/s12879-018-3513-y.

103. Sartorius B, Cohen C, Chirwa T, Ntshoe G, Puren A, Hofman K. Identifying high-risk areas for sporadic measles outbreaks: lessons from South Africa. Bull World Health Organ. 2013. https://doi.org/10.2471/BLT.12.110726.

104. Philippon S, Broutin H, Constantin de Magny G, et al. Meningococcal meningitis in Mali: a long-term study of persistence and spread. Int J Infect Dis. 2009. https://doi.org/10.1016/j.ijid.2008.05.1223.

105. Gething PW, Noor AM, Goodman CA, et al. Information for decision making from imperfect national data: tracking major changes in health care use in Kenya using geostatistics. BMC Med. 2007. https://doi.org/10.1186/1741-7015-5-37.

106. Gething PW, Noor AM, Gikandi PW, et al. Improving imperfect data from health management information systems in Africa using space-time geostatistics. PLoS Med. 2006;3(6 PG-0825-0831):825–31. https://doi.org/10. 1371/journal.pmed.0030271.

107. Gething P, Atkinson P, Noor A, Gikandi P, Hay S, Nixon M. A local space- time kriging approach applied to a national outpatient malaria dataset. Comput Geosci. 2007;33(10 PG-1337–1350):1337–50. https://doi.org/10.1016/ j.cageo.2007.05.006.

108. Ssempiira J, Kasirye I, Kissa J, et al. Measuring health facility readiness and its effects on severe malaria outcomes in Uganda. Sci Rep. 2018;8(1 PG-17928): 17928. https://doi.org/10.1038/s41598-018-36249-8.

109. Bosomprah S, Dotse-Gborgbortsi W, Aboagye P, Matthews Z. Use of a spatial scan statistic to identify clusters of births occurring outside Ghanaian health facilities for targeted intervention. Int J Gynecol Obstet. 2016. https:// doi.org/10.1016/j.ijgo.2016.04.016.

110. Bennett A, Yukich J, Miller JM, et al. A methodological framework for the improved use of routine health system data to evaluate national malaria control programs: evidence from Zambia. Popul Health Metr. 2014. https:// doi.org/10.1186/s12963-014-0030-0.

111. Pires G, Rosa M, Zangarote M, Chicumbe S. Determinants of stillbirths occurred in health facilities in Zambezia province, Mozambique (2013– 2014). Rev Bras Saude Matern Infant. 2016;16(4 PG-415-420):415–20. https:// doi.org/10.1590/1806-93042016000400004.

112. Sebastian MS, Lemma H. Efficiency of the health extension programme in Tigray, Ethiopia: a data envelopment analysis. BMC Int Health Hum Rights. 2010. https://doi.org/10.1186/1472-698X-10-16.

113. Himanshu M, Källestål C. Regional inequity in complete antenatal services and public emergency obstetric care is associated with greater burden of maternal deaths: analysis from consecutive district level facility survey of Karnataka, India. Int J Equity Health. 2017. https://doi.org/10.1186/s12939- 017-0573-3.

114. Souza ML. Registro de distúrbios mentais no Sistema de Informação da Atenção Básica do Brasil, 2014. Epidemiol e Serv saude Rev do Sist Unico

Hung et al. BMC Health Services Research (2020) 20:790 Page 14 of 15

Saude do Bras. 2016;25(2 PG-405-410):405–10. https://doi.org/10.5123/ S1679-49742016000200018.

115. Wanzira H, Muyinda R, Lochoro P, et al. Quality of care for children with acute malnutrition at health center level in Uganda: a cross sectional study in West Nile region during the refugee crisis. BMC Health Serv Res. 2018. https://doi.org/10.1186/s12913-018-3366-5.

116. Nanyunja M, Lewis RF, Makumbi I, et al. Impact of mass measles campaigns among children less than 5 years old in Uganda. J Infect Dis. 2003; 187(SUPPL. 1 PG-S63-S68):S63–8. https://doi.org/10.1086/368026.

117. Gunda R, Chimbari M, Mukaratirwa S. Assessment of burden of malaria in Gwanda District, Zimbabwe, using the disability adjusted life years. Int J Environ Res Public Health. 2016;13(2 PG-244):244. https://doi.org/10.3390/ ijerph13020244.

118. Philippon S, Broutin H, Constantin de Magny G, et al. Meningococcal meningitis in Mali: a long-term study of persistence and spread. Int J Infect Dis. 2009;13(1 PG-103-9):103–9. https://doi.org/10.1016/j.ijid.2008.05.1223.

119. Njuguna J. Effect of eliminating open defecation on diarrhoeal morbidity: an ecological study of Nyando and Nambale sub-counties, Kenya. BMC Public Health. 2016;16(1 PG). https://doi.org/10.1186/s12889-016-3421-2.

120. Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: a tutorial. Int J Epidemiol. 2017. https://doi.org/10.1093/ije/dyw098.

121. Heymann DL, Chen L, Takemi K, et al. Global health security: the wider lessons from the west African Ebola virus disease epidemic. Lancet. 2015. https://doi.org/10.1016/S0140-6736(15)60858-3.

122. Luboga SA, Stover B, Lim TW, et al. Did PEPFAR investments result in health system strengthening? A retrospective longitudinal study measuring non- HIV health service utilization at the district level. Health Policy Plan. 2016. https://doi.org/10.1093/heapol/czw009.

123. Amouzou A, Kachaka W, Banda B, Chimzimu M, Hill K, Bryce J. Monitoring child survival in “real time” using routine health facility records: results from Malawi. Trop Med Int Health. 2013. https://doi.org/10.1111/tmi.12167.

124. Inguane CA, Gloyd S, Manuel JL, et al. Assessment of linkages from HIV testing to enrolment and retention in HIV care in Central Mozambique. J Int AIDS Soc. 2016. https://doi.org/10.7448/IAS.19.5.20846.

125. Wagenaar BH, Gimbel S, Hoek R, et al. Effects of a health information system data quality intervention on concordance in Mozambique: time-series analyses from 2009-2012. Popul Health Metr. 2015. https://doi.org/10.1186/ s12963-015-0043-3.

126. Jat TR, Sebastian MS. Technical efficiency of public district hospitals in Madhya Pradesh, India: a data envelopment analysis. Glob Health Action. 2013. https://doi.org/10.3402/gha.v6i0.21742.

127. Stanaway JD, Afshin A, Gakidou E, et al. Global, regional, and national comparative risk assessment of 84 behavioural, environmental and occupational, and metabolic risks or clusters of risks for 195 countries and territories, 1990-2017: a systematic analysis for the global burden of disease Stu. Lancet. 2018. https://doi.org/10.1016/S0140-6736(18)32225-6.

128. Remais JV, Zeng G, Li G, Tian L, Engelgau MM. Convergence of non- communicable and infectious diseases in low- and middle-income countries. Int J Epidemiol. 2013. https://doi.org/10.1093/ije/dys135.

129. Heller O, Somerville C, Suggs LS, et al. The process of prioritization of non- communicable diseases in the global health policy arena. Health Policy Plan. 2019. https://doi.org/10.1093/heapol/czz043.

130. Victora CG, Black RE, Boerma JT, Bryce J. Measuring impact in the millennium development goal era and beyond: a new approach to large-scale effectiveness evaluations. Lancet. 2011. https://doi.org/10.1016/S0140-6736(10)60810-0.

131. Endriyas M, Alano A, Mekonnen E, et al. Understanding performance data: health management information system data accuracy in southern nations nationalities and People’s region, Ethiopia. BMC Health Serv Res. 2019. https://doi.org/10.1186/s12913-019-3991-7.

132. Ward K, Mugenyi K, Benke A, et al. Enhancing workforce capacity to improve vaccination data quality, Uganda. Emerg Infect Dis. 2017. https://doi. org/10.3201/eid2313.170627.

133. Maïga A, Jiwani SS, Mutua MK, et al. Generating statistics from health facility data: the state of routine health information systems in eastern and southern Africa. BMJ Glob Health. 2019;4(5):e001849. https://doi.org/10.1136/ bmjgh-2019-001849.

134. Necochea E, Tripathi V, Kim YM, et al. Implementation of the standards- based management and recognition approach to quality improvement in maternal, newborn, and child health programs in low-resource countries. Int J Gynecol Obstet. 2015. https://doi.org/10.1016/j.ijgo.2015.04.003.

Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Hung et al. BMC Health Services Research (2020) 20:790 Page 15 of 15

BioMed Central publishes under the Creative Commons Attribution License (CCAL). Under the CCAL, authors retain copyright to the article but users are allowed to download, reprint, distribute and /or copy articles in BioMed Central journals, as long as the original work is properly cited.