Annotated Bibliography

A Framework for Assessing the Socio-Economic Impact of E-Governance Projects in Developing

Countries Sylvester Hatsu

University of South Africa/Accra Polytechnic P.O. Box 561 Accra-Ghana

+233 543937818 [email protected]

Ernest Ketcha Ngassam

University of South Africa P O Box 392, Pretoria, South Africa

+27823552519 [email protected]

Abstract— A study of more than 100 e-Governance

projects showed that impact assessment of rolled out e-

Governance projects remain insignificant. These findings

remain inconclusive notwithstanding the fact that outcomes

of public sector based ICT4D initiatives have not been fully

established and disseminated. This paper proposes a

framework for assessing the socio-economic impact of e-

governance projects in developing countries. Socio-economic

indicators for e-Governance programmes are identified and

grouped into both core and contextual indicators that form

the basis for the development of an evaluation model. The

proposed assessment framework centered on stakeholders’

participation is then subjected to expert evaluation.

Outcome of our evaluation revealed wide acceptance and

acknowledgement of the relevance and importance of the

framework not only by experts, but also through case-study

based validation tests.

Keywords—Framework, e-Governance, Socio-economic

Impact, developing countries, project lifecycle Critical Success

Factors

I. INTRODUCTION

Drawing upon a study of more than 100 e-Gov projects, it was observed in a European report that impact assessment of deployed e-Governance (e-Gov) projects, in terms of tangible and quantifiable socio-economic benefits, was found to be still insignificant [9]. Unfortunately, this situation seems to be in line with findings from other studies [2; 11; 13]. These findings remain inconclusive by virtue of the fact that outcomes of public sector based ICT4D initiatives (e.g. e-Gov) have not been fully established [4].

Impact assessment of e-Gov faces a number of challenges because of certain flaws intrinsic to conventional impact assessment approaches. Some of these challenges include assessing process as against actual impact, placing more weight on external as against community centered indicators of impact. There is also the matter of weak or absence of baselines.

This paper therefore seeks to develop a framework for assessing the socio-economic impact of e-governance projects in developing countries using expert evaluation and case study for its validation and acceptance. Our

proposed framework is premised by the identification of the overall key stakeholders and socio-economic indicators. The latter ought to be considered in quantitatively and qualitatively determine the effect of the intervention to its stakeholders and lesson learnt for improvement thereof.

The remaining part of this paper is structured as follow. In section 2 below, we propose a methodology followed leading to the development of our framework. Then follow in Section 3, a discourse on background and motivation. Section 4 discusses socio-economic indicators for e-Governance programme. The foregoing forms the basis for the grouping of the various core elements required for the development of our framework presented in Section 5. Section 6 covers a discussion of the results of an Expert evaluation. A conclusion and further directions to this work are presented in Section 6.

II. OBJECTIVE AND METHODOLOGY

The main objective of this research was to develop a framework for assessing the socio-economic impact of e- governance projects in developing countries. An extensive literature review was carried out on impact assessment, impact assessment frameworks and e-Gov projects. Key findings made from the foregoing were used and analyzed to propose a framework for assessing the socio-economic impact of e-Gov projects in developing countries. The verification and validation of the framework was done through a semi-structured expert interview. Seven (7) days before commencement of the interview, a gist of the interview was forwarded to experts to help them become aware of and adequately prepare for what is expected of them [14]. A PowerPoint slideshow of the proposed impact assessment framework was presented to participants in order to bring to the fore the understanding that went into the generation of the framework. Though the interview was semi-structured in nature, questions on the expert interview guide were fairly standard and the same for all participants.

The second validation of proposed framework was by means of case study and it was done using the case study evaluation criteria advanced in extant literature [14], a multiple-case study was conducted using specifically two

2017 Conference on Information Communications Technology and Society

978-1-4673-8996-9/17/$31.00 ©2017 IEEE

cases. This involved fieldwork. The first case tested the proposed framework which heavily draws upon the participatory impact assessment approach. The second case represented a replication test of the proposed framework. The purpose of the replication test was to ascertain the consistency of outcomes [13; 14]. During the case studies various sources of evidence were used and triangulated. The sources ranged from internal reports, published reports, newspaper articles and the like. We also had informal exchanges with key informants within the sampled case study organizations.

III. BACKGROUND AND MOTIVATION

Although numerous studies on e-governance have been conducted, there still appears to be no systematic framework for assessing impact of e-governance projects [4], possibly due to the presence of a wide array of impact assessment methodologies [15]. Hence, there is the need for a comprehensive e-Gov impact assessment framework.

Impact assessment of e-Gov projects is a challenging task for many developing countries and even some developed ones. Impact assessment can be defined as the methodical exploration of long-term or major changes – positive or negative, envisioned or not – the lives of individuals triggered by an action or a chain of actions [16]. Projects can have unintended consequences and this may be positive or negative; impact assessment helps bring this to the fore. With regards to e-Gov impact assessment, the level of analysis can either be at project or programme level.

Impact assessment has two key functions – to „prove‟ and to „improve‟ (Table 1). The „prove‟ function of impact assessment involves a test of the significant changes brought about by an intervention (e.g. e-Gov initiative) for reasons of accountability; whereas the „improve‟ function entails improving a practice for the purpose of drawing lessons [12]. The accountability role of impact assessment has to do with relaying information to stakeholders that may be either upward or downward. For example, upward accountability involves government being accountable to donor agencies and development partners with regard to a funded project; but, in downward accountability the project‟s beneficiaries are the focus. Nonetheless, upward accountability is far more favored than downward accountability in many developing countries [5].

Under impact assessment for lesson learning, sponsors and managers of a project are ready to accept mistakes with regards to the project. For this type of impact assessment to work out, there has to be a lot of transparency and trustworthiness [17]. This is because there is the possibility of key decision makers involved in a project having their revealed failures and shortcomings being used against them [5].

Broadly speaking, there are three groupings of impact assessment studies/methodology according to [7], and these groupings depend on the focal point(s) forming the

basis of the exercise. These groups are: (1) Supply-side oriented impact assessment; (2) Anecdotal-oriented impact assessment; (3) Citizens and government agencies-focused impact assessment [4].

TABLE 1. GOALS OF IMPACT ASSESSMENT

Source: After [12], adapted from Hulme, 1997

The supply-side oriented impact assessment methodology focuses on macro-level appraisals of e- governance activity such as web presence, network coverage, institutional and regulatory support and human capital provision [4]. Under the anecdotal-oriented approach, impact assessment is essentially based on anecdotes – „success‟ stories and narrations. Citizens and government agencies-focused impact assessment methodology as the name implies has as its focus citizens and government agencies; under this approach, impact assessment is carried out in relation to how the deployed e-Gov project affects key stakeholders such as citizens, business entities and government agencies. Impact assessment may draw upon either quantitative or qualitative data or both; thus, influencing the type of impact assessment methodology employed. Table 2 below throws more light on this.

End users play a vital role in how impact assessment is structured and implemented [5]. For example, the essential design of impact assessment for illiterate village folks will most likely differ from that for literates dwelling in urban areas, contingent on the type of intervention at play. End user participation or stakeholder involvement has become much evident in the design of impact assessment frameworks these days. A case in point is a study conducted by [3] which measured impact in terms of the total value delivered by an e-Gov project to its diverse stakeholders – client (i.e. citizenry), agency, society and government as a whole; hence, making the

Proving Impact

Improving Practice

Primary Goal

Measuring as accurately as possible the impacts of the interventions

Understanding the processes of the intervention and their impacts so as to improve those processes

Main Audiences

Academics, Policy makers, Evaluation Departments, Programme Managers, Implementing agents

Programme Managers/ Implementing Agents, Donor field staff, Civil Society Organization, Target beneficiaries.

Associated

Approaches/Factors

Objectivity, Theory, External, Top down, Generalization, Academic research, Long timescales, Degree of confidence

Subjectivity, Practice, Internal, Bottom up, Contextualization, Market Research, Short timescales, Level of plausibility

stakeholders a focal point for impact assessment. This makes sense considering the fact that any deployed project aims to cause significant changes, be it socio- economic or otherwise, in the lives of a particular group of people aka 'target beneficiaries'. It therefore follows that the best group of people who can perhaps unequivocally and objectively help determine, via provision of data, whether a project was a success is the target beneficiaries. There is therefore the need to involve them in the project from Day 1.

Source: [12], adapted from Hulme (1997) and [17].

This is a major recommendation by many authors in the socio-economic development space. Consequently, any e-Gov impact assessment framework worth its salt must of necessity incorporate end users or stakeholders; in fact, stakeholder participation has been noted as one of the critical success factors of e-Gov project initiation [10]. The ideology is if stakeholder participation is needed to initiate a project, it therefore stands to reason that stakeholder involvement is equally important for assessment of impact.

In the light of the above, [7] have proposed a Participatory Impact Assessment (PIA) approach, which can also double as a framework (to be discussed in much detail in subsequent sections). This approach perhaps draws upon the Rapid Appraisal IA method shown in Table 2. The PIA however is stakeholder centered; in a sense a philosophy – a way of approaching impact assessment especially in developing countries because of local culture differences.

The functioning of the PIA depends on participation of target beneficiaries in the impact assessment process, combining participatory tools and statistical approaches in a flexible way, thereby ensuring that impact assessment adapts to changing local conditions [7]. This is important because developing economies are characterized by differing local cultures and these local cultures greatly influence how the introduction of new technology initiative will be accepted [10]. In view of this, it is imperative that every decision taken and plan made with regard to the e-Gov project initiative take into account intended beneficiaries. Most often than not, there is a failure on the part of project managers and government to collect pre-intervention baselines, possibly because impact assessment tends to be handled as an afterthought and not built into the project life cycle from start to finish. Refusal to acknowledge the importance of stakeholders to the whole impact assessment process may be another contributory factor to this situation. [6] observes that indigenous people are well able to recognize and measure their peculiar indicators of change.

Consequently, Catley and colleagues advanced a participatory impact assessment approach [7]. The approach contained eight stages: (1) Defining the questions to be answered; (2) Defining the geographical and time limits of the project; (3) Identifying and prioritizing locally-defined impact indicators; (4) Deciding which key impact assessment methods to use, and testing them; (5) Deciding which sampling methods and sampling size to use; (6) Assessment of project attribution; (7) Triangulation; (8) Obtaining feedback and verifying the results with the community.

In 2008, the Department of Information Technology of the Government of India carried out a study to assess the impact of three e-Gov projects namely, Income tax portal, MCA21 e-Governance Project and Online Passport Service. Impact assessment framework for the study was structured along four key dimensions of impact: (1) Cost of availing service; (2) Overall assessment; (3) Quality of Service; (4) Quality of governance, further illustrated in Table 3.

TABLE 3. FRAMEWORK FOR ASSESSING IMPACT OF E-GOV PROJECTS

Dimension of Impact Indicators

Cost of availing

service (Measured directly)

Number of trips made for the service

Average travel cost of making each trip

Average waiting time in each trip

Estimate of wage loss due to time spent in availing the service

Total time elapsed in availing service Amount

Amount paid as bribe to functionaries

Amount paid to agents to facilitate service preference

TABLE 2. COMMON IMPACT ASSESSMENT (IA) METHODS

IA Method

Key Features

Sample Surveys

Collect quantitative data through questionnaires. Usually a random sample and a matched control group are used to measure pre-determined indicators before and after the intervention.

Rapid Appraisal

A range of tools and techniques developed originally as rapid rural appraisal (RRA). Involves the use of focus groups, semi-structured interviews with key informants, case studies, participant observation and secondary sources.

Participa nt Observation

Extended residence in a programme/project community by field researchers using qualitative techniques and mini-scale sample survey.

Case Studies

Detailed studies of a specific unit (e.g. an e-Gov target group, an e-Gov project, organisation) involving open-ended questioning and the preparation of „histories‟.

Participat

ory Learning and Action

The preparation by beneficiaries of a programme of timelines, impact flow charts, village and resource maps, well-being and wealth ranking, seasonal diagrams, problem ranking and institutional assessments through group processes assisted by a facilitator.

Specialize d methods

E.g. Photographic records and video

TABLE 4.1 SOCIO-ECONOMIC IMPACT INDICATORS FOR E- GOVERNANCE PROGRAM WITH RESPECT TO CLIENT

Source: [8]

IV. SOCIO-ECONOMIC INDICATORS FOR E-GOVERNANCE PROGRAMME

Ever since the World Bank in 1996 began taking interest in the governance records of borrowing countries, socio-economic indicators on governance programmes and for that matter e-Gov has become pivotal to investors when deciding the location of foreign direct investment into developing and emerging economies [1]. Spanning a period between 2000 and 2004, loans given out to borrowing countries by the World Bank for the purpose of reforming their economies declined by 14 % every year, whilst loans tied to enhanced governance increased by a margin of 11 % yearly.

Socio-economic indicators are measures that are socio-economic in nature and used to track long term social and economic changes brought about by an intervention (e.g. e-Gov programme). These indicators may either present as core or contextual indicators. Core indicators are universal in nature and so have been standardized; being usually independent of context or the local culture of target beneficiaries. However, context indicators are context-specific and so must be developed at the indigenous level; hence the need for involvement of beneficiaries in impact assessment of e-Gov projects.

For the purpose of this study, three key stakeholders are considered: Client, Agency and Government/society as whole. It should however be noted that the list of stakeholders provided is not exhaustive enough and will vary depending on the project.

1. Client: Dimensions of impact under consideration for this stakeholder are: (i) cost of accessing service measured directly; (ii) Quality of governance measured on a 5-point scale; (iii) Quality of service measured on a 5-point scale; and (iv) overall assessment. Table 4.1 gives more details.

Source: Adapted from [3]

2. Agency: Under this stakeholder, the dimensions of

impact are: (i) Economic impact measured directly; (ii)

Quality of governance measured on a 5-point scale; (iii)

Performance with regard to key non-economic

objectives, such as improved targeting of clients or

equity in coverage, measured on a 5-point scale; (iv)

Process improvements resulting in reduction in

employee workload, improved work environment, and

supervisory control, measured on a 5-point scale.

Overall Assessment

Preference for manual versus computerized system

Composite Score measured on 5-point scale factoring in the key attributes of delivery system seen to be important by users

Quality of Service

Interaction with staff, complaint handling, privacy, accuracy measured on 5-point scale

Quality of Governance

Transparency, participation, accountability, corruption measured on a 5-point scale

Dimensions of Impact

Indicators

Core

Contextual

(i) Travel cost due to the number of trips made to the office for the service and distance traveled

Amount paid as bribes to functionaries or as service charges to agents to facilitate service.

Estimate of wage loss due to time spent traveling to the office for the service and waiting in each trip

Total time elapsed in receiving the service

(ii)

Extent of bribery in the working of the system

Availability of a mechanism to provide feedback to the agency and its effectiveness

Extent to which functionaries can be held accountable for their actions

Transparency of rules and procedures

(iii)

Quality of interaction with functionaries in terms of their courteousness and friendliness

Satisfaction with the mechanism for complaint handling and problem resolution Convenience of working

hours and ease of access to service

Perception about the confidentiality and security of data

(iv) Preference for the computerized system as opposed to the manual system

TABLE 4.2 SOCIO-ECONOMIC IMPACT INDICATORS FOR E- GOVERNANCE PROGRAM WITH RESPECT TO AGENCY

Dimensions of Impact

Indicators

Core Contextual

(i)

Increase in revenue through enhanced compliance by taxpayers

Enlarged base of taxpayers

Collection of user fees from clients

Lowered leakage due to less fraud and corruption

Reduced cost of office space, paper, manpower, and travel

(ii)

Extent of corruption among employees

Transparency of decisions, procedures, and information for internal and external clients.

Accountability, measured as the ability to trace decisions and actions to employees

Participation, measured as the involvement of employees in internal decision processes

Source: Adapted from [3]

3. Society/Government as a whole

For this stakeholder, long-term impact on Millennium Development Goals measured on a 5-point scale and Image of the government measured on a 5-point scale act as the dimensions of impact [3]. Core indicators used to measure the former dimension of impact are reduction in mortality rate, reduction in poverty rate, and reduction in unemployment rate. However, the latter dimension of impact is measured using Credit Ratings (e.g. S&P) and Corruption Perception Index (CPI).

V. PROPOSED IMPACT ASSESSMENT FRAMEWORK

Based on the foregoing, a proposal of an impact

assessment framework is made in Fig 1 above. The Figure

1 above shows that the driving force behind virtually all

impact assessment are goals and expectations and these

lead to the determination of baselines. The goals and

expectations emanate from the funders of the project as

well as the target beneficiaries.

After the execution of an e-Gov project, data is

gathered, analysed, reported and best practices drawn which form the basis for recommendation in terms of upward accountability (i.e. donors and development partners) and downward accountability (e.g. citizens and businesses). The suggested framework reflects the two main functions of impact assessment frameworks – prove and to improve. The „prove‟ component begins with the baseline whilst the „improve‟ aspect starts with the

findings gleaned from the report component of the framework

VI. VERIFICATION OF THE FRAMEWORK

In section 2, it was stated that the proposed framework

will be validated and verified using both expert interviews

and case studies. This section therefore provides a

discussion of the results of the validation process. The

thrust of the validation process was to seek verification

that the framework was useful in the assessment of socio-

economic impact of e-Gov programmes.

The questions posed to the experts were as follows:

(1) Do you agree with the layers of the proposed

framework? (2) Do you agree with the stages of the

proposed framework? (3) Is the framework

comprehensive enough? (4) Do you think the framework

would contribute to the socio-economic impact

assessment of the e-Gov projects? (5) Are there any other

frameworks of which you are aware, to which you can

refer me? (6) Any other comments and suggestions? The

aforementioned questions aimed to authenticate the

layers, stages and extensiveness of the proposed

framework.

Analysis of the results from the semi-structured

interview indicated that both experts agreed with the

layers and stages of the proposed framework. When asked

as to why they agree with the proposed framework, expert

one was of the view that the proposed framework has all

the necessary components of an assessment framework,

whilst the expert two said he agreed because the layers

were clear and comprehensive enough and the fact that

the stages were sequential in nature. As regards the

comprehensiveness of the framework, both experts

attested to the comprehensiveness of the framework. In

the view of expert two, the inclusion of a baseline and

recommendation contributed greatly to the

comprehensiveness of the proposed framework.

Furthermore, they both affirmed that the proposed

framework would contribute to the socio-economic

impact assessment of e-Gov projects.

When asked whether they were aware of any other

frameworks, expert one said, “I don‟t remember”, and

expert two said, “Yes” but could not make reference to

any framework. This situation is perhaps similar to that

reported by [6] who reported that none of the expert

interviews they used for the validation exercise could

make reference to any existing framework. Expert one

recommended that a comparative analysis of practices

underlying that of existing frameworks and that of the

newly proposed approach be carried out to aid fine tuning

of the latter. Expert two suggested that the framework be

deployed using the right data/ scenarios, otherwise its

proper implementation could be greatly hampered. Expert

two went further on to even suggest that if that was done,

generated results could even be ground breaking.

Following [15], the following questions were asked to

decide on the appropriateness of the case strategy (the

answers are provided in parenthesis): (1) Can the

phenomenon of interest be studied outside its natural

setting? (No); (2) Must the study focus on contemporary

events? (Yes); (3) Is control or manipulation of subjects

or events necessary? (Or possible…? No; [15]; (5) Does

the phenomenon of interest enjoy an established

theoretical base? (No, as no cumulative knowledge in

I.S.; Benbasat and Zmud, 1999).

Results from the two case studies showed that impact

assessment of e-Gov projects essentially follow the

pattern of the proposed framework. Some of the

components were observed to hold less importance to

stakeholders, whilst others much importance. The

components, best practices and recommendation seem to

hold less importance to stakeholders. This situation robs

stakeholders of the opportunity to constantly improve

how impact assessment is carried out on e-Gov projects,

via access to pertinent data. Data collected showed that

reporting was very important to stakeholders, perhaps

because of its direct correlation with donors, sponsors or

development partners, considering the fact that release of

funds tends to be tied to reporting and not necessarily best

practices or recommendations.

Figure 1: Proposed Impact Assessment Framework

One of the key essences of the recommendation

component of the proposed framework is to help fine tune

the impact assessment „pipeline‟ by incorporating lesson

learned into best practice. Furthermore, both case studies

scored high on goals and expectation. Consistency of data

gathering was largely an issue. Case study two was found

to be more rigorous in the analysis of its intermittently

collected data. Furthermore, further analysis revealed that

Data gathering and documentation with regard to

assessing the impact of e-Gov most often than not seemed

not to have been planned into the project, thereby

accounting for inconsistent data gathering and by

extension rigorous documentation.

Based on the feedback received from the experts and

the results obtained from the case studies, it can be

concluded that the proposed framework was sufficiently

validated.

VII. CONCLUSION AND FUTURE WORK

In conclusion, an effective impact assessment

framework should be able to take into account of goals

and expectations, the hopes, and aspirations of all

stakeholders or end-users; whilst at the same drawing

lessons from the impact assessment process for onward

recommendation for the fine-tuning of the IA process.

This work presents a phase of a comprehensive

framework for benchmarking e-Governance projects.

With regards to future work, it is suggested that the

proposed framework be subjected to further validation

tests.

REFERENCES

[1] Arndt, C., & Oman, C. (2006). Uses and abuses of governance indicators. OECD Development Studies. doi:10.1787/9789264026865-en

[2] Benjamin, P. (2001). Community Development and Democratisation through Information Technology: Building the New South Africa. In Reinventing Government in the Information Age, ed. R. B. Heeks, 194 – 210. London: Routledge.

[3] Bhatnagar, S. C., & Singh, N. (2004). Results from a Study of Impact of E- government Projects in India.

[4] Bhatnagar, S., Tominaga, J., Madon, S., & Bhatia, D. (2007). Impact Assessment Study of Computerized Services Delivery Projects from India and Chile (No. 2).

[5] Bird, K. (2002). Impact Assessment: an overview for the Chronic Poverty Research Centre (CPRC) Toolbox.

[6] Catley, A. (1999). Monitoring and Impact Assessment of Community-based Animal Health Projects in Southern Sudan: towards participatory approaches and methods. A report for Vétérinaires sans frontières Belgium and Vétérinaires sans frontières Switzerland. Vetwork UK, Musselburgh. Available on http://www.participatoryepidemiology.info/Southern%20Sudan%2 0Impact%20Assessment.pdf, date accessed 7/5/2016.

[7] Catley, A., Burns, J., Abebe, D., & Suji, O. (2007). Participatory Impact Assessment: a guide for practitioners.

[8] Department of Information Technology. (2008). Impact Assessment of e-Governance Projects.

[9] European Union. (2003). Value Creation in eGovernment Projects: An Exploratory Analysis Conducted for the Danish Presidency of the eGovernment Work Group of the Directors General. Danish Ministry of Finance and Accenture.

[10] Hatsu, S. and Ngassam, E. (2015). An Exploration of Critical Success Factors for e-Governance Project Initiation: A Preliminary Framework. Paper presented at IST Conference in South Africa.

[11] Heeks, R. (2002). Information Systems and Developing Countries: Failure, Success, and Local Improvisations. The Information Society, 18(2): 101 – 112.

[12] Herbert, A. with Shepherd, A. (2002) Spin-off study: Impact Assessment for Poverty Reduction. International Development Department, School of Public Policy. Part of DFID funded „Evaluation of DFID Support to Poverty Reduction‟ London, EvD, DFID.

[13] Dube, L., & Pare, G. (2003). Rigor in information systems positivist case research: current practices,trends, and recommendations. MIS Quarterly, 27(4), 597–635. doi:10.2307/30036550

[14] Iacono, J. C., Brown, A., & Holtham, C. (2011). The use of the Case Study Method in Theory Testing : The Example of Steel eMarketplaces. The Electronic Journal of Business Research Methods, 9(1), 57–65.

[15] Benbasat I, Goldstein D and Mead M (1987). The Case Research Strategy in Studies of Information Systems. MIS Quarterly, 11, 369-386.

[16] Benbasat, I. and Zmud, R (1999). Empirical Research in Information Systems: The Practice of Relevance. MIS Quarterly, 23 (1), 3-16.

[17] Montgomery, R. (1995). Guidance note on stakeholder analysis for aid projects and programmes. London, ODA.

1. 1. 2017-05-05T09:55:45-0400
   2. Certified PDF 2 Signature