Global Foundation of Public Administration

EMPIRICAL RESEARCH

Measuring eGovernment success: a public value approach

Murray Scott1, William DeLone2 and William Golden1

1J.E. Cairnes School of Business & Economics, National University of Ireland, Galway, Ireland; 2Kogod School of Business, American University, Washington DC, U.S.A.

Correspondence: Murray Scott, Room 365, J.E. Cairnes School of Business & Economics, National University of Ireland, University Road, Galway, Ireland Tel: +00353 91 495286; E-mail: [email protected]

Received: 17 January 2014 Revised: 6 January 2015 2nd Revision: 28 May 2015 Accepted: 1 June 2015

Abstract Measuring the success of eGovernment systems depends on how citizens perceive their value. Our understanding of success has been hampered however by (i) the rapid development and complexity of Internet technologies and (ii) the lack of conceptual bases necessary to represent the ever expanding range of success dimensions. This study proposes Public Value theory to reposition the DeLone and McLean IS Success Model in order to encompass three essential success or value clusters: efficiency, effectiveness and social value. The efficacy of this approach is demonstrated by creating a Public Value-based (Net Benefits) construct to measure IS success from the citizens' perspective within the context of eGovernment 2.0 systems. Survey responses from 347 experienced users of U.S. government Web 2.0 websites confirm that the proposed success measure is reliable and valid and that the nine-factor structure (Cost, Time, Convenience, Personalisation, Communication, Ease of Information Retrieval, Trust, Well-Inform- edness and Participate in Decision-Making) can explain a major portion of citizens' perceptions of eGovernment success. Additionally, the nine-factor Public Value construct was applied to three identified eGovernment user groups: Passive, Active and Participatory, in order to better understand success in specific usage contexts, including Web 2.0. European Journal of Information Systems (2016) 25(3), 187–208. doi:10.1057/ejis.2015.11; published online 8 December 2015

Keywords: IS success; public value; eGovernment

Introduction Although the evaluation of Information Systems (IS) Success is consistently reported as a major challenge for practitioners and academics, the accurate estimation of value, particularly from the customer perspective, has remained elusive (DeLone & McLean, 1992; Torkzadeh & Dhillon, 2002; DeLone & McLean, 2003; Petter et al, 2012). More than three decades of research has contributed to a vast body of knowledge assessing the success of IS; however, the quest for the dependent variable has resulted in little consensus on appropriate measures for IS Success (Sabherwal et al, 2006), persistent problems over construct validation (Seddon et al, 1999; Petter et al, 2007) and the proclivity of researchers to use single measures of success (Petter & McLean, 2009). Reviewing the previous decade of IS Success research, Petter et al (2008) identify a lack of development in measuring specific dimensions of IS Success; notably, the development of Net Benefits measures has been sparse in studies on IS Success, predominantly taking a narrow, utilitarian focus and a tendency to utilise user satisfaction as a surrogate for success. This is a particular problem for eCommerce and eGovernment research where the primacy of the user is paramount and

European Journal of Information Systems (2016) 25, 187–208 © 2016 Operational Research Society Ltd. All rights reserved 0960-085X/16

www.palgrave-journals.com/ejis/

measures of sophisticated value perceptions are needed (Keeney, 1999; Teo et al, 2008; Wang, 2008). The measurement of success is further complicated as

users of an IS system will often have different attitudes regarding success depending on the specific usage context (Teo et al, 2008). Petter et al (2012) note that in the current ‘Customer-Focused’ era the ability to personalise and customise the user experience of Internet-based systems leads to varying perceptions of value among user groups and individuals. Accounting for this variation is critical as not only may one stakeholder group view the system as a success while others may view it as a failure, but the functionality used by one user may vary considerably to that experienced by others (Myers, 1994; Bartis & Mitev, 2008). New social media technologies have contributed to the

paradigmatic change in the way users interact online with businesses and other organisations (Parameswaran & Whinston, 2007; Kim et al, 2009; McAfee, 2009; Wattal et al, 2010). Web 2.0 technologies have for example provided governments with an unprecedented opportu- nity to provide more personalised, citizen-centric services (Capgemini, 2007; United Nations, 2012; Campbell et al, 2014) and engage the citizen in active, democratic partici- pation (Jaeger, 2005; Peristeras et al, 2009). Such novel forms of interaction enable the co-creation of value in both the private and public sectors (Culnan et al, 2010; Hui & Hayllar, 2010; Mancini, 2012). Changes in online social interaction have thus expanded traditional organisational boundaries to include online community groupings and developed commensurately with an evolving view on value creation. Porter & Kramer (2011) argue that compa- nies should redress the out-dated economic approach to value creation by working to achieve both economic success and social progress. The motivation of this paper is to address Petter et al’s

(2012) call for action: that future development of IS success measures should reflect the current complexity of user online interaction by capturing subjective and intan- gible benefits based on social as well as traditional eco- nomic, utilitarian values. This research focuses on the value perceived by citizens of Web 2.0 eGovernment and adopts Public Value theory (Moore, 1995) to underpin the DeLone and McLean IS Success Model in order to encom- pass both the functional and transactional uses of the Internet and also the participative, collaborative activities of online communities. Public Value provides a framework that distinguishes between clusters of value dimensions: both tangible benefits of improved efficiencies and service effectiveness and also democratic values concerned with participation, engagement and trust (Jorgensen & Bozeman, 2007; Bryson et al, 2014). The objective of this paper is to contribute to eGovern-

ment and IS Success research by (i) developing and validat- ing for the first time a Public Value-based construct to measure Net Benefits of eGovernment 2.0 systems from a citizen's perspective and (ii) stratifying that Public Value construct for different eGovernment user types.

The conceptual model development process used in the definition and confirmation of the Public Value Net Benefits construct was conducted in distinct stages: (i) domain and item identification, (ii) content validation, (iii) exploratory analysis and (iv) confirmatory analysis, based on best practice as advocated by Lewis et al (2005), Straub et al (2004), Straub (1989) and Churchill (1979). The resulting instrument provides specific dimensions useful to both governments and businesses in the quest to understand what value their respective users derive through Internet-based interactions. A comprehensive description of the process employed in

the development of the Public Value Net Benefits scale and the related research instruments is provided in the rest of the paper and developed from previous publications (Scott et al, 2009, 2011). The next section of the paper presents the theoretical background for the new eGovernment Public Value (Public Net Benefits) success construct. The following section then discusses the conceptual develop- ment, testing and validation of the proposed construct. The next two sections review the implications of the validated eGovernment Public Value construct for research and practice. The following section covers the limitations and future research directions. The paper ends with a conclusions section.

Theoretical background

IS success and net benefits The DeLone and McLean IS Success model is the most widely cited framework in the IS discipline (Lowry et al, 2007). In 2003, DeLone and McLean published an updated model and reported that during the period 1993-2002 a total of 285 journal and conference papers were published referencing their original success model (DeLone & McLean, 2003). However, the creation and selection of success constructs is critical to the effective application of the IS success model. Although the DeLone and McLean model has been adapted to a variety of Internet-based systems including eCommerce websites (Wang, 2008) and eGovernment service delivery systems (Teo et al, 2008; Connolly et al, 2010; Sørum et al, 2012), there is a continuing need to develop measures that gauge the success of new IS that aim to deliver hedonic as well as utilitarian benefits (Petter et al, 2008). The Net benefits construct was conceptualised by

DeLone & McLean (2003) in the broadest manner; the construct refers to the extent that IS contributes to the success of individuals, groups, organisations, industries and societies. The subsequent development of measures for Net Benefits has been narrowly conceived however, with the majority of studies focusing almost exclusively on the impact of IS in the work environment (Petter et al, 2008). Net Benefits are typically represented in these studies using perceived usefulness or job impact as the most commonly adopted measure (Adams et al, 1992; Segars & Grover, 1993). At the individual level, this narrow focus is understandable given the purposes for which the

Measuring eGovernment success Murray Scott et al188

European Journal of Information Systems

majority of systems have historically been created. How- ever, the legacy of construct development in this area has yielded Net Benefits which have a singularly utilitarian focus on productivity, efficiency and task improvement (e. g., Gable et al, 2008). As the evolving impact of IS extends beyond the func-

tional work place to social, leisure and personal contexts, the development of new measurement instruments is critical to capture a wider range of value attitudes than simply utilitarian perspectives (van der Heijden, 2004; Petter et al, 2008).

eGovernment success research Although IS Success models have been applied in numer- ous contexts predominantly the private sector, insufficient research has been conducted in identifying measures that determine eGovernment success from a citizen perspec- tive. Belanger & Carter (2012) analyse 30 eGovernment articles published in the AIS Senior Scholar’s Basket of journals from which a minority (five) use citizen-based survey data or the DeLone & McLean IS Success Model (two). The limited research on eGovernment success has been internally focused on employees (Gable et al, 2008; Prybutok et al, 2008), on G2C eGovernment systems (Wang & Liao, 2008) and eGovernment websites (Teo et al, 2008; Connolly et al, 2010). Not only is there a need for more research focusing on the development of eGo- vernment success measures but specifically an approach that examines Net Benefits from a citizen as user perspec- tive that includes new social media environments. The measurement of eGovernment success is therefore not well understood in either practitioner or research communities (Heeks, 2008) and the development of success measures is an urgent task to enable investments in technology to justify their public value (Yildiz, 2007).

Public value net benefits Measuring success in the public sector is a difficult task, while private sector firms focus on efficiency, quality and reliability, public managers must combine these concerns with accountability, the creation of trust and differing public preferences (Hefetz & Warner, 2004). The nature and breadth of the purposes and proposed outcomes of public services serves to distinguish the task of eGovern- ment evaluation from their commercial counterparts (Grimsley & Meehan, 2007; Grimsley et al, 2007). Given the pressing need for the evaluation of eGovernment to include a broader democratic perspective, several authors have called for a broadening and deepening of scholarly perspectives on eGovernment, to adequately provide social scientific theory and insight (Reece, 2006; Heeks & Stanforth, 2007; Yildiz, 2007; Andersen et al, 2010). In response to this challenge, this research proposes the

use of Public Value as a new theoretical framework for measuring efficiency, effectiveness and social value in understanding eGovernment success. First articulated by Moore (1994, 1995), the Public Value approach is an

alternative to previous public management approaches, which have been criticised for emphasising narrow con- cepts of cost-efficiencies (O’Flynn, 2007; Cordella & Bonina, 2012). Public Value can be understood as the value or importance citizens attach to the outcome of government policies and their experience of public ser- vices (Moore, 1994). Public Value provides a new way of thinking about the evaluation of government activity and a new conceptualisation of the public interest is defined in an effort to combine efficiency, effectiveness and the creation of social value (Stoker, 2006; O’Flynn, 2007; Bryson et al, 2014). The promise of Public Value has thus resulted in a growing body of theoretical development (Williams & Shearer, 2011; Pang et al, 2014) and the application of Public Value to specific contexts through empirical research has been encouraged (Benington & Moore, 2010). Much research has focused on identifying the scope of

key value dimensions defining this new emerging public administration movement (Bryson et al, 2014). During the era of Traditional Public Administration, efficiency values were paramount; later, during the New Public Manage- ment period, the greater concern of effectiveness empha- sised the notion of the citizen as customer. The emerging public value movement seeks to amalgamate these two value dimensions (efficiency and effectiveness) but refocus attention to a broader array of values, especially those concerning democracy and social equity. Bryson et al (2014) review the Public Value domain and identify key areas where these value clusters have been identified and operationalised. Jorgenson & Bozeman (2007) for example focus on the policy and societal level; Moore (Moore, 1995, 2013) consistently focuses on the challenge of creating public value from the perspective of public managers; Meynhardt (2009) emphasises the need to assess value from the experience of individuals; and Benington (2011) conceptualises a public sphere encompassing a web of values held in common by individuals, communities, governments and public institutions. The new public administration movement holds much promise in defin- ing a better vision of public value; however, progress in this field will rely on the implementation of accurate measures of value in practice (Bryson et al, 2014). While many studies have espoused the potential bene-

fits of eGovernment, there are few studies that seek to empirically examine or identify dimensions of Net Benefits from the perspective of the citizen (Reddick, 2005). Those that have – (Gilbert et al, 2004; Kolsaker & Lee-Kelley, 2008) – are not comprehensive in their treatment of value and do not canvass the views of experienced users of eGovernment. The lack of theoretical development in this area amplifies the need for more research to study the demand for eGovernment services from a citizen-based perspective, underpinned by solid theoretical concepts (Andersen and Henriksen, 2005; Reddick, 2005; Heeks & Bailur, 2007; Yildiz, 2007; Helbig et al, 2009; Andersen et al, 2010; Chan et al, 2010; Barbosa et al, 2013). This study aims for the first time to measure the success of

Measuring eGovernment success Murray Scott et al 189

European Journal of Information Systems

eGovernment systems by drawing together a comprehen- sive set of Net Benefits measures based on Public Value theory.

Public value net benefits model development and testing The concept of Public Value, as defined by Moore (1995), requires a balancing of efficiency and effectiveness mea- sures with improvements in democratic and social values such as engagement, participation and trust in govern- ment. The creation of Public Value is a function of both the value received from the service or product and the cost of consumption and resources expended to produce the service. The resulting net value closely mirrors the concep- tion of Net Benefits in IS Success research (DeLone & McLean, 2003) and specifically, the costs and benefits of interacting online (Torkzadeh & Dhillon, 2002). There- fore, we define Net Benefits to be ordered around three broad objectives: efficiency, effectiveness and improved democracy. These value clusters reflect similar distinctions proposed in the literature (Jorgensen & Bozeman, 2007; Heeks, 2008; Cordella & Bonina, 2012; Harrison et al, 2012; United Nations, 2012; Bryson et al, 2014). Further, this study identifies web-based eGovernment systems (G2C), which include Web 2.0 functionalities, as the IT artefact from which citizen users develop value percep- tions. As such, the following definition, developed by the researchers, served as a guide to identify relevant dimen- sions of the Net Benefits constructs along with the extant literature: ‘a positive benefit experienced by a citizen resulting from direct interaction with web-based eGovern- ment systems’. This definition serves to confirm the level of analysis for the study, that is, measuring perceptions at the individual level and also the IT artefact under exam- ination, the web-based eGovernment system.

Domain and item identification The first step of construct development aims to establish the domain of the idea (Lewis et al, 2005). This study utilises the conceptual framework of the DeLone and McLean IS Success Model to construct a Net Benefits success measure centred on the perspective of the citizen. The concept of Public Value (Moore, 1995) provides a theoretical framework within which to conceptualise the broad dimensions of eGovernment success from a citizen's perspective. As such, the user defines the context or frame of reference as called for by DeLone & McLean (2003) and the benefits are measured from the individual's perspective defining the level of analysis for this study. Following the deductive approach, this research

employed the technique of content analysis in order to capture the dimensions of Public Value Net Benefits from an extensive review of the literature (Weber, 1985). This process included a careful review of literature from several disciplines, Information Systems (IS) (eGovernment, eCommerce, General IS Success), Public Administration and included journals, academic conferences, practitioner-

oriented magazines, trade magazines, private-sector reports, reports conducted by public and private sector research groups and an extensive review of Government strategy documents and policy material. Keyword searches were performed targeting the title, abstract and keyword list of published material using Web of Science, Scopus and ABI-Inform. The chosen keywords initially reflected the key elements of the construct: ‘eGovernment success’, ‘eGovernment benefits’, ‘eGovernment Net Benefits’, ‘eGovernment efficiency’, ‘eGovernment effectiveness’, ‘eGovernment participation’, ‘Public Value’, ‘Social Value’, ‘Customer and Citizen Value’ and ‘eGovernment Value’. Multiple variations of the term eGovernment were also employed as defined by Gronlund & Horan (2004). This along with the prior development of broad categories relating to the elements of the constructs – efficiency, effectiveness and improved democracy (Lewis et al, 2005) enabled the identification of dimensions within the litera- ture that were conceptually relevant to the eGovernment Net Benefits domain (Tojib et al, 2008). In total, three rounds of literature review and content

analysis were conducted, surveying over 600 separate articles, reports and other documentation and were itera- tively refined until relatively distinct groupings emerged, which in turn embodied the specific dimensions of the Net Benefits construct. Following this process 11 dimensions were identified grouped around the three broad categories: the Dimensions were Cost, Time, Communication, Avoid Personal Interaction, Control, Convenience, Personalisa- tion, Ease of Information Retrieval, Trust, Well-Informed- ness and Participate in Decision-Making falling under the categories of Efficiency, Effectiveness and Improved Democracy (see Table 1 for details). The categories and dimensions correspond to the overall focus on perceived citizen value of eGovernment systems.

Cost The potential for cost-savings is well established from the earliest reviews of eGovernment (Al-Kibisi et al, 2001; Watson & Mundy, 2001) and holds particular bene- fit for citizens as a tangible outcome of using an online channel to interact or transact with government (Lau, 2006). Electronic tax filing, for example, has been a fre- quent area of study within which cost savings has been identified as a consistent benefit (Tan & Pan, 2003; Fu et al, 2004; Fu et al, 2006). Cost savings have further been iden- tified as one of the strongest predictors of willingness to use eGovernment (Gilbert et al, 2004; Norris & Reddick, 2013; Lawson-Body et al, 2014).

Time Time saved by using the online channel was an important early promise of the benefits of using eGovern- ment and has been established as a common perception among end users (Tan & Pan, 2003; Fu et al, 2004; Fu et al, 2006; Gouscos et al, 2007; Andersen et al, 2010; Norris & Reddick, 2013; Lawson-Body et al, 2014). The time saved in technology-based service encounters is rooted in studies of customers in an e-commerce context (Meuter et al, 2000;

Measuring eGovernment success Murray Scott et al190

European Journal of Information Systems

Liao & Cheung, 2001; Liao & Cheung, 2002) and refers to the perception of a faster response to an online interac- tion, particularly in comparison to other offline methods of service delivery.

Communication As a mode of interaction, the Internet is an efficient method of connecting citizens to government

departments (Brown, 2007; Gonzalez et al, 2007). Com- munication methods can take the form of email or online forums and more recently, Web 2.0 technologies have provided a range of new tools such as forums, blogs, chat rooms and other social networking media (Weinberger, 2002; Baumgarten & Chui, 2009; Zimbra et al, 2009; Hui & Hayllar, 2010; Campbell et al, 2014). These new tools pro- vide a unique opportunity to engage the citizen in an

Table 1 eGovernment net benefits

Dimension Definition Category Source(s)

Cost Cost saving to the user from using the online channel

Efficiency Tan & Pan (2003), Fu et al (2004), Gilbert et al (2004), Fu et al (2006), Norris & Reddick (2013), Lawson-Body et al (2014)

Time Time saved by using the online channel

Efficiency Tan & Pan (2003), Fu et al (2004), Gilbert et al (2004), Fu et al (2006), Gouscos et al (2007), Kolsaker & Lee-Kelley (2008), Wang & Liao (2008), Andersen et al (2010), Norris & Reddick (2013), Lawson-Body et al (2014)

Communication Efficient method of communicating with local/ central government

Efficiency (Gonzalez et al, 2007; Kolsaker & Lee-Kelley, 2008; Baumgarten & Chui, 2009; Hui & Hayllar, 2010; Ahn, 2011; Lee & Rao, 2012; Campbell et al, 2014)

Avoid personal interactiona To receive public services without having to interact with service staff

Effectiveness Gilbert et al (2004), Gonzalez et al (2007), Yang & Rho (2007), Chan et al (2010)

Controla The ability to exert personal control over the service

Effectiveness Liao & Cheung (2002), Gilbert et al (2004), Grimsley & Meehan (2007), Kim & Lee (2012), Lawson-Body et al (2014)

Convenience The ability to receive the service how and when the individual wants

Effectiveness Marche & McNiven (2003), Gilbert et al (2004), Chan et al (2010), Norris & Reddick (2013)

Personalisation The ability to tailor the service to the individual

Effectiveness Weinberger (2002), Gilbert et al (2004), Kolsaker & Lee-Kelley (2008), Morgeson & Mithas (2009), Zimbra et al (2009)

Ease of information retrieval Useful and helps the user understand about the service

Effectiveness Thomas & Streib (2003), Wong & Welch (2004), Welch et al (2005), Kolsaker & Lee-Kelley (2008), Teo et al (2008), Ahn & Bretschneider (2011)

Trust Increase in trust and confidence in Government

Improved democracy McKnight et al (2002), Warkentin et al (2002), Welch et al (2005), Belanger & Carter (2008), Teo et al (2008), Alomari et al (2012), Belanche et al (2014)

Well-informedness Better informed, knowledgeable about government policy

Improved democracy Thomas & Streib (2003), Coleman (2004), Grimsley & Meehan (2007), Kolsaker & Lee-Kelley (2008), Lee & Rao (2012)

Participate in decision-making Involved, exert influence in the democratic process

Improved democracy Coleman (2004), Grimsley & Meehan (2007), Kolsaker & Lee-Kelley (2008), Medaglia (2012)

aControl and Avoid Personal Interaction dimensions were dropped after the pilot study; the remaining nine factors were retained. Based on loading scores, one item from ‘Control’ factor was added to Convenience factor and one item from ‘Avoid Personal Interaction’ factor was added to the Time factor.

Measuring eGovernment success Murray Scott et al 191

European Journal of Information Systems

evolving and dynamic dialogue and thus the communica- tion construct is an important dimension of the value of eGovernment (Coleman, 2004; Coleman, 2005; Kolsaker & Lee-Kelley, 2008; Ahn, 2011; Lee & Rao, 2012).

Avoid personal interaction Stemming from the perceived benefits of the online self-service channel (Hansen, 1995; Meuter et al, 2000), the opportunity to expedite an online service, without the need to deal directly with a govern- ment representative, has been identified as a benefit of eGovernment (Gilbert et al, 2004; Gonzalez et al, 2007; Yang & Rho, 2007; Chan et al, 2010).

Control eGovernment has the potential to empower citi- zens. One of the ways in which users can exert power is through their ability to control the delivery of the service (Liao & Cheung, 2002; Zhu et al, 2002; Lawson-Body et al, 2014). Grimsley & Meehan (2007) argue that having per- sonal control over the service needs to complement the other diverse demands made upon them through work, family or other commitments. Being able to choose when to request a service, for example, expresses service empowerment for the individual (Gilbert et al, 2004; Kim & Lee, 2012). Furthermore, the ability of the user to accomplish what they initially set out to achieve is an important determinant of the sense of empowerment and control from using eGovernment.

Convenience The ability of the individual to easily access information and services is an important component of the convenience benefits from self-service technologies (Meuter et al, 2000; Szymanski & Hise, 2000; Zhu et al, 2002; Chan et al, 2010). The Internet provides more accessible and available services than traditional channels, as typically online services can be reached regardless of location and time (Marche & McNiven, 2003; Gilbert et al, 2004; Norris & Reddick, 2013).

Personalisation Several studies have identified that user expectations from e-commerce websites (van Riel et al, 2001) have created a similar requirement for eGovernment websites to cater to personalisation features (Gilbert et al, 2004; Kolsaker & Lee-Kelley, 2008; Morgeson & Mithas, 2009). Kolsaker & Lee-Kelley (2008) found that persona- lised service ranked ahead of other tangible benefit factors of eGovernment. Web 2.0 further provides a new potential method of providing personalisation features by allowing the individual to personalise their use and experience of particular websites, while at the same time participating in a much larger public representation (Weinberger, 2002; Zimbra et al, 2009).

Ease of information retrieval Online information dis- semination is the primary function of eGovernment and information searching accounts for the majority of online activities with eGovernment websites (Teo et al, 2008). The ease with which information can be accessed and the

value of the available information are key determinants of this benefit of eGovernment. Thomas & Streib (2003), Wong & Welch (2004), Welch et al (2005) and Ahn (2011) argue that increased availability and provision of informa- tion through eGovernment can indicate improved open- ness and transparency.

Trust Trust is a complex construct and has been employed in numerous studies in eGovernment (Carter & Belanger, 2005; Grimsley & Meehan, 2007; Belanger & Carter, 2008; Teo et al, 2008; Alomari et al, 2012; Belanche et al, 2014). As a benefit dimension, trust is defined as an outcome variable relating to the direct experience of the user from the eGovernment website, which acts as an information and service provider. Trust includes respond- ing to requests, acting in the best interests of the citizen and reliably providing a service and meeting those obliga- tions (Lau, 2006; Tolbert & Mossberger, 2006; Jorgensen & Bozeman, 2007; Teo et al, 2008). As such, trust in the con- text of eGovernment relationships is understood in terms of mitigating relational risk – the risk that a partner may fail to meet its commitments (Das & Teng, 2001; Ibbott & O’Keefe, 2004; Seltsikas & O’Keefe, 2010). One of the key elements of the theoretical approach

adopted by this research is to view Public Value as created through a process of co-production and cooperation between citizens and government (Moore, 1995). Trust is identified in this view as a central component in the achievement of Public Value (Kelly et al, 2002; Stoker, 2006). From this perspective, it is important to develop a trust benefit that relates to feelings of trust in government as an institutional partner and co-producer of value. Restricting measures of trust to trust in technology or trust in the eGovernment website, as have been previously studied (Carter & Belanger, 2005; Teo et al, 2008), is not sufficient or appropriate to this approach that seeks to focus on partnership with government in the production of Net Benefits, or more generally Public Value. Seltsikas & O’Keefe (2010), who analyse the role of trust

as a benefit outcome of using eGovernment in a Public Value context, support this proposed conception of trust. The items chosen therefore represent trust in government and include references to obligation, duties and citizens' best interests. The importance of trust in government has further been demonstrated by studies (Teo et al, 2008) that similarly use the theoretical model of the DeLone and McLean framework, as being significant in this context, but not however general trust in technology. Various other studies in eGovernment, especially those

using the DeLone and McLean Model and other similar causal models, have adopted trust in government as an important and reliably validated construct, represented by the proposed items for this study (Warkentin et al, 2002; Carter & Belanger, 2005; Belanger & Carter, 2008; Teo et al, 2008; Wang & Liao, 2008). The specific items representing trust were adopted from a seminal article in the IS field on trust in eCommerce (McKnight et al, 2002) and

Measuring eGovernment success Murray Scott et al192

European Journal of Information Systems

subsequently used and validated in various studies in eGovernment. The items chosen for trust in government also relate to perceptions of trust in various forms of interaction and transaction. This is in line with the types of use of eGovernment that this study focuses on and seeks to gather data about, that is, transacting, messaging, interacting, participating and providing information.

Well-informedness Grimsley & Meehan (2007) argue that citizens need to feel well-informed about government and government services. eGovernment provides the opportu- nity for citizens to keep informed, increase their under- standing and build up their knowledge about issues of importance to them. Recent studies reveal that as citizens become more accustomed to searching for information, they become more knowledgeable about issues than non- eGovernment users and as a result, are more able and likely to express their opinions via eGovernment websites (Coleman, 2004; Coleman, 2005; Kolsaker & Lee-Kelley, 2008; Lee & Rao, 2012). By extension, various other studies postulate and identify resultant implications for improved accountability and transparency through eGovernment (Marche & McNiven, 2003; Thomas & Streib, 2003; Wong & Welch, 2004; Gouscos et al, 2007; Pina et al, 2007; Yang & Rho, 2007). As such, well-informedness is a key benefit for the improvement of democratic processes and a core component of public value.

Participate in decision-making Comprising the final stage in the evolution of eGovernment (Moon, 2002), the implementation of online citizen participation is a diffi- cult and challenging task. However, there are a growing number of successful examples among the more advanced countries in eGovernment (e.g., Canada and the U.S.A.) and the expected benefits have been equally reported and anticipated in the literature (Barnes & Vidgen, 2003; Marche & McNiven, 2003; Lau, 2006; Tolbert & Mossberger, 2006; Olphert & Damodaran, 2007; Yang & Rho, 2007; Kolsaker & Lee-Kelley, 2008; Medaglia, 2012). The importance of involvement and the perception of

being able to exert influence with government are impor- tant components of this dimension (Coleman, 2004; Coleman, 2005; Kolsaker & Lee-Kelley, 2008). Such influ- ence can be expressed through comment, discussion or negotiation and is a critical element of active democracy and of Public Value (Grimsley & Meehan, 2007; Jorgensen & Bozeman, 2007). Web 2.0 is an example of the role technology can play in achieving better engagement and participation through the introduction of social network- ing tools in eGovernment, commonly referred to as eParti- cipation (Medaglia, 2012). Web 2.0 can create interactive and collaborative platforms to bring together citizens and public managers in a creative and deliberative process (Hui & Hayllar, 2010).

Generation of measurement items A master list of items was generated from the set of dimensions identified as part of

the Net Benefits construct (Churchill, 1979; Lewis et al, 2005). Multiple items were generated for each dimension to ensure reliability and internal consistency (Nunnally, 1978). The majority of items from existing scales that had been empirically tested were considered and adapted for use in each new measure. The process of item generation and removal followed those used by Moore & Benbasat (1991): firstly, items were categorised according to each dimension; secondly, items considered too narrow in focus and applicable only in particular situations or to particular websites were removed; thirdly, new items were created for dimensions with fewer than three items or where the total meaning of the dimension had not been covered. A total of 48 items were generated for the 11 Public Value Net Benefits dimensions.

Content validation process Following best practice (Straub, 1989; Moore & Benbasat, 1991; Lewis et al, 2005), a group of experts were asked to evaluate the adequacy of items and dimensions of the Net Benefits construct. A mixture of academic and practitioner experts were used to ensure a high degree of representa- tiveness at both the conceptual and practical level (Grant & Davis, 1997). In total, 11 content experts participated in the study; three were academic experts in the areas of IS Success, eGovernment, Public Administration and Public Value and eight were experts with various professional positions within the public sector. The public sector practitioner experts all held leadership positions; three individuals from the General Services Administration (GSA) of the United States of America Federal Govern- ment, two individuals from the Whitehouse Administra- tive staff and one official from the Environmental Protection Agency (EPA) of the Federal Government. Two further individuals participated from the Canadian gov- ernment both with responsibility and expertise in survey- ing citizens and evaluating eGovernment initiatives. The experts were asked (i) to assess whether the item

contents adequately measured all dimensions of the con- struct and overall, how representative the items are of the content domain (Berk, 1990), (ii) to review the consistency of items with the conceptual definition and (iii) to evaluate the entire instrument for completeness and representa- tiveness of the content domain (Lynn, 1986). The expert qualitative review indicated that the list of dimensions was comprehensive and no additional dimensions were required. The group of experts however did provide sub- stantial advice regarding instrument design. Following the expert panel pre-test, in keeping with best

practice (DeVellis, 1991; Grant & Davis, 1997), a second round of content validation was conducted using a sample of 15 eGovernment users drawn from the target popula- tion. The overall purpose of this stage was to pilot test the instrument; accordingly participants were asked to com- plete the questionnaire, then comment on difficulties in completing the instrument, offer suggestions for improve- ment as well as advising on item statements that should be

Measuring eGovernment success Murray Scott et al 193

European Journal of Information Systems

removed or those that were felt to be missing (Lewis et al, 2005; Dillman, 2007). Using the ‘think-aloud protocol’ as proposed by Dillman

(2007), respondents were encouraged to verbalise their thoughts as they completed each stage of the question- naire. On completion of the survey the researcher then interviewed each participant, enabling an exploration or clarification of comments where appropriate, ensuring that the full range of testing issues were covered. The instrument design was significantly improved using the feedback from the second round of content validation. For example, technical expressions such as ‘eGovernment’ that might lead to misinterpretation and confusion among respondents (Grant & Davis, 1997) were replaced with more appropriate expressions. Further, the pilot sample advised, based on their perso-

nal experience of using government websites, that some items may not be applicable to a particular user, based on differing usage scenarios. As such, an additional category containing a ‘Not Applicable’ option was added to the item scale. Corroborating the first round of content validation, this panel of respondents confirmed that the set of dimen- sions was comprehensive from their experiential perspec- tive and adequate to represent the domain of the Net Benefits construct.

User types Accounting for differing user types has been identified as critical given the range of functions available online for example, the ability to personalise Internet-based systems, particularly those using social media technologies (Petter et al, 2012). Teo et al (2008) captured the nature of usage of eGovernment websites grouped into Passive and Active user types. In recogni- tion of the successful use of social media technologies by some governments, this study included for the first time a Participatory user type and extended the Teo et al (2008) measure to account for these activities (see Table 2).

Field study data collection: sample and procedures An exploratory analysis was carried out using pilot survey data collected in the U.S.A. The U.S.A. has consistently ranked in a group of highly developed countries in the United Nation's Global E-government Readiness Report, from 2003 to 2012 (UN, 2005; United Nations, 2012) and in e-participation initiatives (UN, 2005; United Nations, 2012). Since 2002, the U.S. Government has set effi- ciency, effectiveness and recently citizen engagement (Whitehouse, 2010) as key strategic priorities for eGovern- ment programmes (U.S. Government, 2002). From the citizen perspective, there is also evidence to suggest that users willingly engage with eGovernment service offerings in the U.S.A. and show a desire to use the Internet to search for information, transact with government and use this medium to participate in debates on government policy (Reddick, 2005; Whitehouse, 2010). Given the combination of sophisticated e-service development and growing citizen usage, the U.S.A. provides a sufficiently comprehensive and well-developed context to provide a rich set of citizen responses with which to test the research model. Drawing on the definition of the domain as set out in

the research, relevant users should be citizens who have had direct experience of benefits accruing from their usage of eGovernment web-based systems. This study is the first attempt to actively target a range of experienced users of eGovernment websites to develop a measure of Net Bene- fits. Previous studies that evaluate citizens' perspectives on eGovernment either do not attempt to gather data from experienced users (Gilbert et al, 2004; Carter & Belanger, 2005; Belanger & Carter, 2008; Kolsaker & Lee-Kelley, 2008; Kuk & Janssen, 2013) or use a limited sample set and do not control for either experience of usage or recency of use in data collection (Barnes & Vidgen, 2006; Teo et al, 2008; Lee & Rao, 2012; Kuk & Janssen, 2013). In order to develop an accurate measure of Net Benefits, this research places a premium on the level of experience

Table 2 Instruments for measuring user types

Please rate your experience of using U.S. Federal Government websites for the following activities:

(For each row please choose the best answer on a scale of 1 = Very inexperienced to 5 = Very experienced) 1. Browsing Government websites for information 1 2 3 4 5 2. Posting opinions or messages to online groups/blogs 1 2 3 4 5 3. Transacting with a Government agency, for example, for a service or to pay a bill 1 2 3 4 5 4. Interacting with Government officials on the web, for example, by email 1 2 3 4 5 5. Downloading documents, for example, forms, from Government websites. 1 2 3 4 5

Please indicate the types of activities you have used your chosen website for (check √ all the functions that you use):

Browsing Downloading Messaging (e.g. e-mail enquiry) Transacting (e.g. service/financial) Participating (e.g. submitting comments)

Passive Users defined through Browsing and Downloading categories (Teo et al, 1997, 2008). Active Users defined through Messaging and Transacting categories (Teo et al, 1997, 2008). Participatory Users defined through Participating category.

Measuring eGovernment success Murray Scott et al194

European Journal of Information Systems

across a range of activities (active, passive and participa- tory) and the requirement of recent usage to ensure accurate impressions are recorded from the respondent. The University environment was chosen as the most

appropriate site for locating suitable participants for the sampling frame and included undergraduate, graduate and postgraduate students, faculty and staff of specific univer- sities in the U.S.A. The intention was to develop a sample similar in characteristic to reported frequent users of eGovernment services - educated to third level, a worker in the public sector and a highly experienced user of the Internet (Princeton, 2001; Reddick, 2005). Previous studies have shown university students to be an opportune sam- ple in the area of eGovernment as respondents are typi- cally frequent users of eGovernment (Carter & Belanger, 2005; Teo et al, 2008) and frequent, experienced users of the Internet (McKinney et al, 2002; Palmer, 2002). Thus, the sampling strategy of targeting the University environ- ment strongly matches the profile of experienced eGo- vernment users. This study follows the method of previous studies of

eGovernment users, where respondents are provided with a list of websites to choose from (Carter & Belanger, 2005; Barnes & Vidgen, 2006; Belanger & Carter, 2008; Wang & Liao, 2008; Morgeson & Mithas, 2009). In addition, the objective of this study is to develop a comprehensive set of benefit measures and as a result the scope of websites should be sufficiently comprehensive to cover the full range of eGovernment interactions as defined by Moon (2002) – Stage 1 Information Dissemination/catalogue; Stage 2 Two-way communication, for example, email; Stage 3 Service and financial Transaction, for example, pay/file taxes; Stage 4 Vertical and horizontal integration, for example, e.g. one-stop shop supplying federal/state/ local information; Stage 5 Political Participation, for exam- ple, submitting comments online. This research began with an initial list of 61 Federal

Agencies (West, 2008), covering all federal department and agencies in the U.S. Government. The selection process that followed had several objectives: to develop a compre- hensive set of websites representative of all Net Benefits dimensions; to develop a list of sites based on importance, high traffic and representativeness of government services; and to identify sophisticated websites with existing high levels of citizen satisfaction. Initially, all websites were evaluated based on Moon’s

(2002) stage model, in order to establish from the func- tionality present, which stage that particular website had attained. The complete set of stages comprehensively covers all possible usage interactions with eGovernment and hence accounts for the functionality necessary to provide experience of the Net Benefit dimensions. The selected websites were chosen due to the high level of functional consistency and to cover all functions in the stage model, that is, stages 1-5. Next, each site on the list was reviewed for suitability according to the Net Benefit factors reviewed from the literature; then a detailed review of each website was performed to ensure that a sufficient

range of functionality pertaining to Passive, Active and Participatory user types was present. Existing empirical evidence further supported the choice

of websites: the chosen set of six websites were ranked within the top 12 federal government websites (out of 61) for information provision, online services and public out- reach (West, 2008). The websites were also indicated as having achieved high rates of citizen satisfaction and high usage rates from previous empirical analyses (e.g. Morgeson & Mithas, 2009). The set of six Government websites were thus argued to be sufficiently consistent and comprehen- sive with which to test all aspects of the research model and were appropriate for each user type (Passive, Active and Participatory). The list of websites was also corroborated by the pilot test sample and the expert panel for relevance, coverage (important government services), comprehensive- ness (dimensions) and most popular. The list included: U.S. A.gov, Whitehouse.gov, Regulations.gov, ED.gov, IRS.gov, SSA.gov. Each of these websites provided a range of Web 2.0 functionalities, for example, forums, blogs, chat rooms and other social networking media, in addition to traditional internet-based services.

Exploratory analysis – pilot survey A sample of users completed the pilot survey instrument that was used to conduct an exploratory factor analysis. Email invitations were sent to students, faculty and staff from two participating Universities in the U.S.A. In total the pilot survey generated 216 responses of which 90 were complete and usable. Seventy-six related to the pre-defined survey websites for which the respondent indicated the last access date was within 6 months; the remaining 14 responses related to websites not on the pre-defined list but which were subsequently deemed acceptable having sufficiently met the website selection criteria. Respondents were required to indicate their level of experience with various eGovernment activities in order to assess which user type (Passive, Active or Participatory) was appropriate. Net Benefits items were measured using a 7-point Lickert scale. The sample size was considered adequate based on a ratio of 8.2 responses per construct; higher than the minimum ratio of 5 and close to the ideal ratio of 10 (Netemeyer et al, 2003; Lewis et al, 2005; Hair et al, 2006). The sample was well distributed in terms of age demo- graphics: 18-24 (32%); 25-34 (27%); 35-44 (8%); 45-54 (21%); 55-64+ (12%). The Bartlett test of sphericity and the measure of sam-

pling adequacy (MSA) were examined to confirm the appropriateness of exploratory factor analysis (EFA). Hav- ing met the thresholds of sampling adequacy (KMO = 0.65, Bartlett's test of sphericity P<0.001), the EFA process was conducted using principle components analysis using the Varimax orthogonal factor rotation method following an iterative process (Tabachnick & Fidell, 2001; Hair et al, 2006). The iterative process of retaining or removing measure-

ment items was influenced by heuristics developed by

Measuring eGovernment success Murray Scott et al 195

European Journal of Information Systems

Straub et al (2004): factors with eigenvalues greater than 1.0, items with communalities of 0.60 and items with factor loadings of 0.65 or greater were retained. Based on the factor analysis, nine factors were identified from the original 11 dimensions of eGovernment Net Benefits consisting of 30 items overall; namely, Cost, Time, Perso- nalisation, Communication, Convenience, Ease of Infor- mation Retrieval, Trust, Well-Informedness and Participate in Decision-Making (see Table 3). All remaining items loaded significantly on their respective factor, indicating unidimensionality and the absence of cross-loadings sup- porting preliminary discriminant validity of the scale. This factor model accounted for 87% of cumulative variance. Furthermore, the reliability coefficients were acceptable at above 0.74, indicating preliminary reliability (Nunnally, 1978). The results of the factor analysis revealed a different

factor model (nine dimensions) for the Net Benefits con- struct than that of the proposed post-content validation model (11 dimensions). The process of retaining and

removing items was guided by statistical analysis from the exploratory phase and also by ensuring that the resulting items were consistent with theory and the conceptual domain of the dimensions (MacCallum, 1986). The dimension entitled ‘Control’ for example was removed due to insignificant loadings and cross loadings for many of the items. However, based on significant cross-loading scores, one item originally associated with the ‘Control’ dimension was included in the Convenience dimension to form a three-item factor. This revised factor had an accep- table eigenvalue of 1.298 and accounted for 5.901% of variance of the factor model. The coefficient score for this factor also displayed a significant inter correlation value of 0.748. The inclusion of this item, which refers to an action that provides a convenience-based benefit to the user, was conceptually related to the Convenience dimension and consistent with the theoretical domain of this construct. The dimension ‘Avoid Personal Interaction’ was also

removed during exploratory factor analysis, although one of its items significantly cross-loaded with the

Table 3 Summary of items and factor loadings for varimax orthogonal nine-factor solution for e-Government net benefits

Factor

Coma 1 2 3 4 5 6 7 8 9

Cost 1 0.930 0.921 Cost 2 0.827 0.818 Cost 3 0.819 0.761 Time 1 0.814 0.790 Time 2 0.731 0.800 Time 3 0.773 0.837 Time 4 0.851 0.814 Convenience 1 0.832 0.792 Convenience 2 0.796 0.881 Convenience 3 0.701 0.712 Personalisation 1 0.902 0.761 Personalisation 2 0.942 0.908 Personalisation 3 0.968 0.930 Communication 1 0.885 0.879 Communication 2 0.895 0.891 Communication 3 0.913 0.903 Information Retrieval 1 0.821 0.664 Information Retrieval 2 0.881 0.745 Information Retrieval 3 0.775 0.752 Trust 1 0.771 0.840 Trust 2 0.846 0.837 Trust 3 0.902 0.885 Trust 4 0.884 0.901 Well-informedness 1 0.889 0.876 Well-informedness 2 0.916 0.929 Well-informedness 3 0.835 0.896 Participate 1 0.875 0.896 Participate 2 0.903 0.916 Participate 3 0.882 0.911 Participate 4 0.924 0.921

aCommunalities (n = 90). Note: Factor loadings below 0.40 are not shown.

Measuring eGovernment success Murray Scott et al196

European Journal of Information Systems

Time dimension. A new Time dimension was formed with three original items along with the item originally from Avoid Personal Interaction. This set of items displayed significant convergent loadings and communality values. This factor had an acceptable eigenvalue of 1.158 and accounted for 3.859% of variance of the factor model. The coefficient score for this construct also displayed a significant inter correlation value of 0.884. The new Time item can further be logically understood as providing a time-based benefit for the user wishing to avail of the advantages of an automated service from the website instead of having to deal with an official. Therefore this adjustment also displays face and logical validity. All items measuring Cost, Communication, Personalisa-

tion, Ease of Information Retrieval, Trust, Well-Informed- ness and Participate in Decision-Making loaded on their respective factors; thus, each retained their original title based on the post-content analysis and expert review stage with a minimum of three items for each dimension. These results provide initial evidence of discriminant

and convergent validity (Straub, 1989; Straub et al, 2004) and suggest that the nine-factor model is a reliable and valid starting point for the measurement of eGovernment Net Benefits. The items and dimensions relating to the nine-factor model are displayed in Table 4.

Confirmatory analysis – main survey The set of 30 validated items from the exploratory assess- ment were administered in the main survey conducted with students, faculty and staff from eight participating Universities in the U.S.A. New respondents were drawn from undergraduate, graduate and postgraduate students, staff and faculty from a broad variety of disciplines. Student and faculty groups interested in the area of democracy studies at the targeted institutions were also included in the survey invitation. In total, the main survey generated 383 responses of which 347 were complete and usable. All 347 respondents indicated recent use (less than 6 months) of the pre-defined survey websites extended from the exploratory phase to include 10 websites: U.S.A. gov, Whitehouse.gov, Regulations.gov, ED.gov, IRS.gov, SSA.gov, State.gov, Defence.gov, Fafsa.gov, USAjobs.gov. Similar to the pilot stage, it was confirmed that each of these websites also provided a range of Web 2.0 function- alities, for example, forums, blogs, chat rooms and other social networking media, in addition to traditional inter- net-based services. Respondents were also required to indicate their level of experience with various eGovern- ment activities in order to assess which user type (Passive, Active or Participatory) was appropriate. The sample size was considered good, providing a ratio of 38.5 responses per construct (Netemeyer et al, 2003; Lewis et al, 2005; Hair et al, 2006). The sample was once again well distributed in terms of age demographics: 18-24 (35%); 25-34 (21%); 35- 44 (11%); 45-54 (17%); 55-64+ (16%). Confirmatory Factor Analysis (CFA) was performed on

the 347 responses using the maximum likelihood method,

which detects the unidimensionality of each factor, indi- cating the presence of a single construct underlying a set of measures (Anderson & Gerbing, 1988). Following Byrne (2010) and Hair et al (2006), a structural measurement model was developed identifying nine first-order factors that were correlated, each item having a non-zero loading on its designated factor and zero loadings on other factors. The measurement error terms associated with each item were uncorrelated. The hypothesised nine-factor model achieved good fit

(χ2/df = 2.864, GFI = 0.902, RMSEA = 0.077, RMR = 0.12, NFI = 0.904, CFI = 0.934, TLI = 0.921, AGFI = 0.847); the normed chi-square value suggested reasonable fit and all of the other indices were within threshold values. Convergent, discriminant and nomological validities

were checked using the heuristics outlined by Straub et al (2004). Factor loadings, average variance extracted (AVE) and reliability values are presented as indicators of con- vergent validity. Table 5 displays the Cronbach alphas for the main data set; these scores were well above the thresh- old (0.70) for confirmatory research (Nunnally, 1978). The values for the AVE for each factor were also used to assess the internal consistency of each construct. Table 5 indi- cates that all factors met the recommended threshold for AVE (0.50), demonstrating strong reliability levels and good convergence (Hair et al, 2006). Table 6 displays good standardised factor loadings for

each item and construct, indicating that they converge on a common point, the latent construct. All factor loadings exceed the recommended threshold of 0.7 (Hair et al, 2006). The correlation coefficients among all possible pairs

in the Net Benefits construct are presented in Table 7, providing further evidence of construct validity. Statisti- cal significance among all possible pairs of factors indi- cates that all the factors are related to one another and thus measuring aspects of the same construct (Hair et al, 2006). Discriminant validity is the extent to which a construct

is distinct from other constructs and gives evidence that the construct captures some phenomena other measures do not. Table 8 displays the squared correlation estimate for each construct pair and the corresponding AVE value; the results demonstrate that the AVE exceeds all combina- tions of shared variance; therefore, discriminant validity for each of the Net Benefit dimensions is accepted (Fornell & Larcker, 1981; Hair et al, 2006). The assessment of nomological validity is a recom-

mended test in construct evaluation and aims to confirm the behaviour of the newly created construct within a wider theoretical context or network of constructs (Bagozzi, 1981). Accordingly, a multiple regression model was constructed to test the nomological validity of the Public Value Net Benefits scale. This study assumed that positive relationships existed between the Net Benefits construct and Information Quality, System Quality and Service Quality based on prior research in the eGovern- ment domain (e.g. Prybutok et al, 2008; Teo et al, 2008;

Measuring eGovernment success Murray Scott et al 197

European Journal of Information Systems

Wang & Liao, 2008). Using items from previous empirical research in this area, data was collected for each of the independent variables (Information, System and Service Quality) during the survey collection stage. The results

show that Information Quality (β = 0.464; P<0.000), Sys- tem Quality (β = 0.134; P<0.01) and Service Quality (β = 0.360; P < 0.000) participate in a combined significant relationship (R2 = 0.733) with the Net Benefits scale. This

Table 4 Net benefit dimensions and items

Cost 1. Using this government website saves me money. 2. Using this government website reduces the cost of providing the service. 3. I value the cost savings from using this website.

Time

1. Using the website saves me time. 2. This website provides a quicker response to a question or request than other means (e.g. offline interaction). 3. I can accomplish things more quickly because of using this website. 4. Using this government website enables me to avoid having to deal directly with government staff.

Convenience

1. It is important that I can use this website around the clock. 2. It is important that I can access this website from a number of different locations (e.g. home, work, library, post office). 3. This website allows me to terminate what I am doing at any time.

Personalisation

1. I am able to personalise the services offered by this website. 2. I value the personalised services offered by this website. 3. I value the personalised aspects of this website.

Communication

1. Using this website is an efficient way of communicating with government departments. 2. Using this website is a valuable way of communicating with government departments. 3. Using this website is an effective way of communicating with government departments.

Ease of information retrieval

1. This website contains a lot of useful information about government services. 2. This website helps me to understand more about government services. 3. This website answers any queries I might have about government services.

Trust

1. I feel that this website acts in citizens' best interests. 2. I feel comfortable interacting with this website since it generally fulfils its duties efficiently. 3. I always feel confident that I can rely on this website to do its part when I interact with it. 4. I am comfortable relying on the website to meet its obligations.

Well-informedness

1. This website increases my understanding of issues. 2. This website enables me to build up knowledge about issues that are important to me. 3. Because of using this website, I am better informed in general.

Participate in decision-making

1. This website allows me to have my say about things that matter to me. 2. This website enhances my feeling of being part of an active democracy. 3. This website makes me feel that decision-makers listen to me. 4. This website makes me feel that I am being consulted about important issues.

Measuring eGovernment success Murray Scott et al198

European Journal of Information Systems

finding corroborated that of prior research, thereby con- firming the nomological validity of the Net Benefits scale.

Second-order factor model of net benefits A higher-order factor model was created, comprising a second-order factor onto which the nine first-order factors were loaded. This model tests the extent to which the first-order factors were accentuated by the second-order factor and in doing so determines the presence of the theoretical Net Benefits construct and the inter-relationship with the nine first- order dimensions. The second-order model resulted in fit indices values that were within, or close to the threshold values (χ2/df = 2.996, GFI = 0.89, RMSEA = 0.081, RMR = 0.14, NFI = 0.877, CFI = 0.904, TLI = 0.901, AGFI = 0.811). Thus, both the original and the second-order models were adequate for representing the underlying structure of the Net Benefits measure. Although the fit of the higher-order model can never

surpass the original model, it explains the co-variation among first-order factors in a more economical way (Stewart & Segars, 2002). The efficacy of the second-order model can be evaluated through the target (T) coefficient, a value that assesses how effectively the relationships among first-order factors are captured by the higher-order factor. In this case, a target coefficient of 0.873 (the ratio of the χ2 of the original model to the higher-order model) suggests the existence of the second-order factor model and indicates that 87.3% of the variation of the original model was explained by the second-order factor. Hence, the factor co-variances in the original model can be represented in a more parsimonious manner by the higher-order model. As Figure 1 displays, the magnitude for all paths between the higher-order factor and all first- order factors are significant ( > 0.5) providing further evi- dence of the viability of the higher order model. These findings confirm the validity of the 30 items

representing the nine dimensions of Net Benefits pro- posed by the exploratory factor analysis and presented in Table 4.

Public value net benefits for eGovernment user groups In order to explore the multi-dimensionality of the Net Ben- efits construct the CFA process was repeated for three pre- defined eGovernment user groups (Passive, Active and Participatory). The main sample was distinctly categorised into each group using data provided by each respondent for their chosen website: the functionality used on the nominated website; and the self-reported experience level for various eGovernment usage activities. This process ensured that only the most experienced users were inclu- ded in each of the groups. Of the total sample of 347 responses, 105 were Passive users, 125 were Active users and 117 were Participatory users. Specific Net Benefits factor structures were hypothesised

for each User group based on theoretical and logical propositions for each group (see Table 9). For example,

Table 5 Assessment of internal consistency

Construct Number of items

Cronbach's α ( > 0.70)

AVE ( > 0.50)

Cost (3) 0.918 0.799 Time (4) 0.923 0.752 Personalisation (3) 0.930 0.819 Communication (3) 0.934 0.824 Ease of information retrieval

(3) 0.884 0.743

Convenience (3) 0.847 0.675 Trust (4) 0.938 0.795 Well-informedness (3) 0.926 0.810 Participate in decision- making

(4) 0.952 0.853

Table 6 Assessment of convergent validity

Construct Item Factor loading

Cost NBC1 0.91 NBC2 0.87 NBC3 0.91

Time NBT1 0.83 NBT2 0.87 NBT5 0.93 NBAPI1 0.87

Personalisation NBP2 0.88 NBP3 0.93 NBP4 0.92

Communication NBCOMM1 0.92 NBCOMM2 0.88 NBCOMM3 0.92

Ease of information retrieval NBEIR1 0.89 NBEIR2 0.92 NBEIR3 0.77

Convenience NBCN1 0.84 NBCN2 0.89 NBCON1 0.73

Trust NBTR1 0.79 NBTR2 0.88 NBTR3 0.95 NBTR4 0.93

Well-informedness NBWI2 0.87 NBWI4 0.96 NBWI6 0.85

Participate in decision-making NBPDM2 0.86 NBPDM3 0.88 NBPDM4 0.97 NBPDM5 0.97

Measuring eGovernment success Murray Scott et al 199

European Journal of Information Systems

Personalisation benefits require interaction with the web- site and are thus not an outcome benefit possible for a passive user, who is defined as only browsing or downloading information. As well as logical inferences, the theoretical underpinning of Public Value was central to considering the combination of individual factor structures. Multiple regression models were also conducted for each

user group providing further empirical evidence for pro- posing particular Net Benefits dimensions for each user group. For each group, an overall User Satisfaction con- struct was used to identify those Net Benefit factors that significantly correlated with their value perceptions of the website. The results of these analyses are presented in Tables 10, 11 and 12 CFA was repeated for each of the three user groups

following the same structure employed for the overall Net Benefits construct presented above; accordingly, model fit

indices and full construct validities were assessed for each hypothesised model. Construct reliability, convergence and discriminant validity tests yielded satisfactory or good results for each of the three hypothesised models. Table 13 provides a summary of the model fit indices, which confirm the existence of specific factor structures for three eGovernment user groups: Passive, Active and Participatory.

Implications for research This research provides important new knowledge for IS Success research by developing and empirically validating the first comprehensive Net Benefits construct to measure the success of eGovernment. In addition, a valuable con- tribution is the application of Public Value theory to extend the DeLone and McLean Model of IS Success to measure the Net Benefits of eGovernment. This study

Table 7 Assessment of correlation coefficients

Estimate P

Cost <– > Time 0.466 *** Cost <– > Convenience 0.115 ** Cost <– > Personalisation 0.918 *** Cost <– > Communication 0.910 *** Cost <– > Information_Retrieval 0.649 *** Cost <– > Trust 0.882 *** Cost <– > Well-Informedness 0.396 *** Participation <– > Cost 0.818 *** Time <– > Convenience 0.354 *** Time <– > Personalisation 0.616 *** Time <– > Communication 0.793 *** Time <– > Information_Retrieval 0.570 *** Time <– > Trust 0.708 *** Time <– > Well-Informedness 0.504 *** Participation <– > Time 0.252 ** Convenience <– > Personalisation 0.025 ** Convenience <– > Communication 0.255 *** Convenience <– > Information_Retrieval 0.213 *** Trust <– > Convenience 0.286 *** Convenience <– > Well-Informedness 0.192 ** Participation <– > Convenience 0.056 0.073 Personalisation <– > Communication 0.036 *** Personalisation <– > Information_Retrieval 0.525 *** Trust <– > Personalisation 0.772 *** Personalisation <– > Well-Informedness 0.567 *** Participation <– > Personalisation 0.787 *** Communication <– > Information_Retrieval 0.819 *** Trust <– > Communication 0.667 *** Communication <– > Well-Informedness 0.560 *** Participation <– > Communication 0.932 *** Trust <– > Information_Retrieval 0.808 *** Information_Retrieval <– > Well-Informedness 0.616 *** Participation <– > Information_Retrieval 0.409 ** Trust <– > Well-Informedness 0.529 *** Participation <– > Trust 0.619 *** Participation <– > Well-Informedness 0.827 ***

Measuring eGovernment success Murray Scott et al200

European Journal of Information Systems

addresses the call for research by Petter et al (2012) by developing IS Success measures that firstly assess the social impact of IS systems and secondly, accounts for the multi- ple uses of contemporary systems including participatory activities via Web 2.0. Table 14 demonstrates how the Public Value framework of Efficiency, Effectiveness and Social Impact has extended IS Success research to include the value perceptions of Passive, Active and Participatory users. As a result, this study has several implications for the development of eGovernment research and IS Success research in general. This research demonstrates that measuring success in

eGovernment requires multi-dimensional constructs in order to accurately reflect value perceptions stemming from contemporary internet-based systems. Furthermore, success should be understood not just in terms of service- based efficiencies but should properly reflect the personal and societal impact of technology. Findings from the main

survey sample show that citizens perceive more value in benefits such as Trust, Well-Informedness, Participation and Personalisation, than in Convenience or Time saving benefits. The former dimensions together account for 66% of cumulative variance, while the remaining five dimen- sions account for 22% of cumulative variance. This suggests that although success dimensions such as Con- venience and Time are important, experienced eGovern- ment users possess more sophisticated value perceptions than just efficiency and effectiveness. Similarly sophisti- cated value perceptions have been identified in eCom- merce users (Keeney, 1999; Wang, 2008; Mosse & Whitley, 2009) and while there are contextual and operational differences between public and private sector domains, it is likely unavoidable that users' experience of service provision in one area will likely influence their perceptions and expectations in another (Kolsaker & Lee-Kelley, 2008; Venkatesh et al, 2011; Tan et al, 2013). Therefore, the

Table 8 Assessment of discriminant validity

Construct Construct Squared correlation estimate Average variance extracted (AVE)

Cost <– > Time 0.339 0.799/0.752 Cost <– > Convenience 0.102 0.799/0.675 Cost <– > Personalisation 0.444 0.799/0.819 Cost <– > Communication 0.496 0.799/0.824 Cost <– > Information_Retrieval 0.409 0.799/0.743 Cost <– > Trust 0.558 0.799/0.795 Cost <– > Well-Informedness 0.251 0.799/0.810 Participation <– > Cost 0.370 0.853/0.799 Time <– > Convenience 0.439 0.752/0.675 Time <– > Personalisation 0.415 0.752/0.819 Time <– > Communication 0.603 0.752/0.824 Time <– > Information_Retrieval 0.501 0.752/0.743 Time <– > Trust 0.625 0.752/0.795 Time <– > Well-Informedness 0.446 0.752/0.810 Participation <– > Time 0.159 0.853/0.752 Convenience <– > Personalisation 0.021 0.675/0.819 Convenience <– > Communication 0.238 0.675/0.824 Convenience <– > Information_Retrieval 0.230 0.675/0.743 Trust <– > Convenience 0.308 0.795/0.675 Convenience <– > Well-Informedness 0.208 0.675/0.810 Participation <– > Convenience 0.043 0.853/0.675 Personalisation <– > Communication 0.523 0.819/0.824 Personalisation <– > Information_Retrieval 0.307 0.819/0.743 Trust <– > Personalisation 0.452 0.795/0.819 Personalisation <– > Well-Informedness 0.333 0.819/0.810 Participation <– > Personalisation 0.748 0.853/0.819 Communication <– > Information_Retrieval 0.541 0.824/0.743 Trust <– > Communication 0.705 0.795/0.824 Communication <– > Well-Informedness 0.371 0.824/0.810 Participation <– > Communication 0.440 0.853/0.824 Trust <– > Information_Retrieval 0.618 0.795/0.743 Information_Retrieval <– > Well-Informedness 0.472 0.743/0.810 Participation <– > Information_Retrieval 0.224 0.853/0.743 Trust <– > Well-Informedness 0.407 0.795/0.810 Participation <– > Trust 0.339 0.853/0.795 Participation <– > Well-Informedness 0.454 0.853/0.810

Measuring eGovernment success Murray Scott et al 201

European Journal of Information Systems

findings from this research have implications for users of business websites that are designed to encourage customer engagement and participation and we encourage research- ers to test the Public Value Net Benefits construct in business Web 2.0 environments. Citizens’ interaction with eGovernment has become

increasingly complex – some uses can be functional or utilitarian, for example in paying a fine or requesting a service; alternatively, there are increasing opportunities for

citizens to participate in initiatives designed to engage them in democratic endeavours (Gagnon et al, 2010; Ahn & Bretschneider, 2011; Lee et al, 2011; Avgerou, 2013). This study demonstrates that different uses of an eGovern- ment website necessarily contribute to varying value per- ceptions in citizens; some uses result in basic benefits such as time or cost savings, while others may result in more hedonistic or complex value perceptions such as trust. A major contribution of this research, therefore, is the

Figure 1 Second-order factor model for net benefits.

Table 9 Net benefits dimensions according to user type

Passive Active Participatory

Time Time Time Trust Trust Trust Convenience Convenience Convenience Ease of information retrieval

Ease of information retrieval

Participate in decision- making

Well-informedness Cost Well-informedness Personalisation Communication

Table 10 Regression analysis (passive group)

Net benefits User satisfaction R2 = 0.804; F = 98.654

Time β = 0.110; P = 0.000 Ease of information retrieval

β = 0.149; P = 0.001

Convenience β = 0.049; P = 0.05 Trust β = 0.292; P = 0.000 Well-informedness β = 0.215; P = 0.03

Measuring eGovernment success Murray Scott et al202

European Journal of Information Systems

creation of multi-dimensional success measures that effec- tively capture net benefits or value in differing Internet usage contexts. This study extended a previous classification of Internet

users (Teo et al, 2008) to include Participation, which is warranted in the era of Web 2.0. Findings from this study support this extension and further contribute to knowl- edge by providing benefit measures categorised against each usage type: Passive, Active and Participatory. The web sites chosen for this research all possessed the func- tional capability to enable a user to partake in the website passively, actively or in a participatory fashion. The results demonstrate that each of the user groups obtained

significant benefits but from dimensions of benefits based on their specific use of the websites. A key contribution of this research is to show that the relative importance of Net Benefits changed for each usage group; for the Passive users the benefit of Well-Informedness was the construct that explained the largest single amount of variance, for Active users it was Personalisation and for Participa- tory users it was Participating in Decision-Making (see Table 15). In addition, the methodological approach developed in this study to identify specific user types provides a generalisable approach for future research asses- sing public values and is an important, practical contribu- tion to research.

Implications for practice The 30-item eGovernment Net Benefits scale is an acces- sible and easily administered measurement instrument that can be used to evaluate the performance of eGovern- ment websites and examine specific dimensions of per- ceived benefits. As such, this instrument offers a practical means for public sector organisations to evaluate the success of initiatives that may have complex expected

Table 12 Regression analysis (participatory group)

Net benefits User satisfaction R2 = .797; F = 153.800

Time β = 0.132, P = 0.05 Convenience β = 0.116, P = 0.05 Trust β = 0.398, P = 0.000 Well-Informedness β = 0.190, P = 0.000 Participate β = 0.390, P = 0.000

Table 11 Regression analysis (active group)

Net benefits User satisfaction R2 = 0.794; F = 79.887

Cost β = 0.209; P = 0.02 Time β = 0.228; P = 0.001 Personalisation β = 0.415; P = 0.05 Communication β = 0.222; P = 0.001 Ease of information retrieval

β = 0.292; P = 0.007

Convenience β = 0.110; P = 0.01 Trust β = 0.551; P = 0.000

Table 13 Model fit test results of net benefit construct

Index/ Test

Threshold Complete model

Passive group

Active group

Participant group

χ2 Smaller is better

1056.92 240.715 462.533 297.497

df 369 109 209 125 P-value P > 0.05 0.000 0.000 0.000 0.000 χ2/df 1<χ2/df <3 2.864 2.208 2.213 2.380 GFI > 0.90 0.902 0.945 0.932 0.911 RMSEA <0.08 0.077 0.068 0.078 0.072 RMR <0.10 0.11 0.11 0.11 0.13 NFI > 0.90 0.904 0.920 0.914 0.917 CFI > 0.90 0.934 0.954 0.938 0.937 TLI > 0.90 0.921 0.936 0.918 0.913 AGFI > 0.80 0.817 0.896 0.897 0.857

Table 14 Extended net benefits by social impact and participatory user (shown in grey)

Efficiency Effectiveness Social impact

Passive Time Convenience Trust Ease of information retrieval

Well-informedness

Active Time Convenience Trust Cost Personalisation Communication Ease of

information retrieval

Participatory Time Convenience Trust Well-informedness Participate – in - decision-making

Table 15 Net benefits dimensions according to user type

Passive a Active a Participatory a

Well-informedness (35)

Personalisation (28) Participate in decision- making (27)

Trust (15) Trust (20) Trust (23) Ease of information retrieval (12)

Communication (10) Well-informedness (12)

Time (9) Time (8) Time (9) Convenience (6) Cost (6) Convenience (8)

Ease of information retrieval (4) Convenience (4)

aPercentage variance explained.

Measuring eGovernment success Murray Scott et al 203

European Journal of Information Systems

outcomes or benefits for citizens. Also, since the scale has been divided according to user types, the implementation of the instrument can be made more precise to the website purpose. For each usage group or exchange scenario, governments can identify the expected success measures that will influence citizen value perceptions. Strategically, public organisations can therefore focus on creating Public Value through certain web-based initiatives with a degree of certainty as to the resulting benefits perceived by citizens. Government managers can also administer the survey

over successive periods to evaluate performance according to evolutionary models for eGovernment implementation. This enables an assessment of success to be made at successive stages of implementation. Further, the identifi- cation of specific constructs enables the organisation to strategically manage and control the most important aspects of the website during design and implementation. The eGovernment Net Benefits model is constructed

from a variety of success measures, some relating to efficiency, effectiveness and others improved democratic functions. In order to achieve or create value in the public sector, success should be defined according to a blend of these factors. The implication for governments is that citizens view intangible benefits or outcomes as equally important as tangible gains, such as efficiency improve- ments. This requires a view in government that improve- ments in effectiveness and democratic functions are both important to their citizens. This is a more sophisticated view of the role that IT contributes within public sector organisations. IT initiatives measured by ROI methodolo- gies alone only partially tap the potential value of invest- ments to society. The results of this research point to the legitimacy of

citizens in the creation and determination of Public Value. In the context of electronic service delivery, this implies that government agencies be responsive and inclusive of citizens. Rhodes and Wanna (2007) argue that part of the reason for the growing popularity of Public Value is that it restores a more positive view of government and public service provision than previously felt under private-sector styled reforms. Some normality is restored to government action as a return to legitimate public activities is promised through policies emphasising collective preferences and shared aspirations (Smith, 2004; Alford & Hughes, 2008). As Public Value takes hold through policy formulation and implementation, the Net Benefits measure developed in this research is well positioned to aid managers in the evaluations of the value of their eGovernment initiatives.

Limitations and directions for future research The combination of dimensions representing Net Benefits have been constructed and assembled by this study for the first time and further studies are needed to further validate the generalisability, strength and soundness of the mea- sures. Thus, the present study identifies the significant dimensions and associations as a foundation for future

research, which may refine the measures and their relative importance to citizen users. One limitation of this study is its generalisability beyond

the United States. The eGovernment 2.0 systems that were evaluated are U.S. Federal government websites and the citizen users were users of U.S. eGovernment services. These findings may not be applicable to eGovernment services in other countries where the nature of the services and citizen expectations may differ. Therefore an impor- tant follow-up to this research will be studies focused on measuring eGovernment 2.0 success in other countries. This research focused on a representative and compre-

hensive number of Federal level, citizen-facing websites; further research should extend the research to other levels of government, for example, State, County and Local level. Within the context of the U.S. however, this provides a significant challenge for researchers as previous research has identified significant degrees of variation among state and local government websites, in terms of service quality, website sophistication and level of service provision (Norris & Moon, 2005). U.S. Federal Government Depart- ments, on the other hand, display more commonality as they are subject to common strategic service goals and mandates (Morgeson & Mithas, 2009) and have eGovern- ment initiatives controlled and evaluated by a common source (Office of Management and Budget and General Services Administration). This research provides a rigorous set of measures to initiate that process and recommends that future studies also attempt to be representative in evaluating a full range of eGovernment services. The measures for Public Value Net Benefits were con-

ceptualised so as to be applicable in new and emerging usage contexts and as technologies continually advance, the conceptual measures should be evaluated using new technologies (e.g. mobile usage) in order to facilitate theory development and to ensure the efficacy of the constructs to explain contemporary phenomenon. In order to measure Net Benefits comprehensively, the

survey sample gathered users who were highly experi- enced in a range of successful eGovernment services. The resulting dimensions of success are necessarily limited to this sample, characterised as ‘lead users’. Future research studies should attempt to include a broad range of user experience to determine robustness of the Public Value Net Benefits construct. While this is a recognised limita- tion, this study is however motivated by calls for research to advance knowledge in new areas such as Web 2.0 (Majchrzak, 2009; Petter et al, 2012) because accurate evaluations of these initiatives will help governments improve the efficacy of their use and impact to society.

Conclusion Successful Internet-based IS are critical to the development of viable public and private net-enabled enterprises. In measuring eGovernment success, an essential task is to identify the key dimensions that encompass citizen Net Benefits and to develop validated instruments to measure

Measuring eGovernment success Murray Scott et al204

European Journal of Information Systems

them. Yet, current studies on IS Success do not account for the complex, multidimensional nature of Net Benefits in Internet-based systems. This study adopted a new approach to IS Success by incorporating Public Value theory in construct development, resulting in a balanced set of dimensions that include utilitarian benefits of service efficiency but also capture the participative ele- ments present in social media and hence, are reflective of contemporary Internet-based information systems. This study proposed and empirically validated measures

for a multidimensional definition of Public Value Net Benefits comprised of nine cross-disciplinary, literature- grounded sub-constructs. The dimensions of the extended Net Benefits construct include: Cost, Time, Convenience, Personalisation, Communication, Information Retrieval, Trust, Well-Informedness and Participation. In addition, the competing model analysis also revealed the existence of the second-order factor model for the Public Value construct, a unique finding among eGovernment or IS success studies. These findings confirm the view that

Public Value Net Benefits is not a single construct. In doing so, the findings provide a better understanding of the multi-dimensionality of the success of eGovernment. The proposed set of measures in this study yield a strong representation of what experienced citizens value in suc- cessful eGovernment services, statistically explaining 87% of variance in response from this particular sample. In conclusion, this study extends our understanding of

IS Success to Internet-based Information Systems by devel- oping and validating the first comprehensive Net Benefits construct based on Public Value theory. This research represents the first attempt to explore what citizens value in their interactions with eGovernment, by fully capturing the expectations of citizens in passive, active and partici- patory usage roles and in doing so, creates and validates a comprehensive Success model. The validated Public Value Net Benefits construct has important implications for and application to both public and private sector contexts where the users expect more than simply utilitarian value from their usage experience.

About the authors

Murray Scott is a lecturer in Business Information Systems at the J.E. Cairnes School of Business & Economics, National University of Ireland Galway, Ireland. His main research interests lie in IS success, egovernment and innovation in public sector information systems. He obtained a Ph.D. from NUI Galway and has Masters degrees in English and Philosophy from the University of Dundee, Scotland.

William H. DeLone received the BS degree in mathe- matics from Villanova University, the MS degree in indus- trial administration from Carnegie-Mellon University, and the Ph.D. degree in computer information systems from the University of California Los Angeles. He is currently a professor of information technology in the Kogod School

of Business at the American University, Washington, DC. His current research interests include the assessment of information systems’ effectiveness and egovernment and public value.

Willie Golden is Professor of Information Systems at NUI Galway, Ireland. Previously he has held positions as Dean and Director of a research institute at NUI, Galway. He has co-authored a book, contributed numerous chapters to other texts and published journal papers in among others, Information Systems Journal, Omega; The International Jour- nal of Management Science, International Journal of Electronic Commerce, Journal of End User Computing and Journal of Decision Systems.

References ADAMS DA, NELSON RR and TODD PA (1992) Perceived usefulness, ease of

use, and usage of information technology: a replication. MIS Quarterly 16(2), 227–247.

AHN M and BRETSCHNEIDER S (2011) Politics of e-government: e-govern- ment and the political control of bureaucracy. Public Administration Review 71(3), 414–424.

AHN MJ (2011) Adoption of e-communication applications in US munici- palities: the role of political environment, bureaucratic structure, and the nature of applications. American Review of Public Administration 41(4), 428–452.

AL-KIBISI G, DE BOER K, MOURSHED M and REA N (2001) Putting citizens on- line, not inline. The McKinsey Quarterly, Special Edition 2, 64.

ALFORD J and HUGHES O (2008) Public value pragmatism as the next phase of public management. American Review of Public Administration 38(2), 130–148.

ALOMARI M, WOODS P and SANDHU K (2012) Predictors for e-government adoption in Jordan deployment of an empirical evaluation based on a citizen-centric approach. Information Technology & People 25(2), 207–234.

ANDERSEN K-V and HENRIKSEN HZ (2005) The first leg of eGovernment research: domains and application areas 1998–2003. International Journal of Electronic Government Research 1(4), 26–44.

ANDERSEN K-V, HENRIKSEN HZ, MEDAGLIA R, DANZIGER J, SANNARNES M and ENEMAERKE M (2010) Fads and facts of e-government: a review of impacts of e-government (2003–2009). International Journal of Public Administration 33(11), 564–579.

ANDERSON JC and GERBING DW (1988) Structural equation modeling in practice: a review and recommended two-step approach. Psychological Bulletin 103(3), 411–423.

AVGEROU C (2013) Explaining trust in IT-mediated elections: a case study of e-voting in Brazil. Journal of the Association for Information Systems 14(8), 420–451.

BAGOZZI RP (1981) Evaluating structural equation models with unobser- vable variables and measurement error: a comment. Journal of Market- ing Research 18(3), 375–381.

BARBOSA AF, POZZEBON M and DINIZ EH (2013) Rethinking e-government performance assessment from a citizen perspective. Public Administra- tion 91(3), 744–762.

BARNES S and VIDGEN R (2003) Measuring web site quality improvements: a case study of the forum on strategic management knowledge exchange. Industrial Management & Data Systems 103(5), 297–309.

BARNES S and VIDGEN R (2006) Data triangulation and web quality metrics: a case study in e-government. Information & Management 43(6), 767–777.

Measuring eGovernment success Murray Scott et al 205

European Journal of Information Systems

BARTIS E and MITEV N (2008) A multiple narrative approach to information systems failure: a successful system that failed. European Journal of Information Systems 17(2), 112–124.

BAUMGARTEN J and CHUI M (2009) E-Government 2.0. McKinsey Quarterly 4, 26–31.

BELANCHE D, CASALO LV, FLAVIAN C and SCHEPERS J (2014) Trust transfer in the continued usage of public e-services. Information & Management 51(6), 627–640.

BELANGER F and CARTER L (2008) Trust and risk in e-government adoption. Journal of Strategic Information Systems 17(2), 165–176.

BELANGER F and CARTER L (2012) Digitizing government interactions with constituents: an historical review of e-government research in information systems. Journal of the Association for Information Systems 13(5), 363–394.

BENINGTON J (2011) From private choice to public value? In Public Value: Theory and Practice (BENINGTON J and MOORE M, Eds), Palgrave Macmil- lan, Basingstoke, UK.

BENINGTON J and MOORE M (Eds) (2010) Public Value: Theory and Practice. Palgrave Macmillan, Basingstoke, UK.

BERK R (1990) Importance of expert judgement in content-related validity evidence. Western Journal of Nursing Research 12(5), 659–671.

BROWN M (2007) Understanding e-government benefits: an examination of leading-edge local governments. American Review of Public Adminis- tration 37(2), 178–197.

BRYSON J, CROSBY B and BLOOMBERG L (2014) Public value governance: moving beyond traditional public administration and the new public management. Public Administration Review 74(4), 445–456.

BYRNE B. (2010) Structural Equation Modeling with AMOS. Lawrence Erlbaum Associates, Mahwaw, New Jersey.

CAMPBELL DA, LAMBRIGHT KT and WELLS CJ (2014) Looking for friends, fans, and followers? Social media use in public and nonprofit human services. Public Administration Review 74(5), 655–663.

CAPGEMINI (2007) The User Challenge Benchmarking The Supply of Online Public Services. Capgemini.

CARTER L and BELANGER F (2005) The utilization of e-government services: citizen trust, innovation and acceptance factors. Information Systems Journal 15(1), 5–25.

CHAN FKY, THONG JYL, VENKATESH V, BROWN SA, HU PJ-H and TAM KY (2010) Modeling citizen satisfaction with mandatory adoption of an e-govern- ment technology. Journal of the Association for Information Systems 11(10), 519–549.

CHURCHILL G (1979) A paradigm for developing better measures of marketing constructs. Journal of Marketing Research 16(1), 64–73.

COLEMAN S (2004) Connecting parliament to the public via the internet. Information, Communication & Society 7(1), 1–22.

COLEMAN S (2005) The lonely citizen: indirect representation in an age of networks. Political Communication 22(2), 197–214.

CONNOLLY R, BANNISTER F and KEARNEY A (2010) Government website service quality: a study of the Irish revenue online service. European Journal of Information Systems 19(6), 649–667.

CORDELLA A and BONINA CM (2012) A public value perspective for ICT enabled public sector reforms: a theoretical reflection. Government Information Quarterly 29(4), 512–520.

CULNAN MJ, MCHUGH PJ and ZUBILLAGA JI (2010) How large U.S. companies can use twitter and other social media to gain business value. MIS Quarterly Executive 9(4), 243–259.

DAS T and TENG B-S (2001) Trust, control and risk in strategic alliances. Organizational Studies 22(2), 251–283.

DELONE WH and MCLEAN ER (1992) Information systems success – the quest for the dependent variable. Information Systems Research 3(1), 60–95.

DELONE WH and MCLEAN ER (2003) The DeLone and McLean model of information systems success: a ten-year update. Journal of Management Information Systems 19(4), 9–30.

DEVELLIS RF (1991) Scale Development: Theory and Applications. Sage, Newbury Park, CA.

DILLMAN D (2007) Mail and Internet Surveys: The Tailored Design Method, 2nd edn, Wiley.

FORNELL C and LARCKER DF (1981) Evaluating structural equation models with unobservable variables and measurement error. Journal of Market- ing Research 18(February), 39–50.

FU J-R, CHAO W-P and FARN C-K (2004) Determinants of taxpayers' adoption of electronic filing methods in Taiwan: an exploratory study. Journal of Government Information 30(5–6), 658–683.

FU J-R, FARN C-K and CHAO W-P (2006) Acceptance of electronic tax filing: a study of taxpayer intentions. Information & Management 43(1), 109–126.

GABLE G, SEDERA D and CHAN T (2008) Re-conceptualizing information system success: the IS-impact measurement model. Journal of the Association for Information Systems 9(7), 377–408.

GAGNON Y, POSADA E, BOURGAULT M and NAUD A (2010) Multichannel delivery of public services: a new and complex management challenge. International Journal of Public Administration 33(5), 213–222.

GILBERT D, BALESTRINI P and LITTLEBOY D (2004) Barriers and benefits in the adoption of e-government. International Journal of Public Sector Manage- ment 17(4), 286–301.

GONZALEZ R, GASCO J and LLOPIS J (2007) E-government success: some principles from a Spanish case study. Industrial Management & Data Systems 107(6), 845–861.

GOUSCOS D, KALIKAKIS M, LEGAL M and PAPADOPOULOU S (2007) A general model of performance and quality for one-stop e-government service offerings. Government Information Quarterly 24(4), 860–885.

GRANT J and DAVIS L (1997) Selection and use of content experts for instrument development. Research in Nursing & Health 20(3), 269–274.

GRIMSLEY M and MEEHAN A (2007) e-Government information systems: evaluation-led design for public value and client trust. European Journal of Information Systems 16(2), 134–148.

GRIMSLEY M, MEEHAN A and TAN A (2007) Evaluative design of e-govern- ment projects,. Transforming Government: People, Process and Policy 1(2), 174–193.

GRONLUND A and HORAN TA (2004) Introducing e-Gov: history, definitions and issues. Communications of the AIS 15(1), 713–729.

HAIR JF, BLACK WC, BABIN BJ, ANDERSON RE and TATHAM RL (2006) Multi- variate Data Analysis. Pearson Prentice-Hall, New Jersey.

HANSEN H (1995) A case study of a mass information system. Information & Management 28(3), 215–225.

HARRISON TM et al (2012) Open government and e-government: demo- cratic challenges from a public value perspective. Information Polity 17(2), 83–97.

HEEKS R (2008) Benchmarking eGovernment: improving the national and international measurement valuation and comparison of e-govern- ment. In Evaluation of Information Systems: Public and Private Sector (IRANI Z and LOVE P, Eds), pp 236–301, Butterworth-Heinemann, Oxford.

HEEKS R and BAILUR S (2007) Analyzing e-government research: perspec- tives, philosophies, theories, methods, and practice. Government Infor- mation Quarterly 24(2), 243–265.

HEEKS R and STANFORTH C (2007) Understanding e-government project trajectories from an actor-network perspective. European Journal of Information Systems 16(2), 165–177.

HEFETZ A and WARNER M (2004) Privatization and its reverse: explaining the dynamics of the government contracting process. Journal of Public Administration Research & Theory 14(2), 171–190.

HELBIG N, GIL-GARCIA R and FERRO E (2009) Understanding the complexity of electronic government: implications from the digital divide literature. Government Information Quarterly 26(1), 89–97.

HUI G and HAYLLAR MR (2010) Creating public value in E-Government: a public-private-citizen collaboration framework in web 2.0. Australian Journal of Public Administration 69(S1), 120–131.

IBBOTT C and O'KEEFE B (2004) Trust, planning and benefits in a global interorganisational system. Information Systems Journal 14(2), 131–152.

JAEGER P (2005) Deliberative democracy and the conceptual foundations of electronic government. Government Information Quarterly 22(4), 702–719.

JORGENSEN TB and BOZEMAN B (2007) Public values – an inventory. Administration & Society 39(3), 354–381.

KEENEY RL (1999) The value of internet commerce to the customer. Management Science 45(1), 533–542.

KELLY G, MULGAN G and MUERS S (2002) Creating public value: an analytical framework for public service reform. Strategy Unit Discussion Paper, Cabinet Office, London.

KIM D, YUE K-B, HALL S and GATES S (2009) Global diffusion of the internet XV: web 2.0 technologies, principles, and applications: a conceptual framework from technology push and demand pull per- spective. Communications for the Associations of Information Systems 24(1), 657–672.

Measuring eGovernment success Murray Scott et al206

European Journal of Information Systems

KIM S and LEE J (2012) E-participation, transparency, and trust in local government. Public Administration Review 72(6), 819–828.

KOLSAKER A and LEE-KELLEY L (2008) Citizens' attitudes towards e-govern- ment and e-governance: a UK study. International Journal of Public Sector Management 21(7), 723–738.

KUK G and JANSSEN M (2013) Assembling infrastructures and business models for service design and innovation. Information Systems Journal 23(5), 445–469.

LAU E (2006) Electronic government and the drive for growth and equity. In Electronic Government to Information Government (MAYER-SHONBEGER V and LAZER D, Eds), MIT Press, Massachusetts, Cambridge.

LAWSON-BODY A, WILLOUGHBY L, ILLIA A and LEE S (2014) Innovation characteristics influencing veterans' adoption of eGovernment Services. Journal of Computer Information Systems 54(3), 34–44.

LEE C, CHANG K and BERRY F (2011) Testing the development and diffusion of E-government and E-democracy: a global perspective. Public Admin- istration Review 71(3), 444–454.

LEE J and RAO HR (2012) Service source and channel choice in G2C service environments: a model comparison in the anti/counter-terrorism domain. Information Systems Journal 22(4), 313–341.

LEWIS BR, TEMPLETON GF and BYRD TA (2005) A methodology for construct development in MIS research. European Journal of Information Systems 14(4), 388–400.

LIAO Z and CHEUNG M (2001) Internet-based e-shopping and consumer attitudes: an empirical study. Information & Management 38(5), 299–306.

LIAO Z and CHEUNG M (2002) Internet-based e-banking and consumer attitudes: an empirical study. Information & Management 39(4), 283–295.

LOWRY P, KARUGA G and RICHARDSON V (2007) Assessing leading institu- tions, faculty, and articles in premier information systems research journals. Communications of the Association for Information Systems 20(1), 142–203.

LYNN M (1986) Determination and quantification of content validity. Nursing Research 35(6), 382–385.

MACCALLUM RC (1986) Specification searches in covariance structure modeling. Psychological Bulletin 100(1), 107–120.

MAJCHRZAK A (2009) Where is the theory in Wikis? MIS Quarterly 33(1), 19–20.

MANCINI R (2012) Raising the bar for e-government. Public Administration Review 72(6), 829–829.

MARCHE S and MCNIVEN J D (2003) E-government and e-governance: the future isn't what it used to be. Canadian Journal of Administrative Sciences 20(1), 74.

MCAFEE A (2009) Shattering the myths about enterprise 2.0. Harvard Business Review 87(11), 1–6.

MCKINNEY V, YOON K and ZAHEDI F (2002) The measurement of web- customer satisfaction: an expectation and disconfirmation approach. Information Systems Research 13(3), 296–315.

MCKNIGHT DH, CHOUDBURY V and KACMAR C (2002) Developing and validating trust measures for e-commerce: an integrative typology. Information Systems Research 13(3), 334–359.

MEDAGLIA R (2012) eParticipation research: moving characterization for- ward (2006–2011). Government Information Quarterly 29(3), 346–360.

MEUTER M, OSTROM A, ROUNDTREE R and BITNER M (2000) Self-service technologies: understanding customer satisfaction with technology- based service encounters. Journal of Marketing 64(3), 50–64.

MEYNHARDT T (2009) Public value inside: what is public value creation? International Journal of Public Administration 32(3–4), 192–219.

MOON MJ (2002) The evolution of e-government among municipalities: rhetoric or reality? Public Administration Review 62(4), 424–433.

MOORE GC and BENBASAT I (1991) Development of an instrument to measure the perceptions of adopting an information technolog innova- tion. Information Systems Research 2(3), 192–222.

MOORE M (1994) Public value as the focus of strategy. Australian Journal of Public Administration 53(3), 296–304.

MOORE M (1995) Creating Public Value – Strategic Management in Govern- ment. Harvard University Press, Cambridge, MA.

MOORE M (2013) Recognizing Public Value. Harvard University Press, Cam- bridge, MA.

MORGESON F and MITHAS S (2009) Does E-government measure up to e-business? Comparing end user perceptions of U.S. federal

government and E-business web sites. Public Administration Review 69(4), 740–752.

MOSSE B and WHITLEY EA (2009) Critically classifying: UK e-government website benchmarking and the recasting of the citizen as customer. Information Systems Journal 19(2), 149–173.

MYERS MD (1994) Dialectical hermeneutics: a theoretical framework for the implementation of information systems. Information Systems Journal 5(1), 51–70.

NETEMEYER R, BEARDEN W and SHARMA S (2003) Scaling Procedures: Issues and Applications. Sage Publications, London.

NORRIS D and MOON MJ (2005) Advancing E-government at the grass- roots: tortoise or hare? Public Administration Review 65(1), 64–75.

NORRIS D and REDDICK C (2013) Local E-government in the United States: transformation or incremental change? Public Administration Review 73(1), 165–175.

NUNNALLY JC (1978) Psychometric Theory. McGraw-Hill, New York. O'FLYNN J (2007) From new public management to public value: paradig-

matic change and managerial implications. The Australian Journal of Public Administration 66(3), 353–366.

OLPHERT W and DAMODARAN L (2007) Citizen participation and engage- ment in the design of e-government services: the missing link in effective ICT design and delivery. Journal of the Association for Informa- tion Systems 8(9), 491–507.

PALMER J (2002) Web site usability, design, and performance metrics. Information Systems Research 13(2), 151–167.

PANG M-S, LEE G and DELONE WH (2014) In public sector organisations: a public-value management perspective. Journal of Information Technol- ogy 29(3), 187–205.

PARAMESWARAN M and WHINSTON AB (2007) Research issues in social computing. Journal of the Association for Information Systems 8(6), 336–350.

PERISTERAS V, MENTZAS G, TARABANIS K and ABECKER A (2009) Transforming e-government and e-participation through IT. IEEE Intelligent Systems 24(September/October), 14–19.

PETTER S, DELONE WH and MCLEAN ER (2008) Measuring information systems success: models, dimensions, measures, and interrelationships. European Journal of Information Systems 17(3), 236–264.

PETTER S, DELONE WH and MCLEAN ER (2012) The past, present, and future of ‘IS success’. Journal of the Association for Information Systems 13(5), 341–362.

PETTER S and MCLEAN ER (2009) A meta-analytic assessment of the DeLone and McLean IS success model: an examination of IS success at the individual level. Information & Management 46(3), 159–166.

PETTER S, STRAUB D and RAI A (2007) Specifying formative constructs in information systems research. MIS Quarterly 31(4), 623–656.

PINA V, TORRES L and ROYO S (2007) Are ICTs improving transparency and accountability in the EU regional and local governments? An empirical study. Public Administration 85(2), 449–472.

PORTER M and KRAMER M (2011) Creating shared value. Harvard Business Review 89(1/2), 62–77.

PRINCETON (2001) The Internet & American Life Government Survey. Prince- ton Survey Research Associates, Washington, DC.

PRYBUTOK V, ZHANG X and RYAN S (2008) Evaluating leadership, IT quality, and net benefits in an e-government environment. Information & Management 45(4), 143–152.

REDDICK C (2005) Citizen interaction with e-government: from the streets to servers? Government Information Quarterly 22(1), 38–57.

REECE B (2006) E-government literature review. Journal of E-Government 3(1), 69–110.

RHODES R. and WANNA J. (2007) The limits to public value, or rescuing responsible government from the platonic guardians. Australasian Journal of Public Administration 66(4), 406–421.

SABHERWAL R, JEYARAJ A and CHOWA C (2006) Information system success: individual and organizational determinants. Management Science 52(12), 1849–1864.

SCOTT M, DELONE WH and GOLDEN W (2009) Understanding net benefits: A citizen-based perspective on eGovernment success. In: Thirtieth International Conference on Information Systems (ICIS 2009), Phoenix, Arizona: AIS.

SCOTT M, DELONE WH and GOLDEN W (2011) IT quality and eGovernment net benefits: A citizen perspective. In: 19th European Conference on Information Systems (ECIS). Helsinki, Finland.

Measuring eGovernment success Murray Scott et al 207

European Journal of Information Systems

SEDDON PB, STAPLES DS, PATNAYAKUNI R and BOWTELL M (1999) Dimensions of information systems success. Communications for the Associations of Information Systems 2(20), 1–39.

SEGARS AH and GROVER V (1993) Re-examining perceived ease of use and usefulness: a confirmatory factor analysis. MIS Quarterly 17(4), 517–522.

SELTSIKAS P and O'KEEFE B (2010) Expectations and outcomes in electronic identity management: the role of trust and public value. European Journal of Information Systems 19(1), 93–103.

SMITH RFI (2004) Focusing on public value: something new and something old. Australian Journal of Public Administration 63(4), 68–79.

SØRUM H, MEDAGLIA R, ANDERSEN K-V, SCOTT M and DELONE WH (2012) Perceptions of information system success in the public sector: Web- masters at the steering wheel? Transforming Government: People, Process and Policy 6(3),pp 239–257.

STEWART K and SEGARS AH (2002) An empirical examination of the concern for information privacy instrument. Information Systems Research 13(1), 36–49.

STOKER G (2006) Public value management. American Review of Public Administration 36(1), 41–57.

STRAUB D, BOUDREAU M-C and GEFEN D (2004) Validation guidelines for IS positivist research. Communications for the Associations of Information Systems 13(1), 380–427.

STRAUB DW (1989) Validating instruments in MIS research. MIS Quarterly 13(2), 147–169.

SZYMANSKI D and HISE R (2000) e-satisfaction: an initial examination. Journal of Retailing 76(3), 309–322.

TABACHNICK BG and FIDELL LS (2001) Using Multivariate Statistics, 4th edn, Harper Collins, New York.

TAN C-W, BENBASAT I and CENFETELLI RT (2013) IT-mediated customer service content and delivery in electronic governments: an empirical investiga- tion of the antecedents of service quality. MIS Quarterly 37(1), 77.

TAN CW and PAN SL (2003) Managing e-transformation in the public sector: an e-government study of the Inland revenue authority of Singapore (IRAS). European Journal of Information Systems 12(4), 269.

TEO T, LIM V and LAI R (1997) Users and uses of the internet: the case of Singapore. International Journal of Information Management 17(5), 325–336.

TEO T, SRIVASTAVA S and JIANG L (2008) Trust and electronic government success: an empirical study. Journal of Management Information Systems 25(3), 99–131.

THOMAS J and STREIB G (2003) The new face of government: citizen- initiated contacts in the era of e-government. Journal of Public Adminis- tration Research & Theory 13(1), 83–101.

TOJIB D, SUGIANTO L-F and SENDJAYA S (2008) User satisfaction with business-to-employee portals: conceptualization and scale develop- ment. European Journal of Information Systems 17(6), 649–667.

TOLBERT C and MOSSBERGER K (2006) The effects of eGovernment on trust and confidence in government. Public Administration Review 66(3), 354–369.

TORKZADEH G and DHILLON G (2002) Measuring factors that influence the success of internet commerce. Information Systems Research 13(2), 187–204.

UN (2005) Global e-Government Readiness Report 2005: From e-Govern- ment to e-Inclusion. United Nations, New York.

UNITED NATIONS (2012) Global E-Government Survey. United Nations Department of Economic and Social Affairs, New York.

US GOVERNMENT (2002) E-government Strategy: Implementing the Presi- dent's Management Agenda for e-Government – Simplified Delivery of Services to Citizens. US Government.

VAN DER HEIJDEN H (2004) User acceptance of hedontic information systems. MIS Quarterly 28(4), 695–704.

VAN RIEL A, LILJANDER V and JURRIENS P (2001) Exploring consumer evalua- tions of e-services: a portal site. International Journal of Service Industry Management 12(4), 359–377.

VENKATESH V, THONG JYL, CHAN FKY, HU PJ-H and BROWN SA (2011) Extending the two-stage information systems continuance model: incorporating UTAUT predictors and the role of context. Information Systems Journal 21(6), 527–555.

WANG Y-S (2008) Assessing e-commerce systems success: a respecification and validation of the DeLone and McLean model of IS success. Information Systems Journal 18(5), 529–557.

WANG Y-S and LIAO Y-W (2008) Assessing eGovernment systems success: a validation of the DeLone and McLean model of information systems success. Government Information Quarterly 25(4), 717–733.

WARKENTIN M, GEFEN D, PAVLOU P and ROSE GM (2002) Encouraging citizen adoption of e-government by building trust. Electronic Markets 12(3), 157–162.

WATSON RT and MUNDY B (2001) A strategic perspective of electronic democracy. Communications of the ACM 44(1), 27.

WATTAL S, SCHUFF D, MANDVIWALLA M and WILLIAMS CB (2010) Web 2.0 and politics: the 2008 Us presidential election and an e-politics research agenda. MIS Quarterly 34(4), 669–688.

WEBER R (1985) Basic Content Analysis. Sage Publications, London. WEINBERGER D (2002) Small Pieces Loosely Joined: A Unified Theory of the

Web. Perseus Publishing, Cambridge, MA. WELCH EW, HINNANT CC and MOON MJ (2005) Linking citizen satisfaction

with e-government and trust in government. Journal of Public Adminis- tration Research & Theory 15(3), 371–391.

WEST D (2008) State and federal electronic government in the United States’ Governance Studies at Brookings, Washington DC.

WHITEHOUSE (2010) Open Government Directive. WILLIAMS I and SHEARER H (2011) Appraising public value: past, present and

futures. Public Administration 89(4), 1367–1384. WONG W and WELCH EW (2004) Does E-government promote account-

ability? A comparative analysis of website openness and government accountability. Governance 17(2), 275–297.

YANG K and RHO S-Y (2007) E-government for better performance: promises, realities, and challenges. International Journal of Public Man- agement 30(11), 1197–1217.

YILDIZ M (2007) E-government research: reviewing the literature, limitations, and ways forward. Government Information Quarterly 24(3), 646–665.

ZHU F, WYMER W and CHEN I (2002) IT-based services and service quality in consumer banking. International Journal of Service Industry Management 13(1), 69–90.

ZIMBRA D, FU T and LI X (2009) Assessing public opinions through Web 2.0: a case study on Wal-Mart. In Thirtieth International Conference on Information Systems, Phoenix, Arizona.

Measuring eGovernment success Murray Scott et al208

European Journal of Information Systems

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.