Discussion forum on Article readings

The Measurement of End-User Computing Satisfaction Author(s): William J. Doll and Gholamreza Torkzadeh Source: MIS Quarterly, Vol. 12, No. 2 (Jun., 1988), pp. 259-274 Published by: Management Information Systems Research Center, University of Minnesota Stable URL: https://www.jstor.org/stable/248851 Accessed: 08-02-2020 14:07 UTC

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide

range of content in a trusted digital archive. We use information technology and tools to increase productivity and

facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected].

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at

https://about.jstor.org/terms

Management Information Systems Research Center, University of Minnesota is collaborating with JSTOR to digitize, preserve and extend access to MIS Quarterly

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms

End-User Satisfaction

The Measurement of

End-User Computing Satisfaction

By: William J. Doll Professor of MIS and

Strategic Management The University of Toledo

Gholamreza Torkzadeh Assistant Professor of

Information Systems and Management Science

The University of Toledo

2801 West Bancroft Street

Toledo, Ohio 43606

Abstract

This article contrasts traditional versus end-

user computing environments and reports on the development of an instrument which merges ease of use and information product items to measure the satisfaction of users who directly interact with the computer for a specific appli- cation. Using a survey of 618 end users, the researchers conducted a factor analysis and modified the instrument. The results suggest a 12-item instrument that measures five compo- nents of end-user satisfaction - content, accu- racy, format, ease of use, and timeliness. Evi- dence of the instrument's discriminant validity is presented. Reliability and validity is assessed by nature and type of application. Finally, stan- dards for evaluating end-user applications are presented, and the instrument's usefulness for achieving more precision in research questions is explored.

Keywords: End-user computing, user satisfac- tion, end-user computing satisfac- tion, management

ACM Categories: K.6.4, K.6.0

Introduction

End-user computing (EUC) is one of the most significant phenomenon to occur in the informa- tion systems industry in the last ten years (Benson, 1983; Lefkovits, 1979). Although still in its early stages, signs of rapid growth are evi- dent. In the companies they studied, Rockart and Flannery (1983) found annual EUC growth rates of 50 percent to 90 percent. Benjamin (1982) has predicted that by 1990 EUC will absorb as much as 75 percent of the corporate computer budget. Because of these trends, Rockart and Flannery call for better management to improve the success of end-user computing. Without improved management, they see the adverse effects of the Nolan-Gibson (1974) "control" stage constraining development of this new phenomenon.

To improve the management of EUC, Cheney, et al. (1986) call for more empirical research on the factors which influence the success of end-

user computing. Henderson and Treacy (1986) describe a sequence of perspectives (implemen- tation, marketing, operations, and economic) for managing end-user computing and identify ob- jectives for each phase. In the implementation phase, they maintain that objectives should focus on increased usage and user satisfaction. As the organization gains experience with end-user computing, they recommend increased empha- sis on market penetration and objectives that are more difficult to evaluate such as integration, ef- ficiency, and competitive advantage.

Ideally one would like to evaluate EUC based on its degree of use in decision making and the resultant productivity and/or competitive advan- tages. Crandall (1969) describes these resultant benefits as utility in decision making. However, this "decision analysis" approach is generally not feasible (Gallagher, 1974; Nolan and Seward, 1974). End-user computing satisfaction (EUCS) is a potentially measurable surrogate for utility in decision making. An end-user application's util- ity in decision making is enhanced when the out- puts meet the user's information requirements (described by Bailey and Pearson (1983) as "in- formation product") and the application is easy to use. Ease of use or "user friendliness" is es-

pecially important in facilitating voluntary mana- gerial use of inquiry or decision support systems.

MIS Quarterly/June 1988 259

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms

End-User Satisfaction

In a voluntary situation, system usage can also be a surrogate measure of system success. Ives, et al. (1983) argue that usage of an information or decision support system is often not volun- tary (e.g., when usage is mandated by manage- ment). In this involuntary situation, perceptual measures of satisfaction may be more appropri- ate. Also, both theory (Fishbein and Ajzen, 1975) and a recent path analysis (Baroudi, et al., 1986) suggest that satisfaction leads to usage rather than usage stimulating satisfaction. Thus, user satisfaction may be the critical factor.

The growth of end-user computing is presenting new challenges for information system manag- ers. Measures of user information satisfaction

developed for a traditional data processing en- vironment may no longer be appropriate for an end-user environment where users directly in- teract with application software. Indeed, user in- formation satisfaction instruments have not been

designed or validated for measuring end-user satisfaction. They focus on general satisfaction rather than on a specific application, and they omit aspects important to end-user computing such as ease of use. Hence, this study distin- guishes between user information satisfaction and an end user's satisfaction with a specific application.

This article reports on the development of an instrument designed to measure the satisfaction of users who directly interact with a specific ap- plication. The focus is on measuring EUCS among data processing (DP) amateurs and non- DP trained users rather than DP professionals. The explicit goals of this research were to de- velop an instrument that:

1. Focuses on satisfaction with the information

product provided by a specific application;

2. Includes items to evaluate the ease of use

of a specific application;

3. Provides Likert-type scales as an alternative to semantic differential scaling;

4. Is short, easy to use, and appropriate for both academic research and practice;

5. Can be used with confidence across a vari-

ety of applications (i.e., adequate reliability and validity); and

6. Enables researchers to explore the relation- ships between end-user computing satisfac- tion and plausible independent variables (i.e.,

user computing skills, user involvement, EDP support policies and priorities, etc.).

An additional goal was to identify underlying fac- tors or components of end-user computing satisfaction.

The End-User Computing Satisfaction Construct In a traditional data processing environment (see Figure 1), users interact with the computer indi- rectly, through an analyst/programmer or through operations. Routine reports might be requested from operations. For ad hoc or nonroutine re- quests, an analyst/programmer assists the user. In this environment, a user might be unaware of what specific programs are run to produce reports.

In an end-user computing environment (see Figure 2), decision makers interact directly with the application software to enter information or prepare output reports. Decision support and da- tabase applications characterize this emerging end-user phenomenon. The environment typi- cally includes a database, a model base, and an interactive software system that enables the user to directly interact with the computer system (Sprague, 1980). Although vast improvements have been made in end-user software (Canning, 1981; Martin, 1982), efforts to improve the man- machine interface continue (Sondheimer and Relies, 1982; Yavelberg, 1982).

Figures 1 and 2 do not depict all the differences between traditional and end-user computing en- vironments. Other differences such as software, hardware, support requirements, and control pro- cedures are not illustrated. Rather, the intent of these figures is to illustrate that, in an end-user computing environment, analysts/programmers and operations staff are less directly involved in user support; users assume more responsi- bility for their own applications. Systems person- nel might assist in the selection of appropriate software tools, but the end users are largely on their own to design, implement, modify, and run their own applications. Training programs, ex- perienced colleagues, and manuals provide some assistance. However, the goal of informa- tion system staff and service policies typically focuses on enabling end users to function more independently, to solve many problems on their own.

260 MIS Quarterly/June 1988

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms

End-User Satisfaction

(COBOL, ETC.)

KL_J Figure 1. The Traditional DP Environment

The definition of end-user

computing Davis and Olson (1985) describe this changing role of the user. To define end-user computing, they distinguish between primary and secondary user roles. The primary user makes decisions based on the system's output. The secondary user is responsible for interacting with the appli- cation software to enter information or prepare output reports, but does not use the output di- rectly in his or her job. In end-user computing, the two roles are combined: the person who util- izes the system output also develops it.

In contrast, the CODASYL end-user facilities com- mittee (Lefkovits, 1979) provides a broader defi- nition of end-user computing to include: "indi- rect" end users who use computers through other people; "intermediate" end users who spec- ify business information requirements for reports they ultimately receive; and "direct" end users who actually use terminals. However, for the most part, writers in this area such as Martin

(1982), McLean (1979), and Rockart and Flan- nery (1983) limit their definition of end users to individuals who interact directly with the computer.

This research uses the more limited definition.

End-user computing satisfaction is conceptual- ized as the affective attitude towards a specific computer application by someone who interacts with the application directly. End-user satisfac- tion can be evaluated in terms of both the pri- mary and secondary user roles. User informa- tion satisfaction, especially the information product, focuses on the primary role and is in- dependent of the source of the information (i.e., the application). Secondary user satisfaction varies by application; it depends on an applica- tion's ease of use. Despite the growing hands- on use of inquiry and decision support applica- tions by managerial, professional, and operat- ing personnel, research on user information sat- isfaction instruments has emphasized the primary user role, measuring overall user infor- mation satisfaction.

MIS Quarterly/June 1988 261

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms

End-User Satisfaction

Figure 2. The End-User Computing Environment

The Ives, Olson and Baroudi instrument

Focusing on "indirect" or "intermediate" users, Bailey and Pearson (1983) interviewed 32 middle managers and developed a semantic dif- ferential instrument measuring overall computer user satisfaction. Later, Ives, et al., (1983) sur- veyed production managers (e.g., "indirect" or "intermediate" users), conducted a factor analy- sis of the Bailey and Pearson instrument, and reported on a shorter version of this instrument. After two factors identified as "information prod- uct" were combined and a vendor support factor eliminated, the Ives, et al., study suggested three factors: EDP staff and services; information prod- uct; and knowledge or involvement. However, the ratio of sample size to number of scales (7:1) must be regarded with some caution.

Other validation studies have expressed some concerns. Using a sample of "indirect" and "in- termediate" users, Treacy (1985) assessed the reliability and validity of the Ives, et al., instru- ment. He concludes that this instrument is an

important contribution, but has difficulties in three areas: the variables found through exploratory factor analysis were labeled in imprecise and ambiguous terms; many of the questions used were poor operationalizations of their theoreti- cal variables; and the instrument failed to achieve discriminant validity. Also, Galletta and Lederer (1986) found test-retest reliability prob- lems with the Ives, et al., instrument and, be- cause of the heterogeneity of the items (infor- mation product, EDP staff and services, user involvement), expressed the need for caution in interpreting results.

These concerns are not widely shared. The Ives, et al., instrument is frequently used (Barki and Huff, 1985; Mahmood and Becker, 1985-86; Ray- mond, 1985; Galletta, 1986) and is, to date, prob- ably the best available measure of user infor- mation satisfaction (Galletta and Lederer, 1986). However, this instrument has not been used in end-user computing research.

The Ives, et al., instrument was designed for the more traditional data processing environment. It

262 MIS Quarterly/June 1988

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms

End-User Satisfaction

measures general user satisfaction with EDP staff and services, information product, and user involvement/knowledge rather than satisfaction with a specific application. Indeed, it has not been validated for use in assessing specific end- user applications. It also ignores important ease of use aspects of the man-machine interface.

Ease of use has become increasingly important in software design (Branscomb and Thomas, 1984). There is increasing evidence that the ef- fective functioning of an application depends on its ease of use or usability (Goodwin, 1987). If end users find an application easy to use, they may become more advanced users, and therefore, better able to take advantage of the range of capabilities the software has to offer. Also, ease of use may improve productiv- ity or enable decision makers to examine more alternatives.

Both the EDP staff and services items and the

user involvement/knowledge items seem inap- propriate for an end-user environment. The end- user environment requires new EDP staff and service policies. End users have less direct in- teraction with analysts/programmers or opera- tions. Rather than emphasizing direct support for user information requests, EDP staff and serv- ice policies emphasize more indirect and behind the scene technical efforts to improve hardware, languages, data management, privacy, security, and restart/recovery (Rockart and Flannery, 1983). Most end users would not be able to evalu- ate these activities. Thus, several EDP staff and service items in the Ives, et al., instrument seem less appropriate in an end-user environment. These items include:

- Relationship with EDP staff; - Processing of requests for system changes; - Attitude of EDP staff; - Communication with EDP staff; - Time required for system development; and - Personal control of EDP services

By their nature, these items assume a more tra- ditional computing environment and, like the user knowledge /involvement and information product items, are not application specific.

In addition, EDP staff/services and user knowl- edge/involvement items seemed more appropri- ately viewed as independent rather than depend- ent variables in an end-user computing environ- ment. End-user knowledge and involvement in development is generally considered to be posi-

tively correlated with satisfaction. Also, Rockart and Flannery (1983) suggest that end-user skill levels and EDP support policies can affect the success of end-user computing. For these rea- sons, EDP staff/services and user knowledge/ involvement items were excluded when the re-

searchers generated items to measure end- user computing satisfaction.

Research Methods To ensure that a comprehensive list of items was included, the works of previous researchers (Bailey and Pearson, 1983; Debons, et al., 1978; Neuman and Segev, 1980; Nolan and Seward, 1974; Swanson, 1974; Gallagher, 1974) were reviewed. Based on this review, the research- ers generated 31 items to measure end-user per- ceptions. To measure "ease of use" of an appli- cation, a construct which seemed to be missing from the previous works reviewed, seven addi- tional items were also included. Two global meas- ures of perceived overall satisfaction and suc- cess were added to serve as a criterion.

Thus, a 40-item instrument (see the Appendix) was developed using a five point Likert-type scale, where 1 = almost never; 2 = some of the time; 3 = about half of the time; 4 = most of the time; and 5 = almost always. The in- structions requested the users to write in the name of their specific application and, for each question, to circle the response which best de- scribed their satisfaction with this application.

Next, a structured interview questionnaire was developed where users were asked open- ended questions such as: How satisfied were they with the application. What aspects of the application, if any, were they most satisfied with and why. What aspects of the application, if any, were they most dissatisfied with and why?

Pilot study To make the results more generalizable, the re- searchers attempted to gather data from a vari- ety of firms. Five firms - a manufacturing firm, two hospitals, a city government office, and a university - were selected. A sample of 96 end users, with approximately an equal number of responses from each organization, was ob- tained. Data were gathered by research assis- tants through personal interviews with end users.

MIS Quarterly/June 1988 263

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms

End-User Satisfaction

The personal interviews enabled the assistants to verify that the respondent directly interacted with the application software. The research as- sistants first conducted open-ended structured interviews and recorded the end user's com-

ments; then, the Likert-type questionnaire was administered.

To assess whether the instrument was captur- ing the phenomenon desired by the researchers and to verify that important aspects of satisfac- tion were not omitted, qualitative comments from the structured interviews were compared with the responses to the 40 questions. The end users' overall level of satisfaction and the specific as- pects that satisfied or dissatisfied end-users sup- ported the instrument. This also enabled the re- searchers to verify that the respondents knew what the items were asking.

To ensure that the items measured the end-

user computing construct, the construct validity of each item was examined. Kerlinger (1978) cites two methods of construct validation: (1) cor- relations between total scores and item scores, and (2) factor analysis. The first approach as- sumes that the total score is valid; thus, the extent to which the item correlates with the total

score is indicative of construct validity for the item. In this study each item score was sub- tracted from the total score in order to avoid a

spurious part-whole correlation (Cohen and Cohen, 1975); the result is a corrected item total (sum for 37 items) which was then correlated with the item score. In this pilot test, factor analy- sis was not used to assess construct validity be- cause the ratio of sample size to number of items (2:1) was considered too low.

A measure of criterion-related validity (Kerlinger, 1978) was also examined to identify items which were not closely related to the end-user com- puting construct. The two global items measur- ing perceived overall satisfaction and success of the application were assumed to be valid meas- ures, and the sum of the two items was used as a criterion scale. The items comprising this criterion scale were: "Is the system successful?" and "Are you satisfied with the system?" The extent to which each item was correlated with

this two-item criterion scale provided a measure of criterion-related validity.

Items were eliminated if their correlation with the corrected item total was below .5 or if their cor- relation with the two-item criterion scale was

below .4. These cutoffs were arbitrary; there are

no accepted standards. The correlations with the corrected item total (r : .5) and the two item criterion (r : .4) were significant at p < .001 and comparable to those used by other research- ers (Ives, et al., 1983). Thus, the cutoffs were considered high enough to ensure that the items retained were adequate measures of the end- user computing satisfaction construct. These two criteria enabled the researchers to reduce the 38 items to 23. Five additional items were de-

leted because they represented the same as- pects with only slightly different wordings (e.g., "Does the system provide up-to-date informa- tion?" and "Do you find the information up-to- date?"). In each case, the wording with the lowest corrected item total correlation was de-

leted. In the pilot study, the remaining 18 items had a reliability (Cronbach's alpha) of .94 and a correlation of .81 with the two-item criterion scale.

Survey methods To further explore this 18-item instrument, the questionnaire was administered to 44 firms. The sample was select rather than random; however, the large number of firms used supports the gen- eralizability of the findings. In each of these firms, the MIS director was asked to identify the major applications and the major users who directly interact with each application. In many cases, the MIS director consulted with the heads of user

departments to identify major end users. This method may have failed to identify a few major end users, especially microcomputer users. How- ever, working through the MIS director was con- sidered a practical necessity.

In this survey, a separate criterion question ("Overall, how would you rate your satisfaction with this application?") was used. The criterion question used a five point scale: 1 = non- existent; 2 = poor; 3 = fair; 4 = good; 5 = excellent.

Data were gathered by research assistants who first conducted personal interviews with the end users (using the same structured interview proc- ess used in the pilot study) and then admini- stered the questionnaire. Again, the personal in- terviews enabled the research assistants to

verify that the respondents directly interacted with application software. The researchers com- pared the more qualitative interview comments with the questionnaire data to identify inconsis- tencies (i.e., respondents who did not complete

264 MIS Quarterly/June 1988

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms

End-User Satisfaction

the questionnaire carefully). Only about eight re- spondents were discarded because interview com- ments did not correspond with the questionnaire data.

A sample of 618 usable end users' responses was obtained. This sample represented 250 dif- ferent applications with an average of 2.5 re- sponses per application. Bartlett's test of sphe- ricity had a chi-square value of 8033.46 and a significance level of .00000. This suggests that the intercorrelation matrix contains enough common variance to make factor analysis worth pursuing.

The ratio of sample size to number of items (34:1) was well above the minimum 10:1 ratio suggested for factor analysis by Kerlinger (1978). However, in this case, a large sample was con- sidered essential. The items being factor ana- lyzed were selected because they were closely related to each other (i.e., all items were thought to be measures of the same EUCS construct). Thus, the items could be expected to have con- siderable common variance and relatively large error variance compared to their unique variance.

To assess reliability and validity by nature and type of application, users were asked whether their application was: end-user developed; mi- crocomputer or mainframe; and monitor, excep- tion reporting, inquiry or analysis (Alloway and Quillard, 1983).

Sample characteristics The sample contains responses from a variety of industries and management levels (see Table 1). The respondents indicated that 41.9 percent of the applications were "primarily developed by an end user" but only 91 (14.7 percent) had per- sonally developed the application themselves. Twenty five percent were microcomputer appli- cations whereas 75 percent were mini or main- frame applications. The applications were 37.6 percent decision support, 19.3 percent database, 19.8 percent exception reporting, 19.9 percent monitor, and 3.4 percent other (e.g., word processing).

Data Analysis The researchers conducted an exploratory factor analysis and modified the instrument, examined discriminant validity of the modified instrument, and assessed reliability and criterion-related va- lidity by nature and type of application (Kerlin- ger, 1978; Schoenfeldt, 1984). Factor analysis was used to identify the underlying factors or components of end-user satisfaction that com- prise the domain of the end-user satisfaction con- struct. Items which were not factorially pure were deleted to form a modified instrument that

would facilitate the testing of more specific hy- potheses (Weiss, 1970). The researchers at- tempted to avoid the use of imprecise and am- biguous terms to label the factors (Bagozzi,

Table 1. Respondents by Industry and Position

Respondents by Industry Manufacturing 42.6% Finance, banking & insurance 4.5% Education 3.7% Wholesale & retail 6.5%

Transportation, communication & utilities 9.5% Government agencies 9.1% Health services/hospitals 16.7% Other 7.4%

Total 100.0%

Respondents by Position

Top management 4.2% Middle management 31.2% First level supervisor 20.4% Professional employees without supervisory responsibilities 28.7% Other operating personnel 15.5% Total 100.0%

MIS Quarterly/June 1988 265

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms

End-User Satisfaction

1981), and examined discriminant validity (Campbell and Fiske, 1959).

Factor analysis Using the sample of 618 responses, the data was examined using principal components analy- sis as the extraction technique and varimax as a method of rotation. Without specifying the number of factors, three factors with eigen values greater than one emerged. These fac- tors were interpreted as content / format, accuracy timeliness, and ease of use/efficiency.

These labels were considered imprecise be- cause factors appeared to contain two different types of items (e.g., content and format items; accuracy and timeliness items). To achieve more precise and interpretable factors, the analysis was conducted specifying two, four, five and six factors.

The researchers felt that specifying five factors resulted in the most interpretable structure. These factors were interpreted as content, ac- curacy, format, ease of use, and timeliness and explained 78.0 percent of the variance. The load- ings of the 18 items on each factor (for factor loading greater than .30) is depicted in Table 2 and a description of each item (C1 thru T2) is provided in Table 3.

The items are grouped in Table 2 by their high- est (primary) factor loading. A number of items had factor loadings above .3 or .4 on additional (nonprimary) factors. Items with many multiple loadings may be excellent measures of overall end-user satisfaction, but including them in the scale blurs the distinction between factors. To

improve the distinction between factors, items which had factor loadings greater than .3 on three or more items were deleted from the scale -this includes C5, A3, A4, F3, F4, and E3.

These deletions resulted in a 12-item scale for

measuring end-user computing satisfaction and improved the match between the factor labels and the questions. In the modified 12-item in- strument, only one item (C4) had a primary factor loading below .7. Furthermore, none of the items had a secondary loading above .4. Each of these 12 items had a corrected item

total correlation above .63 (a measure of inter- nal consistency) and a correlation with the crite- rion measure of above .51 (see Table 3). Figure 3 illustrates this modified model for measuring end-user computing satisfaction.

This 12-item instrument had a reliability of .92 and a criterion-related validity of .76. The crite- rion was the separate measure of overall end- user satisfaction with the application. The reli- ability (alpha) of each factor was: content = .89;

Table 2. Rotated Factor Matrix of 18-Item Instrument

Item Ease of

Code Content Accuracy Format Use Timeliness C1 .74759 .30926 C2 .73854 C3 .71888 .36191 C4 .66369 .34585 C5 .51188 .36602 .41323 .39061

A1 .85959 A2 .83729 A3 .30158 .73136 .32456 A4 .56169 .34685 .39654

F1 .30357 .78831 F2 .35880 .71263 F3 .42106 .64593 .32791 F4 .32981 .58806 .44132

E1 .82396 E2 .34352 .80421 E3 .40981 .35998 .55695 .34936

T1 .77654 T2 .32913 .77251

266 MIS Quarterly/June 1988

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms

End-User Satisfaction

Table 3. Reliability and Criterion-Related Validity of Measures of End-User Satisfaction

Corrected Item- Correlation

Item Item Total With Code Description Correlation Criterion

C1 Does the system provide the precise information you need? .77 .62

C2 Does the information content

meet your needs? .76 .62

C3 Does the system provide reports that seem to be just about exactly what you need? .72 .60

C4 Does the system provide sufficient information? .70 .55

C5 Do you find the output relevant? .76 .59 A1 Is the system accurate? .69 .54 A2 Are you satisifed with the

accuracy of the system? .68 .51 A3 Do you feel the output is reliable? .73 .54 A4 Do you find the system dependable? .70 .65 F1 Do you think the output is

presented in a useful format? .66 .54 F2 Is the information clear? .72 .55

F3 Are you happy with the layout of the output? .73 .58 F4 Is the output easy to understand? .75 .57 El Is the system user friendly? .63 .52 E2 Is the system easy to use? .67 .57 E3 Is the system efficient? .75 .68 T1 Do you get the information you need in time? .69 .56 T2 Does the system provide up-to-date information? .67 .55

accuracy = .91; format = .78; ease of use = .85; and timeliness = .82. The correlation of each factor with the criterion was: content = .69; accuracy = .55; format = .60; ease of use = .58; and timeliness = .60.

Convergent and discriminant validity analysis Table 4 presents the measure correlation matrix, means, and variances. The multitrait-mul- timethod (MTMM) approach to convergent va- lidity tests that the correlations between meas- ures of the same theoretical construct are

different than zero and large enough to warrant further investigation. The smallest within- variable (factor) correlations are: content = .59;

accuracy = .82; format = 64; ease of use = .75; and timeliness = .70. For a sample of 618, these are significantly different than zero (p < .000) and large enough to encourage further investigation.

Using the MTMM approach, discriminant valid- ity is tested for each item by counting the number of times it correlates more highly with an item of another variable (factor) than with items of its own theoretical variable. Campbell and Fiske (1959) suggest determining whether this count is higher than one-half the potential comparisons. However, in this case, common method variances are present so it is unclear how large a count would be acceptable.

An examination of the matrix in Table 4 reveals

zero violations (out of 112 comparisons) of the

MIS Quarterly/June 1988 267

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms

End-User Satisfaction

condition for discriminant validity. For example, the lowest correlation between C1 and other con- tent items is .67 with C4. This correlation is

higher than Cl's correlation with the other eight noncontent items. Each of the 12 items are more

highly correlated with the other item(s) in its group than with any of the items measuring other variables.

Reliability and criterion validity analysis by nature and type of application

Table 5 describes the reliability and criterion- related validity of the 12-item scale by nature and type of application. The instrument appears

End-User

Computing Satisfaction

CONTENT

C1: Does the system provide the precise information you need? C2: Does the information content meet your needs? C3: Does the system provide reports that seem to be just about exactly what you need? C4: Does the system provide sufficient information?

ACCURACY

A1: Is the system accurate? A2: Are you satisified with the accuracy of the system?

FORMAT

F1: Do you think the output is presented in a useful format? F2: Is the information clear?

EASE OF USE

El: Is the system user friendly? E2: Is the system easy to use?

TIMELINESS

T1: Do you get the information you need in time? T2: Does the system provide up-to-date information?

Figure 3. A Model for Measuring End-User Computing Satisfaction

268 MIS Quarterly/June 1988

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms

End-User Satisfaction

Table 4. Correlation Matrix of Measures

(n = 618)

C2 .72 C3 .68 .68 C4 .67 .66 .59

A1 .49 .49 .41 .55 A2 .48 .45 .41 .48 .82

F1 .52 .56 .56 .56 .42 .48 F2 .56 .55 .54 .55 .53 .57 .64

E1 .51 .51 .46 .41 .37 .39 .37 .44 E2 .52 .51 .47 .41 .39 .39 .43 .56 .75

T1 .53 .53 .47 .50 .53 .51 .43 .46 .46 .44 T2 .52 .51 .45 .55 .57 .54 .44 .48 .44 .37 .70

C1 C2 C3 C4 A1 A2 F1 F2 E1 E2 T1

VARIABLE (ITEM) Mean VARIANCE

C1 3.891 .920 C2 3.972 .822 C3 3.862 1.056 C4 4.037 .799 A1 4.297 .729 A2 4.207 .754 F1 4.099 .668 F2 4.286 .660 El 3.964 1.238 E2 4.080 1.028 T1 4.096 .950 T2 4.247 .853

Table 5. Scale Reliability and Criterion Validity By Nature and Type of Application

Cronbach's Correlation Between

Alpha for Criterion and 12-Item Scale 12-ltem Scale

For All Applications .92 .76*

Micro Computer Application? Yes (n= 147) .91 .64* No (n= 429) .93 .78*

Type of Application? Other (word processing) (n = 19) .94 .85* Monitor Applications (n = 112) .93 .84* Exception Application (n = 117) .90 .65* Inquiry Applications (n = 111) .92 .68* Analysis Applications (n= 223) .94 .79*

End-User Developed Application? Yes (n = 236) .91 .72* No (n =321) .93 .77*

* Significant at p< .000.

MIS Quarterly/June 1988 269

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms

End-User Satisfaction

to have more than acceptable reliability and cri- terion-related validity for microcomputer and main- frame applications, for monitor, exception, inquiry or analysis applications, and for end-user devel- oped applications as well as those developed by more traditional methodology.

The reliability was consistently above .90 and showed little variation by nature and type of ap- plication. With a minimum standard of .80 sug- gested for basic research and .90 suggested for use in applied setting where important decisions will be made with respect to specific test scores (Nunnally, 1978) the instrument's reliability is adequate for both academic research and practice.

The correlations between the criterion question and the 12-item scale were consistently high (greater than .5) but, interestingly, showed more variation by nature and type of application. Mini or mainframe applications had a correlation of .78 with the criterion compared to .64 for micro- computer applications. Analysis (.79) and moni- tor (.84) applications had higher correlations with the criterion than exception (.65) or inquiry (.68) applications. In summary, it is the opinion of the researchers that the instrument presented in this article rep- resents substantial progress towards establish- ment of a standard instrument for measuring end- user satisfaction. The data supports the con- struct and discriminant validity of the instrument. Furthermore, the instrument appears to have ade- quate reliability and criterion-related validity across a variety of applications. However, con- tinuing efforts should be made to validate the instrument. The test-retest reliability of the in- strument should be evaluated and another large multi-organizational sample should be gathered to confirm factor structure and discriminant

validity.

Exploring the Instrument's Practical and Theoretical

Application In this section, suggestions are made for practi- cal application of the instrument. Then tentative standards for more precisely evaluating end- user applications are presented. Next, the use- fulness of the instrument for developing and test- ing more precise research questions is illustrated by exploring some hypotheses concerning the

relationship between end-user involvement in design and end-user satisfaction. Finally, sug- gestions for further research are discussed.

Practical application This 12-item instrument may be utilized to evalu- ate end-user applications. In addition to an over- all assessment, it can be used to compare end- user satisfaction with specific components (i.e., content, format, accuracy, ease of use, or time- liness) across applications. Although there may be reasons to add additional questions to evalu- ate unique features of certain end-user applica- tions, this basic set of 12 items are general in nature, and experience indicates that it can be used for all types of applications. This provides a common framework for comparative analysis.

The sample data used in this study represents the major applications from 44 firms. This cross organizational aspect of the sample makes it ap- propriate for the development of tentative stan- dards. Percentile scores for the 12-item end-

user computing satisfaction instrument are pre- sented in Table 6. Other relevant sample statis- tics are: minimum = 16; maximum = 60; mean - 49.09; median = 51; and standard deviation = 8.302. These statistics may be useful in more precisely evaluating end-user satisfaction with a specific application.

Table 6. Percentile Scores - 12-item Instrument

Percentile Value

10 37 20 43 30 46 40 48 50 51 60 53 70 54 80 57 90 59

Theoretical application In the development of this instrument, items which were not factorially pure were deleted. The five resultant components are relatively independ- ent of each other. With such component meas- ures, researchers may be able to achieve more precision in their research questions. Some com- ponents may be thought more closely associ-

270 MIS Quarterly/June 1988

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms

End-User Satisfaction

ated with specific independent variables than others. The instrument provides a framework for formulating and testing such hypotheses.

User information satisfaction has been used ex-

tensively in studies of user involvement (Ives and Olson, 1984); however, these studies used gen- eral measures and did not explore research ques- tions concerning the components of satisfaction. For example, satisfaction with accuracy and time- liness are affected by how the application is op- erated (i.e., the promptness and care in data entry). In contrast, design rather than operational issues may be the dominant factors affecting sat- isfaction with content, format, and ease of use. Thus, one might expect end-user involvement in design to be more closely associated with con- tent, format, and ease of use than accuracy or timeliness. This suggests two sets of hypothe- ses; the first is general in nature and the second is more precise.

H1: User participation in design is positively cor- related with end-user computing satisfaction and each of its components.

H2: User participation in design is more closely correlated with content, format and ease of use than accuracy or timeliness.

These hypotheses are used to illustrate the use- fulness of the end-user satisfaction instrument

for examining such research questions.

To explore these hypotheses, the researchers developed an eight-item Likert type scale for measuring user involvement in the end-user con- text. End users were asked about the amount

of time they spent in specific design activities (e.g., initiating the project, determining informa- tion needs, developing output format, etc.). This instrument had a reliability (Cronbach alpha) of .96.

The results depicted in Table 7 support the first set of hypotheses. End-user satisfaction and each of its components are significantly corre- lated with the end-user's involvement in the

design of the application.

To examine results for the second set of hy- potheses, absolute differences between corre- lation coefficients were calculated (see Table 8). The results for ease of use do not support the second hypothesis. End-user involvement in design was less positively correlated with ease of use than accuracy or timeliness. With respect to the results for content and format partially sup-

Table 7. Correlation Between End-User

Involvement in Design and End-User Computing Satisfaction Constructs

End-User

Involvement

in Design Overall EUCS .32* Content .30*

Accuracy .21* Format .29* Ease of use .20* Timeliness .25*

* Significant at p = .000.

port the second hypothesis. End-user involve- ment in design was more positively correlated with content and format than accuracy or timeli- ness. Using a test of the difference between cor- relation coefficients (Cohen and Cohen, 1975), two of these differences (content-accuracy and format-accuracy) were found to be significant at p < .05.

Table 8. Matrix of Difference in Correlations

Accuracy Timeliness Content .09* .05 Format .08* .04 Ease of Use .01 .05

* Significant at p = .05.

The intent is not to test hypotheses per se or explain the results obtained, but rather to illus- trate the usefulness of the end-user satisfaction

instrument for developing and testing more pre- cise research questions. The results suggest that some of the end-user satisfaction components derived by factor analysis may be more closely related to independent variables than others. In this illustration, end-user involvement in design was used as the independent variable. Future research efforts might focus on other independ- ent variables such as end-user skill levels, EDP support policies, type of application, or the qual- ity of user documentation.

Conclusions This article presents significant progress towards the development of a standard measure of end- user satisfaction with a specific application. De- signed for an end-user computing environment

MIS QuarterlylJune 1988 271

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms

End-User Satisfaction

rather than traditional data processing, the in- strument merges ease of use and information product items. Whether this instrument is chosen, the authors encourage the MIS research community to move towards a standard instru- ment for measuring end-user satisfaction which includes both information product and ease of use items.

The instrument appears to have adequate reli- ability and validity across a variety of applica- tions. It is short, easy to use, and appropriate for both practical and research purposes. Stan- dards are provided for use by practitioners. Its component factors are distinct, enabling research- ers to develop and test more precise research questions.

The lack of adequate mechanisms to evaluate the effectiveness of end-user computing is evi- dent. End-user satisfaction is only one of sev- eral relevant measures of end-user computing success. Additional work is needed to develop measures of the breadth of end-user computing in an organization (i.e., penetration) and the degree of sophistication (i.e., skill) of individual end users. Research on end-user computing's impact on efficiency, productivity, and competi- tive advantage would benefit from the availabil- ity of such measures.

References

Alloway, R.M. and Quillard, J.A. "User Manag- ers' Systems Needs," MIS Quarterly (7:2), June 1983, pp. 27-41.

Bagozzi, R.P. "An Examination of the Validity of Two Models of Attitude," Multivariate Be- havioral Research (16), 1981, pp. 323-359.

Bailey, J.E. and Pearson, S.W. "Development of a Tool for Measuring and Analyzing Com- puter User Satisfaction," Management Sci- ence (29:5), May 1983, pp. 530-545.

Barki, H. and Huff, S.L. "Change, Attitude to Change, and Decision Support System Suc- cess," Information and Management (9), 1985, pp. 261-268.

Baroudi, J.J., Olson, M.H., and Ives, B. "An Em- pirical Study of the Impact of User Involve- ment on System Usage and Information Sat- isfaction," Communications of the ACM (29:3), March 1986, pp. 232-238.

Benjamin, R.I. "Information Technology in the 1990's: A Long Range Planning Scenario," MIS Quarterly (6:2), June 1982, pp. 11-31.

Benson, D.H. "A Field Study of End-User Com- puting: Findings and Issues," MIS Quarterly'

(7:4), December 1983, pp. 35-45. Branscomb, L.M. and Thomas J.C. "Ease of

Use: A System Design Challenge," IBM Sys- tems Journal (23), 1984, pp. 224-235.

Campbell, D.T. and Fiske, D.W. "Convergent and Discriminant Validation by the Multitrait- Multimethod Matrix," Psychological Bulletin (56:1), 1959, pp. 81-105.

Canning, R.G. "Programming by End Users," EDP Analyzer (19:5), May 1981.

Cheney, P.H., Mann, R.I., and Amoroso, D.L. "Organizational Factors Affecting the Success of End-User Computing," The Journal of Man- agement Information Systems (3:1), Summer 1986, pp. 65-80.

Cohen, J. and Cohen, P. Applied Multiple Re- gression/Correlation Analysis for the Behav- ioral Sciences, Lawrence Erlbaum Assoc., Hillsdale, NJ, 1975.

Crandall, R.H. "Information Economics and Its Implications for the Further Development of Accounting Theory," The Accounting Review (44), 1969, pp. 457-466.

Davis, G.B. and Olson, M.H. Management In- formation Systems: Conceptual Foundations, Structure, and Development, McGraw-Hill Book Co., New York, 1985, pp. 532-533.

Debons, A., Ramage, W., and Orien, J. "Effec- tiveness Model of Productivity," in Research on Productivity Measurement Systems for Ad- ministrative Services: Computing and Informa- tion Services (2), L.F. Hanes and C.H. Kriebel (eds.), July 1978, NSF Grant APR - 20546.

Fishbein, M. and Ajzen, I. Belief, Attitude, Inten- tion and Behavior: An Introduction to Theory and Research, Addison-Wesley, Reading, MA, 1975.

Gallagher, C.A. "Perceptions of the Value of a Management Information System," Academy of Management Journal (17:1), 1974, pp. 46- 55.

Galletta, D.F. "A Longitudinal View of an Office System Failure," SIGOA Bulletin (7:1), 1986, pp. 7-11.

Galletta, D.F. and Lederer, A.L. "Some Cautions of the Measurement of User Information Sat- isfaction," Graduate School of Business, The University of Pittsburgh, Working Paper WP- 643, November 1986.

Goodwin, N.C. "Functionality and Usability," Com- munications of the ACM (30:3), March 1987, pp. 229-333.

Henderson, J.C. and Treacy, M.E. "Managing End-User Computing for Competitive Advan- tage," Sloan Management Review, Winter 1986, pp. 3-14.

272 MIS Quarterly/June 1988

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms

End-User Satisfaction

Ives, B. and Olson, M. "User Involvement and MIS Success: A Review of Research," Man- agement Science (30:5), 1984, pp. 586-603.

Ives, B., Olson, M., and Baroudi, S. "The Meas- urement of User Information Satisfaction," Com- munications of the ACM (26:10), October 1983, pp. 785-793.

Kerlinger, F.N. Foundations of Behavioral Re- search, McGraw-Hill, New York, 1978.

Lefkovits, H.C. "A Status Report on the Activi- ties of Codasyl End-User Facilities Commit- tee (EUFC)," Information and Management (2), 1979, pp. 137-163.

Mahmood, M.A. and Becker, J.D. "Effect of Or- ganizational Maturity on End-User Satisfaction with Information Systems," Journal of Man- agement Information Systems (2:3), Winter 1985-86, pp. 37-64.

Martin, J. Application Development Without Pro- grammers, Prentice Hall, Inc., Englewood Cliffs, NJ, 1982, pp. 102-106.

McLean, E.R. "End-Users as Application Devel- opers," MIS Quarterly (3:4), December 1979, pp. 37-46.

Neumann, S. and Segev, E. "Evaluate Your In- formation System," Journal of Systems Man- agement (31:3), March 1980, pp. 34-41.

Nolan, R. and Gibson, C.F. "Managing the Four Stages of EDP Growth," Harvard Business Review (52:1), January/February 1974, pp. 76- 88.

Nolan, R. and Seward, H. "Measuring User Sat- isfaction to Evaluate Information Systems," in Managing the Data Resource Function, R.L. Nolan (ed.), West Publishing Co., Los Angeles, 1974.

Nunnally, J.C. Psychometric Theory, McGraw- Hill, New York, 1978, p. 245.

Raymond, L. "Organizational Characteristics and MIS Success in the Context of Small Busi-

ness," MIS Quarterly (9:1), 1985, pp. 37-52. Rockart, J.F. and Flannery, L.S. "The Manage-

ment of End User Computing," Communica- tions of the ACM (26:10), October 1983, pp. 776-784.

Schoenfeldt, L.F. "Psychometric Properties of Or- ganizational Research Instruments," in Meth- ods and Analysis in Organizational Research, T.S. Bateman and G.R. Ferris (eds.), Reston Publishing Co., Reston, VA, 1984, pp. 68-80.

Sondheimer, N. and Relies, N. "Human Factors and User Assistance in Interactive Computing Systems: An Introduction," IEEE Transactions on Systems, Man, and Cybernetics SMC-12 (2), March-April 1982, pp. 102-107.

Sprague, R.H. "A Framework for the Develop- ment of Decision Support Systems," MIS Quar- terly (4:4), 1980, pp. 1-26.

Swanson, E.B. "Management Information Sys- tems: Appreciation and Involvement," Manage- ment Science (21:2), October 1974, pp. 178- 188.

Treacy, M.E. "An Empirical Examination of a Causal Model of User Information Satisfac-

tion," Center for Information Systems Re- search, Sloan School of Management, Mas- sachusetts Institute of Technology, April 1985.

Yavelberg, I.S. "Human Performance Engineer- ing Considerations For Very Large Computer- Based Systems: The End User," The Bell System Technical Journal (61:5), May-June 1982, pp. 765-797.

Weiss, D.J. "Factor Analysis in Counseling Re- search," Journal of Counseling Psychology (17), 1970, pp. 477-485.

About the Authors

William J. Doll is a professor of MIS and strate- gic management at The University of Toledo and serves as a management consultant for area com- panies. The author of many articles in academic and professional journals including the Academy of Management Journal, Communications of the ACM, MIS Quarterly, Information & Manage- ment, and the Journal of Systems Management, Dr. Doll has a doctoral degree in business ad- ministration from Kent State University and has worked as a senior management systems ana- lyst on the corporate staff of Burroughs Corporation.

G. Torkzadeh is an assistant professor of infor- mation systems in the Operations Management Department at The University of Toledo. He holds a Ph.D. in operations research from The University of Lancaster, England, and is a member of the O.R. Society of Great Britain, TIMS, DSI, ACM and SIM. He has been involved in research programs pertaining to the applica- tion of O.R. (in the public sector), distribution, resource allocation / re-allocation, and mathemati- cal modelling, and has published in the Journal of Operational Research Society, Communica- tions of the ACM, and Information & Manage- ment. One of his current research interests is

the management of the information systems function.

MIS Quarterly/June 1988 273

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms

End-User Satisfaction

Appendix

Measures of End-User Computing Satisfaction - Forty Items Used in Pilot Study

1. Is the system flexible? 2. Does the system provide out-of-date

information?

3. Is it easy to correct the errors? 4. Do you enjoy using the system? 5. Do you think the output is presented in a

useful format?

6. Is the system difficult to operate? 7. Are you satisfied with the accuracy of the

system? 8. Is the Information clear?

9. Are you happy with the layout of the output? 10. Is the system accurate? 11. Does the system provide sufficient informa-

tion?

12. Does the system provide up-to-date infor- mation?

13. Do you trust the information provided by the system?

14. Do you get the information you need in time?

15. Do you find the output relevant? 16. Do you feel the output is reliable? 17. Does the system provide too much informa-

tion?

18. Do you find the information up-to-date? 19. Does the system provide reports that seem

to be just about exactly what you need?

20. Is the system successful?* 21. Is the system easy to use? 22. Is the system user friendly? 23. Are the reports complete? 24. Does the system provide the precise infor-

mation you need? 25. Is the system efficient? 26. Is the output easy to understand? 27. Is the system troublesome? 28. Is the system convenient? 29. Is the system difficult to interact with? 30. Does the system provide comprehensive

information?

31. Do you think the system is reliable? 32. Would you like more concise output? 33. Does the information content meet your

needs?

34. Does the information you receive require correction?

35. Do you find the system dependable? 36. Would you like the system to be modified

or redesigned? 37. Do you think the reports you receive are

somewhat out-of-date?

38. Are you satisfied with the system?* 39. Would you like the format modified? 40. Do you get information fast enough?

* Criterion question.

274 MIS Quarterly/June 1988

This content downloaded from 198.91.37.2 on Sat, 08 Feb 2020 14:07:56 UTC All use subject to https://about.jstor.org/terms