COMM 600 week 2 individual assignment

Web site credibility: Why do people believe what they believe?

Marie K. Iding Æ Martha E. Crosby Æ Brent Auernheimer Æ E. Barbara Klemm

Received: 3 October 2008 / Accepted: 16 October 2008 / Published online: 27 November 2008 � Springer Science+Business Media B.V. 2008

Abstract This research investigates university students’ determinations of credibility of information on Web sites, confidence in their determinations, and perceptions of Web site

authors’ vested interests. In Study 1, university-level computer science and education

students selected Web sites determined to be credible and Web sites that exemplified

misrepresentations. Categorization of Web site credibility determinations indicated that the

most frequently provided reasons associated with high credibility included information

focus or relevance, educational focus, and name recognition. Reasons for knowing a Web

site’s content is wrong included lack of corroboration with other information, information

focus and bias. Vested interests associated with commercial Web sites were regarded with

distrust and vested interests of educational Web sites were not. In Study 2, credibility

determinations of university students enrolled in computer science courses were examined

for 3 provided Web sites dealing with the same computer science topic. Reasons for

determining Web site inaccuracy included own expertise, information corroboration,

information design and bias. As in Study 1, commercial vested interests were negatively

regarded in contrast to educational interests. Instructional implications and suggestions for

further research are discussed.

Keywords Web site credibility � Web evaluation � Critical information evaluation � College students � Credibility determinations � Web site veracity

Preliminary results from Study 1 were presented as a poster entitled ‘‘Users’ Confidence Levels and Strategies for Determining Web Site Veracity’’ (Iding et al. 2002a) and appeared in associated proceedings for The WWW 2002: The Eleventh International World Wide Web Conference, in Honolulu, Hawaii. Preliminary results from Study 2 were presented as a paper entitled, ‘‘Judging the Veracity of Web Sites’’ (Crosby et al. 2002) and appeared in associated proceedings for the International Conference on Computers in Education (ICCE 2002) in Auckland, New Zealand.

M. K. Iding (&) � M. E. Crosby � E. Barbara Klemm College of Education, University of Hawaii, 1776 University Ave, Honolulu, HI 96822, USA e-mail: [email protected]

B. Auernheimer California State University, Fresno, USA

123

Instr Sci (2009) 37:43–63 DOI 10.1007/s11251-008-9080-7

Introduction

Since the advent of the World Wide Web, the general public has become increasingly

reliant on the Web for information. Upon what bases do people evaluate information on the

Web? How do students and others determine what is credible or scientifically accurate? Do

they consider factors like commercial or vested interests of authors? What aspects

contribute to their judgments?

A recent article in Nature (Giles 2005) magazine highlights aspects of the issue by comparing the accuracy of science information in Wikipedia, a Web-based encyclopedia to which the general public can contribute, to that of Encyclopedia Britannica, long considered the accepted standard. Articles from both sources were sent to experts in

their respective fields without identifying information sources. The experts then rated

the accuracy of the encyclopedia entries. The following conclusion was published in Nature: ‘‘The exercise revealed numerous errors in both encyclopedias, but among the 42 entries tested, the difference in accuracy was not particularly great: the average

science entry in Wikipedia contained around four inaccuracies; Britannica, about three’’ (p. 900).

Despite the fact that Nature’s conclusion could be considered favorable to Wikipedia, Giles (2005) anecdotally highlights the controversial aspects of its open authorship process

by describing the efforts of Wikipedia contributor Connolly, a British climate change researcher. Connolly’s descriptions of global warming were repeatedly edited and coun-

tered by ‘‘climate change skeptics.’’ Finally, Wikipedia administrators mediated the

repeated editing and counter-editing by opposing contributors.

Although the Nature article received wide press, the issue of information accuracy on the Web is of concern in many areas including medical, commercial, and educational

realms. In this article, we define credibility, provide illustrative examples of research

examining information accuracy or credibility judgments in these three areas, and present

two studies in which university students from education and computer science classes

evaluated the credibility of information on Web sites.

What is credibility?

In work by Klemm et al. (2001), credibility is associated with information accuracy or

veracity, and with reputation of Web site authors or institutional affiliations (Klemm et al.

2001). Fogg et al. (2002) provide a concurring definition of credibility in the context of

Web site evaluation:

Credible information is believable information….People perceive credibility by evaluating multiple dimensions simultaneously. In general, these dimensions can be

categorized into two key components: trustworthiness and expertise. The trustwor- thiness component refers to the goodness or morality of the sources and can be

described with terms such as well intentioned, truthful, or unbiased. The expertise

component refers to perceived knowledge of the source and can be described with

terms such as knowledgeable, reputable, or competent. People combine assessments

of both trustworthiness and expertise to arrive at a final credibility perception (p. 9).

In general, this definition provides a useful operational or working definition for

credibility to serve as a foundation for examining credibility research in different

disciplines.

44 M. K. Iding et al.

123

Health care and the Web

An area of particular concern is the public’s acceptance of medical or health care infor-

mation on the Web. To what extent do users accept as credible health care information they

find on the Web? Benotsch et al. (2004) investigated adults with HIV and their determi-

nations of Web site credibility. The authors describe positive aspects of obtaining medical

information on-line: ‘‘For patients with chronic and life-threatening conditions, the Internet

can serve as a source of hope, social support, and empowerment’’ (p. 1004). Further, the

authors point out it can be a source of up-to-date treatment information.

However, Benotsch et al. (2004) argue that the Digital Divide results in more educated

persons of higher socioeconomic status (SES) having access to the information, and

possessing reading skills and comprehension of basic medical terminology that enables

them to more successfully evaluate the credibility of the information. In contrast, less

educated persons of lower SES backgrounds are clearly at a disadvantage.

In their study, they compared the credibility determinations of persons with HIV to

those of medical professionals. The Web sites they used for evaluation included an article

describing treatments for HIV from the Journal of the American Medical Association (JAMA) and a less reputable article from a site describing a supposed cure involving goat

serum extraction.

Results indicated that persons with HIV attributed higher credibility to both sites than

did health care professionals. Furthermore, the more literate and knowledgeable partici-

pants (regarding HIV treatments) tended to rate the JAMA site more highly than did

participants with less knowledge and lower literacy levels. As Benotsch et al. (2004)

explain:

The present findings…suggest that some patients do not always evaluate online information critically and may be vulnerable to misinformation. The nature of the

AIDS epidemic in the United States is such that educationally and economically

disadvantaged groups are increasingly affected by HIV. Such persons are among the

least equipped to critically evaluate the information they receive on-line (p. 1009).

Compounding the seriousness of these findings is the fact that participants with HIV

trusted their physicians most highly as the source of information about health care, then

they information from the Web secondly. The authors conclude that, ‘‘Individuals most in

need of information concerning HIV—those who would most benefit from the opportu-

nities the Internet affords for learning new information may also be the most vulnerable to

misinformation or to unethical peddlers of sham cures sold via the Internet’’ (p. 1009).

Kalichman et al. (2006) subsequently provided a 2 month intervention focusing on

basic computer and ‘‘Internet information consumer skills’’ (p. 545) for persons with HIV/

AIDS. Participants learned critical information evaluation skills, and used the Internet

more to locate health information and for social support than did the control support group.

These findings indicate that instruction can be effective for those with particular healthcare

information needs and specific vulnerabilities.

In other research, Escoffery et al. (2005) found that 74% of the undergraduates they

surveyed found health information on the Internet. While it may seem that undergraduates

would be more similar to the more highly literate and educated participants in the Benotsch

et al. (2004) study, other research by Metzger et al. (2003) indicates that although college

students appear to be aware that information from the Web (not specifically health care

information) may not be highly credible, they tend to verify it less than do members of the

general public. It may be that this finding reflects different purposes for information

Web site credibility 45

123

retrieval. Perhaps college students finding information for course assignments are less

likely to be concerned about credibility than about simple relevance; whereas it may be

that more highly educated and literate people may engage in verification processes if the

information is associated with high levels of importance and could be associated with

potentially costly or disastrous outcomes if wrong, such as may be the case in deciding

upon a health care treatment.

Consumers and the Web

Another area in which people are vulnerable to inaccuracy or non-credibility of Web-based

information is consumerism. In a large-scale study, Fogg et al. (2002) examined 2,684

consumers’ responses to credibility of Web sites in a number of content areas, including

e-commerce, finance, health, entertainment, sports, travel, and news. They found that

consumers’ reasons for making actual credibility determinations were different from what

they said their reasons were for making these determinations (reasons were reported in an

earlier study by Princeton Survey Research Associates 2002).

The authors explain, ‘‘We found a mismatch, as in other areas of life, between what

people say is important and what they actually do’’ (p. 6). For example, when asked

generally what factors they would consider in making Web site credibility determinations,

consumers listed considerations like the presence of a privacy policy. In actual practice,

they found that people rarely if ever referred to these criteria in making determinations.

Instead, the authors found that credibility judgments (in terms of reasons people gave for

making their decisions) focused first upon ‘‘design look’’ (i.e., ‘‘elements of the visual

design, including layout, typography, white space, images, color schemes’’ p. 24).

Other factors that were commented upon by consumers most frequently included

‘‘information structure’’ (i.e., how the information was structured upon the Web site) and

‘‘focus’’ (i.e., depth and breadth of information). It appeared to the authors that there was a

difference between making credibility judgments about information on the Web and ‘‘what

people notice when they evaluate a Web site for credibility’’ (p. 7).

Stanford et al. (2002) carried out another study, in which they examined credibility

judgments of 15 experts in the areas of health and finance and compared their determi-

nations to those of the consumers in the earlier study. They found that, in contrast to

consumers, health experts rated the following factors most highly in determining Web site

credibility: ‘‘name reputation of a site, its operator or its affiliates,’’ ‘‘information source,’’

or references and ‘‘company motive’’ (p. 4). Finance experts focused on ‘‘information

focus,’’ ‘‘quantity of information,’’ and ‘‘company motive’’ (p. 4).

Stanford et al. (2002) conclude by calling for research that examines credibility judg-

ments in other content areas. They also propose that consumer education in credibility

assessment is essential, a conclusion that would concur with educators’ perspectives

as well.

Flanagin and Metzger (2007) examined people’s credibility determinations in response

to Web sites in e-commerce, special interest, news organizations, and personal Web sites.

They found that personal Web sites were rated lowest and news sites the highest. Like

Fogg et al. (2002), they found that design aspects of sites had a greater impact on credi-

bility determinations than knowledge of Web site sponsors. Flanagin and Metzger (2007)

also found a discrepancy between self-reports of behavior and actual behavior, with those

reporting that they had engaged in extensive Web verification actually verifying infor-

mation less than others.

46 M. K. Iding et al.

123

Work by Tormala and Petty (2004) highlights aspects of credibility of information

sources, and while not dealing directly with Web-based information sources, applies

directly to this topic. Their research is important because it addresses peoples’ certainty of

their attitudes and the link to possible behavior after exposure to persuasive arguments

from expert and non-expert sources. For example, in an experiment investigating partici-

pants’ responses to advertising for a fictional aspirin, participants received information

about the product and listed their own counterarguments against it. Findings indicated that

‘‘Participants became more certain of their attitudes when they resisted persuasion from an

expert source, as long as they had sufficient cognitive resources available (presumably to

reflect on the implications of their resistance)’’ (p. 434). The same effect did not occur

when the persuasion was associated with a non-expert source. In a second experiment

involving a fictional proposed change to a university examination policy, they found that

‘‘participants’ attitudes became more predictive of behavioral intentions after they resisted

a persuasive attack from an expert source’’ (p. 438). The implication is that people become

more convinced that what they believe is true when they resist weak arguments from high

credibility sources. This may have implications for those who design Web sites and the

degree of thoughtfulness with which they present information on those sites.

Research in education

One of the general themes emerging from all of this research is the need for relevant

education. Education can take the form of providing guidelines on the Web for evaluating

Web-based information in particular areas, or in instructional interventions. In addition to

educating k-12 and college students, it is also important to work with pre-service and

practicing teachers who will be responsible for facilitating critical credibility determination

skills in their students.

Klemm et al. (2001) were particularly interested in how scientists and pre-service

teachers evaluated information sources. At that time, the authors’ interest was in the

credibility of sources generally, not only in Web-based resources, because of the current

trend in k-12 science instruction away from textbook-based instruction and toward more

constructivist modes that incorporate multiple resources. It appeared that the lack of ref-

ereeing of many of these sources left credibility judgments up to teachers and students

themselves, more so than ever before, when textbooks were assumed to be definitive

sources upon which entire curricula were based.

Klemm et al. (2001) investigated credibility associated with a whole range of infor-

mation sources (31) that might be utilized by teachers and others including tradebooks,

CNN, TV newsmagazines like 20/20, a scientist working on a research question, and

information from the World Wide Web. In order to provide a more veridical task, a context

for credibility determinations was provided: the area of assessment of risks. Results

indicated that pre-service teachers and scientists differed on assessing the credibility of the

majority of the information sources. Elementary pre-service teachers (with fewer required

science courses) were less like scientists in their determinations than secondary science

pre-service teachers. To illustrate, scientists selected ‘‘scientist researching the risk topic in

question’’ as the highest in credibility; secondary teachers rated the same category as fourth

in credibility, and elementary pre-service teachers rated this category as eleventh in

credibility.

Additionally, pre-service elementary teachers selected the following two categories as

having the highest levels of credibility: ‘‘popularized science magazines (e.g., Discover)’’

Web site credibility 47

123

and ‘‘resources from a museum, aquarium, or nature center.’’ Other information sources that

were rated by elementary pre-service teachers as having high levels of credibility included

‘‘CNN Cable News Network,’’ ‘‘weekly newsmagazines (e.g., Times, Newsweek, etc.),’’ and ‘‘TV News Magazine (e.g., 20/20 or Dateline).’’ This high regard for the credibility of the popular media was not shared by scientists, who gave such programs as ‘‘TV News Maga-

zines’’ very low ratings. As a scientist explained, ‘‘I place the most confidence in individuals

with good judgment and broad experience whom I am acquainted with. This option isn’t

listed. I have the least confidence in rapid turn-around sources (e.g., local television and

newspapers that demonstrate a consistent lack of judgment and experience’’ (p. 89).

Despite disagreements about the credibility of popular media, there were some sources of

concurrence. For example, all three groups rated the following sources highly: ‘‘resources

from university cooperative extensions (e.g., Sea Grant, Coop. Agriculture, etc.’’ and

‘‘Resources from a museum, aquarium, or nature center.’’ From an educational perspective

this is encouraging, yet the research indicated a great need for education in informed

credibility judgments for educators, the general public, as well as students in all areas.

In response to the need for education in this area, Iding et al. (2002b) investigated the

effects of instruction upon credibility judgments. For example, in an initial instruction-

oriented study, they worked with high school seniors enrolled in two biology classes over

4 days. The authors were interested both in factors contributing to high school students’

judgments as well as the effects of instruction on their ability to make informed decisions

about the credibility of Web sites and the information contained in Web sites.

As a pretest, the authors asked students to describe characteristics that they consider

when deciding to use a Web site generally, then when deciding to evaluate scientific

information on a Web site. Next, instruction was provided in three aspects of Web eval-

uation, including ‘‘credibility of authors and institutions, validity or accuracy of

information…and presentation aspects…of the Web site and its information’’ (p. 376). These characteristics were adapted from Rader (1998) and Farah (1995) and used in the

work of Nguyen (2000).

Students’ lists of criteria were more complete after instruction, and ‘‘the majority of

students reported learning something new, indicated that they would spend more time

evaluating scientific information on Web sites, and reported increased confidence in their

ability to evaluate scientific information on the Web’’ (p. 373).

Iding et al. (2002b) concluded that it is possible, in a relatively short amount of time, to

impact students’ Web assessment skills in a positive fashion. However, the authors were

aware of the brevity of this instruction and the need for a longer course of instruction and

follow-up evaluation to determine whether long-term transfer would be affected.

In yet another study, Iding and Klemm (2005) worked with pre-service teachers who

identified criteria that they used for determining whether they would cite scientific

information from a Web site, critically evaluated actual scientific Web sites, then created

structured evaluation forms (i.e., rubrics) for their students to use in critical Web site

information evaluation. Students did not agree on the credibility of the science Web sites,

even the NASA site.

The present studies

All of the reviewed research underscores the need for effective Web-based information

credibility determination skills among the general public. The issue is particularly

important, as with the exponential proliferation of information on any topic on the Web,

48 M. K. Iding et al.

123

people become the arbiters of information accuracy—in domains in which they are

knowledgeable and in domains in which they are not. How do people make these deter-

minations? Do they consider potential biases and vested interests of Web site authors as

they determine whether information is acceptable and accurate, believable, or not?

The studies that follow address these questions. Specifically, the credibility determi-

nations of university students in different disciplines, education, and computer science, are

investigated as they make determinations of credibility of various Web sites related to their

disciplines, rate confidence in their own determinations and articulate Web site authors’

vested interests.

Study 1

In the first study, the authors investigated credibility judgments of two groups of university

students: computer science students and education students. The authors were interested in

students’ judgments of characteristics they associated with credible Web sites and Web

sites they determined to be non-credible. The authors were also interested in the students’

confidence in their own credibility judgments and in their determinations of Web site

authors’ vested interests.

Method

Participants

Participants consisted of 84 university students. Forty-seven students were enrolled in a

computer science class, 21 in an educational psychology class, and 16 in a science methods

class.

Materials and procedure

One of the authors, a computer science instructor, developed an exercise related to Web

site selection and evaluation that was relevant to a topic covered in his class, cleanroom

software engineering. The exercise was presented on a worksheet. In the exercise, students

collaborated in small groups to select Web sites relevant to the topic.

They were instructed to ‘‘Select a Web site that your group feels gives the most accurate

and objective representation of cleanroom software engineering.’’ They explained why

they selected that site, and described the author’s vested interests. They were also

instructed to ‘‘Select a Web page that your group feels illustrates a misconception (or leads

to a misconception) about cleanroom software engineering.’’ They explained the follow-

ing: why this site illustrates a misconception; whether they thought the misconception was

due to a deliberate attempt to mislead or to a mistake; and, what they believed this author’s

vested interests were. In the next section of the exercise, they were asked, ‘‘How do you

know when a Web site’s content is wrong?’’ Next, they were asked to answer the following

questions as a group:

On a scale of 1 (no confidence) to 5 (complete confidence), how confident are you about

detecting misrepresentations on Web sites in general?

On the same 1–5 scale, how confident are you in detecting misrepresentations in the

particular Web site you [selected earlier].

Web site credibility 49

123

Finally, each student answered the following questions individually:

On a scale of 1 (no competence) to 5 (complete competence) scale, how competent are

you in evaluating the validity of information about cleanroom software engineering?

On the same 1–5 scale, how competent are you in evaluating the validity of information

on the Web in general?

The exercise was modified to be content-appropriate for the education classes, so that it

focused on science, technology and society (STS) for the science methods class and

collaborative teaching project topics (such as motivation and multicultural education) for

the educational psychology class. In the educational psychology class, instructions were:

‘‘Your task is to find information about the topic that your group has selected for the in-

class cooperative teaching project.’’ Examples of topics included classroom management,

special education, multicultural education, and motivation. Students were instructed to

‘‘Select a WWW page that your group feels gives the most objective and accurate rep-

resentation of your topic.’’ They were also instructed to ‘‘Select a Web page that your

group feels illustrates a misconception (or leads to a misconception) about your topic.’’

For the science methods class, students were instructed to ‘‘Select a WWW page that

your group feels gives the most objective and accurate representation of your STS topic.’’

Examples of topics included ‘‘whale migration and environmental influences’’ and ‘‘genetic

profiles.’’ As in the other groups, students were instructed to ‘‘Select a Web page that your

group feels illustrates a misconception (or leads to a misconception) about the STS topic.’’

All other questions about reasons for accuracy determinations, vested interests, confi-

dence and competence were the same for these students as those asked of the computer

science students.

Coding

Two of the authors collaboratively coded students’ written comments in response to each

question. Each separate statement that represented a major idea was coded, and frequencies

were calculated for each category. The coding scheme serving as the basis for content

analysis consisted of categories that emerged from the data and categories that were

adapted from the work of Fogg et al. (2002). (Categories adapted from the work of Fogg

et al. 2002, are indicated in Tables 1, 3, and 5).

Results and discussion

The authors coded participants’ responses to the question, ‘‘Why did you pick this Web

page?’’ (See Table 1 for categories and results). Notably, the primary reason was infor-

mation focus, or relevance. As one group explained, ‘‘Because it gives you the right

information.’’ Another wrote, ‘‘Gave a detailed description of the cleanroom approach.’’

Other comments indicated that the amount of information affected their choice: ‘‘Extensive

selection of previously done cleanroom projects.’’ ‘‘It had detailed descriptions and history

of cleanroom software engineering.’’

Secondly, participants were interested in Web sites that appeared to be focused on

providing educational information, or having an educational purpose. Name recognition

was also important, as one group explained, ‘‘Carnegie Mellon is a famous educational

institution.’’ Another wrote, ‘‘It is made by IBM and we figured that since they are such a

successful company that they would be a creditable [sic] source.’’

Other reasons included references used in the site, information design, and design look.

To differentiate between information design and design look, comments like the following

50 M. K. Iding et al.

123

characterized information design, ‘‘The design had serious thought behind it which I

credited to the Author’s over all effectiveness presenting the topic.’’ Information look was

reflected in comments like, ‘‘Looks well set out.’’ Additional reasons are listed above are

self-explanatory. Bias was used to characterize comments such as ‘‘Also discusses possible

‘down-sides’ to the technique.’’ Finally, a comment indicating corroboration referred to

‘‘external reviewers from Lockheed Martin.’’

In the next question, participants were asked, ‘‘What are the Web site authors ‘vested

interests’?’’ Their responses are summarized in Table 2. Since participants were searching

for objective and accurate Web sites, it is unsurprising that they described educational

vested interests most frequently. The commercial vested interests of some Web site

authors, the second most frequently described category was unexpected in association with

these highly credible Web sites, as participants expressed general suspicion of the motives/

accuracy of commercial Web sites in other parts of the exercise.

In the next question, participants were asked, ‘‘How do you know?’’ These answers (in

Table 3) appeared to be more indicative of participants’ reasoning processes in deter-

mining Web site authors’ vested interests. We found that students were most likely to

detect commercial interests of authors. This was shown in comments like, ‘‘There is a link

to their products and services….’’

Table 1 Study 1: Participants’ responses to question: Why did you pick this Web page?

Categories Frequency

Information focus a

27

Education 10

Name recognition a

8

Links 6

Commercial interest/bias 5

Reference 4

Information design a

4

Currency of information a

4

Design look a

3

Expertise 2

Bias a

2

Corroboration 1

Other 1

a Indicates categories adapted from Fogg et al. (2002)

Table 2 Study 1: Participants’ responses to question: What are the Web site authors’ vested interests?

Categories Frequency

Education 14

Commercial 9

Other 5

Persuasion 3

Research and development 2

Name recognition*/promotion 1

*Indicates categories adapted from Fogg et al. (2002)

Web site credibility 51

123

Secondly, they mentioned indications of educational vested interests. These comments

were especially interesting, as some participants appeared to associate educational motives

with absence of vested interest, or what appeared to be objectivity or absence of bias. As

one group stated, when asked what the author’s vested interests were, ‘‘None? It’s an

educational source—ERIC.’’ In response to the next question, they said, ‘‘because it say

[sic] it’s from ERIC Clearinghouse on Urban Education.’’ Another group stated, ‘‘It

appears that the authors’ do not have some alterior [sic] motive behind their presentation,

but instead are just trying to educate people on the cleanroom software process.’’ When

asked, ‘‘How do you know?’’ they elaborated, revealing a negative bias against sales

interests, ‘‘First off, there are no banners or pop-up ads. This is usually a big clue that the

presenter doesn’t really care about the content and is just trying to sell you something.’’

It seems that educational vested interests were regarded as acceptable, while com-

mercial-related vested interests were regarded with suspicion. The suspicion related to

commercial interests seemed understandable but the unquestioning acceptance of educa-

tional interests is a promising area for further research. Participants seemed to not consider

possible persuasion and/or political vested interests that can underlie educational Web

sites. However, participants could have been more likely to have demonstrated awareness

of persuasion or political vested interests had we selected controversial topics for them to

examine.

Participants also mentioned authors’ motives in sharing their own experiences with the

cleanroom technique, i.e., personal testimony, and mentioned other aspects that would

typically be associated an expression of vested interests (e.g., persuasion, and name rec-

ognition/reputation). They commented on whether authors appeared to provide a balanced

presentation of benefits and drawbacks to the cleanroom technique or a one-sided per-

spective. Some participants examined links to other sites as well. Finally our participants

(of a notably smaller sample) were less likely than participants in the Fogg et al. (2002)

study, to mention ‘‘design look,’’ which in the present study was referred to as ‘‘infor-

mation design.’’

Participants also responded to the question, ‘‘How do you know when a Web site’s

content is wrong?’’ (See Table 4). The category of responses with the highest frequency

was, ‘‘corroboration,’’ or the tendency of the Web site’s information to contradict users’

knowledge about the topic or to not agree with other known information accepted as valid

Table 3 Study 1: Participants’ responses to question: How do respondents know about Web site authors’ vested interests?

Categories Frequency

Commercial 7

Education 5

Personal testimony 5

Affiliation 5

Name recognition/reputation a

3

Links 3

Other 3

Persuasion 2

Information bias a

or unbiased 2

Information design a

2

a Indicates categories adapted from Fogg et al. (2002)

52 M. K. Iding et al.

123

(e.g., Web sites, etc.). This is reflected in comments like, ‘‘We try to verify it with another

unbiased source.’’ ‘‘You test what they are saying against a source you are sure is cred-

itable [sic] and compare.’’ ‘‘Generally the validity of a Web site’s content comes into

question when the design personal knowledge/experience contradicts what is being pre-

sented on the Web site….’’ Information focus was also associated with discrediting a Web site’s information. This

included insufficient information: ‘‘They don’t have a lot of facts to back up the

information.’’

An awareness of bias was reflected in comments like, ‘‘When the Web sites seems very

narrow minded and only seem to present one way to do it and that must be the right way.’’

Confidence and competence in Web site evaluations

Students provided self-ratings in the following areas, which were generally high: confi-

dence about detecting misrepresentations on Web sites in general (M = 3.56, on a 5-point scale), confidence about detecting misrepresentations on the Web site that they had

selected as exemplifying misrepresentations (M = 3.43), competence in evaluating the validity of information on their group’s topic (M = 3.57), and competence about evalu- ating information on the Web in general (M = 3.43).

It is important to note that these ratings were elicited only to provide general indications

of participants’ levels of certainty of their determinations around credibility. Thus, the

ratings do not permit analyses of confidence calibration (Liberman and Tversky 1993;

Lundeberg et al. 2000) or discrimination (Lundeberg et al 2000; Lundeberg et al. 1994).

To determine whether computer science or education students had higher self-ratings,

we ran one-way ANOVAs with grouping as the independent variable and self-ratings for

detecting misrepresentations on the Web in general and on evaluating the validity of

information on the Web. We expected computer science students to have higher ratings,

Table 4 Study 1: Participants’ responses to question: How do you know when a Web site’s content is wrong?

Categories Frequency

Corroboration 13

Information focus a

10

Bias a

8

Expertise 6

Reference 6

Information design a

4

Name recognition a /affiliation 4

Inaccuracy 3

Currency of information a

2

Information clarity a

2

Commercial interest 2

Tone a

2

Other links 1

Other 1

a Indicates categories adapted from Fogg et al. (2002)

Web site credibility 53

123

since we assumed that they work with Web sites more. There were no significant differ-

ences between groups on detecting misrepresentations on Web sites in general, or on

evaluating the validity of information on the Web in general.

Conclusion

Findings from Study 1 highlighted contradictions between what some students viewed as

credible and others did not. Students appeared to maintain some confusion about Web site

authors’ vested interests, an area that should be investigated further, as users need to make

determinations about impetus behind presentation of information on Web sites, and possible

biases of Web site authors, even if the Web-based information is educational in nature.

Study 2

Study 2 was carried out to further examine computer science students’ determinations of

credibility, vested interests, and confidence in own ratings. In this study, students were

provided with three Web sites regarding cleanroom procedures.

Method

Participants

Participants were 25 students in graduate-level computer science courses on human

computer interaction and information technology.

Materials and procedure

The authors developed a survey for use in the present study. In the survey, participants

were provided with three Web sites on the topic of ‘‘cleanroom’’ procedures for software

development. These Web sites were chosen from the students’ selections of credible Web

sites and ones containing misrepresentations from Study 1 (some of the same Web sites

that had been selected as credible and accurate by some participants in that study had also

been described as inaccurate by other participants).

The first Web site was geared toward business and contained advertising. The author of

the first Web site was a software consulting service specializing in cleanroom software

engineering. The cleanroom approach was briefly described, followed by several para-

graphs outlining the consultants’ services. The layout of the page was spare, with few

graphics, and the customary menu of links down the left side.

A university professor developed the second Web site. It used no graphics at all, and

consisted almost exclusively of links to the author’s descriptions and examples of clean-

room software development.

The author of a widely used software engineering textbook developed the third Web

site. It also used no graphics. This site was a mixture of text and links to additional

cleanroom software engineering sites. A heading at the top of the site says ‘‘A collection of

Web-based and print resources that will help you understand and explore many different

software engineering topics.’’ A list of cleanroom books, and a link to Amazon.com, is at

the bottom of the page.

54 M. K. Iding et al.

123

In the worksheet, cleanroom engineering is briefly defined: ‘‘The cleanroom technique

supports software engineers designing and developing near-zero-defect software. Using the

cleanroom techniques, software engineers certify the quality of software, MTTF (mean

time to failure) in particular. MTTF is a significant part of end-users’ overall satisfaction.’’

The instructions explained that respondents were not expected to be experts on cleanroom

techniques or Web design, but would be asked to give their opinions on whether the Web

sites that followed were deemed to be ‘‘objective and accurate representations of clean-

room software engineering…or mistaken and purposely misleading.’’ They were asked to explain why, then asked to rate their confidence in their categorizations on a scale of 1 (no

confidence) to 5 (complete confidence). They were also asked to describe each author’s

vested interests or motivation.

At the conclusion of the survey, participants were asked to ‘‘Think about Web sites in

general and answer the following questions: How do you know when a Web site’s content

is wrong? On a scale of 1 (no confidence) to 5 (complete confidence), how confident are

you about detecting misrepresentations on Web sites in general? On the same 1–5 scale,

how competent are you [in] evaluating the validity of information on the Web in general?

On a 1 (no competence) to 5 (complete competence) scale, how competent are you

evaluating the validity of information about cleanroom software engineering?’’

Coding

All written responses were coded collaboratively according to a content analysis scheme

developed by two of the authors, and based in part on the scheme used in Study 1 (see

categorization scheme in the tables below). Categories emerged from the data and others

were adapted from the work of Fogg et al. (2002), as they applied to the present data. Each

separate statement that was determined to represent a complete idea was coded, and

frequencies were computed for each category. Subcategories for negative and mixed (both

negative and positive) comments were also coded.

Results and discussion

Were Web sites objective and accurate, mistaken, or purposely misleading? Participants determined whether each Web site was objective and accurate, mistaken, purposely mis-

leading, and other. Table 5 shows the frequencies for accuracy ratings for each Web site. It

is interesting to note that the majority of answers for the Web sites overall were in the

objective and accurate category. Web site 1 (the most obviously commercial site) had the

most distributed answers. It had the lowest number (10 or 40%) of ‘‘objective and accu-

rate’’ answers, while 10 comments indicated that participants thought the Web site was

‘‘purposely misleading’’ and 2 comments considered it ‘‘mistaken.’’ Web site 2 (the edu-

cational Web site) had 14 comments (or 56% of comments) placing it as ‘‘objective and

Table 5 Study 2: Accuracy ratings for clean room Web sites: frequencies and percentages

Web site 1 commercial Web site 2 educational Web site 3 mixed

Objective and accurate 10 (40%) 14 (56%) 17 (71%)

Purposely misleading 10 (40%) 1 (4%) 2 (8%)

Mistaken 2 (8%) 5 (20%) 2 (8%)

Other 3 (12%) 5 (20%) 3 (13%)

Web site credibility 55

123

accurate,’’ yet only 1 comment indicated it was ‘‘purposely misleading,’’ while 5 com-

ments deemed it as ‘‘mistaken.’’ Web site 3 (another commercial Web site) had 17

comments in the ‘‘objective and accurate’’ category, although 2 comments suggested it was

‘‘purposely misleading,’’ and 2 indicated it was ‘‘mistaken.’’ For both of the commercial

Web sites, but particularly Web site 1, the participants attributed an intention to purposely

mislead to the Web site. However, when they rated the educational Web site, the partic-

ipants were more trusting of the Web site designers’ intent and tended to categorize errors

as mistakes.

After rating the specific Web sites, participants wrote responses to the general question,

‘‘How do you know when a Web site’s content is wrong?’’ The authors coded the major

themes that emerged from this data, which are depicted in Table 6. Notably, the category

with the largest number of comments was about whether users believed they had expertise

in the content area addressed by the site. The next largest category of comments was about

corroboration, or whether information agrees with what the user already knows or with

other information that can be found. Finally, participants commented on other issues

presented such as poor design, bias, problems with references, non-working links, lack of

clarity, inaccuracy and sales pitches.

Some participants related strategies that they use for determining Web site accuracy.

One explained, ‘‘I frequently check multiple Web sites when searching for information, or

rely on trusted sites such as the Encyclopedia Britannica, MIT, SourceForge, NY Time-

s.com, etc.’’ Another said, ‘‘By reading the first three paragraphs and subtitles of Web

sites.’’

Others raised issues about accuracy in general. As one said, ‘‘In reality, I don’t [know

when a Web site is wrong]. The only way that I know is by trying to figure out the

Table 6 Study 2: Reasons for knowing Web site information is inaccurate

Category Frequency for category selection

Expertise of user 12

Corroboration 11

Information design a

6

Information bias a

4

References 4

Information clarity a

4

Information focus a

4

Information accuracy a

3

Directory to other links 3

Affiliations a

2

Name recognition/reputation a

2

Performance on test by user a

2

Design look a

2

Readability a

1

Commercial/sales 1

Motive of organization a

1

Not sufficient links 1

Total 63

a Indicates categories adapted from Fogg et al. (2002)

56 M. K. Iding et al.

123

motivation for the site and to cross-check information across a number of sites with

varying motivations.’’

Another participant raised a similar point, ‘‘There is really no way of knowing that a

Web site’s content is wrong, unless I personally know that there is a mistake or inaccuracy

in the content based on my own knowledge or experience. The same is true for any book I

may check out of the library or the newspaper I read every day. Generally, people tend to

believe everything that they see in print, but as an educated human being, I know that this

is not always the case. So, just as I need to watch out for inaccuracies in a book or

newspaper, I also need to use the same caution when reading a webpage.’’

Finally, a participant raised an issue worth considering, ‘‘The last site’s extensive

references to high quality ‘third party’ sites made it seem very credible. This makes me

wonder if we can always be sure of the quality of referenced sites. In addition, how would a

typical Web surfer even be sure that the referenced sites weren’t merely sites produced by

the authors of the original site, designed to mislead and manipulate?’’

Reasons for accuracy decisions In addition to making determinations of Web site accuracy, participants were asked to provide reasons for their answers. Reasons were coded

Table 7 Study 2: Reasons for Web site accuracy determinations

Web site 1 Commercial 2 Educational 3 Mixed

Commercial/sales 12 (10N) 0 4 (4N)

Information accuracy a

6 (5N) b

6 (2N) 3 (1N)

Information usefulness a

6 (5N) 3 (3N) 1

Not sufficient links 4 (4N) 2 (2N) 4 (4N)

Design look a

2 (1N) 3 (2N) 1 (1N)

Expertise of user 2 1 0

Information bias a

4 (3N, 1M) c

6 (3N) 5 (2N, 1M)

Information clarity a

4 (1N) 9 (4N, 1M) 2 (1N)

Information design a

3 (1M) 2 (1N) 3 (1N)

Educational 2 (1N) 6 4

Readability 2 (1M) 1 0

Currency of information a

1 (1N) 1 (1N) 1 (1N)

Directory to other links 0 0 7

References 0 1 8

Motive of organization 0 0 0

Name recognition/reputation a

0 1 2 (1N)

Corroboration 0 0 0

Affiliations a

0 4 0

Personal testimony 0 4 0

Functionality of site 1 1 (1N) 0

Information focus a

0 0 3 (1N)

Totals 49 (31N, 3M) 51 (19N, 1M) 48 (17N, 1M)

a Indicates categories adapted from Fogg et al. (2002)

b Indicates negative comment

c Indicates negative and positive comment

Web site credibility 57

123

according to the general scheme described earlier. (See Table 7.) Subcategories of negative

and mixed comments were also calculated.

One of the most notable findings was the range of discrepant reasons for accuracy

decisions given by participants. For example, with respect to Web site 1 (the commercial

site), 10 of the accuracy ratings (Table 5) had indicated it was objective and accurate,

while 10 of these ratings had rated it as mistaken or purposely misleading. In examining

reasons for the ratings (Table 7), those who regarded the Web site as purposely misleading

frequently referred to the sales aspect in describing it (12 comments fell into this category,

10 were negative). As one participant stated, ‘‘I believe this site is purposefully misleading

because they are selling and basically provide no information about what the cleanroom

software development really is. They attempt to provide the visitor with the feeling that

this technique is necessary in order to sell themselves.’’

In contrast to the suspicion associated with commercial interests, the next largest cat-

egories in terms of kinds of reasons for accuracy ratings were about information accuracy

(6 comments in total; 5 were negative) and information usefulness (6 comments; 5 neg-

ative). One participant made a positive comment about accuracy: ‘‘I believe this Web site

is accurate and objective, because the ideas presented are cogent and unambiguous. I

checked the links and the same observations are true for the material introduced there as

well.’’ Commenting about the accuracy of the same site, another respondent stated, ‘‘It

doesn’t tell me what percent of defect—they say ‘nearly defect free—’ (Not accurately

conveying info[rmation]).’’ These contrasting perspectives on accuracy could reflect dif-

fering levels of familiarity with software engineering procedures, or more simply, different

perspectives on the same information.

Respondents also made comments about information usefulness, such as, ‘‘The web site

didn’t give any details to describe how cleanroom software engineering works and what

are the advantages compared to other software engineering methods no[r] any example to

show how successful[ly] to implement this technique.’’

Other reasons for determinations fell into categories including insufficient links (4

negative), bias (4 comments; 3 negative; 1 mixed), clarity (4 comments; 1 negative), and

information design (3 comments; 1 mixed). This site had the highest number of comments

that could be categorized as negative (31).

With respect to Web site 2, the educational site, 14 (56%) of accuracy ratings (Table 5)

had associated it with the ‘‘objective and accurate’’ category. Most of those who selected

this category associated their reasons (Table 7) with the educational nature of the site (6

comments) and 6 of other comments had to do with accuracy (2 were negative). As one

respondent cautiously stated, ‘‘I don’t really know how accurate it is, but I would believe

that it is for the most part accurate and legitimate since he is coming from an academic

background and not selling anything. It certainly does not seem to be purposely

misleading.’’

Several commented on personal testimony present in the site (4 comments). These

comments were all positive. As one respondent explained, ‘‘From what I read he seems to

know what he is talking about. One particular quote helped to convince me that he is happy

using the CleanRoom product and appears to be an honest person. The quote is ‘For the

first time in my long career, I can honestly say that I have some confidence that my code

will run correctly in most cases.’’’

A high number of comments were about information clarity. Four were positive, 4 were

negative, and 1 was mixed. Positive comments mentioned clear explanations, ‘‘cogent and

unambiguous’’ ideas; whereas negative comments mentioned aspects of user confusion

generated by specific kinds of information left out of the site.

58 M. K. Iding et al.

123

Finally, although Web site 3 (educational combined with commercial) had the highest

number of accuracy ratings (Table 5) describing it as objective and accurate (17 or 71%),

participants were mixed in their reasons as well. As one stated, ‘‘The site links to outside

information extensively, and while much of it appears to be related, I was uncomfortable

with the lack of original content. The link to IBM seemed completely unrelated to

classroom techniques, and made me very nervous about the site overall. Also, broken links

to bibliographic information contributed to this feeling.’’

In contrast, another said, ‘‘I believe that this Web site is objective and accurate. It is

very informative and contains a lot of details, references, suggested readings and links to

other related sites. This site clearly explains what the Cleanroom Software Engineering

approach is, the history, methods, tutorials and even more. Explanations are scientific,

logical and abundant. I feel that this site is honest and trustworthy.’’

Although this site had more objective and accurate ratings than the other sites, it had 17

negative comments associated with it, which was close to the 19 negative comments

associated with Web site 2.

Web site authors’ vested interests For each of the three Web sites, participants were asked to describe the authors’ ‘‘vested interests.’’ Their comments were coded and the

frequencies are shown in Table 8. As expected, the primary category for Web site 1, the

commercial site, was commercial/sales (23 comments or 74%), and for Web site 2,

the educational Web site, was educational (21 comments or 72%). Finally, Web site 3, the

mixed site, was described by 13 comments (45%) as educational, 10 comments (34%) as

commercial/sales, and 4 comments to serve as a directory to other links. Although par-

ticipants were clearly aware of the commercial vested interests associated with Web site 1,

and educational vested interests associated with Web site 2, they appeared divided about

Table 8 Study 2: Categories of comments regarding Web site authors’ vested interests

Web site 1 commercial Web site 2 educational Web site 3 mixed Category Frequency (percentage)

b

Commercial/sales 23 (74%) 0 10 (34%)

Educational 4 (13%) 21 (72%) 13 (45%)

Motive of organization a

0 1 (3%) 1 (3%)

Affiliations 1 (3%) 0 0

Functionality of site a

1 (3%) 0 0

Information design a

1 (3%) 0 0

Name recognition/reputation a

0 0 0

Not sufficient links 1 (3%) 0 0

Personal testimony 0 2 (7%) 0

Information usefulness a

0 3 (10%) 0

Information accuracy a

0 2 (7%) 0

Directory to other links 0 0 4 (14%)

Design look a

0 0 1 (3%)

Information bias a

0 0 0

Total 31 29 29

* Indicates categories adapted from Fogg et al. (2002)

** Indicates frequency per category and percentage of total comments per Web site

Web site credibility 59

123

the nature of vested interests for Web site 3. This is particularly intriguing, because in this

study, as in Study 1, commercial vested interests were regarded with more suspicion than

educational interests, which seemed to be frequently perceived as bias-free. Therefore,

those who were unaware of commercial vested interests in Web site 3 may have associated

it with ‘‘bias free’’ determinations that participants attached to educational Web sites in

general.

Cleanroom competence and confidence ratings As part of the study, participants rated their confidence in determinations of Web site accuracy for each of the three sites. Spe-

cifically, they answered the question, ‘‘On a scale of 1 (no confidence) to 5 (complete

confidence), how confident are you about your categorization and answer above?’’ Mean

ratings were similar for the Web site 1 (commercial site) (M = 3.76) and Web site 2 (educational) (M = 3.74), and were higher for the third Web site (M = 4.25). Since this site received the highest number of ‘‘objective and accurate’’ ratings, it may be that the

association of the site with an academic textbook author on the topic could have been

associated with increased confidence. However, the number of participants in this study

was limited, providing similarly limited indications of differences in confidence ratings.

Participants rated their ‘‘confidence in detecting misrepresentations on Web sites in

general’’ as reasonably high (M = 3.60). Next they rated their competence in detecting the validity of information on the Web in general as high as well (M = 3.71). Finally, they rated their competence in ‘‘evaluating the validity of information about cleanroom software

engineering’’ as lower (M = 3.06). These findings are interpreted cautiously, as they are not tied to actual accuracy ratings,

and have the same limitations that are described with reference to confidence ratings in

Study 1. However, self-determinations of confidence and/or competence may play an

interesting role in Web site credibility determinations. In particular, university students and

others may rely more heavily on self-confidence and competence determinations in

credibility judgments about Web resources, especially in areas where they are just

becoming acclimated into a discipline or attaining competence (initial stages in developing

expertise, according to Alexander’s 2003 model). This would be an interesting area for

further research.

Directions for further research

Areas for further research could also involve examining credibility determinations in

conjunction with levels of expertise in different disciplines and in connection with different

tasks. For example, how do people’s strategies differ in potentially high cost or high stakes

decision-making and in lower cost/stakes scenarios?

Other interesting directions for further research could involve disentangling some of the

seeming contradictions involved in participants’ different ratings of the same sites by

asking more detailed questions, for example, about links to other sites or references.

Although it is possible that participants judged the same links differently, it is also possible

that they visited different links. Highly controlled laboratory studies (outside the scope of

the present research) with sample Web sites and links that are visited by all participants

could provide further answers to this question.

Cross-cultural comparisons could add interesting information in elucidating whether

bases for credibility determinations are the same or different, depending upon cultural

context. Further, do these determinations vary across content areas or disciplines?

60 M. K. Iding et al.

123

Additionally, examining credibility determinations developmentally is imperative. As

Metzger and Flanagin (2008) contend, ‘‘Although research on credibility and new media is

burgeoning, extremely little of it focuses on youth (with the exception of college students),

in spite of this population’s exceptional immersion in digital technologies’’ (p. 1).

Finally, people’s determinations of Web site authors’ vested interests should be studied

further, as it appears to be an area about which participants know little. For example some

of these participants appeared to presume that educational vested interests are neutral and

bias-free. It would be interesting to explore more controversial or political Web sites than

the rather neutral cleanroom software engineering sites.

Instructional implications

Instructional implications of this work are clearly apparent. These include teaching critical

Web site and general information evaluation skills at all levels, practicing with actual Web

sites and actual content, and applying strategies to a range of different kinds of content in

different areas. Rather than performing brief short-term interventions, Web site evaluation

should be addressed repeatedly at appropriate times throughout a school term (Iding et al.

2002b).

As Nguyen (2002) has pointed out, many k-12 teachers mistakenly assume that critical

Web evaluation skills have already been taught, usually by language arts teachers. Edu-

cators may also assume that directly addressing issues of Web credibility is therefore

unnecessary. Our present research indicates that even at the university level, improved

awareness of factors contributing to Web site and information credibility is needed.

In addition to these suggestions, many of which merely reiterate what is known about

effective strategy training generally, some specific recommendations emerge form the

present research:

• Students and others need to learn to critically evaluate information from educational sources as critically as they evaluate information from commercial sites. They need to

be aware that even educational materials are imbued with biases. Simply because a site

has edu or gov attached to a URL, it certainly cannot considered to be free of bias.

• Students also need to consider vested interests of Web site authors. Why was the information placed on the Web? What information is being emphasized and what is

being left out (deliberately or otherwise)? How do presentation aspects like layout and

design look affect what is being emphasized or de-emphasized? Although commercial

vested interests appear to be the easiest to detect, other kinds of influence (e.g.,

political, etc.) should be considered.

• Students need to learn effective information corroboration skills—finding ways to determine information accuracy by comparing it to other information sources. The task

is more difficult in areas where we do not possess expertise, but general critical

information evaluation skills will be helpful in these areas of non-expertise, as we have

seen in the present research.

• Students need to check references and links. • Students need to learn about how research is refereed in academic fields and why non-

refereed information may be suspect.

• Students and general Internet users need to take on roles as arbiters of information accuracy, rather than merely as seekers of relevant information, despite the fact that

making effective accuracy or credibility determinations may take more time and effort.

Web site credibility 61

123

Conclusion

In conclusion, this work contributes to the research on Web site credibility by elucidating

reasons university students provide for determining that Web sites are credible or not

credible, by studying associated self-ratings of confidence in those determinations and by

examining participants’ understandings of vested interests of Web site authors. This

general area of research in credibility judgments and Web site evaluation is important as it

adds to our understanding of how people determine information on the Web is truthful or

not, and provides a basis from which educators can begin to develop ways to address

students’ needs for accurate and useful Web evaluation (as well as general information

evaluation) skills.

References

Alexander, P. A. (2003). The development of expertise: The journey from acclimation to proficiency. Educational Researcher, 32(8), 10–14.

Benotsch, E. G., Kalichman, S., & Weinhardt, L. S. (2004). HIV–AIDS patients’ evaluation of health information on the Internet: The digital divide and vulnerability to fraudulent claims. Journal of Consulting and Clinical Psychology, 72(6), 1004–1011.

Crosby, M. E., Iding, M. K., Auernheimer, B., & Klemm, E. B. (2002). Judging the veracity of Web sites. Proceedings of the International Conference on Computers in Education (ICCE 2002) (pp. 251–252).

Escoffery, C., Miner, K., Adame, D., Butler, S., McCormick, L., & Mendell, E. (2005). Internet use for health information among college students. Journal of American College Health, 53(4), 183–188.

Farah, B. (1995). Information literacy: Retooling evaluation skills in the electronic information environ- ment. Journal of Educational Technology Systems, 24(2), 127.

Flanagin, A. J., & Metzger, M. J. (2007). The role of site features, user attributes, and information verification behaviours on the perceived credibility of web-based information. New Media Society, 9, 319–342.

Fogg, B. J., Soohoo, C., Danielson, D., Marable, L., Stanford, J., & Tauber, E. R. (2002). How do people evaluate a Web site’s credibility: Results from a large study. Retrieved 2 Oct 2008 from http://www. consumerwebwatch.org/dynamic/web-credibility-reports-evaluate-abstract.cfm.

Iding, M., & Klemm, E. B. (2005). Pre-service teachers critically evaluate scientific information on the World Wide Web: What makes information believable? Computers in the Schools, 21(1/2), 7–18.

Iding, M. K., Auernheimer, B., Crosby, M. E., & Klemm, E. B. (2002a). Users’ confidence levels and strategies for determining Web site veracity. Proceedings of the WWW 2002: The Eleventh Interna- tional World Wide Web Conference. [CD ROM-Author index, 1–3].

Iding, M., Landsman, R., & Nguyen, T. (2002b). Critical evaluation of scientific websites by high school students. In D. Watson & J. Anderson (Eds.), Networking the learner: Computers in education: Seventh IFIP World Conference on Computers in Education Conference Proceedings. Dordrecht, Netherlands: Kluwer Academic Publishers.

Kalichman, S. C., Cherry, C., Cain, D., Pope, H., Kalichman, M., Eaton, L., et al. (2006). Internet-based health information consumer skills intervention for people living with HIV/AIDS. Journal of Con- sulting and Clinical Psychology, 74(3), 545–554.

Klemm, E. B., Iding, M., & Speitel, T. (2001). Do scientists and teachers agree on the credibility of media information sources? International Journal of Instructional Media, 28(1), 83–91.

Liberman, V., & Tversky, A. (1993). On the evaluation of probability judgments: Calibration, resolution, and monotonicity. Psychological Bulletin, 114, 162–173.

Lundeberg, M.A., Fox, P. W., Brown, A., & Elbedour, S. (2000). Cultural influences on confidence: Country, and gender. Journal of Educational Psychology, 92, 152–159.

Lundeberg, M., Fox, P. W., & Puncochar, J. (1994). Highly confident, but wrong: Gender differences and similarities in confidence judgments. Journal of Educational Psychology, 86, 114–121.

Metzger, M. J., & Flanagin, A. J. (2008). Introduction. In M. J. Metzger & A. J. Flanagin (Eds.), Digital media, youth, and credibility (pp. 1–4). Cambridge: MIT Press.

Metzger, M., Flanagan, A., & Zwarun, L. (2003). College student Web use, perceptions of information credibility, and verification behavior. Computers and Education, 41, 271–290.

62 M. K. Iding et al.

123

Nguyen, T. T. (2000). OASIS: Student evaluation methods for World Wide Web resources. Unpublished master’s thesis, University of Hawai‘i, Honolulu, Hawaii, USA.

Princeton Survey Research Associates (2002). A matter of trust: What users want from web sites. Retrieved 2 Oct 2008 from http://www.consumerwebwatch.org/dynamic/web-credibility-reports-a-matter-of- trust-abstract.cfm.

Rader, H. (1998). Library instruction and information literacy. Reference Service Review, 26(3/4), 143. Stanford, J., Tauber, E. R., Fogg, B. J., & Marable, L. (2002). Experts vs. online consumers: A comparative

credibility study of health and finance web sites. Retrieved 2 Oct 2008 from http://www.consumer webwatch.org/dynamic/web-credibility-reports-experts-vs-online-abstract.cfm.

Tormala, Z. L., & Petty, R. E. (2004). Source credibility and attitude certainty: A metacognitive analysis of resistance to persuasion. Journal of Consumer Psychology, 14(4), 427–442.

Web site credibility 63

123

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.