reading response

Patient Education and Counseling 85 (2011) e16–e25

Review

The quality of mental disorder information websites: A review

Nicola J. Reavley *, Anthony F. Jorm

Orygen Youth Health Research Centre, Centre for Youth Mental Health, University of Melbourne, Parkville, Parkville, Australia

A R T I C L E I N F O

Article history:

Received 22 June 2010

Received in revised form 10 September 2010

Accepted 14 October 2010

Keywords:

Internet

Website

Mental disorder

Quality

Depression

Anxiety

A B S T R A C T

Objective: This paper reviews studies assessing the quality of websites providing information about

mental disorders.

Methods: The review included 31 articles identified by searching research databases in March 2010.

Topics covered included affective disorders, anxiety disorders, eating disorders, substance use disorders

and schizophrenia/psychosis.

Results: The largest number of articles (13) reported studies assessing affective disorder information

quality. Methodologies varied in site selection and rating methods, with some of limited validity. Most

concluded that quality was poor, although quality of affective disorder sites may be improving.

Conclusion: There is currently very little understanding of the influence of website quality on user

behaviour. Future quality assessments might use the criteria informed by key behaviour change theories.

Practice implications: A possible approach to research on websites and user behaviour might be to

develop an evaluation framework incorporating strategies from behaviour change models, key mental

health literacy elements and health outcomes relevant to mental health promotion.

� 2010 Elsevier Ireland Ltd. All rights reserved.

Contents lists available at ScienceDirect

Patient Education and Counseling

j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / p a t e d u c o u

1. Introduction

Current estimates suggest that over 1.8 billion people have access to over 2.5 million sites on the Internet [1,2]. As many as 80% of Internet users in developed countries use the Internet to search for health information, typically to find information on conditions, symptoms, diseases and treatments [3,4]. Information on mental disorders is commonly accessed online, particularly by those with a psychiatric diagnosis or their carers [5–7]. The increase in health information on the Internet has been closely followed by an increase in the number of studies analysing the quality of that information. Concerns about information quality led to proposals for ways of assessing this, with an early focus on accountability [8]. Since then other assessment measures have been developed and while these vary, most cover one or more of the following: accuracy, completeness, readability, accountability and design and technical criteria [9,10]. One of the most commonly used is DISCERN, a 16 item checklist designed to assist consumers to assess the quality of health information [11]. Codes of conduct have also been developed, for example the e-Health Code of Ethics and the Health on the Net Foundation [12,13]. In a 2002 review of studies of quality, Eysenbach et al. reported that 70% considered quality a problem [9]. However,

* Corresponding author at: Orygen Youth Health Research Centre, Centre for

Youth Mental health, University of Melbourne, Locked Bag 10, Parkville, Victoria

3052, Australia. Tel.: +61 3 9342 3769; fax: +61 3 9342 2941.

E-mail address: [email protected] (N.J. Reavley).

0738-3991/$ – see front matter � 2010 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.pec.2010.10.015

there is evidence that accuracy varies between health domains, with Eysenbach et al. noting that up to 90% of diet and nutrition information was unreliable compared to only 5% of that for cancer. Given the swift evolution of the Internet and a relatively recent focus on information quality, there is a need for a clearer picture of the quality of available mental disorder information, including differ- ences between disorders and types of sites. Such information may be useful for healthcare practitioners in discussions with patients who use the Internet and for organisations involved in mental health promotion. The primary aim of this paper is to review studies assessing the quality of websites providing information about mental disorders, including depression, anxiety disorders, eating disorders, substance use disorders, schizophrenia and psychosis. Further aims are to examine quality assessment methods, correlates of quality, methods of improving it and the influence of quality on user behaviour.

2. Methods

Studies to review were identified by searching PubMed, PsycInfo and the Cochrane Database of Systematic Reviews in March 2010. Search terms were (anxiety OR depression OR bipolar disorder OR psychosis OR schizophrenia OR mental disorder OR alcohol OR substance misuse OR dementia) AND (Internet OR web OR websites) AND (quality OR accuracy OR evaluation) AND (information OR education). Full text articles were retrieved if they were in English. Information from abstracts of articles in languages other than English was included in the review.

N.J. Reavley, A.F. Jorm / Patient Education and Counseling 85 (2011) e16–e25 e17

3. Results

The search retrieved 450 articles and their titles and abstracts were screened for relevance (see Fig. 1). Articles were excluded if they evaluated the effects of web-based interventions. The studies evaluating quality of websites giving information on smoking cessation were also excluded. We also checked the references of all identified studies and retrieved articles not in the original search. This led to the inclusion of further five articles. In total we retrieved 27 full text articles and assessed these for eligibility. Two were subsequently excluded, as they did not involve any assessment of quality. A further article by one of the authors (AFJ) was added when it was accepted for publication. Thirty-one articles were included in the final review. The largest number of articles (13) reported studies assessing the quality of information on affective disorders (see Table 1) [14–26]. Four studies assessed the quality of information on substance misuse [27–30] and three assessed information on anxiety disorders [31–33]. Other topics included schizophrenia/psychosis [34,35], suicide prevention [36], gam- bling [37], eating disorders [38], female hypoactive sexual desire [39], psychological trauma [40], dementia [41], St John’s wort [42] and attention deficit hyperactivity disorder (ADHD) [34,43]. Five of the articles were in languages other than English and the information given in Table 1 is from the English language abstracts [27,33,35,41,44].

3.1. Methods of assessing quality

Study methodologies varied in their site selection and rating methods. Some studies assessed all available websites that met their criteria, which varied according to the study [19,21– 23,25,26,28–30,32,37,39,42,45], while others limited their assess- ment to the top 10, 20 or 50 results given by common search

450 rec ords iden�fied through database searching

450 rec ords screened by �tle and abstract

27 full-text ar� assessed for e

31 studies included in review

22 full-text ar�cles retrie

Abstracts of 5 foreign langu age ar�cles reviewed and included

1 ar�cle by one of the study authors included on acceptance for publica�on

Fig. 1. Flow diagram for review of studies of qu

engines [14–18,20,31,34,36,38,40,43]. Rating methods typically involved the use of expert assessment and rating instruments. All the studies, with the exception of that by Bremner et al. [40], which used a single rater, used rating by two or more experts. Griffiths and Christensen [19,20] involved consumers in the rating and Griffiths et al. [21] used a computer algorithm for the automated assessment of evidence-based quality of depression treatment information.

3.2. Validity of quality assessments

The Silberg scale, which assesses accountability, and DISCERN were the most commonly used quality assessment tools [8,11]. However, evidence for links between these instruments and overall site quality is mixed. In their assessments of depression sites, Griffiths and Christensen [18] and Khazaal et al. [30] did not find significant associations between Silberg scores and site quality. DISCERN is designed to be used by consumers without content expertise and therefore does not assess scientific quality or accuracy of evidence. A small number of studies did not find associations between site quality and DISCERN scores [23,29,30,37] while others did [14,19,20,26,31,32,43].

In addition, a number of studies reviewed here incorporated the development of rating instruments, a practice of questionable value, as such instruments are rarely validated or sustainable. In 1998, Jadad and Gagliardi [10] reviewed 47 Internet health information rating instruments, none of which provided informa- tion on the interobserver reliability and construct validity of the measurements. In a follow-up study done in 2002, 98 instruments were identified. Of 51 newly identified rating instruments, only five provided some information by which they could be evaluated and none of these were validated [46]. This limits the ability to draw conclusions from such studies.

428 rec ords excluded if they did not assess quality, if they ev aluated the effects of interve n�ons or if they evaluated smoking cessa�on websites

cles ligibility

2 full-text ar�cles excluded as they did not involve assessme nt of informa�on quality

ved

5 addi�onal full-text ar�cles retr ieve d a�er checking references

ality of Internet mental health information.

Table 1 Summary of studies of quality of mental disorder information on the Internet.

Material evaluated Time period Measures Mean scores

(where available)

Overall conclusions

All mental disorders Nemoto et al. [45] Japanese language websites dealing

with mental health and mental

disorders (37 websites)

May 2005 DISCERN (15–75)

Global score (1–10)

45.4 (16.8)

4.0 (9.3)

Information on mood disorder,

panic disorder, and

schizophrenia most common.

Quality of information mostly

inadequate, especially

that regarding treatment.

Substance misuse Faden et al. [28] US college online alcohol policy

information (52 websites)

– Comparison of 2002 and 2007 Information more available and

accessible to the students and

other interested parties in

2007 than 2002

Coquard et al.

[27] [French]

Quality of French language

information on alcohol dependence

– Standardised pro forma Overall quality of sites poor

Khazaal et al. [29] Quality of information on cannabis

use (57 websites)

June 2007 Health on the net (HON) quality labela Overall quality of sites poor

Silberg score (0–9) 3.9 (2.4)

Interactivity score (0–6) 1.9 (1.4)

Abbot aesthetic criteria (0–4) 3 (0.8)

Flesch reading ease score (0–100) 43.7 (15.5)

Flesch–Kincaid reading grade

education scores

10.5 (3.3)

Brief DISCERNb (16–20) 30.2 (8.7)

Content quality scores (0–20) 8.0 (3.9)

Global scores (0–39) 16.5 (5.3)

Khazaal et al. [30] Quality of information on cocaine

addiction (120 websites)

July 2007 Health on the net (HON) quality labela Overall quality of sites poor

Silberg score (0–9) 3.8 (2.2)

Interactivity score (0–6) 1.6 (1.1)

Abbot aesthetic criteria (0–4) 3 (0.8)

Flesch reading ease score (0–100) 42.2 (12.2)

Flesch–Kincaid reading grade

education scores

10.5 (2.1)

DISCERNb (16–80) 32.1 (8.9)

Content quality scores (0–20) 9.6 (5)

Global scores (0–39) 18 (6)

Affective disorders Zermatten et al. [26] Quality of information on depression

(45 websites)

February 2009 Health on the net (HON) quality labela Overall quality of sites good

Silberg score (0–9) 6.47 (2.2)

Interactivity score (0–6) 3.2 (1.47)

Abbot aesthetic criteria (0–4) 2.44 (0.76)

Flesch reading ease score (0–4) 40.07 (9.73)

Flesch–Kincaid reading grade

education scores

12.1 (1.78)

DISCERNb (16–80) 30.2 (8.7)

Content quality scores (0–20) 8.0 (3.9)

Global scores (0–39) 16.5 (5.3)

Barnes et al. [14] Quality of information on bipolar

disorder (15 websites)

August 2007 DISCERN

Bipolar Website Quality Checklist (BWQC)

Overall quality of sites good

Websites with an editorial board

or affiliation

to a professional

organisation had higher

quality information

N .J.

R e a

v le

y ,

A .F

. Jo

rm /

P a

tie n

t E

d u

ca tio

n a

n d

C o

u n

se lin

g 8

5 (2

0 1

1 )

e 1

6 –

e 2

5 e

1 8

Morel et al. [23] Quality of information on bipolar

disorder (34 websites)

November 2006 Health on the net (HON) quality labela Overall quality of sites good

Silberg score (0–9) 4.8 (2.2)

Interactivity score (0–6) 1.8 (1.2)

Abbot aesthetic criteria (0–4) 2.8 (0.6)

Flesch reading ease score (0–100) 30.1 (12.7)

Flesch–Kincaid reading grade

education scores

10.3 (2)

DISCERNb (16–80) 43.7 (11.9)

Content quality scores (0–28) 19 (4.8)

Global scores (0–39) 38.3 (6)

Ferreira-Lay and

Miller [17]

Quality of information on depression

(23 websites)

– Depression website content checklist Higher quality information from

websites of government,

professional and charitable

organisations.

Seyringer et al.

[24] [German]

Quality of German language information

on bipolar disorder and manic depressive

disorder

– A range of criteria considering

form and content

Comprehensive information on

the nature of the

illness more frequent in sites

resulting from the search term

‘‘manic-depressive disorder’’.

The term ‘‘bipolar disorder’’

produced more results offering

information on evidence-based

therapeutic

strategies.

Stjernsward and

Ostman [25]

Information for the relatives and significant

others of depressed individuals (22 websites)

November 2004 Grounded Theory methodology Some websites contained valuable

information and

partly addressed known needs.

Griffiths et al. [21] Quality of depression websites Computer algorithm for the automated

assessment of evidence-based quality

of treatment information (AQA)

Correlation between AQA and

human ratings was high leading

to the

conclusion that quality assessment

can be automated.

Griffiths and

Christensen [20]

Quality of depression websites April 2003 Sites rated by health professionals using

evidence-based criteria.

Health professionals and consumers

rated sites using DISCERN

Google PageRank

Overall scores low for both health

professionals

and consumers. Correlation

between evidence-based guideline

score and DISCERN.

Griffiths and

Christensen [19]

Quality and accessibility of Australian

depression sites (15 websites)

November–December 2001 Quality of treatment content (concordance with

evidence-based guidelines, conformity with

DISCERN criteria, citation of scientific evidence)

Accessibility (search engine rank)

Mean content quality scores were

not high and site

accessibility was poor.

Berland et al. [15] Depression information in English

and Spanish (25 websites)

July–December 2000 Accessibility

Quality (topics and questions developed

by expert panel)

Readability (Flesch–Kincaid) criteria

Coverage of key information

generally poor and inconsistent.

Accuracy of the information

generally good.

Lissman and

Boehnlein [22]

Depression information (178 sites) April 2000 Rating according to mention of the nine symptoms

and five major criteria of a major

depressive episode and mention of any of

three basic treatment recommendations.

Overall quality of sites poor.

For-profit websites appeared

much more

frequently than

not-for-profit sites among the

first 20 sites generated by

each search

engine, and they

contained poorer information.

N .J.

R e a

v le

y ,

A .F

. Jo

rm /

P a

tie n

t E

d u

ca tio

n a

n d

C o

u n

se lin

g 8

5 (2

0 1

1 )

e 1

6 –

e 2

5

e 1

9

Table 1 (Continued )

Material evaluated Time period Measures Mean scores

(where available)

Overall conclusions

Christensen

et al. [16]

Depression information (21 websites) March–July 1999 Treatment types and treatment

categories compared across commercial,

professional and consumer websites

Antidepressant medication and

psychotherapy

noted by all sites.

Consumer sites mentioned

psychotherapy

less frequently but did not

mention complementary

treatments

more frequently.

Griffiths and

Christensen [18]

Depression information (21 websites) March 1999 Site characteristics

Quality (concordance with evidence-

based guidelines)

Accountability (Silberg score),

Level of evidence scores

Overall quality poor

Websites with an editorial board

or affiliation

to a professional

organisation had higher quality

information

Mean Silberg 5.4 (2–9)

Global score 2.95/10 (0.5–7.5)

Anxiety disorders Khazaal et al. [32] Quality of information on social

phobia (58 websites)

July 2006 Health on the net (HON) quality labela Overall quality of sites poor

Silberg score (0–9) 3.3 (.7)

Interactivity score (0–6) 1.3 (1.0)

Abbot aesthetic criteria (0–4) 1.9 (0.8)

Flesch reading ease score (0–100) 53.8 (12.7)

Flesch–Kincaid reading grade education scores 7.3 (2)

DISCERNb (16–80) 35.6 (13.9)

Content quality scores (0–28) 11.4 (7.3)

Global scores (0–39) 18 (8.3)

Ipser et al. [31] Quality of information on anxiety

disorders (67 websites)

– DISCERNb

Websites also compared on popularity,

readability and a range of technical criteria

Websites were generally of poor

to moderate quality.

Higher quality scores obtained

by those whose authors attributed

the sources and clearly stated

the purpose of the website.

Serdobbel et al.

[33] [Dutch]

Quality of Dutch-language

information on OCD

– DISCERNb

Google PageRank

Quality of information

generally poor

Schizophrenia/ psychosis

Schrank et al.

[35] [German]

Quality of German-language information

on schizophrenia

– Evaluation of a range of criteria,

including diagnosis

and therapy, links and interactive offer

Evidence-based medical

information was provided

by more than half of the sites

resulting from the search term

schizophrenia and by less

than one-third of psychosis hits.

Kisely et al. [34] Quality of information on the treatment

of schizophrenia

(90 websites)

June–September 2001 Accountability (Silberg score),

Presentation and readability (Flesch–Kincaid

Grade Level score)

3.2 (0–9)

1.9 (0–4)

11.5 (6.25–12.25)

Accountability, presentation and

readability, were poor.

Agreement with evidence-based

practice for low sites

in the top 10% of scores were

significantly

more likely to be owned by an

organisation or have an

editorial board than those

in the bottom 10%.

Other

N .J.

R e a

v le

y ,

A .F

. Jo

rm /

P a

tie n

t E

d u

ca tio

n a

n d

C o

u n

se lin

g 8

5 (2

0 1

1 )

e 1

6 –

e 2

5 e

2 0

Jorm et al. [36] Effect of feedback on the quality of suicide

prevention websites:

randomised control trial (52 websites)

November 2008 Websites scored on a 26 item quality

checklist and feedback

given to 26 randomly assigned websites

Many sites found to be low

quality. Feedback to website

administrators did not improve

quality.

Khazaal et al. [37] Quality of web-based information on

pathological gambling

(75 sites)

September 2007 Health on the net (HON) quality labela Overall quality of sites poor

Silberg score (0–9) 4 (2)

Interactivity score (0–6) 2.07 (1.04)

Abbot aesthetic criteria (0–4) 2.8 (0.6)

Flesch reading ease score (0–100) 48.8 (10.8)

Flesch–Kincaid reading grade education scores 8.5 (2.1)

DISCERNb (16–80) 35.3 (8.2)

Content quality scores (0–28) 12.5 (4.9)

Global scores (0–39) 21.6 (6)

Akram et al. [43] UK websites providing information about ADHD and its

pharmacological management (48 websites)

January 2007 Strathclyde Website evaluation

Form

Flesch reading ease score

DISCERNb

20.4 (5.5) Information basic and incomplete.

Websites by government and

professional bodies better than

other categories

Touchet et al. [39] Quality of web-based information on female hypoactive sexual

desire disorder (HSDD) (24 websites)

– Tool incorporating expert consensus-derived quality

criteria for HSDD, including structural

criteria (currency, authorship, disclosure of competing

interests) and performance criteria

(accuracy, and comprehensiveness).

Websites ranked by quality score.

Results indicated room for

improvement in

the quality of majority of websites

Bremner et al. [40] Quality of Internet sites related to the topic of psychological

trauma (80 websites)

– Published guidelines for evaluating

medical information

Websites providing information

about psychological

trauma are often

not useful, and can sometimes

provide inaccurate and

potentially harmful information.

Seomun et al.

[41] [Korean]

Quality of Korean language information on Dementia – Construction

Accessibility

Operation

Contents

Overall quality of sites poor

Murphy et al. [38] Quality of information on the treatment of

eating disorders (15 websites)

January 2003 Two reviewers evaluated the characteristics,

quality of content, and

accountability of the sites

Silberg score

Global score

2.7 (0–7)

2.0 (0–5) max 10

Overall quality of sites poor

Kisely et al. [34] Quality of information on the treatment of ADHD (92 sites) June–September 2001 Accountability (Silberg score)

Presentation

Readability (Flesch–Kincaid Grade Level score)

3.2 (0–9)

1.9 (0–4)

11.5 (6.25–12.25)

Accountability, presentation and

readability, were poor.

Agreement with evidence-

based practice for

low Sites in the top 10% of scores

were significantly more likely to

be owned by an

organisation or have an editorial

board than those in the

bottom 10%.

Martin-Facklam

et al. [42]

Quality of information on St John’s wort (208 websites) July 2000 HON quality labela

Statement of correct indication and

mentioning of interacting drugs

Quality generally poor.

Non-commercial sites preferable.

a Health on the net (HON) quality label indicates websites which include the author’s credentials, the date of the last modification with respect to clinical documents, confidentiality of data, source data reference,

funding and the advertising policy [65]. b DISCERN – an instrument that aims to identify written health-related material of good content quality [11,66].

N .J.

R e a

v le

y ,

A .F

. Jo

rm /

P a

tie n

t E

d u

ca tio

n a

n d

C o

u n

se lin

g 8

5 (2

0 1

1 )

e 1

6 –

e 2

5

e 2

1

N.J. Reavley, A.F. Jorm / Patient Education and Counseling 85 (2011) e16–e25e22

3.3. Variation in quality across topics

Twenty-three studies concluded that the overall quality of sites was inadequate. This was particularly notable for reviews of studies of the quality of information on anxiety disorders and mental disorders other than affective disorders. The highest quality information appears to be that relating to bipolar disorder, with two studies concluding that information was generally of good quality [14,23]. A comparison of two of the studies using the same methodology reveals that the quality of information on bipolar disorders was higher than that for depression, with mean (SD) DISCERN scores of 43.7 (11.9) and 30.2 (8.7) respectively [23,26]. However, a number of sites giving information on bipolar disorder were assessed in one of these papers [26], which may partly explain the higher quality rating for depression websites in that paper compared to others assessing depression site quality. A number of the papers listed the top-rated websites for the topic area. These are given in Table 2.

3.4. Is quality improving?

Studies reviewing the quality of affective disorder information were carried out between 2000 and 2010. While earlier studies generally concluded that information quality was poor, the four most recent studies, two of which assessed information on bipolar disorder, concluded that the overall quality of website information was good [14,17,23,26]. While study methodologies and quality indicators differed, it may be reasonable to conclude that there has been an improvement in information quality over time, particu- larly in the case of more frequently assessed topics.

Table 2 Top-rated currently available websites for mental disorders.

Depression Top websites (2010) [26]

Medicinenet.com http://www.medicinenet.c

Netdoctor.co.uk http://www.netdoctor.co.u

The Royal College of Psychiatrists http://www.rcpsych.ac.uk/

Top websites (2008) [17]

National Health Service (NHS) www.nhs.uk/depression

National Institutes of Mental Health (NIMH) www.nimh.nih.gov/health

www.nimh.nih.gov/public

www.nimh.nih.gov/medlin

SANE www.sane.org.uk/About__

www.sane.org.uk

Top Australian websites (2002) [19]

beyondblue www.beyondblue.org.au

BluePages www.bluepages.anu.edu.a

University of New South Wales Clinical Research

Unit for Anxiety and Depression (CRUfAD)

www.crufad.unsw.edu.au

Bipolar disorder (2009) [14] Black Dog Institute www.blackdoginstitute.org

National Institutes of Mental Health (NIMH) www.nimh.nih.gov

bipolar.about.com www.bipolar.about.com

ADHD Top UK websites (2008) [43]

BMJ Publishing Group Best Treatments www.besttreatments.co.uk

NHS Direct Health Encyclopaedia ADHD www.nhsdirect.nhs.uk/art

Pharmaceutical company Janssen–Cilag Products www.janssen-cilag.co.uk/b

Top websites (2003) [34]

National Institutes of Mental Health (NIMH) http://www.nimh.nih.gov/

Interactive metronome http://www.interactiveme

Northern County Psychiatric Associates http://www.baltimorepsyc

Schizophrenia (2003) [34] Medical Journal of Australia http://www.mja.com.au/p

Treatment Advocacy Centre http://www.psychlaws.org

Healthcommunities.com http://www.mentalhealthc

Suicide prevention (2010) [36] Suicide.org www.suicide.org

Suicideline www.suicideline.org.au

3.5. Correlates of content quality

Several studies investigated the links between site owner characteristics and content quality. Five studies found that higher quality information came from websites of government, profes- sional and charitable organisations [14,17,22,31,43], while five others found that site ownership did not predict quality [23,29,30,32,37]. Other criteria associated with higher quality included editorial boards, having information on a variety of mental health issues, having internal search engines, mentioning scientific evidence or citation of references and absence of financial interest. Sites by professional organisations tended to recommend one type of provider, perhaps limiting their quality scores. Evidence for association between quality labels (e.g. seals of approval) and quality score was mixed, as was evidence for links between search engine page ranks and quality scores [20,47].

3.6. Methods of improving quality

While there has been discussion of a number of proposals for improving quality, there is no clear consensus on the best way to accomplish this. Early concerns about the quality of health information on the Internet led to proposals for labelling and filtering information [48]. Codes of conduct have also been developed and graphic images denoting an award or approval seal appear on some websites [49]. However, there is no way to stop websites using such images without permission. It is likely that such an approach would be more useful if there were a consistent, universal and well-recognised way of indicating high quality websites. A number of other technological solutions have been proposed, for example, the use of web browser encryption

om/depression/article.htm

k/depression/index.shtml

default.aspx?page50

/publications/depression/complete-publication.shtml N2 10 71

at/depressionmenu.cfm

eplus/depression.html

_Mental___Illness/Depression.htm N3 10 66

u

.au

/btuk/conditions/10235.jsp

icles/article.aspx?articleId=40&sectionId=27010

gdisplay.jhtml?itemname=parents_background_adhd&_requestid=1343815&s=1

publicat/adhd.htm

tronome.com/abstracts/NIH/NIHConcConf.htm

h.com/adhd.htm

ublic/issues/feb3/carr/carr.html

/

hannel.net/schizophrenia

N.J. Reavley, A.F. Jorm / Patient Education and Counseling 85 (2011) e16–e25 e23

icons, and automated quality assessments, but these face significant barriers to implementation [50,51].

In an attempt to assess the effect of feedback on website quality improvement, Jorm et al. [36] scored 52 suicide prevention websites against expert consensus guidelines. Half the websites received feedback on how to improve the sites and half did not. Websites were evaluated again 6 months later. However, feedback did not lead to improvement.

3.7. Influence on user behaviour

Despite the large number of studies assessing the quality of health information on the Internet, there is very little understand- ing of quality in the context of the use that consumers make of such information. There is little evidence of the usefulness to consumers

Fig. 2. Mental health website quality evaluation framework in

of checklists such as DISCERN and it is largely unknown if better quality information leads to better health outcomes [52]. Many of the studies discussed here use expert rating and there is evidence that consumers are influenced by different criteria to experts when evaluating information [53,54]. Design features appear to play a significant part in assessment and consumers have been shown to reject clinically credible sites because of poor design [53]. Readability may also be a factor. Many of the studies reviewed here found reading levels to be relatively high, perhaps limiting the usefulness of sites to those of lower educational levels. In such cases, simplicity and intelligibility may need more emphasis than scientific rigour. Perceived expertise and absence of bias are also relevant, for example, a consumer may mistrust a pharmaceutical company website that an expert may rate highly against evidence- based criteria [54,55]. In a study of mental health-related Internet

corporating behavioural change theory-driven strategies.

N.J. Reavley, A.F. Jorm / Patient Education and Counseling 85 (2011) e16–e25e24

use, Lam-Po-Tang and McKay [56] found that perceived reliability was not associated with perceived influence on decision making.

Assessments of the effects of Internet information quality are further complicated by the fact that consumers typically use many sources of health information and effects may take a long time to develop. Evidence suggests that many of those who source information on the Internet discuss this with their health practitioners [54,56].

In the context of the low rates of help-seeking for mental health problems [57], an aim of some websites is to promote evidence-based treatments, many of which involve seeking professional help. However, research examining the links between Internet use and help seeking has provided mixed results [58–60]. In a more recent study, analysis of data from the Australian National Survey of Mental Health and Wellbeing (NSMHWB) showed that those who used the Internet in the previous 12 months were almost three times as likely to seek help from any service and from GPs. They were just over twice as likely to seek help from mental health professionals. Other information sources, namely, non-fiction book, pamphlet, news- paper, television, radio, were less strongly associated with service use [61]. However, the use of cross-sectional data limits the ability to tell if help-seekers are more likely to use the Internet or if the Internet actually promotes help-seeking and there is a need for further research on this issue.

3.8. Practice implications

There is therefore a need for further research on identifying consumers of mental health information on the Internet and how the various aspects of quality affect health behaviour such as help seeking and use of evidence-based treatments. There may need to be more consideration of factors uniquely relevant to the Internet such as privacy, usability and accessibility.

As the goal of much health education is behavioural change, it can be argued that the extent to which a website applies recognised behaviour change theories is an indicator of quality. Thus, future assessments of quality might use criteria informed by key behaviour change theories. A possible approach to this might be to develop an evaluation framework incorporating strategies from behaviour change models, key mental health literacy elements and health outcomes relevant to mental health promo- tion. A logic model, which specifies principles, elements and outcomes, along with the assumed linkages among these may be a useful approach [62].

An example of a logic model which could be used for an evaluation framework is given in Fig. 2. This example involves a hypothetical website aiming to improve public action for common mental disorders. The first line incorporates strategies from key behaviour change models, such as the Health Belief Model, Theory of Reasoned Action and Planned Behaviour, Transtheoretical (Stages of Change) Model and Social Learning Theory. The second line outlines key messages developed as a result of work on mental health literacy, which is defined as ‘‘knowledge and beliefs about mental disorders which aid their recognition, management or prevention’’ [63]. The third line outlines improvements in mental health literacy, the fourth line outlines intermediate health outcomes and the fifth line outlines final desirable health outcomes. Mental health website quality assessment could be extended to cover the extent to which sites meet the evaluation criteria arising out of such a framework and might include naturalistic reports of user behaviour [54,64]. Such assessments might involve the development of innovative ways of assessing health-related behaviours and outcomes. The move towards greater interactivity, information sharing, and collaboration on the Internet may offer such opportunities.

References

[1] Internet World Statistics. Internet Usage Statistics; 2009 [cited 28 April 2010]. Available from: www.internetworldstats.com.

[2] Netcraft. December 2009 Web Server Survey; 2009 [cited 28 April 2010]. Available from: www.netcraft.com.

[3] Shuyler KS, Knight KM. What are patients seeking when they turn to the Internet? Qualitative content analysis of questions asked by visitors to an orthopaedics Web site. J Med Internet Res 2003;5:e24.

[4] Pew Internet & American Life Project, Online Health Search. Washington, DC: Pew Internet; 2006.

[5] Khazaal Y, Chatton A, Cochand S, Hoch A, Khankarli MB, Khan R, et al. Internet use by patients with psychiatric disorders in search for general and medical informations. Psychiat Quart 2008;79:301–9.

[6] Powell J, Clarke A. Internet information-seeking in mental health: population survey. Brit J Psychiatry 2006;189:273–7.

[7] Ybarra ML, Suman M. Help seeking behavior and the Internet: a national survey. Int J Med Inform 2006;75:29–41.

[8] Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor— let the reader and viewer beware. J Amer Med Assoc 1997;277:1244–5.

[9] Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. J Amer Med Assoc 2002;287:2691–700.

[10] Jadad AR, Gagliardi A. Rating health information on the Internet. Navigating to knowledge or to Babel? J Amer Med Assoc 1998;279.

[11] Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999;53:105–11.

[12] Internet Healthcare Coalition. e-Health Code of Ethics (May 24). J Med Internet Res 2000;2:E9.

[13] Health on the Net Foundation. Health on the Net Foundation code of conduct (HONcode); 2010 [cited 2010 April]. Available from: http://www.hon.ch/ HONcode/Pro/intro.html.

[14] Barnes C, Harvey R, Wilde A, Hadzi-Pavlovic D, Wilhelm K, Mitchell P. Review of the quality of information on bipolar disorder on the Internet. Aust NZ J Psychiatry 2009;43:934–45.

[15] Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. J Amer Med Assoc 2001;285:2612–21.

[16] Christensen H, Griffiths KM, Medway J. Sites for depression on the web: a comparison of consumer, professional and commercial sites. Aust NZ J Public Health 2000;24:396–400.

[17] Ferreira-Lay P, Miller S. The quality of Internet information on depression for lay people. Psychiatry Bull 2008;32:170–3.

[18] Griffiths KM, Christensen H. Quality of web based information on treatment of depression: cross sectional survey. Brit Med J 2000;321:1511–5.

[19] Griffiths KM, Christensen H. The quality and accessibility of Australian de- pression sites on the World Wide Web. Med J Aust 2002;176:S97–104.

[20] Griffiths KM, Christensen H. Website quality indicators for consumers. J Med Internet Res 2005;7:e55.

[21] Griffiths KM, Tang TT, Hawking D, Christensen H. Automated assessment of the quality of depression websites. J Med Internet Res 2005;7:e59.

[22] Lissman TL, Boehnlein JK. A critical review of Internet information about depression. Psychiatry Serv 2001;52:1046–50.

[23] Morel V, Chatton A, Cochand S, Zullino D, Khazaal Y. Quality of web-based information on bipolar disorder. J Affect Disord 2008;110:265–9.

[24] Seyringer ME, Schrank B, Berger P, Katschnig H, Amering M. Bipolar disorder and manic-depressive disorder on the Internet. Neuropsychiatrie 2007;21:172–8.

[25] Stjernsward S, Ostman M. Depression, e-health and family support. What the Internet offers the relatives of depressed persons. Nord J Psychiatry 2007;61:12–8.

[26] Zermatten A, Khazaal Y, Coquard O, Chatton A, Bondolfi G. Quality of web- based information on depression. Depress Anxiety 2010 [Epub ahead of print January 22].

[27] Coquard O, Fernandez S, Khazaal Y. Assessing the quality of French language web sites pertaining to alcohol dependency. Sante Ment Que 2008;33:207–24.

[28] Faden VB, Corey K, Baskin M. An evaluation of college online alcohol-policy information: 2007 compared with 2002. J Stud Alcohol Drugs Suppl 2009;28–33.

[29] Khazaal Y, Chatton A, Cochand S, Zullino D. Quality of web-based information on cannabis addiction. J Drug Educ 2008;38:97–107.

[30] Khazaal Y, Chatton A, Cochand S, Zullino D. Quality of Web-based information on cocaine addiction. Patient Educ Couns 2008;72:336–41.

[31] Ipser JC, Dewing S, Stein DJ. A systematic review of the quality of information on the treatment of anxiety disorders on the Internet. Curr Psychiatry Rep 2007;9:303–9.

[32] Khazaal Y, Fernandez S, Cochand S, Reboh I, Zullino D. Quality of web-based information on social phobia: a cross-sectional study. Depress Anxiety 2008;25:461–5.

[33] Serdobbel Y, Pieters G, Joos S. Obsessive compulsive disorder and the Internet. An evaluation of Dutch-language websites and quality indicators. Tijdschr Psychiatry 2006;48:763–73.

[34] Kisely S, Ong G, Takyar A. A survey of the quality of web based information on the treatment of schizophrenia and Attention Deficit Hyperactivity Disorder. Aust NZ J Psychiatry 2003;37:85–91.

N.J. Reavley, A.F. Jorm / Patient Education and Counseling 85 (2011) e16–e25 e25

[35] Schrank B, Seyringer ME, Berger P, Katschnig H, Amering M. Schizophrenia and psychosis on the Internet. Psychiatry Prax 2006;33:277–81.

[36] Jorm AF, Fisher J, Oh E. Effect of feedback on the quality of suicide prevention websites: randomised control trial. Brit J Psychiatry 2010;197:73–4.

[37] Khazaal Y, Chatton A, Cochand S, Jermann F, Osiek C, Bondolfi G, et al. Quality of web-based information on pathological gambling. J Gambl Stud 2008;24:357–66.

[38] Murphy R, Frost S, Webster P, Schmidt U. An evaluation of web-based information. Int J Eat Disorder 2004;35:145–54.

[39] Touchet BK, Warnock JK, Yates WR, Wilkins KM. Evaluating the quality of websites offering information on female hypoactive sexual desire disorder. J Sex Marital Ther 2007;33:329–42.

[40] Bremner JD, Quinn J, Quinn W, Veledar E. Surfing the net for medical infor- mation about psychological trauma: an empirical study of the quality and accuracy of trauma-related websites. Med Inform Internet Med 2006;31:227– 36.

[41] Seomun GA, Lee SJ, Chang SO, Lee SJ. An evaluation study of dementia information providing websites in Korea. Taehan Kanho Hakhoe Chi 2005;35:631–40.

[42] Martin-Facklam M, Kostrzewa M, Schubert F, Gasse C, Haefeli WE. Quality markers of drug information on the Internet: an evaluation of sites about St John’s wort. Am J Med 2002;113:740–5.

[43] Akram G, Thomson AH, Boyter AC, Morton MJ. Characterisation and evaluation of UK websites on attention deficit hyperactivity disorder. Arch Dis Child 2008;93:695–700.

[44] Khazaal Y, Chatton A, Coquard O, Zullino D. Brief-DISCERN, a possible way to improve patient’s search on the health-related web. Rev Med Suisse 2009;5:1816–9.

[45] Nemoto K, Tachikawa H, Sodeyama N, Endo G, Hashimoto K, Mizukami K, et al. Quality of Internet information referring to mental health and mental dis- orders in Japan. Psychiatry Clin Neurosci 2007;61:243–8.

[46] Gagliardi A, Jadad AR. Examination of instruments used to rate quality of health information on the Internet: chronicle of a voyage with an unclear destination. Brit Med J 2002;324:569–73.

[47] Ipser JC, Stein DJ. A systematic review of the quality and impact of anxiety disorder meta-analyses. Curr Psychiatry Rep 2009;11:302–9.

[48] Eysenbach G, Diepgen TL. Towards quality management of medical informa- tion on the Internet: evaluation, labelling, and filtering of information. Brit Med J 1998;317:1496–500.

[49] Wilson P. How to find the good and avoid the bad or ugly: a short guide to tools for rating quality of health information on the Internet. Brit Med J 2002;324:598–602.

[50] O’Grady L. Future directions for depicting credibility in health care web sites. Int J Med Inform 2006;75:58–65.

[51] Risk A, Petersen C. Health information on the Internet: quality issues and international initiatives. J Amer Med Assoc 2002;287:2713–5.

[52] Bernstam EV, Shelton DM, Walji M, Meric-Bernstam F. Instruments to assess the quality of health information on the World Wide Web: what can our patients actually use? Int J Med Inform 2005;74:13–9.

[53] Harris PR, Sillence E, Briggs P. The effect of credibility-related design cues on responses to a web-based message about the breast cancer risks from alcohol: randomized controlled trial. J Med Internet Res 2009;11:e37.

[54] Sillence E, Briggs P, Harris PR, Fishwick L. How do patients evaluate and make use of online health information? Soc Sci Med 2007;64:1853–62.

[55] Broom A, Tovey P. The role of the Internet in cancer patients’ engagement with complementary and alternative treatments. Health (London) 2008;12:139– 55.

[56] Lam-Po-Tang J, McKay D. Dr Google, MD: a survey of mental health-related Internet use in a private practice sample. Australas Psychiatry 2010;18:130–3.

[57] Burgess PM, Pirkis JE, Slade TN, Johnston AK, Meadows GN, Gunn JM. Service use for mental health problems: findings from the 2007 National Survey of Mental Health and Wellbeing. Aust NZ J Psychiatry 2009;43:615–23.

[58] Christensen H, Leach LS, Barney L, Mackinnon AJ, Griffiths KM. The effect of web based depression interventions on self reported help seeking: random- ised controlled trial [ISRCTN77824516]. BMC Psychiatry 2006;6:13.

[59] Gould MS, Munfakh JL, Lubell K, Kleinman M, Parker S. Seeking help from the Internet during adolescence. J Am Acad Child Psychiatry 2002;41:1182–9.

[60] Santor DA, Poulin C, LeBlanc JC, Kusumakar V. Online health promotion, early identification of difficulties, and help seeking in young people. J Am Acad Child Adolesc Psychiatry 2007;46:50–9.

[61] Reavley N, Cvetkovski S, Jorm A. Sources of information about mental health and links to help seeking: findings from the 2007 Australian National Survey of Mental Health and Wellbeing; in press.

[62] Patton MQ. The Program’s theory of action: conceptualising causal linkages. In: Patton MQ, editor. Utilization focused evaluation. Thousand Oaks: Sage; 1997.

[63] Jorm AF, Korten AE, Jacomb PA, Christensen H, Rodgers B, Pollitt P. ‘‘Mental health literacy’’: a survey of the public’s ability to recognise mental disorders and their beliefs about the effectiveness of treatment. Med J Aust 1997;166:182–6.

[64] Frost JH, Massagli MP, Wicks P, Heywood J. How the Social Web Supports patient experimentation with a new therapy: the demand for patient-controlled and patient-centered informatics. AMIA Annu Symp Proc 2008;217–21.

[65] Boyer C, Selby M, Scherrer JR, Appel RD. The Health On the Net Code of Conduct for medical and health Websites. Comput Biol Med 1998;28:603–10.

[66] Khazaal Y, Chatton A, Cochand S, Coquard O, Fernandez S, Khan R, et al. Brief DISCERN, six questions for the evaluation of evidence-based content of health- related websites. Patient Educ Couns 2009;77:33–7.