QUALITY "A" WORK

ORIGINAL ARTICLE

Evaluating the quality and readability of Internet information sources regarding the treatment of swallowing disorders

Ashley P. O ’Connell Ferster, MD; Am anda Hu, MD, FRCSC

Abstract The Internet has become a popular resource fo r patient edu­ cation. The information it provides, however, is rarely peer-re­ viewed, and its quality m ay be a concern. Since the average American reads at an 8th grade level, the American Medical Association and the National Institutes o f Health have rec­ om m ended that health information be written at a 4th to 6th grade level. We perform ed a study to assess the quality and readability o f online information regarding the treatment o f swallowing disorders. A Google search fo r “swallowing treatm ent” was conducted. We studied the first 50 websites that appeared on the search engines results with the use o f the D ISCERN quality index tool, the Flesch Ease o f Reading Score (FRES), and the Flesch-Kincaid Grade Level (FKGL) readability test. D ISC ERN is a validated 16-item question­ naire used to assess the quality o f written health information; FRES and FKGL are used to assess readability. We classified the websites as either patient-targeted or professional-targeted

From the D ivision of O to laryngology-H ead a nd Neck Surgery, D epartm ent of Surgery, Penn State M ilton S, Hershey Medical Center, Hershey, Pa. (Dr. O’Connell Ferster); and the D epartm ent of Otolaryngology-Head and Neck Surgery, Drexel University College of Medicine, Philadelphia (Dr. Hu). The study described in this article was conducted at Drexel University.

Corresponding author: Ashley P. O ’Connell Ferster, MD, Division of O tolaryngology-Head and Neck Surgery, D epartm ent of Surgery, Penn State Milton S. Hershey Medical Center, 500 University Dr., Hershey, PA 17033. Email: [email protected]

Previous presentation: The information in this article has been edited for publication and updated from its original presentation as a poster at The Triological Society’s Combined Otolaryngology Spring Meetings; May 14-18, 2014; Las Vegas.

sites, as well as either major or minor. The overall D ISC ERN score was 1.61 ± 0.61 (range: 1 to 5), the overall FRES was 39.1 ± 1 9 .0 (range: 1 to 100), and the overall FKGL was 11.8 ± 3.4 (range: 3 to 12). A s would be expected, patient-targeted websites had significantly higher FRES and significantly lower FKGL scores than did the professional-targeted websites (p = 0.01 and p = 0.04, respectively); there was no significant dif­ ference between the two in D ISCERN scores. The major web­ sites had significantly higher D ISC ERN scores than did the m inor sites (p = 0.002); there were no significant differences in FRES and FKGL scores. We conclude that online inform a­ tion sources regarding the treatment o f swallowing disorders were o f suboptimal quality in that information was written at a level too difficult fo r the average American to easily u n ­ derstand. Also, the patient-targeted websites were written at a lower reading level, and the major websites contained a higher quality o f information.

Introduction Dysphagia, or difficulty with swallowing, refers to tran­ sit problems in the passage of food from the mouth to the hypopharynx and through the esophagus.1 With an incidence approaching 15% among community-dwell­ ing individuals and 40% among those in institution­ al settings, swallowing disorders are one of the most common diagnoses in the United States.2 Dysphagia can affect patients of all ages, from pediatric to geri­ atric.

Dysphagia has a wide range of etiologies, including neurologic (e.g., cerebrovascular accident), postinfec- tious (e.g., poliomyelitis), degenerative (e.g., amyo­ trophic lateral sclerosis), neuromuscular (e.g., achalasia), gastrointestinal (e.g., reflux), neoplastic (e.g., esophageal cancer), obstructive (e.g., strictures, rings), autoimmune

128 ■ www.entjournal.com ENT-Ear, Nose & Throat Journal ■ March 2017

EVALUATING THE QUALITY AND READABILITY OF INTERNET INFORMATION SOURCES REGARDING THE __________________________ TREATMENT OF SWALLOWING DISORDERS

(e.g., scleroderma), iatrogenic (e.g., irradiation-induced pathology), and postsurgical (e.g., vocal fold paralysis).

W ith such a large number of patients affected by swallowing disorders, the availability of reliable and understandable literature that addresses treatment options would be of great importance. In our tech­ nology-driven world, the Internet has become one of the most popular sources of information for patients. According to the 2013 report of the Pew Research Cen­ ter’s Internet & American Life Project, 72% of Internet users had searched online for health information, and 77% had started their pursuit at a search engine.3 Since Internet sources are rarely peer-reviewed, quality may be a problem and patients might be provided with misleading information.4

In addition to concerns about reliability, Internet re­ sources are rarely written at an appropriate reading level for the general population. Since the average American adult reads at an 8th grade level, the American Medi­ cal Association (AMA) and the National Institutes of Health (NIH) have recommended that health educa­ tion materials be written at a 4th to 6th grade level.5'6 The National Adult Literacy Survey found that almost one-half of American adults have deficiencies in read­ ing or computational skills.7 This report included the results of a study at two public hospitals, which found that one-third of English-speaking patients exhibited inadequate health literacy. Health literacy was defined in that report as the ability to apply and comprehend published health information. However, studies in various fields have found that most medical literature is written at an exceedingly high reading level.817

In this article, we describe our study to critically eval­ uate the quality and readability of online information sources regarding the treatment of swallowing disorders.

Materials and methods We conducted a Google search using the search term swallowing treatment on May 28, 2013. We limited our study to the first 50 websites listed in the search results, since it has been reported that the quality of websites drastically declines among any remaining sites.16 A study published by Ting and Hu in 2014 used similar methodology in limiting their analysis to the first 50 sites.14 We did not include sites that were ad­ vertisements, those that contained broken links, and duplicate sites, non-English-language sites, and sites without text.

To categorize the websites that we did include, we des­ ignated them as patient-targeted or professional-targeted and as major or minor. The major category included comprehensive sites such as those from academic in­ stitutions and widely recognized sites such as WebMD, emedicinehealth, and Wikipedia. O ur m inor sites in­

cluded those defined in a study by Pusz and Brietzke.17 Measurement tools. We used three tools to evaluate

the quality of each website: the DISCERN quality index tool, the Flesch Ease of Reading Score (FRES), and the Flesch-Kincaid Grade Level (FKGL) readability test.

DISCERN. The DISCERN instrument is a validated, 16-item questionnaire created by researchers at Oxford University in the United Kingdom to determine the quality of written health information.18 Each item is scored on a scale of 1 to 5. We used the mean score for all 16 items to determine a website’s overall DISCERN score. The 16 items cover topics such as the reliability of the publication, details on treatment options, and the overall quality of the publication, with each item assessed by the reader. Higher scores represent high­ er-quality websites.

Scoring was calculated for websites found after an online search using the term treatment. In using the term treatment within the search field, as recommended by the developers of the DISCERN instrument, the lay population can use DISCERN to investigate treatment options in a systematic manner.

FRES and FKGL. The FRES and FKGL readability scores were calculated with the aid of Microsoft Word. For both scoring systems, we assessed readability by reviewing the average number of syllables per word and the average number of words per sentence. FRES generated a number between 0 and 100, with lower scores indicating more difficult reading levels.19 FKGL generated a grade level from 3 to 12 that reflected the difficulty of the material, with higher numbers indi­ cating more difficult reading levels.19

The following formulas represent the statistical equa­ tions used to determine the FRES and FKGL scores:

• FRES = 206.835 - (84.6 x average num ber of syllables per word) - (1.015 x average num ber of words per sentence)

• FKGL = (11.8 x average num ber of syllables per word) + (0.39 + average num ber of words per sentence) - 15.59

Statistical analysis. Microsoft Excel was used for statistical analysis. Calculated DISCERN, FRES, and FKGL scores were expressed as means with standard deviations. Two-tailed Student t tests were conducted to evaluate differences in DISCERN, FRES, and FKGL scores between the patient- and professional-targeted websites and between the major and m inor sites. An a priori probability level was set at <0.05 for statistical significance.

Ethical considerations. The Institutional Review Board at the Drexel University College of Medicine deemed this study exempt from requiring approval

Volume 96, Number 3 www.entjournal.com ■ 129

O 'C O N N E LL FERSTER, HU

from the H um an Research P rotection Program , since public content from the Internet was used to obtain the study’s data.

Results O f the 50 websites we reviewed, we eliminated 5 be­ cause of broken links or a lack of text, leaving us with 45 sites for our final analysis.

Overall, the m ean DISCERN score was 1.61 ± 0.61, the m ean FRES was 39.1 ± 19.0, and the m ean FKGL score was 11.8 ± 3.4 (table 1). The FKGL score was much higher th an the 4th to 6th grade level recom m ended by the A M A /N IH .

Table 1. Overall DISCERN, FRES, and FKGL scores

Score, m ean ± SD

1.61 ±0.61

39.1 ± 19.0

FKGL 11.8 ± 3.4 Key: FRES = Flesch Reading Ease Score; FKGL = Flesch-Kincaid Grade Level.

M easu rem en t tool

DISCERN

FRES

We identified 33 of the 45 websites (73%) as p a­ tient-targeted and 12 (27%) as professional-targeted. The patient-oriented sites had a significantly higher m ean FRES and a significantly lower mean FKGL score (p = 0.01 and p = 0.04, respectively) (table 2). There was no significant difference between the two groups in DISCERN scores.

A total of 30 websites (67%) were classified as m ajor and 15 (33%) as m inor. The m ajor sites had a sig­ nificantly higher DISCERN score (p = 0.002). There was no significant difference in FRES between the m a­ jor and m inor sites (p = 0.10). There was a tren d tow ard a h ig h er FKGL score for the m ajor sites, b u t it was not quite statistica lly sig n ifican t (p = 0.06) (table 3).

the content ranged from poor to moderate. In sum, the inform ation available online regarding the treatm ent of swallowing disorders was suboptimal.

O u r findings are consistent w ith those of previous studies that evaluated online resources in otolaryngolo­ gy for sinus surgery,8 thyroid surgery,9'10 tonsillectomy,11 obstructive sleep apnea,12 tin n itu s,13 and thyroplasty.14 W hile these studies found th at quality ranged from very poor to good, they also revealed difficult to very difficult readability levels. In addition, Eloy et al assessed the readability of patient education m aterials available on major otolaryngology association websites.15 They also reported that these resources were all w ritten above the 6th grade reading level.

Pusz and Brietzke used the DISCERN tool to evaluate websites for informat ion on 10 com m on otolaryngologic problems: ear infection, ringing in ears, sleep apnea, snoring, hearing loss, hoarseness, nasal congestion, postnasal drip, sinus infection, and tonsillitis.1' They found a m arked variability in the quality of Internet inform ation on these com m on ENT problems; the lowest quality o f inform ation pertained to hoarseness and the highest was reported for sleep apnea. Since dysphagia was not included in th eir study, we were inspired to conduct a study looking solely at treatm ents for swallowing disorders.

Table 2. Comparison of DISCERN, FRES, and FKGL scores for patient- and professional-targeted websites ___________________

Patient Professional p Value

Websites,* n (%)

Score, mean ± SD

33 (73%) 12 (27%) N/A

DISCERN 1.58 ±0.54 1.75 ±0.75 0.47

FRES 44.0 ± 16.9 27.0 ± 19.1 O.OD

FKGL 11.1 ±3.1 13.8 ±3.7 0.041

* Five websites were n o t in c lu d e d because o f b roken lin ks o r lack o f text, t Sta tistica lly sig n ifica n t difference. Key: FRES = Flesch R e a d in g Ease Score; FK G L = F lesch-K incaid G rade Level.

Table 3. Comparison of DISCERN, FRES, and FKGL scores for major and minor websites

Discussion Aiming to evaluate the readability and quality of Internet inform ation sources regarding the treatm ent of swallowing disorders, our study yielded some troubling findings. The average FKGL score indicated a reading level of almost the 12th grade, which far exceeds the 4th to 6th grade level recom m ended by the AM A/NIH. Also, the quality of

M ajo r M inor p Value

Websites,* n (%)

Score, mean ± SD

30 (67%) 15(33%) N/A

DISCERN 1.79 ±0.61 1.26 ± 0.44 0.002f

FRES 35.9 ± 19.2 45.4 ± 17.3 0.10

FKGL 12.5 ±19.2 10.5 ±3.0 0.06

* Five w ebsites were n o t in c lu d e d because o f b ro ken lin ks o r lack o f text, f Sta tistica lly sig n ifica n t difference. Key: FR ES = Flesch R e a d in g E ase Score; F K G L = F lesch-K incaid G rade Level.

130 ■ www.entjournal.com ENT-Ear, Nose & Throat Journal ■ March 2017

EVALUATING THE QUALITY AND READABILITY OF INTERNET INFORMATION SOURCES REGARDING THE ____________________________ TREATMENT OF SWALLOWING DISORDERS_______

D ysphagia is one of the m ost com m on presenting com plaints in o u r laryngology practice. It is an inter­ disciplinary condition, w ith elements overlapping in neurology, gastroenterology, oncology, rheumatology, and geriatrics. Since swallowing disorders are a chronic condition, patients have a good deal o f tim e to seek healthcare inform ation.

A nother study conducted in conjunction with the Pew Research C enter’s Internet & A m erican Life Project found that patients w ith chronic disorders were more likely th an others to (1) gather inform ation online about medical problems, treatm ents, and drugs; (2) consult online reviews about drugs and other treatm ents; and (3) read or watch som ething online about someone else’s personal health experience.20

D ysphagia can be difficult to treat, as options are lim ­ ited. F rustrated patients m ay b e m otivated to research treatm ent options themselves. Some researchers may argue that while there is a higher incidence o f dysphagia in the geriatric population, this dem ographic group is less likely th an younger people to use the Internet. However, the Pew Research Center also found that 39% of online health inform ation seekers researched websites for someone other th an themselves.3 For example, chil­ dren often research healthcare topics for th eir parents or grandparents.

In separating patient-targeted from professional-tar­ geted websites, we found th at the patient-targeted web­ sites were w ritten at a significantly lower reading level, as would be expected. However, the readability of the m aterial was still above the recom m ended level o f 4th to 6th grade. Still, the quality of the inform ation for the patient-targeted websites, as reflected in the DISCERN scores, was not significantly different from that of the professional-targeted material.

W ith respect to the m ajor and m in o r websites, DIS­ CERN scores for the major websites were significantly higher th an those for the m inor sites, reflecting a better quality of inform ation. There was also a trend toward higher FKGL scores for the m ajor sites, but there was no significant difference in FRES values. W hile the m ajor websites were generally m ore difficult to read, which can be attributed to the fact that the inform ation was posted by academic institutions, they were also of better quality. To uphold this quality, the authors o f m ost articles on the major websites used m ore sophisticated language, which increased reading difficulty. Nevertheless, even the inform ation on the m in o r websites was w ritten above the recom m ended 4th to 6th grade reading level.

Pusz and Brietzke also categorized their websites as major or m inor.17 Sim ilar to ou r results, they reported that the m ajor sites were associated w ith significantly higher DISCERN scores th a n were the m inor sites (p < 0.001). They also reported that sites about chronic

conditions such as tin n itu s contained a significantly higher n um ber of advertisem ents th a n did sites for acute conditions such as tonsillitis. Since advertisers may be biased toward their own products, they may neglect to m ention other treatm ent options. O u r study purposely excluded sites w ith advertisem ents so that o u r results would not be influenced by poor-quality com m ercialization.

It is a challenge to define the criteria for high-quality health inform ation on the Internet. Kim et al reviewed the literatu re on published criteria for evaluating health-related websites.21 They found 29 published rating tools! The m ost com m on criteria in these rating tools were the content, design, and aesthetics of the site; disclosure of authors, sponsors, and developers; and the currency o f inform ation, authority o f the source, ease of use, accessibility, and availability.

Am ong these num erous rating systems, the DIS­ CERN in stru m en t is regarded as a validated, reliable, and standardized tool. Shortly after its publication, this in stru m en t was used by 15 experts, 15 inform ation p ro ­ viders, and 13 m em bers of self-help groups to evaluate a random sample o f leaflets from 19 major national self- help organizations in an attem pt to establish DISCERN’s reliability and validity.22 Rees et al fu rth e r evaluated DISCERN’s reliability using 31 inform ation leaflets on treatm ents for prostate cancer.23 They concluded th at DISCERN can be used to reliably discrim inate between low- and high-quality publications.

DISCERN has also been com pared to other rating tools. Khazaal et al com pared DISCERN to the H ealth on the Net (HON) ratings.24 The H O N Foundation developed a code of conduct for medical websites, and it grades sites based on the disclosure of authorship, funding, sources, updating of inform ation, disclosure of editorial policy, and confidentiality.25 Khazaal et al used both DISCERN and the “H O N code” to evaluate 388 websites on psychiatric issues such as social phobia, bipolar disorders, pathologic gambling, and addiction to cannabis, alcohol, and cocaine.24 They concluded that HONcode failed to identify websites w ith good-quality content. In contrast, they found that DISCERN could identify quality sites w ith a sensitivity o f 45% and specificity of 96%.

Ademiluyi et al com pared the internal consistency and validity ofDISCERN w ith the Inform ation Quality Tool (IQT) and Quality Scale (QS).26 The IQT is a 21 -item in stru m en t w ith yes/no questions relating to au th o r­ ship, sponsorship, currency, accuracy, confidentiality, and navigability. The QS is a 7-item in stru m en t w ith questions relating to ownership, authorship, source, currency, in teractiv ity , navigability, a n d balance. Each item is graded on a 3-point Likert scale. For this study, the authors evaluated 89 websites on sm oking

Volume 96, Number 3 www.entjournal.com ■ 131

O'CONNELL FERSTER, HU

cessation. DISCERN, IQT, and QS dem onstrated satis­ factory inter-rater reliability, an d the three instrum ents correlated positively w ith each other, which supports the convergent validity o f these tools. DISCERN has also been used widely in various m edical specialties, including urology,23 psychiatry,24 general surgery,27’28 obstetrics,29 and geriatrics.30

Tlie DISCERN tool does have some lim itations. It was designed to allow a layperson to evaluate the quality of health literature, but it does not assess the scientific accuracy o f specialist inform ation. (Authors of previous reports have attem pted to assess scientific accuracy by using self-developed checklists. For exam ­ ple, Strychowsky et al used a self-developed checklist to assess the accuracy of YouTube videos on pediatric tonsillectomy.31 However, self-developed checklists may not be validated or reliable. In contrast, the strength o f DISCERN is that it is validated and reliable.18’22’23) Moreover, the DISCERN tool is subjective and can be tim e-consum ing to use. (The creators ofDISCERN have suggested th at train in g can improve its usefulness.22)

The FRES and FKGL readability assessments can be used as complements to DISCERN. The readability form ulas are objective and can be calculated quickly by a computer. However, the FRES and FKGL instrum ents have th eir lim itations, as well. Since they calculate the num ber of syllables in a word or sentence, they do not take into account shorter words th a t are o f a higher reading level or th at are m ore difficult to understand.

The DISCERN, FRES, and FKGL instrum ents are not used to assess additional m aterials such as videos, diagram s, photographs, and audio presentations. These meffiods may enhance a person’s u nderstanding of the material, but they cannot yet be evaluated because of the lack of tools to assess them.

O u r study also had some lim itations. A lthough we chose to use the Google search engine, m any other search engines are available. We chose Google because it accounts for more th a n 91% of all Internet searches, m aking it the largest Internet search engine.32 In ad­ dition, search term s used by patients may vary, which could expand the potential num ber of sites pertinent to swallowing treatments. We deliberately chose not to use the term dysphagia because we believed that m ost laypersons would not use it.

O u r study was also lim ited to websites w ritten in English. Those th at are w ritten in different languages m ight be o f different quality and readability.

Finally, because DISCERN was designed for layper­ sons, it would not be useful to evaluate its utility as used by a dysphagia expert. Because of this, the senior author (A.H.), who is a fellowship-trained laryngologist and who focuses her practice on dysphagia, was not involved in :he review o f the In tern et articles using DISCERN.

132 ■ w w w . e n t jo u r n a l .c o m

In conclusion, we found th a t online inform ation sources regarding the treatm ent of swallowing disorders were o f suboptim al quality. Inform ation was w ritten at a level too difficult for the average Am erican adult to easily understand. Patient-targeted websites were w ritten at a lower reading level th an were professional-targeted sites, and the major websites had higher quality inform ation th an did the m inor sites.

In view of the lack of high-quality literature available on the Internet, physicians should advise their patients to seek other resources. Physicians should also equip th eir practices with patient-friendly literature to dis­ tribute to patients.

References 1. Hirano I, Kahrilas PJ. Dysphagia. In: Longo DL, Fauci AS, Kasper

DL, et al, eds. Harrison’s Principles of Internal Medicine. 18th ed. New York: McGraw-Hill; 2012:297-300.

2. Robbins J, Hind J, Barczi S. Disorders of swallowing. In: Hal­ ter JB, Ouslander IG, Tinetti ME, et al, eds. Hazzard’s Geriatric Medicine and Gerontology. 6th ed. New York: McGraw-Hill; 2009:483-500.

3. Fox S, Duggan M. Health Online 2013. Pew Research Center website, fan. 15, 2013. http://www.pewinternet.org/2013/01/15/ health-online-2013/. Accessed Jan. 4, 2017.

4. Silberg WM, Lundberg GD, Musacchio RA. Assessing, con­ trolling, and assuring the quality of medical information on the Internet: Caveant lector et viewor—let the reader and viewer be­ ware. JAMA 1997;277( X 5): 1244-5.

5. Weiss BD. Health Literacy: A Manual for Clinicians. Chicago: American Medical Association Foundation and American Medi­ cal Association; 2003.

6. No authors listed. How to write easy-to-read health materials. MedlinePlus website, http://www.nlm.nih.gov/medlineplus/etr. html. Accessed Jan. 4, 2017.

7. No authors listed. Health literacy: Report of the Council on Sci­ entific Affairs. Ad Hoc Committee on Health Literacy for the Council on Scientific Affairs, American Medical Association. JAMA 1999;281(6):552-7.

8. Cherla DV, Sanghvi S, C houdhry OJ, et al. Readability assessment of Internet-based patient education materials related to endo­ scopic sinus surgery. Laryngoscope 2012;122(8): 1649-54.

9. Patel CR, Cherla DV, Sanghvi S, et al. Readability assessment of online thyroid surgery patient education materials. Head Neck 2013;35( 10): 1421 -5.

10. Muthukumarasamy S, Osmani Z, Sharpe A, England RJ. Qual­ ity of information available on the World Wide Web for pa­ tients undergoing thyroidectomy: Review. J Laryngol Otol 2012;126(2):116-19.

11. Roshan A, Agarwal S, England RJ. Role of information avail­ able over the Internet: What are the parents of children un­ dergoing tonsillectomy likely to find? Ann R Coll Surg Engl 2008;90(7):601-5.

12. Langille M, Veldhuyzen van Zanten S, Shanavaz S, Massoud E. Systematic evaluation of obstructive sleep apnea Websites on the Internet. J Otolaryngol Head Neck Surg 2012;41(4):265-72.

13. Fackrell K, Hoare DJ, Smith S, et al. An evaluation of the content and quality of tinnitus information on Websites preferred by gen­ eral practitioners. BMC Med Inform Decis Mak 2012;12:70.

14. Ting K, Hu A. Evaluating the quality and readability of thyroplas- ty information on the Internet. J Voice 2014;28(3):378-81.

Continued on page 138

E N T - E a r , N o s e & T h r o a t J o u r n a l ■ M a r c h 2 0 1 7

MOBASHIR, BASHA, MOHAMED, HASSAAN, ANANY O'CONNELL FERSTER, HU

4. Martinez Devesa P, Ghufoor K, Lloyd S, Howard D. Endo­ scopic C 0 2 laser management of laryngocele. Laryngoscope 2002;112(Pt 1):1426—30.

5. Prasad KC, Vijayalakshmi S, Prasad SC. Laryngoceles-presen- tations and management. Indian J Otolaryngol Head Neck Surg 2008;60(4):303-8.

6. Ingrams D, Hein D, Marks N. Laryngocele: An anatomical vari­ ant. J Laryngol Otol 1999;113(7):675-7.

7. Verret DJ, DeFatta RJ, Sinard R. Combined laryngocele. Ann Otol Rhinol Laryngol 2004;113(7):594-6.

8. Keles E, Alpay HC, Orhan I, Yildirim H. Combined laryngocele: A cause of stridor and cervical swelling. Auris Nasus Larynx 2010;37( 1 ):117-20.

9. Holinger LD, Barnes DR, Smid LJ, Holinger PH. Laryngocele and saccular cysts. Ann Otol Rhinol Laryngol 1978;87(5 Pt 1):675— 85.

10. Dursun G, Ozgursoy OB, Beton S, Batikhan H. Current diagnosis and treatment of laryngocele in adults. Otolaryngol Head Neck Surg 2007;136(2):211-15.

11. Vasileiadis I, Kapetanakis S, Petousis A, et al. Internal laryngo- pyocele as a cause of acute airway obstruction: An extremely rare case and review of the literature. Acta Otorhinolaryngol Ital 2012;32(l):58-62.

12. Felix JA, Felix F, Mello LF. Laryngocele: A cause of upper airway obstruction. Braz J Otorhinolaryngol 2008;74(l):143-6.

13. Amin M, M aran AG. The aetiology of laryngocele. Clin Otolayn- gol Allied Sci 1988;13(4):267-72.

14 DeSanto LW. Laryngocele, laryngeal mucocele, large saccules, and laryngeal saccular cysts: A developmental spectrum. Laryn­ goscope 1974;84(8):1291-6.

15 Dray TG, Waugh PF, Hillel AD. The association of laryngoceles with ventricular phonation. J Voice 2000; 14(2):278—81.

16 Marcotullio D, Paduano F, Magliulo G. Laryngopyocele: An atyp­ ical case. Am J Otolaryngol 1996;17(5):345—8.

17 Macfie DD. Asymptomatic laryngoceles in wind-instrument bandsmen. Arch Otolaryngol 1966;83(3):270-5.

18 Close LG, Merkel M, Burns DK, et al. Asymptomatic laryngocele: Incidence and association with laryngeal cancer. Ann Otol Rhi­ nol Laryngol 1987;96(4):393-9.

19 Harvey RT, Ibrahim H, Yousem DM, Weinstein GS. Radiologic findings in a carcinoma-associated laryngocele. Ann Otol Rhinol Laryngol 1996; 105(5):405-8.

20 Ettema SL, Carothers DG, Hoffman HT. Laryngocele resection by combined external and endoscopic laser approach. Ann Otol Rhinol Laryngol 2003;112(4):361-4.

21 Civantos FJ, Holinger LD. Laryngoceles and saccular cysts in infants and children. Arch Otolaryngol Head Neck Surg 1992;118(3):296—300.

22 Myssiorek D, Persky M. Laser endoscopic treatment of laryn­ goceles and laryngeal cysts. Otolaryngol Head Neck Surg 1989;100(6):538—41.

Continued from page 132

15. Eloy JA, Li S, Kasabwala K, et al. Readability assessment of pa­ tient education materials on major otolaryngology association Websites. Otolaryngol Head Neck Surg 2012;147(5):848-54.

16. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the World Wide Web: A systematic review. JAMA 2002;287(20):2691-2700.

17. Pusz MD, Brietzke SE. How good is Google? The quality of otolaryngology information on the Internet. Otolaryngol Head Neck Surg 2012;147(3):462-5.

18. Charnock D, ed. The DISCERN Handbook: Quality Criteria for Consumer Health Information on Treatment Choices. Abing­ don, U.K.: Radcliffe Medical Press; 1998.

19. Flesch R. A new readability yardstick. J Appl Psychol 1948;32(2):221-33.

20. Fox S, Duggan M. The Diagnosis Difference. Pew Research Center website. Nov. 26, 2013. http://www.pewinternet.org/2013/ll/26/ the-diagnosis-difference/. Accessed Jan. 4, 2017.

21. Kim P, Eng TR, Deering MJ, Maxfield A. Published crite­ ria for evaluating health related Web sites: Review. BMJ 1999;318(7184):647-9.

22. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: An instrument for judging the quality of written consumer health in­ formation on treatment choices. J Epidemiol Community Health 1999;53(2): 105-11.

23. Rees CE, Ford JE, Sheard CE. Evaluating the reliability of DIS­ CERN : A tool for assessing the quality of written patient informa­ tion on treatment choices. Patient Educ Couns 2002;47(3):273-5.

24. Khazaal Y, Chatton A, Zullino D, Khan R. HON label and DIS­ CERN as content quality indicators of health-related Websites. Psychiatr Q 2012;83(l):15-27.

25. Boyer C, Selby M, Appel RD. The Health on the Net Code of Conduct for medical and health Web sites. Stud Health Technol Inform 1998;52(Pt 2):1163-6.

26. Ademiluyi G, Rees CE, Sheard CE. Evaluating the reliability and validity of three tools to assess the quality of health information on the Internet. Patient Educ Couns 2003;50(2): 151-5.

27. Grewal P, Alagaratnam S. The quality and readability of colorec­ tal cancer information on the Internet. Int J Surg 2013;11(5):410- 13.

28. Wasserman M, Baxter NN, Rosen B, et al. Systematic review of Internet patient information on colorectal cancer surgery. Dis Colon Rectum 2014;57(l):64-9.

29. Jaffe J, Tonick S, Angell N. Quality of Web-based information on epidural anesthesia [abstract]. Obstet Gynecol 2014;123(Suppl 1): 115S.

30. Dillon WA, Prorok JC, Seitz DP. Content and quality of infor­ mation provided on Canadian dementia Websites. Can Geriatr J 2013; 16( 1):6-15.

31. Strychowsky JE, Nayan S, Farrokhyar F, MacLean J. YouTube: A good source of information on pediatric tonsillectomy? Int J Pe- diatr Otorhinolaryngol 2013;77(6):972-5.

32. Consumer search engine trends. Experian website. http://annu- alcreditreport.experian.com/marketing-services/online-trends- search-engine.html. Accessed Jan. 16, 2017.

138 ■ w w w .e n tjo u r n a l.c o m E N T -E ar, N o s e & T h r o a t J o u rn a l ■ M a r c h 2 0 1 7

Copyright of ENT: Ear, Nose & Throat Journal is the property of Vendome Group LLC and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.