Reading Analysis Discussion Post
Search as Learning Special Issue
Journal of Information Science
2016, Vol. 42(1) 79–93
� The Author(s) 2016 Reprints and permissions:
sagepub.co.uk/journalsPermissions.nav
DOI: 10.1177/0165551515614472
jis.sagepub.com
The effects of textual environment on reading comprehension: Implications for searching as learning
Luanne Freund The iSchool, University of British Columbia, Canada
Rick Kopak The iSchool, University of British Columbia, Canada
Heather O’Brien The iSchool, University of British Columbia, Canada
Abstract This paper reports on a study of digital reading that investigates the effects of different textual environments on information interac- tion and comprehension outcomes. While there is a large body of literature that compares print and digital reading, research that com- pares differently designed digital reading environments is limited. Such work can inform the design of information and search systems intended to support learning. This study investigated the effects of two design dimensions: Text Presentation (Plain Text vs In-Context) and Interactivity (availability of Reading Tools). Results show that the simplest textual environment (Plain Text presentation with no Interactivity) was associated with the highest comprehension outcomes, but that Interactivity mitigated the negative effects of texts presented In-Context. Both time spent reading and certain reading behaviours varied to some extent by condition and may be associ- ated with comprehension; however, personal characteristics of the readers played little to no role in determining outcomes.
Keywords Construction-Integration Model; information interaction; interactivity; learning; reading; text comprehension
1. Introduction
We are in the midst of a profound shift in information interaction technologies and practices. Whereas the common prac-
tice a decade ago was simply to print digital content before reading it [1], a 2012 Pew Research Centre study found that
43% of adult respondents read e-books or other types of long-form textual content on their digital devices [2]. In acade-
mia, these rates are likely to be much higher, as e-textbooks and e-journals have become commonplace and the quality
and the extent of use of portable electronic devices has risen very quickly. Furthermore, online searching, browsing and
reading, and the concomitant learning supported by these activities, are now intermingled to the point where it is difficult
to distinguish when one activity ends and another begins. As the locus of reading shifts to digital media and to devices
such as laptops, e-book readers and smartphones, important questions remain as to the quality and cognitive effects of the
digital reading experience. These concerns have arisen in the academic literature [3], and have captured public attention
through trade publications such as Carr’s, ‘Is Google making us stupid?’ [4].
Although studies tell us that digital reading differs from reading traditional print, we know substantially less about
digital reading behaviour [5]. Research suggests that it is broader and shallower, and that reflection is more fleeting than
print reading, as readers skip and jump through interconnected texts and media objects in a form of hyper-reading [4, 6].
Corresponding author:
Luanne Freund, University of British Columbia, SLAIS, 1961 East Mall, Vancouver, British Columbia, V6T 1Z1, Canada.
Email: luanne.freund@ubc.ca
Such reading is characterized by browsing and scanning, keyword spotting, one-time reading, non-linear reading and
selective reading [7]. In higher education, the implications of changing reading practices for student learning may be
quite serious, as engagement with knowledge and ideas through deeper interaction with texts is a core component of our
education system [3]. In this context, it is imperative to create digital reading platforms that provide the optimal condi-
tions for human information interaction, supporting reader engagement, comprehension and learning. Pearson et al. [8]
note that reading is ‘strongly influenced by the context in which it occurs’ (p. 33), and Rouet [9] identifies the need to
determine the conditions that promote learning from Web-based and other online reading materials. Yet it is still not clear
what constitutes an optimal textual (reading) environment, that is, an application and its features in which a digital text is
embedded and presented to the reader. Does a more engaging and media-rich environment support comprehension and
learning? Alternately, do these features distract the reader and inhibit learning? What kinds of tools and features encour-
age more active reading strategies and better comprehension outcomes, and under what conditions are these effective?
These are important concerns for designers of information interaction environments, including e-textbooks, e-journal sys-
tems and exploratory search systems.
In addressing such questions, this paper presents the results of an experimental study designed to examine the effects
of the textual environment on comprehension and learning. Two factors were varied: the Presentation Style of the text
(Plain Text and text In-Context) and Interactivity (with and without Reading Tools for highlighting and annotation and
hyperlinks). We focus on text comprehension as the main outcome measure, using the Construction-Integration Model
[10] as a means of modelling and assessing comprehension outcomes. Results indicate that both factors influence com-
prehension, and reveal an interesting relationship between Presentation Style and Interactivity that has implications for
digital information interaction design. The following sections of the paper review the background literature on digital
reading and text comprehension, present the research methods and results, and discuss the findings in the context of
designing platforms for reading and digital information interaction more generally.
2. Background
2.1. Digital reading
Reading is a human ability that has developed in tandem with written language and associated technologies: the scroll,
the codex and, more recently, hypertext and the Web [3, 11]. Studies of reading behaviour on the Web suggest that inter-
action patterns typical of print materials have given way to reading that is faster, less linear and involves more movement
within and between texts [3, 6, 12], raising concerns as to the cognitive and social implications of ‘shallow’ digital read-
ing [3, 4, 7]. Such questions are particularly important within higher education, as reliance upon digital reading to support
learning becomes the norm [13]. The shift began several decades ago with the growing availability of online journals,
although numerous studies showed that, until recently, most users were printing journal articles in order to read them [1].
The recent rise of e-books, e-textbooks and online course management systems, co-occurring with the widespread adop-
tion of a range of hardware devices for e-reading, such as the Kindle and the iPad [14], has resulted in a pervasive digital
context for reading and learning within higher education. Yet even with increased access to digital resources, and clear
indications that university students prefer conducting research online using search engines and scholarly databases [15],
the actual adoption of e-textbooks has been slower than expected [16], suggesting that designing digital reading environ-
ments to support learning is not a solved problem.
To date, most research studies on digital reading have compared the outcomes and processes of reading on screen
with those of print reading. The physical affordances of printed books and our long experience using them make many
of the tasks involved in print reading ‘lightweight’ [8, 17], so that readers perform them with little to no cognitive or
physical effort. In early research, people were known to read digital texts more slowly, by as much as 20–30% [18]
when compared with traditional paper documents. This has been attributed to greater cognitive complexity, owing to the
secondary tasks associated with digital reading, such as scrolling, navigating and hyperlinking, which make demands on
the working memory, as well as ergonomic factors such as screen resolution [7, 18, 19]. The lack of physical cues and
placeholders in digital texts and the use of hypertext make it relatively difficult to assess the length of texts and to situate
information within them [9], so that readers may experience disorientation or ‘lostness’ [18] or have greater difficulty
refinding information than in print environments.
The cognitive outcomes of digital reading are not well understood. Text comprehension, a precursor to learning, is ‘a
complex, memory-based process that allows the integration of information into a meaningful mental representation’ [9].
It is dependent upon reading ability, but also relies upon the reader’s prior knowledge and experience [10, 20]. Some
studies have found lower levels of text comprehension associated with digital reading as compared with print [13, 21];
however it is not clear if this is an effect of the digital display of text or a function of reading behaviours that are more
Freund et al. 80
Journal of Information Science, 42(1) 2016, pp. 79–93 � The Author(s), DOI: 10.1177/0165551515614472
prevalent online, such as browsing, speed reading or non-linear reading [7, 21, 22]. Efforts to compare comprehension
outcomes of print and digital reading are complicated by the wide variation in measures employed and the many possi-
ble interaction effects of perceptual, ergonomic and cognitive factors within testing environments [18, 23].
Distractions seem to pose a significant challenge for readers in digital environments, particularly in online connected
environments [8]. Typically, digital reading is less focused and involves more scanning and topic shifting [7, 12], which
leaves readers more vulnerable to distraction [22]. Furthermore, in work environments there is more competition for the
attention of the reader from other applications such as email and social networking sites, and from related and unrelated
content. Multimedia features, characteristic of Web-based texts, influence reading as well: images relevant to the text
slow readers down as they process the content and make connections with the text, and advertising images distract read-
ers and prompt re-reading of content upon return to the text [24]. In a study of university students, Keller [22] found that
more effort was required to concentrate on reading from the screen as opposed to print, in part owing to distractions
caused by the computer, and in part because students considered online texts to be less authoritative. Echoing the findings
of other recent studies, Keller found that students have developed conscious strategies to avoid distractions by download-
ing and reading offline, or by reading in the library [23]. Web applications, such as the SliceReader, an ‘easy, simple,
distraction-free way to read text’ (http://mthr.me/slicereader/), have arisen in response to this need.
A competing approach to the design of digital reading environments adopts the active reading perspective. Adler [25]
characterized this as reading that is accompanied by thinking and learning and is facilitated through activities such as
note taking and underlining. Rather than emphasizing simplicity, the focus is on the development of tools that allow for
greater interactivity and exploit some of the unique capabilities of digital text. Studies of electronic journal use [1, 26–
28] note the value of advanced features such as integrated information retrieval of related resources; the ability to create
hypertext links to related passages in other journal articles; the ability to take notes and make annotations within articles;
and different article views based on genre-based structural components, for example, showing only the ‘Results’ section.
Kopak and Chiang [28] examined the use of both annotation and linking tools within the Open Journal Systems, with the
intention of having these Reading Tools used to actively create meaning connections both within and between texts in a
collection of journal articles. A more radical approach to supporting comprehension through interaction with texts was
proposed by Kidwa [20], who designed a Web-based environment that allowed readers to break the text into chunks, add
images, create summaries and engage in other metacognitive activities.
Given that digital reading is now well-established, whether as a stand-alone activity or intermingled with Web search-
ing and other online activities, it is necessary to move beyond comparisons of print and digital reading to examine the
effects of such different types of digital reading environments. What kind of environment best supports comprehension
and learning? Prior research suggests a contradiction between the benefit of tools to support active and engaged reading
and the possible negative effect of the increased cognitive load imposed by such tools. Furthermore, the potential benefits
of engaging readers with dynamic multimedia information environments may be undermined by the distractions imposed
by such environments.
2.2. Comprehension: Models and methods
Comprehension is the process of extracting meaning from information [29]. It is a key outcome of reading in the aca-
demic context, and is closely associated with learning and other human information interaction processes such as rele-
vance and credibility assessments. Reading and learning are also central to certain types of searching, notably
exploratory search [30, 31]. Text comprehension has been treated as a dependent variable in studies of reading and inter-
action with different types of online content. Such studies have examined a wide range of independent variables: indi-
vidual differences, including prior knowledge [32, 33], information processing style [34] and need for cognition [35];
and features of texts, including format (print or electronic) [36, 37], level of interactivity [35] and hypertext structure
[32, 33]. Comprehension is sometimes studied in isolation, but is often considered together with other dependent vari-
ables such as navigation behaviour and engagement, which represent complementary aspects of user experiences in
information environments.
In studying the comprehension outcomes of digital reading, we have adopted Kintsch’s Construction–Integration (C-I)
model [10] in our own research. The C-I model stresses the importance of ‘the active construction of meaning during
reading’ [38: 84] through activities such as paraphrasing and summarizing. The model articulates several levels of com-
prehension, including word recognition and decoding, comprehension of the text at the sentence and paragraph level (the
microstructure), comprehension of the network of propositions that extends over the whole text (the macrostructure or
‘gist’) and the situation model: the structure resulting from the integration of the meaning derived from the text with the
reader’s prior knowledge and understanding [38].
Freund et al. 81
Journal of Information Science, 42(1) 2016, pp. 79–93 � The Author(s), DOI: 10.1177/0165551515614472
Assessment of textual comprehension is a non-trivial matter at the best of times, and certainly challenging within
design research studies like the one carried out here. One of the challenges in measuring comprehension, for example, is
that standard instruments and scales used in more controlled experimental environments are difficult to adapt to studies
that are dynamic (i.e. interactive) and that use non-standard texts. In practice, assessment methods range from simple
factual questions to the use of various other question types (e.g. written summaries, sentence recognition, inference veri-
fication) designed to assess different levels (kinds) of comprehension [19, 39, 40].
2.3. The current study
Digital reading research has substantial implications for information search, although we seldom take reading behaviours
into account in modelling and measuring online information seeking. Studies conducted in higher education settings
demonstrate that students prefer to conduct research using online platforms [15], yet reading digitally is more time con-
suming [18] and can result in challenges of disorientation [9] and refinding information [18]. Thus how well do current
exploratory search systems, including digital library platforms and full-text bibliographic databases, support students’
movement between search and active reading activities, and how do we capture and evaluate system- and user-level out-
comes for these tightly coupled behaviours? At this point in time, information behaviour models do not encompass the
array of reading activities involved in information interactions, nor has comprehension as an outcome of search been
explored in depth.
The current study explores the interaction between reading for academic purposes and comprehension, and the role
of electronic reading environments and their affordances in this process. Given the need to assess comprehension using
content-dependent instruments, we did not include a searching component in this study, but rather focus on information
use, a highly important yet less researched component of the information-seeking process. Our work contributes to the
small body of work that focuses on what people ‘do’ with information, and looks at an underutilized outcome measure
of information interaction: comprehension.
Recognizing the need to assess the effects of different digital reading appurtenances and environments on reading
comprehension and bearing in mind the challenges of assessment, this study addresses the following questions:
• What type of digital reading interface better supports user comprehension during academic reading? • What is the effect on comprehension of greater interaction with the text, implemented through Reading Tools
such as highlighting and annotating?
• To what extent do individual differences between readers affect their outcomes in such different reading environments?
3. Methods
To address these questions, we carried out a between-subjects experimental study to simulate the experience of complet-
ing online course readings for a hypothetical ‘Technology and Society’ themed class. Each of the 41 participants read a
set of three thematically related texts of different genres that might be assigned in a course or encountered on the Internet
(newspaper article, scholarly journal article and blog post) from one of two article sets used in the study. To encourage
participants to engage in ‘active reading’ [25], they were instructed to identify the main points of each article and the key
themes that connect them in preparation for a series of questions that would follow. The study was designed to prompt a
quick reading pace, in order to simulate typical digital reading behaviour [21].
3.1. Texts and conditions
We identified and prepared two sets of thematically related readings on multidisciplinary topics (digital activism and
human-robot interaction) so as to avoid advantaging students within a particular discipline. Selected texts represented
both academic (journal articles) and web (online news, blogs) genres and were edited to reduce the overall length and to
ensure that the two sets of readings were comparable in length (about 8000 words in total). Articles were presented as
individual Web pages, with a scrollbar available for navigation within the page. Articles were accessible from a menu
page that contained a description of the task and hyperlinks to each of the three articles.
Two factors were manipulated in the presentation of these texts: Interactivity and Presentation Style. In the In-Context
Presentation Style, texts were presented in their original format and context, with accompanying design, graphical and
paratextual elements; in the Plain Text style, texts were presented in isolation in a plain text format (Figure 1). In the
high-interactivity condition, a small number of hyperlinks were added to connect the three articles in the set and
Freund et al. 82
Journal of Information Science, 42(1) 2016, pp. 79–93 � The Author(s), DOI: 10.1177/0165551515614472
participants were provided with a lightweight and freely available set of Reading Tools in the form of the Diigo Toolbar,
which allowed them to annotate with sticky notes and highlight text. The links and Toolbar were not available in the
low-interactivity condition.
In summary, there were two sets of themed readings, human–robot interaction and digital activism, and four interface
conditions, Plain Text with interactive features, Plain Text without interactive features, In-Context text with interactive
features, and In-Context text without interactive features. This 2 × 4 design resulted in eight distinct experimental groups of four to six students per group. Students were provided with a selection of days and times and they signed up
for the session that was most convenient. They did not have any knowledge of the reading set or interface they would be
interacting with prior to their participation. While we did not practise random assignment to conditions, we did not
observe any statistical differences in our participant groupings to suggest that one experimental group was more homo-
genous than another. For example, males and females were distributed amongst the various groups, as were undergradu-
ate and graduate students.
3.2. Variables and measures
Table 1 summarizes the variables and measures employed. The two manipulated factors are Text Presentation and
Interactivity, as described above. Other independent variables reflect characteristics of the participants (e.g. demo-
graphics, prior knowledge and interest, general reading motivation, reading ability), measured through self-assessment
and standardized tests. Prior knowledge of and interest in the reading set topic was assessed with four self-report items
rated on a seven-point Likert scale. Reading motivation was measured using the Adult Reading Motivation Scale (21
items rated on a five-point Likert scale) [41], as we assumed that more motivated readers may invest more effort and
become more engaged in the reading process, thereby increasing comprehension outcomes [38, 42]. Reading level was
assessed using the Nelson–Denny Standardized Reading Test, a widely used and validated test suitable for children and
adults [43]. The test includes vocabulary, comprehension and reading speed components, which are combined to pro-
duce overall scores that are associated with grade levels [43]. Despite some critiques of the Nelson–Denny test (e.g.
[44]), it has been successfully used in studies of adult reading [45] and served as a means of differentiating between
stronger and weaker readers, who are known to employ different reading strategies [38].
The primary dependent variable is comprehension, measured through tests designed in accordance with the C-I model
of text comprehension to assess both microstructural and macrostructural elements [10]. All test questions were closed
with pre-set correct and incorrect responses; they were developed and refined collaboratively by the research team and
underwent several cycles of pre-testing and revision. Microstructural items tested factual recall (eight true or false items)
and conceptual understanding at the sentence level using the Sentence Verification Technique (SVT) [46]. The SVT
questions for each set of articles consisted of 12 sentences (four per article), two of which accurately represented the
Figure 1. In-Context (left) and Plain Text (right) presentation styles.
Freund et al. 83
Journal of Information Science, 42(1) 2016, pp. 79–93 � The Author(s), DOI: 10.1177/0165551515614472
semantics of the text in the form of one exact phrase and one paraphrase, and two of which falsely represented the content
of the article by changing the meaning or inserting something incongruous. Participants were asked to indicate whether
or not each sentence accurately represented the content. Macrostructural items required participants to connect ideas pre-
sented in the three texts and consisted of six summary statements that varied in centrality and importance [10, 47].
Participants were asked to select the three statements that best depicted the central themes in the article set (Table 2).
The secondary dependent variable is Reading Behaviour, which was measured through a range of process measures
gathered through detailed manual analysis of the screen capture videos of study sessions. Four reading modes or patterns
were identified through analysis of scrolling and navigations patterns observed in the videos, and based on prior work
by Hornbæk and Frøkjær [48]; these are defined in Table 1. A fifth mode, non-linear reading, was also measured, but
only five of 41 participants engaged in this mode, so it was excluded from further analysis. Time spent in various sec-
tions (introduction, body and conclusion) was measured, based on demarcations identified by the researchers in each of
the six texts, for example, headings. Finally, instances of certain types of textual interactions were counted: transitions
between texts, through linking (in the Interactive condition), returning to the menu or some other means; the number of
times participants checked the length of articles by quickly scrolling to the end and back; and the number of times they
highlighted text, either through use of the Reading Tools (in the Interactive condition), or by dragging the cursor to tem-
porarily highlight sections. Rare interaction behaviours, such as increasing the size of the text and adding textual annota-
tions, were noted but not included in the analysis owing to their infrequency.
3.3. Instruments and procedures
Each 2 h session was facilitated by a member of the research team and took place in a seminar room equipped with laptop
computers and Morae screen and event capture software to record on-screen activity. Laptops were pre-set to display the
texts in one of the four combinations of conditions per session. All other materials (instructions, questionnaires) were pro-
vided in print format. Following informed consent, students (in groups of five or six) completed a demographic and read-
ing habits questionnaire, and the Adult Reading Motivation Scale. The demographic questionnaire collected information
Table 1. Variables and measures used in the study
Type of variable Variables Measures
Independent Text Presentation Manipulated Condition: In-Context and Plain Text Independent Interactivity Manipulated Conditions: Interactive and Non-Interactive Independent Prior Knowledge Self-assessed in pre-task questionnaire, seven-point Likert Scale Independent Reading Motivation Adult Reading Motivation Scale [41] Independent Reading Skill Nelson–Denny Reading Test [43, 45] Independent Educational Level Demographics questionnaire: Undergraduate, Master’s, PhD Dependent Comprehension Outcome measures:
(a) Comprehension Test: overall score as percentage of correct responses out of 26
(b) Error rate out of 8 on True–False questions (c) Error rate out of 12 on Sentence Verification questions (d) Error rate out of 6 on Summary Statement questions
Dependent Reading Reading Mode [48] Behaviours Process measures
(a) Orientation (time spent doing initial survey of texts prior to reading) (b) Linear reading (time spent reading in a linear, sequential fashion) (c) Review mode (time spent looking back over the texts and specific sections,
after reaching end of document) Reading times by section (d) Total reading time (time spent reading in all sections) (e) Introduction (time spent in introductions) (f) Body (time spent in body sections) (g) Conclusion (time spent in concluding sections) Interactivity (h) Number of highlights (both permanent and temporary) (i) Number of transitions between articles (j) Number of length checks
Freund et al. 84
Journal of Information Science, 42(1) 2016, pp. 79–93 � The Author(s), DOI: 10.1177/0165551515614472
such as age and education level, while reading habits were surveyed according to time spent per week on academic and
leisure reading in both print and electronic formats (see Section 3.5). Following this, those in the Interactive condition
participated in a tutorial of the Diigo Toolbar. Next, participants were asked about their pre-task knowledge and interest
in the topic of the assigned article set. They were then instructed to move on to the reading task and asked to take about
30 min to complete before moving on to the post-task. Strict time limits were not imposed to allow for more naturalistic
reading behaviours. Participants then completed a self-assessment of knowledge and interest in the topic after interacting
with the readings and completed the comprehension test. They also completed the User Engagement Scale [49], but these
results are reported elsewhere [50]; the User Engagement Scale was included to better appreciate the user experience in
the reading experiment. Lastly, students completed the Nelson–Denny Standardized Reading Test, which took about 30
min. The Nelson–Denny was administered at the end of the experiment to reduce possible fatigue effects on the main
experimental task. At the conclusion of the study, participants were thanked and presented with a $20 honorarium.
3.4. Data analysis
Data collected on paper forms was entered into a spreadsheet and scores were calculated for the comprehension ques-
tions. Manual coding of the Morae screen cast videos was carried out to identify interaction behaviours. A rule-based
codebook for analysis was developed based on a literature review of studies of reading behaviour and an iterative process
of observing the session videos, deriving codes and refining the definitions. Markers were used to code individual events,
such as checking length, and to mark the start and finish of longer activities, such as scrolling or highlighting text. In
addition, each article was divided up into three blocks – introduction, body and conclusion – so that readers’ activities
could be situated within the text. Once the coding was complete, the log files showing the automatically collected inter-
action data and the manually added codes were retrieved from Morae, cleaned and processed for descriptive and statisti-
cal analysis using SPSS. Analyses were conducted to assess the normality of the outcome measures, differences across
Table 2. Comprehension test sample questions
Text comprehension element
Test Sample items (from Digital Activism Reading Set)
Microstructural True or False ‘The activists described in ‘‘Free the Spectrum!’’ portray themselves as digital utopianists.’ T/F ‘As Curiouscatherine has shown in her blog, Clicktivism is a serious medical disorder.’ T/F
Microstructural Sentence Verification Technique (SVT) (Circle ‘yes’ if you think that it accurately reflects the article content and circle ‘no’ if you think that it does not)
Article: ‘Taking the Slack out of Slacktivism’ (a) ‘The availability of electronic forms of ‘‘activism’’ may even lead to
deterioration in the quality of participation, since people who would otherwise get involved through traditional means may instead opt for digital opportunities.’ Y/N
(b) ‘There is an abundance of conflicting evidence on the subject of digital activism, such that both sides end up bolstering their arguments with the latest research findings.’ Y/N
(c) ‘The Internet has greatly reduced the difficulty of arranging mass campaigns. Never before have so many been able to communicate with such a massive audience.’ Y/N
‘As we move further and further along the path of social media and hyper-connectivity, the distinctions between online and offline activism will become even more pronounced.’ Y/N
Macrostructural Summary Statement (Choose 3 statements that reflect the most important themes cutting across all 3 articles.)
(a) ‘The Internet is not the best tool to foster social justice and promote an egalitarian society.’
(b) ‘The Internet is neither a force for liberation nor a force for repression, but rather a tool which is dependent on its proper use.’
(c) The debate over the value of digital activism is unnecessary and divisive.’
(d) ‘Old and new technologies should coexist since both could be used to promote democratic values and principles.’
(e) ‘Digital activism has the capability to raise global awareness and faciliatate local mobilization, which can influence democracy.’
(f ) ‘Digital activism can’t effect real-life change.’
Freund et al. 85
Journal of Information Science, 42(1) 2016, pp. 79–93 � The Author(s), DOI: 10.1177/0165551515614472
conditions (ANOVA) and associations among independent variables (Correlation). Non-parametric tests were used for
measures that are not normally distributed. An alpha level of 0.05 was used for all statistical tests.
3.5. Participants
All 41 participants were university students from a range of degree programmes: 22 undergraduate, 10 masters and five
doctoral students (four did not report their status). Just over half (55%) of the participants were women and most fell into
the 19–24 (54%) and 25–29 (38%) age groups. All but one participant reported spending more than 30 min per day doing
academic reading: 0.5–1 h (N = 9), 1–2 h (N = 15), 2–3 h (N = 7) or > 3 h (N = 9). In addition, the majority of partici-
pants reported doing between 30 min to 2 h of personal reading per day. Responses regarding reading practices show an
overall balance between digital and print reading, but there was substantial variation within the group. On average, parti-
cipants do 49% (range 10–95%) of their academic reading and 51% (range 0–100%) of their personal reading in digital
format. The devices most commonly used for reading were laptop computers (73% of participants), followed by desktop
computers (49%) and mobile devices (29%). Very few respondents ( < 12%) were using tablets or e-book readers at the
time of the study.
Grade level scores from the Nelson–Denny test show a wide range of reading levels within the group, ranging from
public school level (grade 8 and lower) to the PhD level. As shown in Table 3, the Nelson–Denny scores in the columns
do not correspond closely with the actual academic levels of respondents shown in the rows. This may have been influ-
enced by the inclusion of non-native English speakers in the sample, as we did not pre-test for language skills. Similarly,
the reading motivation scores measured on a five-point scale vary substantially (mean = 3.32, SD = 0.57) and do not
appear to be associated with either reading level or academic level.
4. Results
4.1. The effect of textual environment on comprehension
In order to assess the effects of the study conditions on the comprehension measures, a univariate ANOVA using the
GLM procedure in SPSS was conducted. Mean scores are presented in Table 4. The analysis yielded a borderline main
effect for Text Presentation (F(1,37) = 3.97, p = 0.054) such that the average Comprehension Score (expressed as a per-
centage) was significantly higher for Plain Text (M = 76, SD = 13) than the In-Context condition (M = 69, SD = 12).
The main effect for Interactivity was non-significant. However the interaction effect was borderline significant (F(1,37)
= 3.51, p = =.069). The interaction effect is illustrated in Figure 2, showing the estimated Marginal Means for the two
conditions. In the Non-Interactive condition, there is a substantial difference between the comprehension scores of parti-
cipants reading in Plain Text vs In-Context documents, which is mitigated in the Interactive condition.
It is possible to examine these effects at a finer level by focusing on the error rates of participants on the three separate
sets of questions that made up the comprehension test: True–False (8 points); SVT (12 points); and Summary Statements
(6 points). Individual univariate ANOVAs were conducted using the GLM procedure. For the True–False questions, there
was a main effect only for Interactivity (F(1,37) = 7.49, p = 0.009) such that participants made significantly fewer errors
in the Interactive (M = 1.4, SD = 1.5) than in the Non-Interactive condition (M = 2.56, SD = 1.67). There was no interac-
tion effect. For the SVT questions, there was a main effect only for Text Presentation (F(1, 37) = 9.298, p = 0.004) and
there was an interaction effect (F(1, 37) = 6.98, p = 0.012). Participants made fewer errors in the Plain Text (M = 3.4, SD
= 1.5) than in the In-Context (M = 4.8, SD = 1.5) condition, but, as shown in Figure 1, these differences were mitigated
in the Interactive Condition. There were no effects for the Summary Statement questions.
Table 3. Distribution of Nelson–Denny reading grade levels by actual academic status
Nelson–Denny reading grade level
Academic status Public school High school Undergrad Master’s PhD Total
Undergraduate 1 3 9 4 5 22 Master’s 2 2 3 2 1 10 PhD 0 2 1 0 2 5 Total 3 7 13 6 8 37
Freund et al. 86
Journal of Information Science, 42(1) 2016, pp. 79–93 � The Author(s), DOI: 10.1177/0165551515614472
4.2. The role of reading behaviours
Given the observable variation in comprehension outcomes by reading environment, it is interesting to examine the
extent to which reading behaviours, which may be dependent upon the reading environment, are associated with com-
prehension outcomes. We collected data on a number of reading behaviours, as indicated in Table 5. A correlation anal-
ysis reveals only two significant correlations between comprehension and these behaviours. The number of errors in the
SVT questions was negatively correlated with the time spent reviewing the articles after reading them (r (41) =
− 0.334, p = 0.033) and the number of errors in the Summary Statement question was negatively correlated with the time spent in the article conclusions (r (41) = − 0.315, p = 0.045). There were no correlations between reading beha- viours and the overall comprehension score or the error rate on the True–False questions. Table 5 shows that the highest
mean time spent reviewing the articles after reading them was in the Plain Text/Non-Interactive condition, which had
the highest mean comprehension scores overall. These results suggest that there may be an association between particu-
lar reading strategies (e.g. reviewing articles after reading or focusing more attention on the conclusions) and different
levels of comprehension.
The extent to which variations in reading behaviours may be responsible for the differences in comprehension out-
comes across the experimental conditions is unclear, as the measures that are correlated with comprehension do not
appear to vary across conditions. We conducted individual univariate ANOVAs for a subset of reading behaviour
Figure 2. Plot of mean comprehension scores by conditions.
Table 4. Mean comprehension scores and standard deviations by Text Presentation and Interactivity
Text presentation
Interactivity Plain Text In-Context Overall mean
Non-Interactive 77 (15) 63 (13) 71 (16) Interactive 75 (9) 74 (8) 75 (9) Overall Mean 76 (13) 69 (12) 73 (13)
Freund et al. 87
Journal of Information Science, 42(1) 2016, pp. 79–93 � The Author(s), DOI: 10.1177/0165551515614472
measures that were normally distributed. Results (Table 6) show significant differences for Linear Read time, Total
Reading time, Time in Introduction and Time in Body, all of which reflect two patterns: participants spent more time
reading in the Plain Text than the In-Context condition, and the Interactive condition had a differential effect, generally
increasing the time spent reading in the Plain Text condition and decreasing it in the In-Context condition, relative to
the Non-Interactive condition.
4.3. The role of personal characteristics
In order to determine the effect of possible confounding variables on these outcomes, we examined the extent to which
reading level, reading motivation, self-reported prior knowledge and level of interest in the topic may have influenced
comprehension. As some of these measures do not satisfy the assumption of normality, non-parametric tests were used.
Surprisingly, no significant correlations were found between these measures and comprehension scores, indicating that
stronger, more knowledgeable, and more motivated or interested readers were not more likely to obtain higher scores on
the comprehension test. This suggests that, in the context of this study, variations within the sample based on these char-
acteristics were unlikely to have influenced the observed differences in comprehension outcomes across the experimen-
tal conditions.
To better understand this unexpected result, we conducted further analyses on Reading Level. A correlation analysis
of Reading Level and reading behaviours identified a negative correlation between Reading Level and reading times,
specifically, Orientation Time (r(41) = − 0.455, p = 0.003), Time in Introduction (r(41) = − 0.359 p = 0.021), Time in Body (r(41) = − 0.408 p = 0.008) and Total Reading Time (r(41) = − 0.389 p = 0.012), indicating that those reading at a higher level devoted less time to reading the texts. Next we split the sample into two approximately equal sized groups
by Nelson–Denny Reading Level: strong readers were considered to be at Grade 14 (university undergraduate) level and
above (N = 23) and weak readers were considered to be Grade 13 and below (N = 18). Within each group, correlations
between reading behaviours and comprehension outcomes were analysed to determine what, if any, strategies were
employed to reach an understanding of the texts. A different picture emerged for the two groups. In the weak reader
Table 5. Means and standard deviations for reading behaviours by condition (times in minutes).
Plain Text In Context
Measures Non-Interactive Interactive Total Non-Interactive Interactive Total
Reading mode M (SD)
Orientation Time (N = 19/41)
0.17 0.70 0.42 0.77 2.22 1.57 (0.25) (1.01) (0.75) (0.92) (3.71) (2.85)
Linear Read Time (N = 41)
28.59 42.91 35.41 29.51 25.83 27.49 (9.82) (9.56) (11.96) (10.52) (5.36) (8.08)
Reviewing Time (N = 12/41)
1.44 0.22 0.86 0.65 0.22 0.41 (2.18) (0.48) (1.69) (1.07) (0.57) (0.84)
Reading time by section (min)
Time in Introduction (N = 41)
8.64 11.17 9.85 6.36 5.12 5.68 (4.42) (3.03) (3.95) (1.58) (2.10) (1.94)
Time in Body (N = 41)
24.53 33.71 28.90 26.51 17.85 21.75 (5.25) (6.77) (7.51) (10.38) (6.10) (9.19)
Time in Conclusion (N = 41)
5.44 5.41 5.43 6.09 3.71 4.78 (3.76) (2.86) (3.28) (3.28) (1.80) (2.78)
Total Reading Time (N = 41)
38.61 50.30 44.18 38.77 26.68 32.21 (10.60) (9.24) (11.42) (10.70) (7.17) (11.03)
Textual interactions (events)
Number of Highlights (N = 26/41)
1.18 21.90 11.05 4.44 15.27 10.40 (2.56) (23.11) (18.87) (5.36) (15.13) (12.77)
Number Transitions (N = 41)
5.73 13.50 9.43 6.33 4.36 5.25 (3.96) (9.25) (7.88) (2.78) (2.46) (2.73)
Number length checks (N = 20/41)
0.73 1.10 0.90 2.44 0.55 1.40 (1.19) (1.37) (1.26) (2.30) (0.93) (1.90)
Freund et al. 88
Journal of Information Science, 42(1) 2016, pp. 79–93 � The Author(s), DOI: 10.1177/0165551515614472
group, none of the reading behaviours we observed were correlated with comprehension outcomes. However, in the
strong reader group, several were: the Number of Transitions among texts was negatively correlated with errors in the
Summary Statement questions (r(23) = − 0.499 p = 0.015); Reviewing Time was negatively correlated with errors in the SVT questions (r(23) = − 0.446 p = 0.033), and Number of Highlights was negatively correlated with errors in the True–False questions (r(23) = − 0.417 p = 0.048). These findings echo and extend some of the general effects of read- ing behaviours reported in Section 5.2. Limitations of the size of the dataset prevented us from examining differences
between experimental conditions at this level of granularity; however, based on these results, we can conjecture that,
although strong readers read more quickly, which may have undermined their performance on the comprehension test,
they also read more strategically, using highlights, reviewing tactics, etc., to improve their comprehension of the texts.
The weaker readers simply spent more time reading, which seems to have enabled them to perform at a comparable
level with the stronger readers on the comprehension tests.
5. Discussion
Results indicate that comprehension is highest in the simplest reading environment (Plain Text/Non-Interactive). While
we expected the Plain Text condition to have a positive effect on comprehension, it was somewhat surprising that adding
Interactivity to a Plain Text environment was not associated with a further increase in comprehension. Rather, it seems
that in the Plain Text presentation the negative effect of the cognitive load imposed by the Reading Tools, with which
most participants were unfamiliar, outweighed their contribution in supporting active reading. Further evidence of this is
the low uptake of the annotation capability included in the Diigo Toolkit (only five of 21 participants in the Interactive
condition used this feature) in contrast to the simpler highlighting feature that was more heavily used. This reinforces the
claims made [8, 51] that digital reading environments require ‘lightweight’ tools. The benefits of the Interactive condition
became apparent in the In-Context presentation, which has a higher potential for distraction, similar to many Web-based
information environments. Here, the Interactive features of the environment were associated with a marked performance
improvement, bringing the comprehension outcomes up to a level close to that achieved in the Plain Text presentation.
Interestingly, the Text Presentation condition had more of an effect for the SVT questions, which were focused on the
readers’ microstructural conceptual understanding of the text, suggesting that Text Presentation may influence the depth
of the reading experience. Digital reading in general has been criticized for prompting shallower reading behaviours [3,
4], but these results suggest that this may be more an effect of how texts are presented in digital environments than of
the shift from print to digital reading, as suggested by Keller [22]. In contrast, the Interactivity condition had a greater
effect on the recall of specific facts, as assessed through the True and False questions. This suggests that highlighting, in
particular, enabled readers to better focus upon and remember specific content elements [17, 51]. Variations in the read-
ing environment had little apparent effect on macrostructural comprehension as assessed by the Summary Statement
questions. This may point to the limitations of this study design or instruments in assessing macro-comprehension and
learning more generally, as this may require a more longitudinal or individualized approach to allow for synthesis and
personal knowledge construction. However, it may be that other dimensions of the design environment would better con-
tribute to this component of comprehension. For example, this study made very limited use of linking in the Interactive
condition, such that each article contained one or two links to other articles in the set. Increasing the connectivity of texts
through meaningful links may serve as one means of enabling readers to grasp the ‘big picture’ and synthesize learning
[52, 53]. There is some support for this idea in the behaviour of the stronger readers in this sample, as their success on
the Summary Statement questions was correlated with the number of transitions between texts.
The study produced limited evidence of a relationship between reading behaviours and comprehension, apart from
some indication that strategic reading and time spent reading are associated with comprehension. Across all participants,
Table 6. Results of ANOVA for normally distributed reading behaviours by condition
Text presentation Interactivity Interaction effect
Linear Reading Time F(1,37) = 8.29, p = 0.007 n.s. F(1,37) = 10.30, p = 0.003 Time in Introduction F(1,37) = 19.13, p < 0.001 n.s. F(1,37) = 3.92, p = 0.055 Time in Body F(1,37) = 9.44, p = 0.004 n.s. F(1,37) = 15.60, p < 0.001 Time in Conclusion n.s. n.s. n.s. Total Reading Time F(1,37) = 7.41, p = 0.010 F(1,37) = 4.05, p = 0.051 F(1,37) = 8.96, p = 0.005
n.s., Not significant.
Freund et al. 89
Journal of Information Science, 42(1) 2016, pp. 79–93 � The Author(s), DOI: 10.1177/0165551515614472
reviewing the texts after reading seems to have contributed to conceptual understanding, and spending time in the con-
clusion sections of articles better enabled readers to identify the overarching themes. Stronger readers used particular
reading strategies to improve their comprehension outcomes, including reviewing, highlighting and moving between
texts, while weaker readers devoted more time to reading. In terms of how the textual environment affects reading beha-
viours, the main observation from this study was a difference in the time spent reading. Participants spent more time
reading in the Plain Text than the In-Context presentation, but the availability of the Interactive features had a differen-
tial effect, associated with longer reading times for Plain Text and shorter times for the In-Context presentation. This
finding is puzzling, and warrants further research: why would the same set of tools speed up reading in one environment
and slow it down in another? One possibility is that these two reading environments invoked distinct mental models of
reading, one based on print reading, in which Reading Tools are used to facilitate in-depth reading and engagement with
texts, and the other based on web reading, in which Reading Tools are used primarily to facilitate navigation and mark-
ing trails through content. Regardless of the reason, an efficiency-based approach to reading outcomes might consider
the In-Context/Interactive condition to be superior to either of the Plain Text conditions, given that a similar mean com-
prehension score was achieved despite a substantially lower mean reading time (M = 27 min as compared with M = 44
min for Plain Text conditions).
The lack of any effect of personal characteristics on comprehension outcomes constitutes another puzzling result. At
a minimum, we expected that prior knowledge and reading level would be associated with comprehension, based on the
C-I model of comprehension [10, 42]. However, there is some precedent for this result in the literature on browsing and
searching behaviour, where some studies have shown that expert and novice searchers employ different strategies, but
outcomes are similar for certain tasks [54]. This seems to be the case in this study, as the weaker readers spent more time
and effort reading to achieve outcomes similar to those of the stronger readers, who spent less time and employed more
effective reading strategies. This echoes work by Kintsch showing that readers with different abilities will approach read-
ing tasks differently [55].
5.1. Implications for searching as learning
What are the implications of this work for the design of information interaction environments? First, it must be noted
that the generalizability of these results is limited owing to the nature and size of the sample, the domain focus on higher
education, and the artificial and simplified nature of the experiment. Second, this was not a study of searching, but rather
of reading. However, the study was motivated by the recognition that reading is a core component of searching in full-
text retrieval systems, including web search engines, and that reading activities such as the one in which our participants
engaged represents a typical information use situation in academic and workplace environments. As outlined in previous
work, we characterize searching as a form of ‘semantic navigation’ in which people move through the information envi-
ronment, actively constructing meaning [42] as they go. While the search environment scaffolds this process, it is the
interaction with the content, still primarily textual in most systems, that has the most profound impact on learning.
Results of this study serve as a case to demonstrate that comprehension, the essential precursor to learning, is affected
by the manner and context in which texts are presented and by the tools available to the reader to interact with texts.
Therefore, designers of search systems for which learning is an explicit goal, such as digital libraries, may wish to con-
sider the following implications of this study:
• simple options to display texts in isolation from distracting content may strengthen comprehension outcomes, but may also reduce efficiency;
• lightweight (simple, natural and intuitive) Reading Tools may serve to mitigate the negative effects of a distract- ing reading environment;
• particular tools and features (e.g. highlighting, hypertext) may be associated with different levels of comprehen- sion; therefore, tool provision should be aligned with system goals;
• reading level may affect how a system is used – systems should be flexible and provide support for strategic tool use, especially for those reading at a lower level.
Comprehension assessment was a major challenge in conducting this study, and it is one that must be faced by the
information retrieval community given that learning is an important outcome of search. This work suggests that the sim-
ple fact-based instruments often used to assess comprehension only capture a small part of the meaning-making process,
and points to the need to develop and employ more sophisticated methods to assess comprehension and learning in
search.
Freund et al. 90
Journal of Information Science, 42(1) 2016, pp. 79–93 � The Author(s), DOI: 10.1177/0165551515614472
6. Conclusions
Online reading is the reality for most university students, who function within constantly connected, wireless, mobile and
virtual academic environments. Understanding how to facilitate more effective and engaging online reading is extremely
important in this context. Results of this study reinforce and expand on previous research by demonstrating that compre-
hension outcomes and reading behaviours are influenced by differences in how digital texts are presented to readers and
the available tools and features. Both the Plain Text and the Interactive conditions had positive effects on comprehension
outcomes for this study of academic reading, but their contributions were not simply additive. Interactivity proved to add
more value when texts were presented In-Context and was associated with factual knowledge and recall, while Plain
Text contributed more in the Non-Interactive condition, and was associated with conceptual understanding.
There is a pressing need for further work that brings together models and methods from research on reading, human
information interaction and information retrieval to validate and extend these results. Both digital reading and searching
are changing rapidly in response to new technologies, notably the near ubiquitous personal digital devices that offer
high-resolution screens and enable direct manipulation through touch and gesture. Moving forward, we envision research
to validate these results in more naturalistic settings with larger and more diverse samples by examining real-world use
of e-textbooks, digital libraries or exploratory search systems. The use of eye-tracking and other physiological measures
could strengthen these findings by providing more nuanced results with respect to reading behaviours, as we were only
able to track a limited number of behaviours through manual analysis of the videos. Another area of focus for future
work is the development of new evaluation techniques for comprehension and learning, as the approach taken in this
work is text-dependent and would not scale up to more naturalistic studies with a wide range of texts. Concept mapping
is one possibility for consideration. Finally, there is room to expand this line of inquiry beyond comprehension to con-
sider learning in a more holistic manner, taking into account the time, effort, affect and personal experience that goes
into the learning process.
Acknowledgements
This study could not have been completed without the dedicated assistance and contributions of a number of talented research assis-
tants: Florian Ehrensperger, Elizabeth Shaffer, Kimberly Buschert, Ariel Deardorff and Devin Soper. Our thanks also to the study par-
ticipants who contributed their time and effort.
Funding
Funding for this study was provided by the University of British Columbia Hampton Fund grant ‘Digital Information Use
Environments to Support Academic Work’, and from the Graphics, Animation and New Media (GRAND) Network of Centres of
Excellence, Canada.
References
[1] Institute of the Future. Final synthesis report of the e-journal user study. Stanford University Libraries, 2002.
[2] Rainie L, Zickuhr K, Purcell K, Madden M and Brenner J. The rise of e-reading. Pew Research Centre, http://libraries.pewinter-
net.org/2012/04/04/the-rise-of-e-reading/ (2012).
[3] Wolf M and Stoodley CJ. Proust and the squid: The story and science of the reading brain. New York: HarperCollins, 2007.
[4] Carr N. The shallows: What the Internet is doing to our brains. New York: Norton, 2010.
[5] Joint Information Systems Council. Information behaviour of the researcher of the future. London: JISC, 2008.
[6] Nicholas D, Huntington P, Jamali HR, Rowlands I and Fieldhouse M. Student digital information-seeking behaviour in context.
Journal of Documentation 2009; 65(1): 106–132.
[7] Cull BW. Reading revolutions: Online digital text and implications for reading in academe. First Monday 2011; 16(6).
[8] Pearson J, Buchanan G and Thimbleby H. Designing for digital reading. Synthesis Lectures on Information Concepts, Retrieval
and Services, 29. San Rafael, CA: Morgan and Claypool, 2014.
[9] Rouet J-F. The skills of document use: From text comprehension to web-based learning. New York: Routledge, 2006.
[10] Kintsch W. Comprehension: A paradigm for cognition. Cambridge: Cambridge University Press, 1998.
[11] Dehaene S. Reading in the brain: The science and evolution of a human invention. New York: Viking, 2009.
[12] Mangen A. Hypertext fiction reading. Journal of Research in Reading 2008; 31(4): 404–419.
[13] Ben-Yehudah G and Eshet-Alkalai Y. The influence of text annotation tools on print and digital reading comprehension. In:
Proceedings of the 9th Chais conference for the study of innovation and learning technologies: Learning in the technological
era, 2014.
[14] Koolen C, Garnett A and Siemens R. Electronic environments for reading: An annotated bibliography of pertinent hardware and
software. Scholarly and Research Communication 2012; 3(4): 1–62.
Freund et al. 91
Journal of Information Science, 42(1) 2016, pp. 79–93 � The Author(s), DOI: 10.1177/0165551515614472
[15] Purdy JP. Why first-year college students select online research resources as their favorite. First Monday 2012; 17: 9.
[16] Woody WD, Daniel DB and Baker CA. E-Books or textbooks: Students prefer textbooks. Computers & Education 2010; 55(3):
945–948.
[17] Marshall CC. Reading and interactivity in the digital library: Creating an experience that transcends paper. In: Marcum D and
George G (eds), Digital library development: The view from Kanazawa. Westport, CT: Libraries Unlimited, 2005, pp. pp.
127–145.
[18] Dillon A. Reading from paper versus screens: A critical review of the empirical literature. Ergonomics 1992; 35(10):
1297–1326.
[19] Dyson MC. How do we read text on screen? In: van Oostendorp H, Breure L and Dillon A (eds), Creation, use, and deployment
of digital information. Mahwah, NJ: Lawrence Erlbaum, 2005, pp. 279–306.
[20] Kidwa K. A web-based reading environment designed to fundamentally extend readers’ interaction with informational texts. In:
Proceedings of the 9th international conference of the learning sciences, 2010, pp. 778–785.
[21] Dyson MC and Haselgrove M. The effects of reading speed and reading patterns on the understanding of text read from screen.
Journal of Research in Reading 2000; 23(2): 210–223.
[22] Keller A. In print or on screen? Investigating the reading habits of undergraduate students using photo-diaries and photo-inter-
views. Libri 2012; 62: 1–18.
[23] Noyes JM and Garland KJ. Computer vs paper-based tasks: Are they equivalent? Ergonomics 2008; 51(9): 1352–1375.
[24] Beymer D, Orton PZ and Russell DM. An eye tracking study of how pictures influence online reading. In Baranauskas C et al.
(eds), INTERACT 2007. Lecture Notes in Computer Science, Vol. 4663. Berlin: Springer, 2007, Part II, pp. 456–460.
[25] Adler M. How to mark a book. Saturday Review of Literature, 6 July, 1940, pp. 11–12.
[26] Harvel L. Convenience is not enough. Innovative Higher Education 2006; 31(3): 161–174.
[27] Liew CL, Foo S and Chennupati KR. A proposed integrated environment for enhanced user interaction and value-adding of
electronic documents: An empirical evaluation. Journal of the American Society for Information Science and Technology 2001;
52(1): 22–35.
[28] Kopak R and Chiang C. Annotating and linking in the Open Journal Systems. First Monday 2007; 12: 10.
[29] McNamara DS and Magliano JP. Towards a comprehensive model of comprehension. In: Ross B (ed.), The psychology of learn-
ing and motivation, Vol. 51. New York: Elsevier Science, 2009, pp. 297–284.
[30] White R and Roth RA. Exploratory search: Beyond the query-response paradigm. Synthesis Lectures on Information Concepts,
Retrieval and Services. San Rafael, CA: Morgan Claypool, 2009.
[31] Wildemuth BM and Freund L. Assigning search tasks designed to elicit exploratory search behaviours. In: Human computer
interaction and information retrieval symposium (HCIR 2012), Cambridge, MA, 2012.
[32] Amadieu F, Tricot A and Mariné C. Interaction between prior knowledge and concept-map structure on hypertext comprehen-
sion, coherence of reading orders and disorientation. Interacting with Computers 2010; 22(2): 88–97.
[33] Calisir F and Gurel Z. Influence of text structure and prior knowledge of the learner on reading comprehension, browsing and
perceived control. Computers in Human Behaviour 2003; 19: 135–145.
[34] Dalal NP, Quible Z and Wyatt K. Cognitive design of home pages: An experimental study of comprehension on the world wide
web. Information Processing and Management 2000; 36(4): 607–621.
[35] Lustria MLA. Can interactivity make a difference? Effects of interactivity on the comprehension of and attitudes toward online
health content. Journal of the American Society for Information Science and Technology 2007; 58(6): 766–776.
[36] Jeong H. A comparison of the influence of electronic books and paper books on reading comprehension, eye fatigue, and per-
ception. The Electronic Library 2012; 30(3): 390–408.
[37] Maynard S and McKnight C. Children’s comprehension of electronic books: An empirical study. New Review of Children’s
Literature and Librarianship 2001; 7(1): 29–53.
[38] Kintsch W and Kintsch E. Comprehension. In: Paris SG and Stahl SA (eds), Children’s reading comprehension and assessment.
Mahwah, NJ: Lawrence Erlbaum Associates, 2005, pp. 71–92.
[39] Naumann AB, Wechsung I and Krems JF. How to support learning from multiple hypertext sources. Behavior Research
Methods 2009; 41(3): 639–646.
[40] Le Bigot L and Rouet J-F. The impact of presentation format, task assignment, and prior knowledge on students’ comprehension
of multiple online documents. Journal of Literacy Research 2007; 39: 445–469.
[41] Schutte NS and Malouff JM. Dimensions of reading motivation: Development of an adult reading motivation scale. Reading
Psychology 2007; 28(5): 469–489.
[42] Freund L, O’Brien HL and Kopak R. Getting the big picture: Supporting comprehension and learning in search. In: Searching
as learning workshop at the information interaction in context conference (IIiX 2014), Regensburg, Germany, 30 August 2014.
[43] Brown JI, Fishco VV and Hanna G. Nelson–Denny reading test – Manual for scoring and interpretation, forms G and H. Itasca,
IL: Riverside Publishing, 1993.
[44] Coleman C, Lindstrom J, Nelson J, Lindstrom W and Gregg K. Passageless comprehension on the Nelson–Denny Reading test:
Well above chance for university students. Journal of Learning Disabilities 2010; 43: 244–249.
[45] Haught PA and Walla RT. Adult learners: New norms on the Nelson–Denny Reading Test for healthcare workers. Reading
Psychology 2002; 23: 3.
Freund et al. 92
Journal of Information Science, 42(1) 2016, pp. 79–93 � The Author(s), DOI: 10.1177/0165551515614472
[46] Royer JM, Greene BA and Sinatra GM. The sentence verification technique: A practical procedure teachers can use to develop
their own reading and listening comprehension tests. Journal of Reading 1987; 30: 414–423.
[47] van Dijk TA. Macrostructures: An interdisciplinary study of global structures in discourse, interaction, and cognition.
Hillsdale, NJ: Lawrence Erlbaum, 1980.
[48] Hornbæk K and Frøkjær E. Reading patterns and usability in visualizations of electronic documents. ACM Transactions on
Computer–Human Interaction 2003; 10(2): 119–149.
[49] O’Brien HL and Toms EG. The development and evaluation of a survey to measure user engagement in e-commerce environ-
ments. Journal of the American Society for Information Science and Technology 2010; 61(1): 50–69.
[50] O’Brien HL, Freund L. and Kopak R. Investigating the role of user engagement in digital reading environments. In: ACM
SIGIR Conference on Human Information Interaction and Retrieval (CHIIR), Chapel Hill, NC, 13–17 March 2016.
[51] Chiang CA. Multi-dimensional approach to the study of online annotation. Upublished doctoral dissertation, School of Library,
Archival and Information Studies, University of British Columbia, 2010.
[52] Kopak R, Freund L and O’Brien H. Digital information interaction as semantic navigation. In: Foster A and Rafferty P (eds),
Innovations in IR: Perspectives for theory and practice. London: Facet Publishing, 2011, pp. 117–134.
[53] Kopak R and Chiang C. An interactive reading environment for online scholarly journals: The Open Journal Systems reading
tools. OCLC Systems and Services 2009; 25(2): 114–124.
[54] Lazonder AW, Biemans HJA and Worpeis IGJH. Differences between novice and experienced users in searching information
on the World Wide Web. Journal of the American Society for Information Science 2000; 51: 576–581.
[55] Kintsch W. Learning and constructivism. In: Tobias S and Duffy TM (eds), Constructivist instruction: Success or failure. New
York: Routledge, 2009.
Freund et al. 93
Journal of Information Science, 42(1) 2016, pp. 79–93 � The Author(s), DOI: 10.1177/0165551515614472