Reading Analysis Discussion Post

Getting the big picture: supporting comprehension and learning in search

Luanne Freund iSchool

University of British Columbia Vancouver, BC, Canada

604 822 0825 [email protected]

Heather O’Brien iSchool

University of British Columbia Vancouver, BC, Canada

604-822-6365 [email protected]

Rick Kopak iSchool

University of British Columbia Vancouver, BC, Canada

604-822-2898 [email protected]

ABSTRACT This position paper contributes to the development of a research agenda on searching as an informal learning process by applying an established theory of comprehension and learning to the problem of web search. We present a conceptual framework for searching as learning based upon Kintsch’s [10] Construction- Integration Model of text comprehension. Drawing upon this model, we identify and provide a rationale for a number of user and content factors that are likely to be associated with learning while searching: prior knowledge, motivation, knowledge actualization and text coherence. Some implications for the design of web search systems are described.

Categories and Subject Descriptors H.1.2 [User/Machine Systems]: Human Information Processing H.3.3 [Information Search and Retrieval]: Search Process. H.5.2 [User Interfaces]: User-Centered Design

General Terms Design, Human Factors

Keywords web search, learning, search process, Construction-Integration Model, comprehension

1. INTRODUCTION Search systems serve as platforms for information interaction and use in addition to information access. As such, there is a need to extend existing frameworks for modeling, supporting and assessing search success. From an information access perspective, it makes sense to design search systems to display the most relevant results, optimally ranked and represented for efficient selection by users. However, this approach does little to encourage information interaction and use.

In recognition of this limitation, various approaches have been proposed to extend information retrieval (IR) models to incorporate some notion of information use, notably task-based approaches [25,7]. For example, [7] propose that the augmentation of task performance should be considered an

important goal for IR. Others have argued for the substitution of usefulness for relevance as the conceptual basis for measuring search success [6]. These approaches are valuable, although the practicality of assessing a system on the basis of external outcomes (i.e. work task completion) represents an ongoing challenge. On the other hand, models that focus on information interaction and associated outcomes (e.g. comprehension, learning, engagement, pleasure) are only now emerging in IR research. One example is exploratory search, where the goal is to support the searcher in discovery, personal growth, sense making, and learning [27]. The emergence of “searching for learning” as a new direction for information interaction research speaks to the recognition that experiential outcomes are not well understood, and we do not have a good sense of how to measure constructs such as learning or design for them in IR systems [1].

In this position paper, we focus on searching in the context of human information interaction, and consider how systems might be designed to provide enhanced support for outcomes that we consider to be of primary importance, namely: comprehension and learning. In the sections to follow we will define the key concepts, introduce relevant theories, propose an initial framework, and identify a number of design implications for search systems.

2. CONCEPTUAL FRAMEWORK 2.1 Human information interaction Recognizing that conceptions of search vary substantially across disciplines and research fields, we begin by establishing our frame of reference for search as a user-driven, technology supported process of human information interaction (HII). In this process, people engage intellectually, physically and emotionally with digital information. In truly interactive information environments, the effects are mutual: users are influenced by the content and the content is shaped by the users [19]. HII includes searching and browsing as well as other activities.

Useful models of HII in information science include those of Marchionini [18,19], Toms [23] and Dillon [3]. Important components of these models are the motivation and influence of people’s tasks and goals, the roles played by cognitive and perceptual abilities and people’s mental models of the information environment, the importance of cues and landmarks in the information environment, and the iterative nature of the process, which involves querying, browsing, reading, extracting and integrating information. HII has been modeled most often using cognitive and behavioral features, although affect has received considerable attention in recent years [17] building on early foundational work by Kuhlthau [16].

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SAL Workshop, IIiX 2014, August 30, 2014, Regensburg Germany Copyright 2014 ACM 1-58113-000-0/00/0010 …$15.00.

The relationships between HII and comprehension and learning, while implicitly recognized, have received less explicit attention. Kuhlthau’s [16] important work on the Information Seeking Process is an exception, although the implications of that work have focused more on library services and instruction than the design of information systems. A key contribution of Kulthau’s work is the recognition that learning-focused tasks carried out through information seeking have distinct stages. Vakkari [24] applied Kuhlthau’s work to the search process and in a series of studies, identified variations in behavior as searchers progressed through three task stages (Pre-focus, Formulation and Post Focus) and their domain knowledge increased. Based on this work, Vakkari proposed a theory of the task-based IR process, which associates task-based search behavior with the development of searchers’ mental models over time. Other relevant research focuses on semantic navigation [15], a concept that builds upon earlier studies of reading in digital environments [4]. Semantic navigation is described as follows: “as people move through information, they actively construct meaning and make use of both explicit and implicit features of the information and its environment as guides in this process.” [15, p. 118]. Three approaches to support semantic navigation in digital information environments are identified by Freund, Kopak and O’Brien [15]: exposure of users to the cues and meanings latent in information objects through labeling, highlighting and mapping; provision of tools and markers to enable searchers to probe the environment and document their journey; and the design of engaging environments in which the content is interesting, relevant and appealing and the technology is intuitive [15]. Given that learning is an important outcome of semantic navigation and, implicitly, of HII more generally, a clearer grounding in theories of comprehension and learning has the potential to strengthen this research area.

2.2 Comprehension In an extensive review of research on comprehension, McNamara and Magliano [20] define it as follows: “Comprehension, arguably the backbone of cognition, is the processing of information to extract meaning. It is a complex cognitive process that is necessary for virtually all higher-level cognitive activities, including learning, reasoning, problem solving and decision making” [20, p. 298]. Key to the process of comprehension is the emergence of a coherent mental representation based on the texts, stories, sentences or other information objects encountered, built up through connections, or inferences, made between the semantic content, other information in the environment and prior knowledge.

To examine the role of comprehension and learning in search, we take as a starting point one of the earliest and most influential models of text comprehension: the Construction-Integration (CI) model, which has received limited attention in IS&R research to date [2, 26]. The C-I model was proposed by Kintsch [9, 10], based on earlier work on discourse comprehension [12]. A central notion of the C-I model is that comprehension involves the creation of mental representations on several levels, as illustrated in Figure 1. While interacting with a text or information environment at the surface level, a person makes connections and between words and concepts and constructs a mental representation in the form of a textbase: a network of propositions, or pairs of concepts and relations. This textbase is derived from the information environment and consists of a microstructure, “driven by the local structure of the text” and a macrostructure, “driven by the text’s global or hierarchical

structure” [20, p. 311], the latter often referred to as “gist”. The next level, the situation model, is a further developed network of propositions that extends the textbase by drawing upon a person’s background knowledge and experience [20]. Well-structured and coherent situation models produce stronger long-term memories and result in more learning.

Figure 1: Layered model of comprehension (Source: Aschoeke at en.wikibooks)

2.3 Learning As this preceding description suggests, the C-I Model is grounded in a constructivist approach to learning, which frames learning as an active, often effortful, process in which the learner makes connections and draws inferences; thereby building up and altering mental representations stored in long term memory and constructing knowledge over time [11]. For Kintsch, the end result of learning from a text is essentially the same as that of comprehension: “a situation model that faithfully represents the meaning of that text, both at a local and global level, and integrates it with the reader’s prior knowledge and learning goals” [11, p. 224]. Because the creation of a situation model draws heavily upon the reader’s own experience and prior knowledge, such models vary from one individual to the next. Further, the situation model is influenced by a person’s goals, so the same person interacting with content in order to, for example, find facts, prepare for a discussion or solve problems, may produce different types of situation models.

Kintsch [11] identifies a number of conditions that support text- based leaning. Coherence of the information sources is important: connections between concepts and propositions should be clear at the surface level. The learner needs to have sufficient background knowledge of the material presented to enable the construction of a situation model from the text. Too large a knowledge gap between the learner and the text will prevent learning from taking place. If tasks are assigned, the learning goals should be clear so that attention can be focused accordingly. Learners should be sufficiently motivated to invest effort into the process of making connections and constructing knowledge, as greater learning

benefits result from greater effort invested, provided that the necessary skills and knowledge are in place. Finally, learners should be afforded opportunities to engage in a productive way with the information, for example by creating summaries, solving problems or building models.

While most of Kintsch’s own work has focused on the reading of stand-alone texts, the C-I Model has been applied to studies of comprehension and learning in hypertext. Hypertext environments, consisting of many interconnected texts, are a closer analogue to search systems, in which the challenge of constructing coherent mental representations is greater. Foltz [5], for example noted the effect on comprehension of the inherent non-linearity of hypertext, and the consequent burden of choice placed on the reader. Salmerón, Kintsch and Cañas [22] used the C-I model to address the effect of the reading order of nodes in a hypertext on comprehension levels. In the study, participants selected reading order based either on a ‘coherence’ criterion (selecting the related node that seems more semantically related), or on an ‘interest’ criterion (selecting the related node that seemed more interesting). Their results support the position that choosing a reading order on the basis of perceived semantic relationship (i.e. greater cohesion) “is a good strategy to improve comprehension, especially for novice readers” [22, p. 52].

3. SEARCHING AS LEARNING In this section, we pull together the threads from models of HII, related notions such as semantic navigation and task-based IR, and the C-I Model to propose an initial framework for supporting learning in search. Figure 2 illustrates some of the key components, including a set of search-related processes (querying, reading, selecting and navigating) that occur in parallel with comprehension and learning activities. Over time, these processes iterate and produce an increasingly coherent and dense situation model, which represents learning. The human factors that most influence this process are indicated: tasks and goals, motivation, knowledge and experience and skills, including reading, searching and analytical skills. To illustrate the framework, we can use the case of exploratory web search, which has been characterized as a search task type for which learning is an important outcome [27, 28]. Our particular interest is the informal learning that takes place while searching and interacting with online content, as in the following scenario, used as an assigned search task description in [8]:

You have recently moved to Boston and you are interested in buying a home. You have heard that most homes built before 1978 have some lead paint, but that their paint status is often reported as "unknown." You think you should learn about lead paint and housing. The Web seems like a good place to locate this information [8, p. 684].

In this scenario, let us imagine that a searcher, Bob, submits a query to a search engine and then spends about 15 minutes iterating through cycles of selecting, reading and navigating through content and submitting additional queries. Bob does not save or bookmark any documents for later study, but rather aims to get the gist of this issue while moving through the content. Throughout the search session, he processes the textual information automatically, without conscious effort, as he is a strong reader. Also, he is able to establish clear connections between the articles he reads because there is substantial overlap between the points made in the various documents he reads.

However, while Bob has no difficulty comprehending the content at the surface level, he fails to learn much from the experience because he lacks knowledge about home ownership and about Boston that that would allow him to establish a coherent situation model. His inability to gain a “big picture” sense of this issue (situation model) frustrates him to the point where he loses the motivation to search further. So, from a learning perspective, this is an unsuccessful search.

Figure 2. Comprehension and Learning in relation to Search

Bob’s story illustrates some of the assumptions about searching as learning that we derive from the C-I Model. These align with Kintsch’s [11] conditions that support learning, outlined in the previous section. We assume that, in general, learning will be positively influenced by: the coherence of the “text” – in this case a set of webpages retrieved by the search engine, the motivation and effort of the searcher, the prior knowledge of the searcher, and the extent to which the searcher has opportunities to actualize the knowledge gained. Each of these assumptions is discussed briefly below together with some implications for system design.

Coherence is characterized by Kintsch [11] as the extent to which propositions evident in the text “overlap” through grammatical or semantic associations. Because making connections is the central activity in learning, coherence supports learning. In web search, we can consider the set of all documents retrieved over the course of a search session as well as those in close link proximity to constitute the learning environment, equivalent to Kintsch’s “text”. What characteristics of this environment contribute to its coherence and thereby support greater coherence in the textbase and situation model constructed by the searcher? One feature may be relevance, the central concept in IR. We can assume that a set of documents that are highly relevant to a searcher’s query are also likely to be semantically and structurally related to one another, and therefore to be highly coherent as well. Another characteristic of the environment that adds to coherence is framing, which may take place at the document level, through clearly expressed document genres or architectural cues in the form of headings and menus. Kintsch notes that textual schemas are particularly important for the formation of macrostructures: “readers know that particular text types tend to be organized in certain ways and employ this knowledge to construct schema- based macrostructures” (68). Related notions of “information shape” [4] and “data frames” [13] have been associated with navigation, learning and sensemaking. Other techniques that can be applied include: highlighting query terms in retrieved documents, clustering search results, and providing link structure visualizations for search results [15]. The notion of typed links, which identify the functional relationship between the source and target nodes, may offer particular support for the establishment of global coherence within a hypertext environment [14]. Provision

of structural signals of this type will reduce the likelihood of searchers choosing divergent paths and increase their understanding of the whole.

Motivation and Effort are associated with learning, as Kintch [11] notes that even for readers who are skilled at decoding words and extracting meaning from texts, constructing knowledge structures requires effort. Users who are not highly motivated by the task at hand are unlikely to expend the effort necessary to learn. This points to the importance of user engagement in the search process [15], and more specifically, the potentially positive effect of challenge [21]. When search tasks are prompted by important real-world work tasks, such as planning a wedding, motivation is less likely to be a concern, but in other cases, gamification is one design option that may increase positive challenge and motivation to benefit learning.

Prior Knowledge affects learning because searchers with existing knowledge representations will use them to bridge semantic gaps in the information source and to scaffold newly acquired information to create a coherent situation model. If the knowledge gap between searcher and content is too great, learning will not occur. Interestingly, Kintsch [10, 11] reports that there is an interaction effect between prior knowledge, coherence and motivation, because a highly coherent information source may not offer enough diversity and challenge for a high knowledge searcher to stimulate interest and effort on his or her part, which could result in less learning. In contrast, for low knowledge searchers, a highly coherent information source is likely to increase learning, but still may not be sufficient to enable the construction of a situation model, as illustrated by Bob’s story. So, different learning environments are likely to be optimal for high and low knowledge searchers, with the level of coherence serving as one of the differentiating features. Search system features able to bootstrap prior knowledge, for example by displaying dynamically creating overviews or summaries of the topic on the results page or by linking to relevant encyclopedia articles, may increase learning.

Knowledge actualization refers to the extent to which searchers engage in creating tangible representations of knowledge and applying them in the context of their tasks and goals. The relationship between activity or praxis and learning is one of the basic tenets of constructivist learning, and is rooted in the idea that knowledge is personal and tightly coupled with experience. In most search systems, the searcher has considerable agency in formulating queries and selecting content, but limited scope for more creative activities. The provision of tools and workspaces to support annotation, analysis, clipping, composition and other similar activities in the context of search have potential to support learning.

4. CONCLUSION The ideas presented here are meant to stimulate thought, discussion and future research, rather than representing a definitive perspective on these issues. Some of the already apparent limitations of this framework stem from the cognitive and textual orientation of the C-I Model: greater recognition is needed for the emotional and social aspects of searching and learning and for the proliferation of multimedia online content, which is invaluable for informal learning. Recognized limitations of the C-I model are also worthy of further exploration, including the lack of attention to individual differences, learning styles and metacognition in learning [20].

The main contributions of this work are the introduction of an established model of comprehension and the identification of a number of assumptions about the factors that support learning in search for further testing. Among these, the positive relationship between effort and learning has the most potential to challenge accepted models of searching, which are so heavily tuned to ensure ease and efficiency.

5. ACKNOWLEDGMENTS Our thanks to the UBC Hampton Fund and the NCE GRAND for the funding that made this research possible.

6. REFERENCES [1] Allan, J., Croft, W.B., Moffat, A. and Sanderson, M. (2012).

Frontiers, challenges, and opportunities for information retrieval: Report from SWIRL 2012 the second strategic workshop on information retrieval in Lorne. SIGIR Forum, 46(1), 2–32.

[2] Cole, C. & Mandelblatt, B. (2000). Using Kintsch’s discourse comprehension theory to model the user’s coding of an informative message from an enabling information retrieval system. Journal of the American Society for Information Science. 51(11), 1033-1046.

[3] Dillon, A. (1999). TIME - A multi-level framework for the design and evaluation of digital libraries. International Journal of Digital Libraries, 2(2/3), 170-177.

[4] Dillon, A. (2000). Spatial-semantics: how users derive shape from information space. Journal of the American Society for Information Science and Technology. 51(6), 521-528.

[5] Foltz, Peter W. (1996). Comprehension, coherence, and strategies in hypertext and linear text. In Rouet, Jean- Francois, et al., eds., Hypertext and Cognition, 109-136. Mahwah, NJ: Lawrence Erlbaum Associates.

[6] Freund, L. & Berzowska, J. (2010) The goldilocks effect: task-centred assessments of e-government information. Proceedings of the American Society for Information Science and Technology. 47(1), 1–10.

[7] Järvelin, K. & Ingwersen, P. (2004). Information seeking research needs extension towards tasks and technology. Information Research. 10(1). http://www.informationr.net/ir/10-1/paper212.html

[8] Kim, J. (2009). Describing and predicting information- seeking behavior on the Web. Journal of the American Society for Information Science & Technology, 60(4), 679- 693.

[9] Kintsch, W. (1988). The role of knowledge in discourse comprehension: A construction–integration model. Psychological Review, 95, 163–182.

[10] Kintsch, W., Comprehension: a paradigm for cognition. 1998, New York: Cambridge University Press.

[11] Kintsch, W. (2009). Learning and Constructivism. In Constructivist Instruction: Success or Failure. Tobias and Duffy.

[12] Kintsch, W., & van Dijk, T. A. (1978). Toward a model of text comprehension and production. Psychological Review, 85, 363–394.

[13] Klein, G., Phillips, J.K., Rall, E.L. & Pelusi, D.A. (2006) A data/frame theory of sensemaking, in Expertise Out of

Context: Proc. 6th Int’l Conf. Naturalistic Decision Making, R.R. Hoffman, ed.,Lawrence Erlbaum Associates.

[14] Kopak, R. (2000). A Taxonomy of Link Types for use in Hypertext. Unpublished PhD. Dissertation.

[15] Kopak, R., Freund, L. & O’Brien, H.L. (2011) Digital Information Interaction as Semantic Navigation, In A. Foster & P. Rafferty (Eds.) Innovations in IR: Perspectives for Theory and Practice (pp. 117-134). London: Facet Publishing.

[16] Kuhlthau, C. (1991). Inside the search process: Information seeking from the user’s perspective. Journal of the American Society for Information Science, 42(5), 361–371.

[17] Lopatovska, I., & Arapakis, I. (2011). Theories, Methods and Current Research on Emotions in Library and Information Science, Information Retrieval and Human-Computer Interaction. Information Processing and Management. 47(4), 575–592.

[18] Marchionini, G. (1995). Information Seeking in Electronic Environments. Cambridge University Press, Cambridge, UK.

[19] Marchionini, G. (2008). Human-information interaction research and development. Library & Information Science Research, 30(3), 165-174.

[20] McNamara, D. S., & Magliano, J. P. (2009). Towards a comprehensive model of comprehension. In B. Ross (Ed), The psychology of learning and motivation, 51, 297-284. New York, NY, US: Elsevier Science.

[21] O’Brien, H.L. & Toms, E.G. (2008). What is user engagement? A conceptual framework for defining user

engagement with technology. Journal of the American Society for Information Science and Technology. 59(6), 938– 955.

[22] Salmerón, L., Kintsch, W. & Cañas, J. J. (2006). Reading strategies and prior knowledge in learning with hypertext. Memory & Cognition, 34, 1157–1171.

[23] Toms, E. G. (2002). Information interaction: providing a framework for information interaction. Journal of the American Society for Information Science and Technology, 53(10), 855-862.

[24] Vakkari, P. (2001). A theory of the task-based information retrieval process: a summary and generalization of a longitudinal study. Journal of Documentation. 57(1), 44-60.

[25] Vakkari, P. (2003). Task-based information searching. ARIST, 37, 413-464.

[26] Vakkari, P. (2010). Exploratory searching as conceptual exploration. HCIR 20101.

[27] White, R., & Roth, R.A. (2009). Exploratory Search: Beyond the Query-Response Paradigm. Synthesis Lectures on Information Concepts, Retrieval, and Services #3. Morgan & Claypool.

[28] Wildemuth, B.M. & Freund, L. (2012). Assigning search tasks designed to elicit exploratory search behaviors. Human Computer Interaction and Information Retrieval Symposium (HCIR 2012), Cambridge, MA.