Human Computer Interaction

1501

Visualisation and Instructional Design

John Sweller

School of Education

University of New South Wales

Sydney NSW 2052

Australia

[email protected]

University of New South Wales

Abstract

Human cognitive architecture includes a working memory of limited capacity and duration

with partially separate visual and auditory channels, and an effectively infinite long-term

memory holding many schemas that can vary in their degree of automation. These cognitive

structures have evolved to handle information that varies in the extent to which elements can be

processed successively in working memory or, because they interact, must be processed

simultaneously imposing a heavy load on working memory. Cognitive load theory uses this

combination of information and cognitive structures to guide instructional design. Several

designs that rely heavily on visual working memory and its characteristics are discussed in this

paper.

Knowledge of human cognitive architecture is essential for instructional design, and visual

cognition is a central aspect of human cognition. Not surprisingly, there are several instructional

design effects that rely heavily on the manner in which humans visually process information.

This paper discusses some relevant information structures, cognitive structures and instructional

designs that rely on our knowledge of visual information processing.

A. Information structures

While considerable work by many researchers over several decades has been devoted to the

organization of human cognitive architecture, far less effort has gone into investigating the

information structures that must have driven the evolution of that architecture. Some work has

been carried out by Sweller (1994) and Halford, Wilson and Phillips (1998). Sweller (1994)

suggested that all information can be placed on a continuum according to the extent to which the

elements that constitute the information interact. At one extreme, there is no interaction between

the elements that need to be learned. They are independent. Element interactivity is low, or

indeed, non-existent, and that means each element can be considered and learned serially without

reference to any other element. Because elements at the low element interactivity end of the

continuum do not interact with each other, there is no loss of understanding despite each element

being learned individually and in isolation. Understanding is defined as the ability to process all

elements that necessarily interact, simultaneously in working memory. Learning such material

1502

imposes a low cognitive load because each element can be learned without reference to other

elements. At the other extreme of the continuum, there is close interaction between the various

elements that need to be learned. Element interactivity is high which means that if the material is

to be understood, all of the information with its multiple elements must be processed

simultaneously, imposing a heavy cognitive load.

B. Human cognitive architecture

The term “cognitive architecture” refers to the manner in which cognitive structures are

organized. This section describes those aspects of human cognitive architecture relevant to

visually based instructional design and around which there is broad agreement.

1. Working memory. Initially designated short-term memory (e.g. Miller, 1956) it is now

more commonly referred to as working memory (e.g. Baddeley & Hitch, 1974) to reflect the

change in emphasis from a holding store to the cognitive system’s processing engine. Working

memory can be equated with consciousness in that the characteristics of our conscious lives are

the characteristics of working memory. The most commonly expressed attributes of working

memory are its extremely limited capacity, discussed by Miller (1956) and its extremely limited

duration, discussed by Peterson and Peterson (1959). In fact, both of these limitations apply only

to novel information that needs to be processed in a novel way. Well-learned material, held in

long-term memory, suffers from neither of these limitations when brought into working memory

(Ericsson & Kintsch, 1995).

While initially conceptualized as a unitary concept, working memory is now more commonly

assumed to consist of multiple streams, channels or processors. For example, Baddeley (e.g.

Baddeley, 1992; Baddeley & Hitch, 1974) divided working memory into a visuo-spatial

sketchpad for dealing with 2 dimensional diagrams or 3 dimensional information, a phonological

loop for dealing with verbal information and a central executive as a coordinating processor.

A major consequence of the limitations of working memory is that when faced with new, high

element interactivity material, we cannot process it adequately. We invariably fail to understand

new material if it is sufficiently complex. In order to understand such material, other structures

and other mechanisms must be used. Processing high element interactivity material requires the

use of long-term memory and learning mechanisms.

2. Long-term memory. Because we are not conscious of the contents of long-term memory

except when they are brought into working memory, the importance of this store and the extent

to which it dominates our cognitive activity tends to be hidden from us. Given this hidden nature

of long-term memory, it is not surprising that modern research into long-term memory post-dated

research into working memory. It took some time for researchers to realize that long-term

memory is not just used to recognize or recall information but rather, is an integral component of

all cognitive activity including activities such as high-level problem solving. When solving a

problem, it was previously assumed that knowledge stored in long-term memory was of

peripheral, rather than central importance. De Groot’s (1965) work on chess (first published in

1946) demonstrated the critical importance of long-term memory to higher cognitive functioning.

He demonstrated that memory of board configurations taken from real games was critical to the

performance of chess masters who were capable of visualising enormous numbers of board

configurations. This skill depended on schemas held in long-term memory.

3. Schemas. Knowledge is stored in long-term memory in schematic form and schema theory

describes a major learning mechanism. Schemas allow elements of information to be categorized

1503

according to the manner in which they will be used. Thus, for example, we have a schema for the

letter a that allows us to treat each of the infinite number of printed and hand-written variants of

the letter in an identical fashion. Schemas first became important cognitive constructs following

the work of Piaget (1928) and Bartlett (1932). They became central to modern cognitive theory,

especially theories of problem solving, in the 1980’s. As well as the work of de Groot (1965) and

Chase and Simon (1973), Gick and Holyoak (1980; 1983) demonstrated the importance of

schemas during general problem solving and Larkin, McDermott, Simon and Simon (1980) and

Chi, Glaser and Rees (1982) demonstrated the critical role of schemas in expert problem solving.

As a consequence of this work, most researchers now accept that problem solving expertise in

complex areas demands the acquisition of tens of thousands of domain-specific schemas. These

schemas allow expert problem solvers to visually recognize problem states according to the

appropriate moves associated with them. Schema theory assumes that skill in any area is

dependent on the acquisition of specific schemas stored in long-term memory.

Schemas, stored in long-term memory, permit the processing of high element interactivity

material in working memory by permitting working memory to treat the many interacting

elements as a single element. As an example, anyone reading this text has visual schemas for the

complex squiggles that represent a word. Those schemas, stored in long-term memory, allow us

to reproduce and manipulate the squiggles that constitute writing, in working memory, without

strain. But, we are only able to do so after several years of learning.

4. Automation. Everything that is learned can, with practice, become automated. After

practice, specific categories of information can be processed with decreasing conscious effort. In

other words, processing can occur with decreasing working memory load. As an example,

schemas that permit us to read letters and words must initially be processed consciously in

working memory. With practice they can be processed with decreasing conscious effort until

eventually, reading individual letters and words becomes an unconscious activity that does not

require working memory capacity. Schneider and Shiffrin (1977) and Shiffrin and Schneider

(1977) demonstrated the contrast between conscious and automated processing. Kotovsky, Hayes

and Simon (1985) demonstrated the benefits of automated processing to problem solving skill.

High element interactivity material that has been incorporated into an automated schema after

extensive learning episodes can be easily manipulated in working memory to solve problems and

engage in other complex activities.

C. Some Instructional Effects

Cognitive load theory (Sweller, 1988; 1994; 1999; Sweller, van Merrienboer, & Paas, 1998)

deals with the interaction of information and cognitive structures and the implications of that

interaction for instruction. There are many instructional effects that follow from the theory.

Those effects that rely on aspects of visualisation will be discussed in this section.

1. The split-attention effect. Consider a student attempting to study a conventionally

structured worked example such as that of Figure 1. The diagram in isolation provides no

instruction. The associated statements such as Angle DBE = Angle ABC are unintelligible

without a diagram. Meaning can only be derived from the worked example by mentally

integrating the diagram and the statements. Mental integration requires working memory

resources because learners must search for referents. When a geometry statement refers to Angle

ABC, learners must search the diagram for Angle ABC in order to understand the statement.

Locating referents requires working memory resources that are unavailable for learning.

1504

In the above figure, find a value for Angle DBE.

Solution:

Angle ABC = 180 0 – Angle BAC – Angle BCA (Internal angles of a triangle add to 180

0

= 180 0 – 60

0 – 40

0

= 80 0

Angle DBE = Angle ABC (Vertically opposite angles are equal)

= 80 0

Figure 1: An example of a conventional, split-attention diagram and text

The working memory load can be reduced by physically integrating diagrams and statements.

Rather than placing statements below or next to the diagram as normally occurs, relevant

statements can be incorporated within the diagram so that a search for referents is eliminated (see

Figure 2). If conventionally structured worked examples are compared with physically integrated

examples, results normally demonstrate an advantage for the integrated versions resulting in the

split-attention effect. Various versions of the effect have been demonstrated using a wide variety

of instructional materials under a wide variety of conditions (Bobis, Sweller & Cooper, 1993;

Cerpa, Chandler & Sweller, 1996; Chandler & Sweller, 1992; 1996; Mayer & Anderson, 1991;

A

B D

E

600

400 C

1505

1992; Mwangi & Sweller, 1998; Sweller, Chandler, Tierney and Cooper, 1990; Tarmizi &

Sweller, 1988; Ward & Sweller, 1990).

Figure 2: An example of an integrated diagram and text

2. The modality effect. While physical integration of multiple sources of information can be

highly effective, there is an alternative that is equally effective and under some circumstances,

may be preferable. The split-attention effect relies on the visual modality with visual search

being reduced by the use of physical integration. Visual search means that the visual channel

only (the visuo-spatial sketchpad of Baddeley, 1992 and Baddeley & Hitch, 1974) is being used

and overloaded under split-attention conditions. There is considerable evidence that effective

working memory can be increased by using dual rather than a single modality (e.g. Penney,

1989). While the visual and auditory processors of working memory are not fully separate in the

sense that one does not obtain a simple additive increase in processing capacity by presenting

some material visually and some in auditory mode, there is considerable empirical evidence of a

measurable increase in working memory capacity when using both modalities (Allport, Antonis,

A

B D

E

600

400 C

180 0 - 600 - 400

= 80 0

1

800 2

1506

& Reynolds, 1972; Brooks, 1967; Frick, 1984; Levin & Divine-Hawkins, 1974). From a

theoretical perspective, capacity should increase to the extent that visual and auditory processors

can function autonomously without sharing other cognitive structures that limit capacity. Some

of the empirical evidence of an increase in working memory capacity when using both modalities

also provides evidence for a partial autonomy of the auditory and visual channels.

The possibility of increasing working memory capacity by using dual rather than a single

modality should have instructional consequences. For example, under split-attention conditions,

rather than presenting a diagram and written text that should be physically integrated, it may be

possible to present a diagram and spoken text. Because the diagram uses a visual modality while

speech uses the auditory modality, total available working memory capacity should be increased

resulting in enhanced learning.

The instructional modality effect occurs when learners, faced with two sources of information

that refer to each other and are unintelligible in isolation, learn more when presented with one

source in visual mode and the other in auditory mode rather than both in visual mode. This effect

has been demonstrated on a number of occasions (Jeung, Chandler & Sweller, 1997; Mayer &

Moreno, 1998; Moreno & Mayer, 1999; Mousavi, Low & Sweller, 1995; Tindall-Ford, Chandler

& Sweller, 1997).

3. The redundancy effect. Both the split-attention and the modality effects occur under very

specific conditions. They are only obtainable when multiple sources of information refer to each

other and are unintelligible in isolation. For example, a diagram and text will not yield either the

split-attention or the modality effects if the diagram is fully intelligible and fully provides the

information needed, with the text merely recapitulating the information contained in the diagram

in a different form. Under such circumstances, the text is redundant. The redundancy effect

occurs when additional information, rather than having a positive or neutral effect, interferes

with learning. For example, rather than integrating a diagram with redundant text or presenting

the text in auditory form, learning is enhanced by eliminating the text.

There are many different forms of redundancy. As described above, diagram/text redundancy

occurs when a self-explanatory diagram has additional text re-describing the diagram (Chandler

& Sweller, 1991). Mental activity/physical activity redundancy occurs when, for example,

learning how to use a computer application by reading a text has the added physical activity of

using the computer (Cerpa, Chandler & Sweller, 1996; Chandler & Sweller, 1996; Sweller &

Chandler, 1994). Either reading the text in a manual or surprisingly, physically using a computer,

can be redundant and interfere with learning. Summary/detailed exposition redundancy occurs

when a summary alone results in enhanced learning compared to a summary plus a full

exposition (Mayer, Bove, Bryman, Mars & Tapangco, 1996; Reder & Anderson, 1980; 1981)

Lastly, auditory/visual redundancy occurs when the same material, presented simultaneously in

written and spoken form, results in a learning decrement compared to the material presented in

written or auditory form alone (Kalyuga, Chandler, & Sweller, 1999; 2000; Mayer, Heiser &

Lonn, 2001; ).

The redundancy effect is one of the more surprising cognitive load effects with many people

finding it quite counterintuitive. Most of us feel that even if additional explanatory material is not

beneficial, at the very least it should be neutral. In fact, the addition of redundant information can

have strong, negative consequences. The effect can be understood in cognitive load theory terms.

If one form of instruction is intelligible and adequate, providing the same information in a

different form will impose an extraneous cognitive load. Working memory resources will need to

be used to process the additional material and possibly relate it to the initial information. It is

1507

likely to be only after the learner has processed the additional information that he or she will be

aware that the activity was unnecessary. At that point, the damage may have been done.

4. The element interactivity effect. The split-attention, modality and redundancy effects all

occur as a consequence of instructional procedures designed to reduce working memory load. It

might be expected that the instructional procedures would only be effective where the material

being learned imposed an intrinsically high cognitive load. If material does not impose a high

cognitive load, the additional load due to inadequate instructional procedures may not matter a

great deal because working memory capacity may not be exceeded. An extraneous cognitive load

due to inadequate instructional procedures may be irrelevant if the intrinsic cognitive load

imposed by the structure of the information is low. Since low element interactivity material has a

low intrinsic cognitive load, we can predict that cognitive load effects may disappear when

learning such material. The effects may only be obtainable using high element interactivity

material. Chandler and Sweller (1996) and Sweller and Chandler (1994) demonstrated that the

split-attention and redundancy effects could readily be demonstrated using high element

interactivity material but disappeared when low element interactivity material was used. Tindall-

Ford, Chandler and Sweller (1997) similarly found that the modality effect could only be

obtained using high element interactivity material. Marcus, Cooper and Sweller (1996) found

that diagrams for which we have schemas facilitated understanding when compared to text but

only under conditions of high element interactivity.

The finding that cognitive load effects can only be obtained using high element interactivity

material demonstrates the element interactivity effect. It consists of an interaction between the

split-attention, redundancy and modality effects and the complexity (as measured by element

interactivity) of the material being learned. While it has not been tested using other cognitive

load effects, there is every reason to suppose it could be obtained with all other effects based on

a limited working memory.

5. The imagination effect. Assume a novice who has acquired some limited schemas. To

attain relatively high levels of expertise, further learning will need to include automation of the

previously acquired schemas that normally includes continuing to study the material until desired

levels of performance have been attained. An alternative is to attempt to imagine the procedures

that have been learned. Imagining requires the learner to mentally “run through” or visualise the

procedures in working memory. For high element interactivity material, processing information

in working memory is impossible until schemas have been acquired. Once they have been

acquired and the learner has moved towards the right of the matrix of continua, imagination

techniques should be feasible and practice through imagination should assist in automation.

Continuing to study the material should be unnecessary because studying high element

interactivity material is designed to provide the guidance necessary to reduce search while

acquiring schemas. If schemas have already been acquired, there is no longer any need to

provide instructional guidance to reduce search because. Using those schemas to imagine the

procedures learned should facilitate further learning through automation in a manner that

studying the instructions may not.

Cooper, Tindall-Ford, Chandler and Sweller (2001) tested this hypothesis and found that

learners given instructions to “imagine” a set of procedures that needed to be learned performed

better than learners given conventional “study” instructions. This imagination effect was only

obtained using learners with sufficient knowledge to be able to process all of the required

information in working memory. For complete novices who were unable to process the high

element interactivity material in working memory, a reverse imagination or “study” effect was

1508

obtained with “study” instructions proving superior to “imagination” instructions. In other

words, the effect obtained depended on the levels of expertise of the learners. Higher levels of

expertise could reverse the effect obtained. The ideal form of instruction depended on the

expertise of the learners.

The instructional designs described in this section differ from many instructional designs in

that they are very closely tied to our knowledge of information structures and human cognitive

architecture. Indeed, they were generated directly from that knowledge. The designs not only

flow from our knowledge of visual working memory, they can provide additional information

concerning its characteristics.

References

Allport, D., Antonis, B., & Reynolds, P. (1972). On the division of attention: A disproof of the

single channel hypothesis. Quarterly Journal of Experimental Psychology, 24, 225-235.

Baddeley, A. (1992). Working memory. Science, 255, 556-559.

Baddeley, A. & Hitch, G. (1974). Working memory. In G. A. Bower (Ed.) Recent advances in

learning and motivation (Vol 8, pp. 47-90). New York: Academic Press.

Bartlett, F. (1932). Remembering: A study in experimental and social psychology. London:

Cambridge University Press.

Bobis, J., Sweller, J., & Cooper, M. (1993). Cognitive load effects in a primary school geometry

task. Learning and Instruction, 3, 1-21.

Brooks, L. (1967). The suppression of visualization by reading. Quarterly Journal of

Experimental Psychology, 19, 289-235.

Chase, W. G. & Simon, H. A. (1973). Perception in chess. Cognitive Psychology, 4, 55 - 81.

Cerpa, N., Chandler, P., & Sweller, J. (1996). Some conditions under which integrated computer-

based training software can facilitate learning. Journal of Educational Computing

Research., 15, 345-367.

Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction.

Cognition and Instruction, 8, 293-332.

Chandler, P., & Sweller, J. (1992). The split-attention effect as a factor in the design of

instruction, British Journal of Educational Psychology, 62, 233-246.

Chandler, P., & Sweller, J. (1996). Cognitive load while learning to use a computer program.

Applied Cognitive Psychology, 10, 151-170.

Chi, M., Glaser, R. & Rees, E. (1982). Expertise in problem solving. In R. Sternberg (Ed)

Advances in the psychology of human ntelligence (pp. 7-75). Hillsdale, NJ: Erlbaum.

Cooper, G., Tindall-Ford, S., Chandler, P. & Sweller, J. (2001). Learning by imagining. Journal

of Experimental Psychology: Applied, 7, 68-82.

De Groot, A. (1965). Thought and choice in chess. The Hague, Netherlands: Mouton. (Original

work published 1946).

Ericsson, K. A. & Kintsch, W. (1995). Long-term working memory. Psychological Review, 102,

211-245.

Frick. R. (1984). Using both an auditory and a visual short-term store to increase digit span.

Memory & Cognition, 12, 507-514.

1509

Gick, M. L. & Holyoak, K. J. (1980). Analogical problem solving. Cognitive Psychology, 12,

306-355.

Gick, M. L. & Holyoak, K. J. (1983). Schema induction and analogical transfer. Cognitive

Psychology, 15, 1-38.

Halford, G., Wilson, W. & Phillips, S. (1998). Processing capacity defined by relational

complexity: Implications for comparative, developmental, and cognitive psychology.

Behavioral and Brain Sciences, 21, 803-863.

Jeung, H., Chandler, P., & Sweller, J. (1997). The role of visual indicators in dual sensory mode

instruction, Educational Psychology, 17, 329-343.

Kalyuga, S., Chandler, P. & Sweller, J. (1999). Managing split-attention and redundancy in

multimedia instruction. Applied Cognitive Psychology, 13, 351-371.

Kalyuga, S., Chandler, P. & Sweller, J. (2000). Incorporating learner experience into the design

of multimedia instruction. Journal of Educational Psychology, 92, 126-136.

Kotovsky, K., Hayes, J. R. & Simon, H. A. (1985). Why are some problems hard? Evidence

from Tower of Hanoi. Cognitive Psychology, 17, 248 - 294.

Larkin, J., McDermott, J., Simon, D., & Simon, H. (1980). Models of competence in solving

physics problems. Cognitive Science, 4, 317-348.

Levin, J. & Divine-Hawkins, P. (1974). Visual imagery as a prose-learning process. Journal of

Reading Behavior, 6, 23-30.

Marcus, N., Cooper, M., & Sweller, J. (1996). Understanding instructions. Journal of

Educational Psychology, 88, 49-63.

Mayer, R., & Anderson, R. (1991). Animations need narrations: An experimental test of a dual-

coding hypothesis. Journal of Educational Psychology, 83, 484-490.

Mayer, R., & Anderson, R. (1992). The instructive animation: Helping students build

connections between words and pictures in multimedia learning. Journal of Educational

Psychology, 84, 444-452.

Mayer, R., Bove, W., Bryman, A., Mars, R., & Tapangco, L. (1996). When less is more:

Meaningful learning from visual and verbal summaries of science textbook lessons.

Journal of Educational Psychology, 88, 64-73.

Mayer, R., Heiser, J. & Lonn, S. (2001). Cognitive contraints on multimedia learning: When

presenting more material results in less understanding. Journal of Educational Psychology,

93, 187-198.

Mayer, R. & Moreno, R. (1998). A split-attention effect in multimedia learning: Evidence for

dual-processing systems in working memory. Journal of Educational Psychology, 90(2),

312 - 320.

Miller, G.A. (1956). The magical number seven, plus or minus two: Some limits on our capacity

for processing information. Psychological Review, 63, 81-97.

Moreno, R. & Mayer, R. (1999). Cognitive principles of multimedia learning: The role of

modality and contiguity. Journal of Educational Psychology, 91, 358-368.

Mousavi, S. Y., Low, R. & Sweller, J. (1995). Reducing cognitive load by mixing auditory and

visual presentation modes. Journal of Educational Psychology, 87, 319-334.

1510

Mwangi, W. & Sweller, J. (1998). Learning to solve compare word problems: The effect of

example format and generating self-explanations. Cognition and Instruction, 16, 173-199.

Penney, C.G. (1989). Modality effects and the structure of short term verbal memory. Memory

and Cognition, 17, 398-422.

Peterson, L. & Peterson, M. (1959). Short-term retention of individual verbal items. Journal of

Experimental Psychology, 58, 193-198.

Piaget, J. (1928). Judgement and reasoning in the child. New York: Harcourt.

Reder, L. & Anderson, J. R. (1980). A comparison of texts and their summaries: Memorial

consequences. Journal of Verbal Learning and Verbal Behavior, 19, 121-134.

Reder, L. & Anderson, J. R. (1982). Effects of spacing and embellishment on memory for main

points of a text. Memory & Cognition, 10, 97-102.

Schneider, W., & Shiffrin, R. (1977). Controlled and automatic human information processing: I.

Detection, search and attention. Psychological Review, 84, 1-66.

Shiffrin, R., & Schneider, W. (1977). Controlled and automatic human information processing:

II. Perceptual learning, automatic attending, and a general theory. Psychological Review,

84, 127-190.

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive

Science, 12, 257-285.

Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning

and Instruction, 4, 295-312.

Sweller, J. (1999). Instructional design in technical areas. Melbourne: ACER Press.

Sweller, J., & Chandler, P. (1994) Why some material is difficult to learn. Cognition &

Instruction, 12, 185-233.

Sweller, J., Chandler, P., Tierney, P. & Cooper, M. (1990). Cognitive load and selective attention

as factors in the structuring of technical material. Journal of Experimental Psychology:

General, 119, 176-192.

Sweller, J., van Merrienboer, J., & Paas, F. 1998). Cognitive architecture and instructional

design. Educational Psychology Review, 10, 251-296.

Tarmizi, R. & Sweller, J. (1988). Guidance during mathematical problem solving. Journal of

Educational Psychology, 80, 424-436.

Tindall-Ford, S., Chandler, P., & Sweller, J. (1997). When two sensory modes are better than

one. Journal of Experimental Psychology: Applied, 3, 257-287.

Ward, M. & Sweller, J. (1990). Structuring effective worked examples. Cognition and

Instruction, 7, 1-39.