needhelp

Do experts and novices direct attention differently in examining physics diagrams? A study of change detection using the flicker technique

Jason W. Morphew Department of Educational Psychology, University of Illinois at Urbana-Champaign,

Champaign, Illinois 61820, USA

Jose P. Mestre Departments of Physics and Educational Psychology, and Beckman Institute for Advanced Science

and Technology, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801, USA

Brian H. Ross Department of Psychology and Beckman Institute for Advanced Science and Technology,

University of Illinois at Urbana-Champaign, Urbana, Illinois 61820, USA

Natalie E. Strand Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign,

Urbana, Illinois 61801, USA (Received 4 April 2015; published 12 August 2015)

It is known that experts identify or perceive meaningful patterns in visual stimuli related to their domain of expertise. This study explores the speed with which experts and novices detect changes in physics diagrams. Since change detection depends on where individuals direct their attention, differences in the speed with which experts and novices detect changes to diagrams would suggest differences in attention allocation between experts and novices. We present data from an experiment using the “flicker technique,” in which both physics experts and physics novices viewed nearly identical pairs of diagrams that are representative of typical introductory physics situations. The two diagrams in each pair contain a subtle difference that either does or does not change the underlying physics depicted in the diagram. Findings indicate that experts are faster at detecting physics-relevant changes than physics-irrelevant changes; however, there is no difference in response time for novices, suggesting that expertise guides attention for experts when inspecting physics diagrams. We discuss the cognitive implications of our findings.

DOI: 10.1103/PhysRevSTPER.11.020104 PACS numbers: 01.40.Fk

I. INTRODUCTION

The study of expertise has attracted considerable attention in cognitive science for several decades. Various studies across different domains and using different paradigms have revealed important ways in which experts store and use knowledge to solve problems (see Refs. [1,2] for reviews— although some examine expert-novice differences, we focus here on experts). For example, expert physicists possess schemas consisting of principles, contexts, and conditions or constraints for applying them, as well as procedures for applying them, that can all be efficiently retrieved to analyze or solve problems; expert physicists are also adept at performing qualitative analyses of problems and categoriz- ing them according to major principles [3–5].

The kinds of paradigms used to study the nature of expertise have varied, including asking participants to “think out loud” as they attempt to solve problems [6,7], to categorize problems according to similarity of solution [3,8], or to recall problem situations from memory after very short exposures [9,10]. A recent study explored a phenomenon known as “change

blindness” to study expertise in physics [11]. Change blind- ness is a term in visual cognition used to describe a phenomenon whereby a change in a visual scene goes unnoticed. For example, Ref. [12] conducted a study in which an unsuspecting participant was approached by an experimenter holding a map asking for directions to a building on a university campus; then, as the individual was providing directions, two rude confederates holding a door (used to hide the two confederates behind it) walked between the experimenter and the participant and as the door passed the experimenter and one of the confederates changed places. In only about half of these occurrences was the unsuspectingparticipantabletodetectthattheoriginal person asking for directions had changed into a different person.

Published by the American Physical Society under the terms of the Creative Commons Attribution 3.0 License. Further distri- bution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

PHYSICAL REVIEW SPECIAL TOPICS - PHYSICS EDUCATION RESEARCH 11, 020104 (2015)

1554-9178=15=11(2)=020104(6) 020104-1 Published by the American Physical Society

In the Feil and Mestre [11] study, experts and novices were shown physics situations (diagrams and a qualitative question asking the participant to explain the situation), and then while the participant was distracted, a slightly different diagram was substituted for the original. They found that experts noticed the change only when the substituted diagram modified the physics in a significant way; novices, on the other hand, typically did not notice any type of change in the diagram, regardless of whether the change modified the underlying physics, except for those novices who displayed a deep understanding of the physics under- lying the initial situation. One conclusion from that study was that change blindness was a good tool for studying expertise since noticing a physics-relevant change was sensitively dependent on an individuals’ understanding of the underlying physics. In the study reported here, we extend the previous

change-blindness study [11] by investigating experts’ and novices’ ability to notice changes in diagrams using another experimental paradigm from visual cognition used to study change detection called the “flicker technique.” In the flicker technique [13], a participant sitting in front of a computer display is asked to detect a change in a pair of images that quickly alternates on the screen. The original image appears on the screen for a brief time (e.g., 200 ms), followed by a black screen lasting for a smaller amount of time (e.g., 100 ms) followed by the changed version of the original image for a brief time (e.g., 200 ms), followed by the black screen, followed by the original image, and so on. The black screen is used to mask the visual signal to the brain that occurs during an abrupt change in a visual scene. The effect of this paradigm is to create a flickering screen, and hence the name of the technique. The participant is told that his or her task is to find the change and to hit a key as soon as the change is found; the participant is then usually told to use a mouse to click on the screen over the part of the image that had the change to ensure that the participant indeed found the change in the image. The length of time to detect the change is the independent variable in the experiment. Besides the obvious superficial differences between the

two techniques, there are additional important differences between the change-event paradigm used in Ref. [11] and the flicker paradigm used in this study. In Ref. [11], none of the participants knew that the original diagram that they first studied was going to change before they began explaining it. Thus, participants were required first to process the diagram and accompanying qualitative question at a deep enough level to be able to generate an explanation, and it was only then that they were distracted and the changed diagram replaced the original. Immediately after this change, they were asked to provide their qualitative explanation. By structuring the task in this way, the investigators were able to explore if the encoding of the situation included the feature changed; that is, if

participants did not notice the change, then the feature could not have been that salient in the encoding. In the flicker technique used here, the participants were told at the outset of the experiment that there would be a change in each flicker task, and that their job was to detect that change as quickly as possible and to identify it. Thus, in the Feil and Mestre [11] study, the encoding of the diagram and its underlying physics was untimed and deliberate, whereas in this study speed in finding the change is of paramount importance. In other words, while the previous study investigated the effect of knowledge on memory, this study looks at the effect of physics knowledge on perception. The focus on the rate at which an anticipated change is detected allows us to investigate how experts and novices allocate attention when examining physics diagrams [14]. Research in visual cognition suggests that attention is

needed to see a change, and that those changes that are central to interpreting the meaning of a scene are detected faster compared to less crucial changes [14,15]. If expertise in physics helps direct attention to those regions in a physics diagram that display important information, then it is reasonable to expect that the speed of noticing physics- relevant changes to diagrams tracks with expertise. Indeed, two flicker studies using nonphysics topics suggest that expertise or familiarity can help detect relevant changes in scenes faster. One study [16] demonstrated that people who were social users of alcohol or cannabis detected changes faster in photographs compared to nonusers when those changes were alcohol and cannabis related. Another study [17] investigated the ability of experts and novices in American football to detect changes in photographs of a game and nonfootball related scenes (photos of traffic scenes); experts were players, coaches, or referees in an American football league in Germany while novices were students from the University of Göttingen who had little to no experience with the game. Findings revealed that the American football experts were faster at detecting changes that were meaningful to the game’s strategy compared to game-irrelevant changes (e.g., a change whereby a football appeared on the ground versus a change in the shadow cast by a player) or to nonfootball related scenes. Two hypotheses were tested in this study. First, we

hypothesized that physics experts will notice physics- relevant (consequential) changes to diagrams faster than physics-irrelevant (inconsequential) changes, whereas beginning physics students and total novices will not. Second, we hypothesized that time differences between detecting consequential changes compared to inconsequen- tial changes attenuates with diminishing expertise; that is, the time differences will be largest for experts, smallest for total novices, and in between for undergraduates having finished an introductory course. If the hypothesized differences are found, this is another

useful technique to measure expert-novice differences. Measuring differences in reaction times for noticing

MORPHEW et al. PHYS. REV. ST PHYS. EDUC. RES 11, 020104 (2015)

020104-2

changes in physics diagrams are likely due to experts directing their attention differently than novices. In other words, experts likely direct their attention preferentially to regions in diagrams that portray significant physics result- ing in faster times to notice physics-relevant changes compared to regions that contain physics-irrelevant changes. Thus, this study will explore whether knowledge of physics guides attention differentially for experts and novices when they inspect physics diagrams having physics-relevant changes or physics-irrelevant changes. To date, no such study has been conducted in an academic subject like physics which requires highly specialized abstract knowledge.

II. EXPERIMENT

A. Participants

To test the effect that physics knowledge has on response time three groups of participants were recruited: education students, physics novices, and physics experts. The edu- cation participants in the study consisted of 28 undergradu- ate students enrolled in an Educational Psychology course at the University of Illinois. A class-wide email to students was used to recruit the education participants, who received course credit for their participation. Of these students seven had recent college physics experience and were not included in the data analysis. In addition, one student was excluded due to failing to follow the correct procedure resulting in inflated response times. The physics novices in the study consisted of 20 undergraduate students enrolled in an introductory calculus-based mechanics course at the University of Illinois. A class-wide email to students was used to recruit the physics novice participants, who were paid for their participation. The physics experts consisted of 19 graduate student TAs who taught discussion sections in the introductory calculus-based mechanics course. These TA experts were recruited with a similar email solicitation and were paid for their participation in the study.

B. Materials

Our experiment utilized the flicker paradigm, in which one physics diagram (original) and a slightly changed version (changed) are alternated on the screen with a blank, black screen as a separator between each image. Two different types of diagram pairs are used. In one type, which will be referred to as physics-irrelevant change pairs, the changed diagram has an inconsequential change compared to the original in terms of the physics. In the other type, which will be referred to as physics-relevant pairs, there is a consequential change in the physics of the situation depicted in the changed diagram compared to the original. We measured participants’ response time to detect the change in each diagram pair. Thirty pairs of diagrams in the style used in typical physics exercises were created for this study [18]. The two diagrams in each pair contain a subtle

difference. Fifteen of the diagram pairs contained physics- relevant changes (R1–R15), and the remaining fifteen pairs contained physics-irrelevant changes (I1–I15).

C. Procedure

Participants were shown the diagram pairs on a computer screen. The diagram pairs were alternated on a computer screen with a black screen inserted between each diagram slide; the diagrams were displayed for 200 ms, while the black screen appeared for 100 ms. Participants were given the instructions that their task was to identify the change between the two diagrams in each pair as it flickered on the screen. Participants pressed the space bar as soon as they noticed the change between the two diagrams, which stopped the flickering. They then used the cursor to select the region on the diagram that contained the change. After this second click, the program automatically advanced to the next diagram pair. Software recorded the response time (time to identify the change) and the location of the second click (identification of changing region). Every participant saw all thirty items in one of four “scripts” with the order of the presentation of the diagram pairs randomized differ- ently in each script. That is, the diagram pairs with physics- relevant and physics-irrelevant changes were intermixed. Participants were randomly assigned to complete one of the four scripts. The experiment was administered to the novice group after they had the requisite physics in the course related to the diagrams in the experiment.

D. Analysis and Results

The coordinates of the second click were used to exclude data when participants incorrectly identified the changing region. The change in the diagrams was not noticed, as indicated by the participants clicking on a part of the diagrams that did not change, on 71 of the 1770 trials included in the analysis (4%). Only response time data from correct responses were included in the analyses since error responses may be very different from correct responses. Upon inspection, an approximately linear relationship exists between the standard deviation of the reaction times and the means (R2 ¼ 0.84). In addition, the distribution of response times was not normally distributed (skew ¼ 3.30, kurtosis ¼ 16.03). To address this we transformed the data by taking the natural logarithm of each response time. This transformation is common practice in psychology research and is used to yield homogeneous variances and to better approximate normal populations when the raw data exhibits non-normal properties [19]. This transformation reduced the linear relationship between the means and standard deviations (R2 ¼ .26), and the distribution of the ln(response time) was more normally distributed (skew ¼ .78, kurtosis ¼ .30). Because it was expected that participants’ response time

would improve as they advanced through the experiment, a linear regression was conducted with response time as the

DO EXPERTS AND NOVICES DIRECT … PHYS. REV. ST PHYS. EDUC. RES 11, 020104 (2015)

020104-3

criterion variable and question order as the predictor variable. As expected, the average response time decreased as participants progressed through the study [tð28Þ ¼ −3.11, p < 0.01, beta ¼ −0.012]. The order accounted for 25.6% of the variance in ln(response time). In order to analyze the effect of physics relevance without the poten- tially confounding factor of the order effect, the influence of the presentation order was removed from the individual scores as described by Keppel and Wickens (p. 390) [20]. This new distribution had a relatively normal distribution (skew ¼ 0.75, kurtosis ¼ 0.24). Table I shows the average response time and adjusted ln(response time) for the correct responses of both novices and experts. In order to examine the influence of expertise and

physics-relevant changes (Physics) on response time, a 3 × 2 (Expertise x Physics) mixed ANOVA with physics relevance as the repeated measure was conducted. Expertise was significant [Fð2; 56Þ ¼ 8.65, p < 0.001], but physics relevance was not significant [Fð1; 56Þ ¼ 3.73, p ¼ 0.06]. The interaction between physics relevance and expertise was significant [Fð2; 56Þ ¼ 5.36, p < 0.01], indicating that a different pattern of identifying physics- relevant changes compared to physics-irrelevant changes exists between the three groups. This interaction can be seen in Fig. 1. Figure 2 shows the same pattern with the raw response times indicating that the effect is not a function of the log transformation nor the correction for the presentation order. To investigate the effect that expertise has on response

time in more detail, dependent samples t tests were conducted for each group. The results of these tests indicated that there was a significant difference in identi- fying physics-relevant changes compared to physics- irrelevant changes for physics experts [tð18Þ ¼ 3.75, p ¼ 0.002], but not for physics novices [tð19Þ ¼ 1.35, p ¼ 0.19], or education students [tð19Þ ¼ −1.42, p ¼ 0.17]. This indicates that experts noticed physics-relevant changes faster than physics-irrelevant changes, whereas physics

novices and education students did not. Education students, in fact, displayed the opposite pattern, noticing physics- irrelevant changes faster than physics-relevant changes, though not significantly. For physics experts this corre- sponds to a difference of 353 ms when controlling for the order in which the diagrams were presented, which represents a large effect (d ¼ 0.95). Physics novices noticed physics-relevant changes only slightly faster than physics-irrelevant changes, corresponding to a difference of 175 ms when controlling for the presentation order. For education students this corresponds to a difference of −208 ms when controlling for the presentation order in which the diagrams were presented.1

TABLE I. Means and SD of response time and adjusted Ln(response time).

N Mean SD

Response time (in ms) Experts 19 Physics relevant 2467 776 Physics irrelevant 2947 1026

Novice 20 Physics relevant 2601 737 Physics irrelevant 2859 956

Education 20 Physics relevant 3733 952 Physics irrelevant 3391 986

Adjusted Ln(response time) Experts 19 Physics relevant 7.62 0.24 Physics irrelevant 7.78 0.28

Novice 20 Physics relevant 7.65 0.22 Physics irrelevant 7.73 0.25

Education 20 Physics relevant 8.03 0.35 Physics irrelevant 7.96 0.32

1An examination of the average response time by question revealed that two questions (R1 and I14) had response times much higher than the other questions. The difference between the average response times for these questions were more than 3 s (3.5 and 4.2 s, respectively) slower than the overall mean response time and more than 1.5 s slower than that for the next slowest question (R15). In addition, 17 of the 67 participants were unable to correctly identify the change between the I14 diagrams and 9 of the 67 participants were unable to correctly identify the change between the R1 diagrams. This represents almost 40% of the errors in the study. To identify any confounding effect that these questions had on the analysis, the analysis was also conducted with these questions removed. The same findings are found if these diagram pairs are removed from the analysis: 3 × 2 ANOVA: Expertise: Fð2; 56Þ ¼ 8.79, p < 0.001, Physics relevance: Fð1; 56Þ ¼ 3.07, p ¼ 0.09, Interaction: Fð2; 56Þ ¼ 6.36, p ¼ 0.003. t tests: Experts: [tð18Þ ¼ 3.98, p < 0.001, d ¼ 0.89], Beginning Physics: [tð19Þ ¼ 1.44, p ¼ 0.17], Novi- ces: [tð19Þ ¼ −1.77, p ¼ 0.09]. Experts noticed physics-relevant changes (M ¼ 7.58; SD ¼ 0.24) faster than physics-irrelevant changes (M ¼ 7.73;SD ¼ 0.29) corresponding to a difference of 321 ms. Novices noticed physics-relevant changes (M ¼ 7.61; SD ¼ 0.21) faster than physics-irrelevant changes (M ¼ 7.69; SD ¼ 0.24) corresponding to a difference of 168 ms. Education students noticed physics-irrelevant changes (M ¼ 7.91; SD ¼ 0.32) faster than physics-relevant changes (M ¼ 7.99; SD ¼ 0.35) corresponding to a difference of −243 ms.

MORPHEW et al. PHYS. REV. ST PHYS. EDUC. RES 11, 020104 (2015)

020104-4

III. DISCUSSION

This study explored two related hypotheses, namely that experts’ knowledge guides their attention so that they will detect consequential (physics-relevant) changes to physics diagrams faster than inconsequential (physics-irrelevant) changes, and that this advantage diminishes with diminish- ing expertise. To do so we employed the flicker technique to measure speed in detecting two different types of changes to physics diagrams, namely, those that were consequential to the underlying physics and those that were not. Our findings support the view that high knowledge

individuals do detect consequential changes to physics diagrams faster than inconsequential changes, suggesting that their domain knowledge helps guide their attention to relevant portions of physics diagrams. Only the physics experts showed this advantage. This finding suggests that expectations about important features of the diagram seem to be influenced by content expertise and direct the individual’s attention when viewing diagrams depicting a physics situation. Other studies have also demonstrated that

visual attention is allocated differently in physics among high knowledge and low knowledge individuals [21–23]. Further, our findings support the hypothesis that there is

a progression in attention allocation as expertise develops (see the pattern of changing slopes in Figs. 1 and 2). The education students in our study who had not had any recent physics experience, including no college physics experi- ence, not only did not show a speed advantage in detecting consequential changes in the diagrams, but their pattern was opposite of that of the experts’—they noticed incon- sequential physics changes to the diagrams faster than consequential changes, although this difference was not significant. The physics novice’s pattern was closer to that of the expert group than the education group as predicted, although their faster detection of physics-consequential changes was also not significant. The lack of any significant advantage of physics-relevant

information for education students suggests that these students may lack a guiding schema of physics knowledge to direct their attention during the search for changes. This has implications for introductory physics instruction: It may be that for those students who do not do well in physics courses, their difficulty in learning physics is compounded by an inability to extract the appropriate physics from pictorial and other representations used in teaching physics. This implication is also supported by the Feil and Mestre [11] study where ability to answer correctly a question administered to assess understanding of a physics situation was highly correlated with the ability to notice a con- sequential physics change to a diagram in that change- blindness study. Future investigations of how the ability to extract important information from physics representations relates to other skills (e.g., problem solving, problem categorization) could prove fruitful and might suggest ways of improving introductory physics instruction. In summary, the present study applies the flicker

paradigm from visual cognition to the study of expertise in physics. Our findings suggest that the flicker technique could be a potentially useful tool for studying and meas- uring expertise. As we acquire knowledge in a complex domain such as physics, we are better able to direct our attention judiciously and “see” more rapidly relevant features in pictorial and symbolic depictions of situations. This research extends the physics expert-novice change- blindness study [11], which demonstrated that physics knowledge also guides a person’s ability to notice con- sequential changes to physics situations.

ACKNOWLEDGMENTS

The authors are grateful to Adam Feil and Claire McKay Bowen for programming the experiment software and helping develop some of the diagrams.

FIG. 1. Mean adjusted Ln (Response Time) for experts, novices, and education participants.

FIG. 2. Mean Response Time for experts, novices, and educa- tion participants.

DO EXPERTS AND NOVICES DIRECT … PHYS. REV. ST PHYS. EDUC. RES 11, 020104 (2015)

020104-5

[1] J. D. Bransford, A. L. Brown, and R. R. Cocking, How People Learn: Brain, Mind, Experience, and School (National Academy Press, Washington, D.C., 1999).

[2] E. Etkina, J. Mestre, and A. O’Donnell, The impact of the cognitive revolution on science learning and teaching, in The Cognitive Revolution in Educational Psychology, edited by J. M. Royer (Information Age Publishing, Greenwich, CT, 2005), pp. 119–164.

[3] M. T. H. Chi, P. J. Feltovich, and R. Glaser, Categorization and representation of physics problems by experts and novices, Cogn. Sci. 5, 121 (1981).

[4] P. T. Hardiman, R. Dufresne, and J. P. Mestre, The relation between problemcategorizationand problem solvingamong experts and novices, Mem. Cogn. 17, 627 (1989).

[5] J. H. Larkin, J. McDermott, D. P. Simon, and H. A. Simon, Expert and novice performance in solving physics problems, Science 208, 1335 (1980).

[6] J. J. Clement, Solving problems with formulas: Some limitations, Eng. Educ. 72, 158 (1981).

[7] J. Clement, Non-formal reasoning in experts and in science students: The use of analogies, extreme cases, and physical intuition in Informal Reasoning and Education, edited by J. Voss, D. Perkins, and J. Siegel (Lawrence Erlbaum Associates, Hillsdale, NJ, 1991), pp. 344–362.

[8] A. H. Schoenfeld and D. J. Herrmann, Problem perception and knowledge structure in expert and novice mathematical problem solvers, J. Exp. Psychol. Learn. Mem. Cogn. 8, 484 (1982).

[9] D. E. Egan and B. J. Schwartz, Chunking in recall of symbolic drawings, Mem. Cogn. 7, 149 (1979).

[10] W. G. Chase and H. A. Simon, Perception in chess, Cogn. Psychol. 4, 55 (1973).

[11] A. Feil and J. Mestre, Change blindness as a means of studying expertise in physics, J. Learn. Sci. 19, 480 (2010).

[12] D. J. Simons and D. T. Levin, Failure to detect changes to people during a real-world interaction, Psychon. Bull. Rev. 5, 644 (1998).

[13] R. A. Rensink, J. K. O’Regan, and J. J. Clark, To see or not to see: The need for attention to perceive changes in scenes, Psychol. Sci. 8, 368 (1997).

[14] D. Simons and M. Ambinder, Change blindness: Theory and consequences, Curr. Dir. Psychol. Sci. 14, 44 (2005).

[15] D. J. Simons and R. A. Rensink, Change blindness: Past, present, and future, Trends Cogn. Sci. 9, 16 (2005).

[16] B. T. Jones, B. C. Jones, H. Smith, and N. Copley, A flicker paradigm for inducing change blindness reveals alcohol and cannabis information processing biases in social users, Addiction 98, 235 (2003).

[17] S. Werner and B. Thies, Is “change blindness” attenuated by domain-specific expertise? An expert-novices compari- son of change detection in football images, Vis. Cognit. 7, 163 (2000).

[18] See Supplemental Material at http://link.aps.org/ supplemental/10.1103/PhysRevSTPER.11.020104 for the images used in the study.

[19] D. C. Howell, Statistical Methods for Psychology (Wadsworth Group Duxbury imprint, Pacific Grove, CA, 2002).

[20] G. Keppel and T. D. Wickens, Design and Analysis: A Researcher’s Handbook, 4th Ed. (Pearson Education, Upper Saddle River, NJ, 2004).

[21] A. M. Madsen, A. M. Larson, L. C. Loschky, and N. S. Rebello, Differences in visual attention between those who correctly and incorrectly answer physics problems, Phys. Rev. ST Phys. Educ. Res. 8, 010122 (2012).

[22] A. Rouinfar, E. Agra, A. M. Larson, N. S. Rebello, and L. C. Loschky, Linking attentional processes and concep- tual problem solving: Visual cues facilitate the automaticity of extracting relevant information from diagrams, Front. Psychol. 5, 1094 (2014).

[23] A. Carmichael, A. Larson, E. Gire, L. Loschky, and N. S. Rebello, How does visual attention differ between experts and novices on physics problems?, AIP Conf. Proc. 1289, 93 (2010).

MORPHEW et al. PHYS. REV. ST PHYS. EDUC. RES 11, 020104 (2015)

020104-6