You are asked to write an essay to discuss the topic of data mining. Please address each of the following:1.Based on the assigned readings(3 research papers)and textbook, what is data mining? Are the definitions consistent or contradictory?2.Describe the

Computers & Education 113 (2017) 226e242

Contents lists available at ScienceDirect

Computers & Education

journal homepage: www.elsevier.com/locate/compedu

Data mining in educational technology classroom research: Can it make a contribution?

Charoula Angeli a, *, Sarah K. Howard b, Jun Ma b, Jie Yang b, Paul A. Kirschner c, d

a University of Cyprus, Cyprus b University of Wollongong, Australia c Open University of the Netherlands, The Netherlands d University of Oulu, Finland

a r t i c l e i n f o

Article history: Received 18 June 2016 Received in revised form 15 March 2017 Accepted 29 May 2017 Available online 30 May 2017

Keywords: Educational data mining Educational technology research Association rules mining Fuzzy representations

* Corresponding author. 11-13 Dramas street, P.O E-mail address: [email protected] (C. Angeli).

http://dx.doi.org/10.1016/j.compedu.2017.05.021 0360-1315/© 2017 Elsevier Ltd. All rights reserved.

a b s t r a c t

The paper addresses and explains some of the key questions about the use of data mining in educational technology classroom research. Two examples of use of data mining tech- niques, namely, association rules mining and fuzzy representations are presented, from a study conducted in Europe and another in Australia. Both of these studies examine student learning, behaviors, and experiences within computer-supported classroom activities. In the first study, the technique of association rules mining was used to understand better how learners with different cognitive types interacted with a simulation to solve a prob- lem. Association rules mining was found to be a useful method for obtaining reliable data about learners' use of the simulation and their performance with it. The study illustrates how data mining can be used to advance educational software evaluation practices in the field of educational technology. In the second study, the technique of fuzzy representations was employed to inductively explore questionnaire data. The study provides a good example of how educational technologists can use data mining for guiding and monitoring school-based technology integration efforts. Based on the outcomes, the implications of the study are discussed in terms of the need to develop educational data mining tools that can display results, information, explanations, comments, and recommendations in meaningful ways to non-expert users in data mining. Lastly, issues related to data privacy are addressed.

© 2017 Elsevier Ltd. All rights reserved.

1. Introduction

Data mining has long been used in marketing, advertising, health, engineering, and information systems. At its core, data mining is an inductive, analytic, and exploratory approach, which is concerned with knowledge discovery through identi- fication of patterns within large sets of data. In the last 10 years, the field of Educational Data Mining (EDM) has emerged as a distinct area of research concerned with using data mining techniques to answer educational questions, such as, “What are the difficulties students encounter during a learning activity?”, “What sequences of computer interactions lead to successful

. Box 20537, Department of Education, University of Cyprus, CY-1678, Nicosia, Cyprus.

C. Angeli et al. / Computers & Education 113 (2017) 226e242 227

problem-solving performance?”, and “What sequences of actions characterize high performers and low performers in problem-solving activity?” EDM can also provide new insights into “wicked” educational problems, such as, “What are the differences in the ways students experience learning,” and “How can learning designs account for variations in students' learning experiences?”

In particular, EDM is concerned with developing methods for analyzing data from an educational system in order to detect patterns in large datasets that would otherwise be very difficult or even impossible to analyze due to the vast volume of data within which they exist (Romero & Ventura, 2013). Consequently, results from data mining can be used for deciding about how to improve the teaching and learning process as well as how to design or redesign a learning environment (Ingram,1999; Romero & Ventura, 2007). Data mining techniques have been mostly used within the context of web-based or e-learning education in order to: (a) suggest activities, resources, learning paths, and tasks for improving learners' performance and adapting learning experience (Tang & McCalla, 2005); (b) provide feedback to teachers and instructional designers in regards to learners' difficulties with the content and structure of a course, so that revisions can be made to facilitate students' learning (Merceron & Yacef, 2010; Zaiane & Luo, 2001); (c) predict learners' performance (Ahmed & Elaraby, 2014); and (d) inform administrators about the effectiveness of instructional programs, so that better planning and allocation of human and ma- terial resources can be achieved (Romero & Ventura, 2007).

Based on a number of reviews and meta-analyses published (Baker & Yacef, 2009; Mohamad & Tasir, 2013; Romero & Ventura, 2007, 2010), the most popular data mining techniques include: (a) clustering (Amershi & Conati, 2009; Beal, Qu, & Lee, 2006; He, 2013; Perera, Kay, Koprinska, Yacef, & Zaiane, 2009); (b) regression (Buja & Lee, 2001); (c) association rules mining (Lin, Alvarez, & Ruiz, 2002); and (d) sequential pattern mining (Perera et al., 2009). In clustering, the goal is to split the data into clusters, such that, there is homogeneity within clusters and heterogeneity between clusters (Baker & Siemens, 2014). In educational research, clustering procedures have been used to find patterns of effective problem- solving strategies in exploratory computer-based learning environments (Amershi & Conati, 2009; Angeli & Valanides, 2004; Beal et al., 2006; He, 2013). In regression, the goal is to develop a model that can infer or predict something about a dataset. In a regression analysis, a variable is identified as the predicted variable and a set of other variables as the predictors (similar to dependent and independent variables in traditional statistical analyses) (Baker & Siemens, 2014). In association rules mining, the goal is to extract rules of the form if-then, such that if some set of variable values is found, another variable will generally have a specific value (Baker & Siemens, 2014). In sequential pattern mining, the aim is to find temporal as- sociations between events to determine what path of student behaviors leads to a successful group project (Perera et al., 2009).

Currently, most work on data mining has at its base a computer science perspective rather than an educational perspective. Within the educational domain, data mining techniques have been mostly used in e-learning or web-based research, because of the ease of accessing student log data and performing automatic analyses of data. There is, however, also a need to investigate the uses of EDM in real classrooms in order to understand better students' interactions with technology as well as the complexities entailed in investigating how students with diverse needs and cognitive characteristics perform with technology in these settings. The issue then becomes whether EDM can make a contribution to educational technology classroom research in terms of providing tools and techniques that educational technology researchers can easily grasp and apply to their own research in order to answer questions that cannot be easily answered by traditional statistical techniques.

In view of that, in this paper, the authors, within the context of two different studies, describe their efforts in using data mining procedures in educational technology classroom research, and, identify difficulties in applying data mining tech- niques and tools in this research context. The first study was carried out in a European country and sought to investigate how field-dependent and field-independent learners solved a problem using a stand-alone simulation tool. For the purposes of the first study, the authors used a sequence, association, and link analysis for capturing and analyzing learners' interactions with the simulation. The analysis provided a detailed and analytic description of the differences in field-dependent and field- independent learners' problem-solving processes, providing at the same time clear understanding of field-dependent learners' difficulties to take full advantage of the affordances of the simulation in order to maximize learning benefits. The study contributes to educational technology research by presenting evidence about the effectiveness of EDM as an approach for extracting useful process-related knowledge and actual student learning data that can be used for improving the learning design of educational software and systems (Abdous, He, & Yen, 2012; Romero & Ventura, 2013). In turn, EDM can replace traditional approaches to software evaluation, which mostly depend on surveys of students' perceptions of the system (Bayram & Nous, 2004), by providing detailed data about what software features are or are not so successful with learners that instructional designers can use in order to decide how to go about improving their learning designs. Consequently, data mining techniques can become extremely useful in terms of providing ideas for implementing personalized learning to meet students' individual needs (Chen, 2008; Lin, Yeh, Hung, & Chang, 2013). Some preliminary work in this area has been reported by Hsu (2008) who applied association rules algorithms in the development of a personalized English Learning Recom- mendation System, as well as by Chen and Duh (2008) who used a fuzzy technique to determine the difficulty parameters of courseware and decide thereafter the content of courseware for personalized recommendation services.

The second study addresses the use of educational technology in Australian secondary schools. The research considers variations in student experiences in an integrated learning environment and how this may relate to learning. The aim of the study was to understand better the range of students' experiences with technology and accordingly to inform teachers' in- tegrated learning designs. Due to the complexity of the learning environment and the large number of key factors affecting

C. Angeli et al. / Computers & Education 113 (2017) 226e242228

students' experiences in the classroom, association rules mining and fuzzy representations were used to explore relations among students' questionnaire responses and national assessment outcomes. The results showed significantly different patterns of key technology integration factors related to literacy and numeracy outcomes. The findings provide guidance for learning design in relation to how teachers may provide different experiences in technology-integrated learning to support all learners. The study contributes to educational technology research by providing evidence of EDM as a useful approach for (a) understanding school-based technology-related change initiatives, (b) determining where to focus classroom resources and informing choices of technology tools, and (c) developing a deeper understanding of student technology-related experiences (Abdous et al., 2012).

In the general discussion section of the paper, the authors discuss the contribution of data mining in educational tech- nology classroom research, within the context of the two studies, while at the same time they also consider obstacles related to the intrinsic difficulty associated with learning how to use data mining tools and apply EDM techniques to educational data. Research directions aiming at making data mining tools and techniques more accessible to educational researchers are discussed. Lastly, data-privacy issues are also addressed.

2. Study 1

2.1. Theoretical framework and research questions

In the first study, the authors used a data mining technique called sequence, association, and link analysis to understand and best describe how the cognitive style of field dependence-independence (FD-I) affected undergraduate students' ability to solve a problem using a glass-box simulation (Clariana & Strobel, 2008; Landriscina, 2013). According to Landriscina (2013), simulations are distinguished into black-box or model-opaque simulations, and, glass-box or model-transparent simulations. In black-box or model-opaque simulations, learners explore a system's behavior, but the underlying conceptual and computational model of the simulation remains hidden. Thus, learners can only observe the results of the causal relationships between the variables (Landriscina, 2013). Glass-box or model-transparent simulations, on the other hand, make the structure of the model underlying the simulation visible to the learners in the form of a diagram with nodes and connecting links between them (Landriscina, 2013).

FD-I is a cognitive style directly related to how humans perceive, organize, and process information (Morgan, 1997; Price, 2004; Witkin, Moore, Goodenough, & Cox, 1977). It is distinguished from learning styles, in that learning styles are subjective accounts of individuals' instructional preferences across specific domains and tasks (Messick, 1987). FD-I was defined by Witkin et al. (1977) as “the extent to which a person perceives part of a field as discreet from the surrounding field as a whole, rather than embedded in the field; or the extent to which the organization of the prevailing field determines perception of its components; or, to put it in everyday terminology, the extent to which the person perceives analytically” (pp. 6e7). Witkin et al. (1977) conceptualized FD-I as a construct with two discrete modes of perception, such that, at the one extreme end perception is dominated by the prevailing field and is designated as field dependent (FD), and at the other extreme end, perception is more or less separate from the surrounding field and is designated as field independent (FI).

Contemporary research studies have examined the effects of learning with glass-box (model-transparent) simulations on FI and FD learners' performance, and, found that FI learners outperformed FD learners during problem solving with this type of simulation (Angeli, Valanides, & Kirschner, 2009; Burnett, 2010; Dragon, 2009). However, these investigations have pri- marily focused on identifying quantitative differences in performance between FD and FI learners without providing detailed information about FD and FI learners' interactions with the simulation, as well as related difficulties that learners encountered during the problem-solving process with the simulation. While quantitative investigations are in general useful, they do not provide enough insight about how to help those learners, such as for example FD learners, who usually encounter problems during problem solving and need to be supported by the teacher so they can also have successful learning experiences with technology.

Therefore, given the limitations of the existing body of research on FD and FI learners' problem solving with simulations, the present study applied sequence, association, and link analyses to assess and compare FD and FI learners' interactions with a glass-box simulation in order to solve a problem about immigration policy. The research purpose of the study was to identify sequences of interactions with the simulation that were associated with successful performance and whether they differed between FD and FI learners. Analytically, the research questions were stated as follows:

1. What sequences of interactions with the simulation lead to successful problem-solving performance? 2. How do the sequences of interactions with the simulation differ between FD and FI learners? 3. What are the learning difficulties that FD learners encounter during the problem-solving process with the simulation?

Evidently, traditional statistical techniques cannot provide the means for answering these questions, and, thus, the issue becomes whether data mining, and in particular the sequence, association, and link analysis that was employed here, can answer these questions in informative and useful ways for the educational technology researchers.

C. Angeli et al. / Computers & Education 113 (2017) 226e242 229

2.2. Method

2.2.1. Participants One hundred and fifteen freshmen from a teacher education department were recruited to participate in the study.

Students were initially screened based on their scores on the Hidden Figures Test (HFT; French, Ekstrom, & Price, 1963). The HFT was used for identifying students' FD-I. The highest possible score on the HFT is 32 and the lowest zero. In accordance with other research studies (Angeli & Valanides, 2004; Chen & Macredie, 2002; Daniels & Moore, 2000; Khine,1996), the cut- off points for this study were set to two levels of FD-I, namely FD and FI. Students who scored 18 or lower on the HFT were classified as FD learners, while students who scored 19 or higher were classified as FI. Of the 115 students, 45 of them were found to be FI learners, and the remaining 70 FD. Of the 115 participants, 94 (82%) were females, and 21 (18%) males. The average age of the participants was 17.86 years (SD ¼ 0.45). All students had basic computing skills, but no prior experience with problem solving with simulations.

2.2.2. The simulation task All research participants were asked to interact with a glass-box simulation that was specifically developed for the pur-

poses of this study, in order to solve a problem about immigration policy. The researchers explained to the participants that nowadays a lot of people move from one country to another in search of a better life for their children and themselves. Students were given a scenario about people from country A who wanted to move to country B due to a high unemployment rate in country A. The students had to interact with the simulation in order to test hypotheses, and, decide about whether and under what conditions country B could accept immigrants from country A.

The underlying model of the glass-box simulation is depicted in Fig. 1. The model shows how an increase in the number of births in country A will cause an increase in the population of country A. This, in turn, and provided that not enough employment opportunities are created in the interim to cover the new demands for employment in country A, will eventually lead to an increase in the unemployment rate of country A. In contrast, an increase in the number of deaths in country A will eventually cause a decrease in the unemployment rate of country A. In the case of an increase in the unemployment rate of country A, people from country A will eventually seek employment in another country - country B. A movement of people from country A to country B will eventually cause an increase in the unemployment rate of country B, if country B does not create in the meantime enough employment opportunities to cover the increased demand for employment. The model shows how an increase in the number of businesses in country B will cause a decrease in country's B unemployment rate, while a

Fig. 1. The underlying model about immigration policy of the glass-box simulation.

C. Angeli et al. / Computers & Education 113 (2017) 226e242230

movement of businesses from country B to A will cause a decrease in country's A unemployment rate, but in the long run a possible increase in country's B unemployment rate. In total, the tool simulated the phenomenon of immigration using five independent variables, namely number of births in country A, number of births in country B, number of deaths in country A, number of deaths in country B, and movement of businesses from country B to country A. The students had to change the values of the independent variables one at a time to observe the effects on the dependent variables in order to decide, and, propose in writing if and under what conditions country B could possibly accept immigrants from country A.

When the learners run the model, the simulation opens a meter for each dependent and independent variable. As shown in Fig. 2, each meter displays the initial value of each variable and the range of values it can take. At each run time, the learner can change the value of one independent variable at a time and observe how the meters of the affected dependent variables change.

2.2.3. Research instruments

2.2.3.1. Hidden figures test. The Hidden Figures Test (HFT) was administered to determine research participants' field type (French et al., 1963). The test consists of two parts, and each part contains 16 questions. The time allotted for answering each part is 12 min. The scores on the HFT range from zero to 32. Basically, each question on the HFT presents five simple geometric figures and a more complex one. Students are instructed to discover which one of the five simpler figures is embedded in the more complex one. According to Rittschof (2010), the HFT is the most reliable and widely used test for measuring FD-I. It is also highly correlated with the Group Embedded Figures Test (r ¼ 0.67 - 0.88), another popular test for determining FD-I (Witkin, Oltman, Raskin, & Karp, 1971).

2.2.3.2. Assessment rubric. A rubric that was inductively constructed was used to assess the quality of learners' written an- swers to the immigration problem. The scoring rubric assessed three levels of quality ranging from 1 (poor quality) to 3 (high quality). The specific criteria for each level are shown in Table 1. Two independent raters evaluated students' answers to the immigration problem, and Cohen's kappa was used to measure interrater reliability. A satisfactory interrater reliability of k ¼ 0.87 was computed, while noted discrepancies between the two raters were resolved after discussion.

Fig. 2. Simulation run.

Table 1 Rubric for assessing the quality of learners' answers.

3 - High a. The learner's answer is based on a correct interpretation of the simulated outcomes. b. The learner's answer takes into consideration pros and cons of different possible answers. c. The learner's answer takes into consideration possible long-term effects.

2 - Medium a. The learner's answer is based on a correct interpretation of the simulated outcomes. b. The learner's answer takes into consideration pros and cons of different possible answers. c. The learner's answer does not take into consideration possible long-term effects.

1 - Poor a. The learner's answer is not based on a correct interpretation of the simulated outcomes. b. The learner's answer does not take into consideration pros and cons of different possible answers. c. The learner's answer does not take into consideration possible long-term effects.

C. Angeli et al. / Computers & Education 113 (2017) 226e242 231

2.2.4. Research procedures Research data were collected in three different sessions. During the first 25-min research session, the researchers

administered the HFT in order to determine learners' field type. In a follow-up 60-min session, the researchers demonstrated a glass-box simulation, different than the one that was used for collecting research data for this study, and, showed how to use it in order to solve a problem. The students interacted with the simulation individually in order to explore various problem- solving scenarios and learn how to control variables. The researchers explicitly explained the differences between dependent and independent variables, and, demonstrated how changes in the independent variables affected the dependent variables. During the last 60-min session, the researchers collected the data that were used for the analyses of this study. During the session, the participants interacted with the glass-box simulation, observed, organized, and interpreted the simulated out- comes of the system for the purpose of solving the problem about immigration policy.

2.2.5. Data structure and analysis Students' interactions with the simulation were captured into video files with River Past Screen Recorder, a screen

capturing software. Each video file had an average duration of 50 min and a size of about 4GB. A scheme was used for coding learners' interactions in a log file, which took the form of a table with three columns including Student_ID, Time, and Action. Student_ID referred to students' research ID number, Time denoted the start/end time of an event, and Action described what the interaction entailed in terms of a sequence of computer actions. The total number of entries in this table/log file, which constituted the data for the data mining analysis, was 4570 entries. Regarding the Action field in the data table, the simulation afforded five computer actions that the students could employ in order to explore the relationships between all dependent and independent variables, as depicted in Fig. 1, in order to decide if and under what conditions country B could accept immigrants from country A. The first action was about displaying all variables and the relationships amongst them, as represented in the model shown in Fig. 1. The second was about using the test tools in order to run the simulation. The third was about opening the meter of each variable to change the values of the independent variables while observing at the same time the effects on the dependent variables. The fourth was about using the play button for running the simulation, and, lastly, the stop button for stopping the simulation. Thus, the following computer interactions were coded: B for viewing all variables and the relationships between them; T for accessing the test tools needed for a simulation test; M for opening the meter of each variable; P for running/playing the simulation; and S for terminating/stopping the simulation. Additionally, the codes IV1, IV2, IV3, IV4, and IV5 were used for denoting the five independent variables.

A sequence, association, and link analysis (Nisbet, Elder, & Miner, 2009) was used in order to identify unique differences between the FD and FI learners. Specifically, the sequence, association, and link analysis was used for extracting association rules in order to determine which simulation actions were closely associated together. The technique was also used for extracting an immediate subsequent action given a previous one, and for mining patterns of interaction between individuals of different field types and computer actions. In association rules mining, relationships and patterns are expressed in the form of an association rule:

If A then (likely) C Each rule includes an antecedent (A) and a consequent (C). This can be understood as “IF A then C.” Rules may contain single

or multiple antecedents and consequents, such as “IF A and B, then C.” The importance of a rule is determined through critical measurements: support, confidence, and lift (Tan, Kumar, & Srivastava, 2004). The extent to which the antecedent(s) and consequent(s) occur simultaneously in the dataset is indicated through support. The extent to which the consequent(s) oc- cur(s) given the antecedent(s) is indicated through confidence. The correlation between the antecedent(s) and consequent(s) is indicated through lift. For the two sequence, association, and link analyses that were performed, the minimum support was set to 0.55 and the confidence level to 0.95.

The authors employed Statistica Data Miner for conducting the sequence, association, and link analyses. While we experimented with a number of other data mining tools, we ended up using Statistica, because compared to other tools we found it easier to use in preparing the data for mining, as well as easier to integrate with the R programming environment.

C. Angeli et al. / Computers & Education 113 (2017) 226e242232

Statistica Sequence, Association, and Link Analysis is an implementation of several advanced techniques designed for mining rules from datasets that are generally described as “market-baskets”. The “market-basket” metaphor assumes that customers buy products either in a single transaction or in a sequence of transactions. A transaction relates with a subsequent purchase of a product or products given a previous buy. For example, a purchase of flashlights usually coincides with a purchase of batteries in the same basket. In education, the “market-basket” metaphor can be applied to situations where individuals engage in different actions during learning with others or with a computer system. The analysis reveals items in a dataset that occur together extracting patterns and associations between individuals and actions.

2.3. Results and discussion

The quality of FD learners' answers to the immigration problem was found to be 1.43 (SD ¼ 0.63), while the quality of FI learners' answers was found to be 2.10 (SD ¼ 0.75). The time that FD and FI learners spent with the simulation was also measured and no significant differences were found between the two groups of participants. The large mean difference in the quality of FD and FI learners' answers was further investigated using a one-way analysis of variance (ANOVA), and was found to be statistically significant, F (1, 114) ¼ 12.06, p < 0.05, n2 ¼ 0.17, in favour of the FI learners.

In order to further investigate how FD and FI learners interacted with the simulation in order to solve the problem, a separate sequence, association, and link analysis was carried out for each group of FD and FI learners. The outcomes of the sequence, association, and link analysis for the FD learners are shown in Tables 2 and 3, and for the FI learners in Tables 4 and 5.

According to Table 2, FD learners failed to engage in systematic hypotheses testing with the simulation in order to collect data and propose a solution to the problem. This is easily confirmed by the lack of association rules related to controlling the variables IV3, IV4, and IV5, and, the very limited activity about controlling the variables IV1 and IV2. As shown in Table 2, IV1 and IV2 were the only independent variables that FD learners controlled, ignoring the effects of the other three independent variables on the dependent variables. Interestingly, as it is shown in Table 3, which shows the frequencies of each rule for the FD learners, the rules associated with controlling IV1 appear 46 times, and for IV2 39 times, indicating a significant lack of activity related to the control of the independent variables if one considers the fact that there were 70 FD learners partici- pating in the study. This implies that not all FD learners were able to control IV1 or IV2, and none was able to control all five independent variables. This, subsequently, led to answers of poor quality. In addition, FD learners' computer interactions appeared to be repetitions of the same sequences or slightly different sequences of incomplete actions that did not allow the

Table 2 Sequential rules for the FD learners.

Antecedent ¼¼> Consequent Antecedent ¼¼> Consequent 1. (B), (B) ¼¼> (T) 7. (B), (B), (T), (P) ¼¼> (S) 2. (B), (B) ¼¼> (M) 8. (B), (B) ¼¼> (T), (S) 3. (B), (B) ¼¼> (P) 9. (B), (B) ¼¼> (M), (P) 4. (B), (B), (T) ¼¼> (M) 10. (B), (B), (M), (P) ¼¼> (S) 5. (B), (B), (T), (M) ¼¼> (S) 11. (B) ¼¼> (T), (P) 6. (B), (B), (T) ¼¼> (P) 12. (B), (T), (M) ¼¼> (P), (IV2)

13. (B), (T), (M) ¼¼> (P), (IV1) Note: B: BUILD; T: TEST; M: METER; P: PLAY; S: STOP; IV1 ¼ Country A-Number of births; IV2 ¼ Country B-Movement of businesses.

Table 3 Frequent sequences of rules for the FD learners.

Frequent sequences Frequency

(B), (B), (T) 70,00 (B), (B), (M) 70,00 (B), (B), (P) 70,00 (B), (B), (T), (M) 70,00 (B), (B), (T), (P) 70,00 (B), (B), (T), (M), (S) 70,00 (B), (T), (P) 70,00 (B), (B), (T), (P), (S) 70,00 (B), (B), (T), (S) 70,00 (B), (B), (M), (P) 70,00 (B), (B), (M), (P), (S) 70,00 (B), (T), (M), (P), (IV1) 46,00 (B), (T), (M), (P), (IV2) 39,00

Note: B: BUILD; T: TEST; M: METER; S: STOP; P: PLAY; IV1 ¼ Country A-Number of births; IV2 ¼ Country B-Movement of businesses.

Table 4 Sequential rules for the FI learners.

Antecedent ¼¼> Consequent Antecedent ¼¼> Consequent 1. (B), (T), (M), (P) ¼¼> (IV1) 13. (B), (T), (M), (P), (IV1) ¼¼> (IV2), (IV3) 2. (B), (T), (M), (P) ¼¼> (IV2) 14. (B), (T), (M), (P), (IV2) ¼¼> (IV3) 3. (B), (T), (M), (P) ¼¼> (IV3) 15. (B), (T), (M), (P), (IV1) ¼¼> (IV2), (IV4) 4. (B), (T), (M), (P) ¼¼> (IV4) 16. (B), (T), (M), (P), (IV1), (IV3) ¼¼> (IV4) 5. (B), (T), (M), (P) ¼¼> (IV5) 17. (B), (T), (M), (P), (IV1) ¼¼> (IV2), (IV5) 6. (B), (T), (M), (P) ¼¼> (IV1), (IV2) 18. (B), (T), (M), (P), (IV1), (IV3) ¼¼> (IV5) 7. (B), (T), (M), (P), (IV1) ¼¼> (IV5) 19. (B), (T), (M), (P), (IV1) ¼¼> (IV4), (IV5) 8. (B), (T), (M), (P) ¼¼> (IV1), (IV3) 20. (B), (T), (M), (P), (IV4) ¼¼> (IV5) 9. (B), (T), (M), (P), (IV1) ¼¼> (IV2), (IV5) 21. (B), (T), (M), (P), (IV2) ¼¼> (IV5) 10. (B), (T), (M) (P) ¼¼> (IV1), (IV4) 22. (B), (T), (M), (P), (IV3) ¼¼> (IV4) 11. (B), (T), (M), (P) ¼¼> (IV1), (IV5) 23. (B), (T), (M), (P), (IV3) ¼¼> (IV5) 12. (B), (T), (M), (P) ¼¼> (IV2), (IV4)

Note: B: BUILD; T: TEST; M: METER; P: PLAY; IV1 ¼ Country A-Number of births; IV2 ¼ Country B-Movement of businesses; IV3 ¼ Country A-Number of deaths; IV4 ¼ Country B-Number of births; IV5 ¼ Country B-Number of deaths.

Table 5 Frequent sequences of rules for the FI learners.

Frequent sequences Frequency

(B), (T), (M), (P), (IV1) 45,00 (B), (T), (M), (P), (IV2) 45,00 (B), (T), (M), (P), (IV3) 45,00 (B), (T), (M), (P), (IV4) 45,00 (B), (T), (M), (P), (IV5) 45,00 (B), (T), (M), (P), (IV1), (IV2) 30,00 (B), (T), (M), (P), (IV1), (IV3) 30,00 (B), (T), (M), (P), (IV1), (IV4) 30,00 (B), (T), (M), (P), (IV1), (IV5) 30,00 (B), (T), (M), (P), (IV2), (IV3) 30,00 (B), (T), (M), (P), (IV2), (IV4) 30,00 (B), (T), (M), (P), (IV2), (IV5) 30,00 (B), (T), (M), (P), (IV3), (IV4) 30,00 (B), (T), (M), (P), (IV3), (IV5) 30,00 (B), (T), (M), (P), (IV4), (IV5) 30,00 (B), (T), (M), (P), (IV1), (IV2), (IV3) 30,00 (B), (T), (M), (P), (IV1), (IV2), (IV4) 30,00 (B), (T), (M), (P), (IV1), (IV2), (IV5) 30,00 (B), (T), (M), (P), (IV1), (IV3), (IV4) 30,00 (B), (T), (M), (P), (IV1), (IV3), (IV5) 30,00 (B), (T), (M), (P), (IV1), (IV4), (IV5) 30,00

Note: B: BUILD; T: TEST; M: METER; P: PLAY; IV1 ¼ Country A-Number of births; IV2 ¼ Country B-Movement of businesses; IV3 ¼ Country A-Number of deaths; IV4 ¼ Country B-Number of births; IV5 ¼ Country B-Number of deaths.

C. Angeli et al. / Computers & Education 113 (2017) 226e242 233

FD learners to collect useful data for solving the problem. These actions indicate FD learners' uncertainty of what they needed to do to test the model, as well as lack of knowledge in regards to controlling variables and testing hypotheses. All in all, the association rules in Tables 2 and 3 reveal FD learners' weakness to adequately investigate the immigration problem with the glass-box simulation.

In contrast with the rules shown in Table 2, the rules in Table 4 showed that the FI learners interacted with the simulation in a systematic way that led to successful interpretations of the simulated outcomes and provided answers of high quality. According to the rules shown in Table 4, the FI learners followed all necessary steps in order to properly control all five in- dependent variables, collect data, and form conclusions. What is more, according toTable 4, FI learners also engaged in actions demonstrating attempts for examining the effects of several combinations of any two or three independent variables. These actions illustrate FI learners' ability to plan more advanced experimental investigations. Additionally, as it is shown in Table 5, which shows the frequencies of each rule for the FI learners, the rules associated with controlling the independent variables appear 45 times for each independent variable and 30 times for any combination of independent variables. These data show significant differences between the FD and FI learners in regards to their investigations with the glass-box simulation.

C. Angeli et al. / Computers & Education 113 (2017) 226e242234

2.4. Conclusions from Study 1

The results from the first study showed that the FD learners were not able to use the simulation in appropriate ways to control variables, collect useful data, and form appropriate conclusions. Obviously, the FD learners did not cope well with the complexity of the task and failed to develop a step-by-step strategy for solving the problem. In contrast, the FI learners handled successfully the complexity of the problem-solving space, carefully examined the effects of each independent var- iable on the dependent variables, and decided accordingly. The frequencies of the rules as shown in Tables 3 and 5 revealed important differences between the two types of learners in terms of how they interacted with the simulation to investigate the problem at hand. All in all, the FD learners failed to collect useful data with the simulation, as the rules in Table 2 strongly indicated, and thereafter, failed to write an informed answer in regards to the immigration problem given to them.

In the context of the first study, data mining provided the means through which the authors understood better how learners with different cognitive types interacted with the simulation to solve the problem. In essence, association rules mining was found to be a useful method for obtaining reliable data about learners' use of the software and their performance with it. This constitutes a significant departure from current software evaluation approaches that tend to be more or less normative in nature as they often rely on questionnaire data that are subjective, non-reliable, and disconnected from learning behaviors and outcomes (Bangert-Drowns, 2002; Bayram & Nous, 2004; Surry, 1998). In addition, oftentimes, software evaluation is carried out to serve the needs of the organization and its administrators and not necessarily the needs of the individual learners. It is, however, important to evaluate students' learning with educational software, and, thus, it becomes imperative for researchers to develop new techniques for evaluating educational software.

From an educational viewpoint, data mining in study 1 provided the authors with formative evaluation evidence to improve the learning design of the simulation. However, as the sequence, association, and link analysis does not offer specific recommendations in terms of what actions can be taken or need to be taken to improve the learning design of a system, the instructional designer has to decide what modifications to implement in order to facilitate learners' investigations with the software. After a round of revisions and modifications, the instructional designer can employ data mining again as an iterative methodology of testing and improving the system. This methodology constitutes an advancement over traditional formative evaluation methods, as it provides data on actual student performance that can be used to guide software changes and modifications for facilitating in better ways students' learning with the software. In the first study, data mining can offer possible approaches to the optimization of the design and implementation of the revised simulation in terms of its use by learners and how they learn from it. It is, however, imperative after such an exploratory exercise, that specific hypotheses are proposed in each further design and assessment cycle. In this way, specifically targeted data (i.e., data based on insights gained from the mining and analytics specifically implemented in the new design to determine the predictive value of the hypothesis in the environment) are then mined and analyzed. In this study, the data that need to be collected should be directly related to the factors underlying FD-I. For example, the results of such research can be used to design learning an- alytics for remediating FD learners' cognitive deficits during learning with simulations. Learning analytics when applied to underlying factors such as perceptual affordances of the environment and patterns of learning of learners in the environment can be used to capture differences in how learners use and navigate in a simulation, and, as such provide predictions as to effective just-in-time support within the simulation.

From a data mining perspective, the association, sequence, and link analysis produced and showed hundreds of associ- ation rules in a traditional text format. Making sense of these results was a demanding task and could be a daunting task for educational researchers who might not have previous experiences with data mining. In addressing this problem, Romero and Ventura (2013) suggested to integrate recommender systems in data mining tools to display results, information, explana- tions, comments, and recommendations to the non-expert user in data mining. “Thus, instead of showing the obtained DM model, a list of suggestions or conclusions about the results and how to apply them are shown to the users” (p. 21).

In summary, based on the results of the first study, the design and evaluation of educational systems can benefit from integrating data mining tools in them so that through decision support systems, wizard tools, and recommendation engines the learning design of the systems can be improved.

3. Study 2

3.1. Theoretical framework and research questions

In the second study, the authors conducted a preliminary exploratory examination of important technology integration factors and their relation to learning. In education, a common misconception is that young people are confident users of digital technologies, and that use of digital technologies leads to positive learning outcomes Howard & Thompson, 2015; Margaryan, Littlejohn, & Vojt, 2011; Selwyn, 2009; Thompson, 2013). However, research has shown that many students are, in fact, not confident with using technology (e.g., Wang, Hsu, Campbell, Coster, & Longhurst, 2014; Warschauer & Matuchniak, 2010), which suggests a more varied range of student experiences in technology-integrated learning and a more complex relationship with student performance. If difference in perceptions of learning between teachers and students is too large, there is a risk of students becoming unengaged and unmotivated to learn (Vermunt & Verloop,1999). Therefore, a better understanding of student experiences in using digital technologies, and what this means for learning, is needed to

C. Angeli et al. / Computers & Education 113 (2017) 226e242 235

develop more effective and inclusive learning environments (K€onings, Seidel, & van Merri€enboer, 2014; Li, 2007; Pellas, 2014; Skryabin, Zhang, Liu, & Zhang, 2015).

In essence, the second study directly addresses the complexity of technology integration, which according to Borko, Whitcomb, and Liston (2009), has proven to be a “wicked” problem for educational research. One of the reasons for this is because “orderly processes in creating human judgment and intuition lead people to wrong decisions when faced with complex and highly interacting systems” (Forrester, 1971, p. 52). Primarily, it is nearly impossible for the human mind to fully conceptualize complex systems, such as, teaching and learning, and to fully understand dynamic relations and feedback among constituent parts (Howard & Thompson, 2015). Data mining techniques can be used to draw new insights into the important relations and interactions among known key factors of technology integration from school questionnaire data. This is a novel approach, as this type of data can be difficult to use, because it tends to be inconsistent, incomplete, and heterogeneous, particularly, free texts, and varied personal perceptions of questions posing a range of issues for EDM. The knowledge discovery approach of data mining techniques is able to account for numerous factors and complex systems (Fayyad, Piatestsky-Sharpiro, & Smyth, 1996; Papamitsiou & Economides, 2014), and findings from these new approaches can inform and extend the existing body of knowledge (Baker, 2010).

In this study, the authors argue that because students report different experiences in technology-integrated learning environments, the use of data mining approaches can inform new factors affecting learning performance in novel ways. The study was undertaken to examine which factors of students' technology integration, such as positive and negative engage- ment, and high and low confidence in using digital technologies, were meaningfully related to learning outcomes. In particular, eight key factors of digital technology use, engagement with digital technologies, school engagement, and national assessments were explored. The analysis focused on two groups of factors including Information and Communication Technologies (ICTs) Engagement, Computer-Efficacy, and School Engagement, in relation to aggregated school-level perfor- mance on numeracy and literacy assessments. The specific research questions were stated as follows:

1. What are the different patterns occurring among key factors related to students' experiences in technology integration? 2. How do these patterns relate to learning outcomes?

3.2. Method

3.2.1. Data sources The datasets used in this analysis were taken from a large-scale study examining the Australian Digital Education Rev-

olution in New South Wales (DER-NSW), and from the Australian National Assessment Program Literacy and Numeracy (NAPLAN) assessments. The DER-NSW was a federal program (2008e2014) aiming to provide all secondary (Years 9e12) students and teachers with current and up-to-date digital technologies (Department of Education Employment and Workplace Relations, 2012). In New South Wales (NSW), the program was evaluated over four years (2010e2013) through online questionnaires and school case studies. A full description of the study can be found in Howard and Mozejko (2013). The DER-NSW study included all government secondary schools across the state (N ¼ 436). The analysis presented in this paper is drawn from the 2012 Year 9 student questionnaire data. Of the approximately 50,000 Year 9 students in NSW government schools, 21,795 (43%) students completed a two-part questionnaire in 2012; 12,978 students completed Part A, and, 8817 students completed Part B. Responses to Part B of the questionnaire were included in the current analysis. These data included students from 216 NSW secondary schools.

The Australian NAPLAN assessment is administered to students each year, in school Years 3, 5, 7, and 9. The aim of the assessment is to test the types of skills that are essential for every child to progress through school and life (National Assessment Program, 2013). In each year, students' performances on reading, writing, spelling, grammar, punctuation, and numeracy are measured. In alternating years, some schools have also been assessed on civics and citizenship, ICTs, and science literacy. These provide a national “snapshot” of performance in these areas. Tests are administered by each state at individual schools using national protocols. Test results at the school level are made publicly available through the mySchool.com website. For the current study, Reading and Numeracy scores for 195 of schools participating in the Year 9 2012 Student Part B questionnaire were included in the analysis.

3.2.2. Data structure and analysis The DER-NSW Student questionnaire B was administered in 2012 and comprised a total of 147 question items covering five

main subscales: School Engagement, Computer Use, Your Learning, Your Subjects, and Your Intentions. The School Engage- ment subscale was adapted from the NSW DEC Student School Life Survey (SPL-SSL), which provided the department with student feedback on schools. The Program for International Student Assessment (PISA) ICT use and familiarity measure (OECD, 2006) was used as the basis for the Computer Use subscale. Items on frequency of use, confidence (computer-efficacy) in performing tasks using a computer, such as internet searches and data manipulation, were included. The Your Learning

C. Angeli et al. / Computers & Education 113 (2017) 226e242236

subscale addressing students' learning preferences was adapted from the NSW SchoolMap Best Practices Statements (Department of Education and Training, 2002). Your Subjects addressed students' perceptions of success in different subject areas (Lamont & Maton, 2010). Your Intentions was a standard department measure considering students' intentions to leave school early, begin to work, or post-school study after graduation. The questionnaire was pilot tested in 2009 at three schools and was revised. The reliability of the questionnaire was found to be high for each subscale, ranging from Cronbach's alpha 0.83 to 0.93.

In regards to data mining techniques, the current study combined the well-proven association rules analysis and fuzzy representations to answer the research questions. Fuzzy representation techniques aim to describe uncertainties in concepts and perceptions using fuzzy set theory. Combining association rules analysis with fuzzy representations allows for addressing increased sensitivity to variation among participants' responses. An example of this is the use of the standard Likert-type items of “strongly agree,” “agree,” “disagree,” and “strongly disagree,” as responses. These responses do not contain a clear boundary of semantic meaning and can be interpreted differently by different participants. Therefore, each response rep- resents a range of vagueness. A fuzzy concept can be expressed by a fuzzy set to cover possible semantic vagueness in a response, within which each semantic has a value (called membership degree) to indicate to what extent it can be described using the fuzzy concept. For example, we can describe a student's learning performance as “Sound” or “Excellent” based on his or her NAPLAN score. Here, “Sound” or “Excellent” are fuzzy concepts defined on NAPLAN scores. Given a NAPLAN score, say 560, we can determine, for example, 0.8 to “Sound” and “0.4” to “Excellent”. Thus, the use of fuzzy representations allows us to have a better understanding of the collected data and have a tool to handle vagueness in these data.

Specifically, the analysis comprised of three main data mining steps: (a) factor generation, (b) fuzzy representation, and (c) association rule mining (already explained in study 1). The first step focused on generating factors related to students' engagement, performance, and, ICT efficacy. This was the process of identifying key questions from the dataset and construct factors from them. Eight main factors from the questionnaire, covering 16 sub-factors, were constructed: Computer-Efficacy (3 sub-factors), ICT Engagement (3 sub-factors), Learning Preferences (3 sub-factors), Learning Beliefs (3 sub-factors), ICT and Learning Performance (1 sub-factor), School Engagement (1 sub-factor), Teacher Directed ICT Use Frequency (1 sub-factor), and ICT Importance in Subject Areas (1 sub-factor). The eight main factors and the two NAPLAN factors (Numeracy and Reading) are shown in Table 6.

In the second step, raw responses of all factors were rescored to create a fuzzy representation. For the questionnaire data, the fuzzy representation was conducted at the individual level and then aggregated to a school level in order to match the NAPLAN data, which are only available at the school level. In the process of constructing fuzzy representations, numeric data and categorical data were processed in different ways. If a factor was measured using numeric data, the median value of all individual students in a school was used as the school-level value of that factor. If a factor was described as categorical data, the mode value of all individual students in the same school was used as the school-level value of that factor. The Year 9 2012 questionnaire and Year 9 2012 NAPLAN datasets were linked by a national school code.

An example of a fuzzy representation is defining the fuzzy concept “frequent user” of ICT technology in teaching as

frequent user ðhÞ ¼ 8 < :

h 3 ; h < 3

1; h � 3 where h is the participant's hours of ICT use in a day. The more time a user spends on ICT use,

the higher the membership degree. Hence, a user who spends one hour daily on using ICT in teaching will be treated as a “less frequent” user with a membership degree of 0.33. Similarly, we can categorize a user as “less frequent user,” “frequent user,” and “much frequent user.”

In the final step, the dataset was split into three datasets, i.e., all schools (Dataset 1), schools with positive ICT engagement (Dataset 2), and schools with negative ICT engagement (Dataset 3). Students' engagement with ICT was identified as a motivating factor in teachers' use of digital technologies (Ertmer, Ottenbreit-Leftwich, Sadik, Sendurur, & Sendurur, 2012). An association rules analysis was conducted on the three datasets to identify where potential significant associations among factors existed. In a significant association, the antecedent factor is likely to have an effect on the consequent. The apriori algorithm in R was used to implement the association rule mining and adjustable parameters (support degree, confidence degree, and lift) were set and tuned. Rules from the three datasets were converted to a directed graph, in which each factor from the antecedent set and the consequent set was associated with a node in the graph. The stronger the connection, the thicker the arrow line in the graph.

3.3. Results and discussion

Through association rules analysis, the factors that were found to be important in both the positive and negative ICT Engagement datasets and related to students' computer use and beliefs were selected for further examination. There were five ICT Engagement sub-factors with four measurements: ICT Engagement Positive, High (1); ICT Engagement Positive, Medium (2); ICT Engagement Positive, Low (3); ICT Engagement Neutral (4); ICT Engagement Negative, High (5). In more analytical terms, ICT Engagement Positive, High represented schools with students who agreed (Positive) with most engagement statements, and that agreement was strong (High). There were six Computer-efficacy factors: Computer-efficacy Productivity, No knowledge (6); Computer-efficacy Productivity, Low (7); Computer-efficacy Processing, No knowledge (8); Computer-efficacy Processing, Low (9); Computer-efficacy Creating, No knowledge (10); Computer-efficacy Creating, Low (11). Computer-efficacy factors described

Table 6 Key factors and their descriptions.

Factor Description Sample items

ICT Engagement Includes 4 general engagement items: each has 4 meaningful responsesa

It is very important to me to work with a computer.

Computer- Efficacy

Includes 10 items: Productivity tasks (6), Processing tasks (2) and Creating tasks (2); each had 4 meaningful responsesb

Productivity: I am able to take notes using a computer (e.g., recording notes in class). Processing: I am able to edit written work using a computer (e.g., revising writing, spell checking, etc.). Creating: I am able to write a first draft using a computer (e.g., writing in Word rather than on paper first).

School Engagement

Includes 5 items; each has 4 responsesa In my school, I am treated with respect by other students.

Learning Preferences

Includes 3 items: direct, self-paced and collaborative learning; each has 4 meaningful responsesa

I learn more when the teacher talks to the class (e.g., a History lecture, explaining Maths on the board, etc.). I learn more when I am able to explore ideas on my own (e.g., independent research, doing homework, etc.). I learn more when I work in groups with other students (e.g., on a problem set, on a project, etc.).

Learning Beliefs Includes 3 beliefs; self, collaborative and instructed; each has 4 meaningful responsesa

The things I learn in school will prepare me for life as an adult. I am able to contribute when working with other students in a group. I am encouraged to think about things in my own way.

ICT Importance in Subject Areas

Includes 7 school subjects; each has 4 meaningful responsesd How important is it to use computers and ICTs in…English, History, Geography.

Teacher Directed ICT Use

Includes 10 items: each has 9 meaningful responsesc Gather information from different places to solve a problem (e.g., different websites or databases); 2e4 times a week.

ICT Learning Performance

Includes 5 items; each has 4 meaningful responsesa My work is more creative when I use a computer.

NAPLAN Reading Includes 1 school mean Identify the main idea of the poem. NAPLAN

Numeracy Includes 1 school mean Find value of missing angle in a triangle, with access to a calculator.

a 4 ¼ Strongly agree, 3 ¼ Agree, 2 ¼ Disagree, 1 ¼ Strongly disagree. b 4 ¼ I can do this well by myself, 3 ¼ I can do this with help from someone, 2 ¼ I know what this is but can't do it, 1 ¼ I don't know what this means. c 8 ¼ Many times a day, 7 ¼ Once a day, 6 ¼ 2e4 times a day, 5 ¼ Once a week, 4 ¼ 1e3 times a month, 3 ¼ Once a term, 2 ¼ 1e3 times a year, 1 ¼ Never,

0 ¼ I don't know what this means. d 4 ¼ Very important, 3 ¼ Important, 2 ¼ Not very important, 1 ¼ Not at all.

C. Angeli et al. / Computers & Education 113 (2017) 226e242 237

three types of increasingly complex computer-based tasks: Productivity (e.g., email, editing a document, etc.), Processing (e.g., making a simple presentation), and Creating (e.g., making a webpage). Scores were classified as: “No knowledge”, Low, Medium, and High efficacy. Computer-efficacy Productivity, No Knowledge represented those students who selected “I don't know what this means” (No Knowledge) on most of the productivity tasks. A “Low” label represented students who understood most of the tasks, but needed help to perform them. A portion of students did report Medium and High computer-efficacy on all three task types, but, rules containing these factors were not important in either dataset. There were three School Engagement factors, School Engagement, Negative (12), Neutral (13), and Positive (14). NAPLAN Reading, Medium (15) and Numeracy, Medium (16) were important in the datasets. The two factors were categorized into three levels: Low, Medium, and High, based on the schools' mean scores on each assessment.

The two directed graphs show the resulting patterns of rules for positive ICT Engagement (Dataset 2; see Fig. 3) and negative ICT Engagement (Dataset 3; see Fig. 4). The two graphs demonstrate how different patterns of factors that affected Reading and Numeracy have resulted depending on students' engagement with ICTs.

In Fig. 3, nine factors were important, forming 14 rules and two clusters with Reading (Literacy) and Numeracy at the center of each. An association did not exist between Reading (15) and Numeracy (16). All technology integration factors exhibited similar strengths in their associations with Reading and Numeracy, which suggests similar effects of those factors on learning performance in this group. Importantly, all of the computer-efficacy factors were No knowledge (6, 8, and 10). This suggests that the most frequently occurring rules were among schools where students were positive about using ICTs, but, with limited knowledge on how to actually perform different tasks. School Engagement, Negative (12) also appeared as an important factor in this dataset.

In Fig. 4, nine factors were also important, forming 14 rules and two clusters with Reading (16) and Numeracy (15) at the center of each. Reading and Numeracy were also associated with each other. However, there were several key differences in patterns resulting from the two datasets. First, unlike positive ICT engagement, not all technology integration factors were equally related to Reading and Numeracy. Within this group, for the subset of students reporting positive ICT Engagement (1e3), it was likely to have a stronger effect on Reading and Numeracy than other factors (6, 8, and 10). Second, an important Computer-efficacy factor in this dataset was No Knowledge. This suggests, that, similar to the positive group, the most frequently occurring rules were among schools where students were negative about using ICTs and did not feel confident

Fig. 3. Positive ICT engagement.

C. Angeli et al. / Computers & Education 113 (2017) 226e242238

about their knowledge to perform different computer-related tasks. School Engagement, Negative did not appear in the negative ICT engagement dataset, but the Neutral (13) factor did. This suggests that there is a group of students with a strong association between feeling negatively about using ICTs in school, but, more positive about school.

3.4 Conclusions from Study 2

The results from conducting association rules analysis on questionnaire data provided a view of different patterns of technology integration between student groups that highlighted the complexity of technology integration in schools. For the positive ICT engagement schools, it may be important to further address the issue of negative school engagement to un- derstand how it affects their learning, and, ultimately, performance. For the negative ICT engagement group, it may be necessary to examine why students reporting positive ICT engagement are different from the rest of the population. If this group performs better, it may provide justification to address students' ICT engagement through learning design.

Extending this analysis, individual schools could be examined to personalize important factors for students in their own populations. Research has used data mining results to personalize teaching and learning for individuals (Lin et al., 2013), but, the same can be done for groups. In this discussion, results have shown differences between two groups, which could extend to recommendations for addressing important factors in those populations. This could be a useful approach for schools when engaging in technology-related change initiatives to determine where to focus resources and inform choices about additional technology, teacher and student support, curriculum design, etc. In line with other studies of student data, this approach can provide a deeper understanding of student experiences (e.g., Abdous et al., 2012), and school context.

In addition, results from processing student questionnaire data can then be used to design learning analytics. Learning analytics can support exploration of the complexity of students' experiences in technology integration and help identify key points of interaction and effects. Over time, additional questionnaire data can be added to track how students' experiences may change over time. This can also inform learning design, as many teachers continue to struggle with identifying how technology use may improve learning (Perrotta, 2013). For example, in the current analysis, it was identified that students' ICT engagement was more likely to have an effect on standardized tests performance than on computer efficacy. Teachers may adjust their learning design to focus on creating engaging tasks rather than up-skilling students. In the negative group, the

Fig. 4. Negative ICT engagement.

C. Angeli et al. / Computers & Education 113 (2017) 226e242 239

results show that computer-efficacy about Processing tasks had an effect on students' performance. This type of result can help narrow decision-making when teachers design learning tasks.

Further, more objective data can be gleaned from the environment itself with respect to its use and learners' development within the environment. Information gained from keystrokes and mouse moves, for example, can help in determining key interactions between the learner and the environment. Also, physiological measures such as electrodermal activity (skin conductivity) and heart rate can more objectively determine factors such as the extent of engagement (Ali, Hatala, Ga�sevi�c & Jovanovi�c, 2012; Howard-Jones & Demetriou, 2009; Rani, Sarkar, & Liu, 2006).

All in all, in study 2, EDM provided a way to inductively explore questionnaire data to look for unique patterns and trends, which may have otherwise been invisible or neglected (Romero, Ventura, Pechenizkiy, & Baker, 2010). The use of fuzzy representations to explore perceptions and “agreement” allowed for a more nuanced examination of these factors through the creation of categories, such as high, medium, and low, providing a wider range of options for associations. While this method exponentially increases the number of factors being analyzed, which can be problematic in traditional approaches, it is easily handled in data mining. Importantly, in this study, data mining techniques contributed significantly in answering the research questions through the identification of distinctly different patterns among the groups, which allowed the complexity of different learning environments to be observed. As a result, both researchers and teachers would be able to leverage findings to better understand some of the complex effects of digital technologies on learning and inform learning design.

4. General discussion and concluding remarks

In this paper, the contribution of EDM for educational technology classroom research has been examined within the context of two studies with different types of datasets and purposes. The first study, which made use of video data converted first into log-file data before mining, investigated EDM as a potential software evaluation method for improving the design of a stand-alone simulation tool to benefit learners' needs. The second study, which made use of questionnaire data, investigated EDM as a method for providing detailed student data for informing school-based technology integration initiatives.

C. Angeli et al. / Computers & Education 113 (2017) 226e242240

The first study provides a good example of how EDM can be used to advance educational software evaluation practices in the field of educational technology. The employment of association rules mining in this research study provided the authors with (a) reliable data about how learners with different cognitive types interacted with a simulation to solve a problem, and, (b) insights about how learning analytics can be designed and incorporated in the learning design of the simulation. Due to the fact that in this study the association rules mining method produced an enormous body of complicated output e something that can easily discourage educational researchers from employing data mining tools and methods in their research e the authors recommend that educational data mining tools employ alternative ways of reporting results to educational researchers.

The second study provides a good example of how educational technologists can use EDM for guiding and monitoring school-based technology-integration efforts. Taking into consideration the complexity of such efforts (Borko et al., 2009), the results of the second study showed that EDM was quite useful for examining complex interactions and relations among key factors affecting technology integration. What is more, the second study made use of questionnaire data, something un- common for EDM methods due to the nature of this type of data as it tends to be incomplete and inconsistent.

Concisely, based on the findings of the two studies discussed here, EDM can make a significant contribution to educational technology classroom research in terms of providing educational researchers with the tools to study and improve learning design. However, we concur with Hung and Zhang (2008) and Merceron and Yacef (2005) that data mining techniques and tools are not educator-friendly. Specifically, based on the results of the two studies, the employment of data mining in educational research raises some issues of concern. The authors, based on their experiences as discussed in study 1 and study 2, group these issues into two main areas: (a) the structure and organization of the data for mining, and (b) the appropri- ateness of data mining techniques.

In regards to data structure and organization, educational datasets often contain self-reported data (e.g., study 2). Much of these data are usually collected through questionnaires, which often include a range of different types of questions with subjective answers. This issue becomes more complex when different forms of data, such as video (e.g., study 1) and audio are introduced into the analysis. For data mining, these different data types need to be processed into a unified form that can be used for data mining. However, as current data mining techniques are not specifically developed for use with educational datasets, the preparation of classroom data for mining is often done manually by the researcher. This can be a difficult and extremely time-consuming task highly prone, at the same time, to human error. Thus, it may be useful to consider incor- porating data mining tools in educational software and systems to facilitate the preparation and integration of various types of data for mining.

In regards to the second issue about selecting appropriate data mining techniques, the authors found it useful to exper- iment first with different techniques using different software tools before making a final decision. This was done because general-purpose data mining techniques are not specifically designed to answer educational questions, and, thus, may not always produce meaningful results with a specific educational dataset. For example, a high performance data mining tech- nique used on one dataset may be inappropriate for another. One strategy the authors adopted for addressing this was to test and compare results from different data mining techniques before deciding on the techniques to use.

In sum, the authors herein recognize the added value of data mining techniques in opening up new ways of looking at and analyzing classroom data, and recognize at the same time the difficulty for the educational researchers to learn how to employ these techniques in their own research. Thus, for data mining to become main-stream in educational technology research, efforts need to be invested in developing new tools and techniques or refining existing techniques to meet in better ways the needs of educational researchers. These efforts may require broad and sustained collaboration among researchers from various and multiple disciplines.

Finally, the authors recognize the importance of data-privacy issues in data mining. As more and more learners provide personal information to free and accessible online learning systems, their privacy may possibly be at risk if intrusion or use of personal data can impact their life in negative ways (Clifton, Kantarcioglu, Vaidya, Lin, & Zhu, 2002). Despite the fact that currently issues related to data privacy are far from being settled, it is acknowledged that data-privacy issues have been of concern to the data-mining research community and that a considerable number of data mining researchers have proposed ways of how data mining procedures can be redefined in order to preserve data privacy (e.g., Agrawal & Srikant, 2000; Ifenthaler & Tracey, 2016; Nethravathi, Desai, Shenoy, Indiramma, & Venugopal, 2016; Verykios, Bertino et al., 2004; Verykios, Elmagarmid, Bertino, Saygin, & Dasseni, 2004). What is more, as Verykios, Bertino et al. (2004) and Verykios, Elmagarmid et al. (2004) stated, “in order to make a publicly available system secure, we must ensure not only that private sensitive data have been trimmed out, but also to make sure that certain inference channels have been blocked as well” (p. 45). For example, if an association rules mining tool shows that FD-I is also associated with difficulties in understanding literacy texts, then a teacher may quickly and wrongly infer that a learner with problems in literacy is FD. Making this inference is dangerous and can lead to the application of inappropriate remediation techniques, because the learner's difficulty to un- derstand literacy texts may be related to other conditions and not FD-I.

Within the context of data mining for education, data-privacy issues should be considered having in mind that educational researchers are interested in performing data mining analyses for (a) collecting data from organized groups of students in order to improve the design of learning systems, and (b) identifying those learners who face difficulties in order to help them on an individual basis. In the first case, it is possible to preserve students' anonymity through some form of data perturbation, but, in the second case, the issue appears to be more complex as the aim is to access enough student data in order to provide a personalized remediation plan.

C. Angeli et al. / Computers & Education 113 (2017) 226e242 241

In conclusion, the issue of data privacy and protection is complicated and any research efforts devoted toward this di- rection are fully warranted. It is worth noting that the challenge for the educational data mining researcher is to find ways of how to protect learners' data on the one hand, while allowing, on the other hand, enough leeway for the educational researcher to use personal data in order to help learners on an individual basis and in personalized ways.

References

Abdous, M., He, W., & Yen, C.-J. (2012). Using data mining for predicting relationships between online question theme and final grade. Journal of Educational Technology & Society, 15(3), 77e88. Retrieved from: http://www.jstor.org/stable/jeductechsoci.15.3.77.

Agrawal, R., & Srikant, R. (2000). Privacy-preserving data mining. ACM Sigmod Record, 29(2), 439e450. Ahmed, A. B. E. D., & Elaraby, I. S. (2014). Data mining: A prediction for student's performance using classification method. World Journal of Computer

Application and Technology, 2(2), 43e47. Ali, L., Hatala, M., Gasevic, D., & Jovanovic, J. (2012). A qualitative evaluation of evolution of a learning analytics tool. Computers & Education, 58, 470e489.

http://dx.doi.org/10.1016/j.compedu.2011.08.030. Amershi, S., & Conati, C. (2009). Combining unsupervised and supervised machine learning to build user models for exploratory learning environments.

Journal of Educational Data Mining, 1(1), 71e81. Angeli, C., & Valanides, N. (2004). Examining the effects of text-only and text-and-visual instructional materials on the achievement of field-dependent and

field-independent learners during problem-solving with modeling software. Educational Technology Research and Development, 52(4), 23e36. Angeli, C., Valanides, N., & Kirschner, P. (2009). Field dependence-independence and instructional design effects on Learners’ performance with a computer

modeling tool. Computers in Human Behavior, 25, 1355e1366. Baker, R. S. (2010). Data mining for education. In International encyclopedia of education (pp. 112e118). NY: Elsevier. Baker, R. S., & Siemens, G. (2014). Educational data mining and learning analytics. In K. Sawyer (Ed.), Cambridge handbook of the learning sciences (2nd ed.,

pp. 253e274). NY: Cambridge University Press. Baker, R. S., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1(1), 3e17. Bangert-Drowns, R. L. (2002). Teacher ratings of student engagement with educational software:An exploratory study. Educational Technology Research and

Development, 50(2), 23. Bayram, S., & Nous, A. P. (2004). Evolution of educational software evaluation: Instructional software assessment. The Turkish Online Journal of Educational

Technology, 3(2), 21e27. Beal, C. R., Qu, L., & Lee, H. (2006). Classifying learner engagement through integration of multiple data sources. In: Paper presented at the 21st National

Conference on Artificial Intelligence (AAAI-2006), Boston, MA. Borko, H., Whitcomb, J., & Liston, D. (2009). Wicked problems and other thoughts on issues of technology and teacher learning. Journal of Teacher Education,

60(1), 3e7. Buja, A., & Lee, Y. S. (2001, August). Data mining criteria for tree-based regression and classification. In Proceedings of the seventh ACM SIGKDD international

conference on Knowledge discovery and data mining (pp. 27e36). ACM. Burnett, W. C. (2010). Cognitive style: A meta-analysis of the instructional implications for various integrated computer enhanced learning environments.

Doctoral dissertation thesis. Pennsylvania: Indiana University of Pennsylvania. Chen, C. M. (2008). Intelligent web-based learning system with personalized learning path guidance. Computers & Education, 51(2), 787e814. Chen, C. M., & Duh, L. J. (2008). Personalized web-based tutoring system based on fuzzy item response theory. Expert Systems with Applications, 34(4),

2298e2315. Chen, S. Y., & Macredie, R. D. (2002). Cognitive styles and hypermedia navigation: Development of a learning model. Journal of the American Society for

Information Science and Technology, 53(1), 3e15. Clariana, B. R., & Strobel, J. (2008). Modeling technologies. In J. M. Spector, D. M. Merrill, J. van Merrienboer, & M. P. Driscoll (Eds.), Handbook of research on

educational communications and technology (pp. 329e344). NY: Routledge. Clifton, C., Kantarcioglu, M., Vaidya, J., Lin, X., & Zhu, M. Y. (2002). Tools for privacy preserving distributed data mining. ACM Sigkdd Explorations Newsletter,

4(2), 28e34. Daniels, H. L., & Moore, D. M. (2000). Interaction of cognitive style and learner control in a hypermedia environment. International Journal of Instructional

Media, 27, 369e384. Department of Education Employment and Workplace Relations. (2012). Digital education revolution - overview. Retrieved September 20, 2013, from: http://

www.deewr.gov.au/Schooling/DigitalEducationRevolution/Pages/default.aspx. Department of Education and Training. (2002). SchoolMap best practices statements (Sydney). Dragon, K. (2009). Field dependence and student achievement in technology based learning: A meta-analysis. Master thesis. Alberta: University of Alberta. Ertmer, P. A., Ottenbreit-Leftwich, A. T., Sadik, O., Sendurur, E., & Sendurur, P. (2012). Teacher beliefs and technology integration practices: A critical

relationship. Computers & Education, 59(2), 423e435. Fayyad, U., Piatestsky-Sharpiro, G., & Smyth, P. (1996). From data mining to knowledge discovery in databases. AI Magazine, 17(3), 37e54. Forrester, J. W. (1971). Counterintuitive behaviour of social systems. Technology Review, 73(3), 52e68. French, J. W., Ekstrom, R. B., & Price, L. A. (1963). Manual for kit of reference tests for cognitive factors. Princeton, NJ: Educational Testing Service. He, W. (2013). Examining students' online interaction in a live video streaming environment using data mining and text mining. Computers in Human

Behavior, 29(1), 90e102. Howard, S. K., & Mozejko, A. (2013). DER-NSW Evaluation: Conclusions on student and teacher engagement and ICT use. Sydney. Howard, S. K., & Thompson, K. (2015). Seeing the system: Dynamics and complexity of technology integration in secondary schools. Education and In-

formation Technologies. https://doi.org/10.1007/s10639-015-9424-2. Howard-Jones, P. A., & Demetriou, S. (2009). Uncertainty and engagement with learning games. Instructional Science, 37, 519e536. http://dx.doi.org/10.1007/

s11251-008-9073-6. Hsu, M. H. (2008). A personalized English learning recommender system for ESL students. Expert Systems with Applications, 34(1), 683e688. Hung, J. L., & Zhang, K. (2008). Revealing online learning behaviors and activity patterns and making predictions with data mining techniques in online

teaching. MERLOT Journal of Online Learning and Teaching, 4(4), 426e437. Ifenthaler, D., & Tracey, M. W. (2016). Exploring the relationship of ethics and privacy in learning analytics and design: Implications for the field of

educational technology. Educational Technology Research and Development, 64(5), 877e880. Ingram, A. (1999). Using web server logs in evaluating instructional websites. Journal of Educational Technology Systems, 28(2), 137e157. Khine, M. S. (1996). The interaction of cognitive styles with varying levels of feedback in multimedia presentation. International Journal of Instructional

Media, 23, 229e237. K€onings, K., Seidel, T., & van Merri€enboer, J. G. (2014). Participatory design of learning environments: Integrating perspectives of students, teachers, and

designers. Instructional Science, 42(1), 1e9. http://dx.doi.org/10.1007/s11251-013-9305-2. Lamont, A., & Maton, K. (2010). Unpopular music: Beliefs and behaviours towards music in education. In R. Wright (Ed.), Sociology and music education (pp.

63e80). London: Ashgate. Landriscina, F. (2013). Simulation and learning: A model-centered approach. NY: Springer. Li, Q. (2007). Student and teacher views about technology. Journal of Research on Technology in Education, 39(4), 377e397.

C. Angeli et al. / Computers & Education 113 (2017) 226e242242

Lin, W., Alvarez, S. A., & Ruiz, C. (2002). Efficient adaptive-support association rule mining for recommender systems. Data Mining and Knowledge Discovery, 6(1), 83e105.

Lin, C. F., Yeh, Y., Hung, Y. H., & Chang, R. I. (2013). Data mining for providing a personalized learning path in creativity: An application of decision trees. Computers & Education, 68, 199e210.

Margaryan, A., Littlejohn, A., & Vojt, G. (2011). Are digital natives a myth or reality? University students' use of digital technologies. Computers & Education, 56(2), 429e440.

Merceron, A., & Yacef, K. (2005). Tada-ed for educational data mining. Interactive Multimedia Electronic Journal of Computer-enhanced Learning, 7(1), 267e287.

Merceron, A., & Yacef, K. (2010). Measuring correlation of strong symmetric assocation rules in educational data. In C. Romero, S. Ventura, M. Pechenizkiy, & R. S. J. D. Baker (Eds.), Handbook of educational data mining (pp. 245e255). Boca Raton: Taylor & Francis Group.

Messick, S. (1987). Structural relationships across cognition, personality, and style. Aptitude, Learning, and Instruction, 3, 35e75. Mohamad, S. K., & Tasir, Z. (2013). Educational data mining: A review. Procedia-social and Behavioral Sciences, 97, 320e324. Morgan, H. (1997). Cognitive styles and classroom learning. Westport, CT: Praeger. National Assessment Program. (2013). NAPLAN. Retrieved March 4, 2016, from: http://www.nap.edu.au/naplan/naplan.html. Nethravathi, N. P., Desai, V. J., Shenoy, P. D., Indiramma, M., & Venugopal, K. R. (2016). A brief survey on privacy preserving data mining techniques. Data

Mining and Knowledge Engineering, 8(9), 267e273. Nisbet, R., Elder, J., & Miner, G. (2009). Handbook of statistical analysis and data mining applications. NY: Elsevier. OECD. (2006). PISA 2006 information and communication technology questionnaire (OECD Program for International Student Assessment). Papamitsiou, Z. K., & Economides, A. A. (2014). Learning analytics and educational data mining in practice: A systematic literature review of empirical

evidence. Educational Technology & Society, 17(4), 49e64. Pellas, N. (2014). The influence of computer self-efficacy, metacognitive self-regulation and self-esteem on student engagement in online learning pro-

grams: Evidence from the virtual world of Second Life. Computers in Human Behavior, 35, 157e170. Perera, D., Kay, J., Koprinska, I., Yacef, K., & Zaiane, O. R. (2009). Clustering and sequential pattern mining of online collaborative learning data. IEEE

Transactions on Knowledge and Data Engineering, 21(6), 759e772. Perrotta, C. (2013). Do school-level factors influence the educational benefits of digital technology? A critical analysis of teachers' perceptions. British Journal

of Educational Technology, 44(2), 314e327. Price, L. (2004). Individual differences in learning: Cognitive control, cognitive style, and learning style. Educational Psychology, 24(5), 681e698. Rani, P., Sarkar, N., & Liu, C. (2006). Maintaining optimal challenge in computer games through real-time physiological feedback. In D. D. Schmorrow (Ed.),

Task specific information processing in operational and virtual environments, foundations of augmented cognition (pp. 184e192). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.

Rittschof, K. A. (2010). Field dependence-independence as visuospatial and executive functioning in working memory: Implications for instructional systems design and research. Educational Technology Research and Development, 58(1), 99e114.

Romero, C., & Ventura, S. (2007). Educational data mining: A survey from 1995 to 2005. Expert Systems with Applications, 33(1), 135e146. Romero, C., & Ventura, S. (2010). Educational data mining: A review of the state of the art. IEEE Transactions on Systems, Man, and Cybernetics, Part C

(Applications and Reviews), 40, 601e618. Romero, C., & Ventura, S. (2013). Data mining in education. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 3(1), 12e27. Romero, C., Ventura, S., Pechenizkiy, M., & Baker, R. S. J. D. (2010). Handbook of educational data mining. Boca Raton: Taylor & Francis Group. Selwyn, N. (2009). Faceworking: Exploring students' education-related use of facebook. Learning, Media and Technology, 34(2), 157e174. http://dx.doi.org/

10.1080/17439880902923622. Skryabin, M., Zhang, J., Liu, L., & Zhang, D. (2015). How the ICT development level and usage influence student achievement in reading, mathematics, and

science. Computers & Education, 85, 49e58. Surry, D. (1998). Biolab frog & biolab pig. TechTrends, 43(2), 8. Tang, T., & McCalla, G. (2005). Smart recommendation for an evolving e-learning system. International Journal on E-learning, 4(1), 105e129. Tan, P. N., Kumar, V., & Srivastava, J. (2004). Selecting the right objective measure for association analysis. Information Systems, 29(4), 293e313. Thompson, P. (2013). The digital natives as learners: Technology use patterns and approaches to learning. Computers & Education, 65, 12e33. Vermunt, J. D., & Verloop, N. (1999). Congruence and friction between learning and teaching. Learning and Instruction, 9(3), 257e280. Verykios, V. S., Bertino, E., Fovino, I. N., Provenza, L. P., Saygin, Y., & Theodoridis, Y. (2004). State-of-the-art in privacy preserving data mining. ACM Sigmod

Record, 33(1), 50e57. Verykios, V. S., Elmagarmid, A. K., Bertino, E., Saygin, Y., & Dasseni, E. (2004b). Association rule hiding. IEEE Transactions on Knowledge and Data Engineering,

16(4), 434e447. Wang, S.-K., Hsu, H.-Y., Campbell, T., Coster, D., & Longhurst, M. (2014). An investigation of middle school science teachers and students use of technology

inside and outside of classrooms: Considering whether digital natives are more technology savvy than their teachers. Educational Technology Research and Development, 62(6), 637e662.

Warschauer, M., & Matuchniak, T. (2010). New technology and digital worlds: Analyzing evidence of equity in access, use, and outcomes. Review of Research in Education, 34(1), 179e225.

Witkin, H. A., Moore, C. A., Goodenough, D. R., & Cox, P. W. (1977). Field-dependent and field-independent cognitive styles and their educational impli- cations. Review of Educational Research, 47(1), 1e64.

Witkin, H. A., Oltman, P. K., Raskin, E., & Karp, S. A. (1971). Manual for embedded figures test, children's embedded figures test, and group embedded figures test. Palo Alto, CA: Consulting Psychologists Press.

Zaiane, O., & Luo, J. (2001). Web usage mining for a better web-based learning environment. In: Proceedings of the Conference on Advanced Technology for Education, Banff, Alberta (pp. 60e64).