Discussion 2

Computers in Human Behavior xxx (2012) xxx–xxx

Contents lists available at SciVerse ScienceDirect

Computers in Human Behavior

j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / c o m p h u m b e h

Critical thinking in E-learning environments

Raafat George Saadé a,⇑, Danielle Morin a,1, Jennifer D.E. Thomas b,2 a Concordia University, John Molson School of Business, Montreal, Quebec, Canada b Pace University, Ivan Seidenberg School of CSIS, New York, NY, USA

a r t i c l e i n f o

Article history: Available online xxxx

Keywords: E-learning Critical thinking Assessment Information technology

0747-5632/$ - see front matter � 2012 Elsevier Ltd. A http://dx.doi.org/10.1016/j.chb.2012.03.025

⇑ Corresponding author. Tel.: +1 514 848 2424; fax E-mail address: rsaade@jmsb.concordia.ca (R.G. Sa

1 Tel.: +1 514 848 2424; fax: +1 514 848 2824. 2 Tel.: +1 212 346 1569; fax: +1 212 346 1863.

Please cite this article in press as: Saadé, R. G., e 10.1016/j.chb.2012.03.025

a b s t r a c t

One of the primary aims of higher education in today’s information technology enabled classroom is to make students more active in the learning process. The intended outcome of this increased IT-facilitated student engagement is to foster important skills such as critical thinking used in both academia and workplace environments. Critical thinking (CT) skills entails the ability(ies) of mental processes of discern- ment, analysis and evaluation to achieve a logical understanding. Critical thinking in the classroom as well as in the workplace is a central theme; however, with the dramatic increase of IT usage the mechanisms by which critical thinking is fostered and used has changed. This article presents the work and results of critical thinking in a virtual learning environment. We therefore present a web-based course and we assess in which parts of the course, and to what extent, critical thinking was perceived to occur. The course contained two categories of learning modules namely resources and interactive components. Critical thinking was measured subjectively using the ART scale. Results indicate the significance of ‘‘interactivity’’ in what students perceived to be critical-thinking-oriented versus online material as a resource. Results and opportunities that virtual environments present to foster critical thinking are discussed.

� 2012 Elsevier Ltd. All rights reserved.

1. Introduction

One of the primary aims of higher education in today’s informa- tion technology (IT) enabled classroom, is to make students more active in the learning process (Ibrahim & Samsa, 2009). The in- tended outcome of this increased IT-facilitated student engage- ment is to foster important skills such as critical thinking. Given the importance of information technology for critical thinking in learning, it is vital that we understand better the associated key factors related to: background of students, beliefs, perceptions and attitudes and associated antecedents, design related to IT in learning environments, and IT-aligned pedagogical considerations.

In order to learn in the academic environment as well as to per- form well later in the workplace, students need the skills to acquire and absorb knowledge efficiently and effectively. The acquisition, understanding, and use of knowledge require various learning strategies, meta-cognitive skills and the desire to use them. When it comes to scholastic and professional performance, critical think- ing is a key skill that individuals need in order to succeed (Johnson et al., 2010).

Critical thinking (CT) skills entails the ability(ies) of mental pro- cesses of discernment, analysis and evaluation (Ibrahim & Samsa, 2009) applied to information in order to achieve a logical final

ll rights reserved.

: +1 514 848 2824. adé).

t al. Critical thinking in E-learn

understanding and/or judgment. Critical thinking in the classroom is a central theme in the education field; however, with the dra- matic increase of IT usage for teaching and learning, the mecha- nisms by which critical thinking is fostered and used has changed. The potential of IT to create and exploit learning opportu- nities is endless and we have barely scratched the surface. There has been some studies reported on IT for learning and CT in higher education (Akyüz & Samsa, 2009; Krumbacak, 2007; Yang, 2008) but they are scarce. The body of knowledge is relatively small and limited in context and most of CT type of studies is related to elementary and high school education excluding the context of IT usage and CT (Marin & Halpern, 2011).

Considering our information society today, critical thinking is regarded as the most important skill in order to discern false, incomplete, obsolete, etc. information. The internet has become the open medium to hold all types of information. While our understanding of critical thinking has improved significantly in the last two decades, a range of views about its complex structure and many areas of uncertainty and disagreements still remain. Yang (2008) provides a good perspective on those issues between cognitive scientists, educational researchers and philosophers.

This article entails two important concepts: Web-based learn- ing; and Critical thinking. We therefore present herein a web- based course on the fundamentals of Information Technology at a university in Montreal, Canada, and we assess in which parts of the course, and to what extent, critical thinking was perceived to occur. The course contained two categories of learning modules namely resources and interactive components. There were five

ing environments. Computers in Human Behavior (2012), http://dx.doi.org/

2 R.G. Saadé et al. / Computers in Human Behavior xxx (2012) xxx–xxx

different activities and four resources. Critical thinking was mea- sured subjectively using the ART scale (Thomas, 2001). The study aimed at answering the following questions: (1) Do students understand the definition of critical thinking? (2) What is the effect of the learning modules on critical thinking? and (3) What is the relative contribution of the various learning modules on critical thinking skills requirements?

Considering the scarce body of research work on CT in virtual learning environments, this study provides two significant contri- butions: (1) At a macro level, obtain some understanding on CT in the online learning context (higher education), and (2) At a mi- cro level, identify the kinds of resources and activities that fosters/ require CT skills. With this knowledge, practitioners (teachers and online courses designers) can design and implement better online (web-based) courses by integrating learning tools to foster the development of CT skills.

This article starts by defining critical thinking (Section 2), and then goes on to link critical thinking and virtual learning environ- ments (Section 3). In Section 4, the methodology of this research study is described by elaborating on the study context (describing the online course as well), followed by an explanation on how CT was measured and then providing a map on our analysis of results strategy. In Section 5, we present and discuss the results. We finally conclude with recommendations to researchers and practitioners.

2. Critical thinking

According to the Foundation of Critical thinking: ‘‘Critical thinking is that mode of thinking – about any subject, content, or problem – in which the thinker improves the quality of his or her thinking by skillfully taking charge of the structures inherent in thinking and imposing intellectual standards upon them.’’ http://www.criticalthinking.org/articles/27thconf-keynote.cfm. Every day each one of us makes decisions, generates thoughts, draws conclusions and evaluates opinions. In general, people tend to perceive what they want to see, and disregard all facts and evi- dence which doesn’t associate with their way of perceiving things. A skilled critical thinker is the one who can acknowledge the differ- ence between logical reasoning and personal opinion. Critical thinking is an important factor of our life and thinking critically is considered a hard object to achieve. We all agree that CT is af- fected by our point of view and how we see things around us. For example, it’s very likely that the point of view and judgment about the same issue would be different between two people. The main goal of using CT in evaluating thoughts and ideas is nei- ther to ignore one’s personal life experiences nor to let him/her take the outcome as he/she sees without considering other aspects of the environment; on the contrary, the purpose is to generate a balanced action, that merges facts and the skills gained through various experiences, towards continual improvement (Akyüz & Samsa, 2009; Ayad, 2010). Critical thinking is a type of a cognitive ability that has a special importance in decision making and judg- ment processes (Chartrand, Ishikawa, & Flander, 2009).

Critical thinking stems from the ability of higher-order-thinking (HOT), which has been linked to deep learning. Deep learning can be defined as ‘‘the intention to extract meaning which produces active learning processes that involve relating ideas and looking for pat- terns and principles on the one hand (a holist strategy – Pask, 1976; Pask, 1988), and using evidence and examining the logic of the argument on the other (serialist).’’ The approach also involves monitoring the development of one’s own understanding (Entwistle, McCune, & Walker, 2000)’’. (Entwistle, 2000, p. 2). This definition, along with those advocated by Chickering and Gamson (1987), Dan- gel and Wang (2008), Bloom and Krathwohl (1956), and Anderson,

Please cite this article in press as: Saadé, R. G., et al. Critical thinking in E-learn 10.1016/j.chb.2012.03.025

Krathwohl, & Bloom, 2001, led to the definition adopted in other research which considered higher-order thinking skills such as: critical thinking, problem-solving, research, and creative idea generation, and team-building skills (communication skills, work coordination, and team cooperation – (Thomas, 2001)). These are the skills that students are expected to acquire through their university residency, and ultimately to take with them into their careers. Noll and Wilkins (2002) identified these skills as extremely pertinent for the information systems (IS) professional.

There are several definitions of critical thinking (and instru- ments for its measurement), and there seems to be some common grounds around the ideas of analysis, evaluation, inference, and interpretation of CT. Follman, Lavely, and Berger (1997) provides a comprehensive list and associated discussion. The General Edu- cation Critical Thinking Rubric used by faculty to assess students’ critical thinking at Northeastern Illinois University, NEIU, 2006, includes: identifying and explaining issues, distinguishing types of claims, recognizing stakeholders and contests, considering methodol- ogy, framing personal responses and acknowledging other perspec- tives, reconstructing arguments, interpreting content, evaluating assumptions, evaluating evidence and, evaluating inferences.

The Center for Critical Thinking, 2004, (Mandernach, 2006) de- fines, ‘‘Critical thinking is the intellectually disciplines process of actively and skillfully conceptualizing, applying, analyzing, synthe- sizing, and/ or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communica- tion, as a guide to belief and action.’’

A lot of research into critical thinking has focused on the devel- opment of instruments to ‘‘objectively’’ measure this skill based on these differences in understanding of what the term means. Re- search on the impact of course material and technologies, for the development of CT, is still plagued by the lack of consensus. This is due to the fact that there are a number of definitions, and instru- ments stated in various levels of complexities. In fact, most of these instruments are too complex and time-consuming to complete, whether onsite or online, thereby introducing a number of uncer- tainties and noise to the data captured.

In our study presented in this article, we define critical thinking with simpler constructs as follows: ‘‘critical thinking is the process of analysis, evaluation, inference, and interpretation of resources and activities (gathered via online experiences with course mate- rial).’’ Students were asked to identify the extent to which they felt various activities, resources and technologies (Thomas, 2001), were perceived to support their acquisition of these skills so de- fined, in an entirely virtual, online course.

3. Critical thinking and virtual learning environments

The aim of all instruction is to lead the learner to take on their own responsibility for learning, by acquiring the tools and skills for doing so. The advent of technology has permitted this learning to take place virtually, that is, outside the traditional brick-and-mor- tar hallowed halls of learning, in an electronic, distant environ- ment. This distance and electronic format poses challenges particularly in those aspects of learning that require more high- er-level learning (Saadé & Bahli, 2005; Saadé & Galloway, 2005). While computers are known to facilitate lower levels of learning, such as rote memorization of facts, measurable through electroni- cally administered and graded multiple-choice questions, yes/no, and true/false answers, the imparting and measurement of high- er-level cognitive skills is more vexing. These require more open- ended delivery and answers, and may be more problematic in an entirely online environment, notwithstanding the advances in technologies such as wikis, blogs, and discussion boards (Mac- Knight, 2000; Mandernach, 2006; Saadé, 2007, 2010; Saadé &

ing environments. Computers in Human Behavior (2012), http://dx.doi.org/

Table 1 ART in a virtual learning introductory computing course.

Online

Activities � Course readings – text, book author’s PowerPoint slides Y � Case analyses Y � Presentations – written N � Discussions – online Y � Discussions – offline Y � Feedback – students and instructor Y � Teamwork N

Resources � Discussion board, email Y � File exchange, course notes and project repositories Y � Internet Y � Textbook Y � Software – Visio, Word, PowerPoint Y

Y

Technologies � Home/work computer access Y

R.G. Saadé et al. / Computers in Human Behavior xxx (2012) xxx–xxx 3

Huang, 2009). As with the integration of all technology, merit is based more on the instructional design of the course than on the technology employed in, and of, itself.

It goes without saying that critical thinking is an important skill to foster in students engaged in the learning process, for both their professional and personal prosperity (Noll & Wilkins, 2002). An examination of the virtual learning environment literature does not however indicate consensus on how to foster this skill, nor with what modes of deliver, nor how demographics may mediate the outcomes. It is acknowledged that in the online environment, the challenge is even greater than in the traditional setting. Con- straints on the development of critical thinking in virtual environ- ments seems more daunting than in the traditional classroom setting, due to the lack in the body of knowledge regarding the integration of creative instructional strategies, such as constructiv- ist learning philosophies, active learning, team-based learning, and discussion using digital media. MacKnight (2000) expressed an important pitfall in that ‘‘we fall prey to modern communication media, which present a world where the prepackaging of intellec- tual positions and views is so ingenious that thinking seems unnecessary.’’ On the other hand, some researchers suggest online can actually help by giving students opportunities for mastery learning, removal of time-constraints for learning, self-paced learning, anonymous online discussion, etc. (Benson & Samara- wickrema, 2009; Saadé & Kira, 2009; Saadé & Otrakji, 2007).

Burgess (2009), found that integrating WebCT improved read- ing engagement and critical thinking skills and, Thomas and Morin (2010) found that critical thinking was supported by online case study instruction. A significant relationship between GPA’s and students’ perceived improvement in higher-order cognitive skills (HOCS) whereby, students with low GPA’s report a lower level of HOCS than students with high GPA’s, which diminishes as GPA in- creases. This is similar to what Thomas (2001) refers to as HOTS – higher-order thinking skills.

4. Methodology

This complex context (critical thinking in virtual environments) sets the backdrop for the study presented herein which surveys students’ perceptions of the critical thinking skills they may have acquired and/or used via an online course resources and activities. Table 1 below shows the activities, and resources used within the ART framework employed in the course.

Davis (1989) has shown that system use is tied to user’s percep- tions, while Keengwe (2007) and Koohang and Durante (2003) found that a relationship exists between students’ personal com- puter proficiency and students’ perceptions of the effect of com- puter technology to improve their learning. Song, Singleton, Hill, and Koh (2004) focused on students’ perceptions as a way to im- prove online or distance learning. Perceptions are, therefore, important considerations when integrating technology into learn- ing, especially virtual learning. Consequently, the survey used measures the subjective evaluation of the students’ use and/or development of critical thinking while interacting with the course resources, activities and technologies. In the following sections we discuss the case study used to implement the critical thinking study, followed by a description on how it was measured.

4.1. Case study context

The purpose of this research was to investigate what observa- tions could be made about students’ acquisition of one aspect of higher-order learning, namely critical thinking, from the perspec- tive of students’ perceptions and performance in an undergraduate online introductory computer literacy course. In such a course, stu-

Please cite this article in press as: Saadé, R. G., et al. Critical thinking in E-learn 10.1016/j.chb.2012.03.025

dents are required to demonstrate acquisition of lower level skills, such as remembering concepts as well as keystrokes in software, and additionally, how to think critically about the problems which the software is being used to solve. These skills are important for the careers for which they are being prepared, in an area in which obsolescence is an ongoing threat in the digital world, given the speed at which technology changes. Without these skills and the ability to adapt to new innovations in technology, the student will be severely disadvantaged.

Students from the John Molson School of Business, Concordia University, enrolled in an introductory undergraduate course ‘‘Fun- damentals of Information Technology and Business Productivity’’ were asked to complete the survey. Out of 958 students enrolled, four hundred and ninety (490) students completed the survey. Most students are asked to take this course as part of their Bachelors de- gree and during their first or second semester of their program. At the end of the semester students were given a link to the survey (which was to be completed online) and were told that their partic- ipation was on a voluntary basis. They were assured that: the infor- mation they provide will be strictly confidential and used only for statistical purposes; only the results of the statistical analysis will be used for academic and research purposes and that the purpose of the survey is to help the designer of the course to understand the role of critical thinking in the online course and to ameliorate the course by integrating appropriate tools for critical thinking.

The online course uses a web-based learning management sys- tem (LMS) designed with three subsystems: Resource subsystem; Human subsystem; and Implementation subsystem (see Fig. 1 – details are found in Saadé, Nebebe, & Mak, 2011). These three sub- systems comprise the totality of learning elements and learning processes.

The LMS includes tools that fall under three categories: (1) learning tools (resource, implementation or both subsystems), (2) assessment tools (resource, assessment, human or any combina- tion) and (3) support tools (human subsystem).

Due to limitation of space, we will present only some of the fea- tures and selected components of the LMS. In line with the archi- tecture shown in Fig. 1, the design of the LMS is based on defined instructional, administrative and management activities. We briefly present some of those features as follows:

� Generating a course by administrator, and assigning it to an instructor, � Instructor managing different roles: Different users such as

teaching assistants and content managers have different roles and different permissions to different components of the LMS.

ing environments. Computers in Human Behavior (2012), http://dx.doi.org/

Fig. 1. Knowledge architecture of the LMS.

4 R.G. Saadé et al. / Computers in Human Behavior xxx (2012) xxx–xxx

For example, at the beginning of a semester, an administrator gives authorizations for a professor to access a course. Professor then can manage content related to that course, such as define a TA for the course. � Delivering content via various modes of communications: Pub-

lishing content via announcements, lectures, and providing other relevant instructions. Students will be automatically reminded when learning content is published. � Interactive learning modules/tools: Interactive tools are the

most important components for students’ learning. These interactive tools can be standalones or connected to other components. The LMS provides many learning tools, such as EISEL, for students to understand course related material.

Please cite this article in press as: Saadé, R. G., et al. Critical thinking in E-learn 10.1016/j.chb.2012.03.025

� Structured and unstructured discourse: The LMS may provide a forum-based environment for students, TAs and instructors to communicate and discuss relevant subject matter and cases with each other. Participation is monitored by the LMS. � Reporting: In the LMS, academic report services help instructors

to create a feedback mechanism for students and allows stu- dents to check their activities compared to the average of the group. � Support tools: Synchronous as well as asynchronous environ-

ments are created through the use of support tools to foster questions and answers. This is an integral part of the learning activity. For example, a question center provides an environ- ment for students/TAs/professor(s) to collaboratively generate, answer and evaluate question-answer sets.

ing environments. Computers in Human Behavior (2012), http://dx.doi.org/

R.G. Saadé et al. / Computers in Human Behavior xxx (2012) xxx–xxx 5

Based on the pedagogical design, the LMS can include any num- ber of available learning, assessment and support tools. The follow- ing is a description of the type of tools used in the course used for this study:

(1) Learning tools include a set of learning objects (with mea- surable learning outcomes), such as the educational infor- mation system for enhanced learning (EISEL). EISEL is the most mature and used tool in the LMS. Learning with EISEL, students go through the three system-guided following steps (repeated for many teacher defined subject matter): Pre-Test, Review resources, Practice related content, and Post-Test. Students first are evaluated on how much they know in a specific topic via the pre-test. They are then allowed to review and study the material related to the topic. When they are ready, students are allowed to practice the content using an interactive random generator multiple- choice and true-or-false engine. When students feel that they achieved the learning goals of the specific topic, they then complete a post-test to evaluate how much they have learned in this process. The system then opens access to the next topic or subject matter.

(2) Assessment in the LMS includes tools for formative assess- ment, summative assessment, and self-assessment. These three types of assessments are used as quizzes (throughout the semester with the goal to ensure that students actually read the course content as per the suggested schedule), as practice (such as in EISEL and pre-practice exam), and as summative with the goal to test knowledge gained.

(3) Support to students is done through an innovative central- ized question center with private and public zones that may also be conFigured to operate in synchronous and asyn- chronous modes. The dynamic question center is an asyn- chronous environment where students ask questions that they have to categorize. Based on the category of question, different teaching assistants (content, technical, professor, etc. . .) are assigned to the categories to answer. The questions can be labeled private (only student and professor sees) or public (everybody sees the question and answer). Ques- tion–answer sets can also be rated for importance which con- trols the first page content view. Filters are easily available on the first page. Vtutor is the synchronous part of the support tools. Using VTutor office hours, unstructured tutorials and structured sessions can be held and even recorded. The Course Reporting Tool is a knowledge-base connected to an email server which sends messages and reports based on rules. The CRT runs in real-time throughout the semester.

4.2. Measuring critical thinking

There are a number of ways to measure critical thinking. In a typical face-to-face classroom, one way is to use the course evalu- ation survey for quantitative types of responses. Another way can be by asking students about their involvement in discussions through qualitative pen-and-paper feedback (Noruzi, Hernández, & Rahimi, 2010). However, in online courses, measuring critical thinking can be done through online surveys or for qualitative feedback via discussion forums. Another way to measure the CT is through international critical thinking tests, which provides evi- dence of whether and to what extend people, are able to analyze, evaluate and judge information from different source of informa- tion. For example, the California critical thinking skills test (CCTST) is a test to measure performance and general intellectual ability. The CCTST determines results after assessing different factors with repeating the test and eliminating the non-relevant factors.

Please cite this article in press as: Saadé, R. G., et al. Critical thinking in E-learn 10.1016/j.chb.2012.03.025

Measuring critical thinking levels would benefit both the instructors and students. The educational system as a whole should focus on developing critical thinking applications. Although the CCTST is a very good tool, we decided not to use it because it is rather lengthy and students already have many activities to per- form in the course.

4.3. Survey

A survey methodology was used for data collection. The course was offered completely online without any face-to-face interaction with the professor or the teaching assistant. At the end of the semester, students were asked to respond to the survey as candidly as possible. The survey used in this study is based on an instrument developed by the third author. Students were instructed that there were no right or wrong answers and that we were interested pri- marily in their beliefs and perceptions about the course compo- nents and their experiences with the different tools for learning. A portion of the survey used in this study is presented in Table 2 below.

5. Discussion and analysis of results

In order to meet the purpose of this study and to respond the research questions posed earlier, a four-step analytical strategy was devised as followed:

� Demographics � Understanding the definition of critical thinking skills � Students ‘Perceived contribution of activities and resources � Correlation analysis

5.1. Demographics

There was a total of 985 students enrolled in the course and 490 of them completed the survey online for a response rate of 51.2%. Of those who completed the survey, 44% were female students. Most respondents (73.3%) were in the 20–23 age group, 17.4% in the 24–30 age group and 4% and 5.3% were in the below 20 and above 30 categories respectively. The average age is 22.7 years, while the median is 22. Also, 53.3% declared English as one of their first languages. This question was asked because the research took place in Montreal, Canada which is located in a French province in a bilingual country. We discovered that the first language did not have a significant impact on the understanding of the definition of critical thinking.

5.2. Understanding the definition

Fig. 2 below shows the distribution of the scores representing students’ understanding of the critical thinking definition used in the survey. We note that one-third of the students consider that they have a perfect understanding of the definition.

The above frequency distribution of the scores can be split into three categories: ‘perfect understanding’ (score = 10), ‘average understanding’ (scores from 6 to 9) and ‘limited understanding’ of the definition (scores below 5). Considering a cut-off limit of 6 (score of understanding definition), then we observed that approx- imately 86% of students claimed to have at least adequately under- stood the definition. What remains is less than 14% of students who represent the group with limited understanding of the defini- tion (see Table 3).

The mean score for ‘‘understanding of the critical thinking’’ def- inition is calculated at 7.84 and which represents a good level. On

ing environments. Computers in Human Behavior (2012), http://dx.doi.org/

Table 2 Partial survey instrument.

Please take a few moments to answer some questions regarding the areas of learning which you felt were supported by various technologies used in the course, as well as some demographic information.

� Gender: s Male s Female. � Age: s <20; s 20–23; s 24–30; s 30+. � Your first language: s English; s French; s Other.

In answering the questions below, please refer to the definition of critical thinking provided here: � Critical Thinking – analysis, inference, reasoning, evaluation, explanation, interpretation. Indicate to what extent do you understand this definition, 0 = I do not understand at all. . . 10 = I understand it very well. s

How did the following activities, resources and technologies assist you in your development of CRITICAL THINKING as a learning objective? The text book sA lot sSomewhat sNot at all The online book sA lot sSomewhat sNot at all EISEL sA lot sSomewhat sNot at all The assignments sA lot sSomewhat sNot at all The overall online system sA lot sSomewhat sNot at all The excel mini cases sA lot sSomewhat sNot at all The access mini cases sA lot sSomewhat sNot at all The quiz part of the course sA lot sSomewhat sNot at all The course material on the web sA lot sSomewhat sNot at all

Fig. 2. Understanding of ‘‘critical thinking’’ definition.

Table 3 Distribution of the scores of understanding the definition of critical thinking where 10 is very well and 0 is not at all.

Scores Frequency Percentage Combined %

10 163 33.3 33.3 9 66 13.5 52.9 8 79 16.1 7 72 14.7 6 42 8.6

5 25 5.1 13.8 4 10 2.0 3 13 2.7 2 10 2.0 1 5 1.0 0 5 1.0

490 100.0 100.0

Mean score 7.84 Standard Deviation 2.277

6 R.G. Saadé et al. / Computers in Human Behavior xxx (2012) xxx–xxx

this basis, we can continue towards our analysis of students’ perceptions.

5.3. Students perceived contribution of activities and resources

We have categorized our analysis based on learning activities and resources made available as support to the acquisition of knowledge. On the one hand, ‘‘Activities’’ is a category of learning requirements that a student would have to complete in order to gain knowledge and demonstrate that he/she has acquired that knowledge. These activities include assignments, two small pro- jects, one based on Microsoft Excel and the other on Microsoft Ac- cess, quizzes and an interactive tool called EISEL. ‘‘Resources’’ on the other hand are artifacts that students use in the course to sup- port access to content. Resources in this study are identified by the physical textbook (which students have the option to purchase from the bookstore), the digital version of the textbook made avail- able online and through the course website, the overall online sys- tem which includes instructions and navigational elements and any other material on the web directed via the course website.

The following Table 4 presents descriptive statistics and the dis- tribution of students’ perception of the contributions of activities/ resources to critical thinking skills, in percentages. This distribu-

Please cite this article in press as: Saadé, R. G., et al. Critical thinking in E-learn 10.1016/j.chb.2012.03.025

tion provides us with a measure of the perceived relative influence of course elements (activities/resources) on the development of critical thinking skills. The table also presents the Positive impact of each activities/resources defined as the total proportion of an- swers in the categories ‘Moderate’ and ‘A lot’.

The results in Table 4 clearly show that approximately 74% of the students bought the physical textbook and about half of them felt that its perceived contribution to critical thinking skills is mod- erate. Moreover, it is clearly evident that students felt the activities contributed more, compared to resources, to critical thinking. In terms of the strongest positive impact, Assignments are perceived as having contributed the most, followed by the Excel mini case project.

However, although activities in this study refer to learning activities, there are elements in the resources that are interactive on the web: overall online system and material on the web. We note that the contributions of those interactive components (found in the resources group) were close to that of the learning activities. The differences in scores between the interactive and non-interac- tive components of the resources triggered an important query: what then are the factors for critical thinking skills in online learn- ing environments. Consequently, Table 5 presents the characteris- tics of the course components. These characteristics are defined as a course component that can be any combination of: a learning activity; learning content; static material (tangible or digital); and interactive.

ing environments. Computers in Human Behavior (2012), http://dx.doi.org/

Table 4 Distribution of student perception (%) of contribution to critical thinking skills.

N Mean S.D. Not at alla (%) Moderateb (%) A lotc (%) Positive impact (%)

Activities Assignments 483 2.20 0.69 15.7 48.0 36.3 84.3 Excel mini case (project) 488 2.21 0.72 17.8 43.5 38.7 82.2 Access mini case (project) 483 2.18 0.70 18.6 45.1 36.3 81.4 Quiz 488 2.18 0.73 19.3 43.6 37.1 80.7 EISEL 486 2.13 0.72 20.6 46.3 33.1 79.4

Resources Textbook 361 1.95 0.69 26.6 52.1 21.3 73.4 Online book chapters 485 1.90 0.67 28.0 54.0 17.9 71.9 Overall online system 485 2.07 0.70 21.2 50.9 27.8 78.7 Material on the Web 489 2.10 0.68 18.6 52.6 28.8 81.4

Table 5 Characteristics of course components.

Learning activity Learning content Static (tangible and digital) Interactive

Activities Assignments U U Excel mini case (project) U U Access mini case (project) U U Quiz U EISEL U U

Resources Textbook U U Online book chapters U U Overall online system U U Material on the Web U U

Table 6 Student perception distribution (%) support for critical thinking skills.

Pedagogical basis Activities/resources Av. impact Difference from interactive

Interactive Assignments and Material on the Web 82.8 – Project-base Excel and access mini cases 81.8 1% Assessment EISEL and Quiz 80.1 2.7% Static Textbook and online book chapters 72.7 10.1%

R.G. Saadé et al. / Computers in Human Behavior xxx (2012) xxx–xxx 7

Table 5 shows that, as we started off in this article, there are two broad categories for an online course and they are the learning activities and the resources. However, our analysis of the data re- vealed that critical thinking is also dependent on two other charac- teristics, namely static content (tangible or digital) and interactive. Considering Tables 4 and 5, we find that interactive course compo- nents were scored approximately the same regardless of whether they were resources or learning activities. This may lead to the conclusion that interactivity is perceived to be a driver for critical thinking and not necessarily the nature of the component (as a learning activity or as a resource).

It is also worth noting an interesting phenomenon observed from Tables 4 and 5: The score of interactive components were similar (resources or learning activities); and at the same time, the mini cases (which can be considered as static material) scored higher than textbook and online book (static content). In other words, how come static material (in the case of mini cases) scored higher than other static material (such as textbook and online book chapters, when they are both documents)? This can be explained with the concept of ‘‘engagement’’. Engagement is a multidimen- sional construct encompassing affect, behavior, and cognition (Green, Liem, Martin, Colmar, & McInerney, 2012). More specifi- cally to our context is the notion of ‘‘interactive engagement’’ (IE) (Hake, 1997), which entails the methods designed to stimulate the engagement dimensions by promoting conceptual understand-

Please cite this article in press as: Saadé, R. G., et al. Critical thinking in E-learn 10.1016/j.chb.2012.03.025

ing through interactivity. This is done by heads-on (Continually- engaged) and/or hands-on (practice-engaged) interactivity where feedback is immediate in various forms such as discussion (peer to peer, and with instructors), using tools such as chats, email, and forums. In the present study, mini cases entail all the elements of engagement: (1) affect where students are motivated to solve the problem presented to them, (2) behavior where the resulting emotions energize engaged behavior (Slinner, Kindermann, Con- nell, & Wellborn, 2009), and (3) cognition where students engage in an intellectual quest to increase their learning capacity). At the same time, the online environment presenting a multitude of tools (message board, question center, forum, and email) for interactiv- ity among peers, instructor and teaching assistant, both create the perception of mini cases being interactive, as observed from the analytical results.

Furthermore, we looked at the course components from four different pedagogical perspectives as shown in Table 6. It is inter- esting to note from Table 4 that the scores of the course compo- nents (activities/resources) in each pedagogical basis (given in Table 6) are very close.

The scores of each pedagogical basis were calculated by the average of the scores (% positive impact) from the two course com- ponents they are made of. Table 6 shows that interactive compo- nents have the highest impact on CT while static components have the least with approximately 10.1% less impact.

ing environments. Computers in Human Behavior (2012), http://dx.doi.org/

Table 7 Differences in students’ perception of contribution of critical thinking skills.

Activities Resources

Assign. Excel Access Quiz EISEL Textbook OBook OOSystem WebMaterial

Activities Assign. No No No No Yes Yes No No Excel No No No Yes Yes Yes No Access No No Yes Yes No No Quiz No Yes Yes No No EISEL Yes Yes No No

Resources Textbook No No Yes OBook Yes Yes OOSystem No WebMaterial

Note: OBook = online book; OOS = overall online system.

Table 8 Correlation between the understanding of the definition of critical thinking and the perceived contribution of activities and resources.

Correlations

Activities Assignments 0.01958 Excel case (project) 0.0530 ACCESS case (project) 0.0580 QUIZ 0.0419 EISEL 0.0344

Resources Textbook �0.0878 Online book chapters 0.0096 OVERALL ONLINE SYSTEM 0.0462 Web material �0.0234

8 R.G. Saadé et al. / Computers in Human Behavior xxx (2012) xxx–xxx

An overall ANOVA on the five activities and four resources is performed and a p-value of 2.65 � 10�16 is obtained which indi- cates very strong significance of the differences in mean percep- tions. A Tukey–Kramer analysis is performed to identify which pairs of means are significantly different at the same level of signif- icance. Table 7 below indicates by ‘‘yes’’ the significant differences and by ‘‘no’’ those that are not. This table also differentiates be- tween the components identified as ‘Activities’ and those as ‘Re- sources’, and divides the relations by quadrants.

In the first quadrant (top-left), we can observe that among the five activities, there are no significant differences in average per- ception of contributions to critical thinking skills. In the second quadrant (top-right) significant differences between the ‘Text- book’, and the ‘Online Book’ and all of the ‘Activities’ components are observed. However, in the third quadrant (bottom-right), we can observe that among the four resources, there are some signif- icant differences in average perception of contributions to critical thinking skills, such as between ‘Online Book’ and ‘Web Material’, between ‘Online Book’ and ‘Overall Online System’ and also be- tween ‘Textbook’ and ‘Web Material’.

The results in Table 7 confirm our previous findings that the dif- ferences in perceived contribution of course components to critical thinking are significant with regards to learning activities and sta- tic content (regardless of being physical or digital). This phenome- non is observed in both the second and third quadrants.

5.4. Correlation analysis

In order to avoid the possibility that the students’ perceived contributions of different elements of the course could be influ- enced by their level of understanding of the definition, the follow- ing Table 8 gives the correlations between the understanding level

Please cite this article in press as: Saadé, R. G., et al. Critical thinking in E-learn 10.1016/j.chb.2012.03.025

and the perceived contribution of the activities and resources to the acquisition of critical thinking skills. This table was generated in an attempt to understand whether a students’ understanding of the definition has any relationship with their perception. It is important that the instrument can assess the contribution of the activities/resources without being influenced by the proposed understanding of the definition.

We observe that all correlations are close to zero and none of them are significant at the 5% level, supporting the assumption that understanding of the definition is not related (at least linearly) to its use for evaluation of contributions.

6. Limitations and recommendations for future research

We acknowledge that implications are confined to the limits of their interpretation. Consequently, we acknowledged the limita- tion of this study in this section, followed by suggestions for future research. We identify three primary areas that limit the study methodology and interpretation of the results and address these limitations accordingly. These areas are: the respondents, instru- ment used, and online learning related issues.

Usually in similar type of studies, the respondents of question- naires/surveys always present a limitation to the interpretation of the results. The setting in which the study took place is always a factor that should be considered. In the present context, students who participated in the study were taking an online course where face-to-face contact is not part of the instructional design and the survey was completed at the end of the semester online. Although the survey was not mandatory, students’ conditions during the time they were completing it was not controlled. In other words, while completing the survey, respondents may have been in a café, at home watching television or have been studying at school all night. Moreover, the convenience sample of respondents, though addressed by statistical procedures, is problematic on many levels, including bias, size, and demographic controls. There- fore generalizing the findings to other schools, demographic groups or educational levels may be limited.

As a result, there is a great need to identify the boundary condi- tions of the methodology and its interpretation in terms of demo- graphic variables such as age, gender, ethnicities, and cognitive style. For example, the survey timing may be strategically executed to minimize variability. Other possible considerations for future re- search would be to attempt to measure students’ attention and fo- cus while they are completing the survey.

Evidence of critical thinking has been associated with various learning approaches but self-report measures used in this study are not themselves direct indicators of critical thinking in online learning environments. Further, though we have continued to val- idate the instrument, it cannot be assumed that the questions

ing environments. Computers in Human Behavior (2012), http://dx.doi.org/

R.G. Saadé et al. / Computers in Human Behavior xxx (2012) xxx–xxx 9

themselves adequately addressed the constructs for which they were designed.

Regarding online learning, the first and foremost limitation to studies similar to the present one is the design of the online learn- ing system under which the course is running. There are many fac- tors that influence student’s perceptions such as the qualities of the interface, the level of interactivity and the usability of the sys- tem. Learning management systems are complex with regards to making them promote those different methods of learning such as higher order thinking skills, memorization, assessment and others.

Considering the above, limitations are just opportunities for fu- ture research. To that effect, research in online learning and critical thinking has many opportunities in the areas of assessment of crit- ical thinking in online environments, design of interactive learning tools that foster critical thinking, and enhanced online methodolo- gies for self assessment. For example, online learning provides the opportunity to capture objective data such as number of logins to the course, time spent in a specific location in the course website and discussion threads in collaborative environments where stu- dents are helping each other solve a problem. These are the oppor- tunities for research that information technology presents us with to enhance our assessment of critical thinking in online environments.

7. Conclusions

In this article we reported the results of a study on critical thinking in online courses. The study entailed two primary course categories: learning activity and resources course components. The study was conducted using a survey methodology approach with 490 participants taking a first year university course in Montreal, Canada. Critical thinking was measured subjectively using the ART scale. The study aimed at understanding: (1) to what extent students comprehend the definition of critical thinking; (2) the ef- fect of the course components on CT; and (3) the relative contribu- tion of the various learning modules on CT.

Initial testing of understanding confirmed that the definition is satisfactory for usage in the survey. Results clearly indicate that the relationship between students’ perception of learning activities and critical thinking is stronger than that with resources. More specifically, interactive components regardless of their nature (learning activity/resources) were perceived to contribute to CT more than other identified types such as static content, assessment and project-based.

In the context of this study, results indicated that assessment in an online environment, which is interactive, was perceived to con- tribute to critical thinking to a lesser extent than other learning activities, even though the quizzes were summative and EISEL was formative. They both were scored practically equally, thereby indicating the strength of the goal of the activity in it overwhelm- ing its nature (being interactive). EISEL is a tool that allows stu- dents to continue practicing to answer domain specific questions with immediate feedback. Questions come from a fixed pool and are randomized five questions at a time. Therefore, the same ques- tion may appear multiple times. The scoring of performance is the running average to encourage students to use it all semester long. EISEL has been in use in many other courses and has shown to be very effective in helping students learn due to the facts that it is highly interactive. This is why the results were counter intuitive and were not expected. However, it seems that students do not see any contribution to CT when it comes to assessment, regardless of whether it is formative or summative assessment.

Putting everything together, it seems that the world of books for learning is fading away. Our experience, and our attempt in this

Please cite this article in press as: Saadé, R. G., et al. Critical thinking in E-learn 10.1016/j.chb.2012.03.025

study to quantify the relative importance of books, whether if they exist in physical or digital form, indicates that for any content to be perceived as contributing to thinking it must be presented in an interactive mode. When it comes to critical thinking, interactivity was shown to rank at the level of (actually slightly higher) assign- ments to solve a business problem using software (EXCEL and AC- CESS) as a tool.

Today’s students, an interactive environment is very important for their learning. It seems that today’s websites such as those of social media who are highly interconnected and interactive are the primary educational behavior agents to our university stu- dents. We can extract from our study, that critical thinking in on- line learning environments, is the results of interplay between content chunks (as opposed to book), interactivity and design (pedagogy and system).

This study reveals critical research and practical insights for fu- ture studies/projects – we present three recommendations in each case. From a research perspective, it is important to (1) address the design factors and framework that allow for maximum learning utilization of interactive learning environments, (2) synthesize at a more fundamental (psychological) level critical thinking and map it to interactive learning environments, and (3) study the influence of other higher order thinking skills on virtual learning environments. From a practical perspective, designers and teachers of online courses have one primary goal, which is to enhance the student’s learning experience – this includes learning and behav- ior. To that effect, they need to (1) integrate more interactive com- ponents into the course activities, (2) draw on social media principles to keep students connected to each other, and (3) tie 1 and 2 to critical thinking activities.

References

Akyüz, H. I., & Samsa, S. (2009). Critical thinking skills of preserve teachers in the blended learning environment. International Journal of Human Sciences, 6(2), 538–550.

Anderson, L. W., Krathwohl, D. R., & Bloom, B. S. (2001). A taxonomy for learning, teaching and assessing: a revision of Bloom’s Taxonomy of educational objectives: Complete edition. New York: Longman. 2001.

Ayad, A. (2010). Critical thinking and business process improvement. Journal of Management Development, 29(6), 556–564.

Benson, R., & Samarawickrema, G. (2009). Addressing the context of e-learning: Using transactional distance theory to inform design. Distance Education, 30, 5–21.

Bloom, B. S., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational goals, by a committee of college and university examiners. Handbook 1: Cognitive Domain. New York: Longman.

Burgess, M. L. (2009). Using WebCT as a supplemental tool to enhance critical thinking and engagement among developmental reading students. Journal of College Reading and Learning, 39(2), 9–33.

Chartrand, J., Ishikawa, H., Flander, S. (2009). Critical thinking means business: Learn to apply and develop the NEW #1 workplace skill. Pearson Education. <http://www.talentlens.com/en/downloads/whitepapers/Pearson_TalentLens_ Critical_Thinking_Means_Business.pdf>, <http://www.talentlens.com/en/downloads/ whitepapers/Pearson_TalentLens_Critical_Thin> Retrieved 10.12.11.

Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin, 39(7), 8–12.

Dangel, H. L., & Wang, C. X. (2008). Student response systems in higher education: Moving beyond linear teaching and surface learning. Journal of Educational Technology Development and Exchange, 1(1), 93–104.

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance in information technology. MIS Quarterly, 13(3), 319–340.

Entwistle, N. J. (2000). Promoting deep learning through teaching and assessment: Conceptual fameworks and educational contexts. Paper presented at the TLRP conference, Leicester, England, November, 1–11. <http://www.etl.tla.ed.ac.uk/ docs/entwistle2000.pdf> Retrieved 10.12.11.

Entwistle, N. J., McCune, V., & Walker, P. (2000). Conceptions, styles and approaches within higher education: Analytic abstractions and everyday experience. In R. J. Sternberg & L.-F. Zhang (Eds.), Perspectives on cognitive, learning, and thinking styles. Mahwah, N.J.: Lawrence Erlbaum.

Follman, J., Lavely, C., & Berger, N. (1997). Inventory of instruments of critical thinking. Informal Logic, 18(2–3), 261–267.

Green, J., Liem, G. A. D., Martin, A. J., Colmar, H. W., & McInerney, D. (2012). Academic motivation, self-concept, engagement, and performance in high school: Key processes from a longitudinal perspective. Journal of Adolescence. http://dx.doi.org/10.1016/j.adolescence.2012.02.016.

ing environments. Computers in Human Behavior (2012), http://dx.doi.org/

10 R.G. Saadé et al. / Computers in Human Behavior xxx (2012) xxx–xxx

Hake, R. R. (1997). Interacive-engagement versus traditional methods: A six- thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66(1), 64–74.

Johnson, C. S., Dweck, S. C., Chen, S. F., Stern, L. H., Ok, S., & Barth, M. (2010). At the intersection of social and cognitive development: Internal working models of attachment in infancy. Cognitive Science, 34, 807–825.

Keengwe, J. (2007). Faculty integration of technology into instruction and students’ perceptions of computer technology to improve student learning. Journal of Information Technology Education, 6, 169–180.

Koohang, A., & Durante, A. (2003). Learners’ perceptions toward the web-based distance learning activities/assignments portion of an undergraduate hybrid instructional model. Journal of Information Technology Education, 2, 105–113.

Krumbacak, G. (2007). Identify research priorities and needs for mobile learning technologies in open and distance education: A delphi study. International Journal of Teaching and Learning in Higher Education, 19(3), 216–227.

MacKnight, C. B. (2000). Teaching critically about critical thinking through online discussions. EDUCAUSE Quarterly, 4, 38–41.

Mandernach, B. J. (2006). Thinking critically about critical thinking: Integrating online tools to promote critical thinking. Critical Thinking – Insight: A Collection of Faculty Scholarship, 1, 41–50.

Marin, L. M., & Halpern, D. F. (2011). Pedagogy for developing critical thinking in adolescents: Explicit instruction produces greatest gains. Thinking Skills and Creativity, 6(1), 1–13.

Noll, C. L., & Wilkins, M. (2002). Critical skills of IS professionals: A model for curriculum development. Journal of Information Technology Education, 1(3), 144–154.

Northeastern Illinois University (Spring 2006). Measuring critical thinking. Toolkit Newsletter, 4(4).

Noruzi, M. R., Hernández, J. G., & Rahimi, G. R. (2010). An exploration of critical thinking necessities, barriers and CAT MAGIC notion. Interdisciplinary Journal of Comptemporary Research in Business, 2(1), 92–104.

Pask, G. (1976). Styles and strategies of learning. British Journal of Educational Psychology, 46, 128–148.

Pask, G. (1988). Learning strategies, teaching strategies and conceptual or learning style. In R. R. R. R. Schmeck (Ed.), Learning strategies and learning styles. New York: Plenum Press.

Saadé, G. R. (2007). Dimensions to perceived usefulness: Towards an enhanced assessment, decision sciences institute – Decision sciences. Journal of Innovative Education, 5(2).

Saadé, R. (2010). Cognitive mapping decision support for the design of web-based learning environments. International Journal of Web-Based Learning and Teaching Technologies, 5(3), 36–53.

Please cite this article in press as: Saadé, R. G., et al. Critical thinking in E-learn 10.1016/j.chb.2012.03.025

Saadé, R., & Bahli, B. (2005). The impact of cognitive absorption on perceived usefulness and perceived ease of use in on-line learning: An extension of the technology acceptance model. Information and Management, 42, 317–327.

Saadé, R., & Galloway, I. (2005). Understanding the acceptance of multimedia applications for learning. Issues in Informing Science and Information Technology, 2, 287–296.

Saadé, R., & Huang, Q. (2009). Meaningful learning in discussion forums: Towards discourse analysis. Issues in Informing Science and Information Technology, 6, 87–99.

Saadé, R., & Kira, D. (2009). Computer anxiety in E-learning: The effect of computer self-efficacy. Journal of Information Technology in Education, 8, 177–191.

Saadé, R., Nebebe, F., & Mak, T. (2011). Knowledge management systems development: theory and practice. Interdisciplinary Journal of Information, Knowledge and Management, 6, 35–72.

Saadé, R., & Otrakji, C. (2007). First impressions last a lifetime: Effect of disorientation and cognitive load. Computers in Human Behavior, 23(1), 525–535.

Slinner, E. A., Kindermann, T. A., Connell, J. P., & Wellborn, J. G. (2009). Engagement as an organizational construct in the dynamics of motivational development. In K. Wentzel & A. Wigfield (Eds.), Handbook of motivation in school (pp. 223–245). Malwah, NJ: Erlbaum.

Song, L., Singleton, E. S., Hill, J. R., & Koh, M. H. (2004). Improving online learning: Student perceptions of useful and challenging characteristics. Internet & Higher Education, 7(1), 59–70.

Thomas, J. D. E. (2001). Technology integration and higher-order learning. In Proceedings of Conference in Advanced Technology in Education Conference, Banff, Calgary, Canada, May.

Thomas, J. D. E., & Morin, D. (2010). Technological supports for onsite and distance education and students’ perceptions of acquisition of thinking and team- building skills. International Journal of Distance Education Technologies, 8(2), 1–13.

Yang, C. Y. (2008). A catalyst for teaching critical thinking in a large university class in Taiwan: Asynchronous online discussions with the facilitation of teaching assistants. Educational Technology Research and Development, 56, 241–264.

Further reading

Carlson, S. (2011). Student Perceptions of critical thinking Instructional methods: findings in a business curriculum. In Proceedings of the allied academies international conference. Academy of educational leadership. Orlando, USA, January 1.

ing environments. Computers in Human Behavior (2012), http://dx.doi.org/