research

Student evaluations of the credibility and argumentation of online sources

Miika Marttunena , Timo Salminenb , and Jukka Utriainenb

aDepartment of Education, University of Jyv€askyl€a, Jyv€askyl€a, Finland; bFinnish Institute for Educational Research, University of Jyv€askyl€a, Jyv€askyl€a, Finland

ABSTRACT This study investigated upper secondary school students’ skills in evaluating the credibility and argumentative content of a blog text and a YouTube video. Both sources concerned child vaccin- ation, the blog text opposing and the YouTube video supporting it. Students rated each source as credible, fairly credible or non-credible, justified their ratings, and analyzed the argumentation of both sources. Their justifications were analyzed for trustworthiness and expertise and their argu- ment analyses for identification of the main position of the source and the reasons supporting it. Students’ justification skills proved fairly weak, and they also struggled with recognizing unbal- anced argumentation. Students’ skill in analyzing the argumentation used in the sources also proved inadequate, especially in the blog text task. Overall academic achievement significantly predicted students’ credibility evaluation and argument analysis skills. The results suggest that greater emphasis should be placed on tasks involving the interpretation and analysis of online information.

ARTICLE HISTORY Received 10 June 2020 Revised 31 March 2021 Accepted 3 May 2021

KEYWORDS Online sources; multimodality; upper secondary school; argumentation; critical literacy; credibil- ity evaluation

Introduction

The internet offers a favorable environment for disseminat- ing inaccurate and biased information, including misinfor- mation and disinformation (Walsh, 2010). This is possible as anyone can publish freely on the internet without subjec- tion to any quality control mechanisms (Zubiaga et al., 2018) or traditional gatekeeper models (Metzger et al., 2010) that have been instituted to uphold the credibility standards of online information. When defining bias Walton (1991) focuses particularly on the argumentative content of the source. According to him, the basic idea of bias is a lack of appropriate balance or neutrality in argumentation, that is, a tendency to consistently and unduly favor one side over the other, ignoring the arguments on both sides. Further, Walton stresses that bias also involves an absence of critical doubt, which appears in a failure to push the viewpoint one supports, thereby implying a departure from reasoned argu- mentation and critical discussion. Another reason for bias may be that an arguer has some ulterior intentions or some- thing to gain based, e.g., on some ideological or commercial interest (Kammerer et al., 2015).

With the rapid growth of electronic commerce, for example, where purchasing decisions are increasingly made in online environments, lack of objectivity in the informa- tion given about merchandise can be misleading (Xiao & Benbasat, 2015), exposing shoppers to commercial manipu- lation. Inaccurate or biased information is also harmful to citizens and society as it threatens democratic political proc- esses and the democratic values that shape public policies in

such life domains as health, science, and finance (Broniatowski et al., 2018; Kahne & Bowyer, 2017). For example, internet searches are commonly made for health- related information (Chen et al., 2018) despite the fact that many health and nutrition websites are inaccurate or com- mercially biased (see Kammerer et al., 2016) and may even contain distorted information (Chen et al., 2018). Thus, important decisions pertaining to health issues may be more strongly affected by biased and flawed argumentation advanced by manufacturers than by rational argumentation and objective comparison of alternatives.

Because biased online information may have harmful effects, protecting people from deceitful information is important. As digitalization increasingly penetrates all areas of life, the role of schools in promoting the capacity of young people to use and identify accurate and useful information becomes increasingly important (Kahne & Bowyer, 2017). Promoting students’ skills in evaluating the credibility of online information and analyzing the quality of its argumen- tation is of particular importance in helping students discrim- inate between reliable and unreliable online sources. Namely, research has shown analytical argumentation and thinking to be important for the critical evaluation of online information. Pennycook and Rand (2019) found that the ability to discern partisan fake news from real news was associated with the propensity to engage in analytical reasoning. Further, Bronstein et al. (2019) found that people over 18 with reduced analytic thinking were prone to believe in fake news.

The present study investigated upper secondary school students’ ability to evaluate the credibility and argumentative

CONTACT Miika Marttunen miika.marttunen@jyu.fi Department of Education, University of Jyv€askyl€a, PO Box 35, Jyv€askyl€a FI-40014, Finland � 2021 The Author(s). Published with license by Taylor & Francis Group, LLC This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://creativecommons.org/licenses/by-nc-nd/4. 0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way.

THE JOURNAL OF EDUCATIONAL RESEARCH 2021, VOL. 114, NO. 3, 294–305 https://doi.org/10.1080/00220671.2021.1929052

content of two online social media resources familiar to young people, i.e., a blog text, and a YouTube video. The two sources represented opposite viewpoints on a current health-related topic, child vaccination. To find out to what extent students paid attention to source bias, both sources were selected to include flawed and unbalanced argumenta- tion favoring only one side of the question of whether it is wise to vaccinate children or not.

Literature review

Evaluation of credibility and argumentation

Credibility can be defined as the believability of a source or message and has two primary dimensions: trustworthiness and expertise (Flanagin & Metzger, 2008). Tseng and Fogg (1999) define expertise as knowledgeable, experienced, and competent and trustworthiness as well-intentioned, truthful, and unbiased.

When evaluating the credibility of online sources for expertise, various source information and source features (Bråten et al., 2018) are often of interest, including docu- ment type, when and why the document was published, whether the author’s or organization’s contact information is given, and the author’s affiliation and qualifications (Metzger, 2007; Paul et al., 2017). When evaluating trust- worthiness, various content features of the web site are important, e.g., how objective, up to date and in-depth is the information provided (Metzger, 2007) and whether there are indications of source bias. Source bias may appear in unbalanced argumentation when only one side of the issue is covered and favored over the other, or in some hidden intentions or motives of the writer related to e.g. political or commercial interests (Kiili et al., 2018; Walton, 1991).

Evaluating the argumentative quality of a text requires the reader to pay attention to its argumentative structure. Toulmin (1958) defines the following main argumentative elements for analyzing the structure of argumentative texts: a claim is the argument’s conclusion, data are the facts cited as premises for the claim, and a warrant is a general operat- ing principle bridging the data and the claim (Fulkerson, 1996, p. 18; Nussbaum, 2011). In an argumentative text, a claim or a conclusion presents the writer’s main position or message. In a high-quality argumentative text, the main claim is clearly stated and supported by relevant and suffi- cient premises (Fulkerson, 1996).

Argumentation plays an important role when evaluating the trustworthiness of online sources, as one should be able to assess whether the premises underlying the views pre- sented are relevant and sufficient (Fulkerson, 1996), whether the text harbors hidden motives (e.g., economic or ideo- logical), and whether the premises for and against the claim receive equal weight in the source.

The terms grounds (Osborne et al., 2004), evidence (Forzani, 2016) and reason (Means & Voss, 1996) are also used when referring to elements of an argument justifying a claim or a conclusion. In this study, the term “reason” is used to refer to elements of arguments supporting the claim or conclusion, and “argumentation perspective” to refer to

categories of reasons representing a specific perspective, such as a financial, ethical or health-related perspective. Xu and Yao (2015) state that the justifications or reasons repre- senting various argumentation perspectives indicate the breadth of argumentation, a further necessary criterion for high-quality argumentation. Breadth of argumentation, or space of debate as van Amelsvoort et al. (2007) put it, is defined as the number of topics and subtopics mentioned that relate to the issue discussed: The higher the number of topics and subtopics related to the arguments used in the discussion, the higher the argumentative quality of the discussion.

Previous research on evaluation of online information

It is easily assumed that because young people are fluent in using the internet and social media, they are equally savvy about what they find there. However, recent research results show the opposite. Coiro et al. (2015) found that 7th graders were unable to justify whether the author of an online source was an expert in the field the source represented. Further, the Stanford History Education Group (2016) showed that students at different educational levels had poor skills in evaluating information delivered through social media channels, and thus were at risk for being duped by disinformation, including flawed argumentation. For example, Miller and Bartlett (2012) in their survey (n¼ 509) of primary and secondary school teachers reported that stu- dents commonly struggle to recognize biased information and may use information that has been deliberately pro- duced to be misleading. Similarly, Kiili et al. (2018) found that only about half of their sample of 6th grade elementary school students questioned the credibility of the commercial online source and only about a fifth fully recognized com- mercial bias. Similarly, Fogg et al. (2003) asked somewhat older regular web users (n¼ 2 684; average age 39.9) to comment on the credibility of preselected websites repre- senting various content categories. Only 11.6% of the partic- ipants’ credibility comments referred to information bias.

Kiili, Laurinen, and Marttunen (2008) found that upper secondary school students seldom evaluated the credibility of online information and that when they did, they mainly attended to such aspects as the publisher, author or expert interviewed in the source. Cognitively demanding evaluation strategies, such as the evaluation of argumentation or com- parison of texts, were rarely used. At college level, McClure and Clink (2009) examined English composition students’ research essays to determine the amount of attention stu- dents gave to analyzing and crediting online sources, par- ticularly their timeliness, authority, and bias. They found that students not only struggled with understanding bias but also ignored it when it hampered their writing.

Previous research has shown that students’ critical think- ing has a positive association with their academic achieve- ment (e.g., Fong et al., 2017; Ghanizadeh, 2017; Marin & Halpern, 2011). Similarly, university applicants’ argumenta- tive writing skills predicted their academic achievement in university (Preiss et al., 2013). These findings stress the

THE JOURNAL OF EDUCATIONAL RESEARCH 295

importance of academic achievement in terms of students’ argumentation and critical thinking. In this study we explored this issue further by studying whether upper sec- ondary school students’ academic achievement is associated with their credibility evaluation and argument analysis skills.

Prior studies also suggest that girls perform better than boys in critical evaluation of online sources (Forzani, 2016; Taylor & Dalal, 2017), prior knowledge on a topic enhances students’ critical evaluation of sources on similar topics (Forzani, 2016), and that offline and online reading skills are intertwined but their relationship remains unclear (Leu et al., 2013; Kiili et al., 2018). Further, prior knowledge occupies an important role in readers’ activities during off- line reading comprehension (e.g., Means & Voss, 1985) but in online reading contexts its role is more complex (Coiro, 2011). To explore these issues further, this study investigated to what extent students’ performance in the critical and argumentative evaluation of online sources can be explained by their gender, prior knowledge on the topic, and offline reading fluency.

Aim and research questions

Many studies have shown that students commonly struggle in evaluating the credibility of online sources and recogniz- ing bias. Deficiency in credibility evaluation is a serious problem as, e.g., bloggers and video bloggers have a lot of influence particularly on young people who often adopt their ideas despite the absence of guarantees on the trust- worthiness, objectivity, and unbiased nature of information disseminated in social media. Thus, further knowledge is needed on how students evaluate online sources low in credibility.

Students’ recognition of bias in online information has been widely investigated. However, to our knowledge, no studies exist in which students have evaluated the credibility of argumentatively unbalanced online sources representing different modalities (text, video) and addressing the same topic from opposite viewpoints (for and against). Such a comparative study may further understanding of whether young people differ in how they deal with unreliable infor- mation presented through different social media and whether they are more likely to be misled by erroneous and unobjective information presented via one type of social media rather than another. Further, this information is needed in developing pedagogy to help future citizens recog- nize inaccurate information and fake news (Vosoughi et al., 2018).

This study investigated how students deal with two online social media sources frequently used by young people: a written blog and a multimodal video. Upper secondary school students were given a blog text and a YouTube video, one supporting and the other opposing the same currently contested topic (child vaccination). These online sources were deliberately selected for their unbalanced argumenta- tion on the topic and hence questionable credibility. Both sources lacked credibility also in that the author of both sources was not an expert in the field (vaccination) and that

they did not contain any references to support the presented arguments. Students’ skills in evaluating these online sources for argumentative content and credibility were explored. Also examined was whether students’ skills can be explained by gender, prior knowledge on the topic, reading fluency, and previous academic achievement. The research ques- tions were:

1. How skilled are upper secondary school students in evaluating the credibility of unreliable online sources? 1.1 How do students rate the credibility of unreli-

able sources? 1.2 How do students justify the lack of credibility of

unreliable sources? 1.3 Do students identify unbalanced argumentation?

2. How skilled are upper secondary school students in analyzing the argumentative content of unreliable online sources?

3. To what extent are students’ skills explained by gender, prior knowledge on the topic, reading fluency, and pre- vious academic achievement?

Method

Subjects

Participants were 404 upper secondary school students (94.6% aged 17) from seven schools in Finland. There were 60.4% female students and 38.9% male students (missing information 0.7%). Most (79.2%) of the participants were students in three large urban upper secondary schools and the rest (20.8%) students in four small and middle-sized schools in semi-urban and rural areas.

A written consent to participate in the study was obtained from the parents of underage students. Our research was approved by all the participating upper second- ary schools and included only minimum risk for partici- pants. According to the Finnish standard for institutional review board approvals, a statement by the Ethical Board of the University was not required.

Study context

This study is a part of a larger research project on Finnish upper secondary school students’ argumentative online inquiry competencies. In the project, students completed an online task in which they composed an argumentative letter to imaginary Finnish members of parliament on the advan- tages and disadvantages of child vaccination aimed at informing their legislative work on this issue. The online task followed an online research and comprehension frame- work (Leu et al., 2013), including sub-tasks on the informa- tion search, evaluating, synthesis and communication. This article reports the results of the analyses of a sub-task on students’ evaluations of the credibility and argumentative content of two online sources utilizing the evaluation com- ponent of the framework by Leu et al. (2013).

296 M. MARTTUNEN ET AL.

The two sources, a blog text and a multimodal YouTube video, viewed child vaccination from opposite standpoints. The Finnish blog text (3 pages, 967 words) presented several potential disadvantages of vaccination by appealing to vari- ous side effects of vaccines, the financial benefits of vaccin- ation for the pharmaceutical industry, and emotional arguments by describing situations in which parents have reported harmful effects of vaccines on their children. The blog text concluded that children should not be vaccinated. In contrast, the YouTube video presented arguments for vaccination, concluding that it is wise to vaccinate children. In the video, two US comedians (Penn and Teller) argued for vaccination by means of humor, music, sound effects, and visual demonstrations such as gestures and motion. The comedians also cited statistics on both the effectiveness of vaccinations and their minimal injurious effects and described the potential negative effects of not vaccinating children. The video, in English with Finnish subtitles, lasted for 1.5minutes.

Data collection

Prior knowledge and reading fluency In this study, students’ prior knowledge on the child vaccin- ation was assessed with ten statements, three of which were correct and seven false. The statements were selected based on both current scientific knowledge and common misinfor- mation disseminated by anti-vaccination movements. To strengthen content validity, two university lecturers in health sciences and a medical expert reviewed the statements. Students were asked to select the three statements they thought were correct. Students’ prior knowledge scores var- ied from 0 to 3 depending on the number of correct state- ments selected.

Reading fluency was measured by a word chain test (Holopainen et al., 2004). The task was to separate as many of a series of 100 words, written in four-word clusters with- out spaces in between, as possible in 90 seconds (Kiuru et al., 2011).

Credibility evaluation and argument analysis tasks To evaluate the credibility of the online sources the students completed two tasks, one for the blog text and one for the YouTube video. In both tasks, the students first rated the credibility of the source by answering a multiple choice question (How credible do you think the text/video is?) by selecting one of three responses (credible, fairly credible, not credible), and then justifying their credibility evaluation in writing by answering the question “Justify your choice, why do you think this?.”

Students also completed two tasks on the argumentative content of the online sources, one for the blog and one for the video. In both tasks, the students wrote down the main position on child vaccination they identified in the source and the most important justifications for that position given in the source.

Data analysis

Students’ credibility evaluation skills First, the frequencies of the students’ responses (n¼ 404) to the multiple-choice questions on the credibility of each online source were calculated. Next, the students’ justifica- tions for their credibility ratings (n¼ 403) were analyzed for trustworthiness and expertise. On trustworthiness, students scored points when they appealed to content-related issues: use of citations, research-based information, and unbalanced argumentation (Tseng & Fogg, 1999; Metzger, 2007; Walton, 1991). On expertise, the students scored points when they cited external characteristics of the source, e.g., author attributes, form of the web source, and organization(s) affili- ated with the web page (Metzger, 2007).

As both sources included unbalanced argumentation on the topic their credibility was principally deemed question- able. Consequently, students scored 0 points if they had not questioned the credibility of the source, i.e., selected the response option “credible”. When they had questioned cred- ibility, i.e., chosen the options “fairly credible” or “not credi- ble”, the response was scored from 1 to 4 depending on how well the lack of credibility had been justified. No justifi- cations scored 1 point and one or more justifications, relat- ing either to expertise or to trustworthiness, scored 2 points. Identification of unbalanced argumentation was emphasized in the scoring. Thus, 3 points was scored when lack of cred- ibility had been justified by appealing to unbalanced argu- mentation. Identification of unbalanced argumentation is illustrated in the following two examples of students’ justifi- cations (see Table 1): The text is based only on the disadvan- tages of vaccination and totally ignores its advantages (Student 428, Blog text); The issue was treated only from one side and the attitude for not using vaccines was initially nega- tive (Student 174, YouTube video). Justifications including both expertise and trustworthiness, although without refer- ence to unbalanced argumentation, also earned 3 points. The highest score (4 points) was given when the justifica- tions included, at the least, both expertise and reference to unbalanced argumentation (Table 1).

To test the reliability of the analysis criteria of students’ justifications, 15% of the data were selected at random and analyzed by two independent raters, reaching 82% agree- ment (Cohen’s kappa, Œ ¼.74).

Students’ argument analysis skills Analysis of students’ skills to analyze the argumentation of the blog text and the YouTube video proceeded in two phases. First, the researchers analyzed the argumentation of these two online sources. Next, the students’ responses to the argument analysis tasks were compared with the argu- mentation in the sources.

Argumentation of the online sources. First, all the reasons against vaccination in the blog text (42 reasons) and all those for vaccination in the video (5 reasons) were identi- fied. The identified reasons were then classified into

THE JOURNAL OF EDUCATIONAL RESEARCH 297

argumentation perspectives on a data-driven basis (Hsieh & Shannon, 2005).

For the blog text, the argumentation perspectives (an example of a reason in parenthesis) were: 1) Consequences of vaccination (Vaccination results in serious, even fatal injury); 2) Effectiveness of vaccination (Going through things notices that vaccinations do not protect from diseases); 3) Consistency of vaccination (Viral vaccines can never be safe); and 4) Support in problematic situations (Children falling ill from vaccinations are given help by official bodies). Similarly, for the video, the argumentation perspectives were: 1) Consequences of vaccination (Unvaccinated people are more likely to contract diseases); 2) Effectiveness of vac- cination (Before Salk’s vaccine there were approximately 58,000 cases of polio).

Analyzing students’ responses. Students’ skills in analyzing the argumentation in the source were scored based on their responses to each task (397 for the blog and 403 for the video). Students who failed to correctly identify the position taken in the source scored 0 points. Students who correctly identified the source position scored from 1 to 4 depending on the quality of their identification (position quality) and how many argumentation perspectives they found to sup- port that position.

Position quality was analyzed by investigating how well the position identified by a student corresponded to the source position, i.e., against vaccination in the blog and for vaccination in the video. Position quality indicated how well

a student had understood (partially vs. fully) the main mes- sage of the source. In responses indicating partial under- standing the source position was correctly identified but either only implicitly embedded in the response or expressed in the form of a descriptive claim by referring, for example, to specific harms or benefits of vaccines. Examples of descriptive claims were “Vaccines cause autism” (Student 416, Blog text) and “Vaccines protect people from severe dis- eases” (Student 95, YouTube video). Full understanding of the main message of the source was indicated by responses in which the source position was expressed in a form of a correct conclusion expressed in the source. Examples of cor- rect conclusions were “Children should not be vaccinated” (Student 335, Blog text) and “It is wise to vaccinate children” (Student 217, YouTube video).

The minimum requirement for a score of 1 in the blog task was that the student’s response included a position on child vaccination corresponding to that presented in the source (i.e., against vaccination, Table 2). Students scored 2 when the source position (SP) was correctly identified and justified from two argumentation perspectives (AP), as illus- trated in the following response by student 296 (analysis markings of the authors in parentheses): “The author is clearly against vaccinations (SP) and presents many justifica- tions for her stand. She states that more children vaccinated against whooping cough contracted the disease than non-vacci- nated children (AP 1, Effectiveness of vaccination). She also extensively describes the harm caused by vaccinations” (AP 2, Consequences of vaccination). Students scored 2 also when

Table 1. Examples of scoring students’ responses to the credibility evaluation tasks.

Examples of student responses

Score Scoring criteria Blog text task YouTube video task

0 Credibility of source not questioned The facts prove that the claims of the author are credible. (St. 390)

The video was convincing. (St. 263)

1 Credibility questioned without relevant justifications for the lack of credibility

Lots of facts. (St. 417) Everybody can find such things on the internet because it is so wide (St. 151)

The video is quite credible because it includes facts and exact figures. (St. 37)

2 Credibility questioned, and justification(s) relating either to expertise (E) or trustworthiness (T) presented

Because this is a blog (E) and the writer is necessarily not an expert (E) in this field (St. 219) No references (T) listed at the end of the text, and the studies were somewhat vague (St. 290)

Penn and Teller are magicians, not vaccination guys (E) (St. 422) There are no references (T) presented on the video. (St. 199)

3 Credibility questioned, and justifications relating both to expertise (E) and trustworthiness (T) presented, or unbalanced argumentation (UA) identified

The text is based only on the disadvantages of vaccination and totally ignores its advantages (UA) (St. 428) Because the text is a blog written by one person (E), an opinion without reliable references (T) (St. 409)

The issue was treated only from one side (UA) and the attitude for not using vaccines was initially negative (St. 174)

4 Credibility questioned, and unbalanced argumentation (UA) identified, and justification(s) relating to expertise (E) presented

The source is a website of a natural products company, and the text is a blog (E) which makes is primarily rather subjective. Discussion on the issue is very one-sided (UA), and different approaches to the issue are not presented. (St. 436)

I think like this because the video does not tell whether the persons are professionals or not (E), or whether they are only expressing their own opinion. Although the video also includes facts, no references (T) are used. The video is solely about the advantages of vaccines, nothing about disadvantages (UA), although they must certainly also exist. (St. 171)

Note. St. ¼ Student; T¼ Justification relating to trustworthiness; E¼ Justification relating to expertise; UA¼ Justification relating to unbalanced argumentation; The core words guiding the scoring are italicized.

298 M. MARTTUNEN ET AL.

the response included a correct conclusion justified from one argumentation perspective. Score 3 was given when the source position was correct and justified from three argumen- tation perspectives, or a correct conclusion was justified from two argumentation perspectives. Score 4 was given when the response included a correct conclusion justified from at least three argumentation perspectives (Table 2).

Students scored 1 on the YouTube video task when they correctly identified the source position taken on child vac- cination (i.e., for vaccination, see Table 3). Students scored 2 when either the source position was correctly identified and justified from one argumentation perspective, or the conclu- sion was correct but lacked justifications. Score 3 was given when the source position was correct and justified from two argumentation perspectives, or the conclusion (CO) was cor- rect and justified from one argumentation perspective (AP). The latter option is illustrated in the response by student

354 (analysis markings of the authors in parentheses): “The main message is that vaccination is not useless (CO). The most important justification is the survival of vaccinated children” (AP, Effectiveness of vaccination). Students scored 4 when their response included a correct conclusion justified from both argumentation perspectives (Table 3).

To test the reliability of the analysis criteria for both the blog and the video, two independent raters classified 15% of the data selected at random, reaching 87% agreement (Cohen’s kappa, Œ ¼.79).

Variables used in the study The dependent variables used in this study were formed based on the students’ credibility ratings, credibility evalu- ation, identification of unbalanced argumentation, and argu- ment analysis for each source (Table 4).

Table 2. Examples of scoring of students’ responses to the blog text argumentation analysis task.

Score Student response Interpretation

0 The author thinks vaccination is useful but not necessary. She justifies her opinion by facts. (St. 122)

Source position (Vaccination is useful) not correctly identified

1 The author finds vaccines dangerous and that one should take them with a pinch of salt. (St. 303)

Source position (against vaccination) correctly identified (Vaccines are dangerous).

2 The author is clearly against vaccinations and presents many justifications for her stand. She states that more children vaccinated against whooping cough contracted the disease than non-vaccinated children. She also extensively the describes the harm caused by vaccinations (St. 296)

Source position correctly identified (against vaccination) and justified from two argumentation perspectives:

1. Effectiveness of vaccination (Vaccinated children contracted the disease they were vaccinated against�)

2. Consequences of vaccination (Vaccinations cause harm�) 3 The author thinks vaccines are completely useless and might even be

dangerous. She justifies her claim by examples on how vaccines have several injurious effects and that they do not always protect people from diseases they are vaccinated against. (St. 337)

Correct conclusion (Vaccines should not be used) justified with two argumentation perspectives:

1. Consequences of vaccination (Vaccines have many different harmful effects�)

2. Effectiveness of vaccination (Vaccines do not always protect people from diseases they are vaccinated against�)

4 The author thinks vaccines are harmful and should not be used. She justifies her claim by giving examples of people who have contracted the disease they were vaccinated against. She explains why she thinks that viral vaccines are dangerous by describing how they are produced. At the end she also tells about a child who became autistic after vaccination, and about a child who had narcolepsy after vaccination. (St. 385)

Correct conclusion (Vaccines should not be used) justified with three argumentation perspectives:

1. Effectiveness of vaccination (Vaccines are not effective, because vaccinated people contracted the disease�)

2. Consistency of vaccination (Viral vaccinations are dangerous because of the production process�)

3. Consequences of vaccination (A child became autistic after vaccination and another child had narcolepsy�)

Note. St. ¼ Student; � ¼ Authentic reason included in the blog text.

Table 3. Examples of scoring of students’ responses to the YouTube video argumentation analysis task.

Score Student response Interpretation

0 The key point is that vaccines are complete bullshit :) (St. 134) Source position not correctly identified (Vaccines are bullshit) 1 Vaccines protect against many diseases and deaths (St. 115) Source position (for vaccination) correctly identified (Vaccines

protect against diseases). 2 Vaccines do not cause autism and vaccines save people from bad

diseases. The most important justifications were the use of visuals showing how vaccinated people can protect themselves from diseases and demonstrating the number of deaths from these diseases when not vaccinated. (St. 95)

A descriptive claim (Vaccines save people from bad diseases) justified from one argumentation perspective:

1. Effectiveness of vaccination (Vaccines are effective, because they protect people from diseases�)

3 The main message is that vaccination is not useless. The most important justification is the survival of vaccinated children. (St. 354)

Correct conclusion (Vaccination is not useless) justified with one argumentation perspective:

1. Effectiveness of vaccination (The most important justification is the survival of vaccinated children�)

4 The main point is that it is stupid not to vaccinate a child. Therefore, the video supports vaccines. The opinion was justified by listing many diseases against which vaccinations are used and saying that one can get the disease if not vaccinated. However, the video mentions that research exists claiming that vaccinations cause autism. However, this claim was completely refuted in the video. (St. 17)

Correct conclusion (It is stupid to not vaccinate a child) justified with two argumentation perspectives:

1. Effectiveness of vaccination (Vaccinations protect children from many diseases�)

2. Consequences of vaccination (Vaccinations are claimed to cause autism, which is refuted in the video�)

Note. St. ¼ Student; � ¼ Authentic reason included in the video.

THE JOURNAL OF EDUCATIONAL RESEARCH 299

Students’ most recent grades in various upper secondary school subjects, performance in the word chain test, prior knowledge on vaccination, and gender were used as inde- pendent variables (Table 4). In Finland, academic perform- ance in various subjects in upper secondary schools is rated from 4 to 10. Moreover, in the Finnish course-based school system at upper secondary level (European Commission, 2019), both schools and students have considerable freedom in organizing studies. Hence, the number of courses taken in different subjects may vary considerably between stu- dents, as was the case here. Moreover, some students did not consent to the use of their school grades for research purposes, which increased the amount of missing data. The proportion of missing data for the grades ranged from 15% to 32%.

Statistical analyses To answer research questions (RQ) one and two, descriptive statistics on the level of students’ credibility evaluation and argument analysis skills were used. Additionally, Wilcoxon’s signed ranks test was applied to compare students’ credibil- ity ratings for the blog text and video, and McNemar’s test was used to compare how students identified unbalanced argumentation in the sources (Davis, 2013). Furthermore, paired samples t-test was applied to compare differences between the blog text and the video in students’ overall per- formance in the credibility assessments and argument ana- lysis skills (Davis, 2013).

To answer RQ 3, Structural Equation Modeling (SEM) was applied (Kline, 2011). All SEM analyses were conducted using Mplus 7.4. Maximum likelihood estimation with robust standard errors (MLR) was used as an estimation method, as the variables showed slight non-normality (skew- ness values from �.68 to .45 and kurtosis values from �.80 to 1.20).

In the analysis, Overall academic achievement, Prior knowledge of the topic, Reading fluency and Gender were independent variables, and Argument analysis and

Credibility evaluation skills dependent variables. In addition, school grades were indicators for the bi-factor model: Overall academic achievement was modeled as a generic fac- tor for which all the individual grades were indicators. To complement the generic factor, specific factors for each indi- vidual grade were estimated to model variance not accounted for by the general factor.

The following fit indices and cutoff values for acceptable fit of the model were used: v2-test, ns, p > .05, Root Mean Square Error of Approximation (RMSEA) value under .06, and Tucker-Lewis Index (TLI) and Comparative Fit Index (CFI) values over .95 (Hu & Bentler, 1999). Effect sizes used for the repeated measures tests were reported as Cohen’s d (Cooligan, 2017). For the non-parametric tests, the effect sizes were transformed to Cohen’s d. For the SEM-analyses, squared r’s were calculated.

Results

Students’ credibility ratings of unreliable online sources

More than a half of the students (59.1%) considered the blog text fairly credible, slightly above one-third (37.5%) not credible, and only a few (3.5%) credible. Similarly, the majority (53.1%) judged the video to be fairly credible and 38.2% non-credible. In contrast to the blog, nearly a tenth (8.7%) of the students considered the video credible. The difference between the students’ credibility ratings for the blog and the video was not statistically significant (Z ¼ �1.20, p ¼ .232).

Students’ skills in evaluating the credibility and analyzing the argumentative content of unreliable online sources

Students’ skill in evaluating the credibility of the online sources proved rather weak. Although the students com- monly questioned the credibility of both sources, they justi- fied the non-credible nature of the sources poorly. More than a half of the students, 50.3% for the blog and 58.8% for the video, offered no justifications or appealed only to either authority or trustworthiness (1-2 points, Table 5). Students’ performance on the blog (M¼ 2.4) and video (M¼ 2.1) tasks differed significantly (t¼ 4.68, df¼ 402, p <

.001, d ¼ .29). As in the credibility evaluation, students’ skill in analyz-

ing the argumentation in the online sources was also fairly weak. This appeared especially in the blog text task. The majority (76.3%; points 0-2 in total) failed to identify the correct position taken in the text (0 points), or identified it but with either no or one justification (1 point), or found at most two of the four argumentation perspectives (2 points; Table 5). Further, only 2.8% of students correctly identified the main conclusion and several justifications for it (4 points; Table 5). Students performed somewhat better in the video task, as the corresponding proportions were 53.3% (0� 2 points) and 10.1% (4 points). The difference between

Table 4. Variables used in the study.

Variable N Scoring range

Dependent skill variables Credibility rating for the blog 403 1–3 Credibility rating for the video 403 1–3 Credibility evaluation for the blog 403 0–4 Credibility evaluation for the video 403 0–4 Identifying unbalanced argumentation in the blog 403 0¼No; 1¼ Yes Identifying unbalanced argumentation in the video 403 0¼No; 1¼ Yes Argument analysis for the blog 397 0–4 Argument analysis for the video 403 0–4

Independent variables Grades Philosophy 273 4–10 Psychology 337 4–10 Health education 311 4–10 Mother tongue and literature 344 4–10 Mathematics 331 4–10 History 342 4–10

Other Reading fluency 404 0–100 Prior knowledge 404 0–3 Gender 401�

Note. � Females, n¼ 244 (60.6%); Males, n¼ 157 (39.2%).

300 M. MARTTUNEN ET AL.

the blog (M¼ 1.9) and video (M¼ 2.3) was statistically sig- nificant (t ¼ �7.14, df¼ 395, p < .001, d ¼ .47).

Students’ identification of unbalanced argumentation

The study also investigated whether students had identified unbalanced argumentation as a justification for the non- credible nature of the sources. The results showed that this was mentioned by only slightly more than a third of the stu- dents (37.4%) in the blog text and about a fifth (21.0%) in the video. The difference between the blog and the video was significant (v2(1) ¼ 31.06, p < .001, d ¼ .57).

Factors explaining students’ skills in credibility evaluation and argument analysis

The structural equation model describing the factors explaining students’ argument analysis and credibility evalu- ation skills, and its parameter estimates, is presented in

Figure 1. The fit indices of the model were: v2(51) ¼ 66.31, p ¼ .073, RMSEA ¼ .03, CFI ¼ .98, TLI ¼ .97. Furthermore, all school subjects had significant and reason- able loadings, ranging from .64 to .73, for the Overall aca- demic achievement factor. Correlations between independent variables were estimated as suggested by the modification indices.

The structural equation model (Figure 1) shows that, for both online sources, students’ overall academic achievement was the only significant predictor of their argument ana- lysis and credibility evaluation skills. However, its explana- tory power for the video was rather weak, as it predicted only 2.9% to 4.2% of the variances for the argument ana- lysis and credibility evaluation tasks. Instead, overall aca- demic achievement was a somewhat stronger predictor of students’ skill in evaluating the credibility of the blog text, explaining 12.3% of the variance. Finally, the scores for the credibility evaluation of the blog text correlated positively (.24) with the scores for the credibility evaluation of the video.

Table 5. Level of credibility evaluation and argument analysis for the blog text and the YouTube video.

0 points 1 point 2 points 3 points 4 points M SD

Skill variable f % f % f % f % f %

Credibility evaluation for the blog 14 3.5 55 13.6 148 36.7 127 31.5 59 14.6 2.4 1.0 Credibility evaluation for the video 35 8.7 32 7.9 205 50.9 108 26.8 23 5.7 2.1 1.0 Argument analysis for the blog 5 1.3 130 32.7 168 42.3 83 20.9 11 2.8 1.9 0.8 Argument analysis for the video 4 1.0 83 21.0 124 31.3 145 36.6 40 10.1 2.3 1.0

Figure 1. Standardized parameter estimates (significant at the.05 level) for the structural equation model explaining students’ skills in argument analysis and cred- ibility evaluation. Note. Ovals to the left of the grades refer to the specific factors for each grade (e.g., Phil_SF¼ specific factor for philosophy grade).

THE JOURNAL OF EDUCATIONAL RESEARCH 301

Discussion

Although the students in this study commonly questioned the credibility of the online sources, their justifications for the non-credible nature of the sources were fairly weak, especially in the case of the YouTube video. As many as 67.5% of students either presented no justifications or appealed only to either authority or trustworthiness. Students also seemed to struggle in identifying unbalanced argumentation, as only slightly more than a third for the blog text and about a fifth for the video mentioned unbal- anced argumentation as a reason for the non-credibility of the source. This result supports previous findings on stu- dents’ difficulty in recognizing bias in online sources at the primary and secondary levels (Kiili et al., 2018; Miller & Bartlett, 2012), and at college level (McClure & Clink, 2009).

To understand the weakness of students’ justifications for the non-credibility of the YouTube video, it is important to note that the video dealt solely with the advantages of vac- cination, and thus echoed to the official stand and a positive public attitude toward vaccination (e.g., The National Institute for Health and Welfare, 2019). The students prob- ably found it unnecessary to put forward reasons for this generally accepted stand despite the obviously one-sided nature of information source. Another explanation may be myside bias (Greene et al., 2019; Perkins, 1989), that is, a tendency to prefer one’s own position on an issue and ignore evidence for the opposite position. van Strien et al. (2016) found that students seem to prefer, and to evaluate as more credible, information consistent with their own atti- tudes. Furthermore, research on motivated reasoning (e.g., Ditto et al., 1998; Kunda, 1990) suggests that pro-attitudinal information will be evaluated less thoughtfully than informa- tion that runs contrary to one’s beliefs and attitudes. This is especially true for social-scientific issues such as vaccination (Sinatra et al., 2014). People seem also to evaluate informa- tion differently if it comes from entertainment media com- pared to news media; entertainment media facilitate fairly stable attitudes, such as stereotypical or ideology-based judg- ments (Kim & Vishak, 2008).

Students’ skill in analyzing the argumentation contained in the online sources also proved insufficient, especially in the blog text task. This result is in line with previous find- ings that students have difficulties in identifying and evalu- ating arguments in linear texts (e.g., Larson et al., 2004; Marttunen et al., 2005). Although the students’ analysis of the argumentation of the video was also weak, it was never- theless better than that of the blog text. One reason for this may be that the argumentative structure of the video was simpler than that of the blog text: the video contained far less reasons than the blog and only two argumentation per- spectives in support of its main position in contrast to the four argumentation perspectives in the blog text. The multi- modal nature of the video may also have affected the result. Whereas the argumentation in the blog was mainly expressed via linear text, that in the video comprised both verbal and nonverbal communication, such as visuals, motion, and voice, used simultaneously to express its con- tent. The multimodality of the video may have helped the

students to understand its content and analyze it (see Lee & List, 2019). Tseronis (2018) states that visuals and other nonverbal means can be seen as argumentatively relevant because they can simply depict the argument, or elements of the propositions that constitute the argument, and convey the meaning of the source in more diverse ways than are possible with plain text.

Students’ lack of skills in analyzing argumentation and the credibility of online sources may, on the other hand, mirror a ‘why bother’ attitude stemming from skeptical or even cynical views (Vraga & Tully, 2021) that much online information is somehow biased in any case. Skepticism toward, e.g., politics and media in general is, however, con- sidered a ‘healthy’ trait as it includes doubt and a question- ing of certainty that encourages participation in a democratic society (Vraga & Tully, 2021) and enhances crit- ical discussion on societal issues. For example, Maksl et al. (2015) found that highly news media literate teens were more skeptical of the media and more motivated to con- sume news than their less news literate counterparts. Therefore, one objective of education should be to enhance students’ ability to engage actively in society by teaching them to be sufficiently critical when searching, evaluating, and utilizing online information.

In educational settings, students often evaluate online information through argumentative writing. Wolfe and Britt (2008) found that when students prepare themselves for essay writing, for example, by taking notes that fail to include information incongruent with their own opinions, their essays tend to be one-sided. Students seem to have misconceptions about good argumentation and argumenta- tive writing, possibly thinking that sound argumentation involves generating arguments only in support of one’s own position. However, high quality argumentation also requires analyzing information from the other side and combining arguments and counterarguments for a final position (Nussbaum & Schraw, 2007). For this reason, developing students’ understanding of more sophisticated and versatile argumentation thinking is an important educational goal that should be accorded more attention to prevent students being misled by inaccurate and flawed information. Previous studies (Bronstein et al., 2019; Pennycook & Rand, 2019) have shown, for example, that deficiencies in analytical thinking and reasoning are related to believing in biased information such as fake news.

The results also showed that students’ previous perform- ance in various individual school subjects did not explain their skills in either evaluating the credibility or analyzing the argumentative content of the two online sources. However, students’ overall academic achievement was found to be significantly associated with these skills. This result supports the study by Utriainen et al. (2017), who found that university applicants’ overall performance in the matriculation examination predicted their argument analysis skills. The present finding that students’ overall academic achievement, but not their achievement in individual school subjects, prior knowledge on the task topic, or reading flu- ency, explained their evaluation and analyzing skills (cf. Kiili

302 M. MARTTUNEN ET AL.

et al., 2018; Leu et al., 2015) refer to the generic nature of credibility evaluation and argument analysis skills. However, it must be noted that, as in the study by Utriainen et al. (2017), the proportions of explanations remained rather low.

Owing to the rapid increase in internet use both in and out of school, it is very likely that current teaching practices do not yet adequately support students’ learning of the skills they need when they work internet-sourced knowledge. This concern is supported by the weak associations found in this study between students’ previous school achievement and their credibility evaluation and argument analysis skills with online sources. These findings suggest that tasks in which students interpret and analyze online information presented in various modes, both written and multimodal, should be more integrated into the instruction given in different sub- jects in schools.

Limitations

This study was carried out in an authentic context as a part of students’ normal schoolwork in upper secondary school classrooms. Hence, the participating schools were not ran- domly selected and the data are not necessarily representa- tive of all Finnish upper secondary school students. Thus, the results cannot be generalized beyond the present sample. Further, another limitation of the study is that arguments and information presented were not counterbalanced across the blog and the video, due to the authentic study situation in schools. Same task repeated, although using different medium, would have possibly diminished students’ motiv- ation to complete the task. The task used in this study was also a part of a large and time-consuming online inquiry task, and thus we did not find it possible to increase the stu- dents’ workload to complete the task.

Further, it should also be noted that the two online sour- ces used in this study differed in mode, language, length, and argumentative content which complicates their compari- son. To facilitate comparison of the sources, the focus was on only the verbal argumentation included in them, and thus many multimodal arguments based on, e.g., visuals, sounds, and gestures in the video were not taken into account. However, the comparison of the sources should be treated with caution. The study, nevertheless, opened inter- esting topics for further research to compare the credibility and argumentation of multimodal and text-based online sources.

Conclusion

Limitations aside, this study showed that evaluating the credibility and analyzing the argumentative content of two online sources commonly used among young people in social media—a written blog text and a YouTube video— was poorly accomplished by many students. The fact that many major life decisions are made based on information acquired from the internet (Horrigan & Rainie, 2006) exposes many individuals to the risk that these decisions are likely to be based on inaccurate or even flawed information.

Thus, critical evaluation of information should be increas- ingly emphasized in schools to promote students’ skills in differentiating reliable from inaccurate information and to protect future citizens from being misled by the disinforma- tion increasingly encountered on the internet.

Disclosure statement

None of the authors have conflicts of interest to declare.

Funding

Academy of Finland (Project 285 817).

ORCID

Miika Marttunen http://orcid.org/0000-0002-4554-9764 Timo Salminen http://orcid.org/0000-0001-7798-3779 Jukka Utriainen http://orcid.org/0000-0002-9325-2893

References

Bråten, I., Stadtler, M., & Salmer�on, L. (2018). The role of sourcing in discourse comprehension. In M. F. Schober, D. N. Rapp, & M. A. Britt (Eds.), The Routledge handbook of discourse processes (2nd ed., pp. 141–166). Routledge.

Broniatowski, D. A., Jamison, A. M., Qi, S., AlKulaib, L., Chen, T., Benton, A., Quinn, S. C., & Dredze, M. (2018). Weaponized health communication: Twitter bots and Russian trolls amplify the vaccine debate. American Journal of Public Health, 108(10), 1378–1384. https://doi.org/10.2105/AJPH.2018.304567

Bronstein, M. V., Pennycook, G., Bear, A., Rand, D. G., & Cannon, T. D. (2019). Belief in fake news is associated with delusionality, dogmatism, religious fundamentalism, and reduced analytic think- ing. Journal of Applied Research in Memory and Cognition, 8(1), 108–117. https://doi.org/10.1016/j.jarmac.2018.09.005

Chen, Y.-Y., Li, C.-M., Liang, J.-C., & Tsai, C.-C. (2018). Health infor- mation obtained from the Internet and changes in medical decision making: Questionnaire development and cross-sectional survey. Journal of Medical Internet Research, 20(2), e47. https://doi.org/10. 2196/jmir.9370

Coiro, J. (2011). Predicting reading comprehension on the Internet: Contributions of offline reading skills, online reading skills, and prior knowledge. Journal of Literacy Research, 43(4), 352–392. https://doi.org/10.1177/1086296X11421979

Coiro, J., Coscarelli, C., Maykel, C., & Forzani, E. (2015). Investigating criteria that seventh graders use to evaluate the quality of online information. Journal of Adolescent & Adult Literacy, 59(3), 287–297. https://doi.org/10.1002/jaal.448

Cooligan, H. (2017). Research methods and statistics in psychology (6th ed.). Psychology Press.

Davis, C. (2013). SPSS for applied sciences: Basic statistical testing. CSIRO Publishing.

Ditto, P. H., Scepansky, J. A., Munro, G. D., Apanovitch, A. M., & Lockhart, L. K. (1998). Motivated sensitivity to preference-inconsist- ent information. Journal of Personality and Social Psychology, 75(1), 53–69. https://doi.org/10.1037/0022-3514.75.1.53

European Commission. (2019). Organisation of general upper second- ary education. https://eacea.ec.europa.eu/national-policies/eurydice/ content/organisation-general-upper-secondary-education-15_en

Flanagin, A., & Metzger, M. (2008). Digital media and youth: Unparalleled opportunity and unprecedented responsibility. In M. J. Metzger, & A. J. Flanagin (Eds.), Digital media, youth, and credibil- ity (pp. 5–28). The MIT Press, The John D. and Catherine T. MacArthur Foundation Series on Digital Media and Learning.

THE JOURNAL OF EDUCATIONAL RESEARCH 303

Fong, C. J., Kim, Y., Davis, C. W., Hoang, T., & Kim, Y. W. (2017). A meta-analysis on critical thinking and community college student achievement. Thinking Skills and Creativity, 26, 71–83. https://doi. org/10.1016/j.tsc.2017.06.002

Fogg, B. J., Soohoo, C., Danielson, D. R., Marable, L., Stanford, J., & Tauber, E. R. (2003). How do users evaluate the credibility of Web sites? A study with over 2,500 participants [Paper presentation]. The Proceedings of the Conference on Designing for User Experiences, San Francisco, CA, June 6-7 (pp. 1–15). https://doi.org/10.1145/ 997078.997097

Forzani, E. (2016). Individual differences in evaluating the credibility of online information in science: Contributions of prior knowledge, gen- der, socioeconomic status, and offline reading ability [Doctoral disser- tation]. University of Connecticut. https://www.researchgate.net/ profile/ElenaForzani2/publication/311559938

Fulkerson, R. (1996). Teaching the argument in writing. National Council of Teachers of English.

Ghanizadeh, A. (2017). The interplay between reflective thinking, crit- ical thinking, self-monitoring, and academic achievement in higher education. Higher Education, 74(1), 101–114. https://doi.org/10.1007/ s10734-016-0031-y

Greene, J. A., Cartiff, B. M., Duke, R. F., & Deekens, V. M. (2019). A nation of curators. Educating students to be critical consumers and users of online information. In P. Kendeou, D. H. Robinson, & M. T. McCrudden (Eds.), Misinformation and fake news in educa- tion (pp. 187–206). Information age publishing.

Holopainen, L., Kairaluoma, L., Nevala, J., Ahonen, T., & Aro, M. (2004). Lukivaikeuksien seulontatesti nuorille ja aikuisille [Dyslexia screening test for youth and adults]. Niilo M€aki Institute.

Horrigan, J., & Rainie, L. (2006). The internet’s growing role in life’s major moments. Pew Research Center Publication, Pew Internet and American Life Project. http://www.pewinternet.org/2006/04/19/the- internets-growing-role-in-lifes-major-moments/

Hsieh, H.-F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288. https://doi.org/10.1177/1049732305276687

Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alter- natives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

Kahne, J., & Bowyer, B. (2017). Educating for democracy in a partisan age: Confronting the challenges of motivated reasoning and misin- formation. American Educational Research Journal, 54(1), 3–34. https://doi.org/10.3102/0002831216679817

Kammerer, Y., Amann, D., & Gerjets, P. (2015). When adults without university education search the Internet for health information: The roles of Internet-specific epistemic beliefs and a source evaluation intervention. Computers in Human Behavior, 48, 297–309. https:// doi.org/10.1016/j.chb.2015.01.045

Kammerer, Y., Kalbfell, E., & Gerjets, P. (2016). Is this information source commercially biased? How contradictions between web pages stimulate the consideration of source information. Discourse Processes, 53(5-6), 430–456. https://doi.org/10.1080/0163853X.2016. 1169968

Kiili, C., Laurinen, L., & Marttunen, M. (2008). Students evaluating Internet sources: From versatile evaluators to uncritical readers. Journal of Educational Computing Research, 39(1), 75–95. https:// doi.org/10.2190/EC.39.1.e

Kiili, C., Leu, D. J., Marttunen, M., Hautala, J., & Lepp€anen, P. H. T. (2018). Exploring early adolescents’ evaluation of academic and commercial online resources related to health. Reading and Writing, 31(3), 533–557. https://doi.org/10.1007/s11145-017-9797-2

Kim, Y. M., & Vishak, J. (2008). Just laugh! You don’t need to remem- ber: The effects of entertainment media on political information acquisition and information processing in political judgment. Journal of Communication, 58(2), 338–360. https://doi.org/10.1111/j. 1460-2466.2008.00388.x

Kiuru, N., Haverinen, K., Salmela-Aro, K., Nurmi, J.-E., Savolainen, H., & Holopainen, L. (2011). Students with reading and spelling disabil- ities: Peer groups and educational attainment in secondary

education. Journal of Learning Disabilities, 44(6), 556–569. https:// doi.org/10.1177/0022219410392043

Kline, R. B. (2011). Principles and practice of structural equation model- ing. The Guilford Press.

Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108(3), 480–498. https://doi.org/10.1037/0033-2909.108.3.480

Larson, M., Britt, M. A., & Larson, A. A. (2004). Disfluencies in com- prehending argumentative texts. Reading Psychology, 25(3), 205–224. https://doi.org/10.1080/02702710490489908

Lee, H. Y., & List, A. (2019). Processing of texts and videos: A strat- egy-focused analysis. Journal of Computer Assisted Learning, 35(2), 268–282. https://doi.org/10.1111/jcal.12328

Leu, D. J., Forzani, E., Rhoads, C., Maykel, C., Kennedy, C., & Timbrell, N. (2015). The new literacies of online research and com- prehension: Rethinking the reading achievement gap. Reading Research Quarterly, 50(1), 37–59. https://doi.org/10.1002/rrq.85

Leu, D. J., Kinzer, C. K., Coiro, J., Castek, J., & Henry, L. A. (2013). New literacies and the new literacies of online reading comprehen- sion: A dual level theory. In N. Unrau, & D. Alvermann (Eds.), Theoretical models and process of reading (6th ed., pp. 1150–1181). International Reading Association.

Maksl, A., Ashley, S., & Craft, S. (2015). Measuring news media liter- acy. Journal of Media Literacy Education, 6(3), 29–45. https://digi- talcommons.uri.edu/jmle/vol6/iss3/3

Marin, L. M., & Halpern, D. F. (2011). Pedagogy for developing critical thinking in adolescents: Explicit instruction produces greatest gains. Thinking Skills and Creativity, 6(1), 1–13. https://doi.org/10.1016/j. tsc.2010.08.002

Marttunen, M., Laurinen, L., Litosseliti, L., & Lund, K. (2005). Argumentation skills as prerequisites for collaborative learning among Finnish, French, and English secondary school students. Educational Research and Evaluation, 11(4), 365–384. https://doi. org/10.1080/13803610500110588

McClure, R., & Clink, K. (2009). How do you know that?: An investi- gation of student research practices in the digital age. Portal: Libraries and the Academy, 9(1), 115–132. https:/doi.org/10.1353/ pla.0.0033

Means, M. L., & Voss, J. F. (1985). Star Wars: A developmental study of expert and novice knowledge structures. Journal of Memory and Language, 24(6), 746–757. https://doi.org/10.1016/0749-596X(85)90057-9

Means, M. L., & Voss, J. F. (1996). Who reasons well? Two studies of informal reasoning among children of different grade, ability, and knowledge levels. Cognition and Instruction, 14(2), 139–178. https:// doi.org/10.1207/s1532690xci1402_1

Metzger, M. J. (2007). Making sense of credibility on the Web: Models for evaluating online information and recommendations for future research. Journal of the American Society for Information Science and Technology, 58(13), 2078–2091. https://doi.org/10.1002/asi.20672

Metzger, M. J., Flanagin, A. J., & Medders, R. B. (2010). Social and heuristic approaches to credibility evaluation online. Journal of Communication, 60(3), 413–439. https://doi.org/10.1111/j.1460-2466. 2010.01488.x

Miller, C., & Bartlett, J. (2012). Digital fluency’: Towards young peo- ple’s critical use of the internet. Journal of Information Literacy, 6(2), 35–55. https://doi.org/10.11645/6.2.1714

Nussbaum, E. M. (2011). Argumentation, dialogue theory, and prob- ability modeling: Alternative frameworks for argumentation research in education. Educational Psychologist, 46(2), 84–106. https://doi.org/ 10.1080/00461520.2011.558816

Nussbaum, E. M., & Schraw, G. (2007). Promoting argument–counter- argument integration in students’ writing. The Journal of Experimental Education, 76(1), 59–92. https://doi.org/10.3200/JEXE. 76.1.59-92

Osborne, J., Erduran, S., & Simon, S. (2004). Enhancing the quality of argumentation in science classrooms. Journal of Research in Science Teaching, 41(10), 994–1020. https://doi.org/10.1002/tea.20035

Paul, J., Macedo-Rouet, M., Rouet, J.-R., & Stadtler, M. (2017). Why attend to source information when reading online? The perspective of ninth grade students from two different countries. Computers & Education, 113, 339–354. https://doi.org/10.1016/j.compedu.2017.05.020

304 M. MARTTUNEN ET AL.

Pennycook, G., & Rand, D. G. (2019). Lazy, not biased: Susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoning. Cognition, 188, 39–50. https://doi.org/10. 1016/j.cognition.2018.06.011

Perkins, D. N. (1989). Reasoning as it is and could be: An empirical perspective. In D. M. Topping, D. C. Crowell, & V. N. Kobayashi (Eds.), Thinking across cultures: The third international conference on thinking (pp. 175–195). Erlbaum.

Preiss, D. D., Castillo, J. C., Flotts, P., & San Mart�ın, E. (2013). Assessment of argumentative writing and critical thinking in higher education: Educational correlates and gender differences. Learning and Individual Differences, 28, 193–203. https://doi.org/10.1016/j.lin- dif.2013.06.004

Sinatra, G. M., Kienhues, D., & Hofer, B. K. (2014). Addressing chal- lenges to public understanding of science: Epistemic cognition, motivated reasoning, and conceptual change. Educational Psychologist, 49(2), 123–138. https://doi.org/10.1080/00461520.2014. 916216

Stanford History Education Group. (2016). Evaluating information: The cornerstone of civic responsibility. An executive summary. https:// sheg.stanford.edu/upload/V3LessonPlans/Executive%20Summary% 2011.21.16.pdf

Taylor, A., & Dalal, H. A. (2017). Gender and information literacy: Evaluation of gender differences in a student survey of information sources. College & Research Libraries, 78(1), 90–113. https://doi.org/ 10.5860/crl.78.1.90

The National Institute for Health and Welfare. (2019). Vaccination coverage. https://thl.fi/en/web/vaccination/vaccination-coverage

Tseng, S., & Fogg, B. J. (1999). Credibility and computing technology. Communications of the ACM, 42(5), 39–44. https://doi.org/10.1145/ 301353.301402

Tseronis, A. (2018). Multimodal argumentation: Beyond the verbal/vis- ual divide. Semiotica, 2018(220), 41–67. https://doi.org/10.1515/sem- 2015-0144

Toulmin, S. E. (1958). The uses of argument. Cambridge University Press.

Utriainen, J., Marttunen, M., Kallio, E., & Tynj€al€a, P. (2017). University applicants’ critical thinking skills: The case of the Finnish

educational sciences. Scandinavian Journal of Educational Research, 61(6), 629–649. https:/doi.org/10.1080/00313831.2016.1173092

van Amelsvoort, M., Andriessen, J., & Kanselaar, G. (2007). Representational tools in computer-supported collaborative argu- mentation-based learning: How dyads work with constructed and inspected argumentative diagrams. Journal of the Learning Sciences, 16(4), 485–521. https://doi.org/10.1080/10508400701524785

van Strien, J. L., Kammerer, Y., Brand-Gruwel, S., & Boshuizen, H. P. (2016). How attitude strength biases information processing and evaluation on the web. Computers in Human Behavior, 60, 245–252. https://doi.org/10.1016/j.chb.2016.02.057

Vosoughi, S., Roy, D., & Aral, S. (2018). The spread of true and false news online. Science, 359(6380), 1146–1151. https://doi.org/10.1126/ science.aap9559

Vraga, E. K., & Tully, M. (2021). News literacy, social media behaviors, and skepticism toward information on social media. Information, Communication & Society, 24(2), 150–166. https://doi.org/10.1080/ 1369118X.2019.1637445

Walsh, J. (2010). Librarians and controlling disinformation: Is multi-lit- eracy instruction the answer? Library Review, 59(7), 498–511. https://doi.org/10.1108/00242531011065091

Walton, D. N. (1991). Bias, critical doubt and fallacies. Argumentation and Advocacy, 28(1), 1–22. https://doi.org/10.1080/00028533.1991. 11951525

Wolfe, C. R., & Britt, M. A. (2008). The locus of the myside bias in written argumentation. Thinking & Reasoning, 14(1), 1–27. https:// doi.org/10.1080/13546780701527674

Xu, X., & Yao, Z. (2015). Understanding the role of argument quality in the adoption of online reviews: An empirical study integrating value-based decision and needs theory. Online Information Review, 39(7), 885–902. https://doi.org/10.1108/OIR-05-2015-0149

Xiao, B., & Benbasat, I. (2015). Designing warning messages for detect- ing biased online product recommendations: An empirical investiga- tion. Information Systems Research, 26(4), 793–811. https://doi.org/ 10.1287/isre.2015.0592

Zubiaga, A., Aker, A., Bontcheva, K., Liakata, M., & Procter, R. (2018). Detection and resolution of rumours in social media: A survey. ACM Computing Surveys, 51(2), 1–36. https://doi.org/10.1145/ 3161603

THE JOURNAL OF EDUCATIONAL RESEARCH 305

Copyright of Journal of Educational Research is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.