helpfn

IEEE REVISTA IBEROAMERICANA DE TECNOLOGIAS DEL APRENDIZAJE, VOL. 14, NO. 4, NOVEMBER 2019 177

A Content Analysis System That Supports Sentiment Analysis for Subjectivity and

Polarity Detection in Online Courses Ruth Cobos, Francisco Jurado, and Alberto Blázquez-Herranz

Abstract— Given the current and increasing relevance of research aimed towards the optimization of teaching and learning experiences in online Education, a plethora of studies regarding the application of different technologies to this purpose have been developed. Specifically, Natural Language Processing (NLP) has been used to detect potential sentiments and opinions in texts, enabling a broader scope for making inferences. At Universidad Autónoma of Madrid, Spain, we have designed and developed a tool for the application of NLP techniques to analyse the contents of online courses and the contributions of their learners (video transcriptions, readings, questions and answers of the evaluation activities and learner’s posts in forums, among others) to improve the teaching material and the teaching-learning processes of these courses. This tool is called edX-CAS (“Content Analyser System for edX MOOCs”). In this paper, we provide a detailed description of the tool, its functionalities and its NPL processes that support Sentiment Analysis for Subjectivity and Polarity detection. Moreover, we present a review of current research in the field of application of NLP in the improvement of teaching and learning experiences in MOOCs.

Index Terms— MOOC, sentiment analysis, opinion mining, natural language processing, polarity, subjectivity.

I. INTRODUCTION

NOWADAYS, there is a great variety of approaches aimedat facilitating the effectiveness of the teaching-learning processes in online courses. These approaches make use of Learning Analytics techniques to analyse learners’ interac- tions, such as the number of contributions in forums, the time the users spend in specific assignments, how long they work with learning material like videos or readings, and so on. Just to mention a few examples [1]–[4].

More recently, researchers are looking to introduce new metrics, tools and techniques providing insights for the recog- nition and analysis of users’ emotional dispositions within the context of these online courses.

This article is an extended version of the work published in the 2019 IEEE Global Engineering Education Conference

Manuscript received September 9, 2019; revised September 24, 2019; accepted October 24, 2019; date of publication November 8, 2019; date of current version December 9, 2019. (Spanish version received October 8, 2019; revised October 19, 2019; accepted October 19, 2019). This work was supported by the Madrid Regional Government, through the Project e-Madrid-CM under Grant P2018/TCS-4307; the e-Madrid-CM project also co-financed by the Structural Funds (FSE and FEDER). (Corresponding author: Ruth Cobos.)

The authors are with the Computer Science Department, Universidad Autónoma de Madrid, 28049 Madrid, Spain (e-mail: [email protected]; [email protected]; [email protected]).

There exists a Spanish version of this article available at http://rita.det.uvigo.es/VAEPRITA/V7N4/A9.pdf

Digital Object Identifier 10.1109/RITA.2019.2952298

(EDUCON2019) [5]. We extend our previous work by a complementary literature review and an extended and detailed explanation of the proposed tool edX-CAS: Content Analyser System for edX courses (https://www.edx.org/). This tool was designed and developed at Universidad Autónoma of Madrid (UAM, Spain).

More in detail, we have designed and developed the edX-CAS for identifying traits of subjectivity and polarity in the contents of online courses and contributions of their learn- ers. Thus, we have aimed the generated processes towards the extraction and analysis of potential sentiments and opinions and its polarity in the learning material of our online courses, such as SPOCs (Small Private Online Courses) and MOOCs (Massive Open Online Courses). Particularly, with the use of Natural Language Processing (NLP) techniques, we mine the opinion about specific parts of the course (like learning material, services, and even instructors), as well as other clues that the instructors and learners may have reflected in the text they write in a non-intrusive way.

Thus, by using NLP it is possible to extract features from the text that can be used as input in other Learning Analytics processes. In fact, there is already conducted research based on the enrichment of input variables from the implementa- tion of NLP processes for success prediction [6], evaluation of learners’ reflective capabilities [7], automated scoring of assessments [8], and so on.

Most of these studies set the emphasis of their feature extraction processes in learners’ textual data, such as that provided in written assessments, forum discussions or their profile information, with fewer efforts being placed in the inspection of the course itself.

Video-transcriptions and text-format files (in HTML, pdf, etc.) are just part of the plethora of “extra” information inherent to all courses, that is susceptible to being processed and analysed. By processing and analysing these features the pool of sources from which to extract knowledge expands, as well as its associated potential: allows to infer decisions to improve or modify online courses materials, get insights about the learning services and its effect on learners, further develop measurements of the effects of content on specific learners’ facets, etc.

Taking all these points into consideration, we can sum- marize the intendments of this study as aimed to fos- ter the adaptability of the students’ learning experience by inspecting a course’s content and highlighting potential

1932-8540 © 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:30:18 UTC from IEEE Xplore. Restrictions apply.

178 IEEE REVISTA IBEROAMERICANA DE TECNOLOGIAS DEL APRENDIZAJE, VOL. 14, NO. 4, NOVEMBER 2019

points of improvement related to its subjectivity and polarity.

The rest of the article is structured as follows: the next section presents state of the art and terminology in the area of this research work; in section 3, we present a detailed description of the proposed tool edX-CAS for extracting and analysing the sentiments and opinions of online courses; in section 4, we present the NLP processes supported by edX-CAS; in section 5, we present the visualization provided by edX-CAS; and in section 6, we present the use of the proposed tool with several online courses provided by UAM. Finally, the article ends with conclusions and future works.

II. STATE OF THE ART

Within the field of Natural Language Processing (NLP), the terms Sentiment Analysis (SA) and Opinion Mining (OM), frequently conceived as a unique discipline, have been jointly defined as the “computational treatment of opinion, sentiment and subjectivity in text” [9]–[11]. This involves three main concepts: subjectivity, and opinion or sentiment. In this context, subjectivity analysis allows classifying a given text into subjective or objective. Once a given text has been labelled as subjective, we can identify the opinion adhered to it, computed as the polarity degree (positive or negative) against something. However, it is important to remark that the diversity of ambits where these techniques have been applied and the associated lack of consensus on their distinction may lead to a misunderstanding of what these fields are really concerned with.

A. Terminology

It is interesting to provide some nuances in terminology. In particular, [12] distinguishes the next concepts:

• Feelings: the sensations caused by physiological states (e.g. anger, pain, fatigue…) and influenced by previous experience, which makes them personal.

• Emotion: the corporal response to an event linked to the social facet of expressing an internal state, thus influenced by culture.

• Sentiment: the tendency towards a type of reaction to a stimulus while being aware of both the reaction and its cause.

• Opinion: the owned knowledge with respect to some- thing, not enough founded to be May imply emotions or feelings.

With this scope set, we have considered of special impor- tance to clarify the terms around which our research is going to take place. First, tackling the difference between sentiments and emotions, they differ in durability and stability, with the former lasts over long time periods, while the latter relates to specific moments and can be overlapped with an existing sentiment [12]. This puts grounds enough to objectively set a preference for the analysis of sentiments since they are associated with an invariance that allows for its more feasible recognition.

Furthermore, it is interesting to recall Wierzbicka’s study with respect to the influence of culture on emotions [13]: “feel- ing is universal and can be safely used in the investigation of

Fig. 1. Sentiment Analusis workflow chart.

human experience and human nature, the concept of emotion is culture-bound and cannot be similarly relied on”. Thus, the set of factors on which emotions may rely lies outside of the scope of this research.

Consequently, and summing up our conclusions in this respect, we have identified four key terms in our work: Sub- jectivity Analysis, Sentiment Analysis (SA), Opinion Mining (OM) and Polarity Analysis (PA).

However, if attending to the scope of application of these techniques, it is important to remark that, in which respects to the technological application of SA and OM, they have been developed and investigated as equivalent [11]. In addition, the Opinion measurement against something is performed by identifying its specific polarity degree. Coherently, for the sake of simplicity and from this point on, we are to uniquely use the term, namely: Sentiment Analysis (SA).

Consequently, we have defined a hierarchical process in which SA is conducted (see Figure 1). After performing the pre-processing of the text by tokenizing it, removing the stopwords, lemmatizing the relevant word, and doing a part-of-speech tagging (POS-tagging), then we start with a classification process to categorize the text as objective or subjective. Whether the text is subjective, we can consider to identify the polarity as one of the many factors involved in the Polarity Analysis.

B. Uses of NLP in MOOCs

Narrowing down the scope to the learning environment, we can find in [14] a recent Systematic Literature Review on the application of SA to education. This review reveals that the main uses of SA within the educational domain are the improvement of the teaching-learning process and the reduc- tion of dropouts. Moreover, it is also pointed out MOOCs’ forums and social networks are the most used resources to perform the SA process. Given this scope, the general

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:30:18 UTC from IEEE Xplore. Restrictions apply.

COBOS et al.: CONTENT ANALYSIS SYSTEM THAT SUPPORTS SA FOR SUBJECTIVITY AND POLARITY DETECTION 179

approach set in these investigations aims to the detection, distinction and labelling of sentiments and opinions present in the mentioned sources (using classification techniques such as Naive Bayes and Support Vector Machines).

In order to conduct a literature review insightful enough to serve as the first approach to currently developed applications of NLP in MOOCs, the investigation in [15] helps to identify guidelines for the distinction of content related to this field.

In accordance with [15], we identified the following cate- gories as the general and most common intendments on the field of NLP in MOOCs: the insights into student performance, the learners’ interaction and feedback, and the improvement of teaching material.

Thus, the reviewed research, related to each of the pre- vious categories, is shown and summarized in the following subsections.

1) Insights Into Student Performance: In this case, we reviewed research aimed at evaluating the practicality of including NLP-obtained features in common Learning Ana- lytics’ tasks, such as the prediction of certain performative measures (e.g. success or graduation).

In [16] a SA model for the analysis of learners’ behav- iour data in a MOOC is presented. This way, students are distinguished in terms of their emotional tendencies and course participation, allowing for an in-depth assessment of the relationship between their emotional facets and learning effects (such as completion and graduation rates). A method for predicting graduation probability by analysing learners’ sentiment changes is subsequently developed.

Besides, the research conducted in [17] explores the parallel field of Affective Computing (AC) as a means to identify the emotional charge (happiness, sadness, fear, anger-passion, etc.) that is reflected in the text, depending on the words used and regardless of the presence of an opinion in it.

In [18] we can find an example of AC experience where authors perform an analysis to identify if there are correlations between the academic marks of students and the sentiment traces they revealed in their collaborative writing assignments. Their results reveal correlations between the students’ marks and sentiments in highest and lowest marks.

2) Learners’ Interaction and Feedback: The research per- formed for this category involved investigation centred on the assessment of textual content generated by students with the purpose of further analysing its implications (e.g. sentiment labelling, networks’ assessment, etc.).

Among the examples found in our literature review, [19] and [20] inspect the application of SA on the feedback that students provide to the teacher via social networks such as Twitter so that the teacher can adapt the teaching process.

Also analysing the students’ feedback in [21] we can find a lexicon-based approach to identify their positive or negative attitude, and so, to predict the level of teaching performance. With a similar goal, [22] perform SA using Latent Dirichlet Allocation (LDA) as sentiment grabber to detect students’ opinion on various topics, so that the teacher can better tune the teaching-learning process.

Not to improve the teaching material but the teacher perfor- mance itself, [23]–[26] carry out SA students’ comments to

evaluate the teacher performance and to implement construc- tive strategies.

For its part, [27] analyse educational posts from social media, particularly Twitter. Similarly, [28] use Support Vector Machines (SVMs) to perform SA with the aim to evaluate students’ interaction in these type of networks by mining the students’ tweets and email messages within the context of a MOOC.

Although in initial stages, [29] proposes TutorAlert, an NLP-based tool to identify traits such as confusion or frus- tration in students by using SVM, Naïve Bayes and Random Forest algorithms to classify students’ discussion posts.

Similarly, [30] compares several machine learning algo- rithms and lexicon-based approaches to perform SA for stu- dents’ feedback in order to identify the best (most relevant) features and algorithm combination.

For its part [31] developed SentBuk to support SA on Facebook. The authors follow a lexicon-based approach, combining lexical and syntactical analysis along with other techniques to process the messages written by the users in the social network. Although initially designed for its use on Facebook, the authors provide some insights for exploiting their approach in the context of e-learning.

3) Improvement of Teaching Material: Finally, this section’s review was aimed at identifying studies centred in the proposal and development of strategies for improving the adaptability of MOOC Content.

As scarce as it has been found to be, there already are research lines focused on the development of strategies for a course’s content evaluation (and further optimization).

An interesting example is the work conducted in [32] in which, apart from inspecting feedback from students’ posts in a course’s forum, they apply Henri’s content analysis framework [33] to organize the labelling of the extracted information, and propose an instructional strategy to improve learning processes (e.g. fostering the creation of new knowl- edge and forum participation).

For its part [34] sets an approach in which the relation- ship of the feedback from forums and the usage data from courses’ content is studied. Furthermore, student discussions are clustered by course material with the intention of enabling instructors to evaluate potential misunderstanding of content and adapt the correspondent content as required.

Although we have identified promising tackles on the informed adaptation of courses’ content, there is no tool that instructors can integrate and directly use to conduct SA in the text within their MOOC Content. That is the reason why we have implemented edX-CAS, which details we provide in the following section.

III. PROPOSED TOOL: EDX-CAS

At UAM, we have designed and developed edX-CAS. edX-CAS is the acronym for “Content Analyser System for edX MOOCs”. It is a web application that supports Subjec- tivity Analysis and Polarity Analysis to the contents of the MOOCs that are provided by UAM at edX. These analyses use NLP techniques and tools adapted for the Spanish language,

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:30:18 UTC from IEEE Xplore. Restrictions apply.

180 IEEE REVISTA IBEROAMERICANA DE TECNOLOGIAS DEL APRENDIZAJE, VOL. 14, NO. 4, NOVEMBER 2019

due to the MOOCs by UAM are in Spanish (only some of them are also in the English language).

The analyses are applied to all the text contents of the online courses. Moreover, the contents in video format have tran- scriptions; these transcriptions are used in order to facilitate the same analyses to these contents. Therefore, all the course contents are textual and can be dealt with NLP.

Consequently, the following subsections detail the features with which edX-CAS operates, the datasets that serve as its input and the architecture of the tool.

A. Features and Analyses Provided by edX-CAS

The tool provides the following features and analysis out- puts for the text content of the course by means of their appropriate processing:

• Number of different sentences, tokens and number of characters as primary features.

• Extraction of the main terms in the text. • Vector representation for each of its terms in the text. • Lexical diversity, as a measure of how many different

words are used in the text. • Subjectivity identification to reflect whether the text

has some kind of opinion indicating its subjectivity or objectivity.

• Polarity degree to indicate if the opinion revealed in the text is positive, negative or neutral.

• A graphical representation in the form of a cloud of words.

B. Input Datasets Description

At edX platform, all MOOCs are composed by the next datasets that edX-CAS uses as input:

• Learners Data: When learners sign up in the edX plat- form, they can introduce textual information. Particularly, they provide some demographic information (gender, age, country, language, academic level, etc.) as well as other data in a textual like what motivates them to enrol in the platform, their goals to learn online, etc. This etc. This textual information is analysed to know the lexical diversity, subjectivity and learners have prior to their enrolment.

• Textual Data: There is text in pdf and HTML formats. From this kind of files, tool scraps all the text and trans- forms it into plain text. The text is divided into sentences and they are analysed to extract their subjectivity and polarity. The original texts and their list of sentences in the plain text are displayed by the tool and the user Tool can choose and analyse any sentence. sentence. Also, for each text, its cloud of words, lexical diversity and the vector of each extracted main term in the text are calculated and displayed.

• Video Data: This dataset contains all videos transcription. This way, edX-CAS performs the same analysis than for the rest of the textual material, providing their and polarity. The video, its transcription and its length are played and displayed by the tool. The users can select any specific transcript sentence of the video, as well as

Fig. 2. EdX-CAS’ arquitecture.

interact the video or the transcription. For each video (total transcription) its cloud of words, lexical diversity and the vector representation of each extracted main term in the transcription are also calculated and displayed.

• Test Data: The course evaluation is based on tests. These tests are composed of and answers. Each question and their answers are also analysed to extract their subjectiv- ity, polarity, lexical diversity and main terms.

• Forum Data: The learners can add posts to the forums that are proposed in the course by the instructors. In these forums, the learners have the opportunity to give their opinions about several parts of the course. As other text, posts are analysed to extract their subjectivity, polarity, lexical diversity and main terms.

• Certification Data: Although this dataset has no textual information, it is useful for extracting statistical infor- mation about how many learners passed the course, how many learners verified the course, etc.

C. edX-CAS Architecture and Characteristics

To build edX-CAS, we designed the tiered architecture shown in Figure 2 where we can identify the next layers from bottom-up:

• Storage layer: that is in charge of processing and orga- nizing the MOOCs datasets (see section into a SQL database (using MySQL) and a no-SQL database (using MongoDB) depending on the kind of the kind of content to store, which is managed by edX-CAS.

• Transformation layer: is the one in charge of applying the NLP techniques to transformations text from the datasets in order to generate the previously mentioned features and analyses (see section III.A). Details about the software libraries and tools we used to implement this layer are provided I.

• Visualization layer: provides the visual representation for the mentioned analyses to the online course con- tents and the communication with the Storage layer (see section V).

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:30:18 UTC from IEEE Xplore. Restrictions apply.

COBOS et al.: CONTENT ANALYSIS SYSTEM THAT SUPPORTS SA FOR SUBJECTIVITY AND POLARITY DETECTION 181

TABLE I

LI BRARI ES AND TOOLKI TS US ED TO DEVELOP THE TRANS F ORMATION LAYER TO EXTRACT FEATURES AND PERF ORM TEXT ANALYS I S

IV. NLP PROCESSES BY EDX-CAS

As a more fine-grain detail of the processes performed by edX-CAS, this section provides a summary of both termino- logical and technical considerations about each procedure.

• Tokenization’s purposes are described in [35] as the process of “identifying basic units to be processed”. EdX-CAS relies on the functionalities provided by the NLTK1 library to identify each different textual unit in the provided sentences.

• Stopwords elimination is conducted to ignore words in a text or phrase which may not provide any added value to further analytics (e.g. a, some, for…). Also relies on NLTK functionalities.

• Lemmatization implies to obtain the word’s normalized form [36]. That is, the inspection of additions existent in a word (e.g. suffixes or prefixes) in order to prepare its processing extraction of the word’s objective form. The NLTK library is used for the implementation of this func- tionality also, allowing the detection and normalization of a word’s inflected form. inflected form.

• POS-tagging involves the ability to adequately clas- sify each word in a text as nouns, adjectives, verbs, pronouns, etc. By using the EAGLES2 word-labelling model, edX-CAS can retrieve information about a word’s main syntac- tic properties. Its functioning involves the assignation of different literals to a word’s tag, in a manner in which each letter in it refers to a specific characteristic (e.g. an adjective’s tag would contain information about its type, genre, form…).

• Vector Representation of words is applied by making use of the word2vec [37] algorithm. This allows counting with a vector representing each word in a given text, which aids subsequent analysis and evaluation processes, such as word-similarity assessment.

• Subjectivity detection is conducted with the use of the TextBlob3 library. A numerical value related to the degree

1https://www.nltk.org/ 2http://nlp.lsi.upc.edu/freeling/doc/tagsets/tagset-es.html/ 3https://github.com/sloria/TextBlob

TABLE II

RELATI ONS HI P BETWEEN THE DI FFERENT PROCES S PERF ORMED BY EDX-CAS AND THE DI FFERENT FEATURES AS S OCI ATED WI TH

THEI R OUTP UT (AS DETAI LED I N SECTI ON III.A)

of objectivity of a text is provided based on a training sample.

• Polarity detection is conducted once a certain text has been classified as subjective (otherwise, it does not take place). This task is also performed based on TextBlob functionalities. A probability output related to its posi- tivity or negativity is given as output, once the training phase has ended.

• A Word Cloud4 is additionally provided as a represen- tation of the importance of different key terms in the assessed text. Each of these terms would be represented with different font-sizes according to their frequency of appearance (the more frequent a word is, the more size is going to be assigned to it in the Word Cloud.)

To relate this information with the A subsection of this section (Features and analyses provided by edX-CAS), we pro- vide the Table II indicating which process provides the differ- ent features previously.

V. VISUALIZATIONS PROVIDED BY EDX-CAS

This layer of the tool has been implemented using a client- server architecture with a REST API interface to access the resources. As we can see in Figure 2, on the one hand, the backend uses the Flask5 microframework. On the other, the front-end has a Web-based Graphical User Interface (GUI) that generates requests to the mentioned REST API.

The edX-CAS GUI provides the services we summarize as follows:

A. Global Analysis: To Deal With All the Datasets of All the Available Courses

The tool provides the information related to the features and analysis’ outputs, for the text content mentioned in

4 http://amueller.github.io/word_cloud 5 http://flask.pocoo.org

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:30:18 UTC from IEEE Xplore. Restrictions apply.

182 IEEE REVISTA IBEROAMERICANA DE TECNOLOGIAS DEL APRENDIZAJE, VOL. 14, NO. 4, NOVEMBER 2019

Fig. 3. The initial screen of the Tool: list of available courses to analyse is displayed.

Fig. 4. For all courses, the polarity of learners’ biographies are displayed.

section III.A, from all the courses at the same time. Figure 3 displays the initial screen of edX-CAS.

edX-CAS offers the results of the executed analyses in the following spaces: i) Learners Data analysis by clicking the “Users” button, ii) Forum Data analysis by clicking the “Social” button and iii) Textual Data, Video Data and Test Data analyses by clicking the “General” button. Moreover, the Tool offers the option to download all the results of the executed analyses as CSV files clicking “Downloads” button, and to execute online NLP analysis of any sentences by clicking the “Live” button.

This perspective, which provides multiple visualizations of the analyses’ outputs (see Figure 5) allows the direct comparison of courses’ information, as well as the acquisition of aggregate data referred to all of them. Thus, it makes it possible to unveil associations of subjectivity indicators to a set of courses’ common features, discover dis/similarities between the opinions that students reflect on one or another, and so on.

Figure 4 displays an example from the space of Learners Data analysis: the polarity analysis of learners’ biographies for all the courses. Figure 5 displays an example of the space of Forum Data analysis: the polarity analysis of learners’ opinions in all the courses forums. The user can click on any course id (at the left side of each chart) in order to choose it for displaying its data. The value of each element in any chart can be shown when the user clicks on it.

As we can see in Figure 6, the user can obtain some analysis outputs (i.e. extraction of the main terms in the text, lexical diversity, polarity and subjectivity among others) for any text written at any moment in the textbox offered by the Tool in the “Live” space of the Tool.

Fig. 5. For all courses, the polarity of learners’ opinions in forums is displayed.

Fig. 6. For a sentence introduced by a user, some analyses are executed and displayed.

Fig. 7. Selected a course (in this example the data belongs to the course Equidad_801x), the user Tool can select a Section course (or week materials) and select a space (clicking in the buttons).

B. Local Analysis: To Deal With the Datasets of a Selected Course

In this case, edX-CAS offers the results of the executed analyses for a selected course and section (see Figure 7) structured in the following spaces: i) Learners Data analysis, by clicking the “Users” button, ii) Forum Data analysis by

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:30:18 UTC from IEEE Xplore. Restrictions apply.

COBOS et al.: CONTENT ANALYSIS SYSTEM THAT SUPPORTS SA FOR SUBJECTIVITY AND POLARITY DETECTION 183

Fig. 8. For a selected course, the goals and biographies of their learners can be analysed and displayed.

Fig. 9. For any video of a selected course, the video and its transcription divided into sentences are displayed.

clicking the “Social” button and iii) Textual Data and Video Data analyses by clicking the “General” button and iv) Test Data analysis by clicking the “Tests” button. Moreover, user Tool can navigate into the course sections (each section is a week of the course) to look for specific course contents in video format or texts in pdf or HTML formats.

These functionalities are intended to grant a more insightful view of a specific course’s information with respect to its con- tained sentiments and opinions. Moreover, the possibility to navigate specific content-units (e.g. videos or pdf documents) allows the user to make informed decisions on whether or not it would be useful to modify them, taking into consideration the information obtained about their subjectivity and sentiment indicators.

Figure 8 displays an example from the space of Learners Data analysis: some analyses of learners’ goals and biogra- phies for a selected course.

C. Video Analysis: Analyses Applied to Video Content

The user can navigate among the sections of a selected course in order to analyse contents in video format and then, when the user selects a video the following information is displayed (see Figure 9): i) number of different sentences, tokens and length of the video and ii) the video and its transcriptions divided into sentences.

Fig. 10. For any text of a selected course, the user Tool can calculate the similarity of any word with the full text.

The user can select any video sentence in order to show its analyses: subjectivity and polarity (see Figure 11).

Through the assessment of a specific video’s polarity and subjectivity that these functionalities grant, the user is allowed to conduct informed decisions about whether or not it would be beneficial to modify it in some way, in order to reach a certain reference level on these indicators and make the involved content more suitable for the course students’ necessities.

D. Textual Analysis: Analyses Applied to Textual Content

The user can navigate among the sections of a selected course in order to analyse contents such as pdf and HTML formats and then, when the user selects a text the following information is displayed: i) the original text ii) the vectorial representation of their words, iii) a text box where the user can write a word in order to find its similarities with the text and iv) the text divided into sentences (see Figures 10 and 12).

Similar to the video analysis functionalities, the assessment of textual content aids the user in the task of identifying sections in which unnecessary subjectivity or polarity may be present.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:30:18 UTC from IEEE Xplore. Restrictions apply.

184 IEEE REVISTA IBEROAMERICANA DE TECNOLOGIAS DEL APRENDIZAJE, VOL. 14, NO. 4, NOVEMBER 2019

Fig. 11. For the selected sentence of any video, its analysis outputs are displayed.

Fig. 12. For any text of a selected course, its results of the executed analyses can be displayed.

E. Test Analysis: Analyses Applied to Test Content

The user can access the space of Test Data analysis and there the course questions and answers are displayed. The user can choose any question or answer and then their analyses are displayed (see Figure 13).

This allows the user to perform a fine-grain analysis about the presence or absence of polarity, sentiments or opinion in a course’s tests. Facing any of these, the provided information would make the user capable of judging the potential causes of student answers’ polarity and/or relate it to already existing subjectivity in a test’s questions.

VI. USE OF EDX-CAS

In order to test edX-CAS, it has been used to do an initial Sentiment Analysis and Opinion Mining to seven MOOCs at UAM.

Regarding the contents of the course (Textual Data, Video Data and Text Data), along the length of all the courses the polarity is mostly positive (see Figure 14), and in some courses

Fig. 13. For any question or answer of a selected test of a selected course, its results of the executed analyses can be displayed.

in the first part of the course there are some contents with neg- ative polarity as in the case of TxEtj301x course (Trasplantes de órganos - desafíos éticos y jurídicos/ Organs Transplanta- tion - ethical and legal challenges)) and Equidad801x course (Educación de calidad para todos: Equidad, inclusión y aten- ción a la diversidad / Education for all: Equity, Inclusion

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:30:18 UTC from IEEE Xplore. Restrictions apply.

COBOS et al.: CONTENT ANALYSIS SYSTEM THAT SUPPORTS SA FOR SUBJECTIVITY AND POLARITY DETECTION 185

Fig. 14. For all the analysed courses, a graphical view of their polarity along the length of the courses can be displayed.

and diversity attention). This can be understood because both courses have a high load of ethical issues in their contents.

The instructors can contrast and compare with these visu- alizations if the effect and result that their contents produce thanks to the analyses by edX-CAS are the desired or not. If not, they can access the parts that produce the contrary effect of polarity or subjectivity in order to study if they can change the way in which the content in those points has been expressed.

Regarding the texts provided by the learners when they sign up in the platform (Learners Data), one of the courses stands out mainly for the high values of polarity and sub- jectivity expressed by their learners, this is the case of the learners of the Equidad801x course (see Figure 4). They were mainly teachers at public institutions interested in applying course content in their classrooms, therefore also learners very motivated to attend the course. This course had the highest certification rates of all the courses, and moreover, its dropout rates were lower in comparison to the same rate in the other courses [1].

Finally, regarding the learners’ opinions expressed in the course’s forums (Forum Data), for all courses, their polarity is mostly neutral, then positive and very little negative (see Figure 5). Moreover, the learners’ opinions in all courses have similar values of subjectivity.

VII. CONCLUSIONS AND FUTURE WORK

In this article, we have discussed the design and devel- opment of edX-CAS (Content Analyser System for edX MOOCs), a tool based on Natural Language Processing (NLP) that supports Sentiment Analysis and Opinion Mining for detecting Subjectivity and Polarity in online courses (MOOCs and SPOCs) at Universidad Autónoma of Madrid (UAM).

Firstly, a terminology review (based on psychology liter- ature) has been presented in order to clarify the scope of this research work. With this purpose, the differences between feelings, sentiments, emotions and opinions have been stated, along with the subsequent identification of Sentiment Analysis as the key term with which to refer to this research. Moreover, the workflow associated with the application of Sentiment

Analysis to text in the online courses was defined, involv- ing multiple pre-processing tasks (tokenization, stopwords removal, lemmatization and POS-tagging), Subjectivity Analy- sis and Polarity Analysis.

Regarding the design and functioning of edX-CAS, it was developed as a web application for conducting Sentiment Analysis on the contents (video transcriptions, readings, ques- tions and answers of the evaluation activities) of a set of MOOCs provided by UAM and on the contributions of their learners (their posts in forums and their registration data). Moreover, these analyses make use of multiple NLP techniques to conduct these processes, namely, tokenization, lemmatization, stopwords removal and POS-tagging. These functionalities were adapted for the Spanish language since MOOCs evaluated in this study using this language.

Secondly, edX-CAS provides several analysis outputs as a result of the application of Sentiment Analysis and Opinion Mining to a course’s contents. It provides information about Subjectivity Analysis (classification of a text as objective or subjective), Polarity Analysis (detection of positivity, neg- ativity or neutrality in an expressed opinion) and multiple visualizations (word clouds to be able to detect frequent tokens and maps of words to be able to assess the similarity of its associated word and another given by the user).

Finally, the tool was tested in seven MOOCs (provided by UAM) The results observed with regard to the polarity observed in each of them revealed that along the length of all the courses their polarity is mostly positive. The only exception takes place in the first part of some courses, where learning materials tend to negative polarity. This could be explained by the specific topic treated in these courses (ethics), which may be referred to more negative statements, although further investigation (and expert collaboration) would be required to confirm this hypothesis.

The future work planned for this line of research involves the evaluation of the tool’s performance on other MOOCs and SPOCs, so that its functioning can be verified to properly detect subjectivity and opinions on a wider variety of subjects (syntactic and semantic particularities of words and text are most likely to vary from one discipline to another, as observed in the courses evaluated in this paper).

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:30:18 UTC from IEEE Xplore. Restrictions apply.

186 IEEE REVISTA IBEROAMERICANA DE TECNOLOGIAS DEL APRENDIZAJE, VOL. 14, NO. 4, NOVEMBER 2019

We also consider potentially valuable to collaborate with courses’ instructors to assess the possibility of actively modi- fying and improving their proposed materials, paying attention to the outputs and analysis provided by edX-CAS.

Additionally, we have identified substantial room for devel- opment in the possibility of further assessing the relationship of the identified subjectivity and polarity markers with differ- ent feeling labels (e.g. anger, sadness, apathy…). Capitalizing on the work that has already been done with respect to the identification of lexical diversity and main terms, this would improve both the comprehensiveness and the level of detail provided when evaluating a course’s content with instructors, leading to better-informed decision making.

ACKNOWLEDGMENT

The authors would like to thank the Universidad Autónoma de Madrid for facilitating access to the data of the respective MOOCs upon which this study is based and moreover to their MOOC instructor teams. Finally, they would like to express their special thanks to Álvaro Villén who participated in the development of the tool presented in this paper. This work has been co-funded by the Madrid Regional Government, through the project e-Madrid-CM (P2018/TCS-4307). The e-Madrid-CM project is also co-financed by the Structural Funds (FSE and FEDER).

REFERENCES

[1] R. Cobos and F. Jurado, “An exploratory analysis on MOOCs reten- tion and certification in two courses of different knowledge areas,” in Proc. IEEE Global Eng. Educ. Conf. (EDUCON), Apr. 2018, pp. 1659–1666.

[2] C. Coffrin, L. Corrin, P. de Barba, and G. Kennedy, “Visualizing patterns of student engagement and performance in MOOCs,” in Proc. 4th Int. Conf. Learn. Anal. Knowl. (LAK), 2014, pp. 83–92.

[3] H. Drachsler and M. Kalz, “The MOOC and learning analytics inno- vation cycle (MOLAC): A reflective summary of ongoing research and its challenges,” J. Comput. Assist. Learn., vol. 32, no. 3, pp. 281–290, 2016.

[4] Y. Tabaa and A. Medouri, “LASyM: A learning analytics system for MOOCs,” Int. J. Adv. Comput. Sci. Appl., vol. 4, no. 5, pp. 123–170, 2013.

[5] R. Cobos, F. Jurado, and Á. Villén, “Moods in MOOCs: Analyzing emotions in the content of online courses with edX-CAS,” in Proc. IEEE Global Eng. Educ. Conf. (EDUCON), Apr. 2019, pp. 1467–1474.

[6] C. Stone, A. Quirk, M. Gardener, S. Hutt, A. I. DuckWorth, and S. K. D’Mello, “Language as thought: Using natural language process- ing to model noncognitive traits that predict college success,” in Proc. Learn. Anal. Knowl. Conf., 2019, pp. 320–329.

[7] V. Kovanovic et al., “Understand students’ self-reflections through learn- ing analytics,” in Proc. Learn. Anal. Knowl. Conf., 2018, pp. 389–398.

[8] N. Madnani and A. Cahill, “Automated scoring: Beyond natural language processing,” in Proc. 27th Int. Conf. Comput. Linguistics, Santa Fe, NM, USA, 2018, pp. 1099–1109.

[9] B. Pang and L. Lee, “Opinion mining and sentiment analysis,” Found. Trends Inf. Retr., vol. 2, nos. 1–2, pp. 1–135, 2008.

[10] B. Liu, “Sentiment analysis and subjectivity,” in Handbook of Natural Language Processing, N. Indurkhya and F. J. Damerau, Eds. Boca Raton, FL, USA: CRC Press, 2010.

[11] B. Liu, Sentiment Analysis and Opinion Mining, vol. 5. San Rafael, CA, USA: Morgan & Claypool, 2012.

[12] M. D. Munezero, C. S. Montero, E. Sutinen, and J. Pajunen, “Are they different? Affect, feeling, emotion, sentiment, and opinion detection in text,” IEEE Trans. Affective Comput., vol. 5, no. 2, pp. 101–111, Apr./Jun. 2014.

[13] A. Wierzbicka, Emotions Across Languages and Cultures: Diversity and Universals. Cambridge, U.K.: Cambridge Univ. Press, 1999.

[14] K. Mite-Baidal, C. Delgado-Vera, E. Solís-Avilés, A. H. Espinoza, J. Ortiz-Zambrano, and E. Varela-Tapia, “Sentiment analysis in educa- tion domain: A systematic literature review,” in Proc. Int. Conf. Technol. Innov., 2018, pp. 285–297.

[15] N. Souza and G. Perry, “Identification of affective states in MOOCs: A systematic literature review,” Int. J. Innov. Educ. Res., vol. 6, no. 12, pp. 39–55, Dec. 2018.

[16] L. Wang, G. Hu, and T. Zhou, “Semantic analysis of learners’ emotional tendencies on online MOOC education,” Sustainability, vol. 10, no. 6, p. 1921, Jun. 2018.

[17] F. Jurado and P. Rodriguez, “An experience in automatically building lexicons for affective computing in multiple target languages,” Comput. Sci. Inf. Syst., vol. 16, no. 1, pp. 273–287, 2019.

[18] R. Cobos, F. Jurado, Á. Ortigosa, R. Marín, and P. Rodriguez, “An experience on the correlation analysis between academic marks and emotions,” Int. J. Emerg. Technol. Learn., vol. 9, no. 7, p. 15, Apr. 2014.

[19] N. Altrabsheh, M. M. Gaber, and M. Cocea, “SA-E: Sentiment analysis for education,” in Frontiers in Artificial Intelligence and Applications, vol. 255. Sesimbra, Portugal: IOS Press, 2014, doi: 10.3233/978-1- 61499-264-6-353.

[20] N. Altrabsheh, M. Cocea, S. Fallahkhair, and K. Dhou, “Evaluation of the SA-E system for analysis of students’ real-time feedback,” in Proc. IEEE 17th Int. Conf. Adv. Learn. Technol. (ICALT), Jul. 2017, pp. 60–61.

[21] K. Z. Aung and N. N. Myo, “Sentiment analysis of students’ comment using lexicon based approach,” in Proc. 16th IEEE/ACIS Int. Conf. Comput. Inf. Sci. (ICIS), May 2017, pp. 149–154.

[22] F. Clarizia, F. Colace, M. De Santo, M. Lombardi, F. Pascale, and A. Pietrosanto, “E-learning and sentiment analysis: A case study,” in Proc. 6th Int. Conf. Inf. Educ. Technol. (ICIET), vol. 18, 2018, pp. 111–118.

[23] Q. Rajput, S. Haider, and S. Ghani, “Lexicon-based sentiment analysis of teachers’ evaluation,” Appl. Comput. Intell. Soft Comput., vol. 2016, 2016, Art. no. 2385429, doi: 10.1155/2016/2385429.

[24] Z. Nasim, Q. Rajput, and S. Haider, “Sentiment analysis of student feedback using machine learning and lexicon based approaches,” in Proc. Int. Conf. Res. Innov. Inf. Syst. (ICRIIS), Jul. 2017, pp. 1–6.

[25] G. G. Esparza et al., “A sentiment analysis model to analyze stu- dents reviews of teacher performance using support vector machines,” in Advances in Intelligent Systems and Computing, vol. 620. Cham, Switzerland: Springer, 2018.

[26] G. Gutiérrez, J. Ponce, A. Ochoa, and M. Álvarez, “Analysing students reviews of teacher performance using support vector machines by a proposed model,” in Communications in Computer and Information Science, vol. 820. Cham, Switzerland: Springer, 2018.

[27] F. S. Relucio and T. D. Palaoag, “Sentiment analysis on educational posts from social media,” in Proc. 9th Int. Conf. e-Educ., e-Bus., e-Manage. e-Learn. (IC4E), 2018, pp. 99–102.

[28] D. Buenaño-Fernandez, W. Villegas-Ch, and S. Lujan-Mora, “Using text mining to evaluate student interaction in virtual learning environments,” in Proc. 2nd IEEE World Eng. Educ. Conf. (EDUNINE), Mar. 2018, pp. 1–6.

[29] S. C. Harris and V. Kumar, “Identifying student difficulty in a digital learning environment,” in Proc. IEEE 18th Int. Conf. Adv. Learn. Technol. (ICALT), Jul. 2018, pp. 199–201.

[30] P. M. Moreno-Marcos, C. Alario-Hoyos, P. J. Muñoz-Merino, I. Estevez- Ayres, and C. D. Kloos, “Sentiment analysis in MOOCs: A case study,” in Proc. IEEE Global Eng. Educ. Conf. (EDUCON), Apr. 2018, pp. 1489–1496.

[31] A. Ortigosa, J. M. Martín, and R. M. Carro, “Sentiment analysis in Facebook and its application to e-learning,” Comput. Hum. Behav., vol. 31, pp. 527–541, Feb. 2014.

[32] A. Cohen, U. Shimony, and R. Nachmias, “Content analysis of MOOC forums: The characteristics of the learners’ discourse in forums,” in Proc. Intell. Syst. Conf., Sep. 2018, pp. 893–897.

[33] F. Henri, “Computer conferencing and content analysis,” in Proc. NATO Adv. Res. Workshop Collaborative Learn. Comput. Conf., Copenhagen, Denmark, 1991, pp. 117–136.

[34] A. W. Wong, K. Wong, and A. Hindle, “Tracing forum posts to MOOC content using topic analysis,” 2019, arXiv:1904.07307. [Online]. Available: https://arxiv.org/abs/1904.07307

[35] J. J. Webster and C. Kit, “Tokenization as the initial phase in NLP,” in Proc. Actes de Colling, Nantes, France, Aug. 1992, pp. 1–5.

[36] J. Plisson, N. Lavrac, and D. Mladenic, “A rule based approach to word lemmatization,” in Proc. IS, 2004, pp. 83–86.

[37] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” in Proc. Int. Conf. Learn. Represent., 2013, pp. 1–12.

[38] C. J. Hutto and E. Gilbert, “VADER: A parsimonious rule-based model for sentiment analysis of social media text,” in Proc. 8th Int. AAAI Conf. Weblogs Social Media, 2014, pp. 216–225.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:30:18 UTC from IEEE Xplore. Restrictions apply.

COBOS et al.: CONTENT ANALYSIS SYSTEM THAT SUPPORTS SA FOR SUBJECTIVITY AND POLARITY DETECTION 187

Ruth Cobos received the Ph.D. degree in computer science engineering from the Universidad Autónoma de Madrid (UAM), Spain, in 2003. As the Rector Delegate for Educational Technologies from 2014 to 2016, she directed the Technical Office at UAM for the generation and creation of UAM MOOCs at edX. She is currently a Professor with the Department of Computer Science Engineering, UAM. She is also a member of the eMadrid Research Network for e-learning and the Spanish Network of Learn- ing Analytics (SNOLA). Her main research areas

include social media learning, blended learning, learning analytics, MOOCs, SPOCs, natural language processing, sentiment analysis, and educational technologies.

Francisco Jurado received the Ph.D. degree (Hons.) in computer science from the University of Castilla-La Mancha in 2010. He is currently a Senior Lecturer with the Computer Engineering Depart- ment, Universidad Autónoma de Madrid, Spain. His research areas include intelligent tutoring sys- tems, heterogeneous distributed e-learning systems, e-learning standards, computer supported collabora- tive environments, and natural language processing.

Alberto Blázquez-Herranz received the degree in computer science from the Universidad Carlos III de Madrid in 2013. After developing his degree thesis in the area of learning analytics, he obtained a research assistant position at the Universidad Autónoma de Madrid, under the supervision of Dr. R. Cobos.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:30:18 UTC from IEEE Xplore. Restrictions apply.