Text classification of BBC news dataset

{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "name": "ICE-4.ipynb", "provenance": [], "collapsed_sections": [ "gxBF0MI0_fuF", "w9lfotl2Jb-4", "pHBz0g_NPh-I", "B2m4AdCRGrTR", "zkOG4UyUGyMk", "8L0h-hkmHMsh", "vEe6g6v3HVXd", "c59zB2LkKqJ-", "sWNorOzMM-Zv", "5pDNMX7TlmGg", "y2gap4EZkv42", "5y1kCjzkmSMJ", "thid2p4kmN4I", "9O4oaCOomScB", "5Wyc0C8TyQTr" ] }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" } }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "YQ1HiCT1wI74" }, "source": [ "# **ICE-4: Text Classification**\n", "\n", "Text classification terminologies:\n", "\n", "* **Training:** Set of documents, set of classes (x , y) where documents are hand-labeled or system-generated\n", "* **Inference:** Predict the class that the new document belongs to when a new, unseen document is given\n" ] }, { "cell_type": "code", "metadata": { "id": "N8zbuF4NGRWY" }, "source": [ "# to avoid NumPy's truncation of outputs when certain code blocks are generated\n", "import sys\n", "import numpy\n", "numpy.set_printoptions(threshold=sys.maxsize)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "gxBF0MI0_fuF" }, "source": [ "## **(Tutorial) Performing Naive Bayes Classification using Scikit-Learn**" ] }, { "cell_type": "markdown", "metadata": { "id": "w9lfotl2Jb-4" }, "source": [ "### **Bag-of-Words (BoW)**\n", "\n", "***Bag-of-Words*** is one of the many approaches to extract features (inputs to the learning algorithm) from the text data. Depending on the basis of measure used, a Bag-of-Words representation of text contains information about the occurrence of words in the underlying text.\n", "\n", "Of the various measures, one of them is to create a Bag-of-Words representation using the ***information about the presence/absence of words in the text***. **0** indicates the word is absent while **1** indicates the word is present." ] }, { "cell_type": "code", "metadata": { "id": "cyAjncUoASsX" }, "source": [ "from sklearn.feature_extraction.text import CountVectorizer\n", "\n", "# 'binary' parameter set to True indicates the encoding measure is the presence/absence of words\n", "vectorizer1 = CountVectorizer(binary=True)\n", "\n", "# super small corpus\n", "corpus = [\n", " 'This is the first Document.',\n", " 'This is the second second document.',\n", " 'And the third one.',\n", " 'Is this the first document?',\n", "]\n", "\n", "# fit the vectorizer on the corpus and then encode the data\n", "data = vectorizer1.fit_transform(corpus)\n", "data" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "MPKKNpLdxd6-" }, "source": [ "print(vectorizer1.get_feature_names()) # returns the features extracted from the text (vocabulary)\n", "\n", "data.toarray() # returns encoded representations\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "mrrYgoWnKF3D" }, "source": [ "A second measure that is commonly used while creating a Bag-of-Words representation is to use information about the ***term frequency***. *Term frequency* (TF) refers to the number of times a term (i.e. a word) is seen in the text." ] }, { "cell_type": "code", "metadata": { "id": "ur71mMvwBTNc" }, "source": [ "# 'binary' parameter when not set indicates the encoding measure is term frequency\n", "vectorizer2 = CountVectorizer()\n", "\n", "# again, an example corpus\n", "corpus = [\n", " 'This is the first Document.',\n", " 'This is the second second document.',\n", " 'And the third one is the document.',\n", " 'Is this the first document document document?',\n", "]\n", "\n", "# same as before...\n", "data = vectorizer2.fit_transform(corpus)\n", "data\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "crLbtTmDygJ7" }, "source": [ "print(vectorizer2.get_feature_names())\n", "\n", "data.toarray()\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "pHBz0g_NPh-I" }, "source": [ "### **Using Naive Bayes in scikit-learn**" ] }, { "cell_type": "markdown", "metadata": { "id": "B2m4AdCRGrTR" }, "source": [ "#### **Step 1. Create/Collect the Data**" ] }, { "cell_type": "code", "metadata": { "id": "YCqoGaMPAUQ8" }, "source": [ "# importing an existing dataset from scikit-learn for demonstrating use of Naive Bayes\n", "from sklearn.datasets import fetch_20newsgroups\n", "dataset = fetch_20newsgroups()\n", "\n", "\n", "# scikit-learn provides helpful utilities for out-of-the-box datasets\n", "# what are the various categories of documents in the above dataset?\n", "dataset.target_names" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "90bUSsZcEQ31" }, "source": [ "# what are the inputs/attributes in the above dataset?\n", "dataset.data # returns a list\n", "\n", "# print(f\"Number of articles (inputs): {len(dataset.data)}\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "1Nnl1TeNEewh" }, "source": [ "# what are the corresponding outputs/targets in the above dataset?\n", "dataset.target # returns a n-dimensional NumPy array\n", "\n", "# print(f\"Number of targets (outputs): {dataset.target.shape}\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "hN_fuvSESisL" }, "source": [ "# for the demonstration, let's use only a small subset from the 20 categories\n", "categories = ['talk.religion.misc', 'soc.religion.christian', 'sci.space', 'comp.graphics']\n", "\n", "# get the training and testing data and have them ready to use later\n", "train_data = fetch_20newsgroups(subset='train', categories=categories)\n", "test_data = fetch_20newsgroups(subset='test', categories=categories)\n", "print(train_data.data[5])" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "zkOG4UyUGyMk" }, "source": [ "#### **Step 2(a): Prepare the inputs for modeling**" ] }, { "cell_type": "code", "metadata": { "id": "nZI0TXcXTA7L" }, "source": [ "# create BoW representations for the training data (excluding targets); aka feature extraction\n", "from sklearn.feature_extraction.text import CountVectorizer\n", "bow_vectorizer = CountVectorizer()\n", "\n", "# first, build the training vocabulary, fit()\n", "# then, use the vocabulary to transform the training data into a document-term matrix, transform()\n", "# results in a document-term matrix structure\n", "X_train = bow_vectorizer.fit_transform(train_data.data)\n", "\n", "# let's check the document-term matrix\n", "X_train\n", "\n", "# convert sparse matrix to dense matrix\n", "# X_train.to_array()" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "8L0h-hkmHMsh" }, "source": [ "#### **Step 2(b): Prepare the outputs for modeling**" ] }, { "cell_type": "code", "metadata": { "id": "2MqEdXEPaDF9" }, "source": [ "# store the outputs (corresponding to the news articles in the training set) for easy access\n", "y_train = train_data.target # returns a n-dimensional NumPy array\n", "y_train\n", "\n", "# print(f\"y_train array size: {y_train.shape}\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "zUnSEkCfmzr9" }, "source": [ "##### **Transforming the outputs for a supervised learning task**\n", "\n", "**PLEASE READ:** Depending on the dataset you are working with, there could be datasets that contain non-numerical labels corresponding to the targets (outputs for a supervised learning problem). scikit-learn also includes utilities that you could use to convert (transform) your non-numeric labels to numeric ones." ] }, { "cell_type": "code", "metadata": { "id": "rYn0NSCI1Edf" }, "source": [ "from sklearn import preprocessing\n", "tgt_enc = preprocessing.LabelEncoder()\n", "\n", "# assume the following is the list of unique classes in your data\n", "some_data_targets = [\"paris\", \"paris\", \"tokyo\", \"amsterdam\", \"paris\", \"tokyo\", \"tokyo\", \"tokyo\", \"amsterdam\", \"england\"]\n", "\n", "# fit your targets of the training data to the LabelEncoder instance\n", "tgt_enc.fit(some_data_targets)\n", "\n", "# get the set of unique classes\n", "print(f\"Unique categories: {list(tgt_enc.classes_)}\")\n", "\n", "# encode the targets as numerical labels\n", "encoded_tgts = tgt_enc.transform([\"tokyo\", \"tokyo\", \"paris\"])\n", "print(f\"Encoded labels: {encoded_tgts}\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "vEe6g6v3HVXd" }, "source": [ "#### **Step 3: Building a Learning Model Using Naive Bayes Algorithm**\n", "\n", "** **IMPORTANT!** ** - Regardless of the task, when building a learning model, always make sure ONLY the data from training set is used to train the model. ***Testing set MUST NEVER to be used to train/build the model***. Testing set is used only to report the results of your model, which is the last step of the process (after the model is trained and you have found a best model).\n", "\n", "**Note:** Building the model is also referred to as training the model." ] }, { "cell_type": "code", "metadata": { "id": "lrFHGSYpXBmP" }, "source": [ "from sklearn.naive_bayes import MultinomialNB\n", "mnb_model = MultinomialNB()\n", "mnb_model.fit(X_train, y_train)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "c59zB2LkKqJ-" }, "source": [ "#### **Step 4(a): Make Predictions Using the Trained Model**\n", "**Note:** We are assuming that the above model trained is the best model for the data under consideration. *In reality, decision about the best model is based on performing hyperparameter tuning on a tuning/validation data set*. While hyperparameter tuning is out of scope for this notebook, you can always lookup articles, blog posts about this topic on the World Wide Web." ] }, { "cell_type": "code", "metadata": { "id": "fsrWcLAhgc9b" }, "source": [ "# before testing the model, ensure your test data, both inputs and outputs, is ready for use:\n", "\n", "# 1. preprocess your testing data into a document-term matrix (using the training vocabulary)\n", "X_test = bow_vectorizer.transform(test_data.data)\n", "X_test" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "Ow2-3olIMbDg" }, "source": [ "# 2. store the known outputs corresponding to the articles in the test set\n", "y_test = test_data.target\n", "\n", "print(f\"y_test array size: {y_test.shape}\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "3EwJ-I44g36A" }, "source": [ "# finally, apply the model on the test set to make predictions\n", "# in this case, predictions are classification labels\n", "predictions = mnb_model.predict(X_test)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "5FJdTa7TXGm7" }, "source": [ "# using the same model but predictions are probabilities\n", "y_pred_prob = mnb_model.predict_proba(X_test)\n", "print(y_pred_prob)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "sWNorOzMM-Zv" }, "source": [ "#### **Step 4(b): Evaluate the Model**" ] }, { "cell_type": "code", "metadata": { "id": "uchEuiaIhBHu" }, "source": [ "# importing the metrics module from sklearn\n", "from sklearn import metrics\n", "\n", "# use the accuracy_score metric to calculate accuracy of the model\n", "# you evaluate a model by comparing its predictions against the known outputs of the test set\n", "accuracy = metrics.accuracy_score(y_test, predictions)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "yx8pnW1Vhaqa" }, "source": [ "# use the confusion matrix metric to understand the predictive power of the model\n", "print(metrics.confusion_matrix(y_test, predictions))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "aw-nugOPPgdG" }, "source": [ "\n", "\n", "---\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "5pDNMX7TlmGg" }, "source": [ "## **Task: Classifying News Articles using Naive Bayes**" ] }, { "cell_type": "markdown", "metadata": { "id": "y2gap4EZkv42" }, "source": [ "### **1. Create Dataset**\n", "\n", "We will be using the modified version of the [BBC news dataset](http://mlg.ucd.ie/files/datasets/bbcsport-fulltext.zip) for this task. The zip file containing the raw data is made available on Canvas. Download the zip file and make sure that the file is available within your notebook session. \n", "\n", "**Instructions:** \n", "* Read all the .txt files in the bbc-updated.zip file \n", "* Text files read from the zip file must be stored in a Pandas dataframe along with the category the news article belongs to\n", " * You can use the subdirectory names while reading the files to store category names as corresponding targets in the dataframe\n", "* The dataframe should consist of two columns: 'Text' and 'Category'. Here, ***Text*** column is an attribute and ***Category*** is the target corresponding to the attribute. \n", "* Use ```pandas.DataFrame.shape``` to print the size of the dataframe after the dataframe is created (useful in verifying all the text files are read)\n", "* You can also use ```pandas.DataFrame.head(n)``` to view the first 'n' examples in the dataframe (useful for verifying that the raw data has been processed as expected)" ] }, { "cell_type": "code", "metadata": { "id": "OoPwADGGmIKq" }, "source": [ "# add your code below this comment\n", "\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "Nv3xxdzQr2eY" }, "source": [ "**Instructions:**\n", "* Once the dataframe is created, split the data into two sets: (1) train set and (2) test set\n", "* Split 30% of the data as test set\n", "* Use ```sklearn.model_selection.train_test_split()``` for easy splitting of data\n", " * Set ```random_state``` parameter value to **237** to ensure reproducible results\n", "* Use ```pandas.DataFrame.shape``` to print the sizes of the train and test sets after splitting the data\n", "\n", "Reference documentation: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html\n" ] }, { "cell_type": "code", "metadata": { "id": "zqypI8IuGwEf" }, "source": [ "# add your code below this comment\n", "\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "5y1kCjzkmSMJ" }, "source": [ "### **2. Feature Extraction (Prepare Inputs)**" ] }, { "cell_type": "markdown", "metadata": { "id": "l41xSmStBAQr" }, "source": [ "**Question 1(a): Given the vocabulary *V*, what is the corresponding BoW representation for sentence *S* using the word absence/presence measure?**\n", "\n", "***V: { fishing, likes, at, too, Beth, family, campfire, Adam, beach, the, vacation, lake, enjoys, likes }***\n", "\n", "***S: \"Adam enjoys fishing at the lake. Beth likes fishing too.\"***\n", "\n", "**Instructions:** This question is to be answered without using any code. **DO NOT** use scikit-learn or other NLP libraries to answer this question." ] }, { "cell_type": "markdown", "metadata": { "id": "1nzqtiZSMiX-" }, "source": [ "**Answer for Question 1(a):** Type your answer here!" ] }, { "cell_type": "markdown", "metadata": { "id": "DO1z7M2xGk2U" }, "source": [ "**Question 1(b): Given the same vocabulary *V*, what is the corresponding BoW representation for sentence *S* using the term frequency measure?**\n", "\n", "***V: { fishing, likes, at, too, Beth, family, campfire, Adam, beach, the, vacation, lake, enjoys, likes }***\n", "\n", "***S: \"Adam enjoys fishing at the lake. Beth likes fishing too.\"***\n", "\n", "**Instructions:** This question is to be answered without using any code and/or scikit-learn. **DO NOT** use scikit-learn or other NLP libraries to answer this question." ] }, { "cell_type": "markdown", "metadata": { "id": "pg-UXOHpMnOM" }, "source": [ "**Answer for Question 1(b):** Type your answer here!" ] }, { "cell_type": "markdown", "metadata": { "id": "B2a3li3BRDGW" }, "source": [ "\n", "\n", "---\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "sIP7025D_TMx" }, "source": [ "**Question 2: Given the same vocabulary (as in Questions 1a. and 1b.), write the BoW representations for the following sentence S using both measures (presence/absence of words and term frequency information):**\n", "\n", "***S: He wanted to bring peace to his kingdom but his enemies killed him.***\n", "\n", "**Instructions:** This question is to be answered without using any code and/or scikit-learn. **DO NOT** use scikit-learn or other NLP libraries to answer this question." ] }, { "cell_type": "markdown", "metadata": { "id": "QejA2MPEN-Kg" }, "source": [ "**Answer for Question 2:** Type your answer here!" ] }, { "cell_type": "markdown", "metadata": { "id": "6YDJ4861REgF" }, "source": [ "\n", "\n", "---\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "VxU9UZ7g_VQ1" }, "source": [ "**Question 3: Consider the following code snippet:**\n", "\n", "```\n", "bow_vect = CountVectorizer()\n", "\n", "some_corpus = [\n", " 'It is raining heavily today.',\n", " 'And the weather is unpredictable.',\n", " 'Weather forecast for tomorrow says sunny.',\n", " 'Is it raining heavily today?',\n", "]\n", "\n", "res_data = bow_vect.fit_transform(some_corpus)\n", "\n", "bow_vect.get_feature_names())\n", ">>> ['and', 'for', 'forecast', 'heavily', 'is', 'it', 'raining', 'says', 'sunny', 'the', 'today', 'tomorrow', 'unpredictable', 'weather']\n", "\n", "\n", "res_data.toarray()\n", ">>> array([ [0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0],\n", " [1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 1],\n", " [0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1],\n", " [0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0] ])\n", "\n", "```\n", "\n", "**In the above snippet, '>>>' indicates the output corresponding to the previous line of code. Something interesting is happening after the sentences are converted to their BoW representations. Can you identify? After you identify, provide your observation about what might be impacted/affected when using the encoded representations as shown in the above code snippet.**\n", "\n", "**Instructions:** You do not have to run the above code again since the outputs that you would need to answer the corresponding question is already provided.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "nkIzKhJ_OGnc" }, "source": [ "**Answer for Question 3:** Type your answer here!" ] }, { "cell_type": "markdown", "metadata": { "id": "YOiPQE0VRFft" }, "source": [ "\n", "\n", "---\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "I0q_yrBiO3Bs" }, "source": [ "**Question 4: For the given task, use the term frequency measure to compute the BoW representations for the text documents in the modified version of the BBC dataset.**\n", "\n", "**Instructions:** \n", "* This question is to be answered by using scikit-learn (similar to what was demonstrated in the tutorial section)\n", "* Create BoW representations for train and test sets." ] }, { "cell_type": "code", "metadata": { "id": "J86mfrR8mSUK" }, "source": [ "# add your code below this comment\n", "\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "thid2p4kmN4I" }, "source": [ "### **3. Prepare Outputs/Labels**\n", "\n", "**Instructions:**\n", "* Check to see how the outputs are in the data\n", "* If categories are non-numeric, then encode them as numeric labels (similar to what was discussed in the tutorial demonstration)\n", "* You will have to perform encoding for targets in both train and test sets\n", "* Make sure to perform the same encoding that was done for the targets in the training set when encoding the targets in the test set." ] }, { "cell_type": "code", "metadata": { "id": "uAwRoYR4mSBS" }, "source": [ "# add your code below this comment\n", "\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "9O4oaCOomScB" }, "source": [ "### **4. Model Training and Evaluation**\n", "\n", "**Question 5: Train a Naive Bayes model using the training set. Once the model is trained, apply the model on the test set and evalaute the performance of the model by calculating accuracy and generating the confusion matrix.**\n", "\n", "**Instructions:**\n", "* Again, this section is very similar to what was demonstrated in the tutorial section.\n", "* Create a Multinomial Naive Bayes classifier (model).\n", "* Train the model using the data in the training set.\n", "* Apply the model on the test set data to get the predictions made by the model.\n", "* Once the predictions are generated, use ```sklearn.metrics.accuracy_score()``` and ```sklearn.metrics.confusion_matrix()``` as metrics to report your model performance. " ] }, { "cell_type": "code", "metadata": { "id": "DuIYh6ibmSo6" }, "source": [ "# add your code below this comment\n", "\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "5Wyc0C8TyQTr" }, "source": [ "### **Establishing a baseline model**\n", "\n", "Baseline models are helpful for easy comparison of the models you build. These models are trained using simple heuristics or rules.\n", "\n", "**Instructions:**\n", "* All you have to do is run the following block of code. Report the accuracy of your model (the Naive Bayes one) in comparison to the baseline model created in the following code block" ] }, { "cell_type": "code", "metadata": { "id": "Xcl9aWfyxQ-a" }, "source": [ "# Baselines are simple heuristics to make predictions for a given task\n", "# just execute this code block; nothing needs to be added/modified\n", "from sklearn.dummy import DummyClassifier\n", "from sklearn.metrics import accuracy_score\n", "# choose 'most-frequent class' as the baseline method\n", "baseline_model = DummyClassifier(strategy=\"most_frequent\")\n", "\n", "# fit the baseline model on the training data\n", "baseline_model.fit(X_train, y_train)\n", "\n", "# make predictions on the test data using the created baseline model\n", "baseline_preds = baseline_model.predict(X_test)\n", "\n", "# compute the accuracy of the baseline model\n", "print(accuracy_score(y_test, baseline_preds))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "2f6zub7yz-D7" }, "source": [ "**Report the accuracies for the baseline and NB models here. Type your answer below! Indicate clearly the numbers corresponding to the models.**" ] }, { "cell_type": "markdown", "metadata": { "id": "28HWq6NunZYm" }, "source": [ "\n", "\n", "---\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "4FeOGdxynaQd" }, "source": [ "# **References**\n", "\n", "* D. Greene and P. Cunningham. \"Practical Solutions to the Problem of Diagonal Dominance in Kernel Document Clustering\", Proc. ICML 2006.\n", "* [Datasets (scikit-learn)](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.datasets)\n", "* [All about Naive Bayes (from a scikit-learn perspective)](https://scikit-learn.org/stable/modules/naive_bayes.html)\n", "* [Multinomial Naive Bayes API](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html#sklearn-naive-bayes-multinomialnb)\n", "* [Evaluation metrics](https://scikit-learn.org/stable/modules/classes.html#sklearn-metrics-metrics)\n", "* [Feature extraction module](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.feature_extraction)\n", "* [Transforming prediction targets using LabelEncoder](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html)\n", "\n", "\n" ] } ] }