Computer science project for besiness questions

PREDICTIVE MODELING PROJECT TEMPLATE & HELLO WORLD Section 1: Predictive Modeling Project Template Section 2: Hello World of Machine Learning

Predictive Modeling Project Template

One of the best ways to practice predictive modeling machine learning projects is to use standardized datasets from the UCI Machine Learning Repository.

Once you have a practice dataset and a bunch of Python recipes, how do you put it all together and work through the problem end-to-end?

Any predictive modeling machine learning project can be broken down into six common tasks:

1. Define Problem.

2. Summarize Data.

3. Prepare Data.

4. Evaluate Algorithms.

5. Improve Results

6. Present Results.

Tasks can be combined or broken down further, but this is the general structure.

To work through predictive modeling machine learning problems in Python, you need to map Python onto this process.

The tasks may need to be adapted or renamed slightly to suit the Python way of doing things (e.g. Pandas for data loading and scikit-learn for modeling).

The next section provides exactly this mapping and elaborates each task and the types of sub- tasks and libraries that you can use

Section 1: Machine Learning Template How to use the project template:

1. Create a new file for your project (e.g. project name.py). 2. Copy the project template. 3. Paste it into your empty project file 4. Start to fill it in, using recipes from this guide.

1) PREPARE THE PROBLEM

Depending on the nature of what’s being explored, you may need to use the techniques discussed in the first two classes. Creating a problem statement, iterating over a problem and determining various paths of execution. Remember that outlining the problem statement for what you’re trying to achieve AND defining the criteria for success is critical.

During this phase you are also loading everything that you need to start working on your problem. This includes:

§ Python modules, classes and functions that you intend to use § Loading your dataset (in this case) from CSV

This is also the home of any global configuration that you might need to do. It is also the place where you might need to make a reduced sample of your dataset if it is too large to work with. Ideally, your dataset should be small enough to build a model or create a visualization within a minute, ideally 30 seconds. You can always scale up well performing models later.

2) SUMMARIZE THE DATA

This step is about better understanding the data that you have available. This includes understanding your data using:

§ Descriptive statistics such as summaries § Data visualizations such as plots with Matplotlib, ideally using convenience

functions from Pandas

Take your time and use the results to prompt questions, assumptions and hypotheses that you can investigate later with specialized models.

3) PREPARE THE DATA

This step is about preparing the data in such a way that it best exposes the structure of the problem and the relationships between your input attributes with the output variable. This includes task such as:

§ Cleaning data by removing duplicates, marking missing values and even imputing missing values

§ Feature selection where redundant features may be removed and new features developed

§ Data transforms where attributes are scaled or redistributed in order to best expose the structure of the problem later to learning algorithms

Start simple. Revisit this step often and cycle with the next step until you converge on a subset of algorithms and a presentation of the data that results in accurate or accurate-enough models to proceed.

4) EVALUATE ALGORITHMS

This step is about finding a subset of machine learning algorithms that are good at exploiting the structure of your data (e.g. have better than average skill) .. this involves steps such as :

§ Separating out a validation dataset to use for later confirmation of the skill of your developed model.

§ Defining test options using scikit-learn such as cross-validation and the evaluation metric to use.

§ Spot-checking a suite of linear and nonlinear machine learning algorithms. § Comparing the estimated accuracy of algorithms.

On a given problem you will likely spend most of your time on this and the previous step until you converge on a set of 3-to-5 well performing machine learning algorithms.

5) IMPROVE ACCURACY

Once you have a short list of machine learning algorithms, you need to get the most out of them. There are two different ways to improve the accuracy of your models:

§ Search for a combination of parameters for each algorithm using scikit-learn that yields the best results.

§ Combine the prediction of multiple models into an ensemble prediction using ensemble techniques.

The line between this and the previous step can blur when a project becomes concrete. There may be a little algorithm tuning in the previous step. And in the case of ensembles, you may bring more than a shortlist of algorithms forward to combine their predictions.

6) FINALIZE THE MODEL

Once you have found a model that you believe can make accurate predictions on unseen data, you are ready to finalize it. Finalizing a model may involve sub-tasks such as:

§ Using an optimal model tuned by scikit-learn to make predictions on unseen data. § Creating a standalone model using the parameters tuned by scikit-learn. § Saving an optimal model to file for later use.

Once you make it this far you are ready to present results to stakeholders and/or deploy your model to start making predictions on unseen data.

Remember that this entire process is iterative for each initiative / attempt at solving the problem statement. Within each initiative is additional iterations in order to determine the best course and results.

Tips for using the template

§ Fast First Pass. Make a first-pass through the project steps as fast as possible. This will give you confidence that you have all the parts that you need and a baseline from which to improve.

§ Cycles. The process in not linear but cyclic. You will loop between steps, and probably spend most of your time in tight loops between steps 3-4 or 3-4-5 until you achieve a level of accuracy that is sufficient or you run out of time.

§ Attempt Every Step. It is easy to skip steps, especially if you are not confident or familiar with the tasks of that step. Try and do something at each step in the process, even if it does not improve accuracy. You can always build upon it later. Don’t skip steps, just reduce their contribution.

§ Ratchet Accuracy. The goal of the project is model accuracy. Every step contributes towards this goal. Treat changes that you make as experiments that increase accuracy as the golden path in the process and reorganize other steps around them. Accuracy is a ratchet that can only move in one direction (better, not worse).

§ Adapt As Needed. Modify the steps as you need on a project, especially as you become more experienced with the template. Blur the edges of tasks, such as steps 4-5 to best serve model accuracy.

Hello World of Machine Learning

The best small project to start with on a new tool is the classification of iris flowers.

This is a good dataset for your first project because it is so well understood. Available on blackboard.

§ Attributes are numeric so you do not have to figure out how to load and handle data. § It is a classification problem, allowing you to practice with an easier type of supervised

learning algorithm. § It is a multiclass classification problem (multi-nominal) that may require some

specialized handling. § It only has 4 attributes and 150 rows, meaning it is small and easily fits into memory

(and a screen or single sheet of paper). § All of the numeric attributes are in the same units and the same scale not requiring any

special scaling or transforms to get started.

TUTORIAL

In this tutorial we are going to work through a small machine learning project end-to-end.

Here is an overview of what we are going to cover:

1) Loading the dataset. 2) Summarizing the dataset. 3) Visualizing the dataset. 4) Evaluating some algorithms. 5) Making some predictions.

Take your time and work through each step. Try to type in the commands yourself or copy-and- paste the commands to speed things up. Start your Python interactive environment and let’s get started with your hello world machine learning project in Python.

LOAD THE DATA

First, let’s import all of the modules, functions and objects we are going to use in this tutorial.

Import Libraries

# Load libraries

from pandas import read_csv from pandas.plotting import scatter_matrix from matplotlib import pyplot from sklearn.model_selection import train_test_split from sklearn.model_selection import KFold from sklearn.model_selection import cross_val_score from sklearn.metrics import classification_report from sklearn.metrics import confusion_matrix from sklearn.metrics import accuracy_score from sklearn.linear_model import LogisticRegression from sklearn.tree import DecisionTreeClassifier from sklearn.neighbors import KNeighborsClassifier from sklearn.discriminant_analysis import LinearDiscriminantAnalysis from sklearn.naive_bayes import GaussianNB from sklearn.svm import SVC

Everything should load without error. If you have an error, stop. You need a working SciPy

environment before continuing.

Load the dataset

The iris dataset can be downloaded from blackboard . We are using Pandas to load the data. We will also use Pandas next to explore the data both with descriptive tatistics and data visualization. Note that we are specifying the names of each column when loading the data. This will help later when we explore the data.

# Load dataset

filename = 'iris.data.csv' names = ['sepal-length', 'sepal-width', 'petal-length', 'petal-width', 'class'] dataset = read_csv(filename, names=names)

Summarize the Dataset

Nowitistimetotakealookatthedata. Inthisstepwearegoingtotakealookatthedataa few different ways:

§ Dimensions of the dataset.

§ Peek at the data itself.

§ Statistical summary of all attributes.

§ Breakdown of the data by the class variable.

Don’t worry, each look at the data is one command. These are useful commands that you can use again and again on future projects.

We can get a quick idea of how many instances (rows) and how many attributes (columns) the data contains with the shape property.

# shape

print(dataset.shape)

you should see 150 instances and 5 attributes

Peek at the Data

It is also always a good idea to actually eyeball your data

# head

print(dataset.head(20))

you should see the first 20 rows of data

Statistical Summary

Now we can take a look at a summary of each attribute. This includes the count, mean, the min and max values as well as some percentiles.

# descriptions

print(dataset.describe())

We can see that all of the numerical values have the same scale (centimeters) and similar ranges between 0 and 8 centimeters

Class Distribution

Let’s take a look at the number of instances ( rows) that belong to each class. We can view this as an absolute count.

# class distribution

print(dataset.groupby('class').size())

We can see that each class has the same number of instances ( 50 or 33% of the dataset)

Data Visualization

We now have a basic idea about the data. We need to extend this with some visualizations. We are going to look at two types of plots:

§ Univariate plots to better understand each attribute. § Multivariate plots to better understand the relationships between attributes.

Univariate Plots

We will start with some univariate plots, that is, plots of each individual variable. Given that the input variables are numeric, we can create box and whisker plots of each.

box and whisker plots

dataset.plot(kind='box', subplots=True, layout=(2,2), sharex=False, sharey=False)

pyplot.show()

We can also create a histogram of each input variable to get an idea of the distribution

# histograms

dataset.hist()

pyplot.show()

It looks like perhaps two of the input variables have a Gaussian distribution. This is useful to note as we can use algorithms that can exploit this assumption.

Multivariate Plots

Now we can look at the interactions between the variables. Let’s look at scatter plots of all pairs of attributes. This can be helpful to spot structured relationships between input variables.

# scatter plot matrix

scatter_matrix(dataset)

pyplot.show()

Note the diagonal grouping of some pairs of attributes. This suggests a high correlation and a predictable relationship.

EVALUATE SOME ALGORITHMS

Now it is time to create some models of the data and estimate their accuracy on unseen data. Here is what we are going to cover in this step:

§ Separate out a validation dataset § Setup the test harness to use 10-fold cross-validation. § Build 5 different models to predict species from flower measurements § Select the best model.

CREATE A VALIDATION DATASET

We need to know whether or not the model that we created is any good. Later, we will use statistical methods to estimate the accuracy of the models that we create on unseen data. We also want a more concrete estimate of the accuracy of the best model on unseen data by evaluating it on actual unseen data. That is, we are going to hold back some data that the algorithms will not get to see and we will use this data to get a second and independent idea of

how accurate the best model might actually be. We will split the loaded dataset into two, 80%

of which we will use to train our models and 20% that we will hold back as a validation dataset.

# Split-out validation dataset

array = dataset.values X = array[:,0:4] Y = array[:,4] validation_size = 0.20 seed = 7

X_train, X_validation, Y_train, Y_validation = train_test_split(X, Y, test_size=validation_size, random_state=seed)

You now have training data in the X train and Y train for preparing models and a

X validation and Y validation sets that we can use later.

We will use 10-fold cross-validation to estimate accuracy on unseen data. This will split our dataset into 10 parts, e.g. the model will train on 9 and test on 1 and repeat for all combinations of train-test splits. We are using the metric of accuracy to evaluate models. This is a proportion of the number of correctly predicted instances divided by the total number of

instances in the dataset multiplied by 100 to give a percentage (e.g. 95% accurate). We will be using the scoring variable when we run build and evaluate each model next.

Build Models

We don’t know which algorithms would be good on this problem or what configurations to use. We got an idea from the plots that some of the classes are partially linearly separable in some dimensions, so we are expecting generally good results. Let’s evaluate six different algorithms:

§ Logistic Regression (LR). § Linear Discriminant Analysis (LDA). § k-Nearest Neighbors (KNN). § Classification and Regression Trees (CART). § Gaussian Naive Bayes (NB). § Support Vector Machines (SVM).

This list is a good mixture of simple linear (LR and LDA), and nonlinear (KNN, CART, NB and SVM) algorithms. We reset the random number seed before each run to ensure that the evaluation of each algorithm is performed using exactly the same data splits. It ensures the results are directly comparable. Let’s build and evaluate our five models:

# Spot-Check Algorithms

models = [] models.append(('LR', LogisticRegression(solver='liblinear', multi_class='ovr'))) models.append(('LDA', LinearDiscriminantAnalysis())) models.append(('KNN', KNeighborsClassifier())) models.append(('CART', DecisionTreeClassifier())) models.append(('NB', GaussianNB())) models.append(('SVM', SVC(gamma='auto'))) # evaluate each model in turn results = [] names = [] for name, model in models:

kfold = KFold(n_splits=10, random_state=seed) cv_results = cross_val_score(model, X_train, Y_train, cv=kfold, scoring='accuracy') results.append(cv_results) names.append(name) msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std()) print(msg)

Select The Best Model

We now have 6 models and accuracy estimations for each. We need to compare the models to each other and select the most accurate. Running the example above, we get the following raw results:

§ LR: 0.966667 (0.040825) § LDA: 0.975000 (0.038188) § KNN: 0.983333 (0.033333) § CART: 0.975000 (0.038188) § NB: 0.975000 (0.053359) § SVM: 0.981667 (0.025000)

We can see that it looks like KNN has the largest estimated accuracy score. We can also create a plot of the model evaluation results and compare the spread and the mean accuracy of each model. There is a population of accuracy measures for each algorithm because each algorithm was evaluated 10 times (10 fold cross-validation).

# Compare Algorithms

fig = pyplot.figure()

fig.suptitle('Algorithm Comparison')

ax = fig.add_subplot(111)

pyplot.boxplot(results)

ax.set_xticklabels(names)

pyplot.show()

You can see that the box and whisker plots are squashed at the top of the range, with many samples achieving 100% accuracy.

Make Predicitons

The KNN algorithm was the most accurate model that we tested. Now we want to get an idea of the accuracy of the model on our validation dataset. This will give us an independent final check on the accuracy of the best model. It is important to keep a validation set just in case you made a slip during training, such as overfitting to the training set or a data leak. Both will result in an overly optimistic result. We can run the KNN model directly on the validation set and summarize the results as a final accuracy score, a confusion matrix and a classification report.

# Make predictions on validation dataset

knn = KNeighborsClassifier() knn.fit(X_train, Y_train) predictions = knn.predict(X_validation) print(accuracy_score(Y_validation, predictions)) print(confusion_matrix(Y_validation, predictions)) print(classification_report(Y_validation, predictions))

We can see that the accuracy is 0.9 or 90%. The confusion matrix provides an indication of the three errors made. Finally the classification report provides a breakdown of each class by precision, recall, f1-score and support showing excellent results (granted the validation dataset was small).