Do on 4/23/2020 2:20PM 20-25 questions Multiple choice timed Chapter 11 Predictive Machine Learning and Chapter 12 Analytics in Practice

4/8/2020

1

Practical Analytics Chapter 11: Predictive Models for Data Mining

Confusion Matrix Actual Response Yes No

Yes True positives (hit) False negatives (miss)

No False positives (false alarm) True negatives (correct rejection)

Predicted Response

Chapter 11 Learning Objectives

Explain the term “machine learning.”

Discuss various predictive data models.

Identify which models are applicable for which types of predictive scenarios.

1

2

4/8/2020

2

Outline - Predictive Data Models

ESTIMATION CLASSIFICATION

Analytics Techniques

Overview of Analytics Techniques

Exploration and Reporting

• Slicing/dicing

• Multidimensional analysis

• Reporting

Visualization

• Charts

• Dashboards

Knowledge discovery

• Forecasting

• Unsupervised machine learning

• Predictive machine learning

D a

ta S

ta g

in g

P u

b li

sh in

g

3

4

4/8/2020

3

Machine Learning • Predictive data mining involves the

partitioning of datasets with known target variables into three subsets to train, validate, and test a model.

• This process is known as supervision of the model.

• Supervised data models are also known as “machine learning.” This term highlights the capabilities of the model to “learn” and adapt to new data feeds.

• Machine learning is ideal for large, complex problems, and it is at the heart of artificial intelligence (AI).

Predictive models are of two types: ◦ Estimation models - attempt to approximate or otherwise determine

outcomes based on multiple parameters and known relationships expressed as mathematical algorithms or parametric equations; that is, equations that express a set of quantities as functions of independent variables.

◦ Classification models or classifiers - models to classify or categorize data, entities, and events to identify patterns that explain how different variables in a model contribute to an outcome.

Predictive Data Models

5

6

4/8/2020

4

Estimation • Estimation models are used to predict a specific value of a

variable.

• For example, Nina may wish to predict the revenue for electric bikes for next year

• Simple linear regression is a mathematical model that creates an arithmetic equation to explain the relationship be

• The goal of simple linear regression is to fit a straight line through the points on a chart between the dependent and independent variables between independent and dependent variables.

• Once the equation for the straight line is known, then you can estimate the value of the dependent variable for any given value of independent variable (within its interval of validity)

The process of estimating or defining the relationships between and among variables and developing a model of cause and effect.

In other words, it answers the question of which variable(s) affect another variable and in what way.

◦ Acidity and Wine score ◦ Age and Weight, Height and Weight

The target (dependent) variable is numeric. The predictor (independent) variable is usually numeric.

Simple Linear Regression

Multiple Linear Regression

Regression

7

8

4/8/2020

5

Simple Linear Regression Simple linear regression creates an arithmetic equation to explain the relationship between independent and dependent variables.

The independent variable is also called the predictor or explanatory variable, and the dependent variable is also called the target variable.

The goal of simple linear regression is to fit a straight line through the points on a chart between the dependent and independent variables

The chart displays a natural phenomenon of the snowy tree cricket’s chirps.

The number of chirps are a decent measure of temperature within the bounds of applicability.

The scatter plot in the figure was created by counting the number of chirps per 15 seconds and plotting that number against the actual temperature when the chirps occurred.

A regression line shows the relationship between the two.

The line is expressed as an equation 𝑌 = 𝑎 + 𝑏𝑋

𝑇𝑒𝑚𝑝𝑒𝑟𝑎𝑡𝑢𝑟𝑒 = 𝑎 + 𝑏 ∗ 𝑐ℎ𝑖𝑟𝑝𝑠

Where y is the temperature, a is the x axis intercept and b is the slope of the line.

Note that this line should only be extrapolated within a certain range because the cricket stops chirping outside of its temperature of habitation.

Maybe other factors should be taken into consideration such as humidity, time of day, etc.

Simple Linear Regression

9

10

4/8/2020

6

Scatter Chart of Temperature vs Chirps

Simple Linear Regression

The equation for the line is computed by fitting the line the minimizes Root Mean Square Error (RMSE)

𝑆𝑆𝐸 = ෍

𝑖=1

𝑛

(𝑦𝑖 − ො𝑦𝑖) 2

Where SSE is sum of squared errors, n is the number of data points, ො𝑦 is the predicted target value, 𝑦𝑖 is the observed target value.

𝑅𝑀𝑆𝐸 = 𝑆𝑆 Τ𝐸 𝑛

11

12

4/8/2020

7

Errors In the absence of regression, our best prediction would simply be the mean of all observations. We call this value ത𝑦.

ത𝑦 = ෍

𝑖=1

𝑛

𝑦𝑖

The sum of squared errors without regression is called total sum of squares (SST).

𝑆𝑆𝑇 = ෍

𝑖=1

𝑛

(𝑦𝑖 − ത𝑦) 2

Sum of Squares Explained by Regression

sum of squares explained by regression (SSR) : 𝑆𝑆𝑅 = 𝑆𝑆𝑇 − 𝑆𝑆𝐸

𝑆𝑆𝑅 = ෍

𝑖=1

𝑛

(ො𝑦𝑖 − ത𝑦) 2

The formula for total sum of squares (SST) is: 𝑆𝑆𝑇 = 𝑆𝑆𝑅 + 𝑆𝑆𝐸

R-squared (R2), also known as the coefficient of determination, is a statistical measure of goodness-of-fit

𝑅2 = 𝑆𝑆𝑅

𝑆𝑆𝑇 = 𝑆𝑆𝑇 − 𝑆𝑆𝐸

𝑆𝑆𝑇

13

14

4/8/2020

8

In most real-life cases more than one predictor influences the target variable.

To make predictions in the case of multiple independent variables, we would use multiple linear regression.

𝑍 = 𝑎 + 𝑏𝑋 + 𝑐𝑌 + ⋯

Z is the value we want to predict, X and Y are independent variables, a is the Z intercept, and b and c are the slopes of the line with respect to X, Y.

Examples: ◦ Acidity and Alcohol vs wine score

◦ Age and Height vs weight

𝑤𝑒𝑖𝑔ℎ𝑡 = 𝑎 + 𝑏𝑥 𝑎𝑔𝑒 + 𝑐 𝑥 ℎ𝑒𝑖𝑔ℎ𝑡

Multiple Linear Regression

Classification Store

Number

Annual

Revenue City State

Number of

Brands Carried

Years in

Existence

Number of

Clear

Weather

Days

Number of

Sunny Hours

Per Capita

Income in

City

Response

1 $ 2,000,000.00 New York City NY 3 4 107 2535 $32,000.00 Yes

2 $ 1,500,000.00 Los Angeles CA 10 15 147 3100 $27,000.00 No

3 $ 550,000.00 Chicago IL 7 12 84 2508 $28,000.00 No

4 $ 1,200,000.00 Dallas TX 8 1 135 2850 $27,000.00 Yes

5 $ 505,000.00 Miami FL 9 2 74 3154 $21,000.00 No

6 $ 376,000.00 Cincinnati OH 15 3 81 2335 $25,000.00 No

7 $ 670,000.00 San Diego CA 2 8 146 3055 $33,000.00 Yes

8 $ 1,110,000.00 Austin TX 4 8 115 2644 $32,000.00 Yes

9 $ 454,000.00 Seattle WA 4 4 58 2170 $43,000.00 Yes

10 $ 500,000.00 Boston MA 5 1 98 2634 $34,000.00 No

Use existing data to train the model to predict a categorical variable such as Response to marketing attempts. The table shows data about bicycle stores. Several independent variables are used to predict the Response of a customer. Yes/No

We want to be able to predict the Response of new/other customers using classification

15

16

4/8/2020

9

Performance of Classification on Validation Data After the classifier has been trained, it is tested on holdout data. The result is presented as a confusion matrix

The objective of the classification algorithm is to maximize the true positives and true negatives and to minimize the false negatives and false positives.

Confusion Matrix Actual Response Yes No

Yes 32 8

No 3 7

Predicted Response

Confusion Matrix Actual Response Yes No

Yes True positives (hit) False negatives (miss)

No False positives (false alarm) True negatives (correct rejection)

Predicted Response

Classification Models

Several models exist for classify the target variable. Here are a few of them:

Naïve Bayes

K-nearest neighbors (KNN)

Logistics regression

Decision trees

Neural networks

Genetic algorithms

Support vector machines (SVM)

17

18

4/8/2020

10

Naïve Bayes Classifier

The Naïve Bayes model assumes that the impact of the value of one independent attribute is independent of the value of other independent attribute

For instance, the city is independent of the revenue for a particular customer.

Because of this assumption, the Naïve Bayes theorem can be used to estimate the probability of every combination of independent attributes.

After the probabilities are estimated, any new case can be predicted for its outcome.

Model is based on a simple observation that a case is most likely to be similar to its nearest neighbors.

For example, the price of a house is most likely to be similar to the price of other homes in its immediate vicinity.

The number of nearest neighbors we use to predict a case is called K.

One or more independent variables are used as predictors. The algorithm has to scale each factor to the same scale.

◦ Example: income is 0-100000 but age is 0-100, both need to be scaled to 0-1

After we have scaled all of the factors, we measure the distance of the new customer from all other customers.

We then note the K nearest neighbors’ classification.

If the majority of the neighbors are one type of customer, then we classify the new customer as such.

K-nearest Neighbors (KNN).

19

20

4/8/2020

11

Decision Trees – Classification Trees Classification trees are used to classify new cases into categories. Therefore the target variable is categorical

The simplest of them being a binary category such as true/false, Yes/No.

Let’s build a simple decision tree used to determine “survivability” of passengers on the Titanic.

The classification is either survived or died.

Variable

Name Definition Key

Target Variable

Survival Survived 0 = No, 1 = Yes

Predictor Variables

Pclass Ticket class 1 = 1st, 2 = 2nd, 3 = 3rd

sex Gender M = male, F = female

Age Age in years

sibsp # of siblings / spouses aboard the

Titanic

parch # of parents / children aboard the

Titanic

fare Passenger fare

embarked Port of Embarkation C = Cherbourg, Q = Queenstown, S =

Southampton

Titanic Decision Tree A decision tree to classify the Titanic data has been trained using the standard training and validation data partitions.

The training dataset has 891 passengers; the validation dataset has 418 passengers.

Here’s how you read a tree: ◦ Start at the top, where the value of a predictor variable leads you to the next

variable by following one branch of the tree. Then, go to the next branch, and so on.

◦ The tree tips are classification points where the case—in our example, individual passenger—is classified into Survived or Not Survived. 1 represents survived, and 0 represents Not Survived.

21

22

4/8/2020

12

Logistic Regression

Logistic regression is a classification model in

which the dependent, or predicted variable is

categorical; that is, there are groupings into which a

case is classified.

This method is used to classify a case into one of two categories based on a

number of predictor variables or attributes.

It differs from simple linear regression, where the

dependent (or predicted) variable is numeric and

continuous.

Neural Networks

Artificial neural networks (ANN), are a type of machine learning that is based on biological neural networks such as a human (or animal) brain.

The brain is made up of a vast network of interconnected neurons that transmit messages to one another. This interconnectedness makes the brain capable of learning and adapting over time.

ANN try to mimic this capability using computer programs. They are applied primarily in tasks such as image recognition that are easy for humans to perform but very difficult for computers.

23

24

4/8/2020

13

Genetic Algorithms

Computer scientists who work in artificial intelligence or in

machine learning have devised methods to mimic

evolution in their programs.

These methods are called genetic algorithms (GA). GA

use the same process of selection, recombination, and mutation to find solutions to

complex problems.

Support Vector Machines (SVM) Can be used for both estimation and classification.

When used in classification, SVM groups new cases into one of two classes.

For example, GB could categorize customers into premier and standard customers based on the number of years they have been with GB, the amount of revenue they bring, and other factors.

SVM then classifies a new GB customer as either a premier or standard customer.

SVM FOR SPAM DETECTION

25

26

4/8/2020

14

Summary

We examined the various data models that allow us to make forecasts and predictions.

For forecasting, we discussed the use of time series analysis to identify patterns, trends, and seasonality as well as the modeling techniques that enable us to separate the random values in a time series from those we can explain.

For predictive modeling, we considered the two basic types of predictive models, estimations and classifications.

It is important to note that predictive data models are supervised; that is, they need to be trained, validated, tested, and run in real-world scenarios.

Models are evaluated and retrained from time to time.

27