Research Paper

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 153

theoretical regression line). Even though there are several methods/algorithms proposed to identify the regression line, the one that is most commonly used is called the ordinary least squares (OLS) method. The OLS method aims to minimize the sum of squared residuals (squared vertical distances between the observation and the regression point) and leads to a mathematical expression for the estimated value of the regression line (which are known as b parameters). For simple linear regression, the aforementioned relationship between the response variable 1y2 and the explanatory variable(s) 1x2 can be shown as a simple equation as follows:

y = b0 + b1x

In this equation, b0 is called the intercept, and b1 is called the slope. Once OLS deter- mines the values of these two coefficients, the simple equation can be used to forecast the values of y for given values of x. The sign and the value of b1 also reveal the direc- tion and the strengths of relationship between the two variables.

If the model is of a multiple linear regression type, then there would be more coef- ficients to be determined, one for each additional explanatory variable. As the following formula shows, the additional explanatory variable would be multiplied with the new bi coefficients and summed together to establish a linear additive representation of the response variable.

y = b0 + b1x1 + b2x2 + b3x3 + # + bnxn

How Do We Know If the Model Is Good Enough?

Because of a variety of reasons, sometimes models as representations of the reality do not prove to be good. Regardless of the number of explanatory variables included, there is always a possibility of not having a good model, and therefore the linear regression model needs to be assessed for its fit (the degree to which it represents the response variable). In the simplest sense, a well-fitting regression model results in predicted values close to the observed data values. For the numerical assessment, three statistical measures are often used in evaluating the fit of a regression model: R 2(R - squared), the overall F-test, and the root mean square error (RMSE). All three of these measures are based on the sums of the square errors (how far the data are from the mean and how far the data are from the model’s predicted values). Different combinations of these two values pro- vide different information about how the regression model compares to the mean model.

Of the three, R 2 has the most useful and understandable meaning because of its intuitive scale. The value of R 2 ranges from 0 to 1 (corresponding to the amount of vari- ability explained in percentage) with 0 indicating that the relationship and the prediction power of the proposed model is not good, and 1 indicating that the proposed model is a perfect fit that produces exact predictions (which is almost never the case). The good R 2 values would usually come close to one, and the closeness is a matter of the phe- nomenon being modeled—whereas an R 2 value of 0.3 for a linear regression model in social sciences can be considered good enough, an R 2 value of 0.7 in engineering might be considered as not a good enough fit. The improvement in the regression model can be achieved by adding more explanatory variables or using different data transforma- tion techniques, which would result in comparative increases in an R 2 value. Figure 3.14 shows the process flow of developing regression models. As can be seen in the process flow, the model development task is followed by the model assessment task in which not only is the fit of the model assessed, but because of restrictive assumptions with which the linear models have to comply, the validity of the model also needs to be put under the microscope.

154 Part I • Introduction to Analytics and AI

What Are the Most Important Assumptions in Linear Regression?

Even though they are still the choice of many for data analyses (both for explanatory and for predictive modeling purposes), linear regression models suffer from several highly restrictive assumptions. The validity of the linear model built depends on its ability to comply with these assumptions. Here are the most commonly pronounced assumptions:

1. Linearity. This assumption states that the relationship between the response variable and the explanatory variables is linear. That is, the expected value of the response variable is a straight-line function of each explanatory variable while holding all other explanatory variables fixed. Also, the slope of the line does not depend on the values of the other variables. It also implies that the effects of dif- ferent explanatory variables on the expected value of the response variable are additive in nature.

2. Independence (of errors). This assumption states that the errors of the response variable are uncorrelated with each other. This independence of the errors is weaker

Tabulated Data

Data Assessment

Scatter plot

Correlations

Model Fitting

Transform data

Estimate parameters

Model Assessment

Test assumptions

Assess model fit

Deployment

One-time use

Recurrent use

FIGURE 3.14 A Process Flow for Developing Regression Models.

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 155

than actual statistical independence, which is a stronger condition and is often not needed for linear regression analysis.

3. Normality (of errors). This assumption states that the errors of the response vari- able are normally distributed. That is, they are supposed to be totally random and should not represent any nonrandom patterns.

4. Constant variance (of errors). This assumption, also called homoscedasticity, states that the response variables have the same variance in their error regardless of the values of the explanatory variables. In practice, this assumption is invalid if the response variable varies over a wide enough range/scale.

5. Multicollinearity. This assumption states that the explanatory variables are not correlated (i.e., do not replicate the same but provide a different perspective of the information needed for the model). Multicollinearity can be triggered by having two or more perfectly correlated explanatory variables presented to the model (e.g., if the same explanatory variable is mistakenly included in the model twice, one with a slight transformation of the same variable). A correlation-based data assessment usually catches this error.

There are statistical techniques developed to identify the violation of these assump- tions and techniques to mitigate them. The most important part for a modeler is to be aware of their existence and to put in place the means to assess the models to make sure that they are compliant with the assumptions they are built on.

Logistic Regression

Logistic regression is a very popular, statistically sound, probability-based classifica- tion algorithm that employs supervised learning. It was developed in the 1940s as a complement to linear regression and linear discriminant analysis methods. It has been used extensively in numerous disciplines, including the medical and social sciences fields. Logistic regression is similar to linear regression in that it also aims to regress to a mathematical function that explains the relationship between the response vari- able and the explanatory variables using a sample of past observations (training data). Logistic regression differs from linear regression with one major point: its output (re- sponse variable) is a class as opposed to a numerical variable. That is, whereas linear regression is used to estimate a continuous numerical variable, logistic regression is used to classify a categorical variable. Even though the original form of logistic regres- sion was developed for a binary output variable (e.g., 1/0, yes/no, pass/fail, accept/ reject), the present-day modified version is capable of predicting multiclass output variables (i.e., multinomial logistic regression). If there is only one predictor variable and one predicted variable, the method is called simple logistic regression (similar to calling linear regression models with only one independent variable simple linear regression).

In predictive analytics, logistic regression models are used to develop probabilis- tic models between one or more explanatory/predictor variables (which can be a mix of both continuous and categorical in nature) and a class/response variable (which can be binomial/binary or multinomial/multiclass). Unlike ordinary linear regression, logis- tic regression is used for predicting categorical (often binary) outcomes of the response variable—treating the response variable as the outcome of a Bernoulli trial. Therefore, logistic regression takes the natural logarithm of the odds of the response variable to create a continuous criterion as a transformed version of the response variable. Thus, the logit transformation is referred to as the link function in logistic regression—even though the response variable in logistic regression is categorical or binomial, the logit is the con- tinuous criterion on which linear regression is conducted. Figure 3.15 shows a logistic

156 Part I • Introduction to Analytics and AI

regression function where the odds are represented in the x-axis (a linear function of the independent variables), whereas the probabilistic outcome is shown in the y-axis (i.e., response variable values change between 0 and 1).

The logistic function, f1y2 in Figure 3.15 is the core of logistic regression, which can take values only between 0 and 1. The following equation is a simple mathematical representation of this function:

f1y2 = 1 1 + e-1b0 +b1x 2

The logistic regression coefficients (the bs) are usually estimated using the maximum likelihood estimation method. Unlike linear regression with normally distributed residu- als, it is not possible to find a closed-form expression for the coefficient values that maxi- mizes the likelihood function, so an iterative process must be used instead. This process begins with a tentative starting solution, then revises the parameters slightly to see if the solution can be improved, and repeats this iterative revision until no improvement can be achieved or is very minimal, at which point the process is said to have completed/ converged.

Sports analytics—use of data and statistical/analytics techniques to better manage sports teams/organizations—has been gaining tremendous popularity. Use of data-driven analytics techniques has become mainstream for not only professional teams but also col- lege and amateur sports. Application Case 3.4 is an example of how existing and readily available public data sources can be used to predict college football bowl game outcomes using both classification and regression-type prediction models.

Time-Series Forecasting

Sometimes the variable that we are interested in (i.e., the response variable) might not have distinctly identifiable explanatory variables, or there might be too many of them in a highly complex relationship. In such cases, if the data are available in a desired format, a prediction model, the so-called time series, can be developed. A time series is a sequence of data points of the variable of interest, measured and represented at successive points in time spaced at uniform time intervals. Examples of time series include monthly rain volumes in a geographic area, the daily closing value of the stock market indexes, and

f (y) 1

26 24 22 0 2 4 6

b0 1 b1x

0.5

FIGURE 3.15 The Logistic Function.

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 157

Predicting the outcome of a college football game (or any sports game, for that matter) is an interesting and challenging problem. Therefore, challenge-seeking researchers from both academics and industry have spent a great deal of effort on forecasting the out- come of sporting events. Large amounts of historic data exist in different media outlets (often publicly available) regarding the structure and outcomes of sporting events in the form of a variety of numeri- cally or symbolically represented factors that are assumed to contribute to those outcomes.

The end-of-season bowl games are very impor- tant to colleges in terms of both finance (bring- ing in millions of dollars of additional revenue) and reputation—for recruiting quality students and highly regarded high school athletes for their athletic pro- grams (Freeman & Brewer, 2016). Teams that are selected to compete in a given bowl game split a purse, the size of which depends on the specific bowl (some bowls are more prestigious and have higher payouts for the two teams), and therefore securing an invitation to a bowl game is the main goal of any division I-A college football program. The decision makers of the bowl games are given the authority to select and invite bowl-eligible (a team that has six

wins against its Division I-A opponents in that season) successful teams (as per the ratings and rankings) that will play in an exciting and competitive game, attract fans of both schools, and keep the remaining fans tuned in via a variety of media outlets for advertising.

In a recent data mining study, Delen et al. (2012) used eight years of bowl game data along with three popular data mining techniques (decision trees, neural networks, and support vector machines) to predict both the classification-type outcome of a game (win versus loss) and the regression-type out- come (projected point difference between the scores of the two opponents). What follows is a shorthand description of their study.

The Methodology

In this research, Delen and his colleagues followed a popular data mining methodology, CRISP-DM (Cross-Industry Standard Process for Data Mining), which is a six-step process. This popular meth- odology, which is covered in detail in Chapter 4, provided them with a systematic and structured way to conduct the underlying data mining study and hence improved the likelihood of obtaining accurate

Application Case 3.4 Predicting NCAA Bowl Game Outcomes

(Continued )

158 Part I • Introduction to Analytics and AI

and reliable results. To objectively assess the pre- diction power of the different model types, they used a cross-validation methodology k-fold cross- validation. Details on k-fold cross-validation can be found in Chapter 4. Figure 3.16 graphically illustrates the methodology employed by the researchers.

Data Acquisition and Data Preprocessing

The sample data for this study are collected from a variety of sports databases available on the Web,

including jhowel.net, ESPN.com, Covers.com, ncaa.org, and rauzulusstreet.com. The data set included 244 bowl games representing a com- plete set of eight seasons of college football bowl games played between 2002 and 2009. Delen et al. also included an out-of-sample data set (2010– 2011 bowl games) for additional validation pur- poses. Exercising one of the popular data mining rules of thumb, they included as much relevant information in the model as possible. Therefore, after an in-depth variable identification and

Classification & Regression Trees

Neural Networks

X1

X2

Support Vector Machines

M ax

im um

-m ar

gin h yp

er pla

ne

M argin

Data Collection, Organization, Cleaning, and Transformation

Raw Data Sources

Built Classification

Models

Test Model

Tabulate the Results

Built Regression

Models

Transform and Tabulate Results

Compare the Prediction Results

Test Model

Classification Modeling

Regression Modeling

DBs

Output: Binary (win/loss) Output: Integer (point difference)

Win Loss

Win

Loss

...

......

...

10 %

10 %

10 %

10 % 10 %

10 %

10 %

10 %

10 % 10 % 10 %

10 %

10 %

10 % 10 %

10 %

10 %

10 %

10 % 10 %

FIGURE 3.16 The Graphical Illustration of the Methodology Employed in the Study.

Application Case 3.4 (Continued)

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 159

collection process, they ended up with a data set that included 36 variables, of which the first 6 were the identifying variables (i.e., name and the year of the bowl game, home and away team names, and their athletic conferences—see variables 1–6 in

Table 3.5), followed by 28 input variables (which included variables delineating a team’s seasonal sta- tistics on offense and defense, game outcomes, team composition characteristics, athletic conference char- acteristics, and how they fared against the odds—see

TABLE 3.5 Description of Variables Used in the Study

No Cat Variable Name Description

1 ID1 YEAR Year of the bowl game

2 ID BOWLGAME Name of the bowl game

3 ID HOMETEAM Home team (as listed by the bowl organizers)

4 ID AWAYTEAM Away team (as listed by the bowl organizers)

5 ID HOMECONFERENCE Conference of the home team

6 ID AWAYCONFERENCE Conference of the away team

7 I12 DEFPTPGM Defensive points per game

8 I1 DEFRYDPGM Defensive rush yards per game

9 I1 DEFYDPGM Defensive yards per game

10 I1 PPG Average number of points a given team scored per game

11 I1 PYDPGM Average total pass yards per game

12 I1 RYDPGM Team’s average total rush yards per game

13 I1 YRDPGM Average total offensive yards per game

14 I2 HMWIN% Home winning percentage

15 I2 LAST7 How many games the team won out of their last 7 games

16 I2 MARGOVIC Average margin of victory

17 I2 NCTW Nonconference team winning percentage

18 I2 PREVAPP Did the team appear in a bowl game previous year

19 I2 RDWIN% Road winning percentage

20 I2 SEASTW Winning percentage for the year

21 I2 TOP25 Winning percentage against AP top 25 teams for the year

22 I3 TSOS Strength of schedule for the year

23 I3 FR% Percentage of games played by freshmen class players for the year

24 I3 SO% Percentage of games played by sophomore class players for the year

25 I3 JR% Percentage of games played by junior class players for the year

26 I3 SR% Percentage of games played by senior class players for the year

27 I4 SEASOvUn% Percentage of times a team went over the O/U3 in the current season

28 I4 ATSCOV% Against the spread cover percentage of the team in previous bowl games

(Continued )

160 Part I • Introduction to Analytics and AI

TABLE 3.5 (Continued)

No Cat Variable Name Description

29 I4 UNDER% Percentage of times a team went under in previous bowl games

30 I4 OVER% Percentage of times a team went over in previous bowl games

31 I4 SEASATS% Percentage of covering against the spread for the current season

32 I5 CONCH Did the team win their respective conference championship game

33 I5 CONFSOS Conference strength of schedule

34 I5 CONFWIN% Conference winning percentage

35 O1 ScoreDiff4 Score difference (HomeTeamScore – AwayTeamScore)

36 O2 WinLoss4 Whether the home team wins or loses the game

1ID: Identifier variables; O1: output variable for regression models; O2: output variable for classification models. 2Offense/defense; I2: game outcome; I3: team configuration; I4: against the odds; I5: conference stats. 3Over/Under—Whether or not a team will go over or under the expected score difference. 4Output variables—ScoreDiff for regression models and WinLoss for binary classification models.

variables 7–34 in Table 3.5), and finally the last two were the output variables (i.e., ScoreDiff—the score difference between the home team and the away team represented with an integer number—and WinLoss—whether the home team won or lost the bowl game represented with a nominal label).

In the formulation of the data set, each row (a.k.a. tuple, case, sample, example, etc.) represented a bowl game, and each column stood for a variable (i.e., identifier/input or output type). To represent the game-related comparative characteristics of the two opponent teams in the input variables, Delen et al. calculated and used the differences between the measures of the home and away teams. All these variable values are calculated from the home team’s perspective. For instance, the variable PPG (average number of points a team scored per game) repre- sents the difference between the home team’s PPG and away team’s PPG. The output variables repre- sent whether the home team wins or loses the bowl game. That is, if the ScoreDiff variable takes a posi- tive integer number, then the home team is expected to win the game by that margin; otherwise (if the ScoreDiff variable takes a negative integer number), the home team is expected to lose the game by that margin. In the case of WinLoss, the value of the out- put variable is a binary label, “Win” or “Loss,” indi- cating the outcome of the game for the home team.

The Results and Evaluation

In this study, three popular prediction techniques are used to build models (and to compare them to each other): artificial neural networks, decision trees, and support vector machines. These prediction techniques are selected based on their capability of modeling both classification and regression-type prediction problems and their popularity in recently published data mining literature. More details about these popular data min- ing methods can be found in Chapter 4.

To compare predictive accuracy of all models to one another, the researchers used a stratified k-fold cross-validation methodology. In a stratified version of k-fold cross-validation, the folds are created in a way that they contain approximately the same proportion of predictor labels (i.e., classes) as the original data set. In this study, the value of k is set to 10 (i.e., the com- plete set of 244 samples are split into 10 subsets, each having about 25 samples), which is a common prac- tice in predictive data mining applications. A graphical depiction of the 10-fold cross-validations was shown earlier in this chapter. To compare the prediction mod- els that were developed using the aforementioned three data mining techniques, the researchers chose to use three common performance criteria: accuracy, sen- sitivity, and specificity. The simple formulas for these metrics were also explained earlier in this chapter.

Application Case 3.4 (Continued)

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 161

The prediction results of the three model- ing techniques are presented in Tables 3.6 and 3.7. Table 3.6 presents the 10-fold cross-validation results of the classification methodology in which the three data mining techniques are formulated to have a binary-nominal output variable (i.e., WinLoss). Table 3.7 presents the 10-fold cross- validation results of the regression-based classifica- tion methodology in which the three data mining techniques are formulated to have a numerical out- put variable (i.e., ScoreDiff). In the regression-based classification prediction, the numerical output of the models is converted to a classification type by label- ing the positive WinLoss numbers with a “Win” and

negative WinLoss numbers with a “Loss” and then tabulating them in the confusion matrixes. Using the confusion matrices, the overall prediction accuracy, sensitivity, and specificity of each model type are calculated and presented in Tables 3.6 and 3.7. As the results indicate, the classification-type prediction methods performed better than regression-based classification-type prediction methodology. Among the three data mining technologies, classification and regression trees produced better prediction accuracy in both prediction methodologies. Overall, classification and regression tree classification mod- els produced a 10-fold cross-validation accuracy of 86.48 percent followed by support vector machines

TABLE 3.6 Prediction Results for the Direct Classification Methodology

Prediction Method (classification1)

Confusion Matrix

Accuracy2 (in %)

Sensitivity (in %)

Specificity (in %)

Win Loss

ANN (MLP) Win 92 42 75.00 68.66 82.73

Loss 19 91

SVM (RBF) Win 105 29 79.51 78.36 80.91

Loss 21 89

DT (C&RT) Win 113 21 86.48 84.33 89.09

Loss 12 98

1The output variable is a binary categorical variable (Win or Loss). 2Differences were significant.

TABLE 3.7 Prediction Results for the Regression-Based Classification Methodology

Prediction Method (regression based1)

Confusion Matrix

Accuracy2

Sensitivity

Specificity

Win Loss

ANN (MLP) Win 94 40 72.54 70.15 75.45

Loss 27 83

SVM (RBF) Win 100 34 74.59 74.63 74.55

Loss 28 82

DT (C&RT) Win 106 28 77.87 76.36 79.10

Loss 26 84

1The output variable is a numerical/integer variable (point-diff). 2Differences were sig p 6 0.01.

(Continued )

162 Part I • Introduction to Analytics and AI

daily sales totals for a grocery store. Often, time series are visualized using a line chart. Figure 3.17 shows an example time series of sales volumes for the years 2008 through 2012 on a quarterly basis.

Time-series forecasting is the use of mathematical modeling to predict future values of the variable of interest based on previously observed values. The time-series plots/charts look and feel very similar to simple linear regression in that, as was the case in simple linear regression, in time series there are two variables: the response variable and the time variable presented in a scatter plot. Beyond this appearance similarity, there is hardly any other commonality between the two. Although regression analysis is often employed in testing theories to see if current values of one or more explanatory variables explain (and hence predict) the response variable, the time-series models are focused on extrapolating on their time-varying behavior to estimate the future values.

Time-series-forecasting assumes that all of the explanatory variables are aggregated into the response variable as a time-variant behavior. Therefore, capturing the time- variant behavior is the way to predict the future values of the response variable. To do that, the pattern is analyzed and decomposed into its main components: random variations, time trends, and seasonal cycles. The time-series example shown in Figure 3.17 illustrates all of these distinct patterns.

The techniques used to develop time-series forecasts range from very simple (the naïve forecast that suggests today’s forecast is the same as yesterday’s actual) to very complex like ARIMA (a method that combines autoregressive and moving average pat- terns in data). Most popular techniques are perhaps the averaging methods that include simple average, moving average, weighted moving average, and exponential smoothing. Many of these techniques also have advanced versions when seasonality and trend can also be taken into account for better and more accurate forecasting. The accuracy of a method is usually assessed by computing its error (calculated deviation between actuals and forecasts for the past observations) via mean absolute error (MAE), mean squared error (MSE), or mean absolute percent error (MAPE). Even though they all use the same

(with a 10-fold cross-validation accuracy of 79.51 percent) and neural networks (with a 10-fold cross- validation accuracy of 75.00 percent). Using a t-test, researchers found that these accuracy values were significantly different at 0.05 alpha level; that is, the decision tree is a significantly better predictor of this domain than the neural network and support vec- tor machine, and the support vector machine is a significantly better predictor than neural networks.

The results of the study showed that the classification-type models predict the game out- comes better than regression-based classification models. Even though these results are specific to the application domain and the data used in this study and therefore should not be generalized beyond the scope of the study, they are exciting because deci- sion trees are not only the best predictors but also the best in understanding and deployment, com- pared to the other two machine-learning techniques

employed in this study. More details about this study can be found in Delen et al. (2012).

Questions for Case 3.4

1. What are the foreseeable challenges in predicting sporting event outcomes (e.g., college bowl games)?

2. How did the researchers formulate/design the prediction problem (i.e., what were the inputs and output, and what was the representation of a single sample—row of data)?

3. How successful were the prediction results? What else can they do to improve the accuracy?

Sources: D. Delen, D. Cogdell, and N. Kasap, “A Comparative Analysis of Data Mining Methods in Predicting NCAA Bowl Outcomes,” International Journal of Forecasting, 28, 2012, pp. 543–552; K. M. Freeman, and R. M. Brewer, “The Politics of American College Football,” Journal of Applied Business and Economics, 18(2), 2016, pp. 97–101.

Application Case 3.4 (Continued)

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 163

core error measure, these three assessment methods emphasize different aspects of the error, some penalizing larger errors more so than the others.

u SECTION 3.6 REVIEW QUESTIONS

1. What is regression, and what statistical purpose does it serve? 2. What are the commonalities and differences between regression and correlation? 3. What is OLS? How does OLS determine the linear regression line? 4. List and describe the main steps to follow in developing a linear repression model. 5. What are the most commonly pronounced assumptions for linear regression? 6. What is logistics regression? How does it differ from linear regression? 7. What is time series? What are the main forecasting techniques for time-series data?

3.7 BUSINESS REPORTING

Decision makers need information to make accurate and timely decisions. Information is essentially the contextualization of data. In addition to statistical means that were ex- plained in the previous section, information (descriptive analytics) can also be obtained using OLTP systems (see the simple taxonomy of descriptive analytics in Figure 3.7). The information is usually provided to decision makers in the form of a written report (digital or on paper), although it can also be provided orally. Simply put, a report is any com- munication artifact prepared with the specific intention of conveying information in a di- gestible form to whoever needs it whenever and wherever. It is typically a document that contains information (usually driven from data) organized in a narrative, graphic, and/or tabular form, prepared periodically (recurring) or on an as-needed (ad hoc) basis, refer- ring to specific time periods, events, occurrences, or subjects. Business reports can fulfill many different (but often related) functions. Here are a few of the most prevailing ones:

• To ensure that all departments are functioning properly. • To provide information.

0

1

2

3

4

5

6

7

8

9

10

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4

2008 2009 2010 2011 2012

Quarterly Product Sales (in millions)

FIGURE 3.17 A Sample Time Series of Data on Quarterly Sales Volumes.

164 Part I • Introduction to Analytics and AI

• To provide the results of an analysis. • To persuade others to act. • To create an organizational memory (as part of a knowledge management

system).

Business reporting (also called OLAP or BI) is an essential part of the larger drive to- ward improved, evidence-based, optimal managerial decision making. The foundation of these business reports is various sources of data coming from both inside and outside the organization (OLTP systems). Creation of these reports involves extract, transform, and load (ETL) procedures in coordination with a data warehouse and then using one or more reporting tools.

Due to the rapid expansion of IT coupled with the need for improved competitive- ness in business, there has been an increase in the use of computing power to produce unified reports that join different views of the enterprise in one place. Usually, this report- ing process involves querying structured data sources, most of which were created using different logical data models and data dictionaries, to produce a human-readable, easily digestible report. These types of business reports allow managers and coworkers to stay informed and involved, review options and alternatives, and make informed decisions. Figure 3.18 shows the continuous cycle of data acquisition S information generation S decision-making S business process management. Perhaps the most critical task in this cyclical process is the reporting (i.e., information generation)—converting data from dif- ferent sources into actionable information.

Key to any successful report are clarity, brevity, completeness, and correctness. The nature of the report and the level of importance of these success factors changes significantly based on for whom the report is created. Most of the research in effective reporting is dedicated to internal reports that inform stakeholders and decision makers within the organization. There are also external reports between businesses and the government (e.g., for tax purposes or for regular filings to the Securities and Exchange Commission). Even though there is a wide variety of business reports, the ones that are often used for managerial purposes can be grouped into three major categories (Hill, 2016).

Data Repositories

Business Functions

UOB 1.0 X

UOB 2.2

UOB 2.1 X UOB 3.0

Symbol Count Description

1 Machine Failure

Exception Event

Transactional Records

Information (reporting)

Decision Maker

Action (decision)

Data

1 2 3 4

5

FIGURE 3.18 The Role of Information Reporting in Managerial Decision Making.

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 165

METRIC MANAGEMENT REPORTS In many organizations, business performance is man- aged through outcome-oriented metrics. For external groups, these are service-level agreements. For internal management, they are key performance indicators (KPIs). Typically, there are enterprise-wide agreed upon targets to be tracked against over a pe- riod of time. They can be used as part of other management strategies such as Six Sigma or total quality management.

DASHBOARD-TYPE REPORTS A popular idea in business reporting in recent years has been to present a range of different performance indicators on one page like a dashboard in a car. Typically, dashboard vendors would provide a set of predefined reports with static elements and fixed structure but also allow for customization of the dashboard wid- gets, views, and set targets for various metrics. It is common to have color-coded traffic lights defined for performance (red, orange, green) to draw management’s attention to particular areas. A more detailed description of dashboards can be found in a later section of this chapter.

BALANCED SCORECARD–TYPE REPORTS This is a method developed by Kaplan and Norton that attempts to present an integrated view of success in an organization. In addition to financial performance, balanced scorecard–type reports also include cus- tomer, business process, and learning and growth perspectives. More details on balanced scorecards are provided in a later section in this chapter.

Application Case 3.5 is an example to illustrate the power and the utility of auto- mated report generation for a large (and, at a time of natural crisis, somewhat chaotic) organization such as the Federal Emergency Management Agency.

Staff at the Federal Emergency Management Agency (FEMA), the U.S. federal agency that coordinates disaster response when the president declares a national disaster, always got two floods at once. First, water covered the land. Next, a flood of paper required to administer the National Flood Insurance Program (NFIP) covered their desks— pallets and pallets of green-striped reports poured off a mainframe printer and into their offices. Individual reports were sometimes 18 inches thick with a nugget of information about insurance claims, premiums, or payments buried in them somewhere.

Bill Barton and Mike Miles do not claim to be able to do anything about the weather, but the project manager and computer scientist, respec- tively, from Computer Sciences Corporation (CSC) have used WebFOCUS software from Information Builders to turn back the flood of paper generated

by the NFIP. The program allows the government to work with national insurance companies to col- lect flood insurance premiums and pay claims for flooding in communities that adopt flood control measures. As a result of CSC’s work, FEMA staffs no longer leaf through paper reports to find the data they need. Instead, they browse insurance data posted on NFIP’s BureauNet intranet site, select just the information they want to see, and get an on-screen report or download the data as a spread- sheet. And that is only the start of the savings that WebFOCUS has provided. The number of times that NFIP staff ask CSC for special reports has dropped in half because NFIP staff can generate many of the special reports they need without calling on a pro- grammer to develop them. Then there is the cost of creating BureauNet in the first place. Barton esti- mates that using conventional Web and database software to export data from FEMA’s mainframe,

Application Case 3.5 Flood of Paper Ends at FEMA

(Continued )

166 Part I • Introduction to Analytics and AI

u SECTION 3.7 REVIEW QUESTIONS

1. What is a report? What are reports used for? 2. What is a business report? What are the main characteristics of a good business report? 3. Describe the cyclic process of management, and comment on the role of business

reports.

4. List and describe the three major categories of business reports. 5. What are the main components of a business reporting system?

3.8 DATA VISUALIZATION

Data visualization (or more appropriately, information visualization) has been defined as “the use of visual representations to explore, make sense of, and communicate data” (Few, 2007). Although the name that is commonly used is data visualization, usually what this means is information visualization. Because information is the aggregation, summarization, and contextualization of data (raw facts), what is portrayed in visualiza- tions is the information, not the data. However, because the two terms data visualiza- tion and information visualization are used interchangeably and synonymously, in this chapter we will follow suit.

Data visualization is closely related to the fields of information graphics, information visualization, scientific visualization, and statistical graphics. Until recently, the major forms of data visualization available in both BI applications have included charts and graphs as well as the other types of visual elements used to create scorecards and dashboards.

To better understand the current and future trends in the field of data visualization, it helps to begin with some historical context.

store it in a new database, and link that to a Web server would have cost about 100 times as much— more than $500,000—and taken about two years to complete compared with the few months Miles spent on the WebFOCUS solution.

When Tropical Storm Allison, a huge slug of sodden, swirling cloud, moved out of the Gulf of Mexico onto the Texas and Louisiana coastline in June 2001, it killed 34 people, most from drowning; damaged or destroyed 16,000 homes and businesses; and displaced more than 10,000 families. President George W. Bush declared 28 Texas counties disaster areas, and FEMA moved in to help. This was the first serious test for BureauNet, and it delivered. This first comprehensive use of BureauNet resulted in FEMA field staff readily accessing what they needed when they needed it and asking for many new types of reports. Fortunately, Miles and WebFOCUS were up to the task. In some cases, Barton says, “FEMA would ask for a new type of report one day, and Miles would have it on BureauNet the next day,

thanks to the speed with which he could create new reports in WebFOCUS.”

The sudden demand on the system had little impact on its performance, noted Barton. “It han- dled the demand just fine,” he says. “We had no problems with it at all. And it made a huge differ- ence to FEMA and the job they had to do. They had never had that level of access before, never had been able to just click on their desktop and generate such detailed and specific reports.”

Questions for Case 3.5

1. What is FEMA, and what does it do?

2. What are the main challenges that FEMA faces?

3. How did FEMA improve its inefficient reporting practices?

Source: Used with permission from Information Builders. Useful information flows at disaster response agency. informationbuild- ers.com/applications/fema (accessed July 2018); and fema.gov.

Application Case 3.5 (Continued)

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 167

Brief History of Data Visualization

Despite the fact that predecessors to data visualization date back to the second century AD, most developments have occurred in the last two and a half centuries, predominantly during the last 30 years (Few, 2007). Although visualization has not been widely recognized as a discipline until fairly recently, today’s most popular visual forms date back a few centuries. Geographical exploration, mathematics, and popularized history spurred the creation of early maps, graphs, and timelines as far back as the 1600s, but William Playfair is widely credited as the inventor of the modern chart, having created the first widely distributed line and bar charts in his Commercial and Political Atlas of 1786 and what is generally considered to be the first time-series line chart in his Statistical Breviary published in 1801 (see Figure 3.19).

Perhaps the most notable innovator of information graphics during this period was Charles Joseph Minard, who graphically portrayed the losses suffered by Napoleon’s army in the Russian campaign of 1812 (see Figure 3.20). Beginning at the Polish–Russian border, the thick band shows the size of the army at each position. The path of Napoleon’s retreat from Moscow in the bitterly cold winter is depicted by the dark lower band, which is tied to tem- perature and time scales. Popular visualization expert, author, and critic Edward Tufte says that this “may well be the best statistical graphic ever drawn.” In this graphic, Minard man- aged to simultaneously represent several data dimensions (the size of the army, direction of movement, geographic locations, outside temperature, etc.) in an artistic and informative

FIGURE 3.19 The First Time-Series Line Chart Created by William PlayFair in 1801.

168 Part I • Introduction to Analytics and AI

manner. Many more excellent visualizations were created in the 1800s, and most of them are chronicled on Tufte’s Web site (edwardtufte.com) and his visualization books.

The 1900s saw the rise of a more formal, empirical attitude toward visualization, which tended to focus on aspects such as color, value scales, and labeling. In the mid- 1900s, cartographer and theorist Jacques Bertin published his Semiologie Graphique, which some say serves as the theoretical foundation of modern information visualization. Although most of his patterns are either outdated by more recent research or completely inapplicable to digital media, many are still very relevant.

In the 2000s, the Internet emerged as a new medium for visualization and brought with it many new tricks and capabilities. Not only has the worldwide, digital distribution of both data and visualization made them more accessible to a broader audience (raising visual literacy along the way), but also it has spurred the design of new forms that incorporate interaction, animation, and graphics-rendering technology unique to screen media and real- time data feeds to create immersive environments for communicating and consuming data.

Companies and individuals are, seemingly all of a sudden, interested in data; that in- terest has in turn sparked a need for visual tools that help them understand it. Cheap hard- ware sensors and do-it-yourself frameworks for building your own system are driving down the costs of collecting and processing data. Countless other applications, software tools, and low-level code libraries are springing up to help people collect, organize, manipulate, visualize, and understand data from practically any source. The Internet has also served as a fantastic distribution channel for visualizations; a diverse community of designers, program- mers, cartographers, tinkerers, and data wonks has assembled to disseminate all sorts of new ideas and tools for working with data in both visual and nonvisual forms.

Google Maps has also single-handedly democratized both the interface conven- tions (click to pan, double-click to zoom) and the technology (256-pixel square map tiles with predictable file names) for displaying interactive geography online to the extent that most people just know what to do when they are presented with a map online. Flash has served well as a cross-browser platform on which to design and develop rich, beautiful Internet applications incorporating interactive data visualization and maps; now, new

FIGURE 3.20 Decimation of Napoleon’s Army during the 1812 Russian Campaign.

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 169

browser-native technologies such as canvas and SVG (sometimes collectively included under the umbrella of HTML5) are emerging to challenge Flash’s supremacy and extend the reach of dynamic visualization interfaces to mobile devices.

The future of data/information visualization is very hard to predict. We can only extrapolate from what has already been invented: more three-dimensional visualization, immersive experience with multidimensional data in a virtual reality environment, and holographic visualization of information. There is a pretty good chance that we will see something that we have never seen in the information visualization realm invented be- fore the end of this decade. Application Case 3.6 shows how visual analytics/reporting tools such as Tableau can help facilitate effective and efficient decision making through information/insight creation and sharing.

The Background

Macfarlan Smith has earned its place in medical history. The company held a royal appointment to provide medicine to Her Majesty Queen Victoria and supplied groundbreaking obstetrician Sir James Simpson with chloroform for his experiments in pain relief during labor and delivery. Today, Macfarlan Smith is a sub- sidiary of the Fine Chemical and Catalysts division of Johnson Matthey plc. The pharmaceutical manufac- turer is the world’s leading manufacturer of opiate narcotics such as codeine and morphine.

Every day, Macfarlan Smith is making decisions based on its data. The company collects and ana- lyzes manufacturing operational data, for example, to allow it to meet continuous improvement goals. Sales, marketing, and finance rely on data to identify new pharmaceutical business opportunities, grow revenues, and satisfy customer needs. Additionally, the company’s manufacturing facility in Edinburgh needs to monitor, trend, and report quality data to ensure the identity, quality, and purity of its phar- maceutical ingredients for customers and regulatory authorities such as the U.S. FDA and others as part of current good manufacturing practice (CGMP).

Challenges: Multiple Sources of Truth and Slow, Onerous Reporting Processes

The process of gathering that data, making decisions, and reporting was not easy, though. The data were

scattered across the business including in the compa- ny’s bespoke enterprise resource planning (ERP) plat- form, inside legacy departmental databases such as SQL, Access databases, and stand-alone spreadsheets. When those data were needed for decision mak- ing, excessive time and resources were devoted to extracting the data, integrating them, and presenting them in a spreadsheet or other presentation outlet.

Data quality was another concern. Because teams relied on their own individual sources of data, there were multiple versions of the truth and con- flicts between the data. And it was sometimes hard to tell which version of the data was correct and which was not.

It didn’t stop there. Even once the data had been gathered and presented, making changes “on the fly” was slow and difficult. In fact, whenever a member of the Macfarlan Smith team wanted to per- form trend or other analysis, the changes to the data needed to be approved. The end result was that the data were frequently out of date by the time they were used for decision making.

Liam Mills, Head of Continuous Improvement at Macfarlan Smith highlights a typical reporting scenario:

One of our main reporting processes is the “Corrective Action and Preventive Action,” or CAPA, which is an analysis of Macfarlan Smith’s manufacturing processes taken to eliminate causes of non-conformities or other unde- sirable situations. Hundreds of hours every month were devoted to pulling data together for CAPA—and it took days to produce each

Application Case 3.6 Macfarlan Smith Improves Operational Performance Insight with Tableau Online

(Continued )

170 Part I • Introduction to Analytics and AI

report. Trend analysis was tricky too, because the data was static. In other reporting scenar- ios, we often had to wait for spreadsheet pivot table analysis; which was then presented on a graph, printed out, and pinned to a wall for everyone to review.

Slow, labor-intensive reporting processes, dif- ferent versions of the truth, and static data were all catalysts for change. “Many people were frustrated because they believed they didn’t have a complete picture of the business,” says Mills. “We were having more and more discussions about issues we faced— when we should have been talking about business intelligence reporting.”

The Solution: Interactive Data Visualizations

One of the Macfarlan Smith team had previous expe- rience in using Tableau and recommended Mills explore the solution further. A free trial of Tableau Online quickly convinced Mills that the hosted inter- active data visualization solution could conquer the data battles the company was facing.

“I was won over almost immediately,” he says. “The ease of use, the functionality and the breadth of data visualizations are all very impressive. And of course being a software-as-a-service (SaaS)-based solution, there’s no technology infrastructure invest- ment, we can be live almost immediately, and we have the flexibility to add users whenever we need.”

One of the key questions that needed to be answered concerned the security of the online data. “Our parent company Johnson Matthey has a cloud- first strategy, but has to be certain that any hosted solution is completely secure. Tableau Online fea- tures like single sign-on and allowing only autho- rized users to interact with the data provide that watertight security and confidence.”

The other security question that Macfarlan Smith and Johnson Matthey wanted answered was this: Where are the data physically stored? Mills again: “We are satisfied Tableau Online meets our criteria for data security and privacy. The data and workbooks are all hosted in Tableau’s new Dublin data center, so it never leaves Europe.”

Following a six-week trial, the Tableau sales manager worked with Mills and his team to build a

business case for Tableau Online. The management team approved it almost straight away, and a pilot program involving 10 users began. The pilot involved a manufacturing quality improvement initiative: look- ing at deviations from the norm, such as when a heat- ing device used in the opiate narcotics manufacturing process exceeds a temperature threshold. From this, a “quality operations” dashboard was created to track and measure deviations and put in place measures to improve operational quality and performance.

“That dashboard immediately signaled where deviations might be. We weren’t ploughing through rows of data—we reached answers straight away,” says Mills.

Throughout this initial trial and pilot, the team used Tableau training aids, such as the free training videos, product walk-throughs, and live online train- ing. They also participated in a two-day “fundamen- tals training” event in London. According to Mills, “The training was expert, precise and pitched just at the right level. It demonstrated to everyone just how intuitive Tableau Online is. We can visualize 10 years’ worth of data in just a few clicks.” The com- pany now has five Tableau Desktop users and up to 200 Tableau Online licensed users.

Mills and his team particularly like the Tableau Union feature in Version 9.3, which allows them to piece together data that have been split into little files. “It’s sometimes hard to bring together the data we use for analysis. The Union feature lets us work with data spread across multiple tabs or files, reducing the time we spend on prepping the data,” he says.

The Results: Cloud Analytics Transform Decision Making and Reporting

By standardizing on Tableau Online, Macfarlan Smith has transformed the speed and accuracy of its deci- sion making and business reporting. This includes:

• New interactive dashboards can be produced within one hour. Previously, it used to take days to integrate and present data in a static spreadsheet.

• The CAPA manufacturing process report, which used to absorb hundreds of man-hours every month and days to produce, can now be produced in minutes—with insights shared in the cloud.

Application Case 3.6 (Continued)

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 171

u SECTION 3.8 REVIEW QUESTIONS

1. What is data visualization? Why is it needed? 2. What are the historical roots of data visualization? 3. Carefully analyze Charles Joseph Minard’s graphical portrayal of Napoleon’s march.

Identify and comment on all the information dimensions captured in this ancient diagram.

4. Who is Edward Tufte? Why do you think we should know about his work? 5. What do you think is the “next big thing” in data visualization?

3.9 DIFFERENT TYPES OF CHARTS AND GRAPHS

Often end users of business analytics systems are not sure what type of chart or graph to use for a specific purpose. Some charts or graphs are better at answering certain types of questions. Some look better than others. Some are simple; some are rather complex and crowded. What follows is a short description of the types of charts and/or graphs com- monly found in most business analytics tools and the types of questions they are better at answering/analyzing. This material is compiled from several published articles and other literature (Abela, 2008; Hardin et al., 2012; SAS, 2014).

Basic Charts and Graphs

What follows are the basic charts and graphs that are commonly used for information visualization.

LINE CHART The line chart is perhaps the most frequently used graphical visuals for time-series data. Line charts (or line graphs) show the relationship between two variables; they are most often used to track changes or trends over time (having one of the vari- ables set to time on the x-axis). Line charts sequentially connect individual data points to help infer changing trends over a period of time. Line charts are often used to show time-dependent changes in the values of some measure, such as changes in a specific

• Reports can be changed and interrogated “on the fly” quickly and easily, without technical intervention. Macfarlan Smith has the flexibility to publish dashboards with Tableau Desktop and share them with colleagues, partners, or customers.

• The company has one, single, trusted version of the truth.

• Macfarlan Smith is now having discussions about its data—not about the issues surround- ing data integration and data quality.

• New users can be brought online almost instantly—and there’s no technical infrastruc- ture to manage.

Following this initial success, Macfarlan Smith is now extending Tableau Online to financial report- ing, supply chain analytics, and sales forecasting. Mills

concludes, “Our business strategy is now based on data-driven decisions, not opinions. The interactive visualizations enable us to spot trends instantly, identify process improvements and take business intelligence to the next level. I’ll define my career by Tableau.”

Questions for Case 3.6

1. What were the data and reporting related chal- lenges that Macfarlan Smith faced?

2. What were the solution and the obtained results/ benefits?

Source: Tableau Customer Case Study, “Macfarlan Smith improves operational performance insight with Tableau Online,” http://www. tableau.com/stories/customer/macfarlan-smith- improves- operational-performance-insight-tableau-online (accessed June 2018). Used with permission from Tableau Software, Inc.

172 Part I • Introduction to Analytics and AI

stock price over a five-year period or changes in the number of daily customer service calls over a month.

BAR CHART The bar chart is among the most basic visuals used for data representation. They are effective when you have nominal data or numerical data that split nicely into different categories so you can quickly see comparative results and trends within your data. Bar charts are often used to compare data across multiple categories such as the percentage of advertising spending by departments or by product categories. Bar charts can be vertically or horizontally oriented. They can also be stacked on top of each other to show multiple dimensions in a single chart.

PIE CHART The pie chart is visually appealing, as the name implies, pie-looking charts. Because they are so visually attractive, they are often incorrectly used. Pie charts should be used only to illustrate relative proportions of a specific measure. For instance, they can be used to show the relative percentage of an advertising budget spent on differ- ent product lines, or they can show relative proportions of majors declared by college students in their sophomore year. If the number of categories to show is more than just a few (say more than four), one should seriously consider using a bar chart instead of a pie chart.

SCATTER PLOT The scatter plot is often used to explore the relationship between two or three variables (in 2D or 3D visuals). Because scatter plots are visual exploration tools, translating more than three variables into more than three dimensions is not easily achiev- able. Scatter plots are an effective way to explore the existence of trends, concentrations, and outliers. For instance, in a two-variable (two-axis) graph, a scatter plot can be used to illustrate the co-relationship between age and weight of heart disease patients, or it can illustrate the relationship between the number of customer care representatives and the number of open customer service claims. Often, a trend line is superimposed on a two- dimensional scatter plot to illustrate the nature of the relationship.

BUBBLE CHART The bubble chart is often an enhanced version of scatter plots. Bubble charts, though, are not a new visualization type; instead, they should be viewed as a tech- nique to enrich data illustrated in scatter plots (or even geographic maps). By varying the size and/or color of the circles, one can add additional data dimensions, offering more enriched meaning about the data. For instance, a bubble chart can be used to show a competitive view of college-level class attendance by major and by time of the day, and it can be used to show profit margin by product type and by geographic region.

Specialized Charts and Graphs

The graphs and charts that we review in this section are either derived from the basic charts as special cases or they are relatively new and are specific to a problem type and/ or an application area.

HISTOGRAM Graphically speaking, a histogram looks just like a bar chart. The dif- ference between histograms and generic bar charts is the information that is portrayed. Histograms are used to show the frequency distribution of one variable or several vari- ables. In a histogram, the x-axis is often used to show the categories or ranges, and the y-axis is used to show the measures/values/frequencies. Histograms show the distribu- tional shape of the data. That way, one can visually examine whether the data are nor- mally or exponentially distributed. For instance, one can use a histogram to illustrate the

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 173

exam performance of a class, to show distribution of the grades as well as comparative analysis of individual results, or to show the age distribution of the customer base.

GANTT CHART A Gantt chart is a special case of horizontal bar charts used to portray project timelines, project tasks/activity durations, and overlap among the tasks/activities. By showing start and end dates/times of tasks/activities and the overlapping relation- ships, Gantt charts provide an invaluable aid for management and control of projects. For instance, Gantt charts are often used to show project timelines, task overlaps, relative task completions (a partial bar illustrating the completion percentage inside a bar that shows the actual task duration), resources assigned to each task, milestones, and deliverables.

PERT CHART The PERT chart (also called a network diagram) is developed primarily to simplify the planning and scheduling of large and complex projects. A PERT chart shows precedence relationships among project activities/tasks. It is composed of nodes (rep- resented as circles or rectangles) and edges (represented with directed arrows). Based on the selected PERT chart convention, either nodes or the edges can be used to repre- sent the project activities/tasks (activity-on-node versus activity-on-arrow representation schema).

GEOGRAPHIC MAP When the data set includes any kind of location data (e.g., physical addresses, postal codes, state names or abbreviations, country names, latitude/longitude, or some type of custom geographic encoding), it is better and more informative to see the data on a map. Maps usually are used in conjunction with other charts and graphs rather than by themselves. For instance, one can use maps to show the distribution of customer service requests by product type (depicted in pie charts) by geographic locations. Often a large variety of information (e.g., age distribution, income distribution, education, eco- nomic growth, population changes) can be portrayed in a geographic map to help decide where to open a new restaurant or a new service station. These types of systems are often called geographic information systems (GIS).

BULLET A bullet graph is often used to show progress toward a goal. This graph is essentially a variation of a bar chart. Often bullet graphs are used in place of gauges, meters, and thermometers in a dashboard to more intuitively convey the meaning within a much smaller space. Bullet graphs compare a primary measure (e.g., year-to-date rev- enue) to one or more other measures (e.g., annual revenue target) and present this in the context of defined performance metrics (e.g., sales quotas). A bullet graph can intuitively illustrate how the primary measure is performing against overall goals (e.g., how close a sales representative is to achieving his or her annual quota).

HEAT MAP The heat map is a great visual to illustrate the comparison of continuous values across two categories using color. The goal is to help the user quickly see where the intersection of the categories is strongest and weakest in terms of numerical values of the measure being analyzed. For instance, one can use a heat map to show segmenta- tion analysis of target markets where the measure (color gradient would be the purchase amount) and the dimensions would be age and income distribution.

HIGHLIGHT TABLE The highlight table is intended to take heat maps one step further. In addition to showing how data intersect by using color, highlight tables add a number on top to provide additional detail. That is, they are two-dimensional tables with cells popu- lated with numerical values and gradients of colors. For instance, one can show sales representatives’ performance by product type and by sales volume.

174 Part I • Introduction to Analytics and AI

TREE MAP A tree map displays hierarchical (tree-structured) data as a set of nested rectangles. Each branch of the tree is given a rectangle, which is then tiled with smaller rectangles representing subbranches. A leaf node’s rectangle has an area proportional to a specified dimension on the data. Often the leaf nodes are colored to show a separate dimension of the data. When the color and size dimensions are correlated in some way with the tree structure, one can often easily see patterns that would be difficult to spot in other ways, such as a certain color that is particularly relevant. A second advantage of tree maps is that, by construction, they make efficient use of space. As a result, they can legibly display thousands of items on the screen simultaneously.

Which Chart or Graph Should You Use?

Which chart or graph that we explained in the previous section is the best? The answer is rather easy: There is not one best chart or graph because if there were, we would not have so many chart and graph types. They all have somewhat different data representa- tion “skills.” Therefore, the right question should be, “Which chart or graph is the best for a given task?” The capabilities of the charts given in the previous section can help in selecting and using the proper chart/graph for a specific task, but doing so still is not easy to sort out. Several different chart/graph types can be used for the same visualiza- tion task. One rule of thumb is to select and use the simplest one from the alternatives to make it easy for the intended audience to understand and digest.

Although there is not a widely accepted, all-encompassing chart selection algorithm or chart/graph taxonomy, Figure 3.21 presents a rather comprehensive and highly logical

Single Variable

What would you like to show in your chart or graph?

Composition

DistributionRelationship

Two Variables

Three Variables

Changing over Time

Static Few Periods Many Periods

Only Relative Difference Matters

Relative and Absolute Difference

Matter

Only Relative Difference Matters

Relative and Absolute Difference

Matter

Simple Share

of Total

Accumulation or Subtraction

to Total

Components of

Components

Two Variables

Three Variables

Among Items Over Time

Two Variables per Item

One Variable per Item

Many Categories Few Categories

Few ItemsMany Items

Many Periods Few Periods

Cyclic Data Non-Cyclic Data Single or Few Categories

Many Categories

Many Data Points

Few Data Points

Comparison

FIGURE 3.21 A Taxonomy of Charts and Graphs. Source: Adapted from Abela, A. (2008). Advanced Presentations by Design: Creating Communication That Drives Action. New York: Wiley.

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 175

organization of chart/graph types in a taxonomy-like structure (the original version was published in Abela, 2008). The taxonomic structure is organized around the questions of “What would you like to show in your chart or graph?”—that is, what the purpose of the chart or graph will be. At that level, the taxonomy divides the purpose into four different types—relationship, comparison, distribution, and composition—and further divides the branches into subcategories based on the number of variables involved and time depen- dency of the visualization.

Even though these charts and graphs cover a major part of what is commonly used in information visualization, they by no means cover all. Today, one can find many other specialized graphs and charts that serve a specific purpose. Furthermore, the cur- rent trend is to combine/hybridize and animate these charts for better-looking and more intuitive visualization of today’s complex and volatile data sources. For instance, the interactive, animated, bubble charts available at the Gapminder Web site (gapminder. org) provide an intriguing way of exploring world health, wealth, and population data from a multidimensional perspective. Figure 3.22 depicts the types of displays available at that site. In this graph, population size, life expectancy, and per capita income at the continent level are shown; also given is a time-varying animation that shows how these variables change over time.

FIGURE 3.22 A Gapminder Chart That Shows the Wealth and Health of Nations. Source: gapminder.org.

176 Part I • Introduction to Analytics and AI

u SECTION 3.9 REVIEW QUESTIONS

1. Why do you think there are many different types of charts and graphs? 2. What are the main differences among line, bar, and pie charts? When should you use

one over the others?

3. Why would you use a geographic map? What other types of charts can be combined with it?

4. Find and explain the role of two types of charts that are not covered in this section.

3.10 EMERGENCE OF VISUAL ANALYTICS

As Seth Grimes (2009a, b) has noted, there is a “growing palate” of data visualization tech- niques and tools that enable the users of business analytics and BI systems to better “commu- nicate relationships, add historical context, uncover hidden correlations, and tell persuasive stories that clarify and call to action.” The latest Magic Quadrant for Business Intelligence and Analytics Platforms released by Gartner in February 2016 further emphasizes the impor- tance of data visualization in BI and analytics. As the chart in Figure 3.23 shows, all solution

FIGURE 3.23 Magic Quadrant for Business Intelligence and Analytics Platforms. Source: Used with permission from Gartner Inc.

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 177

providers in the Leaders and Visionaries quadrants are either relatively recently founded information visualization companies (e.g., Tableau Software, QlikTech) or well-established large analytics companies (e.g., Microsoft, SAS, IBM, SAP, MicroStrategy, Alteryx) that are increasingly focusing their efforts on information visualization and visual analytics. More de- tails on Gartner’s latest Magic Quadrant are given in Technology Insights 3.2.

In BI and analytics, the key challenges for visualization have revolved around the intuitive representation of large, complex data sets with multiple dimensions and mea- sures. For the most part, the typical charts, graphs, and other visual elements used in these applications usually involve two dimensions, sometimes three, and fairly small sub- sets of data sets. In contrast, the data in these systems reside in a data warehouse. At a

TECHNOLOGY INSIGHTS 3.2 Gartner Magic Quadrant for Business Intelligence and Analytics Platforms

Gartner, Inc., the creator of Magic Quadrants, is the leading IT research and advisory company publically traded in the United States with over $2 billion annual revenues in 2015. Founded in 1979, Gartner has 7,600 associates, including 1,600 research analysts and consultants and numer- ous clients in 90 countries.

Magic Quadrant is a research method designed and implemented by Gartner to monitor and evaluate the progress and positions of companies in a specific, technology-based market. By applying a graphical treatment and a uniform set of evaluation criteria, Magic Quadrant helps users to understand how technology providers are positioned within a market.

Gartner changed the name of this Magic Quadrant from “Business Intelligence Platforms” to “Business Intelligence and Analytics Platforms” to emphasize the growing importance of ana- lytics capabilities to the information systems that organizations are now building. Gartner defines the BI and analytics platform market as a software platform that delivers 15 capabilities across three categories: integration, information delivery, and analysis. These capabilities enable orga- nizations to build precise systems of classification and measurement to support decision making and improve performance.

Figure 3.23 illustrates the latest Magic Quadrant for Business Intelligence and Analytics Platforms. Magic Quadrant places providers in four groups (niche players, challengers, visionar- ies, and leaders) along two dimensions: completeness of vision (x-axis) and ability to execute (y-axis). As the quadrant clearly shows, most of the well-known BI/BA (business analytics) providers are positioned in the “leaders” category while many of the less known, relatively new, emerging providers are positioned in the “niche players” category.

The BI and analytics platform market’s multiyear shift from IT-led enterprise reporting to business-led self-service analytics seems to have passed the tipping point. Most new buying is of modern, business-user-centric visual analytics platforms forcing a new market perspective, sig- nificantly reordering the vendor landscape. Most of the activity in the BI and analytics platform market is from organizations that are trying to mature their visualization capabilities and to move from descriptive to predictive and prescriptive analytics echelons. The vendors in the market have overwhelmingly concentrated on meeting this user demand. If there were a single market theme in 2015, it would be that data discovery/visualization became a mainstream architec- ture. While data discovery/visualization vendors such as Tableau, Qlik, and Microsoft are solidi- fying their position in the Leaders quadrant, others (both emerging and large, well-established tool/solution providers) are trying to move out of Visionaries into the Leaders quadrant.

This emphasis on data discovery/visualization from most of the leaders and visionar- ies in the market—which are now promoting tools with business-user-friendly data integration coupled with embedded storage and computing layers and unfettered drilling—continues to accelerate the trend toward decentralization and user empowerment of BI and analytics and greatly enables organizations’ ability to perform diagnostic analytics.

Source: Gartner Magic Quadrant, released on February 4, 2016, gartner.com (accessed August 2016). Used with permission from Gartner Inc.

178 Part I • Introduction to Analytics and AI

minimum, these warehouses involve a range of dimensions (e.g., product, location, orga- nizational structure, time), a range of measures, and millions of cells of data. In an effort to address these challenges, a number of researchers have developed a variety of new visualization techniques.

Visual Analytics

Visual analytics is a recently coined term that is often used loosely to mean nothing more than information visualization. What is meant by visual analytics is the combi- nation of visualization and predictive analytics. Whereas information visualization is aimed at answering “What happened?” and “What is happening?” and is closely associ- ated with BI (routine reports, scorecards, and dashboards), visual analytics is aimed at answering “Why is it happening?” and “What is more likely to happen?” and is usu- ally associated with business analytics (forecasting, segmentation, correlation analysis). Many of the information visualization vendors are adding the capabilities to call them- selves visual analytics solution providers. One of the top, long-time analytics solution providers, SAS Institute, is approaching it from another direction. It is embedding its analytics capabilities into a high-performance data visualization environment that it calls visual analytics.

Visual or not visual, automated or manual, online or paper based, business report- ing is not much different than telling a story. Technology Insights 3.3 provides a different, unorthodox viewpoint on better business reporting.

TECHNOLOGY INSIGHTS 3.3 Telling Great Stories with Data and Visualization

Everyone who has data to analyze has stories to tell, whether it’s diagnosing the reasons for manufacturing defects, selling a new idea in a way that captures the imagination of your target audience, or informing colleagues about a particular customer service improvement program. And when it’s telling the story behind a big strategic choice so that you and your senior management team can make a solid decision, providing a fact-based story can be es- pecially challenging. In all cases, it’s a big job. You want to be interesting and memorable; you know you need to keep it simple for your busy executives and colleagues. Yet you also know you have to be factual, detail oriented, and data driven, especially in today’s metric- centric world.

It’s tempting to present just the data and facts, but when colleagues and senior manage- ment are overwhelmed by data and facts without context, you lose. We have all experienced presentations with large slide decks only to find that the audience is so overwhelmed with data that they don’t know what to think, or they are so completely tuned out that they take away only a fraction of the key points.

Start engaging your executive team and explaining your strategies and results more powerfully by approaching your assignment as a story. You will need the “what” of your story (the facts and data), but you also need the “Who?” “How?” “Why?” and the often-missed “So what?” It’s these story elements that will make your data relevant and tangible for your audience. Creating a good story can aid you and senior management in focusing on what is important.

Why Story? Stories bring life to data and facts. They can help you make sense and order out of a disparate collection of facts. They make it easier to remember key points and can paint a vivid picture of what the future can look like. Stories also create interactivity—people put themselves into stories and can relate to the situation.

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 179

Cultures have long used storytelling to pass on knowledge and content. In some cultures, storytelling is critical to their identity. For example, in New Zealand, some of the Maori people tattoo their faces with mokus. A moku is a facial tattoo containing a story about ancestors—the family tribe. A man may have a tattoo design on his face that shows features of a hammerhead to highlight unique qualities about his lineage. The design he chooses signifies what is part of his “true self” and his ancestral home.

Likewise, when we are trying to understand a story, the storyteller navigates to finding the “true north.” If senior management is looking to discuss how they will respond to a competitive change, a good story can make sense and order out of a lot of noise. For example, you may have facts and data from two studies, one including results from an advertising study and one from a product satisfaction study. Developing a story for what you measured across both studies can help people see the whole where there were disparate parts. For rallying your distributors around a new product, you can employ a story to give vision to what the future can look like. Most important, storytelling is interactive—typically, the presenter uses words and pictures that audience members can put themselves into. As a result, they become more engaged and better understand the information.

So What Is a Good Story? Most people can easily rattle off their favorite film or book. Or they remember a funny story that a colleague recently shared. Why do people remember these stories? Because they contain cer- tain characteristics. First, a good story has great characters. In some cases, the reader or viewer has a vicarious experience where they become involved with the character. The character then has to be faced with a challenge that is difficult but believable. There must be hurdles that the character overcomes. And finally, the outcome or prognosis is clear by the end of the story. The situation may not be resolved—but the story has a clear endpoint.

Think of Your Analysis as a Story—Use a Story Structure When crafting a data-rich story, the first objective is to find the story. Who are the characters? What is the drama or challenge? What hurdles have to be overcome? And at the end of your story, what do you want your audience to do as a result?

Once you know the core story, craft your other story elements: define your characters, understand the challenge, identify the hurdles, and crystallize the outcome or decision question. Make sure you are clear with what you want people to do as a result. This will shape how your audience will recall your story. With the story elements in place, write out the storyboard, which represents the structure and form of your story. Although it’s tempting to skip this step, it is bet- ter first to understand the story you are telling and then to focus on the presentation structure and form. Once the storyboard is in place, the other elements will fall into place. The storyboard will help you think about the best analogies or metaphors, clearly set up challenge or oppor- tunity, and finally see the flow and transitions needed. The storyboard also helps you focus on key visuals (graphs, charts, and graphics) that you need your executives to recall. Figure 3.24 shows a storyline for the impact of small loans in a worldwide view within the Tableau visual analytics environment.

In summary, do not be afraid to use data to tell great stories. Being factual, detail oriented, and data driven is critical in today’s metric-centric world, but it does not have to mean being boring and lengthy. In fact, by finding the real stories in your data and following the best prac- tices, you can get people to focus on your message—and thus on what’s important. Here are those best practices:

1. Think of your analysis as a story—use a story structure. 2. Be authentic—your story will flow. 3. Be visual—think of yourself as a film editor. 4. Make it easy for your audience and you. 5. Invite and direct discussion.

Source: Fink, E., & Moore, S. J. (2012). “Five Best Practices for Telling Great Stories with Data.” White paper by Tableau Software, Inc., www.tableau.com/whitepapers/telling-data-stories (accessed May 2016).

180 Part I • Introduction to Analytics and AI

High-Powered Visual Analytics Environments

Due to the increasing demand for visual analytics coupled with fast-growing data volumes, there is an exponential movement toward investing in highly efficient visualization sys- tems. With its latest move into visual analytics, the statistical software giant SAS Institute is now among those who are leading this wave. Its new product, SAS Visual Analytics, is a very high-performance computing, in-memory solution for exploring massive amounts of data in a very short time (almost instantaneously). It empowers users to spot patterns, identify opportunities for further analysis, and convey visual results via Web reports or mobile platforms such as tablets and smartphones. Figure 3.24 shows the high-level ar- chitecture of the SAS Visual Analytics platform. On one end of the architecture, there are universal data builder and administrator capabilities, leading into explorer, report designer, and mobile BI modules, collectively providing an end-to-end visual analytics solution.

Some of the key benefits proposed by the SAS analytics platform (see Figure 3.25) are the following:

• Empowers all users with data exploration techniques and approachable analytics to drive improved decision making. SAS Visual Analytics enables different types of users to conduct fast, thorough explorations on all available data. Sampling to re- duce the data is not required and not preferred.

• Has easy-to-use, interactive Web interfaces that broaden the audience for analyt- ics, enabling everyone to glean new insights. Users can look at additional options, make more precise decisions, and drive success even faster than before.

FIGURE 3.24 A Storyline Visualization in Tableau Software. Source: Used with permission from Tableau Software, Inc.

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 181

• Answers complex questions faster, enhancing the contributions from your analytic talent. SAS Visual Analytics augments the data discovery and exploration process by providing extremely fast results to enable better, more focused analysis. Analytically savvy users can identify areas of opportunity or concern from vast amounts of data so further investigation can take place quickly.

• Improves information sharing and collaboration. Large numbers of users, including those with limited analytical skills, can quickly view and interact with reports and charts via the Web, Adobe PDF files, and iPad mobile devices while IT maintains control of the underlying data and security. SAS Visual Analytics provides the right information to the right person at the right time to improve productivity and orga- nizational knowledge.

• Liberates IT by giving users a new way to access the information they need. Frees IT from the constant barrage of demands from users who need access to different amounts of data, different data views, ad hoc reports, and one-off requests for infor- mation. SAS Visual Analytics enables IT to easily load and prepare data for multiple users. Once data are loaded and available, users can dynamically explore data, cre- ate reports, and share information on their own.

• Provides room to grow at a self-determined pace. SAS Visual Analytics provides the option of using commodity hardware or database appliances from EMC Greenplum and Teradata. It is designed from the ground up for performance optimization and scalability to meet the needs of any size organization.

Figure 3.26 shows a screenshot of a SAS Analytics platform where time-series fore- casting and confidence interval around the forecast are depicted.

u SECTION 3.10 REVIEW QUESTIONS

1. What are the main reasons for the recent emergence of visual analytics? 2. Look at Gartner’s Magic Quadrant for Business Intelligence and Analytics Platforms.

What do you see? Discuss and justify your observations.

3. What is the difference between information visualization and visual analytics?

FIGURE 3.25 An Overview of SAS Visual Analytics Architecture. Source: Copyright © SAS Institute, Inc. Used with permission.

182 Part I • Introduction to Analytics and AI

4. Why should storytelling be a part of your reporting and data visualization? 5. What is a high-powered visual analytics environment? Why do we need it?

3.11 INFORMATION DASHBOARDS

Information dashboards are common components of most, if not all, BI or business ana- lytics platforms, business performance management systems, and performance measure- ment software suites. Dashboards provide visual displays of important information that is consolidated and arranged on a single screen so that the information can be digested at a single glance and easily drilled in and further explored. A typical dashboard is shown in Figure 3.27. This particular executive dashboard displays a variety of key per- formance indicators (KPIs) for a hypothetical software company called Sonatica (selling audio tools). This executive dashboard shows a high-level view of the different functional groups surrounding the products, starting from a general overview to the marketing ef- forts, sales, finance, and support departments. All of this is intended to give executive

FIGURE 3.26 A Screenshot from SAS Visual Analytics. Source: Copyright © SAS Institute, Inc. Used with permission.

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 183

decision makers a quick and accurate idea of what is going on within the organization. On the left side of the dashboard, we can see (in a time-series fashion) the quarterly changes in revenues, expenses, and margins as well as the comparison of those figures to previous years’ monthly numbers. On the upper-right side are two dials with color- coded regions showing the amount of monthly expenses for support services (dial on the left) and the amount of other expenses (dial on the right). As the color coding in- dicates, although the monthly support expenses are well within the normal ranges, the other expenses are in the red region, indicating excessive values. The geographic map on the bottom right shows the distribution of sales at the country level throughout the world. Behind these graphical icons there are various mathematical functions aggregating numerous data points to their highest level of meaningful figures. By clicking on these graphical icons, the consumer of this information can drill down to more granular levels of information and data.

Dashboards are used in a wide variety of businesses for a wide variety of reasons. For instance, in Application Case 3.7, you will find the summary of a successful imple- mentation of information dashboards by the Dallas Cowboys football team.

FIGURE 3.27 A Sample Executive Dashboard. Source: A Sample Executive Dashboard from Dundas Data Visualization, Inc., www.dundas.com, reprinted with permission.

184 Part I • Introduction to Analytics and AI

Founded in 1960, the Dallas Cowboys are a pro- fessional American football team headquartered in Irving, Texas. The team has a large national follow- ing, which is perhaps best represented by their NFL record for number of consecutive games at sold-out stadiums.

The Challenge

Bill Priakos, chief operating officer (COO) of the Dallas Cowboys Merchandising Division, and his team needed more visibility into their data so they could run it more profitably. Microsoft was selected as the baseline platform for this upgrade as well as a number of other sales, logistics, and e-commerce (per MW) applications. The Cowboys expected that this new information architecture would provide the needed analytics and reporting. Unfortunately, this was not the case, and the search began for a robust dashboarding, analytics, and reporting tool to fill this gap.

The Solution and Results

Tableau and Teknion together provided real-time reporting and dashboard capabilities that exceeded the Cowboys’ requirements. Systematically and methodically, the Teknion team worked side by side with data owners and data users within the Dallas Cowboys to deliver all required functionality on time and under budget. “Early in the process, we were able to get a clear understanding of what it would take to run a more profitable operation for the Cowboys,” said Teknion Vice President Bill Luisi. “This process step is a key step in Teknion’s approach with any client, and it always pays huge

dividends as the implementation plan progresses.” Added Luisi, “Of course, Tableau worked very closely with us and the Cowboys during the entire project. Together, we made sure that the Cowboys could achieve their reporting and analytical goals in record time.”

Now, for the first time, the Dallas Cowboys are able to monitor their complete merchandising activi- ties from manufacture to end customer and not only see what is happening across the life cycle but also drill down even further into why it is happening.

Today, this BI solution is used to report and analyze the business activities of the Merchandising Division, which is responsible for all of the Dallas Cowboys’ brand sales. Industry estimates say that the Cowboys generate 20 percent of all NFL mer- chandise sales, which reflects the fact that they are the most recognized sports franchise in the world.

According to Eric Lai, a ComputerWorld reporter, Tony Romo and the rest of the Dallas Cowboys may have been only average on the foot- ball field in the last few years, but off the field, especially in the merchandising arena, they remain America’s team.

Questions for Case 3.7

1. How did the Dallas Cowboys use information visualization?

2. What were the challenge, the proposed solution, and the obtained results?

Sources: Lai, E. (2009, October 8). “BI Visualization Tool Helps Dallas Cowboys Sell More Tony Romo Jerseys,” ComputerWorld. Tableau case study. tableau.com/blog/computerworld-dallas- cowboys-business-intelligence (accessed July 2018).

Application Case 3.7 Dallas Cowboys Score Big with Tableau and Teknion

Dashboard Design

Dashboards are not a new concept. Their roots can be traced at least to the executive information system of the 1980s. Today, dashboards are ubiquitous. For example, a few years back, Forrester Research estimated that over 40 percent of the largest 2,000 com- panies in the world used the technology (Ante & McGregor, 2006). Since then, one can safely assume that this number has gone up quite significantly. In fact, today it would be rather unusual to see a large company using a BI system that does not employ some sort of performance dashboards. The Dashboard Spy Web site (dashboardspy.com/about) provides further evidence of their ubiquity. The site contains descriptions and screenshots

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 185

of thousands of BI dashboards, scorecards, and BI interfaces used by businesses of all sizes and industries, nonprofits, and government agencies.

According to Eckerson (2006), a well-known expert on BI in general and dash- boards in particular, the most distinctive feature of a dashboard is its three layers of information:

1. Monitoring: Graphical, abstracted data to monitor key performance metrics. 2. Analysis: Summarized dimensional data to analyze the root cause of problems. 3. Management: Detailed operational data that identify what actions to take to re-

solve a problem.

Because of these layers, dashboards pack a large amount of information into a sin- gle screen. According to Few (2005), “The fundamental challenge of dashboard design is to display all the required information on a single screen, clearly and without distraction, in a manner that can be assimilated quickly.” To speed assimilation of the numbers, they need to be placed in context. This can be done by comparing the numbers of interest to other baseline or target numbers, by indicating whether the numbers are good or bad, by denoting whether a trend is better or worse, and by using specialized display widgets or components to set the comparative and evaluative context. Some of the common comparisons that are typically made in BI systems include comparisons against past val- ues, forecasted values, targeted values, benchmark or average values, multiple instances of the same measure, and the values of other measures (e.g., revenues versus costs).

Even with comparative measures, it is important to specifically point out whether a particular number is good or bad and whether it is trending in the right direction. Without these types of evaluative designations, it can be time consuming to determine the status of a particular number or result. Typically, either specialized visual objects (e.g., traffic lights, dials, and gauges) or visual attributes (e.g., color coding) are used to set the evalu- ative context. An interactive dashboard-driven reporting data exploration solution built by an energy company is featured in Application Case 3.8.

Energy markets all around the world are going through a significant change and transformation, creating ample opportunities along with significant challenges. As is the case in any industry, oppor- tunities are attracting more players in the market- place, increasing the competition, and reducing the tolerances for less-than-optimal business decision making. Success requires creating and disseminat- ing accurate and timely information to whomever whenever it is needed. For instance, if you need to easily track marketing budgets, balance employee workloads, and target customers with tailored mar- keting messages, you would need three different reporting solutions. Electrabel GDF SUEZ is doing all of that for its marketing and sales business unit with SAS'Analytics Visual Analytics platform.

The one-solution approach is a great time-saver for marketing professionals in an industry that is undergoing tremendous change. “It is a huge chal- lenge to stabilize our market position in the energy market. That includes volume, prices, and margins for both retail and business customers,” notes Danny Noppe, manager of Reporting Architecture and Development in the Electrabel Marketing and Sales business unit. The company is the largest supplier of electricity in Belgium and the largest producer of elec- tricity for Belgium and the Netherlands. Noppe says it is critical that Electrabel increase the efficiency of its customer communications as it explores new digital channels and develops new energy-related services.

“The better we know the customer, the bet- ter our likelihood of success,” he says. “That is why

Application Case 3.8 Visual Analytics Helps Energy Supplier Make Better Connections

(Continued )

186 Part I • Introduction to Analytics and AI

What to Look for in a Dashboard

Although performance dashboards and other information visualization frameworks differ, they all share some common design characteristics. First, they all fit within the larger BI and/or performance measurement system. This means that their underlying architecture is the BI or performance management architecture of the larger system. Second, all well- designed dashboards and other information visualizations possess the following charac- teristics (Novell, 2009):

we combine information from various sources— phone traffic with the customer, online questions, text messages, and mail campaigns. This enhanced knowledge of our customer and prospect base will be an additional advantage within our competitive market.”

One Version of the Truth

Electrabel was using various platforms and tools for reporting purposes. This sometimes led to ambigu- ity in the reported figures. The utility also had per- formance issues in processing large data volumes. SAS Visual Analytics with in-memory technology removes the ambiguity and the performance issues. “We have the autonomy and flexibility to respond to the need for customer insight and data visualization internally,” Noppe says. “After all, fast reporting is an essential requirement for action-oriented depart- ments such as sales and marketing.”

Working More Efficiently at a Lower Cost

SAS Visual Analytics automates the process of updating information in reports. Instead of building a report that is out of date by the time it is com- pleted, the data are refreshed for all the reports once a week and is available on dashboards. In deploying the solution, Electrabel chose a phased approach, starting with simple reports and moving on to more complex ones. The first report took a few weeks to build, and the rest came quickly. The successes include the following:

• Reduction of data preparation from two days to only two hours.

• Clear graphic insight into the invoicing and composition of invoices for business-to-busi- ness (B2B) customers.

• A workload management report by the op- erational teams. Managers can evaluate team

workloads on a weekly or long-term basis and can make adjustments accordingly.

“We have significantly improved our effi- ciency and can deliver quality data and reports more frequently, and at a significantly lower cost,” says Noppe. And if the company needs to combine data from multiple sources, the process is equally easy. “Building visual reports, based on these data marts, can be achieved in a few days, or even a few hours.”

Noppe says the company plans to continue broadening its insight into the digital behavior of its customers, combining data from Web analytics, e-mail, and social media with data from back-end systems. “Eventually, we want to replace all labor- intensive reporting with SAS Visual Analytics,” he says, adding that the flexibility of SAS Visual Analytics is critical for his department. “This will give us more time to tackle other challenges. We also want to make this tool available on our mobile devices. This will allow our account managers to use up-to-date, insightful, and adaptable reports when visiting cus- tomers. We’ve got a future-oriented reporting plat- form to do all we need.”

Questions for Case 3.8

1. Why do you think energy supply companies are among the prime users of information visualiza- tion tools?

2. How did Electrabel use information visualization for the single version of the truth?

3. What were their challenges, the proposed solu- tion, and the obtained results?

Source: SAS Customer Story, “Visual Analytics Helps Energy Supplier Make Better Connections.” http://www.sas.com/ en_us/customers/electrabel-be.html (accessed July 2018). Copyright © 2018 SAS Institute Inc., Cary, NC, United States. Reprinted with permission. All rights reserved.

Application Case 3.8 (Continued)

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 187

• They use visual components (e.g., charts, performance bars, sparklines, gauges, meters, stoplights) to highlight, at a glance, the data and exceptions that require action.

• They are transparent to the user, meaning that they require minimal training and are extremely easy to use.

• They combine data from a variety of systems into a single, summarized, unified view of the business.

• They enable drill-down or drill-through to underlying data sources or reports, pro- viding more detail about the underlying comparative and evaluative context.

• They present a dynamic, real-world view with timely data refreshes, enabling the end user to stay up-to-date with any recent changes in the business.

• They require little, if any, customized coding to implement, deploy, and maintain.

Best Practices in Dashboard Design

The real estate saying “location, location, location” makes it obvious that the most im- portant attribute for a piece of real estate property is where it is located. For dashboards, it is “data, data, data.” Often overlooked, data are considered one of the most important things to focus on in designing dashboards (Carotenuto, 2007). Even if a dashboard’s ap- pearance looks professional, is aesthetically pleasing, and includes graphs and tables cre- ated according to accepted visual design standards, it is also important to ask about the data: Are they reliable? Are they timely? Are any data missing? Are they consistent across all dashboards? Here are some of the experience-driven best practices in dashboard de- sign (Radha, 2008).

Benchmark Key Performance Indicators with Industry Standards

Many customers, at some point in time, want to know if the metrics they are measuring are the right metrics to monitor. Sometimes customers have found that the metrics they are tracking are not the right ones to track. Doing a gap assessment with industry bench- marks aligns you with industry best practices.

Wrap the Dashboard Metrics with Contextual Metadata

Often when a report or a visual dashboard/scorecard is presented to business users, questions remain unanswered. The following are some examples:

• Where did you source these data? • While loading the data warehouse, what percentage of the data was rejected/

encountered data quality problems? • Is the dashboard presenting “fresh” information or “stale” information? • When was the data warehouse last refreshed? • When is it going to be refreshed next? • Were any high-value transactions that would skew the overall trends rejected as a

part of the loading process?

Validate the Dashboard Design by a Usability Specialist

In most dashboard environments, the dashboard is designed by a tool specialist without giving consideration to usability principles. Even though it is a well-engineered data warehouse that can perform well, many business users do not use the dashboard because it is perceived as not being user friendly, leading to poor adoption of the infrastructure and change management issues. Up-front validation of the dashboard design by a usabil- ity specialist can mitigate this risk.

188 Part I • Introduction to Analytics and AI

Prioritize and Rank Alerts/Exceptions Streamed to the Dashboard

Because there are tons of raw data, having a mechanism by which important exceptions/ behaviors are proactively pushed to the information consumers is important. A business rule can be codified, which detects the alert pattern of interest. It can be coded into a pro- gram, using database-stored procedures, which can crawl through the fact tables and detect patterns that need immediate attention. This way, information finds the business user as opposed to the business user polling the fact tables for the occurrence of critical patterns.

Enrich the Dashboard with Business-User Comments

When the same dashboard information is presented to multiple business users, a small text box can be provided that can capture the comments from an end user’s perspective. This can often be tagged to the dashboard to put the information in context, adding per- spective to the structured KPIs being rendered.

Present Information in Three Different Levels

Information can be presented in three layers depending on the granularity of the infor- mation: the visual dashboard level, the static report level, and the self-service cube level. When a user navigates the dashboard, a simple set of 8 to 12 KPIs can be presented, which would give a sense of what is going well and what is not.

Pick the Right Visual Construct Using Dashboard Design Principles

In presenting information in a dashboard, some information is presented best with bar charts and some with time-series line graphs, and when presenting correlations, a scatter plot is useful. Sometimes merely rendering it as simple tables is effective. Once the dashboard design principles are explicitly documented, all the developers working on the front end can adhere to the same principles while rendering the reports and dashboard.

Provide for Guided Analytics

In a typical organization, business users can be at various levels of analytical maturity. The capability of the dashboard can be used to guide the “average” business user to access the same navigational path as that of an analytically savvy business user.

u SECTION 3.11 REVIEW QUESTIONS

1. What is an information dashboard? Why is it so popular? 2. What are the graphical widgets commonly used in dashboards? Why? 3. List and describe the three layers of information portrayed on dashboards. 4. What are the common characteristics of dashboards and other information visuals? 5. What are the best practices in dashboard design?

Chapter Highlights

• Data have become one of the most valuable assets of today’s organizations.

• Data are the main ingredient for any BI, data science, and business analytics initiative.

• Although its value proposition is undeniable, to live up its promise, the data must comply with some basic usability and quality metrics.

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 189

• The term data (datum in singular form) refers to a collection of facts usually obtained as the result of experiments, observations, transactions, or experiences.

• At the highest level of abstraction, data can be classified as structured and unstructured.

• Data in original/raw form are not usually ready to be useful in analytics tasks.

• Data preprocessing is a tedious, time-demanding, yet crucial task in business analytics.

• Statistics is a collection of mathematical tech- niques to characterize and interpret data.

• Statistical methods can be classified as either de- scriptive or inferential.

• Statistics in general, as well as descriptive statistics in particular, is a critical part of BI and business analytics.

• Descriptive statistics methods can be used to mea- sure central tendency, dispersion, or the shape of a given data set.

• Regression, especially linear regression, is per- haps the most widely known and used analytics technique in statistics.

• Linear regression and logistic regression are the two major regression types in statistics.

• Logistics regression is a probability-based classifi- cation algorithm.

• Time series is a sequence of data points of a vari- able, measured and recorded at successive points in time spaced at uniform time intervals.

• A report is any communication artifact prepared with the specific intention of conveying informa- tion in a presentable form.

• A business report is a written document that con- tains information regarding business matters.

• The key to any successful business report is clar- ity, brevity, completeness, and correctness.

• Data visualization is the use of visual representations to explore, make sense of, and communicate data.

• Perhaps the most notable information graphic of the past was developed by Charles J. Minard, who graphically portrayed the losses suffered by Napoleon’s army in the Russian campaign of 1812.

• Basic chart types include line, bar, and pie chart. • Specialized charts are often derived from the

basic charts as exceptional cases. • Data visualization techniques and tools make the

users of business analytics and BI systems better information consumers.

• Visual analytics is the combination of visualiza- tion and predictive analytics.

• Increasing demand for visual analytics coupled with fast-growing data volumes led to exponen- tial growth in highly efficient visualization sys- tems investment.

• Dashboards provide visual displays of important information that is consolidated and arranged on a single screen so that information can be di- gested at a single glance and easily drilled in and further explored.

Key terms

analytics ready arithmetic mean box-and-whiskers plot box plot bubble chart business report categorical data centrality correlation dashboards data preprocessing data quality data security data taxonomy data visualization datum descriptive statistics dimensional reduction

dispersion high-performance computing histogram inferential statistics key performance indicator

(KPI) knowledge kurtosis learning linear regression logistic regression mean absolute deviation median mode nominal data online analytics processing

(OLAP) ordinal data

ordinary least squares (OLS) pie chart quartile range ratio data regression report scatter plot skewness standard deviation statistics storytelling structured data time-series forecasting unstructured data variable selection variance visual analytics

190 Part I • Introduction to Analytics and AI

Questions for Discussion

1. How do you describe the importance of data in analyt- ics? Can we think of analytics without data? Explain.

2. Considering the new and broad definition of business analytics, what are the main inputs and outputs to the analytics continuum?

3. Where do the data for business analytics come from? What are the sources and the nature of those incoming data?

4. What are the most common metrics that make for analytics-ready data?

5. What are the main categories of data? What types of data can we use for BI and analytics?

6. Can we use the same data representation for all analytics models (i.e., do different analytics models require different data representation schema)? Why, or why not?

7. Why are the original/raw data not readily usable by ana- lytics tasks?

8. What are the main data preprocessing steps? List and explain their importance in analytics.

9. What does it mean to clean/scrub the data? What activi- ties are performed in this phase?

10. Data reduction can be applied to rows (sampling) and/ or columns (variable selection). Which is more chal- lenging? Explain.

11. What is the relationship between statistics and business analytics? (Consider the placement of statistics in a busi- ness analytics taxonomy.)

12. What are the main differences between descriptive and inferential statistics?

13. What is a box-and-whiskers plot? What types of statisti- cal information does it represent?

14. What are the two most commonly used shape character- istics to describe a data distribution?

15. List and briefly define the central tendency measures of descriptive statistics.

16. What are the commonalities and differences between regression and correlation?

17. List and describe the main steps to follow in developing a linear regression model.

18. What are the most commonly pronounced assumptions for linear regression? What is crucial to the regression models against these assumptions?

19. What are the commonalities and differences between linear regression and logistic regression?

20. What is time series? What are the main forecasting tech- niques for time-series data?

21. What is a business report? Why is it needed? 22. What are the best practices in business reporting? How

can we make our reports stand out? 23. Describe the cyclic process of management, and com-

ment on the role of business reports. 24. List and describe the three major categories of business

reports. 25. Why has information visualization become a center-

piece in BI and business analytics? Is there a difference between information visualization and visual analytics?

26. What are the main types of charts/graphs? Why are there so many of them?

27. How do you determine the right chart for a job? Explain and defend your reasoning.

28. What is the difference between information visualiza- tion and visual analytics?

29. Why should storytelling be a part of your reporting and data visualization?

30. What is an information dashboard? What does it present? 31. What are the best practices in designing highly informa-

tive dashboards? 32. Do you think information/performance dashboards are

here to stay? Or are they about to be outdated? What do you think will be the next big wave in BI and business analytics in terms of data/information visualization?

Exercises

Teradata University and Other Hands-on Exercises

1. Download the “Voting Behavior” data and the brief data description from the book’s Web site. This is a data set manually compiled from counties all around the United States. The data are partially processed, that is, some derived variables have been created. Your task is to thoroughly preprocess the data by identifying the error and anomalies and proposing remedies and solutions. At the end, you should have an analytics-ready version of these data. Once the pre- processing is completed, pull these data into Tableau (or into some other data visualization software tool)

to extract useful visual information from it. To do so, conceptualize relevant questions and hypotheses (come up with at least three of them) and create proper visualizations that address those questions of “tests” of those hypotheses.

2. Download Tableau (at tableau.com, following aca- demic free software download instructions on the site). Using the Visualization_MFG_Sample data set (available as an Excel file on this book’s Web site), answer the fol- lowing questions: a. What is the relationship between gross box office

revenue and other movie-related parameters given in the data set?

Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 191

b. How does this relationship vary across different years? Prepare a professional-looking written report that is enhanced with screenshots of your graphic findings.

3. Go to teradatauniversitynetwork.com. Look for an article that deals with the nature of data, management of data, and/or governance of data as it relates to BI and analytics, and critically analyze the content of the article.

4. Go to UCI data repository (archive.ics.uci.edu/ml/ datasets.html) and identify a large data set that con- tains both numeric and nominal values. Using Microsoft Excel or any other statistical software: a. Calculate and interpret central tendency measures

for each and every variable. b. Calculate and interpret the dispersion/spread mea-

sures for each and every variable. 5. Go to UCI data repository (archive.ics.uci.edu/ml/

datasets.html) and identify two data sets, one for estimation/regression and one for classification. Using Microsoft Excel or any other statistical software: a. Develop and interpret a linear regression model. b. Develop and interpret a logistic regression model.

6. Go to KDnuggest.com and become familiar with the range of analytics resources available on this portal. Then identify an article, a white paper, or an interview script that deals with the nature of data, management of data, and/or governance of data as they relate to BI and business analytics, and critically analyze the content of the article.

7. Go to Stephen Few’s blog, “The Perceptual Edge” (perceptualedge.com). Go to the section of “Examples.” In this section, he provides critiques of various dashboard examples. Read a handful of these examples. Now go to dundas.com. Select the “Gallery” section of the site. Once there, click the “Digital Dashboard” selection. You will be shown a variety of different dashboard demos. Run a couple of them. a. What types of information and metrics are shown on

the demos? What types of actions can you take? b. Using some of the basic concepts from Few’s cri-

tiques, describe some of the good design points and bad design points of the demos.

8. Download an information visualization tool, such as Tableau, QlikView, or Spotfire. If your school does not have an educational agreement with these companies, a trial version would be sufficient for this exercise. Use your own data (if you have any) or use one of the data sets that comes with the tool (such tools usually have one or more data sets for demonstration purposes). Study the data, come up with several business prob- lems, and use data visualization to analyze, visualize, and potentially solve those problems.

9. Go to teradatauniversitynetwork.com. Find the “Tableau Software Project.” Read the description, exe- cute the tasks, and answer the questions.

10. Go to teradatauniversitynetwork.com. Find the assignments for SAS Visual Analytics. Using the infor- mation and step-by-step instructions provided in the

assignment, execute the analysis on the SAS Visual Analytics tool (which is a Web-enabled system that does not require any local installation). Answer the questions posed in the assignment.

11. Find at least two articles (one journal article and one white paper) that talk about storytelling, especially within the context of analytics (i.e., data-driven storytell- ing). Read and critically analyze the article and paper, and write a report to reflect your understanding and opinions about the importance of storytelling in BI and business analytics.

12. Go to data.gov—a U.S. government–sponsored data portal that has a very large number of data sets on a wide variety of topics ranging from healthcare to edu- cation, climate to public safety. Pick a topic that you are most passionate about. Go through the topic- specific information and explanation provided on the site. Explore the possibilities of downloading the data, and use your favorite data visualization tool to create your own meaningful information and visualizations.

Team Assignments and Role-Playing Projects

1. Analytics starts with data. Identifying, accessing, obtain- ing, and processing of relevant data is the most essential task in any analytics study. As a team, you are tasked to find a large enough real-world data (either from your own organization, which is the most preferred, or from the Internet that can start with a simple search, or from the data links posted on KDnuggets.com), one that has tens of thousands of rows and more than 20 vari- ables to go through, and document a thorough data preprocessing project. In your processing of the data, identify anomalies and discrepancies using descriptive statistics methods and measures, and make the data an- alytics ready. List and justify your preprocessing steps and decisions in a comprehensive report.

2. Go to a well-known information dashboard provider Web site (dundas.com, idashboards.com, enterprise- dashboard.com). These sites provide a number of exam- ples of executive dashboards. As a team, select a particular industry (e.g., healthcare, banking, airline). Locate a hand- ful of example dashboards for that industry. Describe the types of metrics found on the dashboards. What types of displays are used to provide the information? Using what you know about dashboard design, provide a paper pro- totype of a dashboard for this information.

3. Go to teradatauniversitynetwork.com. From there, go to University of Arkansas data sources. Choose one of the large data sets, and download a large number of records (this could require you to write an SQL state- ment that creates the variables that you want to include in the data set). Come up with at least 10 questions that can be addressed with information visualization. Using your favorite data visualization tool (e.g., Tableau), ana- lyze the data, and prepare a detailed report that includes screenshots and other visuals.

192 Part I • Introduction to Analytics and AI

References

Abela, A. (2008). Advanced Presentations by Design: Creating Communication That Drives Action. New York, NY: Wiley.

Annas, G. (2003). “HIPAA Regulations—A New Era of Medical-Record Privacy?” New England Journal of Medicine, 348(15), 1486–1490.

Ante, S., & J. McGregor. (2006). “Giving the Boss the Big Pic- ture: A Dashboard Pulls Up Everything the CEO Needs to Run the Show.” Business Week, 43–51.

Carotenuto, D. (2007). “Business Intelligence Best Practices for Dashboard Design.” WebFOCUS. www. datawarehouse. inf.br/papers/information_builders_dashboard_ best_practices.pdf (accessed August 2016).

Dell Customer Case Study. “Medical Device Company Ensures Product Quality While Saving Hundreds of Thousands of Dollars.” https://software.dell.com/ documents/instrumentation-laboratory-medical- device- companyensures-product-quality-while- saving-hundreds-of thousands-of-dollars-case- study-80048.pdf (accessed August 2016).

Delen, D. (2010). “A Comparative Analysis of Machine Learn- ing Techniques for Student Retention Management.” Deci- sion Support Systems, 49(4), 498–506.

Delen, D. (2011). “Predicting Student Attrition with Data Min- ing Methods.” Journal of College Student Retention 13(1), 17–35.

Delen, D. (2015). Real-World Data Mining: Applied Business Analytics and Decision Making. Upper Saddle River, NJ: Financial Times Press (A Pearson Company).

Delen, D., D. Cogdell, & N. Kasap. (2012). “A Comparative Analysis of Data Mining Methods in Predicting NCAA Bowl Outcomes.” International Journal of Forecasting, 28, 543–552.

Eckerson, W. (2006). Performance Dashboards. New York: Wiley.

Few, S. (2005, Winter). “Dashboard Design: Beyond Meters, Gauges, and Traffic Lights.” Business Intelligence Journal, 10(1).

Few, S. (2007). “Data Visualization: Past, Present and Future.” Perceptualedge.com/articles/Whitepapers/Data_ Visualization.pdf (accessed July 2016).

Fink, E., & S. J. Moore. (2012). “Five Best Practices for Telling Great Stories with Data.” Tableau Software, Inc. www. tableau.com/whitepapers/telling-data-stories (accessed May 2016).

Freeman, K., & R. M. Brewer. (2016). “The Politics of Ameri- can College Football.” Journal of Applied Business and Economics, 18(2), 97–101.

Gartner Magic Quadrant. (2016, February 4). gartner.com (accessed August 2016).

Grimes, S. (2009a, May 2). “Seeing Connections: Visualizations Makes Sense of Data. Intelligent Enterprise.” i.cmpnet. com/intelligententer pr ise/next-era-business- intelligence/Intelligent_Enterprise_Next_Era_BI_ Visualization.pdf (accessed January 2010).

Grimes, S. (2009b). Text “Analytics 2009: User Perspectives on Solutions and Providers.” Alta Plana. altaplana.com/ TextAnalyticsPerspectives2009.pdf (accessed July, 2016).

Hardin, M. Hom, R. Perez, & Williams L. (2012). “Which Chart or Graph Is Right for You?” Tableau Software. http:// www.tableau.com/sites/default/files/media/which_ chart_v6_final_0.pdf (accessed August 2016).

Hernández, M., & S. J. Stolfo. (1998, January). “Real-World Data Is Dirty: Data Cleansing and the Merge/Purge Problem.” Data Mining and Knowledge Discovery, 2(1), 9–37.

Hill, G. (2016). “A Guide to Enterprise Reporting.” Ghill. customer.netspace.net.au/reporting/definition.html (accessed July 2016).

Kim, W., B. J. Choi, E. K. Hong, S. K. Kim, & D. Lee. (2003). “A Taxonomy of Dirty Data.” Data Mining and Knowledge Discovery, 7(1), 81–99.

Kock, N. F., R. J. McQueen, & J. L. Corner. (1997). “The Na- ture of Data, Information and Knowledge Exchanges in Business Processes: Implications for Process Improvement and Organizational Learning.” The Learning Organization, 4(2), 70–80.

Kotsiantis, S., D. Kanellopoulos, & P. E. Pintelas. (2006). “Data Preprocessing for Supervised Leaning.” International Jour- nal of Computer Science, 1(2), 111–117.

Lai, E. (2009, October 8). “BI Visualization Tool Helps Dallas Cowboys Sell More Tony Romo Jerseys.” Com- puterWorld.

Quinn, C. (2016). “Data-Driven Marketing at SiriusXM,” Teradata Articles & News. http://bigdata.teradata. com/US/Articles-News/Data-Driven-Marketing-At- SiriusXM/ (accessed August 2016); “SiriusXM Attracts and Engages a New Generation of Radio Consumers.” http:// assets. teradata.com/resourceCenter/downloads/ CaseStudies/EB8597.pdf?processed=1 (accessed August 2018).

Novell. (2009, April). “Executive Dashboards Elements of Success.” Novell white paper. www.novell.com/ docrep/documents/3rkw3et fc3/Executive%20 Dashboards_Elements_of_Success_White_Paper_ en.pdf (accessed June 2016).

Radha, R. (2008). “Eight Best Practices in Dashboard Design.” Information Management. www.information- management.com/news/columns/-10001129-1.html (accessed July 2016).

SAS. (2014). “Data Visualization Techniques: From Basics to Big Data.” http://www.sas.com/content/dam/ SAS/en_us/doc/whitepaper1/data-visualization- techniques-106006.pdf (accessed July 2016).

Thammasiri, D., D. Delen, P. Meesad, & N. Kasap. (2014). “A Critical Assessment of Imbalanced Class Distribution Prob- lem: The Case of Predicting Freshmen Student Attrition.” Expert Systems with Applications, 41(2), 321–330.

193

P A R T

Predictive Analytics/ Machine Learning

II

194

LEARNING OBJECTIVES

Data Mining Process, Methods, and Algorithms

4 C H A P T E R

■■ Define data mining as an enabling technology for business analytics

■■ Understand the objectives and benefits of data mining

■■ Become familiar with the wide range of applications of data mining

■■ Learn the standardized data mining processes

■■ Learn different methods and algorithms of data mining

■■ Build awareness of existing data mining software tools

■■ Understand the privacy issues, pitfalls, and myths of data mining

G enerally speaking, data mining is a way to develop intelligence (i.e., actionable information or knowledge) from data that an organization collects, organizes, and stores. A wide range of data mining techniques is being used by organiza- tions to gain a better understanding of their customers and their operations and to solve complex organizational problems. In this chapter, we study data mining as an enabling technology for business analytics and predictive analytics; learn about the standard pro- cesses of conducting data mining projects; understand and build expertise in the use of major data mining techniques; develop awareness of the existing software tools; and explore privacy issues, common myths, and pitfalls that are often associated with data mining.

4.1 Opening Vignette: Miami-Dade Police Department Is Using Predictive Analytics to Foresee and Fight Crime 195

4.2 Data Mining Concepts 198 4.3 Data Mining Applications 208 4.4 Data Mining Process 211 4.5 Data Mining Methods 220 4.6 Data Mining Software Tools 236 4.7 Data Mining Privacy Issues, Myths, and Blunders 242

Chapter 4 • Data Mining Process, Methods, and Algorithms 195

4.1 OPENING VIGNETTE: Miami-Dade Police Department Is Using Predictive Analytics to Foresee and Fight Crime

Predictive analytics and data mining have become an integral part of many law enforce- ment agencies including the Miami-Dade Police Department whose mission is not only to protect the safety of Florida’s largest county, with 2.5 million citizens (making it the seventh largest in the United States), but also to provide a safe and inviting climate for the millions of tourists who come from around the world to enjoy the county’s natural beauty, warm climate, and stunning beaches. With tourists spending nearly US$20 billion every year and generating nearly one-third of Florida’s sales taxes, it is hard to overstate the importance of tourism to the region’s economy. So although few of the county’s police officers would likely list economic development in their job description, nearly all grasp the vital link between safe streets and the region’s tourist-driven prosperity.

That connection is paramount for Lieutenant Arnold Palmer, currently supervising the Robbery Investigations Section and a former supervisor of the department’s Robbery Intervention Detail. This specialized team of detectives is focused on intensely policing the county’s robbery hot spots and worst repeat offenders. He and the team occupy mod- est offices on the second floor of a modern-looking concrete building set back from a palm-lined street on the western edge of Miami. In his 10 years in the unit and 23 in total on the force, Palmer has seen many changes—not just in policing practices like the way his team used to mark street crime hot spots with colored pushpins on a map.

POLICING WITH LESS

Palmer and the team have also seen the impact of a growing population, shifting demo- graphics, and a changing economy on the streets they patrol. Like any good police force officers, they have continually adapted their methods and practices to meet a policing challenge that has grown in scope and complexity. But like nearly all branches of the county’s government, intensifying budget pressures have placed the department in a squeeze between rising demands and shrinking resources.

Palmer, who sees detectives as front-line fighters against a rising tide of street crime and the looming prospect of ever-tightening resources, put it this way: “Our basic chal- lenge was how to cut street crime even as tighter resources have reduced the number of cops on the street.” Over the years, the team has been open to trying new tools, the most notable of which is a program called “analysis-driven enforcement,” which used crime history data as the basis for positioning teams of detectives. “We’ve evolved a lot since then in our ability to predict where robberies are likely to occur, both through the use of analysis and our own collective experience.”

NEW THINKING ON COLD CASES

The more confounding challenge for Palmer and his team of investigators, one shared with the police of all major urban areas, is in closing the hardest cases whose leads, wit- nesses, video—any facts or evidence that can help solve a case—are lacking. This is not surprising, explains Palmer, because “the standard practices we used to generate leads, like talking to informants or to the community or to patrol officers, haven’t changed much, if at all. That kind of an approach works okay, but it relies a lot on the experience our detectives carry in their head. When the detectives retire or move on, that experience goes with them.”

Palmer’s conundrum was that turnover resulting from the retirement of many of his most experienced detectives was on an upward trend. True, he saw the infusion of young blood as an inherently good thing, especially given this group’s increased comfort with

196 Part II • Predictive Analytics/Machine Learning

the new types of information—from e-mails, social media, and traffic cameras, to name a few—to which his team had access. But as Palmer recounts, the problem came when the handful of new detectives coming into the unit turned to look for guidance from the senior officers “and it’s just not there. We knew at that point we needed a different way to fill the experience gap going forward.”

His ad hoc efforts to come up with a solution led to blue-sky speculation. What if new detectives on the squad could pose the same questions to a computer database as they would to a veteran detective? That speculation planted a seed in Palmer’s mind that wouldn’t go away.

THE BIG PICTURE STARTS SMALL

What was taking shape within the robbery unit demonstrated how big ideas can come from small places. But more importantly, it showed that for these ideas to reach frui- tion, the “right” conditions need to be in alignment at the right time. On a leadership level, this means a driving figure in the organization who knows what it takes to nurture top-down support as well as crucial bottom-up buy-in from the ranks while keeping the department’s information technology (IT) personnel on the same page. That person was Palmer. At the organizational level, the robbery unit served as a particularly good launching point for lead modeling because of the prevalence of repeat offenders among perpetrators. Ultimately, the department’s ability to unleash the broader transformative potential of lead modeling would hinge in large part on the team’s ability to deliver results on a small scale.

When early tests and demos proved encouraging—with the model yielding accu- rate results when the details of solved cases were fed into it—the team started gaining attention. The initiative received a critical boost when the robbery bureau’s unit major and captain voiced their support for the direction of the project, telling Palmer that “if you can make this work, run with it.” But more important than the encouragement, Palmer explains, was their willingness to advocate for the project among the department’s higher-ups. “I can’t get it off the ground if the brass doesn’t buy in,” says Palmer. “So their support was crucial.”

SUCCESS BRINGS CREDIBILITY

Having been appointed the official liaison between IT and the robbery unit, Palmer set out to strengthen the case for the lead modeling tool—now officially called Blue PALMS (for Predictive Analytics Lead Modeling Software)—by building a series of successes. His constituency was not only the department brass but also the detectives whose support would be critical to its successful adoption as a robbery-solving tool. In his attempts to introduce Blue PALMS, resistance was predictably stronger among veteran detectives who saw no reason to give up their long-standing practices. Palmer knew that dictates or coercion would not win their hearts and minds. He would need to build a beachhead of credibility.

Palmer found that opportunity in one of his best and most experienced detectives. Early in a robbery investigation, the detective indicated to Palmer that he had a strong hunch who the perpetrator was and wanted, in essence, to test the Blue PALMS system. At the detective’s request, the department analyst fed key details of the crime into the sys- tem, including the modus operandi (MO). The system’s statistical models compared these details to a database of historical data, looking for important correlations and similarities in the crime’s signature. The report that came out of the process included a list of 20 suspects ranked in order of match strength, or likelihood. When the analyst handed the detective the report, his “hunch” suspect was listed in the top five. Soon after his arrest, the suspect confessed, and Palmer had gained a solid convert.

Chapter 4 • Data Mining Process, Methods, and Algorithms 197

Although it was a useful exercise, Palmer realized that the true test was not in con- firming hunches but in breaking cases that had come to a dead end. Such was the situ- ation in a carjacking that had, in Palmer’s words, “no witnesses, no video, and no crime scene—nothing to go on.” When the senior detective on the stalled case went on leave after three months, the junior detective to whom it was assigned requested a Blue PALMS report. Shown photographs of the top people on the suspect list, the victim made a posi- tive identification of the suspect leading to the successful conclusion of the case. That suspect was number one on the list.

JUST THE FACTS

The success that Blue PALMS continues to build has been a major factor in Palmer’s get- ting his detectives on board. But if there is a part of his message that resonates even more with his detectives, it is the fact that Blue PALMS is designed not to change the basics of policing practices but to enhance them by giving them a second chance of cracking the case. “Police work is at the core about human relations—about talking to witnesses, to victims, to the community—and we’re not out to change that,” says Palmer. “Our aim is to give investigators factual insights from information we already have that might make a difference, so even if we’re successful 5 percent of the time, we’re going to take a lot of offenders off the street.”

The growing list of cold cases solved has helped Palmer in his efforts to reinforce the merits of Blue PALMS. But, in showing where his loyalty lies, he sees the detectives who have closed these cold cases—not the program—as most deserving of the spotlight, and that approach has gone over well. At his chief’s request, Palmer is beginning to use his liai- son role as a platform for reaching out to other areas in the Miami-Dade Police Department.

SAFER STREETS FOR A SMARTER CITY

When he speaks of the impact of tourism, a thread that runs through Miami-Dade’s Smarter Cities vision, Palmer sees Blue PALMS as an important tool to protect one of the county’s greatest assets. “The threat to tourism posed by rising street crime was a big rea- son the unit was established,” says Palmer. “The fact that we’re able to use analytics and intelligence to help us close more cases and keep more criminals off the street is good news for our citizens and our tourist industry.”

u QUESTIONS FOR THE OPENING VIGNETTE

1. Why do law enforcement agencies and departments like the Miami-Dade Police Department embrace advanced analytics and data mining?

2. What are the top challenges for law enforcement agencies and departments like the Miami-Dade Police Department? Can you think of other challenges (not mentioned in this case) that can benefit from data mining?

3. What are the sources of data that law enforcement agencies and departments like the Miami-Dade Police Department use for their predictive modeling and data mining projects?

4. What type of analytics do law enforcement agencies and departments like the Miami-Dade Police Department use to fight crime?

5. What does “the big picture starts small” mean in this case? Explain.

WHAT WE CAN LEARN FROM THIS VIGNETTE

Law enforcement agencies and departments are under tremendous pressure to carry out their mission of safeguarding people with limited resources. The environment within which they perform their duties is becoming increasingly more challenging so that they

198 Part II • Predictive Analytics/Machine Learning

have to constantly adopt and perhaps stay a few steps ahead to prevent the likelihood of catastrophes. Understanding the changing nature of crime and criminals is an ongo- ing challenge. In the midst of these challenges, what works in favor of these agencies is the availability of the data and analytics technologies to better analyze past occur- rences and to foresee future events. Data have become available more now than in the past. Applying advanced analytics and data mining tools (i.e., knowledge discovery techniques) to these large and rich data sources provides them with the insight that they need to better prepare and act on their duties. Therefore, law enforcement agencies are becoming one of the leading users of the new face of analytics. Data mining is a prime candidate for better understanding and management of these mission-critical tasks with a high level of accuracy and timeliness. The study described here clearly illustrates the power of analytics and data mining to create a holistic view of the world of crime and criminals for better and faster reaction and management. In this chapter, you will see a wide variety of data mining applications solving complex problems in a variety of indus- tries and organizational settings where the data are used to discover actionable insight to improve mission readiness, operational efficiency, and competitive advantage.

Sources: “Miami-Dade Police Department: Predictive modeling pinpoints likely suspects based on common crime signatures of previous crimes,” IBM Customer Case Studies. www-03.ibm.com/software/ businesscasestudies/om/en/corp?synkey=C894638H25952N07; “Law Enforcement Analytics: Intelligence- Led and Predictive Policing by Information Builder.” www.informationbuilders.com/solutions/gov-lea.

4.2 DATA MINING CONCEPTS

Data mining, a relatively new and exciting technology, has become a common practice for a vast majority of organizations. In an interview with Computerworld magazine in January 1999, Dr. Arno Penzias (Nobel laureate and former chief scientist of Bell Labs) identified data mining from organizational databases as a key application for corporations of the near future. In response to Computerworld’s age-old question of “What will be the killer applica- tions in the corporation?” Dr. Penzias replied, “Data mining.” He then added, “Data mining will become much more important and companies will throw away nothing about their customers because it will be so valuable. If you’re not doing this, you’re out of business.” Similarly, in an article in Harvard Business Review, Thomas Davenport (2006) argued that the latest strategic weapon for companies is analytical decision making, providing examples of companies such as Amazon.com, Capital One, Marriott International, and others that have used analytics to better understand their customers and optimize their extended sup- ply chains to maximize their returns on investment while providing the best customer ser- vice. This level of success is highly dependent on a company’s thorough understanding of its customers, vendors, business processes, and the extended supply chain.

A large portion of “understanding the customer” can come from analyzing the vast amount of data that a company collects. The cost of storing and processing data has decreased dramatically in the recent past, and, as a result, the amount of data stored in electronic form has grown at an explosive rate. With the creation of large databases, the possibility of ana- lyzing the data stored in them has emerged. The term data mining was originally used to describe the process through which previously unknown patterns in data were discovered. This definition has since been stretched beyond those limits by some software vendors to include most forms of data analysis in order to increase sales with the popularity of the data mining label. In this chapter, we accept the original definition of data mining.

Although the term data mining is relatively new, the ideas behind it are not. Many of the techniques used in data mining have their roots in traditional statistical analysis and artificial intelligence work done since the early part of the 1980s. Why, then, has it

Chapter 4 • Data Mining Process, Methods, and Algorithms 199

suddenly gained the attention of the business world? Following are some of the most important reasons:

• More intense competition at the global scale driven by customers’ ever-changing needs and wants in an increasingly saturated marketplace.

• General recognition of the untapped value hidden in large data sources. • Consolidation and integration of database records, which enables a single view of

customers, vendors, transactions, and so on. • Consolidation of databases and other data repositories into a single location in the

form of a data warehouse. • The exponential increase in data processing and storage technologies. • Significant reduction in the cost of hardware and software for data storage and

processing. • Movement toward the demassification (conversion of information resources into

nonphysical form) of business practices.

Data generated by the Internet are increasing rapidly in both volume and complex- ity. Large amounts of genomic data are being generated and accumulated all over the world. Disciplines such as astronomy and nuclear physics create huge quantities of data on a regular basis. Medical and pharmaceutical researchers constantly generate and store data that can then be used in data mining applications to identify better ways to accu- rately diagnose and treat illnesses and to discover new and improved drugs.

On the commercial side, perhaps the most common use of data mining has been in the finance, retail, and healthcare sectors. Data mining is used to detect and reduce fraudulent activities, especially in insurance claims and credit card use (Chan et al., 1999); to identify customer buying patterns (Hoffman, 1999); to reclaim profitable customers (Hoffman, 1998); to identify trading rules from historical data; and to aid in increased profitability using market-basket analysis. Data mining is already widely used to bet- ter target clients, and with the widespread development of e-commerce, this can only become more imperative with time. See Application Case 4.1 for information on how Infinity P&C has used predictive analytics and data mining to improve customer service, combat fraud, and increase profit.

When card issuers first started using automated busi- ness rules software to counter debit and credit card fraud, the limits on that technology were quickly evident: Customers reported frustrating payment rejections on dream vacations or critical business trips. Visa works with its clients to improve cus- tomer experience by providing cutting-edge fraud risk tools and consulting services that make its strategies more effective. Through this approach, Visa enhances customer experience and minimizes invalid transaction declines.

The company’s global network connects thousands of financial institutions with millions of merchants and cardholders every day. It has been

a pioneer in cashless payments for more than 50 years. By using SAS® Analytics, Visa is supporting financial institutions to reduce fraud without upset- ting customers with unnecessary payment rejections. Whenever it processes a transaction, Visa analyzes up to 500 unique variables in real time to assess the risk of that transaction. Using vast data sets, includ- ing global fraud hot spots and transactional patterns, the company can more accurately assess whether you’re buying escargot in Paris or someone who stole your credit card is.

“What that means is that if you are likely to travel we know it, and we tell your financial insti- tution so you’re not declined at the point of sale,”

Application Case 4.1 Visa Is Enhancing the Customer Experience while Reducing Fraud with Predictive Analytics and Data Mining

(Continued )

200 Part II • Predictive Analytics/Machine Learning

says Nathan Falkenborg, head of Visa Performance Solutions for North Asia. “We also will assist your bank in developing the right strategies for using the Visa tools and scoring systems,” he adds. Visa esti- mates that Big Data analytics works; state-of-the-art models and scoring systems have the potential to prevent an incremental $2 billion of fraudulent pay- ment volume annually.

A globally recognized name, Visa facilitates electronic funds transfer through branded products that are issued by its thousands of financial institu- tion partners. The company processed 64.9 billion transactions in 2014, and $4.7 trillion in purchases were made with Visa cards in the same year.

Visa has the computing capability to process 56,000 transaction messages per second, which is more than four times the actual peak transaction rate to date. Visa does not just process and compute—it is continually using analytics to share strategic and operational insights with its partner financial insti- tutions and assist them in improving performance. This business goal is supported by a robust data management system. Visa also assists its clients in improving performance by developing and deliver- ing deep analytical insight.

“We understand patterns of behavior by per- forming clustering and segmentation at a granular level, and we provide this insight to our financial institution partners,” says Falkenborg. “It’s an effec- tive way to help our clients communicate better and deepen their understanding of the customer.”

As an example of marketing support, Visa has assisted clients globally in identifying segments of customers that should be offered a different Visa product. “Understanding the customer lifecycle is incredibly important, and Visa provides information to clients that help them take action and offer the right product to the right customer before a value proposition becomes stale,” says Falkenborg.

How Can Using In-Memory Analytics Make a Difference?

In a recent proof of concept, Visa used a high- performance solution from SAS that relies on in-memory computing to power statistical and machine-learning algorithms and then present the information visually. In-memory analytics reduces

the need to move data and perform additional model iterations, making it much faster and accurate.

Falkenborg describes the solution as like having the information memorized versus having to get up and go to a filing cabinet to retrieve it. “In-memory analytics is just taking your brain and making it bigger. Everything is instantly accessible.”

Ultimately, solid analytics helps the company do more than just process payments. “We can deepen the client conversation and serve our clients even better with our incredible big data set and expertise in mining transaction data,” says Falkenborg. “We use our consulting and analytics capabilities to assist our clients in tackling business challenges and pro- tect the payment ecosystem. And that’s what we do with high-performance analytics.”

Falkenborg elaborates,

The challenge that we have, as with any com- pany managing and using massive data sets, is how we use all necessary information to solve a business challenge—whether that is improving our fraud models, or assisting a cli- ent to more effectively communicate with its customers. In-memory analytics enables us to be more nimble; with a 100* analytical system processing speed improvement, our data and decision scientists can iterate much faster.

Fast and accurate predictive analytics allows Visa to better serve clients with tailored consult- ing services, helping them succeed in today’s fast- changing payments industry.

Questions for Case 4.1

1. What challenges were Visa and the rest of the credit card industry facing?

2. How did Visa improve customer service while also improving concepts related to retention of fraud?

3. What is in-memory analytics, and why was it necessary?

Source: “Enhancing the Customer Experience While Reducing Fraud (SAS® Analytics) High-Performance Analytics Empowers Visa to Enhance Customer Experience While Reducing Debit and Credit Card Fraud.” Copyright © 2018 SAS Institute Inc., Cary, NC, USA. Reprinted with permission. All rights reserved.

Application Case 4.1 (Continued)

Chapter 4 • Data Mining Process, Methods, and Algorithms 201

Definitions, Characteristics, and Benefits

Simply defined, data mining is a term used to describe discovering or “mining” knowl- edge from large amounts of data. When considered by analogy, one can easily realize that the term data mining is a misnomer; that is, mining of gold from within rocks or dirt is referred to as “gold” mining rather than “rock” or “dirt” mining. Therefore, data min- ing perhaps should have been named “knowledge mining” or “knowledge discovery.” Despite the mismatch between the term and its meaning, data mining has become the choice of the community. Many other names that are associated with data mining include knowledge extraction, pattern analysis, data archaeology, information harvesting, pattern searching, and data dredging.

Technically speaking, data mining is a process that uses statistical, mathematical, and artificial intelligence techniques to extract and identify useful information and subse- quent knowledge (or patterns) from large sets of data. These patterns can be in the form of business rules, affinities, correlations, trends, or prediction models (see Nemati and Barko, 2001). Most literature defines data mining as “the nontrivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data stored in structured databases,” where the data are organized in records structured by categori- cal, ordinal, and continuous variables (Fayyad, Piatetsky-Shapiro, and Smyth, 1996, pp. 40–41). In this definition, the meanings of the key term are as follows:

• Process implies that data mining comprises many iterative steps. • Nontrivial means that some experimental type search or inference is involved; that

is, it is not as straightforward as a computation of predefined quantities. • Valid means that the discovered patterns should hold true on new data with a suf-

ficient degree of certainty. • Novel means that the patterns are not previously known to the user within the con-

text of the system being analyzed. • Potentially useful means that the discovered patterns should lead to some benefit to

the user or task. • Ultimately understandable means that the pattern should make business sense that

leads to the user saying, “Mmm! It makes sense; why didn’t I think of that,” if not immediately, at least after some postprocessing.

Data mining is not a new discipline but rather a new definition for the use of many disciplines. Data mining is tightly positioned at the intersection of many disci- plines, including statistics, artificial intelligence, machine learning, management sci- ence, information systems (IS), and databases (see Figure 4.1). Using advances in all of these disciplines, data mining strives to make progress in extracting useful information and knowledge from large databases. It is an emerging field that has attracted much attention in a very short time.

The following are the major characteristics and objectives of data mining:

• Data are often buried deep within very large databases, which sometimes contain data from several years. In many cases, the data are cleansed and consolidated into a data warehouse. Data can be presented in a variety of formats (see Chapter 3 for a brief taxonomy of data).

• The data mining environment is usually a client/server architecture or a Web-based IS architecture.

• Sophisticated new tools, including advanced visualization tools, help remove the information ore buried in corporate files or archival public records. Finding it involves massaging and synchronizing the data to get the right results. Cutting-edge data min- ers are also exploring the usefulness of soft data (i.e., unstructured text stored in such places as Lotus Notes databases, text files on the Internet, or enterprisewide intranets).

202 Part II • Predictive Analytics/Machine Learning

• The miner is often an end user empowered by data drills and other powerful query tools to ask ad hoc questions and obtain answers quickly with little or no programming skill.

• “Striking it rich” often involves finding an unexpected result and requires end users to think creatively throughout the process, including the interpretation of the findings.

• Data mining tools are readily combined with spreadsheets and other software devel- opment tools. Thus, the mined data can be analyzed and deployed quickly and easily.

• Because of the large amounts of data and massive search efforts, it is sometimes necessary to use parallel processing for data mining.

A company that effectively leverages data mining tools and technologies can ac- quire and maintain a strategic competitive advantage. Data mining offers organizations an indispensable decision-enhancing environment to exploit new opportunities by trans- forming data into a strategic weapon. See Nemati and Barko (2001) for a more detailed discussion on the strategic benefits of data mining.

How Data Mining Works

Using existing and relevant data obtained from within and outside the organization, data mining builds models to discover patterns among the attributes presented in the data set. Models are the mathematical representations (simple linear relationships/affinities and/or complex and highly nonlinear relationships) that identify the patterns among the attributes of the things (e.g., customers, events) described within the data set. Some of these patterns are explanatory (explaining the interrelationships and affinities among the attributes), whereas others are predictive (foretelling future values of certain attributes). In general, data mining seeks to identify four major types of patterns:

1. Associations find the commonly co-occurring groupings of things, such as beer and diapers going together in market-basket analysis.

Statistics

Artificial Intelligence

Information Visualization

Database Management

& Data Warehousing

Management Science & Information Systems

Machine Learning &

Pattern Recognition

DATA MINING (Knowledge Discovery)

FIGURE 4.1 Data Mining is a Blend of Multiple Disciplines.

Chapter 4 • Data Mining Process, Methods, and Algorithms 203

2. Predictions tell the nature of future occurrences of certain events based on what has happened in the past, such as predicting the winner of the Super Bowl or forecasting the absolute temperature of a particular day.

3. Clusters identify natural groupings of things based on their known characteristics, such as assigning customers in different segments based on their demographics and past purchase behaviors.

4. Sequential relationships discover time-ordered events, such as predicting that an existing banking customer who already has a checking account will open a savings account followed by an investment account within a year.

Application Case 4.2 shows how American Honda uses data mining (a critical component of advanced analytics tools) to enhance their understanding of the warranty claims, forecast feature part and resource needs, and better understand customer needs, wants, and opinions.

Background

When a car or truck owner brings a vehicle into an Acura or Honda dealership in the United States, there’s more to the visit than a repair or a service check. During each visit, the service technicians gen- erate data on the repairs, including any warranty claims to American Honda Motor Co., Inc., that feed directly into its database. This includes what type of work was performed, what the customer paid, ser- vice advisor comments, and many other data points.

Now, multiply this process by dozens of visits a day at over 1,200 dealerships nationwide, and it’s clear—American Honda has big data. It’s up to peo- ple like Kendrick Kau, assistant manager of American Honda’s Advanced Analytics group, to draw insights from this data and turn it into a useful asset.

Examining Warranty Data to Make Maintenance More Efficient

Like any other major automobile distributor, American Honda works with a network of dealer- ships that perform warrantied repair work on its vehicles. This can be a significant cost for the com- pany, so American Honda uses analytics to make sure that warranty claims are complete and accurate upon submission.

In the case of warranty claims, Kau’s team helps empower dealers to understand the appro- priate warranty processes by providing them with useful information via an online report. To support a goal of reducing inappropriate warranty costs,

Kau and his team must sift through information on repairs, parts, customers, and other details. They chose a visual approach to business intelligence and analytics, powered by SAS, to identify cost reduction opportunities.

To decrease warranty expense, the Advanced Analytics team used SAS Analytics to create a propri- etary process to surface suspicious warranty claims for scrutiny on a daily basis to make sure they are in compliance with existing guidelines. The effort to identify and scrutinize claims was once fairly man- ual, tedious, and time-intensive.

“Before SAS, it took one of our staff members one week out of each month to aggregate and report warranty data within Microsoft Excel spreadsheets,” Kau says. “Now, with SAS, we populate those same reports on an easily accessible online dashboard automatically, and we recovered a week of man- power that we could put on other projects.”

By applying SAS Analytics to warranty data, the Advanced Analytics group gave the Claims group and field personnel the ability to quickly and accu- rately identify claims that were incomplete, inaccu- rate, or noncompliant. The results were impressive.

“Initially, it took our examiners over three min- utes on average to identify a potentially noncompli- ant claim, and even then, they were only finding a truly noncompliant claim 35 percent of the time,” Kau says. “Now, with SAS, it takes less than a minute to identify a suspicious claim. And in that time, they are finding a noncompliant claim 76 percent of the time.”

Application Case 4.2 American Honda Uses Advanced Analytics to Improve Warranty Claims

(Continued )

204 Part II • Predictive Analytics/Machine Learning

The effort to increase warranty compliance has paid off for American Honda. Through more com- plete analysis of warranty claims—and more edu- cation at the dealerships—American Honda saw a reduction in labor costs for 52 percent of its available labor codes.

Using Service Data to Forecast Future Needs

The American Honda Advanced Analytics team also uses service and parts data to develop stronger bonds with customers by ensuring dealers have in-demand parts available for customer repairs. Having the right parts available—at the right time—is paramount, so vehicle repairs data feed directly into American Honda’s marketing and customer retention efforts.

“For the marketing team, we provide strate- gic insight to help shape their programs that are designed to drive customers to the dealers, and ulti- mately, keep them loyal to our brand,” Kau says. “The goal of Honda is lifetime owner loyalty. We want our customers to have a good experience, and one of the ways to do that is through exceptional service.”

American Honda uses SAS Forecast Server to assist with business planning to ensure adequate resources are available to meet future demands for services. Using historical information on repair orders and certifications, they developed a time series using years of previous repairs. By combining time series information with sales data, Kau’s team can project where the company’s greatest opportu- nities are in the years ahead.

“Our goal is to forecast the number of vehi- cles in operation in order to predict the volume of customers coming into the dealerships,” Kau says.

“And that translates to how many parts we should have on hand and helps us to plan staffing to meet customer demands. Looking backward on a year-by- year basis, we’ve been within 1 percent of where we forecast to be. That’s extremely good for a forecast, and I attribute much of that to the abilities of the SAS software.”

Customer Feedback that Drives the Business

Another way American Honda uses analytics is to quickly evaluate customer survey data. Using SAS, the Advanced Analytics team mines survey data to gain insight into how vehicles are being used and identify design changes that are most likely to improve customer satisfaction.

On a weekly basis, the analytics team exam- ines customer survey data. Kau’s team uses SAS to flag emerging trends that may require the attention of design, manufacturing, engineering, or other groups. With SAS technology, users can drill down from high-level issues to more specific responses to understand a potential root cause.

“We can look into the data and see what the customers are saying,” Kau says. “And that leads to a number of questions that we can tackle. Is a component designed in the most optimal way? Is it a customer education issue? Is it something that we should address at the manufacturing process? Because of SAS, these are critical questions that we can now identify using our data.”

Questions for Case 4.2

1. How does American Honda use analytics to improve warranty claims?

2. In addition to warranty claims, for what other purposes does American Honda use advanced analytics methods?

3. Can you think of other uses of advanced analyt- ics in the automotive industry? You can search the Web to find some answers to this question.

Source: SAS Case Study “American Honda Motor Co., Inc. uses SAS advanced analytics to improve warranty claims” https:// www.sas.com/en_us/customers/american-honda.html (accessed June 2018)

Application Case 4.2 (Continued)

Honda–Facts & Figures

Faster claims analysis

Reduced labor costs

3X $$ 1200

Dealerships nationwide

Chapter 4 • Data Mining Process, Methods, and Algorithms 205

These types of patterns have been manually extracted from data by humans for centuries, but the increasing volume of data in modern times has created the need for more automatic approaches. As data sets have grown in size and complexity, direct manual data analysis has increasingly been augmented with indirect, automatic data- processing tools that use sophisticated methodologies, methods, and algorithms. The manifestation of such evolution of automated and semiautomated means of processing large data sets is now commonly referred to as data mining.

Generally speaking, data mining tasks can be classified into three main categories: prediction, association, and clustering. Based on the way in which the patterns are ex- tracted from the historical data, the learning algorithms of data mining methods can be classified as either supervised or unsupervised. With supervised learning algorithms, the training data includes both the descriptive attribute (i.e., independent variables or deci- sion variables) and the class attribute (i.e., output variable or result variable). In contrast, with unsupervised learning, the training includes only descriptive attributes. Figure 4.2 shows a simple taxonomy for data mining tasks along with the learning methods and popular algorithms for each of the data mining tasks.

PREDICTION Prediction is commonly referred to as the act of telling about the future. It differs from simple guessing by taking into account the experiences, opinions, and other relevant information in conducting the task of foretelling. A term that is commonly associ- ated with prediction is forecasting. Even though many believe that these two terms are synonymous, there is a subtle but critical difference between the two. Whereas prediction is largely experience and opinion based, forecasting is data and model based. That is, in order of increasing reliability, one might list the relevant terms as guessing, predicting, and forecasting, respectively. In data mining terminology, prediction and forecasting are used synonymously, and the term prediction is used as the common representation of the act. Depending on the nature of what is being predicted, prediction can be named more specifically as classification (where the predicted thing, such as tomorrow’s forecast, is a class label such as “rainy” or “sunny”) or regression (where the predicted thing, such as tomorrow’s temperature, is a real number, such as “65°F”).

CLASSIFICATION Classification, or supervised induction, is perhaps the most common of all data mining tasks. The objective of classification is to analyze historical data stored in a database and automatically generate a model that can predict future behavior. This induced model consists of generalizations over the records of a training data set, which help distin- guish predefined classes. The hope is that the model can then be used to predict the classes of other unclassified records and, more importantly, to accurately predict actual future events.

Common classification tools include neural networks and decision trees (from machine learning), logistic regression and discriminant analysis (from traditional statis- tics), and emerging tools such as rough sets, support vector machines (SVMs), and ge- netic algorithms. Statistics-based classification techniques (e.g., logistic regression and discriminant analysis) have received their share of criticism—that they make unrealistic assumptions about the data, such as independence and normality—which limit their use in classification-type data mining projects.

Neural networks involve the development of mathematical structures (somewhat resembling the biological neural networks in the human brain) that have the capability to learn from past experiences presented in the form of well-structured data sets. They tend to be more effective when the number of variables involved is rather large and the relationships among them are complex and imprecise. Neural networks have disadvan- tages as well as advantages. For example, providing a good rationale for the predictions made by a neural network is usually very difficult. Also, training neural networks usually

206 Part II • Predictive Analytics/Machine Learning

Data Mining Algorithms

K-means, Expectation Maximization (EM)

Autoregressive Methods, Averaging Methods, Exponential Smoothing, ARIMA

Expectation Maximization, Apriori Algorithm, Graph-Based Matching

Apriori, OneR, ZeroR, Eclat, GA

Linear/Nonlinear Regression, ANN, Regression Trees, SVM, kNN, GA

Decision Trees, Neural Networks, Support Vector Machines, kNN, Naïve Bayes, GA

Data Mining Tasks and Methods

Prediction

Classification

Regression

Segmentation

Association

Link Analysis

Sequence Analysis

Clustering

Apriori Algorithm, FP-Growth, Graph-Based Matching

Time series

Market-Basket

Outlier Analysis

Learning Type

K-means, Expectation Maximization (EM)

Supervised

Unsupervised

Supervised

Supervised

Unsupervised

Unsupervised

Unsupervised

Unsupervised

FIGURE 4.2 Simple Taxonomy for Data Mining Tasks, Methods, and Algorithms.

takes a considerable amount of time. Unfortunately, the time needed for training tends to increase exponentially as the volume of data increases, and in general, neural networks cannot be trained on very large databases. These and other factors have limited the ap- plicability of neural networks in data-rich domains.

Decision trees classify data into a finite number of classes based on the values of the input variables. Decision trees are essentially a hierarchy of if-then statements and are thus significantly faster than neural networks. They are most appropriate for categorical data and interval data. Therefore, incorporating continuous variables into a decision

Chapter 4 • Data Mining Process, Methods, and Algorithms 207

tree framework requires discretization, that is, converting continuous valued numerical variables to ranges and categories.

A related category of classification tools is rule induction. Unlike with a decision tree, with rule induction the if-then statements are induced from the training data directly, and they need not be hierarchical in nature. Other, more recent techniques such as SVM, rough sets, and genetic algorithms are gradually finding their way into the arsenal of clas- sification algorithms.

CLUSTERING Clustering partitions a collection of things (e.g., objects, events, presented in a structured data set) into segments (or natural groupings) whose members share simi- lar characteristics. Unlike in classification, in clustering, the class labels are unknown. As the selected algorithm goes through the data set, identifying the commonalities of things based on their characteristics, the clusters are established. Because the clusters are de- termined using a heuristic-type algorithm and because different algorithms could end up with different sets of clusters for the same data set, before the results of clustering tech- niques are put to actual use, it could be necessary for an expert to interpret, and poten- tially modify, the suggested clusters. After reasonable clusters have been identified, they can be used to classify and interpret new data.

Not surprisingly, clustering techniques include optimization. The goal of clustering is to create groups so that the members within each group have maximum similarity and the members across groups have minimum similarity. The most commonly used cluster- ing techniques include k-means (from statistics) and self-organizing maps (from machine learning), which is a unique neural network architecture developed by Kohonen (1982).

Firms often effectively use their data mining systems to perform market segmenta- tion with cluster analysis. Cluster analysis is a means of identifying classes of items so that items in a cluster have more in common with each other than with items in other clusters. Cluster analysis can be used in segmenting customers and directing appropriate market- ing products to the segments at the right time in the right format at the right price. Cluster analysis is also used to identify natural groupings of events or objects so that a common set of characteristics of these groups can be identified to describe them.

ASSOCIATIONS Associations, or association rule learning in data mining, is a popular and well-researched technique for discovering interesting relationships among variables in large databases. Thanks to automated data-gathering technologies such as bar code scanners, the use of association rules for discovering regularities among products in large- scale transactions recorded by point-of-sale systems in supermarkets has become a com- mon knowledge discovery task in the retail industry. In the context of the retail industry, association rule mining is often called market-basket analysis.

Two commonly used derivatives of association rule mining are link analysis and sequence mining. With link analysis, the linkage among many objects of interest is dis- covered automatically, such as the link between Web pages and referential relationships among groups of academic publication authors. With sequence mining, relationships are examined in terms of their order of occurrence to identify associations over time. Algorithms used in association rule mining include the popular Apriori (where frequent itemsets are identified) and FP-Growth, OneR, ZeroR, and Eclat.

VISUALIZATION AND TIME-SERIES FORECASTING Two techniques often associated with data mining are visualization and time-series forecasting. Visualization can be used in conjunction with other data mining techniques to gain a clearer understanding of underly- ing relationships. As the importance of visualization has increased in recent years, a new term, visual analytics, has emerged. The idea is to combine analytics and visualization in a single environment for easier and faster knowledge creation. Visual analytics is covered in

208 Part II • Predictive Analytics/Machine Learning

detail in Chapter 3. In time-series forecasting, the data consist of values of the same vari- able that are captured and stored over time in regular intervals. These data are then used to develop forecasting models to extrapolate the future values of the same variable.

Data Mining Versus Statistics

Data mining and statistics have a lot in common. They both look for relationships within data. Most people call statistics the “foundation of data mining.” The main difference between the two is that statistics starts with a well-defined proposition and hypothesis whereas data mining starts with a loosely defined discovery statement. Statistics collects sample data (i.e., primary data) to test the hypothesis whereas data mining and analyt- ics use all the existing data (i.e., often observational, secondary data) to discover novel patterns and relationships. Another difference comes from the size of data that they use. Data mining looks for data sets that are as “big” as possible, whereas statistics looks for the right size of data (if the data are larger than what is needed/required for the statisti- cal analysis, a sample of them is used). The meaning of “large data” is rather different between statistics and data mining. A few hundred to a thousand data points are large enough to a statistician, but several million to a few billion data points are considered large for data mining studies.

uSECTION 4.2 REVIEW QUESTIONS

1. Define data mining. Why are there many different names and definitions for data mining?

2. What recent factors have increased the popularity of data mining? 3. Is data mining a new discipline? Explain. 4. What are some major data mining methods and algorithms? 5. What are the key differences between the major data mining tasks?

4.3 DATA MINING APPLICATIONS

Data mining has become a popular tool in addressing many complex business problems and opportunities. It has been proven to be very successful and helpful in many areas, some of which are shown by the following representative examples. The goal of many of these business data mining applications is to solve a pressing problem or to explore an emerging business opportunity to create a sustainable competitive advantage.

• Customer relationship management. CRM is the extension of traditional mar- keting. The goal of CRM is to create one-on-one relationships with customers by developing an intimate understanding of their needs and wants. As businesses build relationships with their customers over time through a variety of interactions (e.g., product inquiries, sales, service requests, warranty calls, product reviews, social media connections), they accumulate tremendous amounts of data. When combined with demographic and socioeconomic attributes, this information-rich data can be used to (1) identify most likely responders/buyers of new products/services (i.e., customer profiling), (2) understand the root causes of customer attrition to improve customer retention (i.e., churn analysis), (3) discover time-variant associations be- tween products and services to maximize sales and customer value, and (4) identify the most profitable customers and their preferential needs to strengthen relation- ships and to maximize sales.

• Banking. Data mining can help banks with the following: (1) automating the loan application process by accurately predicting the most probable defaulters,

Chapter 4 • Data Mining Process, Methods, and Algorithms 209

(2) detecting fraudulent credit card and online banking transactions, (3) identifying ways to maximize value for customers by selling them products and services that they are most likely to buy, and (4) optimizing the cash return by accurately fore- casting the cash flow on banking entities (e.g., ATM machines, banking branches).

• Retailing and logistics. In the retailing industry, data mining can be used to (1) predict accurate sales volumes at specific retail locations to determine correct inven- tory levels, (2) identify sales relationships between different products (with market- basket analysis) to improve the store layout and optimize sales promotions, (3) forecast consumption levels of different product types (based on seasonal and environmental conditions) to optimize logistics and, hence, maximize sales, and (4) discover interest- ing patterns in the movement of products (especially for products that have a limited shelf life because they are prone to expiration, perishability, and contamination) in a supply chain by analyzing sensory and radio-frequency identification (RFID) data.

• Manufacturing and production. Manufacturers can use data mining to (1) pre- dict machinery failures before they occur through the use of sensory data (enabling what is called condition-based maintenance), (2) identify anomalies and common- alities in production systems to optimize manufacturing capacity, and (3) discover novel patterns to identify and improve product quality.

• Brokerage and securities trading. Brokers and traders use data mining to (1) predict when and how much certain bond prices will change, (2) forecast the range and direction of stock fluctuations, (3) assess the effect of particular issues and events on overall market movements, and (4) identify and prevent fraudulent activities in securities trading.

• Insurance. The insurance industry uses data mining techniques to (1) forecast claim amounts for property and medical coverage costs for better business planning, (2) determine optimal rate plans based on the analysis of claims and customer data, (3) predict which customers are more likely to buy new policies with special fea- tures, and (4) identify and prevent incorrect claim payments and fraudulent activities.

• Computer hardware and software. Data mining can be used to (1) predict disk drive failures well before they actually occur, (2) identify and filter unwanted Web content and e-mail messages, (3) detect and prevent computer network security breaches, and (4) identify potentially unsecure software products.

• Government and defense. Data mining also has a number of military appli- cations. It can be used to (1) forecast the cost of moving military personnel and equipment, (2) predict an adversary’s moves and, hence, develop more successful strategies for military engagements, (3) predict resource consumption for better planning and budgeting, and (4) identify classes of unique experiences, strategies, and lessons learned from military operations for better knowledge sharing through- out the organization.

• Travel industry (airlines, hotels/resorts, rental car companies). Data mining has a variety of uses in the travel industry. It is successfully used to (1) predict sales of different services (seat types in airplanes, room types in hotels/resorts, car types in rental car companies) in order to optimally price services to maximize revenues as a function of time-varying transactions (commonly referred to as yield management), (2) forecast demand at different locations to better allocate limited organizational resources, (3) identify the most profitable customers and provide them with person- alized services to maintain their repeat business, and (4) retain valuable employees by identifying and acting on the root causes for attrition.

• Healthcare. Data mining has a number of healthcare applications. It can be used to (1) identify people without health insurance and the factors underlying this unde- sired phenomenon, (2) identify novel cost–benefit relationships between different

210 Part II • Predictive Analytics/Machine Learning

treatments to develop more effective strategies, (3) forecast the level and the time of demand at different service locations to optimally allocate organizational resources, and (4) understand the underlying reasons for customer and employee attrition.

• Medicine. Use of data mining in medicine should be viewed as an invaluable complement to traditional medical research, which is mainly clinical and biological in nature. Data mining analyses can (1) identify novel patterns to improve surviv- ability of patients with cancer, (2) predict success rates of organ transplantation patients to develop better organ donor matching policies, (3) identify the functions of different genes in the human chromosome (known as genomics), and (4) dis- cover the relationships between symptoms and illnesses (as well as illnesses and successful treatments) to help medical professionals make informed and correct decisions in a timely manner.

• Entertainment industry. Data mining is successfully used by the entertainment industry to (1) analyze viewer data to decide what programs to show during prime time and how to maximize returns by knowing where to insert advertisements, (2) predict the financial success of movies before they are produced to make invest- ment decisions and to optimize the returns, (3) forecast the demand at different loca- tions and different times to better schedule entertainment events and to optimally allocate resources, and (4) develop optimal pricing policies to maximize revenues.

• Homeland security and law enforcement. Data mining has a number of home- land security and law enforcement applications. It is often used to (1) identify patterns of terrorist behaviors (see Application Case 4.3 for an example of the use of data min- ing to track funding of terrorists’ activities), (2) discover crime patterns (e.g., locations, timings, criminal behaviors, and other related attributes) to help solve criminal cases in a timely manner, (3) predict and eliminate potential biological and chemical attacks to the nation’s critical infrastructure by analyzing special-purpose sensory data, and (4) identify and stop malicious attacks on critical information infrastructures (often called information warfare).

The terrorist attack on the World Trade Center on September 11, 2001, underlined the importance of open source intelligence. The USA PATRIOT Act and the creation of the U.S. Department of Homeland Security heralded the potential application of infor- mation technology and data mining techniques to detect money laundering and other forms of terror- ist financing. Law enforcement agencies had been focusing on money laundering activities via normal transactions through banks and other financial ser- vice organizations.

Law enforcement agencies are now focusing on international trade pricing as a terrorism fund- ing tool. Money launderers have used international trade to move money silently out of a country with- out attracting government attention. They achieve

this transfer by overvaluing imports and undervalu- ing exports. For example, a domestic importer and foreign exporter could form a partnership and over- value imports, thereby transferring money from the home country, resulting in crimes related to customs fraud, income tax evasion, and money laundering. The foreign exporter could be a member of a terror- ist organization.

Data mining techniques focus on analysis of data on import and export transactions from the U.S. Department of Commerce and commerce- related entities. Import prices that exceed the upper quartile of import prices and export prices that are lower than the lower quartile of export prices are tracked. The focus is on abnormal transfer prices between corporations that might result in shifting

Application Case 4.3 Predictive Analytic and Data Mining Help Stop Terrorist Funding

Chapter 4 • Data Mining Process, Methods, and Algorithms 211

taxable income and taxes out of the United States. An observed price deviation could be related to income tax avoidance/evasion, money laundering, or terrorist financing. The observed price devia- tion could also be due to an error in the U.S. trade database.

Data mining will result in efficient evalua- tion of data, which, in turn, will aid in the fight against terrorism. The application of information technology and data mining techniques to financial transactions can contribute to better intelligence information.

Questions for Case 4.3

1. How can data mining be used to fight terrorism? Comment on what else can be done beyond what is covered in this short application case.

2. Do you think data mining, although essential for fighting terrorist cells, also jeopardizes individu- als’ rights of privacy?

Sources: J. S. Zdanowic, “Detecting Money Laundering and Terrorist Financing via Data Mining,” Communications of the ACM, 47(5), May 2004, p. 53; R. J. Bolton, “Statistical Fraud Detection: A Review,” Statistical Science, 17(3), January 2002, p. 235.

• Sports. Data mining was used to improve the performance of National Basketball Association (NBA) teams in the United States. Major League Baseball teams are into predictive analytics and data mining to optimally utilize their limited resources for a winning season. In fact, most, if not all, professional sports today employ data crunchers and use data mining to increase their chances of winning. Data mining applications are not limited to professional sports. In an article, Delen at al. (2012) developed data mining models to predict National Collegiate Athletic Association (NCAA) Bowl Game outcomes using a wide range of variables about the two op- posing teams’ previous games statistics (more details about this case study are pro- vided in Chapter 3). Wright (2012) used a variety of predictors for examination of the NCAA men’s basketball championship (a.k.a. March Madness) bracket.

uSECTION 4.3 REVIEW QUESTIONS

1. What are the major application areas for data mining? 2. Identify at least five specific applications of data mining and list five common charac-

teristics of these applications.

3. What do you think is the most prominent application area for data mining? Why? 4. Can you think of other application areas for data mining not discussed in this section?

Explain.

4.4 DATA MINING PROCESS

To systematically carry out data mining projects, a general process is usually followed. Based on best practices, data mining researchers and practitioners have proposed sev- eral processes (workflows or simple step-by-step approaches) to maximize the chances of success in conducting data mining projects. These efforts have led to several standard- ized processes, some of which (a few of the most popular ones) are described in this section.

One such standardized process, arguably the most popular one, the Cross-Industry Standard Process for Data Mining—CRISP-DM—was proposed in the mid-1990s by a European consortium of companies to serve as a nonproprietary standard methodology for data mining (CRISP-DM, 2013). Figure 4.3 illustrates this proposed process, which is a sequence of six steps that starts with a good understanding of the business and the

212 Part II • Predictive Analytics/Machine Learning

need for the data mining project (i.e., the application domain) and ends with the deploy- ment of the solution that satisfies the specific business need. Even though these steps are sequential in nature, there is usually a great deal of backtracking. Because data mining is driven by experience and experimentation, depending on the problem situation and the knowledge/experience of the analyst, the whole process can be very iterative (i.e., one should expect to go back and forth through the steps quite a few times) and time consuming. Because later steps are built on the outcomes of the former ones, one should pay extra attention to the earlier steps in order to not put the whole study on an incorrect path from the onset.

Step 1: Business Understanding

The key element of any data mining study is to know what the study is for. Determining this begins with a thorough understanding of the managerial need for new knowledge and an explicit specification of the business objective regarding the study to be con- ducted. Specific goals answering questions such as “What are the common characteristics of the customers we have lost to our competitors recently?” or “What are typical profiles of our customers, and how much value does each of them provide to us?” are needed. Then a project plan for finding such knowledge is developed that specifies the people responsible for collecting the data, analyzing the data, and reporting the findings. At this early stage, a budget to support the study should also be established at least at a high level with rough numbers.

Step 2: Data Understanding

A data mining study is specific to addressing a well-defined business task, and differ- ent business tasks require different sets of data. Following the business understanding

Business Understanding

1

Data Understanding

2

Data Preparation

3

Model Building

4

Testing and Evaluation

5

Deployment

6

Data

FIGURE 4.3 Six-Step CRISP-DM Data Mining Process.

Chapter 4 • Data Mining Process, Methods, and Algorithms 213

step, the main activity of the data mining process is to identify the relevant data from many available databases. Some key points must be considered in the data iden- tification and selection phase. First and foremost, the analyst should be clear and concise about the description of the data mining task so that the most relevant data can be identified. For example, a retail data mining project could seek to identify spending behaviors of female shoppers who purchase seasonal clothes based on their demographics, credit card transactions, and socioeconomic attributes. Furthermore, the analyst should build an intimate understanding of the data sources (e.g., where the relevant data are stored and in what form; what the process of collecting the data is—automated versus manual; who the collectors of the data are and how often the data are updated) and the variables (e.g., What are the most relevant variables? Are there any synonymous and/or homonymous variables? Are the variables independent of each other—do they stand as a complete information source without overlapping or conflicting information?).

To better understand the data, the analyst often uses a variety of statistical and graphical techniques, such as simple statistical summaries of each variable (e.g., for nu- meric variables, the average, minimum/maximum, median, and standard deviation are among the calculated measures whereas for categorical variables, the mode and fre- quency tables are calculated), and correlation analysis, scatterplots, histograms, and box plots can be used. A careful identification and selection of data sources and the most relevant variables can make it easier for data mining algorithms to quickly discover useful knowledge patterns.

Data sources for data selection can vary. Traditionally, data sources for business applications include demographic data (such as income, education, number of house- holds, and age), sociographic data (such as hobby, club membership, and entertain- ment), transactional data (sales record, credit card spending, issued checks), and so on. Today, data sources also use external (open or commercial) data repositories, social media, and machine-generated data.

Data can be categorized as quantitative and qualitative. Quantitative data are mea- sured using numeric values, or numeric data. They can be discrete (such as integers) or continuous (such as real numbers). Qualitative data, also known as categorical data, contain both nominal and ordinal data. Nominal data have finite nonordered values (e.g., gender data, which have two values: male and female). Ordinal data have finite ordered values. For example, customer credit ratings are considered ordinal data because the ratings can be excellent, fair, and bad. A simple taxonomy of data (i.e., the nature of data) is provided in Chapter 3.

Quantitative data can be readily represented by some sort of probability distribu- tion. A probability distribution describes how the data are dispersed and shaped. For instance, normally distributed data are symmetric and are commonly referred to as being a bell-shaped curve. Qualitative data can be coded to numbers and then described by frequency distributions. Once the relevant data are selected according to the data mining business objective, data preprocessing should be pursued.

Step 3: Data Preparation

The purpose of data preparation (more commonly called data preprocessing) is to take the data identified in the previous step and prepare it for analysis by data mining meth- ods. Compared to the other steps in CRISP-DM, data preprocessing consumes the most time and effort; most people believe that this step accounts for roughly 80 percent of the total time spent on a data mining project. The reason for such an enormous effort spent on this step is the fact that real-world data are generally incomplete (lacking at- tribute values, lacking certain attributes of interest, or containing only aggregate data),

214 Part II • Predictive Analytics/Machine Learning

noisy (containing errors or outliers), and inconsistent (containing discrepancies in codes or names). The nature of the data and the issues related to the preprocessing of data for analytics are explained in detail in Chapter 3.

Step 4: Model Building

In this step, various modeling techniques are selected and applied to an already prepared data set to address the specific business need. The model-building step also encompasses the assessment and comparative analysis of the various models built. Because there is not a universally known best method or algorithm for a data mining task, one should use a variety of viable model types along with a well-defined experimentation and assessment strategy to identify the “best” method for a given purpose. Even for a single method or algorithm, a number of parameters need to be calibrated to obtain optimal results. Some methods could have specific requirements in the way that the data are to be formatted; thus, stepping back to the data preparation step is often necessary. Application Case 4.4 presents a research study in which a number of model types are developed and com- pared to each other.

According to the American Cancer Society, half of all men and one-third of all women in the United States will develop cancer during their lifetimes; approxi- mately 1.5 million new cancer cases were expected to be diagnosed in 2013. Cancer is the second most common cause of death in the United States and in the world, exceeded only by cardiovascular dis- ease. This year, more than 500,000 Americans are expected to die of cancer—more than 1,300 peo- ple a day—accounting for nearly one of every four deaths.

Cancer is a group of diseases generally char- acterized by uncontrolled growth and spread of abnormal cells. If the growth and/or spread are not controlled, cancer can result in death. Even though the exact reasons are not known, cancer is believed to be caused by both external factors (e.g., tobacco, infectious organisms, chemicals, and radiation) and internal factors (e.g., inherited mutations, hormones, immune conditions, and mutations that occur from metabolism). These causal factors can act together or in sequence to initiate or promote carcinogenesis. Cancer is treated with surgery, radiation, chemo- therapy, hormone therapy, biological therapy, and targeted therapy. Survival statistics vary greatly by cancer type and stage at diagnosis.

The five-year relative survival rate for all can- cers is improving, and the decline in cancer mortal- ity had reached 20 percent in 2013, translating into the avoidance of about 1.2 million deaths from can- cer since 1991. That’s more than 400 lives saved per day! The improvement in survival reflects progress in diagnosing certain cancers at an earlier stage and improvements in treatment. Further improvements are needed to prevent and treat cancer.

Even though cancer research has traditionally been clinical and biological in nature, in recent years, data-driven analytic studies have become a common complement. In medical domains where data- and analytics-driven research has been applied success- fully, novel research directions have been identified to further advance the clinical and biological stud- ies. Using various types of data, including molecular, clinical, literature-based, and clinical trial data, along with suitable data mining tools and techniques, researchers have been able to identify novel pat- terns, paving the road toward a cancer-free society.

In one study, Delen (2009) used three popular data mining techniques (decision trees, artificial neu- ral networks, and SVMs) in conjunction with logistic regression to develop prediction models for prostate cancer survivability. The data set contained around

Application Case 4.4 Data Mining Helps in Cancer Research

Chapter 4 • Data Mining Process, Methods, and Algorithms 215

120,000 records and 77 variables. A k-fold cross- validation methodology was used in model build- ing, evaluation, and comparison. The results showed that support vector models are the most accurate predictor (with a test set accuracy of 92.85%) for this domain followed by artificial neural networks and decision trees. Furthermore, using a sensitivity– analysis-based evaluation method, the study also revealed novel patterns related to prognostic factors of prostate cancer.

In a related study, Delen, Walker, and Kadam (2005) used two data mining algorithms (artificial neural networks and decision trees) and logistic regression to develop prediction models for breast cancer survival using a large data set (more than 200,000 cases). Using a 10-fold cross-validation method to measure the unbiased estimate of the prediction models for performance comparison pur- poses, the researchers determined that the results indicated that the decision tree (C5 algorithm) was the best predictor with 93.6 percent accuracy on the holdout sample (which was the best prediction accuracy reported in the literature) followed by arti- ficial neural networks with 91.2 percent accuracy, and logistic regression, with 89.2 percent accu- racy. Further analysis of prediction models revealed prioritized importance of the prognostic factors, which can then be used as a basis for further clinical and biological research studies.

In the most recent study, Zolbanin et al. (2015) studied the impact of comorbidity in cancer sur- vivability. Although prior research has shown that diagnostic and treatment recommendations might be altered based on the severity of comorbidities, chronic diseases are still being investigated in isola- tion from one another in most cases. To illustrate the significance of concurrent chronic diseases in the course of treatment, their study used the Surveillance, Epidemiology, and End Results (SEER) Program’s cancer data to create two comorbid data sets: one for breast and female genital cancers and another for prostate and urinal cancers. Several pop- ular machine-learning techniques are then applied to the resultant data sets to build predictive mod- els (see Figure 4.4). Comparison of the results has

shown that having more information about comor- bid conditions of patients can improve models’ pre- dictive power, which in turn can help practitioners make better diagnostic and treatment decisions. Therefore, the study suggested that proper identi- fication, recording, and use of patients’ comorbidity status can potentially lower treatment costs and ease the healthcare-related economic challenges.

These examples (among many others in the medical literature) show that advanced data min- ing techniques can be used to develop models that possess a high degree of predictive as well as explanatory power. Although data mining meth- ods are capable of extracting patterns and relation- ships hidden deep in large and complex medical databases, without the cooperation and feedback from medical experts, their results are not of much use. The patterns found via data mining methods should be evaluated by medical professionals who have years of experience in the problem domain to decide whether they are logical, actionable, and novel enough to warrant new research directions. In short, data mining is not meant to replace medi- cal professionals and researchers but to comple- ment their invaluable efforts to provide data-driven new research directions and to ultimately save more human lives.

Questions for Case 4.4

1. How can data mining be used for ultimately cur- ing illnesses like cancer?

2. What do you think are the promises and major challenges for data miners in contributing to medical and biological research endeavors?

Sources: H. M. Zolbanin, D. Delen, & A. H. Zadeh, “Predicting Overall Survivability in Comorbidity of Cancers: A Data Mining Approach,” Decision Support Systems, 74, 2015, pp. 150–161; D. Delen, “Analysis of Cancer Data: A Data Mining Approach,” Expert Systems, 26(1), 2009, pp. 100–112; J. Thongkam, G. Xu, Y. Zhang, & F. Huang, “Toward Breast Cancer Survivability Prediction Models Through Improving Training Space,” Expert Systems with Applications, 36(10), 2009, pp. 12200–12209; D. Delen, G. Walker, & A. Kadam, “Predicting Breast Cancer Survivability: A Comparison of Three Data Mining Methods,” Artificial Intelligence in Medicine, 34(2), 2005, pp. 113–127.

(Continued )

216 Part II • Predictive Analytics/Machine Learning

Training and calibrating the

model

Testing the model

Artificial Neural Networks (ANN)

Tabulated Model Testing Results

(Accuracy, Sensitivity, and

Specificity)

Partitioned data (training &

testing)

Partitioned data (training

& testing)

Training and calibrating the

model

Testing the model

Logistic Regression (LR)

Training and calibrating the

model

Testing the model

Random Forest (RF)

Assess variable

importance

Tabulated Relative Variable

Importance Results

Data Preprocessing Cleaning Selecting Transforming

Cancer DB 1 Cancer DB 2 Cancer DB n

Combined Cancer DB

Partitioned data (training

& testing)

FIGURE 4.4 Data Mining Methodology for Investigation of Comorbidity in Cancer Survivability.

Depending on the business need, the data mining task can be of a prediction (either classification or regression), an association, or a clustering type. Each of these data min- ing tasks can use a variety of data mining methods and algorithms. Some of these data mining methods were explained earlier in this chapter, and some of the most popular

Application Case 4.4 (Continued)

Chapter 4 • Data Mining Process, Methods, and Algorithms 217

algorithms, including decision trees for classification, k-means for clustering, and the Apriori algorithm for association rule mining, are described later in this chapter.

Step 5: Testing and Evaluation

In step 5, the developed models are assessed and evaluated for their accuracy and gen- erality. This step assesses the degree to which the selected model (or models) meets the business objectives and, if so, to what extent (i.e., Do more models need to be developed and assessed?). Another option is to test the developed model(s) in a real- world scenario if time and budget constraints permit. Even though the outcome of the developed models is expected to relate to the original business objectives, other find- ings that are not necessarily related to the original business objectives but that might also unveil additional information or hints for future directions often are discovered.

The testing and evaluation step is a critical and challenging task. No value is added by the data mining task until the business value obtained from discovered knowledge patterns is identified and recognized. Determining the business value from discovered knowledge patterns is somewhat similar to playing with puzzles. The extracted knowledge patterns are pieces of the puzzle that need to be put together in the context of the specific business purpose. The success of this identification operation depends on the interaction among data analysts, business analysts, and decision makers (such as business managers). Because data analysts might not have the full understanding of the data mining objectives and what they mean to the business and the business analysts, and decision makers might not have the technical knowledge to interpret the results of sophisticated mathematical solutions, interaction among them is necessary. To properly interpret knowledge patterns, it is often necessary to use a variety of tabulation and visualization techniques (e.g., pivot tables, cross-tabulation of findings, pie charts, histograms, box plots, scatterplots).

Step 6: Deployment

Development and assessment of the models is not the end of the data mining project. Even if the purpose of the model is to have a simple exploration of the data, the knowl- edge gained from such exploration will need to be organized and presented in a way that the end user can understand and benefit from. Depending on the requirements, the de- ployment phase can be as simple as generating a report or as complex as implementing a repeatable data mining process across the enterprise. In many cases, it is the customer, not the data analyst, who carries out the deployment steps. However, even if the analyst will not carry out the deployment effort, it is important for the customer to understand up front what actions need to be carried out to actually make use of the created models.

The deployment step can also include maintenance activities for the deployed models. Because everything about the business is constantly changing, the data that reflect the busi- ness activities also are changing. Over time, the models (and the patterns embedded within them) built on the old data can become obsolete, irrelevant, or misleading. Therefore, moni- toring and maintenance of the models are important if the data mining results are to become a part of the day-to-day business and its environment. A careful preparation of a maintenance strategy helps avoid unnecessarily long periods of incorrect usage of data mining results. To monitor the deployment of the data mining result(s), the project needs a detailed plan on the monitoring process, which might not be a trivial task for complex data mining models.

Other Data Mining Standardized Processes and Methodologies

To be applied successfully, a data mining study must be viewed as a process that follows a standardized methodology rather than as a set of automated software tools and tech- niques. In addition to CRISP-DM, there is another well-known methodology developed

218 Part II • Predictive Analytics/Machine Learning

by the SAS Institute, called SEMMA (2009). The acronym SEMMA stands for “sample, explore, modify, model, and assess.”

Beginning with a statistically representative sample of the data, SEMMA makes it easy to apply exploratory statistical and visualization techniques, select and transform the most significant predictive variables, model the variables to predict outcomes, and confirm a model’s accuracy. A pictorial representation of SEMMA is given in Figure 4.5.

By assessing the outcome of each stage in the SEMMA process, the model devel- oper can determine how to model new questions raised by the previous results and thus proceed back to the exploration phase for additional refinement of the data; that is, as with CRISP-DM, SEMMA is driven by a highly iterative experimentation cycle. The main difference between CRISP-DM and SEMMA is that CRISP-DM takes a more comprehensive approach—including understanding of the business and the relevant data—to data mining projects whereas SEMMA implicitly assumes that the data mining project’s goals and objectives along with the appropriate data sources have been identi- fied and understood.

Some practitioners commonly use the term knowledge discovery in databases (KDD) as a synonym for data mining. Fayyad et al. (1996) defined knowledge discov- ery in databases as a process of using data mining methods to find useful information and patterns in the data as opposed to data mining, which involves using algorithms to identify patterns in data derived through the KDD process (see Figure 4.6). KDD is a comprehensive process that encompasses data mining. The input to the KDD pro- cess consists of organizational data. The enterprise data warehouse enables KDD to be implemented efficiently because it provides a single source for data to be mined. Dunham (2003) summarized the KDD process as consisting of the following steps: data selection, data preprocessing, data transformation, data mining, and interpretation/ evaluation.

Figure 4.7 shows the polling results for the question, “What main methodology are you using for data mining?” (conducted by KDnuggets.com in August 2007).

SEMMA

Feedback

Sample (Generate a representative

sample of the data.)

Explore (Visualize and provide a

basic description of the data.)

Assess (Evaluate the accuracy and usefulness of the models.)

Modify (Select variables, transform variable representations.)

Model (Use a variety of statistical

and machine learning models.)

FIGURE 4.5 SEMMA Data Mining Process.

Chapter 4 • Data Mining Process, Methods, and Algorithms 219

Target Data

Preprocessed Data

1 2 3 4 5

Transformed Data

Extracted Patterns

Knowledge “Actionable

Insight”

Data Selection

Data Cleaning

Data Transformation

Data Mining

Internalization

Feedback

Sources for Raw Data

FIGURE 4.6 KDD (Knowledge Discovery in Databases) Process.

CRISP-DM

My own

SEMMA

KDD Process

Domain-specific methodology

None

Other methodology (not domain specific)

My organization’s

0 10 20 30 40 50 60 70

FIGURE 4.7 Ranking of Data Mining Methodologies/Processes. Source: Used with permission from KDnuggets.com.

220 Part II • Predictive Analytics/Machine Learning

uSECTION 4.4 REVIEW QUESTIONS

1. What are the major data mining processes? 2. Why do you think the early phases (understanding of the business and understand-

ing of the data) take the longest amount of time in data mining projects?

3. List and briefly define the phases in the CRISP-DM process. 4. What are the main data-preprocessing steps? Briefly describe each step and provide

relevant examples.

5. How does CRISP-DM differ from SEMMA?

4.5 DATA MINING METHODS

Various methods are available for performing data mining studies, including classification, regression, clustering, and association. Most data mining software tools employ more than one technique (or algorithm) for each of these methods. This section describes the most popular data mining methods and explains their representative techniques.

Classification

Classification is perhaps the most frequently used data mining method for real-world problems. As a popular member of the machine-learning family of techniques, classifica- tion learns patterns from past data (a set of information—traits, variables, features—on characteristics of the previously labeled items, objects, or events) to place new instances (with unknown labels) into their respective groups or classes. For example, one could use classification to predict whether the weather on a particular day will be “sunny,” “rainy,” or “cloudy.” Popular classification tasks include credit approval (i.e., good or bad credit risk), store location (e.g., good, moderate, bad), target marketing (e.g., likely customer, no hope), fraud detection (i.e., yes/no), and telecommunication (e.g., likely to turn to another phone company, yes/no). If what is being predicted is a class label (e.g., “sunny,” “rainy,” or “cloudy”), the prediction problem is called a classification; if it is a numeric value (e.g., temperature, such as 68°F), the prediction problem is called a regression.

Even though clustering (another popular data mining method) can also be used to determine groups (or class memberships) of things, there is a significant difference between the two. Classification learns the function between the characteristics of things (i.e., independent variables) and their membership (i.e., output variable) through a su- pervised learning process in which both types (input and output) of variables are pre- sented to the algorithm; in clustering, the membership of the objects is learned through an unsupervised learning process by which only the input variables are presented to the algorithm. Unlike classification, clustering does not have a supervising (or controlling) mechanism that enforces the learning process; instead, clustering algorithms use one or more heuristics (e.g., multidimensional distance measure) to discover natural groupings of objects.

The most common two-step methodology of classification-type prediction involves model development/training and model testing/deployment. In the model development phase, a collection of input data, including the actual class labels, is used. After a model has been trained, the model is tested against the holdout sample for accuracy assessment and eventually deployed for actual use where it is to predict classes of new data instances (where the class label is unknown). Several factors are considered in assessing the model, including the following.

• Predictive accuracy. The model’s ability to correctly predict the class label of new or previously unseen data. Prediction accuracy is the most commonly used assessment factor for classification models. To compute this measure, actual class

Chapter 4 • Data Mining Process, Methods, and Algorithms 221

labels of a test data set are matched against the class labels predicted by the model. The accuracy can then be computed as the accuracy rate, which is the percentage of test data set samples correctly classified by the model (more on this topic is pro- vided later in the chapter).

• Speed. The computational costs involved in generating and using the model where faster is deemed to be better.

• Robustness. The model’s ability to make reasonably accurate predictions given noisy data or data with missing and erroneous values.

• Scalability. The ability to construct a prediction model efficiently given a rather large amount of data.

• Interpretability. The level of understanding and insight provided by the model (e.g., how and/or what the model concludes on certain predictions).

Estimating the True Accuracy of Classification Models

In classification problems, the primary source for accuracy estimation is the confusion matrix (also called a classification matrix or a contingency table). Figure 4.8 shows a confusion matrix for a two-class classification problem. The numbers along the diagonal from the upper left to the lower right represent correct decisions, and the numbers out- side this diagonal represent the errors.

Table 4.1 provides equations for common accuracy metrics for classification models. When the classification problem is not binary, the confusion matrix gets bigger (a

square matrix with the size of the unique number of class labels), and accuracy metrics become limited to per class accuracy rates and the overall classifier accuracy.

(True Classification Rate)i = (True Classification)

a n

i = 1 1False Classification2

(Overall Classifier Accuracy)i = a n

i = 1 (Ture Classification)i

Total Number of Cases

Estimating the accuracy of a classification model (or classifier) induced by a super- vised learning algorithm is important for the following two reasons: First, it can be used

True Positive (TP)

Count

False Negative (FN)

Count

True Negative (TN)

Count

False Positive (FP)

Count

Positive

True/Observed Class

N e g a ti

ve P

o s it

iv e

Negative

P re

d ic

te d C

la s s

FIGURE 4.8 Simple Confusion Matrix for Tabulation of Two-Class Classification Results.

222 Part II • Predictive Analytics/Machine Learning

to estimate its future prediction accuracy, which could imply the level of confidence one should have in the classifier’s output in the prediction system. Second, it can be used for choosing a classifier from a given set (identifying the “best” classification model among the many trained). The following are among the most popular estimation methodologies used for classification-type data mining models.

SIMPLE SPLIT The simple split (or holdout or test sample estimation) partitions the data into two mutually exclusive subsets called a training set and a test set (or holdout set). It is common to designate two-thirds of the data as the training set and the remaining one- third as the test set. The training set is used by the inducer (model builder), and the built classifier is then tested on the test set. An exception to this rule occurs when the classifier is an artificial neural network. In this case, the data are partitioned into three mutually ex- clusive subsets: training, validation, and testing. The validation set is used during model building to prevent overfitting. Figure 4.9 shows the simple split methodology.

The main criticism of this method is that it makes the assumption that the data in the two subsets are of the same kind (i.e., have the exact same properties). Because this is a simple random partitioning, in most realistic data sets where the data are skewed on the classification variable, such an assumption might not hold true. To improve this

TABLE 4.1 Common Accuracy Metrics for Classification Models

Metric Description

Accuracy = TP + TN

TP + TN + FP + FN The ratio of correctly classified instances (positives and negatives) divided by the total numbers of instances

True Positive Rate = TP

TP + FN (a.k.a. Sensitivity) The ratio of correctly classified positives divided by the total positive count (i.e., hit rate or recall)

True Negative Rate = TN

TN + FP (a.k.a. Specificity) The ratio of correctly classified negatives divided by the total negative count (i.e., false alarm rate)

Precision = TP

TP + FP The ratio of correctly classified positives divided by the sum of correctly classified positives and incorrectly classified positives

Recall = TP

TP + FN Ratio of correctly classified positives divided by the sum of correctly classified positives and incorrectly classified negatives

Preprocessed Data

Training Data

Model Development

Model Assessment (scoring)

Testing Data

1/3

2/3

Trained Classifier

Prediction Accuracy

FN TN

TP FP

FIGURE 4.9 Simple Random Data Splitting.

Chapter 4 • Data Mining Process, Methods, and Algorithms 223

situation, stratified sampling is suggested, where the strata become the output variable. Even though this is an improvement over the simple split, it still has a bias associated from the single random partitioning.

K-FOLD CROSS-VALIDATION To minimize the bias associated with the random sampling of the training and holdout data samples in comparing the predictive accuracy of two or more methods, one can use a methodology called k-fold cross-validation. In k-fold cross-validation, also called rotation estimation, the complete data set is randomly split into k mutually exclusive subsets of approximately equal size. The classification model is trained and tested k times. Each time it is trained on all but one fold and then tested on the remaining single fold. The cross-validation estimate of the overall accuracy of a model is calculated by simply averaging the k individual accuracy measures, as shown in the following equation:

CVA = 1

k a k

i = 1 Ai

where CVA stands for cross-validation accuracy, k is the number of folds used, and A is the accuracy measure (e.g., hit rate, sensitivity, specificity) of each fold. Figure 4.10 shows a graphical illustration of k-fold cross-validation where k is set to 10.

ADDITIONAL CLASSIFICATION ASSESSMENT METHODOLOGIES Other popular assessment methodologies include the following:

• Leave one out. The leave-one-out method is similar to the k-fold cross-validation where the k takes the value of 1; that is, every data point is used for testing once as many models are developed as there are data points. This is a time-consuming methodology, but sometimes for small data sets, it is a viable option.

• Bootstrapping. With bootstrapping, a fixed number of instances from the origi- nal data are sampled (with replacement) for training, and the rest of the data set is used for testing. This process is repeated as many times as desired.

• Jackknifing. Though similar to the leave-one-out methodology, with jackknifing, the accuracy is calculated by leaving one sample out at each iteration of the estima- tion process.

• Area under the ROC curve. The area under the ROC curve is a graphical assessment technique that plots the true positive rate on the y-axis and the false positive rate on the x-axis. The area under the ROC curve determines the accuracy

...

10% 10%10% 10%

10%10%

10%10%

10%10% 10%

10% 10% 10%

10%10%

10%10%

10%10% 10%

10%

10%

10%10%

10%10%

10%10% 10% Repeated for

all 10 folds

FIGURE 4.10 Graphical Depiction of k-Fold Cross-Validation.

224 Part II • Predictive Analytics/Machine Learning

measure of a classifier: A value of 1 indicates a perfect classifier; 0.5 indicates no better than random chance; in reality, the values would range between the two extreme cases. For example, in Figure 4.11, A has a better classification performance than B, whereas C is not any better than the random chance of flipping a coin.

Estimating the Relative Importance of Predictor Variables

Data mining methods (i.e., machine-learning algorithms) are really good at capturing complex relationships between input and output variables (producing very accurate prediction models) but are not nearly as good at explaining how they do what they do (i.e., model transparency). To mitigate this deficiency (also called the black-box syn- drome), the machine-learning community proposed several methods, most of which are characterized as sensitivity analysis. In the context of predictive modeling, sensitivity analysis refers to an exclusive experimentation process aimed at discovering the cause- and-effect relationship between the input variables and output variable. Some of the variable importance methods are algorithm specific (i.e., applied to decision trees) and some are algorithm agnostic. Here are the most commonly used variable importance methods employed in machine learning and predictive modeling:

1. Developing and observing a well-trained decision tree model to see the relative dis- cernibility of the input variables—the closer to the root of the tree a variable is used to split, the greater is its importance/relative-contribution to the prediction model.

2. Developing and observing a rich and large random forest model and assessing the variable split statistics. If the ratio of a given variable’s selection into candidate counts (i.e., number of times a variable selected as the level-0 splitter is divided by number of times it was picked randomly as one of split candidates) is larger, its importance/ relative-contribution is also greater.

10.90.80.70.60.50.40.30.20.10 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1

0.9

0.8

False Positive Rate (1-Specificity)

T ru

e P

o s it

iv e R

a te

( S

e n s it

iv it

y) A

B

C

FIGURE 4.11 Sample ROC Curve.

Chapter 4 • Data Mining Process, Methods, and Algorithms 225

3. Sensitivity analysis based on input value perturbation by which the input variables are gradually changed/perturbed one at a time and the relative change in the output is observed—the larger the change in the output, the greater the importance of the perturbed variable. This method is often used in feed-forward neural network mod- eling when all of the variables are numeric and standardized/normalized. Because this method is covered in Chapter 6 within the context of deep learning and deep neural networks, it is not be explained here.

4. Sensitivity analysis based on leave-one-out methodology. This method can be used for any type of predictive analytics method and therefore is further explained as follows.

The sensitivity analysis (based on leave-one-out methodology) relies on the experi- mental process of systematically removing input variables, one at a time, from the input variable set, developing and testing a model, and observing the impact of the absence of this variable on the predictive performance of the machine-learning model. The model is trained and tested (often using a k-fold cross validation) for each input variable (i.e., its absence in the input variable collection) to measure its contribution/importance to the model. A graphical depiction of the process is shown in Figure 4.12.

This method is often used for support vector machines, decision trees, logistic re- gression, and artificial neural networks. In his sensitivity analysis book, Saltelli (2002) formalized the algebraic representation of this measurement process:

Si = Vi

V (Ft) =

V (E(Ft|Xi))

V (Ft)

In the denominator of the equation, V(Ft) refers to the variance in the output vari- able. In the numerator, V(E(Ft|Xi)), E is the expectation operator to call for an integral over parameter Xi; that is, inclusive of all input variables except Xi, the V, the variance operator, applies a further integral over Xi. The variable contribution (i.e., importance), represented as Si for the i

th variable, is calculated as the normalized sensitivity mea- sure. In a later study, Saltelli et al. (2004) proved that this equation is the most probable

Systematically Perturb, Add/ Remove Inputs

Trained ML Model – “the black-box” Observed Change in Outputs

w∙ x–

b= –1

w∙ x–

b= 0

w∙ x–

b= 1

2 w

–6 –4X1

X2

–2 0 2

1

0.5

0 4 6

Im p o rt

a n c e

Variable Names

100 90

80 70

60 50 50

40 30

40

1

2 3

7654

8 9 10 11 12 13 14 15

D1

FIGURE 4.12 Graphical Depiction of the Sensitivity Analysis Process.

226 Part II • Predictive Analytics/Machine Learning

measure of model sensitivity that is capable of ranking input variables (i.e., the predic- tors) in the order of importance for any combination of interactions including the non- orthogonal relationships among the input variables. To properly combine the sensitivity analysis results for several prediction methods, one can use an information fusion–based methodology, particularly by modifying the preceding equation in such a way that the sensitivity measure of an input variable n obtained based on the information combined (i.e., fused) from m number of prediction models. The following equation represents this weighted summation function.

Sn(fused ) = a m

i = 1 viSin = v1S1n + v2S2n + ... + vmSmn

In this equation, vi represents the normalized contribution/weight for each predic- tion model in which the level of contribution/weight of a model is calculated as a func- tion of its relative predictive power—the larger the prediction power (i.e., accuracy) is, the higher is the value of v.

CLASSIFICATION TECHNIQUES A number of techniques (or algorithms) are used for clas- sification modeling, including the following:

• Decision tree analysis. Decision tree analysis (a machine-learning technique) is arguably the most popular classification technique in the data mining arena. A detailed description of this technique is given in the following section.

• Statistical analysis. Statistical techniques were the primary classification algo- rithm for many years until the emergence of machine-learning techniques. Statistical classification techniques include logistic regression and discriminant analysis, both of which make the assumptions that the relationships between the input and output variables are linear in nature, the data are normally distributed, and the variables are not correlated and are independent of each other. The questionable nature of these assumptions has led to the shift toward machine-learning techniques.

• Neural networks. These are among the most popular machine-learning tech- niques that can be used for classification-type problems.

• Case-based reasoning. This approach uses historical cases to recognize com- monalities to assign a new case into the most probable category.

• Bayesian classifiers. This approach uses probability theory to build classifi- cation models based on the past occurrences that are capable of placing a new instance into a most probable class (or category).

• Genetic algorithms. This is the use of the analogy of natural evolution to build directed-search-based mechanisms to classify data samples.

• Rough sets. This method takes into account the partial membership of class labels to predefined categories in building models (collection of rules) for classifica- tion problems.

A complete description of all of these classification techniques is beyond the scope of this book; thus, only several of the most popular ones are presented here.

DECISION TREES Before describing the details of decision trees, we need to discuss some simple terminology. First, decision trees include many input variables that might have an impact on the classification of different patterns. These input variables are usu- ally called attributes. For example, if we were to build a model to classify loan risks on the basis of just two characteristics—income and a credit rating—these two characteristics would be the attributes, and the resulting output would be the class label (e.g., low, me- dium, or high risk). Second, a tree consists of branches and nodes. A branch represents

Chapter 4 • Data Mining Process, Methods, and Algorithms 227

the outcome of a test to classify a pattern using one of the attributes. A leaf node at the end represents the final class choice for a pattern (a chain of branches from the root node to the leaf node, which can be represented as a complex if-then statement).

The basic idea behind a decision tree is that it recursively divides a training set until each division consists entirely or primarily of examples from one class. Each nonleaf node of the tree contains a split point, which is a test of one or more attributes and deter- mines how the data are to be divided further. Decision tree algorithms, in general, build an initial tree from training data such that each leaf node is pure, and they then prune the tree to increase its generalization, and hence, the prediction accuracy of test data.

In the growth phase, the tree is built by recursively dividing the data until each divi- sion is either pure (i.e., contains members of the same class) or relatively small. The basic idea is to ask questions whose answers would provide the most information, similar to what we do when playing the game “Twenty Questions.”

The split used to partition the data depends on the type of the attribute used in the split. For a continuous attribute A, splits are of the form value(A) 6 x, where x is some “optimal” split value of A. For example, the split based on income could be “Income 6 50,000.” For the categorical attribute A, splits are of the form that value(A) belongs to x where x is a subset of A. As an example, the split could be on the basis of gender: “Male versus Female.”

A general algorithm for building a decision tree is as follows:

1. Create a root node and assign all of the training data to it. 2. Select the best splitting attribute. 3. Add a branch to the root node for each value of the split. Split the data into mutually

exclusive (nonoverlapping) subsets along the lines of the specific split and move to the branches.

4. Repeat steps 2 and 3 for each and every leaf node until a stopping criterion is reached (e.g., the node is dominated by a single class label).

Many different algorithms have been proposed for creating decision trees. These algorithms differ primarily in terms of the way in which they determine the splitting attri- bute (and its split values), the order of splitting the attributes (splitting the same attribute only once or many times), the number of splits at each node (binary versus ternary), the stopping criteria, and the pruning of the tree (pre- versus postpruning). Some of the most well-known algorithms are ID3 (followed by C4.5 and C5 as the improved versions of ID3) from machine learning, classification and regression trees (CART) from statistics, and the chi-squared automatic interaction detector (CHAID) from pattern recognition.

When building a decision tree, the goal at each node is to determine the attribute and the split point of that attribute that best divides the training records to purify the class representation at that node. To evaluate the goodness of the split, some splitting indices have been proposed. Two of the most common ones are the Gini index and information gain. The Gini index is used in CART and Scalable PaRallelizable INduction of Decision Trees (SPRINT) algorithms. Versions of information gain are used in ID3 (and its newer versions, C4.5 and C5).

The Gini index has been used in economics to measure the diversity of a popu- lation. The same concept can be used to determine the purity of a specific class as the result of a decision to branch along a particular attribute or variable. The best split is the one that increases the purity of the sets resulting from a proposed split. Let us briefly look into a simple calculation of the Gini index.

If a data set S contains examples from n classes, the Gini index is defined as

gini(S) = 1 - a n

j = 1 p2j

228 Part II • Predictive Analytics/Machine Learning

where pj is a relative frequency of class j in S. If a data set S is split into two subsets, S1 and S2 with sizes N1 and N2, respectively, the Gini index of the split data contains examples from n classes, and the Gini index is defined as

ginisplit(S) = N1 N

gini(S1) + N2 N

gini(S2)

The attribute/split combination that provides the smallest ginisplit(S) is chosen to split the node. In such a determination, one should enumerate all possible splitting points for each attribute.

Information gain is the splitting mechanism used in ID3, which is perhaps the most widely known decision tree algorithm. It was developed by Ross Quinlan in 1986, and since then, he has evolved this algorithm into the C4.5 and C5 algorithms. The basic idea behind ID3 (and its variants) is to use a concept called entropy in place of the Gini index. Entropy measures the extent of uncertainty or randomness in a data set. If all the data in a subset belong to just one class, there is no uncertainty or randomness in that data set, so the entropy is zero. The objective of this approach is to build subtrees so that the entropy of each final subset is zero (or close to zero). Let us also look at the calcula- tion of the information gain.

Assume that there are two classes: P (positive) and N (negative). Let the set of ex- amples S contain p counts of class P and n counts of class N. The amount of information needed to decide if an arbitrary example in S belongs to P or N is defined as

I(p, n) = - p

p + n log2

p

p + n -

n

p + n log2

n

p + n

Assume that using attribute A, the set S will be partitioned into sets {S1, S2, . . ., g}. If Si contains pi examples of P and ni examples of N, the entropy, or the expected information needed to classify objects in all subtrees, Si, is

E(A) = a n

i = 1

pi + ni p + n

I(pi,ni)

Then the information that would be gained by branching on attribute A would be

Gain(A) = I(p,n) - E(A)

These calculations are repeated for each and every attribute, and the one with the highest information gain is selected as the splitting attribute. The basic ideas behind these splitting indices are rather similar, but the specific algorithmic details vary. A detailed definition of the ID3 algorithm and its splitting mechanism can be found in Quinlan (1986).

Application Case 4.5 illustrates how significant the gains can be if the right data min- ing techniques are used for a well-defined business problem.

Cluster Analysis for Data Mining

Cluster analysis is an essential data mining method for classifying items, events, or con- cepts into common groupings called clusters. The method is commonly used in biology, medicine, genetics, social network analysis, anthropology, archaeology, astronomy, char- acter recognition, and even in management information systems (MIS) development. As data mining has increased in popularity, its underlying techniques have been applied to business, especially to marketing. Cluster analysis has been used extensively for fraud

Chapter 4 • Data Mining Process, Methods, and Algorithms 229

Influence Health, Inc. provides the healthcare industry’s only integrated digital consumer engage- ment and activation platform. It enables providers, employers, and payers to positively influence con- sumer decision making and health behaviors well beyond the physical care setting through personal- ized and interactive multichannel engagement. Since 1996, the Birmingham, Alabama–based company has helped more than 1,100 provider organizations influence consumers in a way that transforms finan- cial and quality outcomes.

Healthcare is a personal business. Each patient’s needs are different and require an indi- vidual response. On the other hand—as the cost of providing healthcare services continues to rise— hospitals and health systems increasingly need to harness economies of scale by catering to larger and larger populations. The challenge then becomes to provide a personalized approach while operat- ing on a large scale. Influence Health specializes in helping its healthcare sector clients solve this chal- lenge by getting to know their existing and poten- tial patients better and targeting each individual with the appropriate health services at the right time. Advanced predictive analytics technology from IBM allows Influence Health to help its clients discover the factors that have the most influence on patients’ healthcare decisions. By assessing the propensity of hundreds of millions of prospects to require spe- cific healthcare services, Influence Health is able to boost revenues and response rates for healthcare campaigns, improving outcomes for its clients and their patients alike.

Targeting the Savvy Consumer

Today’s healthcare industry is becoming more com- petitive than ever before. If the use of an organiza- tion’s services drops, so do its profits. Rather than simply seeking out the nearest hospital or clinic, consumers are now more likely to make positive choices among healthcare providers. Paralleling efforts that are common in other industries, health- care organizations must make increased efforts to market themselves effectively to both existing and potential patients, building long-term engagement and loyalty.

The keys to successful healthcare marketing are timeliness and relevance. If you can predict what kind of health services an individual prospect might need, you can engage and influence her or him much more effectively for wellness care.

Venky Ravirala, chief analytics officer at Influence Health, explains, “Healthcare organiza- tions risk losing people’s attention if they bombard them with irrelevant messaging. We help our clients avoid this risk by using analytics to segment their existing and potential prospects and market to them in a much more personal and relevant way.”

Faster and More Flexible Analytics

As its client base has expanded, the total volume of data in Influence Health’s analytics systems has grown to include over 195 million patient records with a detailed disease encounter history for several million patients. Ravirala comments, “With so much data to analyze, our existing method of scoring data was becoming too complex and time-consuming. We wanted to be able to extract insights at greater speed and accuracy.”

By leveraging predictive analytics software from IBM, Influence Health is now able to develop models that calculate how likely each patient is to require particular services and express this likelihood as a percentage score. Microsegmentation and numerous disease-specific models draw on demographic, socio- economic, geographical, behavioral, disease history, and census data and examine different aspects of each patient’s predicted healthcare needs.

“The IBM solution allows us to combine all these models using an ensemble technique, which helps to overcome the limitations of individual mod- els and provide more accurate results,” comments Venky Ravirala, chief analytics officer at Influence Health. “It gives us the flexibility to apply multiple techniques to solve a problem and arrive at the best solution. It also automates much of the analytics process, enabling us to respond to clients’ requests faster than before, and often give them a much deeper level of insight into their patient population.”

For example, Influence Health decided to find out how disease prevalence and risk vary between different cohorts within the general population. By

Application Case 4.5 Influence Health Uses Advanced Predictive Analytics to Focus on the Factors That Really Influence People’s Healthcare Decisions

(Continued )

230 Part II • Predictive Analytics/Machine Learning

detection (both credit card and e-commerce) and market segmentation of customers in contemporary CRM systems. More applications in business continue to be developed as the strength of cluster analysis is recognized and used.

Cluster analysis is an exploratory data analysis tool for solving classification prob- lems. The objective is to sort cases (e.g., people, things, events) into groups, or clusters, so that the degree of association is strong among members of the same cluster and weak among members of different clusters. Each cluster describes the class to which its mem- bers belong. An obvious one-dimensional example of cluster analysis is to establish score ranges into which to assign class grades for a college class. This is similar to the cluster analysis problem that the U.S. Treasury faced when establishing new tax brackets in the 1980s. A fictional example of clustering occurs in J. K. Rowling’s Harry Potter books. The Sorting Hat determines to which House (e.g., dormitory) to assign first-year students at the Hogwarts School. Another example involves determining how to seat guests at a wed- ding. As far as data mining goes, the importance of cluster analysis is that it can reveal

using very sophisticated cluster analysis techniques, the company was able to discover new comorbidity patterns that improve risk predictability for over 100 common diseases by up to 800 percent.

This helps to reliably differentiate between high-risk and very high-risk patients—making it eas- ier to target campaigns at the patients and prospects who need them most. With insights like these in hand, Influence Health is able to use its healthcare marketing expertise to advise its clients on how best to allocate marketing resources.

“Our clients make significant budgeting deci- sions based on the guidance we give them,” states Ravirala. “We help them maximize the impact of one-off campaigns—such as health insurance mar- ketplace campaigns when Obamacare began—as well as their long-term strategic plans and ongoing marketing communications.”

Reaching the Right Audience

By enabling its clients to target their marketing activi- ties more effectively, Influence Health is helping to drive increased revenue and enhance population health. “Working with us, clients have been able to achieve return on investment of up to 12 to 1 through better targeted marketing,” elaborates Ravirala. “And it’s not just about revenues: by ensuring that vital healthcare information gets sent to the people who need it, we are helping our clients improve general health levels in the communities they serve.”

Influence Health continues to refine its modeling techniques, gaining an ever-deeper understanding of the critical attributes that influence healthcare deci- sions. With a flexible analytics toolset at its fingertips, the company is well equipped to keep improving its service to clients. Ravirala explains, “In the future, we want to take our understanding of patient and prospect data to the next level, identifying patterns in behavior and incorporating analysis with machine- learning libraries. IBM SPSS has already given us the ability to apply and combine multiple models without writing a single line of code. We’re eager to further leverage this IBM solution as we expand our health- care analytics to support clinical outcomes and popu- lation health management services.”

“We are achieving analytics on an unprec- edented scale. Today, we can analyze 195 million records with 35 different models in less than two days—a task which was simply not possible for us in the past,” says Ravirala.

Questions for Case 4.5

1. What does Influence Health do?

2. What were the company’s challenges, proposed solutions, and obtained results?

3. How can data mining help companies in the healthcare industry (in ways other than the ones mentioned in this case)?

Source: Reprint Courtesy of International Business Machines Corporation, © (2018) International Business Machines Corporation.

Application Case 4.5 (Continued)

Chapter 4 • Data Mining Process, Methods, and Algorithms 231

associations and structures in data that were not previously apparent but are sensible and useful once found.

Cluster analysis results can be used to

• Identify a classification scheme (e.g., types of customers). • Suggest statistical models to describe populations. • Indicate rules for assigning new cases to classes for identification, targeting, and

diagnostic purposes. • Provide measures of definition, size, and change in what were previously broad

concepts. • Find typical cases to label and represent classes. • Decrease the size and complexity of the problem space for other data mining

methods. • Identify outliers in a specific domain (e.g., rare-event detection).

DETERMINING THE OPTIMAL NUMBER OF CLUSTERS Clustering algorithms usually re- quire one to specify the number of clusters to find. If this number is not known from prior knowledge, it should be chosen in some way. Unfortunately, there is not an op- timal way to calculate what this number is supposed to be. Therefore, several different heuristic methods have been proposed. The following are among the most commonly referenced ones:

• Look at the percentage of variance explained as a function of the number of clus- ters; that is, choose a number of clusters so that adding another cluster would not give much better modeling of the data. Specifically, if one graphs the percentage of variance explained by the clusters, there is a point at which the marginal gain will drop (giving an angle in the graph), indicating the number of clusters to be chosen.

• Set the number of clusters to (n>2)1>2, where n is the number of data points. • Use the Akaike information criterion (AIC), which is a measure of the goodness of

fit (based on the concept of entropy), to determine the number of clusters. • Use Bayesian information criterion, a model-selection criterion (based on maximum

likelihood estimation), to determine the number of clusters.

ANALYSIS METHODS Cluster analysis might be based on one or more of the following general methods:

• Statistical methods (including both hierarchical and nonhierarchical), such as k- means or k-modes.

• Neural networks (with the architecture called self-organizing map). • Fuzzy logic (e.g., fuzzy c-means algorithm). • Genetic algorithms.

Each of these methods generally works with one of two general method classes:

• Divisive. With divisive classes, all items start in one cluster and are broken apart. • Agglomerative. With agglomerative classes, all items start in individual clusters,

and the clusters are joined together.

Most cluster analysis methods involve the use of a distance measure to calculate the closeness between pairs of items. Popular distance measures include Euclidian dis- tance (the ordinary distance between two points that one would measure with a ruler) and Manhattan distance (also called the rectilinear distance or taxicab distance) between two points. Often, they are based on true distances that are measured, but this need not be so, as is typically the case in IS development. Weighted averages can be used to es- tablish these distances. For example, in an IS development project, individual modules of

232 Part II • Predictive Analytics/Machine Learning

the system can be related by the similarity between their inputs, outputs, processes, and the specific data used. These factors are then aggregated, pairwise by item, into a single distance measure.

K-MEANS CLUSTERING ALGORITHM The k-means algorithm (where k stands for the pre- determined number of clusters) is arguably the most referenced clustering algorithm. It has its roots in traditional statistical analysis. As the name implies, the algorithm assigns each data point (customer, event, object, etc.) to the cluster whose center (also called the cen- troid) is the nearest. The center is calculated as the average of all the points in the cluster; that is, its coordinates are the arithmetic mean for each dimension separately over all the points in the cluster. The algorithm steps follow and are shown graphically in Figure 4.13:

Initialization step: Choose the number of clusters (i.e., the value of K ).

Step 1: Randomly generate k random points as initial cluster centers. Step 2: Assign each point to the nearest cluster center. Step 3: Recompute the new cluster centers.

Repetition step: Repeat steps 2 and 3 until some convergence criterion is met (usually that the assignment of points to clusters becomes stable).

Association Rule Mining

Association rule mining (also known as affinity analysis or market-basket analysis) is a popular data mining method that is commonly used as an example to explain what data mining is and what it can do to a technologically less-savvy audience. Most of you might have heard the famous (or infamous, depending on how you look at it) relationship dis- covered between the sales of beer and diapers at grocery stores. As the story goes, a large supermarket chain (maybe Walmart, maybe not; there is no consensus on which super- market chain it was) did an analysis of customers’ buying habits and found a statistically significant correlation between purchases of beer and purchases of diapers. It was theo- rized that the reason for this was that fathers (presumably young men) were stopping off at the supermarket to buy diapers for their babies (especially on Thursdays), and because they could no longer go down to the sports bar as often, would buy beer as well. As a result of this finding, the supermarket chain is alleged to have placed the diapers next to the beer, resulting in increased sales of both.

In essence, association rule mining aims to find interesting relationships (affinities) between variables (items) in large databases. Because of its successful application to retail business problems, it is commonly called market-basket analysis. The main idea

Step 1 Step 2 Step 3

FIGURE 4.13 Graphical Illustration of the Steps in the k-Means Algorithm.

Chapter 4 • Data Mining Process, Methods, and Algorithms 233

in market-basket analysis is to identify strong relationships among different products (or services) that are usually purchased together (show up in the same basket together, ei- ther a physical basket at a grocery store or a virtual basket at an e-commerce Web site). For example, 65 percent of those who buy comprehensive automobile insurance also buy health insurance; 80 percent of those who buy books online also buy music online; 60 percent of those who have high blood pressure and are overweight have high cho- lesterol; 70 percent of the customers who buy a laptop computer and virus protection software also buy extended service plans.

The input to market-basket analysis is the simple point-of-sale transaction data when a number of products and/or services purchased together (just like the content of a purchase receipt) are tabulated under a single transaction instance. The outcome of the analysis is invaluable information that can be used to better understand customer- purchase behavior to maximize the profit from business transactions. A business can take advantage of such knowledge by (1) putting the items next to each other to make it more convenient for the customers to pick them up together and not forget to buy one when buying the others (increasing sales volume), (2) promoting the items as a package—do not put one on sale if the other(s) are on sale, and (3) placing them apart from each other so that the customer has to walk the aisles to search for it, and by doing so, potentially seeing and buying other items.

Applications of market-basket analysis include cross-marketing, cross-selling, store design, catalog design, e-commerce site design, optimization of online advertising, prod- uct pricing, and sales/promotion configuration. In essence, market-basket analysis helps businesses infer customer needs and preferences from their purchase patterns. Outside the business realm, association rules are successfully used to discover relationships between symptoms and illnesses, diagnosis and patient characteristics and treatments (which can be used in a medical decision support system), and genes and their functions (which can be used in genomics projects), among others. Here are a few common areas and uses for association rule mining:

• Sales transactions: Combinations of retail products purchased together can be used to improve product placement on the sales floor (placing products that go to- gether in close proximity) and promotional pricing of products (not having promo- tions on both products that are often purchased together).

• Credit card transactions: Items purchased with a credit card provide insight into other products the customer is likely to purchase or fraudulent use of credit card numbers.

• Banking services: The sequential patterns of services used by customers (check- ing account followed by savings account) can be used to identify other services they might be interested in (investment account).

• Insurance service products: Bundles of insurance products bought by custom- ers (car insurance followed by home insurance) can be used to propose additional insurance products (life insurance), or unusual combinations of insurance claims can be a sign of fraud.

• Telecommunication services: Commonly purchased groups of options (e.g., call waiting, caller ID, three-way calling) help better structure product bundles to maximize revenue; the same is also applicable to multichannel telecom providers with phone, television, and Internet service offerings.

• Medical records: Certain combinations of conditions can indicate increased risk of various complications; or, certain treatment procedures at certain medical facili- ties can be tied to certain types of infections.

A good question to ask with respect to the patterns/relationships that association rule mining can discover is “Are all association rules interesting and useful?” To answer

234 Part II • Predictive Analytics/Machine Learning

such a question, association rule mining uses two common metrics: support, and con- fidence and lift. Before defining these terms, let’s get a little technical by showing what an association rule looks like:

X 1 Y 3Supp(%), Conf (%)4 {Laptop Computer, Antivirus software} 1 {Extended Service Plan} [30%, 70%]

Here, X (products and/or service—called the left-hand side, LHS, or antecedent) is associated with Y (products and/or service—called the right-hand side, RHS, or con- sequent). S is the support, and C is the confidence for this particular rule. Here are the simple formulas for Supp, Conf, and Lift.

Support = Supp(X 1 Y ) = Number of Baskets that contains both X and Y

Total Number of Baskets

Confidence = Conf (X 1 Y) = Supp (X 1 Y )

Supp (X)

Lift(X 1 Y ) = Conf (X 1 Y )

Expected Conf (X 1 Y ) =

S(X 1 Y ) S(X)

S(X) * S(Y )

S(X)

= S(X 1 Y )

S(X) * S(Y )

The support (S) of a collection of products is the measure of how often these products and/or services (i.e., LHS + RHS = Laptop Computer, Antivirus Software, and Extended Service Plan) appear together in the same transaction; that is, the proportion of transactions in the data set that contain all of the products and/or services mentioned in a specific rule. In this example, 30 percent of all transactions in the hypothetical store database had all three products present in a single sales ticket. The confidence of a rule is the measure of how often the products and/or services on the RHS (consequent) go together with the products and/or services on the LHS (antecedent), that is, the propor- tion of transactions that include LHS while also including the RHS. In other words, it is the conditional probability of finding the RHS of the rule present in transactions where the LHS of the rule already exists. The lift value of an association rule is the ratio of the confidence of the rule and the expected confidence of the rule. The expected confidence of a rule is defined as the product of the support values of the LHS and the RHS divided by the support of the LHS.

Several algorithms are available for discovering association rules. Some well-known algorithms include Apriori, Eclat, and FP-Growth. These algorithms only do half the job, which is to identify the frequent itemsets in the database. Once the frequent itemsets are identified, they need to be converted into rules with antecedent and consequent parts. Determination of the rules from frequent itemsets is a straightforward matching process, but the process can be time consuming with large transaction databases. Even though there can be many items on each section of the rule, in practice the consequent part usu- ally contains a single item. In the following section, one of the most popular algorithms for identification of frequent itemsets is explained.

APRIORI ALGORITHM The Apriori algorithm is the most commonly used algorithm to discover association rules. Given a set of itemsets (e.g., sets of retail transactions, each listing individual items purchased), the algorithm attempts to find subsets that are common to at least a minimum number of the itemsets (i.e., complies with a minimum support). Apriori uses a bottom-up approach by which frequent subsets are extended

Chapter 4 • Data Mining Process, Methods, and Algorithms 235

one item at a time (a method known as candidate generation by which the size of frequent subsets increases from one-item subsets to two-item subsets, then three-item subsets, etc.), and groups of candidates at each level are tested against the data for minimum support. The algorithm terminates when no further successful extensions are found.

As an illustrative example, consider the following. A grocery store tracks sales trans- actions by SKU (stock-keeping unit) and thus knows which items are typically purchased together. The database of transactions along with the subsequent steps in identifying the frequent itemsets is shown in Figure 4.14. Each SKU in the transaction database cor- responds to a product, such as “1 = butter,” “2 = bread,” “3 = water,” and so on. The first step in Apriori is to count the frequencies (i.e., the supports) of each item (one-item itemsets). For this overly simplified example, let us set the minimum support to 3 (or 50, meaning an itemset is considered to be a frequent itemset if it shows up in at least 3 of 6 transactions in the database). Because all the one-item itemsets have at least 3 in the sup- port column, they are all considered frequent itemsets. However, had any of the one-item itemsets not been frequent, they would not have been included as a possible member of possible two-item pairs. In this way, Apriori prunes the tree of all possible itemsets. As Figure 4.14 shows, using one-item itemsets, all possible two-item itemsets are generated and the transaction database is used to calculate their support values. Because the two- item itemset {1, 3} has a support less than 3, it should not be included in the frequent itemsets that will be used to generate the next-level itemsets (three-item itemsets). The algorithm seems deceivingly simple, but only for small data sets. In much larger data sets, especially those with huge amounts of items present in low quantities and small amounts of items present in big quantities, the search and calculation become a computationally intensive process.

uSECTION 4.5 REVIEW QUESTIONS

1. Identify at least three of the main data mining methods. 2. Give examples of situations in which classification would be an appropriate data

mining technique. Give examples of situations in which regression would be an appropriate data mining technique.

3. List and briefly define at least two classification techniques. 4. What are some of the criteria for comparing and selecting the best classification

technique?

Itemset (SKUs)

Support Transaction

No SKUs

(item no.)

1001234

1001235

1001236

1001237

1001238

1001239

1, 2, 3, 4

2, 3, 4

2, 3

1, 2, 4

1, 2, 3, 4

2, 4

Raw Transaction Data

1

2

3

4

3

6

4

5

Itemset (SKUs)

Support

1, 2

1, 3

1, 4

2, 3

3

2

3

4

3, 4

5

3

2, 4

Itemset (SKUs)

Support

1, 2, 4

2, 3, 4

3

3

One-Item Itemsets Two-Item Itemsets Three-Item Itemsets

FIGURE 4.14 Identification of Frequent Itemsets in the Apriori Algorithm.

236 Part II • Predictive Analytics/Machine Learning

5. Briefly describe the general algorithm used in decision trees. 6. Define Gini index. What does it measure? 7. Give examples of situations in which cluster analysis would be an appropriate data

mining technique.

8. What is the major difference between cluster analysis and classification? 9. What are some of the methods for cluster analysis?

10. Give examples of situations in which association would be an appropriate data min- ing technique.

4.6 DATA MINING SOFTWARE TOOLS

Many software vendors provide powerful data mining tools. Examples of these ven- dors include IBM (IBM SPSS Modeler, formerly known as SPSS PASW Modeler and Clementine), SAS (Enterprise Miner), Dell (Statistica, formerly known as StatSoft Statistica Data Miner), SAP (Infinite Insight, formerly known as KXEN Infinite Insight), Salford Systems (CART, MARS, TreeNet, RandomForest), Angoss (KnowledgeSTUDIO, KnowledgeSEEKER), and Megaputer (PolyAnalyst). Noticeably but not surprisingly, the most popular data mining tools are developed by the well-established statistical soft- ware companies (SAS, SPSS, and StatSoft)—largely because statistics is the foundation of data mining, and these companies have the means to cost-effectively develop them into full-scale data mining systems. Most of the business intelligence tool vendors (e.g., IBM Cognos, Oracle Hyperion, SAP Business Objects, Tableau, Tibco, Qlik, MicroStrategy, Teradata, and Microsoft) also have some level of data mining capabilities integrated into their software offerings. These BI tools are still primarily focused on multidimensional modeling and data visualization and are not considered to be direct competitors of the data mining tool vendors.

In addition to these commercial tools, several open source and/or free data mining software tools are available online. Traditionally, especially in educational circles, the most popular free and open source data mining tool is Weka, which was developed by a number of researchers from the University of Waikato in New Zealand (the tool can be downloaded from cs.waikato.ac.nz/ml/weka). Weka includes a large number of algo- rithms for different data mining tasks and has an intuitive user interface. Recently, a num- ber of free open source, highly capable data mining tools emerged: leading the pack are KNIME (knime.org) and RapidMiner (rapidminer.com). Their graphically enhanced user interfaces, employment of a rather large number of algorithms, and incorporation of a variety of data visualization features set them apart from the rest of the free tools. These two free software tools are also platform agnostic (i.e., can natively run on both Windows and Mac operating systems). With a recent change in its offerings, RapidMiner has created a scaled-down version of its analytics tool for free (i.e., community edition) while making the full commercial product. Therefore, once listed under the free/open source tools cat- egory, RapidMiner today is often listed under commercial tools. The main difference be- tween commercial tools, such as SAS Enterprise Miner, IBM SPSS Modeler, and Statistica, and free tools, such as Weka, RapidMiner (community edition), and KNIME, is the com- putational efficiency. The same data mining task involving a rather large and feature-rich data set can take much longer to complete with the free software tools, and for some algorithms, the job might not even be completed (i.e., crashing due to the inefficient use of computer memory). Table 4.2 lists a few of the major products and their Web sites.

A suite of business intelligence and analytics capabilities that has become increas- ingly more popular for data mining studies is Microsoft’s SQL Server (it has included

Chapter 4 • Data Mining Process, Methods, and Algorithms 237

increasingly more analytics capabilities, such as BI and predictive modeling modules, starting with the SQL Server 2012 version) where data and the models are stored in the same relational database environment, making model management a considerably easier task. Microsoft Enterprise Consortium serves as the worldwide source for access to Microsoft’s SQL Server software suite for academic purposes—teaching and research. The consortium has been established to enable universities around the world to access en- terprise technology without having to maintain the necessary hardware and software on their own campus. The consortium provides a wide range of business intelligence devel- opment tools (e.g., data mining, cube building, business reporting) as well as a number of large, realistic data sets from Sam’s Club, Dillard’s, and Tyson Foods. The Microsoft Enterprise Consortium is free of charge and can be used only for academic purposes. The Sam M. Walton College of Business at the University of Arkansas hosts the enterprise sys- tem and allows consortium members and their students to access these resources using a simple remote desktop connection. The details about becoming a part of the consortium along with easy-to-follow tutorials and examples can be found at walton.uark.edu/ enterprise/.

In May 2016, KDnuggets.com conducted the 13th Annual Software Poll on the following question: “What software have you used for Analytics, Data Mining, Data Science, Machine Learning projects in the past 12 months?” The poll received remarkable participation from analytics and data science community and vendors, attracting 2,895

TABLE 4.2 Selected Data Mining Software

Product Name Web Site (URL)

IBM SPSS Modeler www-01.ibm.com/software/analytics/spss/products/ modeler/

IBM Watson Analytics ibm.com/analytics/watson-analytics/

SAS Enterprise Miner sas.com/en_id/software/analytics/enterprise-miner.html

Dell Statistica statsoft.com/products/statistica/product-index

PolyAnalyst megaputer.com/site/polyanalyst.php

CART, RandomForest salford-systems.com

Insightful Miner solutionmetrics.com.au/products/iminer/default.html

XLMiner solver.com/xlminer-data-mining

SAP InfiniteInsight (KXEN) help.sap.com/ii

GhostMiner qs.pl/ghostminer

SQL Server Data Mining msdn.microsoft.com/en-us/library/bb510516.aspx

Knowledge Miner knowledgeminer.com

Teradata Warehouse Miner teradata.com/products-and-services/teradata-warehouse- miner/

Oracle Data Mining (ODM) oracle.com/technetwork/database/options/odm/

FICO Decision Management fico.com/en/analytics/decision-management-suite/

Orange Data Mining Tool orange.biolab.si/

Zementis Predictive Analytics zementis.com

238 Part II • Predictive Analytics/Machine Learning

voters, who chose from a record number of 102 different tools. Here are some of the interesting findings that came from the poll:

• R remains the leading tool, with 49 percent shares (up from 46.9% in 2015), but Python usage grew faster and almost caught up to R with 45.8 percent shares (up from 30.3%).

• RapidMiner remains the most popular general platform for data mining/data science with about 33 percent shares. Notable tools with the most growth in popularity include g, Dataiku, MLlib, H2O, Amazon Machine Learning, scikit-learn, and IBM Watson.

• The increased choice of tools is reflected in wider usage. The average number of tools used was 6.0 (versus 4.8 in May 2015).

• The usage of Hadoop/Big Data tools increased to 39 percent up from 29 percent in 2015 (and 17% in 2014) driven by Apache Spark, MLlib (Spark Machine Learning Library), and H2O.

• The participation by region was United States/Canada (40%), Europe (39%), Asia (9.4%), Latin America (5.8%), Africa/MidEast (2.9%), and Australia/New Zealand (2.2%).

• This year, 86 percent of voters used commercial software, and 75 percent used free software. About 25 percent used only commercial software, and 13 percent used only open source/free software. A majority of 61 percent used both free and com- mercial software, similar to 64 percent in 2015.

• The use of Hadoop/Big Data tools increased to 39 percent, up from 29 percent in 2015 and 17 percent in 2014, driven mainly by big growth in Apache Spark, MLlib (Spark Machine Learning Library), and H2O, which we include among Big Data tools.

• For the second year, KDnuggets.com’s poll included Deep Learning tools. This year, 18 percent of voters used Deep Learning tools, doubling the 9 percent in 2015—Google Tensorflow jumped to first place, displacing last year’s leader, Theano/Pylearn2 ecosystem.

• In the programming languages category, Python, Java, Unix tools, and Scala grew in popularity, while C/C++, Perl, Julia, F#, Clojure, and Lisp declined.

To reduce bias through multiple voting, in this poll KDnuggets.com used e-mail verification and, by doing so, aimed to make results more representative of the reality in the analytics world. The results for the top 40 software tools (as per total number of votes received) are shown in Figure 4.15. The horizontal bar chart also makes a differentiation among free/open source, commercial, and Big Data/Hadoop tools using a color-coding schema.

Application Case 4.6 is about a research study in which a number of software tools and data mining techniques were used to build data mining models to predict financial success (box-office receipts) of Hollywood movies while they are nothing more than ideas.

uSECTION 4.6 REVIEW QUESTIONS

1. What are the most popular commercial data mining tools? 2. Why do you think the most popular tools are developed by statistics-based

companies?

3. What are the most popular free data mining tools? Why are they gaining overwhelm- ing popularity (especially R)?

4. What are the main differences between commercial and free data mining software tools?

5. What would be your top five selection criteria for a data mining tool? Explain.

Chapter 4 • Data Mining Process, Methods, and Algorithms 239

0

Orange 89 Gnu Octave 89

Salford SPM/CART/RF/MARS/TreeNet 100 Rattle 103

121 Apache Pig 132

IBM Watson

Other Hadoop/HDFS-based tools 141 Microsoft Azure Machine Learning 147

QlikView 153 Hbase 158

Microsoft Power BI 161

Scala 180

SAS Enterprise Miner 162

H2O 193 Other programming and data languages 197

Other free analytics/data mining tools 198 C/C++ 210

211 222 225 227 242 263

301 314 315 337 359

462 487 497 521 536

624

944 972

1,029 1,325

1,419

641

SQL on Hadoop tools IBM SPSS Modeler

SAS base Dataiku

IBM SPSS Statistics MATLAB

Unix shell/awk/gawk Microsoft SQL Server

Weka Mllib Hive

Anaconda Java

SciKit-Learn KNIME Tableau

Spark Hadoop

RapidMiner Excel SQL

Python R

300 600 900 1200 1500

Free/Open Source tools

Commercial tools Hadoop/Big Data tools

FIGURE 4.15 Popular Data Mining Software Tools (Poll Results). Source: Used with permission from KDnuggets.com.

Predicting box-office receipts (i.e., financial success) of a particular motion picture is an interesting and challenging problem. According to some domain

experts, the movie industry is the “land of hunches and wild guesses” due to the difficulty associated with forecasting product demand, making the movie

Application Case 4.6 Data Mining goes to Hollywood: Predicting Financial Success of Movies

(Continued )

240 Part II • Predictive Analytics/Machine Learning

business in Hollywood a risky endeavor. In sup- port of such observations, Jack Valenti (the longtime president and CEO of the Motion Picture Association of America) once mentioned that “no one can tell you how a movie is going to do in the market- place . . . not until the film opens in darkened the- atre and sparks fly up between the screen and the audience.” Entertainment industry trade journals and magazines have been full of examples, statements, and experiences that support such a claim.

Like many other researchers who have attempted to shed light on this challenging real-world problem, Ramesh Sharda and Dursun Delen have been explor- ing the use of data mining to predict the financial per- formance of a motion picture at the box office before it even enters production (while the movie is nothing more than a conceptual idea). In their highly publi- cized prediction models, they convert the forecasting (or regression) problem into a classification problem; that is, rather than forecasting the point estimate of box-office receipts, they classify a movie based on its box-office receipts in one of nine categories, rang- ing from “flop” to “blockbuster,” making the problem a multinomial classification problem. Table 4.3 illus- trates the definition of the nine classes in terms of the range of box-office receipts.

Data

Data were collected from a variety of movie-related databases (e.g., ShowBiz, IMDb, IMSDb, AllMovie, BoxofficeMojo) and consolidated into a single data set. The data set for the most recently developed models contained 2,632 movies released between 1998 and 2006. A summary of the independent vari- ables along with their specifications is provided in Table 4.4. For more descriptive details and justifi- cation for inclusion of these independent variables, the reader is referred to Sharda and Delen (2006).

The Methodology

Using a variety of data mining methods, including neural networks, decision trees, SVMs, and three types of ensembles, Sharda and Delen (2006) devel- oped the prediction models. The data from 1998 to 2005 were used as training data to build the pre- diction models, and the data from 2006 were used as the test data to assess and compare the models’ prediction accuracy. Figure 4.16 shows a screenshot of IBM SPSS Modeler (formerly Clementine data mining tool) depicting the process map employed for the prediction problem. The upper-left side of the process map shows the model development

Application Case 4.6 (Continued)

TABLE 4.3 Movie Classification based on Receipts

Class No. 1 2 3 4 5 6 7 8 9

Range (in millions of dollars)

71 71 710 720 740 765 7100 7150 7200 (Flop) 66 10 620 66 40 66 65 66 100 66 150 66 200 (Blockbuster)

TABLE 4.4 Summary of Independent Variables

Independent Variable Number of Values Possible Values

MPAA Rating 5 G, PG, PG-13, R, NR

Competition 3 High, medium, low

Star value 3 High, medium, low

Genre 10 Sci-Fi, Historic Epic Drama, Modern Drama, Politically Related, Thriller, Horror, Comedy, Cartoon, Action, Documentary

Special effects 3 High, medium, low

Sequel 2 Yes, no

Number of screens 1 A positive integer between 1 and 3,876

Chapter 4 • Data Mining Process, Methods, and Algorithms 241

process, and the lower-right corner of the process map shows the model assessment (i.e., testing or scoring) process (more details on the IBM SPSS Modeler tool and its usage can be found on the book’s Web site).

The Results

Table 4.5 provides the prediction results of all three data mining methods as well as the results of the three different ensembles. The first performance measure is the percentage of correct classification rate, which is called Bingo. Also reported in the table is the 1-Away correct classification rate (i.e., within one category). The results indicate that SVM performed the best among the individual prediction models followed by ANN; the worst of the three was the CART decision tree algorithm. In general, the ensemble models per- formed better than the individual prediction models of which the fusion algorithm performed the best. What is probably more important to decision makers and standing out in the results table is the significantly

low standard deviation obtained from the ensembles compared to the individual models.

The Conclusion

The researchers claim that these prediction results are better than any reported in the published literature for this problem domain. Beyond the attractive accuracy of their prediction results of the box-office receipts, these models could also be used to further analyze (and potentially optimize) the decision variables to maximize the financial return. Specifically, the param- eters used for modeling could be altered using the already trained prediction models to better understand the impact of different parameters on the end results. During this process, which is commonly referred to as sensitivity analysis, the decision maker of a given enter- tainment firm could find out, with a fairly high accuracy level, how much value a specific actor (or a specific release date, or the addition of more technical effects, etc.) brings to the financial success of a film, making the underlying system an invaluable decision aid.

Model Development Process

Model Assessment Process

Table

DataMining_Movie_All.

1996–2005 Data

SVM

2006 Data

Class

Class

Class

Class Analysis

Analysis

Analysis

CART Decision Tree

Neural Net

FIGURE 4.16 Process Flow Screenshot for the Box-Office Prediction System. Source: Reprint Courtesy of International Business Machines Corporation, © International Business Machines Corporation.

(Continued )

242 Part II • Predictive Analytics/Machine Learning

Questions for Case 4.6

1. Why is it important for many Hollywood pro- fessionals to predict the financial success of movies?

2. How can data mining be used for predicting financial success of movies before the start of their production process?

3. How do you think Hollywood performed, and perhaps still is performing, this task without the help of data mining tools and techniques?

Sources: R. Sharda & D. Delen, “Predicting Box-Office Success of Motion Pictures with Neural Networks,” Expert Systems with Applications, 30, 2006, pp. 243–254; D. Delen, R. Sharda, & P. Kumar, “Movie Forecast Guru: A Web-Based DSS for Hollywood Managers,” Decision Support Systems, 43(4), 2007, pp. 1151–1170.

Application Case 4.6 (Continued)

4.7 DATA MINING PRIVACY ISSUES, MYTHS, AND BLUNDERS

Data that are collected, stored, and analyzed in data mining often contain information about real people. Such information can include identification data (name, address, Social Security number, driver’s license number, employee number, etc.), demographic data (e.g., age, sex, ethnicity, marital status, number of children), financial data (e.g., salary, gross family income, checking or savings account balance, home ownership, mortgage or loan account specifics, credit card limits and balances, investment account specifics), purchase history (i.e., what is bought from where and when—either from vendor’s transaction records or from credit card transaction specifics), and other personal data (e.g., anniversary, preg- nancy, illness, loss in the family, bankruptcy filings). Most of these data can be accessed through some third-party data providers. The main question here is the privacy of the per- son to whom the data belong. To maintain the privacy and protection of individuals’ rights, data mining professionals have ethical (and often legal) obligations. One way to accomplish this is the process of de-identification of the customer records prior to applying data mining applications so that the records cannot be traced to an individual. Many publicly available data sources (e.g., CDC data, SEER data, UNOS data) are already de-identified. Prior to ac- cessing these data sources, users are often asked to consent that under no circumstances will they try to identify the individuals behind those figures.

There have been a number of instances in the recent past when companies shared their customer data with others without seeking the explicit consent of their customers. For instance, as most of you might recall, in 2003, JetBlue Airlines provided more than 1 million passenger records of customers to Torch Concepts, a U.S. government contractor. Torch

TABLE 4.5 Tabulated Prediction Results for Individual and Ensemble Models

Prediction Models

Individual Models Ensemble Models

Performance Measure SVM ANN CART Random Forest Boosted Tree Fusion (average)

Count (Bingo) 192 182 140 189 187 194

Count (1-Away) 104 120 126 121 104 120

Accuracy (% Bingo) 55.49% 52.60% 40.46% 54.62% 54.05% 56.07%

Accuracy (% 1-Away) 85.55% 87.28% 76.88% 89.60% 84.10% 90.75%

Standard deviation 0.93 0.87 1.05 0.76 0.84 0.63

Chapter 4 • Data Mining Process, Methods, and Algorithms 243

then subsequently augmented the passenger data with additional information such as fam- ily sizes and Social Security numbers—information purchased from the data broker Acxiom. The consolidated personal database was intended to be used for a data mining project to develop potential terrorist profiles. All of this was done without notification or consent of passengers. When news of the activities got out, however, dozens of privacy lawsuits were filed against JetBlue, Torch, and Acxiom, and several U.S. senators called for an investiga- tion into the incident (Wald, 2004). Similar, but not as dramatic, privacy-related news was reported in the recent past about popular social network companies that allegedly were selling customer-specific data to other companies for personalized target marketing.

Another peculiar story about privacy concerns made it to the headlines in 2012. In this instance, the company, Target, did not even use any private and/or personal data. Legally speaking, there was no violation of any laws. The story is summarized in Application Case 4.7.

In early 2012, an infamous story appeared concern- ing Target’s practice of predictive analytics. The story was about a teenage girl who was being sent adver- tising flyers and coupons by Target for the kinds of things that a mother-to-be would buy from a store like Target. The story goes like this: An angry man went into a Target outside of Minneapolis, demanding to talk to a manager: “My daughter got this in the mail!” he said. “She’s still in high school, and you’re sending her coupons for baby clothes and cribs? Are you trying to encourage her to get pregnant?” The manager had no idea what the man was talking about. He looked at the mailer. Sure enough, it was addressed to the man’s daughter and contained advertisements for maternity clothing, nursery furniture, and pictures of smiling infants. The manager apologized and then called a few days later to apologize again. On the phone, though, the father was somewhat abashed. “I had a talk with my daughter,” he said. “It turns out there’s been some activities in my house I haven’t been completely aware of. She’s due in August. I owe you an apology.”

As it turns out, Target figured out a teen girl was pregnant before her father did! Here is how the company did it. Target assigns every customer a Guest ID number (tied to his or her credit card, name, or e-mail address) that becomes a placeholder that keeps a history of everything the person has bought. Target augments these data with any demographic information that it had collected from the customer or had bought from other information sources. Using this information, Target looked at historical buying

data for all the females who had signed up for Target baby registries in the past. They analyzed the data from all directions, and soon enough, some useful patterns emerged. For example, lotions and special vitamins were among the products with interesting purchase patterns. Lots of people buy lotion, but what an analyst noticed was that women on the baby registry were buying larger quantities of unscented lotion around the beginning of their second trimester. Another analyst noted that sometime in the first 20 weeks, pregnant women loaded up on supplements like calcium, magnesium, and zinc. Many shoppers purchase soap and cotton balls, but when someone suddenly starts buying lots of scent-free soap and extra-large bags of cotton balls, in addition to hand sanitizers and washcloths, it signals that they could be getting close to their delivery date. In the end, the analysts were able to identify about 25 products that, when analyzed together, allowed them to assign each shopper a “pregnancy prediction” score. More impor- tant, they could also estimate a woman’s due date to within a small window, so Target could send cou- pons timed to very specific stages of her pregnancy.

If you look at this practice from a legal perspec- tive, you would conclude that Target did not use any information that violates customer privacy; rather, they used transactional data that almost every other retail chain is collecting and storing (and perhaps analyzing) about their customers. What was disturb- ing in this scenario was perhaps the targeted con- cept: pregnancy. Certain events or concepts should

Application Case 4.7 Predicting Customer Buying Patterns—The Target Story

(Continued )

244 Part II • Predictive Analytics/Machine Learning

Data Mining Myths and Blunders

Data mining is a powerful analytical tool that enables business executives to advance from describing the nature of the past (looking at a rearview mirror) to predicting the future (looking ahead) to better manage their business operations (making accurate and timely decisions). Data mining helps marketers find patterns that unlock the mysteries of customer behavior. The results of data mining can be used to increase revenue and reduce cost by identifying fraud and discovering business opportunities, offering a whole new realm of competitive advantage. As an evolving and maturing field, data mining is often associated with a number of myths, including those listed in Table 4.6 (Delen, 2014; Zaima, 2003).

Data mining visionaries have gained enormous competitive advantage by under- standing that these myths are just that: myths.

Although the value proposition and therefore its necessity are obvious to anyone, those who carry out data mining projects (from novice to seasoned data scientist) some- times make mistakes that result in projects with less-than-desirable outcomes. The follow- ing 16 data mining mistakes (also called blunders, pitfalls, or bloopers) are often made in practice (Nisbet et al., 2009; Shultz, 2004; Skalak, 2001), and data scientists should be aware of them and, to the extent that is possible, do their best to avoid them:

1. Selecting the wrong problem for data mining. Not every business problem can be solved with data mining (i.e., the magic bullet syndrome). When there are no represen- tative data (large and feature rich), there cannot be a practicable data mining project.

2. Ignoring what your sponsor thinks data mining is and what it really can and cannot do. Expectation management is the key for successful data mining projects.

TABLE 4.6 Data Mining Myths

Myth Reality

Data mining provides instant, crystal-ball- like predictions.

Data mining is a multistep process that requires deliberate, proactive design and use.

Data mining is not yet viable for mainstream business applications.

The current state of the art is ready for almost any business type and/or size.

Data mining requires a separate, dedicated database.

Because of the advances in database technology, a dedicated database is not required.

Only those with advanced degrees can do data mining.

Newer Web-based tools enable managers of all educational levels to do data mining.

Data mining is only for large firms that have lots of customer data.

If the data accurately reflect the business or its customers, any company can use data mining.

be off limits or treated extremely cautiously, such as terminal disease, divorce, and bankruptcy.

Questions for Case 4.7

1. What do you think about data mining and its implication for privacy? What is the threshold between discovery of knowledge and infringe- ment of privacy?

2. Did Target go too far? Did it do anything ille- gal? What do you think Target should have done? What do you think Target should do next (quit these types of practices)?

Sources: K. Hill, “How Target Figured Out a Teen Girl Was Pregnant Before Her Father Did,” Forbes, February 16, 2012; R. Nolan, “Behind the Cover Story: How Much Does Target Know?”, February 21, 2012. NYTimes.com.

Application Case 4.7 (Continued)

Chapter 4 • Data Mining Process, Methods, and Algorithms 245

3. Beginning without the end in mind. Although data mining is a process of knowledge discovery, one should have a goal/objective (a stated business problem) in mind to succeed. Because, as the saying goes, “If you don’t know where you are going, you will never get there.”

4. Defining the project around a foundation that your data cannot support. Data mining is all about data; that is, the biggest constraint that you have in a data mining project is the richness of the data. Knowing what the limitations of data are helps you craft feasible projects that deliver results and meet expectations.

5. Leaving insufficient time for data preparation. It takes more effort than is generally understood. The common knowledge suggests that up to one-third of the total proj- ect time is spent on data acquisition, understanding, and preparation tasks. To suc- ceed, avoid proceeding into modeling until after your data are properly processed (aggregated, cleaned, and transformed).

6. Looking only at aggregated results, not at individual records. Data mining is at its best when the data are at a granular representation. Try to avoid unnecessarily ag- gregating and overly simplifying data to help data mining algorithms—they don’t really need your help; they are more than capable of figuring it out themselves.

7. Being sloppy about keeping track of the data mining procedure and results. Because data mining is a discovery process that involves many iterations and experimenta- tions, its user is highly likely to lose track of the findings. Success requires a system- atic and orderly planning, execution, and tracking/recording of all data mining tasks.

8. Using data from the future to predict the future. Because of the lack of description and understanding of the data, oftentimes analysts include variables that are unknown at the time when the prediction is supposed to be made. By doing so, their prediction models produce unbelievably accurate results (a phenomenon that is often called fool’s gold). If your prediction results are too good to be true, they usually are; in that case, the first thing that you need to look for is the incorrect use of a variable from the future.

9. Ignoring suspicious findings and quickly moving on. The unexpected findings are often the indicators of real novelties in data mining projects. Proper investigation of such oddities can lead to surprisingly pleasing discoveries.

10. Starting with a high-profile complex project that will make you a superstar. Data mining projects often fail if they are not thought out carefully from start to end. Success often comes with a systematic and orderly progression of projects from smaller/simpler to larger/complex ones. The goal should be to show incremental and continuous value added as opposed to taking on a large project that will con- sume resources without producing any valuable outcomes.

11. Running data mining algorithms repeatedly and blindly. Although today’s data mining tools are capable of consuming data and setting algorithmic parameters to produce results, one should know how to transform the data and set the proper parameter values to obtain the best possible results. Each algorithm has its own unique way to process data, and knowing that is necessary to get the most out of each model type.

12. Ignore the subject matter experts. Understanding the problem domain and the related data requires a highly involved collaboration between the data mining and the domain experts. Working together helps the data mining expert to go beyond the syntactic representation and to obtain semantic nature (i.e., the true meaning of the variables) of the data.

13. Believing everything you are told about the data. Although it is necessary to talk to domain experts to better understand the data and the business problem, the data scientist should not take anything for granted. Validation and verification through a critical analysis is the key to intimate understanding and processing of the data.

14. Assuming that the keepers of the data will be fully on board with cooperation. Many data mining projects fail because the data mining expert did not know/understand the organizational politics. One of the biggest obstacles in data mining projects can

246 Part II • Predictive Analytics/Machine Learning

be the people who own and control the data. Understanding and managing the politics is a key to identify, access, and properly understand the data to produce a successful data mining project.

15. Measuring your results differently from the way your sponsor measures them. The results should talk/appeal to the end user (manager/decision maker) who will be using them. Therefore, producing the results in a measure and format that appeals to the end user tremendously increases the likelihood of true understanding and proper use of the data mining outcomes.

16. Follow the advice in a well-known quote: “If you build it, they will come”: don’t worry about how to serve it up. Usually, data mining experts think they have finished once they build models that meet and hopefully exceed the needs/wants/expectations of the end user (i.e., the customer). Without a proper deployment, the value deliverance of data mining outcomes is rather limited. Therefore, deployment is a necessary last step in the data mining process in which models are integrated into the organizational decision support infrastructure for enablement of better and faster decision making.

uSECTION 4.7 REVIEW QUESTIONS

1. What are the privacy issues in data mining? 2. How do you think the discussion between privacy and data mining will progress? Why? 3. What are the most common myths about data mining? 4. What do you think are the reasons for these myths about data mining? 5. What are the most common data mining mistakes/blunders? How can they be allevi-

ated or completely eliminated?

Chapter Highlights

• Data mining is the process of discovering new knowledge from databases.

• Data mining can use simple flat files as data sources, or it can be performed on data in data warehouses.

• There are many alternative names and definitions for data mining.

• Data mining is at the intersection of many dis- ciplines, including statistics, artificial intelligence, and mathematical modeling.

• Companies use data mining to better understand their customers and optimize their operations.

• Data mining applications can be found in virtually every area of business and government, including healthcare, finance, marketing, and homeland security.

• Three broad categories of data mining tasks are prediction (classification or regression), cluster- ing, and association.

• Similar to other IS initiatives, a data mining proj- ect must follow a systematic project management process to be successful.

• Several data mining processes have been pro- posed: CRISP-DM, SEMMA, KDD, for example.

• CRISP-DM provides a systematic and orderly way to conduct data mining projects.

• The earlier steps in data mining projects (i.e., un- derstanding the domain and the relevant data) consume most of the total project time (often more than 80% of the total time).

• Data preprocessing is essential to any successful data mining study. Good data lead to good infor- mation; good information leads to good decisions.

• Data preprocessing includes four main steps: data consolidation, data cleaning, data transformation, and data reduction.

• Classification methods learn from previous ex- amples containing inputs and the resulting class labels, and once properly trained, they are able to classify future cases.

• Clustering partitions pattern records into natural segments or clusters. Each segment’s members share similar characteristics.

• A number of different algorithms are commonly used for classification. Commercial implementa- tions include ID3, C4.5, C5, CART, CHAID, and SPRINT.

• Decision trees partition data by branching along different attributes so that each leaf node has all the patterns of one class.

Chapter 4 • Data Mining Process, Methods, and Algorithms 247

• The Gini index and information gain (entropy) are two popular ways to determine branching choices in a decision tree.

• The Gini index measures the purity of a sample. If everything in a sample belongs to one class, the Gini index value is zero.

• Several assessment techniques can measure the prediction accuracy of classification models, in- cluding simple split, k-fold cross-validation, boot- strapping, and the area under the ROC curve.

• There are a number of methods to assess the vari- able importance of data mining models. Some of these methods are model type specific, some are model type agnostic.

• Cluster algorithms are used when data records do not have predefined class identifiers (i.e., it is not known to what class a particular record belongs).

• Cluster algorithms compute measures of similarity in order to group similar cases into clusters.

• The most commonly used similarity measure in cluster analysis is a distance measure.

• The most commonly used clustering algorithms are k-means and self-organizing maps.

• Association rule mining is used to discover two or more items (or events or concepts) that go together.

• Association rule mining is commonly referred to as market-basket analysis.

• The most commonly used association algorithm is Apriori by which frequent itemsets are identi- fied through a bottom-up approach.

• Association rules are assessed based on their sup- port and confidence measures.

• Many commercial and free data mining tools are available.

• The most popular commercial data mining tools are IBM SPSS Modeler and SAS Enterprise Miner.

• The most popular free data mining tools are KNIME, RapidMiner, and Weka.

Key Terms

Apriori algorithm area under the ROC curve association bootstrapping categorical data classification clustering confidence CRISP-DM data mining

decision tree distance measure ensemble entropy Gini index information gain interval data k-fold cross-

validation KNIME

knowledge discovery in databases (KDD)

lift link analysis Microsoft Enterprise

Consortium Microsoft SQL Server nominal data numeric data ordinal data

prediction RapidMiner regression SEMMA sensitivity analysis sequence mining simple split support Weka

Questions for Discussion

1. Define data mining. Why are there many names and definitions for data mining?

2. What are the main reasons for the recent popularity of data mining?

3. Discuss what an organization should consider before making a decision to purchase data mining software.

4. Distinguish data mining from other analytical tools and techniques.

5. Discuss the main data mining methods. What are the fundamental differences among them?

6. What are the main data mining application areas? Discuss the commonalities of these areas that make them a prospect for data mining studies.

7. Why do we need a standardized data mining pro- cess? What are the most commonly used data mining processes?

8. Discuss the differences between the two most com- monly used data mining processes.

9. Are data mining processes a mere sequential set of activities? Explain.

10. Why do we need data preprocessing? What are the main tasks and relevant techniques used in data preprocessing?

11. Discuss the reasoning behind the assessment of clas- sification models.

12. What is the main difference between classification and clustering? Explain using concrete examples.

13. Moving beyond the chapter discussion, where else can association be used?

14. What are the privacy issues with data mining? Do you think they are substantiated?

15. What are the most common myths and mistakes about data mining?

248 Part II • Predictive Analytics/Machine Learning

Exercises

Teradata University Network (TUN) and Other Hands-On Exercises

1. Visit teradatauniversitynetwork.com. Identify case studies and white papers about data mining. Describe recent developments in the field of data mining and predictive modeling.

2. Go to teradatauniversitynetwork.com. Locate Web seminars related to data mining. In particular, locate and watch a seminar given by C. Imhoff and T. Zouqes. Then answer the following questions: a. What are some of the interesting applications of data

mining? b. What types of payoffs and costs can organizations

expect from data mining initiatives? 3. For this exercise, your goal is to build a model to iden-

tify inputs or predictors that differentiate risky custom- ers from others (based on patterns pertaining to previous customers) and then use those inputs to predict new risky customers. This sample case is typical for this domain.

The sample data to be used in this exercise are in Online File W4.1 in the file CreditRisk.xlsx. The data set has 425 cases and 15 variables pertaining to past and current customers who have borrowed from a bank for various reasons. The data set contains customer-related information such as financial standing, reason for the loan, employment, demographic information, and the outcome or dependent variable for credit standing, clas- sifying each case as good or bad based on the institu- tion’s past experience.

Take 400 of the cases as training cases and set aside the other 25 for testing. Build a decision tree mod- el to learn the characteristics of the problem. Test its per- formance on the other 25 cases. Report on your model’s learning and testing performance. Prepare a report that identifies the decision tree model and training param- eters as well as the resulting performance on the test set. Use any decision tree software. (This exercise is cour- tesy of StatSoft, Inc., based on a German data set from ftp.ics.uc,i.edu/pub/machine-learning-databases/ statlog/german renamed CreditRisk and altered.)

4. For this exercise, you will replicate (on a smaller scale) the box-office prediction modeling explained in Application Case 4.6. Download the training data set from Online File W4.2, MovieTrain.xlsx, which is in Microsoft Excel format. Use the data description given in Application Case 4.6 to understand the domain and the problem you are trying to solve. Pick and choose your independent variables. Develop at least three clas- sification models (e.g., decision tree, logistic regression, neural networks). Compare the accuracy results using 10-fold cross-validation and percentage split techniques, use confusion matrices, and comment on the outcome. Test the models you have developed on the test set (see Online File W4.3, MovieTest.xlsx). Analyze the results

with different models, and find the best classification model, supporting it with your results.

5. This exercise introduces you to association rule min- ing. The Excel data set baskets1ntrans.xlsx has around 2,800 observations/records of supermarket transaction products data. Each record contains the customer’s ID and products that they have purchased. Use this data set to understand the relationships among products (i.e., which products are purchased together). Look for inter- esting relationships and add screenshots of any subtle association patterns that you might find. More specifi- cally, answer the following questions. • Which association rules do you think are most important? • Based on some of the association rules you found,

make at least three business recommendations that might be beneficial to the company. These recom- mendations can include ideas about shelf organiza- tion, up-selling, or cross-selling products. (Bonus points will be given to new/innovative ideas.)

• What are the Support, Confidence, and Lift values for the following rule?

Wine, Canned Veg S Frozen Meal 6. In this assignment, you will use a free/open source

data mining tool, KNIME (knime.org), to build pre- dictive models for a relatively small Customer Churn Analysis data set. You are to analyze the given data set (about the customer retention/attrition behavior for 1,000 customers) to develop and compare at least three prediction (i.e., classification) models. For exam- ple, you can include decision trees, neural networks, SVM, k nearest neighbor, and/or logistic regression models in your comparison. Here are the specifics for this assignment: • Install and use the KNIME software tool from

(knime.org). • You can also use MS Excel to preprocess the data (if

you need to/want to). • Download CustomerChurnData.csv data file from the

book’s Web site. • The data are given in comma-separated value (CSV)

format. This format is the most common flat-file for- mat that many software tools can easily open/handle (including KNIME and MS Excel).

• Present your results in a well-organized professional document.

• Include a cover page (with proper information about you and the assignment).

• Make sure to nicely integrate figures (graphs, charts, tables, screenshots) within your textual description in a professional manner. The report should have six main sections (resembling CRISP-DM phases).

• Try not to exceed 15 pages in total, including the cover (use 12-point Times New Roman fonts, and 1.5- line spacing).

Chapter 4 • Data Mining Process, Methods, and Algorithms 249

Team Assignments and Role-Playing Projects

1. Examine how new data capture devices such as RFID tags help organizations accurately identify and segment their customers for activities such as targeted marketing. Many of these applications involve data mining. Scan the literature and the Web and then propose five potential new data mining applications that can use the data cre- ated with RFID technology. What issues could arise if a country’s laws required such devices to be embedded in everyone’s body for a national identification system?

2. Interview administrators in your college or executives in your organization to determine how data mining, data warehousing, Online Analytics Processing (OLAP), and visualization tools could assist them in their work. Write a proposal describing your findings. Include cost esti- mates and benefits in your report.

3. A very good repository of data that has been used to test the performance of many data mining algorithms is available at ics.uci.edu/~mlearn/MLRepository.html. Some of the data sets are meant to test the limits of cur- rent machine-learning algorithms and to compare their performance with new approaches to learning. However, some of the smaller data sets can be useful for exploring the functionality of any data mining software, such as RapidMiner or KNIME. Download at least one data set from this repository (e.g., Credit Screening Databases, Housing Database) and apply decision tree or clustering methods, as appropriate. Prepare a report based on your results. (Some of these exercises, especially the ones that involve large/challenging data/problem may be used as semester-long term projects.)

4. Large and feature-rich data sets are made available by the U.S. government or its subsidiaries on the Internet. For instance, see a large collection of government data sets (data.gov), the Centers for Disease Control and Prevention data sets (www.cdc.gov/DataStatistics), Surveillance, Cancer.org’s Epidemiology and End Results data sets (http://seer.cancer.gov/data), and the Department of Transportation’s Fatality Analysis Reporting System crash data sets (www.nhtsa.gov/FARS). These data sets are not preprocessed for data mining, which makes them a great resource to experience the complete data mining process. Another rich source for a collec- tion of analytics data sets is listed on KDnuggets.com (KDnuggets.com/datasets/index.html).

5. Consider the following data set, which includes three attributes and a classification for admission decisions into an MBA program:

GMAT GPA Quantitative GMAT Score (percentile) Decision

650 2.75 35 No

580 3.50 70 No

600 3.50 75 Yes

450 2.95 80 No

700 3.25 90 Yes

GMAT GPA Quantitative GMAT Score (percentile) Decision

590 3.50 80 Yes

400 3.85 45 No

640 3.50 75 Yes

540 3.00 60 ?

690 2.85 80 ?

490 4.00 65 ?

a. Using the data shown, develop your own manual expert rules for decision making.

b. Use the Gini index to build a decision tree. You can use manual calculations or a spreadsheet to perform the basic calculations.

c. Use an automated decision tree software program to build a tree for the same data.

Internet Exercises

1. Visit the AI Exploratorium at cs.ualberta.ca/~aixplore. Click the Decision Tree link. Read the narrative on bas- ketball game statistics. Examine the data, and then build a decision tree. Report your impressions of its accuracy. Also explore the effects of different algorithms.

2. Survey some data mining tools and vendors. Start with fico.com and egain.com. Consult dmreview.com, and identify some data mining products and service providers that are not mentioned in this chapter.

3. Find recent cases of successful data mining applications. Visit the Web sites of some data mining vendors, and look for cases or success stories. Prepare a report sum- marizing five new case studies.

4. Go to vendor Web sites (especially those of SAS, SPSS, Cognos, Teradata, StatSoft, and Fair Isaac) and look at success stories for BI (OLAP and data mining) tools. What do the various success stories have in common? How do they differ?

5. Go to statsoft.com (now a Dell company). Download at least three white papers on applications. Which of these applications might have used the data/text/Web mining techniques discussed in this chapter?

6. Go to sas.com. Download at least three white papers on applications. Which of these applications could have used the data/text/Web mining techniques discussed in this chapter?

7. Go to spss.com (an IBM company). Download at least three white papers on applications. Which of these applications could have used the data/text/Web mining techniques discussed in this chapter?

8. Go to teradata.com. Download at least three white papers on applications. Which of these applications could have used the data/text/Web mining techniques discussed in this chapter?

9. Go to fico.com. Download at least three white papers on applications. Which of these applications could have used the data/text/Web mining techniques discussed in this chapter?

250 Part II • Predictive Analytics/Machine Learning

10. Go to salfordsystems.com. Download at least three white papers on applications. Which of these applica- tions could have used the data/text/Web mining tech- niques discussed in this chapter?

11. Go to rulequest.com. Download at least three white papers on applications. Which of these applications

could have used the data/text/Web mining techniques discussed in this chapter?

12. Go to KDnuggets.com. Explore the sections on applications as well as software. Find names of at least three additional packages for data mining and text mining.

References

Chan, P., Phan, W., Prodromidis, A., & Stolfo, S. (1999). “Dis- tributed Data Mining in Credit Card Fraud Detection.” IEEE Intelligent Systems, 14(6), 67–74.

CRISP-DM. (2013). “Cross-Industry Standard Process for Data Min- ing (CRISP-DM).” http://crisp-dm.orgwww.the- modeling- agency.com/crisp-dm.pdf (accessed February 2, 2013).

Davenport, T. (2006, January). “Competing on Analytics.” Harvard Business Review, 99–107.

Delen, D. (2009). “Analysis of Cancer Data: A Data Mining Approach.” Expert Systems, 26(1), 100–112.

Delen, D. (2014). Real-World Data Mining: Applied Business Analytics and Decision Making. Upper Saddle River, NJ: Pearson.

Delen, D., Cogdell, D., & Kasap, N. (2012). “A Comparative Analysis of Data Mining Methods in Predicting NCAA Bowl Outcomes.” International Journal of Forecasting, 28, 543–552.

Delen, D., Sharda, R., & Kumar, P. (2007). “Movie Forecast Guru: A Web-Based DSS for Hollywood Managers.” Deci- sion Support Systems, 43(4), 1151–1170.

Delen, D., Walker, G., & Kadam, A. (2005). “Predicting Breast Cancer Survivability: A Comparison of Three Data Mining Methods.” Artificial Intelligence in Medicine, 34(2), 113–127.

Dunham, M. (2003). Data Mining: Introductory and Ad- vanced Topics. Upper Saddle River, NJ: Prentice Hall.

Fayyad, U., Piatetsky-Shapiro, G., & Smyth, P. (1996). “From Knowl- edge Discovery in Databases.” AI Magazine, 17(3), 37–54.

Hoffman, T. (1998, December 7). “Banks Turn to IT to Re- claim Most Profitable Customers.” Computerworld.

Hoffman, T. (1999, April 19). “Insurers Mine for Age- Appropriate Offering.” Computerworld.

Kohonen, T. (1982). “Self-Organized Formation of Topologically Correct Feature Maps.” Biological Cybernetics, 43(1), 59–69.

Nemati, H., & Barko, C. (2001). “Issues in Organizational Data Mining: A Survey of Current Practices.” Journal of Data Warehousing, 6(1), 25–36.

Nisbet, R., Miner, G., & Elder IV, J. (2009). “Top 10 Data Min- ing Mistakes.” Handbook of Statistical Analysis and Data Mining Applications. Academic Press, pp. 733–754.

Quinlan, J. (1986). “Induction of Decision Trees.” Machine Learning, 1, 81–106.

Saltelli, A. (2002). “Making Best Use of Model Evaluations to Compute Sensitivity Indices,” Computer Physics Communi- cations, 145, 280–297.

Saltelli, A., Tarantola, S., Campolongo, F., & Ratto, M. (2004). Sensitivity Analysis in Practice – A Guide to Assessing Sci- entific Models. Hoboken, NJ: John Wiley.

SEMMA. (2009). “SAS’s Data Mining Process: Sample, Explore, Modify, Model, Assess.” sas.com/offices/europe/uk/ technologies/analytics/datamining/miner/semma. html (accessed August 2009).

Sharda, R., & Delen, D. (2006). “Predicting Box-Office Success of Motion Pictures with Neural Networks.” Expert Systems with Applications, 30, 243–254.

Shultz, R. (2004, December 7). “Live from NCDM: Tales of Database Buffoonery.” directmag.com/news/ncdm-12- 07-04/index.html (accessed April 2009).

Skalak, D. (2001). “Data Mining Blunders Exposed!” DB2 Magazine, 6(2), 10–13.

Thongkam, J., Xu, G., Zhang, Y., & Huang, F. (2009). “Toward Breast Cancer Survivability Prediction Models Through Im- proving Training Space.” Expert Systems with Applications, 36(10), 12200–12209.

Wald, M. (2004, February 21). “U.S. Calls Release of JetBlue Data Improper.” The New York Times.

Wright, C. (2012). “Statistical Predictors of March Mad- ness: An Examination of the NCAA Men’s Basketball Championship.” http://economics-files.pomona. edu/GarySmith/Econ190/Wright%20March%20 Madness%20Final%20Paper.pdf (accessed February 2, 2013).

Zaima, A. (2003). “The Five Myths of Data Mining.” What Works: Best Practices in Business Intelligence and Data Warehousing, Vol. 15. Chatsworth, CA: The Data Ware- housing Institute, pp. 42–43.

Zolbanin, H., Delen, D., & Zadeh, A. (2015). “Predicting Over- all Survivability in Comorbidity of Cancers: A Data Mining Approach.” Decision Support Systems, 74, 150–161.

251

Machine-Learning Techniques for Predictive Analytics

LEARNING OBJECTIVES

■■ Understand the basic concepts and definitions of artificial neural networks (ANN)

■■ Learn the different types of ANN architectures ■■ Understand the concept and structure of support vector machines (SVM)

■■ Learn the advantages and disadvantages of SVM compared to ANN

■■ Understand the concept and formulation of k@nearest neighbor (kNN) algorithm

■■ Learn the advantages and disadvantages of kNN compared to ANN and SVM

■■ Understand the basic principles of Bayesian learning and Naïve Bayes algorithm

■■ Learn the basics of Bayesian Belief Networks and how they are used in predictive analytics

■■ Understand different types of ensemble models and their pros and cons in predictive analytics

Predictive modeling is perhaps the most commonly practiced branch in data science and business analytics. It allows decision makers to estimate what the future holds by means of learning from the past (i.e., historical data). In this chapter, we study the internal structures, capabilities/limitations, and applications of the most popular pre- dictive modeling techniques, such as artificial neural networks, support vector machines, k@nearest neighbor, Bayesian learning, and ensemble models. Most of these techniques are capable of addressing both classification- and regression-type prediction problems. Often, they are applied to complex prediction problems where other, more traditional techniques are not capable of producing satisfactory results. In addition to the ones cov- ered in this chapter, other notable prediction modeling techniques include regression (linear or nonlinear), logistic regression (for classification-type prediction problems), and different types of decision trees (covered in Chapter 4).

5.1 Opening Vignette: Predictive Modeling Helps Better Understand and Manage Complex Medical Procedures 252

5.2 Basic Concepts of Neural Networks 255

C H A P T E R

5

252 Part II • Predictive Analytics/Machine Learning

5.3 Neural Network Architectures 259 5.4 Support Vector Machines 263 5.5 Process-Based Approach to the Use of SVM 271 5.6 Nearest Neighbor Method for Prediction 274 5.7 Naïve Bayes Method for Classification 278 5.8 Bayesian Networks 287 5.9 Ensemble Modeling 293

5.1 OPENING VIGNETTE: Predictive Modeling Helps Better Understand and Manage Complex Medical Procedures

Healthcare has become one of the most important issues to have a direct impact on the quality of life in the United States and around the world. While the demand for healthcare services is increasing because of the aging population, the supply side is having problems keeping up with the level and quality of service. To close the gap, healthcare systems ought to significantly improve their operational effectiveness and efficiency. Effectiveness (doing the right thing, such as diagnosing and treating accurately) and efficiency (doing it the right way, such as using the least amount of resources and time) are the two fundamental pillars upon which the healthcare sys- tem can be revived. A promising way to improve healthcare is to take advantage of predictive modeling techniques along with large and feature-rich data sources (true reflections of medical and healthcare experiences) to support accurate and timely decision making.

According to the American Heart Association, cardiovascular disease (CVD) is the underlying cause for over 20 percent of deaths in the United States. Since 1900, CVD has been the number-one killer every year except 1918, which was the year of the great flu pandemic. CVD kills more people than the next four leading causes of deaths combined: cancer, chronic lower respiratory disease, accidents, and diabetes mellitus. Of all CVD deaths, more than half are attributed to coronary diseases. Not only does CVD take a huge toll on the personal health and well-being of the population, but also it is a great drain on the healthcare resources in the United States and elsewhere in the world. The direct and indirect costs associated with CVD for a year are estimated to be in excess of $500 billion . A common surgical procedure to cure a large variant of CVD is called coronary artery bypass grafting (CABG). Even though the cost of a CABG surgery depends on the patient and service provider–related factors, the average rate is between $50,000 and $100,000 in the United States. As an illustrative example, Delen et al. (2012) carried out an analytics study using various predictive modeling methods to predict the outcome of a CABG and applied an information fusion–based sensitivity analysis on the trained models to better understand the importance of the prognostic factors. The main goal was to illustrate that predictive and explanatory analysis of large and feature-rich data sets provides invaluable information to make more efficient and effective decisions in healthcare.

THE RESEARCH METHOD

Figure 5.1 shows the model development and testing process used by Delen et al. (2012). They employed four different types of prediction models (artificial neural networks, sup- port vector machines, and two types of decision trees—C5 and CART) and went through

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 253

a large number of experimental runs to calibrate the modeling parameters for each model type. Once the models were developed, the researchers went to the text data set. Finally, the trained models were exposed to a sensitivity analysis procedure that measured the contribution of the variables. Table 5.1 shows the test results for the four different types of prediction models.

Training and calibrating the

model

Training and calibrating the

model

Preprocessed Data

(in Excel format)

Partitioned data (training, testing,

and validation)

Partitioned data (training, testing,

and validation)

Training and calibrating the

model

ANN

SVM

DT/C5

DT/CART

Testing the model

Testing the model

Conducting sensitivity analysis

Tabulated Model Testing Results

(Accuracy, Sensitivity, and

Specificity)

Integrated (fused) Sensitivity

Analysis Results

Conducting sensitivity analysis

Testing the model

Conducting sensitivity analysis

Training and calibrating the

model

Testing the model

Conducting sensitivity analysis

FIGURE 5.1 Process Map for Training and Testing of the Four Predictive Models.

254 Part II • Predictive Analytics/Machine Learning

THE RESULTS

In this study, Delen et al. (2012) showed the power of data mining in predicting the outcome and in analyzing the prognostic factors of complex medical procedures such as CABG surgery. The researchers showed that using a number of prediction methods (as opposed to only one) in a competitive experimental setting has the potential to produce better predictive as well as explanatory results. Among the four methods that they used, SVM produced the best results with prediction accuracy of 88 percent on the test data sample. The information fusion–based sensitivity analysis revealed the ranked importance of the independent variables. Some of the top variables identified in this analysis having to overlap with the most important variables identified in previously conducted clinical and biological studies confirm the validity and effectiveness of the proposed data mining methodology.

From the managerial standpoint, clinical decision support systems that use the out- come of data mining studies (such as the ones presented in this case study) are not meant to replace healthcare managers and/or medical professionals. Rather, they intend to support them in making accurate and timely decisions to optimally allocate resources to increase the quantity and quality of medical services. There still is a long way to go before we can see these decision aids being used extensively in healthcare practices. Among others, there are behavioral, ethical, and political reasons for this resistance to adoption. Maybe the need and government incentives for better healthcare systems will expedite the adoption.

u QUESTIONS FOR THE OPENING VIGNETTE

1. Why is it important to study medical procedures? What is the value in predicting outcomes?

2. What factors do you think are the most important in better understanding and managing healthcare? Consider both managerial and clinical aspects of healthcare.

3. What would be the impact of predictive modeling on healthcare and medicine? Can predictive modeling replace medical or managerial personnel?

TABLE 5.1 Prediction Accuracy Results for All Four Model Types Based on the Test Data Set

Model Type1 Confusion Matrices2

Pos (1) Neg (0) Accuracy3 Sensitivity3 Specificity3

ANN Pos (1) 749 230

74.72% 76.51% 72.93% Neg (0) 265 714

SVM Pos (1) 876 103

87.74% 89.48% 86.01% Neg (0) 137 842

C5 Pos (1) 876 103

79.62% 80.29% 78.96% Neg (0) 137 842

CART Pos (1) 660 319

71.15% 67.42% 74.87% Neg (0) 246 733

1Acronyms for model types: artificial neural networks (ANN), support vector machines (SVM), popular decision tree algorithm (C5), classification and regression trees (CART). 2Prediction results for the test data samples are shown in a confusion matrix where the rows represent the actuals and columns represent the predicted cases. 3Accuracy, sensitivity, and specificity are the three performance measures that were used in comparing the four prediction models.

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 255

4. What were the outcomes of the study? Who can use these results? How can the results be implemented?

5. Search the Internet to locate two additional cases that used predictive modeling to understand and manage complex medical procedures.

WHAT WE CAN LEARN FROM THIS VIGNETTE

As you will see in this chapter, predictive modeling techniques can be applied to a wide range of problem areas, from standard business problems of assessing cus- tomer needs to understanding and enhancing the efficiency of production processes to improving healthcare and medicine. This vignette illustrates an innovative application of predictive modeling to better predict, understand, and manage CABG procedures. As the results indicate, these sophisticated analytics techniques are capable of predict- ing and explaining such complex phenomena. Evidence-based medicine is a relatively new term coined in the healthcare arena where the main idea is to dig deeply into past experiences to discover new and useful knowledge to improve medical and manage- rial procedures in healthcare. As we all know, healthcare needs all the help that it can get. Compared to traditional research, which is clinical and biological in nature, data-driven studies provide an out-of-the-box view to medicine and management of medical systems.

Sources: D. Delen, A. Oztekin, and L. Tomak, “An Analytic Approach to Better Understanding and Management of Coronary Surgeries,” Decision Support Systems, Vol. 52, No. 3, 2012, pp. 698–705; and American Heart Association, “Heart Disease and Stroke Statistics,” heart.org (accessed May 2018).

5.2 BASIC CONCEPTS OF NEURAL NETWORKS

Neural networks represent a brain metaphor for information processing. These models are biologically inspired rather than an exact replica of how the brain actually functions. Neural networks have been shown to be very promising systems in many forecasting and business classification applications due to their ability to “learn” from the data, their nonparametric nature (i.e., no rigid assumptions), and their ability to generalize. Neural computing refers to a pattern-recognition methodology for machine learning. The result- ing model from neural computing is often called an artificial neural network (ANN) or a neural network. Neural networks have been used in many business applications for pattern recognition, forecasting, prediction, and classification. Neural network comput- ing is a key component of any data science and business analytics toolkit. Applications of neural networks abound in finance, marketing, manufacturing, operations, information systems, and so on.

Because we cover neural networks, especially the feed-forward, multi-layer, perception-type prediction modeling–specific neural network architecture, in Chapter 6 (which is dedicated to deep learning and cognitive computing) as a primer to un- derstanding deep learning and deep neural networks, in this section, we provide only a brief introduction to the vast variety of neural network models, methods, and applications.

The human brain possesses bewildering capabilities for information processing and problem solving that modern computers cannot compete with in many aspects. It has been postulated that a model or a system that is enlightened and supported by results from brain research and has a structure similar to that of biological neural net- works could exhibit similar intelligent functionality. Based on this bottom-up approach,

256 Part II • Predictive Analytics/Machine Learning

ANN (also known as connectionist models, parallel distributed processing models, neuro- morphic systems, or simply neural networks) has been developed as biologically inspired and plausible models for various tasks.

Biological neural networks are composed of many massively interconnected neurons. Each neuron possesses axons and dendrites, fingerlike projections that enable a neuron to communicate with its neighboring neurons by transmitting and receiving electrical and chemical signals. More or less resembling the structure of their biological counterparts, ANN are composed of interconnected, simple processing elements called artificial neurons. When processing information, the processing elements in ANN operate concurrently and collectively, similar to biological neurons. ANN possess some desirable traits similar to those of biological neural networks, such as the abilities to learn, self- organize, and support fault tolerance.

Coming along a winding journey, ANN have been investigated by researchers for more than half a century. The formal study of ANN began with the pioneering work of McCulloch and Pitts in 1943. Inspired by the results of biological experiments and obser- vations, McCulloch and Pitts (1943) introduced a simple model of a binary artificial neuron that captured some of the functions of biological neurons. Using information- processing machines to model the brain, McCulloch and Pitts built their neural network model using a large number of interconnected artificial binary neurons. From these beginnings, neural network research became quite popular in the late 1950s and early 1960s. After a thor- ough analysis of an early neural network model (called the perceptron, which used no hidden layer) as well as a pessimistic evaluation of the research potential by Minsky and Papert in 1969, interest in neural networks diminished.

During the past two decades, there has been an exciting resurgence in ANN studies due to the introduction of new network topologies, new activation functions, and new learning algorithms as well as progress in neuroscience and cognitive science. Advances in theory and methodology have overcome many of the obstacles that hindered neural network research a few decades ago. Evidenced by the appealing results of numer- ous studies, neural networks are gaining in acceptance and popularity. In addition, the desirable features in neural information processing make neural networks attractive for solving complex problems. ANN have been applied to numerous complex problems in a variety of application settings. The successful use of neural network applications has inspired renewed interest from industry and business. With the emergence of deep neural networks (as part of the rather recent deep learning phenomenon), the popularity of neu- ral networks (with a “deeper” architectural representation and much-enhanced analytics capabilities) hit an unprecedented high, creating mile-high expectations from this new generation of neural networks. Deep neural networks are covered in detail in Chapter 6.

Biological versus Artificial Neural Networks

The human brain is composed of special cells called neurons. These cells do not die and replenish when a person is injured (all other cells reproduce to replace themselves and then die). This phenomenon might explain why humans retain information for an extended period of time and start to lose it when they get old—as the brain cells gradu- ally start to die. Information storage spans sets of neurons. The brain has anywhere from 50  billion to 150 billion neurons of which there are more than 100 different kinds. Neurons are partitioned into groups called networks. Each network contains several thousand highly interconnected neurons. Thus, the brain can be viewed as a collection of neural networks.

The ability to learn and to react to changes in our environment requires intelligence. The brain and the central nervous system control thinking and intelligent behavior. People who suffer brain damage have difficulty learning and reacting to changing environments. Even so, undamaged parts of the brain can often compensate with new learning.

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 257

A portion of a network composed of two cells is shown in Figure 5.2. The cell itself includes a nucleus (the central processing portion of the neuron). To the left of cell 1, the dendrites provide input signals to the cell. To the right, the axon sends output signals to cell 2 via the axon terminals. These axon terminals merge with the dendrites of cell 2. Signals can be transmitted unchanged, or they can be altered by synapses. A synapse is able to increase or decrease the strength of the connection between neurons and cause excitation or inhibition of a subsequent neuron. This is how information is stored in the neural networks.

An ANN emulates a biological neural network. Neural computing actually uses a very limited set of concepts from biological neural systems (see Technology Insights 5.1). It is more of an analogy to the human brain than an accurate model of it. Neural concepts usually are implemented as software simulations of the massively parallel processes in- volved in processing interconnected elements (also called artificial neurons, or neurodes) in a network architecture. The artificial neuron receives inputs analogous to the electro- chemical impulses that dendrites of biological neurons receive from other neurons. The output of the artificial neuron corresponds to signals sent from a biological neuron over its axon. These artificial signals can be changed by weights in a manner similar to the physical changes that occur in the synapses (see Figure 5.3).

Several ANN paradigms have been proposed for applications in a variety of prob- lem domains. Perhaps the easiest way to differentiate among the various neural models is on the basis of the way they structurally emulate the human brain, process information, and learn to perform their designated tasks.

Dendrites

Synapse

Dendrites

Soma

Axon Axon

Soma

Synapse

1 2

FIGURE 5.2 Portion of a Biological Neural Network: Two Interconnected Cells/Neurons.

1

2

n n

Inputs

X Y

Y

Y

X

X

Weights

Summation Transfer Function

1

2

Outputs

2

1

Neuron (or PE)

S X W n

i i i5

5 1 S

Y

W

W

Wn

f(S)

FIGURE 5.3 Processing Information in an Artificial Neuron.

258 Part II • Predictive Analytics/Machine Learning

In the mining industry, most of the underground injuries and fatalities are due to rock falls (i.e., fall of hanging wall/roof). The method that has been used for many years in the mines when determin- ing the integrity of the hanging wall is to tap the hanging wall with a sounding bar and listen to the sound emitted. An experienced miner can differenti- ate an intact/solid hanging wall from a detached/ loose hanging wall by the sound that is emitted. This method is subjective. The Council for Scientific and Industrial Research (CSIR) in South Africa has devel- oped a device that assists any miner in making an

objective decision when determining the integrity of the hanging wall. A trained neural network model is embedded into the device. The device then records the sound emitted when a hanging wall is tapped. The sound is then preprocessed before being input into a trained neural network model, which classi- fies the hanging wall as either intact or detached.

Teboho Nyareli, who holds a master’s degree in electronic engineering from the University of Cape Town in South Africa and works as a research engineer at CSIR, used NeuroSolutions, a popu- lar artificial neural network modeling software

Application Case 5.1 Neural Networks Are Helping to Save Lives in the Mining Industry

Application Case 5.1 provides an interesting example of the use of neural networks as a prediction tool in the mining industry.

TECHNOLOGY INSIGHTS 5.1 The Relationship between Biological and Artificial Neural Networks

The following list shows some of the relationships between biological and artificial networks.

Biological Artificial

Soma Node

Dendrites Input

Axon Output

Synapse Weight

Slow Fast

Many neurons (109) Few neurons (a dozen to hundreds of thousands)

Sources: L. Medsker and J. Liebowitz, Design and Development of Expert Systems and Neural Networks, Macmillan, New York, 1994, p. 163; and F. Zahedi, Intelligent Systems for Business: Expert Systems with Neural Networks, Wadsworth, Belmont, CA, 1993.

Because they are biologically inspired, the main processing elements of a neural network are individual neurons, analogous to the brain’s neurons. These artificial neurons receive the in- formation from other neurons or external input stimuli, perform a transformation on the inputs, and then pass on the transformed information to other neurons or external outputs. This is simi- lar to how it is currently thought that the human brain works. Passing information from neuron to neuron can be thought of as a way to activate, or trigger, a response from certain neurons based on the information or stimulus received.

How information is processed by a neural network is inherently a function of its structure. Neural networks can have one or more layers of neurons. These neurons can be highly or fully interconnected, or only certain layers can be connected. Connections between neurons have an associated weight. In essence, the “knowledge” possessed by the network is encapsulated in these interconnection weights. Each neuron calculates a weighted sum of the incoming neuron values, transforms this input, and passes on its neural value as the input to subsequent neurons. Typically, although not always, this input/output transformation process at the individual neuron level is performed in a nonlinear fashion.

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 259

u SECTION 5.2 REVIEW QUESTIONS

1. What is an ANN? 2. What are the commonalities and differences between biological and artificial neural

networks?

3. What types of business problems can be solved with ANN?

5.3 NEURAL NETWORK ARCHITECTURES

There are several neural network architectures designed to solve different types of prob- lems (Haykin, 2009). The most common ones include feedforward (multilayer perceptron with backpropagation), associative memory, recurrent networks, Kohonen’s self- organizing feature maps, and Hopfield networks. The feedforward multi-layer perceptron-type network architecture activates the neurons (and learns the relationship between input variables and the output variable) in one direction (from input layer to the output layer, going through one or more middle/hidden layers). This neural network architecture will be covered in detail in Chapter 6; hence, the details will be skipped in this section. In contrast to feedforward neural network architecture, Figure 5.4 shows a pictorial representation of a recurrent neural net- work architecture where the connections between the layers are not unidirectional; rather, there are many connections in every direction between the layers and neurons, creating a complex connection structure. Many experts believe that this multidirectional connectedness better mimics the way biological neurons are structured in the human brain.

Kohonen’s Self-Organizing Feature Maps

First introduced by the Finnish professor Teuvo Kohonen, Kohonen’s self- organizing feature map (Kohonen networks, or SOM in short) provides a way to represent mul- tidimensional data in much lower dimensional spaces, usually one or two dimensions.

developed by NeuroDimensions, Inc., to develop the classification-type prediction models. The multi- layer perceptron-type ANN architecture that he built achieved better than 70 percent prediction accu- racy on the hold-out sample. In 2018, the prototype system was undergoing a final set of tests before it was deployed as a decision aid and then the com- mercialization phase followed. The following figure shows a snapshot of NeuroSolution’s model build- ing workspace, called the breadboard.

Source: Used with permission from NeuroSolutions, customer success story, neurosolutions.com/resources/nyareli.html (accessed May 2018).

Questions for Case 5.1

1. How did neural networks help save lives in the mining industry?

2. What were the challenges, the proposed solu- tion, and the results?

260 Part II • Predictive Analytics/Machine Learning

One of the most interesting aspects of SOM is that they learn to classify data without supervision (i.e., there is no output vector). Remember that in supervised learning techniques, such as backpropagation, the training data consist of vector pairs—an input vector and a target vector. Because of its self-organizing capability, SOM are commonly used for clustering tasks where a group of cases is assigned an arbitrary number of naturals groups. Figure 5.5a illustrates a very small Kohonen network of 4 * 4 nodes connected to the input layer (with three inputs), representing a two- dimensional vector.

Hopfield Networks

The Hopfield network is another interesting neural network architecture, first introduced by John Hopfield (1982). Hopfield demonstrated in a series of research articles in the early 1980s how highly interconnected networks of nonlinear neurons can be extremely

Input 1

Input 2

Input 3

Input n

*H: indicates a “hidden” neuron without a target output

Output 1

Output 2

H

H

FIGURE 5.4 Recurrent Neural Network Architecture.

I N P U T

O U T P U T

Input 1

Input 2

Input 3

(a) Kohonen Network (SOM) (b) Hopfield Network

FIGURE 5.5 Graphical Depiction of Kohonen and Hopfield ANN Structures.

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 261

effective in solving complex computational problems. These networks were shown to provide novel and quick solutions to a family of problems stated in terms of a desired objective subject to a number of constraints (i.e., constraint optimization problems). One of the major advantages of Hopfield neural networks is the fact that their structure can be realized on an electronic circuit board, possibly on a very large-scale integration (VLSI) circuit, to be used as an online solver with a parallel-distributed process. Architecturally, a general Hopfield network is represented as a single large layer of neurons with total inter- connectivity; that is, each neuron is connected to every other neuron within the network (see Figure 5.5b).

Ultimately, the architecture of a neural network model is driven by the task it is in- tended to carry out. For instance, neural network models have been used as classifiers, as forecasting tools, as customer segmentation mechanisms, and as general optimizers. As shown later in this chapter, neural network classifiers are typically multilayer models in which information is passed from one layer to the next, with the ultimate goal of map- ping an input to the network to a specific category, as identified by an output of the net- work. A neural model used as an optimizer, in contrast, can be a single layer of neurons, can be highly interconnected, and can compute neuron values iteratively until the model converges to a stable state. This stable state represents an optimal solution to the problem under analysis.

Application Case 5.2 summarizes the use of predictive modeling (e.g., neural net- works) in addressing emerging problems in the electric power industry.

The electrical power industry produces and deliv- ers electric energy (electricity or power) to both residential and business customers wherever and whenever they need it. Electricity can be generated from a multitude of sources. Most often, electricity is produced at a power station using electromechanical generators that are driven by heat engines fueled by chemical combustion (by burning coal, petroleum, or natural gas) or nuclear fusion (by a nuclear reac- tor). Generation of electricity can also be accom- plished by other means, such as kinetic energy (through falling/flowing water or wind that activates turbines), solar energy (through the energy emitted by sun, either light or heat), or geothermal energy (through the steam or hot water coming from deep layers of the earth). Once generated, electric energy is distributed through a power grid infrastructure.

Even though some energy-generation methods are favored over others, all forms of electricity gen- eration have positive and negative aspects. Some are environmentally favored but are economically unjustifiable; others are economically superior but environmentally prohibitive. In a market economy,

the options with fewer overall costs are generally chosen above all other sources. It is not clear yet which form can best meet the necessary demand for electricity without permanently damaging the environment. Current trends indicate that increas- ing the shares of renewable energy and distributed generation from mixed sources has the promise of reducing/balancing environmental and economic risks.

The electrical power industry is a highly regu- lated, complex business endeavor. There are four distinct roles that companies choose to participate in: power producers, transmitters, distributers, and retail- ers. Connecting all of the producers to all of the cus- tomers is accomplished through a complex structure, called the power grid. Although all aspects of the elec- tricity industry are witnessing stiff competition, power generators are perhaps the ones getting the lion’s share of it. To be competitive, producers of power need to maximize the use of their variety of resources by making the right decisions at the right rime.

StatSoft, one of the fastest growing provid- ers of customized analytics solutions, developed

Application Case 5.2 Predictive Modeling Is Powering the Power Generators

(Continued )

262 Part II • Predictive Analytics/Machine Learning

integrated decision support tools for power gen- erators. Leveraging the data that come from the production process, these data mining–driven soft- ware tools help technicians and managers rapidly optimize the process parameters to maximize the power output while minimizing the risk of adverse effects. Following are a few examples of what these advanced analytics tools, which include ANN and SVM, can accomplish for power generators.

• Optimize Operation Parameters Problem: A coal-burning 300 MW multi- cyclone unit required optimization for consis- tent high flame temperatures to avoid forming slag and burning excess fuel oil. Solution: Using StatSoft’s predictive model- ing tools (along with 12 months of three-minute historical data), optimized control parameter settings for stoichiometric ratios, coal flows, primary air, tertiary air, and split secondary air damper flows were identified and implemented. Results: After optimizing the control pa- rameters, flame temperatures showed strong responses, resulting in cleaner combustion for higher and more stable flame temperatures.

• Predict Problems Before They Happen Problem: A 400 MW coal-fired DRB-4Z burner required optimization for consistent and robust low NOx operations to avoid ex- cursions and expensive downtime. Identify root causes of ammonia slip in a selective non- catalytic reduction process for NOx reduction. Solution: Apply predictive analytics method- ologies (along with historical process data) to predict and control variability; then target pro- cesses for better performance, thereby r educing both average NOx and variability. Results: Optimized settings for combinations of control parameters resulted in consistently lower NOx emissions with less variability (and

no excursions) over continued operations at low load, including predicting failures or unexpected maintenance issues.

• Reduce Emission (NOx, CO) Problem: While NOx emissions for higher loads were within acceptable ranges, a 400 MW coal-fired DRB-4Z burner was not optimized for low-NOx operations under low load (50–175 MW). Solution: Using data-driven predictive modeling technologies with historical data, optimized parameter settings for changes to airflow were identified, resulting in a set of specific, achievable input parameter ranges that were easily implemented into the existing DCS (digital control system). Results: After optimization, NOx emissions under low-load operations were comparable to NOx emissions under higher loads.

As these specific examples illustrate, there are numerous opportunities for advanced analytics to make a significant contribution to the power industry. Using data and predictive models could help deci- sion makers get the best efficiency from their pro- duction system while minimizing the impact on the environment.

Questions for Case 5.2

1. What are the key environmental concerns in the electric power industry?

2. What are the main application areas for predic- tive modeling in the electric power industry?

3. How was predictive modeling used to address a variety of problems in the electric power industry?

Source: Based on the StatSoft, Success Stories, statsoft.com/Portals/ 0/Downloads/EPRI.pdf (accessed June 2018) and the statsoft.fr/ pdf/QualityDigest_Dec2008.pdf (accessed February 2018).

u SECTION 5.3 REVIEW QUESTIONS

1. What are the most popular neural network architectures? 2. What types of problems are solved with Kohonen SOM ANN architecture? 3. How does Hopfield ANN architecture work? To what type of problems can it be

applied?

Application Case 5.2 (Continued)

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 263

5.4 SUPPORT VECTOR MACHINES

Support vector machines are among the popular machine-learning techniques, mostly because of their superior predictive power and their theoretical foundation. SVM are among the supervised learning techniques that produce input-output functions from a set of labeled training data. The function between the input and output vectors can be either a classification (used to assign cases into predefined classes) or a regression (used to estimate the continuous numerical value of the desired output). For classification, nonlinear kernel functions are often used to transform input data (naturally representing highly complex nonlinear relationships) to a high-dimensional feature space in which the input data become linearly separable. Then, the maximum-margin hyperplanes are constructed to optimally separate the output classes from each other in the training data.

Given a classification-type prediction problem, generally speaking, many linear classifiers (hyperplanes) can separate the data into multiple subsections, each represent- ing one of the classes (see Figure 5.6a where the two classes are represented with circles [“ ”] and squares [“ ”]). However, only one hyperplane achieves the maximum separa- tion between the classes (see Figure 5.6b where the hyperplane and the two maximum margin hyperplanes are separating the two classes).

Data used in SVM can have more than two dimensions (i.e., two distinct classes). In that case, we would be interested in separating data using the n - 1 dimensional hyperplane, where n is the number of dimensions (i.e., class labels). This can be seen as a typical form of linear classifier where we are interested in finding the n - 1 hyperplane so that the distance from the hyperplanes to the nearest data points are maximized. The assumption is that the larger the margin or distance between these parallel hyperplanes, the better the generalization power of the classifier (i.e., pre- diction power of the SVM model). If such hyperplanes exist, they can be mathe- matically represented using quadratic optimization modeling. These hyperplanes are known as the maximum-margin hyperplane, and such a linear classifier is known as a maximum-margin classifier.

In addition to their solid mathematical foundation in statistical learning theory, SVM have also demonstrated highly competitive performance in numerous real-world predic- tion problems, such as medical diagnosis, bioinformatics, face/voice recognition, demand

L1

L2 L3

2 w

w . x 2

b 5

2 1

w . x 2

b 5

0

w . x 2

b 5

1

(a) (b)

X2 X2

X1 X1

FIGURE 5.6 Separation of the Two Classes Using Hyperplanes.

264 Part II • Predictive Analytics/Machine Learning

As technology keeps advancing, new and improved safety measures are being developed and incorpo- rated into vehicles and roads to prevent crashes from happening and/or reduce the impact of the injury sustained by passengers caused by such inci- dents. Despite the extent of these efforts, the num- ber of vehicle crashes and the resulting injuries are increasing worldwide. For instance, according to the National Highway Traffic Safety Administration (NHTSA), in the United States more than 6 million traffic accidents claim over 30,000 lives and injure more than 2 million people each year (NHTSA, 2014). The latest NHTSA report presented to the U.S. Congress in April 2014 stated that in 2012, highway fatalities in the United States reached 33,561, which is an increase of 1,082 over the pre- vious year (Friedman, 2014). In the same year, an estimated 2.36 million people were injured in motor vehicle traffic crashes compared to 2.22 million in 2011. As a result, an average of nearly four lives were lost and nearly 270 people were injured on America’s roadways every hour in 2012. In addition to the staggering number of fatalities and injuries, these traffic accidents also cost the taxpayers more than $230 billion. Hence, addressing road safety is a major problem in the United States.

Root causes of traffic accidents and crash- related injury severity are of special concern to the general public and to researchers (in academia, government, and industry) because such investiga- tion aimed not only at prevention of crashes but also at reduction of their severe outcomes, poten- tially saving many lives and money. In addition to laboratory- and experimentation-based engineer- ing research methods, another way to address the issue is to identify the most probable factors that affect injury severity by mining the historical data

on vehicle crashes. Thorough understanding of the complex circumstances in which drivers and/or passengers are more likely to sustain severe inju- ries or even be killed in a vehicle crash can miti- gate the risks involved to a great extent, thereby saving lives due to crashes. Many factors were found to have an impact on the severity of injury sustained by occupants in the event of a vehi- cle accident. These factors include behavioral or demographic features of the occupants (e.g., drug and/or alcohol levels, seatbelt or other restrain- ing system usage, gender and age of the driver), crash-related situational characteristics (e.g., road surface/type/situation, direction of impact, strike versus struck, number of cars and/or other objects involved), environmental factors at the time of the accident (weather conditions, visibility and/or light conditions, time of the day, etc.), and the techni- cal characteristics of the vehicle itself (age, weight, body type, etc.).

The main goal of this analytic study was to determine the most prevailing risk factors and their relative importance/significance in influencing the likelihood of increasing severity of injury caused by vehicle crashes. The crashes examined in this study included a collection of geographically well- represented samples. To have a consistent sample, the data set comprised only collations of specific types: single or multi-vehicle head-on collisions, single or multi-vehicle angled collisions, and single- vehicle fixed-object collisions. To obtain reliable and accurate results, this investigative study employed the most prevalent machine-learning techniques to identify the significance of crash-related factors as they relate to the changing levels of injury sever- ity in vehicle crashes and compared the different machine-learning techniques.

Application Case 5.3 Identifying Injury Severity Risk Factors in Vehicle Crashes with Predictive Analytics

forecasting, image processing, and text mining, which has established SVM as among the most popular analytics tools for knowledge discovery and data mining. Similar to artificial neural networks, SVM possess the well-known ability of being universal approximators of any multivariate function to any desired degree of accuracy. Therefore, SVM are of par- ticular interest to modeling highly nonlinear, complex problems, systems, and processes. In the research study summarized in Application Case 5.3, SVM were better than other machine-learning methods in predicting and characterizing injury severity risk factors in automobile crashes.

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 265

The Research Method

The methodology employed in this study follows a very well-known standardized analytics process, namely cross-industry standard process for data mining (CRISP-DM). As is the case in any analyt- ics project, a significant portion of the project time was devoted to the acquisition, integration, and preprocessing of data. Then, the preprocessed, analytics-ready data were used to build several dif- ferent prediction models. Using a set of standard metrics, researchers assessed the outcomes of these models and compared them. In the final stage, sen- sitivity analyses were used to identify the most pre- vailing injury-severity related risk factors.

To effectively and efficiently perform the indi- vidual tasks in the proposed methodology, several statistical and data mining software tools were used. Specifically, JMP (a statistical and data mining soft- ware tool developed by SAS Institute), Microsoft Excel, and Tableau were used for inspecting, under- standing, and preprocessing the data; IBM SPSS Modeler and KNIME were used for data merging, predictive model building, and sensitivity analysis.

The National Automotive Sampling System General Estimates System (NASS GES) data set was used for covered accidents in the years 2011 and 2012. The complete data set was obtained in the form of three separate flat/text files—accident, vehi- cle, and person. The accident files contained spe- cific characteristics about road conditions, environ- mental conditions, and crash-related settings. The vehicle files included a large number of variables about the specific features of the vehicle involved in the crash. The person files provided detailed demographics, injury, and situational information about the occupants (i.e., driver and the passengers) impacted in the crash. To consolidate the data into a single database, the two years of data were merged within each file types (i.e., accident, person, vehi- cle), and the resulting files were combined using unique accident, vehicle, and person identifiers to create a single data set. After the data consolidation/ aggregation, the resulting data set included person- level records—one record per person involved in a reported vehicle crash. At this point in the process (before the data cleaning, preprocessing and slic- ing/dicing), the complete data set included 279,470 unique records (i.e., persons/occupants involved in crashes) and more than 150 variables (a combination

of accident, person, and vehicle related characteris- tics). Figure 5.7 graphically illustrates the individual steps involved in the processing of data.

Of all the variables—directly obtained from the GES databases and the ones that were derived/ recalculated using the existing GES variables—29 were selected as relevant and potentially influential in determining the varying levels of injury severity involved in vehicle crashes. This extent of variables was expected to provide a rich description of the people and the vehicle involved in the accident: the specifics of the environmental conditions at the time of the crash, the settings surrounding the crash itself, and the time and place of the crash. Table 5.2 lists

Data Preprocessing Selecting Characterizing Aggregating

Accident DB Vehicle DB Person DB

Combined DB (279,470 rows,

152 cols)

2011 2012

Pre-processed Data

2011 2012 2011 2012

FIGURE 5.7 Data Acquisition/Merging/Preparation Process.

(Continued )

Source: Microsoft Excel 2010, Microsoft Corporation.

266 Part II • Predictive Analytics/Machine Learning

TABLE 5.2 List of Variables Included in the Study

Variable Description Data Type Descriptive Statistics1 Missing (%)

AIR_BAG Airbag deployed Binary Yes: 52, no: 26 5.2

ALC_RES Alcohol test results Numeric 12.68 115.052 0.4 BDYTYP_IMN Vehicle body type Nominal Sedan: 34, Sm-SUV: 13 3.2

DEFORMED Extent of damage Nominal Major: 43, minor: 22 3.7

DRINKING Alcohol involvement Binary Yes: 4, no: 67 28.8

AGE Age of person Numeric 36.45 (18.49) 6.9

DRUGRES1 Drug test results Binary Yes: 2, no: 72 25.5

EJECT_IM Ejection Binary Yes: 2, no: 93 4.9

FIRE_EXP Fire occurred Binary Yes: 3, no: 97 0.0

GVWR Vehicle weight category Nominal Small: 92, large: 5 2.9

HAZ_INV Hazmat involved Binary Yes: 1, no: 99 0.0

HOUR_IMN Hour of day Nominal Evening: 39, noon: 32 1.2

INT_HWY Interstate highway Binary Yes: 13, no: 86 0.7

J_KNIFE Jackknife Binary Yes: 4, no: 95 0.2

LGTCON_IM Light conditions Nominal Daylight: 70, dark: 25 0.3

MANCOL_IM Manner of collision Nominal Front: 34, angle: 28 0.0

MONTH Month of year Nominal Oct: 10, Dec: 9 0.0

NUMINJ_IM Number of injured Numeric 1.23 14.132 0.0 PCRASH1_IMN Precrash movement Nominal Going str.: 52, stopped: 14 1.3

REGION Geographic region Nominal South: 42, Midwest: 24 0.0

REL_ROAD Relation to traffic way Nominal Roadway: 85, median: 9 0.1

RELJCT1_IM At a junction Binary Yes: 4, no: 96 0.0

REST_USE_N Restraint system used Nominal Yes: 76, no: 4 7.4

SEX_IMN Gender of driver Binary Male: 54, female: 43 3.1

TOWED_N Car towed Binary Yes: 49, no: 51 0.0

VEH_AGE Age of vehicle Numeric 8.96 14.182 0.0 WEATHR_IM Weather condition Nominal Clear: 73, cloudy: 14 0.0

WKDY_IM Weekday Nominal Friday: 17, Thursday 15 0.0

WRK_ZONE Work zone Binary Yes: 2, no: 98 0.0

INJ_SEV Injury severity (Dependent Variable)

Binary Low: 79, high: 21 0.0

1For numeric variables: mean (st. dev.); for binary or nominal variables: % frequency of the top two classes.

Application Case 5.3 (Continued)

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 267

and briefly describes the variables created and used for this study.

Table 5.3 shows the predictive accuracies of all four model types. It shows the confusion matrices, overall accuracy, sensitivity, specificity, and area under the receiver operating characteristics (ROC) curve measures obtained using 10-fold cross-validation for all four mode types. As the results indicate, SVM was the most accurate classification technique with bet- ter than 90 percent overall accuracy, comparably high sensitivity and specificity, and an area under the curve (AUC) value of 0 .928 (of maximum 1 .000) . The next best model type was C5 decision tree algorithms with slightly better accuracy than ANN. The last in the accuracy ranking was LR, also with fairly good accuracy measures but not as good as the machine- learning methods.

Even though the accuracy measures obtained from all four model types were high enough to validate the proposed methodology, the main goal of this study was to identify and prioritize the sig- nificant risk factors influencing the level of injury severity sustained by drivers during a vehicle crash. To achieve this goal, a sensitivity analysis on all of the developed prediction models was con- ducted. Focusing on each model type individually, the variable importance measures for each fold were calculated using leave-one-out method, and then the results obtained were summed for each

model type. To properly fuse (i.e., ensemble) the sensitivity analysis results for all four model types, the models’ contribution to the fused/combined variable importance values were determined based on their cross-validation accuracy. That is, the best performing model type had the largest weight/con- tribution while the worst performing model type had the smallest weight/contribution. The fused variable importance values were tabulated, normal- ized, and then graphically presented in Figure 5.8.

Examination of the sensitivity analysis results revealed four somewhat distinct risk groups, each comprising four to eight variables. The top group, in an order from most to least importance, included REST_USE_N (whether the seat belt of any other restraining system was used), MANCOL_IM (man- ner of collision), EJECT_IM (whether the driver was ejected from the car), and DRUGRES1 (results of the drug test). According to the combined sensitivity analysis results of all prediction models, these four risk factors seemed to be significantly more impor- tant than the rest.

Questions for Case 5.3

1. What are the most important motivations behind analytically investigating car crashes?

2. How were the data in the Application Case acquired, merged, and reprocessed?

TABLE 5.3 Tabulation of All Prediction Results Based on 10-fold Cross-Validation

Model Type

Confusion Matrices Accuracy (%)

Sensitivity (%)

Specificity (%) AUCLow High

Artificial neural networks (ANN)

Low 12,864 1,464 85.77 81.31 89.78 0.865

High 2,409 10,477

Support vector machines (SVM)

Low 13,192 1,136 90.41 88.55 92.07 0.928

High 1,475 11,411

Decision trees (DT>C5)

Low 12,675 1,653 86.61 84.55 88.46 0.8790

High 1,991 10,895

Logistic regression (LR)

Low 8,961 2,742 76.97 77.27 76.57 0.827

High 3,525 11,986

(Continued )

268 Part II • Predictive Analytics/Machine Learning

3. What were the results of this study? How can these findings be used for practical purposes?

Sources: D. Delen, L. Tomak, K. Topuz, & E. Eryarsoy, “Investigating Injury Severity Risk Factors in Automobile Crashes with Predictive Analytics and Sensitivity Analysis Methods,” Journal of Transport

& Health, 4, 2017, pp. 118–131; D. Friedman, “Oral Testimony Before the House Committee on Energy and Commerce, by the Subcommittee on Oversight and Investigations,” April 1, 2014, www. nhtsa.gov/Testimony (accessed October 2017); National Highway Traffic Safety Administration (NHTSA’s) (2018) General Estimate System (GES), www.nhtsa.gov (accessed January 20, 2018).

FIGURE 5.8 Variable Importance Values.

0 20 40 60 80 100

GVWR

WRK_ZONE

TOWED_N

DRINKING_N

BDYTYP_IMN

HAZ_INV

J_KNIFE

ALC_RES_N

REGION

NUMINJ_IM

LGTCON_IM

FIRE_EXP

VEH_AGE

RELJCT1_IM

HOUR_IMN

MONTH

DEFORMED

SEX_IMN

WEATHR_IM

PCRASH1_IMN

INT_HWY

WKDY_IM

DRIVER_AGE

REL_ROAD

AIRBAG_DEPL_N

DRUGRES1

EJECT_IM

MANCOL_IM

REST_USE_N

0

0

0

0

0

0

1

2

2

2

4

4

6

8

8

9

11

14

19

20

24

24

30

32

39

65

76

86

100

Normalized Importance Measure

Group 1

Group 2

Group 3

Group 4

Application Case 5.3 (Continued)

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 269

Mathematical Formulation of SVM

Consider data points in the training data set of the form:

{(x1,c1), (x2,c2), . . . , (xn,cn)}

where the c is the class label taking a value of either 1 (i.e., “yes”) or 0 (i.e., “no”) while x is the input variable vector. That is, each data point is an m- dimensional real vector, usually of scaled 30, 14 or 3-1, 14 values. The normalization and/or scaling are important steps to guard against variables/attributes with larger variance that might otherwise dominate the classification formulae. We can view this as training data, which denote the correct classification (something that we would like the SVM to eventually achieve) by means of a dividing hyperplane, which takes the mathematical form

w # x - b = 0. The vector w points perpendicularly to the separating hyperplane. Adding the offset

parameter b allows us to increase the margin. In its absence, the hyperplane is forced to pass through the origin, restricting the solution. Because we are interested in the maxi- mum margin, we also are interested in the support vectors and the parallel hyperplanes (to the optimal hyperplane) closest to these support vectors in either class. It can be shown that these parallel hyperplanes can be described by equations

w # x - b = 1, w # x - b = -1.

If the training data are linearly separable, we can select these hyperplanes so that there are no points between them and then try to maximize their distance (see Figure 5.6b). By using geometry, we find the distance between the hyperplanes is 2> � w �, so we want to minimize � w �. To exclude data points, we need to ensure that for all i either

w # xi - b Ú 1 or

w # xi - b … -1. This can be rewritten as:

ci (w # xi - b) Ú 1, 1 … i … n Primal Form

The problem now is to minimize � w � subject to the constraint ci(w # xi - b) Ú 1, 1 … i … n. This is a quadratic programming (QP) optimization problem. More clearly,

Minimize 11>22 7w 72 Subject to ci(w # xi - b) Ú 1, 1 … i … n

The factor of 1>2 is used for mathematical convenience. Dual Form

Writing the classification rule in its dual form reveals that classification is only a function of the support vectors, that is, the training data that lie on the margin. The dual of the SVM can be shown to be:

max a n

i = 1 ai - a

i, j ai aj ci cj xi

T xj

270 Part II • Predictive Analytics/Machine Learning

where the a terms constitute a dual representation for the weight vector in terms of the training set:

w = a i ai ci xi

Soft Margin

In 1995, Cortes and Vapnik suggested a modified maximum margin idea that allows for mislabeled examples. If there exists no hyperplane that can split the “yes” and “no” examples, the soft margin method will choose a hyperplane that splits the examples as cleanly as possible while still maximizing the distance to the nearest cleanly split examples. This work popularized the expression support vector machine or SVM. The method introduces slack variables, ji, which measure the degree of misclassification of the datum.

ci(w # xi - b) Ú 1 - ji 1 … i … n The objective function is then increased by a function that penalizes non-zero ji,

and the optimization becomes a trade-off between a large margin and a small error pen- alty. If the penalty function is linear, the equation then transforms to

min 7w 72 + C a i ji such that ci (w # xi - b) Ú 1 - ji 1 … i … n

This constraint along with the objective of minimizing � w � can be solved using Lagrange multipliers. The key advantage of a linear penalty function is that the slack vari- ables vanish from the dual problem with the constant C appearing only as a v@additional constraint on the Lagrange multipliers. Nonlinear penalty functions have been used, partic- ularly to reduce the effect of outliers on the classifier, but unless care is taken, the problem becomes nonconvex, and thus it is considerably more difficult to find a global solution.

Nonlinear Classification

The original optimal hyperplane algorithm proposed by Vladimir Vapnik in 1963 while he was a doctoral student at the Institute of Control Science in Moscow was a linear clas- sifier. However, in 1992, Boser, Guyon, and Vapnik suggested a way to create nonlinear classifiers by applying the kernel trick (originally proposed by Aizerman et al., 1964) to maximum-margin hyperplanes. The resulting algorithm is formally similar, except that every dot product is replaced by a nonlinear kernel function. This allows the algorithm to fit the maximum-margin hyperplane in the transformed feature space. The transformation can be nonlinear and the transformed space high dimensional; thus, although the classifier is a hyperplane in the high-dimensional feature space, it can be nonlinear in the original input space.

If the kernel used is a Gaussian radial basis function, the corresponding feature space is a Hilbert space of infinite dimension. Maximum margin classifiers are well regular- ized, so the infinite dimension does not spoil the results. Some common kernels include:

Polynomial (homogeneous): k(x, x′) = (x # x′) Polynomial (inhomogeneous): k(x, x′) = (x # x′ + 1) Radial basis function: k(x, x′) = exp (-g 7x - x′ 72), for g 7 0 Gaussian radial basis function: k(x, x′) = expa-

7x - x′ 72 2s2

b

Sigmoid: k(x, x′) = tan h (kx # x′ + c) for some k 7 0 and c 6 0

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 271

Kernel Trick

In machine learning, the kernel trick is a method for converting a linear classifier algo- rithm into a nonlinear one by using a nonlinear function to map the original observa- tions into a higher-dimensional space; this makes a linear classification in the new space equivalent to nonlinear classification in the original space.

This is done using Mercer’s theorem, which states that any continuous, symmetric, positive semi-definite kernel function K1x, y2 can be expressed as a dot product in a high-dimensional space. More specifically, if the arguments to the kernel are in a measur- able space X and if the kernel is positive semi-definite—that is,

a i, j

K (xi, xj)ci cj Ú 0

for any finite subset 5x1, c, xn6 of X and subset 5c1, c, cn6of objects (typically real numbers or even molecules)—then there exists a function w1x2 whose range is in an inner product space of possibly high dimension, such that

K(x, y) = w(x) # w( y) The kernel trick transforms any algorithm that solely depends on the dot product

between two vectors. Wherever a dot product is used, it is replaced with the kernel func- tion. Thus, a linear algorithm can easily be transformed into a nonlinear algorithm. This nonlinear algorithm is equivalent to the linear algorithm operating in the range space of w. However, because kernels are used, the w function is never explicitly computed. This is desirable because the high-dimensional space could be infinite-dimensional (as is the case when the kernel is a Gaussian).

Although the origin of the term kernel trick is not known, it was first published by Aizerman et al. (Aizerman et al., 1964). It has been applied to several kinds of algorithm in machine learning and statistics, including:

• Perceptrons • Support vector machines • Principal components analysis • Fisher’s linear discriminant analysis • Clustering

u SECTION 5.4 REVIEW QUESTIONS

1. How do SVM work? 2. What are the advantages and disadvantages of SVM? 3. What is the meaning of “maximum-margin hyperplanes”? Why are they important

in SVM?

4. What is the “kernel trick”? How is it used in SVM?

5.5 PROCESS-BASED APPROACH TO THE USE OF SVM

Due largely to the better classification results, SVM recently have become a popular tech- nique for classification-type problems. Even though people consider them as being easier to use than artificial neural networks, users who are not familiar with the intricacies of SVM often get unsatisfactory results. In this section, we provide a process-based approach to the use of SVM, which is more likely to produce better results. A pictorial representa- tion of the three-step process is given in Figure 5.9.

272 Part II • Predictive Analytics/Machine Learning

NUMERICIZING THE DATA SVM require that each data instance be represented as a vec- tor of real numbers. Hence, if there are categorical attributes, we first have to convert them into numeric data. A common recommendation is to use m pseudo-binary variables to represent an m-class attribute (where m Ú 3). In practice, only one of the m variables assumes the value of 1 and others assume the value of 0 based on the actual class of the case (this is also called 1-of-m representation). For example, a three-category attribute such as {red, green, blue} can be represented as (0,0,1), (0,1,0), and (1,0,0) .

NORMALIZING THE DATA As was the case for artificial neural networks, SVM also require normalization and/or scaling of numerical values. The main advantage of normalization is to avoid attributes in greater numeric ranges dominating those in smaller numeric ranges. Another advantage is that it helps performing numerical calculations during the iterative process of model building. Because kernel values usually depend on the inner products of feature vectors (e.g., the linear kernel and the polynomial kernel), large attribute val- ues might slow the training process. Use recommendations to normalize each attribute to the range [-1, +1] or [0, 1] . Of course, we have to use the same normalization method to scale testing data before testing.

SELECT THE KERNEL TYPE AND KERNEL PARAMETERS Even though there are only four common kernels mentioned in the previous section, one must decide which one to use

Preprocess the Data

Scrub the data “Identify and handle missing, incorrect, and noisy” Transform the data “Numerisize, normalize, and standardize the data”

Develop the Model

Select the kernel type “Choose from RBF, sigmoid, or polynomial kernel types” Determine the kernel values “Use v -fold cross validation or employ ‘grid-search’”

Deploy the Model

Extract the model coefficients Code the trained model into the decision support system Monitor and maintain the model

Training Data

Preprocessed Data

Validated SVM Model

Prediction Model

Experimentation “Training/Testing”

DB

FIGURE 5.9 Simple Process Description for Developing SVM Models.

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 273

(or whether to try them all, one at a time, using a simple experimental design approach). Once the kernel type is selected, then one needs to select the value of penalty parameter C and kernel parameters. Generally speaking, Radial Basis Function (RBF) is a reason- able first choice for the kernel type. The RBF kernel aims to nonlinearly map data into a higher dimensional space; by doing so (unlike with a linear kernel), it handles the cases in which the relation between input and output vectors is highly nonlinear. Besides, one should note that the linear kernel is just a special case of RBF kernel. There are two pa- rameters to choose for RBF kernels: C and g. It is not known beforehand which C and g are the best for a given prediction problem; therefore, some kind of parameter search method needs to be used. The goal for the search is to identify optimal values for C and g so that the classifier can accurately predict unknown data (i.e., testing data). The two most commonly used search methods are cross-validation and grid search.

DEPLOY THE MODEL Once an “optimal” SVM prediction model has been developed, the next step is to integrate it into the decision support system. For that, there are two options: (1) converting the model into a computational object (e.g., a Web service, Java Bean, or COM object) that takes the input parameter values and provides output prediction and (2) extracting the model coefficients and integrating them directly into the decision sup- port system. The SVM models are useful (i.e., accurate, actionable) only if the behavior of the underlying domain stays the same. For some reason, if it changes, so does the ac- curacy of the model. Therefore, one should continuously assess the performance of the models and decide when they no longer are accurate, and, hence, need to be retrained.

Support Vector Machines versus Artificial Neural Networks

Even though some people characterize SVM as a special case of ANN, most recognize them as two competing machine-learning techniques with different qualities. Here are a few points that help SVM stand out against ANN. Historically, the development of ANN followed a heuristic path with applications and extensive experimentation preced- ing theory. In contrast, the development of SVM involved sound statistical learning theory first and then implementation and experiments. A significant advantage of SVM is that while ANN could suffer from multiple local minima, the solutions to SVM are global and unique. Two more advantages of SVM are that they have a simple geometric interpreta- tion and give a sparse solution. The reason that SVM often outperform ANN in practice is that they successfully deal with the “over fitting” problem, which is a big issue with ANN.

Although SVM have these advantages (from a practical point of view), they have some limitations. An important issue that is not entirely solved is the selection of the ker- nel type and kernel function parameters. A second and perhaps more important limitation of SVM involves its speed and size, both in the training and testing cycles. Model building in SVM involves complex and time-demanding calculations. From the practical point of view, perhaps the most serious problem with SVM is the high algorithmic complexity and extensive memory requirements of the required quadratic programming in large-scale tasks. Despite these limitations, because SVM are based on a sound theoretical founda- tion and the solutions they produce are global and unique in nature (as opposed to get- ting stuck in a suboptimal alternative such as a local minimum), today they are arguably among the most popular prediction modeling techniques in the data mining arena. Their use and popularity will only increase as the popular commercial data mining tools start to incorporate them into their modeling arsenal.

u SECTION 5.5 REVIEW QUESTIONS

1. What are the main steps and decision points in developing an SVM model? 2. How do you determine the optimal kernel type and kernel parameters?

274 Part II • Predictive Analytics/Machine Learning

3. Compared to ANN, what are the advantages of SVM? 4. What are the common application areas for SVM? Search the Internet to identify

popular application areas and specific SVM software tools used in those applications.

5.6 NEAREST NEIGHBOR METHOD FOR PREDICTION

Data mining algorithms tend to be highly mathematical and computationally intensive. The two popular ones that are covered in the previous section (i.e., ANN and SVM) involve time- demanding, computationally intensive iterative mathematical derivations. In contrast, the k-nearest neighbor algorithm (or kNN in short) seems overly simplistic for a competitive prediction method. What it does and how it does it are so easy to understand (and explain to others). k-NN is a prediction method for classification—as well as regression-type prediction problems. k-NN is a type of instance-based learning (or lazy learning) since the function is approximated only local and all computations are deferred until the actual prediction.

The k-nearest neighbor algorithm is among the simplest of all machine-learning al- gorithms: For instance, in the classification-type prediction, a case is classified by a major- ity vote of its neighbors with the object being assigned to the class most common among its k nearest neighbors (where k is a positive integer). If k = 1, then the case is simply assigned to the class of its nearest neighbor. To illustrate the concept with an example, let us look at Figure 5.10 where a simple two-dimensional space represents the values for the two variables (x, y); the star represents a new case (or object); and circles and squares represent known cases (or examples). The task is to assign the new case to either circles or squares based on its closeness (similarity) to one or the other. If you set the value of k to 1 (k = 1), the assignment should be made to square because the closest example to star is a square. If you set the value of k to 3 (k = 3), the assignment should be made to circle because there two circles and one square; hence, from the simple majority vote rule, the circle gets the assignment of the new case. Similarly, if you set the value of k to 5 (k = 5), then the assignment should be made to square class. This overly simplified example is meant to illustrate the importance of the value that one assigns to k.

The same method can be used for regression-type prediction tasks by simply av- eraging the values of its k nearest neighbors and assigning this result to the case being predicted. It can be useful to weight the contributions of the neighbors so that the nearer

y

yi

xi x

k 5 3

k 5 5

FIGURE 5.10 The Importance of the Value of k in kNN Algorithm.

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 275

neighbors contribute more to the average than the more distant ones. A common weight- ing scheme is to give each neighbor a weight of 1>d, where d is the distance to the neighbor. This scheme is essentially a generalization of linear interpolation.

The neighbors are taken from a set of cases for which the correct classification (or, in the case of regression, the numerical value of the output value) is known. This can be thought of as the training set for the algorithm even though no explicit training step is required. The k-nearest neighbor algorithm is sensitive to the local structure of the data.

Similarity Measure: The Distance Metric

One of the two critical decisions that an analyst has to make while using kNN is to determine the similarity measure (the other is to determine the value of k, which is explained next). In the kNN algorithm, the similarity measure is a mathematically calcu- lable distance metric. Given a new case, kNN makes predictions based on the outcome of the k neighbors closest in distance to that point. Therefore, to make predictions with kNN, we need to define a metric for measuring the distance between the new case and the cases from the examples. One of the most popular choices to measure this dis- tance is known as Euclidean (Eq. 2), which is simply the linear distance between two points in a dimensional space; the other popular one is the rectilinear (a.k.a. city-block or Manhattan distance) (Eq. 3). Both of these distance measures are special cases of Minkowski distance (Eq. 1).

Minkowski distance

d (i, j) = 3( � xi1 - xj 1 �q + � xi 2 - xj 2 �q + . . . + � xip - xjp �q ) (Eq. 1) where i = (xi1, xi2, . . . , xip ) and j = (xj1, xj2, . . . , xjp ) are two p-dimensional data objects (e.g., a new case and an example in the data set), and q is a positive integer.

If q = 1, then d is called Manhattan distance

d(i, j) = 3 � xi1 - xj1 � + � xi 2 - xj 2 � + . . . + � xip - xjp � (Eq. 2) If q = 2, then d is called Euclidean distance

d(i, j) = 3( � xi1 - xj1 �2 + �xi 2 - xj 2 �2 + . . . + � xip - xjp �2 ) (Eq. 3) Obviously, these measures apply only to numerically represented data. What

about nominal data? There are ways to measure distance for non-numerical data as well. In the simplest case, for a multi-value nominal variable, if the value of that vari- able for the new case and that for the example case are the same, the distance would be 0, otherwise 1. In cases such as text classification, more sophisticated metrics exist, such as the overlap metric (or Hamming distance). Often, the classification accuracy of kNN can be improved significantly if the distance metric is determined through an experimental design in which different metrics are tried and tested to identify the best one for the given problem.

Parameter Selection

The best choice of k depends upon the data. Generally, larger values of k reduce the effect of noise on the classification (or regression) but also make boundaries between classes less distinct. An “optimal” value of k can be found by some heuristic techniques, for in- stance, cross-validation. The special case in which the class is predicted to be the class of the closest training sample (i.e., when k = 1) is called the nearest neighbor algorithm.

CROSS-VALIDATION Cross-validation is a well-established experimentation technique that can be used to determine optimal values for a set of unknown model parameters.

276 Part II • Predictive Analytics/Machine Learning

It applies to most, if not all, of the machine-learning techniques that have a number of model parameters to be determined. The general idea of this experimentation method is to divide the data sample into a number of randomly drawn, disjointed subsamples (i.e., v number of folds). For each potential value of k, the kNN model is used to make predic- tions on the vth fold while using the v - 1 folds as the examples and to evaluate the error. The common choice for this error is the root-mean-squared-error (RMSE) for regression- type predictions and percentage of correctly classified instances (i.e., hit rate) for the classification-type predictions. This process of testing each fold against the remaining examples repeats v times. At the end of the v number of cycles, the computed errors are accumulated to yield a goodness measure of the model (i.e., how well the model predicts with the current value of the k). At the end, the k value that produces the smallest overall error is chosen as the optimal value for that problem. Figure 5.11 shows a simple process that uses the training data to determine optimal values for k and distance metric, which are then used to predict new incoming cases.

As we observed in the simple example given earlier, the accuracy of the kNN algo- rithm can be significantly different with different values of k. Furthermore, the predictive power of the kNN algorithm degrades with the presence of noisy, inaccurate, or irrelevant features. Much research effort has been put into feature selection and normalization/ scaling to ensure reliable prediction results. A particularly popular approach is the use of evolutionary algorithms (e.g., genetic algorithms) to optimize the set of features included in the kNN prediction system. In binary (two-class) classification problems, it is helpful to choose k to be an odd number because this would avoid tied votes.

A drawback to the basic majority voting classification in kNN is that the classes with the more frequent examples tend to dominate the prediction of the new vector because they tend to come up in the k nearest neighbors when the neighbors are computed due to their large number. One way to overcome this problem is to weigh the classification taking into account the distance from the test point to each of its k nearest neighbors. Another way to overcome this drawback is by one level of abstraction in data representation.

The naïve version of the algorithm is easy to implement by computing the distances from the test sample to all stored vectors, but it is computationally intensive, especially when the size of the training set grows. Many nearest neighbor search algorithms have

Historical Data

New Data

Parameter Setting

Distance metric Value of “k”

Training Set

Validation Set

Predicting

Classify (or forecast) new cases using k number of most similar cases

FIGURE 5.11 Process of Determining the Optimal Values for Distance Metric and k.

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 277

been proposed over the years; these generally seek to reduce the number of distance evaluations actually performed. Using an appropriate nearest neighbor search algo- rithm makes kNN computationally tractable even for large data sets. Refer to Application Case 5.4 about the superior capabilities of kNN in image recognition and categorization.

Image recognition is an emerging data mining appli- cation field involved in processing, analyzing, and categorizing visual objects such as pictures. In the process of recognition (or categorization), images are first transformed to a multidimensional feature space and then, using machine-learning techniques, are categorized into a finite number of classes. Application areas of image recognition and categori- zation range from agriculture to homeland security, personalized marketing to environmental protection. Image recognition is an integral part of an artifi- cial intelligence field called computer vision. As a technological discipline, computer vision seeks to develop computer systems that are capable of “see- ing” and reacting to their environment. Examples of applications of computer vision include systems for process automation (industrial robots), navigation (autonomous vehicles), monitoring/detecting (visual surveillance), searching and sorting visuals (indexing databases of images and image sequences), engag- ing (computer–human interaction), and inspection (manufacturing processes).

While the field of visual recognition and cate- gory recognition has been progressing rapidly, much remains to be done to reach human-level perfor- mance. Current approaches are capable of dealing with only a limited number of categories (100 or so) and are computationally expensive. Many machine- learning techniques (including ANN, SVM, and kNN) are used to develop computer systems for visual rec- ognition and categorization. Although commend- able results have been obtained, generally speaking, none of these tools in their current form is capable of developing systems that can compete with humans.

Several researchers from the Computer Science Division of the Electrical Engineering and Computer Science Department at the University of California–Berkeley used an innovative ensemble approach to image categorization (Zhang et al.,

2006). They considered visual category recognition in the framework of measuring similarities, or per- ceptual distances, to develop examples of catego- ries. The recognition and categorization approach the researchers used was quite flexible, permitting recognition based on color, texture, and particu- larly shape. While nearest neighbor classifiers (i.e., kNN) are natural in this setting, they suffered from the problem of high variance (in bias-variance decomposition) in the case of limited sampling. Alternatively, one could choose to use SVM, but they also involve time-consuming optimization and computations. The researchers proposed a hybrid of these two methods, which deals naturally with the multiclass setting, has reasonable computa- tional complexity both in training and at run time, and yields excellent results in practice. The basic idea was to find close neighbors to a query sample and train a local support vector machine that pre- serves the distance function on the collection of neighbors.

The researchers’ method can be applied to large, multiclass data sets when it outperforms near- est neighbor and SVM and remains efficient when the problem becomes intractable. A wide variety of distance functions were used, and their experiments showed state-of-the-art performance on a number of benchmark data sets for shape and texture classifica- tion (MNIST, USPS, CUReT) and object recognition (Caltech-101).

Another group of researchers (Boiman and Irani, 2008) argued that two practices commonly used in image classification methods (namely, SVM- and ANN-type model-driven approaches and kNN-type nonparametric approaches) have led to less-than-desired performance outcomes. These researchers also claimed that a hybrid method can improve the performance of image recognition and categorization. They proposed a trivial Naïve Bayes

Application Case 5.4 Efficient Image Recognition and Categorization with knn

(Continued )

278 Part II • Predictive Analytics/Machine Learning

u SECTION 5.6 REVIEW QUESTIONS

1. What is special about the kNN algorithm? 2. What are the advantages and disadvantages of kNN as compared to ANN and SVM? 3. What are the critical success factors for a kNN implementation? 4. What is a similarity (or distance) measure? How can it be applied to both numerical

and nominal valued variables?

5. What are the common applications of kNN?

5.7 NAÏVE BAYES METHOD FOR CLASSIFICATION

Naïve Bayes is a simple probability-based classification method (a machine-learning tech- nique that is applied to classification-type prediction problems) derived from the well- known Bayes theorem. The method requires the output variable to have nominal values. Although the input variables can be a mix of numeric and nominal types, the numeric output variable needs to be discretized via some type of binning method before it can be used in a Bayes classifier. The word “Naïve” comes from its strong, somewhat unrealistic, assumption of independence among the input variables. Simply put, a Naïve Bayes clas- sifier assumes that the input variables do not depend on each other, and the presence (or absence) of a particular variable in the mix of the predictors does not have anything to do with the presence or absence of any other variables.

Naïve Bayes classification models can be developed very efficiently (rather rapidly with very little computational effort) and effectively (quite accurately) in a supervised machine-learning environment. That is, by using a set of training data (not necessarily very large), the parameters for Naïve Bayes classification models can be obtained using the maximum likelihood method. In other words, because of the independence assump- tion, we can develop Naïve Bayes models without strictly complying with all of the rules and requirements of Bayes theorem. First let us review the Bayes theorem.

kNN-based classifier, which employs kNN distances in the space of the local image descriptors (not in the space of images). The researchers claimed that, although the modified kNN method is extremely simple and efficient and requires no learning/train- ing phase, its performance ranks among the top leading learning-based parametric image classifiers. Empirical comparisons of their method were shown on several challenging image categorization data- bases (Caltech-101, Caltech-256, and Graz-01).

In addition to image recognition and catego- rization, kNN is successfully applied to complex classification problems, such as content retrieval (handwriting detection, video content analysis, body and sign language (where communication is done using body or hand gestures), gene expression (another area in which kNN tends to perform bet- ter than other state-of-the-art techniques; in fact, a

combination of kNN-SVM is one of the most popular techniques used here), and protein-to-protein inter- action and 3D structure prediction (graph-based kNN is often used for interaction structure prediction).

Questions for Case 5.4

1. Why is image recognition/classification a worthy but difficult problem?

2. How can kNN be effectively used for image rec- ognition/classification applications?

Sources: H. Zhang, A. C. Berg, M. Maire, & J. Malik, “SVM- KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition,” Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), Vol. 2, 2006, pp. 2126–2136; O. Boiman, E. Shechtman, & M. Irani, “In Defense of Nearest-Neighbor Based Image Classification,” IEEE Conference on Computer Vision and Pattern Recognition, 2008 (CVPR), 2008, pp. 1–8.

Application Case 5.4 (Continued)

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 279

Bayes Theorem

To appreciate Naïve Bayes classification method, one would need to understand the basic definition of the Bayes theorem and the exact Bayes classifier (the one without the strong “Naïve” independence assumption). The Bayes theorem (also called Bayes Rule), named after the British mathematician Thomas Bayes (1701–1761), is a mathematical formula for determining conditional probabilities (the formula follows). In this formula, Y denotes the hypothesis and X denotes the data/evidence. This vastly popular theorem/rule provides a way to revise/improve prediction probabilities by using additional evidence.

The following formula shows the relationship between the probabilities of two events, Y and X. P (Y ) is the prior probability of Y. It is “prior” in the sense that it does not take into account any information about X. P(X>Y ) is the conditional probability of Y, given X. It is also called the posterior probability because it is derived from (or depends upon) the specified value of X. P(X>Y) is the conditional probability of X given Y. It is also called the likelihood. P (X ) is the prior probability of X, which is also called the evi- dence, and acts as the normalizing constant.

P (Y � X ) = P (X � Y )P (Y )

P (X ) S Posterior =

Likelihood * Prior

Evidence

P (Y � X ): Posterior probability of Y given X P (X � Y ): Conditional probability of X given Y (likelihood ) P(Y ): Prior probability of Y

P (X): Prior probability of X (evidence, or unconditional probability of X)

To numerically illustrate these formulas, let us look at a simple example. Based on the weather report, we know that there is a 40 percent chance of rain on Saturday. From the historical data, we also know that if it rains on Saturday, there is a 10 percent chance it will rain on Sunday; and if doesn’t rain on Saturday, there is an 80 percent chance it will rain on Sunday. Let us say that “Raining on Sunday” is event Y , and “Raining on Monday” is event X. Based on the description we can write the following:

P (Y ) = Probability of raining on Saturday = 0.40 P (X � Y ) = Probability of raining on Sunday if it rained on Saturday = 0.10 P(X) = Probability of raining on Monday = Sum of the probability of “Raining on Saturday and Raining on Sunday” and “Not Raining on Saturday and Raining on Sunday” = 0.40 * 0.10 + 0.60 * 0.80 = 0.52

Now if we were to calculate the probability for “It rained on Saturday?” given that it “Rained on Sunday,” we would use Bayes theorem. It would allow us to calculate the probability of an earlier event given the result of a later event.

P (X � Y ) = P ( X � Y )P (Y )

P (X ) =

0.10 * 0.40

0.52 = 0.0769

Therefore, in this example, if it rained on Sunday, there’s a 7.69 percent chance that it rained on Saturday.

Naïve Bayes Classifier

The Bayes classifier uses the Bayes theorem without the simplifying strong independence assumption. In a classification-type prediction problem, the Bayes classifier works as fol- lows: Given a new sample to classify, it finds all other samples exactly like it (i.e., all predic- tor variables having the same values as the sample being classified); determines the class labels that they all belong to; and classifies the new sample into the most representative class. If none of the samples has the exact value match with the new class, then the classifier

280 Part II • Predictive Analytics/Machine Learning

will fail in assigning the new sample into a class label (because the classifier could not find any strong evidence to do so). Here is a very simple example. Using the Bayes classifier, we are to decide whether to play golf (Yes or No) for the following situation (Outlook is Sunny, Temperature is Hot, Humidity is High, and Windy is No). Table 5.4 presents histori-

cal samples that will be used to illustrate the specifics of our classification process. Based on the historical data, three samples seem to match the situation (sample

numbers 1, 6, and 7 as highlighted in Table 5.4). Of the three, two of the samples have the class label “No” and one has the label “Yes.” Because the majority of the matched samples indicated “No,” the new sample/situation is to be classified as “No.”

Now let us consider a situation in which Outlook is Sunny, Temperature is Hot, Humidity is High, and Windy is Yes. Because there is no sample matching this value set, the Bayes classifier will not return a result. To find exact matches, there needs to be a very big data set. Even for the big data sets, as the number of predictor variables increases, the possibility of not finding an exact match increases significantly. When the data set and the number of predictor variables get larger, so does the time it takes to search for an exact match. All of these are the reasons why the Naïve Bayes classifier, a derivative of the Bayes classifier, is often used in predictive analytics and data mining practices. In the Naïve Bayes classifier, the exact match requirement is no longer needed. The Naïve Bayes classifier treats each predictor variable as an independent contributor to the prediction of the output variable and, hence, significantly increases its practicality and usefulness as a classification-type prediction tool.

Process of Developing a Naïve Bayes Classifier

Similar to other machine-learning methods, Naïve Bayes employs a two-phase model de- velopment and scoring/deployment process: (1) training in which the model/parameters are estimated and (2) testing in which the classification/prediction is performed on new cases. The process can be described as follows.

Training phase

Step 1. Obtain the data, clean the data, and organize them in a flat file format (i.e., col- umns as variables and rows as cases).

TABLE 5.4 Sample Data Set for the Classification-Type Prediction Methods

Input Variables (X) Output

Variable (Y)

Sample No. Outlook Temperature Humidity Windy Play Golf

1 Sunny Hot High No No

2 Overcast Hot High No Yes

3 Rainy Cool Normal No Yes

4 Rainy Cool Normal Yes No

5 Overcast Cool Normal Yes No

6 Sunny Hot High No No

7 Sunny Hot High No Yes

8 Rainy Mild Normal No Yes

9 Sunny Mild Normal Yes Yes

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 281

Step 2. Make sure that the variables are a nominal; if not (i.e., if any one of the variables is numeric/continuous), the numeric variables need to go through a data trans- formation (i.e., converting the numerical variable into nominal types by using discretization, such as binning).

Step 3. Calculate the prior probability of all class labels for the dependent variable. Step 4. Calculate the likelihood for all predictor variables and their possible values with

respect to the dependent variable. In the case of mixed variable types (categorical and continuous), each variable’s likelihood (conditional probability) is estimated with the proper method that applies to the specific variable type. Likelihoods for nominal and numeric predictor variables are calculated as follows:

• For categorical variables, the likelihood (the conditional probability) is esti- mated as the simple fraction of the training samples for the variable value with respect to the dependent variable.

• For numerical variables, the likelihood is calculated by (1) calculating the mean and variance for each predictor variable for each dependent variable value (i.e., class) and then (2) calculating the likelihood using the following formula:

P (x = v � c) = 122psc2 e - (v - mc )22sc2

Quite often, the continuous/numerical independent/input variables are discretized (using an appropriate binning method), and then the categorical variable estimation method is used to calculate the conditional probabilities (likelihood parameters). If performed properly, this method tends to produce better predicting Naïve Bayes models.

Testing Phase

Using the two sets of parameters produced in steps 3 and 4 in the training phase, any new sample can be put into a class label using the following formula:

Posterior = Prior x Likelihood

Evidence

P (C � F1, . . . , Fn ) = P (C ) P (F1, . . . , Fn � C )

P (F1, . . . , Fn)

Because the denominator is constant (the same for all class labels), we can remove it from the formula, leading to the following simpler formula, which is essentially nothing but the joint probability.

classify ( f1, . . . , fn ) = argmax p (C = c) q n

i = 1 p(Fi = fi � C = c)

This is a simple example to illustrate these calculations. In this example, we use the same data as shown in Table 5.4. The goal is to classify the following case: given that Outlook is Sunny, Temperature is Hot, Humidity is High, and Windy is No, what would be the class for the dependent variable (Play = Yes or No)?

From the data, we can observe that Prior (Yes) = 5>9 and Prior (No) = 4>9. For the Outlook variable, the likelihoods are Likelihood (No/Sunny) = 2/3;

Likelihood (No/Overcast); = ½; Likelihood (No/Rainy) = 1/3 . The likelihood values of the other variables (Temperature, Humidity, and Wind) can be determined/calculated similarly. Again, the case we are trying to classify is Outlook is Sunny, Temperature is Hot, Humidity is High, and Windy is No. The results are shown in Table 5.5.

c

282 Part II • Predictive Analytics/Machine Learning

Based on the results shown in Table 5.5, the answer would be Play Golf = Yes because it produces a larger value, 0.031 (compared to 0.025 for “No”) as per the joint probabilities (the simplified calculation without the inclusion of the denominator). If we were to use the full posterior formula for the two class labels, which requires the inclu- sion of the denominator in the calculations, we observe 0.07 for “Yes” and 0.056 for “No.” Because the denominator is common to all class labels, it will change the numerical out- put but not the class assignment.

Although Naïve Bayes is not very commonly used in predictive analytics projects today (because of its relatively poor prediction performance in a wide variety of applica- tion domains), one of its extensions, namely Bayesian network (see the next section), is gaining surprisingly rapid popularity among data scientists in the analytics world.

Application Case 5.5 provides an interesting example when many predictive an- alytics techniques are used to determine the changing condition of Crohn’s disease patients in order to better manage this debilitating chronic disease. Along with Naïve Bayes, several statistical and machine-learning methods were developed, tested, and compared. The best performing model was then used to explain the rank-ordered im- portance (i.e., relative contribution) of all independent variables used in predicting the disease progress.

TABLE 5.5 Naïve Bayes Classification Calculations

1Does not include the denominator/evidence for the calculations; hence, it is a partial calculation. 2 Include the denominator/evidence. Because the evidence is the same for all class labels (i.e., Yes and No), it does not make a difference in the classification result because both of the measures indicate class label as Yes.

Ratio Fraction (%>100) Play: Yes Play: No Play = Yes Play = No

Li k

e li

h o

o d

Outlook = Sunny 1/3 2/3 0.33 0.67

Temperature = Hot 2/4 2/4 0.50 0.50

Humidity = High 2/4 2/4 0.50 0.50

Wind = No 4/6 2/6 0.67 0.33

Prior 5/9 4/9 0.56 0.44

Product (multiply all)1 0.031 0.025

Divide by the evidence2 0.070 0.056

Introduction and Motivation

Inflammatory bowel disease (IBD), which includes Crohn’s disease and ulcerative colitis (UC), impacts 1.6 million Americans, according to the Crohn’s and Colitis Foundation (crohnscolitisfoundation. org). Crohn’s disease causes chronic inflammation and damages the gastrointestinal tract. It can impact any part of the gastrointestinal tract. The cause of

the disease is not entirely known, but some knowl- edge from research suggests that it could be caused by a combination of factors that include genetic makeup, immune system, and environmental set- tings. Systems that can detect disease progression or early disease onset can help in optimal utilization of healthcare resources and can result in better patient outcomes. The goal of this case study was to use

Application Case 5.5 Predicting Disease Progress in Crohn’s Disease Patients: A Comparison of Analytics Methods

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 283

electronic medical records (EMRs) to predict and explain inflammation in Crohn’s disease patients.

The Methodology

The data used in this study were from one of the nation’s largest EMR databases, Cerner Health Facts EMR. It houses rich and varied information related to patients, healthcare settings, costs, reimburse- ment type, and prescription ordered data from mul- tiple healthcare providers and hospitals in the United States. Data stored in the EMR database consists of patient-level data that were captured when a patient visited hospitals, urgent care centers, specialty clinics, general clinics, and nursing homes. The Health Facts database contains patient-level de-identified longitu- dinal data that were time stamped. The database was organized in the data tables as shown in Table 5.6.

A high-level process flow of the research meth- odology is shown in Figure 5.12. Although the pro- cess flow diagram did not provide the details of each step, it gave a high-level view of the sequence of the steps performed in the current predictive modeling study using EMR data. The three model types shown in the diagram were selected based on their com- paratively better performance over other machine- learning methods such as Naïve Bayes, nearest neighbor, and neural networks. Detailed steps of data balancing and data standardization are explained in the paper by Reddy, Delen, and Agrawal (2018).

The Results

Prediction results were generated using the test set applying the repeated 10 times run on the 10-fold cross-validation method. The performance of each model was assessed by the metric AUC, which was

the preferred performance metric over the prediction accuracy because the ROC curve, which generated the AUC, compared the classifier performance across the entire range of class distributions and error costs and, hence, is widely accepted as the performance measure for machine-learning applications. The mean AUC from the 10 run on the 10-fold cross-validation was generated (and shown in Table 5.7) for the three final model types— logistic regression, regularized regression, and gradient boosting machines (GBM).

Upon generation of the AUC for 100 models, researchers performed a post hoc analysis of variance (ANOVA) test and applied Tukey’s Honest Significant Difference (HSD) test for multiple comparison tests to determine which classifier method’s performance differed from the others based on the AUC. The test results showed that the mean AUC for regularized regression and the logistic regression did not differ significantly. However, the AUC from regularized regression and logistic regression were significantly different from the GBM model as seen in Table 5.8.

The relative importance of the independent variables was computed by adding the total amount of decrease in Gini index by the splits over a given predictor, averaged across all trees specified in the GBM tuning parameter, 1,000 trees in this research. This average decrease in GINI was normalized to a 0-100 scale on which a higher number indicates a stronger predictor. The variable importance results are shown in Figure 5.13.

Relative importance was computed by adding the total amount of decrease in Gini index by the splits over a given predictor averaged across all trees specified in the GBM tuning parameter, 1,000 trees in this research. This average decrease in GINI was nor- malized to a 0-100. scale on which a higher number

TABLE 5.6 Metadata of the Tables Extracted from EMR Database

Data Set (table) Description

Encounter Encounters including demographics, billing, healthcare setting, payer type, etc.

Medication Medication orders sent by the healthcare provider

Laboratory Laboratory data including blood chemistry, hematology, and urinalysis

Clinical Event Clinical events data containing information about various metrics including body mass index, smoking status, pain score, etc.

Procedure Clinical procedures performed on the patient

(Continued )

284 Part II • Predictive Analytics/Machine Learning

indicates a stronger predictor. The model results in Figure 5.13 showed that there was not one single predictor but a combination of predictors driving the predictions. Crohn’s disease location at diagnosis such as small intestine and large intestine; lab parameters

at baseline such as white blood cell (WBC) count; mean corpuscular hemoglobin (MCH); mean corpus- cular volume; sodium; red blood cell (RBC); distri- bution of platelet count; creatinine; hematocrit; and hemoglobin were the strongest  predictors. One  of

Encounter Procedure Medication

Combined Patient-Level

Data Set

Data Preprocessing

Gradient Boosting Machine

Results (mean AUC) Variable Selection

Variable Importance

Results (mean AUC) Variable Selection

Results (mean AUC) Variable Selection

Regularized Regression Logistic Regression

Lab Clinical Event

Selecting/filtering Aggregating

10 Replications of 10-fold Cross Validation

Training and Testing

Training and Testing

10%

10%

10%

10% 10%

10%

10%

10%

10% 10%

Training and Testing

Transforming

Comparative Analyses

FIGURE 5.12 Process Flow Diagram of the High-Level Steps Involved in the Data Mining Research.

Application Case 5.4 (Continued)

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 285

the strongest demographic predictors of the inflam- mation severity doubling was age. Other healthcare- setting and encounter-related variables such as hospital bed size, diagnosis priority, and region, whether south or not, predicting whether inflam- mation severity doubled or not, also had some pre- dictive ability. The majority of the Crohn’s disease researchers identified the location of the disease, age at diagnosis, smoking status, biologic markers, and tumor necrosis factor (TNF) levels to predict the response to treatment; these are some of the iden- tifiers that also predicted the inflammation severity.

Logistic regression and regularized regression cannot produce a similar relative variable importance plot. However, the odds ratio and standardized coeffi- cients generated were used to identify the stronger predictors of inflammation severity.

This study was able to show that disease can be managed in real time by using decision sup- port tools that rely on advanced analytics to pre- dict the future inflammation state, which would then allow for medical intervention prospectively. With this information, healthcare providers can improve patient outcomes by intervening early and making

TABLE 5.7 AUC for Each Repeated Run Across Three Models

Repeated Run Logistic Regression Regularized Regression

Gradient Boosting Machines (GBM)

1 0.7929 0.8267 0.9393

2 0.7878 0.8078 0.9262

3 0.8080 0.8145 0.9369

4 0.8461 0.8487 0.9124

5 0.8243 0.8281 0.9414

6 0.7681 0.8543 0.8878

7 0.8167 0.8154 0.9356

8 0.8174 0.8176 0.9330

9 0.8452 0.8281 0.9467

10 0.8050 0.8294 0.9230

Mean AUC 0.8131 0.8271 0.9282

Median AUC 0.8167 0.8274 0.9343

TABLE 5.8 ANOVA with Multiple Comparisons Using Tukey’s Test

Tukey Grouping Mean AUC No. of Observations Model Type

A 0.928 100 GBM

B 0.827 100 Regularized regression

B 0.812 100 Logistic regression

Means with the Same Letter Are Not Significantly Different

(Continued )

286 Part II • Predictive Analytics/Machine Learning

necessary therapeutic adjustments that would work for the specific patient.

Questions for Case 5.5

1. What is Crohn’s disease and why is it important?

2. Based on the findings of this Application Case, what can you tell about the use of analytics in chronic disease management?

3. What other methods and data sets might be used to better predict the outcomes of this chronic disease?

Source: B. K. Reddy, D. Delen, & R. K. Agrawal, “Predicting and Explaining Inflammation in Crohn’s Disease Patients Using Predictive Analytics Methods and Electronic Medical Record Data,” Health Informatics Journal, 2018.

30

25

20

15

10

5

0

W B

C

M C

H

M C

V

A g e

S o d iu

m

R B

C

P la

te le

ts

C re

a ti n in

e

C h lo

ri d e

B lo

o d U

re a N

it ro

g e n

H e m

a to

c ri t

H e m

o g lo

b in

D ia

g n o s is

P ri o ri ty

B e d s iz

e 3

0 0

to 4

9 9

M a ri ta

lS ta

tu s S

in g le

M a ri ta

lS ta

tu s N

u ll

M a ri ta

lS ta

tu s M

a rr

ie d

B e d s iz

e R

a n g e 5

0 0

.

C e n s u s R

e g io

n S

o u th

G e n d e rM

a le

D ia

g n o s is

U n s p e c ifi

e d S

it e

B e d S

iz e R

a n g e 2

0 0

to 2

9 9

C e n s u s R

e g io

n N

o rt

h e a s t

D ia

g n o s is

S m

a llI

n te

s ti n e

D ia

g n o s is

S m

a llW

it h L a rg

e ln

t. ..

R a c e C

a u c a s ia

n

R a c e H

is p a n ic

R a c e N

a ti ve

A m

e ri c a n

R a c e O

th e r

M a ri ta

lS ta

tu s L e g a lly

S e p a ra

te d

M a ri ta

lS ta

tu s W

id o w

e d

100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

FIGURE 5.13 Relative Variable Importance for GBM Model.

u SECTION 5.7 REVIEW QUESTIONS

1. What is special about the Naïve Bayes algorithm? What is the meaning of “Naïve” in this algorithm?

2. What are the advantages and disadvantages of Naïve Bayes compared to other machine-learning methods?

3. What type of data can be used in Naïve Bayes algorithm? What type of predictions can be obtained from it?

4. What is the process of developing and testing a Naïve Bayes classifier?

Application Case 5.5 (Continued)

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 287

5.8 BAYESIAN NETWORKS

Bayesian belief networks or Bayesian networks (BN) were first defined in an early paper of Judea Pearl as “supportive of self-activated, multidirectional propagation of evidence that converges rapidly to a globally-consistent equilibrium” (Pearl, 1985). Later on, with his continuing work in this area, Pearl won the prestigious ACM’s A.M. Turing Award for his contributions to the field of artificial intelligence and the development of BN. With this success, BN has received more public recognition than ever before, establishing it as a new paradigm in artificial intelligence, predictive analytics, and data science.

BN is a powerful tool for representing dependency structure in a graphical, ex- plicit, and intuitive way. It reflects the various states of a multivariate model and their probabilistic relationships. Theoretically, any system can be modeled with BN. In a given model, some states will occur more frequently when others are also present; for example, if a freshman student is not registered for next fall (a presumed freshman stu- dent dropout case), the chances of the student’s having financial aid is lower, indicating a relationship between the two variables. This is where the conditional probabilities (the basic theory that underlies BN) come to play to analyze and characterize the situation.

BNs have become popular among probabilistic graphical models because they have been shown to be able to capture and reason with complex, nonlinear, and partially uncertain situations and interactions (Koller and Friedman, 2009). While their solid, probability-based theoretical properties made Bayesian networks immediately attractive for academic research, especially for studying causality, their use in practical data science and business analytics domains is relatively new. For instance, researchers recently have developed data analytics– driven BN models in such domains that include predicting and understanding the graft survival for kidney transplantations (Topuz et al., 2018), predicting failures in the rail in- dustry caused by weather-related issues (Wang et al., 2017), predicting food fraud type (Bouzembrak et al., 2016), and detecting diseases (Meyfroidt et al., 2009).

Essentially, the BN model is a directed acyclic graph whose nodes correspond to the variables and arcs that signify conditional dependencies between variables and their possible values (Pearl, 2009). Here is a simple example, which was previously used as Application Case 3.2. For details of the example, please reread this Application Case. Let us say that the goal was to predict whether a freshman student will stay or drop out of college (presented in the graph as SecondFallRegistered) using some data/information about the student such as (1) the declared college type (a number of states/options exists for potential colleges) and (2) whether the student received financial aid in the first fall semester (two states exists, Yes or No), both of which can be characterized probabilistically using the his- torical data. One might think that there exist some causal links among the three variables, both college type and financial aid relating to whether the student comes back for the second fall semester and that it is reasonable to think that some colleges historically have more financial support than others (see Figure 5.14 for the presumed causal relationships).

The direction of links in BN graphs corresponded to the probabilistic or condi- tional dependencies between any two variables. Calculating actual conditional prob- abilities using historical data would help predict and understand student retention (SecondFallRegistered) using two variables, “financial aid” and “college type.” Such a network can then be used to answer questions such as these:

• Is the college type “engineering”? • What are the chances the student will register next fall? • How will financial aid affect the outcome?

How Does BN Work?

Building probabilistic models such as BN of complex real-world situations/problems using historical data can help in predicting what is likely to happen when something else would have happened. Essentially, BN typically tries to represent interrelationships

288 Part II • Predictive Analytics/Machine Learning

among the variables (both input and output variables) using a probabilistic structure that is often called the joint distribution. Joint distributions can be presented as a table consisting of all possible combinations of states (variable values) in a given model. For complex models, such a table can easily become rather large because it stores one prob- ability value for every combination of states. To mitigate the situation, BN does not con- nect all of nodes in the model to each other; rather, it connects only the nodes that are probabilistically related by some sort of conditional and/or logical dependency, resulting in significant savings on the computations.

The naturally complex probability distributions can be represented in a relatively compact way using BNs’ conditional independence formula. In the following formula, each xi represents a variable and Paxi represents the parents of that variable; by using these rep- resentations, the BN chain rule can be expressed as follows (Koller and Friedman, 2009):

P(x1, . . . , xn) = q n

i = 1 P(xi �Paxi).

Let’s look at an example by building a simple network for the student retention pre- diction problem. Remember that our problem is to predict whether a freshman student will stay for the second fall semester or drop out of college by using some data/information from student records (i.e., declared college type and whether the student received finan- cial aid for the first semester). The constructed BN graphical model shown in Figure 5.15 exhibits the relationships and conditional probabilities among all three nodes.

How Can BN Be Constructed?

There are two common methods available to construct the network: (1) manually with the help of a domain expert and (2) analytically by learning the structure of the network from the historical data by using advanced mathematical methods. Building a network manually, even for a modest size network, requires a skilled knowledgeable engineer spending several hours with the domain expert. As the size of the network gets larger, the time spent by the engineer and the domain expert increases exponentially. In some cases, it is very difficult to find a knowledgeable expert for a particular domain. Even if such a domain expert exists, he or she might not have the time to devote to the model building effort and/or might not be explicit and articulate enough (i.e., explaining tacit knowledge is always a difficult task) to be of much use as a source of knowledge. Therefore, most of the previous studies developed and offered various techniques that can be used to learn the structure of the network automatically from the data.

One of the earlier methods used to learn the structure of the network automatically from the data is the Naïve Bayes method. The Naïve Bayes classification method is a simple probabilistic model that assumes conditional independence between all predictor

SecondFallRegistered (binary)

FinancialAid (binary)

CollegeType (nominal)

FIGURE 5.14 Simple Illustration of the Partial Causality Relationships in Student Retention.

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 289

variables and the given class/target variable to learn the structure. The classification algo- rithm is based on the Bayes rule that the probability of class/target value is computed for each given attribute variable and then the highest prediction is chosen for the structure.

A more recent and popular method for learning the structure of the network is called Tree Augmented Naïve (TAN) Bayes. The TAN method is an updated version of the Naïve Bayes classifier that uses tree structure to approximate the interactions between predictor variables and the target variable (Friedman, Geiger, and Goldszmidt, 1997). In the TAN model structure, class variable has no parent, and each and every predictor variable has the class variable as its parent along with at most one other predictor variable (i.e., attribute) as shown in Figure 5.16. Thus, an arc between two variables indicates a directional and causal relationship between them. Formal representation of parents for a variable xi can be shown by:

Paxi = 5C, xd(i)6 where the tree is a function over d1i2 7 0, and Paxi is the set of parents for each xi. A class variable (C ) has no parents, namely PaC = ∅. It is empirically and theoretically shown that TAN performs better than Naïve Bayes and maintains simplicity in the compu- tations because it does not require a search process (Friedman et al., 1997).

The procedure for constructing a TAN uses Chow and Liu’s tree Bayesian con- cept. Finding a maximally weighted spanning tree in a graph is an optimization problem

FinancialAid (binary)

CollegeType (nominal)

SecondFallRegistered (binary)

FIGURE 5.15 Conditional Probability Tables for the Two Predictor and One Target Variables.

290 Part II • Predictive Analytics/Machine Learning

whose objective is to maximize log likelihood of d(i) (Chow and Liu 1968). Then the TAN construction steps can be as described as follows (Friedman, et al., 1997):

Step 1. Compute the conditional mutual information function for each (i, j ) pair as

IP (xi : xj �C) = a xi,xj,C

P (xi, xj, C )log P(xi, xj �C )

P(xi �C)P(xj �C ) , i ≠ j

This function indicates how much information is provided when the class variable is known.

Step 2. Build a complete undirected graph and use a conditional mutual information function to annotate the weight of an edge connecting xi to xj.

Step 3. Build a maximum weighted spanning tree. Step 4. Convert the undirected graph into a directed one by choosing a root variable

and setting the direction of all edges to be outward from it. Step 5. Construct a TAN model by adding a vertex labeled by C and an arc from C to

each xi.

One of the superior features of BN is its ease of adaptability. While building a BN, one can start the network as small with a limited knowledge and then expand on it as new information becomes available. In such situation, having missing values in the data set might not be a major issue because it can use the available portion of the data/values/ knowledge to create the probabilities. Figure 5.17 shows a fully developed, data-driven BN example for the student retention project.

From the applicability perspective, such a fully constructed BN can be of great use to practitioners (i.e., administrators and managers in educational institutions) because it offers a holistic view of all relationships and provides the means to explore detailed information using a variety of “what-if” analysis. In fact, with this network model, it is possible to cal- culate the student-specific risk probability of attrition, which is the likelihood or posterior probability of the student who would drop out, by systematically selecting and changing the value of a predictor variable within its value domain (assessing how much the dropout risk of a student changes as the value of a given predictor variable such as Fall-GPA changes).

When interpreting the BN model shown in Figure 5.17, one should con- sider the arcs, directions of the arrows on those arcs, direct interactions, and indi- rect relationships. For example, the fall grant/tuition waiver/scholarship category

C

X1 X2 X3

FIGURE 5.16 Tree Augmented Naïve Bayes Network Structure.

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 291

P e rs

is ta

n c e F a ll

0 t

o 0

.5 0

.5 t

o 0

.7 5

0 .7

5 t

o 0

.8 0

.8 t

o 1

1

8 .4

3 Y e s

7 7

.7 N

o 2

2 .3

Y e s

7 4

.0 N

o

Y e s

N o

7 8

.8 2

1 .2

Y e s

N o

Y e s

N o

2 6

.0 7

9 .1

2 0

.9

8 5

.2 1

4 .8

9 .1

3 7

.9 5

1 0

.1 6

4 .3

0 .8

7 5

6 0

.2 3

8 .6

4 6

4 .5

1 3

.1 6

4 .4

R e c e iv

e d S

p ri n g A

id S

p ri n g G

ra n tT

u it io

n W

a iv

e rS

c h o la

rs h ip

F a llG

ra n tT

u it io

n W

a iv

e rS

c h o la

rs h ip

R e c e iv

e d F a llA

id

Y e s

N o

4 1

.0 5

9 .0

F a llS

tu d e n tL

o a n

Y e s

N o

3 8

.1

0 t

o 2

2 2 t

o 1

5 1

5 1

5 t

o 5

4

W I B A M H N D X O C P

A F H X K J I G R

9 0

.8 7

.8 4

0 .4

8 0

.2 4

.0 8

1 0

.2 7

0 .1

1 .0

7 6

.0 7

6

E N

A G

A S

B U

H E

S G

U E

D

A B C D F

A B C D F

1 6

.6 4

7 .7

2 3

.1 7

.7 6

4 .7

4

1 6

.2 4

5 .4

2 4

.3 8

.5 6

5 .5

8

0 t

o 1

0 1

0 o

t 1

3 1

3 t

o 1

4 1

4 1

4 t

o 2

2

1 5

.0 2

0 .4

1 7

.3 2

0 .9

2 6

.4

1 7

.5 1

1 .3

2 6

.2 1

5 .6

9 .4

9 1

1 .7

8 .0

7

7 7

.7 9

.1 3

4 .5

4 1

.5 1

2 .2

5 3

.1 2

1 .0

4 0

.1 5

0 .3

0 0

.1 3

.0 7

6 .0

7 6

0 .2

5 1

.6 5

9 6

.2 1

.2 1

0 .7

1

6 1

.9

S p ri n g S

tu d e n tL

o a n

M o n th

s A

ft e rH

ig h S

c h o o lG

ra d u a ti o n

E th

n ic

it y

A d d m

is io

n T

yp e

C o lle

g e

F a llE

a rn

e d H

o u rs

F a llG

P A

L

F a llC

u m

u la

ti ve

G P

.. .

S e c o n d F a llR

re g is

te re

d

F IG

U R

E 5

.1 7

B

a y e si

a n

B e li e f

N e tw

o rk

f o

r P re

d ic

ti n

g F

re sh

m e n

S tu

d e n

t A

tt ri

ti o

n .

292 Part II • Predictive Analytics/Machine Learning

(i.e., FallGrantTuitionWaiverScholarship) and all the nodes linked to FallGrantTuition WaiverScholarship are related to student attrition (i.e., SecondFallRegistered). Moreover, while FallGrantTuitionWaiverScholarship interacts with college (College) and spring grant/tuition waiver/scholarship (i.e., SpringGrantTuitionWaiverScholarship) directly, it also interacts with admission type (AdmissionType) indirectly through College. According to the BN model, one of the most interactive predictors is the student’s earned credit hours by registered rate (i.e., PersistanceFall), which contributes to the effect of the student’s fall GPA (FallGPA) and student attrition. As such, if the PersistanceFall of the student is less than 0.8, then College type has an effect on student attrition. However, if the PersistanceFall of the student is 1.0, the College type does not impact the student attrition in a noteworthy manner.

As a collective view to what-if scenarios, Figure 5.18 summarizes the most posi- tive and most negative levels within each predictor with its posterior probabilities. For instance, getting an A for the Fall GPA decreases the posterior probability of student attri- tion to 7.3 percent, or conversely, getting an F increases the probability of attrition to 87.8 percent where the baseline is 21.2 percent.

Some people have had doubts about using BN because they thought that BN does not work well if the probabilities upon which it is constructed are not exact. However, it turns out that in most cases, approximate probabilities driven from data and even the subjective ones that are guessed by domain experts provide reasonably good results. BN are shown to be quite robust toward imperfect, inaccurate, and incomplete knowledge. Often the combination of several strands of imperfect knowledge allows BN to make sur- prisingly good predictions and conclusions. Studies have shown that people are better at estimating probabilities “in the forward direction.” For example, managers are quite good at providing probability estimates for “If the student has dropped out the college, what are the chances his or her college type is Art & Sciences?” rather than the reverse, “If the

21.2

0%

Baseline (Second fall registered5No)

Fall GPA (A vs F)

Fall cumulative GPA (A vs F)

Spring grant/ tuition waiver/ scholarship (Yes vs No)

College (HES vs GU)

Admission type (H vs K)

Fall grant/ tuition waiver/ scholarship (Yes vs No)

Fall student loan (No vs Yes)

Months after high school (2to15 vs .15)

Ethnicity (White vs Pacific)

Spring student loan (Yes vs No)

Received fall financial aid (Yes vs No)

Earned by registered hours in fall (,0.5 vs 1)

Fall earned hours (,10 vs .14)

Received spring financial aid (Yes vs No)

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

87.87.3

88.47.2

81

64.3

42.7

39.8

10.5

9.2

15.1

14.7

15.9

19.2

19.5

18.6

31

49.7

27.8

25

20.7 46.9

19.9

18.2

45.1

23.2

20.3 26.8

FIGURE 5.18 Probability of Student Attrition for Risk Factors—What-If Analysis on Individual Factors.

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 293

student goes to Art & Sciences college, what are the chances that this student will not register the next fall?”

u SECTION 5.8 REVIEW QUESTIONS

1. What are Bayesian networks? What is special about them? 2. What is the relationship between Naïve Bayes and Bayesian networks? 3. What is the process of developing a Bayesian networks model? 4. What are the advantages and disadvantages of Bayesian networks compared to other

machine-learning methods?

5. What is Tree Augmented Naïve (TAN) Bayes and how does it relate to Bayesian networks?

5.9 ENSEMBLE MODELING

Ensembles (or more appropriately called model ensembles or ensemble modeling) are combinations of the outcomes produced by two or more analytics models into a com- pound output. Ensembles are primarily used for prediction modeling when the scores of two or more models are combined to produce a better prediction. The prediction can be either classification or regression/estimation type (i.e., the former predicting a class label and the latter estimating a numerical output variable). Although use of ensembles has been dominated by prediction-type modeling, it can also be used for other analytics tasks such as clustering and association rule mining. That is, model ensembles can be used for supervised as well as unsupervised machine-learning tasks. Traditionally, these machine- learning procedures focused on identifying and building the best possible model (often the most accurate predictor of the holdout data) from a large number of alternative model types. To do so, analysts and scientists used an elaborate experimental process that mainly relied on trial and error to improve each single model’s performance (defined by some predetermined metrics, e.g., prediction accuracy) to its best possible level so that the best of the models can be used/deployed for the task at hand. The ensemble ap- proach turns this thinking around. Rather than building models and selecting the single best model to use/deploy, it proposes to build many models and use them all for the task they are intended to perform (e.g., prediction).

Motivation—Why Do We Need to Use Ensembles?

Usually researchers and practitioners build ensembles for two main reasons: for bet- ter accuracy and more stable/robust/consistent/reliable outcomes. Numerous research studies and publications over the past two decades have shown that ensembles almost always improve predictive accuracy for the given problem and rarely predict worse than the single models (Abbott, 2014). Ensembles began to appear in the data mining/analyt- ics literature in 1990s, motivated by the limited success obtained by the earlier works on combining forecasts that dated a couple or more decades. By the early- to mid-2000s, ensembles had become popular and almost essential to winning data mining and predic- tive modeling competitions. One of the most popular awards for ensemble competitions is perhaps the famous Netflix prize, which was an open competition that solicited re- searchers and practitioners to predict user ratings of films based on historical ratings. The prize was US$1 million for a team that could reduce the RMSE of the then-existing Netflix internal prediction algorithm by the largest margin but no less than 10 percentage points. The winner, runner-up, and nearly all the teams at the top of the leaderboard used model ensembles in their submissions. As a result, the winning submission was the result of an ensemble containing hundreds of predictive models.

294 Part II • Predictive Analytics/Machine Learning

When it comes to justifying the use of ensembles, Vorhies (2016) put it the best—if you want to win a predictive analytics competition (at Kaggle or at anywhere else) or at least get a respectable place on the leaderboard, you need to embrace and intelligently use model ensembles. Kaggle has become the premier platform for data scientists to showcase their talents. According to Vorhies, the Kaggle competitions are like Formula One racing for data science. Winners edge out competitors at the fourth decimal place and, like Formula One race cars, not many of us would mistake them for daily drivers. The amount of time devoted and the extreme techniques used would not always be ap- propriate for an ordinary data science production project, but like paddle shifters and ex- otic suspensions, some of those improvements and advanced features find their way into the day-to-day life and practice of analytics professionals. In addition to Kaggle competi- tions, reputable organizations such as the Association for Computing Machinery (ACM)’s Special Interest Group (SIG) on Knowledge Discovery and Data Mining (SIGKDD) and Pacific-Asia Conference in Knowledge Discovery and Data Mining (PAKDD) regularly organize competitions (often called “cups”) for the community of data scientists to dem- onstrate their competence, sometimes for monetary rewards but most often for simple bragging rights. Some of the popular analytics companies like the SAS Institute and Teradata Corporation organize similar competitions for (and extend a variety of relatively modest awards to) both graduate and undergraduate students in universities all over the world, usually in concert with their regular analytics conferences.

It is not just the accuracy that makes model ensembles popular and unavoidably necessary. It has been shown time and time again that ensembles can improve model accuracy, but they can also improve model robustness, stability, and, hence, reliabil- ity. This advantage of model ensembles is equally (or perhaps more) important and invaluable than accuracy in situations for which reliable prediction is of the essence. In ensemble models, by combining (some form of averaging) multiple models into a single prediction outcome, no single model dominates the final predicted value of the models, which in turn, reduces the likelihood of making a way-off-target “wacky” predic- tion. Figure 5.19 shows a graphical illustration of model ensembles for classification-type prediction problems. Although some varieties exist, most ensemble modeling methods

Data

M1T

Training & Calibrating

Trained Model

Combining Ensembling

Predicting Deploying

Processed Data

Raw Data Collection

Cross- Validation

1 0

1

0

...

......

...

... ...

10 %

10 %

10 %10

% 10 %

10 % 10 %10

% 10 %

10 %

M1

M2T M2

M3T M3

MnT Mn

FIGURE 5.19 Graphical Depiction of Model Ensembles for Prediction Modeling.

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 295

follow this generalized process. From left to right, Figure 5.19 illustrates the general tasks of data acquisition and data preparation followed by cross-validation and model building and testing, and finally assembling/combining the individual model outcomes and assess- ing the resultant predictions.

Another way to look at ensembles is from the perspective of “collective wisdom” or “crowdsourcing.” In the popular book The Wisdom of Crowds (Surowiecki, 2005), the au- thor proposes that better decisions can be made if rather than relying on a single expert, many (even uninformed) opinions (obtained by a process called crowdsourcing) can be aggregated into a decision that is superior to the best expert’s opinion. In his book, Surowiecki describes four characteristics necessary for the group opinion to work well and not degenerate into the opposite effect of poor decisions as evidenced by the “mad- ness of crowds”: diversity of opinion, independence, decentralization, and aggregation. The first three characteristics relate to how individual decisions are made—they must have information that differs from that of others in the group and is not affected by the others in the group. The last characteristic merely states that the decisions must be combined. These four principles/characteristics seem to lay the foundation for building better model ensembles as well. Each predictive model has a voice in the final decision. The diversity of opinion can be measured by the correlation of the predictive values themselves—if all of the predictions are highly correlated, or in other words, if the models nearly all agree, there is no foreseeable advantage in combining them. The decentralization characteristic can be achieved by resampling data or case weights; each model uses either different records from a common data set or at least uses the records with weights that differ from the other models (Abbott, 2014).

One of the prevalent concepts in statistics and predictive modeling that is highly relevant to model ensembles is the bias-variance trade-off. Therefore, before delving into the different types of model ensembles, it is necessary to review and understand the bias- variance trade-off principle (as it applies to the field of statistics or machine learning). In predictive analytics, bias refers to the error and variance refers to the consistency (or lack thereof) in predictive accuracy of models applied to other data sets. The best models are expected to have low bias (low error, high accuracy) and low variance (consistency of ac- curacy from data set to data set). Unfortunately, there is always a trade-off between these two metrics in building predictive models—improving one results in worsening the other. You can achieve low bias on training data, but the model could suffer from high variance on hold-out/validation data because the models could have been overtrained/overfit. For instance, the kNN algorithm with k = 1 is an example of a low bias model (perfect on training data set) but is susceptible to high variance on a test/validation data set. Use of cross-validation along with proper model ensembles seems to be the current best practice in handling such trade-offs between bias and variance in predictive modeling.

Different Types of Ensembles

Ensembles or teams of predictive models working together have been the fundamental strategy for developing accurate and robust analytics models. Although ensembles have been around for quite a while, their popularity and effectiveness has surfaced in a sig- nificant way only within the last decade as they continually improved in parallel with the rapidly improving software and hardware capabilities. When we refer to model ensembles, many of us immediately think of decision tree ensembles like random forest and boosted trees; however, generally speaking, the model ensembles can be classified into four groups in two dimensions as shown in Figure 5.20. The first dimension is the method type (the x-axis in Figure 5.20) in which the ensembles can be grouped into bagging or boosting types. The second dimension is the model type (the y-axis in Figure 5.20) in which the ensembles can be grouped into homogeneous or heterogeneous types (Abbott, 2014).

296 Part II • Predictive Analytics/Machine Learning

As the name implies, homogeneous-type ensembles combine the outcomes of two or more of the same type of models such as decision trees. In fact, a vast majority of homo- geneous model ensembles are developed using a combination of decision tree structures. The two most common categories of homogeneous type ensembles that use decision trees are bagging and boosting (more information on these are given in subsequent sec- tions). Heterogeneous model ensembles combine the outcomes of two or more different types of models such as decision trees, artificial neural networks, logistic regression, SVM, and others. As mentioned in the context of “the wisdom of crowds,” one of the key suc- cess factors in ensemble modeling is to use models that are fundamentally different from one another, ones that look at the data from a different perspective. Because of the way it combines the outcomes of different model types, heterogeneous model ensembles are also called information fusion models (Delen and Sharda, 2010) or stacking (more infor- mation on these is given later in this chapter).

Bagging

Bagging is the simplest and most common ensemble method. Leo Breiman, a very well- respected scholar in the world of statistics and analytics, is known to have first pub- lished a description of bagging (i.e., Bootstrap Aggregating) algorithm at the University of California–Berkeley in 1996 (Breiman, 1996). The idea behind bagging is quite simple yet powerful: build multiple decision trees from resampled data and combine the predicted values through averaging or voting. The resampling method Breiman used was bootstrap sampling (sampling with replacement), which creates replicates of some records in the training data. With this selection method, on average, about 37 percent of the records will not be included at all in the training data set (Abbott, 2014).

Although bagging was first developed for decision trees, the idea can be applied to any predictive modeling algorithm that produces outcomes with sufficient variation in the predicted values. Although rare in practice, the other predictive modeling algorithms that are potential candidates for bagging-type model ensembles include neural networks, Naïve Bayes, k-nearest neighbor (for low values of k), and, to a lesser degree, even logis- tic regression. k-nearest neighbor is not a good candidate for bagging if the value of k is

[Rare - Active Research Area] Systematically weighing data samples for better prediction modeling

[Rare] Other types of single- model-type bagging (e.g., Ann)

Bagging H

o m

o g e n e o u s

H e te

ro g e n e o u s

Boosting

Stacking (meta- modeling)

Simple/Complex model weighing

Ensemble trees Random forest

Information fusion

AdaBoost XGBoost [Rare - Active Research Area] Other types of single-model- type boosting

Method Type

Model Type

FIGURE 5.20 Simple Taxonomy for Model Ensembles.

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 297

already large; the algorithm already votes or averages predictions and with larger values of k, so predictions are already very stable with low variance.

Bagging can be used for both classification- and regression/estimation–type predic- tion problems. In classification-type prediction problems, all of the participant models’ outcomes (class assignments) are combined using either a simple or complex/weighted majority voting mechanism. The class label that gets the most/highest votes becomes the aggregated/ensemble prediction for that sample/record. In regression/estimation–type prediction problems, when the output/target variable is a number, all of the participant models’ outcomes (numerical estimations) are combined using either a simple or com- plex/weighted–averaging mechanism. Figure 5.21 illustrates the graphical depiction of a decision tree–type bagging algorithm.

One of the key questions in bagging is, “How many bootstrap samples, also called replicates, should be created?” Brieman stated, “My sense of it is that fewer are required when y [dependent variable] is numerical and more are required with an increasing number of classes [for classification type prediction problems].” He typically used 10–25 bootstrap replicates with significant improvements occurring with as few as 10 replicates. Overfitting the models is an important requirement to building good bagged ensembles. By overfitting each model, the bias is low, but the decision tree generally has worse ac- curacy on held-out data. But bagging is a variance reduction technique; the averaging of predictions smooths the predictions to behave in a more stable way on new data.

As mentioned before, the diversity of model predictions is a key factor in creating effective ensembles. One way to measure the diversity of predictions is to examine the correlation of predicted values. If the correlations between model predictions are always very high, more than 0.95, each model brings little additional predictive information to the ensemble and therefore little improvement in accuracy is achievable. Generally, it is best to have correlations of less than 0.9. The correlations should be computed from the model propensities or predicted probabilities rather than the 50,16 classification value itself. Bootstrap sampling in bagging is the key to introducing diversity in the models. One can think of the bootstrap sampling methodology as creating case weights for each record—some records are included multiple times in the training data (their weights are

Data

Sample1 Sample2 Samplen

Final Prediction (voting/average)

Bootstrap Sampling

FIGURE 5.21 Bagging-Type Decision Tree Ensembles.

298 Part II • Predictive Analytics/Machine Learning

1, 2, 3, or more), and other records are not included at all (their weights are equal to 0) (Abbott, 2014).

Boosting

Boosting is perhaps the second most common ensemble method after bagging. Yoav Freund and Robert E. Schapire are known to have first introduced the boosting algo- rithm in separate publications in the early 1990s and then in a 1996 joint publication (Freund and Schapire, 1996). They introduced the well-known boosting algorithm, called AdaBoost. As with bagging, the idea behind boosting is also quite straightforward. First, build a rather simple classification model; it needs to be only slightly better than random chance, so for a binary classification problem, it needs to be only slightly better than a 50 percent correct classification. In this first step, each record is used in the algorithm with equal case weights as one would do normally in building a predictive model. The errors in the predicted values for each case are noted. The case weights of correctly classified records/cases/samples will stay the same or perhaps be reduced, and the case weights of the records that are incorrectly classified will have increased, and then a second sim- ple model is built on these weighted cases (i.e., the transformed/weighted–training data set). In other words, for the second model, records that were incorrectly classified are “boosted” through case weights to be considered more strongly or seriously in the con- struction of the new prediction model. In each iteration, the records that are incorrectly predicted (the ones that are difficult to classify) keep having their case weights increased, communicating to the algorithm to pay more attention to these records until, hopefully, they are finally classified correctly.

This process of boosting is often repeated tens or even hundreds of times. After the tens or hundreds of iterations, the final predictions are made based on a weighted aver- age of the predictions from all the models. Figure 5.22 illustrates the simple process of boosting in building decision tree–type ensemble models. As shown, each tree takes the most current data set (one of equal size, but with the most recently boosted case weights)

2 1

Data

4

Final Prediction (voting/average)

n 3

FIGURE 5.22 Boosting-Type Ensembles for Decision Trees.

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 299

to build another tree. The feedback of incorrectly predicted cases is used as an indicator to determine which cases and to what extent (direction and magnitude) to boost (update the weights) for the training samples/cases.

Although they look quite similar in structure and purpose, bagging and boosting employ slightly different strategies to utilize the training data set and to achieve the goal of building the best possible prediction model ensemble. The two key differences be- tween bagging and boosting are as follows. Bagging uses a bootstrap sample of cases to build decision trees whereas boosting uses the complete training data set. Whereas bagging creates independent, simple trees to ensemble, boosting creates dependent trees (each tree “learning” from the previous one to pay more attention to the incorrectly pre- dicted cases) that collectively contribute to the final ensemble.

Boosting methods are designed to work with weak learners, that is, simple models; the component models in a boosted ensemble are simple models with high bias although low variance. The improvement with boosting is better, as with bagging, when algorithms that are unstable predictors are used. Decision trees are most often used in boosted models. Naïve Bayes is also used but with fewer improvements over a single model. Empirically speaking, boosting typically produces better model accuracy than single deci- sion trees or even bagging-type ensembles.

Variants of Bagging and Boosting

Bagging and boosting were the first ensemble methods that appeared in predictive ana- lytics software, primarily with decision tree algorithms. Since their introduction, many other approaches to building ensembles have been developed and made available, par- ticularly in open source software (both as part of open analytics platforms like KNIME and Orange and as class libraries in R and Python). The most popular and successful [advanced] variants of bagging and boosting are random forest and stochastic gradient boosting, respectively.

RANDOM FOREST The random forest (RF) model was first introduced by Breiman (2001) as a modification to the simple bagging algorithm. As with bagging, the RF algorithm be- gins with a bootstrap-sampled data set and builds one decision tree from each bootstrap sample. Compared to simple bagging, there is, however, an important twist to the RF algorithm: at each split in the tree, staring from the very first split, rather than considering all input variables as candidates, only a random subset of variables is considered. Hence, in RF, the bootstrap sampling technique applies to both a random selection of cases and a random selection of features (i.e., input variables).

The number of cases and the number of variables to consider along with how many trees to construct are all parameters used to decide in building RF models. Common prac- tice suggests that the default number of variables to consider as candidates at each split point should be the square root of the total number of candidate inputs. For example, if there were 100 candidate inputs for the model, a random 10 inputs are candidates for each split. This also means that it is unlikely that the same inputs will be available for splits at parent and children nodes in a given tree, forcing the tree to find alternate ways to maxi- mize the accuracy of subsequent splits. Therefore, there is an intentionally created twofold diversity mechanism built into the tree construction process—random selection of cases and variables. RF models produce prediction outcomes that are usually more accurate than simple bagging and are often more accurate than simple boosting (i.e., AdaBoost).

STOCHASTIC GRADIENT BOOSTING The simple boosting algorithm AdaBoost is only one of many boosting algorithms currently documented in the literature. In commercial soft- ware, AdaBoost is still the most commonly used boosting technique; however, dozens

300 Part II • Predictive Analytics/Machine Learning

of boosting variants can be found in open source software packages. One interesting boosting algorithm that has recently gained popularity due to its superior performance is the stochastic gradient boosting (SGB) algorithm created by Jerry Friedman at Stanford University. Then Friedman developed an advanced version of this algorithm (Friedman, 2001) called multiple additive regression trees (MART) and later branded as TreeNet' by Salford Systems in its software tool. Like other boosting algorithms, the MART algorithm builds successive, simple trees and combines them additively. Typically, the simple trees are more than stumps and contain up to six terminal nodes. Procedurally, after building the first tree, errors (also called residuals) are computed. The second tree and all subse- quent trees then use residuals as the target variable. Subsequent trees identify patterns that relate inputs to small and large errors. Poor prediction of the errors results in large errors to predict in the next tree, and good predictions of the errors result in small errors to predict in the next tree. Typically, hundreds of trees are built, and the final predictions are additive combinations of the predictions that are, interestingly, piecewise constant models because each tree is itself a piecewise constant model. However, one rarely notices these intricacies about the individual trees because typically hundreds of trees are included in the ensemble (Abbott, 2014). The TreeNet algorithm, an example of sto- chastic gradient boosting, has won multiple data mining modeling competitions since its introduction and has proven to be an accurate predictor with the benefit that very little data cleanup is needed for the trees before modeling.

Stacking

Stacking (a.k.a. stacked generalization or super learner) is a part of heterogeneous en- semble methods. To some analytics professionals, it could be the optimum ensemble technique but is also the least understood (and the most difficult to explain). Due to its two-step model training procedure, some think of it as an overly complicated ensemble modeling. Simply put, stacking creates an ensemble from a diverse group of strong learn- ers. In the process, it interjects a metadata step called super learner or meta learner. These intermediate meta classifiers forecast how accurate the primary classifiers have become and are used as the basis for adjustments and corrections (Vorhies, 2016). The process of stacking is figuratively illustrated in Figure 5.23.

As shown in Figure 5.23, in constructing a stacking-type model ensemble, a number of diverse strong classifiers are first trained using bootstrapped samples of the training data, creating tier 1 classifiers (each optimized to its full potential for the best possible prediction outcomes). The outputs of the tier 1 classifiers are then used to train a tier 2 classifier (i.e., a metaclassifier) (Wolpert, 1992). The underlying idea is to learn whether training data have been properly learned. For example, if a particular classifier incorrectly learned a certain region of the feature space and hence consistently misclassifies instances coming from that region, the tier 2 classifier might be able to learn this behavior and along with the learned behaviors of other classifiers, it can correct such improper training. Cross-validation–type selection is typically used for training the tier 1 classifiers—the en- tire training data set is divided into k numbers of mutually exclusive subsets, and each tier 1 classifier is first trained on (a different set of) k - 1 subsets of the training data. Each classifier is then evaluated on the kth subset, which was not seen during training. The outputs of these classifiers on their pseudo-training blocks, along with the actual correct labels for those blocks, constitute the training data set for the tier 2 classifier.

Information Fusion

As part of heterogeneous model ensembles, information fusion combines (fuses) the out- put (i.e., predictions) of different types of models such as decision trees, artificial neural networks, logistic regression, SVM, Naïve Bayes, and k-nearest neighbor, among others,

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 301

and their variants. The difference between stacking and information fusion is the fact that information fusion has no “meta modeling” or “superlearners.” It simply combines the outcomes of heterogeneous strong classifiers using simple or weighted voting (for clas- sification) or simple or weighted averaging (for regression). Therefore, it is simpler and less computationally demanding than stacking. In the process of combining the outcomes of multiple models, either a simple voting (each model contributes equally one vote) or a weighted combination of voting (each model contributes based on its prediction accuracy—more accurate models have higher weight value) can be used. Regardless of the combination method, this type of heterogeneous ensemble has been shown to be an invaluable addition to any data mining and predictive modeling project. Figure 5.24 graphically illustrates the process of building information fusion–type model ensembles.

Summary—Ensembles are not Perfect!

As a prospective data scientist, if you are asked to build a prediction model (or any other analytics model for that matter), you are expected to develop some of the popular model ensembles along with the standard individual models. If done properly, you will realize that ensembles are often more accurate and almost always more robust and reliable than

Preprocessed Data

Bootstrap Sampling

Sample1 Sample2

Built and Test the Meta Model

Final Prediction

Samplek

Built Model1

Built Model2

Built Modeln

T ie

r- 2

M e ta

M o d e l B

u ild

in g

T ie

r- 1

In d iv

id u a l M

o d e l B

u ild

in g

k-fold Cross-Validation

FIGURE 5.23 Stacking-Type Model Ensembles.

302 Part II • Predictive Analytics/Machine Learning

the individual models. Although they seem like silver bullets, model ensembles are not without shortcomings; the following are the two most common ones.

COMPLEXITY Model ensembles are more complex than individual models. Occam’s razor is a core principle that many data scientists use; the idea is that simpler models are more likely to generalize better, so it is better to reduce/regularize complexity, or in other words, simplify the models so that the inclusion of each term, coefficient, or split in a model is justified by its power of reducing the error at a sufficient amount. One way to quantify the relationship between accuracy and complexity is taken from information theory in the form of information theoretic criteria, such as the Akaike information cri- terion (AIC), the Bayesian information criterion (BIC), and minimum description length (MDL). Traditionally, statisticians—more recently, data scientists—are using these criteria to select variables in predictive modeling. Information theoretic criteria require a reduc- tion in model error to justify additional model complexity. So the question is, “Do model ensembles violate Occam’s razor?” Ensembles, after all, are much more complex than single models. According to Abbott (2014), if the ensemble accuracy is better on held-out data than single models, then the answer is “no” as long as we think about the complex- ity of a model in different terms—not just computational complexity but also behavioral complexity. Therefore, we should not fear that adding computational complexity (more terms, splits, or weights) will necessarily increase the complexity of models because sometimes the ensemble will significantly reduce the behavioral complexity.

Preprocessed Data

Decision logic (voting/weighting)

Ensemble (the prediction)

26 24 22 0

0.5

1

2 4 6

M argin

M ax

im um

-m ar

gin h

yp er

pla ne

X1

X2

k-fold Cross-Validation

FIGURE 5.24 Illustration of Building Process for Information Fusion–Type Model Ensemble.

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 303

TRANSPARENCY The interpretation of ensembles can become quite difficult. If you build an RF ensemble containing 200 trees, how do you describe why a prediction has a particular value? You can examine each of the trees individually, although this is clearly not practical. For this reason, ensembles are often considered black-box models, mean- ing that what they do is not transparent to the modeler or domain expert. Although you can look at the split statistics (which variables are more often picked to split early in those 200 trees) to artificially judge the level of contribution (a pseudo-variable of importance measure), each variable is contributing to the trained model ensemble. Compared to a single decision tree, such an investigation of 200 trees will be too difficult and not an intuitive way to interpret how the model comes up with a specific prediction. Another way to determine which inputs to the model are most important is to perform a sensitivity analysis.

In addition to complexity and transparency, model ensembles are also more dif- ficult and computationally more expensive to build and much more difficult to deploy. Table 5.9 shows the pros and cons of ensemble models compared with individual models.

In summary, model ensembles are the new frontier for predictive modelers who are interested in accuracy by reducing either errors in the models or the risk that the models behave erratically. Evidence for this is clear from the dominance of ensembles in predic- tive analytics and data mining competitions: ensembles always win.

The good news for predictive modelers is that many techniques for building en- sembles are built into software already. The most popular ensemble algorithms (bagging, boosting, stacking, and their variants) are available in nearly every commercial or open source software tool. Building customized ensembles is also supported in many software products whether based on a single algorithm or through a heterogeneous ensemble.

Ensembles are not appropriate for every solution—their applicability is deter- mined by the modeling objectives defined during business understanding and problem definition—but they should be part of every predictive modeler’s and data scientist’s modeling arsenal.

TABLE 5.9 Brief List of Pros and Cons of Model Ensembles Compared to Individual Models

PROS (Advantages) Description

✓ Accuracy Model ensembles usually result in more accurate models than individual models.

✓ Robustness Model ensembles tend to be more robust against outliers and noise in the data set than individual models.

✓ Reliability (stable) Because of the variance reduction, model ensembles tend to produce more stable, reliable, and believable results than individual models.

✓ Coverage Model ensembles tend to have a better coverage of the hidden complex patterns in the data set than individual models.

CONS (Shortcomings) Description

✓ Complexity Model ensembles are much more complex than individual models.

✓ Computationally expensive Compared to individual models, ensembles require more time and computational power to build.

✓ Lack of transparency (explainability) Because of their complexity, it is more difficult to understand the inner structure of model ensembles (how they do what they do) than individual models.

✓ Harder to deploy Model ensembles are much more difficult to deploy in an analytics-based managerial decision-support system than single models.

304 Part II • Predictive Analytics/Machine Learning

Introduction and Motivation

Analytics has been used by many businesses, organi- zations, and government agencies to learn from past experiences to more effectively and efficiently use their limited resources to achieve their goals and objec- tives. Despite all the promises of analytics, however, its multidimensional and multidisciplinary nature can sometimes disserve its proper, full-fledged application. This is particularly true for the use of predictive analyt- ics in several social science disciplines because these domains are traditionally dominated by descriptive analytics (causal-explanatory statistical modeling) and might not have easy access to the set of skills required to build predictive analytics models. A review of the extant literature shows that drug court is one such area. While many researchers have studied this social phenomenon, its characteristics, its requirements, and its outcomes from a descriptive analytics perspective, there currently is a dearth of predictive analytics mod- els that can accurately and appropriately predict who would (or would not) graduate from intervention and treatment programs. To fill this gap and to help author- ities better manage the resources, and to improve the outcomes, this study sought to develop and compare several predictive analytics models (both single models and ensembles) to identify who would graduate from these treatment programs.

Ten years after President Richard Nixon first declared a “war on drugs,” President Ronald Reagan signed an executive order leading to stricter drug enforcement, stating, “We’re taking down the surren- der flag that has flown over so many drug efforts; we are running up a battle flag.” The reinforcement of the war on drugs resulted in an unprecedented 10-fold surge in the number of citizens incarcerated for drug offences during the following two decades. The sky- rocketing number of drug cases inundated court dockets, overloaded the criminal justice system, and overcrowded prisons. The abundance of drug-related caseloads, aggravated by a longer processing time than that for most other felonies, imposed tremen- dous costs on state and federal departments of justice. Regarding the increased demand, court systems started to look for innovative ways to accelerate the inquest of drug-related cases. Perhaps analytics-driven deci- sion support systems are the solution to the problem.

To support this claim, the current study’s goal was to build and compare several predictive models that use a large sample of data from drug courts across different locations to predict who is more likely to complete the treatment successfully. The researchers believed that this endeavor might reduce the costs to the criminal justice system and local communities.

Methodology

The methodology used in this research effort included a multi-step process that employed pre- dictive analytics methods in a social science con- text. The first step of this process, which focused on understanding the problem domain and the need to conduct this study, was presented in the previous section. For the steps of the process, the research- ers employed a structured and systematic approach to develop and evaluate a set of predictive models using a large and feature-rich real-world data set. The steps included data understanding, data pre- processing, model building, and model evaluation; they are reviewed in this section. The approach also involved multiple iterations of experimentations and numerous modifications to improve individual tasks and to optimize the modeling parameters to achieve the best possible outcomes. A pictorial depiction of the methodology is given in Figure 5.25.

The Results

A summary of the models’ performances based on accuracy, sensitivity, specificity, and AUC is pre- sented in Table 5.10. As the results show, RF has the best classification accuracy and the greatest AUC among the models. The heterogeneous ensemble_ (HE) model closely follows RF, and SVM, ANN, and LR rank third to last based on their classifica- tion performances. RF also has the highest specific- ity and the second highest sensitivity. Sensitivity in the context of this study is an indicator of a model’s ability in correctly predicting the outcome for suc- cessfully graduated participants. Specificity, on the other hand, determines how a model performs in predicting the end results for those who do not suc- cessfully complete the treatment. Consequently, it can be concluded that RF outperforms other models for the drug courts data set used in this study.

Application Case 5.6 To Imprison or Not to Imprison: A Predictive Analytics-Based Decision Support System for Drug Courts

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 305

10-fold Cross-Validation

Data Preprocessing Merging Aggregating Cleaning Binning Selecting

True Positive Count (TP)

False Positive Count (FP)

True Negative Count (TN)

False Negative Count (FN)

True/Observed Class Positive Negative

P os

it iv

e N

e ga

ti ve

P re

di c te

d C

la s s

100 90

80 70

60 50

40 30

40 50

Variable Names

Im po

rt a nc

e

X1

X2

ANN

LR

SVM

RF

Pre-processed Data

SplittingData Preparation Modeling Assessment

Treat. DB Case DB Court DB

Domain Expert(s)

X1

X2

M ax

im um

-m ar gi n hy

pe rp lan

e

M argin

HE

Individual Models

Ensemble Models

10 %

10 %

10 %

10 % 10 %

10% 10%

10%

10%

10%

10%

10% 10%

10%

10%

Variable Importance

Prediction Accuracy 26 24 22 0

0.5

1

2 4 6

26 24 22 0

0.5

1

2 4 6

FIGURE 5.25 Research Methodology Depicted as a Workflow.

TABLE 5.10 Performance of Predictive Models Using 10-Fold Cross-Validation on the Balanced Data Set

Model Type

Confusion Matrix Accuracy (%)

Sensitivity (%)

Specificity (%) AUCG T

In d

iv id

u a l M

o d

e ls ANN

G 6,831 1,072 86.63 86.76 86.49 0.909

T 1,042 6,861

SVM G 6,911 992

88.67 89.63 87.75 0.917 T 799 7,104

LR G 6,321 1,582

85.13 86.16 81.85 0.859 T 768 7,135

E n

se m

b le

s

RF G 6,998 905

91.16 93.44 89.12 0.927 T 491 7,412

HE G 6,885 1,018

90.61 93.66 87.96 0.916 T 466 7,437

ANN: artificial neural networks; DT: decision trees; LR: logistic regression; RF: random forest; HE: heterogeneous ensemble; AUC: area under the curve; G: graduated; T: terminated

(Continued )

306 Part II • Predictive Analytics/Machine Learning

u SECTION 5.9 REVIEW QUESTIONS

1. What is a model ensemble, and where can it be used analytically? 2. What are the different types of model ensembles? 3. Why are ensembles gaining popularity over all other machine-learning trends? 4. What is the difference between bagging- and boosting-type ensemble models? 5. What are the advantages and disadvantages of ensemble models?

Although the RF model performs better than the other models in general, it falls second to the HE model in the number of false negative predictions. Similarly, the HE model has a slightly better performance in true negative predictions. False positive predictions represent participants who were terminated from the treatment, but the models mistakenly classified them as successful graduates. False negatives pertain to individuals who graduated, but the models predicted them as dropouts. False positive predictions are syn- onymous with increased costs and opportunity losses whereas false negatives carry social impacts. Spending resources on those offenders who would recidivate at some point in time during the treatment and, hence, be terminated from the program prevented a number of (potentially successful) prospective offenders from participating in the treatment. Conspicuously, depriv- ing potentially successful offenders from the treatment is against the initial objective of drug courts in reinte- grating nonviolent offenders into their communities.

In summary, traditional causal-explanatory sta- tistical modeling, or descriptive analytics, uses sta- tistical inference and significance levels to test and evaluate the explanatory power of hypothesized underlying models or to investigate the association

between variables retrospectively. Although a legiti- mate approach for understanding the relationships within the data used to build the model, descriptive analytics falls short in predicting outcomes for pro- spective observations. In other words, partial explana- tory power does not imply predictive power, and predictive analytics is a must for building empirical models that predict well. Therefore, relying on the findings of this study, application of predictive analyt- ics (rather than the sole use of descriptive analytics) to predict the outcomes of drug courts is well grounded.

Questions for Case 5.6

1. What are drug courts and what do they do for the society?

2. What are the commonalities and differences between traditional (theoretical) and modern (machine-learning) base methods in studying drug courts?

3. Can you think of other social situations and sys- tems for which predictive analytics can be used?

Source: Zolbanin, H., and Delen, D. (2018). To Imprison or Not to Imprison: An Analytics-Based Decision Support System for Drug Courts. The Journal of Business Analytics (forthcoming).

Chapter Highlights

• Neural computing involves a set of methods that emulates the way the human brain works. The basic processing unit is a neuron. Multiple neurons are grouped into layers and linked together.

• There are differences between biological and arti- ficial neural networks.

• In an artificial neural network, knowledge is stored in the weight associated with each connec- tion between two neurons.

• Neural network applications abound in almost all business disciplines as well as in virtually all other functional areas.

• Business applications of neural networks include finance, bankruptcy prediction, time-series fore- casting, and so on.

• There are various neural network architectures for different types of problems.

• Neural network architectures can be applied not only to prediction (classification or estimation)

Application Case 5.6 (Continued)

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 307

but also to clustering and optimization-type problems.

• SVM are among popular machine-learning techniques, mostly because of their superior predictive performance and their theoretical foundation.

• Although SVM can use a radial-basis function as a kernel, they are not very similar to neural networks.

• SVM can be used for both classification- and estimation/regression-type prediction problems.

• SVM use only numerical variables and the super- vised machine-learning method.

• Plenty of SVM applications exist, and new ones are emerging in a variety of domains including healthcare, finance, security, and energy.

• The nearest neighbor (or k-nearest neighbor) al- gorithm is a simple machine-learning technique that is used for both classification- and estima- tion/regression-type prediction problems.

• The nearest neighbor algorithm is a type of instance-based learning (or lazy learning) algo- rithm in which all computations are deferred until the actual prediction.

• The parameter k signifies the number of neigh- bors to use in a given prediction problem.

• Determining the “optimal” value of k requires a cross-validation–type experimentation.

• The nearest neighbor algorithm uses a dis- tance measure to identify close-by/appropriate neighbors.

• The input variables to the nearest neighbor al- gorithm must be in numeric format; all non- numeric/nominal variables need to be converted to pseudo-binary numeric variables.

• Bayesian classifiers are built on the foundation of the Bayes theorem (i.e., conditional probabilities).

• Naïve Bayes is a simple probability-based clas- sification method that is applied to classification- type prediction problems.

• The Naïve Bayes method requires input and out- put variables to have nominal values; numeric ones need to be discretized.

• Naïve keyword refers to the somewhat unrealistic yet practical assumption of independence (of the predictor/input variables).

• The Bayesian network (or Bayesian belief network) is a relatively new machine-learning technique that is gaining popularity among data scientists, academics, and theorists.

• The Bayesian network is a powerful tool for rep- resenting dependency structure in a graphical, explicit, and intuitive way.

• The Bayesian network can be used for prediction and explanation (or the interrelationships among the variables).

• Bayesian networks can be constructed manually (based on a domain expert’s knowledge) or auto- matically using historical data.

• While constructing a Bayesian network automat- ically, one can use regular Naïve Bayes or the tree-augmented Naïve (TAN) Bayes.

• Bayesian networks provide an excellent model for conducting what-if analyses for a variety of hypothetical scenarios.

• Ensembles (or more appropriately model ensem- bles or ensemble modeling) are combinations of the outcomes produced by two or more analytics models into a compound output.

• Although ensembles are primarily used for pre- diction modeling when the scores of two or more models are combined to produce a better pre- diction, they can also be used for clustering and association.

• Ensembles can be applied to both classification (via voting) and estimation/regression-type (via averaging) prediction problems.

• Ensembles are used mainly for two reasons: to obtain better accuracy and achieve more stable/ reliable outcomes.

• Recent history in data science has shown that en- sembles win competitions.

• There are homogeneous and heterogeneous en- sembles; if the combined models are of the same type (e.g., decision trees), the ensemble is homo- geneous; if not, it is heterogeneous.

• There are three methods in ensemble modeling: bagging, boosting, and stacking.

• Random forest is a bagging-type, homogeneous, decision tree–based ensemble method.

• Stochastic gradient boosting is a boosting type that is a homogeneous, decision tree–based en- semble method.

• Information fusion and stacking are heteroge- neous ensembles in which different types of models are combined.

• The disadvantages of ensembles include com- plexity and lack of transparency.

308 Part II • Predictive Analytics/Machine Learning

Key Terms

AdaBoost artificial neural network

(ANN) attrition axon backpropagation bagging Bayesian belief network

(BNN) Bayesian network (BN) Bayes theorem boosting conditional probability cross-validation

dendrites distance metric Euclidean distance heterogeneous ensemble hidden layer Hopfield network hyperplane information fusion k-fold cross-validation k-nearest neighbor

(kNN) kernel trick Kohonen’s self-organizing

feature map

Manhattan distance maximum margin Minkowski distance multi-layer perceptron Naïve Bayes neural computing neural network neuron nucleus pattern recognition perceptron processing element (PE) radial basis function

(RBF)

random forest retention stacking supervised learning stochastic gradient

boosting synapse transformation (transfer)

function voting weights what-if scenario

Questions for Discussion

1. What is an artificial neural network and for what types of problems can it be used?

2. Compare artificial and biological neural networks. What aspects of biological networks are not mimicked by arti- ficial ones? What aspects are similar?

3. What are the most common ANN architectures? For what types of problems can they be used?

4. ANN can be used for both supervised and unsupervised learning. Explain how they learn in a supervised mode and in an unsupervised mode.

5. What are SVM? How do they work? 6. What are the types of problems that can be solved by SVM? 7. What is the meaning of “maximum-margin hyper-

planes”? Why are they important in SVM? 8. What is the kernel trick and how does it relate to SVM? 9. What are the specific steps to follow in developing an

SVM model? 10. How can the optimal kernel type and kernel parameters

be determined? 11. What are the common application areas for SVM?

Conduct a search on the Internet to identify popular application areas and specific SVM software tools used in those applications.

12. What are the commonalities and differences, advan- taged and disadvantages between ANN and SVM?

13. Explain the difference between a training and a testing data set in ANN and SVM. Why do we need to differen- tiate them? Can the same set be used for both purposes? Why or why not?

14. Everyone would like to make a great deal of money on the stock market. Only a few are very successful. Why is using an SVM or ANN a promising approach? What can they do that other decision support technologies cannot do? How could SVM or ANN fail?

15. What is special about the kNN algorithm?

16. What are the advantages and disadvantages of kNN as compared to ANN and SVM?

17. What are the critical success factors for a kNN implementation?

18. What is a similarity (or distance) measure? How can it be applied to both numerical and nominal valued variables?

19. What are the common (business and scientific) applica- tions of kNN? Conduct a Web search to find three real- world applications that use kNN to solve the problem.

20. What is special about the Naïve Bayes algorithm? What is the meaning of “Naïve” in this algorithm?

21. What are the advantages and disadvantages of Naïve Bayes compared to other machine-learning methods?

22. What type of data can be used in a Naïve Bayes algo- rithm? What type of predictions can be obtained from it?

23. What is the process of developing and testing a Naïve Bayes classifier?

24. What are Bayesian networks? What is special about them? 25. What is the relationship between Naïve Bayes and

Bayesian networks? 26. What is the process of developing a Bayesian networks

model? 27. What are the advantages and disadvantages of Bayesian

networks compared to other machine-learning methods? 28. What is Tree Augmented Naïve (TAN) Bayes and how

does it relate to Bayesian networks? 29. What is a model ensemble, and analytically where can

it be used? 30. What are the different types of model ensembles? 31. Why are ensembles gaining popularity over all other

machine-learning trends? 32. What is the difference between bagging- and boosting-

type ensemble models? 33. What are the advantages and disadvantages of ensem-

ble models?

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 309

Exercises

Teradata University Network (TUN) and Other Hands-On Exercises

1. Go to the Teradata University Network Web site (teradatauniversitynetwork.com) or a URL given by your instructor. Locate Web seminars related to data mining and neural networks. Specifically, view the sem- inar given by Professor Hugh Watson at the SPIRIT2005 conference at Oklahoma State University; then, answer the following questions: a. Which real-time application at Continental Airlines

might have used a neural network? b. What inputs and outputs can be used in building a

neural network application? c. Given that its data mining applications are in real

time, how might Continental implement a neural net- work in practice?

d. What other neural network applications would you propose for the airline industry?

2. Go to the Teradata University Network Web site (teradatauniversitynetwork.com) or a URL given by your instructor. Locate the Harrah’s case. Read the case and answer the following questions:

a. Which of the Harrah’s data applications are most likely implemented using neural networks?

b. What other applications could Harrah’s develop using the data it collects from its customers?

c. What are some concerns you might have as a cus- tomer at this casino?

3. A bankruptcy-prediction problem can be viewed as a problem of classification. The data set you will be using for this problem includes five ratios that have been computed from the financial statements of real-world firms. These five ratios have been used in studies involving bankruptcy pre- diction. The first sample includes data on firms that went bankrupt and firms that did not. This will be your training sample for the neural network. The second sample of 10 firms also consists of some bankrupt firms and some non- bankrupt firms. Your goal is to use neural networks, SVM, and nearest neighbor algorithms to build a model using the first 20 data points and then to test its performance on the other 10 data points. (Try to analyze the new cases yourself manually before you run the neural network and see how well you do.) The following tables show the train- ing sample and test data you should use for this exercise.

Training Sample

Firm WC/TA RE/TA EBIT/TA MVE/TD S/TA BR/NB

1 0.1650 0.1192 0.2035 0.8130 1.6702 1

2 0.1415 0.3868 0.0681 0.5755 1.0579 1

3 0.5804 0.3331 0.0810 1.1964 1.3572 1

4 0.2304 0.2960 0.1225 0.4102 3.0809 1

5 0.3684 0.3913 0.0524 0.1658 1.1533 1

6 0.1527 0.3344 0.0783 0.7736 1.5046 1

7 0.1126 0.3071 0.0839 1.3429 1.5736 1

8 0.0141 0.2366 0.0905 0.5863 1.4651 1

9 0.2220 0.1797 0.1526 0.3459 1.7237 1

10 0.2776 0.2567 0.1642 0.2968 1.8904 1

11 0.2689 0.1729 0.0287 0.1224 0.9277 0

12 0.2039 -0.0476 0.1263 0.8965 1.0457 0

13 0.5056 -0.1951 0.2026 0.5380 1.9514 0

14 0.1759 0.1343 0.0946 0.1955 1.9218 0

15 0.3579 0.1515 0.0812 0.1991 1.4582 0

16 0.2845 0.2038 0.0171 0.3357 1.3258 0

17 0.1209 0.2823 -0.0113 0.3157 2.3219 0

18 0.1254 0.1956 0.0079 0.2073 1.4890 0

19 0.1777 0.0891 0.0695 0.1924 1.6871 0

20 0.2409 0.1660 0.0746 0.2516 1.8524 0

310 Part II • Predictive Analytics/Machine Learning

Test Data

Firm WC/TA RE/TA EBIT/TA MVE/TD S/TA BR/NB

A 0.1759 0.1343 0.0946 0.1955 1.9218 ?

B 0.3732 0.3483 -0.0013 0.3483 1.8223 ?

C 0.1725 0.3238 0.1040 0.8847 0.5576 ?

D 0.1630 0.3555 0.0110 0.3730 2.8307 ?

E 0.1904 0.2011 0.1329 0.5580 1.6623 ?

F 0.1123 0.2288 0.0100 0.1884 2.7186 ?

G 0.0732 0.3526 0.0587 0.2349 1.7432 ?

H 0.2653 0.2683 0.0235 0.5118 1.8350 ?

I 0.1070 0.0787 0.0433 0.1083 1.2051 ?

J 0.2921 0.2390 0.0673 0.3402 0.9277

Describe the results of the neural network, SVM, and nearest neighbor model predictions, including  soft- ware, architecture, and training information.

4. The purpose of this exercise is to develop models to predict the type of forest cover using a number of car- tographic measures. The given data set (see Online Supplements) includes four wilderness areas found in the Roosevelt National Forest of northern Colorado. A total of 12 cartographic measures were utilized as inde- pendent variables; seven major forest cover types were used as dependent variables.

This is an excellent example for a multi-class classification problem. The data set is rather large (with 581,012 unique instances) and feature rich. As you will see, the data are also raw and skewed (un- balanced for different cover types). As a model build- er, you are to make necessary decisions to preprocess the data and build the best possible predictor. Use your favorite tool to build the models for neural net- works, SVM, and nearest neighbor algorithms, and document the details of your results and experiences in a written report. Use screenshots within your re- port to illustrate important and interesting findings. You are expected to discuss and justify any decision that you make along the way.

5. Go to UCI Machine-Learning Repository (archive.ics. uci.edu/ml/index.php), identify four data sets for classification-type problems, and use these data sets to build and compare ANN, SVM, kNN, and Naïve Bayes models. To do so, you can use any analytics tool. We suggest you use a free, open-source ana- lytics tool such as KNIME (knime.org) or Orange

(orange.biolab.si). Prepare a well-written report to summarize your findings.

6. Go to Google Scholar (scholar.google.com). Conduct a search to find two papers written in the last five years that compare and contrast multiple machine-learning methods for a given problem domain. Observe com- monalities and differences among their findings and prepare a report to summarize your understanding.

Team Assignments and Role-Playing Projects

1. Consider the following set of data that relates daily elec- tricity usage to a function of the outside high tempera- ture (for the day):

Temperature, X Kilowatts, Y

46.8 12,530

52.1 10,800

55.1 10,180

59.2 9,730

61.9 9,750

66.2 10,230

69.9 11,160

76.8 13,910

79.7 15,110

79.3 15,690

80.2 17,020

83.3 17,880

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 311

Number Name Independent Variables

1 Elevation Elevation in meters

2 Aspect Aspect in degrees azimuth

3 Slope Slope in degrees

4 Horizontal_Distance_To_Hydrology Horizontal distance to nearest surface water features

5 Vertical_Distance_To_Hydrology Vertical distance to nearest surface water features

6 Horizontal_Distance_To_Roadways Horizontal distance to nearest roadway

7 Hillshade_9am Hill shade index at 9 a.m., summer solstice

8 Hillshade_Noon Hill shade index at noon, summer solstice

9 Hillshade_3pm Hill shade index at 3 p.m., summer solstice

10 Horizontal_Distance_To_Fire_Points Horizontal distance to nearest wildfire ignition points

11 Wilderness_Area (4 binary variables) Wilderness area designation

12 Soil_Type (40 binary variables) Soil-type designation

Number Dependent Variable

1 Cover_Type (7 unique types) Forest cover–type designation

Note: More details about the data set (variables and observations) can be found in the online file.

a. Plot the raw data. What pattern do you see? What do you think is really affecting electricity usage?

b. Solve this problem with linear regression Y = a + bX (in a spreadsheet). How well does this work? Plot your results. What is wrong? Calculate the sum-of- the-squares error and R2.

c. Solve this problem by using nonlinear regres- sion. We recommend a quadratic function, Y = a + b1X + b2X2. How well does this work? Plot your results. Is anything wrong? Calculate the sum-of- squares error and R2.

d. Break the problem into three sections (look at the plot). Solve it using three linear regression models, one for each section. How well does this work? Plot your results. Calculate the sum-of-squares error and R2. Is this modeling approach appropriate? Why or why not?

e. Build a neural network to solve the original problem. (You might have to scale the X and Y values to be between 0 and 1.) Train the network (on the entire set of data) and solve the problem (i.e., make predic- tions for each of the original data items). How well does this work? Plot your results. Calculate the sum- of-squares error and R2.

f. Which method works best and why? 2. Build a real-world neural network. Using demo soft-

ware downloaded from the Web (e.g., NeuroSolutions at neurodimension.com or another neural network tool/site), identify real-world data (e.g., start searching on the Web at archive.ics.uci.edu/ml/index.php

or use data from an organization with which some- one in your group has a contact) and build a neural network to make predictions. Topics might include sales forecasts, predicting success in an academic pro- gram (e.g., predict GPA from high school ranking and SAT scores, being careful to look out for “bad” data, such as GPAs of 0.0) or housing prices; or survey the class for weight, gender, and height and try to predict height based on the other two factors. You could also use U.S. Census data by state on this book’s Web site or at census.gov to identify a relationship between education level and income. How good are your pre- dictions? Compare the results to predictions generated using standard statistical methods (regression). Which method is better? How could your system be embed- ded in a decision support system (DSS) for real deci- sion making?

3. For each of the following applications, would it be bet- ter to use a neural network or an expert system? Explain your answers, including possible exceptions or special conditions. a. Diagnosis of a well-established but complex disease b. Price lookup subsystem for a high-volume merchan-

dise seller c. Automated voice inquiry processing system d. Training of new employees e. Handwriting recognition

4. Consider the following data set, which includes three attributes and a classification for admission decisions into an MBA program:

312 Part II • Predictive Analytics/Machine Learning

GMAT GPA Quantitative

GMAT Decision

650 2.75 35 NO

580 3.50 70 NO

600 3.50 75 YES

450 2.95 80 NO

700 3.25 90 YES

590 3.50 80 YES

400 3.85 45 NO

640 3.50 75 YES

540 3.00 60 ?

690 2.85 80 ?

490 4.00 65 ?

a. Using the data given here as examples, develop your own manual expert rules for decision making.

b. Build and test a neural network model using your favorite data mining tool. Experiment with differ- ent model parameters to “optimize” the predictive power of your model.

c. Build and test a support vector machine model using your favorite data mining tool. Experiment

with different model parameters to “optimize” your model’s predictive power. Compare the results with ANN and SVM.

d. Report the predictions on the last three observations from each of the three classification approaches (ANN, SVM, and kNN). Comment on the results.

e. Comment on the similarity and differences of these three prediction approaches. What did you learn from this exercise?

5. You have worked on neural networks and other data mining techniques. Give examples of the use of each of them. Based on your knowledge, how would you differentiate among these techniques? Assume that a few years from now you will come across a situation in which neural network or other data mining tech- niques could be used to build an interesting applica- tion for your organization. You have an intern working with you to do the grunt work. How will you decide whether the application is appropriate for a neural net- work or another data mining model? Based on your homework assignments, what specific software guid- ance can you provide so that your intern is productive for you quickly? Your answer for this question might mention the specific software, describe how to go about setting up the model/neural network, and vali- date the application.

Internet Exercises

1. Explore the Web sites of several neural network ven- dors, such as California Scientific Software (calsci.com), NeuralWare (neuralware.com), and Ward Systems Group (wardsystems.comv), and review some of their products. Download at least two demos and install, run, and compare them.

2. A very good repository of data that have been used to test the performance of neural network and other machine-learning algorithms can be accessed at https://archive.ics.uci.edu/ml/index.php. Some of the data sets are really meant to test the limits of current machine-learning algorithms and compare their performance against new approaches to learning. However, some of the smaller data sets can be use- ful for exploring the functionality of the software you might download in Internet Exercise 1 or the software that is available at StatSoft.com (i.e., Statistica Data Miner with extensive neural network capabilities). Download at least one data set from the UCI repository (e.g., Credit Screening Databases, Housing Database). Then apply neural networks as well as decision tree methods as appropriate. Prepare a report on your results. (Some of these exercises could also be com- pleted in a group or even as semester-long projects for term papers and so on.)

3. Go to calsci.com and read about the company’s vari- ous business applications. Prepare a report that summa- rizes the applications.

4. Go to nd.com. Read about the company’s applica- tions in investment and trading. Prepare a report about them.

5. Go to nd.com. Download the trial version of NeuroSolutions for Excel and experiment with it using one of the data sets from the exercises in this chapter. Prepare a report about your experience with the tool.

6. Go to neoxi.com. Identify at least two software tools that have not been mentioned in this chapter. Visit Web sites of those tools and prepare a brief report on their capabilities.

7. Go to neuroshell.com. Look at Gee Whiz examples. Comment on the feasibility of achieving the results claimed by the developers of this neural network model.

8. Go to easynn.com. Download the trial version of the software. After the installation of the software, find the sample file called Houseprices.tvq. Retrain the neu- ral network and test the model by supplying some data. Prepare a report about your experience with this software.

9. Visit tibco.com. Download at least three white papers of applications. Which of these applications might have used neural networks?

Chapter 5 • Machine-Learning Techniques for Predictive Analytics 313

10. Go to neuralware.com. Prepare a report about the products the company offers.

11. Go to ibm.com. Download at least two customer suc- cess stories or case studies that use advanced analyt- ics or machine learning. Prepare a presentation for your understanding of these application cases.

12. Go to sas.com. Download at least two customer suc- cess stories or case studies that use advanced analytics

or machine learning. Prepare a presentation for your understanding of these application cases.

13. Go to teradata.com. Download at least two cus- tomer success stories or case studies where advanced analytics or machine learning is used. Prepare a pre- sentation for your understanding of these application cases.

References

Abbott, D. (2014). Applied Predictive Analytics: Principles and Techniques for the Professional Data Analyst. Hoboken, NJ: John Wiley.

Aizerman, M., E. Braverman, & L. Rozonoer. (1964). “Theo- retical Foundations of the Potential Function Method in Pattern Recognition Learning.” Automation and Remote Control, Vol. 25, pp. 821–837.

American Heart Association, “Heart Disease and Stroke Statis- tics,” heart.org (accessed May 2018).

Boiman, E. S., & M. Irani. (2008). “In Defense of Nearest-Neighbor Based Image Classification,” IEEE Conference on Computer Vi- sion and Pattern Recognition, 2008 (CVPR), 2008, pp. 1–8.

Bouzembrak, Y., & H. J. Marvin. (2016). “Prediction of Food Fraud Type Using Data from Rapid Alert System for Food and Feed (RASFF) and Bayesian Network Modelling.” Food Control, 61, 180–187.

Breiman, L. (1996). Bagging Predictors. Machine Learning, 24(2), 123–140.

Breiman, L. (2001). “Random Forests.” Machine Learning, 45 (1), 5–32.

Chow, C., & C. Liu (1968). “Approximating Discrete Probabil- ity Distributions with Dependence Trees.” IEEE Transac- tions on Information Theory, 14(3), 462–473.

Delen, D., & R. Sharda. (2010). “Predicting the Financial Success of Hollywood Movies Using an Information Fusion Approach.” Indus Eng J, 21 (1), 30–37.

Delen, D., L. Tomak, K. Topuz, & E. Eryarsoy (2017). Investigat- ing Injury Severity Risk Factors in Automobile Crashes with Predictive Analytics and Sensitivity Analysis Methods. Jour- nal of Transport & Health, 4, 118–131.

Delen, D., A. Oztekin, & L. Tomak. (2012). “An Analytic Approach to Better Understanding and Management of Coronary Surgeries.” Decision Support Systems, 52 (3), 698–705.

Friedman, J. (2001). Greedy Function Approximation: A Gra- dient Boosting Machine. Annals of Statistics, 1189–1232.

Freund, Y., & R. E. Schapire. (1996, July). “Experiments with a New Boosting Algorithm.” In Icml (Vol. 96, pp. 148–156).

Friedman, D. (2014). “Oral Testimony before the House Committee on Energy and Commerce, by the Subcommit- tee on Oversight and Investigations,” April 1, 2014, www. nhtsa.gov/Testimony (accessed October 2014).

Friedman, N., D. Geiger, & M. Goldszmidt. (1997). “Bayesian Net- work Classifiers.” Machine Learning, Vol. 29, No. 2–3, 131–163.

Haykin, S. (2009). Neural Networks and Learning Machines, 3rd ed. Upper Saddle River, NJ: Prentice Hall.

Hopfield, J. (1982, April). “Neural Networks and Physical Systems with Emergent Collective Computational Abilities.” Proceed- ings of National Academy of Science, Vol. 79, No. 8, 2554–2558.

Koller, D., & N. Friedman. (2009). Probabilistic Graphical Models: Principles and Techniques. Boston, MA: MIT Press.

McCulloch, W., & W. Pitts. (1943). “A Logical Calculus of the Ideas Imminent in Nervous Activity.” Bulletin of Mathemat- ical Biophysics, Vol. 5.

Medsker, L., & J. Liebowitz. (1994). Design and Development of Expert Systems and Neural Networks. New York, NY: Macmillan, p. 163.

Meyfroidt, G., F. Güiza, J. Ramon, & M. Bruynooghe. (2009). “Machine Learning Techniques to Examine Large Patient Databases.” Best Practice & Research Clinical Anaesthesiol- ogy, 23(1), 127–143.

Minsky, M., & S. Papert. (1969). Perceptrons. Cambridge, MA: MIT Press.

Neural Technologies. “Combating Fraud: How a Leading Telecom Company Solved a Growing Problem.” neuralt. com/iqs/dlsfa.list/dlcpti.7/downloads.html (accessed March 2009).

NHTSA (2018) National Highway Traffic Safety Administra- tion (NHTSA’s) General Estimate System (GES), www. nhtsa.gov (accessed January 20, 2017).

Pearl, J. (1985). “Bayesian Networks: A Model of Self-Activated Memory for Evidential Reasoning.” Proceedings of the Seventh Conference of the Cognitive Science Society, 1985, pp. 329–334.

Pearl, J. (2009). Causality. Cambridge University Press, Cam- bridge, England.

Principe, J., N. Euliano, & W. Lefebvre. (2000). Neural and Adaptive Systems: Fundamentals Through Simulations. New York, NY: Wiley.

Reddy, B. K., D. Delen, & R. K. Agrawal. (2018). “Pre- dicting and Explaining Inflammation in Crohn’s Disease Patients Using Predictive Analytics Methods and Electronic Medical Record Data.” Health Informat- ics Journal, 1460458217751015.

Reagan, R. (1982). “Remarks on Signing Executive Order 12368, Concerning Federal Drug Abuse Policy Func- tions,” June 24, 1982. Online by Gerhard Peters and John

314 Part II • Predictive Analytics/Machine Learning

T. Woolley, The American Presidency Project. http:// www. presidency.ucsb.edu/ws/?pid=42671.

Surowiecki, J. (2006). The Wisdom of Crowds. New York, NY: Penguin Random House.

Topuz, K., F. Zengul, A. Dag, A. Almehmi, & M. Yildirim (2018). “Predicting Graft Survival Among Kidney Transplant Recipients: A Bayesian Decision Support Model.” Decision Support Systems, 106, 97–109.

Vorhies, W. (2016). “Want to Win Competitions? Pay Atten- tion to Your Ensembles.” Data Science Central Web Portal, www.datasciencecentral.com/profiles/blogs/want- to-win-at-kaggle-pay-attention-to-your-ensembles (accessed July 2018).

Wang, G., T. Xu, T. Tang, T. Yuan, & H. Wang. (2017). A “Bayes- ian Network Model for Prediction of Weather-Related Fail- ures in Railway Turnout Systems.” Expert Systems with Applications, 69, 247–256.

Wolpert, D. (1992). “Stacked Generalization.” Neural Networks, 5(2), 241–260.

Zahedi, F. (1993). Intelligent Systems for Business: Expert Systems with Neural Networks, Wadsworth, Belmont, CA.

Zhang, H., A. C. Berg, M. Maire, & J. Malik. (2006). “SVM-KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition.” In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on (Vol. 2, pp. 2126–2136). IEEE.

315

Deep Learning and Cognitive Computing

LEARNING OBJECTIVES

■■ Learn what deep learning is and how it is changing the world of computing

■■ Know the placement of deep learning within the broad family of artificial intelligence (AI) learning methods

■■ Understand how traditional “shallow” artificial neural networks (ANN) work

■■ Become familiar with the development and learning processes of ANN

■■ Develop an understanding of the methods to shed light into the ANN black box

■■ Know the underlying concept and methods for deep neural networks

■■ Become familiar with different types of deep learning methods

■■ Understand how convolutional neural networks (CNN) work

■■ Learn how recurrent neural networks (RNN) and long short-memory networks (LSTM) work

■■ Become familiar with the computer frameworks for implementing deep learning

■■ Know the foundational details about cognitive computing

■■ Learn how IBM Watson works and what types of application it can be used for

A rtificial intelligence (AI) is making a re-entrance into the world of commuting and in our lives, this time far stronger and much more promising than before. This unprecedented re-emergence and the new level of expectations can largely be attributed to deep learning and cognitive computing. These two latest buzzwords de- fine the leading edge of AI and machine learning today. Evolving out of the traditional artificial neural networks (ANN), deep learning is changing the very foundation of how machine learning works. Thanks to large collections of data and improved computational resources, deep learning is making a profound impact on how computers can discover complex patterns using the self-extracted features from the data (as opposed to a data scientist providing the feature vector to the learning algorithm). Cognitive computing— first popularized by IBM Watson and its success against the best human players in the game show Jeopardy!—makes it possible to deal with a new class of problems, the type

C H A P T E R

6

316 Part II • Predictive Analytics/Machine Learning

of problems that are thought to be solvable only by human ingenuity and creativity, ones that are characterized by ambiguity and uncertainty. This chapter covers the concepts, methods, and application of these two cutting-edge AI technology trends.

6.1 Opening Vignette: Fighting Fraud with Deep Learning and Artificial Intelligence 316

6.2 Introduction to Deep Learning 320 6.3 Basics of “Shallow” Neural Networks 325 6.4 Process of Developing Neural Network–Based Systems 334 6.5 Illuminating the Black Box of ANN 340 6.6 Deep Neural Networks 343 6.7 Convolutional Neural Networks 349 6.8 Recurrent Networks and Long Short-Term Memory Networks 360 6.9 Computer Frameworks for Implementation of Deep Learning 368

6.10 Cognitive Computing 370

6.1 OPENING VIGNETTE: Fighting Fraud with Deep Learning and Artificial Intelligence

THE BUSINESS PROBLEM

Danske Bank is a Nordic universal bank with strong local roots and bridges to the rest of the world. Founded in October 1871, Danske Bank has helped people and businesses in the Nordics realize their ambitions for over 145 years. Its headquarters is in Denmark, with core markets in Denmark, Finland, Norway, and Sweden.

Mitigating fraud is a top priority for banks. According to the Association of Certified Fraud Examiners, businesses lose more than $3.5 trillion each year to fraud. The problem is pervasive across the financial industry and is becoming more prevalent and sophis- ticated each month. As customers conduct more banking online across a wider variety of channels and devices, there are more opportunities for fraud to occur. Adding to the problem, fraudsters are becoming more creative and technologically savvy—they are also using advanced technologies such as machine learning—and new schemes to defraud banks are evolving rapidly.

Old methods for identifying fraud, such as using human-written rules engines, catch only a small percentage of fraud cases and produce a significantly high number of false positives. While false negatives end up costing money to the bank, chasing after a large number of false positives not only costs time and money but also blemishes customer trust and satisfaction. To improve probability predictions and identify a much higher per- centage of actual cases of fraud while reducing false alarms, banks need new forms of analytics. This includes using artificial intelligence.

Danske Bank, like other global banks, is seeing a seismic shift in customer interac- tions. In the past, most customers handled their transactions in a bank branch. Today, almost all interactions take place digitally through a mobile phone, tablet, ATM, or call center. This provides more “surface area” for fraud to occur. The bank needed to mod- ernize its fraud detection defenses. It struggled with a low 40 percent fraud detection rate and was managing up to 1,200 false positives per day—and 99.5 percent of all cases the bank was investigating were not fraud related. That large number of false alarms required a substantial investment of people, time, and money to investigate what turned out to be dead ends. Working with Think Big Analytics, a Teradata company, Danske Bank made a strategic decision to apply innovative analytic techniques, including AI, to better identify instances of fraud while reducing false positives.

Chapter 6 • Deep Learning and Cognitive Computing 317

THE SOLUTION: DEEP LEARNING ENHANCES FRAUD DETECTION

Danske Bank integrated deep learning with graphics processing unit (GPU) appliances that were also optimized for deep learning. The new software system helps the analyt- ics team to identify potential cases of fraud while intelligently avoiding false positives. Operational decisions are shifted from users to AI systems. However, human interven- tion is still necessary in some cases. For example, the model can identify anomalies, such as debit card purchases taking place around the world, but analysts are needed to determine whether that is fraud or a bank customer simply made an online purchase that sent a payment to China and then bought an item the next day from a retailer based in London.

Danske Bank’s analytic approach employs a “champion/challenger” methodology. With this approach, deep learning systems compare models in real time to determine which one is most effective. Each challenger processes data in real time, learning as it goes which traits are more likely to indicate fraud. If a process dips below a certain threshold, the model is fed more data, such as the geolocation of customers or recent ATM transactions. When a challenger outperforms other challengers, it transforms into a champion, giving the other models a roadmap to successful fraud detection.

THE RESULTS

Danske Bank implemented a modern enterprise analytic solution leveraging AI and deep learning, and it has paid big dividends. The bank was able to:

• Realize a 60 percent reduction in false positives with an expectation to reach as high as 80 percent.

• Increase true positives by 50 percent. • Focus resources on actual cases of fraud.

The following graph (see Figure 6.1) shows how true and false positive rates improved with advanced analytics (including deep learning). The red dot represents the old rules engine, which caught only about 40 percent of all fraud. Deep learning improved signifi- cantly upon machine learning, allowing Danske Bank to better detect fraud with much lower false positives.

Enterprise analytics is rapidly evolving and moving into new learning systems enabled by AI. At the same time, hardware and processors are becoming more powerful and spe- cialized, and algorithms more accessible, including those available through open source. This gives banks the powerful solutions needed to identify and mitigate fraud. As Danske Bank learned, building and deploying an enterprise-grade analytics solution that meets its specific needs and leverages its data sources deliver more value than traditional off- the-shelf tools could have provided. With AI and deep learning, Danske Bank now has the ability to better uncover fraud without being burdened by an unacceptable amount of false positives. The solution also allows the bank’s engineers, data scientists, lines of business, and investigative officers from Interpol, local police, and other agencies to col- laborate to uncover fraud, including sophisticated fraud rings. With its enhanced capabili- ties, the enterprise analytic solution is now being used across other business areas of the bank to deliver additional value.

Because these technologies are still evolving, implementing deep learning and AI solutions can be difficult for companies to achieve on their own. They can benefit by partnering with a company that has the proven capabilities to implement technology- enabled solutions that deliver high-value outcomes. As shown in this case, Think Big Analytics, a Teradata company, has the expertise to configure specialized hardware and software frameworks to enable new operational processes. The project entailed integrat- ing open-source solutions, deploying production models, and then applying deep learning

318 Part II • Predictive Analytics/Machine Learning

analytics to extend and improve the models. A framework was created to manage and track the models in the production system and to make sure the models could be trusted. These models enabled the underlying system to make autonomous decisions in real time that aligned with the bank’s procedural, security, and high-availability guidelines. The solution provided new levels of detail, such as time series and sequences of events, to better assist the bank with its fraud investigations. The entire solution was implemented very quickly—from kickoff to live in only five months. Figure 6.2 shows a generalized framework for AI and deep learning–based enterprise-level analytics solutions.

In summary, Danske Bank undertook a multi-step project to productionize machine- learning techniques while developing deep learning models to test those techniques. The integrated models helped identify the growing problem of fraud. For a visual summary, watch the video (https://www.teradata.com/Resources/Videos/Danske-Bank- Innovating-in-Artificial-Intelligence) and/or read the blog (http://blogs.teradata. com/customers/danske-bank-innovating-artificial-intelligence-deep-learning- detect-sophisticated-fraud/).

Deep Learning

21.0

20.8

20.6

20.4

20.2

0.0

0.0 0.02

Tr u e P

o s it iv

e R

a te

0.04 0.06

False Negative Rate

Random pred iction

0.08 0.10

Classic Machine Learning

Rules Engine

Ensemble (area = 0.89) CNN (area = 0.95) ResNet (area = 0.94) LSTM (area = 0.90) Rule Engine Random predictions

FIGURE 6.1 Deep Learning Improves Both True Positives and True Negatives.

Chapter 6 • Deep Learning and Cognitive Computing 319

u QUESTIONS FOR THE OPENING VIGNETTE

1. What is fraud in banking? 2. What are the types of fraud that banking firms are facing today? 3. What do you think are the implications of fraud on banks and on their customers? 4. Compare the old and new methods for identifying and mitigating fraud. 5. Why do you think deep learning methods provided better prediction accuracy? 6. Discuss the trade-off between false positive and false negative (type 1 and type 2

errors) within the context of predicting fraudulent activities.

WHAT WE CAN LEARN FROM THIS VIGNETTE

As you will see in this chapter, AI in general and the methods of machine learning in specific are evolving and advancing rapidly. The use of large digitized data sources, both from inside and outside the organization, both structured and unstructured, along with advanced computing systems (software and hardware combinations), has paved the way toward dealing with problems that were thought to be unsolvable just a few years ago. Deep learning and cognitive computing (as the ramifications of the cutting edge in AI systems) are helping enterprises to make accurate and timely decisions by harnessing the rapidly expanding Big Data resources. As shown in this opening vignette, this new generation of AI systems is capable of solving problems much bet- ter than their older counterparts. In the domain of fraud detection, traditional methods have always been marginally useful, having higher than desired false positive rates and causing unnecessary investigations and thereby dissatisfaction for their customers. As difficult problems such as fraud detection are, new AI technologies like deep learn- ing are making them solvable with a high level of accuracy and applicability.

Source: Teradata Case Study. “Danske Bank Fights Fraud with Deep Learning and AI.” https://www.teradata. com/Resources/Case-Studies/Danske-Bank-Fight-Fraud-With-Deep-Learning-and-AI (accessed August 2018). Used with permission.

Engineer

Simulate

M entoring

Handover Investigate

Cross-Functional Teams

4

3 2

1

Cross-Functional Teams

Leveragable

APIs

Validate

InsightsLive Test

Production

Test

Integrate

Analyze Data

Go Live

Tr

ai ni

ng

Ini tia

l W in

s

Al as-a-Service Manage iterative, stage-gate process for analytic models from development to handover to operations

Al Strategy Analyze business priorities and identify Al use cases. Review key enterprise AI capabilities and provide recommendations and next steps for customers to successfully get value from AI.

Al Rapid Analytic Consulting EngagementTM (Race) Use AI exploration to test use cases and provide a proof of value for AI approaches.

Al Foundation Operationalize use cases through data science and engineering; build and deploy a deep learning platform, integrating data sources, models, and business processes.

FIGURE 6.2 A Generalized Framework for AI and Deep Learning–Based Analytics Solutions.

320 Part II • Predictive Analytics/Machine Learning

6.2 INTRODUCTION TO DEEP LEARNING

About a decade ago, conversing with an electronic device (in human language, intelligently) would have been unconceivable, something that could only be seen in SciFi movies. Today, however, thanks to the advances in AI methods and technologies, almost everyone has ex- perienced this unthinkable phenomenon. You probably have already asked Siri or Google Assistant several times to dial a number from your phone address book or to find an address and give you the specific directions while you were driving. Sometimes when you were bored in the afternoon, you may have asked the Google Home or Amazon’s Alexa to play some music in your favorite genre on the device or your TV. You might have been surprised at times when you uploaded a group photo of your friends on Facebook and observed its tagging suggestions where the name tags often exactly match your friends’ faces in the pic- ture. Translating a manuscript from a foreign language does not require hours of struggling with a dictionary; it is as easy as taking a picture of that manuscript in the Google Translate mobile app and giving it a fraction of a second. These are only a few of the many, ever- increasing applications of deep learning that have promised to make life easier for people.

Deep learning, as the newest and perhaps at this moment the most popular member of the AI and machine-learning family, has a goal similar to those of the other machine- learning methods that came before it: mimic the thought process of humans—using math- ematical algorithms to learn from data pretty much the same way that humans learn. So, what is really different (and advanced) in deep learning? Here is the most commonly pronounced differentiating characteristic of deep learning over traditional machine learn- ing. The performance of traditional machine-learning algorithms such as decision trees, support vector machines, logistic regression, and neural networks relies heavily on the representation of the data. That is, only if we (analytics professionals or data scientists) provide those traditional machine-learning algorithms with relevant and sufficient pieces of information (a.k.a. features) in proper format are they able to “learn” the patterns and thereby perform their prediction (classification or estimation), clustering, or association tasks with an acceptable level of accuracy. In other words, these algorithms need humans to manually identify and derive features that are theoretically and/or logically relevant to the objectives of the problem on hand and feed these features into the algorithm in a proper format. For example, in order to use a decision tree to predict whether a given customer will return (or churn), the marketing manager needs to provide the algorithm with information such as the customer’s socioeconomic characteristics—income, occupa- tion, educational level, and so on (along with demographic and historical interactions/ transactions with the company). But the algorithm itself is not able to define such socio- economic characteristics and extract such features, for instance, from survey forms com- pleted by the customer or obtained from social media.

While such a structured, human-mediated machine-learning approach has been working fine for rather abstract and formal tasks, it is extremely challenging to have the approach work for some informal, yet seemingly easy (to humans), tasks such as face identification or speech recognition since such tasks require a great deal of knowledge about the world (Goodfellow et al., 2016). It is not straightforward, for instance, to train a machine-learning algorithm to accurately recognize the real meaning of a sentence spo- ken by a person just by manually providing it with a number of grammatical or semantic features. Accomplishing such a task requires a “deep” knowledge about the world that is not easy to formalize and explicitly present. What deep learning has added to the classic machine-learning methods is in fact the ability to automatically acquire the knowledge required to accomplish such informal tasks and consequently extract some advanced fea- tures that contribute to the superior system performance.

To develop an intimate understanding of deep learning, one should learn where it fits in the big picture of all other AI family of methods. A simple hierarchical relationship diagram,

Chapter 6 • Deep Learning and Cognitive Computing 321

or a taxonomy-like representation, may in fact provide such a holistic understanding. In an attempt to do this, Goodfellow and his colleagues (2016) categorized deep learning as part of the representation learning family of methods. Representation learning techniques entail one type of machine learning (which is also a part of AI) in which the emphasis is on learn- ing and discovering features by the system in addition to discovering the mapping from those features to the output/target. Figure 6.3 uses a Venn diagram to illustrate the place- ment of deep learning within the overarching family of AI-based learning methods.

Figure 6.4 highlights the differences in the steps/tasks that need to be performed when building a typical deep learning model versus the steps/tasks performed when building models with classic machine-learning algorithms. As shown in the top two work- flows, knowledge-based systems and classic machine-learning methods require data sci- entists to manually create the features (i.e., the representation) to achieve the desired output. The bottommost workflows show that deep learning enables the computer to derive some complex features from simple concepts that would be very effort intensive (or perhaps impossible in some problem situations) to be discovered by humans manu- ally, and then it maps those advanced features to the desired output.

From a methodological viewpoint, although deep learning is generally believed to be a new area in machine learning, its initial idea goes back to the late 1980s, just a few decades after the emergence of artificial neural networks when LeCun and colleagues (1989) published an article about applying backpropagation networks for recognizing handwritten ZIP codes. In fact, as it is being practiced today, deep learning seems to be nothing but an extension of neural networks with the idea that deep learning is able to deal with more complicated tasks with a higher level of sophistication by employing many layers of connected neurons along with much larger data sets to automatically character- ized variables and solve the problems but only at the expense of a great deal of compu- tational effort. This very high computational requirement and the need for very large data sets were the two main reasons why the initial idea had to wait more than two decades until some advanced computational and technological infrastructure emerged for deep

Artificial Intelligence

Machine Learning

Representation Learning

Deep Learning

CNN RNN

LSTM

Autoencoders Decision trees

Logistic regression

... ... Robotics Fuzzy logic

Knowledge- based/expert systems

Clustering PCA/ICA ......

FIGURE 6.3 A Venn Diagram Showing the Placement of Deep Learning within the Overarching AI-Based Learning Methods.

322 Part II • Predictive Analytics/Machine Learning

learning’s practical realization. Although the scale of neural networks has dramatically in- creased in the past decade by the advancement of related technologies, it is still estimated that having artificial deep neural networks with the comparable number of neurons and level of complexity existing in the human brain will take several more decades.

In addition to the computer infrastructures, as mentioned, the availability of large and feature-rich digitized data sets was another key reason for the development of suc- cessful deep learning applications in recent years. Obtaining good performance from a deep learning algorithm used to be a very difficult task that required extensive skills and experience/understanding to design task-specific networks, and therefore, not many were able to develop deep learning for practical and/or research purposes. Large training data sets, however, have greatly compensated for the lack of intimate knowledge and reduced the level of skill needed for implementing deep neural networks. Nevertheless, although the size of available data sets has exponentially increased in recent years, a great chal- lenge, especially for supervised learning of deep networks, is now the labeling of the cases in these huge data sets. As a result, a great deal of research is ongoing, focusing on how we can take advantage of large quantities of unlabeled data for semisupervised or unsupervised learning or how we can develop methods to label examples in bulk in a reasonable time.

The following section of this chapter provides a general introduction to neural networks from where deep learning has originated. Following the overview of these “shallow” neural networks, the chapter introduces different types of deep learning archi- tectures and how they work, some common applications of these deep learning architec- tures, and some popular computer frameworks to use in implementing deep learning in practice. Since, as mentioned, the basics of deep learning are the same as those of arti- ficial neural networks, in the following section, we provide a brief coverage of the neu- ral network architecture (namely, multilayered perceptron [MLP]-type neural networks, which was omitted in the neural network section in Chapter 5 because it was to be covered here) to focus on their mathematical principles and then explain how the vari- ous types of deep learning architectures/approaches were derived from these founda- tions. Application Case 6.1 provides an interesting example of what deep learning and advanced analytics techniques can achieve in the field of football.

Input Knowledge-

Based Systems

Classic Machine Learning

Generic

Deep Learning

R e p re

s e n ta

ti o n L

e a rn

in g

Input

Input

Input

Manually Created

Representation Output

Manually Created Features

Auto- Created Features

Simple Features

More Advanced Features

Mapping from

Features

Mapping from

Features

Mapping from Features

Output

Output

Output

FIGURE 6.4 Illustration of the Key Differences between Classic Machine-Learning Methods and Representation Learning/Deep Learning (shaded boxes indicate components that are able to learn

directly from data).

Chapter 6 • Deep Learning and Cognitive Computing 323

Football. Soccer. The beautiful game. Whatever you call it, the world’s most popular sport is being trans- formed by a Dutch start-up bringing AI to the pitch. SciSports, founded in 2012 by two self-proclaimed football addicts and data geeks, is innovating on the edge of what is possible. The sports analytics com- pany uses streaming data and applies machine learn- ing, deep learning, and AI to capture and analyze these data, making way for innovations in everything from player recruitment to virtual reality for fans.

Player Selection Goes High Tech

In the era of eight-figure contracts, player recruitment is a high-stakes game. The best teams are not those with the best players but the best combination of players. Scouts and coaches have used observation, rudimentary data, and intuition for decades, but savvy clubs now are using advanced analytics to identify rising stars and undervalued players. “The SciSkill Index evaluates every professional football player in the world in one universal index,” says SciSports founder and CEO Giels Brouwer. The company uses machine-learning algorithms to calculate the quality, talent, and value of more than 200,000 players. This

helps clubs find talent, look for players who fit a cer- tain profile, and analyze their opponents.

Every week, more than 1,500 matches in 210 leagues are analyzed by the SciSkill technology. Armed with this insight, SciSports partners with elite football clubs across Europe and other continents to help them sign the right players. This has led to several unexpected—and in some cases lucrative— player acquisitions. For example, a second-division Dutch player did not want to renew his contract, so he went out as a free agent. A new club reviewed the SciSkill index and found his data intriguing. That club was not too sure at first because it thought he looked clumsy in scouting—but the data told the true story. The club signed him as the third striker, and he quickly moved into a starting role and became its top goal scorer. His rights were sold at a large premium within two years, and now he is one of the top goal scorers in Dutch professional football.

Real-Time 3D Game Analysis

Traditional football data companies generate data only on players who have the ball, leaving everything else

Application Case 6.1 Finding the Next Football Star with Artificial Intelligence

(Continued )

324 Part II • Predictive Analytics/Machine Learning

undocumented. This provides an incomplete picture of player quality. Seeing an opportunity to capture the immense amount of data regarding what happens away from the ball, SciSports developed a camera sys- tem called BallJames.

BallJames is a real-time tracking technology that automatically generates 3D data from video. Fourteen cameras placed around a stadium record every movement on the field. BallJames then gen- erates data such as the precision, direction, and speed of the passing, sprinting strength, and jump- ing strength. “This forms a complete picture of the game,” says Brouwer. “The data can be used in lots of cool ways, from allowing fans to experience the game from any angle using virtual reality, to sports betting and fantasy sports.” He added that the data can even help coaches on the bench. “When they want to know if a player is getting tired, they can substitute players based on analytics.”

Machine Learning and Deep Learning

SciSports models on-field movements using machine- learning algorithms, which by nature improve on performing a task as the player gains more experi- ence. On the pitch, BallJames works by automati- cally assigning a value to each action, such as a cor- ner kick. Over time, these values change based on their success rate. A goal, for example, has a high value, but a contributing action—which may have previously had a low value—can become more valuable as the platform masters the game. Wouter Roosenburg, SciSports chief technology officer, says AI and machine learning will play an important role in the future of SciSports and football analyt- ics in general. “Existing mathematical models model

existing knowledge and insights in football, while artificial intelligence and machine learning will make it possible to discover new connections that people wouldn’t make themselves.”

To accurately compile 3D images, BallJames must distinguish between players, referees, and the ball. SAS Event Stream Processing enables real-time image recognition using deep learning models. “By combining our deep learning models into SAS'Viya', we can train our models in-memory in the cloud, on our cameras or wherever our resources are,” says Roosenburg. The ability to deploy deep learn- ing models in memory onto cameras and then do the inferencing in real time is cutting-edge sci- ence. “Having one uniform platform to manage the entire 3-D production chain is invaluable,” says Roosenburg. “Without SAS Viya, this project would not be possible.”

Adding Oomph to Open Source

Previously SciSports exclusively used open source to build models. It now benefits from an end-to- end platform that allows analytical teams to work in their language of choice and share a single, man- aged analytical asset inventory across the organiza- tion. According to Brouwer, this enables the firm to attract employees with different open-source skills yet still manage the production chain using one platform. “My CTO tells me he loves that our data scientists can do all the research in open source and he doesn’t have to worry about the produc- tion of the models,” says Brouwer. “What takes 100 lines of code in Python only takes five in SAS. This speeds our time to market, which is crucial in sports analytics.”

SciSports - Facts & Figures

1

Universal Index with Every Professional

Football Player

Players Analyzed in SciSkill Index

Cameras Around the Pitch Enable

Real-Time Analysis

200,000 14

Application Case 6.1 (Continued)

Chapter 6 • Deep Learning and Cognitive Computing 325

u SECTION 6.2 REVIEW QUESTIONS

1. What is deep learning? What can deep learning do? 2. Compared to traditional machine learning, what is the most prominent difference of

deep learning?

3. List and briefly explain different learning methods in AI. 4. What is representation learning, and how does it relate to deep learning?

6.3 BASICS OF “SHALLOW” NEURAL NETWORKS

Artificial neural networks are essentially simplified abstractions of the human brain and its complex biological networks of neurons. The human brain has a set of billions of interconnected neurons that facilitate our thinking, learning, and understanding of the world around us. Theoretically speaking, learning is nothing but the establishment and adaptation of new or existing interneuron connections. In the artificial neural networks, however, neurons are processing units (also called processing elements [PEs]) that perform a set of predefined mathematical operations on the numerical values coming from the input variables or from the other neuron outputs to create and push out its own outputs. Figure 6.5 shows a schematic representation of a single-input and single-output neuron (more accurately, the processing element in artificial neural networks).

In this figure, p represents a numerical input. Each input goes into the neuron with an adjustable weight w and a bias term b. A multiplication weight function applies the weight to the input, and a net input function shown by g adds the bias term to the weighted input z. The output of the net input function (n, known as the net input) then goes through another function called the transfer (a.k.a. activation) function (shown by f ) for conversion and the production of the actual output a. In other words:

a = f (wp + b)

Since its inception, SciSports has quickly become one of the world’s fastest-growing sports analytics companies. Brouwer says the versatility of the SAS Platform has also been a major factor. “With SAS, we’ve got the ability to scale processing power up or down as needed, put models into production in real time, develop everything in one platform and integrate with open source. Our ambition is to bring real-time data analytics to billions of soccer fans all over the world. By partnering with SAS, we can make that happen.”

Questions for Case 6.1

1. What does SciSports do? Look at its Web site for more information.

2. How can advanced analytics help football teams?

3. What is the role of deep learning in solutions provided by SciSports?

Sources: SAS Customer Stories. “Finding the Next Football Star with Artificial Intelligence.” www.sas.com/en_us/customers/ scisports. html (accessed August 2018).Copyright (c) 2018 SAS Institute Inc., Cary, NC, USA. All Rights Reserved. Used with permission.

X fSp

Input Single-Input Neuron a 5 f(wp 1 b)

w b

nz a

FIGURE 6.5 General Single-Input Artificial Neuron Representation.

326 Part II • Predictive Analytics/Machine Learning

A numerical example: if w = 2, p = 3, and b = -1, then a = f (2 * 3 - 1) = f (5). Various types of transfer functions are commonly used in the design of neural

networks. Table 6.1 shows some of the most common transfer functions and their cor- responding operations. Note that in practice, selection of proper transfer functions for a network requires a broad knowledge of neural networks—characteristics of the data as well as the specific purpose for which the network is created.

Just to provide an illustration, if in the previous example we had a hard limit trans- fer function, the actual output a would be a = hardlim(5) = 1. There are some guide- lines for choosing the appropriate transfer function for each set of neurons in a network. These guidelines are especially robust for the neurons located at the output layer of the network. For example, if the nature of the output for a model is binary, we are advised to use Sigmoid transfer functions at the output layer so that it produces an output between 0 and 1, which represents the conditional probability of y = 1 given x or P (y = 1 � x). Many neural network textbooks provide and elaborate on those guidelines at different layers in a neural network with some consistency and much disagreement, suggesting that the best practices should (and usually does) come from experience.

TABLE 6.1 Common Transfer (Activation) Functions in Neural Networks

Transfer Function Form Operation

Hard limit

a

n 0

21

11

a 5 hardlim (n)

a = +1 if n 7 0

a = 0 if n 6 0

Linear

21

11

a 5 purelin (n)

n

a

0 a = n

Log-Sigmoid

21

11

a 5 logsig (n)

n

a

0 a = 1

1 + e-n

Positive linear (a.k.a. rectified linear or ReLU)

21

11

a 5 poslin(n)

n

a

0

a = n if n 7 0

a = 0 if n 6 0

Chapter 6 • Deep Learning and Cognitive Computing 327

Typically, a neuron has more than a single input. In that case, each individual input pi can be shown as an element of the input vector p. Each of the individual input values would have its own adjustable weight wi of the weight vector W. Figure 6.6 shows a multiple-input neuron with R individual inputs.

For this neuron, the net input n can be expressed as:

n = w1,1 p1 + w1,2 p2 + w1,3 p3 + . . . + w1,R pR + b

Considering the input vector p as a R * 1 vector and the weight vector W as a 1 * R vector, then n can be written in matrix form as:

n = Wp + b

where Wp is a scalar (i.e., 1 * 1 vector). Moreover, each neural network is typically composed of multiple neurons connected

to each other and structured in consecutive layers so that the outputs of a layer work as the inputs to the next layer. Figure 6.7 shows a typical neural network with four neurons

X fSpR

p1

p1

Inputs Single-Input Neuron a 5 f(wp 1 b)

WRx1 bRx1

nz a

FIGURE 6.6 Typical Multiple-Input Neuron with R Individual Inputs.

Hidden Layer

Output Layer

Input Layer

In p u ts

Output

FIGURE 6.7 Typical Neural Network with Three Layers and Eight Neurons.

328 Part II • Predictive Analytics/Machine Learning

at the input (i.e., first) layer, four neurons at the hidden (i.e., middle) layer, and a single neuron at the output (i.e., last) layer. Each of the neurons has its own weight, weighting function, bias, and transfer function and processes its own input(s) as described.

While the inputs, weighting functions, and transfer functions in a given network are fixed, the values of the weights and biases are adjustable. The process of adjusting weights and biases in a neural network is what is commonly called training. In fact, in practice, a neural network cannot be used effectively for a prediction problem unless it is well trained by a sufficient number of examples with known actual outputs (a.k.a. targets). The goal of the training process is to adjust network weights and biases such that the network output for each set of inputs (i.e., each sample) is adequately close to its corresponding target value.

Application Case 6.2 provides a case where computer gaming companies are using advanced analytics to better understand and engage with their customers.

Video gamers are a special breed. Sure, they spend a lot of time playing games, but they’re also build- ing social networks. Like sports athletes, video game players thrive on competition. They play against other gamers online. Those who earn first place, or even second or third place, have bragging rights. And like athletes who invest a lot of time training, video gamers take pride in the number of hours they spend playing. Furthermore, as games increase in complexity, gamers take pride in developing unique skills to best their compatriots.

A New Level of Gaming

Video gaming has evolved from the days of PAC-MAN and arcades. The widespread availability of the Internet has fueled the popularity of video games by bringing them into people’s homes via a wide range of electronics such as the personal computer and mobile devices. The world of computer games is now a powerful and profitable business.

According to NewZoo’s Global Games Market Report from April 2017, the global games market in 2017 saw:

• $109 billion in revenues.

• 7.8 percent increase from the previous year.

• 2.2 billion gamers globally.

• 42 percent of the market being mobile.

Video game companies can tap into this envi- ronment and learn valuable information about their customers, especially their behaviors and the under- lying motivations. These customer data enable com- panies to improve the gaming experience and better engage players.

Traditionally, the gaming industry appealed to its customers—the gamers—by offering striking graphics and captivating visualizations. As technol- ogy advanced, the graphics became more vivid with hi-def renditions. Companies have continued to use technology in highly creative ways to develop games that attract customers and capture their inter- ests, which results in more time spent playing and higher affinity levels. What video game companies have not done as well is to fully utilize technology to understand the contextual factors that drive sus- tained brand engagement.

Know the Players

In today’s gaming world, creating an exciting prod- uct is no longer enough. Games must strongly appeal to the visual and auditory senses in an era when people expect cool graphics and cutting-edge sound effects. Games must also be properly mar- keted to reach highly targeted player groups. There are also opportunities to monetize gaming characters in the form of commercially available merchandise (e.g., toy store characters) or movie rights. Making a game successful requires programmers, designers,

Application Case 6.2 Gaming Companies Use Data Analytics to Score Points with Players

Chapter 6 • Deep Learning and Cognitive Computing 329

scenarists, musicians, and marketers to work together and share information. That is where gamer and gaming data come into play.

For example, the size of a gamer’s network— the number and types of people a gamer plays with or against—usually correlates with more time spent playing and more money that is spent. The more relationships gamers have, the higher the likeli- hood they will play more games with more people because they enjoy the experience. Network effects amplify engagement volumes.

These data also help companies better under- stand the types of games each individual likes to play. These insights enable the company to recom- mend additional games across other genres that will likely exert a positive impact on player engagement and satisfaction. Companies can also use these data in marketing campaigns to target new gamers or entice existing gamers to upgrade their member- ships, for example, to premium levels.

Monetize Player Behaviors

Collaborative filtering (cFilter) is an advanced ana- lytic function that makes automatic predictions (fil- tering) about the interests of a user by collecting preferences or taste information from many users (collaborating). The cFilter function supposes that if User A has the same opinion as User B on one issue, then User A is more likely to have User B’s opinion on a different issue when compared to a random user. This shows that predictions are specific to a gamer based on data from many other gamers.

Filtering systems are often used by online retailers to make product recommendations. The analytics can determine products that a customer will like based on what other shoppers who made similar purchases also bought, liked, or rated highly. There are many examples across other industries such as healthcare, finance, manufacturing, and telecommunication.

The cFilter analytic function offers several ben- efits to online video game companies:

• Marketers can run more effective cam- paigns. Connections between gamers nat- urally form to create clusters. Marketers can isolate common player characteristics and

leverage those insights for campaigns. Con- versely, they can isolate players who do not belong to a cluster and determine what unique characteristics contribute to their nonconform- ing behaviors.

• Companies can improve player retention. A strong membership in a community of gam- ers decreases the chances of churn. The greater the incentives for gamers to belong to a group of active participants, the more desire they have to engage in competitions. This increases the “stickiness” of customers and can lead to more game subscriptions.

• Data insights lead to improved customer satisfaction. Clusters indicate a desire for certain types of games that correspond to dis- tinct gamer interests and behaviors. Compa- nies can create gaming experiences that are unique to each player. Enticing more peo- ple to play and play longer enhances gamer satisfaction.

Once they understand why customers want to play games and uncover their relationships with other gamers, companies can create the right incentives for players to keep returning. This ensures a sus- tained customer base and stable revenue streams.

Boost Loyalty and Revenue

Regardless of the genre, each video game has pas- sionate players who seek each other for competi- tions. The thrill of a conquest attracts avid engage- ment. Over time, distinct networks of gamers are formed, with each participant constructing social relationships that often lead to more frequent and intense gaming interactions.

The gaming industry is now utilizing data analytics and visualizations to discern customer behaviors better and uncover player motivations. Looking at customer segments is no longer enough. Companies are now looking at microsegments that go beyond traditional demographics like age or geo- graphic location to understand customer preferences such as favorite games, preferred levels of difficulty, or game genres.

By gaining analytic insights into gamer strat- egies and behaviors, companies can create unique

(Continued )

330 Part II • Predictive Analytics/Machine Learning

gaming experiences that are attuned to these behaviors. By engaging players with the games and features they desire, video game compa- nies gain a devoted following, grow profits, and develop new revenue streams through merchan- dising ventures.

For a visual treat, watch a short video (https:// www.teradata.com/Resources/Videos/Art-of- Analytics-The-Sword) to see how the companies can use analytics to decipher gamer relationships that drive user behaviors and lead to better games.

Questions for Case 6.2

1. What are the main challenges for gaming companies?

2. How can analytics help gaming companies stay competitive?

3. What types of data can gaming companies obtain and use for analytics?

Source: Teradata Case Study. https://www.teradata.com/ Resources/Case-Studies/Gaming-Companies-Use-Data- Analytics (accessed August 2018).

Technology Insight 6.1 briefly describes the common components (or elements) of a typical artificial neural network along with their functional relationships.

TECHNOLOGY INSIGHT 6.1 Elements of an Artificial Neural Network

A neural network is composed of processing elements that are organized in different ways to form the network’s structure. The basic processing unit in a neural network is the neuron. A number of neurons are then organized to establish a network of neurons. Neurons can be orga- nized in a number of different ways; these various network patterns are referred to as topologies or network architectures (some of the most common architectures are summarized in Chapter 5). One of the most popular approaches, known as the feedforward-multilayered perceptron, allows all neurons to link the output in one layer to the input of the next layer, but it does not allow any feedback linkage (Haykin, 2009).

Processing Element (PE) The PE of an ANN is an artificial neuron. Each neuron receives inputs, processes them, and de- livers a single output as shown in Figure 6.5. The input can be raw input data or the output of other processing elements. The output can be the final result (e.g., 1 means yes, 0 means no), or it can be input to other neurons.

Network Structure

Each ANN is composed of a collection of neurons that are grouped into layers. A typical struc- ture is shown in Figure 6.8. Note the three layers: input, intermediate (called the hidden layer), and output. A hidden layer is a layer of neurons that takes input from the previous layer and converts those inputs into outputs for further processing. Several hidden layers can be placed between the input and output layers, although it is common to use only one hidden layer. In that case, the hidden layer simply converts inputs into a nonlinear combination and passes the transformed inputs to the output layer. The most common interpretation of the hidden layer is as a feature-extraction mechanism; that is, the hidden layer converts the original inputs in the problem into a higher-level combination of such inputs.

In ANN, when information is processed, many of the processing elements perform their computations at the same time. This parallel processing resembles the way the human brain works, and it differs from the serial processing of conventional computing.

Application Case 6.2 (Continued)

Chapter 6 • Deep Learning and Cognitive Computing 331

Input Layer

Y1

X1

X2 X3

(PE)

(PE)

(PE) Hidden Layer

(PE)

(PE) Output Layer

(PE)

(PE)

Weighted Sum ( )

Transfer Function

( )fS

FIGURE 6.8 Neural Network with One Hidden Layer. PE: processing element (an artificial representation of a biological neuron); Xi: inputs to a PE; y: output generated by a PE; g : summation function; and f : activation/transfer function.

Input Each input corresponds to a single attribute. For example, if the problem is to decide on ap- proval or disapproval of a loan, attributes could include the applicant’s income level, age, and home ownership status. The numeric value, or the numeric representation of non-numeric value, of an attribute is the input to the network. Several types of data, such as text, picture, and voice, can be used as inputs. Preprocessing may be needed to convert the data into meaningful inputs from symbolic/non-numeric data or to numeric/scale data.

Outputs The output of a network contains the solution to a problem. For example, in the case of a loan application, the output can be “yes” or “no.” The ANN assigns numeric values to the output, which may then need to be converted into categorical output using a threshold value so that the results would be 1 for “yes” and 0 for “no.”

Connection Weights Connection weights are the key elements of an ANN. They express the relative strength (or mathematical value) of the input data or the many connections that transfer data from layer to layer. In other words, weights express the relative importance of each input to a processing element and, ultimately, to the output. Weights are crucial in that they store learned patterns of information. It is through repeated adjustments of weights that a network learns.

Summation Function The summation function computes the weighted sums of all input elements entering each pro- cessing element. A summation function multiplies each input value by its weight and totals the values for a weighted sum. The formula for n inputs (represented with X ) in one processing element is shown in Figure 6.9a, and for several processing elements, the summation function formulas are shown in Figure 6.9b.

Transfer Function

The summation function computes the internal stimulation, or activation level, of the neuron. Based on this level, the neuron may or may not produce an output. The relationship between the

332 Part II • Predictive Analytics/Machine Learning

(a) Single Neuron

(PE)

PE: Processing Element (or neuron)

Y1

Y2 X2

W2

W1 W11

W21

W12

W22

W23

X1

X2

X1

Y3

Y

Y 5 X1W1 1 X2W2

Y1 5 X1W11 1 X2W21 Y2 5 X1W12 1 X2W22 Y3 5 X2W23

(PE)

(PE)

(PE)

(b) Multiple Neurons

FIGURE 6.9 Summation Function for (a) a Single Neuron/PE and (b) Several Neurons/PEs.

Processing Element (PE)

Y 5 1.2

Summation Function: Y 5 3(0.2) 1 1(0.4) 1 2(0.1) 5 1.2

Transfer Function: YT 5 1/(1 1 e 21.2) 5 0.77

YT 5 0.77

X1 5 3

X2 5 1

X3 5 2

W 1 5 0.2

W2 5 0.4

W 3 5

0. 1

FIGURE 6.10 Example of ANN Transfer Function.

internal activation level and the output can be linear or nonlinear. The relationship is expressed by one of several types of transformation (transfer) functions (see Table 6.1 for a list of commonly used activation functions). Selection of the specific activation function affects the network’s op- eration. Figure 6.10 shows the calculation for a simple sigmoid-type activation function example.

The transformation modifies the output levels to fit within a reasonable range of values (typically between 0 and 1). This transformation is performed before the output reaches the next level. Without such a transformation, the value of the output becomes very large, especially when there are several layers of neurons. Sometimes a threshold value is used instead of a trans- formation function. A threshold value is a hurdle value for the output of a neuron to trigger the next level of neurons. If an output value is smaller than the threshold value, it will not be passed to the next level of neurons. For example, any value of 0.5 or less becomes 0, and any value above 0.5 becomes 1. A transformation can occur at the output of each processing element, or it can be performed only at the final output nodes.

Chapter 6 • Deep Learning and Cognitive Computing 333

Application Case 6.3 provides an interesting use case where advanced analytics and deep learning are being used to prevent the extinction of rare animals.

“There are some people who want to kill animals like the lions and cheetahs. We would like to teach them, there are not many left,” says WildTrack offi- cials. The more we can study their behavior, the more we can help to protect them—and sustain the earth’s biodiversity that supports us all. Their tracks tell a collective story that holds incredible value in conservation. Where are they going? How many are left? There is much to be learned by monitoring footprints of endangered species like the cheetah.

WildTrack, a nonprofit organization, was founded in 2004 by Zoe Jewell and Sky Alibhai, a vet- erinarian and a wildlife biologist, respectively, who had been working for many years in Africa monitor- ing black and white rhinos. While in Zimbabwe, in the early 1990s, they collected and presented data to show that invasive monitoring techniques used for black rhinos were negatively impacting female fertility and began to develop a footprint identifica- tion technique. Interest from researchers around the world who needed a cost-effective and noninvasive approach to wildlife monitoring sparked WildTrack.

Artificial intelligence may help people recre- ate some of the skills used by indigenous trackers. WildTrack researchers are exploring the value AI can bring to conservation. They think that AI solu- tions are designed to enhance human efforts—not replace them. With deep learning, given enough data, a computer can be trained to perform human- like tasks such as identifying footprint images and recognizing patterns in a similar way to indigenous trackers—but with the added ability to apply these concepts at a much larger scale and more rapid pace. Analytics really underpins the whole thing, potentially giving insights into species populations that WildTrack never had before.

The WildTrack footprint identification tech- nique is a tool for noninvasive monitoring of endan- gered species through digital images of footprints. Measurements from these images are analyzed by customized mathematical models that help to identify the species, individual, sex, and age class. AI could add the ability to adapt through progressive learning algorithms and tell an even more complete story.

Obtaining crowdsourcing data is the next important step toward redefining what con- servation looks like in the future. Ordinary people would not necessarily be able to dart a rhino, but they can take an image of a foot- print. WildTrack has data coming in from everywhere—too much to manage traditionally. That’s really where AI comes in. It can automate repetitive learning through data, performing fre- quent, high- volume, computerized tasks reliably and without fatigue.

“Our challenge is how to harness artificial intelligence to create an environment where there’s room for us, and all species in this world,” says Alibhai.

Questions for Case 6.3

1. What is WildTrack and what does it do?

2. How can advanced analytics help WildTrack?

3. What are the roles that deep learning plays in this application case?

Source: SAS Customer Story. “Can Artificial Intelligence Help Protect These Animals from Extinction? The Answer May Lie in Their Footprints.” https://www.sas.com/en_us/explore/ analytics-in-action/impact/WildTrack.html (accessed August 2018); WildTrack.org.

Application Case 6.3 Artificial Intelligence Helps Protect Animals from Extinction

u SECTION 6.3 REVIEW QUESTIONS

1. How does a single artificial neuron (i.e., PE) work? 2. List and briefly describe the most commonly used ANN activation functions. 3. What is MLP, and how does it work? 4. Explain the function of weights in ANN. 5. Describe the summation and activation functions in MLP-type ANN architecture.

334 Part II • Predictive Analytics/Machine Learning

6.4 PROCESS OF DEVELOPING NEURAL NETWORK–BASED SYSTEMS

Although the development process of ANN is similar to the structured design methodolo- gies of traditional computer-based information systems, some phases are unique or have some unique aspects. In the process described here, we assume that the preliminary steps of system development, such as determining information requirements, conducting a fea- sibility analysis, and gaining a champion in top management for the project, have been completed successfully. Such steps are generic to any information system.

As shown in Figure 6.11, the development process for an ANN application in- cludes nine steps. In step 1, the data to be used for training and testing the network

Get more data; reformat data

Collect, organize and format the data 1

Step

Re-separate data into subsets

Separate data into training, validation, and testing sets

2

Change network architecture

Decide on a network architecture and structure 3

Change learning algorithm

Select a learning algorithm 4

Change network parameters

Reset and restart the training

Set network parameters and initialize their values

5

Initialize weights and start training (and validation)

6

Stop training, freeze the network weights

7

Test the trained network 8

Deploy the network for use on unknown new cases

9

FIGURE 6.11 Development Process of an ANN Model.

Chapter 6 • Deep Learning and Cognitive Computing 335

are collected. Important considerations are that the particular problem is amenable to a neural network solution and that adequate data exist and can be obtained. In step 2, training data must be identified, and a plan must be made for testing the performance of the network.

In steps 3 and 4, a network architecture and a learning method are selected. The availability of a particular development tool or the capabilities of the development personnel may determine the type of neural network to be constructed. Also, certain problem types have demonstrated high success rates with certain configurations (e.g., multilayer feedforward neural networks for bankruptcy prediction [Altman (1968), Wilson and Sharda (1994), and Olson, Delen, and Meng (2012)]). Important considerations are the exact number of neurons and the number of layers. Some packages use genetic algo- rithms to select the network design.

There are several parameters for tuning the network to the desired learning perfor- mance level. Part of the process in step 5 is the initialization of the network weights and pa- rameters followed by the modification of the parameters as training performance feedback is received. Often, the initial values are important in determining the efficiency and length of training. Some methods change the parameters during training to enhance performance.

Step 6 transforms the application data into the type and format required by the neu- ral network. This may require writing software to preprocess the data or performing these operations directly in an ANN package. Data storage and manipulation techniques and processes must be designed for conveniently and efficiently retraining the neural network when needed. The application data representation and ordering often influence the ef- ficiency and possibly the accuracy of the results.

In steps 7 and 8, training and testing are conducted iteratively by presenting input and desired or known output data to the network. The network computes the outputs and adjusts the weights until the computed outputs are within an acceptable tolerance of the known outputs for the input cases. The desired outputs and their relationships to input data are derived from historical data (i.e., a portion of the data collected in step 1).

In step 9, a stable set of weights is obtained. Then the network can reproduce the desired outputs given inputs such as those in the training set. The network is ready for use as a stand-alone system or as part of another software system where new input data will be presented to it and its output will be a recommended decision.

Learning Process in ANN

In supervised learning, the learning process is inductive; that is, connection weights are de- rived from existing cases. The usual process of learning involves three tasks (see Figure 6.12):

1. Compute temporary outputs. 2. Compare outputs with desired targets. 3. Adjust the weights and repeat the process.

Like any other supervised machine-learning technique, neural network training is usually done by defining a performance function (F) (a.k.a. cost function or loss func- tion) and optimizing (minimizing) that function by changing model parameters. Usually, the performance function is nothing but a measure of error (i.e., the difference between the actual input and the target) across all inputs of a network. There are several types of error measures (e.g., sum square errors, mean square errors, cross entropy, or even cus- tom measures) all of which are designed to capture the difference between the network outputs and the actual outputs.

The training process begins by calculating outputs for a given set of inputs using some random weights and biases. Once the network outputs are on hand, the performance

336 Part II • Predictive Analytics/Machine Learning

function can be computed. The difference between the actual output (Y or YT) and the desired output (Z) for a given set of inputs is an error called delta (in calculus, the Greek symbol delta, ∆, means “difference”).

The objective is to minimize delta (i.e., reduce it to 0 if possible), which is done by adjusting the network’s weights. The key is to change the weights in the proper direction, making changes that reduce delta (i.e., error). Different ANNs compute delta in different ways, depending on the learning algorithm being used. Hundreds of learning algorithms are available for various situations and configurations of ANN.

Backpropagation for ANN Training

The optimization of performance (i.e., minimization of the error or delta) in the neural network is usually done by an algorithm called stochastic gradient descent (SGD), which is an iterative gradient-based optimizer used for finding the minimum (i.e., the lowest point) in performance functions, as in the case of neural networks. The idea behind the SGD algorithm is that the derivative of the performance function with respect to each current weight or bias indicates the amount of change in the error measure by each unit of change in that weight or bias element. These derivatives are referred to as network gradients. Calculation of network gradients in the neural networks requires application of an algorithm called backpropagation, which is the most popular neural network learning algorithm, that applies the chain rule of calculus to compute the deriv- atives of functions formed by composing other functions whose derivatives are known [more on the mathematical details of this algorithm can be found in Rumelhart, Hinton, and Williams (1986)].

Compute the output

ANN Model

Stop the learning

and freeze the weights

Is the desired output

achieved?

Yes

NoAdjust the weights

FIGURE 6.12 Supervised Learning Process of an ANN.

Chapter 6 • Deep Learning and Cognitive Computing 337

Backpropagation (short for back-error propagation) is the most widely used supervised learning algorithm in neural computing (Principe, Euliano, and Lefebvre, 2000). By using the SGD mentioned previously, the implementation of backpropaga- tion algorithms is relatively straightforward. A neural network with backpropagation learning includes one or more hidden layers. This type of network is considered feed- forward because there are no interconnections between the output of a processing element and the input of a node in the same layer or in a preceding layer. Externally provided correct patterns are compared with the neural network’s output during (su- pervised) training, and feedback is used to adjust the weights until the network has categorized all training patterns as correctly as possible (the error tolerance is set in advance).

Starting with the output layer, errors between network-generated actual output and the desired outputs are used to correct/adjust the weights for the connections between the neurons (see Figure 6.13). For any output neuron j, the error (delta) = (Zj - Yj) (df/dx), where Z and Y are the desired and actual outputs, respectively. Using the sigmoid func- tion, f = 31 + exp(-x)4-1, where x is proportional to the sum of the weighted inputs to the neuron, is an effective way to compute the output of a neuron in practice. With this function, the derivative of the sigmoid function df/dx = f (1 - f ) and of the error is a simple function of the desired and actual outputs. The factor f (1 - f ) is the logistic func- tion, which serves to keep the error correction well bounded. The weight of each input to the jth neuron is then changed in proportion to this calculated error. A more complicated expression can be derived to work backward in a similar way from the output neurons through the hidden layers to calculate the corrections to the associated weights of the inner neurons. This complicated method is an iterative approach to solving a nonlinear optimization problem that is very similar in meaning to the one characterizing multiple linear regression.

In backpropagation, the learning algorithm includes the following procedures:

1. Initialize weights with random values and set other parameters. 2. Read in the input vector and the desired output. 3. Compute the actual output via the calculations, working forward through the layers. 4. Compute the error. 5. Change the weights by working backward from the output layer through the hidden

layers.

W1

W2 Yi

Summation Transfer Function

f(S) Y 5 f(S)

a( Zi 2 Yi ) error

Wn Xn

X2

X1

Neuron (or PE)

S 5 i 5 1

n XiWiπ

FIGURE 6.13 Backpropagation of Error for a Single Neuron.

338 Part II • Predictive Analytics/Machine Learning

This procedure is repeated for the entire set of input vectors until the desired output and the actual output agree within some predetermined tolerance. Given the calcula- tion requirements for one iteration, training a large network can take a very long time; therefore, in one variation, a set of cases is run forward and an aggregated error is fed backward to speed the learning. Sometimes, depending on the initial random weights and network parameters, the network does not converge to a satisfactory performance level. When this is the case, new random weights must be generated, and the network parameters, or even its structure, may have to be modified before another attempt is made. Current research is aimed at developing algorithms and using parallel computers to improve this process. For example, genetic algorithms (GA) can be used to guide the selection of the network parameters to maximize the performance of the desired output. In fact, most commercial ANN software tools are now using GA to help users “optimize” the network parameters in a semiautomated manner.

A central concern in the training of any type of machine-learning model is over- fitting. It happens when the trained model is highly fitted to the training data set but performs poorly with regard to external data sets. Overfitting causes serious issues with respect to the generalizability of the model. A large group of strategies known as regu- larization strategies is designed to prevent models from overfitting by making changes or defining constraints for the model parameters or the performance function.

In the classic ANN models of small size, a common regularization strategy to avoid overfitting is to assess the performance function for a separate validation data set as well as the training data set after each iteration. Whenever the performance stopped improv- ing for the validation data, the training process would be stopped. Figure 6.14 shows a

Error Reduction on the Validation Set

Error Reduction on the Training Set

The Best Model

Training Iterations0 0

E rr

o r

FIGURE 6.14 Illustration of the Overfitting in ANN—Gradually Changing Error Rates in the Training and Validation Data Sets As the Number of Iterations Increases.

Chapter 6 • Deep Learning and Cognitive Computing 339

typical graph of the error measure by the number of iterations of training. As shown, in the beginning, the error decreases in both training and validation data by running more and more iterations; but from a specific point (shown by the dashed line), the error starts increasing in the validation set while still decreasing in the training set. It means that be- yond that number of iterations, the model becomes overfitted to the data set with which it is trained and cannot necessarily perform well when it is fed with some external data. That point actually represents the recommended number of iterations for training a given neural network.

Technology Insight 6.2 discusses some of the popular neural network software and offers some Web links to more comprehensive ANN-related software sites.

TECHNOLOGY INSIGHT 6.2 ANN Software

Many tools are available for developing neural networks (see this book’s Web site and the re- source lists at PC AI, pcai.com). Some of these tools function like software shells. They provide a set of standard architectures, learning algorithms, and parameters, along with the ability to manipulate the data. Some development tools can support several network paradigms and learn- ing algorithms.

Neural network implementations are also available in most of the comprehensive pre- dictive analytics and data mining tools, such as the SAS Enterprise Miner, IBM SPSS Modeler (formerly Clementine), and Statistica Data Miner. Weka, RapidMiner, Orange, and KNIME are open-source free data mining software tools that include neural network capabilities. These free tools can be downloaded from their respective Web sites; simple Internet searches on the names of these tools should lead you to the download pages. Also, most of the commercial software tools are available for download and use for evaluation purposes (usually they are limited on time of availability and/or functionality).

Many specialized neural network tools make the building and deployment of a neural network model an easier undertaking in practice. Any listing of such tools would be in- complete. Online resources such as Wikipedia (en.wikipedia.org/wiki/Artificial_neural_net- work), Google’s or Yahoo!’s software directory, and the vendor listings on pcai.com are good places to locate the latest information on neural network software vendors. Some of the vendors that have been around for a while and have reported industrial applications of their neural network software include California Scientific (BrainMaker), NeuralWare, NeuroDimension Inc., Ward Systems Group (Neuroshell), and Megaputer. Again, the list can never be complete.

Some ANN development tools are spreadsheet add-ins. Most can read spreadsheet, database, and text files. Some are freeware or shareware. Some ANN systems have been developed in Java to run directly on the Web and are accessible through a Web browser interface. Other ANN products are designed to interface with expert systems as hybrid de- velopment products.

Developers may instead prefer to use more general programming languages, such as C, C#, C++, Java, and so on, readily available R and Python libraries, or spreadsheets to program the model, perform the calculations, and deploy the results. A common practice in this area is to use a library of ANN routines. Many ANN software providers and open-source platforms provide such programmable libraries. For example, hav.Software (hav.com) provides a library of C++ classes for implementing stand-alone or embedded feedforward, simple recurrent, and random- order recurrent neural networks. Computational software such as MATLAB also includes neural network–specific libraries.

340 Part II • Predictive Analytics/Machine Learning

u SECTION 6.4 REVIEW QUESTIONS

1. List the nine steps in conducting a neural network project. 2. What are some of the design parameters for developing a neural network? 3. Draw and briefly explain the three-step process of learning in ANN. 4. How does backpropagation learning work? 5. What is overfitting in ANN learning? How does it happen, and how can it be mitigated? 6. Describe the different types of neural network software available today.

6.5 ILLUMINATING THE BLACK BOX OF ANN

Neural networks have been used as an effective tool for solving highly complex real- world problems in a wide range of application areas. Even though ANN have been proven to be superior predictors and/or cluster identifiers in many problem scenarios (compared to their traditional counterparts), in some applications, there exists an ad- ditional need to know “how the model does what it does.” ANN are typically known as black boxes, capable of solving complex problems but lacking the explanation of their capabilities. This lack of transparency situation is commonly referred to as the “black- box” syndrome.

It is important to be able to explain a model’s “inner being”; such an explanation offers assurance that the network has been properly trained and will behave as desired once deployed in a business analytics environment. Such a need to “look under the hood” might be attributable to a relatively small training set (as a result of the high cost of data acquisition) or a very high liability in case of a system error. One example of such an application is the deployment of airbags in vehicles. Here, both the cost of data acquisition (crashing vehicles) and the liability concerns (danger to human lives) are rather significant. Another representative example for the importance of explana- tion is loan-application processing. If an applicant is refused a loan, he or she has the right to know why. Having a prediction system that does a good job on differentiating good and bad applications may not be sufficient if it does not also provide the justifi- cation of its predictions.

A variety of techniques have been proposed for analysis and evaluation of trained neural networks. These techniques provide a clear interpretation of how a neural net- work does what it does; that is, specifically how (and to what extent) the individual inputs factor into the generation of specific network output. Sensitivity analysis has been the front-runner of the techniques proposed for shedding light into the black-box charac- terization of trained neural networks.

Sensitivity analysis is a method for extracting the cause-and-effect relationships among the inputs and the outputs of a trained neural network model. In the process of performing sensitivity analysis, the trained neural network’s learning capability is disabled so that the network weights are not affected. The basic procedure behind sensitivity analysis is that the inputs to the network are systematically perturbed within the allowable value ranges, and the corresponding change in the output is recorded for each and every input variable (Principe et al., 2000). Figure 6.15 shows a graphical illustration of this process. The first input is varied between its mean plus and minus of a user-defined number of standard deviations (or for categorical variables, all of its possible values are used) while all other input variables are fixed at their respective means (or modes). The network output is computed for a user-defined number of steps above and below the mean. This process is repeated for each input. As a result, a report is generated to summarize the variation of each output with respect to the variation in each input. The generated report often contains a column plot (along with

Chapter 6 • Deep Learning and Cognitive Computing 341

Systematically Perturbed

Inputs

Observed Change in Outputs

Trained ANN, the “Black Box”

D1

FIGURE 6.15 A Figurative Illustration of Sensitivity Analysis on an ANN Model.

According to the National Highway Traffic Safety Administration (NHTSA), over 6 million traffic acci- dents claim more than 41,000 lives each year in the United States. Causes of accidents and related injury severity are of special interest to traffic safety researchers. Such research is aimed at reducing not only the number of accidents but also the severity of injury. One way to accomplish the latter is to identify the most profound factors that affect injury severity. Understanding the circumstances under which driv- ers and passengers are more likely to be severely injured (or killed) in a vehicle accident can help improve the overall driving safety situation. Factors that potentially elevate the risk of injury severity of vehicle occupants in the event of an accident include demographic and/or behavioral characteris- tics of the person (e.g., age, gender, seatbelt usage, use of drugs or alcohol while driving), environmen- tal factors, and/or roadway conditions at the time of the accident (e.g., surface conditions, weather or light conditions, direction of the impact, vehicle ori- entation in the crash, occurrence of a rollover), as well as technical characteristics of the vehicle itself (e.g., age, body type).

In an exploratory data mining study, Delen et al. (2006) used a large sample of data—30,358 police- reported accident records obtained from the General Estimates System of NHTSA—to identify which fac- tors become increasingly more important in escalating the probability of injury severity during a traffic crash. Accidents examined in this study included a geograph- ically representative sample of multiple-vehicle colli- sion accidents, single-vehicle fixed-object collisions, and single-vehicle noncollision (rollover) crashes.

Contrary to many of the previous studies con- ducted in this domain, which have primarily used regression-type generalized linear models where the functional relationships between injury severity and crash-related factors are assumed to be linear (which is an oversimplification of the reality in most real-world situations), Delen and his colleagues (2006) decided to go in a different direction. Because ANN are known to be superior in capturing highly nonlinear complex relationships between the predictor variables (crash factors) and the target variable (severity level of the injuries), they decided to use a series of ANN models to estimate the significance of the crash factors on the level of injury severity sustained by the driver.

Application Case 6.4 Sensitivity Analysis Reveals Injury Severity Factors in Traffic Accidents

(Continued )

numeric values presented on the x-axis), reporting the relative sensitivity values for each input variable. A representative example of sensitivity analysis on ANN models is provided in Application Case 6.4.

342 Part II • Predictive Analytics/Machine Learning

From a methodological standpoint, Delen et al. (2006) followed a two-step process. In the first step, they developed a series of prediction models (one for each injury severity level) to capture the in-depth relation- ships between the crash-related factors and a specific level of injury severity. In the second step, they con- ducted sensitivity analysis on the trained neural network models to identify the prioritized importance of crash- related factors as they relate to different injury severity levels. In the formulation of the study, the five-class prediction problem was decomposed into a number of binary classification models to obtain the granularity of information needed to identify the “true” cause-and- effect relationships between the crash-related factors and different levels of injury severity. As shown in Figure 6.16, eight different neural network models have been developed and used in the sensitivity analy- sis to identify the key determinants of increased injury severity levels.

The results revealed considerable differences among the models built for different injury severity levels. This implies that the most influential factors in prediction models highly depend on the level of injury severity. For example, the study revealed that the variable seatbelt use was the most impor- tant determinant for predicting higher levels of injury severity (such as incapacitating injury or fatality), but it was one of the least significant pre- dictors for lower levels of injury severity (such as non-incapacitating injury and minor injury). Another interesting finding involved gender: The

driver’s gender was among the significant predic- tors for lower levels of injury severity, but it was not among the significant factors for higher lev- els of injury severity, indicating that more serious injuries do not depend on the driver being a male or a female. Another interesting and somewhat intuitive finding of the study indicated that age becomes an increasingly more significant factor as the level of injury severity increases, implying that older people are more likely to incur severe inju- ries (and fatalities) in serious vehicle crashes than younger people.

Questions for Case 6.4

1. How does sensitivity analysis shed light on the black box (i.e., neural networks)?

2. Why would someone choose to use a black-box tool such as neural networks over theoretically sound, mostly transparent statistical tools like logistic regression?

3. In this case, how did neural networks and sensi- tivity analysis help identify injury-severity factors in traffic accidents?

Sources: Delen, D., R. Sharda, & M. Bessonov. (2006). “Identifying Significant Predictors of Injury Severity in Traffic Accidents Using a Series of Artificial Neural Networks.” Accident Analysis and Prevention, 38(3), pp. 434–444; Delen, D., L. Tomak, K. Topuz, & E. Eryarsoy (2017). “Investigating Injury Severity Risk Factors in Automobile Crashes with Predictive Analytics and Sensitivity Analysis Methods.” Journal of Transport & Health, 4, pp. 118–131.

Model Label

1.1

1.2

1.3

1.4

2.1

2.2

2.3

2.4

No Injury (35.4%)

Probable Injury (23.6%)

Non-Incapacitating (19.6%)

Incapacitating (17.8%)

Fatal Injury (3.6%)

Binary category label 0 Binary category label 1

FIGURE 6.16 Graphical Representation of the Eight Binary ANN Model Configurations.

Application Case 6.4 (Continued)

Chapter 6 • Deep Learning and Cognitive Computing 343

u SECTION 6.5 REVIEW QUESTIONS

1. What is the so-called black-box syndrome? 2. Why is it important to be able to explain an ANN’s model structure? 3. How does sensitivity analysis work in ANN? 4. Search the Internet to find other methods to explain ANN methods. Report the results.

6.6 DEEP NEURAL NETWORKS

Until recently (before the advent of deep learning phenomenon), most neural network applications involved network architectures with only a few hidden layers and a limited number of neurons in each layer. Even in relatively complex business applications of neural networks, the number of neurons in networks hardly exceeded a few thousands. In fact, the processing capability of computers at the time was such a limiting factor that central processing units (CPU) were hardly able to run networks involving more than a couple of layers in a reasonable time. In recent years, development of graphics processing units (GPUs) along with the associated programming languages (e.g., CUDA by NVIDIA) that enable people to use them for data analysis purposes has led to more advanced appli- cations of neural networks. GPU technology has enabled us to successfully run neural net- works with over a million neurons. These larger networks are able to go deeper into the data features and extract more sophisticated patterns that could not be detected otherwise.

While deep networks can handle a considerably larger number of input variables, they also need relatively larger data sets to be trained satisfactorily; using small data sets for training deep networks typically leads to overfitting of the model to the training data and poor and unreliable results in case of applying to external data. Thanks to the Internet- and Internet of Things (IoT)-based data-capturing tools and technologies, larger data sets are now available in many application domains for deeper neural network training.

The input to a regular ANN model is typically an array of size R * 1, where R is the number of input variables. In the deep networks, however, we are able to use tensors (i.e., N-dimensional arrays) as input. For example, in image recognition networks, each input (i.e., image) can be represented by a matrix indicating the color codes used in the image pixels; or for video processing purposes, each video can be represented by several matrices (i.e., a 3D tensor), each representing an image involved in the video. In other words, tensors provide us with the ability to include additional dimensions (e.g., time, location) in analyzing the data sets.

Except for these general differences, the different types of deep networks involve various modifications to the architecture of standard neural networks that equip them with distinct capabilities of dealing with particular data types for advanced purposes. In the fol- lowing section, we discuss some of these special network types and their characteristics.

Feedforward Multilayer Perceptron (MLP)-Type Deep Networks

MLP deep networks, also known as deep feedforward networks, are the most general type of deep networks. These networks are simply large-scale neural networks that can contain many layers of neurons and handle tensors as their input. The types and characteristics of the network elements (i.e., weight functions, transfer functions) are pretty much the same as in the standard ANN models. These models are called feedforward because the flow of information that goes through them is always forwarding and no feedback connections (i.e., connections in which outputs of a model are fed back to itself) are allowed. The neural networks in which feedback connections are allowed are called recurrent neural networks (RNN). General RNN architectures, as well as a specific variation of RNNs called long short-term memory networks, are discussed in later sections of this chapter.

344 Part II • Predictive Analytics/Machine Learning

Generally, a sequential order of layers has to be held between the input and the output layers in the MLP-type network architecture. This means that the input vector has to pass through all layers sequentially and cannot skip any of them; moreover, it cannot be directly connected to any layer except for the very first one; the output of each layer is the input to the subsequent layer. Figure 6.17 demonstrates a vector representation of the first three layers of a typical MLP network. As shown, there is only one vector going into each layer, which is either the original input vector ( p for the first layer) or the output vector from the previous hidden layer in the network architecture (ai - 1 for the ith layer). There are, however, some special variations of MLP network architectures designed for specialized purposes in which these principles can be violated.

Impact of Random Weights in Deep MLP

Optimization of the performance (loss) function in many real applications of deep MLPs is a challenging issue. The problem is that applying the common gradient-based train- ing algorithms with random initialization of weights and biases that is very efficient for finding the optimal set of parameters in shallow neural networks most of the time could lead to getting stuck in the locally optimal solutions rather than catching the global opti- mum values for the parameters. As the depth of network increases, chances of reaching a global optimum using random initializations with the gradient-based algorithms decrease. In such cases, usually pretraining the network parameters using some unsupervised deep learning methods such as deep belief networks (DBNs) can be helpful (Hinton, Osindero, and Teh, 2006). DBNs are a type of a large class of deep neural networks called generative models. Introduction of DBNs in 2006 is considered as the beginning of the current deep learning renaissance (Goodfellow et al., 2016), since prior to that, deep models were considered too difficult to optimize. In fact, the primary application of DBNs today is to improve classification models by pretraining of their parameters.

Using these unsupervised learning methods, we can train the MLP layers, one at a time, starting from the first layer, and use the output of each layer as the input to the subsequent layer and initialize that layer with an unsupervised learning algorithm. At the end, we will have a set of initialized values for the parameters across the whole network. Those pre- trained parameters, instead of random initialized parameters, then can be used as the initial values in the supervised learning of the MLP. This pretraining procedure has been shown to cause significant improvements to the deep classification applications. Figure 6.18 illustrates the classification errors that resulted from training a deep MLP network with (blue circles) and without (black triangles) pretraining of parameters (Bengio, 2009). In this example, the blue line represents the observed error rates of testing a classification model (on 1,000 heldout examples) trained using a purely supervised approach with 10 million examples,

x p z1

a1 5 f1(w1p 1 b1) a2 5 f2(w2a1 1 b2)

a3 5 f 3(w3f 2(w2f 1(w1p1b1)1b2)1b3)

a3 5 f3(w3a2 1 b3)

w1 b1 w2 b2 w3 b3

n1 a1 z2 n2 a2 z3 n3 a3 S S Sf 1 x f 2 x f 3

I

n

p

u

t

FIGURE 6.17 Vector Representation of the First Three Layers in a Typical MLP Network.

Chapter 6 • Deep Learning and Cognitive Computing 345

whereas the black line indicates the error rates on the same testing data set when 2.5 million examples were initially used for unsupervised training of network parameters (using DBN) and then the other 7.5 million examples along with the initialized parameters were used to train a supervised classification model. The diagrams clearly show a significant improvement in terms of the classification error rate in the model pretrained by a deep belief network.

More Hidden Layers versus More Neurons?

An important question regarding the deep MLP models is “Would it make sense (and produce better results) to restructure such networks with only a few layers, but many neurons in each?” In other words, the question is why do we need deep MLP networks with many layers when we can include the same number of neurons in just a few layers (i.e., wide networks instead of deep networks). According to the universal approximation theorem (Cybenko, 1989; Hornik, 1991), a sufficiently large single-layer MLP network will be able to approximate any function. Although theoretically founded, such a layer with many neurons may be prohibitively large and hence may fail to learn the underlying pat- terns correctly. A deeper network can reduce the number of neurons required at each layer and hence decrease the generalization error. Whereas theoretically it is still an open research question, practically using more layers in a network seems to be more effective and computationally more efficient than using many neurons in a few layers.

Like typical artificial neural networks, multilayer perceptron networks can also be used for various prediction, classification, and clustering purposes. Especially when a large number of input variables are involved or in cases that the nature of input has to be an N -dimensional array, a deep multilayer network design needs to be employed.

Application Case 6.5 provides an excellent case for the use of advanced analytics to better manage traffic flows in crowded cities.

0 1024

1023

1022

1021

100

1 2 3 4 5 6 7 8 9

Number of Examples Seen (3106)

C la

s s ifi

c a ti

o n E

rr o r

10

FIGURE 6.18 The Effect of Pretraining Network Parameters on Improving Results of a Classification- Type Deep Neural Network.

346 Part II • Predictive Analytics/Machine Learning

The Background

When the Georgia Department of Transportation (GDOT) wanted to optimize the use of Big Data and advanced analytics to gain insight into transporta- tion, it worked with Teradata to develop a proof of concept evaluation of GDOT’s variable speed limit (VSL) pilot project.

The VSL concept has been adopted in many parts of the world, but it is still relatively new in the United States. As GDOT explains,

VSL are speed limits that change based on road, traffic, and weather conditions. Electronic signs slow down traffic ahead of congestion or bad weather to smooth out flow, diminish stop-and- go conditions, and reduce crashes. This low- cost, cutting edge technology alerts drivers in real time to speed changes due to conditions down the road. More consistent speeds improve safety by helping to prevent rear-end and lane changing collisions due to sudden stops.

Quantifying the customer service, safety, and efficiency benefits of VSL is extremely important to GDOT. This fits within a wider need to understand the effects of investments in intelligent transporta- tion systems as well as other transportation systems and infrastructures.

VSL Pilot Project on I-285 in Atlanta

GDOT conducted a VSL pilot project on the north- ern half, or “top end,” of I-285 that encircles Atlanta. This 36-mile stretch of highway was equipped with 88 electronic speed limit signs that adjusted speed limits in 10 mph increments from 65 miles per hour (mph) to the minimum of 35 mph. The objectives were twofold:

1. Analyze speeds on the highway before versus after implementation of VSL.

2. Measure the impact of VSL on driving conditions.

To obtain an initial view of the traffic, the Teradata data science solution identified the loca- tions and durations of “persistent slowdowns.” If highway speeds are above “reference speed,” then

traffic is considered freely flowing. Falling below the reference speed at any point on the highway is considered a slowdown. When slowdowns per- sist across multiple consecutive minutes, a persistent slowdown can be defined.

By creating an analytic definition of slow- downs, it is possible to convert voluminous and highly variable speed data into patterns to support closer investigation. The early analyses of the data revealed that the clockwise and counterclockwise directions of the same highway may show signifi- cantly different frequency and duration of slow- downs. To better understand how slowdowns affect highway traffic, it is useful to take our new defini- tion and zoom in on a specific situation. Figure 6.19 shows a specific but typical Atlanta afternoon on I-285, at a section of highway where traffic is mov- ing clockwise, from west to east, between mile markers MM10 in the west to the east end at MM46.

The first significant slowdown occurred at 3:00 p.m. near MM32. The size of the circles repre- sents duration (measured in minutes). The slowdown at MM32 was nearly four hours long. As the slow- down “persisted,” traffic speed diminished behind it. The slowdown formed on MM32 became a bottle- neck that caused traffic behind it to slow down as well. The “comet trail” of backed-up traffic at the top left of Figure 6.20 illustrates the sequential formation of slowdowns at MM32 and then farther west, each starting later in the afternoon and not lasting as long.

Measuring Highway Speed Variability

The patterns of slowdowns on the highway as well as their different timings and locations led us to ques- tion their impact on drivers. If VSL could help driv- ers better anticipate the stop-and-go nature of the slowdowns, then being able to quantify the impact would be of interest to GDOT. GDOT was particu- larly concerned about what happens when a driver first encounters a slowdown. “While we do not know what causes the slowdown, we do know that driv- ers have made speed adjustments. If the slowdown was caused by an accident, then the speed reduction could be quite sudden; alternatively, if the slowdown was just caused by growing volumes of traffic, then the speed reduction might be much more gradual.”

Application Case 6.5 Georgia DOT Variable Speed Limit Analytics Help Solve Traffic Congestions

Chapter 6 • Deep Learning and Cognitive Computing 347

Identifying Bottlenecks and Traffic Turbulence

A bottleneck starts as a slowdown at a particular loca- tion. Something like a “pinch point” occurs on the highway. Then, over a period of time, traffic slows down behind the original pinch point. A  bottle- neck is a length of highway where traffic falls below

60  percent of reverence speed and can stay at that level for miles. Figure 6.20 shows a conceptual repre- sentation of a bottleneck.

While bottlenecks are initiated by a pitch point, or slowdown, that forms the head of the queue, it is the end of the queue that is the most interest- ing. The area at the back of a queue is where traf- fic encounters a transition from free flow to slowly

14

2 PM

3 PM

M in

u te

o f B

o tt

le n e c k

S u s p e c te

d (D

e c e m

b e r

1 1

, 2

0 1

4 )

T im

e o

f D

a y

from West to East, by Mile MarkerDirection of Traffic

4 PM

5 PM

6 PM

7 PM

16 18 20 22 24 26 28

Pseudo Mile Marker

30 32 34 36 38 30 32

Slowdown Duration

100.0 200.0

219.0

FIGURE 6.19 Traffic Moving Clockwise during the Afternoon.

Turbulence Reduction Opportunity

Bottleneck (queuing traffic)

Tr a ffi

c S

p e e d (

m p h )

Zone of Influence

Bottleneck End

Bottleneck End

Direction of Travel

60% of Reference Speed

Speed of Traffic

Normal Traffic

FIGURE 6.20 Graphical Depiction of a Bottleneck on a Highway.

(Continued )

348 Part II • Predictive Analytics/Machine Learning

moving congested conditions. In the worst condi- tions, the end of the queue can experience a rapid transition. Drivers moving at highway speed may unexpectedly encounter slower traffic. This condi- tion is ripe for accidents and is the place where VSL can deliver real value.

Powerful New Insight on Highway Congestion

The availability of new Big Data sources that describe the “ground truth” of traffic conditions on highways provides rich new opportunities for developing and analyzing highway performance metrics. Using just a single data source on detailed highway speeds, we produced two new and distinctive metrics using Teradata advanced data science capabilities.

First, by defining and measuring persistent slowdowns, we helped traffic engineers understand the frequency and duration of slow speed locations on a highway. The distinction of measuring a per- sistent slowdown versus a fleeting one is uniquely challenging and requires data science. It provides the ability to compare the number, duration, and location of slowdowns in a way that is more infor- mative and compelling than simple averages, vari- ances, and outliers in highway speeds.

The second metric was the ability to measure turbulence caused by bottlenecks. By identifying where bottlenecks occur and then narrowing in on their very critical zones of influence, we can make measurements of speeds and traffic deceleration tur- bulence within those zones. Data science and ana- lytics capabilities demonstrated reduced turbulence when VSL is active in the critical zone of a bottleneck.

There is much more that could be explored within this context. For example, it is natural to assume that because most traffic is on the road dur- ing rush hours, VSL provides the most benefits dur- ing these high-traffic periods. However, the opposite may be true, which could provide a very important benefit of the VSL program.

Although this project was small in size and was just a proof of concept, a combination of similar projects beyond just transportation under the name of “smart cities” is underway around the United States and abroad. The goal is to use a variety of data from sensors to multimedia, rare event reports to satellite images along with advanced analytics that include deep learning and cognitive computing to transform the dynamic nature of cities toward bet- ter to best for all stakeholders.

Questions for Case 6.5

1. What was the nature of the problems that GDOT was trying to solve with data science?

2. What type of data do you think was used for the analytics?

3. What were the data science metrics developed in this pilot project? Can you think of other metrics that can be used in this context?

Source: Teradata Case Study. “Georgia DOT Variable Speed Limit Analytics Help Solve Traffic Congestion.” https:// www.teradata. com/Resources/Case-Studies/Georgia-DOT-Variable-Speed- Limit-Analytics (accessed July 2018); “Georgia DOT Variable Speed Limits.” www.dot.ga.gov/ DriveSmart/SafetyOperation/ Pages/VSL.aspx (accessed August 2018).Used with permission from Teradata.

In the next section, we discuss a very popular variation of deep MLP architecture called convolutional neural network (CNN) specifically designed for computer vision applications (e.g., image recognition, handwritten text processing).

u SECTION 6.6 REVIEW QUESTIONS

1. What is meant by “deep” in deep neural networks? Compare deep neural networks to shallow neural networks.

2. What is GPU? How does it relate to deep neural networks? 3. How does a feedforward multilayer perceptron-type deep network work?

Application Case 6.5 (Continued)

Chapter 6 • Deep Learning and Cognitive Computing 349

4. Comment on the impact of random weights in developing deep MLP. 5. Which strategy is better: more hidden layers versus more neurons?

6.7 CONVOLUTIONAL NEURAL NETWORKS

CNNs (LeCun et al., 1989) are among the most popular types of deep learning methods. CNNs are in essence variations of the deep MLP architecture, initially designed for com- puter vision applications (e.g., image processing, video processing, text recognition) but are also applicable to nonimage data sets.

The main characteristic of the convolutional networks is having at least one layer in- volving a convolution weight function instead of general matrix multiplication. Figure 6.21 illustrates a typical convolutional unit.

Convolution, typically shown by the symbol, is a linear operation that essentially aims at extracting simple patterns from sophisticated data patterns. For instance, in pro- cessing an image containing several objects and colors, convolution functions can extract simple patterns like the existence of horizontal or vertical lines or edges in different parts of the picture. We discuss convolution functions in more detail in the next section.

A layer containing a convolution function in a CNN is called a convolution layer. This layer is often followed by a pooling (a.k.a. subsampling) layer. Pooling layers are in charge of consolidating the large tensors to one with a smaller size and reducing the number of model parameters while keeping their important features. Different types of pooling layers are also discussed in the following sections.

Convolution Function

In the description of MLP networks, it was said that the weight function is generally a matrix manipulation function that multiplies the weight vector into the input vector to produce the output vector in each layer. Having a very large input vector/tensor, which is the case in most deep learning applications, we need a large number of weight pa- rameters so that each single input to each neuron could be assigned a single weight pa- rameter. For instance, in an image-processing task using a neural network for images of size 150 * 150 pixels, each input matrix will contain 22,500 (i.e., 150 times 150) integers, each of which should be assigned its own weight parameter per each neuron it goes into throughout the network. Therefore, having even only a single layer requires thousands of weight parameters to be defined and trained. As one might guess, this fact would dramatically increase the required time and processing power to train a network, since in each training iteration, all of those weight parameters have to be updated by the SGD algorithm. The solution to this problem is the convolution function.

S fp

Input

w b

z n a

Convolutional Unit a 5 f(w p 1 b)

FIGURE 6.21 Typical Convolutional Network Unit.

350 Part II • Predictive Analytics/Machine Learning

The convolution function can be thought of as a trick to address the issue defined in the previous paragraph. The trick is called parameter sharing, which in addition to computational efficiency provides additional benefits. Specifically, in a convolution layer, instead of having a weight for each input, there is a set of weights referred to as the convolution kernel or filter, which is shared between inputs and moves around the input matrix to produce the outputs. The kernel is typically represented as a small matrix of size Wr * c; for a given input matrix V , then, the convolution function can be stated as:

zi, j = a r

k = 1 a c

l = 1 wk,l vi + k - 1, j + l - 1

For example, assume that the input matrix to a layer and the convolution kernel is

V = £ 1 0 1

1 1 0

1 1 0

0 1 1

1 1 1

0 0 1

§ W = c0 1 1 1

d

Figure 6.22 illustrates how the convolution output can be computed. As shown, each element of the output matrix results from summing up the one-by-one point mul- tiplications of the kernel elements into a corresponding r * c (in this example, 2 * 2 because the kernel is 2 * 2) subset of the input matrix elements. So, in the example shown, the element at the second column of the first row of the output matrix is in fact 0(0) + 1(1) + 1(1) + 1(0) = 2.

It can be seen that the magnitude of each element in the output matrix directly depends on how the matched kernel (with the 2 * 2 matrix) and the input matrix are involved in calculation of that element. For example, the element at the fourth column of the first row of the output matrix is the result of convoluting the kernel by a part of the input matrix, which is exactly the same as the kernel (shown in Figure 6.23). This suggests that by applying the convolution operation, we actually are converting the input matrix into an output in which the parts that have a particular feature (reflected by the kernel) are placed in the square box.

This characteristic of convolution functions is especially useful in practical image- processing applications. For instance, if the input matrix represents the pixels of an image,

1

1

1 1

0

1

1

0

0

0

1

0

1

1

0

1

1

1 0 1

1 1

Kernel (W)

Input matrix (V) Output matrix (Z)

2

3

2

1

1

1

3

1

3

2

FIGURE 6.22 Convolution of a 2 : 2 Kernel by a 3 : 6 Input Matrix.

1

1

1 1

0

1

1

0

0

0

1

0

1

1

0

1

1

1

FIGURE 6.23 The Output of Convolution Operation Is Maximized When the Kernel Exactly Matches the Part of Input Matrix That Is Being Convoluted by.

Chapter 6 • Deep Learning and Cognitive Computing 351

a particular kernel representing a specific shape (e.g., a diagonal line) may be convoluted into that image to extract parts of the image involving that specific shape. Figure 6.24, for example, shows the result of applying a 3 * 3 horizontal line kernel to a 15 * 15 image of a square.

Clearly, the horizontal kernel produces an output in which the location of horizon- tal lines (as a feature) in the original input image is identified.

Convolution using a kernel of size r * c will reduce the number of rows and columns in the output by r - 1 and c - 1, respectively. In the recent case, for exam- ple, using a 2 * 2 kernel for convolution, the output matrix has 1 row and 1 column less than the input matrix. To prevent this change of size, we can pad the outside of the input matrix with zeros before convolving, that is, to add r - 1 rows and c - 1 columns of zeros to the input matrix. On the other hand, if we want the output matrix to be even smaller, we can have the kernel to take larger strides, or kernel movements. Normally, the kernel is moved one step at a time (i.e., stride = 1) when performing the convolution. By increasing this stride to 2, the size of the output matrix is reduced by a factor of 2.

Although the main benefit of employing convolution in the deep networks is pa- rameter sharing, which effectively reduces the required time and processing power to train the network by reducing the number of weight parameters, it involves some other benefits as well. A convolution layer in a network will have a property called equivari- ance for translation purposes (Goodfellow et al., 2016). It simply means that any changes in the input will lead to a change in the output in the same way. For instance, moving an object in the input image by 10 pixels in a particular direction will lead to moving its representation in the output image by 10 pixels in the same direction. Apart from image- processing applications, this feature is especially useful for analyzing time-series data using convolutional networks where convolution can produce a kind of timeline that shows when each feature appears in the input.

It should be noted that in almost all of the practical applications of convolutional networks, many convolution operations are used in parallel to extract various kinds of features from the data, because a single feature is hardly enough to fully describe the inputs for the classification or recognition purposes. Also, as noted before, in most real- world applications, we have to represent the inputs as multi-dimensional tensors. For instance, in the processing of color images as opposed to gray scale pictures, instead of having 2D tensors (i.e., matrices) that represent the color of pixels (i.e., black or white), one will have to use 3D tensors because each pixel should be defined using the intensity of red, blue, and green colors.

1

1

2

3

2

Horizontal Kernel

Input Image Output Image

3

1 2 3 4 5 6 7 8 9 101112131415 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 2 3 4 5 6 7 8 5 9

10 11 12 13 14 15

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15

FIGURE 6.24 Example of Using Convolution for Extracting Features (Horizontal Lines in This Example) from Images.

352 Part II • Predictive Analytics/Machine Learning

Pooling

Most of the times, a convolution layer is followed by another layer known as the pooling (a.k.a. subsampling) layer. The purpose of a pooling layer is to consolidate elements in the input matrix to produce a smaller output matrix while maintaining the important fea- tures. Normally, a pooling function involves an r * c consolidation window (similar to a kernel in the convolution function) that moves around the input matrix and in each move calculates some summary statistics of the elements involved in the consolidation window so that it can be put in the output image. For example, a particular type of pooling func- tion called average pooling takes the average of the input matrix elements involved in the consolidation window and puts that average value as an element of the output matrix in the corresponding location. Similarly, the max pooling function (Zhou et al.) takes the maximum of the values in the window as the output element. Unlike convolution, for the pooling function, given the size of the consolidation window (i.e., r and c), stride should be carefully selected so that there would be no overlaps in the consolidations. The pooling operation using an r * c consolidation window reduces the number of rows and columns of the input matrix by a factor of r and c, respectively. For example, using a 3 * 3 consolidation window, a 15 * 15 matrix will be consolidated to a 5 * 5 matrix.

Pooling, in addition to reducing the number of parameters, is especially useful in the image-processing applications of deep learning in which the critical task is to determine whether a feature (e.g., a particular animal) is present in an image while the exact spatial lo- cation of the same in the picture is not important. However, if the location of features is im- portant in a particular context, applying a pooling function could potentially be misleading.

You can think of pooling as an operation that summarizes large inputs whose fea- tures are already extracted by the convolution layer and shows us just the important parts (i.e., features) in each small neighborhood in the input space. For instance, in the case of the image-processing example shown in Figure 6.24, if we place a max pooling layer after the convolution layer using a 3 * 3 consolidation window, the output will be like what is shown in Figure 6.25. As shown, the 15 * 15 already convoluted image is consolidated in a 5 * 5 image while the main features (i.e., horizontal lines) are maintained therein.

Sometimes pooling is used just to modify the size of matrices coming from the pre- vious layer and convert them to a specified size required by the following layer in the network.

There are various types of pooling operations such as max pooling, average pool- ing, the L2 norm of a rectangular neighborhood, and weighted average pooling. The

Horizontal Convoluted Square

Output

1 2 3 4 5 6 7 8 9 101112131415

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15

Max Pooling

FIGURE 6.25 An Example of Applying Max Pooling on an Output Image to Reduce Its Size.

Chapter 6 • Deep Learning and Cognitive Computing 353

choice of proper pooling operation as well as the decision to include a pooling layer in the network at all depends highly on the context and properties of the problem that the network is solving. There are some guidelines in the literature to help the network de- signers in making such decisions (Boureau et al., 2011; Boureau, Ponce, and LeCun, 2010; Scherer, Müller, and Behnke, 2010).

Image Processing Using Convolutional Networks

Real applications of deep learning in general and CNNs in particular highly depend on the availability of large, annotated data sets. Theoretically, CNNs can be applied to many practical problems, and today there are many large and feature-rich databases for such applications available. Nevertheless, the biggest challenge is that in supervised learning applications, one needs an already annotated (i.e., labeled) data set to train the model be- fore we can use it for prediction/identification of other unknown cases. Whereas extract- ing features of data sets using CNN layers is an unsupervised task, the extracted features will not be of much use without having labeled cases to develop a classification network in a supervised learning fashion. That is why image classification networks traditionally involve two pipelines: visual feature extraction and image classification.

ImageNet (http://www.image-net.org) is an ongoing research project that pro- vides researchers with a large database of images, each linked to a set of synonym words (known as synset) from WordNet (a word hierarchy database). Each synset represents a particular concept in the WordNet. Currently, WordNet includes more than 100,000 synsets, each of which is supposed to be illustrated by an average of 1,000 images in the ImageNet. ImageNet is a huge database for developing image processing–type deep networks. It contains more than 15 million labeled images in 22,000 categories. Because of its sheer size and proper categorization, ImageNet is by far the most widely used benchmarking data set to assess the efficiency and accuracy of deep networks designed by deep learning researchers.

One of the first convolutional networks designed for image classification using the ImageNet data set was AlexNet (Krizhevsky, Sutskever, and Hinton, 2012). It was com- posed of five convolution layers followed by three fully connected (a.k.a. dense) layers (see Figure 6.26 for a schematic representation of AlexNet). One of the contributions of this relatively simple architecture that made its training remarkably faster and com- putationally efficient was the use of rectified linear unit (ReLu) transfer functions in the convolution layers instead of the traditional sigmoid functions. By doing so, the designers

3

C1

C2

96

55

55

5

5

27

3 13 13

13 13

13

3

3

3

13 27 384 384 256

4,096 4,096

1,000

3

256

C3 C4 C5

FC6 FC7

FC8

FIGURE 6.26 Architecture of AlexNet, a Convolutional Network for Image Classification.

354 Part II • Predictive Analytics/Machine Learning

addressed the issue called the vanishing gradient problem caused by very small deriva- tives of sigmoid functions in some regions of the images. The other important contribu- tion of this network that has a dramatic role in improving the efficiency of deep networks was the introduction of the concept of dropout layers to the CNNs as a regularization technique to reduce overfitting. A dropout layer typically comes after the fully connected layers and applies a random probability to the neurons to switch off some of them and make the network sparser.

In the recent years, in addition to a large number of data scientists who showcase their deep learning capabilities, a number of well-known industry-leading companies such as Microsoft, Google, and Facebook have participated in the annual ImageNet Large Scale Visual Recognition Challenge (ILSVRC). The goal in the ILSVRC classification task is to design and train networks that are capable of classifying 1.2 million input images into one of the 1,000 image categories. For instance, GoogLeNet (a.k.a. Inception), a deep convolutional network architecture designed by Google researchers, was the win- ning architecture of ILSVRC 2014 with a 22-layer network and only a 6.66 percent clas- sification error rate, only slightly 15.1%2 worse than the human-level classification error (Russakovsky et al., 2015). The main contribution of the GoogLeNet architecture was to introduce a module called Inception. The idea of Inception is that because one would have no idea of the size of convolution kernel that would perform best on a particular data set, it is better to include multiple convolutions and let the network decide which one to use. Therefore, as shown in Figure 6.27, in each convolution layer, the data com- ing from the previous layer is passed through multiple types of convolution and the out- puts are concatenated before going to the next layer. Such architecture allows the model to take into account both local features via smaller convolutions and high abstracted features via larger ones.

Google recently launched a new service, Google Lens, that uses deep learning arti- ficial neural network algorithms (along with other AI techniques) to deliver information about the images captured by users from their nearby objects. This involves identifying the objects, products, plants, animals, and locations and providing information about them on the Internet. Some other features of this service are the capability of saving

Filter Concatenation

3 3 3 Convolutions

5 3 5 Convolutions

1 3 1 Convolutions

1 3 1 Convolutions

1 3 1 Convolutions

3 3 3 Max Pooling

131 Convolutions

Previous Layer

FIGURE 6.27 Conceptual Representation of the Inception Feature in GoogLeNet.

Chapter 6 • Deep Learning and Cognitive Computing 355

contact information from a business card image on the phone, identifying type of plants and breed of animals, identifying books and movies from their cover photos, and provid- ing information (e.g., stores, theaters, shopping, reservations) about them. Figure 6.28 shows two examples of using the Google Lens app on an Android mobile device.

Even though later more accurate networks have been developed (e.g., He, Zhang, Ren, & Sun, 2015) in terms of efficiency and processing requirements (i.e., smaller num- ber of layers and parameters), GoogLeNet is considered to be one of the best architec- tures to date. Apart from AlexNet and GoogLeNet, several other convolutional network architectures such as Residual Networks (ResNet), VGGNet, and Xception have been developed and contributed to the image-processing area, all relying on the ImageNet database.

In a May 2018 effort to address the labor-intensive task of labeling images on a large scale, Facebook published a weakly supervised training image recognition deep learning project (Mahajan et al., 2018). This project used hashtags made by the users on the im- ages posted on Instagram as labels and trained a deep learning image recognition model based on that. The model was trained using 3.5 billion Instagram images labeled with around 17,000 hashtags using 336 GPUs working in parallel; the training procedure took a few weeks to be accomplished. A preliminary version of the model (trained using only 1 billion images and 1,500 hashtags) was then tested on the ImageNet benchmark data set and is reported to have outperformed the state-of-the-art models in terms of accuracy by more than 2 percent. This big achievement by Facebook surely will open doors to a new world of image processing using deep learning since it can dramatically increase the size of available image data sets that are labeled for training purposes.

Use of deep learning and advanced analytics methods to classify images has evolved into the recognition of human faces and has become a very popular application for a variety of purposes. It is discussed in Application Case 6.6.

FIGURE 6.28 Two Examples of Using the Google Lens, a Service Based on Convolutional Deep Networks for Image Recognition. Source: ©2018 Google LLC, used with permission. Google and the

Google logo are registered trademarks of Google LLC.

356 Part II • Predictive Analytics/Machine Learning

Face recognition, although seemingly similar to image recognition, is a much more complicated undertaking. The goal of face recognition is to identify the individ- ual as opposed to the class it belongs to (human), and this identification task needs to be performed on a nonstatic (i.e., moving person) 3D environment. Face recognition has been an active research field in AI for many decades with limited success until recently. Thanks to the new generation of algorithms (i.e., deep learning) coupled with large data sets and computa- tional power, face recognition technology is starting to make a significant impact on real-world applications. From security to marketing, face recognition and the variety of applications/use cases of this technology are increasing at an astounding pace.

Some of the premier examples of face recogni- tion (both in advancements in technology and in the creative use of the technology perspectives) come from China. Today in China, face recognition is a very hot topic both from business development and from application development perspectives. Face recognition has become a fruitful ecosystem with hundreds of start-ups in China. In personal and/or business settings, people in China are widely using and relying on devices whose security is based on automatic recognition of their faces.

As perhaps the largest scale practical applica- tion case of deep learning and face recognition in the world today, the Chinese government recently started a project known as “Sharp Eyes” that aims at establishing a nationwide surveillance system based on face recognition. The project plans to integrate security cameras already installed in public places with private cameras on buildings and to utilize AI and deep learning to analyze the videos from those cameras. With millions of cameras and billions of lines of code, China is building a high-tech authori- tarian future. With this system, cameras in some cit- ies can scan train and bus stations as well as airports to identify and catch China’s most wanted suspected criminals. Billboard-size displays can show the faces of jaywalkers and list the names and pictures of peo- ple who do not pay their debts. Facial recognition scanners guard the entrances to housing complexes.

An interesting example of this surveillance system is the “shame game” (Mozur, 2018). An

intersection south of Changhong Bridge in the city of Xiangyang previously was a nightmare. Cars drove fast, and jaywalkers darted into the street. Then, in the summer of 2017, the police put up cameras linked to facial recognition technology and a big out- door screen. Photos of lawbreakers were displayed alongside their names and government identifica- tion numbers. People were initially excited to see their faces on the screen until propaganda outlets told them that this was a form of punishment. Using this, citizens not only became a subject of this shame game but also were assigned negative citizenship points. Conversely, on the positive side, if people are caught on camera showing good behavior, like pick- ing up a piece of trash from the road and putting it into a trash can or helping an elderly person cross an intersection, they get positive citizenship points that can be used for a variety of small awards.

China already has an estimated 200 million sur- veillance cameras—four times as many as the United States. The system is mainly intended to be used for tracking suspects, spotting suspicious behavior, and predicting crimes. For instance, to find a criminal, the image of a suspect can be uploaded to the system, matching it against millions of faces recognized from videos of millions of active security cameras across the country. This can find individuals with a high degree of similarity. The system also is merged with a huge database of information on medical records, travel bookings, online purchases, and even social media activities of every citizen and can monitor practically everyone in the country (with 1.4 billion people), tracking where they are and what they are doing each moment (Denyer, 2018). Going beyond narrowly defined security purposes, the govern- ment expects Sharp Eyes to ultimately assign every individual in the country a “social credit score” that specifies to what extent she or he is trustworthy.

While such an unrestricted application of deep learning (i.e., spying on citizens) is against the privacy and ethical norms and regulations of many western countries, including the United States, it is becoming a common practice in countries with less restrictive privacy laws and concerns as in China. Even western countries have begun to plan on employing similar technologies in limited scales only for security and

Application Case 6.6 From Image Recognition to Face Recognition

Chapter 6 • Deep Learning and Cognitive Computing 357

Text Processing Using Convolutional Networks

In addition to image processing, which was in fact the main reason for the popularity and development of convolutional networks, they have been shown to be useful in some large-scale text mining tasks as well. Especially since 2013, when Google published its word2vec project (Mikolov et al., 2013; Mikolov, Sutskever, Chen, Corrado, and Dean, 2013), the applications of deep learning for text mining have increased remarkably.

Word2vec is a two-layer neural network that gets a large text corpus as the input and converts each word in the corpus to a numeric vector of any given size (typically ranging from 100 to 1,000) with very interesting features. Although word2vec itself is not a deep learning algorithm, its outputs (word vectors also known as word embeddings) already have been widely used in many deep learning research and commercial projects as inputs.

One of the most interesting properties of word vectors created by the word2vec algorithm is maintaining the words’ relative associations. For example, vector operations

vector (‘King’) - vector (‘Man’) + vector (‘Woman’)

and

vector (‘London’) - vector (‘England’) + vector (‘France’)

will result in a vector very close to vector (‘Queen’) and vector (‘Paris’), respectively. Figure 6.29 shows a simple vector representation of the first example in a two-dimensional vector space.

Moreover, the vectors are specified in such a way that those of a similar context are placed very close to each other in the n-dimensional vector space. For instance, in the word2vec model pretrained by Google using a corpus including about 100 billion words (taken from Google News), the closest vectors to the vector (‘Sweden’) in terms of cosine distance, as shown in Table 6.2, identify European country names near the Scandinavian region, the same region in which Sweden is located.

Additionally, since word2vec takes into account the contexts in which a word has been used and the frequency of using it in each context in guessing the meaning of the word, it enables us to represent each term with its semantic context instead of just the syntactic/symbolic term itself. As a result, word2vec addresses several word variation issues that used to be problematic in traditional text mining activities. In other words,

crime prevention purposes. The FBI’s Next Generation Identification System, for instance, is a lawful appli- cation of facial recognition and deep learning that compares images from crime scenes with a national database of mug shots to identify potential suspects.

Questions for Case 6.6

1. What are the technical challenges in face recognition?

2. Beyond security and surveillance purposes, where else do you think face recognition can be used?

3. What are the foreseeable social and cultural problems with developing and using face recog- nition technology?

Sources: Mozur, P. (2018, June 8). “Inside China’s Dystopian Dreams: A.I., Shame and Lots of Cameras.” The New York Times. https://www.nytimes.com/2018/07/08/business/china- surveillance-technology.html; Denyer, S. (2018, January). “Beijing Bets on Facial Recognition in a Big Drive for Total Surveillance.” The Washington Post. https://www.washing- tonpost.com/news/world/wp/2018/01/07/feature/in- china-facial-recognition-is-sharp-end-of-a-drive-for-total- surveillance/?noredirect=on&utm_term=.e73091681b31.

358 Part II • Predictive Analytics/Machine Learning

word2vec is able to handle and correctly represent words including typos, abbreviations, and informal conversations. For instance, the words Frnce, Franse, and Frans would all get roughly the same word embeddings as their original counterpart France. Word embeddings are also able to determine other interesting types of associations such as distinction of entities (e.g., vector3‘human’4 - vector3‘animal’4~vector3‘ethics’4) or geopolitical associations (e.g., vector3‘Iraq’4 - vector3‘violence’4~vector3‘Jordan’4).

By providing such a meaningful representation of textual data, in recent years, word2vec has driven many deep learning–based text mining projects in a wide range of contexts (e.g., medical, computer science, social media, marketing), and various types of deep networks have been applied to the word embeddings created by this algorithm to accomplish different objectives. Particularly, a large group of studies had developed convolutional networks applied to the word embeddings with the aim of relation extrac- tion from textual data sets. Relation extraction is one of the subtasks of natural language processing (NLP) that focuses on determining whether two or more named entities rec- ognized in the text form specific relationships (e.g., “A causes B”; “B is caused by A”). For instance, Zeng et al. (2014) developed a deep convolutional network (see Figure 6.30) to classify relations between specified entities in sentences. To this end, these researchers

King

Queen

ManKing-Man

Woman

FIGURE 6.29 Typical Vector Representation of Word Embeddings in a Two-Dimensional Space

TABLE 6.2 Example of the word2vec Project Indicating the Closest Word Vectors to the Word “Sweden”

Word Cosine Distance

Norway 0.760124

Denmark 0.715460

Finland 0.620022

Switzerland 0.588132

Belgium 0.585635

Netherlands 0.574631

Iceland 0.562368

Estonia 0.547621

Slovenia 0.531408

Chapter 6 • Deep Learning and Cognitive Computing 359

used a matrix format to represent each sentence. Each column of the input matrices is in fact the word embedding (i.e., vector) associated with one of the words involved in the sentence. Zeng et al. then used a convolutional network, shown in the right box in Figure 6.30, to automatically learn the sentence-level features and concatenate those features (i.e., the output vector of the CNN) with some basic lexical features (e.g., the order of the two words of interest within the sentence and the left and right tokens for each of them). The concatenated feature vector then is fed into a classification layer with a softmax transfer function, which determines the type of relationship between the two words of interest among multiple predefined types. The softmax transfer function is the most com- mon type of function to be used for classification layers, especially when the number of classes is more than two. For classification problems with only two outcome categories, log-sigmoid transfer functions are also very popular. The proposed approach by Zeng et al. was shown to correctly classify the relation between the marked terms in sentences of a sample data set with an 82.7 percent accuracy.

In a similar study, Nguyen and Grishman (2015) used a four-layer convolutional net- work with multiple kernel sizes in each convolution layer fed by the real-valued vectors of words included in sentences to classify the type of relationship between the two marked words in each sentence. In the input matrix, each row was the word embedding associated with a word in the same sequence in the sentence as the row number. In addition, these researchers included two more columns to the input matrices to represent the relative posi- tion of each word (either positive or negative) with regard to each of the marked terms. The automatically extracted features then were passed through a classification layer with soft- max function for the type of relationship to be determined. Nguyen and Grishman trained their model using 8,000 annotated examples (with 19 predefined classes of relationships) and tested the trained model on a set of 2,717 validation data sets and achieved a classifica- tion accuracy of 61.32 percent (i.e., more than 11 times better performance than guessing).

Such text mining approaches using convolutional deep networks can be extended to various practical contexts. Again, the big challenge here, just as in image processing, is lack of sufficient large annotated data sets for supervised training of deep networks. A distant supervision method of training has been proposed (Mintz et al., 2009) to ad- dress this challenge. It suggests that large amounts of training data can be produced by aligning knowledge base (KB) facts with texts. In fact, this approach is based on the assumption that if a particular type of relation exists between an entity pair (e.g., “A” is a component of “B”) in the KB, then every text document containing the mention of the

tanh(W2x )

Word Representation

[People] have been moving back into [downtown]

Window Processing

Sentence level Features

Convolution

Lexical level features

Feature Extraction

Output

W3X

W1

WF

PF

Max over times

Sentence level features

FIGURE 6.30 CNN Architecture for Relation Extraction Task in Text Mining.

360 Part II • Predictive Analytics/Machine Learning

entity pair would express that relation. However, since this assumption was not very realistic, Riedel, Yao, and McCallum (2010) later relaxed it by modeling the problem as a multi-instance learning problem. They suggest assigning labels to a bag of instances rather than a single instance that can reduce the noise of the distant supervision method and create more realistic labeled training data sets (Kumar, 2017).

u SECTION 6.7 REVIEW QUESTIONS

1. What is CNN? 2. For what type of applications can CNN be used? 3. What is convolution function in CNN and how does it work? 4. What is pooling in CNN? How does it work? 5. What is ImageNet and how does it relate to deep learning? 6. What is the significance of AlexNet? Draw and describe its architecture. 7. What is GoogLeNet? How does it work? 8. How does CNN process text? What are word embeddings, and how do they work? 9. What is word2vec, and what does it add to traditional text mining?

6.8 RECURRENT NETWORKS AND LONG SHORT-TERM MEMORY NETWORKS

Human thinking and understanding to a great extent relies on context. It is crucial for us, for example, to know that a particular speaker uses very sarcastic language (based on his previous speeches) to fully catch all the jokes that he makes. Or to understand the real meaning of the word fall (i.e., either the season or to collapse) in the sentence “It is a nice day of fall” without knowledge about the other words in the surrounding sentences would only be guessing, not necessarily understanding. Knowledge of context is typically formed based on observing events that happened in the past. In fact, human thoughts are persistent, and we use every piece of information we previously acquired about an event in the process of analyzing it rather than throwing away our past knowledge and thinking from scratch every time we face similar events or situations. Hence, there seems to be a recurrence in the way humans process information.

While deep MLP and convolutional networks are specialized for processing a static grid of values like an image or a matrix of word embeddings, sometimes the sequence of input values is also important to the operation of the network to accomplish a given task and hence should be taken into account. Another popular type of neural networks is recurrent neural network (RNN) (Rumelhart et al., 1986), which is specifically de- signed to process sequential inputs. An RNN basically models a dynamic system where (at least in one of its hidden neurons) the state of the system (i.e., output of a hidden neuron) at each time point t depends on both the inputs to the system at that time and its state at the previous time point t - 1. In other words, RNNs are the type of neural networks that have memory and that apply that memory to determine their future out- puts. For instance, in designing a neural network to play chess, it is important to take into account several previous moves while training the network, because a wrong move by a player can lead to the eventual loss of the game in the subsequent 10–15 plays. Also, to understand the real meaning of a sentence in an essay, sometimes we need to rely on the information portrayed in the previous several sentences or paragraphs. That is, for a true understanding, we need the context built sequentially and collectively over time. Therefore, it is crucial to consider a memory element for the neural network that takes into account the effect of prior moves (in the chess example) and prior sentences and paragraphs (in the essay example) to determine the best output. This memory portrays and creates the context required for the learning and understanding.

Chapter 6 • Deep Learning and Cognitive Computing 361

In static networks like MLP-type CNNs, we are trying to find some functions (i.e., network weights and biases) that map the inputs to some outputs that are as close as possible to the actual target. In dynamic networks like RNNs, on the other hand, both inputs and outputs are sequences (patterns). Therefore, a dynamic network is a dynamic system rather than a function because its output depends not only on the input but also on the previous outputs. Most of the RNNs use the following general equation to define the values of their hidden units (Goodfellow et al., 2016).

a(t) = f (a(t - 1), p(t),u)

In this equation, a(t) represents the state of the system at time t, and p(t) and u rep- resent the input to the unit at time t and the parameters, respectively. Applying the same general equation for calculating the state of system at time t - 1, we will have:

a(t - 1) = f (a(t - 2), p(t - 1),u)

In other words:

a(t) = f ( f (a(t - 2), p(t - 1),u), p(t),u)

And this equation can be extended multiple times for any given sequence length. Graphically, a recurrent unit in a network can be depicted in a circuit diagram like the one shown in Figure 6.31. In this figure, D represents the tap delay lines, or simply the delay element of the network that, at each time point t, contains a1t2, the previous output value of the unit. Sometimes instead of just one value, we store several previous output values in D to account for the effect of all of them. Also iw and lw represent the weight vectors applied to the input and the delay, respectively.

Technically speaking, any network with feedback can actually be called a deep net- work, because even with a single layer, the loop created by the feedback can be thought of as a static MLP-type network with many layers (see Figure 6.32 for a graphical illustra- tion of this structure). However, in practice, each recurrent neural network would involve dozens of layers, each with feedback to itself, or even to the previous layers, which makes a recurrent neural network even deeper and more complicated.

Because of the feedbacks, computation of gradients in the recurrent neural net- works would be somewhat different from the general backpropagation algorithm used

X

XIw

Input Recurrent Neuron a(t) 5 f(iw.p(t) 1 lw.a(t) 1 b)

iw b

p(t) n(t) a(t)

D

fS

FIGURE 6.31 Typical Recurrent Unit.

362 Part II • Predictive Analytics/Machine Learning

for the static MLP networks. There are two alternative approaches for computing the gradients in the RNNs, namely, real-time recurrent learning (RTRL) and backpropagation through time (BTT), whose explanation is beyond the scope of this chapter. Nevertheless, the general purpose remains the same; once the gradients have been computed, the same procedures are applied to optimize the learning of the network parameters.

The LSTM networks (Hochreiter & Schmidhuber, 1997) are variations of recurrent neural networks that today are known as the most effective sequence modeling tech- nique and are the base of many practical applications. In a dynamic network, the weights are called the long-term memory while the feedbacks role is the short-term memory.

In essence, only the short-term memory (i.e., feedbacks; previous events) provides a network with the context. In a typical RNN, the information in the short-term memory is continuously replaced as new information is fed back into the network over time. That is why RNNs perform well when the gap between the relevant information and the place that is needed is small. For instance, for predicting the last word in the sentence “The referee blew his whistle,” we just need to know a few words back (i.e., the referee) to correctly predict. Since in this case the gap between the relevant information (i.e., the ref- eree) and where it is needed (i.e., to predict whistle) is small, an RNN network can easily perform this learning and prediction task.

However, sometimes the relevant information required to perform a task is far away from where it is needed (i.e., the gap is large). Therefore, it is quite likely that it would have already been replaced by other information in the short-term memory by the time it is needed for the creation of the proper context. For instance, to predict the last word in “I went to a carwash yesterday. It cost $5 to wash my car,” there is a relatively larger gap between the relevant information (i.e., carwash) and where it is needed. Sometimes we may even need to refer to the previous paragraphs to reach the relevant information for predicting the true meaning of a word. In such cases, RNNs usually do not perform well since they cannot keep the information in their short-term memory for a long enough time. Fortunately, LSTM networks do not have such a shortcoming. The term long short- term memory network then refers to a network in which we are trying to remember what happened in the past (i.e., feedbacks; previous outputs of the layers) for a long enough time so that it can be used/leveraged in accomplishing the task when needed.

From an architectural viewpoint, the memory concept (i.e., remembering “what happened in the past”) is incorporated in LSTM networks by incorporating four addi- tional layers into the typical recurrent network architecture: three gate layers, namely input gate, forget (a.k.a. feedback) gate, and output gate, and an additional layer called Constant Error Carousel (CEC), also known as the state unit that integrates those gates and interacts them with the other layers. Each gate is nothing but a layer with two inputs, one from the network input and the other a feedback from the final output of the whole network. The gates involve log-sigmoid transfer functions. Therefore, their outputs will be between 0 and 1 and describe how much of each component (either input, feedback, or output) should be let through the network. Also, CEC is a layer that falls between the

a(...)

x(...)

a(...)a(t21)

x(t11) x(t12) x(t13)x(t)

a(t11) a(t12)a(t) f f f f f

FIGURE 6.32 Unfolded View of a Typical Recurrent Network.

Chapter 6 • Deep Learning and Cognitive Computing 363

input and the output layers in a recurrent network architecture and applies the gates out- puts to make the short-term memory long.

To have a long short-term memory means that we want to keep the effect of previ- ous outputs for a longer time. However, we typically do not want to indiscriminately re- member everything that has happened in the past. Therefore, gating provides us with the capability of remembering prior outputs selectively. The input gate will allow selective inputs to the CEC; the forget gate will clear the CEC from the unwanted previous feed- backs; and the output gate will allow selective outputs from the CEC. Figure 6.33 shows a simple depiction of a typical LSTM architecture.

In summary, the gates in the LSTM are in charge of controlling the flow of informa- tion through the network and dynamically change the time scale of integration based on the input sequence. As a result, LSTM networks are able to learn long-term dependencies among the sequence of inputs more easily than the regular RNNs.

Application Case 6.7 illustrates the use of text processing in the context of under- standing customer opinions and sentiments toward innovatively designing and develop- ing new and improved products and services.

CECx

1 x Input Layer

Input Gate

Forget (feedback) Gate

Output Gate

Output Layer

a7(t)

a7(t)

a7(t)

a5(t) a6(t) a7(t)

a1(t)

a2(t)

a3(t)

x

p(t)

p(t)

p(t)

a7(t) a4(t)

p(t)

FIGURE 6.33 Typical Long Short-Term Memory (LSTM) Network Architecture.

Analyzing product and customer behavior provides valuable insights into what consumers want, how they interact with products, and where they encoun- ter usability issues. These insights can lead to new feature designs and development or even new products.

Understanding customer sentiment and know- ing what consumers truly think about products or a brand are traditional pain points. Customer jour- ney analytics provides insights into these areas, yet these solutions are not all designed to integrate vital sources of unstructured data such as call center

Application Case 6.7 Deliver Innovation by Understanding Customer Sentiments

(Continued )