Data Mining

\ ( (

P A N G . N I N G T A N M i c h i g a n S t a t e U n i v e r s i t y

M I C H A E L S T E I N B A C H U n i v e r s i t y o f M i n n e s o t a

V I P I N K U M A R U n i v e r s i t y o f M i n n e s o t a a n d A r m y H i g h P e r f o r m a n c e

C o m p u t i n g R e s e a r c h C e n t e r

+f.f_l crf.rfh. .W if f

aqtY 6l$

t . T . R . C .

i'&'ufe61ttt1/. Y \ t.\ $t,/,1'

n,5 \ . 7 \ V ' 4 8 !

Boston San Francisco NewYork

London Toronto Sydney Tokyo Singapore Madrid

MexicoCity Munich Paris CapeTown HongKong Montreal

G.R r+6,q

If you purchased this book within the United States or Canada you should be aware that it has been wrongfirlly imported without the approval of the Publishel or the Author.

T3 Loo 6

- {)gq* 3 AcquisitionsEditor Matt Goldstein

ProjectEditor Katherine Harutunian Production Supervisor Marilyn Lloyd Production Services Paul C. Anagnostopoulos of Windfall Software Marketing Manager Michelle Brown Copyeditor Kathy Smith Proofreader IenniferMcClain Technicallllustration GeorgeNichols Cover Design Supervisor Joyce Cosentino Wells Cover Design Night & Day Design Cover Image @ 2005 Rob Casey/Brand X pictures hepress and Manufacturing Caroline Fell Printer HamiltonPrinting

Access the latest information about Addison-Wesley titles from our iWorld Wide Web site: http : //www. aw-bc.com/computing

Many of the designations used by manufacturers and sellers to distiriguish their products are claimed as trademarks. where those designations appear in this book, and Addison- Wesley was aware of a trademark claim, the designations have been printed in initial caps or all caps.

The programs and applications presented in this book have been incl,[rded for their instructional value. They have been tested with care, but are not guatanteed for any particular purpose. The publisher does not offer any warranties or representations, nor does it accept any liabilities with respect to the programs or applications.

Copyright @ 2006 by Pearson Education, Inc.

For information on obtaining permission for use of material in this work, please submit a written request to Pearson Education, Inc., Rights and Contract Department, 75 Arlington Street, Suite 300, Boston, MA02II6 or fax your request to (617) g4g-j047.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or any other media embodiments now known or hereafter to become known, without the prior written permission of the publisher. printed in the united States of America.

lsBN 0-321-42052-7

2 3 4 5 67 8 9 10-HAM-O8 07 06

our famili,es

Preface

Advances in data generation and collection are producing data sets of mas- sive size in commerce and a variety of scientific disciplines. Data warehouses store details of the sales and operations of businesses, Earth-orbiting satellites beam high-resolution images and sensor data back to Earth, and genomics ex- periments generate sequence, structural, and functional data for an increasing number of organisms. The ease with which data can now be gathered and stored has created a new attitude toward data analysis: Gather whatever data you can whenever and wherever possible. It has become an article of faith that the gathered data will have value, either for the purpose that initially motivated its collection or for purposes not yet envisioned.

The field of data mining grew out of the limitations of current data anal- ysis techniques in handling the challenges posedl by these new types of data sets. Data mining does not replace other areas of data analysis, but rather takes them as the foundation for much of its work. While some areas of data mining, such as association analysis, are unique to the field, other areas, such as clustering, classification, and anomaly detection, build upon a long history of work on these topics in other fields. Indeed, the willingness of data mining researchers to draw upon existing techniques has contributed to the strength and breadth of the field, as well as to its rapid growth.

Another strength of the field has been its emphasis on collaboration with researchers in other areas. The challenges of analyzing new types of data cannot be met by simply applying data analysis techniques in isolation from those who understand the data and the domain in which it resides. Often, skill in building multidisciplinary teams has been as responsible for the success of data mining projects as the creation of new and innovative algorithms. Just as, historically, many developments in statistics were driven by the needs of agriculture, industry, medicine, and business, rxrany of the developments in data mining are being driven by the needs of those same fields.

This book began as a set of notes and lecture slides for a data mining course that has been offered at the University of Minnesota since Spring 1998 to upper-division undergraduate and graduate Students. Presentation slides

viii Preface

and exercises developed in these offerings grew with time and served as a basis for the book. A survey of clustering techniques in data mining, originally written in preparation for research in the area, served as a starting point for one of the chapters in the book. Over time, the clustering chapter was joined by chapters on data, classification, association analysis, and anomaly detection. The book in its current form has been class tested at the home institutions of the authors-the University of Minnesota and Michigan State University-as well as several other universities.

A number of data mining books appeared in the meantime, but were not completely satisfactory for our students primarily graduate and undergrad- uate students in computer science, but including students from industry and a wide variety of other disciplines. Their mathematical and computer back- grounds varied considerably, but they shared a common goal: to learn about data mining as directly as possible in order to quickly apply it to problems in their own domains. Thus, texts with extensive mathematical or statistical prerequisites were unappealing to many of them, as were texts that required a substantial database background. The book that evolved in response to these students needs focuses as directly as possible on the key concepts of data min- ing by illustrating them with examples, simple descriptions of key algorithms, and exercises.

Overview Specifically, this book provides a comprehensive introduction to data mining and is designed to be accessible and useful to students, instructors, researchers, and professionals. Areas covered include data preprocessing, vi- sualization, predictive modeling, association analysis, clustering, and anomaly detection. The goal is to present fundamental concepts and algorithms for each topic, thus providing the reader with the necessary background for the application of data mining to real problems. In addition, this book also pro- vides a starting point for those readers who are interested in pursuing research in data mining or related fields.

The book covers five main topics: data, classification, association analysis, clustering, and anomaly detection. Except for anomaly detection, each of these areas is covered in a pair of chapters. For classification, association analysis, and clustering, the introductory chapter covers basic concepts, representative algorithms, and evaluation techniques, while the more advanced chapter dis- cusses advanced concepts and algorithms. The objective is to provide the reader with a sound understanding of the foundations of data mining, while still covering many important advanced topics. Because of this approach, the book is useful both as a learning tool and as a reference.

Preface ix

To help the readers better understand the concepts that have been pre- sented, we provide an extensive set of examples, figures, and exercises. Bib- Iiographic notes are included at the end of each chapter for readers who are interested in more advanced topics, historically important papers, and recent trends. The book also contains a comprehensive subject and author index.

To the Instructor As a textbook, this book is suitable for a wide range of students at the advanced undergraduate or graduate level. Since students come to this subject with diverse backgrounds that may not include extensive knowledge of statistics or databases, our book requires minimal prerequisites- no database knowledge is needed and we assume only a modest background in statistics or mathematics. To this end, the book was designed to be as self-contained as possible. Necessary material from statistics, linear algebra, and machine learning is either integrated into the body of the text, or for some advanced topics, covered in the appendices.

Since the chapters covering major data mining topics are self-contained, the order in which topics can be covered is quite flexible. The core material is covered in Chapters 2, 4, 6, 8, and 10. Although the introductory data chapter (2) should be covered first, the basic classification, association analy- sis, and clustering chapters (4, 6, and 8, respectively) can be covered in any order. Because of the relationship of anomaly detection (10) to classification (4) and clustering (8), these chapters should precede Chapter 10. Various topics can be selected from the advanced classification, association analysis, and clustering chapters (5, 7, and 9, respectively) to fit the schedule and in- terests of the instructor and students. We also advise that the lectures be augmented by projects or practical exercises in data mining. Although they are time consuming, such hands-on assignments greatly enhance the value of the course.

Support Materials The supplements for the book are available at Addison- Wesley's Website www.aw.con/cssupport. Support materials available to all readers of this book include

PowerPoint lecture slides

Suggestions for student projects

Data mining resources such as data mining algorithms and data sets

On-line tutorials that give step-by-step examples for selected data mining techniques described in the book using actual data sets and data analysis software

o

o

o

o

x Preface

Additional support materials, including solutions to exercises, are available only to instructors adopting this textbook for classroom use. Please contact your school's Addison-Wesley representative for information on obtaining ac- cess to this material. Comments and suggestions, as well as reports of errors, can be sent to the authors through [email protected].

Acknowledgments Many people contributed to this book. We begin by acknowledging our families to whom this book is dedicated. Without their patience and support, this project would have been impossible.

We would like to thank the current and former students of our data mining groups at the University of Minnesota and Michigan State for their contribu- tions. Eui-Hong (Sam) Han and Mahesh Joshi helped with the initial data min- ing classes. Some ofthe exercises and presentation slides that they created can be found in the book and its accompanying slides. Students in our data min- ing groups who provided comments on drafts of the book or who contributed in other ways include Shyam Boriah, Haibin Cheng, Varun Chandola, Eric Eilertson, Levent Ertoz, Jing Gao, Rohit Gupta, Sridhar Iyer, Jung-Eun Lee, Benjamin Mayer, Aysel Ozgur, Uygar Oztekin, Gaurav Pandey, Kashif Riaz, Jerry Scripps, Gyorgy Simon, Hui Xiong, Jieping Ye, and Pusheng Zhang. We would also like to thank the students of our data mining classes at the Univer- sity of Minnesota and Michigan State University who worked with early drafbs of the book and provided invaluable feedback. We specifically note the helpful suggestions of Bernardo Craemer, Arifin Ruslim, Jamshid Vayghan, and Yu Wei.

Joydeep Ghosh (University of Texas) and Sanjay Ranka (University of Florida) class tested early versions of the book. We also received many useful suggestions directly from the following UT students: Pankaj Adhikari, Ra- jiv Bhatia, Fbederic Bosche, Arindam Chakraborty, Meghana Deodhar, Chris Everson, David Gardner, Saad Godil, Todd Hay, Clint Jones, Ajay Joshi, Joonsoo Lee, Yue Luo, Anuj Nanavati, Tyler Olsen, Sunyoung Park, Aashish Phansalkar, Geoff Prewett, Michael Ryoo, Daryl Shannon, and Mei Yang.

Ronald Kostoff (ONR) read an early version of the clustering chapter and offered numerous suggestions. George Karypis provided invaluable IATEX as- sistance in creating an author index. Irene Moulitsas also provided assistance with IATEX and reviewed some of the appendices. Musetta Steinbach was very helpful in finding errors in the figures.

We would like to acknowledge our colleagues at the University of Min- nesota and Michigan State who have helped create a positive environment for data mining research. They include Dan Boley, Joyce Chai, Anil Jain, Ravi

Preface xi

Janardan, Rong Jin, George Karypis, Haesun Park, William F. Punch, Shashi Shekhar, and Jaideep Srivastava. The collaborators on our many data mining projects, who also have our gratitude, include Ramesh Agrawal, Steve Can- non, Piet C. de Groen, FYan Hill, Yongdae Kim, Steve Klooster, Kerry Long, Nihar Mahapatra, Chris Potter, Jonathan Shapiro, Kevin Silverstein, Nevin Young, and Zhi-Li Zhang.

The departments of Computer Science and Engineering at the University of Minnesota and Michigan State University provided computing resources and a supportive environment for this project. ARDA, ARL, ARO, DOE, NASA, and NSF provided research support for Pang-Ning Tan, Michael Steinbach, and Vipin Kumar. In particular, Kamal Abdali, Dick Brackney, Jagdish Chan- dra, Joe Coughlan, Michael Coyle, Stephen Davis, Flederica Darema, Richard Hirsch, Chandrika Kamath, Raju Namburu, N. Radhakrishnan, James Sido- ran, Bhavani Thuraisingham, Walt Tiernin, Maria Zemankova, and Xiaodong Zhanghave been supportive of our research in data mining and high-performance computing.

It was a pleasure working with the helpful staff at Pearson Education. In particular, we would like to thank Michelle Brown, Matt Goldstein, Katherine Harutunian, Marilyn Lloyd, Kathy Smith, and Joyce Wells. We would also like to thank George Nichols, who helped with the art work and Paul Anag- nostopoulos, who provided I4.T[X support. We are grateful to the following Pearson reviewers: Chien-Chung Chan (University of Akron), Zhengxin Chen (University of Nebraska at Omaha), Chris Clifton (Purdue University), Joy- deep Ghosh (University of Texas, Austin), Nazli Goharian (Illinois Institute of Technology), J. Michael Hardin (University of Alabama), James Hearne (Western Washington University), Hillol Kargupta (University of Maryland, Baltimore County and Agnik, LLC), Eamonn Keogh (University of California- Riverside), Bing Liu (University of Illinois at Chicago), Mariofanna Milanova (University of Arkansas at Little Rock), Srinivasan Parthasarathy (Ohio State University), Zbigniew W. Ras (University of North Carolina at Charlotte), Xintao Wu (University of North Carolina at Charlotte), and Mohammed J. Zaki (Rensselaer Polvtechnic Institute).

Gontents

Preface

Introduction 1 1.1 What Is Data Mining? 2 7.2 Motivating Challenges 4 1.3 The Origins of Data Mining 6 1.4 Data Mining Tasks 7 1.5 Scope and Organization of the Book 11 1.6 Bibliographic Notes 13

v l l

t.7 Exercises

Data

1 6

1 9 2.I Types of Data 22

2.1.I Attributes and Measurement 23 2.L.2 Types of Data Sets . 29

2.2 Data Quality 36 2.2.I Measurement and Data Collection Issues 37 2.2.2 Issues Related to Applications

2.3 Data Preprocessing 2.3.L Aggregation 2.3.2 Sampling 2.3.3 Dimensionality Reduction 2.3.4 Feature Subset Selection 2.3.5 Feature Creation 2.3.6 Discretization and Binarization 2.3:7 Variable Tlansformation .

2.4 Measures of Similarity and Dissimilarity . . . 2.4.L Basics 2.4.2 Similarity and Dissimilarity between Simple Attributes . 2.4.3 Dissimilarities between Data Objects . 2.4.4 Similarities between Data Objects

43 44 45 47 50 5 2 5 5 5 7 63 6 5 66 6 7 6 9 72

xiv Contents

2.4.5 Examples of Proximity Measures 2.4.6 Issues in Proximity Calculation 2.4.7 Selecting the Right Proximity Measure

2.5 BibliographicNotes 2.6 Exercises

Exploring Data 3.i The Iris Data Set 3.2 Summary Statistics

3.2.L Frequencies and the Mode 3.2.2 Percentiles 3.2.3 Measures of Location: Mean and Median 3.2.4 Measures of Spread: Range and Variance 3.2.5 Multivariate Summary Statistics 3.2.6 Other Ways to Summarize the Data

3.3 Visualization 3.3.1 Motivations for Visualization 3.3.2 General Concepts 3.3.3 Techniques 3.3.4 Visualizing Higher-Dimensional Data . 3.3.5 Do's and Don'ts

3.4 OLAP and Multidimensional Data Analysis 3.4.I Representing Iris Data as a Multidimensional Array 3.4.2 Multidimensional Data: The General Case . 3.4.3 Analyzing Multidimensional Data 3.4.4 Final Comments on Multidimensional Data Analysis Bibliographic Notes Exercises

Classification: Basic Concepts, Decision Tlees, and Model Evaluation 4.1 Preliminaries 4.2 General Approach to Solving a Classification Problem 4.3 Decision Tlee Induction

4.3.1 How a Decision Tlee Works 4.3.2 How to Build a Decision TYee 4.3.3 Methods for Expressing Attribute Test Conditions 4.3.4 Measures for Selecting the Best Split . 4.3.5 Algorithm for Decision Tlee Induction 4.3.6 An Examole: Web Robot Detection

3 . 5 3 . 6

73 80 83 84 88

9 7 98 98 99

1 0 0 1 0 1 102 704 1 0 5 1 0 5 1 0 5 1 0 6 1 1 0 724 1 3 0 1 3 1 1 3 1 1 3 3 1 3 5 1 3 9 1 3 9 747

L45 746 748 1 5 0 150 1 5 1 1 5 5 1 5 8 164 1 6 6

Contents xv

4.3.7 Characteristics of Decision Tlee Induction 4.4 Model Overfitting

4.4.L Overfitting Due to Presence of Noise 4.4.2 Overfitting Due to Lack of Representative Samples 4.4.3 Overfitting and the Multiple Comparison Procedure 4.4.4 Estimation of Generalization Errors 4.4.5 Handling Overfitting in Decision Tlee Induction

4.5 Evaluating the Performance of a Classifier 4.5.I Holdout Method 4.5.2 Random Subsampling . . . 4.5.3 Cross-Validation 4.5.4 Bootstrap

4.6 Methods for Comparing Classifiers 4.6.L Estimating a Confidence Interval for Accuracy 4.6.2 Comparing the Performance of Two Models . 4.6.3 Comparing the Performance of Two Classifiers

4.7 BibliographicNotes 4.8 Exercises

5 Classification: Alternative Techniques 5.1 Rule-Based Classifier

5.1.1 How a Rule-Based Classifier Works 5.1.2 Rule-Ordering Schemes 5.1.3 How to Build a Rule-Based Classifier 5.1.4 Direct Methods for Rule Extraction 5.1.5 Indirect Methods for Rule Extraction 5.1.6 Characteristics of Rule-Based Classifiers

5.2 Nearest-Neighbor classifiers 5.2.L Algorithm 5.2.2 Characteristics of Nearest-Neighbor Classifiers

5.3 Bayesian Classifiers 5.3.1 Bayes Theorem 5.3.2 Using the Bayes Theorem for Classification 5.3.3 Naive Bayes Classifier 5.3.4 Bayes Error Rate 5.3.5 Bayesian Belief Networks

5.4 Artificial Neural Network (ANN) 5.4.I Perceptron 5.4.2 Multilayer Artificial Neural Network 5.4.3 Characteristics of ANN

1 6 8 1 7 2 L75 L 7 7 178 179 184 186 1 8 6 1 8 7 1 8 7 1 8 8 1 8 8 1 8 9 1 9 1 192 193 1 9 8

207 207 209 2 I I 2r2 2r3 2 2 L 223 223 225 226 227 228 229 23L 238 240 246 247 25r 255

xvi Contents

5.5 Support Vector Machine (SVM) 5.5.1 Maximum Margin Hyperplanes 5.5.2 Linear SVM: Separable Case 5.5.3 Linear SVM: Nonseparable Case 5.5.4 Nonlinear SVM . 5.5.5 Characteristics of SVM Ensemble Methods 5.6.1 Rationale for Ensemble Method 5.6.2 Methods for Constructing an Ensemble Classifier 5.6.3 Bias-Variance Decomposition 5.6.4 Bagging 5.6.5 Boosting 5.6.6 Random Forests 5.6.7 Empirical Comparison among Ensemble Methods Class Imbalance Problem 5.7.1 Alternative Metrics 5.7.2 The Receiver Operating Characteristic Curve 5.7.3 Cost-Sensitive Learning . . 5.7.4 Sampling-Based Approaches . Multiclass Problem Bibliographic Notes Exercises

5 . 6

o . t

256 256 259 266 270 276 276 277 278 28r 2 8 3 285 290 294 294 295 298 302 305 306 309 3 1 5

c . 6

5 . 9 5 . 1 0

Association Analysis: Basic Concepts and Algorithms 327 6.1 Problem Definition . 328 6.2 Flequent Itemset Generation 332

6.2.I The Apri,ori Principle 333 6.2.2 Fbequent Itemset Generation in the Apri,ori, Algorithm . 335 6.2.3 Candidate Generation and Pruning . . . 338 6.2.4 Support Counting 342 6.2.5 Computational Complexity 345

6.3 Rule Generatiorr 349 6.3.1 Confidence-Based Pruning 350 6.3.2 Rule Generation in Apri,ori, Algorithm 350 6.3.3 An Example: Congressional Voting Records 352

6.4 Compact Representation of Fbequent Itemsets 353 6.4.7 Maximal Flequent Itemsets 354 6.4.2 Closed Frequent Itemsets 355

6.5 Alternative Methods for Generating Frequent Itemsets 359 6.6 FP-Growth Alsorithm 363

Contents xvii

6.6.1 FP-tee Representation 6.6.2 Frequent Itemset Generation in FP-Growth Algorithm .

6.7 Evaluation of Association Patterns 6.7.l Objective Measures of Interestingness 6.7.2 Measures beyond Pairs of Binary Variables 6.7.3 Simpson's Paradox

6.8 Effect of Skewed Support Distribution 6.9 Bibliographic Notes

363 366 370 37r 382 384 386 390 404

4L5 415 4t8 4 1 8 422 424 426 429 429 431 436 439 442 443 444 447 448 453 457 457 458 458

460 461 463 465 469 473

6.10 Exercises

7 Association Analysis: Advanced 7.I Handling Categorical Attributes 7.2 Handling Continuous Attributes

Concepts

7.2.I Discretization-Based Methods 7.2.2 Statistics-Based Methods 7.2.3 Non-discretizalion Methods Handling a Concept Hierarchy Seouential Patterns 7.4.7 Problem Formulation 7.4.2 Sequential Pattern Discovery 7.4.3 Timing Constraints 7.4.4 Alternative Counting Schemes

7.5 Subgraph Patterns 7.5.1 Graphs and Subgraphs . 7.5.2 Frequent Subgraph Mining 7.5.3 Apri,od-like Method 7.5.4 Candidate Generation 7.5.5 Candidate Pruning 7.5.6 Support Counting

7.6 Infrequent Patterns 7.6.7 Negative Patterns 7.6.2 Negatively Correlated Patterns 7.6.3 Comparisons among Infrequent Patterns, Negative Pat-

terns, and Negatively Correlated Patterns 7.6.4 Techniques for Mining Interesting Infrequent Patterns 7.6.5 Techniques Based on Mining Negative Patterns 7.6.6 Techniques Based on Support Expectation .

7.7 Bibliographic Notes 7.8 Exercises

7 . 3 7 . 4

xviii Contents

Cluster Analysis: Basic Concepts and Algorithms 8.1 Overview

8.1.1 What Is Cluster Analysis? 8.I.2 Different Types of Clusterings . 8.1.3 Different Types of Clusters

8.2 K-means 8.2.7 The Basic K-means Algorithm 8.2.2 K-means: Additional Issues 8.2.3 Bisecting K-means 8.2.4 K-means and Different Types of Clusters 8.2.5 Strengths and Weaknesses 8.2.6 K-means as an Optimization Problem

8.3 Agglomerative Hierarchical Clustering 8.3.1 Basic Agglomerative Hierarchical Clustering Algorithm 8.3.2 Specific Techniques 8.3.3 The Lance-Williams Formula for Cluster Proximity . 8.3.4 Key Issues in Hierarchical Clustering . 8.3.5 Strengths and Weaknesses DBSCAN 8.4.1 Tladitional Density: Center-Based Approach 8.4.2 The DBSCAN Algorithm 8.4.3 Strengths and Weaknesses Cluster Evaluation 8.5.1 Overview 8.5.2 Unsupervised Cluster Evaluation Using Cohesion and

Separation 8.5.3 Unsupervised Cluster Evaluation Using the Proximity

Matrix 8.5.4 Unsupervised Evaluation of Hierarchical Clustering . 8.5.5 Determining the Correct Number of Clusters 8.5.6 Clustering Tendency 8.5.7 Supervised Measures of Cluster Validity 8.5.8 Assessing the Significance of Cluster Validity Measures .

8 . 4

8 . 5

487 490 490 49r 493 496 497 506 508 5 1 0 5 1 0 5 1 3 5 1 5 5 1 6 5 1 8 524 524 526 526 5 2 7 528 530 532 5 3 3

536

542 544 546 547 548 5 5 3 o o o

5 5 9 8.6 Bibliograph 8.7 Exercises

ic Notes

Cluster Analysis: Additional Issues and Algorithms 569 9.1 Characteristics of Data, Clusters, and Clustering Algorithms . 570

9.1.1 Example: Comparing K-means and DBSCAN . . . . . . 570 9.1.2 Data Characteristics 577

Contents xix

9.1.3 Cluster Characteristics . . 573 9.L.4 General Characteristics of Clustering Algorithms 575

9.2 Prototype-Based Clustering 577 9.2.1 F\zzy Clustering 577 9.2.2 Clustering Using Mixture Models 583 9.2.3 Self-Organizing Maps (SOM) 594

9.3 Density-Based Clustering 600 9.3.1 Grid-Based Clustering 601 9.3.2 Subspace Clustering 604 9.3.3 DENCLUE: A Kernel-Based Scheme for Density-Based

Clustering 608 9.4 Graph-Based Clustering 612

9.4.1 Sparsification 613 9.4.2 Minimum Spanning Tlee (MST) Clustering . . . 674 9.4.3 OPOSSUM: Optimal Partitioning of Sparse Similarities

Using METIS 616 9.4.4 Chameleon: Hierarchical Clustering with Dynamic

Modeling 9.4.5 Shared Nearest Neighbor Similarity 9.4.6 The Jarvis-Patrick Clustering Algorithm 9.4.7 SNN Density 9.4.8 SNN Density-Based Clustering

9.5 Scalable Clustering Algorithms 9.5.1 Scalability: General Issues and Approaches 9 . 5 . 2 B I R C H 9.5.3 CURE

9.6 Which Clustering Algorithm? 9.7 Bibliographic Notes 9.8 Exercises

6 1 6 622 625 627 629 630 630 633 635 639 643 647

10 Anomaly Detection 651 10.1 Preliminaries 653

10.1.1 Causes of Anomalies 653 10.1.2 Approaches to Anomaly Detection 654 10.1.3 The Use of Class Labels 655 1 0 . 1 . 4 I s s u e s 6 5 6

10.2 Statistical Approaches 658 t0.2.7 Detecting Outliers in a Univariate Normal Distribution 659 1 0 . 2 . 2 O u t l i e r s i n a M u l t i v a r i a t e N o r m a l D i s t r i b u t i o n . . . . . 6 6 1 10.2.3 A Mixture Model Approach for Anomaly Detection. 662

xx Contents

10.2.4 Strengths and Weaknesses 10.3 Proximity-Based Outlier Detection

10.3.1 Strengths and Weaknesses 10.4 Density-Based Outlier Detection

10.4.1 Detection of Outliers Using Relative Density 70.4.2 Strengths and Weaknesses

10.5 Clustering-Based Techniques 10.5.1 Assessing the Extent to Which an Object Belongs to a

Cluster 10.5.2 Impact of Outliers on the Initial Clustering 10.5.3 The Number of Clusters to Use 10.5.4 Strengths and Weaknesses

665 666 666 668 669 670 67L

672 674 674 674 675 680

6 8 5 b6i)

10.6 Bibliograph 10.7 Exercises

ic Notes

Appendix A Linear Algebra A.1 Vectors

A.1.1 Definition 685 4.I.2 Vector Addition and Multiplication by a Scalar 685 A.1.3 Vector Spaces 687 4.7.4 The Dot Product, Orthogonality, and Orthogonal

Projections 688 A.1.5 Vectors and Data Analysis 690

42 Matrices 691 A.2.1 Matrices: Definitions 691 A-2.2 Matrices: Addition and Multiplication by a Scalar 692 4.2.3 Matrices: Multiplication 693 4.2.4 Linear tansformations and Inverse Matrices 695 4.2.5 Eigenvalue and Singular Value Decomposition . 697 4.2.6 Matrices and Data Analysis 699

A.3 Bibliographic Notes 700

Appendix B Dimensionality Reduction 7OL 8.1 PCA and SVD 70I

B.1.1 Principal Components Analysis (PCA) 70L 8 . 7 . 2 S V D . 7 0 6

8.2 Other Dimensionality Reduction Techniques 708 8.2.I Factor Analysis 708 8.2.2 Locally Linear Embedding (LLE) . 770 8.2.3 Multidimensional Scaling, FastMap, and ISOMAP 7I2

Contents xxi

8.2.4 Common Issues B.3 Bibliographic Notes

Appendix C Probability and Statistics C.1 Probability

C.1.1 Expected Values C.2 Statistics

C.2.L Point Estimation C.2.2 Central Limit Theorem C.2.3 Interval Estimation

C.3 Hypothesis Testing

Appendix D Regression D.1 Preliminaries D.2 Simple Linear Regression

D.2.L Least Square Method D.2.2 Analyzing Regression Errors D.2.3 Analyzing Goodness of Fit

D.3 Multivariate Linear Regression D.4 Alternative Least-Square Regression Methods

Appendix E Optimization E.1 Unconstrained Optimizafion

E.1.1 Numerical Methods 8.2 Constrained Optimization

E.2.I Equality Constraints 8.2.2 Inequality Constraints

Author Index

Subject Index

Copyright Permissions

715 7L6

7L9 7L9 722 723 724 724 725 726

739 739 742 746 746 747

750

758

769

729 729 730 731 733 735 736 737

1

Introduction

Rapid advances in data collection and storage technology have enabled or- ganizations to accumulate vast amounts of data. However, extracting useful information has proven extremely challenging. Often, traditional data analy- sis tools and techniques cannot be used because of the massive size of a data set. Sometimes, the non-traditional nature of the data means that traditional approaches cannot be applied even if the data set is relatively small. In other situations, the questions that need to be answered cannot be addressed using existing data analysis techniques, and thus, new methods need to be devel- oped.

Data mining is a technology that blends traditional data analysis methods with sophisticated algorithms for processing large volumes of data. It has also opened up exciting opportunities for exploring and analyzing new types of data and for analyzing old types of data in new ways. In this introductory chapter, we present an overview of data mining and outline the key topics to be covered in this book. We start with a description of some well-known applications that require new techniques for data analysis.

Business Point-of-sale data collection (bar code scanners, radio frequency identification (RFID), and smart card technology) have allowed retailers to collect up-to-the-minute data about customer purchases at the checkout coun- ters of their stores. Retailers can utilize this information, along with other business-critical data such as Web logs from e-commerce Web sites and cus- tomer service records from call centers, to help them better understand the needs of their customers and make more informed business decisions.

Data mining techniques can be used to support a wide range of business intelligence applications such as customer profiling, targeted marketing, work- flow management, store layout, and fraud detection. It can also help retailers

2 Chapter 1 lntroduction

answer important business questions such as "Who are the most profitable customers?" "What products can be cross-sold or up-sold?" and "What is the revenue outlook of the company for next year?)) Some of these questions mo- tivated the creation of association analvsis (Chapters 6 and 7), a new data analysis technique.

Medicine, Science, and Engineering Researchers in medicine, science, and engineering are rapidly accumulating data that is key to important new discoveries. For example, as an important step toward improving our under- standing of the Earth's climate system, NASA has deployed a series of Earth- orbiting satellites that continuously generate global observations of the Iand surface, oceans, and atmosphere. However, because of the size and spatio- temporal nature of the data, traditional methods are often not suitable for analyzing these data sets. Techniques developed in data mining can aid Earth scientists in answering questions such as "What is the relationship between the frequency and intensity of ecosystem disturbances such as droughts and hurricanes to global warming?" "How is land surface precipitation and temper- ature affected by ocean surface temperature?" and "How well can we predict the beginning and end of the growing season for a region?"

As another example, researchers in molecular biology hope to use the large amounts of genomic data currently being gathered to better understand the structure and function of genes. In the past, traditional methods in molecu- lar biology allowed scientists to study only a few genes at a time in a given experiment. Recent breakthroughs in microarray technology have enabled sci- entists to compare the behavior of thousands of genes under various situations. Such comparisons can help determine the function of each gene and perhaps isolate the genes responsible for certain diseases. However, the noisy and high- dimensional nature of data requires new types of data analysis. In addition to analyzing gene array data, data mining can also be used to address other important biological challenges such as protein structure prediction, multiple sequence alignment, the modeling of biochemical pathways, and phylogenetics.

1.1 What Is Data Mining?

Data mining is the process of automatically discovering useful information in large data repositories. Data mining techniques are deployed to scour large databases in order to find novel and useful patterns that might otherwise remain unknown. They also provide capabilities to predict the outcome of a

1.1 What Is Data Mining? 3

future observation, such as predicting whether a newly arrived. customer will spend more than $100 at a department store.

Not all information discovery tasks are considered to be data mining. For example, Iooking up individual records using a database managemenr sysrem or finding particular Web pages via a query to an Internet search engine are tasks related to the area of information retrieval. Although such tasks are important and may involve the use of the sophisticated algorithms and data structures, they rely on traditional computer science techniques and obvious features of the data to create index structures for efficiently organizing and retrieving information. Nonetheless, data mining techniques have been used to enhance information retrieval systems.

Data Mining and Knowledge Discovery

Data mining is an integral part of knowledge discovery in databases (KDD), which is the overall process of converting raw data into useful in- formation, as shown in Figure 1.1. This process consists of a series of trans- formation steps, from data preprocessing to postprocessing of data mining results.

Information

Figure 1 ,1. The process of knowledge discovery in databases (KDD).

The input data can be stored in a variety of formats (flat files, spread- sheets, or relational tables) and may reside in a centralized data repository or be distributed across multiple sites. The purpose of preprocessing is to transform the raw input data into an appropriate format for subsequent analysis. The steps involved in data preprocessing include fusing data from multiple sources, cleaning data to remove noise and duplicate observations, and selecting records and features that are relevant to the data mining task at hand. Because of the many ways data can be collected and stored, data

4 Chapter 1 Introduction

preprocessing is perhaps the most laborious and time-consuming step in the

overall knowledge discovery process. ,,Closing the loop" is the phrase often used to refer to the process of in-

tegrating data mining results into decision support systems. For example,

in business applications, the insights offered by data mining results can be

integrated with campaign management tools so that effective marketing pro-

motions can be conducted and tested. Such integration requires a postpro-

cessing step that ensures that only valid and useful results are incorporated

into the decision support system. An example of postprocessing is visualiza-

tion (see Chapter 3), which allows analysts to explore the data and the data

mining results from a variety of viewpoints. Statistical measures or hypoth-

esis testing methods can also be applied during postprocessing to eliminate

spurious data mining results.

L.2 Motivating Challenges

As mentioned earlier, traditional data analysis techniques have often encoun-

tered practical difficulties in meeting the challenges posed by new data sets.

The following are some of the specific challenges that motivated the develop-

ment of data mining.

Scalability Because of advances in data generation and collection, data sets

with sizes of gigabytes, terabytes, or even petabytes are becoming common.

If data mining algorithms are to handle these massive data sets, then they

must be scalable. Many data mining algorithms employ special search strate-

gies to handle exponential search problems. Scalability may also require the

implementation of novel data structures to access individual records in an ef-

ficient manner. For instance, out-of-core algorithms may be necessary when

processing data sets that cannot fit into main memory. Scalability can also be

improved by using sampling or developing parallel and distributed algorithms.

High Dimensionality It is now common to encounter data sets with hun-

dreds or thousands of attributes instead of the handful common a few decades

ago. In bioinformatics, progress in microarray technology has produced gene

expression data involving thousands of features. Data sets with temporal

or spatial components also tend to have high dimensionality. For example,

consider a data set that contains measurements of temperature at various

locations. If the temperature measurements are taken repeatedly for an ex-

tended period, the number of dimensions (features) increases in proportion to

L.2 Motivating Challenges 5

the number of measurements taken. Tladitional data analysis techniques that were developed for low-dimensional data often do not work well for such high- dimensional data. Also, for some data analysis algorithms, the computational complexity increases rapidly as the dimensionality (the number of features) increases.

Heterogeneous and Complex Data TYaditional data analysis methods often deal with data sets containing attributes of the same type, either contin- uous or categorical. As the role of data mining in business, science, medicine, and other flelds has grown, so has the need for techniques that can handle heterogeneous attributes. Recent years have also seen the emergence of more complex data objects. Examples of such non-traditional types of data include collections of Web pages containing semi-structured text and hyperlinks; DNA data with sequential and three-dimensional structure; and climate data that consists of time series measurements (temperature, pressure, etc.) at various locations on the Earth's surface. Techniques developed for mining such com- plex objects should take into consideration relationships in the data, such as temporal and spatial autocorrelation, graph connectivity, and parent-child re- lationships between the elements in semi-structured text and XML documents.

Data ownership and Distribution Sometimes, the data needed for an analysis is not stored in one location or owned by one organization. Instead, the data is geographically distributed among resources belonging to multiple entities. This requires the development of distributed data mining techniques. Among the key challenges faced by distributed data mining algorithms in- clude (1) how to reduce the amount of communication needed to perform the distributed computatior, (2) how to effectively consolidate the data mining results obtained from multiple sources, and (3) how to address data security issues.

Non-traditional Analysis The traditional statistical approach is based on a hypothesize-and-test paradigm. In other words, a hypothesis is proposed, an experiment is designed to gather the data, and then the data is analyzed with respect to the hypothesis. Unfortunately, this process is extremely labor- intensive. Current data analysis tasks often require the generation and evalu- ation of thousands of hypotheses, and consequently, the development of some data mining techniques has been motivated by the desire to automate the process of hypothesis generation and evaluation. Furthermore, the data sets analyzed in data mining are typically not the result of a carefully designed

6 Chapter 1 Introduction

experiment and often represent opportunistic samples of the data, rather than

random samples. Also, the data sets frequently involve non-traditional types

of data and data distributions.

1.3 The Origins of Data Mining

Brought together by the goal of meeting the challenges of the previous sec-

tion, researchers from different disciplines began to focus on developing more

efficient and scalable tools that could handle diverse types of data. This work,

which culminated in the field of data mining, built upon the methodology and

algorithms that researchers had previously used. In particular, data mining

draws upon ideas, such as (1) sampling, estimation, and hypothesis testing

from statistics and (2) search algorithms, modeling techniques, and learning

theories from artificial intelligence, pattern recognition, and machine learning.

Data mining has also been quick to adopt ideas from other areas, including

optimization, evolutionary computing, information theory, signal processing,

visualization, and information retrieval. A number of other areas also play key supporting roles. In particular,

database systems are needed to provide support for efficient storage, index-

ing, and query processing. Techniques from high performance (parallel) com-

puting are often important in addressing the massive size of some data sets.

Distributed techniques can also help address the issue of size and are essential

when the data cannot be gathered in one location. Figure 1.2 shows the relationship of data mining to other areas.

Figure 1.2. Data mining as a conlluence of many disciplines.

Data Mining Tasks 7

1.4 Data Mining Tasks

Data mining tasks are generally divided into two major categories:

Predictive tasks. The objective of these tasks is to predict the value of a par- ticular attribute based on the values of other attributes. The attribute to be predicted is commonly known as the target or dependent vari- able, while the attributes used for making the prediction are known as the explanatory or independent variables.

Descriptive tasks. Here, the objective is to derive patterns (correlations, trends, clusters, trajectories, and anomalies) that summarize the un- derlying relationships in data. Descriptive data mining tasks are often exploratory in nature and frequently require postprocessing techniques to validate and explain the results.

Figure 1.3 illustrates four of the core data mining tasks that are described in the remainder of this book.

Four of the core data mining tasks.

L . 4

I

Figure 1.3.

8 Chapter 1 Introduction

Predictive modeling refers to the task of building a model for the target

variable as a function of the explanatory variables. There are two types of

predictive modeling tasks: classification, which is used for discrete target

variables, and regression, which is used for continuous target variables. For

example, predicting whether a Web user will make a purchase at an online

bookstore is a classification task because the target variable is binary-valued.

On the other hand, forecasting the future price of a stock is a regression task

because price is a continuous-valued attribute. The goal of both tasks is to

learn a model that minimizes the error between the predicted and true values

of the target variable. Predictive modeling can be used to identify customers

that will respond to a marketing campaign, predict disturbances in the Earth's

ecosystem, or judge whether a patient has a particular disease based on the

results of medical tests.

Example 1.1 (Predicting the Type of a Flower). Consider the task of

predicting a species of flower based on the characteristics of the flower. In

particular, consider classifying an Iris flower as to whether it belongs to one

of the following three Iris species: Setosa, Versicolour, or Virginica. To per-

form this task, we need a data set containing the characteristics of various

flowers of these three species. A data set with this type of information is

the well-known Iris data set from the UCI Machine Learning Repository at

http: /hrurw.ics.uci.edu/-mlearn. In addition to the species of a flower,

this data set contains four other attributes: sepal width, sepal length, petal

length, and petal width. (The Iris data set and its attributes are described

further in Section 3.1.) Figure 1.4 shows a plot of petal width versus petal

length for the 150 flowers in the Iris data set. Petal width is broken into the

categories low, med'ium, and hi'gh, which correspond to the intervals [0' 0.75),

[0.75, 1.75), [1.75, oo), respectively. Also, petal length is broken into categories

low, med,'ium, and hi,gh, which correspond to the intervals [0' 2.5), [2.5,5), [5' oo), respectively. Based on these categories of petal width and length, the

following rules can be derived:

Petal width low and petal length low implies Setosa. Petal width medium and petal length medium implies Versicolour. Petal width high and petal length high implies Virginica.

While these rules do not classify all the flowers, they do a good (but not

perfect) job of classifying most of the flowers. Note that flowers from the

Setosa species are well separated from the Versicolour and Virginica species

with respect to petal width and length, but the latter two species overlap

somewhat with respect to these attributes. I

r Setosa . Versicolour o Virginica

L.4 Data Mining Tasks I

l - - - - a - - f o - - - - - - - i l a o r , f t f o o t o a i : o o o I I

' t 0 f 0 a o 0?oo r a a r f I

? 1 . 7 5 E() r 1 . 5 E

= (t' ()

( L l

!0_l! _.! o_ _o. t a a r O

. .4. a?o o a a a a

a aaaaaaa a a a a a

aa a a a a a

I

I

l l t l l

l l t I

I l l l l t t I

I t !

1 2 2 . 5 3 4 5 ( Petal Length (cm)

Figure 1.4. Petal width versus petal length for 1 50 lris flowers,

Association analysis is used to discover patterns that describe strongly as- sociated features in the data. The discovered patterns are typically represented in the form of implication rules or feature subsets. Because of the exponential size of its search space, the goal of association analysis is to extract the most interesting patterns in an efficient manner. Useful applications of association analysis include finding groups of genes that have related functionality, identi- fying Web pages that are accessed together, or understanding the relationships between different elements of Earth's climate system.

Example 1.2 (Market Basket Analysis). The transactions shown in Ta- ble 1.1 illustrate point-of-sale data collected at the checkout counters of a grocery store. Association analysis can be applied to find items that are fre- quently bought together by customers. For example, we may discover the rule {Diapers} -----* {lt:.ft}, which suggests that customers who buy diapers also tend to buy milk. This type of rule can be used to identify potential cross-selling opportunities among related items. I

Cluster analysis seeks to find groups of closely related observations so that observations that belong to the same cluster are more similar to each other

10 Chapter 1 Introduction

Table 1 .1. Market basket data.

Tlansaction ID Items 1 2 3 4 r

o 7 8 9 1 0

{Bread, Butter, Diapers, Milk}

{Coffee, Sugar, Cookies, Sakoon}

{Bread, Butter, Coffee, Diapers, Milk, Eggs}

{Bread, Butter, Salmon, Chicken}

{fgg", Bread, Butter}

{Salmon, Diapers, Milk}

{Bread, Tea, Sugar, Eggs}

{Coffee, Sugar, Chicken, Eggs}

{Bread, Diapers, Mi1k, Salt}

{Tea, Eggs, Cookies, Diapers, Milk}

than observations that belong to other clusters. Clustering has been used to

group sets of related customers, find areas of the ocean that have a significant

impact on the Earth's climate, and compress data.

Example 1.3 (Document Clustering). The collection of news articles

shown in Table 1.2 can be grouped based on their respective topics. Each

article is represented as a set of word-frequency pairs (r, "),

where tu is a word

and c is the number of times the word appears in the article. There are two

natural clusters in the data set. The first cluster consists of the first four ar-

ticles, which correspond to news about the economy, while the second cluster

contains the last four articles, which correspond to news about health care. A

good clustering algorithm should be able to identify these two clusters based

on the similarity between words that appear in the articles.

Table 1.2. Collection of news articles.

Article Words I 2 .) A

r J

o

7 8

dollar: 1, industry: 4, country: 2, loan: 3, deal: 2, government: 2

machinery: 2, labor: 3, market: 4, industry: 2, work: 3, country: 1 job: 5, inflation: 3, rise: 2, jobless: 2, market: 3, country: 2, index: 3

domestic: 3, forecast: 2, gain: 1, market: 2, sale: 3, price: 2 patient: 4, symptom: 2, drug: 3, health: 2, clinic: 2, doctor: 2 pharmaceutical:2, company: 3, drug: 2,vaccine:1, flu: 3

death: 2, cancer: 4, drug: 3, public: 4, health: 3, director: 2

medical: 2, cost: 3, increase: 2, patient: 2, health: 3, care: 1

1.5 Scope and Organization of the Book 11

Anomaly detection is the task of identifying observations whose character- istics are significantly different from the rest of the data. Such observations are known as anomalies or outliers. The goal of an anomaly detection al- gorithm is to discover the real anomalies and avoid falsely labeling normal objects as anomalous. In other words, a good anomaly detector must have a high detection rate and a low false alarm rate. Applications of anomaly detection include the detection of fraud, network intrusions, unusual patterns of disease, and ecosystem disturbances.

Example 1.4 (Credit Card trYaud Detection). A credit card company records the transactions made by every credit card holder, along with personal information such as credit limit, age, annual income, and address. since the number of fraudulent cases is relatively small compared to the number of legitimate transactions, anomaly detection techniques can be applied to build a profile of legitimate transactions for the users. When a new transaction arrives, it is compared against the profile of the user. If the characteristics of the transaction are very different from the previously created profile, then the transaction is flagged as potentially fraudulent. I

1.5 Scope and Organization of the Book

This book introduces the major principles and techniques used in data mining from an algorithmic perspective. A study of these principles and techniques is essential for developing a better understanding of how data mining technology can be applied to various kinds of data. This book also serves as a starting point for readers who are interested in doing research in this field.

We begin the technical discussion of this book with a chapter on data (Chapter 2), which discusses the basic types of data, data quality, prepro- cessing techniques, and measures of similarity and dissimilarity. Although this material can be covered quickly, it provides an essential foundation for data analysis. Chapter 3, on data exploration, discusses summary statistics, visualization techniques, and On-Line Analytical Processing (OLAP). These techniques provide the means for quickly gaining insight into a data set.

Chapters 4 and 5 cover classification. Chapter 4 provides a foundation by discussing decision tree classifiers and several issues that are important to all classification: overfitting, performance evaluation, and the comparison of different classification models. Using this foundation, Chapter 5 describes a number of other important classification techniques: rule-based systems, nearest-neighbor classifiers, Bayesian classifiers, artificial neural networks, sup- port vector machines, and ensemble classifiers, which are collections of classi-

!2 Chapter 1 lntroduction

fiers. The multiclass and imbalanced class problems are also discussed. These

topics can be covered independently. Association analysis is explored in Chapters 6 and 7. Chapter 6 describes

the basics of association analysis: frequent itemsets, association rules, and

some of the algorithms used to generate them. Specific types of frequent

itemsets-maximal, closed, and hyperclique-that are important for data min-

ing are also discussed, and the chapter concludes with a discussion of evalua-

tion measures for association analysis. Chapter 7 considers a variety of more

advanced topics, including how association analysis can be applied to categor-

ical and continuous data or to data that has a concept hierarchy. (A concept

hierarchy is a hierarchical categorization of objects, e.g., store items, clothing,

shoes, sneakers.) This chapter also describes how association analysis can be

extended to find sequential patterns (patterns involving order), patterns in

graphs, and negative relationships (if one item is present, then the other is

n o t ) . Cluster analysis is discussed in Chapters 8 and 9. Chapter 8 first describes

the different types of clusters and then presents three specific clustering tech-

niques: K-means, agglomerative hierarchical clustering, and DBSCAN. This

is followed by a discussion of techniques for validating the results of a cluster-

ing algorithm. Additional clustering concepts and techniques are explored in

Chapter 9, including fiszzy and probabilistic clustering, Self-Organizing Maps

(SOM), graph-based clustering, and density-based clustering. There is also a

discussion of scalability issues and factors to consider when selecting a clus-

tering algorithm. The last chapter, Chapter 10, is on anomaly detection. After some basic

definitions, several different types of anomaly detection are considered: sta-

tistical, distance-based, density-based, and clustering-based. Appendices A

through E give a brief review of important topics that are used in portions of

the book: linear algebra, dimensionality reduction, statistics, regression, and

optimization. The subject of data mining, while relatively young compared to statistics

or machine learning, is already too large to cover in a single book. Selected

references to topics that are only briefly covered, such as data quality' are

provided in the bibliographic notes of the appropriate chapter. References to

topics not covered in this book, such as data mining for streams and privacy-

preserving data mining, are provided in the bibliographic notes of this chapter.

Bibliographic Notes 13

1.6 Bibliographic Notes

The topic of data mining has inspired many textbooks. Introductory text- books include those by Dunham [10], Han and Kamber l2L), Hand et al. [23], and Roiger and Geatz [36]. Data mining books with a stronger emphasis on business applications include the works by Berry and Linoff [2], Pyle [34], and Parr Rud [33]. Books with an emphasis on statistical learning include those by Cherkassky and Mulier [6], and Hastie et al. 124]. Some books with an emphasis on machine learning or pattern recognition are those by Duda et al. [9], Kantardzic [25], Mitchell [31], Webb [41], and Witten and F]ank [42]. There are also some more specialized books: Chakrabarti [a] (web mining), Fayyad et al. [13] (collection of early articles on data mining), Fayyad et al.

111] (visualization), Grossman et al. [18] (science and engineering), Kargupta and Chan [26] (distributed data mining), Wang et al. [a0] (bioinformatics), and Zaki and Ho [44] (parallel data mining).

There are several conferences related to data mining. Some of the main conferences dedicated to this field include the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), the IEEE In- ternational Conference on Data Mining (ICDM), the SIAM International Con- ference on Data Mining (SDM), the European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD), and the Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD). Data min- ing papers can also be found in other major conferences such as the ACM SIGMOD/PODS conference, the International Conference on Very Large Data Bases (VLDB), the Conference on Information and Knowledge Management (CIKM), the International Conference on Data Engineering (ICDE), the In- ternational Conference on Machine Learning (ICML), and the National Con- ference on Artificial Intelligence (AAAI).

Journal publications on data mining include IEEE Transact'ions on Knowl- edge and Data Engi,neering, Data Mi,ning and Knowledge Discouery, Knowl- edge and Information Systems, Intelli,gent Data Analysi,s, Inforrnati,on Sys- tems, and lhe Journal of Intelligent Informati,on Systems.

There have been a number of general articles on data mining that define the field or its relationship to other fields, particularly statistics. Fayyad et al. [12] describe data mining and how it fits into the total knowledge discovery process. Chen et al. [5] give a database perspective on data mining. Ramakrishnan and Grama [35] provide a general discussion of data mining and present several viewpoints. Hand [22] describes how data mining differs from statistics, as does Fliedman lf 4]. Lambert [29] explores the use of statistics for large data sets and provides some comments on the respective roles of data mining and statistics.

1_.6

L4 Chapter 1 Introduction

Glymour et al. 116] consider the lessons that statistics may have for data

mining. Smyth et aL [38] describe how the evolution of data mining is being

driven by new types of data and applications, such as those involving streams,

graphs, and text. Emerging applications in data mining are considered by Han

et al. [20] and Smyth [37] describes some research challenges in data mining.

A discussion of how developments in data mining research can be turned into

practical tools is given by Wu et al. [43]. Data mining standards are the

subject of a paper by Grossman et al. [17]. Bradley [3] discusses how data

mining algorithms can be scaled to large data sets. With the emergence of new data mining applications have come new chal-

lenges that need to be addressed. For instance, concerns about privacy breaches

as a result of data mining have escalated in recent years, particularly in ap-

plication domains such as Web commerce and health care. As a result, there

is growing interest in developing data mining algorithms that maintain user

privacy. Developing techniques for mining encrypted or randomized data is

known as privacy-preserving data mining. Some general references in this

area include papers by Agrawal and Srikant l1], Clifton et al. [7] and Kargupta

et al. [27]. Vassilios et al. [39] provide a survey. Recent years have witnessed a growing number of applications that rapidly

generate continuous streams of data. Examples of stream data include network

traffic, multimedia streams, and stock prices. Several issues must be considered

when mining data streams, such as the limited amount of memory available,

the need for online analysis, and the change of the data over time. Data

mining for stream data has become an important area in data mining. Some

selected publications are Domingos and Hulten [8] (classification), Giannella

et al. [15] (association analysis), Guha et al. [19] (clustering), Kifer et al. [28] (change detection), Papadimitriou et al. [32] (time series), and Law et al. [30] (dimensionality reduction).

[1]

12l

l o l [')]

[4]

Bibliography R. Agrawal and R. Srikant. Privacy-preserving data mining. ln Proc. of 2000 ACM-

SIGMOD IntI. Conf. on Management of Data, pages 439-450, Dallas, Texas, 2000.

ACM Press.

M. J. A. Berry and G. Linofi. Data Mtni,ng Technr,ques: For Marketing, Sales, and'

Customer Relati,onship Management. Wiley Computer Publishing, 2nd edition, 2004.

P. S. Bradley, J. Gehrke, R. Ramakrishnan, and R. Srikant. Scaling mining algorithms

to large databases. Communicati,ons of the ACM, 45(8):38 43,2002.

S. Chakrabarti. Mini.ng the Web: Di.scoueri.ng Knouledge from Hypertert Data' Morgan

Kaufmann, San Flancisco, CA, 2003.

Bibliography 15

[5] M.-s. chen, J. Han, and P. s. Yu. Data Mining: An overview from a Database Perspective. IEEE Transact'ions on Knowled,ge abd Data Engineering, g(6):g66-gg3, 1 9 9 6 .

[6] v. cherkassky and F. Mulier. Learn'ing from Data: concepts, Theory, and, Method,s. Wiley Interscience, 1g98.

[7] c. clifton, M. Kantarcioglu, and J. vaidya. Defining privacy for data mining. In National Sc'ience Foundat'ion workshop on Nert Generation Data Mining, pages 126- 133, Baltimore, MD, November 2002.

f8] P' Domingos and G. Hulten. Mining high-speed data streams. In Proc. of the 6th Intt. conf. on Knowled,ge Discouery and Data M'in'ing, pages z1-80, Boston, Massachusetts, 2000. ACM Press.

J9] R. o' Duda, P. E. Hart, and D. G. stork. Pattern classification John wiley & sons, Inc., New York, 2nd edition, 2001.

[10] M. H. Dunham. Data Mini,ng: Introd,uctory and, Ad,uanced ropics. prentice Hall, 2002. f11] U. M. Fayyad, G. G. Grinstein, and A. Wierse, editors. Information Visualization in

Data Mining and, Knowled,ge Discouery. Morgan Kaufmann Publishers, San Ftancisco, CA, September 200I.

112] u. M. Fayyad, G. Piatetsky-Shapiro, and P. smyth. Fyom Data Mining to Knowledge Discovery: An overview. rn Ad,aances in Knowledge Discouery and Data M'ining, pages 1-34. AAAI Press, 1996.

[13] u. M. Fayyad, G. Piatetsky-shapiro, P. Smyth, and R. uthurusamy, editors. Aduances 'in Knowled,ge Discouery and Data Mini.ng. AAAI/MIT press, 1g96.

[14] J. H. Friedman. Data Mining and Statistics: What's the Connection? Unpublished. www-stat.stanford.edu/-jhf/ftp/dm-stat.ps, 1992.

[15] c. Giannella, J. Han, J. Pei, X. Yan, and P. s. Yu. Mining Fyequent patterns in Data streams at Multiple Time Granularities. In H. Kargupta, A. Joshi, K. sivakumar, and Y. Yesha, editors, Nert Generation Data M,ining, pages ISI-2I2. AAAI/MIT,2003.

116] c. Glymour, D. Madigan, D. Pregibon, and P. smyth. statistical rhemes and Lessons for Data Mining. Data Mining and Knowleilge D,iscouerg, 1(1):11-28, 1992.

[17] R. L. Grossman, M. F. Hornick, and G. Meyer. Data mining standards initiatives. c omtnunications of the A c M, 45(g) :59-6I, 2002.

[18] R. L. Grossman, c. Kamath, P. Kegelmeyer, v. Kumar, and R. Namburu, editors. Data Mini;ng for Sci,entific and Engineering Applicati,ons. Kluwer Academic Publishers, 2001.

119] s. Guha, A. Meyerson, N. Mishra, R. Motwani, and L. o'callaghan. clustering Data Streams: Theory and Practice. IEEE Tbansact'ions on Knowledge and, Data Engineering, 15(3) :515-528, May/June 2003.

[20] J. Han, R. B. Altman, V. Kumar, H. Mannila, and D. pregibon. Emerging scientific applications in data mining. Communications of the ACM, 4S(8):54-b8,2002.

[21] J. Han and M. Kamber. Data Mining: concepts and, Techniques. Morgan Kaufmann Publishers, San Francisco, 2001.

[22] D. J. Hand. Data Mining: Statistics and More? The American Statistician, 52(2): 1 1 2 - 1 1 8 , 1 9 9 8 .

[23] D. J. Hand, H. Mannila, and P. smyth. Principles of Data Mining. MIT press, 2001.

l24l T. Hastie, R. Tibshirani, and J. H. Fliedman. The Elements of Stati.stical Learning: Data Mini,ng, Inference, Pred,iction. Springer, New York, 2001.

[25] M. Kantardzic. Data Mini,ng: concepts, Mod,el.s, Method,s, and Algorithms. wiley-IEEE Press, Piscataway NJ, 2003.

16 Chapter I Introduction

[26] H. Kargupta and P. K. Chan, editors. Aduances in Di,stributed and Parallel Knowledge

Discouery. AAAI Press, September 2002.

l27l H. Kargupta, s. Datta, Q. wang, and K. sivakumar. on the Privacy Preserving Prop-

erties of Random Data Perturbation Techniques. In Proc. of the 2003 IEEE IntI. Conf.

on Data Mi,n'ing, pages 99 106, Melbourne, Florida, December 2003. IEEE Computer

Society.

f28] D. Kifer, s. Ben-David, and J. Gehrke. Detecting change in Data streams. In Proc. of

the 30th VLDB Conf., pages 180 191, Toronto, Canada,2004. Morgan Kaufmann.

f29] D. Lambert. What Use is Statistics for Massive Data? In ACM SIGMOD Workshop

on Research Issues in Data Mini'ng and Knowledge Di'scouery, pages 54-62, 2000.

[30] M H. C. Law, N. Zhang, and A. K. Jain. Nonlinear Manifold Learning for Data

Streams. In Proc. of the SIAM Intl. Conf. on Data Mi.ning, Lake Buena Vista, Florida,

A p r i l 2 0 0 4 . S I A M .

[31] T. Mitchell. Mach'ine Learning. McGraw-Hill, Boston, MA, 1997.

[32] S. Papadimitriou, A. Brockwell, and C. Faloutsos. Adaptive, unsupervised stream min-

ing. VLDB Journal, 13(3):222 239.2004.

f33l O. Parr Rud. Data Mi,ning Cookbook: Modeling Data for Marleet'ing, Risk and, Customer

Relationship Management John Wiley & Sons, New York, NY, 2001.

[34] D. Pyle. Business Modeling and, Data Mining. Morgan Kaufmann, san FYancisco, cA,

2003.

135] N. Ramakrishnan and A. Grama. Data Mining: From Serendipity to Science-Guest

Editors' Introduction. IEEE Computer, S2(8):34 37, 1999.

[36] R. Roiger and M. Geatz. Data Mzni,ng: A Tutorial Based, Primer. Addison-Wesley,

2002.

137] P. Smyth. Breaking out of the Black-Box: Research Challenges in Data Mining. In

Proc. of the 2001 ACM SIGMOD Workshop on Research Issues i.n Data Mining and

Knowledg e Discouerg, 2OOL.

[38] P. Smyth, D. Pregibon, and C. Faloutsos. Data-driven evolution of data mining algo-

rithms. Commun'ications of the ACM, 45(8):33-37, 2002.

[39] V. S. Verykios, E. Bertino, I. N. Fovino, L. P. Provenza, Y. Saygin, and Y. Theodoridis.

State-of-the-art in privacy preserving data mining. SIGMOD Record,,33(1):50-57' 2004.

[40] J. T. L. Wang, M. J. Zaki, H. Toivonen, and D. tr. Shasha, editors. Data Mining in

Bi,oi,nformatics. Springer, September 2004.

[41] A. R. Webb. Statistical Pattern Recogn'iti'on. John Wiley & Sons, 2nd edition, 2002.

[42] I.H. Witten and E. Frank. Data Mini,ng: Practzcal Machine Learn'ing Tools and Tech-

niques with Jaaa Implernentat'ions. Morgan Kaufmann, 1999.

[43] X. Wu, P. S. Yu, and G. Piatetsky-Shapiro. Data Mining: How Research Meets Practical

Development ? Knowledg e and Inf ormati'on Sy stems, 5 (2) :248-261, 2003.

l44l M. J. Zaki and C.-T. Ho, editors. Large-Scale Parallel Data Mining. Springer, September

2002.

L.7 Exercises

1. Discuss whether or not each of the following activities is a data mining task.

2 .

J .

L.7 Exercises L7

(a) Dividing the customers of a company according to their gender.

(b) Dividing the customers of a company according to their profitability.

(c) Computing the total sales of a company.

(d) Sorting a student database based on student identification numbers.

(e) Predicting the outcomes of tossing a (fair) pair of dice.

(f) Predicting the future stock price of a company using historical records.

(g) Monitoring the heart rate of a patient for abnormalities.

(h) Monitoring seismic waves for earthquake activities.

(i) Extracting the frequencies of a sound wave.

Suppose that you are employed as a data mining consultant for an Internet search engine company. Describe how data mining can help the company by giving specific examples of how techniques, such as clustering, classification, association rule mining, and anomaly detection can be applied.

For each of the following data sets, explain whether or not data privacy is an important issue.

(a) Census data collected from 1900-1950.

(b) IP addresses and visit times of Web users who visit your Website.

(c) Images from Earth-orbiting satellites.

(d) Names and addresses of people from the telephone book.

(e) Names and email addresses collected from the Web.

Data This chapter discusses several data-related issues that are important for suc- cessful data mining:

The Type of Data Data sets differ in a number of ways. For example, the attributes used to describe data objects can be of different types-quantitative or qualitative-and data sets may have special characteristics; e.g., some data sets contain time series or objects with explicit relationships to one another. Not surprisingly, the type of data determines which tools and techniques can be used to analyze the data. F\rrthermore, new research in data mining is often driven by the need to accommodate new application areas and their new types of data.

The Quality of the Data Data is often far from perfect. while most data mining techniques can tolerate some level of imperfection in the data, a focus on understanding and improving data quality typically improves the quality of the resulting analysis. Data quality issues that often need to be addressed include the presence of noise and outliers; missing, inconsistent, or duplicate data; and data that is biased or, in some other way, unrepresentative of the phenomenon or population that the data is supposed to describe.

Preprocessing Steps to Make the Data More suitable for Data Min- ing often, the raw data must be processed in order to make it suitable for analysis. While one objective may be to improve data quality, other goals focus on modifying the data so that it better fits a specified data mining tech- nique or tool. For example, a continuous attribute, e.g., length, m&y need to be transformed into an attribute with discrete categories, e.g., short, med,ium, or long, in order to apply a particular technique. As another example, the

20 Chapter 2 Data

number of attributes in a data set is often reduced because many techniques

are more effective when the data has a relatively small number of attributes.

Analyzing Data in Terms of Its Relationships One approach to data

analysis is to find relationships among the data objects and then perform

the remaining analysis using these relationships rather than the data objects

themselves. For instance, we can compute the similarity or distance between pairs of objects and then perform the analysis-clustering, classification, or

anomaly detection-based on these similarities or distances. There are many

such similarity or distance measures) and the proper choice depends on the

type of data and the particular application.

Example 2.1 (An Illustration of Data-Related Issues). To further il-

Iustrate the importance of these issues, consider the following hypothetical sit-

uation. You receive an email from a medical researcher concerning a project

that you are eager to work on.

H i ,

I've attached the data file that I mentioned in my previous email. Each line contains the information for a single patient and consists of five fields. We want to predict the last field using the other fields. I don't have time to provide any more information about the data since I'm going out of town for a couple of days, but hopefully that won't slow you down too much. And if you don't mind, could we

meet when I get back to discuss your preliminary results? I might invite a few other members of mv team.

Thanks and see you in a couple of days.

Despite some misgivings, you proceed to analyze the data. The first few

rows of the fiIe are as follows:

2 3 2 3 3 . 5 0 1 0 . 7 7 2 r 1 6 . 9 2 2 L 0 . L 1 6 5 2 4 . 0 0 4 2 7 . 6

A brieflook at the data reveals nothing strange. You put your doubts aside

and start the analysis. There are only 1000 lines, a smaller data file than you

had hoped for, but two days later, you feel that you have made some progress.

You arrive for the meeting, and while waiting for others to arrive, you strike

0r2 020 027

2 L

up a conversation with a statistician who is working on the project. When she learns that you have also been analyzing the data from the project, she asks if you would mind giving her a brief overview of your results.

Statistician: So, you got the data for all the patients? Data Miner: Yes. I haven't had much time for analysis, but I

do have a few interesting results. Statistician: Amazing. There were so many data issues with

this set of patients that I couldn't do much. Data Miner: Oh? I didn't hear about any possible problems. Statistician: Well, first there is field 5, the variable we want to

predict. It's common knowledge among people who analyze this type of data that results are better if you work with the log of the values, but I didn't discover this until later. Was it mentioned to you?

Data Miner: No. Statistician: But surely you heard about what happened to field

4? It's supposed to be measured on a scale from 1 to 10, with 0 indicating a missing value, but because of a data entry error, all 10's were changed into 0's. Unfortunately, since some of the patients have missing values for this field, it's impossible to say whether a 0 in this field is a real 0 or a 10. Quite a few of the records have that problem.

Data Miner: Interesting. Were there any other problems? Statistician: Yes, fields 2 and 3 are basically the same, but I

assume that you probably noticed that. Data Miner: Yes, but these fields were only weak predictors of

field 5. Statistician: Anyway, given all those problems, I'm surprised

you were able to accomplish anything. Data Miner: Thue, but my results are really quite good. Field 1

is a very strong predictor of field 5. I'm surprised that this wasn't noticed before.

Statistician: What? Field 1 is just an identification number. Data Miner: Nonetheless, my results speak for themselves. Statistician: Oh, no! I just remembered. We assigned ID

numbers after we sorted the records based on field 5. There is a strong connection, but it's meaningless. Sorry.

Table 2,1. A sample data set containing student information.

22 Chapter 2 Data

Although this scenario represents an extreme situation, it emphasizes the

importance of "knowing your data." To that end, this chapter will address

each of the four issues mentioned above, outlining some of the basic challenges

and standard approaches.

2.L Types of Data

A data set can often be viewed as a collection of data objects. Other

names for a data object are record, po'int, uector, pattern, euent, case, sample,

obseruat'ion, or ent'ity. In turn, data objects are described by a number of

attributes that capture the basic characteristics of an object, such as the

mass of a physical object or the time at which an event occurred. Other

names for an attribute are uariable, characteristi,c, field, feature, ot d'imens'ion.

Example 2.2 (Student Information). Often, a data set is a file, in which

the objects are records (or rows) in the file and each field (or column) corre-

sponds to an attribute. For example, Table 2.1 shows a data set that consists

of student information. Each row corresponds to a student and each column

is an attribute that describes some aspect of a student, such as grade point

average (GPA) or identification number (ID).

Student ID Year Grade Point Average (GPA)

Senior Sophomore Fleshman

I

Although record-based data sets are common, either in flat files or rela-

tional database systems, there are other important types of data sets and

systems for storing data. In Section 2.I.2,we will discuss some of the types of

data sets that are commonly encountered in data mining. However, we first

consider attributes.

1034262 1052663 1082246

3 . 2 4 3 . 5 1 3 . 6 2

2.L Types of Data 23

2.L.t Attributes and Measurement

In this section we address the issue of describing data by considering what types of attributes are used to describe data objects. We first define an at- tribute, then consider what we mean by the type of an attribute, and finally describe the types of attributes that are commonly encountered.

What Is an attribute?

We start with a more detailed definition of an attribute.

Definition 2.1. An attribute is a property or characteristic of an object that may vary; either from one object to another or from one time to another.

For example, eye color varies from person to person, while the temperature of an object varies over time. Note that eye color is a symbolic attribute with a small number of possible values {brown,black,blue, green, hazel, etc.}, while temperature is a numerical attribute with a potentially unlimited number of values.

At the most basic level, attributes are not about numbers or symbols. However, to {iscuss and more precisely analyze the characteristics of objects, we assign numbers or symbols to them. To do this in a well-defined way, we need a measurement scale.

Definition 2.2. A measurement scale is a rule (function) that associates a numerical or symbolic value with an attribute of an object.

Formally, the process of measurement is the application of a measure- ment scale to associate a value with a particular attribute of a specific object. While this may seem a bit abstract, we engage in the process of measurement all the time. For instance, we step on a bathroom scale to determine our weight, we classify someone as male or female, or we count the number of chairs in a room to see if there will be enough to seat all the people coming to a meeting. In all these cases) the "physical value" of an attribute of an object is mapped to a numerical or symbolic value.

With this background, we can now discuss the type of an attribute, a concept that is important in determining if a particular data analysis technique is consistent with a specific type of attribute.

The Type of an Attribute

It should be apparent from the previous discussion that the properties of an attribute need not be the same as the properties of the values used to mea-

24 Chapter 2 Data

sure it. In other words, the values used to represent an attribute may have

properties that are not properties of the attribute itself, and vice versa. This

is illustrated with two examples.

Example 2.3 (Employee Age and ID Number). Two attributes that

might be associated with an employee are ID and age (in years). Both of these

attributes can be represented as integers. However, while it is reasonable to

talk about the average age of an employee, it makes no sense to talk about

the average employee ID. Indeed, the only aspect of employees that we want

to capture with the ID attribute is that they are distinct. Consequently, the

only valid operation for employee IDs is to test whether they are equal. There

is no hint of this limitation, however, when integers are used to represent the

employee ID attribute. For the age attribute, the properties of the integers

used to represent age are very much the properties of the attribute. Even so,

the correspondence is not complete since, for example, ages have a maximum'

while integers do not.

Example 2.4 (Length of Line Segments). Consider Figure 2.1, which

shows some objects-line segments and how the length attribute of these

objects can be mapped to numbers in two different ways. Each successive

line segment, going from the top to the bottom, is formed by appending the

topmost line segment to itself. Thus, the second line segment from the top is

formed by appending the topmost line segment to itself twice, the third line

segment from the top is formed by appending the topmost line segment to

itself three times, and so forth. In a very real (physical) sense, all the line

segments are multiples of the first. This fact is captured by the measurements

on the right-hand side of the figure, but not by those on the left hand-side.

More specifically, the measurement scale on the left-hand side captures only

the ordering of the length attribute, while the scale on the right-hand side

captures both the ordering and additivity properties. Thus, an attribute can be

measured in a way that does not capture all the properties of the attribute. t

The type of an attribute should tell us what properties of the attribute are

reflected in the values used to measure it. Knowing the type of an attribute

is important because it tells us which properties of the measured values are

consistent with the underlying properties of the attribute, and therefore, it

allows us to avoid foolish actions, such as computing the average employee ID.

Note that it is common to refer to the type of an attribute as the type of a

measurement scale.

2.1 Types of Data 25

----> 1

----> 2

--> 3

--> 5

A mapping of lengths to numbers

propertiesof rensth. nffii?;'fi"::ilin""till8in*o Figure 2.1. The measurement of the length of line segments on two different scales of measurement.

The Different Types of Attributes

A useful (and simple) way to specify the type of an attribute is to identify the properties of numbers that correspond to underlying properties of the attribute. For example, an attribute such as length has many of the properties of numbers. It makes sense to compare and order objects by length, as well as to talk about the differences and ratios of length. The following properties (operations) of numbers are typically used to describe attributes.

1. Distinctness : and *

2 . O r d e r < ) < , > , a n d )

3. Addition * and -

4. Multiplication x and /

Given these properties, we can define four types of attributes: nominal, ordinal, interval, and ratio. Table 2.2 gives the definitions of these types, along with information about the statistical operations that are valid for each type. Each attribute type possesses all of the properties and operations of the attribute types above it. Consequently, any property or operation that is valid for nominal, ordinal, and interval attributes is also valid for ratio attributes. In other words, the definition of the attribute types is cumulative. However,

26 Chapter 2 Data

this does not mean that the operations appropriate for one attribute type are

appropriate for the attribute types above it. Nominal and ordinal attributes are collectively referred to as categorical

or qualitative attributes. As the name suggests, qualitative attributes, such

as employee ID, lack most of the properties of numbers. Even if they are rep-

resented by numbers, i.e., integers, they should be treated more like symbols.

The remaining two types of attributes, interval and ratio, are collectively re-

ferred to as quantitative or numeric attributes. Quantitative attributes are

represented by numbers and have most of the properties of numbers. Note

that quantitative attributes can be integer-valued or continuous.

The types of attributes can also be described in terms of transformations

that do not change the meaning of an attribute. Indeed, S. Smith Stevens, the

psychologist who originally defined the types of attributes shown in Table 2.2,

defined them in terms of these permissible transformations. For example,

Table 2.2, Different attribute types.

Attribute Type Description Examples Operations

Nominal The values of a nominal attribute are just different names; i.e., nominal values provide only enough information to distinguish one object from another. t - + \ \ - ) T l

codes, employee ID numbers, eye color, gender

zrp mode, entropy, contingency correlation, y2 test

Ordinal The values of an ordinal attribute provide enough information to order objects. ( < , > )

hardness of minerals,

{good,better,best}, grades, street numbers

median, percentiles, rank correlation, run tests, siqn tests

lnterval For interval attributes, the differences between values are meaningful, i.e., a unit of measurement exists. ( + , - )

calendar dates, temperature in Celsius or Fahrenheit

mean, standard deviation, Pearson's correlation, t and F tests

Katto For ratio variables, both differences and ratios are meaningful. ( +, l )

temperature in Kelvin. monetary quantities, counts, age, mass, length, electrical current

geometric mean, harmonic mean, percent variation

Table 2,3. Transformations that define attribute levels, Attribute Typ" Tlansformation Comment

Nominal Any one-to-one mapping, €.g., & permutation of values

It all employee IIJ numbers are reassigned, it will not make any differcnce

()rdinal An order-preserving change of values. i.e.. new _u alue : f (old _u alue), where / is a monotonic function.

An attribute encompassing the notion of good, better, best can be represented equally well by the values {1,2,3} or by

{ 0 . 5 , 1 , 1 0 } . Interval new -ualue : a * old-talue I b,

o. and b constants. The Fahrenheit and Celsius temperature scales differ in the Iocation of their zero value and the size of a degree (unit).

Ratio new -ualue : a * ol,d-ua|ue Length can be measured in meters or feet.

2 . L Types of Data 27

the meaning of a length attribute is unchanged if it is measured in meters instead of feet.

The statistical operations that make sense for a particular type of attribute are those that will yield the same results when the attribute is transformed us- ing a transformation that preserves the attribute's meaning. To illustrate, the average length of a set of objects is different when measured in meters rather than in feet, but both averages represent the same length. Table 2.3 shows the permissible (meaning-preserving) transformations for the four attribute types of Table 2.2.

Example 2.5 (Temperature Scales). Temperature provides a good illus- tration of some of the concepts that have been described. First, temperature can be either an interval or a ratio attribute, depending on its measurement scale. When measured on the Kelvin scale, a temperature of 2o is, in a physi- cally meaningful way, twice that of a temperature of 1o. This is not true when temperature is measured on either the Celsius or Fahrenheit scales, because, physically, a temperature of 1o Fahrenheit (Celsius) is not much different than a temperature of 2" Fahrenheit (Celsius). The problem is that the zero points of the Fahrenheit and Celsius scales are, in a physical sense, arbitrary, and therefore, the ratio of two Celsius or Fahrenheit temperatures is not physi- cally meaningful.

28 Chapter 2 Data

Describing Attributes by the Number of Values

An independent way of distinguishing between attributes is by the number of values they can take.

Discrete A discrete attribute has a finite or countably infinite set of values. Such attributes can be categorical, such as zip codes or ID numbers, or numeric, such as counts. Discrete attributes are often represented using integer variables. Binary attributes are a special case of dis- crete attributes and assume only two values, e.g., true/false, yes/no, male/female, or 0f 1. Binary attributes are often represented as Boolean variables, or as integer variables that only take the values 0 or 1.

Continuous A continuous attribute is one whose values are real numbers. Ex- amples include attributes such as temperature, height, or weight. Con- tinuous attributes are typically represented as floating-point variables. Practically, real values can only be measured and represented with lim- ited precision.

In theory, any of the measurement scale types-nominal, ordinal, interval, and ratio could be combined with any of the types based on the number of at- tribute values-binary, discrete, and continuous. However, some combinations occur only infrequently or do not make much sense. For instance, it is difficult to think of a realistic data set that contains a continuous binary attribute. Typically, nominal and ordinal attributes are binary or discrete, while interval and ratio attributes are continuous. However, count attributes, which are discrete, are also ratio attributes.

Asymmetric Attributes

For asymmetric attributes, only presence a non-zero attribute value-is re- garded as important. Consider a data set where each object is a student and each attribute records whether or not a student took a particular course at a university. For a specific student, an attribute has a value of 1 if the stu- dent took the course associated with that attribute and a value of 0 otherwise. Because students take only a small fraction of all available courses, most of the values in such a data set would be 0. Therefore, it is more meaningful and more efficient to focus on the non-zero values. To illustrate, if students are compared on the basis of the courses they don't take, then most students would seem very similar, at least if the number of courses is large. Binary attributes where only non-zero values are important are called asymmetric

2 . L Types of Data 29

binary attributes. This type of attribute is particularly important for as- sociation analysis, which is discussed in Chapter 6. It is also possible to have discrete or continuous asymmetric features. For instance, if the number of credits associated with each course is recorded, then the resulting data set will consist of asymmetric discrete or continuous attributes.

2.L.2 Types of Data Sets

There are many types of data sets, and as the field of data mining develops and matures, a greater variety of data sets become available for analysis. In this section, we describe some of the most common types. For convenience, we have grouped the types of data sets into three groups: record data, graph- based data, and ordered data. These categories do not cover all possibilities and other groupings are certainly possible.

General Characteristics of Data Sets

Before providing details of specific kinds of data sets, we discuss three char- acteristics that apply to many data sets and have a significant impact on the data mining techniques that are used: dimensionality, sparsity, and resolution.

Dimensionality The dimensionality of a data set is the number of attributes that the objects in the data set possess. Data with a small number of dimen- sions tends to be qualitatively different than moderate or high-dimensional data. Indeed, the difficulties associated with analyzing high-dimensional data are sometimes referred to as the curse of dimensionality. Because of this, an important motivation in preprocessing the data is dimensionality reduc- tion. These issues are discussed in more depth later in this chapter and in Appendix B.

Sparsity For some data sets, such as those with asymmetric features, most attributes of an object have values of 0; in many casesT fewer than 1% of the entries are non-zero. In practical terms, sparsity is an advantage because usually only the non-zero values need to be stored and manipulated. This results in significant savings with respect to computation time and storage. FurthermoreT some data mining algorithms work well only for sparse data.

Resolution It is frequently possible to obtain data at different levels of reso- Iution, and often the properties ofthe data are different at different resolutions. For instance, the surface of the Earth seems very uneven at a resolution of a

30 Chapter 2 Data

few meters, but is relatively smooth at a resolution of tens of kilometers. The patterns in the data also depend on the level of resolution. If the resolution is too fine, a pattern may not be visible or may be buried in noise; if the resolution is too coarse, the pattern may disappear. For example, variations in atmospheric pressure on a scale of hours reflect the movement of storms and other weather systems. On a scale of months, such phenomena are not detectable.

Record Data

Much data mining work assumes that the data set is a collection of records (data objects), each of which consists of a fixed set of data fields (attributes).

See Figure 2.2(a). For the most basic form of record data, there is no explicit relationship among records or data fields, and every record (object) has the same set of attributes. Record data is usually stored either in flat files or in relational databases. Relational databases are certainly more than a collection of records, but data mining often does not use any of the additional information available in a relational database. Rather, the database serves as a convenient place to find records. Different types of record data are described below and are illustrated in Figure 2.2.

Tbansaction or Market Basket Data Tbansaction data is a special type of record data, where each record (transaction) involves a set of items. Con- sider a grocery store. The set of products purchased by a customer during one shopping trip constitutes a transaction, while the individual products that were purchased are the items. This type of data is called market basket data because the items in each record are the products in a person's "mar- ket basket." Tlansaction data is a collection of sets of items, but it can be viewed as a set of records whose fields are asymmetric attributes. Most often, the attributes are binary, indicating whether or not an item was purchased, but more generally, the attributes can be discrete or continuous, such as the number of items purchased or the amount spent on those items. Figure 2.2(b) shows a sample transaction data set. Each row represents the purchases of a particular customer at a particular time.

The Data Matrix If the data objects in a collection of data all have the same fixed set of numeric attributes, then the data objects can be thought of as points (vectors) in a multidimensional space, where each dimension represents a distinct attribute describing the object. A set of such data objects can be interpreted as an n'L by n matrix, where there are rn rows, one for each object,

2.L Types of Data 31

(a) Record data. (b) Ttansaction data.

Document 1 0 0 2 o 0 0 2

Document 2 0 7 0 0 0 0 0

Document 3 0 I 0 0 2 2 0 o 0

(c) Data matrix. (d) Document-term matrix.

Figure 2.2, Different variations of record data.

and n columns, one for each attribute. (A representation that has data objects as columns and attributes as rows is also fine.) This matrix is called a data matrix or a pattern matrix. A data matrix is a variation of record data, but because it consists of numeric attributes, standard matrix operation can be applied to transform and manipulate the data. Therefore, the data matrix is the standard data format for most statistical data. Figure 2.2(c) shows a sample data matrix.

The Sparse Data Matrix A sparse data matrix is a special case of a data matrix in which the attributes are of the same type and are asymmetric; i.e., only non-zero values are important. Transaction data is an example of a sparse data matrix that has only 0 1 entries. Another common example is document data. In particular, if the order of the terms (words) in a document is ignored,

32 Chapter 2 Data

then a document can be represented as a term vector, where each term is a component (attribute) of the vector and the value of each component is the number of times the corresponding term occurs in the document. This representation of a collection of documents is often called a document-term matrix. Figure 2.2(d) shows a sample document-term matrix. The documents are the rows of this matrix, while the terms are the columns. In practice, only the non-zero entries of sparse data matrices are stored.

Graph-Based Data

A graph can sometimes be a convenient and powerful representation for data. We consider two specific cases: (1) the graph captures relationships among data objects and (2) the data objects themselves are represented as graphs.

Data with Relationships among Objects The relationships among ob- jects frequently convey important information. In such cases, the data is often represented as a graph. In particular, the data objects are mapped to nodes of the graph, while the relationships among objects are captured by the links between objects and link properties, such as direction and weight. Consider Web pages on the World Wide Web, which contain both text and links to other pages. In order to process search queries, Web search engines collect and process Web pages to extract their contents. It is well known, however, that the links to and from each page provide a great deal of information about the relevance of a Web page to a query, and thus, must also be taken into consideration. Figure 2.3(a) shows a set of linked Web pages.

Data with Objects That Are Graphs If objects have structure, that is, the objects contain subobjects that have relationships, then such objects are frequently represented as graphs. For example, the structure of chemical compounds can be represented by a graph, where the nodes are atoms and the links between nodes are chemical bonds. Figure 2.3(b) shows a ball-and-stick diagram of the chemical compound benzene, which contains atoms of carbon (black) and hydrogen (gray). A graph representation makes it possible to determine which substructures occur frequently in a set of compounds and to ascertain whether the presence of any of these substructures is associated with the presence or absence of certain chemical properties, such as melting point

or heat of formation. Substructure mining, which is a branch of data mining that analyzes such data, is considered in Section 7.5.

2 . 1 Types of Data 33

(a) Linked Web pages. (b) Benzene molecule.

Figure 2.3. Different variations of graph data.

Ordered Data

For some types of data, the attributes have relationships that involve order in time or space. Different types of ordered data are described next and are shown in Figure 2.4.

Sequential Data Sequential data, also referred to as temporal data, can be thought of as an extension of record data, where each record has a time associated with it. Consider a retail transaction data set that also stores the time at which the transaction took place. This time information makes it possible to find patterns such as "candy sales peak before Halloween." A time can also be associated with each attribute. For example, each record could be the purchase history of a customer, with a listing of items purchased at different times. Using this information, it is possible to find patterns such as "people who buy DVD players tend to buy DVDs in the period immediately following the purchase."

Figure 2.a@) shows an example of sequential transaction data. There are fi.ve different times-/7, t2, t3, tl, and t5; three different customers-Cl,

Useful Links: . Bbuoq@hv -

. mer Useful Web sib

o ACM SIGmD

o onuqqets

o fteDahh€

Knowledge Discovery and Data Mining Bibliography

(GeB up&td frequenily, so dsironenl)

Bd Refereffi in Dab MilDg and Knwled$ Dlsc@ry

Us@ Fayyad, cregory HateBky-Shapirc, Ptrdc Smyfr, ed Rmmy ud|many, "Advses in kowledge Dhcovery dd De Mining", MI hess/the Mnkss, 1996

J Ross Quinlm, "g 5i kogms ftr Mehne hing", Mqil Kilfmmn hblishers, 1993 Michael Bery ild ftdon Linon "Dau Mining T€hniques (For kkdng, Sales, md Custom Suppd). John Wiley & Sons, 197

Usam Fayyad, "Mining Databiles: Towtrds Algdhms ftr howl€dge D$overy", Bulleun oI he EEE ConpuFr Smery Technical CoM@ o n & B E n g i n e d n g , v o l 2 t , n o I , M m h 1 9 9 8

CMstopher kheus, Philip Chd, ild Gretory Habbky-Shph, "Systems for knowledge Dis@veqf itr dMbes", IEEE Trmsacdons on howledge sd Data Engineedng, 5(6)1903-913, tuembr 193

T i m e Customer Items Purchased t1 c 1 A , B 12 c3 A , C 12 c 1 c , D t3 c2 A , D t4 c2 E r5 c 1 A , E

34 Chapter 2 Data

(a) Sequential transaction data.

GGTTCCGCCTTCAGCCCCGCGCC CGCAGGGCCCGCCCCGCGCCGTC GAGAAGGGC CCGCCTGGCGGGCG GGGGGAGGCGGGGCCGCCCGAGC CCAACCGAGT ECGACCAGGTGCC CCCTCTGCT CGGCCTAGACCTGA GCTCATTAGGCGGCAGCGGACAG GC CAAGTAGAAEAEG CGAAGCGC TGGGCTGCCTGCTGCGACCAGGG

(b) Genomic sequence data.

Minneapolis Av€rage Monthly Temperature (1S2-1993)

1983 19& 1985 1986 1987 1984 1989 1990 1991 lS2 1993 1994 y€r

( c ) T e m p e r a t u r e t i m e s e r i e s . (d) Spatial temperature data.

Figure 2.4. Different variations of ordered data.

C2, and C3; and five different items A, B, C, D, and E. In the top table, each row corresponds to the items purchased at a particular time by each customer. For instance, at time f3, customer C2 purchased items A and D. In the bottom table, the same information is displayed, but each row corresponds to a particular customer. Each row contains information on each transaction involving the customer, where a transaction is considered to be a set of items and the time at which those items were purchased. For example, customer C3 bought items A and C at time t2.

E

E

@

25

20

1 5

1 0

5

0

Customer Time and ltems Purchased c 1 (t1: A,B) (t2:C,D) (ts:A,E) c2 (t3: A, D) (t4: E) c3 ( t 2 : A , C )

2 . 1 Types of Data 35

Sequence Data Sequence data consists of a data set that is a sequence of individual entities, such as a sequence of words or letters. It is quite similar to sequential data, except that there are no time stamps; instead, there are posi- tions in an ordered sequence. For example, the genetic information of plants and animals can be represented in the form of sequences of nucleotides that are known as genes. Many of the problems associated with genetic sequence data involve predicting similarities in the structure and function of genes from similarities in nucleotide sequences. Figure 2.4(b) shows a section of the hu- man genetic code expressed using the four nucleotides from which all DNA is constructed: A, T, G, and C.

Time Series Data Time series data is a special type of sequential data in which each record is a time series, i.e., a series of measurements taken over time. For example, a financial data set might contain objects that are time series of the daily prices of various stocks. As another example, consider Figure 2.4(c), which shows a time series of the average monthly temperature for Minneapolis during the years 1982 to 1994. When working with temporal data, it is important to consider temporal autocorrelation; i.e., if two measurements are close in time, then the values of those measurements are often very similar.

Spatial Data Some objects have spatial attributes, such as positions or ar- eas, as well as other types of attributes. An example of spatial data is weather data (precipitation, temperature, pressure) that is collected for a variety of geographical locations. An important aspect of spatial data is spatial auto- correlation; i.e., objects that are physically close tend to be similar in other ways as well. Thus, two points on the Earth that are close to each other usually have similar values for temperature and rainfall.

Important examples of spatial data are the science and engineering data sets that are the result of measurements or model output taken at regularly or irregularly distributed points on a two- or three-dimensional grid or mesh. For instance, Earth science data sets record the temperature or pressure mea- sured at points (grid cells) on latitude-longitude spherical grids of various resolutions, e.8., 1o by 1o. (See Figure 2.4(d).) As another example, in the simulation of the flow of a gas, the speed and direction of flow can be recorded for each grid point in the simulation.

36 Chapter 2 Data

Handling Non-Record Data

Most data mining algorithms are designed for record data or its variations, such as transaction data and data matrices. Record-oriented techniques can be applied to non-record data by extracting features from data objects and using these features to create a record corresponding to each object. Consider the chemical structure data that was described earlier. Given a set of common substructures, each compound can be represented as a record with binary attributes that indicate whether a compound contains a specific substructure. Such a representation is actually a transaction data set, where the transactions are the compounds and the items are the substructures.

In some cases, it is easy to represent the data in a record format, but this type of representation does not capture all the information in the data. Consider spatio-temporal data consisting of a time series from each point on a spatial grid. This data is often stored in a data matrix, where each row represents a location and each column represents a particular point in time. However, such a representation does not explicitly capture the time relation- ships that are present among attributes and the spatial relationships that exist among objects. This does not mean that such a representation is inap- propriate, but rather that these relationships must be taken into consideration during the analysis. For example, it would not be a good idea to use a data mining technique that assumes the attributes are statistically independent of one another.

2.2 Data Quality

Data mining applications are often applied to data that was collected for an- other purpose, or for future, but unspecified applications. For that reasonT data mining cannot usually take advantage of the significant benefits of "ad- dressing quality issues at the source." In contrast, much of statistics deals with the design of experiments or surveys that achieve a prespecified level of data quality. Because preventing data quality problems is typically not an op- tion, data mining focuses on (1) the detection and correction of data quality problems and (2) the use of algorithms that can tolerate poor data quality. The first step, detection and correction, is often called data cleaning.

The following sections discuss specific aspects of data quality. The focus is on measurement and data collection issues, although some application-related issues are also discussed.

2.2 Data Quality 37

2.2.L Measurement and Data Collection Issues

It is unrealistic to expect that data will be perfect. There may be problems due to human error, limitations of measuring devices, or flaws in the data collection process. Values or even entire data objects may be missing. In other cases, there may be spurious or duplicate objects; i.e., multiple data objects that all correspond to a single "real" object. For example, there might be two different records for a person who has recently lived at two different addresses. Even if all the data is present and "looks fine," there may be inconsistencies-a person has a height of 2 meters, but weighs only 2 kilograms.

In the next few sections, we focus on aspects ofdata quality that are related to data measurement and collection. We begin with a definition of measure- ment and data collection errors and then consider a variety of problems that involve measurement error: noise, artifacts, bias, precision, and accuracy. We conclude by discussing data quality issues that may involve both measurement and data collection problems: outliers, missing and inconsistent values, and duplicate data.

Measurement and Data Collection Errors

The term measurement error refers to any problem resulting from the mea- surement process. A common problem is that the value recorded differs from the true value to some extent. For continuous attributes, the numerical dif- ference of the measured and true value is called the error. The term data collection error refers to errors such as omitting data objects or attribute values, or inappropriately including a data object. For example, a study of animals of a certain species might include animals of a related species that are similar in appearance to the species of interest. Both measurement errors and data collection errors can be either systematic or random.

We will only consider general types of errors. Within particular domains, there are certain types of data errors that are commonplace, and there ofben exist well-developed techniques for detecting and/or correcting these errors. For example, keyboard errors are common when data is entered manually, and as a result, many data entry programs have techniques for detecting and, with human intervention, correcting such errors.

Noise and Artifacts

Noise is the random component of a measurement error. It may involve the distortion of a value or the addition of spurious objects. Figure 2.5 shows a time series before and after it has been disrupted by random noise. If a bit

38 Chapter 2 Data

(a) Time series.

Figure 2.5. Noise in a time series context.

(b) Time series with noise.

. i*. i

^ + , +

T

i a t

(a) Three groups of points. (b) With noise points (+) added.

Figure 2.6. Noise in a spatial context.

more noise were added to the time series, its shape would be lost. Figure 2.6 shows a set of data points before and after some noise points (indicated by '+'s) have been added. Notice that some of the noise points are intermixed with the non-noise points.

The term noise is often used in connection with data that has a spatial or temporal component. In such cases, techniques from signal or image process- ing can frequently be used to reduce noise and thus, help to discover patterns (signals) that might be "lost in the noise." Nonetheless, the elimination of noise is frequently difficult, and much work in data mining focuses on devis- ing robust algorithms that produce acceptable results even when noise is present.

T +

2.2 Data Quality 39

Data errors may be the result of a more deterministic phenomenon, such as a streak in the same place on a set of photographs. Such deterministic distortions of the data are often referred to as artifacts.

Precision, Bias, and Accuracy

In statistics and experimental science, the quality of the measurement process and the resulting data are measured by precision and bias. We provide the standard definitions, followed by a brief discussion. For the following defini- tions, we assume that we make repeated measurements of the same underlying quantity and use this set of values to calculate a mean (average) value that serves as our estimate of the true value.

Definition 2.3 (Precision). The closeness of repeated measurements (of the same quantity) to one another.

Deffnition 2.4 (Bias). A systematic quantity being measured.

Precision is often measured by the standard deviation of a set of values, while bias is measured by taking the difference between the mean of the set of values and the known value of the quantity being measured. Bias can only be determined for objects whose measured quantity is known by means external to the current situation. Suppose that we have a standard laboratory weight with a mass of 1g and want to assess the precision and bias of our new Iaboratory scale. We weigh the mass five times, and obtain the following five v a l u e s : { 1 . 0 1 5 , 0 . 9 9 0 , 1 . 0 1 3 , 1 . 0 0 1 , 0 . 9 8 6 } . T h e m e a n o f t h e s e v a l u e s i s 1 . 0 0 1 , and hence, the bias is 0.001. The precision, as measured by the standard deviation, is 0.013.

It is common to use the more general term, accuracy, to refer to the degree of measurement error in data.

Definition 2.5 (Accuracy). The closeness of measurements to the true value of the quantity being measured.

Accuracy depends on precision and bias, but since it is a general concept, there is no specific formula for accuracy in terms of these two quantities.

One important aspect of accuracy is the use of significant digits. The goal is to use only as many digits to represent the result of a measurement or calculation as are justified by the precision of the data. For example, if the Iength of an object is measured with a meter stick whose smallest markings are millimeters, then we should only record the length of data to the nearest mil- limeter. The precision of such a measurement #ould be * 0.5mm. We do not

of measurements from-the

40 Chapter 2 Data

review the details of working with significant digits, as most readers will have encountered them in previous courses, and they are covered in considerable depth in science, engineering, and statistics textbooks.

Issues such as significant digits, precision, bias, and accuracy are sometimes overlooked, but they are important for data mining as well as statistics and science. Many times, data sets do not come with information on the precision of the data, and furthermore, the programs used for analysis return results without any such information. Nonetheless, without some understanding of the accuracy of the data and the results, an analyst runs the risk of committing serious data analysis blunders.

Outliers

Outliers are either (1) data objects that, in some sense, have characteristics that are different from most of the other data objects in the data set, or (2) values of an attribute that are unusual with respect to the typical values for that attribute. Alternatively, we can speak of anomalous objects or values. There is considerable leeway in the definition of an outlier, and many different definitions have been proposed by the statistics and data mining communities. Furthermore, it is important to distinguish between the notions of noise and outliers. Outliers can be legitimate data objects or values. Thus, unlike noise, outliers may sometimes be of interest. In fraud and network intrusion detection, for example, the goal is to find unusual objects or events from among a large number of normal ones. Chapter 10 discusses anomaly detection in more detail.

Missing Values

It is not unusual for an object to be missing one or more attribute values. In some cases, the information was not collected; e.g., some people decline to give their age or weight. In other cases, some attributes are not applicable to all objects; e.g., often, forms have conditional parts that are filled out only when a person answers a previous question in a certain way, but for simplicity, all fields are stored. Regardless, missing values should be taken into account during the data analysis.

There are several strategies (and variations on these strategies) for dealing with missing data, each of which may be appropriate in certain circumstances. These strategies are listed next, along with an indication of their advantages and disadvantages.

2 . 2 Data Quality 4L

Eliminate Data Objects or Attributes A simple and effective strategy is to eliminate objects with missing values. However, even a partially speci- fied data object contains some information, and if many objects have missing values, then a reliable analysis can be difficult or impossible. Nonetheless, if a data set has only a few objects that have missing values, then it may be expedient to omit them. A related strategy is to eliminate attributes that have missing values. This should be done with caution, however, since the eliminated attributes may be the ones that are critical to the analysis.

Estimate Missing Values Sometimes missing data can be reliably esti- mated. For example, consider a time series that changes in a reasonably smooth fashion, but has a few, widely scattered missing values. In such cases, the missing values can be estimated (interpolated) bV using the remaining values. As another example, consider a data set that has many similar data points. In this situation, the attribute values of the points closest to the point with the missing value are often used to estimate the missing value. If the attribute is continuous, then the average attribute value of the nearest neigh- bors is used; if the attribute is categorical, then the most commonly occurring attribute value can be taken. For a concrete illustration, consider precipitation measurements that are recorded by ground stations. For areas not containing a ground station, the precipitation can be estimated using values observed at nearby ground stations.

Ignore the Missing Value during Analysis Many data mining approaches can be modified to ignore missing values. For example, suppose that objects are being clustered and the similarity between pairs of data objects needs to be calculated. If one or both objects of a pair have missing values for some attributes, then the similarity can be calculated by using only the attributes that do not have missing values. It is true that the similarity will only be approximate, but unless the total number of attributes is small or the num- ber of missing values is high, this degree of inaccuracy may not matter much. Likewise, many classification schemes can be modified to work with missing values.

Inconsistent Values

Data can contain inconsistent values. Consider an address field, where both a zip code and city are listed, but the specified zip code area is not contained in that city. It may be that the individual entering this information transposed two digits, or perhaps a digit was misread when the information was scanned

42 Chapter 2 Data

from a handwritten form. Regardless of the cause of the inconsistent values, it is important to detect and, if possible, correct such problems.

Some types of inconsistences are easy to detect. For instance, a person's height should not be negative. In other cases, it can be necessary to consult an external source of information. For example, when an insurance company processes claims for reimbursement, it checks the names and addresses on the reimbursement forms against a database of its customers.

Once an inconsistency has been detected, it is sometimes possible to correct the data. A product code may have "check" digits, or it may be possible to double-check a product code against a list of known product codes, and then correct the code if it is incorrect, but close to a known code. The correction of an inconsistency requires additional or redundant information.

Example 2.6 (Inconsistent Sea Surface Temperature). This example illustrates an inconsistency in actual time series data that measures the sea surface temperature (SST) at various points on the ocean. SST data was origi- nally collected using ocean-based measurements from ships or buoys, but more recently, satellites have been used to gather the data. To create a long-term data set, both sources of data must be used. However, because the data comes from different sources, the two parts of the data are subtly different. This discrepancy is visually displayed in Figure 2.7, which shows the correlation of SST values between pairs of years. If a pair of years has a positive correlation, then the location corresponding to the pair of years is colored white; otherwise it is colored black. (Seasonal variations were removed from the data since, oth- erwise, all the years would be highly correlated.) There is a distinct change in behavior where the data has been put together in 1983. Years within each of the two groups, 1958-1982 and 1983-1999, tend to have a positive correlation with one another, but a negative correlation with years in the other group. This does not mean that this data should not be used, only that the analyst should consider the potential impact of such discrepancies on the data mining analysis.

Duplicate Data

A data set may include data objects that are duplicates, or almost duplicates, of one another. Many people receive duplicate mailings because they appear in a database multiple times under slightly different names. To detect and eliminate such duplicates, two main issues must be addressed. First, if there are two objects that actually represent a single object, then the values of corresponding attributes may differ, and these inconsistent values must be

2.2 Data Quality 43

60 65 70 75 80 85 90 95

Year

Figure 2.7, Conelation of SST data between pairs of years. White areas indicate positive correlation. Black areas indicate negative correlation.

resolved. Second, care needs to be taken to avoid accidentally combining data objects that are similar, but not duplicates, such as two distinct people with identical names. The term deduplication is often used to refer to the process of dealing with these issues.

In some cases, two or more objects are identical with respect to the at- tributes measured by the database, but they still represent different objects. Here, the duplicates are legitimate, but may still cause problems for some al- gorithms if the possibility of identical objects is not specifically accounted for in their design. An example of this is given in Exercise 13 on page 91.

2.2.2 Issues Related to Applications

Data quality issues can also be considered from an application viewpoint as expressed by the statement "data is of high quality if it is suitable for its intended use." This approach to data quality has proven quite useful, particu- Iarly in business and industry. A similar viewpoint is also present in statistics and the experimental sciences, with their emphasis on the careful design of ex- periments to collect the data relevant to a specific hypothesis. As with quality

b I

44 Chapter 2 Data

issues at the measurement and data collection level, there are many issues that are specific to particular applications and fields. Again, we consider only a few of the general issues.

Timeliness Some data starts to age as soon as it has been collected. In particular, if the data provides a snapshot of some ongoing phenomenon or process, such as the purchasing behavior of customers or Web browsing pat- terns, then this snapshot represents reality for only a limited time. If the data is out of date, then so are the models and patterns that are based on it.

Relevance The available data must contain the information necessary for the application. Consider the task of building a model that predicts the acci- dent rate for drivers. If information about the age and gender of the driver is omitted, then it is likely that the model will have limited accuracy unless this information is indirectly available through other attributes.

Making sure that the objects in a data set are relevant is also challenging. A common problem is sampling bias, which occurs when a sample does not contain different types of objects in proportion to their actual occurrence in the population. For example, survey data describes only those who respond to the survey. (Other aspects of sampling are discussed further in Section 2.3.2.) Because the results of a data analysis can reflect only the data that is present, sampling bias will typically result in an erroneous analysis.

Knowledge about the Data Ideally, data sets are accompanied by doc- umentation that describes different aspects of the data; the quality of this documentation can either aid or hinder the subsequent analysis. For example, if the documentation identifies several attributes as being strongly related, these attributes are likely to provide highly redundant information, and we may decide to keep just one. (Consider sales tax and purchase price.) If the documentation is poor, however, and fails to tell us, for example, that the missing values for a particular field are indicated with a -9999, then our analy- sis of the data may be faulty. Other important characteristics are the precision of the data, the type of features (nominal, ordinal, interval, ratio), the scale of measurement (e.g., meters or feet for length), and the origin of the data.

2.3 Data Preprocessing

In this section, we address the issue of which preprocessing steps should be applied to make the data more suitable for data mining. Data preprocessing

2.3 Data Preprocessing 45

is a broad area and consists of a number of different strategies and techniques that are interrelated in complex ways. We will present some of the most important ideas and approaches, and try to point out the interrelationships among them. Specifically, we will discuss the following topics:

o Aggregation

o Sampling

o Dimensionality reduction

o Feature subset selection o Feature creation

o Discretization and binarization

o Variable transformation

Roughly speaking, these items fall into two categories: selecting data ob- jects and attributes for the analysis or creating/changing the attributes. In both cases the goal is to improve the data mining analysis with respect to time, cost, and quality. Details are provided in the following sections.

A quick note on terminology: In the following, we sometimes use synonyms for attribute, such as feature or variable, in order to follow common usage.

2.3.L Aggregation

Sometimes "less is more" and this is the case with aggregation, the combining of two or more objects into a single object. Consider a data set consisting of transactions (data objects) recording the daily sales of products in various store locations (Minneapolis, Chicago, Paris, ...) for different days over the course of a year. See Table 2.4. One way to aggregate transactions for this data set is to replace all the transactions of a single store with a single storewide transaction. This reduces the hundreds or thousands of transactions that occur daily at a specific store to a single daily transaction, and the number of data objects is reduced to the number of stores.

An obvious issue is how an aggregate transaction is created; i.e., how the values of each attribute are combined across all the records corresponding to a particular location to create the aggregate transaction that represents the sales of a single store or date. Quantitative attributes, such as price, are typically aggregated by taking a sum or an average. A qualitative attribute, such as item, can either be omitted or summarized as the set of all the items that were sold at that location.

The data in Table 2.4 can also be viewed as a multidimensional array, where each attribute is a dimension. FYom this viewpoint, aggregation is the

Table2.4. Data set containing information about customer purchases.

46 Chapter 2 Data

Ttansaction ID Item I Store Location

L01r23 r0rl23 t 0 r r 2 4

Watch Battery Shoes

: Chicago Chicago

Minneapolis

Date

: 0 e 1 0 6 / 0 4 0 e / 0 6 1 0 4 0 s 1 0 6 1 0 4

process of eliminating attributes, such as the type of item, or reducing the number of values for a particular attribute; e.g., reducing the possible values for date from 365 days to 12 months. This type of aggregation is commonly used in Online Analytical Processing (OLAP), which is discussed further in Chapter 3.

There are several motivations for aggregation. First, the smaller data sets resulting from data reduction require less memory and processing time, and hence, aggregation may permit the use of more expensive data mining algo- rithms. Second, aggregation can act as a change ofscope or scale by providing a high-level view of the data instead of a low-level view. In the previous ex- ample, aggregating over store locations and months gives us a monthly, per store view of the data instead of a daily, per item view. Finally, the behavior of groups of objects or attributes is often more stable than that of individual objects or attributes. This statement reflects the statistical fact that aggregate quantities, such as averages or totals, have less variability than the individ- ual objects being aggregated. For totals, the actual amount of variation is larger than that of individual objects (on average), but the percentage of the variation is smaller, while for means, the actual amount of variation is less than that of individual objects (on average). A disadvantage of aggregation is the potential loss of interesting details. In the store example aggregating over months loses information about which day of the week has the highest sales.

Example 2.7 (Australian Precipitation). This example is based on pre- cipitation in Australia from the period 1982 to 1993. Figure 2.8(a) shows a histogram for the standard deviation of average monthly precipitation for 3,030 0.5' by 0.5' grid cells in Australia, while Figure 2.8(b) shows a histogram for the standard deviation of the average yearly precipitation for the same lo- cations. The average yearly precipitation has less variability than the average monthly precipitation. All precipitation measurements (and their standard deviations) are in centimeters.

I

(a) Histogram of standard deviation of average monthly precipitation

Data Preprocessing 47

(b) Histogram of standard deviation of average yearly precipitation

2 . 3

z

; .5

6

z

Figure 2.8. Histograms of standard deviation for monthly and yearly precipitation in Australia for the oeriod 1982 to 1993.

2 . 3 . 2 S a m p l i n g

Sampling is a commonly used approach for selecting a subset of the data objects to be analyzed. In statistics, it has long been used for both the pre- Iiminary investigation of the data and the final data analysis. Sampling can also be very useful in data mining. However, the motivations for sampling in statistics and data mining are ofben different. Statisticians use sampling because obtaining the entire set of data of interest is too expensive or time consuming, while data miners sample because it is too expensive or time con- suming to process all the data. In some cases, using a sampling algorithm can reduce the data size to the point where a better, but more expensive algorithm can be used.

The key principle for effective sampling is the following: Using a sample will work almost as well as using the entire data set if the sample is repre- sentative. fn turn, a sample is representative if it has approximately the same property (of interest) as the original set of data. If the mean (average) of the data objects is the property of interest, then a sample is representative if it has a mean that is close to that of the original data. Because sampling is a statistical process, the representativeness of any particular sample will vary, and the best that we can do is choose a sampling scheme that guarantees a high probability of getting a representative sample. As discussed next, this involves choosing the appropriate sample size and sampling techniques.

48 Chapter 2

Sampling Approaches

There are many sampling techniques, but only a few of the most basic ones and their variations will be covered here. The simplest type of sampling is simple random sampling. For this type of sampling, there is an equal prob- ability of selecting any particular item. There are two variations on random sampling (and other sampling techniques as well): (1) sampling without re- placement-as each item is selected, it is removed from the set of all objects that together constitute the population, and (2) sampling with replace- ment-objects are not removed from the population as they are selected for the sample. In sampling with replacement, the same object can be picked more than once. The samples produced by the two methods are not much different when samples are relatively small compared to the data set size, but sampling with replacement is simpler to analyze since the probability of selecting any object remains constant during the sampling process.

When the population consists of different types of objects, with widely different numbers of objects, simple random sampling can fail to adequately represent those types of objects that are less frequent. This can cause prob- lems when the analysis requires proper representation of all object types. For example, when building classification models for rare classes, it is critical that the rare classes be adequately represented in the sample. Hence, a sampling scheme that can accommodate differing frequencies for the items of interest is needed. Stratified sampling, which starts with prespecified groups of ob- jects, is such an approach. In the simplest version, equal numbers of objects are drawn from each group even though the groups are ofdifferent sizes. In an- other variation, the number of objects drawn from each group is proportional to the size of that group.

Example 2.8 (Sampling and Loss of Information). Once a sampling technique has been selected, it is still necessary to choose the sample size. Larger sample sizes increase the probability that a sample will be representa- tive, but they also eliminate much of the advantage of sampling. Conversely, with smaller sample sizes, patterns may be missed or erroneous patterns can be detected. Figure 2.9(a) shows a data set that contains 8000 two-dimensional points, while Figures 2.9(b) and 2.9(c) show samples from this data set of size 2000 and 500, respectively. Although most of the structure of this data set is present in the sample of 2000 points, much of the structure is missing in the sample of 500 points. r

Data

2 . 3 Data Preprocessing 49

(a) 8000 points (b) 2000 points (c) 500 points

Figure 2.9. Example of the loss of structure with sampling,

Example 2.9 (Determining the Proper Sample Size). To illustrate that determining the proper sample size requires a methodical approach, consider the following task.

Given a set of data that consists of a small number of almost equal- sized groups, find at least one representative point for each of the groups. Assume that the objects in each group are highly similar to each other, but not very similar to objects in different groups. Also assume that there are a relatively small number of groups, e.g., 10. Figure 2.10(a) shows an idealized set of clusters (groups) from which these points might be drawn.

This problem can be efficiently solved using sampling. One approach is to take a small sample of data points, compute the pairwise similarities between points, and then form groups of points that are highly similar. The desired set of representative points is then obtained by taking one point from each of these groups. To follow this approach, however, we need to determine a sample size that would guarantee, with a high probability, the desired outcome; that is, that at least one point will be obtained from each cluster. Figure 2.10(b) shows the probability of getting one object from each of the 10 groups as the sample size runs from 10 to 60. Interestingly, with a sample size of 20, there is little chance (20%) of getting a sample that includes all 10 clusters. Even with a sample size of 30, there is still a moderate chance (almost a0%) of getting a sample that doesn't contain objects from all 10 clusters. This issue is further explored in the context of clustering by Exercise 4 on page 559.

I

ii

50 Chapter 2 Data

oo oo oo

(a) ren sroups or points' f,tl-T:i:1ti[ ;,""T"1"

contains points

Figure 2.10. Finding representative points from 10 groups.

Progressive Sampling

The proper sample size can be difficult to determine, so adaptive or progres- sive sampling schemes are sometimes used. These approaches start with a small sample, and then increase the sample size until a sample of sufficient size has been obtained. While this technique eliminates the need to determine the correct sample size initially, it requires that there be a way to evaluate the sample to judge if it is large enough.

Suppose, for instance, that progressive sampling is used to learn a pre- dictive model. Although the accuracy of predictive models increases as the sample size increases, at some point the increase in accuracy levels off. We want to stop increasing the sample size at this leveling-off point. By keeping track of the change in accuracy of the model as we take progressively larger samples, and by taking other samples close to the size of the current one, we can get an estimate as to how close we are to this leveling-off point, and thus, stop sampling.

2.3.3 Dimensionality Reduction

Data sets can have a large number of features. Consider a set of documents, where each document is represented by a vector whose components are the frequencies with which each word occurs in the document. In such cases,

(d

e n

o o

o o

Sample Size

2.3 Data Preprocessing 51

there are typically thousands or tens of thousands of attributes (components), one for each word in the vocabulary. As another example, consider a set of time series consisting of the daily closing price of various stocks over a period of 30 years. In this case, the attributes, which are the prices on specific days, again number in the thousands.

There are a variety of benefits to dimensionality reduction. A key benefit is that many data mining algorithms work better if the dimensionality the number of attributes in the data-is lower. This is partly because dimension- ality reduction can eliminate irrelevant features and reduce noise and partly because of the curse of dimensionality, which is explained below. Another ben- efit is that a reduction of dimensionality can lead to a more understandable model because the model may involve fewer attributes. Also, dimensionality reduction may allow the data to be more easily visualized. Even if dimen- sionality reduction doesn't reduce the data to two or three dimensions, data is often visualized by looking at pairs or triplets of attributes, and the num- ber of such combinations is greatly reduced. Finally, the amount of time and memory required by the data mining algorithm is reduced with a reduction in dimensionality.

The term dimensionality reduction is often reserved for those techniques that reduce the dimensionality of a data set by creating new attributes that are a combination of the old attributes. The reduction of dimensionality by selecting new attributes that are a subset of the old is known as feature subset selection or feature selection. It will be discussed in Section 2.3.4.

In the remainder of this section, we briefly introduce two important topics: the curse of dimensionality and dimensionality reduction techniques based on linear algebra approaches such as principal components analysis (PCA). More details on dimensionality reduction can be found in Appendix B.

The Curse of Dimensionality

llhe curse of dimensionality refers to the phenomenon that many types of data analysis become significantly harder as the dimensionality of the data increases. Specifically, as dimensionality increases, the data becomes increas- ingly sparse in the space that it occupies. For classification, this can mean that there are not enough data objects to allow the creation of a model that reliably assigns a class to all possible objects. For clustering, the definitions of density and the distance between points, which are critical for clustering, become less meaningful. (This is discussed further in Sections g.1.2, 9.4.5, and 9.4.7.) As a result) many clustering and classification algorithms (and other

52 Chapter 2 Data

data analysis algorithms) have trouble with high-dimensional data-reduced

classification accuracy and poor quality clusters.

Linear Algebra Techniques for Dimensionality Reduction

Some of the most common approaches for dimensionality reduction, partic-

ularly for continuous data, use techniques from linear algebra to project the

data from a high-dimensional space into a lower-dimensional space. Principal

Components Analysis (PCA) is a linear algebra technique for continuous

attributes that finds new attributes (principal components) that (1) are linear

combinations of the original attributes, (2) are orthogonal (perpendicular) to

each other, and (3) capture the maximum amount of variation in the data. For

example, the first two principal components capture as much of the variation

in the data as is possible with two orthogonal attributes that are linear combi-

nations of the original attributes. Singular Value Decomposition (SVD)

is a linear algebra technique that is related to PCA and is also commonly used

for dimensionality reduction. For additional details, see Appendices A and B.

2.3.4 Feature Subset Selection

Another way to reduce the dimensionality is to use only a subset of the fea-

tures. While it might seem that such an approach would lose information, this

is not the case if redundant and irrelevant features are present. Redundant

features duplicate much or all of the information contained in one or more

other attributes. For example, the purchase price of a product and the amount

of sales tax paid contain much of the same information. Irrelevant features

Oontain almost no useful information for the data mining task at hand. For

instance, students' ID numbers are irrelevant to the task of predicting stu-

dents' grade point averages. Redundant and irrelevant features can reduce

classification accuracy and the quality of the clusters that are found.

While some irrelevant and redundant attributes can be eliminated imme-

diately by using common sense oI domain knowledge, selecting the best subset

of features frequently requires a systematic approach. The ideal approach to

feature selection is to try all possible subsets of features as input to the data

mining aigorithm of interest, and then take the subset that produces the best

results. This method has the advantage of reflecting the objective and bias of

the data mining algorithm that will eventually be used. Unfortunately, since

the number of subsets involving n attributes is 2n, such an approach is imprac-

tical in most situations and alternative strategies are needed. There are three

standard approaches to feature selection: embedded, filter, and wrapper.

Data Preprocessing 53

Embedded approaches Feature selection occurs naturally as part of the data mining algorithm. Specifically, during the operation of the data mining algorithm, the algorithm itself decides which attributes to use and which to ignore. Algorithms for building decision tree classifiers, which are discussed in Chapter 4, often operate in this manner.

Filter approaches Features are selected before the data mining algorithm is run, using some approach that is independent of the data mining task. For example, we might select sets of attributes whose pairwise correlation is as low as possible.

Wrapper approaches These methods use the target data mining algorithm as a black box to find the best subset of attributes, in a way similar to that of the ideal algorithm described above, but typically without enumerating all possible subsets.

Since the embedded approaches are algorithm-specific, only the filter and wrapper approaches will be discussed further here.

An Architecture for Feature Subset Selection

It is possible to encompass both the filter and wrapper approaches within a common architecture. The feature selection process is viewed as consisting of four parts: a measure for evaluating a subset, a search strategy that controls the generation of a new subset of features, a stopping criterion, and a valida- tion procedure. Filter methods and wrapper methods differ only in the way in which they evaluate a subset of features. For a wrapper method, subset evaluation uses the target data mining algorithm, while for a filter approach, the evaluation technique is distinct from the target data mining algorithm. The following discussion provides some details of this approach, which is sum- marized in Figure 2.11.

Conceptually, feature subset selection is a search over all possible subsets of features. Many different types of search strategies can be used, but the search strategy should be computationally inexpensive and should find optimal or near optimal sets of features. It is usually not possible to satisfy both requirements, and thus, tradeoffs are necessary.

An integral part ofthe search is an evaluation step tojudge how the current subset of features compares to others that have been considered. This requires an evaluation measure that attempts to determine the goodness of a subset of attributes with respect to a particular data mining task, such as classification

2 . 3

54 Chapter 2 Data

Figure 2,11. Flowchart of a feature subset selection process.

or clustering. For the filter approach, such measures attempt to predict how

well the actual data mining algorithm will perform on a given set of attributes. For the wrapper approach, where evaluation consists of actually running the

target data mining application, the subset evaluation function is simply the

criterion normally used to measure the result of the data mining. Because the number of subsets can be enormous and it is impractical to

examine them all, some sort of stopping criterion is necessary. This strategy is

usualiy based on one or more conditions involving the following: the number

of iterations, whether the value of the subset evaluation measure is optimal or

exceeds a certain threshold, whether a subset of a certain size has been ob-

tained, whether simultaneous size and evaluation criteria have been achieved, and whether any improvement can be achieved by the options available to the

search strategy. Finally, once a subset of features has been selected, the results of the

target data mining algorithm on the selected subset should be validated. A

straightforward evaluation approach is to run the algorithm with the full set

of features and compare the full results to results obtained using the subset of

features. Hopefully, the subset of features will produce results that are better

than or almost as good as those produced when using all features. Another validation approach is to use a number of different feature selection algorithms to obtain subsets of features and then compare the results of running the data

mining algorithm on each subset.

2.3 Data Preprocessing 55

Feature Weighting

Feature weighting is an alternative to keeping or eliminating features. More important features are assigned a higher weight, while less important features are given a lower weight. These weights are sometimes assigned based on do- main knowledge about the relative importance of features. Alternatively, they may be determined automatically. For example, some classification schemes, such as support vector machines (Chapter 5), produce classification models in which each feature is given a weight. Features with larger weights play a more important role in the model. The normalization of objects that takes place when computing the cosine similarity (Section 2.4.5) can also be regarded as a type of feature weighting.

2.3.5 Feature Creation

It is frequently possible to create, from the original attributes, a new set of attributes that captures the important information in a data set much more effectively. Furthermore, the number of new attributes can be smaller than the number of original attributes, allowing us to reap all the previously described benefits of dimensionality reduction. Three related methodologies for creating new attributes are described next: feature extraction, mapping the data to a new space, and feature construction.

Feature Extraction

The creation of a new set of features from the original raw data is known as feature extraction. Consider a set of photographs, where each photograph is to be classified according to whether or not it contains a human face. The raw data is a set of pixels, and as such, is not suitable for many types of classification algorithms. However, if the data is processed to provide higher- level features, such as the presence or absence of certain types of edges and areas that are highly correlated with the presence of human faces, then a much broader set of classification techniques can be applied to this problem.

Unfortunately, in the sense in which it is most commonly used, feature extraction is highly domain-specific. For a particular field, such as image processing, various features and the techniques to extract them have been developed over a period of time, and often these techniques have limited ap- plicability to other fields. Consequently, whenever data mining is applied to a relatively new area, a key task is the development of new features and feature extraction methods.

56 Chapter 2 Data

(a) Two time series. (b) Noisy time series. (c) Power spectrum

Figure 2.12. Application of the Fourier transform to identify the underlying frequencies in time series

data.

Mapping the Data to a New Space

A totally different view of the data can reveal important and interesting fea-

tures. Consider, for example, time series data, which often contains periodic

patterns. If there is only a single periodic pattern and not much noise' then

the pattern is easily detected. If, on the other hand, there are a number of periodic patterns and a significant amount of noise is present, then these pat-

terns are hard to detect. Such patterns can, nonetheless, often be detected

by applying a Fourier transform to the time series in order to change to a

representation in which frequency information is explicit. In the example that

follows, it will not be necessary to know the details of the Fourier transform.

It is enough to know that, for each time series, the Fourier transform produces

a new data object whose attributes are related to frequencies.

Example 2.10 (Fourier Analysis). The time series presented in Figure

2.I2(b) is the sum of three other time series, two of which are shown in Figure

2.12(a) and have frequencies of 7 and 17 cycles per second, respectively. The

third time series is random noise. Figure 2.12(c) shows the power spectrum

that can be computed after applying a Fourier transform to the original time

series. (Informally, the power spectrum is proportional to the square of each

frequency attribute.) In spite ofthe noise, there are two peaks that correspond to the periods of the two original, non-noisy time series. Again, the main point

is that better features can reveal important aspects of the data. I

2.3 Data Preprocessing 57

Many other sorts of transformations are also possible. Besides the Fourier transform, the wavelet transform has also proven very useful for time series and other types of data.

Feature Construction

Sometimes the features in the original data sets have the necessary information, but it is not in a form suitable for the data mining algorithm. In this situation, one or more new features constructed out of the original features can be more useful than the original features.

Example 2.11- (Density). To illustrate this, consider a data set consisting of information about historical artifacts, which, along with other information, contains the volume and mass of each artifact. For simplicity, assume that these artifacts are made of a small number of materials (wood, clay, bronze, gold) and that we want to classify the artifacts with respect to the material of which they are made. In this case, a density feature constructed from the mass and volume features, i.e., density : mass/uolume, would most directly yield an accurate classification. Although there have been some attempts to automatically perform feature construction by exploring simple mathematical combinations of existing attributes, the most common approach is to construct features using domain expertise.

2.3.6 Discretization and Binarization

Some data mining algorithms, especially certain classification algorithms, re- quire that the data be in the form of categorical attributes. Algorithms that find association patterns require that the data be in the form of binary at- tributes. Thus, it is often necessary to transform a continuous attribute into a categorical attribute (discretization), and both continuous and discrete attributes may need to be transformed into one or more binary attributes (binarization). Additionally, if a categorical attribute has a large number of values (categories), or some values occur infrequently, then it may be beneficial for certain data mining tasks to reduce the number of categories by combining some of the values.

As with feature selection, the best discretization and binarization approach is the one that "produces the best result for the data mining algorithm that will be used to analyze the data." It is typically not practical to apply such a criterion directly. Consequently, discretization or binarization is performed in

Categorical Value Integer Value :x7 u 5 tr4 awtuL poor OK good great

0 I 2 3 4

1 0 0 0 0

U 1 0 0 0

0 0 1 0 0

0 0 0 1 0

0 0 0 0 1

58 Chapter 2 Data

Table 2.5. Conversion of a categorical attribute to three binary attributes.

Catesorical Value Integer Value f i 1 T a

awful poor OK good great

0 I 2 3 4

0 0 0 0 1

0 0 1 1 0

0 1 0 1 0

Table 2.6. Conversion of a categorical attribute to five asymmetric binary attributes.

a way that satisfies a criterion that is thought to have a relationship to good

performance for the data mining task being considered.

Binarization

A simple technique to binarize a categorical attribute is the following: If there

are rn categorical values, then uniquely assign each original value to an integer

in the interval [0,rn - 1]. If the attribute is ordinal, then order must be

maintained by the assignment. (Note that even if the attribute is originally

represented using integers, this process is necessary ifthe integers are not in the

interval 10, ̂ - 1] . ) Next, convert each of these m integers to a binary number.

Since n : flogr(m)l binary digits are required to represent these integers,

represent these binary numbers using n binary attributes. To illustrate, a

categorical variable with 5 values {awful, poor, OK, good, great} would require

three binary variables frrt fr2t and 23. The conversion is shown in Table 2.5.

Such a transformation can cause complications, such as creating unin-

tended relationships among the transformed attributes. For example, in Table

2.5, attributes 12 and 13 are correlated because information about the good

value is encoded using both attributes. Furthermore, association analysis re-

quires asymmetric binary attributes, where only the presence of the attribute (value : 1) is important. For association problems, it is therefore necessary to

introduce one binary attribute for each categorical value) as in Table 2.6. Ifthe

2.3 Data Preprocessing 59

number of resulting attributes is too large, then the techniques described below can be used to reduce the number of categorical values before binarization.

Likewise, for association problems, it may be necessary to replace a single binary attribute with two asymmetric binary attributes. Consider a binary attribute that records a person's gender, male or female. For traditional as- sociation rule algorithms, this information needs to be transformed into two asymmetric binary attributes, one that is a 1 only when the person is male and one that is a 1 only when the person is female. (For asymmetric binary attributes, the information representation is somewhat inefficient in that two bits of storage are required to represent each bit of information.)

Discretization of Continuous Attributes

Discretization is typically applied to attributes that are used in classification or association analysis. In general, the best discretization depends on the algo- rithm being used, as well as the other attributes being considered. Typically, however, the discretization of an attribute is considered in isolation.

tansformation of a continuous attribute to a categorical attribute involves two subtasks: deciding how many categories to have and determining how to map the values ofthe continuous attribute to these categories. In the first step, after the values of the continuous attribute are sorted, they are then divided into n intervals by specifying n - r split points. In the second, rather trivial step, all the values in one interval are mapped to the same categorical value. Therefore, the problem of discretization is one of deciding how many split points to choose and where to place them. The result can be represented either as a set of intervals {(ro, ,t], (*t, ,zl, . . . , (*n-t, r",) }, wher e ro and rn may be *oo or -oo, respectively, or equivalently, as a series of inequalities : t g 1 i t 1 I I t . . . t I n - I 1 r 1 r n .

IJnsupervised Discretization A basic distinction between discretization methods for classification is whether class information is used (supervised) or not (unsupervised). If class information is not used, then relatively simple approaches are common. For instance, the equal width approach divides the range of the attribute into a user-specified number of intervals each having the same width. Such an approach can be badly affected by outliers, and for that reason, an equal frequency (equal depth) approach, which tries to put the same number of objects into each interval, is often preferred. As another example of unsupervised discretization, a clustering method, such as K-means (see Chapter 8), can also be used. Finally, visually inspecting the data can sometimes be an effective approach.

60 Chapter 2 Data

Example 2.12 (Discretization Techniques). This example demonstrates how these approaches work on an actual data set. Figure 2.13(a) shows data points belonging to four different groups, along with two outliers-the large

dots on either end. The techniques of the previous paragraph were applied

to discretize the r values of these data points into four categorical values. (Points in the data set have a random g component to make it easy to see

how many points are in each group.) Visually inspecting the data works quite

well, but is not automatic, and thus, we focus on the other three approaches. The split points produced by the techniques equal width, equal frequency, and

K-means are shown in Figures 2.13(b), 2.13(c), and 2.13(d), respectively. The

split points are represented as dashed lines. If we measure the performance of

a discretization technique by the extent to which different objects in different groups are assigned the same categorical value, then K-means performs best, followed by equal frequency, and finally, equal width. I

Supervised Discretization The discretization methods described above

are usually better than no discretization, but keeping the end purpose in mind

and using additional information (class labels) often produces better results. This should not be surprising, since an interval constructed with no knowledge

of class labels often contains a mixture of class labels. A conceptually simple approach is to place the splits in a way that maximizes the purity of the intervals. In practice, however, such an approach requires potentially arbitrary decisions about the purity of an interval and the minimum size of an interval. To overcome such concerns, some statistically based approaches start with each attribute value as a separate interval and create larger intervals by merging adjacent intervals that are similar according to a statistical test. Entropy- based approaches are one of the most promising approaches to discretization, and a simple approach based on entropy will be presented.

First, it is necessary to define entropy. Let k be the number of different

class labels, mibe the number of values in the eth interval of a partition, and

mai be the number of values of class j in interval i. Then the entropy ei of the 'ith interval is given by the equation

k

e r : l p i i l o g z p i j , i : l

where pq : m4 fm2 is the probability (fraction of values) of class j in the i'th interval. The total entropy, e, of the partition is the weighted average of the individual interval entropies, i.e.,

2 . 3

:1. t r .

Data Preprocessing 61,

t

s- 3'

t

. r i I

! , . 3

l .

. ; . !

.ti i . .

i..

r ; .

'l

? r

2 . 1

1 0 1 5 1 0 1 5 20

(a) Original data. (b) Equal width discretization.

i :-i i *| ' . . . I . i .

i .'T

i ,i. : i.' : g

| . , ' I . . .

! : .

i i,:, 1 0 1 5 20

(c) Equal frequency discretization. (d) K-means discretization.

Figure 2.1 3. Different discretization techniques.

" - \ -" : / , = i t o ' o '

where rn is the number of values, wi : mif m is the fraction of values in the ith interval. and n is the number of intervals. Intuitively, the entropy of an interval is a measure of the purity of an interval. If an interval contains only values ofone class (is perfectly pure), then the entropy is 0 and it contributes

62 Chapter 2 Data

nothing to the overall entropy. If the classes of values in an interval occur

equally often (the interval is as impure as possible), then the entropy is a

maximum. A simple approach for partitioning a continuous attribute starts by bisect-

ing the initial values so that the resulting two intervals give minimum entropy.

This technique only needs to consider each value as a possible split point, be-

cause it is assumed that intervals contain ordered sets of values. The splitting process is then repeated with another interval, typically choosing the interval

with the worst (highest) entropy, until a user-specified number of intervals is

reached, or a stopping criterion is satisfied.

Example 2.13 (Discretization of Two Attributes). This method was

used to independently discretize both the z and y attributes of the two-

dimensional data shown in Figure 2.I4. In the first discretization, shown in

Figure 2.14(a), the r and g attributes were both split into three intervals. (The

dashed lines indicate the split points.) In the second discretization, shown in

Figure 2.L4(b), the r and gr attributes were both split into five intervals. I

This simple example illustrates two aspects of discretization. First, in two

dimensions, the classes of points are well separated, but in one dimension, this

is not so. In general, discretizing each attribute separately often guarantees

suboptimal results. Second, five intervals work better than three, but six

intervals do not improve the discretization much, at least in terms of entropy. (Entropy values and results for six intervals are not shown.) Consequently, it is desirable to have a stopping criterion that automatically finds the right

number of partitions.

Categorical Attributes with Too Many Values

Categorical attributes can sometimes have too many values. If the categorical attribute is an ordinal attribute, then techniques similar to those for con-

tinuous attributes can be used to reduce the number of categories. If the categorical attribute is nominal, however, then other approaches are needed.

Consider a university that has a large number of departments. Consequently, a department name attribute might have dozens of different values. In this

situation, we could use our knowledge of the relationships among different departments to combine departments into larger groups, such as eng'ineering, soc'ial sciences, or biological sc'iences. If domain knowledge does not serve as

a useful guide or such an approach results in poor classification performance,

then it is necessary to use a more empirical approach, such as grouping values

Data Preprocessing 63

(a) Three intervals

Figure 2.14. Discretizing r and y attributes for four groups (classes) of points.

together only if such a grouping results in improved classification accuracy or achieves some other data mining objective.

2.3.7 Variable Tlansformation

A variable transformation refers to a transformation that is applied to all the values of a variable. (We use the term variable instead of attribute to ad- here to common usage, although we will also refer to attribute transformation on occasion.) In other words, for each object, the transformation is applied to the value of the variable for that object. For example, if only the magnitude of a variable is important, then the values of the variable can be transformed by taking the absolute value. In the following section, we discuss two impor- tant types of variable transformations: simple functional transformations and normalization.

Simple Functions

For this type of variable transformation, a simple mathematical function is applied to each value individually. If r is a variable, then examples of such transformations include rk, log tr, e', yE,If n, sinr, or lrl. In statistics, vari- able transformations, especially sqrt, log , and 7 f r , are often used to transform data that does not have a Gaussian (normal) distribution into data that does. While this can be important, other reasons often take precedence in data min-

2 . 3

- - - - -xg : :""" *S.;t-

- - - - - :"-,$

64 Chapter 2 Data

ing. Suppose the variable of interest is the number of data bytes in a session)

and the number of bytes ranges from 1 to 1 billion. This is a huge range, and

it may be advantageous to compress it by using a lo916 transformation. In

this case, sessions that transferred 108 and 10e bytes would be more similar

to each other than sessions that transferred 10 and 1000 bytes (9 - 8 : 1

versus 3 - 1 : 2). For some applications, such as network intrusion detection,

this may be what is desired, since the first two sessions most likely represent

transfers of large files, while the latter two sessions could be two quite distinct

types of sessions. Variable transformations should be applied with caution since they change

the nature of the data. While this is what is desired, there can be problems

if the nature of the transformation is not fully appreciated. For instance, the

transformation If r reduces the magnitude of values that are 1 or larger, but

increases the magnitude of values between 0 and 1. To illustrate, the values

{ L , 2 , 3 } g o t o { 1 , + , + } , b u t t h e v a l u e s { 1 , ; , + } g o t o { 1 , 2 , 3 } . T h u s , f o r all sets of values, the transformation If r reverces the order. To help clarify

the effect of a transformation, it is important to ask questions such as the

following: Does the order need to be maintained? Does the transformation apply to all values, especially negative values and 0? What is the effect of

the transformation on the values between 0 and 1? Exercise 17 on page 92

explores other aspects of variable transformation.

Normalization or Standardization

Another common type of variable transformation is the standardization or

normalization of a variable. (In the data mining community the terms are

often used interchangeably. In statistics, however, the term normalization can

be confused with the transformations used for making a variable normal, i.e',

Gaussian.) The goal of standardization or normalization is to make an en-

tire set of values have a particular property. A traditional example is that

of "standardizing a variable" in statistics. If 7 is the mean (average) of the

attribute values and s, is their standard deviation, then the transformation r' : (r -?)lt" creates a new variable that has a mean of 0 and a standard deviation of 1. If different variables are to be combined in some way, then

such a transformation is often necessary to avoid having a variable with large

values dominate the results of the calculation. To illustrate, consider compar- ing people based on two variables: age and income. For any two people, the

difference in income will likely be much higher in absolute terms (hundreds or

thousands of dollars) than the difference in age (less than 150). If the differ- ences in the range ofvalues ofage and income are not taken into account, then

Measures of Similaritv and Dissimilaritv 65

the comparison between people will be dominated by differences in income. In particular, if the similarity or dissimilarity of two people is calculated using the similarity or dissimilarity measures defined later in this chapter, then in many cases, such as that of Euclidean distance, the income values will dominate the calculation.

The mean and standard deviation are strongly affected by outliers, so the above transformation is often modified. First, the mean is replaced by the median, i.e., the middle value. Second, the standard deviation is replaced by the absolute standard deviation. Specifically, if r is a variable, then the absolute standard deviation of r is given by oa : Dlrl*n - ltl, where ri is lhe 'ith value of the variable, rn is the number of objects, and. p, is either the mean or median. Other approaches for computing estimates of the location (center) and spread of a set of values in the presence of outliers are described in Sections 3.2.3 and 3.2.4, respectively. These measures can also be used to define a standardi zation transformation.

2.4 Measures of Similarity and Dissimilarity

Similarity and dissimilarity are important because they are used by a number of data mining techniques, such as clustering, nearest neighbor classification, and anomaly detection. In many cases) the initial data set is not needed once these similarities or dissimilarities have been computed. Such approaches can be viewed as transforming the data to a similarity (dissimilarity) space and then performing the analysis.

We begin with a discussion of the basics: high-level definitions of similarity and dissimilarity, and a discussion of how they are related. For convenience, the term proximity is used to refer to either similarity or dissimilarity. Since the proximity between two objects is a function of the proximity between the corresponding attributes of the two objects, we first describe how to measure the proximity between objects having only one simple attribute, and then consider proximity measures for objects with multiple attributes. This in- cludes measures such as correlation and Euclidean distance, which are useful for dense data such as time series or two-dimensional points, as well as the Jaccard and cosine similarity measures, which are useful for sparse data like documents. Next, we consider several important issues concerning proximity measures. The section concludes with a brief discussion of how to select the right proximity measure.

2 . 4

66 Chapter 2 Data

2.4.L Basics

Definitions

Informally, the similarity between two objects is a numerical measure of the

degree to which the two objects are alike. Consequently, similarities are hi,gher for pairs of objects that are more alike. Similarities are usually non-negative and are often between 0 (no similarity) and 1 (complete similarity).

The dissimilarity between two objects is a numerical measure of the de- gree to which the two objects are different. Dissimilarities are lower for more similar pairs of objects. FYequently, the term distance is used as a synonym for dissimilarity, although, as we shall see, distance is often used to refer to a special class of dissimilarities. Dissimilarities sometimes fall in the interval

[0,1], but it is also common for them to range from 0 to oo.

TYansformations

Thansformations are often applied to convert a similarity to a dissimilarity, or vice versa, or to transform a proximity measure to fall within a particular

range, such as [0,1]. For instance, we may have similarities that range from 1 to 10, but the particular algorithm or software package that we want to use may be designed to only work with dissimilarities, or it may only work with similarities in the interval 10,1]. We discuss these issues here because we will employ such transformations later in our discussion of proximity. In addi- tion, these issues are relatively independent of the details of specific proximity

measures. Frequently, proximity measures, especially similarities, are defined or trans-

formed to have values in the interval [0,1]. Informally, the motivation for this is to use a scale in which a proximity value indicates the fraction of similarity (or dissimilarity) between two objects. Such a transformation is often rela- tively straightforward. For example, if the similarities between objects range from 1 (not at all similar) to 10 (completely similar), we can make them fall within the range 10, 1] by using the transformation s' : (s

- 1)/9, where s and s/ are the original and new similarity values, respectively. In the more general case, the transformation of similarities to the interval [0,1] is given by the expression

"' : (" - mi,n-s) I (mar -s - mi,n-s) , where mar -s and m'in-s are the

maximum and minimum similarity values, respectively. Likewise, dissimilarity measures with a finite range can be mapped to the interval [0,1] by using the formula d' : (d - rni,n-d)l(mar-d - mi,n-d).

There can be various complications in mapping proximity measures to the interval 10,1], however. If, for example, the proximity measure originally takes

Measures of Similaritv and Dissimilaritv 67

values in the interval [0,-], then a non-linear transformation is needed and values will not have the same relationship to one another on the new scale. Consider the transformation d,' : dl(I* d) for a dissimilarity measure that ranges from 0 to oo. The dissimilarities 0, 0.5, 2, 10, 100, and 1000 will be transformed into the new dissimilarities 0, 0.33, 0.67, 0.90, 0.99, and 0.999, respectively. Larger values on the original dissimilarity scale are compressed into the range of values near 1, but whether or not this is desirable depends on the application. Another complication is that the meaning of the proximity measure may be changed. For example, correlation, which is discussed later, is a measure of similarity that takes values in the interval [-1,1]. Mapping these values to the interval [0,1] by taking the absolute value loses information about the sign, which can be important in some applications. See Exercise 22 on page 94.

Tlansforming similarities to dissimilarities and vice versa is also relatively straightforward, although we again face the issues of preserving meaning and changing a linear scale into a non-linear scale. If the similarity (or dissimilar- ity) falls in the interval [0,1], then the dissimilarity can be defined as d : 1- s (s : 1 - d). Another simple approach is to define similarity as the nega- tive of the dissimilarity (or vice versa). To illustrate, the dissimilarities 0, 1, 10, and 100 can be transformed into the similarities 0, -1, -10, and -100,

respectively. The similarities resulting from the negation transformation are not re-

stricted to the range [0,1], but if that is desired, then transformations such as

" :

;{1, s : e-d, or s : 1- *mZ can be used. For the transformation s : 7fi, the dissimilarities 0, 1, 10, 100 are transformed into 1, 0.5, 0.09, 0.01, respectively. For s : e-d, they become 1.00, 0.37, 0.00, 0.00, respectively, while for s: 1- *ffin they become 1.00, 0.99, 0.00, 0.00, respectively. In this discussion, we have focused on converting dissimilarities to similarities. Conversion in the opposite direction is considered in Exercise 23 on page 94.

In general, any monotonic decreasing function can be used to convert dis- similarities to similarities, or vice versa. Of course, other factors also must be considered when transforming similarities to dissimilarities, or vice versa, or when transforming the values of a proximity measure to a new scale. We have mentioned issues related to preserving meaning, distortion of scale, and requirements of data analysis tools, but this list is certainly not exhaustive.

2.4.2 Similarity and Dissimilarity between Simple Attributes

The proximity of objects with a number of attributes is typically defined by combining the proximities of individual attributes, and thus, we first discuss

2 . 4

68 Chapter 2 Data

proximity between objects having a single attribute. Consider objects de-

scribed by one nominal attribute. What would it mean for two such objects

to be similar? Since nominal attributes only convey information about the

distinctness of objects, all we can say is that two objects either have the same value or they do not. Hence, in this case similarity is traditionally defined as 1

if attribute values match, and as 0 otherwise. A dissimilarity would be defined

in the opposite way: 0 if the attribute values match, and 1 if they do not. For objects with a single ordinal attribute, the situation is more compli-

cated because information about order should be taken into account. Consider an attribute that measures the quality of a product, €.8., a candy bar, on the

scale {poor, fai,r, OK, good, wonderful).It would seem reasonable that a prod-

uct, P1, which is rated wonderful, would be closer to a produclP2, which is

rated good, than it would be to a product P3, which is rated OK. To make this observation quantitative, the values of the ordinal attribute are often mapped

to successive integers, beginning at 0 or 1, e.g., {poor:O, fair:|, OK:2, good:3, wonderful:4). Then, d(Pl,P2) - 3 - 2 : 7 or, if we want the dis-

similarity to fall between 0 and 1, d(P1, P2) : TZ :0.25. A similarity for

ordinal attributes can then be defined as s : 7 - d. This definition of similarity (dissimilarity) for an ordinal attribute should

make the reader a bit uneasy since this assumes equal intervals, and this is not

so. Otherwise, we would have an interval or ratio attribute. Is the difference between the values fair and good really the same as that between the values

OK and wonderful? Probably not, but in practice, our options are limited, and in the absence of more information, this is the standard approach for defining proximity between ordinal attributes.

For interval or ratio attributes, the natural measure of dissimilarity be- tween two objects is the absolute difference of their values. For example, we might compare our current weight and our weight a year ago by saying "I am

ten pounds heavier." In cases such as these, the dissimilarities typically range from 0 to oo, rather than from 0 to 1. The similarity of interval or ratio at- tributes is typically expressed by transforming a similarity into a dissimilarity, as previously described.

Table 2.7 summarizes this discussion. In this table, r and g are two objects that have one attribute of the indicated type. AIso, d(r,a) and s(r,gr) are the dissimilarity and similarity between r and g, respectively. Other approaches are possible; these are the most common ones.

The following two sections consider more complicated measures of prox-

imity between objects that involve multiple attributes: (1) dissimilarities be- tween data objects and (2) similarities between data objects. This division

2 . 4 Measures of Similaritv and Dissimilaritv 69

allows us to more naturally display the underlying motivations for employing various proximity measures. We emphasize, however, that similarities can be transformed into dissimilarities and vice versa using the approaches described earlier.

2.4.3 Dissimilarities between Data Objects

In this section, we discuss various kinds of dissimilarities. We begin with a discussion of distances, which are dissimilarities with certain properties, and then provide examples of more general kinds of dissimilarities.

Distances

We first present some examples, and then offer a more formal description of distances in terms of the properties common to all distances. The Euclidean distance, d, between two points, x and y, in one-, two-, three-, or higher- dimensional space, is given by the following familiar formula:

d ( * , y ) : n

\ - r \ t ) \ t n

- a k ) ' , K = l

( 2 . 1 )

where n is the number of dimensions and rp and.Ak are) respectively,the kth attributes (components) of r and g. We illustrate this formula with Figure 2.15 and Tables 2.8 and 2.9, which show a set of points, the e and gr coordinates of these points, and the distance matrix containing the pairwise distances of these points.

Table 2.7 . Similarity and dissimilarity for simple attributes

Attribute T'ype

Dissimilarity Similaritv

Nominal ) _11, - 0 1 f r : A I i f r l y

s : I \ f r : y 0 i f n l y

Ordinal d : l r - a l l ( " - t ) (values mapped to integers 0 to n-1 where n is the number of values)

s : I - d

Interval or Ratio d : l r - a l s : - d , s : ; i , s : e - o ,' L t a

^ - 1 d - m i n - d-

70 Chapter 2 Data

The Euclidean distance measure given the Minkowski distance metric shown in

in Equation 2.1 is generalized Equation 2.2,

by

d ( x , y ) : ( , , \

where r is a parameter. The following are the three most common examples of Minkowski distances.

. r :1. City block (Manhattan, taxicab, L1 norm) distance. A common example is the Hamming distance, which is the number of bits that are different between two objects that have only binary attributes, i.e., between two binary vectors.

o r :2. Euclidean distance (L2 norm).

. r : oo. Supremum (L*o, or L- norm) distance. This is the maximum difference between any attribute of the objects. More formally, the L- distance is defined by Equation 2.3

( 2 . 3 )

(Lor-orr)''"

J* (U'wr-rrt')"'d ( * , Y ) :

The r parameter should not be confused with the number of dimensions (at- tributes) n. The Euclidean, Manhattan, and supremum distances are defined for all values of n: I,2,3,..., and specify different ways of combining the differences in each dimension (attribute) into an overall distance.

Tables 2.10 and 2.11, respectively, give the proximity matrices for the L1 and Loo distances using data from Table 2.8. Notice that all these distance matrices are symmetric; i.e., the ijth entry is the same as the jith entry. In Table 2.9, for instance, the fourth row of the first column and the fourth column of the first row both contain the value 5.1.

Distances, such as the Euclidean distance, have some well-known proper- ties. If d(*, y) is the distance between two points, x and y, then the following properties hold.

1. Positivity

(a) d(x, x) > 0 for all x and y,

(b) d(x, Y) : 0 onlY if x : Y.

Measures of Similaritv and Dissimilaritv 7L

Figure 2.15. Four two-dimensional points.

Tabfe 2.8. r and y coordinates of four points. Table 2.9. Euclidean distance matrix for Table 2.g. point z coordinate y coordinate

p 1 0 2 p2 2 0 p3 .) 1 p4 1

Table 2.10. L1 distance matrix for Table 2.8. Table 2,11. L- distance matrix for Table 2.8.

2. Symmetry d ( * , Y ) : d ( Y , x ) f o r a l l x a n d Y .

3. T[iangle Inequality d(x,z) < d(*, y) + d(y, z) for all points x, y,, and z.

Measures that satisfy all three properties are known as metrics. Some people only use the term distance for dissimilarity measures that satisfy these properties, but that practice is often violated. The three properties described here are useful, as well as mathematically pleasing. AIso, if the triangle in- equality holds, then this property can be used to increase the efficiency of tech- niques (including clustering) that depend on distances possessing this property. (See Exercise 25.) Nonetheless, many dissimilarities do not satisfy one or more of the metric properties. We give two examples of such measures.

2 . 4

1 2 3 4 5 6 X

p l p2 p3 p4 p l 0 . 0 2 . 8 3 . 2 5 . 1 p2 2 . 8 0 . 0 t . 4 3 . 2 p3 3 . 2 L . 4 0 . 0 2 . 0 p4 c . r 3 . 2 2 . 0 0 . 0

L 1 p 1 p2 p3 p4 p l 0 . 0 4 . 0 4 . 0 6 . 0 p2 4 . 0 0 . 0 2 . 0 4 . 0 p3 4 . 0 2 . 0 0 . 0 2 . 0 p4 6 . 0 4 . 0 2 . 0 0 . 0

L- p l p2 p3 p4 p1 0 . 0 2 . 0 3 . 0 5 . 0 p2 2 . 0 0 . 0 1 . 0 3 . 0 p3 3 . 0 1 . 0 0 . 0 2 . 0 p4 5 . 0 3 . 0 2 . 0 0 . 0

72 Chapter 2 Data

Example 2.L4 (Non-metric Dissimilarities: Set Differences). This ex- ample is based on the notion of the difference of two sets, as defined in set theory. Given two sets -4 and B, A - B is the set of elements of A that are n o t i n B . F o r e x a m p l e , i f A : { I , 2 , 3 , 4 } a n d B : { 2 , 3 , 4 } , t h e n A - B : { 1 } and B - A - 0, the empty set. We can define the distance d between two sets A and B as d(A,B): size(A- B), where s'ize ts a function returning the number of elements in a set. This distance measure, which is an integer value greater than or equal to 0, does not satisfy the second part of the pos- itivity property the symmetry property, or the triangle inequality. However, these properties can be made to hold if the dissimilarity measure is modified a s f o l l o w s : d ( A , B ) : s i z e ( A - B ) + s i , z e ( B - A ) . S e e E x e r c i s e 2 1 o n p a g e 94. r

Example 2.15 (Non-metric Dissimilarities: Time). This example gives

a more everyday example of a dissimilarity measure that is not a metric, but that is still useful. Define a measure of the distance between times of the day as follows:

d,(t1,t2) : {7^ *t,az _ tr) li ll i I:}

To illustrate, d(lPM, 2PM) : t hour, while d(2PM, 1PM) : 23 hours. Such a definition would make sense, for example, when answering the question:

"ff an event occurs at lPM every day, and it is now 2PM, how Iong do I have to wait for that event to occur again?"

2.4.4 Similarities between Data Objects

For similarities, the triangle inequality (or the analogous property) typically does not hold, but symmetry and positivity typically do. To be explicit, if s(x, y) is the similarity between points x and y, then the typical properties of similarities are the following:

1 . s ( x , y ) : 1 o n l y i f x : y . ( 0 < s S 1 )

2. s(x,y) : s(y, x) for all x and y. (Symmetry)

There is no general analog of the triangle inequality for similarity mea- sures. It is sometimes possible, however, to show that a similarity measure can easily be converted to a metric distance. The cosine and Jaccard similarity measures, which are discussed shortly, are two examples. Also, for specific sim- ilarity measures, it is possible to derive mathematical bounds on the similarity between two objects that are similar in spirit to the triangle inequality.

( 2 . 4 )

Measures of Similarity and 73

Example 2.L6 (A Non-symmetric Similarity Measure). Consider an experiment in which people are asked to classify a small set of characters as they flash on a screen. The confusion matrix for this experiment records how often each character is classified as itself, and how often each is classified as another character. For instance, suppose that "0" appeared 200 times and was classified as a "0" 160 times, but as an "o" 40 times. Likewise, suppose that 'o' appeared 200 times and was classified as an "o" 170 times, but as "0" only 30 times. If we take these counts as a measure of the similarity between two characters, then we have a similarity measure, but one that is not symmetric. In such situations, the similarity measure is often made symmetric by setting s'(x, y) : s'(y, x) : (s(x, y)+ s(y,

"D 12, where s/ indicates the new similarity

measure,

2 . 4 . 5

This section provides specific examples of some similarity and dissimilarity measures.

Similarity Measures for Binary Data

Similarity measures between objects that contain only binary attributes are called similarity coefficients, and typically have values between 0 and 1. A value of 1 indicates that the two objects are completely similar, while a value of 0 indicates that the objects are not at all similar. There are many rationales for why one coefificient is better than another in specific instances.

Let x and y be two objects that consist of n binary attributes. The com- parison of two such objects, i.e., two binary vectors, Ieads to the following four quantities (frequencies) :

.foo : the number of attributes where x is 0 and y is 0

.for : the number of attributes where x is 0 and y is 1

,fio : the number of attributes where x is 1 and y is 0

"fir : the number of attributes where x is 1 and y is 1

Simple Matching Coefficient One commonly used similarity coefficient is the simple matching coefficient (SMC), which is defined as

number of matching attribute values ft * fss

2 . 4

Examples of Proximity Measures

S M C : number of attributes for -l fn * ,frr * ,foo'

( 2 . 5 )

74 Chapter 2 Data

This measure counts both presences and absences equally. Consequently, the SMC could be used to find students who had answered questions similarly on a test that consisted only of true/false questions.

Jaccard Coefficient Suppose that x and y are data objects that represent two rows (two transactions) of a transaction matrix (see Section 2.1.2). If each asymmetric binary attribute corresponds to an item in a store, then a 1 indi- cates that the item was purchased, while a 0 indicates that the product was not purchased. Sincb the number of products not purchased by any customer far outnumbers the number of products that were purchased, a similarity measure such as SMC would say that all transactions are very similar. As a result, the Jaccard coefficient is frequently used to handle objects consisting of asymmet- ric binary attributes. The Jaccard coefficient, which is often symbolized by ,./,'is given by the following equation:

J : number of matching presences

( 2 . 6 ) number of attributes not involved in 00 matches fot * fn -t ft

Example 2.17 (The SMC and Jaccard Similarity Coefficients). To illustrate the difference between these two similarity measures, we calculate SMC and -I for the following two binary vectors.

x : ( 1 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 ) y : ( 0 , 0 , 0 , 0 , 0 , 0 , 1 , 0 , 0 , 1 )

for :2 the number of attributes where x was 0 and y was 1

frc: I the number of attributes where x was 1 and y was 0

foo :7 the number of attributes where x was 0 and y was 0

/rr : 0 the number of attributes where x was 1 and y was 1

S M C : - ; - , = - - ; t ' - u . l

J : - T - , + - - - - ; - : - - 4 - : g J O I T J I O T J I I Z T I T U

Cosine Similarity

Documents are often represented as vectors, where each attribute represents the frequency with which a particular term (word) occurs in the document. It is more complicated than this, of course, since certain common words are ig-

J 7 I

Measures of Similaritv and Dissimilaritv 75

nored and various processing techniques are used to account for different forms of the same word, differing document lengths, and different word frequencies.

Even though documents have thousands or tens of thousands of attributes (terms), each document is sparse since it has relatively few non-zero attributes. (The normalizations used for documents do not create a non-zero entry where there was azero entry; i.e., they preserve sparsity.) Thus, as with transaction data, similarity should not depend on the number of shared 0 values since any two documents are likely to "not contain" many of the same words, and therefore, if 0-0 matches are counted, most documents will be highly similar to most other documents. Therefore, a similarity measure for documents needs to ignores 0-0 matches like the Jaccard measure, but also must be able to handle non-binary vectors. The cosine similarity, defined next, is one of the most common measure of document similarity. If x and y are two document vectors, then

cos(x,y) : ffi,

2 . 4

( , 7 \

where . indicates the vector dot product, x .y : D[:trplp, and llxll is the

length of vector x, ll*ll : 1f D|:rr2r: 1/x4.

Example 2.18 (Cosine Similarity of Two Document Vectors). This example calculates the cosine similarity for the following two data objects, which might represent document vectors:

* : ( 3 , 2 , 0 , 5 , 0 , 0 , 0 , 2 , 0 , 0 ) y : ( 1 , 0 , 0 , 0 , 0 , 0 , 0 , 1 , 0 , 2 )

x . y : 3 i . 1 * 2 x 0 * 0 * 0 * 5 x 0 * 0 x 0 f 0 * 0 * 0 * 0 * 2 * l * 0 x 0 * 0 x 2 : 5

l l x l l : r o x o * o x o * 2 x 2 * o x o * o x o : 6 . 4 8 l l y l l : : 2 . 2 4 cos(x, y) : 0.31

I

As indicated by Figure 2.16, cosine similarity really is a measure of the (cosine of the) angle between x and y. Thus, if the cosine similarity is 1, the angle between x and y is 0o, and x and y are the same except for magnitude (length). If the cosine similarity is 0, then the angle between x and y is 90o, and they do not share any terms (words).

76 Chapter 2 Data

Figule 2.16. Geometric illustration of the cosine measure.

Equation 2.7 can be written as Equation 2.8.

c o s ( x , y ) : + . , , Y , , : x ' . y ' , l l x l l l l v l l

( 2 . 8 )

where x' : x/llxll and y/ : V lllyll. Dividing x and y by their lengths normal- izes them to have a length of 1. This means that cosine similarity does not take the magnitude ofthe two data objects into account when computing similarity. (Euclidean distance might be a better choice when magnitude is important.) For vectors with a length of 1, the cosine measure can be calculated by taking a simple dot product. Consequently, when many cosine similarities between objects are being computed, normalizing the objects to have unit length can reduce the time required.

Extended Jaccard Coefficient (Tanimoto Coefficient)

The extended Jaccard coefficient can be used for document data and that re- duces to the Jaccard coefficient in the case of binary attributes. The extended Jaccard coefficient is also known as the Tanimoto coefficient. (However, there is another coefficient that is also known as the Tanimoto coefficient.) This co- efficient, which we shall represent as E J , is defined by the following equation:

E J ( x , y ) : x . y

( 2 . e ) l l " l l ' + l l v l l ' - * . v

Correlation

The correlation between two data objects that have binary or continuous vari- ables is a measure of the linear relationship between the attributes of the objects. (The calculation of correlation between attributes, which is more common, can be defined similarly.) More precisely, Pearson's correlation

Measures of Similaritv and Dissimilaritv 77

coefficient between two data objects, x and y, is defined by the following equation:

corr(x, y) : covariance(x, y) StA

( 2 . 1 0 ) standard-deviation(x) xstandard-deviation(y) tr ta'

where we are using the following standard statistical notation and definitions:

2 . 4

1 n

covariance(x,y) : s,s: :+ I("u -z)(yx -9)n - r - . E : L

( 2 . 1 1 )

standard-deviation(x) : su :

standard-deviation(y) : su:

1 f l

i : I ) - r * i s t h e m e a n o f x n 4

k : 1

1 f l

g : 1 f y * i s t h e m e a n o f y h : 1

Example 2.19 (Perfect Correlation). Correlation is always in the range -1 to 1. A correlation of 1 (-1) means that x and y have a perfect positive (negative) linear relationship; that is, 16 - aAx -f b, where a, and b are con- stants. The following two sets of values for x and y indicate cases where the correlation is -1 and *1, respectively. In the first case, the means of x and y were chosen to be 0, for simplicity.

x : ( - 3 , 6 , 0 , 3 , - 6 )

y : ( 1 , - 2 , 0 , - 7 , 2 )

x : ( 3 , 6 , 0 , 3 , 6 ) y : ( 1 , 2 , 0 , L , 2 )

I

fil@n-n)'

; \ l @ * - v t '

78 Chapter 2 Data

-1.00 -.0.90 4.70 --0.60

0.300.200 . 1 0

1 . 0 00.80

- o . 1 0

Figure 2.17. Scatter plots illustrating correlations from -1 to 1.

Example 2.20 (Non-linear Relationships). If the correlation is 0, then there is no linear relationship between the attributes of the two data objects. However, non-linear relationships may still exist. In the following example, n*: A7, but their correlation is 0.

* : ( - 3 , - 2 , - 1 , 0 , I , 2 , 3 ) Y : ( 9 , 4 , 1 , 0 , 1 , 4 , 9 )

I

Example 2.21 (Visualizing Correlation). It is also easy to judge the cor- relation between two data objects x and y by plotting pairs of corresponding attribute values. Figure 2.17 shows a number of these plots when x and y have 30 attributes and the values of these attributes are randomly generated (with a normal distribution) so that the correlation of x and y ranges from -1

to 1. Each circle in a plot represents one of the 30 attributes; its r coordinate is the value of one of the attributes for x, while its 3r coordinate is the value of the same attribute for y. I

If we transform x and y by subtracting off their means and then normaliz- ing them so that their lengths are 1, then their correlation can be calculated by

Measures of Similaritv and Dissimilaritv 79

taking the dot product. Notice that this is not the same as the standardization used in other contexts, where we make the transformations, r'* : (rp - ,) lt" and y'r : (A* - T) I sa.

Bregman Divergence* This section provides a brief description of Breg- man divergences, which are a family of proximity functions that share some common properties. As a result, it is possible to construct general data min- ing algorithms, such as clustering algorithms, that work with any Bregman divergence. A concrete example is the K-means clustering algorithm (Section

8.2). Note that this section requires knowledge of vector calculus. Bregman divergences are loss or distortion functions. To understand the

idea of a loss function, consider the following. Let x and y be two points, where y is regarded as the original point and x is some distortion or approximation of it. For example, x may be a point that was generated, for example, by adding random noise to y. The goal is to measure the resulting distortion or Ioss that results if y is approximated by x. Of course, the more similar x and y are, the smaller the loss or distortion. Thus, Bregman divergences can be used as dissimilarity functions.

More formally, we have the following definition.

Definition 2.6 (Bregman Divergence). Given a strictly convex function

@ (with a few modest restrictions that are generally satisfied), the Bregman divergence (loss function) D(x, y) generated by that function is given by the following equation:

D(*,y) : d(x) - Q0) - (Vd(v), (* - y)) (2.r2)

where Vd(V) is the gradient of / evaluated at y, x - y, is the vector difference between x and y, and (Vd(V), (" - V)) is the inner product between Vd(*) and (x-y). For points in Euclidean space, the inner product is just the dot product.

D ( * , y ) c a n b e w r i t t e n a s D ( x , V ) : d ( x ) - L ( * ) , w h e r e . L ( x ) : d $ ) + (Vd(V), (* - V)) and represents the equation of a plane that is tangent to the function Q at y. Using calculus terminology, L(x) is the linearization of. Q around the point y and the Bregman divergence is just the difference between a function and a linear approximation to that function. Different Bregman divergences are obtained by using different choices for S.

Example 2.22. We provide a concrete example using squared Euclidean dis- tance, but restrict ourselves to one dimension to simplify the mathematics. Let

2 . 4

80 Chapter 2 Data

r and y be real numbers and /(t) be the real valued function, d(t) : t2. In that case, the gradient reduces to the derivative and the dot product reduces to multiplication. Specifically, Equation 2.L2 becomes Equation 2.13.

D ( * , i l : 1 2 - A 2 - 2 A @ - A ) : @ - i l ' ( 2 . 1 3 )

The graph for this example, with 3r : 1, is shown in Figure 2.18. The Bregman divergence is shown for two values of r: r :2 and r :3. r

Figure 2.18. lllustration of Bregman divergence.

2.4.6 fssues in Proximity Calculation

This section discusses several important issues related to proximity measures: (1) how to handle the case in which attributes have different scales and/or are correlated, (2) how to calculate proximity between objects that are composed of different types of attributes, e.g., quantitative and qualitative, (3) and how to handle proximity calculation when attributes have different weights; i.e., when not all attributes contribute equally to the proximity of objects.

- 1 0 1 2 3 4 x

2.4 Measures of Similarity and Dissimilarity 81

Standardization and Correlation for Distance Measures

An important issue with distance measures is how to handle the situation when attributes do not have the same range of values. (This situation is often described by saying that "the variables have different scales.") Earlier, Euclidean distance was used to measure the distance between people based on two attributes: age and income. Unless these two attributes are standardized, the distance between two people will be dominated by income.

A related issue is how to compute distance when there is correlation be- tween some of the attributes, perhaps in addition to differences in the ranges of values. A generalization of Euclidean distance, the Mahalanobis distance, is useful when attributes are correlated, have different ranges of values (dif- ferent variances), and the distribution of the data is approximately Gaussian (normal). Specifically, the Mahalanobis distance between two objects (vectors) x and y is defined as

mahalanobis(x,y) : (x - y)>-1(x -y)r, ( 2 . 1 4 )

where E-1 is the inverse of the covariance matrix of the data. Note that the covariance matrix E is the matrix whose ijth entry is the covariance of the ith and jth attributes as defined by Equation 2.II.

Example 2.23. In Figure 2.19, there are 1000 points, whose r and g at- tributes have a correlation of 0.6. The distance between the two large points

at the opposite ends of the long axis of the ellipse is 14.7 in terms of Euclidean distance, but only 6 with respect to Mahalanobis distance. In practice, com- puting the Mahalanobis distance is expensive, but can be worthwhile for data whose attributes are correlated. If the attributes are relatively uncorrelated, but have different ranges, then standardizing the variables is sufficient.

I

Combining Similarities for fleterogeneous Attributes

The previous definitions of similarity were based on approaches that assumed all the attributes were of the same type. A general approach is needed when the attributes are of different types. One straightforward approach is to compute the similarity between each attribute separately using Table 2.7, and then combine these similarities using a method that results in a similarity between 0 and 1. Typically, the overall similarity is defined as the average of all the individual attribute similarities.

82 Chapter 2 Data

Figure 2.19. Set of two-dimensional points. The Mahalanobis distance between the two points repre- sented by large dots is 6;their Euclidean distance is 14.7.

Unfortunately, this approach does not work well if some of the attributes are asymmetric attributes. For example, if all the attributes are asymmetric binary attributes, then the similarity measure suggested previously reduces to the simple matching coefficient, a measure that is not appropriate for asym- metric binary attributes. The easiest way to fix this problem is to omit asym- metric attributes from the similarity calculation when their values are 0 for both of the objects whose similarity is being computed. A similar approach also works well for handling missing values.

In summary, Algorithm 2.7 is effective for computing an overall similar- ity between two objects, x and y, with different types of attributes. This procedure can be easily modified to work with dissimilarities.

Using Weights

In much of the previous discussion, all attributes were treated equally when computing proximity. This is not desirable when some attributes are more im- portant to the definition of proximity than others. To address these situations,

2.4 Measures of Similaritv and Dissimilaritv 83

Algorithm 2.1 Similarities of heterogeneous objects.

1-: 2 :

For the kth attribute, compute a similarity, s,r(x,y), in the range [0, 1]. Define an indicator variable, d'7., for the kth attribute as follows:

3: Compute the overall similarity between the two objects using the following for- mula:

similarity(x,") :$ffi ( 2 . 1 5 )

the formulas for proximity can each attribute.

If the weights u.r6 sum to L,

be modified by weighting the contribution of

bhen (2.15) becomes

( 2 . 1 6 )

The definition of the Minkowski distance can also be modified as follows:

d ( x , y ) : (2.r7)

2.4.7 Selecting the Right Proximity Measure

The following are a few general observations that may be helpful. First, the type of proximity measure should fit the type of data. For many types of dense, continuous data, metric distance measures such as Euclidean distance are of- ten used. Proximity between continuous attributes is most often expressed in terms of differences, and distance measures provide a well-defined way of combining these differences into an overall proximity measure. Although at- tributes can have different scales and be of differing importance, these issues can often be dealt with as described earlier.

For sparse data, which ofben consists of asymmetric attributes, we typi- cally employ similarity measures that ignore 0-0 matches. Conceptually, this reflects the fact that, for a pair of complex objects, similarity depends on the number of characteristics they both share, rather than the number of charac- teristics they both lack. More specifically, for sparse, asymmetric data, most

similarity(x, y) : D?:tl|lns !(*' Y) DT:'6x

/ n \ 1 / '

( D,'ol"r - akl'I \ t : r /

84 Chapter 2 Data

objects have only a few of the characteristics described by the attributes, and thus, are highly similar in terms of the characteristics they do not have. The cosine, Jaccard, and extended Jaccard measures are appropriate for such data.

There are other characteristics of data vectors that may need to be consid- ered. Suppose, for example, that we are interested in comparing time series. If the magnitude of the time series is important (for example, each time series represent total sales of the same organization for a different year), then we could use Euclidean distance. If the time series represent different quantities (for example, blood pressure and oxygen consumption), then we usually want to determine if the time series have the same shape, not the same magnitude. Correlation, which uses a built-in normalization that accounts for differences in magnitude and level, would be more appropriate.

In some cases, transformation or normalization of the data is important for obtaining a proper similarity measure since such transformations are not always present in proximity measures. For instance, time series may have trends or periodic patterns that significantly impact similarity. Also, a proper computation of similarity may require that time lags be taken into account. Finally, two time series may only be similar over specific periods of time. For example, there is a strong relationship between temperature and the use of natural gas, but only during the heating season.

Practical consideration can also be important. Sometimes, a one or more proximity measures are already in use in a particular field, and thus, others will have answered the question of which proximity measures should be used. Other times, the software package or clustering algorithm being used may drastically limit the choices. If efficiency is a concern, then we may want to choose a proximity measure that has a property, such as the triangle inequality, that can be used to reduce the number of proximity calculations. (See Exercise 25.)

However, if common practice or practical restrictions do not dictate a choice, then the proper choice of a proximity measure can be a time-consuming task that requires careful consideration of both domain knowledge and the purpose for which the measure is being used. A number of different similarity measures may need to be evaluated to see which ones produce results that make the most sense.

2.5 Bibliographic Notes

It is essential to understand the nature of the data that is being analyzed, and at a fundamental level, this is the subject of measurement theory. In

2.5 Bibliographic Notes 85

particular, one of the initial motivations for defining types of attributes was to be precise about which statistical operations were valid for what sorts of data. We have presented the view of measurement theory that was initially described in a classic paper by S. S. Stevens 179]. (Tables 2.2 and 2.3 are derived from those presented by Stevens [80].) While this is the most common view and is reasonably easy to understand and apply, there is, of course, much more to measurement theory. An authoritative discussion can be found in a three-volume series on the foundations of measurement theory [63, 69, 81]. AIso of interest is a wide-ranging article by Hand [55], which discusses measurement theory and statistics, and is accompanied by comments from other researchers in the field. Finally, there are many books and articles that describe measurement issues for particular areas of science and engineering.

Data quality is a broad subject that spans every discipline that uses data. Discussions of precision, bias, accuracy, and significant figures can be found in many introductory science, engineering, and statistics textbooks. The view of data quality as "fitness for use" is explained in more detail in the book by Redman [76]. Those interested in data quality may also be interested in MIT's Total Data Quality Management program [70, 84]. However, the knowledge needed to deal with specific data quality issues in a particular domain is often best obtained by investigating the data quality practices of researchers in that field.

Aggregation is a less well-defined subject than many other preprocessing

tasks. However, aggregation is one of the main techniques used by the database area of Online Analytical Processing (OLAP), which is discussed in Chapter 3. There has also been relevant work in the area of symbolic data analysis (Bock and Diday [a7]). One of the goals in this area is to summarize traditional record data in terms of symbolic data objects whose attributes are more complex than traditional attributes. Specifically, these attributes can have values that are sets of values (categories), intervals, or sets of values with weights (histograms). Another goal of symbolic data analysis is to be able to perform clustering, classification, and other kinds of data analysis on data that consists of symbolic data objects.

Sampling is a subject that has been well studied in statistics and related fields. Many introductory statistics books, such as the one by Lindgren [65], have some discussion on sampling, and there are entire books devoted to the subject, such as the classic text by Cochran [49]. A survey of sampling for data mining is provided by Gu and Liu [54], while a survey of sampling for databases is provided by Olken and Rotem [ZZ]. There are a number of other data mining and database-related sampling references that may be of interest,

86 Chapter 2 Data

including papers by Palmer and Faloutsos [74], Provost et al. [75], Toivonen

[82], and Zakiet al. [85]. In statistics, the traditional techniques that have been used for dimension-

ality reduction are multidimensional scaling (MDS) (Borg and Groenen [48], Kruskal and Uslaner [6a]) and principal component analysis (PCA) (Jolliffe

[58]), which is similar to singular value decomposition (SVD) (Demmel [50]). Dimensionality reduction is discussed in more detail in Appendix B.

Discretization is a topic that has been extensively investigated in data mining. Some classification algorithms only work with categorical data, and association analysis requires binary data, and thus, there is a significant moti- vation to investigate how to best binarize or discretize continuous attributes. For association analysis, we refer the reader to work by Srikant and Agrawal

[78], while some useful references for discretization in the area of classification include work by Dougherty et al. [51], Elomaa and Rousu [SZ], Fayyad and Irani [53], and Hussain et al. [56].

Feature selection is another topic well investigated in data mining. A broad coverage of this topic is provided in a survey by Molina et al. [71] and two books by Liu and Motada [66, 67]. Other useful paperc include those by Blum and Langley 1461, Kohavi and John [62], and Liu et al. [68].

It is difficult to provide references for the subject of feature transformations because practices vary from one discipline to another. Many statistics books have a discussion of transformations, but typically the discussion is restricted to a particular purpose, such as ensuring the normality of a variable or making sure that variables have equal variance. We offer two references: Osborne [73] and Ttrkey [83].

While we have covered some of the most commonly used distance and similarity measures, there are hundreds of such measures and more are being created all the time. As with so many other topics in this chapter, many of these measures are specific to particular fields; e.g., in the area of time series see papers by Kalpakis et al. [59] and Keogh and Pazzani [61]. Clustering books provide the best general discussions. In particular, see the books by Anderberg

[45], Jain and Dubes [57], Kaufman and Rousseeuw [00], and Sneath and Sokal

1771.

[45]

[461

Bibliography M. R. Anderberg. Cluster Analysis for Appli,cati,ons. Academic Press, New York, De- cember 1973.

A. BIum and P. Langley. Selection of Relevant Features and Examples in Machine Learning. Artificial Intellig ence, 97 (l=2) :245-27 l, 1997 .

Bibliography 87

l47l H. H. Bock and E. Diday. Analysis of Sgmbolic Data: Exploratory Methods for Ertract- ing Statistical Information from Complen Data (Studi,es in Classifi,cation, Data Analys'is, and, Know ledg e Org an'izat'ionl. Springer-Verlag Telos, January 2000.

[48] I. Borg and P. Groenen. Modern Multidimensional Scaling Theory and, Applications. Springer-Verlag, February 1997.

[49] W. G. Cochran. Sampling Techniques. John Wiley & Sons, 3rd edition, JuJy 1977.

[50] J. W. Demmel. Applied, Numerical Linear Algebra. Society for Industrial & Applied Mathematics, September 1997.

[51] J. Dougherty, R. Kohavi, and M. Sahami. Supervised and Unsupervised Discretization

of Continuous Features. In Proc. of the 12th Intl. Conf. on Machine Learni,ng, pages

L94-202, t995.

[52] T. Elomaa and J. Rousu. General and Efficient Multisplitting of Numerical Attributes.

M achine Learni,ng, 36(3):201 244, 1999.

[53] U. M. Fayyad and K. B. Irani. Multi-interval discretization of continuousvalued at- tributes for classification learning. In Proc. 13th Int. Joint Conf. on Arti;fi,cial Intelli,- gence, pages lO22-L027. Morgan Kaufman, 1993.

154] F. H. Gaohua Gu and H. Liu. Sampling and Its Application in Data Mining: A Survey. Technical Report TRA6/00, National University of Singapore, Singapore, 2000.

f55] D. J. Hand. Statistics and the Theory of Measurement. Jountal of the Rogal Statistical Societg: Series A (Statistics in Societg),159(3):445-492, 1996.

[56] F. Hussain, H. Liu, C. L. Tan, and M. Dash. TRC6/99: Discretization: an enabling technique. Technical report, National University of Singapore, Singapore, 1999.

[57j A. K. Jain and R. C. Dubes. Algorithrns for Clustering Data. Prentice Hall

Advanced Reference Series. Prentice Hall, March 1988. Book available online at

http: //www.cse.msu.edu/-jain/Clustering-Jain-Dubes.pdf.

[58] I. T. Jolliffe. Principal Cornponent Analys'is. Springer Verlag, 2nd edition, October 2002.

[59] K. Kalpakis, D. Gada, and V. Puttagunta. Distance Measures for Effective Clustering of ARIMA Time-Series. In Proc. of the 2001 IEEE Intl. Conf. on Data Mini'ng, pages

273-280. IEEE Computer Society, 2001.

[60] L. Kaufman and P. J. Rousseeuw. Findi,ng Groups in Data: An Introduction to Cluster Analysi.s. Wiley Series in Probability and Statistics. John Wiley and Sons, New York,

November 1990.

[61] E. J. Keogh and M. J. Pazzani. Scaling up dynamic time warping for datamining

applications. In KDD, pages 285-289, 2000.

[62] R. Kohavi and G. H. John. Wrappers for Feature Subset Selection. Artificial Intell'igence,

97 {I-2) :27 3-324, 1997 .

f63] D. Krantz, R. D. Luce, P. Suppes, and A. Tversky. Foundations of Measurements:

Volume 1: Additiue and polgnomial representations. Academic Press, New York, 1971.

[64] J. B. Kruskal and E. M. Uslaner. Multiilimensional Scal'ing. Sage Publications, August

1 9 7 8 .

[65] B. W. Lindgren. Statistical Theory CRC Press, January 1993.

f66] H. Liu and H. Motoda, editors. Feature Ertract'ion, Constr-uction and Select'ion: A Data

Mini,ng Perspectiue. Kluwer International Series in Engineering and Computer Science, 453. Kluwer Academic Publishers, July 1998.

[67] H. Liu and H. Motoda. Feature Selection for Knowleilge Discouery and Data Mi'n-

ing. Kluwer International Series in Engineering and Computer Science, 454. Kluwer

Academic Publishers. Julv 1998.

88 Chapter 2 Data

168] H. Liu, H. Motoda, and L. Yu. Feature Extraction, Selection, and Construction. In N. Ye, editor, The Handbook of Data Mi,ning, pages22 41. Lawrence Erlbaum Asso- ciates, Inc., Mahwah, NJ, 2003.

[69] R. D. Luce, D. Krantz, P. Suppes, and A. Tversky. Foundat'ions of Measurements: Volume 3: Representation, Ariomatizati,on, and Inuariance. Academic Press, New York, 1 9 9 0 .

[70] MIT Total Data Quality Management Program. web.mit.edu/tdqm/www/index.shtml, 2 0 0 3 .

171] L. C. Molina, L. Belanche, and A. Nebot. Feature Selection Algorithms: A Survey and Experimental Evaluation. ln Proc. of the 2002 IEEE Intl. Conf. on Data M'ining,2OO2.

1721 F. Olken and D. Rotem. Random Sampling from Databases-A Survey. Stati,stics I Comput'ing, 5(l) :25-42, March 1995.

[73] J. Osborne. Notes on the Use of Data TYansformations. Practical Assessment, Research €i Eualuation, 28(6), 2002.

l74l C R. Palmer and C. Faloutsos. Density biased sampling: An improved method for data mining and clusterin g. AC M S I G M O D Record, 29(2) :82-92, 2000.

[75] F. J. Provost, D. Jensen, and T. Oates. Efficient Progressive Sampling. In Proc. of the 5th IntI. Conf. on Knowled,ge Discouery and, Data Mining, pages 23-32, 1999.

[76] T. C. Redman. Data Qualitg: The Field Gui.d,e. Digital Press, January 2001.

l77l P H. A. Sneath and R. R. Sokal. Numerical Taronomy. FYeeman, San Francisco, 1971.

[78] R. Srikant and R. Agrawal. Mining Quantitative Association Rules in Large Relational Tables. In Proc. of 1996 ACM-SIGMOD Intl. Conf. on Management of Data, pages 1-12, Montreal, Quebec, Canada, August 1996.

[79] S. S. Stevens. On the Theory of Scales of Measurement. Science, 103(2684):677-680, June 1946.

[80] S. S. Stevens. Measurement. In G. M. Maranell, editor, Scali,ng: A Sourceboole for Behauioral Scientists, pages 22-4L Aldine Publishing Co., Chicago, 1974.

f81] P. Suppes, D. Krantz, R. D. Luce, and A. Tversky. Found,at'ions of Measurements: Volume 2: Geometrical, Threshold, and Probab'ilistic Representations. Academic Press, New York, 1989.

f82] H. Toivonen. Sampling Large Databases for Association Rules. In VLDB96, pages L34-I45. Morgan Kaufman, September 1996.

[83] J. W. Tukey. On the Comparative Anatomy of TYansformations. Annals of Mathematical Stat'istics, 28(3):602-632, September 1957.

f84] R. Y. Wang, M. Ziad, Y. W. Lee, and Y. R. Wang. Data Quali,ty. The Kluwer In- ternational Series on Advances in Database Systems, Volume 23. Kluwer Academic Publishers, January 2001.

[85] M. J. Zaki, S. Parthasarathy, W. Li, and M. Ogihara. Evaluation of Sampling for Data Mining of Association Rules. Technical Report TR617, Renssela,er Polytechnic Institute, 1 9 9 6 .

2.6 Exercises

1. In the initial example of Chapter 2, the statistician says, ((Yes,

fields 2 and 3 are basically the same." Can you tell from the three lines of sample data that are shown why she says that?

Exercises 89

2. Classify the following attributes as binary, discrete, or continuous. Also classifu them as qualitative (nominal or ordinal) or quantitative (interval or ratio).

Some cases may have more than one interpretation, so briefly indicate your

reasoning if you think there may be some ambiguity.

Example: Age in years. Answer: Discrete, quantitative, ratio

2 . 6

(u)

(b)

(")

(d)

(e)

(f)

(s) (h)

(i)

Time in terms of AM or PM.

Brightness as measured by a light meter.

Brightness as measured by people's judgments.

Angles as measured in degrees between 0 and 360.

Bronze, Silver, and Gold medals as awarded at the Olympics.

Height above sea level.

Number of patients in a hospital.

ISBN numbers for books. (Look up the format on the Web.)

Ability to pass light in terms of the following values: opaque, translucent' transparent.

Military rank.

Distance from the center of campus.

Density of a substance in grams per cubic centimeter.

Coat check number. (When you attend an event, you can often give your

coat to someone who, in turn, gives you a number that you can use to

claim your coat when you leave.)

0 ) (k)

(l)

(-)

3. You are approached by the marketing director of a local company, who believes

that he has devised a foolproof way to measure customer satisfaction' He

explains his scheme as follows: "It's so simple that I can't believe that no one

has thought of it before. I just keep track of the number of customer complaints for each product. I read in a data mining book that counts are ratio attributes, and so, my measure of product satisfaction must be a ratio attribute. But

when I rated the products based on my new customer satisfaction measure and

showed them to my boss, he told me that I had overlooked the obvious, and that my measure was worthless. I think that he was just mad because our best-

selling product had the worst satisfaction since it had the most complaints. Could you help me set him straight?"

(a) Who is right, the marketing director or his boss? If you answered, his

boss, what would you do to fix the meaaure of satisfaction?

(b) What can you say about the attribute type of the original product satis- faction attribute?

90 Chapter 2 Data

7

A few months later, you are again approached by the same marketing director as in Exercise 3. This time, he has devised a better approach to measure the extent to which a customer prefers one product over other, similar products. He explains, "When we develop new products, we typically create several variations and evaluate which one customers prefer. Our standard procedure is to give our test subjects all ofthe product variations at one time and then ask them to rank the product variations in order of preference. However, our test subjects are very indecisive, especially when there are more than two products. As a result, testing takes forever. I suggested that we perform the comparisons in pairs and then use these comparisons to get the rankings. Thus, if we have three product variations, we have the customers compare variations I and 2, then 2 and 3, and finally 3 and 1. Our testing time with my new procedure is a third of what it was for the old procedure, but the employees conducting the tests complain that they cannot come up with a consistent ranking from the results. And my boss wants the latest product evaluations, yesterday. I should also mention that he was the person who came up with the old product evaluation approach. Can you help me?"

(a) Is the marketing director in trouble? Will his approach work for gener- ating an ordinal ranking of the product variations in terms of customer preference? Explain.

(b) Is there a way to fix the marketing director's approach? More generally, what can you say about trying to create an ordinal measurement scale based on pairwise comparisons?

(c) For the original product evaluation scheme, the overall rankings of each product variation are found by computing its average over all test subjects. Comment on whether you think that this is a reasonable approach. What other approaches might you take?

Can you think of a situation in which identification numbers would be useful for prediction?

An educational psychologist wants to use association analysis to analyze test results. The test consists of 100 questions with four possible answers each.

(a) How would you convert this data into a form suitable for association analysis?

(b) In particular, what type of attributes would you have and how many of them are there?

Which of the following quantities is likely to show more temporal autocorrela- tion: daily rainfall or daily temperature? Why?

Discuss why a document-term matrix is an example of a data set that has asymmetric discrete or asymmetric continuous features.

I

5 .

o .

8 .

9 .

1 0 .

1 1 .

12.

2 . 6 Exercises 9L

Many sciences rely on observation instead of (or in addition to) designed ex- periments. Compare the data quality issues involved in observational science with those of experimental science and data mining.

Discuss the difference between the precision of a measurement and the terms single and double precision, as they are used in computer science, typically to represent floating-point numbers that require 32 and 64 bits, respectively.

Give at least two advantages to working with data stored in text files instead of in a binary format.

Distinguish between noise and outliers. Be sure to consider the following ques- tions.

(a) Is noise ever interesting or desirable? Outliers?

(b) Can noise objects be outliers?

(c) Are noise objects always outliers?

(d) Are outliers always noise objects?

(e) Can noise make a typical value into an unusual one, or vice versa?

13. Consider the problem of finding the K nearest neighbors of a data object. A programmer designs Algorithm 2.2 for this task.

Algorithm 2.2 Algorithm for finding K nearest neighbors. 1: for ri : 1 to number of data objects do 2: Find the distances of the ith object to all other objects. 3: Sort these distances in decreasing order.

(Keep track of which object is associated with each distance.) 4: return the objects associated with the first K distances of the sorted list 5: end for

Describe the potential problems with this algorithm if there are duplicate objects in the data set. Assume the distance function will only return a distance of 0 for objects that are the same.

How would you fix this problem?

14. The following attributes are measured for members of a herd of Asian ele- phants: wei,ght, hei,ght, tusk length, trunk length, and ear area. Based on these measurements, what sort of similarity mea"sure from Section 2.4 would you use to compare or group these elephants? Justify your answer and explain any special circumstances.

(u)

(b)

92 Chapter 2 Data

You are given a set of rn objects that is divided into K groups, where the ith group is of size mi. If. the goal is to obtain a sample of size fl I ffi, what is the difference between the following two sampling schemes? (Assume sampling with replacement.)

(a) We randomly select n *mif m elements from each group.

(b) We randomly select n elements from the data set, without regard for the group to which an object belongs.

Consider a document-term matrix, where tfii isthe frequency of the rith word (term) in the jth document and m is the number of documents. Consider the variable transformation that is defined by

1 5 .

1 6 .

( 2 . 1 8 )

where dfi is the number of documents in which the i.th term appears, which is known as the document frequency of the term. This transformation is known as the inverse document frequency transformation.

(a) What is the effect of this transformation if a term occurs in one document? In every document?

(b) What might be the purpose of this transformation?

Assume that we apply a square root transformation to a ratio attribute r to obtain the new attribute r*. As part of your analysis, you identify an interval (o, b) in which r* has a linear relationship to another attribute gr.

(a) What is the corresponding interval (o, b) in terms of r?

(b) Give an equation that relates y to r.

This exercise compares and contrasts some similarity and distance measures.

(a) For binary data, the Ll distance corresponds to the Hamming distance; that is, the number of bits that are different between two binary vectors. The Jaccard similarity is a measure of the similarity between two bina,ry vectors. Compute the Hamming distance and the Jaccard similarity be- tween the following two binary vectors.

x : 0 1 0 1 0 1 0 0 0 1 y : 0100011000

(b) Which approach, Jaccard or Hamming distance, is more similar to the Simple Matching Coefficient, and which approach is more similar to the cosine measure? Explain. (Note: The Hamming mea,sure is a distance, while the other three measures are similarities, but don't let this confuse you.)

tf'ti:tfti *nsffi,

1 7 .

1 8 .

1 9 .

2.6 Exercises 93

(c) Suppose that you are comparing how similar two organisms of different species are in terms of the number of genes they share. Describe which measure, Hamming or Jaccard, you think would be more appropriate for comparing the genetic makeup of two organisms. Explain. (Assume that each animal is represented as a binary vector, where each attribute is 1 if a particular gene is present in the organism and 0 otherwise.)

(d) If you wanted to compare the genetic makeup of two organisms of the same species, e.g., two human beings, would you use the Hamming distance, the Jaccard coefficient, or a different measure of similarity or distance? Explain. (Note that two human beings share > 99.9% of the same genes.)

For the following vectors, x and y, calculate the indicated similarity or distance measures.

(a) x : (1, 1, 1, 1), y : (2,2,2,2) cosine, correlation, Euclidean

(b) x : (0, 1,0, 1), y : (1,0, 1,0) cosine, correlation, Euclidean, Jaccard

( c ) x : ( 0 , - 1 , 0 , 1 ) , y : ( 1 , 0 , - 1 , 0 ) c o s i n e , c o r r e l a t i o n , E u c l i d e a n

( d ) x : ( 1 , 1 , 0 , 1 , 0 , 1 ) , y : ( 1 , 1 , 1 , 0 , 0 , 1 ) c o s i n e , c o r r e l a t i o n , J a c c a r d

( e ) x : ( 2 , - 7 , 0 , 2 , 0 , - 3 ) , y : ( - 1 , 1 , - 1 , 0 , 0 , - 1 ) c o s i n e , c o r r e l a t i o n

Here, we further explore the cosine and correlation measures.

(a) What is the range of values that are possible for the cosine measure?

(b) If two objects have a cosine measure of 1, are they identical? Explain.

(c) What is the relationship of the cosine mea,sure to correlation, if any? (Hint: Look at statistical measures such as mean and standard deviation in cases where cosine and correlation are the same and different.)

(d) Figure 2.20(a) shows the relationship of the cosine measure to Euclidean distance for 100,000 randomly generated points that have been normalized to have an L2 length of 1. What general observation can you make about the relationship between Euclidean distance and cosine similarity when vectors have an L2 norm of 1?

(e) Figure 2.20(b) shows the relationship of correlation to Euclidean distance for 100,000 randomly generated points that have been standardized to have a mean of 0 and a standard deviation of 1. What general observa- tion can you make about the relationship between Euclidean distance and correlation when the vectors have been standardized to have a mean of 0 and a standard deviation of 1?

(f) Derive the mathematical relationship between cosine similarity and Eu- clidean distance when each data object has an L2 length of 1.

(g) Derive the mathematical relationship between correlation and Euclidean distance when each data point has been been standardized by subtracting its mean and dividing by its standard deviation.

20.

1 . 4

1 . 2

1

0 . 8

0 . 6

o . 4

o . 2

Cosine Similarity

94 Chapter 2 Data

(a) Relationship between Euclidean distance and the cosine measure.

(b) Relationship between Euclidean distance and correlation.

Figure 2.20. Graphs for Exercise 20.

Show that the set difference metric given by

d(A, B) : size(A - B) + si,ze(B - A) (2.19)

satisfies the metric axioms given on page 70. ,4. and B are sets and A - B is the set difference.

Discuss how you might map correlation values from the interval l-1,1] to the interval [0,1]. Note that the type of transformation that you use might depend on the application that you have in mind. Thus, consider two applications: clustering time series and predicting the behavior of one time series given an- other.

Given a similarity measure with values in the interval [0,1] describe two ways to transform this similarity value into a dissimilarity value in the interval l0,oo].

Proximity is typically defined between a pair of objects.

(a) Define two ways in which you might define the proximity among a group of objects.

(b) How might you define the distance between two sets of points in Euclidean space?

(c) How might you define the proximity between two sets of data objects? (Make no assumption about the data objects, except that a proximity measure is defined between any pair of objects.)

You are given a set of points ,9 in Euclidean space, as well as the distance of each point in ,S to a point x. (It does not matter if x e S.)

o o

q

o G o

l,rJ

o

o o c (5 o

o

I.JJ

2T,

22.

23.

, A

1 . 4

1 . 2

1

0 . 8

0 . 6

0 . 4

o . 2

Correlation

2 5 .

26.

27.

,9,

2.6, Exercises 95

(a) If the goal is to find all points within a specified distance e of point y, y * x, explain how you could use the triangle inequality and the already calculated distances to x to potentially reduce the number of distance calculations necessary? Hint: The triangle inequality, d(x,z) < d(x,y)* d(y,*), can be rewritten as d(x,y) 2 d(x, z) - d(y,z).

(b) In general, how would the distance between x and y affect the number of distance calculations?

(c) Suppose that you can find a small subset of points ,5', from the original data set, such that every point in the data set is within a specified distance e of at least one of the points in ^91 and that you also have the pairwise

distance matrix for 51 Describe a technique that uses this information to compute, with a minimum of distance calculations, the set of all points

within a distance of B of a specified point from the data set.

Show that 1 minus the Jaccard similarity is a distance measure between two data objects, x and y, that satisfies the metric axioms given on page 70. Specifically, d ( * , y ) : 1 - J ( x , y ) .

Show that the distance measure defined as the angle between two data vectors, x and y, satisfies the metric axioms given on page 70. Specifically, d(*, y) :

arccos(cos(x, y)).

Explain why computing the proximity between two attributes is often simpler than computing the similarity between two objects.

Exploring Data

The previous chapter addressed high-level data issues that are important in the knowledge discovery process. This chapter provides an introduction to data exploration, which is a preliminary investigation of the data in order to better understand its specific characteristics. Data exploration can aid in selecting the appropriate preprocessing and data analysis techniques. It can even address some of the questions typically answered by data mining. For example, patterns can sometimes be found by visually inspecting the data. Also, some of the techniques used in data exploration, such as visualization, can be used to understand and interpret data mining results.

This chapter covers three major topics: summary statistics, visualization, and On-Line Analytical Processing (OLAP). Summary statistics, such as the mean and standard deviation of a set of values, and visualization techniques, such as histograms and scatter plots, are standard methods that are widely employed for data exploration. OLAP, which is a more recent development, consists of a set of techniques for exploring multidimensional arrays of values. OlAP-related analysis functions focus on various ways to create summary data tables from a multidimensional data array. These techniques include aggregating data either across various dimensions or across various attribute values. For instance, if we are given sales information reported according to product, location, and date, OLAP techniques can be used to create a summary that describes the sales activity at a particular location by month and product category.

The topics covered in this chapter have considerable overlap with the area known as Exploratory Data Analysis (EDA), which was created in the 1970s by the prominent statistician, John Tirkey. This chapter, like EDA, places a heavy emphasis on visualization. Unlike EDA, this chapter does not include topics such as cluster analysis or anomaly detection. There are two

98 Chapter 3 Exploring Data

reasons for this. First, data mining views descriptive data analysis techniques as an end in themselves, whereas statistics, from which EDA originated, tends to view hypothesis-based testing as the final goal. Second, cluster analysis and anomaly detection are large areas and require full chapters for an in- depth discussion. Hence, cluster analysis is covered in Chapters 8 and 9, while anomaly detection is discussed in Chapter 10.

3.1- The Iris Data Set

In the following discussion, we will often refer to the Iris data set that is available from the University of California at Irvine (UCI) Machine Learn- ing Repository. It consists of information on 150 Iris flowers, 50 each from one of three Iris species: Setosa, Versicolour, and Virginica. Each flower is characterized by five attributes:

1. sepal length in centimeters

2. sepal width in centimeters

3. petal length in centimeters

4. petal width in centimeters

5. class (Setosa, Versicolour, Virginica)

The sepals of a flower are the outer structures that protect the more fragile parts of the flower, such as the petals. In many flowers, the sepals are green, and only the petals are colorful. For Irises, however, the sepals are also colorful. As illustrated by the picture of a Virginica Iris in Figure 3.1, the sepals of an Iris are larger than the petals and are drooping, while the petals are upright.

3.2 Summary Statistics

Summary statistics are quantities, such as the mean and standard deviation, that capture various characteristics of a potentially large set of values with a single number or a small set of numbers. Everyday examples of summary statistics are the average household income or the fraction of college students who complete an undergraduate degree in four years. Indeed, for many people, summary statistics are the most visible manifestation of statistics. We will concentrate on summary statistics for the values of a single attribute, but will provide a brief description of some multivariate summary statistics.

3.2 Summary Statistics 99

Figure 3.1. Picture of lris Virginica. Robert H. Mohlenbrock @ USDA-NRCS PLANTS Database/ USDA NRCS. 1995. Northeast wetland flora: Field office guide to plant species. Northeast National Technical Center, Chester, PA. Background removed.

This section considers only the descriptive nature of summary statistics. However, as described in Appendix C, statistics views data as arising from an underlying statistical process that is characterized by various parameters, and some of the summary statistics discussed here can be viewed as estimates of statistical parameters of the underlying distribution that generated the data.

3.2.L Flequencies and the Mode

Given a set of unordered categorical values, there is not much that can be done to further characterize the values except to compute the frequency with which each value occurs for a particular set of data. Given a categorical attribute r, which can take values {rt,. . . ,1ri,. .. u7r} and a set of rn objects, the frequency of a value u; is defined as

frequency(u5) : number of objects with attribute value u;

( 3 . 1 )

The mode of a categorical attribute is the value that has the highest frequency.

100 Chapter 3 Exploring Data

Example 3.1. Consider a set of students who have an attribute, class, which can take values from the set {f reshman, sophomore, junior, senior}. Table 3.1 shows the number of students for each value of the class attribute. The

mode of the class attribute is f reshman, with a frequency of 0.33. This may indicate dropouts due to attrition or a larger than usual freshman class.

Table 3.1. Class size for students in a hypothetical college.

Class Size FYequency freshman sophomore junior

senior

t40 160 130 170

0 . 3 3 0 . 2 7 0 . 2 2 0 . 1 8

T

Categorical attributes often, but not always, have a small number of values, and consequently, the mode and frequencies of these values can be interesting and useful. Notice, though, that for the Iris data set and the class attribute, the three types of flower all have the same frequency, and therefore, the notion of a mode is not interesting.

For continuous data, the mode, as currently defined, is often not useful because a single value may not occur more than once. Nonetheless, in some cases, the mode may indicate important information about the nature of the values or the presence of missing values. For example, the heights of 20 people

measured to the nearest millimeter will typically not repeat, but if the heights are measured to the nearest tenth of a meter, then some people may have the same height. Also, if a unique value is used to indicate a missing value, then this value will often show up as the mode.

3 . 2 . 2 P e r c e n t i l e s

For ordered data, it is more useful to consider the percentiles of a set of values. In particular, given an ordinal or continuous attribute r and a number p between 0 and 100, the pth percentile ro is a value of z such that pTo of the observed values of r are less than ro. For instance, the 50th percentile is the valte r5so/o such that 50% of all values of r are less than r5sv. Table 3.2 shows the percentiles for the four quantitative attributes of the Iris data set.

3.2 Summary Statistics 101-

Table 3.2. Percentiles for sepal length, sepal width, petal length, and petal width. (All values are in centimeters.)

Percentile Sepal Length Sepal Width Petal Length Petal Width 0 1 0 20 30 40 50 60 70 80 90 100

4 . 3 4 . 8 5 . 0 5 . 2 D.r) 5 . 8 6 . 1 6 . 3 6 . 6 6 . 9 7 . 9

2 . 0 2 . 5 2 . 7 2 . 8 3 . 0 3 . 0 3 . 1 3 . 2 3 . 4 3 . 6 4 . 4

1 . 0 t . 4 r . o t . 7 3 . 9 4 . 4 4 . 6 5 . 0 o . 4

c . 6

6 . 9

0 . 1 0 . 2 0 . 2 0 . 4 7 . 2 1 . 3 - t . l )

1 . 8 1 . 9 2 . 2 2 . 5

Example 3.2. The percentiles, ryyo,rro4o,.. . ,frgyyo,rno% of the integers from 1 t o 1 0 a r e , i n o r d e r , t h e f o l l o w i n g : 1 . 0 , 7 . 5 , 2 . 5 , 3 . 5 , 4 . 5 , 5 . 5 , 6 . 5 , 7 . 5 , 8 . 5 , 9 . 5 , 10.0. By tradition, min(r) : troyo and max(r) : *too%. r

3.2.3 Measures of Location: Mean and Median

For continuous data, two of the most widely used summary statistics are the mean and median, which are measures of the locati,on of a set of values. C o n s i d e r a s e t o f n l o b j e c t s a n d a n a t t r i b u t e r . L e t { r 1 , . . . , r ^ ) b e t h e attribute values of r for these zn objects. As a concrete example, these values might be the heights of rn children. Let {rg1,...,:xOd} represent the values of z after they have been sorted in non-decreasing order. Thus, ro): min(z) and r1*1: max(r). Then, the mean and median are defined as follows:

* ^ ^ - / - \ = I \ - . .m e a n ( r ) : I : * L * 'm,

_ ^ r : ^ - / \ f * , r * r l i f r n i s o d d , i . e . , r n : 2 r * |m e o r a n ( r / : 1 ! . ' i * , ' \ 2 \ * ( r ) * r 1 " + r 1 ) i f r n i s e v e n , i ' e ' , m : 2 r

(3.2)

( 3 . 3 )

To summarize, the median is the middle value if there are an odd number of values, and the average of the two middle values if the number of values is even. Thus, for seven values, the median is 1141, while for ten values, the m e d i a n i s | ( r 1 5 ; + r f o l ) .

LO2 Chapter 3 Exploring Data

Although the mean is sometimes interpreted as the middle of a set of values, this is only correct if the values are distributed in a symmetric manner. If the distribution of values is skewed, then the median is a better indicator of the middle. AIso, the mean is sensitive to the presence of outliers. For data with outliers, the median again provides a more robust estimate of the middle of a set of values.

To overcome problems with the traditional definition of a mean, the notion

of a trimmed mean is sometimes used. A percentage p between 0 and 100 is specified, the top and bottom (pl2)% of the data is thrown out, and the mean is then calculated in the normal way. The median is a trimmed mean with p -- L00yo, while the standard mean corresponds to p: go/o.

Example 3.3. Consider the set of values {L,2,3,4, 5,90}. The mean of these values is 17.5, while the median is 3.5. The trimmed mean with p : 40To is also 3.5. r

Example 3.4. The means, medians, and trimmed means (p : 20%) of the four quantitative attributes of the Iris data are given in Table 3.3. The three measures of location have similar values except for the attribute petal length.

Table 3.3. Means and medians for sepal length, sepal width, petal length, and petal width. (All values are in centimeters.)

3.2.4 Measures of Spread: Range and Variance

Another set of commonly used summary statistics for continuous data are those that measure the dispersion or spread of a set of values. Such measures indicate if the attribute values are widely spread out or if they are relatively concentrated around a single point such as the mean.

The simplest measure of spread is the range, which, given an attribute r with a set of rn values {rr, . . . , r*}, is defined as

Measure Sepal Length Sepal Width Petal Length Petal Width mean

median trimmed mean (20To)

5 . 8 4 5 . 8 0 5 . ( 9

3 . 0 5 3.00 3 . 0 2

3 . 7 6 4 . 3 5 3 . 7 2

I 20 1 . 3 0 r . 7 2

range(r) : max(r) - min(r) : r(^) - r(t). ( 3 . 4 )

3.2 Summary Statistics 103

Table 3.4. Range, standard deviation (std), absolute average difference (AAD), median absolute difier- ence (MAD), and interquartile range (lQR)for sepal length, sepal width, petal length, and petal width. (Allvalues are in centimeters.)

Although the range identifies the maximum spread, it can be misleading if most of the values are concentrated in a narrow band of values, but there are also a relatively small number of more extreme values. Hence, the variance is preferred as a measure of spread. The variance of the (observed) values of an attribute r is typically written as sl and is defined below. The standard deviation, which is the square root of the variance, is written as su and has the same units as r.

Measure Sepal Length Sepal Width Petal Length Petal Width range std

AAD MAD IQR

3 . 6 0 . 8 0 . 7 0 . 7 1 . 3

, A

0 . 4 0 . 3 0 . 3 0 . 5

5 . 9 1 . 8 1 . 6 r . 2 3 . 5

, A

0 . 8 0 . 6 u . ( 1 . 5

1 m --- \-(2, - z)2 m , - l z - / ' '

1 f f i

A A D ( z ) : ' t l * i - n l m -

MAD(z) : med'ian( ft, - rl,. . ., l"- - rl)) \ /

interquartile range(r) : r1sTo - r2s%

variance(z) - s7: ( 3 . 5 )

The mean can be distorted by outliers, and since the variance is computed using the mean, it is also sensitive to outliers. Indeed, the variance is particu- larly sensitive to outliers since it uses the squared difference between the mean and other values. As a result, more robust estimates of the spread of a set of values are often used. Following are the definitions of three such measures: the absolute average deviation (AAD), the median absolute deviation (MAD), and the interquartile range(IQR). Table 3.4 shows these measures for the Iris data set.

( 3 . 6 )

( 3 . 7 )

( 3 . 8 )

LO4 Chapter 3 Exploring Data

3.2.5 Multivariate Summary Statistics

Measures of location for data that consists of several attributes (multivariate

data) can be obtained by computing the mean or median separately for each

attribute. Thus, given a data set the mean of the data objects, x, is given by

(3.e)

where 4 is the mean of the i,th attribute r;. For multivariate data, the spread of each attribute can be computed in-

dependently of the other attributes using any of the approaches described in

Section 3.2.4. However, for data with continuous variables, the spread of the

data is most commonly captured. by the covariance matrix S, whose iith

entry sii is the covariance of the i}h and jth attributes of the data. Thus, if ai

and ri are the ith and jth attributes, then

sij : covafi ance(rrr t i).

In turn, couariance(q,ri) is given by

( 3 . 1 0 )

covariance(ri, r j) 1 nr \ -

., l\rnt. - ri)\rki - ri), ( 3 . 1 1 )

v 2 - \ z - t K : 1

where rpi arrd, rkj are the values of the ith andj'h attributes for the kth object. Notice that covariance(r6,rt) : variance(r1). Thus, the covariance matrix has

the variances of the attributes along the diagonal. The covariance of two attributes is a measure of the degree to which two

attributes vary together and depends on the magnitudes of the variables. A

value near 0 indicates that two attributes do not have a (linear) relationship,

but it is not possible to judge the degree of relationship between two variables

by looking only at the value of the covariance. Because the correlation of two

attributes immediately gives an indication of how strongly two attributes are (linearly) related, correlation is preferred to covariance for data exploration. (AIso see the discussion of correlation in Section 2.4.5.) The ijth entry of the

correlation matrix R, is the correlation between I'he ith and jth attributes of the data. If rt arrd. rj are the i,th and jth attributes, then

ri.j : corcelntion(r6, ,j) : **Xy-f, (3.12)

Visualization 105

where s2 and sy are the variances of r; and rjj respectively. The diagonal entries of R are correlation(u,rt): 1, while the other entries are between -1 and 1. It is also useful to consider correlation matrices that contain the pairwise correlations of objects instead of attributes.

3.2.6 Other Ways to Summarize the Data

There are, of course, other types of summary statistics. For instance, the skewness of a set of values measures the degree to which the values are sym- metrically distributed around the mean. There are also other characteristics of the data that are not easy to measure quantitatively, such as whether the distribution of values is multimodal; i.e., the data has multiple "bumps" where most of the values are concentrated. In many cases, however, the most effec- tive approach to understanding the more complicated or subtle aspects of how the values of an attribute are distributed, is to view the values graphically in the form of a histogram. (Histograms are discussed in the next section.)

3.3 Visualization

Data visualization is the display of information in a graphic or tabular format. Successful visualization requires that the data (information) be converted into a visual format so that the characteristics of the data and the relationships among data items or attributes can be analyzed or reported. The goal of visualization is the interpretation of the visualized information by a person and the formation of a mental model of the information.

In everyday life, visual techniques such as graphs and tables are often the preferred approach used to explain the weather, the economy, and the results of political elections. Likewise, while algorithmic or mathematical approaches are often emphasized in most technical disciplines-data mining included- visual techniques can play a key role in data analysis. In fact, sometimes the use of visualization techniques in data mining is referred to as visual data mining.

3.3.1 Motivations for Visualization

The overriding motivation for using visualization is that people can quickly absorb large amounts of visual information and find patterns in it. Consider Figure 3.2, which shows the Sea Surface Temperature (SST) in degrees Celsius for July, 1982. This picture summarizes the information from approximately 250,000 numbers and is readily interpreted in a few seconds. For example, it

3 . 3

106 Chapter 3 Exploring Data

Longitude

Figure 3,2. Sea Surface Temperature (SST) for July, 1982.

is easy to see that the ocean temperature is highest at the equator and lowest at the poles.

Another general motivation for visualization is to make use of the domain knowledge that is "locked up in people's heads." While the use of domain knowledge is an important task in data mining, it is often difficult or impossible

to fully utilize such knowledge in statistical or algorithmic tools. In some cases, an analysis can be performed using non-visual tools, and then the results presented visually for evaluation by the domain expert. In other cases, having

a domain specialist examine visualizations of the data may be the best way

of finding patterns of interest since, by using domain knowledge, a person can

often quickly eliminate many uninteresting patterns and direct the focus to the patterns that are important.

3 . 3 . 2 G e n e r a l C o n c e p t s

This section explores some of the general concepts related to visualization, in particular, general approaches for visualizing the data and its attributes. A

number of visualization techniques are mentioned briefly and will be described in more detail when we discuss specific approaches later on. We assume that

the reader is familiar with line graphs, bar charts, and scatter plots.

Temp

1 5

1 0

3.3 Visualization 1.O7

Representation: Mapping Data to Graphical Elements

The first step in visualization is the mapping of information to a visual format; i.e., mapping the objects, attributes, and relationships in a set of information to visual objects, attributes, and relationships. That is, data objects, their at- tributes, and the relationships among data objects are translated into graphical elements such as points, lines, shapes, and colors.

Objects are usually represented in one of three ways. First, if only a single categorical attribute of the object is being considered, then objects are often lumped into categories based on the value of that attribute, and these categories are displayed as an entry in a table or arr area on a screen. (Examples shown later in this chapter are a cross-tabulation table and a bar chart.) Second, if an object has multiple attributes, then the object can be displayed as a row (or column) of a table or as a line on a graph. Finally, an object is often interpreted as a point in two- or three-dimensional space, where graphically, the point might be represented by a geometric figure, such as a circle. cross. or box.

For attributes, the representation depends on the type of attribute, i.e., nominal, ordinal, or continuous (interval or ratio). Ordinal and continuous attributes can be mapped to continuous, ordered graphical features such as location along the x:) A) or z axes; intensity; color; or size (diameter, width, height, etc.). For categorical attributes, each category can be mapped to a distinct position, color, shape, orientation, embellishment, or column in a table. However, for nominal attributes, whose values are unordered, care should be taken when using graphical features, such as color and position that have an inherent ordering associated with their values. In other words, the graphical elements used to represent the ordinal values often have an order, but ordinal values do not.

The representation of relationships via graphical elements occurs either explicitly or implicitly. For graph data, the standard graph representation- a set of nodes with links between the nodes-is normally used. If the nodes (data objects) or links (relationships) have attributes or characteristics oftheir own, then this is represented graphically. To illustrate, if the nodes are cities and the links are highways, then the diameter of the nodes might represent population, while the width of the links might represent the volume of traffic.

In most cases, though, mapping objects and attributes to graphical el- ements implicitly maps the relationships in the data to relationships among graphical elements. To illustrate, if the data object represents a physical object that has a location, such as a city, then the relative positions of the graphical objects corresponding to the data objects tend to naturally preserve the actual

1-08 Chapter 3 Exploring Data

relative positions of the objects. Likewise, if there are two or three continuous attributes that are taken as the coordinates ofthe data points, then the result-

ing plot often gives considerable insight into the relationships of the attributes and the data points because data points that are visually close to each other have similar values for their attributes.

In general, it is difficult to ensure that a mapping of objects and attributes will result in the relationships being mapped to easily observed relationships among graphical elements. Indeed, this is one of the most challenging aspects

of visualization. In any given set of data, there are many implicit relationships, and hence, a key challenge of visualization is to choose a technique that makes

the relationships of interest easily observable.

Arrangement

As discussed earlier, the proper choice of visual representation of objects and attributes is essential for good visualization. The arrangement of items within

the visual display is also crucial. We illustrate this with two examples.

Example 3.5. This example illustrates the importance of rearranging a table of data. In Table 3.5, which shows nine objects with six binary attributes, there is no clear relationship between objects and attributes, at least at first glance. If the rows and columns of this table are permuted, however, as shown in Table 3.6, then it is clear that there are really only two types of objects in the table-one that has all ones for the first three attributes and one that has only ones for the last three attributes. I

Table 3.5, A table of nine objects (rows) with six binary attributes (columns).

r 2 3 4 5 6

Table 3.6. A table of nine objects (rows) with six binary attributes (columns) permuted so that the relationships of the rows and columns are clear.

6 1 3 2 5 4 1 2 J

4 5 o ,7

8 q

0 1 0 1 1 0 1 0 1 0 0 1 0 1 0 1 1 0 1 0 1 0 0 1 0 1 0 1 1 0 1 0 1 0 0 1 0 1 0 1 1 0 1 0 1 0 0 1 0 1 0 1 1 0

0 0 0 1 I

1 1 I

4 2 6 8 r

3 9 1 7

1 1 1 0 0 1 1 1 0 0 1 1 1 0 0 1 1 1 0 0 0 0 0 1 1 0 0 0 1 1 0 0 0 1 1 0 0 0 1 1 0 0 0 1 1

3.3 Visualization 109

Example 3.6. Consider Figure 3.3(a), which shows a visualization of a graph. If the connected components of the graph are separated, as in Figure 3.3(b), then the relationships between nodes and graphs become much simpler to understand.

V (a) Original view of a graph. (b) Uncoupled view ofconnected components

of the graph.

Two visualizations of a graph.Figure 3.3.

Selection

Another key concept in visualization is selection, which is the elimination or the de-emphasis of certain objects and attributes. Specifically, while data objects that only have a few dimensions can often be mapped to a two- or three-dimensional graphical representation in a straightforward way, there is no completely satisfactory and general approach to represent data with many attributes. Likewise, if there are many data objects, then visualizing all the objects can result in a display that is too crowded. If there are many attributes and many objects, then the situation is even more challenging.

The most common approach to handling many attributes is to choose a subset of attributes-usually two-for display. If the dimensionality is not too high, a matrix of bivariate (two-attribute) plots can be constructed for simul- taneous viewing. (Figure 3.16 shows a matrix of scatter plots for the pairs of attributes of the Iris data set.) Alternatively, a visualization program can automatically show a series of two-dimensional plots, in which the sequence is user directed or based on some predefined strategy. The hope is that visualiz- ing a collection of two-dimensional plots will provide a more complete view of the data.

L10 Chapter 3 Exploring Data

The technique of selecting a pair (or small number) of attributes is a type of dimensionality reduction, and there are many more sophisticated dimension- ality reduction techniques that can be employed, e.g., principal components analysis (PCA). Consult Appendices A (Linear Algebra) and B (Dimension-

ality Reduction) for more information. When the number of data points is high, e.9., more than a few hundred,

or if the range of the data is large, it is difficult to display enough information about each object. Some data points can obscure other data points, or a data object may not occupy enough pixels to allow its features to be clearly

displayed. For example, the shape of an object cannot be used to encode a characteristic of that object if there is only one pixel available to display it. In

these situations, it is useful to be able to eliminate some of the objects, either by zooming in on a particular region of the data or by taking a sample of the data points.

3.3.3 Techniques

Visualization techniques are often specialized to the type of data being ana- lyzed. Indeed, new visualization techniques and approaches, as well as special- ized variations ofexisting approaches, are being continuously created, typically in response to new kinds of data and visualization tasks.

Despite this specialization and the ad hoc nature of visualization, there are some generic ways to classify visualization techniques. One such classification is based on the number of attributes involved (1,2,3, or many) or whether the data has some special characteristic, such as a hierarchical or graph structure. Visualization methods can also be classified according to the type of attributes involved. Yet another classification is based on the type of application: scien- tific, statistical, or information visualization. The following discussion will use three categories: visualization of a small number of attributes, visualization of data with spatial andf or temporal attributes, and visualization of data with many attributes.

Most of the visualization techniques discussed here can be found in a wide variety of mathematical and statistical packages, some of which are freely available. There are also a number of data sets that are freely available on the World Wide Web. Readers are encouraged to try these visualization techniques as they proceed through the following sections.

3.3 Visualization 111

Visualizing Small Numbers of Attributes

This section examines techniques for visualizing data with respect to a small number of attributes. Some of these techniques, such as histograms, give insight into the distribution of the observed values for a single attribute. Other techniques, such as scatter plots, are intended to display the relationships between the values of two attributes.

Stem and Leaf Plots Stem and leaf plots can be used to provide insight into the distribution of one-dimensional integer or continuous data. (We will assume integer data initially, and then explain how stem and leaf plots can be applied to continuous data.) For the simplest type of stem and leaf plot, we split the values into groups, where each group contains those values that are the same except for the last digit. Each group becomes a stem, while the last digits of a group are the leaves. Hence, if the values are two-digit integers, e.g., 35, 36, 42, and 51, then the stems will be the high-order digits, e.g., 3, 4, and 5, while the leaves are the low-order digits, e.g., 1, 2, 5, and 6. By plotting the stems vertically and leaves horizontally, we can provide a visual representation of the distribution of the data.

Example 3.7. The set of integers shown in Figure 3.4 is the sepal length in centimeters (multiplied by 10 to make the values integers) taken from the Iris data set. For convenience, the values have also been sorted.

The stem and leaf plot for this data is shown in Figure 3.5. Each number in Figure 3.4 is first put into one of the vertical groups-4, 5, 6, or 7-according to its ten's digit. Its last digit is then placed to the right of the colon. Often, especially if the amount of data is larger, it is desirable to split the stems. For example, instead of placing all values whose ten's digit is 4 in the same "bucket," the stem 4 is repeated twice; all values 40-44 are put in the bucket corresponding to the first stem and all values 45-49 are put in the bucket corresponding to the second stem. This approach is shown in the stem and leaf plot of Figure 3.6. Other variations are also possible. I

Histograms Stem and leaf plots are a type of histogram, a plot that dis- plays the distribution of values for attributes by dividing the possible values into bins and showing the number of objects that fall into each bin. For cate- gorical data, each value is a bin. If this results in too many values, then values are combined in some way. For continuous attributes, the range of values is di- vided into bins-typically, but not necessarily, of equal width-and the values in each bin are counted.

A

LL2 Chapter 3 Exploring Data

43 44 44 44 45 46 46 46 46 47 47 48 48 48 48 48 49 49 49 49 49 49 50 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 2 5 2 5 2 5 2 5 3 54 54 54 54 54 54 55 55 55 55 55 55 55 56 56 56 56 56 56 57 57 57 57 57 57 57 57 58 58 58 58 58 58 58 59 59 59 60 60 60 60 60 60 61 61 61 61 61 61 62 62 62 62 63 63 63 63 63 63 63 63 63 64 64 64 64 64 64 64 65 65 65 65 65 66 66 67 67 67 67 67 67 67 67 68 68 68 69 69 69 69 70 7t 72 72 72 73 74 76 77 77 77 77 79

Figure 3.4. Sepal length data from the lris data set.

34444566667788888999999 0000000000 Lt t l1tfl L222234444445555555666 6667 7 7 7 7 7 7 78888888999 000000 1 1 1 1 t1222233333333344444445555566777 7777 7 8889999 0t22234677779

Figure 3.5. Stem and leaf plot for the sepal length from the lris data set.

3444 566667788888999999 00000000001 1 1 11 1 1 1 122223444++4 5555555666 6667 7 77 7 7778888888999 000000 1 1 1 1 t t22223333333334444444 5 5 5 55 6 67 7 77 7 7 7 7 8889 999 0122234 677779

Figure 3.6. Stem and leaf plot for the sepal length from the lris data set when buckets conesponding to digits are split.

Once the counts are available for each bin, a bar plot is constructed such that each bin is represented by one bar and the area of each bar is proportional

to the number of values (objects) that fall into the corresponding range. If all

intervals are of equal width, then all bars are the same width and the height of a bar is proportional to the number of values in the corresponding bin.

Exarnple 3.8. Figure 3.7 shows histograms (with 10 bins) for sepal length, sepal width, petal length, and petal width. Since the shape of a histogram can depend on the number of bins, histograms for the same data, but with 20

bins, are shown in Figure 3.8. I

There are variations of the histogram plot. A relative (frequency) his- togram replaces the count by the relative frequency. However, this is just a

4 : t r .

t r .

A

A

7 .

Visualization 1-1-3

(a) Sepal length. (b) Sepal width. (c) Petal length (d) Petal width.

Figure 3.7. Histograms of four lris attributes (10 bins).

(a) Sepal length. (b) Sepal width. (c) Petal length.

Figure 3.8. Histograms of four lris attributes (20 bins).

change in scale of the g axis, and the shape of the histogram does not change. Another common variation, especially for unordered categorical data, is the Pareto histogram, which is the same as a normal histogram except that the categories are sorted by count so that the count is decreasing from left to right.

Two-Dimensional Histograms Two-dimensional histograms are also pos- sible. Each attribute is divided into intervals and the two sets of intervals define two-dimensional rectangles of values.

Example 3.9. Figure 3.9 shows a two-dimensional histogram of petal length and petal width. Because each attribute is split into three bins, there are nine rectangular two-dimensional bins. The height of each rectangular bar indicates the number of objects (flowers in this case) that fall into each bin. Most of the flowers fall into only three of the bins-those along the diagonal. It is not possible to see this by looking at the one-dimensional distributions. r

3 . 3

) 0 5 I 1 5 2 Petwnh

(d) Petal width.

LL4 Chapter 3 Exploring Data

Figure 3.9. Two-dimensional histogram of petal length and width in the lris data set.

While two-dimensional histograms can be used to discover interesting facts

about how the values of two attributes co-occur, they are visually more com- plicated. For instance, it is easy to imagine a situation in which some of the columns are hidden bv others.

Box Plots Box plots are another method for showing the distribution of the values of a single numerical attribute. Figure 3.10 shows a labeled box plot for sepal length. The lower and upper ends of the box indicate the 25th and 75th percentiles, respectively, while the line inside the box indicates the value of the 50th percentile. The top and bottom lines of the tails indicate the 10'h and 90th percentiles. Outliers are shown by "+" marks. Box plots are relatively compact, and thus, many of them can be shown on the same plot. Simplified versions of the box plot, which take less space, can also be used.

Example 3.1-0. The box plots for the first four attributes of the Iris data set are shown in Figure 3.11. Box plots can also be used to compare how attributes vary between different classes of objects, as shown in Figure 3.12.

T

Pie Chart A pie chart is similar to a histogram, but is typically used with categorical attributes that have a relatively small number of values. Instead of showing the relative frequency of different values with the area or height of a bar, as in a histogram, a pie chart uses the relative area of a circle to indicate relative frequency. Although pie charts are common in popular articles, they

<- Outlier

<- 90th percentile

<- 75rh percentile

<-- sOth percentile

<_ 25rh percentile

<- 1Oth percentile

Figure 3.10. Description of box plot for sepal length,

Visualization 115

Sepal Length Sepal Width Petal Length Petal Width

Figure 3,11. Box plot for lris attributes.

3 . 3

g o o E

o

o

d

widh M L€ngth &lwidh Spalkngih Sepalwdlh told Length Peblwidh

(b) Versicolour.(a) Setosa. (c) Virginica.

Figure 3.12. Box plots of attributes by lris species.

are used less frequently in technical publications because the size of relative areas can be hard to judge. Histograms are preferred for technical work.

Example 3.11. Figure 3.13 displays a pie chart that shows the distribution of Iris species in the Iris data set. In this case, all three flower types have the same freouencv. r

Percentile Plots and Empirical Cumulative Distribution Functions A type of diagram that shows the distribution of the data more quantitatively is the plot of an empirical cumulative distribution function. While this type of plot may sound complicated, the concept is straightforward. For each value of a statistical distribution, a cumulative distribution function (CDF) shows

T

= LJ

l -

=

S6pal bngth S€palwdh bd bngth &l Wfr

116 Chapter 3 Exploring Data

Versicolour

Figure 3.13. Distribution of the types of lris flowers.

the probability that a point is less than that value. For each observed value, an empirical cumulative distribution function (ECDF) shows the fraction of points that are less than this value. Since the number of points is finite, the empirical cumulative distribution function is a step function.

Example 3.12. Figure 3.14 shows the ECDFs of the Iris attributes. The percentiles of an attribute provide similar information. Figure 3.15 shows the percentile plots of the four continuous attributes of the Iris data set from Table 3.2. The reader should compare these figures with the histograms given in Figures 3.7 and 3.8. r

Scatter Plots Most people are familiar with scatter plots to some extent, and they were used in Section 2.4.5 to illustrate linear correlation. Each data object is plotted as a point in the plane using the values of the two attributes as r and y coordinates. It is assumed that the attributes are either integer- or real-valued.

Example 3.13. Figure 3.16 shows a scatter plot for each pair of attributes of the Iris data set. The different species of Iris are indicated by different markers. The arrangement of the scatter plots of pairs of attributes in this type of tabular format, which is known as a scatter plot matrix, provides an organized way to examine a number of scatter plots simultaneously. I

3.3 Visualization LL7

(a) Sepal Length.

(c) Petal Length.

Figure 3.14.

(b) Sepal Width.

(d) Petal Width.

Empirical CDFs of four lris attributes.

0 2 0 4 6 0 8 0 1 0

Figure 3.15. Percentile plots for sepal length, sepal width, petal length, and petal width.

118 Chapter 3 Exploring Data

@ F - @ r O r Q \ t | r ) ( f ) l r | c \ l \ f ( r ) o l

q$uag ledes qprm ledes

( o S O l t Q c \ l t Q - l O O N O

q$ue; le1ed qptrr le1ed

c\l

o = (d q) o-

-c c')

s 9 (U o o-

ol

s -c '=

- E o- o o

ol

@

t\ -C o) c q)

@ _ (d o- o

g r @

ov) ct (o

E at)

o -c

.9 (t

o o_ (D

(6 <) <t>

o x

(5

<rt Grt o = ctt u-

X X X X X X X X

X f f i X X X X

X X X g X

X X W X X g

p

x

x x

x X X

x X X )

x x g x x x X

x x

X X x

x x g X X

x x X

oo f

+

o o goo

oce9

3.3 Visualization 119

There are two main uses for scatter plots. First, they graphically show the relationship between two attributes. In Section 2.4.5, we saw how scatter plots could be used to judge the degree of linear correlation. (See Figure 2.17.) Scatter plots can also be used to detect non-linear relationships, either directly or by using a scatter plot of the transformed attributes.

Second, when class labels are available, they can be used to investigate the degree to which two attributes separate the classes. If is possible to draw a line (or a more complicated curve) that divides the plane defined by the two attributes into separate regions that contain mostly objects of one class, then it is possible to construct an accurate classifier based on the specified pair of attributes. If not, then more attributes or more sophisticated methods are needed to build a classifier. In Figure 3.16, many of the pairs of attributes (for example, petal width and petal length) provide a moderate separation of the Iris species.

Example 3.14. There are two separate approaches for displaying three at- tributes of a data set with a scatter plot. First, each object can be displayed according to the values of three, instead of two attributes. F igure 3.17 shows a three-dimensional scatter plot for three attributes in the Iris data set. Second, one of the attributes can be associated with some characteristic of the marker, such as its size, color, or shape. Figure 3.18 shows a plot of three attributes of the Iris data set, where one of the attributes, sepal width, is mapped to the size of the marker. r

Extending Two- and Three-Dimensional Plots As illustrated by Fig- ure 3.18, two- or three-dimensional plots can be extended to represent a few additional attributes. For example, scatter plots can display up to three ad- ditional attributes using color or shading, size, and shape, allowing five or six dimensions to be represented. There is a need for caution, however. As the complexity of a visual representation of the data increases, it becomes harder for the intended audience to interpret the information. There is no benefit in packing six dimensions' worth of information into a two- or three-dimensional plot, if doing so makes it impossible to understand.

Visualizing Spatio-temporal Data

Data often has spatial or temporal attributes. For instance, the data may consist of a set of observations on a spatial grid, such as observations of pres- sure on the surface of the Earth or the modeled temperature at various grid points in the simulation of a physical object. These observations can also be

LzO Chapter 3 Exploring Data

Figure 3.17. Three-dimensional scatter plot ol sepal width, sepal length, and petalwidth.

4 Petal Length

Figure 3.18, Scatter plot of petal length versus petalwidth, with the size of the marker indicating sepal width.

2

c 1 . 5

c o

J 1

o o @ 0 . 5

0

*:;'H*: "**-i,S#".*f

3.3 Visualization LzL

Figure 3.19, Contour plot of SST for December 1 998.

made at various points in time. In addition, data may have only a temporal component, such as time series data that gives the daily prices of stocks.

Contour Plots For some three-dimensional data, two attributes specify a position in a plane, while the third has a continuous value, such as temper- ature or elevation. A useful visualization for such data is a contour plot, which breaks the plane into separate regions where the values of the third attribute (temperature, elevation) are roughly the same. A common example of a contour plot is a contour map that shows the elevation of land locations.

Example 3.15. Figure 3.19 shows a contour plot of the average sea surface temperature (SST) for December 1998. The land is arbitrarily set to have a temperature of 0oC. In many contour maps, such as that of Figure 3.19, the contour lines that separate two regions are labeled with the value used to separate the regions. For clarity, some of these labels have been deleted. r

Surface Plots Like contour plots, surface plots use two attributes for'the r and 3l coordinates. The third attribute is used to indicate the height above

I22 Chapter 3 Exploring Data

t

(a) Set of 12 points. (b) Overall density function----surface plot.

Figure 3,20. Density of a set of 12 points.

the plane defined by the first two attributes. While such graphs can be useful, they require that a value of the third attribute be defined for all combinations of values for the first two attributes, at least over some range. AIso, if the surface is too irregular, then it can be difficult to see all the information, unless the plot is viewed interactively. Thus, surface plots are often used to describe mathematical functions or physical surfaces that vary in a relatively smooth manner.

Example 3.16. Figure 3.20 shows a surface plot of the density around a set of 12 points. This example is further discussed in Section 9.3.3. r

Vector Field Plots In some data, a characteristic may have both a mag- nitude and a direction associated with it. For example, consider the flow of a substance or the change of density with location. In these situations, it can be useful to have a plot that displays both direction and magnitude. This type of plot is known as a vector plot.

Example 3.17. Figure 3.2I shows a contour plot of the density of the two smaller density peaks from Figure 3.20(b), annotated with the density gradient

vectors.

Lower-Dimensional Slices Consider a spatio-temporal data set that records sorne quantity, such as temperature or pressure) at various locations over time. Such a data set has four dimensions and cannot be easily displayed by the types

3.3 Visualization L23

\ \ \ l l l l t r r \ \ \ l l l . t

t t l l l r t t l l l \

Figure 3.21, Vector plot of the gradient (change) in density for the bottom two density peaks of Figure 3.20.

of plots that we have described so far. However, separate "slices" of the data can be displayed by showing a set of plots, one for each month. By examining the change in a particular area from one month to another, it is possible to notice changes that occur, including those that may be due to seasonal factors.

Example 3.18. The underlying data set for this example consists of the av- erage monthly sea level pressure (SLP) from 1982 to 1999 on a 2.5o by 2.5' Iatitude-longitude grid. The twelve monthly plots of pressure for one year are shown in Figure 3.22. In this example, we are interested in slices for a par- ticular month in the year 1982. More generally, we can consider slices bf the data along any arbitrary dimension.

Animation Another approach to dealing with slices of data, whether or not time is involved, is to employ animation. The idea is to display successive two-dimensional slices of the data. The human visual system is well suited to detecting visual changes and can often notice changes that might be difficult to detect in another manner. Despite the visual appeal of animation, a set of still plots, such as those of Figure 3.22, can be more useful since this type of visualization allows the information to be studied in arbitrary order and for arbitrary amounts of time.

L24 Chapter 3

January

Exploring Data

February

April

July

May

August

Figure 3,22. Monthly plots of sea level pressure over the 12 months of 1982.

3.3.4 Visualizing Higher-Dimensional Data

This section considers visualization techniques that can display more than the handful of dimensions that can be observed with the techniques just discussed. However, even these techniques are somewhat limited in that they only show some aspects of the data.

Matrices An image can be regarded as a rectangular array of pixels, where each pixel is characterized by its color and brightness. A data matrix is a rectangular array of values. Thus, a data matrix can be visualized as an image by associating each entry of the data matrix with a pixel in the image. The brightness or color of the pixel is determined by the value of the corresponding entry of the matrix.

W,8;, 3.3 Visualization I25

Figure 3.23. Plot of the lris data matrix where columns have been standardized to have a mean of 0 and standard deviation of 1.

Ssrca V€cidour Virginl€

Figure 3.24. Plot of the lris conelation matrix.

There are some important practical considerations when visualizing a data matrix. If class labels are known, then it is useful to reorder the data matrix so that all objects of a class are together. This makes it easier, for example, to detect if all objects in a class have similar attribute values for some attributes. If different attributes have different ranges, then the attributes are ofben stan- dardized to have a mean of zero and a standard deviation of 1. This prevents the attribute with the largest magnitude values from visually dominating the plot.

Example 3.19. Figure 3.23 shows the standardized data matrix for the Iris data set. The first 50 rows represent Iris flowers ofthe species Setosa, the next 50 Versicolour, and the last 50 Virginica. The Setosa flowers have petal width and length well below the average, while the Versicolour flowers have petal width and length around average. The Virginica flowers have petal width and length above average. l

It can also be useful to look for structure in the plot of a proximity matrix for a set of data objects. Again, it is useful to sort the rows and columns of the similarity matrix (when class labels are known) so that all the objects of a class are together. This allows a visual evaluation of the cohesiveness of each class and its separation from other classes.

Example 3.20. Figure 3.24 shows the correlation matrix for the Iris data set. Again, the rows and columns are organized so that all the flowers of a particular species are together. The flowers in each group are most similar

L26 Chapter 3 Exploring Data

to each other, but Versicolour and Virginica are more similar to one another than to Setosa. r

If class labels are not known, various techniques (matrix reordering and seriation) can be used to rearrange the rows and columns of the similarity matrix so that groups of highly similar objects and attributes are together and can be visually identified. Effectively, this is a simple kind of clustering. See Section 8.5.3 for a discussion of how a proximity matrix can be used to investigate the cluster structure of data.

Parallel Coordinates Parallel coordinates have one coordinate axis for each attribute, but the different axes are parallel to one other instead of per- pendicular, as is traditional. Furthermore, an object is represented as a line instead of as a point. Specifically, the value of each attribute of an object is mapped to a point on the coordinate axis associated with that attribute, and these points are then connected to form the line that represents the object.

It might be feared that this would yield quite a mess. However, in many cases, objects tend to fall into a small number of groups, where the points in each group have similar values for their attributes. If so, and if the number of data objects is not too large, then the resulting parallel coordinates plot can reveal interesting patterns.

Example 3.2L. Figure 3.25 shows a parallel coordinates plot of the four nu- merical attributes of the Iris data set. The lines representing objects of differ- ent classes are distinguished by their shading and the use of three different line styles-solid, dotted, and dashed. The parallel coordinates plot shows that the classes are reasonably well separated for petal width and petal length, but less well separated for sepal length and sepal width. Figure 3.25 is another parallel coordinates plot of the same data, but with a different ordering of the axes. r

One of the drawbacks of parallel coordinates is that the detection of pat- terns in such a plot may depend on the order. For instance, if lines cross a Iot, the picture can become confusing, and thus, it can be desirable to order the coordinate axes to obtain sequences of axes with less crossover. Compare Figure 3.26, where sepal width (the attribute that is most mixed) is at the left of the figure, to Figure 3.25, where this attribute is in the middle.

Star Coordinates and Chernoff Faces

Another approach to displaying multidimensional data is to encode objects as glyphs or icons-symbols that impart information non-verbally. More

3.3 Visualization L27

I o o E

c o o

o 5

Sepal Width Petal Length Petal Width

I o o E c c) o

o f (!

Sepal Width Sepal Length Petal Length Petal Width

Figure 3.26. A parallel coordinates plot of the four lris attributes with the attributes reordered to emphasize similarities and dissimilarities of groups

Figure 3.25. A parallel coordinates plot of the four lris attributes.

1-28 Chapter 3 Exploring Data

specifically, each attribute of an object is mapped to a particular feature of a glyph, so that the value of the attribute determines the exact nature of the feature. Thus, at a glance, we can distinguish how two objects differ.

Star coordinates are one example of this approach. This technique uses one axis for each attribute. These axes all radiate from a center point, like the spokes of a wheel, and are evenly spaced. Typically, all the attribute values are mapped to the range [0,1].

An object is mapped onto this star-shaped set of axes using the following process: Each attribute value of the object is converted to a fraction that represents its distance between the minimum and maximum values of the attribute. This fraction is mapped to a point on the axis corresponding to this attribute. Each point is connected with a line segment to the point on the axis preceding or following its own axis; this forms a polygon. The size and shape of this polygon gives a visual description of the attribute values of the object. For ease of interpretation, a separate set of axes is used for each object. In other words, each object is mapped to a polygon. An example of a star coordinates plot of flower 150 is given in Figure 3.27(a).

It is also possible to map the values of features to those of more familiar objects, such as faces. This technique is named Chernoff faces for its creator, Herman Chernoff. In this technique, each attribute is associated with a specific feature of a face, and the attribute value is used to determine the way that the facial feature is expressed. Thus, the shape of the face may become more elongated as the value of the corresponding data feature increases. An example of a Chernoff face for flower 150 is given in Figure 3.27(b).

The program that we used to make this face mapped the features to the four features listed below. Other features of the face, such as width between the eyes and length of the mouth, are given default values.

Data Feature Facial Feature sepal length sepal width petal length petal width

size of face forehead/jaw relative arc length shape of forehead shape ofjaw

Example 3.22. A more extensive illustration of these two approaches to view- ing multidimensional data is provided by Figures 3.28 and 3.29, which shows the star and face plots, respectively, of 15 flowers from the Iris data set. The first 5 flowers are of species Setosa, the second 5 are Versicolour, and the last 5 are Virginica. r

3.3 Visualization L2g

oro

(b) Chernoff face of Iris 150.

Figure 3.27. Star coordinates graph and Chernoff face of the 150th flower of the lris data set.

I\ AJ 5

<t> \,/ 55

-/t\s \t/

1 0 5

Figute 3.28. Plot of 15 lris flowers using star coordinates.

/Cit /-F\ v v

2 3 4 5

/ o r o \ \:./ 55

/oro) \7 1 0 5

0 4 3 4

,,f\<vv 53 54

,,'1t\ ,-T\ \ | / / \ l , / \Z \'/

'103 104

rD 54

/\ /oro) \, '104

/ o r o \ \_-/ 53

/ o r o \ t l \7 1 0 3

(a) Star graph of Iris 150.

A 2

/4\ €--F v/

52

+ \t/ 102

Z\ I o r e \ \:-/ 52

/eto) f ,1 \, \:/ 101 102

il\ 4J

,4\=vz 5 1

,,,,\ Y-T-7 \l/

J 0 l

'I

l^\

/ o r o \ \_-/

5 1

:-;4"r6$h fp+;fr;o[,

Figure 3.29. A plot of 15 lris flowers using Chernoff faces.

130 Chapter 3 Exploring Data

Despite the visual appeal of these sorts of diagrams, they do not scale well, and thus, they are of limited use for many data mining problems. Nonetheless, they may still be of use as a means to quickly compare small sets of objects that have been selected by other techniques.

3 . 3 . 5 D o ' s a n d D o n ' t s

To conclude this section on visualization, we provide a short list of visualiza- tion do's and don'ts. While these guidelines incorporate a lot of visualization wisdom, they should not be followed blindly. As always, guidelines are no substitute for thoughtful consideration of the problem at hand.

ACCENT Principles The follov,ring are the ACCEN? principles for ef- fective graphical display put forth by D. A. Burn (as adapted by Michael Friendlv):

Apprehension Ability to correctly perceive relations among variables. Does the graph maximize apprehension of the relations among variables?

Clarity Ability to visually distinguish all the elements of a graph. Are the most important elements or relations visually most prominent?

Consistency Ability to interpret a graph based on similarity to previous graphs. Are the elements, symbol shapes, and colors consistent with their use in previous graphs?

Efficiency Ability to portray a possibly complex relation in as simple a way as possible. Are the elements of the graph economically used? Is the graph easy to interpret?

Necessity The need for the graph, and the graphical elements. Is the graph a more useful way to represent the data than alternatives (table, text)? Are all the graph elements necessary to convey the relations?

Tbuthfulness Ability to determine the true value represented by any graph- ical element by its magnitude relative to the implicit or explicit scale. Are the graph elements accurately positioned and scaled?

T\rfte's Guidelines Edward R. Tufte has also enumerated the following principles for graphical excellence;

3.4 OLAP and Multidimensional Data Analysis 131

Graphical excellence is the well-designed presentation of interesting data- a matter of substance, of statistics, and of design.

Graphical excellence consists of complex ideas communicated with clar- ity, precision, and efficiency.

Graphical excellence is that which gives to the viewer the greatest num- ber of ideas in the shortest time with the least ink in the smallest space.

Graphical excellence is nearly always multivariate.

And graphical excellence requires telling the truth about the data.

3.4 OLAP and Multidimensional Data Analysis

In this section, we investigate the techniques and insights that come from viewing data sets as multidimensional arrays. A number of database sys- tems support such a viewpoint, most notably, On-Line Analytical Processing (OLAP) systems. Indeed, some of the terminology and capabilities of OLAP systems have made their way into spreadsheet programs that are used by mil- Iions of people. OLAP systems also have a strong focus on the interactive analysis of data and typically provide extensive capabilities for visualizing the data and generating summary statistics. For these reasons, our approach to multidimensional data analysis will be based on the terminology and concepts common to OLAP systems.

3.4.L Representing Iris Data as a Multidimensional Array

Most data sets can be represented as a table, where each row is an object and each column is an attribute. In many cases, it is also possible to view the data as a multidimensional array. We illustrate this approach by representing the Iris data set as a multidimensional array.

Table 3.7 was created by discretizing the petal length and petal width attributes to have values of low, med'ium, and hi,gh and then counting the number of flowers from the Iris data set that have particular combinations of petal width, petal length, and species type. (For petal width, the cat- egories low, med'ium, and hi,gh correspond to the intervals [0, 0.75), [0.75, 1.75), [7.75, oo), respectively. For petal length, the categories low, med'ium, and hi,gh correspond to the intervals 10, 2.5), 12.5, 5), [5, m), respectively.)

I32 Chapter 3 Exploring Data

Table 3.7. Number of flowers having a particular combination of petal width, petal length, and species

rype. Petal Length Petal Width Species T Count

Iov,r low

medium medium medium medium

high high high high

low medium

low medium

high high

medium medium

high high

Setosa Setosa Setosa

Versicolour Versicolour Virginica

Versicolour Virginica

Versicolour Virginica

4 0

2 2

+L)

3 3 2 a

2 44

Petal widrh

= o

E . = E o E

-c . 9 -c

Virginica Versicolour Setosa

h i g h

medium

low

Petal width

Figure 3.30. A multidimensional data representation for the lris data set.

3.4 OLAP and Multidimensional Data Analysis 133

Table 3.8. Crosstabulation of flowers accord- ing to petal length and width for flowers of the Setosa species.

Table 3.9, Cross-tabulation of flowers accord- ing to petal length and width for flowers of the Versicolour species.

width width Iow medium high

low medium

high

Table 3.10. Cross-tabulation of flowers ac- cording to petal length and width for flowers of the Virginica species.

width

Empty combinations-those combinations that do not correspond to at least one flower-are not shown.

The data can be organized as a multidimensional array with three dimen- sions cortesponding to petal width, petal length, and species type, as illus- trated in Figure 3.30. For clarity, slices of this array are shown as a set of three two-dimensional tables, one for each species-see Tables 3.8, 3.9, and 3.10. The information contained in both Table 3.7 and Figure 3.30 is the same. However, in the multidimensional representation shown in Figure 3.30 (and Tables 3.8, 3.9, and 3.10), the values of the attributes-petal width, petal length, and species type-are array indices.

What is important are the insights can be gained by looking at data from a multidimensional viewpoint. Tables 3.8, 3.9, and 3.10 show that each species of Iris is characterized by a different combination of values of petal length and width. Setosa flowers have low width and length, Versicolour flowers have medium width and length, and Virginica flowers have high width and length.

3.4.2 Multidimensional Data: The General Case

The previous section gave a specific example of using a multidimensional ap- proach to represent and analyze a familiar data set. Here we describe the general approach in more detail.

0 0 0 0 4 3 3 0 2 2

+l b0 et

j

L34 Chapter 3 Exploring Data

The starting point is usually a tabular representation of the data, such as that of Table 3.7, which is called a fact table. Two steps are necessary in order to represent data as a multidimensional arrayi identification of the dimensions and identification of an attribute that is the focus of the analy- sis. The dimensions are categorical attributes or, as in the previous example, continuous attributes that have been converted to categorical attributes. The values of an attribute serve as indices into the array for the dimension corre- sponding to the attribute, and the number of attribute values is the size of that dimension. In the previous example, each attribute had three possible values, and thus, each dimension was of size three and could be indexed by threevalues. Thisproduceda3 x 3 x 3multidimensional array.

Each combination of attribute values (one value for each difierent attribute) defines a cell of the multidimensional array. To illustrate using the previous example, if petal length : lou), petal width : mediutr\ and species : Setosa, a specific cell containing the value 2 is identified. That is, there are only two flowers in the data set that have the specified attribute values. Notice that each row (object) of the data set in Table 3.7 corresponds to a cell in the multidimensional array.

The contents of each cell represents the value of a target quantity (target variable or attribute) that we are interested in analyzing. In the Iris example, the target quantity is the nurnber of flowers whose petal width and length fall within certain limits. The target attribute is quantitative because a key goal of multidimensional data analysis is to look aggregate quantities, such as totals or averages.

The following summarizes the procedure for creating a multidimensional data representation from a data set represented in tabular form. First, identify the categorical attributes to be used as the dimensions and a quantitative attribute to be used as the target of the analysis. Each row (object) in the table is mapped to a cell of the multidimensional array. The indices of the cell are specified by the values of the attributes that were selected as dimensions, while the value of the cell is the value of the target attribute. Cells not defined by the data are assumed to have a value of 0.

Example 3.23. To further illustrate the ideas just discussed, we present a more traditional example involving the sale of products.The fact table for this example is given by Table 3.11. The dimensions of the multidimensional rep- resentation are the product ID, locati,on, and date attributes, while the target attribute is the reaenue. Figure 3.31 shows the multidimensional representa- tion of this data set. This larger and more complicated data set will be used to illustrate additional concepts of multidimensional data analysis. r

3.4 OLAP and Multidimensional Data Analysis 135

3.4.3 Analyzing Multidimensional Data

In this section, we describe different multidimensional analysis techniques. In particular, we discuss the creation of data cubes, and related operations, such as slicing, dicing, dimensionality reduction, roll-up, and drill down.

Data Cubes: Computing Aggregate Quantities

A key motivation for taking a multidimensional viewpoint of data is the im- portance of aggregating data in various ways. In the sales example, we might wish to find the total sales revenue for a specific year and a specific product. Or we might wish to see the yearly sales revenue for each location across all products. Computing aggregate totals involves fixing specific values for some of the attributes that are being used as dimensions and then summing over all possible values for the attributes that make up the remaining dimensions. There are other types of aggregate quantities that are also of interest, but for simplicity, this discussion will use totals (sums).

Table 3.12 shows the result of summing over all locations for various com- binations of date and product. For simplicity, assume that all the dates are within one year. Ifthere are 365 days in a year and 1000 products, then Table 3.12 has 365,000 entries (totals), one for each product-data pair. We could also specify the store location and date and sum over products, or specify the location and product and sum over all dates.

Table 3.13 shows the marginal totals of Table 3.12. These totals are the result of further summing over either dates or products. In Table 3.13, the total sales revenue due to product 1, which is obtained by summing across row 1 (over all dates), is $370,000. The total sales revenue on January 1, 2004, which is obtained by summing down column 1 (over all products), is

$527,362. The total sales revenue, which is obtained by summing over all rows and columns (all times and products) is $227,352,127. All of these totals are for all locations because the entries of Table 3.13 include all locations.

A key point of this example is that there are a number of different totals (aggregates) that can be computed for a multidimensional array, depending on how many attributes we sum over. Assume that there are n dimensions and that the ith dimension (attribute) has si possible values. There are n different ways to sum only over a single attribute. If we sum over dimension j, then we obtain s1 x ... * sj-1 * tj+t * ... * s' totals, one for each possible combination of attribute values of the n- l other attributes (dimensions). The totals that result from summing over one attribute form a multidimensional array of n-I dimensions and there are n such arrays of totals. In the sales example, there

136 Chapter 3 Exploring Data

Table 3.11, Sales revenue of products (in dollars) for various locations and times.

Product ID Location Date Revenue

: : 1 Minneapolis 1 Chicago

i p*i.

27 Minneapolis 27 Chicago

n Paris

i i Oct. 18, 2004 $250 O c t . 1 8 , 2 0 0 4 $ 7 9

Oct. 18, 2004 301

: : Oct. 18, 2004 $2,321 Oct. 18, 2004 $3,278

Oct. 18, 2004 $1,325 : :

{"'v

Product lD

Figure 3.31. Multidimensional data representation for sales data.

tr

2 7 tr

3.4 OLAP and Multidimensional Data Analvsis L37

Table 3.12. Totals that result from summing over all locations for a fixed time and product.

date

Table 3.13. Table 3.12 with marginaltotals.

date J a n 1 . 2 0 0 4 J a n 2 , 2 0 0 4 Dec 31. 2004 | total

$3.800.020

27,362 , r 2 7

are three sets of totals that result from summing over only one dimension and

each set of totals can be displayed as a two-dimensional table.

If we sum over two dimensions (perhaps starting with one of the arrays

of totals obtained by summing over one dimension), then we will obtain a

multidimensional array of totals with rz - 2 dimensions. There will be (!)

distinct anays of such totals. For the sales examples, there will be () : g

arays of totals that result from summing over location and product, Iocation

and time, or product and time. In general, summing over ,k dimensions yields

([) arrays of totals, each with dimension n - k.

A multidimensional representation of the data, together with all possible

totals (aggregates), is known as a data cube. Despite the name, the size of

each dimension-the number of attribute values-does not need to be equal.

AIso, a data cube may have either more or fewer than three dimensions. More

importantly, a data cube is a generalization of what is known in statistical

terminology as a cross-tabulation. If marginal totals were added, Tables

3.8, 3.9, or 3.10 would be typical examples of cross tabulations.

J a n I . 2 O 0 4 J a n 2 , 2 0 0 4

138 Chapter 3 Exploring Data

Dimensionality Reduction and Pivoting

The aggregation described in the last section can be viewed as a form of dimensionality reduction. Specifically, the jth dimension is eliminated by summing over it. Conceptually, this collapses each "column" of cells in the jth

dimension into a single cell. For both the sales and Iris examples, aggregating over one dimension reduces the dimensionality of the data from 3 to 2. If si is the number of possible values of the 7'h dimension, the number of cells is reduced by a factor of sr. Exercise 17 on page 143 asks the reader to explore the difference between this type of dimensionality reduction and that of PCA.

Pivoting refers to aggregating over all dimensions except two. The result is a two-dimensional cross tabulation with the two specified dimensions as the only remaining dimensions. Table 3.13 is an example of pivoting on date and product.

Slicing and Dicing

These two colorful names refer to rather straightforward operations. Slicing is selecting a group of cells from the entire multidimensional array by specifying a specific value for one or more dimensions. Tables 3.8, 3.9, and 3.10 are three slices from the Iris set that were obtained by specifying three separate values for the species dimension. Dicing involves selecting a subset of cells by specifying a range of attribute values. This is equivalent to defining a subarray from the complete array. In practice, both operations can also be accompanied by aggregation over some dimensions.

Roll-Up and Drill-Down

In Chapter 2, attribute values were regarded as being "atomic" in some sense. However, this is not always the case. In particular, each date has a number of properties associated with it such as the year, month, and week. The data can also be identified as belonging to a particular business quarter) or if the application relates to education, a school quarter or semester. A location also has various properties: continent, country, state (province, etc.), and city. Products can also be divided into various categories, such as clothing, electronics, and furniture.

Often these categories can be organized as a hierarchical tree or lattice. For instance) years consist of months or weeks, both of which consist of days. Locations can be divided into nations, which contain states (or other units of local government), which in turn contain cities. Likewise, any category

3.5 Bibliographic Notes 1-39

of products can be further subdivided. For example, the product category,

furniture, can be subdivided into the subcategories, chairs, tables, sofas, etc.

This hierarchical structure gives rise to the roll-up and drill-down opera-

tions. To illustrate, starting with the original sales data, which is a multidi-

mensional array with entries for each date, we can aggregate (roll up) the

sales across all the dates in a month. Conversely, given a representation of the

data where the time dimension is broken into months, we might want to split

the monthly sales totals (drill down) into daily sales totals. Of course, this

requires that the underlying sales data be available at a daily granularity.

Thus, roll-up and drill-down operations are related to aggregation. No'

tice, however, that they differ from the aggregation operations discussed until

now in that they aggregate cells within a dimension, not across the entire

dimension.

3.4.4 Final Comments on Multidimensional Data Analysis

Multidimensional data analysis, in the sense implied by OLAP and related sys-

tems, consists of viewing the data as a multidimensional array and aggregating

data in order to better analyze the structure of the data. For the Iris data,

the differences in petal width and length are clearly shown by such an anal-

ysis. The analysis of business data, such as sales data, can also reveal many

interesting patterns, such as profitable (or unprofitable) stores or products.

As mentioned, there are various types of database systems that support

the analysis of multidimensional data. Some of these systems are based on

relational databases and are known as ROLAP systems. More specialized

database systems that specifically employ a multidimensional data represen-

tation as their fundamental data model have also been designed. Such systems

are known as MOLAP systems. In addition to these types of systems, statisti-

cal databases (SDBs) have been developed to store and analyze various types

of statistical data, e.g., census and public health data, that are collected by

governments or other large organizations. References to OLAP and SDBs are

provided in the bibliographic notes.

3.5 Bibliographic Notes

Summary statistics are discussed in detail in most introductory statistics

books, such as 192]. References for exploratory data analysis are the classic

text by Tirkey [104] and the book by Velleman and Hoaglin [105]. The basic visualization techniques are readily available, being an integral

part of most spreadsheets (Microsoft EXCEL [95]), statistics programs (sAS

1,4O Chapter 3 Exploring Data

[99], SPSS [102], R [96], and S-PLUS [98]), and mathematics software (MAT- LAB [94] and Mathematica [93]). Most of the graphics in this chapter were generated using MATLAB. The statistics package R is freely available as an open source software package from the R project.

The literature on visualization is extensive, covering many fields and many decades. One of the classics of the field is the book by Tufte [103]. The book by Spence [tOt], which strongly influenced the visualization portion of this chapter, is a useful reference for information visualization-both principles and techniques. This book also provides a thorough discussion of many dynamic visualization techniques that were not covered in this chapter. Two other books on visualization that may also be of interest are those by Card et al.

[87] and Fayyad et al. [S9]. Finally, there is a great deal of information available about data visualiza-

tion on the World Wide Web. Since Web sites come and go frequently, the best strategy is a search using "information visualization," "data visualization," or "statistical graphics." However, we do want to single out for attention "The Gallery of Data Visualization," by Friendly [90]. The ACCENT Principles for effective graphical display as stated in this chapter can be found there, or as originally presented in the article by Burn [86].

There are a variety of graphical techniques that can be used to explore whether the distribution of the data is Gaussian or some other specified dis- tribution. Also, there are plots that display whether the observed values are statistically significant in some sense. We have not covered any of these tech- niques here and refer the reader to the previously mentioned statistical and mathematical packages.

Multidimensional analysis has been around in a variety of forms for some time. One of the original papers was a white paper by Codd [88], the father of relational databases. The data cube was introduced by Gray et al. [91], who described various operations for creating and manipulating data cubes within a relational database framework. A comparison of statistical databases and OLAP is given by Shoshani [100]. Specific information on OLAP can be found in documentation from database vendors and many popular books. Many database textbooks also have general discussions of OLAP, often in the context of data warehousing. For example, see the text by Ramakrishnan and Gehrke [97].

Bibliography [86] D. A. Burn. Designing Effective Statistical Graphs. In C. R. Rao, editor, Hand,book of

Stati,stics 9. Elsevier/North-Holland, Amsterdam, The Netherlands, September 1993.

3.6 Exercises L4L

[87] S. K. Card, J. D. MacKinlay, and B. Shneiderman, editors. Read,ings ,in Informat'ion Visualization: Using Vision to Thznlc. Morgan Kaufmann Publishers, San Francisco, CA, January 1999.

[88] E. F. Codd, S. B. Codd, and C. T. Smalley. Providing OLAP (On-line Analytical Processing) to User- Analysts: An IT Mandate. White Paper, E.F. Codd and Associates, 1 9 9 3 .

f89] U. M. Fayyad, G. G. Grinstein, and A. Wierse, editors. Information V'isualization'in Data Mining and, Knowled,ge Discouery. Morgan Kaufmann Publishers, San FYancisco, CA, September 2001.

[90] M. F]iendly. Gallery of Data Visualization. http://www.math.yorku.ca/SCS/Gallery/, 2 0 0 5 .

[91] J. Gray, S. Chaudhuri, A. Bosworth, A. Layman, D. Reichart, M. Venkatrao, F. Pellow, and H. Pirahesh. Data Cube: A Relational Aggregation Operator Generalizing Group- By, Cross-Tab, and Sub-Totals. Journal Data Mining and Knouledge Discouerg, l(I): 29-53, 1997.

f92] B. W. Lindgren. Stat'istical Theory. CRC Press, January 1993.

[93] Mathematica 5.1. Wolfram Research, Inc. http://www.wolfram.comf ,2005.

[94] MATLAB 7.0. The MathWorks, Inc. http://www.mathworks.com, 2005.

[95] Microsoft Excel 2003. Microsoft, Inc. http://www.microsoft.comf ,2003.

[96] R: A language and environment for statistical computing and graphics. The R Project for Statistical Computing. http: / /www.r-project.org/, 2005.

[97] R Ramakrishnan and J. Gehrke. Database Management Systems. McGraw-Hill, 3rd edition, August 2002.

198] S-PLUS. Insightful Corporation. http: //www.insightful.com, 2005.

[99] SAS: Statistical Analysis System. SAS Institute Inc. http:f f www.sas.com/, 2005.

[100] A. Shoshani. OLAP and statistical databases: similarities and differences. In Proc. of the Siateenth ACM SIGACT-SIGMOD-SIGART Symp. on Princ,i,ples of Database Sgstems, pages 185-196. ACM Press, 1997.

[101] R. Spence. Inforrnation Visual'izati,on ACM Press, New York, December 2000.

[102] SPSS: Statistical Package for the Social Sciences. SPSS, lnc. http://www.spss.com/, 2 0 0 5 .

f103] E. R. Tufte. The Visual Di.splag of Quantitatiue Informat'ion. Graphics Press, Cheshire, CT, March 1986.

[104] J. W. T\rkey. Exploratory data analys,is. Addison-Wesley, 1977.

[105] P. Velleman and D. Hoaglin. The ABC's of EDA: Applications, Basics, and Computing of Exploratorg Data Analysis. Duxbury, 1981.

3.6 Exercises

1. Obtain one of the data sets available at the UCI Machine Learning Repository and apply as many of the different visualization techniques described in the chapter as possible. The bibliographic notes and book Web site provide pointers to visualization software.

L42 Chapter 3 Exploring Data

2. Identify at least two advantages and two disadvantages ofusing color to visually represent information.

E d .

What are the arrangement issues that arise with respect to three-dimensional plots?

Discuss the advantages and disadvantages of using sampling to reduce the num- ber of data objects that need to be displayed. Would simple random sampling (without replacement) be a good approach to sampling? Why or why not?

Describe how you would create visualizations to display information that de- scribes the following types of systems.

(a) Computer networks. Be sure to include both the static aspects of the network, such as connectivity, and the dynamic aspects, such as traffic.

(b) The distribution of specific plant and animal species around the world for a specific moment in time.

(c) The use of computer resources, such as processor time, main memory, and disk, for a set of benchmark database programs.

(d) The change in occupation of workers in a particular country over the last thirty years. Assume that you have yearly information about each person that also includes gender and level of education.

Be sure to address the following issues:

o Representation. How will you map objects, attributes, and relation- ships to visual elements?

o Arrangement. Are there any special considerations that need to be taken into account with respect to how visual elements are displayed? Spe- cific examples might be the choice of viewpoint, the use of transparency, or the separation of certain groups of objects.

o Selection. How will you handle a larqe number of attributes and data objects?

Describe one advantage and one disadvantage of a stem and leaf plot with respect to a standard histogram.

How might you address the problem that a histogram depends on the number and location of the bins?

Describe how a box plot can give information about whether the value of an attribute is symmetrically distributed. What can you say about the symmetry of the distributions of the attributes shown in Fieure 3.11?

9. Compare sepal length, sepal width, petal length, and petal width, using Figure 3 . r 2 .

3 .

A

6 .

7 .

8 .

1 0 .

1 1 .

1 2 .

1 3 .

r4.

1 5 .

1 6 .

3.6 Exercises L43

Comment on the use of a box plot to explore a data set with four attributes: age, weight, height, and income.

Give a possible explanation as to why most of the values of petal length and width fall in the buckets along the diagonal in Figure 3.9.

Use Figures 3.14 and 3.15 to identify a characteristic shared by the petal width and petal length attributes.

Simple line plots, such as that displayed in Figure 2.L2 on page 56, which shows two time series, can be used to effectively display high-dimensional data. For example, in Figure 2.72 iI is easy to tell that the frequencies of the two time series are different. What characteristic of time series allows the effective visualization of high-dimensional data?

Describe the types of situations that produce sparse or dense data cubes. Illus- trate with examples other than those used in the book.

How might you extend the notion of multidimensional data analysis so that the target variable is a qualitative variable? In other words, what sorts of summary statistics or data visualizations would be of interest?

Construct a data cube from Table 3.14. Is this a dense or sparse data cube? If it is sparse, identify the cells that empty.

Product ID Location ID Number Sold I 1 2 2

I 3 1 2

1 0 6 5 22

17. Discuss the differences between dimensionality reduction based on aggregation and dimensionality reduction based on techniques such as PCA and SVD.

Table 3.14. Fact table for Exercise 16.

Classification: Basic Concepts, Decision Trees, and Model Evaluation

Classification, which is the task of assigning objects to one of several predefined categories, is a pervasive problem that encompasses many diverse applications. Examples include detecting spam email messages based upon the message header and content, categorizing cells as malignant or benign based upon the results of MRI scans, and classifying galaxies based upon their shapes (see F i g u r e 4 . 1 ) .

(a) A spiral galaxy (b) An elliptical galaxy.

Figure 4.1. Classification of galaxies. The images are from the NASA website.

L46 Chapter 4 Classification

Input Output

onr.'l1,," set l---) ---) Class labelClassilication model

Figure 4.2. Classification as the task of mapping an input attribute set x into its class label g.

This chapter introduces the basic concepts of classification, describes some of the key issues such as model overfitting, and presents methods for evaluating and comparing the performance of a classification technique. While it focuses mainly on a technique known as decision tree induction, most of the discussion in this chapter is also applicable to other classification techniques, many of which are covered in Chapter 5.

4.L Preliminaries

The input data for a classification task is a collection of records. Each record, also known as an instance or example, is characterized by a tuple (*,A), where x is the attribute set and gr is a special attribute, designated as the class label (also known as category or target attribute). Table 4.1 shows a sample data set used for classifying vertebrates into one of the following categories: mammal, bird, fish, reptile, or amphibian. The attribute set includes properties of a vertebrate such as its body temperature, skin cover, method of reproduction, ability to fly, and ability to live in water. Although the attributes presented in Table 4.L are mostly discrete, the attribute set can also contain continuous features. The class label, on the other hand, must be a discrete attribute. This is a key characteristic that distinguishes classification from regression, a predictive modeling task in which g is a continuous attribute. Regression techniques are covered in Appendix D.

Definition 4.1 (Classification). Classification is the task of learning a tar- get function / that maps each attribute set x to one of the predefined class Iabels g.

The target function is also known informally as a classification model. A classification model is useful for the following purposes.

Descriptive Modeling A classification model can serve as an explanatory tool to distinguish between objects of different classes. For example, it would be useful-for both biologists and others-to have a descriptive model that

4 . r

Table 4.1. The vertebrate data set.

Preliminaries 147

summarizes the data shown in Table 4.1 and explains what features define a vertebrate as a mammal, reptile, bird, fish, or amphibian.

Predictive Modeling A classification model can also be used to predict the class iabel of unknown records. As shown in Figure 4.2, a classification model can be treated as a black box that automatically assigns a class label when presented with the attribute set of an unknown record. Suppose we are given the following characteristics of a creature known as a gila monster:

We can use a classification model built from the data set shown in Table 4.1 to determine the class to which the creature belongs.

Classification techniques are most suited for predicting or describing data sets with binary or nominal categories. They are less effective for ordinal categories (e.g., to classify a person as a member of high-, medium-, or low- income group) because they do not consider the implicit order among the categories. Other forms of relationships, such as the subclass-superclass re- lationships among categories (e.g., humans and apes are primates, which in

Name lJody

Temperature

Skin Cover

Gives Birth

Aquatic Creature

Aerral Creature

Has Legs nates

beH Ulass Label

human python salmon whale frog komodo dragon bat plgeon cat leopard shark turtle penguln porcuptne eel salamander

warm-blooded cold-blooded cold-blooded

warm-blooded cold-blooded cold-blooded

warm-blooded warm-blooded warm-blooded cold-blooded

cold-blooded warm-blooded warm-blooded cold-blooded cold-blooded

hair feathers

fur scales

scales feathers

quills scales none

hair scales scales hair none scales

yes no no yes no no

yes no yes yes

no n o yes no no

no no yes yes

seml no

no no no yes

semt seml no yes

semr

no no no no no no

yes yes no no

no n o no no no

yes no no no yes yes

yes yes yes no

yes yes yes no yes

no yes no no yes no

yes no no no

no no yes no yes

mammal reptile

fish mammal

amphibian reptile

mammal bird

mammal fish

reptile bird

mammal fish

amphibian

Name lJody Iemperature

Dlfln

Cover Gives Birth

Aquatic Creature

Aerial Creature

Has Legs

Hiber- nates

Ulass Label

gila monster cold-blooded scales no no no yes yes

1,48 Chapter 4 Classification

turn, is a subclass of mammals) are also ignored. The remainder of this chapter focuses only on binary or nominal class labels.

4.2 General Approach to Solving a Classification Problem

A classification technique (or classifier) is a systematic approach to building classification models from an input data set. Examples include decision tree

classifiers, rule-based classifiers, neural networks, support vector machines,

and naive Bayes classifiers. Each technique employs a learning algorithm to identify a model that best fits the relationship between the attribute set and class label of the input data. The model generated by a learning algorithm should both fit the input data well and correctly predict the class labels of records it has never seen before. Therefore, a key objective of the learning algorithm is to build models with good generalization capability; i.e., models that accurately predict the class labels of previously unknown records.

Figure 4.3 shows a general approach for solving classification problems. First, a training set consisting of records whose class labels are known must

t ;t'' I I ruooel IK",':

Figure 4.3. General approach for building a classification model.

4.2 General Approach to Solving a Classification Problem L4g

Table 4.2. Confusion matrix for a 2-class problem.

Predicted Class C l a s s : 7 U l a s s : U

Actual Class

Class : I T. t 7 7 J I O U l a s s : 0 ./ 01 .Ioo

be provided. The training set is used to build a classification model, which is subsequently applied to the test set, which consists of records with unknown class labels.

Evaluation of the performance of a classification model is based on the counts of test records correctly and incorrectly predicted by the model. These counts are tabulated in a table known as a confusion matrix. TabIe 4.2 depicts the confusion matrix for a binary classification problem. Each entry f4 in this table denotes the number of records from class e predicted to be of class 7. For instance, /s1 is the number of records from class 0 incorrectly predicted as class 1. Based on the entries in the confusion matrix, the total number of correct predictions made by the model ir ("ftr + /oo) and the total number of incorrect predictions ir (/ro + /or).

Although a confusion matrix provides the information needed to determine how well a classification model performs, summarizing this information with a single number would make it more convenient to compare the performance of different models. This can be done using a performance metric such as accuracy, which is defined as follows:

Accuracy: Number of correct oredictions f n * f o o ( 4 . 1 ) Total number of predictions fr l- fn * ,for * ,foo'

Equivalently, the performance of a model can be expressed error rate, which is given by the following equation:

terms of its

n__^_- _^r^ Number of wrong predictionsH : r r n r r o f 6 Total number of predictions

. t r r J r c - T J O 7

f : - r * f r c * , f o r * . f o o ( 4 . 2 )

Most classification algorithms seek models that attain the highest accuracy, or equivalently, the lowest error rate when applied to the test set. We will revisit the topic of model evaluation in Section 4.5.

1-50 Chapter 4 Classification

4.3 Decision Tree Induction

This section introduces a decision tree classifier, which is a simple yet widely

used classification technique.

4.3.L How a Decision TYee Works

To illustrate how classification with a decision tree works, consider a simpler

version of the vertebrate classification problem described in the previous sec-

tion. Instead of classifying the vertebrates into five distinct groups of species,

we assign them to two categories: mammals and non-mammals.

Suppose a new species is discovered by scientists. How can we tell whether

it is a mammal or a non-mammal? One approach is to pose a series of questions

about the characteristics of the species. The first question we may ask is

whether the species is cold- or warm-blooded. If it is cold-blooded, then it is

definitely not a mammal. Otherwise, it is either a bird or a mammal. In the

latter case, we need to ask a follow-up question: Do the females of the species give birth to their young? Those that do give birth are definitely mammals,

while those that do not are likely to be non-mammals (with the exception of

egg-laying mammals such as the platypus and spiny anteater). The previous example illustrates how we can solve a classification problem

by asking a series of carefully crafted questions about the attributes of the

test record. Each time we receive an answer? a follow-up question is asked

until we reach a conclusion about the class label of the record. The series of questions and their possible answers can be organized in the form of a decision

tree, which is a hierarchical structure consisting of nodes and directed edges.

Figure 4.4 shows the decision tree for the mammal classification problem. The

tree has three types of nodes:

o A root node that has no incoming edges and zero or more outgoing

edges.

o Internal nodes, each of which has exactly one incoming edge and two

or more outgoing edges.

o Leaf or terminal nodes, each of which has exactly one incoming edge

and no outgoing edges.

In a decision tree, each leaf node is assigned a class label. The non-

terminal nodes, which include the root and other internal nodes, contain

attribute test conditions to separate records that have different characteris- tics. For example, the root node shown in Figure 4.4 uses the attribute Body

4.3 Decision Thee Induction 151

Figure 4.4. A decision tree for the mammal classification problem.

Temperature to separate warm-blooded from cold-blooded vertebrates. Since all cold-blooded vertebrates are non-mammals, a leaf node labeled Non-mamma]s is created as the right child of the root node. If the vertebrate is warm-blooded, a subsequent attribute, Gives Birth, is used to distinguish mammals from other warm-blooded creatures, which are mostly birds.

Classifying a test record is straightforward once a decision tree has been constructed. Starting from the root node, we apply the test condition to the record and follow the appropriate branch based on the outcome of the test. This will lead us either to another internal node, for which a new test condition is applied, or to a leaf node. The class label associated with the leaf node is then assigned to the record. As an illustration, Figure 4.5 traces the path in the decision tree that is used to predict the class label of a flamingo. The path terminates at a leaf node labeled Non-rna-nmal-s.

4.3.2 How to Build a Decision Tbee

In principle, there are exponentially many decision trees that can be con- structed from a given set of attributes. While some of the trees are more accu- rate than others, finding the optimal tree is computationally infeasible because of the exponential size of the search space. Nevertheless, efficient algorithms have been developed to induce a reasonably accurate, albeit suboptimal, de- cision tree in a reasonable amount of time. These algorithms usually employ a greedy strategy that grows a decision tree by making a series of locally op-

L52 Chapter 4 Classification

Unlabeled data

Non- mammats

Figure 4.5. Classifying an unlabeled vertebrate. The dashed lines represent the outcomes of applying various attribute test conditions on the unlabeled vertebrate. The vertebrate is eventually assigned to lIg lrf 6a-aammal class.

timum decisions about which attribute to use for partitioning the data. One

such algorithm is fluntts algorithm, which is the basis of many existing de-

cision tree induction algorithms, including ID3, C4.5, and CART. This section presents a high-level discussion of Hunt's algorithm and illustrates some of its

design issues.

flunt's Algorithm

In Hunt's algorithm, a decision tree is grown in a recursive fashion by parti-

tioning the training records into successively purer subsets. Let Dl be the set

of training records that are associated with node t and g : {At,U2, . . . ,A"} be

the class labels. The following is a recursive definition of Hunt's algorithm.

Step 1: If ail the records in Dt belong to the same class y1, then t is a leaf

node labeled as y.

Step 2: If D; contains records that belong to more than one class, an at-

tribute test condition is selected to partition the records into smaller

subsets. A child node is created for each outcome of the test condi-

tion and the records in D1 are distributed to the children based on the

outcomes. The algorithm is then recursively applied to each child node.

Name Body temperature Gives Birth Class Flamingo Warm N o

4.3 Decision TYee Induction 1-53

a."" """""""J"

Figure 4.6. Training set for predicting borrowers who will default on loan payments.

To illustrate how the algorithm works, consider the problem of predicting whether a loan applicant will repay her loan obligations or become delinquent, subsequently defaulting on her loan. A training set for this problem can be constructed by examining the records of previous borrowers. In the example shown in Figure 4.6, each record contains the personal information of a bor- rower along with a class label indicating whether the borrower has defaulted on loan payments.

The initial tree for the classification problem contains a single node with class label Defaulted = No (see Figure a.7@)), which means that most of the borrowers successfully repaid their loans. The tree, however, needs to be refined since the root node contains records from both classes. The records are subsequently divided into smaller subsets based on the outcomes of the Hone Owner test condition) as shown in Figure 4.7(b). The justification for choosing this attribute test condition will be discussed later. For now, we will assume that this is the best criterion for splitting the data at this point. Hunt's algorithm is then applied recursively to each child of the root node. From the training set given in Figure 4.6, notice that all borrowers who are home owners successfully repaid their loans. The left child of the root is therefore a leaf node labeled Def aulted = No (see Figure 4.7(b)). For the right child, we need to continue applying the recursive step of Hunt's algorithm until all the records belong to the same class. The trees resulting from each recursive step are shown in Figures a.7@) and (d).

Yes No No Yes No No Yes No No No

125K 1 0 0 K 70K 120K 95K 60K 220K 85K 75K 90K

L54 Chapter 4 Classification

Figure 4.7. Hunt's algorithm for inducing decision trees.

Hunt's algorithm will work if every combination of attribute values is present in the training data and each combination has a unique class label. These assumptions are too stringent for use in most practical situations. Ad- ditional conditions are needed to handle the following cases:

1. It is possible for some of the child nodes created in Step 2 to be empty; i.e., there are no records associated with these nodes. This can happen if none of the training records have the combination of attribute values associated with such nodes. In this case the node is declared a leaf node with the same class label as the majority class of training records associated with its parent node.

2. In Step 2, if all the records associated with D; have identical attribute values (except for the class label), then it is not possible to split these records any further. In this case, the node is declared a leaf node with the same class label as the majority class of training records associated with this node.

4.3 Decision TYee Induction 155

Design Issues of Decision TYee Induction

A learning algorithm for inducing decision trees must address the following two issues.

1. How should the training records be split? Each recursive step of the tree-growing process must select an attribute test condition to divide the records into smaller subsets. To implement this step, the algorithm must provide a method for specifying the test condition for different attribute types as well as an objective measure for evaluating the goodness of each test condition.

2. How should the splitting procedure stop? A stopping condition is needed to terminate the tree-growing process. A possible strategy is to continue expanding a node until either all the records belong to the same class or all the records have identical attribute values. Although both conditions are sufficient to stop any decision tree induction algorithm, other criteria can be imposed to allow the tree-growing procedure to terminate earlier. The advantages of early termination will be discussed later in Section 4.4.5.

4.3.3 Methods for Expressing Attribute Test Conditions

Decision tree induction algorithms must provide a method for expressing an attribute test condition and its correspondins outcomes for different attribute types.

Binary Attributes The test condition for a binary attribute generates two potential outcomes, as shown in Figure 4.8.

Warm- Cold- blooded blooded

Figure 4,8. Test condition for binary attributes.

156 Chapter 4 Classification

{Manied} {Single, Divorced)

Divorced

(a) Multiway split

{Single} {Married, Divorced)

{Single, {Divorced} Married)

Single

O R

(b) Binary split {by grouping attribute values}

Figure 4.9. Test conditions for nominal attributes.

Nominal Attributes Since a nominal attribute can have many values, its

test condition can be expressed in two ways, as shown in Figure 4.9. For

a multiway split (Figure 4.9(a)), the number of outcomes depends on the

number of distinct values for the corresponding attribute. For example, if

an attribute such as marital status has three distinct values-single, married,

or divorced-its test condition will produce a three-way split. On the other

hand, some decision tree algorithms, such as CART, produce only binary splits

by considering all 2k-1 - 1 ways of creating a binary partition of k attribute

values. Figure 4.9(b) illustrates three different ways of grouping the attribute

values for marital status into two subsets.

Ordinal Attributes Ordinal attributes can also produce binary or multiway

splits. Ordinal attribute values can be grouped as long as the grouping does

not violate the order property of the attribute values. Figure 4.10 illustrates

various ways of splitting training records based on the Shirt Size attribute.

The groupings shown in Figures 4.10(a) and (b) preserve the order among

the attribute values, whereas the grouping shown in Figure a.10(c) violates

this property because it combines the attribute values Small and Large into

{Small, {Large, Medium) Extra Large)

(b) (c)

Figure 4.10. Different ways of grouping ordinal attribute values.

the same partition while Mediun and Extra Large are combined into another partition.

Continuous Attributes For continuous attributes, the test condition can be expressed as a comparison test (A < u) or (A 2 ,) with binary outcomes, or a range query with outcomes of the form ui I A l ut+t, fot 'i: L,... ,k. The difference between these approaches is shown in Figure 4.11. For the binary case, the decision tree algorithm must consider all possible split positions u, and it selects the one that produces the best partition. For the multiway split, the algorithm must consider all possible ranges of continuous values. One approach is to apply the discretization strategies described in Section 2.3.6 on page 57. After discretization, a new ordinal value will be assigned to each discretized interval. Adjacent intervals can also be aggregated into wider ranges as long as the order property is preserved.

(b)

Figure 4.1 1. Test condition for continuous attributes.

{Small, {Medium, Large) Extra Large)

4.3 Decision Tlee Induction L57

{ S m a l l } { M e d i u m , L a r g e , Extra Large)

(a)

{10K, 25K} {25K, 50K} {50K, 80K}

(a)

158 Chapter 4 Classification

Figure 4.12. Multiway versus binary splits.

4.3.4 Measures for Selecting the Best Split

There are many measures that can be used to determine the best way to split

the records. These measures are defined in terms of the class distribution of

the records before and after splitting. Let p(i.lt) denote the fraction of records belonging to class i at a given node

t. We sometimes omit the reference to node I and express the fraction as p,;.

In a two-class problem, the class distribution at any node can be written as

(po,pt), where Pt:7 -Po. To illustrate, consider the test conditions shown

in Figure 4.12. The class distribution before splitting is (0.5,0.5) because

there are an equal number of records from each class. If we split the data

using the Gender attribute, then the class distributions of the child nodes are

(0.6,0.4) and (0.4,0.6), respectively. Although the classes are no longer evenly

distributed, the child nodes still contain records from both classes. Splitting

on the second attribute, Car Type, will result in purer partitions.

The measures developed for selecting the best split are often based on the

degree of impurity of the child nodes. The smaller the degree of impurity, the

more skewed the class distribution. For example, a node with class distribu-

tion (0,1) has zero impurity, whereas a node with uniform class distribution (0.5,0.5) has the highest impurity. Examples of impurity measures include

c - l

Entropy(t) : -ln?.lt)rog2pllt), o1_,

Gini(r) : L -|,lp11t)12, i,:o

Classification error(t) : 1 -max[p(ilt)]'

where c is the number of classes and 0log2 0 : O ," entropy calculations.

( 4 . 3 )

( 4 . 4 )

( 4 . 5 )

C 0 : 1 C 1 : 3

C 0 : 8 C 1 : 0

C 0 : 0 C 1 : 1

C 0 : 0 C 1 : 1

C 0 : 1 C 1 : 0

0.7

o.4

Entropy

eini -.'-' )':-'l'l-- ' - ) . ' . . ' . . . - - .

- a a t . a , t

- ' \ . \ - ' a a

- a t - . a ' - ' a . - a a

, ' . ' t ' \ ,

- r " - . a ' t

\ ' \ . \

,' ,'- Misclassification error '\.

4.3 Decision Tlee Induction 159

Gini : r - (016)2 - (616)2 : 0 Entropy : -(016) Iog2(0/6) - (616) log2(6/6) :0 Error : 1 - maxl0/6,6/6] : 0

Gini : | - Gl6)2 - Fl6)2 : 0.278 Entropy : -(Il6)togr(716) - (516)ro92616): 0.650 Error : 1 - maxfl/6,516]l : 9.167

Gini : r - (Jl6)2 - (3/6)2 : 9.5 Entropy : -(316) Iog2(3/6) - (3/6)logr(3/6) : 1 Error : 1 - maxlS I 6, 3 I 6l : 0.b

0 . 1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 o

Figure 4.13. Comparison among the impurity measures for binary classification problems.

Figure 4.13 compares the values of the impurity measures for binary classi- fication problems. p refers to the fraction of records that belong to one of the two classes. Observe that all three measures attain their maximum value when the class distribution is uniform (i.e., when p : 0.5). The minimum values for the measures are attained when all the records belong to the same class (i.e., when p equals 0 or 1). We next provide several examples of computing the different impuritv measures.

Node Nr Uount Class:0 0 Class:1 6

Node Ah Count CIass:0 I Class:1

Node l/a Uount Class:0 .) Ulass:1 3

160 Chapter 4 Classification

The preceding examples, along with Figure 4.13, illustrate the consistency

among different impurity measures. Based on these calculations, node lft has

the lowest impurity value, followed by l{z and lf3. Despite their consistency,

the attribute chosen as the test condition may vary depending on the choice

of impurity measure, as will be shown in Exercise 3 on page 198. To determine how well a test condition performs, we need to compare the

degree of impurity of the parent node (before splitting) with the degree of

impurity of the child nodes (after splitting). The larger their difference, the

better the test condition. The gain, A, is a criterion that can be used to

determine the goodness of a split:

( 4 . 6 ) j : t

where 1(.) is the impurity measure of a given node, ly' is the total number of

records at the parent node, k is the number of attribute values, and l/(u3)

is the number of records associated with the child node, u7. Decision tree

induction algorithms often choose a test condition that maximizes the gain

A. Since /(parent) is the same for all test conditions, maximizing the gain is

equivalent to minimizing the weighted average impurity measures of the child

nodes. Finally, when entropy is used as the impurity measure in Equation 4.6,

the difference in entropy is known as the information gain, 461o.

Splitting of Binary Attributes

Consider the diagram shown in Figure 4.14. Suppose there are two ways to

split the data into smaller subsets. Before splitting, the Gini index is 0.5 since

there are an equal number of records from both classes. If attribute A is chosen

to split the data, the Gini index for node N1 is 0.4898, and for node N2, it

is 0.480. The weighted average of the Gini index for the descendent nodes is (71L2) x 0.4898 + (5112) x 0.480 : 0.486. Similarly, we can show that the

weighted average of the Gini index for attribute B is 0.375. Since the subsets

for attribute B have a smaller Gini index, it is preferred over attribute A.

Splitting of Nominal Attributes

As previously noted, a nominal attribute can produce either binary or multi-

way splits, as shown in Figure 4.15. The computation of the Gini index for a

binary split is similar to that shown for determining binary attributes. For the

first binary grouping of the Car Type attribute, the Gini index of {Sports,

k

A:I(pare"t) - i Yttr),

4 . 3 Decision Tree Induction L6L

Figure 4.14. Splitting binary attributes.

6''A 6;) {Sports,

--:-sl,- {Family,

tuxut).,/ \Tmrry) nxuryl.z \<{orts}

(a) Binary split (b) Multiway split

Figure 4.15. Splitting nominal attributes.

Luxury) is 0.4922 and the Gini index of {Fa:rily} is 0.3750. The weighted average Gini index for the grouping is equal to

16120 x 0.4922 + 4120 x 0.3750 : 0.468.

Similarly, for the second binary grouping of {Sports} and {Fanily, Luxury}, the weighted average Gini index is 0.167. The second grouping has a lower Gini index because its corresponding subsets are much purer.

L62 Chapter 4 Classification

Figure 4.16. Splitting continuous attributes.

For the multiway split, the Gini index is computed for every attribute value.

Since Gini({raniry}) : 0.375, Gini({Sports}) : 0, and Gini({Luxury}) :

0.219, the overall Gini index for the multiway split is equal to

4120 x 0.375 + 8120 x 0 + 8120 x 0.219 : 0.163.

The multiway split has a smaller Gini index compared to both two-way splits.

This result is not surprising because the two-way split actually merges some

of the outcomes of a multiway split, and thus, results in less pure subsets.

Splitting of Continuous Attributes

Consider the example shown in Figure 4.16, in which the test condition Annual

Income ( u is used to split the training records for the loan default classifica-

tion problem. A brute-force method for finding u is to consider every value of

the attribute in the ly' records as a candidate split position. For each candidate

u, the data set is scanned once to count the number of records with annual

income less than or greater than u. We then compute the Gini index for each

candidate and choose the one that gives the lowest value. This approach is

computationally expensive because it requires O(,nf) operations to compute

the Gini index at each candidate split position. Since there are -l[ candidates,

the overall complexity of this task is O(N\. To reduce the complexity, the

training records are sorted based on their annual income, a computation that

requires O(,n/logli) time. Candidate split positions are identified by taking

the midpoints between two adjacent sorted values: 55, 65, 72, and so on. How-

ever, unlike the brute-force approach, we do not have to examine all ly' records

when evaluating the Gini index of a candidate split position. For the first candidate. u : 55. none of the records has annual income less

than $55K. As a result, the Gini index for the descendent node with Annual

Decision Tlee Induction 163

fncome < $55K is zero. On the other hand, the number of records with annual income greater than or equal to $55K is 3 (for class Yes) and 7 (for class No), respectively. Thus, the Gini index for this node is 0.420. The overall Gini index for this candidate split position is equal to 0 x 0 + 1 x 0.420 :0.420.

For the second candidate. u : 65. we can determine its class distribution by updating the distribution of the previous candidate. More specifically, the new distribution is obtained by examining the class label of the record with the lowest annual income (i.e., $60K). Since the class label for this record is No, the count for class No is increased from 0 to 1 (for Annual Income < $65K) and is decreased from 7 to 6 (for Annual- Incone > $65K). The distribution for class Yes remains unchanged. The new weighted-average Gini index for this candidate split position is 0.400.

This procedure is repeated until the Gini index values for all candidates are computed, as shown in Figure 4.16. The best split position corresponds to the one that produces the smallest Gini index, i.e., u:97. This procedure is less expensive because it requires a constant amount of time to update the class distribution at each candidate split position. It can be further optimized by considering only candidate split positions located between two adjacent records with different class labels. For example, because the first three sorted records (with annual incomes $60K, $70K, and $75K) have identical class labels, the best split position should not reside between $60K and $75K. Therefore, the candidate split positions at a : $55K, $65K, $72K, $87K, $92K, $110K, $I22K, $772K, and $230K are ignored because they are located between two adjacent records with the same class labels. This approach allows us to reduce the number of candidate split positions from 11 to 2.

Gain Ratio

Impurity measures such as entropy and Gini index tend to favor attributes that have a large number of distinct values. Figure 4.12 shows three alternative test conditions for partitioning the data set given in Exercise 2 on page 198. Comparing the first test condition, Gender, with the second, Car Type, it is easy to see that Car Type seems to provide a better way of splitting the data since it produces purer descendent nodes. However, if we compare both conditions with Customer ID, the latter appears to produce purer partitions. Yet Custoner ID is not a predictive attribute because its value is unique for each record. Even in a less extreme situation, a test condition that results in a large number of outcomes may not be desirable because the number of records associated with each partition is too small to enable us to make anv reliable predictions.

4 . 3

L64 Chapter 4 Classification

There are two strategies for overcoming this problem. The first strategy is

to restrict the test conditions to binary splits only. This strategy is employed

by decision tree algorithms such as CART. Another strategy is to modify the

splitting criterion to take into account the number of outcomes produced by

the attribute test condition. For example, in the C4.5 decision tree algorithm,

a splitting criterion known as gain ratio is used to deterrnine the goodness

of a split. This criterion is defined as follows:

" Ai"foUaln ratlo : ;--;.,--*-. 5pt1t rnlo

( 4 . 7 )

Here, Split Info: -Df:rP(ui)logrP(u6) and /c is the total number of splits. For example, if each attribute value has the same number of records, then

Yi, : P(u,;) : Llk and the split information would be equal to log2 k. This

example suggests that if an attribute produces a large number of splits, its

split information will also be large, which in turn reduces its gain ratio.

4.3.5 Algorithm for Decision Tlee Induction

A skeleton decision tree induction algorithm called TreeGrowth is shown in Algorithm 4.7. The input to this algorithm consists of the training records

E and the attribute set F. The algorithm works by recursively selecting the

best attribute to split the data (Step 7) and expanding the leaf nodes of the

Algorithm 4.L A skeleton decision tree induction algorithm. TreeGrowth (8, F)

1: if stopping-cond(E,f') : true t}nen 2: leaf : createNode$. 3 : l e a f . l a b e l : C t a s s i f y ( E ) . 4: rcturn leaf . 5: else 6: root : createNode0. 7'. root.test-cond: f ind-best-split(E, F). 8: let V : {T.'lo is a possible outcome of root.test-cond }. 9: for each u €V do

10: Eo : {e I root.test-cond(e) : u and e e E}. 11: chi,ld: TreeGrowth(8", F). 12: add chi,ld as descendent of root and Iabel the edge (root -- chi,ld) as u. 13: end for 14: end if I5: return root.

Decision Tlee Induction 165

tree (Steps 11 and 12) until the stopping criterion is met (Step 1). The details of this algorithm are explained below:

1. The createNode$ function extends the decision tree by creating a new node. A node in the decision tree has either a test condition, denoted as node.test-cond, or a class label, denoted as node.label.

2. The f ind-best-split0 function determines which attribute should be selected as the test condition for splitting the training records. As pre- viously noted, the choice of test condition depends on which impurity measure is used to determine the goodness of a split. Some widely used measures include entropy, the Gini index, and the 12 statistic.

3. The Cl-assifyQ function determines the class label to be assigned to a leaf node. For each leaf node t,let p(ilt) denote the fraction of training records from class i associated with the node f. In most cases? the leaf node is assigned to the class that has the majority number of training records:

leaf .label: argmax p(i,lt), ( 4 . 8 )

where the argmax operator returns the argument i that maximizes the expression p(i,lt). Besides providing the information needed to determine the class label of a leaf node, the fraction p(i,lt) can also be used to es- timate the probability that a record assigned to the leaf node t belongs to class z. Sections 5.7.2 and 5.7.3 describe how such probability esti mates can be used to determine the oerformance of a decision tree under different cost functions.

4. The stopping-cond0 function is used to terminate the tree-growing pro- cess by testing whether all the records have either the same class label or the same attribute values. Another way to terminate the recursive function is to test whether the number of records have fallen below some minimum threshold.

After building the decision tree, a tree-pruning step can be performed to reduce the size of the decision tree. Decision trees that are too large are susceptible to a phenomenon known as overfitting. Pruning helps by trim- ming the branches of the initial tree in a way that improves the generalization capability of the decision tree. The issues of overfitting and tree pruning are discussed in more detail in Section 4.4.

4 . 3

166 Chapter 4 Classification

htto://www.cs. u mn.edu/-kumar

Ml NDS/Ml NDS_papers.htm

(b) Graph of a Web session. (c) Derived attributes for Web robot detection.

Input data for Web robot detection.Figure 4.17.

4.3.6 An Example: Web Robot Detection

Web usage mining is the task of applying data mining techniques to extract useful patterns from Web access logs. These patterns can reveal interesting characteristics of site visitors; e.g., people who repeatedly visit a Web site and view the same product description page are more likely to buy the product if certain incentives such as rebates or free shipping are offered.

In Web usage mining, it is important to distinguish accesses made by hu- man users from those due to Web robots. A Web robot (also known as a Web crawler) is a software program that automatically locates and retrieves infor- mation from the Internet by following the hyperlinks embedded in Web pages. These programs are deployed by search engine portals to gather the documents necessary for indexing the Web. Web robot accesses must be discarded before applying Web mining techniques to analyze human browsing behavior.

1 6 0 1 1 1 1 1 1 08/Au9/2004 1 0 : 1 5 : 2 1

http://www cs umn edu/ -kumar

HTTP/1 1 200 u24 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)

1 6 0 . 1 1 1 1 . 1 108/Aug/2004 1 0 : 1 5 : 3 4

u E l http://www cs.umn.edu/ -kumar/MINDS

HTTPN,1200 41378 *umar

Mozilla/4 0 (compatible; MSIE 6 0; Windows NT 5.0)

6 0 1 1 1 1 , 1 1uu/AUg/zuu4 1 0:15:41

GET nIIP://WWW.CS,Umn.eOU/ -kumar/MINDS/MINDS

naners hlm

HTTPN.l 200 1 0 1 8 5 1 crftp://www cs umn eou/ ,kumar/MINDS

MOZila/4.U (compatible; MSIE 6.0; Windows NT 5.0)

6 0 1 1 1 1 , 1 1JU/AUg/ZUU4 1 0 : 1 6 : 1 1

GET nttp//www uS,urln.euu/ -kumar/papers/papers html

t P t 1 . 1 200 7463 )ttp://wwwcs.umn edu/ .kumar

M0zlila/4.u (compatible; MSIE 6 0; Windows NT 5.0)

3 5 9 2 2 08/Aug/2004 1 0 : 1 6 : 1 5

u E l http://www cs umn edu/ -sieinbac

HTTPN ( 200 3149 Mozilla/s.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/2004061 6

(a) Example of a Web server log

Total number of pages retrieved in a Web session Total number of imaqe paqes retrieved in a Web session Total amount ol time sDent bv Web site visitor

more than once in a Web session

of requests made using HEAD method

4.3 Decision Tree Induction 167

This section describes how a decision tree classifier can be used to distin- guish between accesses by human users and those by Web robots. The input data was obtained from a Web server log, a sample of which is shown in Figure a.I7(a). Each line corresponds to a single page request made by a Web client (a user or a Web robot). The fields recorded in the Web log include the IP address of the client, timestamp of the request, Web address of the requested document, size of the document, and the client's identity (via the user agent field). A Web session is a sequence of requests made by a client during a single visit to a Web site. Each Web session can be modeled as a directed graph, in which the nodes correspond to Web pages and the edges correspond to hyper- links connecting one Web page to another. Figure 4.L7(b) shows a graphical representation of the first Web session given in the Web server log.

To classify the Web sessions, features are constructed to describe the char- acteristics of each session. Figure a.fi(c) shows some of the features used for the Web robot detection task. Among the notable features include the depth and breadth of the traversal. Depth determines the maximum dis- tance of a requested page, where distance is measured in terms of the num- ber of hyperlinks away from the entry point of the Web site. For example, the home page http://vutw.cs.umn.edu/-lparar is assumed to be at depth 0, whereas http : / /wuw. cs. unn. edu/kunar/MINDS/MINDS-papers.htn is lo- cated at depth 2. Based on the Web graph shown in Figure 4.I7(b), the depth attribute for the first session is equal to two. The breadth attribute measures the width of the corresponding Web graph. For example, the breadth of the Web session shown in Figure 4.17(b) is equal to two.

The data set for classification contains 2916 records, with equal numbers of sessions due to Web robots (class 1) and human users (class 0). L0% of the data were reserved for training while the remaining 90% were used for testing. The induced decision tree model is shown in Figure 4.18. The tree has an error rate equal to 3.8% on the training set and 5.37o on the test set.

The model suggests that Web robots can be distinguished from human users in the following way:

1 . Accesses by Web robots tend to be broad but shallow, whereas accesses by human users tend to be more focused (narrow but deep).

Unlike human users, Web robots seldom retrieve the image pages asso- ciated with a Web document.

Sessions due to Web robots tend to be long and contain a large number of requested pages.

2 .

3 .

168 Chapter 4 Classification

Decision Tree: d e p t h = 1 :

breadth> 7: class 1 breadth<= 7:

breadth <= 3: lmagePages> 0.375: class 0 lmagePages<= 0.375:

totalPages<= 6: class 1 totalPages> 6: I breadth <= 1: class 1 I breadth > 1: class 0

width > 3: M u l t i l P = 0 :

lmagePages<= 0.1333: class 1 lmagePages> 0.1333: breadth <= 6: class 0 breadth > 6: class 1

M u l t i l P = 1 : I TotalTime <= 361: class 0 I TotalTime > 361: class 1

d e p t h > 1 : MultiAgent = 0:

deoth > 2: class 0 deoth < 2:

MultiAgent = 1: I totalPages <= 81 : class 0 I totalPages > 81 : class 1

M u l t i l P = 1 : c l a s s 0 M u l t i l P = 0 :

breadth <= 6: class 0 breadth > 6: I RepeatedAccess <= 0.322: class 0 I RepeatedAccess > 0.322: class 1

Figure 4.18. Decision tree modelfor Web robot detection,

4. Web robots are more likely to make repeated requests for the same doc-

ument since the Web pages retrieved by human users are often cached by the browser.

4.3.7 Characteristics of Decision Tbee Induction

The following is a summary of the important characteristics of decision tree induction algorithms.

1. Decision tree induction is a nonparametric approach for building classifi-

cation models. In other words, it does not require any prior assumptions regarding the type of probability distributions satisfied by the class and other attributes (unlike some of the techniques described in Chapter 5).

2 .

4.3 Decision Tbee Induction L69

Finding an optimal decision tree is an NP-complete problem. Many de- cision tree algorithms employ a heuristic-based approach to guide their search in the vast hypothesis space. For example, the algorithm pre- sented in Section 4.3.5 uses a greedy, top-down, recursive partitioning strategy for growing a decision tree.

Techniques developed for constructing decision trees are computationally inexpensive, making it possible to quickly construct models even when the training set size is very large. Furthermore, once a decision tree has been built, classifying a test record is extremely fast, with a worst-case complexity of O(to), where ,u.r is the maximum depth of the tree.

Decision trees, especially smaller-sized trees, are relatively easy to inter- pret. The accuracies of the trees are also comparable to other classifica- tion techniques for many simple data sets.

Decision trees provide an expressive representation for learning discrete- valued functions. However, they do not generalize well to certain types of Boolean problems. One notable example is the parity function, whose value is 0 (1) when there is an odd (even) number of Boolean attributes with the valueTrue. Accurate modeling of such a function requires a full decision tree with 2d nodes, where d is the number of Boolean attributes (see Exercise 1 on page 198).

Decision tree algorithms are quite robust to the presence of noise, espe- cially when methods for avoiding overfitting, as described in Section 4.4, are employed.

The presence of redundant attributes does not adversely affect the ac- curacy of decision trees. An attribute is redundant if it is strongly cor- related with another attribute in the data. One of the two redundant attributes will not be used for splitting once the other attribute has been chosen. However, if the data set contains many irrelevant attributes, i.e., attributes that are not useful for the classification task, then some of the irrelevant attributes may be accidently chosen during the tree-growing process, which results in a decision tree that is larger than necessary. Feature selection techniques can help to improve the accuracy of deci- sion trees by eliminating the irrelevant attributes during preprocessing. We will investigate the issue of too many irrelevant attributes in Section 4 . 4 . 3 .

3 .

4 .

5 .

6 .

7 .

170

8 .

Chapter4 Classification

Since most decision tree algorithms employ a top-down, recursive parti-

tioning approach, the number of records becomes smaller as we traverse

down the tree. At the leaf nodes, the number of records may be too

small to make a statistically significant decision about the class rep-

resentation of the nodes. This is known as the data fragmentation problem. One possible solution is to disallow further splitting when the

number of records falls below a certain threshold.

A subtree can be replicated multiple times in a decision tree, as illus-

trated in Figure 4.19. This makes the decision tree more complex than

necessary and perhaps more difficult to interpret. Such a situation can

arise from decision tree implementations that rely on a single attribute

test condition at each internal node. Since most of the decision tree al- gorithms use a divide-and-conquer partitioning strategy, the same test

condition can be applied to different parts of the attribute space, thus

Ieading to the subtree replication problem.

9 .

Figure 4.19. Tree replication problem. The same subtree can appear at different branches.

10. The test conditions described so far in this chapter involve using only a

single attribute at a time. As a consequence, the tree-growing procedure

can be viewed as the process of partitioning the attribute space into

disjoint regions until each region contains records of the same class (see

Figure 4.20). The border between two neighboring regions of different

classes is known as a decision boundary. Since the test condition in- volves only a single attribute, the decision boundaries are rectilinear; i.e., parallel to the "coordinate axes." This limits the expressiveness of the

4 . 3 Decision Tree Induction L7L

1

0.9

0.8

0.7

0 . 6

> 0.5

0.4

0.3

0.2

0 . 1

o i v

V

i o i

a l V

v i V i' i v

- - - - - - - - - - - d - - - -

v i v i o

0 r , 0 0 . 1 0.3 0.4 0.5 0.6 0.7 0.8 0.9

x

Figure 4.20. Example of a decision tree and its decision boundaries for a two-dimensional data set.

'I

0.9

0 . 8

o . 7

u,o

o.4

i: .::... .'l ..' '.' ' ; \ . . 1 . ' - '

t a t a a t a t .

t 3

t : . t t . . t '

t

: r t . t '

a ' a o - t

t . . t . " t a

j l t t ' r o o a a- a

t o '

. l

+ + + + \ .

. . i . . 1 ; j - ; - \

0

o.2

0 . 1

oot 0 . 1 o.2 0 . 3 0 . 4 0 . 6 o.7 0 . 9

Figure 4.21. Example of data set that cannot be partitioned optimally using test conditions involving single attributes.

decision tree representation for modeling complex relationships among continuous attributes. Figure 4.21 illustrates a data set that cannot be classified effectively by a decision tree algorithm that uses test conditions involving only a single attribute at a time.

L72 Chapter 4 Classification

An oblique decision tree can be used to overcome this limitation because it allows test conditions that involve more than one attribute. The data set given in Figure 4.21can be easily represented by an oblique decision tree containing a single node with test condition

r + A < I .

Although such techniques are more expressive and can produce more compact trees, finding the optimal test condition for a given node can be computationally expensive.

Constructive induction provides another way to partition the data into homogeneous) nonrectangular regions (see Section 2.3.5 on page 57). This approach creates composite attributes representing an arithmetic or logical combination of the existing attributes. The new attributes provide a better discrimination of the classes and are augmented to the data set prior to decision tree induction. Unlike the oblique decision tree approach, constructive induction is less expensive because it identifies all the relevant combinations of attributes once, prior to constructing the decision tree. In contrast, an oblique decision tree must determine the right attribute combination dynamically, every time an internal node is expanded. However, constructive induction can introduce attribute re- dundancy in the data since the new attribute is a combination of several existing attributes.

11. Studies have shown that the choice of impurity measure has little effect on the performance of decision tree induction algorithms. This is because many impurity measures are quite consistent with each other, as shown in Figure 4.13 on page 159. Indeed, the strategy used to prune the tree has a greater impact on the final tree than the choice of impurity measure.

4.4 Model Overfitting

The errors committed by a classification model are generally divided into two types: training errors and generalization errors. TYaining error, also known as resubstitution error or apparent error, is the number of misclas- sification errors committed on training records, whereas generalization error is the expected error of the model on previously unseen records.

Recall from Section 4.2 that a good classification model must not only fit the training data well, it must also accurately classify records it has never

4.4 Model Overfitting 173

1 8

1 6

1 4

't2

(\l -

1 0

8

b

4

2

0 E 0 1 0

x 1

Figure 4.22. Example of a data set with binary classes.

seen before. In other words, a good model must have low training error as well as low generalization error. This is important because a model that fits

the training data too well can have a poorer generalization error than a model with a higher training error. Such a situation is known as model overfitting.

Overfftting Example in Two-Dimensional Data For a more concrete example of the overfitting problem, consider the two-dimensional data set shown in Figure 4.22. The data set contains data points that belong to two different classes, denoted as class o and class +, respectively. The data points

for the o class are generated from a mixture of three Gaussian distributions, while a uniform distribution is used to generate the data points for the + class. There are altogether 1200 points belonging to the o class and 1800 points be- Ionging to the + class. 30% of. the points are chosen for training, while the remaining 70%o are used for testing. A decision tree classifier that uses the

Gini index as its impurity measure is then applied to the training set. To

investigate the effect of overfitting, different levels of pruning are applied to

the initial, fully-grown tree. Figure .23(b) shows the training and test error rates of the decision tree.

Training sel

:r E ..;* ";T 1i.'_ : ";:'.-tsi.i *i..j..rr ;liS $ .gJ' j:.,. .'^i:#.$"-i."kii

,{. }tll.* fi : rl iT "f, - :-g:'i; it .. *'-+ * **- * 1 ++ +t*+ * , * * * ; . * * - * 3 S * * o * l + * * * +* I

* .^ .dr; qoF oL-Y o r^**l *

** *

* l * * * * t

* a o i * * 3 o - . . * i a : . . - i . ^ o o . i 1 81 61 2

L74 Chapter 4 Classification

0.35 - Training Error .- - Test Error

100 150 200 250 300 Number of Nodes

Figure 4.23. Training and test error rates.

Notice that the training and test error rates of the model are large when the size of the tree is very small. This situation is known as model underfitting. Underfitting occurs because the model has yet to learn the true structure of the data. As a result, it performs poorly on both the training and the test sets. As the number of nodes in the decision tree increases, the tree will have fewer training and test enors. However, once the tree becomes too large, its test error rate begins to increase even though its training error rate continues to decrease. This phenomenon is known as model overfitting.

To understand the overfitting phenomenon, note that the training error of a model can be reduced by increasing the model complexity. For example, the Ieaf nodes of the tree can be expanded until it perfectly fits the training data. Although the training error for such a complex tree is zero, the test error can be large because the tree may contain nodes that accidently fit some of the noise points in the training data. Such nodes can degrade the performance of the tree because they do not generalize well to the test examples. Figure 4.24 shows the structure of two decision trees with different number of nodes. The tree that contains the smaller number of nodes has a higher training error rate, but a lower test error rate compared to the more complex tree.

Overfitting and underfitting are two pathologies that are related to the model complexity. The remainder of this section examines some of the poten- tial causes of model overfitting.

0.25 o Gr h 0.2 Y

ul

o.4

0 . 1 s

0 . 1

0.05

4.4 Model Overfitting L75

(a) Decision tree with 11 leaf nodes.

(b) Decision tree with 24 leaf nodes.

Figure 4.24. Decision trees with ditferent model complexities.

4.4.L Overfitting Due to Presence of Noise

Consider the training and test sets shown in Tables 4.3 and 4.4 for the mammal classification problem. Two of the ten training records are mislabeled: bats and whales are classifi.ed as non-mammals instead of mammals.

A decision tree that perfectly fits the training data is shown in Figure a.25(a). Although the training error for the tree is zero, its error rate on

Table 4.3. An example training set for classifying mammals. Class labels with asterisk symbols repre- sent mislabeled records.

Name Body Temperature

Grves Birth

Four- Iegged

Hibernates Olass Label

porcupme

cat bat whale salamander komodo dragon python salmon eagle guppy

warm-blooded warm-blooded warm-blooded warm-blooded cold-blooded cold-blooded cold-blooded cold-blooded

warm-blooded cold-blooded

yes yes yes yes no no no no no yes

yes yes no no yes yes no no no no

yes no yes no yes no yes no no no

yes yes no no no no no no no no

+

2 n

+

L76 Chapter 4 Classification

(a) Model Ml (b) Model M2

Figure 4.25. Decision tree induced from the data set shown in Table 4.3.

the test set is 30%. Both humans and dolphins were misclassified as non- mammals because their attribute values for Body Tenperature, Gives Birth, and Four-legged are identical to the mislabeled records in the training set. Spiny anteaters, on the other hand, represent an exceptional case in which the class label of a test record contradicts the class labels of other similar records in the training set. Errors due to exceptional cases are often unavoidable and establish the minimum error rate achievable bv anv classifier.

Table 4.4. An example test set for classifying mammals. Name tsody

Temperature G l ves Birth

Four- legged

Hibernates UIaSS Label

human plgeon elephant leopard shark turtle penguin eel dolphin spiny anteater gila monster

warm-blooded warm-blooded warm-blooded coid-blooded cold-blooded cold-blooded cold-blooded

warm-blooded warm-blooded cold-blooded

yes no yes yes no no no yes no no

no no yes no yes no no no yes yes

no no no no no no no no yes yes

yes no yes no no no no yes yes no

4.4 Model Overfitting L77

In contrast, the decision tree M2 shown in Figure 4.25(b) has a lower test error rate (I0%) even though its training error rate is somewhat higher (20%). It is evident that the first decision tree, MI, has overfitted the training data because there is a simpler model with lower error rate on the test set. The Four-legged attribute test condition in model MI is spurious because it fits the mislabeled training records, which leads to the misclassification of records in the test set.

4.4.2 Overfitting Due to Lack of Representative Samples

Models that make their classification decisions based on a small number of training records are also susceptible to overfitting. Such models can be gener- ated because of lack of representative samples in the training data and learning algorithms that continue to refine their models even when few training records are available. We illustrate these effects in the example below.

Consider the five training records shown in Table 4.5. All of these training records are labeled correctly and the corresponding decision tree is depicted in Figure 4.26. Although its training error is zero, its error rate on the test set is 30%.

Table 4.5. An example training set for classifying mammals.

Name Body Temperature

Lt lves

Birth Four- Iegged

Hibernates Class Label

salamander guppy eagle poorwill platypus

blooded cold-blooded

warm-blooded warm-blooded warm-blooded

cold- no yes n o no no

yes no no no yes

yes no no yes

Yes

no no no no yes

Humans, elephants, and dolphins are misclassified because the decision tree classifies all warm-blooded vertebrates that do not hibernate as non-mammals. The tree arrives at this classification decision because there is only one training record, which is an eagle, with such characteristics. This example clearly demonstrates the danger of making wrong predictions when there are not enough representative examples at the leaf nodes of a decision tree.

I78 Chapter 4 Classification

Figure 4.26. Decision tree induced from the data set shown in Table 4.5.

4.4.3 Overfitting and the Multiple Comparison Procedure

Model overfitting may arise in learning algorithms that employ a methodology known as multiple comparison procedure. To understand multiple comparison procedure, consider the task of predicting whether the stock market will rise or fall in the next ten trading days. If a stock analyst simply makes random guesses, the probability that her prediction is correct on any trading day is 0.5. However, the probability that she will predict correctly at least eight out of the ten times is

( ' r ' ) + ( ' r ) + ( 1 3 ) : o . o b 4 l , 2 r o

which seems quite unlikely. Suppose we are interested in choosing an investment advisor from a pool of

fifty stock analysts. Our strategy is to select the analyst who makes the most correct predictions in the next ten trading days. The flaw in this strategy is that even if all the analysts had made their predictions in a random fashion, the probability that at least one of them makes at least eight correct predictions rS

1 - ( 1 - 0 . 0 5 4 7 ) 5 0 : 0 . 9 3 9 9 ,

which is very high. Although each analyst has a low probability of predicting at least eight times correctly, putting them together, we have a high probability of finding an analyst who can do so. Furthermore, there is no guarantee in the

4 . 4 Model Overfitting LTg

future that such an analyst will continue to make accurate predictions through random guessing.

How does the multiple comparison procedure relate to model overfitting? Many learning algorithms explore a set of independent alternatives, {7a}, and then choose an alternative, 'y*ur., that maximizes a given criterion function. The algorithm will add 7*ur" to the current model in order to improve its overall performance. This procedure is repeated until no further improvement is observed. As an example, during decision tree growing, multiple tests are performed to determine which attribute can best split the training data. The attribute that leads to the best split is chosen to extend the tree as long as the observed improvement is statistically significant.

Let ?o be the initial decision tree and Trbe the new tree after inserting an internal node for attribute r. In principle, r can be added to the tree if the observed gain, A(?s, T*), is greater than some predefined threshold a. If there is only one attribute test condition to be evaluated, then we can avoid inserting spurious nodes by choosing a Iarge enough value of a. However, in practice,

more than one test condition is available and the decision tree algorithm must choose the best attribute r,,,u* from a set of candidates, {*r,*2,. ..,rp}, to partition the data. In this situation, the algorithm is actually using a multiple comparison procedure to decide whether a decision tree should be extended. More specifically, it is testing for A(?s, T*^u*) > a instead of A(?0, T") > o. As the number of alternatives, k, increases, so does our chance of finding A(fo, T*^u*) ) a. Unless the gain function A or threshold a is modified to account for k, the algorithm may inadvertently add spurious nodes to the model, which leads to model overfitting.

This effect becomes more pronounced when the number of training records from which z-ur. is chosen is small, because the variance of A(Tg, Tr^u*) is high when fewer examples are available for training. As a result, the probability of finding A(fo, Tr^u*) ) c increases when there are very few training records. This often happens when the decision tree grows deeper, which in turn reduces the number of records covered by the nodes and increases the likelihood of adding unnecessary nodes into the tree. Failure to compensate for the large number of alternatives or the small number of training records will therefore lead to model overfitting.

4.4.4 Estimation of Generalization Errors

Although the primary reason for overfitting is still a subject of debate, it is generally agreed that the complexity of a model has an impact on model overfitting, as was illustrated in Figure 4.23. The question is, how do we

180 Chapter 4 Classification

determine the right model complexity? The ideal complexity is that of a model that produces the lowest generalization error. The problem is that the learning algorithm has access only to the training set during model building (see Figure 4.3). It has no knowledge of the test set, and thus, does not know how well the tree will perform on records it has never seen before. The best it can do is to estimate the generalization error of the induced tree. This section presents several methods for doing the estimation.

Using Resubstitution Estimate

The resubstitution estimate approach assumes that the training set is a good representation of the overall data. Consequently, the training error, otherwise known as resubstitution error, can be used to provide an optimistic estimate for the generalization error. Under this assumption, a decision tree induction algorithm simply selects the model that produces the lowest training error rate as its final model. However, the training error is usually a poor estimate of generalization error.

Example 4.1. Consider the binary decision trees shown in Figure 4.27. As- sume that both trees are generated from the same training data and both make their classification decisions at each leaf node according to the majority class. Note that the left tree, Ty, is more complex because it expands some of the leaf nodes in the right tree, ?a. The training error rate for the left tree is e(?:7):4124:0.167, while the training error rate for the right tree is

Decision Tree, Ta Decision Tree, Ta

Figure 4.27. Example of two decision trees generated from the same training data.

4.4 Model Overfitting L81

e(Tp) :6f24:0.25. Based on their resubstitution estimate, the left

considered better than the right tree.

Incorporating Model Complexity

As previously noted, the chance for model overfitting increases as the model

becomes more complex. For this reason, we should prefer simpler models, a

strategy that agrees with a well-known principle known as Occam's razor or

the principle of parsimony:

Definition 4.2. Occam's Razor: Given two models with the same general-

ization errors, the simpler model is preferred over the more complex model.

Occam's razor is intuitive because the additional components in a complex model stand a greater chance of being fitted purely by chance. In the words of

Einstein, "Everything should be made as simple as possible, but not simpler." Next, we present two methods for incorporating model complexity into the

evaluation of classification models.

Pessimistic Error Estimate The first approach explicitly computes gener-

alization error as the sum of training error and a penalty term for model com- plexity. The resulting generalization error can be considered its pessimistic

error estimate. For instance, let n(t) be the number of training records classi-

fied by node t and e(t) be the number of misclassified records. The pessimistic

error estimate of a decision tree 7, "s(T),

can be computed as follows:

tree is I

e.(T\ : If:, [".(t') + cl(to)] -

Y ' D f : r n ( t t )

/ m \ 6 + 4 x 0 . 5 e s \ t R ) :

2 4

eQ) + o(r) l/,

where k is the number of leaf nodes, e(") is the overall training error of the

decision tree, N1 is the number of training records, and O(t6) is the penalty

term associated with each node l;.

Example 4.2. Consider the binary decision trees shown in Figure 4.27. If

the penalty term is equal to 0.5, then the pessimistic error estimate for the

left tree is 4 - t 7 x 0 . 5 7 . 5: 0 . 3 1 2 5e g ( T r ) :

24 and the pessimistic error estimate for the right tree is

24

: * , : 0 ' 3 3 3 3 '

182 Chapter 4 Classification

Unlabeled

The minimum description length (MDL) principle.

Thus, the left tree has a better pessimistic error rate than the right tree. For binary trees, a penalty term of 0.5 means a node should always be expanded into its two child nodes as long as it improves the classification of at least one training record because expanding a node, which is equivalent to adding 0.5 to the overall error, is less costly than committing one training error.

If fr(r) : 1 for all the nodes t, the pessimistic error estimate for the left tree is es(Tr) : 71124: 0.458, while the pessimistic error estimate for the r i g h t t r e e i s e n ( T p ) : 1 0 1 2 4 : 0 . 4 L 7 . T h e r i g h t t r e e t h e r e f o r e h a s a b e t t e r pessimistic error rate than the left tree. Thus, a node should not be expanded into its child nodes unless it reduces the misclassification error for more than one training record.

Minimum Description Length Principle Another way to incorporate model complexity is based on an information-theoretic approach known as the minimum description length or MDL principle. To illustrate this principle, consider the example shown in Figure 4.28. ln this example, both A and B are given a set of records with known attribute values x. In addition, person A knows the exact class label for each record, while person B knows none of this information. B can obtain the classification of each record by requesting that A transmits the class labels sequentially. Such a message would require @(n) bits of information, where n is the total number of records.

Alternatively, A may decide to build a classification model that summarizes the relationship between x and g. The model can be encoded in a compact

B

.)

T A

.)

T

Figure 4.28.

4.4 Model Overfitting 183

form before being transmitted to B. If the model is 100% accurate, then the cost of transmission is equivalent to the cost of encoding the model. Otherwise, A must also transmit information about which record is classified incorrectly by the model. Thus, the overall cost of transmission is

C o st(model, dat a) : C o st (model) + C o st (datalmodel), (4.e)

where the first term on the right-hand side is the cost of encoding the model, while the second term represents the cost of encoding the mislabeled records. According to the MDL principle, we should seek a model that minimizes the overall cost function. An example showing how to compute the total descrip- tion length of a decision tree is given by Exercise 9 on page 202.

Estimating Statistical Bounds

The generalization error can also be estimated as a statistical correction to the training error. Since generalization error tends to be larger than training error, the statistical correction is usually computed as an upper bound to the training error, taking into account the number of training records that reach a particular leaf node. For instance, in the C4.5 decision tree algorithm, the number of errors committed by each Ieaf node is assumed to follow a binomial distribution. To compute its generalization error, we must determine the upper bound limit to the observed training error, as illustrated in the next example.

Example 4.3. Consider the left-most branch of the binary decision trees shown in Figure 4.27. Observe that the left-most leaf node of Tp has been expanded into two child nodes in 72. Before splitting, the error rate of the node is 217 :0.286. By approximating a binomial distribution with a normal distribution, the following upper bound of the error rate e can be derived:

e u w . r ( N r e r c r ) :

_2 T: .2 , + 7 # a " , / 2 \ , 1 t 1 + = / + f f i

, ( 4 . 1 0 ), 1 -l-

-d/2

r r l y '

where c is the confidence \evel, zo12 is the standardized value from a standard normal distribution, and -Ay' is the total number of training records used to compute e. By replacing a : 25To, N : 7, and e : 217, the upper bound for the error rate is eupp.r(7,217,0.25):0.503, which corresponds to 7 x 0.503: 3.521 errors. If we expand the node into its child nodes as shown in ft,, the training error rates for the child nodes are Lf 4: 0.250 and 1/3 : 0.333,

L84 Chapter 4 Classification

respectively. Using Equation 4.10, the upper bounds of these error rates are e u p p . , ( 4 , 7 1 4 , 0 . 2 5 ) : 0 . 5 3 7 a n d e , , * " , ( 3 , 1 f 3 , 0 . 2 5 ) : 0 . 6 5 0 , r e s p e c t i v e l y . T h e overall training error of the child nodes is 4 x 0.537+3 x 0.650:4.098, which is larger than the estimated error for the corresponding node in 76. r

Using a Validation Set

In this approach, instead of using the training set to estimate the generalization error, the original training data is divided into two smaller subsets. One of the subsets is used for training, while the other, known as the validation set, is used for estimating the generalization error. Typically, two-thirds of the training set is reserved for model building, while the remaining one-third is used for error estimation.

This approach is typically used with classification techniques that can be parameterized to obtain models with different levels of complexity. The com- plexity of the best model can be estimated by adjusting the parameter of the learning algorithm (e.g., the pruning level of a decision tree) until the empir- ical model produced by the learning algorithm attains the lowest error rate on the validation set. Although this approach provides a better way for esti- mating how well the model performs on previously unseen records, less data is available for training.

4.4.5 Handling Overfitting in Decision Tree Induction

In the previous section, we described several methods for estimating the gen- eralization error of a classification model. Having a reliable estimate of gener- alization error allows the learning algorithm to search for an accurate model without overfitting the training data. This section presents two strategies for avoiding model overfitting in the context of decision tree induction.

Prepruning (Early Stopping Rule) In this approach, the tree-growing algorithm is halted before generating a fully grown tree that perfectly fits the entire training data. To do this, a more restrictive stopping condition must be used; e.g., stop expanding a leaf node when the observed gain in impurity measure (or improvement in the estimated generalization error) falls below a certain threshold. The advantage of this approach is that it avoids generating overly complex subtrees that overfit the training data. Nevertheless, it is difficult to choose the right threshold for early termination. Too high of a threshold will result in underfitted models, while a threshold that is set too low may not be sufficient to overcome the model overfitting problem. Furthermore,

Model Overfitting L85

Figure 4,29. Post-pruning of the decision tree for Web robot detection,

even if no significant gain is obtained using one of the existing attribute test conditions, subsequent splitting may result in better subtrees.

Post-pruning In this approach, the decision tree is initially grown to its

maximum size. This is followed by a tree-pruning step, which proceeds to trim the fully grown tree in a bottom-up fashion. Ttimming can be done by replacing a subtree with (1) a new leaf node whose class label is determined from the majority class of records affiliated with the subtree, or (2) the most frequently used branch of the subtree. The tree-pruning step terminates when no further improvement is observed. Post-pruning tends to give better results than prepruning because it makes pruning decisions based on a fully grown

tree, unlike prepruning, which can suffer from premature termination of the tree-growing process. However, for post-pruning, the additional computations needed to grow the full tree may be wasted when the subtree is pruned.

Figure 4.29 illustrates the simplified decision tree model for the Web robot

detection example given in Section 4.3.6. Notice that the subtrees rooted at

4 . 4

lmagePages> 0.375: class 0 lmagePages<= 0.375:

totalPages> 6: I breadth <= 1: class 1 I breadth > 1: class 0

1\Lul!ilP-=-o---- t ilmagePages:f o.i$3: Gtas-sT'i I i lmagePages> 0.1 333: I i breadth <= 6: class 0 i I Lbjqaltl

"]j-

jbgs_l_ _ _ _ _ _ _ _l M u l t i l P = 1 : I TotalTime <= 361: class 0 I TotalTime > 361: class 1

NLul!i4s_e!t_=_ot _ I ideoth"iz-:

- c-la-ss-il

- - - - - - - - - - - - | ^ l

| | deplh <= 2: I i I MultilP = 1: class 0 l r l M u l t i l P = 0 : I I | | breadth <= 6: class 0 l r l I b r e a d t h > 6 : lll | | RepeatedAccess<=0.322: classo i liLIlEgp_eet9dAc9es1>_q jq2_2i_c!_a9s_1__l MultiAgent = 1: I t o t a l P a g e s < = 8 1 : c l a s s 0 I totalPages > 81: class I

Simplified Decision Tree:

d e p t h = 1 : t iimageP-a-ges-<---o leds:

-c-tass T--l

I ilmagePages > 0.1333: I I breadth <= 6: class 0

I i I breadth > 6: class I

| | t o t a l P a g e s < = 8 1 : c l a s s 0 | | t o t a l P a g e s > 8 1 : c l a s s l

d e D t h > 1 :

uMuiriAsaT = oa EE;to ------'l

T tr,tuniaGn-t-=l:

186 Chapter 4 Classification

depth : t have been replaced by one of the branches involving the attribute InagePages. This approach is also known as subtree raising. The depth ) 1 and MultiAgent : 0 subtree has been replaced by a leaf node assigned to class 0. This approach is known as subtree replacement. The subtree for depth ) 1 and MultiAgent : 1 remains intact.

4.5 Evaluating the Performance of a Classifier

Section 4.4.4 described several methods for estimating the generalization error of a model during training. The estimated error helps the learning algorithm to do model selection; i.e., to find a model of the right complexity that is not susceptible to overfitting. Once the model has been constructed, it can be applied to the test set to predict the class labels of previously unseen records.

It is often useful to measure the performance of the model on the test set because such a measure provides an unbiased estimate of its generalization error. The accuracy or error rate computed from the test set can also be used to compare the relative performance of different classifiers on the same domain. However, in order to do this, the class Iabels of the test records must be known. This section reviews some of the methods commonly used to evaluate the performance of a classifier.

4 . 5 . L H o l d o u t M e t h o d

In the holdout method, the original data with labeled examples is partitioned into two disjoint sets, called the training and the test sets, respectively. A classification model is then induced from the training set and its performance is evaluated on the test set. The proportion of data reserved for training and for testing is typically at the discretion of the analysts (e.g., 50-50 or two- thirds for training and one-third for testing). The accuracy of the classifier can be estimated based on the accuracy of the induced model on the test set.

The holdout method has several well-known limitations. First, fewer la- beled examples are available for training because some of the records are with- held for testing. As a result, the induced model may not be as good as when all the labeled examples are used for training. Second, the model may be highly dependent on the composition of the training and test sets. The smaller the training set size, the larger the variance of the model. On the other hand, if the training set is too large, then the estimated accuracy computed from the smaller test set is Iess reliable. Such an estimate is said to have a wide con- fidence interval. Finally, the training and test sets are no longer independent

4.5 Evaluating the Performance of a Classifier 187

of each other. Because the training and test sets are subsets of the original data, a class that is overrepresented in one subset will be underrepresented in the other, and vice versa.

4.5.2 Random Subsampling

The holdout method can be repeated several times to improve the estimation of a classifier's performance. This approach is known as random subsampling. Let acc.i be the model accuracy during the i,th iteration. The overall accuracy is given by acq,,6 : Ditaccif k. Random subsampling still encounters some of the problems associated with the holdout method because it does not utilize

as much data as possible for training. It also has no control over the number of times each record is used for testing and training. Consequently, some records might be used for training more often than others.

4.5.3 Cross-Validation

An alternative to random subsampling is cross-validation. In this approach,

each record is used the same number of times for training and exactly once for testing. To illustrate this method, suppose we partition the data into two equal-sized subsets. First, we choose one of the subsets for training and the other for testing. We then swap the roles of the subsets so that the previous

training set becomes the test set and vice versa. This approach is called a two- fold cross-validation. The total error is obtained by summing up the errors for both runs. In this example, each record is used exactly once for training and once for testing. The k-fold cross-validation method generalizes this approach

by segmenting the data into k equal-sized partitions. During each run, one of the partitions is chosen for testing, while the rest of them are used for training.

This procedure is repeated k times so that each partition is used for testing exactly once. Again, the total error is found by summing up the errors for all k runs. A special case of the k-fold cross-validation method sets k : N,

the size of the data set. In this so-called leave-one-out approach, each test set contains only one record. This approach has the advantage of utilizing

as much data as possible for training. In addition, the test sets are mutually exclusive and they effectively cover the entire data set. The drawback of this approach is that it is computationally expensive to repeat the procedure ly'

times. Furthermore, since each test set contains only one record, the variance

of the estimated performance metric tends to be high.

188 Chapter 4 Classification

4.5.4 Bootstrap

The methods presented so far assume that the training records are sampled without replacement. As a result, there are no duplicate records in the training and test sets. In the bootstrap approach, the training records are sampled with replacement; i.e., a record already chosen for training is put back into the original pool of records so that it is equally likely to be redrawn. If the original data has .ly' records, it can be shown that, on average, a bootstrap sample of size ly' contains about 63.2% of the records in the original data. This approximation follows from the fact that the probability a record is chosen by a bootstrap sample is 1 - (1 - 1/,^f)^i. When l,r is sufficiently large, the probability asymptotically approaches 1 - e-r :0.632. Records that are not included in the bootstrap sample become part of the test set. The model induced from the training set is then applied to the test set to obtain an estimate of the accuracy of the bootstrap sample, e.;. The sampling procedure is then repeated b times to generate b bootstrap samples.

There are several variations to the bootstrap sampling approach in terms of how the overall accuracy of the classifier is computed. One of the more widely used approaches is the .632 bootstrap, which computes the overall accuracy by combining the accuracies of each bootstrap sample (e;) with the accuracy computed from a training set that contains all the labeled examples in the original data (acc"):

1 - a Accuracy, (rcc6661 :

i ),{0.U32 x e6+ 0.368 x acc"). ( 4 . 1 1 )

4.6 Methods for Comparing Classifiers

It is often useful to compare the performance of different classifiers to deter- mine which classifier works better on a given data set. However, depending on the size of the data, the observed difference in accuracy between two clas- sifiers may not be statistically significant. This section examines some of the statistical tests available to compare the performance of different models and classifiers.

For illustrative purposes, consider a pair of classification models, M4 and M6. Suppose Mn achieves 85To accuracy when evaluated on a test set con- taining 30 records, while Il[B achieves 75To accvacy on a different test set containing 5000 records. Based on this information, is M4 a better model than MB?

4.6 Methods for Comparing Classifiers 189

The preceding example raises two key questions regarding the statistical significance of the performance metrics:

1. Although Me has a higher accuracy than Ms, it was tested on a smaller test set. How much confidence can we place on the accuracy for Ma?

2. Is it possible to explain the difference in accuracy as a result of variations in the composition of the test sets?

The first question relates to the issue of estimating the confidence interval of a given model accuracy. The second question relates to the issue of testing the statistical significance of the observed deviation. These issues are investigated in the remainder of this section.

4.6.1 Estimating a Confidence Interval for Accuracy

To determine the confidence interval, we need to establish the probability

distribution that governs the accuracy measure. This section describes an ap- proach for deriving the confidence interval by modeling the classification task as a binomial experiment. Following is a list of characteristics of a binomial experiment:

1. The experiment consists of l/ independent trials, where each trial has

two possible outcomes: success or failure.

2. The probability of success, p, in each trial is constant.

An example of a binomial experiment is counting the number of heads that turn up when a coin is flipped N times. If X is the number of successes observed in l/ trials, then the probability that X takes a particular value is given by a binomial distribution with mean l/p and variance l/p(l - p):

P ( X : u ) : - p ) N - "

For example, if the coin is fair (p : 0.5) and is flipped fifty times, then the probability that the head shows up 20 times is

P ( X : 2 0 ) :

If the experiment is repeated many times, then the average number of heads

expected to show up is 50 x 0.5 : 25, while its variance is 50 x 0.5 x 0.5 : I2.5.

On,'

( 1 3 ; 0 . t " , 1 - 0 . 5 ) 3 0 : 0 . 0 4 1 e .

190 Chapter 4 Classification

The task of predicting the class labels of test records can also be consid- ered as a binomial experiment. Given a test set that contains l{ records, let X be the number of records correctly predicted by a model and p be the true accuracy of the model. By modeling the prediction task as a binomial experi- ment, X has a binomial distribution with mean I/p and variance Np(L - p). It can be shown that the empirical accuracy? acc : X lN , also has a binomial distribution with mean p and variance p(t-p)lN (see Exercise 12). Although the binomial distribution can be used to estimate the confidence interval for acc, it is often approximated by a normal distribution when N is sufficiently large. Based on the normal distribution, the following confidence interval for acc car:' be derived:

a c c - p (4.12)

where Zo12 and Zt-o/z are the upper and lower bounds obtained from a stan- dard normal distribution at confidence level (1 - a). Since a standard normal distribution is symmetric around Z:0, it follows that Zop: Zt-o/z.Rear- ranging this inequality leads to the following confidence interval for p:

< z r _ . p ) : 7 _ ( r ,

2 x N x a c c * Z ' , t r + Z * / r @ ( 4 . 1 3 )

2 ( N + 2 2 ^ , r ) " t -

The following table shows the values of Zo12 at different confidence levels:

l , - a 0 . 9 9 0 . 9 8 0 . 9 5 0 . 9 0 . 8 u . l 0 . 5 Z o / 2 2 . 5 8 2 . 3 3 1 . 9 6 1 . 6 5 I . 2 8 r . 0 4 0 . 6 7

Example 4.4. Consider a model that has an accuracy of 80% when evaluated on 100 test records. What is the confidence interval for its true accuracy at a 95% confidence level? The confidence level of 95% correspondsto Zo12:1.96 according to the table given above. Inserting this term into Equation 4.13 yields a confidence interval between 71.1% and 86.7%. The following table shows the confidence interval when the number of records, l/, increases:

t ( - z o t z I

N 2 0 5 0 1 0 0 500 1000 5000 Confidence

Interval 0 . 5 8 4

- 0 . 9 1 9 0 . 6 7 0

- 0 . 8 8 8 0.71.t

- 0 . 8 6 7 0 . 7 6 3

- 0 . 8 3 3 0.774

- 0.824 0 . 7 8 9

- 0 . 8 1 1

Note that the confidence interval becomes tiehter when N increases.

Methods for Comparing Classifiers 1-91

4.6.2 Comparing the Performance of Two Models

Consider a pair of models, M1 and M2, that are evaluated on two independent

test sets, D1 and D2. Let n1 denote the number of records in D1 and n2 denote the number of records in D2. In addition, suppose the error rate for M1 oL Dl is el and the error rate for Mz on D2 is e2. Our goal is to test whether the observed difference between e1 and e2 is statisticaliy significant.

Assuming that n1 and n2 are sufficiently large, the error rates e1 and e2 can be approximated using normal distributions. If the observed difference in

the error rate is denoted as d : er - €2t then d is also normally distributed

with mean d1,its true difference, and variance, o]. Tne variance of d can be

computed as follows:

e 1 ( L - e 1 ) , e 2 ( l - e 2 )- 7 - 1 nL n2

(4.r4)

where "t(I

- et)lu and e2(1 - ez)lnz are the variances of the error rates. Finally, at the (t - a)% confidence level, it can be shown that the confidence interval for the true difference dl is given by the following equation:

d t : d L z o 1 2 G 6 . ( 4 . 1 5 )

Example 4.5. Consider the problem described at the beginning of this sec-

tion. Model Ma has an error rate of er : 0.15 when applied to A1 : 39

test records, while model Ms has an error rate of ez : 0.25 when applied

to Ab : 5000 test records. The observed difference in their error rates is

d : 10.15 - 0.251 : 0.1. In this example, we are performing a two-sided test

to check whether dt : 0 or d,6 I 0. The estimated variance of the observed

difference in error rates can be computed as follows:

4 . 6

n 2 . - i ? - " d

- " d

^, w d -

0 . 1 5 ( 1 - 0 . 1 5 ).qag#a:ooo43 30

or 64 :0.0655. Inserting this value into Equation 4.L5, we obtain the following

confidence interval for d+ at 95% confidence level:

d + : 0 . 1 + 1 . 9 6 x 0 . 0 6 5 5 : 0 . 1 * 0 . 1 2 8 .

As the interval spans the value zero) we can conclude that the observed differ-

ence is not statistically significant at a g5% confidence level. I

L92 Chapter 4 Classification

At what confi.dence level can we reject the hypothesis that dt:0? To do this, we need to determine the value of Zop such that the confidence interval for d1 does not span the value zero. We can reverse the preceding computation and look for the valte Zo12 such that d > Zo126a. Replacing the values of d and 66 gives Zo12 < 7.527. This value first occurs when (1 - a) S 0.936 (for a two-sided test). The result suggests that the null hypothesis can be rejected at confidence level of 93.6% or lower.

4.6.3 Comparing the Performance of Two Classifiers

Suppose we want to compare the performance of two classifiers using the /c-fold cross-validation approach. Initially, the data set D is divided into /c equal-sized partitions. We then apply each classifier to construct a model from k - 1 of the partitions and test it on the remaining partition. This step is repeated /c times, each time using a different partition as the test set.

Let Mii denote the model induced by classification technique tr; during the jth iteration. Note that each pair of models M1i and, M2i are tested on the same partition j. Let e1i and e2i be their respective error rates. The difference between their error rates durin g the jth fold can be written as di - eU - e2j. If k is sufficiently large, then d7 is normally distributed with mean dfu, which is the true difference in their error rates, and variance a"'. Unlike the previous approach, the overall variance in the observed differences is estimated using the following formula:

( 4 . 1 6 )

where d is the average difference. For this approach, we need to use a f- distribution to compute the confidence interval for df":

d t " : d I t g _ . ' 1 , t _ r 6 a . " .

The coefficient t1r-a;,r-1 is obtained from a probability table with two input parameters, its confidence level (1 -

") and the number of degrees of freedom,

k - I. The probability table for the t-distribution is shown in Table 4.6.

Example 4.6. Suppose the estimated difference in the accuracy of models generated by two classification techniques has a mean equal to 0.05 and a standard deviation equal to 0.002. If the accuracy is estimated using a 30-fold cross-validation approach, then at a gbTo confidence level, the true accuracy difference is

d f " : 0 . 0 5 + 2 . 0 4 x 0 . 0 0 2 . ( 4 . 1 7 )

4.7 Bibliographic Notes 193

Table 4.6. Probability table for t-distribution.

k - 7 ( 1 - o

0 . 9 9 0 . 9 8 0 . 9 5 0 . 9 0 . 8 1 l-

2 A

9 I 4 1 9 24 2 9

3 . 0 8 1 . 8 9 1 . 5 3 1 . 3 8 7 . 3 4 1 . 3 3 r . 3 2 1 . 3 1

6 . 3 1 2 . 9 2 2 . L 3 1 . 8 3 r . 7 6 1 . 7 3 7 . 7 r 7 . 7 0

72.7 4 . 3 0 2 . 7 8 2 . 2 6 2 . L 4 2 . 0 9 2 . 0 6 2 . 0 4

3 1 . 8 6 . 9 6 3 . 7 5 2 . 8 2 2 . 6 2 2 . 5 4 2 . 4 9 2 . 4 6

63.7 9 . 9 2 4 . 6 0 3 . 2 5 2 . 9 8 2 . 8 6 2 . 8 0 2 . 7 6

Since the confidence interval does not span the value zero, the observed dif- ference between the techniques is statistically significant. r

4.7 Bibliographic Notes

Early classification systems were developed to organize a large collection of objects. For example, the Dewey Decimal and Library of Congress classifica- tion systems were designed to catalog and index the vast number of library books. The categories are typically identified in a manual fashion, with the help of domain experts.

Automated classification has been a subject of intensive research for many yearc. The study of classification in classical statistics is sometimes known as discriminant analysis, where the objective is to predict the group member- ship of an object based on a set of predictor variables. A well-known classical method is Fisher's linear discriminant analysis [117], which seeks to find a lin- ear projection of the data that produces the greatest discrimination between objects that belong to different classes.

Many pattern recognition problems also require the discrimination of ob- jects from different classes. Examples include speech recognition, handwritten character identification, and image classification. Readers who are interested in the application of classification techniques for pattern recognition can refer to the survey articles by Jain et al. 1722] and Kulkarni et al. [128] or classic pattern recognition books by Bishop [107], Duda et al. [114], and Fukunaga

[118]. The subject of classification is also a major research topic in the fields of neural networks, statistical learning, and machine learning. An in-depth treat-

L94 Chapter 4 Classification

ment of various classification techniques is given in the books by Cherkassky and Mulier [112], Hastie et al. [120], Michie et al. [133], and Mitchell [136].

An overview of decision tree induction algorithms can be found in the survey articles by Buntine [110], Moret [137], Murthy [tSS], and Safavian et aL l7a7l. Examples of some well-known decision tree algorithms include CART

[108], ID3 [143], C4.5 [ta5], and CHAID [125]. Both ID3 and C4.5 employ the entropy measure as their splitting function. An in-depth discussion of the C4.5 decision tree algorithm is given by Quinlan [145]. Besides explaining the methodology for decision tree growing and tree pruning, Quinlan [145] also described how the algorithm can be modified to handle data sets with missing values. The CART algorithm was developed by Breiman et al. [108] and uses the Gini index as its splitting function. CHAID [tZ5] uses the statistical y2 test to determine the best split during the tree-growing process.

The decision tree algorithm presented in this chapter assumes that the splitting condition is specified one attribute at a time. An oblique decision tree can use multiple attributes to form the attribute test condition in the internal nodes [121, 152]. Breiman et al. [108] provide an option for using linear combinations of attributes in their CARf implementation. Other approaches for inducing oblique decision trees were proposed by Heath et al. [121], Murthy et al. [139], Cantil-Paz and Kamath 1111], and Utgoff and Brodley 11.52]1. Although oblique decision trees help to improve the expressiveness of a decision tree representation, learning the appropriate test condition at each node is computationally challenging. Another way to improve the expressiveness of a decision tree without using oblique decision trees is to apply a method known as constructive induction [132]. This method simplifies the task of learning complex splitting functions by creating compound features from the original attributes.

Besides the top-down approach, other strategies for growing a decision tree include the bottom-up approach by Landeweerd et al. 1130] and Pattipati and Alexandridis [142], as well as the bidirectional approach by Kim and Landgrebe

[126]. Schuermann and Doster [150] and Wang and Suen [154] proposed using a soft splitting criterion to address the data fragmentation problem. In this approach, each record is assigned to different branches of the decision tree with different probabilities.

Model overfitting is an important issue that must be addressed to ensure that a decision tree classifier performs equally well on previously unknown records. The model overfitting problem has been investigated by many authors including Breiman et al. [108], Schaffer [148], Mingers [135], and Jensen and Cohen [123]. While the presence of noise is often regarded as one of the

Bibliography 195

primary reasons for overfitting [t35, 140], Jensen and Cohen [123] argued that overfitting is the result of using incorrect hypothesis tests in a multiple comparison procedure.

Schapire 1149] defined generalization error as "the probability of misclas- sifying a new example" and test error as "the fraction of mistakes on a newly sampled test set." Generalization error can therefore be considered as the ex- pected test error of a classifier. Generalization error may sometimes refer to the true error 1136] of a model, i.e., its expected error for randomly drawn data points from the same population distribution where the training set is sampled. These definitions are in fact equivalent if both the training and test sets are gathered from the same population distribution, which is often the case in many data mining and machine learning applications.

The Occam's razor principle is often attributed to the philosopher William of Occam. Domingos [113] cautioned against the pitfall of misinterpreting Occam's razor as comparing models with similar training errors, instead of generalization errors. A survey on decision tree-pruning methods to avoid overfitting is given by Breslow and Aha [109] and Esposito et al. [116]. Some of the typical pruning methods include reduced error pruning [144], pessimistic error pruninglL44], minimum error pruning [141], critical value pruning 1134], cost-complexity pruning [108], and error-based pruning [1a5]. Quinlan and Rivest proposed using the minimum description length principle for decision tree pruning in [146].

Kohavi [127] had performed an extensive empirical study to compare the performance metrics obtained using different estimation methods such as ran- dom subsampling, bootstrapping, and k-fold cross-validation. Their results suggest that the best estimation method is based on the ten-fold stratified cross-validation. Efron and Tibshirani [115] provided a theoretical and empir- ical comparison between cross-validation and a bootstrap method known as the 632* rule.

Current techniques such as C4.5 require that the entire training data set fit into main memory. There has been considerable effort to develop parallel and scalable versions of decision tree induction algorithms. Some of the proposed algorithms include SLIQ by Mehta et al. [131], SPRINT by Shafer et al. [151], CMP by Wang and Zaniolo [153], CLOUDS by Alsabti et al. [106], RainForest by Gehrke et al. [119], and ScalParC by Joshi et al. j2al. A general survey of parallel algorithms for data mining is available in 1129].

11071

l1o8l

l1Oel

196 Chapter 4 Classification

Bibliography [106] K. Alsabti, S. Ranka, and V. Singh. CLOUDS: A Decision TYee Classifier for Large

Datasets. In Proc. of the lth Intl. Conf. on Knowledge D'iscouery and Data M'ining, pages 2-8, New York. NY, August 1998.

C. M. Bishop. Neural Networks for Pattern Recogniti,on. Oxford University Press,

O x f o r d , U . K . , 1 9 9 5 .

L. Breiman, J. H. Friedman, R. Olshen, and C. J. Stone. Classi,fi'cati'on and Regression Trees. Chaprnan & Hall, New York, 1984.

L. A. Breslow and D. W. Aha. Simplifying Decision Tlees: A Survey. Knowledge

Engineering Reui,ew, L2(l):L-40, 1997.

[110] W. Buntine. Learning classification trees. In Artifici,al Intelligence Frontiers i;n Statis-

lics, pages I82 20I. Chapman & Hall, London, 1993.

1111] E. Cantri-Paz and C. Kamath. Using evolutionary algorithms to induce oblique decision

trees. In Proc. of the Genetic and Euoluti,onarg Computation Conf., pages 1053-1060,

San Flancisco, CA, 2000.

[112] V. Cherkassky and F. Mulier. Learning from Data: Concepts, Theory, and Method's.

Wiley Interscience, 1998.

[113] P. Domingos. The Role of Occam's Razor in Knowledge Discovery. Data Mi'ning and

Knouledg e Discouery, 3(4) :409-425, t999.

[114] R. O. Duda, P. E. Hart, and D. G. Stork. Pattern Class'ification John Wiley & Sons, Inc., New York, 2nd edition, 2001.

[115] B. Efron and R. Tibshirani. Cross-validation and the Bootstrap: Estimating the Error

Rate of a Prediction Rule. Technical report, Stanford University, 1995.

[116] F. Esposito, D. Malerba, and G. Semeraro. A Comparative Analysis of Methods for

Pruning Decision Trees. IEEE Trans. Pattern Analysis and Machine Intelli,gence, 19 (5):476-491, May 1997. R. A. Fisher. The use of multiple measurements in taxonomic problems. Annals of

Eugenics, 7:179 188, 1936. K. Fukunaga. Introduct'ion to Statist'ical Pattern Recognit'i,on. Academic Press, New

York, 1990.

1119] J. Gehrke, R. Ramakrishnan, and V. Ganti. RainForest-A Framework for Fast De-

cision Tree Construction of Large Datasets. Data Mi,ning and Knowledge D'iscouerg, 4

( 2 1 3 ) : 1 2 7 - 1 6 2 , 2 0 0 0 .

[120] T. Hastie, R. Tibshirani, and J. H. Friedman. The Elements of Stati.stical Leanting:

Data Mi,ning, Inference, Pred'icti,on. Springer, New York, 2001.

f121] D. Heath, S. Kasif, and S. Salzberg. Induction of Oblique Decision Trees. In Proc. of

the 13th IntI. Joint Conf. on Arti,fici,al Intelligence, pages 1002-1007, Chambery, France,

A u g u s t 1 9 9 3 .

ll22] A. K. Jain, R. P. W. Duin, and J. Mao. Statistical Pattern Recognition: A Review.

I E E E T f a n . P a t t . A n a l . a n d , M a c h . I n t e l l i g . , 2 2 ( I ) : 4 3 7 , 2 0 0 0 .

f123] D. Jensen and P. R. Cohen. Multiple Comparisons in Induction Algorithms. Mach'ine

Learning, 38(3) :309-338, March 2000.

ll24l M. V. Joshi, G. Karypis, and V. Kumar. ScalParC: A New Scalable and Efficient Parallel Classification Algorithm for Mining Large Datasets. In Proc. of 12th IntI.

Parallel Processing Sgmp. (IPPS/SPDP), pages 573 579, Orlando, FL, April 1998.

[125] G. V. Kass. An Exploratory Technique for Investigating Large Quantities of Categor- ical Data. Appli,ed Statistics, 29:Ll9-127, 1980.

[1 17]

11181

[126]

lr27l

u28l

[12e]

Bibtiography Lg7

[130j G. Landeweerd, T. Timmers, E. Gersema, M. Bins, and M. Halic. Binary tree versussingle level tree classification of white blood cells. pattern Recognzt,ion, L6:bTr-bTZ, 1983.

[131]-M' Mehta, R. Agrawal, and J. Rissanen. sLIQ: A Fast scalable classifier for DataMining. ln Proc. of -the

sth Intt. conf. on Extend,,ing Database Technorogy,pages 1g-32,Avignon, Flance, March 19g6. [132] R' S' Michalski' A theory and methodology of inductive learning. Arti,fi,cial Intell,igence,

2 0 : 1 1 1 - 1 1 6 , 1 9 8 8 .

[133j D. Michie, D. J. Spiegelhalter, and C. C. Taylor. Machine Learning, .

Statistical Classi,fi,cati,oz. Ellis Horwood, Upper Saddle River, NJ, 1g94. [134] -J'

Mingers. Expert systems-Rule Induction with statisticar Data. J Research Soc'ietg, Bg:89 47, 19g2.

[135] J' Mingers' An empirical comparison of pruning methods for decision tree induction.Machine Learni,ng, 4:227_24J, Iggg. [136] T. Mitchell. Machine Learn,ing. McGraw_Hill, Boston, MA, IggT. 1137] B. M. E' Moret. Decision T}ees and Diagrams. computing suraegs, r4(4):893-62J,

t982.

I138]-S' K' Murthy. Automatic Construction of Decision TYees from Data: A Multi-Disciplinary survey. Data Mi,ning and Knowredge Discouery,2(4):34b-3gg, 1ggg. [139] S' K' Murthy, s. r.<3if,

_and S. salzberg. A system for induction of oblique decisiontrees. ,.I of Artificial Intelligence Research,2:I,JJ, Igg4. [140] T' Niblett. constructing decision trees in noisy domain s. In proc. of the 2nd. EuropeanWorking Session on Leaming, pages 67_2g, Bled, yugoslavia, May 19g7. [141] T' Niblett and I. Bratko' Learning Decision Rules in Noisy Domai ns. rn Research and,Deuelopment in Er,pert systems r11, cambridge, 1gg6. cu-bridg" University press. [142] K' R' Pattipati and M. G. Alexandri<Iis. Application of heuristic search and information

theory to sequentiar fault diagnosis. IEEE Trans. on sgstems, Mon, ord, cgbernetics,2o(4):872-s82, rss}. [143] J' R' Quinlan. Discovering rules by induction from large collection of examples. InD' Michie, editot, Eupert systems ,in the M,icro Erectronic Ag". Edinburgh universityPress, Edinburgh, UK, 1g79. Ir44) J- R' Quinlan. simplifying Decision Ttees. 1nrl. J. Man-Machine stud,ies,2T:22r_284,

1987.

[145] J. R. Quinlan. cl.s: progr.ams for Machine Learni,ng. Morgan-Kaufmann publishers, San Mateo, CA, 1gg3.

[146] J' R' Quinlan and R. L. Rivest. Inferring Decision Tlees using the Minimum Descrip-tion Length Principle. Information and Computation, g0(B):2i7_24g, 19gg.

Neural and

Operational

1-98 Chapter 4 Classification

[147] S. R. Safavian and D. Landgrebe. A Survey of Decision Ttee Classifier Methodology.

IEEE Trans. Sgstems, Man and Cyber"netics, 22:660-674, May/June 1998.

[148] C. Schaffer. Overfitting avoidence as bias. Machine Learn'ing, 10:153-178, 1993.

[149] R. E. Schapire. The Boosting Approach to Machine Learning: An Overview. In MSRI

Workshop on Nonlinear Estimati'on and Classificati'on, 2002.

[150] J. Schuermann and W. Doster. A decision-theoretic approach in hierarchical classifier

design. P attern Recogn'ition, 17:359-369, 1984.

[151] J. C. Shafer, R. Agrawal, and M. Mehta. SPRINT: A Scalable Parallel Classifier

for Data Mining. In Proc. of the 22nd VLDB Conf., pages 544-555, Bombay, India,

September 1996.

1152] P. E. Utgoff and C. E. Brodley. An incremental method for finding multivariate splits

for decision trees. In Proc. of the 7th IntI. Conf. on Mach'ine Learning, pages 58-65,

Austin. TX. June 1990.

1153] H. Wang and C. Zaniolo. CMP: A Fast Decision Tlee Classifier Using Multivariate

Predictions. In Proc. of the 16th IntI. Conf. on Data Engineering, pages 449-460, San

Diego, CA, March 2000.

1154] Q. R. Wang and C. Y. Suen. Large tree classifier with heuristic search and global

training. IEEE Trans. on Pattern Analys'is and Machine Intell'igence,9(1):91-102, 1987.

4.8 Exercises

Draw the full decision tree for the parity function of four Boolean attributes, A, B, C, and D. Is it possible to simplify the tree?

Consider the training examples shown in Table 4.7 for a binary classification problem.

(a) Compute the Gini index for the overall collection of training examples.

(b) Compute the Gini index for the Custoner ID attribute.

(c) Compute the Gini index for the Gender attribute.

(d) Compute the Gini index for the Car Type attribute using multiway split.

(e) Compute the Gini index for the Shirt Size attribute using multiway split.

(f) Which attribute is better, Gender, Car Type, or Shirt Size?

(g) Explain why Custoner ID should not be used as the attribute test con- dition even though it has the lowest Gini.

Consider the training examples shown in Table 4.8 for a binary classification problem.

(a) What is the entropy of this collection of training examples with respect to the positive class?

1 .

2 .

. f .

Table 4.7. Data set for Exercise 2. Customer ID Gender Car Type Shirt Size Class

1 2 3 4 b

6 F7 I

8 9 1 0 1 1 t 2 1 3 t 4 1 5 1 6 7 7 1 8 1 9 20

M

M M M M M F F F F M M M M F F F F F F

Family Sports Sports Sports Sports Sports Sports Sports Sports Luxury Family Family Family Luxury Luxury Luxury Luxury Luxury Luxury Luxury

Small Medium Medium

Large Extra Large Extra Large

Small Small

Medium Large Large

Extra Large Medium

Extra Large Small Small

Medium Medium Medium

Larqe

CO CO CO CO CO CO CO CO CO CO C 1 C 1 C 1 C 1 C 1 C 1 C 1 C 1 C 1 C 1

4.8 Exercises 199

Table 4,8. Data set for Exercise 3. Instance aL a2 as Target Class

I 2 3 4 5 6 a T

8 o

T T 1 . O T T 6 . 0 T F 5 . O F F 4 . O F T 7 . 0 F T 3 . O F F 8 . O T F 7 . 0 F T 5 . O

T

-T

-T-

-T-

(b)

(")

What are the information gains of o1 and o2 relative to these training examples?

For o3, which is a continuous attribute, compute the information gain for every possible split.

200 Chapter 4 Classification

(d) What is the best split (among e7t a2t and o3) according to the information gain?

(e) What is the best split (between 01 and o2) according to the classification error rate?

(f) What is the best split (between 01 and o2) according to the Gini index?

4. Show that the entropy of a node never increases after splitting it into smaller successor nodes.

5. Consider the following data set for a binary class problem.

(a) Calculate the information gain when splitting on ,4 and B. Which at- tribute would the decision tree induction algorithm choose?

(b) Calculate the gain in the Gini index when splitting on .A and B. Which attribute would the decision tree induction algorithm choose?

(c) Figure 4.13 shows that entropy and the Gini index are both monotonously increasing on the range [0, 0.5] and they are both monotonously decreasing on the range [0.5, 1]. Is it possible that information gain and the gain in the Gini index favor different attributes? Explain.

Consider the following set of training examples.

X Y Z No. of Class C1 Examples No. of Class C2 Examples 0 0 0 5 40 0 0 1 0 1 5 0 1 0 1 0 5 0 I I 45 0 1 0 0 1 0 I 0 I 2 5 0 I 1I 0 5 20 1 1 1 0 1 5

o .

A B Class Label I

T T T T F F F T T

! '

T T F T F F F T F

-r -r

+

+

7

4.8 Exercises zOL

(a) Compute a two-level decision tree using the greedy approach described in this chapter. Use the classification error rate as the criterion for splitting. What is the overall error rate of the induced tree?

(b) Repeat part (a) using X as the first splitting attribute and then choose the best remaining attribute for splitting at each of the two successor nodes. What is the error rate of the induced tree?

(c) Compare the results of parts (a) and (b). Comment on the suitability of the greedy heuristic used for splitting attribute selection.

The following table summarizes a data set with three attributes A, B. C and two class labels *, -. Build a two-level decision tree.

A B C Number of Instances +

T F T F T F T F

T T F F T T F F

T T T T F F F F

(

0 2 0 0 0

2 5 0 0

0 2 0 0 K

0 0 0

2 5

(a) According to the classification error rate, which attribute would be chosen as the first splitting attribute? For each attribute, show the contingency table and the gains in classification error rate.

(b) Repeat for the two children of the root node.

(c) How many instances are misclassified by the resulting decision tree?

(d) Repeat parts (a), (b), and (c) using C as the splitting attribute.

(e) Use the results in parts (c) and (d) to conclude about the greedy nature of the decision tree induction algorithm.

8. Consider the decision tree shown in Figure 4.30.

(a) Compute the generalization error rate of the tree using the optimistic approach.

(b) Compute the generalization error rate of the tree using the pessimistic approach. (For simplicity, use the strategy of adding a factor of 0.5 to each leaf node.)

(c) Compute the generalization error rate of the tree using the validation set shown above. This approach is known as reduced error pruning.

2O2 Chapter 4 Classification

Figure 4.30. Decision tree and data sets for Exercise 8.

Consider the decision trees shown in Figure 4.31. Assume they are generated

from a data set that contains 16 binarv attributes and 3 classes, C1, C2, and C3.

(a) Decision tree with 7 errors (b) Decision tree with 4 errors

Figure 4.31. Decision trees for Exercise 9.

9 .

Instance A B c Class +

2 0 0 + 3 1 + 4 1 5 1 0 0 + 6 1 0 0 + 7 1 1 0 8 1 0 + I 1 1 0 1 0 1 1 0

Validation: lnstance A B c Class

0 0 0 + 1 2 0 1 1 + 1 3 1 1 0 + 1 4 1 0 1 1 5 1 0 0 +

4.8 Exercises 2Og

Compute the total description length of each minimum description length principle.

tree according to the

o The total description length of a tree is given by:

C o st (tr e e, dat a) : C o st (tr ee) -f C o st (dat altr ee) .

Each internal node of the tree is encoded by the ID of the splitting at- tribute. If there are rn attributes, the cost of encodins each attribute is Iog, rn bits.

Each leaf is encoded using the ID of the class it is associated with. If there are k classes, the cost of encoding a class is log, k bits.

Cost(tree) is the cost of encoding all the nodes in the tree. To simplify the computation, you can assume that the total cost of the tree is obtained by adding up the costs of encoding each internal node and each leaf node.

Cost(dataltree) is encoded using the classification errors the tree commits on the training set. Each error is encoded by log2 n bits, where n is the total number of training instances.

Which decision tree is better, according to the MDL principle?

10. While the .632 bootstrap approach is useful for obtaining a reliable estimate of model accuracy, it has a known limitation 1127]. Consider a two-class problem, where there are equal number of positive and negative examples in the data. Suppose the class labels for the examples are generated randomly. The classifier used is an unpruned decision tree (i.e., a perfect memorizer). Determine the accuracy of the classifier using each of the following methods.

The holdout method, where two-thirds of the data are used for training and the remaining one-third are used for testing.

Ten-fold cross-validation.

The .632 bootstrap method.

Flom the results in parts (u), (b), and (c), which method provides a more reliable evaluation of the classifier's accuracy?

11. Consider the following approach for testing whether a classifier A beats another classifier B. Let l[ be the size of a given data set, pa be the accuracy of classifier A, ps be the accuracy of classifier B, and p: (pt +pB)12 be the average accuracy for both classifiers. To test whether classifier A is significantly better than B, the following Z-statistic is used:

P A - P B

( a )

(b)

( c )

( d )

Classifier A is assumed to be better than classifier B if Z >

2O4 Chapter 4 Classification

Table 4.9 compares the accuracies of three different classifiers, decision tree classifiers, naive Bayes classifiers, and support vector machines, on various data sets. (The latter two classifiers are described in Chapter 5.)

Table 4.9. Comparing the accuracy of various classification methods.

Data Set Size (r/)

Decision Tlee (%)

nalve

Bayes (%)

Support vector machine (%)

nneal Australia Auto Breast Cleve Credit Diabetes German GIass Heart Hepatitis Horse Ionosphere Iris Labor LedT Lymphography Pima Sonar Tic-tac-toe Vehicle Wine Zoo

898 690 205 699 303 690 768 1000 214 270 1 5 5 368 3 5 1 150

F A

3200 L48 / b 6

208 958 846 178 1 0 1

92.09 6 b . b l

8 1 . 9 5 9 5 . 1 4 76.24 8 5 . 8 0 72.40 70.90 67.29 80.00 8 1 . 9 4 85.33 8 9 . 1 7 94.67 7 8 . 9 5 73.34 77.03 1 4 . ; t O

78.85 83.72 7 t . 0 4 94.38 93.07

79.62 76.81 58.05 95.99 83.50 77.54 7 5 . 9 1 74.70 48.59 84.07 8 3 . 2 3 78.80 82.34 9 5 . 3 3 94.74 73.16 8 3 . 1 1 76.04 69.7L 70.04 45.04 96.63 93.07

8 7 . 1 9 84.78 1 U . 1 J

96.42 84.49 8 5 . 0 7 76.82 74.40 5 9 . 8 1 83.70 8 7 . 1 0 82.6r 88.89 96.00 9 2 . 9 8 73.56 86.49 76.95 76.92 98.33 74.94 98.88 96.04

Summarize the performance of the classifiers given in Table 4.9 using the fol- l o w i n g 3 x 3 t a b l e :

win-loss-draw Decision tree Naive Bayes Support vector machine

Decision tree 0 - 0 - 2 3 NaiVe tsayes 0 - 0 - 2 3 Support vector machine 0 - 0 - 2 3

Each cell in the table contains the number of wins. losses, and draws when comparing the classifier in a given row to the classifier in a given column.

1 2 .

4.8 Exercises 2O5

Let X be a binomial random variable with mean lr'p and variance lfp(l -p).

Show that the ratio Xf N also has a binomial distribution with mean p and v a r i a n c e p ( t - p ) l N .

Classification: Alternative Tech n iq ues

The previous chapter described a simple, yet quite effective, classification tech- nique known as decision tree induction. Issues such as model overfitting and classifier evaluation were also discussed in great detail. This chapter presents alternative techniques for building classification models-from simple tech- niques such as rule-based and nearest-neighbor classifiers to more advanced techniques such as support vector machines and ensemble methods. Other key issues such as the class imbalance and multiclass problems are also dis- cussed at the end of the chapter.

5.1 Rule-Based Classifier

A rule-based classifier is a technique for classifying records using a collection of "if . . .then. . ." rules. Table 5.1 shows an example of a model generated by a rule-based classifier for the vertebrate classification problem. The rules for the model are represented in a disjunctive normal form, .R : (r1Yr2V...rn), where R is known as the rule set and r;'s are the classification rules or disjuncts.

Table 5.1. Example of a rule set for the vertebrate classification problem.

rri (Gives Birth : no) n (Aerial Creature : yes) ------+ Birds r2i (Gives Birth : no) n (Aquatic Creature : yes) ------+ Fishes 13: (Gives Birth : yes) n (Body Temperature : warm-blooded) ------+ Mammals 14: (Gives Birth : no) n (Aerial Creature : no) ----- Reptiles rbi (Aquatic Creature : semi) ------+ Amphibians

208 Chapter 5 Classification: Alternative Techniques

Each classification rule can be expressed in the following way:

ri: (Condi,ti,ont) - gr. ( 5 . 1 )

The left-hand side of the rule is called the rule antecedent or precondition.

It contains a conjunction of attribute tests:

Condi,tion,i : (At op ur) A (Az op u2) A . . .(An oP u*), ( 5 . 2 )

where (Ai,ui) is an attribute-value pair and op is a logical operator chosen from the set {:, 1,1,},<,>}. Each attribute test (Ai op u7) is known as a conjunct. The right-hand side of the rule is called the rule consequent, which contains the predicted class g;.

A rule r covers a record r if the precondition of r matches the attributes of r. r is also said to be fired or triggered whenever it covers a given record.

For an illustration, consider the rule 11 given in Table 5.1 and the following attributes for two vertebrates: hawk and grizzly bear.

Name Body Temperature

SK1n Cover

Glves Birth

Aquatic Creature

Ae r1aI Creature

Has Legs

Hiber- nates

hawk qrizzly bear

warm-blooded warm-blooded

feather fur

no yes

no no

yes no

yes yes

n o ves

11 covers the first vertebrate because its precondition is satisfied by the hawk's attributes. The rule does not cover the second vertebrate because grizzly bears give birth to their young and cannot fly, thus violating the precondition of 11.

The quality of a classification rule can be evaluated using measures such as coverage and accuracy. Given a data set D and a classification rule r : A -+ At the coverage of the rule is defined as the fraction of records in D that trigger the rule r. On the other hand, its accuracy or confidence factor is defined as the fraction of records triggered by r whose class labels are equal to gr. The formal definitions of these measures are

Coverage(r ) !!l t D l

. l A n a lAccuracy(r) : T

( 5 . 3 )

where lAl is the number of records that satisfy the rule antecedent, lA n gl is the number of records that satisfy both the antecedent and consequent, and lDl is the total number of records.

5.1 Rule-Based Classifier 2Og

Table 5,2. The vertebrate data set.

Example 5.1. Consider the data set shown in Table 5.2. The rule

(G:.ves Birth : yes) A (Body Temperature : warn-blooded) ------+ Mammals

has a coverage of 33% since five of the fifteen records support the rule an- tecedent. The rule accuracy is 100% because all five vertebrates covered by the rule are mammals. r

5.1.1 How a Rule-Based Classifier Works

A rule-based classifier classifies a test record based on the rule triggered by the record. To illustrate how a rule-based classifier works, consider the rule set shown in Table 5.1 and the followins vertebrates:

o The first vertebrate, which is a lemur, is warm-blooded and gives birth to its young. It triggers the rule 13, and thus, is classified as a mammal.

l\ame lJody

Temperature

Skin Cover

Gives Birth

Aquatic Creature

Aerial Creature

-nas Legs

lllDer-

nates

C'lass Label

numan python salmon whale froo

komodo dragon bat

guppv alligator penguln porcuplne eel salamander

prgeon cat

warm-blooded cold-blooded cold-blooded

warm-blooded cold-blooded cold-blooded

warm-blooded warm-blooded warm-blooded cold-blooded cold-blooded

warm-blooded warm-blooded cold-blooded cold-blooded

scales

hair feathers

fur scales scales

feathers quills scales none

nalr scales scales hair none

yes no no yes no no

yes no yes yes no no yes no no

no no yes yes

semr no

no no no yes

seml semr no yes

semr

no no no no no no

yes yes no no no no no no no

yes no no no yes yes

yes yes yes no yes yes yes no yes

no yes no no yes no

yes no no no no no yes

no yes

IVlammals Reptiles Fishes

Mammals Amphibians

Reptiles

Mammals Birds

Mammals Fishes

Reptiles Birds

Mammals Fishes

Amphibians

Name lJody Temperature

n Cover Skt Glves

Birth Aquatic Creature

Aerial Creature

Has Legs

Hiber- nates

lemur turtle dogfish shark

warm-blooded cold-blooded cold-blooded

fur scales scales

yes no yes

no semr yes

no no n o

yes yes no

yes no no

zLO Chapter 5 Classification: Alternative Techniques

o The second vertebrate, which is a turtle, triggers the rules 14 and rs. Since the classes predicted by the rules are contradictory (reptiles versus amphibians), their conflicting classes must be resolved.

o None of the rules are applicable to a dogfish shark. In this case, we need to ensure that the classifier can still make a reliable prediction even though a test record is not covered by any rule.

The previous example illustrates two important properties of the rule set gen-

erated by a rule-based classifier.

Mutually Exclusive Rules The rules in a rule set .R are mutually exclusive if no two rules in .R are triggered by the same record. This property ensures that every record is covered by at most one rule in R. An example of a mutually exclusive rule set is shown in Table 5.3.

Exhaustive Rules A rule set -R has exhaustive coverage if there is a rule for each combination of attribute values. This property ensures that every record is covered by at least one rule in -R. Assuming that Body Tenperature and Gives Birth are binarv variables. the rule set shown in Table 5.3 has exhaustive coverage.

Table 5.3. Example of a mutually exclusive and exhaustive rule set.

11: (Body Temperature : cold-blooded) ---- Non-mammals 12: (Body Temperature : warm-blooded) A (Gives Birth : yes) .* Mammals 13: (Body Temperature : warm-blooded) A (Gives Birth : no) --' Non-mammals

Together, these properties ensure that every record is covered by exactly one rule. Unfortunately, many rule-based classifiers, including the one shown in Table 5.1, do not have such properties. If the rule set is not exhaustive, then a default rt7e, r4: 0 .----+ !4, must be added to cover the remaining cases. A default rule has an empty antecedent and is triggered when all other rules have failed. gr4 is known as the default class and is typically assigned to the majority class of training records not covered by the existing rules.

If the rule set is not mutually exclusive, then a record can be covered by several rules, some of which may predict conflicting classes. There are two ways to overcome this problem.

5 . 1 Rule-BasedClassifier 2LI

Ordered Rules In this approach, the rules in a rule set are ordered in decreasing order of their priority, which can be defined in many ways (e.g., based on accuracy, coverage, total description length, or the order in which the rules are generated). An ordered rule set is also known as a decision list. When a test record is presented, it is classified by the highest-ranked rule that covers the record. This avoids the problem of having conflicting classes predicted by multiple classification rules.

IJnordered Rules This approach allows a test record to trigger multiple classification rules and considers the consequent of each rule as a vote for a particular class. The votes are then tallied to determine the class label of the test record. The record is usually assigned to the class that receives the highest number of votes. In some cases, the vote may be weighted by the rule's accuracy. Using unordered rules to build a rule-based classifier has both advantages and disadvantages. Unordered rules are less susceptible to errors caused by the wrong rule being selected to classify a test record (unlike classifiers based on ordered rules, which are sensitive to the choice of rule- ordering criteria). Model building is also less expensive because the rules do not have to be kept in sorted order. Nevertheless, classifying a test record can be quite an expensive task because the attributes of the test record must be compared against the precondition of every rule in the rule set.

In the remainder of this section, we will focus on rule-based classifiers that use ordered rules.

5.L.2 Rule-Ordering Schemes

Rule ordering can be implemented on a rule-by-rule basis or on a class-by-class basis. The difference between these schemes is illustrated in Figure 5.1.

Rule-Based Ordering Scheme This approach orders the individual rules by some rule quality measure. This ordering scheme ensures that every test record is classified by the "best" rule covering it. A potential drawback ofthis scheme is that lower-ranked rules are much harder to interpret because they assume the negation of the rules preceding them. For example, the fourth rule shown in Figure 5.1 for rule-based ordering,

Aquatic Creature : semi ------+ Amphibians,

has the following interpretation: If the vertebrate does not have any feathers or cannot fly, and is cold-blooded and semi-aquatic, then it is an amphibian.

2L2 Chapter 5 Classification: Alternative Techniques

Rule-Based Ordering

(Skin Cover=feathers, Aerial 9r"31u1s=yes) ==> Birds

(Body temperature=warm-blooded, Gives Birth=yes) ==> 1446t";.

(Body temperature=warm-blooded, Gives Birth=no) =-> Birds

(Aquatic Creature=semi)) ==> 4606;6;"n.

(Skin Cover=scales, Aquatic Creature=no) =-> Reptiles

(Skin Cover=scales, Aquatic gt"u1u1s=yes) ==> Fishes

(Skin Cover=none) ==> Amphibians

Class-Based Ordering

(Skin Cover=feathers, Aerial gte2luvs=yes) ==> Birds

(Body temperature=warm-blooded, Gives Birth=no) ==> Birds

(Body temperature=warm-blooded, Gives Birth=yes) ==> J1l36P41t

(Aquatic Creature=semi)) ==> 4601'';6'"n.

(Skin Cover=none) ==> Amphibians

(Skin Cover=scales, Aquatic gr"s1u1e=no) ==> Reptiles

(Skin Cover=scales, Aquatic gtsslups=yes) ==> Fishes

Figure 5.1. Comparison between rule-based and class-based ordering schemes.

The additional conditions (that the vertebrate does not have any feathers or cannot fly, and is cold-blooded) are due to the fact that the vertebrate does not satisfy the first three rules. If the number of rules is large, interpreting the meaning of the rules residing near the bottom of the list can be a cumbersome task.

Class-Based Ordering Scheme In this approach, rules that belong to the same class appear together in the rule set R. The rules are then collectively sorted on the basis of their class information. The relative ordering among the rules from the same class is not important; as long as one of the rules fires, the class will be assigned to the test record. This makes rule interpretation slightly easier. However, it is possible for a high-quality rule to be overlooked in favor of an inferior rule that happens to predict the higher-ranked class.

Since most of the well-known rule-based classifiers (such as C4.5rules and RIPPER) employ the class-based ordering scheme, the discussion in the re- mainder of this section focuses mainly on this type of ordering scheme.

5.1.3 How to Build a Rule-Based Classifier

To build a rule-based classifier, we need to extract a set of rules that identifies key relationships between the attributes of a data set and the class label.

5 . 1 Rule-BasedClassifier 2I3

There are two broad classes of methods for extracting classification rules: (1) direct methods, which extract classification rules directly from data, and (2) indirect methods, which extract classification rules from other classification models, such as decision trees and neural networks.

Direct methods partition the attribute space into smaller subspaces so that all the records that belong to a subspace can be classified using a single classi- fication rule. Indirect methods use the classification rules to provide a succinct description of more complex classification models. Detailed discussions of these methods are presented in Sections 5.1.4 and 5.1.5, respectively.

5.L.4 Direct Methods for Rule Extraction

The sequential covering algorithm is often used to extract rules directly from data. Rules are grown in a greedy fashion based on a certain evaluation measure. The algorithm extracts the rules one class at a time for data sets that contain more than two classes. For the vertebrate classification problem, the sequential covering algorithm may generate rules for classifying birds first, followed by rules for classifying mammals, amphibians, reptiles, and finally, fishes (see Figure 5.1). The criterion for deciding which class should be gen- erated first depends on a number of factors, such as the class prevalence (i.e., fraction of training records that belong to a particular class) or the cost of misclassifyirig records from a given class.

A summary of the sequential covering algorithm is given in Algorithm 5.1. The algorithm starts with an empty decision list, .R. The Learn-One- Rule function is then used to extract the best rule for class y that covers the current set of training records. During rule extraction, all training records for class gr are considered to be positive examples, while those that belong to

Algorithm 5.1 Sequential covering algorithm. t: Let E be the training records and ,4 be the set of attribute-value pairs, {(,4i, u7)}. 2: Let Y, be an ordered set of classes {yt,yz,. . . ,a*}. 3: Let R: { } be the initial rule list. 4: for each class U e Yo - {gr} do 5: while stopping condition is not met do 6: r +- Learn-One-Rule (E, A, y). 7: Remove training records from -E that are covered by r. 8 : A d d r t o t h e b o t t o m o f t h e r u l e l i s t : - R - - ' R V r . 9: end while

10: end for 11: Insert the default rule, {} + Uk, to the bottom of the rule list R.

2L4 Chapter 5 Classification: Alternative Techniques

other classes are considered to be negative examples. A rule is desirable if it

covers most of the positive examples and none (or very few) of the negative examples. Once such a rule is found, the training records covered by the rule are eliminated. The new rule is added to the bottom of the decision list R. This procedure is repeated until the stopping criterion is met. The algorithm then proceeds to generate rules for the next class.

Figure 5.2 demonstrates how the sequential covering algorithm works for

a data set that contains a collection of positive and negative examples. The rule ,R1, whose coverage is shown in Figure 5.2(b), is extracted first because it covers the largest fraction of positive examples. All the training records covered by .Rl are subsequently removed and the algorithm proceeds to Iook for the next best rule. which is R2.

- : " " " " ' : : + + : i + + i : + - r :

- : - . - . - - . - 1 . - ;

(a)Original Data

(c) Step 2

Figure 5.2. An example of the sequential covering algorithm.

i R l :

- + + , +

T

( b ) S t e p 1

: R 2

: R 1 i - : . . , . . . . . . . i

- + * + *

(d) Step 3

Rule-BasedClassifier 2L5

Learn-One-Rule F\rnction

The objective of the Learn-One-Rule function is to extract a classification rule that covers many of the positive examples and none (or very few) of the negative examples in the training set. However, finding an optimal rule is computationally expensive given the exponential size of the search space. The Learn-one-Rule function addresses the exponential search problem by growing the rules in a greedy fashion. It generates an initial rule r and keeps refining the rule until a certain stopping criterion is met. The rule is then pruned to improve its generalization error.

Rule-Growing Strategy There are two common strategies for growing a classification rule: general-to-specific or specific-to-general. Under the general- to-specific strategy, an initial rule r , {} - 3r is created, where the left-hand side is an empty set and the right-hand side contains the target class. The rule has poor quality because it covers all the examples in the training set. New

(b) Specificto-general

Figure 5.3. General{o-specific and specific-to-general rule-growing strategies.

5 . 1

Body Temperature = warm-blooded

Body Temperature = warm-blooded, Has Legs = yes => Mammals

Body Temperature = warm-blooded, - Gives Birth = yes => Mammals _

(a) General-to-specif ic

Body Temperature=warm-blooded, Skin Cover=hair, Gives Birth=yes, Aquatic creature=no, Aerial Creature=no

Has Legs=yes, Hjbernales=no => Mammals

Skin Cover=hair, Gives Birth=yes Aquatic Creature=no, Aerial Creature=no,

Has Legs=yes, Hibernates=no => Mammals

Body Temperature=warm-blooded, Skin Cover=hair, Gives Birth=yes,

Aquatic creature=no, Aeilal Creature=no Has Legs=yes => Mammals

2L6 Chapter 5 Classification: Alternative Techniques

conjuncts are subsequently added to improve the rule's quality. Figure 5.3(a) shows the general-to-specific rule-growing strategy for the vertebrate classifi- cation problem. The conjunct Body Tenperature=warn-blooded is initially chosen to form the rule antecedent. The algorithm then explores all the possi-

ble candidates and greedily chooses the next conjunct, Gives Birth=yes, to be added into the rule antecedent. This process continues until the stopping criterion is met (e.g., when the added conjunct does not improve the quality of the rule).

For the specific-to-general strategy, one of the positive examples is ran- domly chosen as the initial seed for the rule-growing process. During the refinement step, the rule is generalized by removing one of its conjuncts so that it can cover more positive examples. Figure 5.3(b) shows the specific-to- general approach for the vertebrate classification problem. Suppose a positive example for mammals is chosen as the initial seed. The initial rule contains the same conjuncts as the attribute values of the seed. To improve its cov- erage, the rule is generalized by removing the conjunct Hibernate=no. The refinement step is repeated until the stopping criterion is met, e.g., when the rule starts covering negative examples.

The previous approaches may produce suboptimal rules because the rules are grown in a greedy fashion. To avoid this problem, a beam search may be used, where k of the best candidate rules are maintained by the algorithm. Each candidate rule is then grown separately by adding (or removing) a con- junct from its antecedent. The quality ofthe candidates are evaluated and the k best candidates are chosen for the next iteration.

Rule Evaluation An evaluation metric is needed to determine which con- junct should be added (or removed) during the rule-growing process. Accu- racy is an obvious choice because it explicitly measures the fraction of training examples classified correctly by the rule. However, a potential limitation of ac- curacy is that it does not take into account the rule's coverage. For example, consider a training set that contains 60 positive examples and 100 negative examples. Suppose we are given the following two candidate rules:

Rule 11 : covers 50 positive examples and 5 negative examples, Rule 12: covers 2 positive examples and no negative examples.

The accuracies for 11 and 12 are 90.9% and 100%, respectively. However, 11 is the better rule despite its lower accuracy. The high accuracy for 12 is potentially spurious because the coverage of the rule is too low.

5 . 1 Rule-BasedClassifier 2L7

The following approaches can be used to handle this problem.

1. A statistical test can be used to prune rules that have poor coverage. For example, we may compute the following likelihood ratio statistic:

K

R : 2 \ f i t o s f f , ; l e ) , i : I

where k is the number of classes, fi is the observed frequency of class e examples that are covered by the rule, and el is the expected frequency of a rule that makes random predictions. Note that ,R has a chi-square distribution with k - 1 degrees of freedom. A large l? value suggests that the number of correct predictions made by the rule is significantly larger than that expected by random guessing. For example, since 11 covers 55 examples, the expected frequency for the positive class is ea :

55x601160:20.625, while the expected frequency for the negative class is e- : 55 x 100/160 : 34.375. Thus, the likelihood ratio for 11 is

E(r1) : 2 x [50 x log2(50120.625) f 5 x logr(5134.375)): 99.9.

Similarly, the expected frequencies for 12 a,re e+ : 2 x 60/160 : 0.75 a n d e - : 2 x 1 0 0 / 1 6 0 : 1 . 2 5 . T h e l i k e l i h o o d r a t i o s t a t i s t i c f o r r e i s

R(r2) : 2 x [2 x log2(210.75) + 0 x log2(0/1.25)] : 5.66.

This statistic therefore suggests that rr is a better rule than 12.

2. At evaluation metric that takes into account the rule coverage can be used. Consider the following evaluation metrics:

f , - r 1 Laplace :

m-estimate :

n l k '

f+ -t kp+

( 5 . 4 )

( O . D J n - f k

I

where n is the number of examples covered by the rule, /-p is the number of positive examples covered by the rule, k is the total number of classes, and p1 is the prior probability for the positive class. Note that the m- estimate is equivalent to the Laplace measure by choosing p+ : llk. Depending on the rule coverage, these measures capture the trade-off

218 Chapter 5 Classification: Alternative Techniques

between rule accuracy and the prior probability of the positive class. If

the rule does not cover any training example, then the Laplace mea-

sure reduces to lf k, which is the prior probability of the positive class assuming a uniform class distribution. The m-estimate also reduces to the prior probability (p1) when n : 0. However, if the rule coverage is large, then both measures asymptotically approach the rule accuracy,

f+ln. Going back to the previous example, the Laplace measure for 11 is 51157 : 89.47To, which is quite close to its accuracy. Conversely, the Laplace measure for 12 (75%) is significantly lower than its accuracy because 12 has a much lower coverage.

3. An evaluation metric that takes into account the support count of the rule can be used. One such metric is the FOILts information gain.

The support count of a rule corresponds to the number of positive exam- ples covered by the rule. Suppose the rule r : A ------ * covers ps positive

examples and ns negative examples. After adding a new conjunct B, the extended rule r' : A tt, B ------+ * covers p1 positive examples and n1 neg- ative examples. Given this information, the FOIL's information gain of the extended rule is defined as follows:

FoIL's information gain: pt x ( bsr=!= - logz - 1o- ). (5.6) \

" ' P r + n r " - P o l n s l

Since the measure is proportional to pr and ptl(n*nr), it prefers rules that have high support count and accuracy. The FOIL's information gains for rules 11 and 12 given in the preceding example are 43.12 and 2, respectively. Therefore, 11 is a better rule than 12.

Rule Pruning The rules generated by the Learn-One-Rule function can be pruned to improve their generalization errors. To determine whether pruning is necessary, w€ may apply the methods described in Section 4.4 on page

172 to estimate the generalization error of a rule. For example, if the error on validation set decreases after pruning, we should keep the simplified rule. Another approach is to compare the pessimistic error of the rule before and after pruning (see Section 4.4.4on page 179). The simplified rule is retained in place of the original rule if the pessimistic error improves after pruning.

5.1 Rule-Based Classifier 2I9

Rationale for Sequential Covering

After a rule is extracted, the sequential covering algorithm must eliminate all the positive and negative examples covered by the rule. The rationale for doing this is given in the next example.

class = +

T

+ +

+ + + +

class =

Figure 5.4. Elimination of training records by the sequential covering algorithm. R7, R2, and R3 represent regions covered by three different rules.

Figure 5.4 shows three possible rules, R7, R2, and R3, extracted from a data set that contains 29 positive examples and 21 negative examples. The accuracies of .R1, R2, and E3 are I2115 (80%), 7lI0 (70%), and 8f L2 (66.7%), respectively. .R1 is generated first because it has the highest accuracy. After generating R1, it is clear that the positive examples covered by the rule must be removed so that the next rule generated by the algorithm is different than .R1. Next, suppose the algorithm is given the choice of generating either R2 or R3. Even though R2 has higher accuracy than -R3, Rl and -R3 together cover 18 positive examples and 5 negative examples (resulting in an overail accuracy of 78.3%), whereas R1 and .R2 together cover 19 positive examples and 6 negative examples (resulting in an overall accuracy of 76%). The incremental impact of R2 or -R3 on accuracy is more evident when the positive and negative exarnples covered by -Rl are removed before computing their accuracies. In particular, if positive examples covered by R1 are not removed, then we may overestimate the effective accuracy of ,R3, and if negative examples are not removed, then we may underestimate the accuracy of R3. In the latter caseT we might end up preferring R2 over fi3 even though half of the false positive errors committed by E3 have already been accounted for by the preceding rule, .R1.

R2R3

22O Chapter 5 Classification: Alternative Techniques

RIPPER Algorithm

To illustrate the direct method, we consider a widely used rule induction algo- rithm called RIPPER. This algorithm scales almost linearly with the number of training examples and is particularly suited for building models from data sets with imbalanced class distributions. RIPPER also works well with noisy data sets because it uses a validation set to prevent model overfitting.

For two-class problems, RIPPER chooses the majority class as its default class and learns the rules for detecting the minority class. For multiclass prob- lems, the classes are ordered according to their frequencies. Let (y,Az, . . . ,U") be the ordered classes, where 91 is the least frequent class and g" is the most frequent class. During the first iteration, instances that belong to 91 are Ia- beled as positive examples, while those that belong to other classes are labeled as negative examples. The sequential covering method is used to generate rules that discriminate between the positive and negative examples. Next, RIPPER extracts rules that distinguish y2 frorn other remaining classes. This process is repeated until we are left with g., which is designated as the default class.

Rule Growing RIPPER employs a general-to-specific strategy to grow a rule and the FOIL's information gain measure to choose the best conjunct to be added into the rule antecedent. It stops adding conjuncts when the rule starts covering negative examples. The new rule is then pruned based on its performance on the validation set. The following metric is computed to determine whether pruning is needed: (p-") l@+n), where p (n) is the number of positive (negative) examples in the validation set covered by the rule. This metric is monotonically related to the rule's accuracy on the validation set. If the metric improves after pruning, then the conjunct is removed. Pruning is done starting from the last conjunct added to the rule. For example, given a rde ABCD + a, RIPPER checks whether D should be pruned first, followed by CD, BCD, etc. While the original rule covers only positive examples, the pruned rule may cover some of the negative examples in the training set.

Building the Rule Set After generating a rule, all the positive and negative examples covered by the rule are eliminated. The rule is then added into the rule set as long as it does not violate the stopping condition, which is based on the minimum description length principle. If the new rule increases the total description length of the rule set by at least d bits, then RIPPER stops adding rules into its rule set (by default, d is chosen to be 64 bits). Another stopping condition used by RIPPER is that the error rate of the rule on the validation set must not exceed 50%.

5 . 1 Rule-BasedClassifier 22L

RIPPER also performs additional optimization steps to determine whether some of the existing rules in the rule set can be replaced by better alternative rules. Readers who are interested in the details of the optimization method may refer to the reference cited at the end of this chapter.

5.1.5 Indirect Methods for Rule Extraction

This section presents a method for generating a rule set from a decision tree. In principle, every path from the root node to the Ieaf node of a decision tree can be expressed as a classification rule. The test conditions encountered along the path form the conjuncts ofthe rule antecedent, while the class label at the leaf node is assigned to the rule consequent. Figure 5.5 shows an example of a rule set generated from a decision tree. Notice that the rule set is exhaustive and contains mutually exclusive rules. However, some of the rules can be simplified as shown in the next example.

Rule Set

rl: (P=No,Q=No) =-> -

12: (P=No,Q=Yes) ==2 'i r3; (P=Yes,Q=No) ==1 .' 14: (P=Yes,R=Yes,Q=No) ==> -

r5: (P=Yes,R=Yes,Q=Yes) ==> a

Flgure 5.5. Converting a decision tree into classification rules.

Example 5.2. Consider the following three rules from Figure 5.5:

12: (P : No) A (Q : Yes) ------+ -1-

r3: (P : Yes) n (R: No) -----+ 1 r5: (P : Yes) A (R: Yes) n (Q : Yes) ------+ f

Observe that the rule set always predicts a positive class when the value of Q is Yes. Therefore, we may simplify the rules as follows:

r2t; (Q : Yes) ----+ f r3: (P : Yes) A (R: No) ------+ 1.

222 Chapter 5 Classification: Alternative Techniques

Rul+Based Classifier: (Gives Birth=No, Aerial Creature=Yes) => Birds

(Gives Birth=No, Aquatic Creature=Yes) => Fishes (Gives Birth=Yes) => Mammals

(Gives Birth=No, Aerial Creature=No, Aquatic Creature=No) => Reptiles

( ) => Amphibians

Figure 5.6. Classification rules extracted from a decision tree for the vertebrate classification problem.

13 is retained to cover the remaining instances of the positive class. Although the rules obtained after simplification are no longer mutually exclusive, they are less complex and are easier to interpret. I

In the following, we describe an approach used by the C4.5rules algorithm to generate a rule set from a decision tree. Figure 5.6 shows the decision tree and resulting classification rules obtained for the data set given in Table 5.2.

Rule Generation Classification rules are extracted for every path from the root to one of the leaf nodes in the decision tree. Given a classification rule r : A -------+ gr, we consider a simplified rule, r' : A' + A, where A/ is obtained by removing one of the conjuncts in A. The simplified rule with the lowest pessimistic error rate is retained provided its error rate is less than that of the original rule. The rule-pruning step is repeated until the pessimistic error of the rule cannot be improved further. Because some of the rules may become identical after pruning, the duplicate rules must be discarded.

Rule Ordering After generating the rule set, C4.5rules uses the class-based ordering scheme to order the extracted rules. Rules that predict the same class are grouped together into the same subset. The total description length for each subset is computed, and the classes are arranged in increasing order of their total description length. The class that has the smallest description

5 . 2 Nearest-Neighbor classifiers 223

length is given the highest priority because it is expected to contain the best set of rules. The total description length for a class is given by tr"*ception * g X

Imodel, where trexception is the number of bits needed to encode the misclassified

examples, Zmodel is the number of bits needed to encode the model, and g is a

tuning parameter whose default value is 0.5. The tuning parameter depends on the number of redundant attributes present in the model. The value of the

tuning parameter is small if the model contains many redundant attributes.

5.1.6 Characteristics of Rule-Based Classifiers

A rule-based classifier has the following characteristics:

o The expressiveness of a rule set is almost equivalent to that of a decision

tree because a decision tree can be represented by a set of mutually ex-

clusive and exhaustive rules. Both rule-based and decision tree classifiers

create rectilinear partitions of the attribute space and assign a class to

each partition. Nevertheless, if the rule-based classifier allows multiple

rules to be triggered for a given record, then a more complex decision

boundary can be constructed.

o Rule-based classifiers are generally used to produce descriptive models

that are easier to interpret, but gives comparable performance to the

decision tree classifier.

o The class-based ordering approach adopted by many rule-based classi-

fiers (such as RIPPER) is well suited for handling data sets with imbal-

anced class distributions.

5.2 Nearest-Neighbor classifiers

The classification framework shown in Figure 4.3 involves a two-step process:

(1) an inductive step for constructing a classification model from data, and (2) a deductive step for applying the model to test examples. Decision tree

and rule-based classifiers are examples of eager learners because they are

designed to learn a model that maps the input attributes to the class label as

soon as the training data becomes available. An opposite strategy would be to

delay the process of modeling the training data until it is needed to classify the

test examples. Techniques that employ this strategy are known as lazy learn-

ers. An example of alazy learner is the Rote classifier, which memorizes the

entire training data and performs classification only if the attributes of a test

instance match one of the training examples exactly. An obvious drawback of

224 Chapter 5 Classification: Alternative Techniques

++ (a) 1-nearest neighbor

Figure 5.7. The

(b) 2-nearest neighbor (c) 3-nearest neighbor

1-,2-,and 3-nearest neighbors of an instance.

this approach is that some test records may not be classified because they do not match any training example.

One way to make this approach more flexible is to find all the training examples that are relatively similar to the attributes of the test example. These examples, which are known as nearest neighbors, can be used to determine the class label of the test example. The justification for using nearest neighbors is best exemplified by the following saying: "If i,t walks li,ke a duck, quacks li,lce a duck, and looks li,ke a duck, then i,t's probably a duck." A nearest- neighbor classifier represents each example as a data point in a d-dimensional space, where d is the number of attributes. Given a test example, we compute its proximity to the rest of the data points in the training set, using one of the proximity measures described in Section 2.4 on page 65. The k-nearest neighbors of a given example z refer to the k points that are closest to z.

Figure 5.7 illustrates the L-, 2-, and 3-nearest neighbors of a data point located at the center of each circle. The data point is classified based on the class labels of its neighbors. In the case where the neighbors have more than one label, the data point is assigned to the majority class of its nearest neighbors. In Figure 5.7(a), the l-nearest neighbor of the data point is a negative example. Therefore the data point is assigned to the negative class. If the number of nearest neighbors is three, as shown in Figure 5.7(c), then the neighborhood contains two positive examples and one negative example. Using the majority voting scheme, the data point is assigned to the positive class. In the case where there is a tie between the classes (see Figure 5.7(b)), we may randomly choose one of them to classify the data point.

The preceding discussion underscores the importance of choosing the right value for k. If k is too small, then the nearest-neighbor classifier may be

+ +

5 . 2 Nearest-Neighborclassifiers 225

Figure 5.8. k-nearest neighbor classification with large /c.

susceptible to overfitting because of noise in the training data. On the other hand, if k is too large, the nearest-neighbor classifier may misclassify the test instance because its list of nearest neighbors may include data points that are located far away from its neighborhood (see Figure 5.8).

5.2.L Algorithm

A high-level summary of the nearest-neighbor classification method is given in Algorithm 5.2. The algorithm computes the distance (or similarity) between each test example , : (*',y') and all the training examples (x, g) e D to determine its nearest-neighbor list, D". Such computation can be costly if the number of training examples is large. However, efficient indexing techniques are available to reduce the amount of comoutations needed to find the nearest neighbors of a test example.

Algorithm 5.2 The k-nearest neighbor classification algorithm. 7: Let k be the number of nearest neighbors and D be the set of training examples. 2: fot each test example

" : (x' ,E') do

3: Compute d(x',x), the distance between z and every example, (x,y) e D. 4: Select D, ! D, the set of k closest training examples to z. 5: y' : argmax-D6u,ouy.n" I(u : yu1

6: end for

226 Chapter 5 Classification: Alternative Techniques

Once the nearest-neighbor list is obtained, the test example is classified based on the majority class of its nearest neighbors:

Majority Voting: g' : argmax I ( , : A ) , ( 5 . 7 ) U" (xi,g6)€D.

where u is a class label, 916 is the class label for one of the nearest neighbors, and 1(.) is an indicator function that returns the value 1 if its argument is true and 0 otherwise.

In the majority voting approach, every neighbor has the same impact on the classification. This makes the algorithm sensitive to the choice of k, as shown in Figure 5.7. One way to reduce the impact of k is to weight the influence of each nearest neighbor x; according to its distance: wt.:7ld(x',x4)2. As a result, training examples that are located far away from z have a weaker impact on the classification compared to those that are located close to z. Using the distance-weighted voting scheme, the class label can be determined as follows:

Distance-Weighted Voting: A' : argrnax u; x I(u : Ai). (5.8) ( x ; , v 6 ) € D .

5.2.2 Characteristics of Nearest-Neighbor Classifiers

The characteristics of the nearest-neighbor classifier are summarized below:

o Nearest-neighbor classification is part of a more general technique known as instance-based learning, which uses specific training instances to make predictions without having to maintain an abstraction (or model) de- rived from data. Instance-based learning algorithms require a proximity measure to determine the similarity or distance between instances and a classification function that returns the predicted class of a test instance based on its proximity to other instances.

o Lazy learners such as nearest-neighbor classifiers do not require model building. However, classifying a test example can be quite expensive because we need to compute the proximity values individually between the test and training examples. In contrast, eager learners often spend the bulk of their computing resources for model building. Once a model has been built, classifying a test example is extremely fast.

o Nearest-neighbor classifiers make their predictions based on local infor- mation, whereas decision tree and rule-based classifiers attempt to find

BayesianClassifiers 227

a global model that fits the entire input space. Because the classification decisions are made locally, nearest-neighbor classifiers (with small values of /c) are quite susceptible to noise.

Nearest-neighbor classifiers can produce arbitrarily shaped decision bound- aries. Such boundaries provide a more flexible model representation compared to decision tree and rule-based classifiers that are often con- strained to rectilinear decision boundaries. The decision boundaries of nearest-neighbor classifiers also have high variability because they de- pend on the composition of training examples. Increasing the number of nearest neighbors may reduce such variability.

Nearest-neighbor classifiers can produce wrong predictions unless the appropriate proximity measure and data preprocessing steps are taken. For example, suppose we want to classify a group of people based on attributes such as height (measured in meters) and weight (measured in pounds). The height attribute has a low variability, ranging from 1.5 m to 1.85 m, whereas the weight attribute may vary from 90 lb. to 250 lb. If the scale of the attributes are not taken into consideration, the proximity measure may be dominated by differences in the weights of a person.

5.3 Bayesian Classifiers

In many applications the relationship between the attribute set and the class variable is non-deterministic. In other words, the class label of a test record cannot be predicted with certainty even though its attribute set is identical to some of the training examples. This situation may arise because of noisy data or the presence of certain confounding factors that affect classification but are not included in the analysis. For example, consider the task of pre-

dicting whether a person is at risk for heart disease based on the person's diet and workout frequency. Although most people who eat healthily and exercise regularly have less chance of developing heart disease, they may still do so be- cause of other factors such as heredity excessive smoking, and alcohol abuse. Determining whether a person's diet is healthy or the workout frequency is sufficient is also subject to interpretation, which in turn may introduce uncer- tainties into the learning problem.

This section presents an approach for modeling probabilistic relationships between the attribute set and the class variable. The section begins with an introduction to the Bayes theorem, a statistical principle for combining prior

5 . 3

228 Chapter 5 Classification: Alternative Techniques

knowledge of the classes with new evidence gathered from data. The use of the Bayes theorem for solving classification problems will be explained, followed by a description of two implementations of Bayesian classifiers: naiVe Bayes and the Bayesian belief network.

5.3.1 Bayes Theorem

Cons'ider a football game between two riual teams: Team 0 and Team 1. Suppose Team 0 wins 65%o of the ti,me and Team 1 w,ins the remaining matches. Among the games won by Team 0, only 30To of them come

from playing on Team 1's football field. On the other hand,,75To of the u'ictories for Team 1 are obta'ined whi,le playi.ng at home. If Team f is to host the nert match between the two teams, wh,ich team wi,ll most likelu etnerge as the winner?

This question can be answered by using the well-known Bayes theorem. For completeness, we begin with some basic definitions from probability theory. Readers who are unfamiliar with concepts in probability may refer to Appendix C for a brief review of this topic.

Let X and Y be a pair of random variables. Their joint probability, P(X :

r,Y : g), refers to the probability that variable X will take on the value r and variable Y will take on the value g. A conditional probability is the probability that a random variable will take on a particular value given that the outcome for another random variable is known. For example, the conditional probability P(Y :UlX: r) refers to the probability that the variable Y will take on the value g, given that the variable X is observed to have the value r. The joint and conditional probabilities for X and Y are related in the following way:

P(x,Y) : P(Ylx) x P(X) : P(XIY) x P(Y). ( 5 . e )

Rearranging the last two expressions in Equation 5.9 leads to the following formula, known as the Bayes theorem:

P(Ylx) : P(xlY)P(Y)

( 5 . 1 0 ) P ( X )

The Bayes theorem can be used to solve the prediction problem stated at the beginning of this section. For notational convenience, let X be the random variable that represents the team hosting the match and Y be the random variable that represents the winner of the match. Both X and Y can

5.3 Bayesian Classifiers 229

take on values from the set {0,1}. We can summarize the information given in the problem as follows:

Probability Team 0 wins is P(Y :0) : 0.65. Probability Team 1 wins is P(Y :1) : 1 - P(Y : 0) : 0.35. Probability Team t hosted the match it won is P(X : llY :1) : 0.75. Probability Team t hosted the match won by Team 0 is P(X : llY :0) : 0.9.

Our objective is to compute P(Y : llx : 1), which is the conditional probability that Team 1 wins the next match it will be hosting, and compares it against P(Y :OlX: 1). Using the Bayes theorem, we obtain

P ( Y : l l x : 1 ) : P ( X : r l Y : 1 ) x P ( Y : 1 )

P ( X : 1 )

P ( X : L l Y : 1 ) x P ( Y : 1 ) P ( X : ! , Y : 1 ) + P ( X : 1 , Y : 0 )

P ( X : L I Y : 1 ) x P ( Y : 1 ) P ( X : I I Y : I ) P ( Y : 1 ) * P ( X : r l Y : O ) P ( Y : 0 )

0.75 x 0.35 0 . 7 5 x 0 . 3 5 + 0 . 3 x 0 . 6 5

: 0 . 5 7 3 8 .

where the law of total probability (see Equation C.5 on page 722) was applied in the second line. Furthermore, P(Y :OlX : 1) : t - P(Y : llx - 1) :

0.4262. Since P(Y : llx : 1) > P(Y : OlX : 1), Team t has a better chance than Team 0 of winning the next match.

5.3.2 Using the Bayes Theorem for Classification

Before describing how the Bayes theorem can be used for classification, let us formalize the classification problem from a statistical perspective. Let X denote the attribute set and Y denote the class variable. If the class variable has a non-deterministic relationship with the attributes, then we can treat X and Y as random variables and capture their relationship probabilistically using P(YIX). This conditional probability is also known as the posterior probability for Y, as opposed to its prior probability, P(Y).

During the training phase, we need to learn the posterior probabilities P(ylX) for every combination of X and Y based on information gathered from the training data. By knowing these probabilities, a test record X' can be classified by finding the class Yt that maximizes the posterior probability,

230 Chapter 5 Classification: Alternative Techniques

P(Y'lX'). To illustrate this approach, consider the task of predicting whether a loan borrower will default on their payments. Figure 5.9 shows a training set with the following attributes: Hone Owner, Marital Status, and Annual Income. Loan borrowers who defaulted on their payments are classified as Yes, while those who repaid their loans are classifi.ed as No.

a."' """""od"

Figure 5.9. Training set for predicting the loan default problem.

Suppose we are given a test record with the following attribute set: X : (Hone Owner : No, Marital Status : Married, Annual Income : $120K). To classify the record, we need to compute the posterior probabilities P(YeslX) and P(NolX) based on information available in the training data. If P(YeslX) > P(NolX), then the record is classified as Yes; otherwise, it is classified as No.

Estimating the posterior probabilities accurately for every possible combi- nation of class labiel and attribute value is a difficult problem because it re- quires a very large training set, even for a moderate number of attributes. The Bayes theorem is useful because it allows us to express the posterior probabil- ity in terms of the prior probability P(f), the class-conditional probability P(X|Y), and the evidence, P(X):

P ( y l x ) : P ( x l Y ) x P ( Y )

( 5 . 1 1 ) P(x)

When comparing the posterior probabilities for different values of Y, the de- nominator term, P(X), is always constant, and thus, can be ignored. The

BayesianClassifiers 23L

prior probability P(f) can be easily estimated from the training set by com- puting the fraction of training records that belong to each class. To estimate the class-conditional probabilities P(Xlf), we present two implementations of Bayesian classification methods: the naiVe Bayes classifier and the Bayesian belief network. These implementations are described in Sections 5.3.3 and 5.3.5, respectively.

5.3.3 NaiVe Bayes Classifier

A naive Bayes classifier estimates the class-conditional probability by assuming that the attributes are conditionally independent, given the class label g. The

conditional independence assumption can be formally stated as follows:

&

P ( X I Y : a ) : f r l x S v : 1 1 , ( 5 . 1 2 ) i : l

where each attribute set X : {Xr, X2,..., X4} consists of d attributes.

Conditional Independence

Before delving into the details of how a naive Bayes classifier works, let us examine the notion of conditional independence. Let X, Y, and Z denote three sets of random variables. The variables in X are said to be conditionally independent of Y, given Z, 1f the following condition holds:

P(xlY, z) : P(xlz). ( 5 . 1 3 )

An example of conditional independence is the relationship between a person's

arm length and his or her reading skills. One might observe that people with longer arms tend to have higher levels of reading skills. This relationship can

be explained by the presence of a confounding factor, which is age. A young

child tends to have short arms and lacks the reading skills of an adult. If the age of a person is fixed, then the observed relationship between arm length

and reading skills disappears. Thus, we can conclude that arm length and reading skills are conditionally independent when the age variable is fixed.

5 . 3

232 Chapter 5 Classification: Alternative Techniques

The conditional independence between X and Y can also be written into a form that looks similar to Equation 5.12:

P ( x , Y l z ) : P \ X , Y , Z ) P ( Z )

P ( X . Y , Z ) , . P ( Y . Z )-FEq ^ P(z) P(xlY, z) x P(vlz) P(xlz) x P(Ylz), ( 5 . 1 4 )

where Equation 5.13 was used to obtain the last line of Equation 5.14.

How a Nai've Bayes Classifier Works

With the conditional independence assumption, instead of computing the class-conditional probability for every combination of X, we only have to esti- mate the conditional probability of each X4, given Y. The latter approach is more practical because it does not require a very large training set to obtain a good estimate of the probability.

To classify a test record, the naive Bayes classifier computes the posterior probability for each class Y:

( 5 . 1 5 )

Since P(X) is fixed for every Y, it is sufficient to choose the class that maxi- mizes the numerator term, p(V)l[i:tP(X,lY). In the next two subsections, we describe several approaches for estimating the conditional probabilities P(X,lY) for categorical and continuous attributes.

Estimating Conditional Probabilities for Categorical Attributes

For a categorical attribute Xa, the conditional probability P(Xi : rilY : A) is estimated according to the fraction of training instances in class g that take on a particular attribute value ri. For example, in the training set given in Figure 5.9, three out of the seven people who repaid their loans also own a home. As a result, the conditional probability for P(Home Owner:Yeslno) is equal to 3/7. Similarly, the conditional probability for defaulted borrowers who are single is given by P(Marital Status : SinglelYes) : 213.

p ( y t x ) - P V ) n ! : l P ( x i l Y )' P ( x )

5.3 Bayesian Classifiers 233

Estimating Conditional Probabilities for Continuous Attributes

There are two ways to estimate the class-conditional probabilities for contin- uous attributes in naive Bayes classifiers:

1. We can discretize each continuous attribute and then replace the con- tinuous attribute value with its corresponding discrete interval. This approach transforms the continuous attributes into ordinal attributes. The conditional probability P(X,IY : U) is estimated by computing the fraction of training records belonging to class g that falls within the corresponding interval for Xi. The estimation error depends on the dis- cretization strategy (as described in Section 2.3.6 on page 57), as well as the number of discrete intervals. If the number of intervals is too large, there are too few training records in each interval to provide a reliable estimate for P(XrlY). On the other hand, if the number of intervals is too small, then some intervals may aggregate records from different classes and we may miss the correct decision boundary.

2. We can assume a certain form of probability distribution for the contin- uous variable and estimate the parameters of the distribution using the training data. A Gaussian distribution is usually chosen to represent the class-conditional probability for continuous attributes. The distribution is characterized by two parameters, its mean, p,, and variance, o2. For each class Aj, the class-conditional probability for attribute Xi is

. , _ ( t t _ t t : j ) 2

P(Xi: r,ilY : y) : -)- exp zofi

1/2troii ( 5 . 1 6 )

The parameter p,ii can be estimated based on the sample mean of Xt (z) for all training record.s that belong to the class gt . Similarly, ol, can

be estimated from the sample variance (s2) of such training records. For

example, consider the annual income attribute shown in Figure 5.9. The sample mean and variance for this attribute with respect to the class No

are

r 2 5 + 1 0 0 + 7 0 + . . . + 7 5

,

: 1 1 0 (

( 1 2 5 - 1 1 0 ) 2 + ( 1 0 0 - 1 1 0 ) 2 + . . . + ( 7 5 - 1 1 0 ) 2

7 ( 6 )

s : t / 2 9 7 5 : 5 4 . 5 4 .

: 2 9 7 5

234 Chapter 5 Classification: Alternative Techniques

Given a test record with taxable income equal to $120K, we can compute its class-conditional probability as follows:

P(rncome=12olNo) : 6h.b4)"*p-95#f

: 0.0072.

Note that the preceding interpretation of class-conditional probability is somewhat misleading. The right-hand side of Equation 5.16 corre- sponds to a probability density function, f (X;pti,o;7). Since the function is continuous, the probability that the random variable Xl takes a particular value is zero. Instead, we should compute the conditional probability that Xi lies within some interval, ri and rt t e , where e is a small constant:

f r t l e

P ( * o < X ; I r i * e l y : y r 1 : I f q o ; t t i j , o i j ) d x i J : r '

= f ( r t ; t t t i , o , i i ) x e . ( 5 . 1 7 )

Since e appears as a constant multiplicative factor for each class, it cancels out when we normalize the posterior probability for P(flX). Therefore, we can still apply Equation 5.16 to approximate the class- conditional probability P (X,lY).

Example of the Naive Bayes Classifier

Consider the data set shown in Figure 5.10(a). We can compute the class- conditional probability for each categorical attribute, along with the sample mean and variance for the continuous attribute using the methodology de- scribed in the previous subsections. These probabilities are summarized in F i g u r e 5 . 1 0 ( b ) .

To predict the class label of a test record ;q : (Home Owner:No, Marital Status : Married, Income : $120K), we need to compute the posterior prob- abilities P(UolX) and P(YeslX). Recall from our earlier discussion that these posterior probabilities can be estimated by computing the product between the prior probability P(Y) and the class-conditional probabilitiesll P(X,lY), which corresponds to the numerator of the right-hand side term in Equation 5 . 1 5 .

The prior probabilities of each class can be estimated by calculating the fraction of training records that belong to each class. Since there are three records that belong to the class Yes and seven records that belong to the class

5.3 Bayesian Classifiers 235

P(Home Owner=YeslNo) = 317 P(Home Owner=NolNo) = 4fl P(Home Owner=YeslYes) = 0 P(Home Owner=NolYes) = 1 P(Marital Status=SinglelNo) = 2n P(Marital Status=Divorcedl No) = 1 /7 P(Marital Status=MarriedlNo) = 4t7 P(Marital Status=SinglelYes) = 2/3 P(Marital Status=DivorcedlYes) = 1 /3 P(Marital Status=MarriedlYes) = 0

For Annual Income: lf class=No: sample mean=1 10

sample variance=2975 lf class=Yes: sample medn=90

sample variance=2S

(a) (b)

Figure 5.10. The nalve Bayes classifier for the loan classification problem.

No, P(Yes) :0.3 and P(no) :0.7. Using the information provided in Figure 5.10(b), the class-conditional probabilities can be computed as follows:

P(Xluo) : P(Hone 0wner : NolNo) x P(status : MarriedlNo)

x P(Annual fncome : $120KlNo)

: 4 1 7 x 4 1 7 x 0 . 0 0 7 2 : 0 . 0 0 2 4 .

P(XlYes) : P(Home 0wner : IrtolYes) x P(Status : MarriedlYes)

x P(AnnuaI Income : $120KlYes)

: 1 x 0 x 1 . 2 x 1 0 - e : 0 .

Putting them together, the posterior probability for class No is P(NolX) :

a x 7 l l 0 x 0 . 0 0 2 4 : 0 . 0 0 1 6 a , w h e r e a : l l P ( X ) i s a c o n s t a n t t e r m . U s i n g a similar approach, we can show that the posterior probability for class Yes is zero because its class-conditional probability is zero. Since P(NolX) > P(YeslX), the record is classified as No.

Yes N o N o Yes No N o Yes N o N o N o

125K 1 0 0 K 70K 120K 95K 60K 220K 85K 75K 90K

236 Chapter 5 Classification: Alternative Techniques

M-estimate of Conditional Probability

The preceding example illustrates a potential problem with estimating poste- rior probabilities from training data. If the class-conditional probability for one of the attributes is zero, then the overall posterior probability for the class vanishes. This approach of estimating class-conditional probabilities using simple fractions may seem too brittle, especially when there are few training examples available and the number of attributes is large.

In a more extreme case, if the training examples do not cover many of the attribute values, we may not be able to classify some of the test records. For example, if P(Marital Status : DivorcedlNo) is zero instead of If7, then a record with attribute set 1(: (Home Owner - yes, Marital Status : Divorced, Income : $120K) has the following class-conditional probabilities:

P ( X l n o ) : 3 / 7 x 0 x 0 . 0 0 7 2 : 0 .

P ( X l v e s ) : 0 x 7 1 3 x 7 . 2 x 1 0 - e : 0 .

The naive Bayes classifier will not be able to classify the record. This prob- lem can be addressed by using the m-estimate approach for estimating the conditional probabilities :

P ( r , l a ) : ? s ! ! !

,- n + T n ( 5 . 1 8 )

where n is the total number of instances from class 3ry, n" is the number of training examples from class gi that take on the value ri, rrl is a parameter known as the equivalent sample size, and p is a user-specified parameter. If there is no training set available (i.e., n:0), then P(rilyi) : p. Therefore p can be regarded as the prior probability of observing the attribute value ri among records with class 97. The equivalent sample size determines the tradeoff between the prior probability p and the observed probability n.f n.

In the example given in the previous section, the conditional probability P(Status : MarriedlYes) : 0 because none of the training records for the class has the particular attribute value. Using the m-estimate approach with m:3 and p :113, the conditional probability is no longer zero:

P ( M a r i t a l S t a t u s : M a r r i e d l Y e s ) : ( 0 + 3 x t l S ) / ( J + 3 ) : 1 7 6 .

5.3 Bayesian Classifiers 237

If we assume p : If 3 for all attributes of class Yes and p : 213 for all attributes of class No. then

P(Xluo) : P(Home Owner : NolNo) x P(status : MarriedlNo)

x P(Annual Incone : $120KlNo)

: 6lto x 6/10 x o.oo72 : o.oo26.

P(XlYes) : P(Home 0tmer : ttolYes) x P(status : MarriedlYes)

x P ( A n n u a I I n c o m e : $ 1 2 0 K l Y e s )

: 4 / 6 x 1 1 6 x 7 . 2 x 1 0 - e : 1 . 3 x 1 0 - 1 0 .

The posterior probability for class No is P(llolx) : (t x 7110 x 0.0026 :

0.0018o, while the posterior probability for class Yes is P(YeslX) : o x 3/10 x 1.3 x 10-10 : 4.0 x 10-11a. Atthough the classification decision has not changed, the m-estimate approach generally provides a more robust way for estimating probabilities when the number of training examples is small.

Characteristics of Naive Bayes Classifiers

NaiVe Bayes classifiers generally have the following characteristics:

o They are robust to isolated noise points because such points are averaged out when estimating conditional probabilities from data. Naive Bayes classifiers can also handle missing values by ignoring the example during model building and classification.

o They are robust to irrelevant attributes. If Xi is an irrelevant at- tribute, then P(XrlY) becomes almost uniformly distributed. The class- conditional probability for Xi has no impact on the overall computation of the posterior probability.

o Correlated attributes can degrade the performance of naive Bayes clas- sifiers because the conditional independence assumption no longer holds for such attributes. For example, consider the following probabilities:

P ( A : 0 l Y : 0 ) : 0 . 4 , P ( A : 1 l Y : 0 ) : 0 . 6 ,

P ( A : 0 l Y : 1 ) : 0 . 6 , P ( A : L I Y : 1 ) : 0 . 4 ,

where A is a binary attribute and Y is a binary class variable. Suppose there is another binary attribute B that is perfectly correlated with A

238 Chapter 5 Classification: Alternative Techniques

when Y : 0, but is independent of -4 when Y : I. For simplicity, assume that the class-conditional probabilities for B are the same as for A . G i v e n a r e c o r d w i t h a t t r i b u t e s , 4 : 0 . 8 : 0 . w e c a n c o m o u t e i t s posterior probabilities as follows:

P ( Y : 0 l A : 0 , B : 0 ) : P ( A : O l y : 0 ) P ( B : O l y : O ) P ( Y : 0 )

P ( A : 0 , B : 0 ) 0 . 1 6 x P ( Y : 0 )

P ( A : 0 , B : 0 ) '

P ( A : O l y : I ) P ( B : O l y : l ) P ( Y : 1 ) P ( Y : I l A : 0 , 8 : 0 ) :

P ( A : 0 , B : 0 ) 0 . 3 6 x P ( Y : 1 )

P ( A : 0 , B : 0 ) '

If P(Y - 0) : P(Y : 1), then the naiVe Bayes classifier would assign the record to class 1. However, the truth is,

P ( A : 0 , B : O l Y : 0 ) : P ( A : 0 l ) ' : 0 ) : 0 . 4 ,

because A and B are perfectly correlated when Y : 0. As a result, the posterior probability for Y : 0 is

P ( Y : 0 l A : 0 , B : 0 ) : P ( A : 0 , 8 : O l Y : 0 ) P ( Y : 0 )

P ( A : 0 , 8 : 0 ) 0 . 4 x P ( Y : 0 )

P ( A : 0 , 8 : 0 ) '

which is larger than that for Y : 1. The record should have been classified as class 0.

5.3.4 Bayes Error Rate

Suppose we know the true probability distribution that governs P(Xlf). The Bayesian classification method allows us to determine the ideal decision bound- ary for the classification task, as illustrated in the following example.

Example 5.3. Consider the task of identifying alligators and crocodiles based on their respective lengths. The average length of an adult crocodile is about 15 feet, while the average length of an adult alligator is about 12 feet. Assuming

5.3 Bayesian Classifiers 239

5 10 Length, *

tu

Figure 5.11. Comparing the likelihood functions of a crocodile and an alligator.

that their length z follows a Gaussian distribution with a standard deviation equal to 2 feet, we can express their class-conditional probabilities as follows:

P(Xlcrocodile) : #"""0 1

"(ry)') P(Xlnrri.gator) :

#""*o[ ;(ry)'l

( 5 . 1 e )

(5.20)

Figure 5.11 shows a comparison between the class-conditional probabilities

for a crocodile and an alligator. Assuming that their prior probabilities are the same, the ideal decision boundary is located at some length i such that

P ( X : i l C r o c o d i l e ) : P ( X : f l A l l i g a t o r ) .

Using Equations 5.19 and 5.20, we obtain

( f t - r b \ 2 / i - r 2 \ 2

\ , / : \

, / '

which can be solved to yield f : 13.5. The decision boundary for this example is located halfway between the two means. r

\, Crocodile \ \ \ \ \ \ \ \ \ \ \ \ \

24O Chapter 5 Classification: Alternative Techniques

(b) (c)

Figure 5.12. Representing probabilistic relationships using directed acyclic graphs.

When the prior probabilities are different, the decision boundary shifts toward the class with lower prior probability (see Exercise 10 on page 319). Furthermore, the minimum error rate attainable by any classifier on the given data can also be computed. The ideal decision boundary in the preceding example classifies all creatures whose lengths are less than ft as alligators and those whose lengths are greater than 0 as crocodiles. The error rate of the classifier is given by the sum of the area under the posterior probability curve for crocodiles (from length 0 to i) and the area under the posterior probability curve for alligators (from f to oo):

Error : fo"

et 'o.odirelX)d , * Ir*

P(Alri-gat orlx)d,X.

The total error rate is known as the Bayes error rate.

5.3.5 Bayesian Belief Networks

The conditional independence assumption made by naive Bayes classifiers may seem too rigid, especially for classification problems in which the attributes are somewhat correlated. This section presents a more flexible approach for modeling the class-conditional probabilities P(Xlf). Instead of requiring all the attributes to be conditionally independent given the class, this approach allows us to specify which pair of attributes are conditionally independent. We begin with a discussion on how to represent and build such a probabilistic model, followed by an example of how to make inferences from the model.

(a)

5.3 Bayesian Classifiers 24L

Model Representation

A Bayesian belief network (BBN), or simply, Bayesian network, provides a graphical representation of the probabilistic relationships among a set of ran- dom variables. There are two key elements of a Bayesian network:

1. A directed acyclic graph (dag) encoding the dependence relationships among a set of variables.

2. A probability table associating each node to its immediate parent nodes.

Consider three random variables, -4, B, and C, in which A and B are independent variables and each has a direct influence on a third variable, C. The relationships among the variables can be summarized into the directed

acyclic graph shown in Figure 5.12(a). Each node in the graph represents a variable, and each arc asserts the dependence relationship between the pair

of variables. If there is a directed arc from X to Y, then X is the parent of Y and Y is the child of X. F\rrthermore, if there is a directed path in the network from X to Z, then X is an ancestor of Z, whlle Z is a descendant

of X. For example, in the diagram shown in Figure 5.12(b), A is a descendant of D and D is an ancestor of B. Both B and D arc also non-descendants of A. An important property of the Bayesian network can be stated as follows:

Property 1 (Conditional Independence). A node in a Bayesi'an network

i,s condi,tionally i,ndependent of its non-descendants, i'f i,ts parents are lcnown.

In the diagram shown in Figure 5.12(b), A is conditionally independent of both B and D given C because the nodes for B and D are non-descendants of node A. The conditional independence assumption made by a naive Bayes classifier can also be represented using a Bayesian network, as shown in Figure

5 . 1 2 ( c ) , w h e r e g r i s t h e t a r g e t c l a s s a n d { X t , X z , . . . , X a } i s t h e a t t r i b u t e s e t . Besides the conditional independence conditions imposed by the network

topology, each node is also associated with a probability table.

1. If a node X does not have any parents, then the table contains only the prior probability P(X).

2. If a node X has only one parent, Y, then the table contains the condi- tional probability P(XIY).

3. If a node X has multiple parents, {Yt,Yz, . . . ,Yn}, then the table contains t h e c o n d i t i o n a l p r o b a b i l i t y P ( X l Y , Y z , . . . , Y r . ) .

242 Chapter 5 Classification: Alternative Techniques

Hb=Yes D=Healthy 0.2 D=Unhealthy 0 . 8 5

CP=Yes HD=Yes Hb=Yes

0.8

HD=Yes Hh=Nn 0.6

H D = N o Hb=Yes

0.4

H D = N o Hb=No 0 . 1

Figure 5.13. A Bayesian belief network for detecting heart disease and heartburn in patients.

Figure 5.13 shows an example of a Bayesian network for modeling patients with heart disease or heartburn problems. Each variable in the diagram is assumed to be binary-valued. The parent nodes for heart disease (HD) cor- respond to risk factors that may affect the disease, such as exercise (E) and diet (D). The child nodes for heart disease correspond to symptoms of the disease, such as chest pain (CP) and high blood pressure (BP). For example, the diagram shows that heartburn (Hb) may result from an unhealthy diet and may lead to chest pain.

The nodes associated with the risk factors contain only the prior proba- bilities, whereas the nodes for heart disease, heartburn, and their correspond- ing symptoms contain the conditional probabilities. To save space, some of the probabilities have been omitted from the diagram. The omitted prob- abilities can be recovered by noting that P(X - *) : 1 - P(X : r) and P(X : TIY) : 1 - P(X : rlY), where 7 denotes the opposite outcome of r. For example, the conditional probability

P ( H e a r t D i s e a s e : N o l E x e r c i s e : N o , D i e t : H e a l t h y )

1 - P ( H e a r t D i s e a s e : Y e s l E x e r c i s e : N o , D i e t : H e a l t h y )

1 - 0 . 5 5 : 0 . 4 5 .

HD=Yes E=Yes D=Heatthy 0.25

E=Yes D=Unhealthl0.45

E=No D=Healthy 0 . 5 5

E=No D=Unhealth!

0 . 7 5

BayesianClassifiers 243

Model Building

Model building in Bayesian networks involves two steps: (1) creating the struc- ture of the network, and (2) estimating the probability values in the tables associated with each node. The network topology can be obtained by encod- ing the subjective knowledge of domain experts. Algorithm 5.3 presents a systematic procedure for inducing the topology of a Bayesian network.

Algorithm 5.3 Algorithm for generating the topology of a Bayesian network.

1: Let T: (XuXz,...,X4) denote a total order of the variables. 2 : f o r j : T t o d d o 3: Let X76 denote the jth highest order variable in ?. 4: Let r(X711) : {Xz(r), Xrp1, . . . , Xr1-t)} denote the set of variables preced-

ing X713y. 5: Remove the variables from r(Xrti>) that do not affect X1 (using prior knowl-

edge). 6: Create an arc between X71y; and the remaining variables in r(X711). 7: end for

Example 5.4. Consider the variables shown in Figure 5.13. After performing Step 1, Iet us assume that the variables are ordered in the following way: (E,D,HD,H\,CP,BP). From Steps 2 to 7, starting with variable D, we obtain the following conditional probabilities:

. P(DIE) is simplified to P(D).

o P(HDlE, D) cannot be simplified.

o P ( H b l H D , E , D ) i s s i m p l i f i e d t o P ( H b l D ) .

o P(C PlHb, H D, E, D) is simplified to P(C PlHb, H D).

o P ( B P I C P , H b , H D , E , D ) i s s i m p l i f i e d t o P ( B P I H D ) .

Based on these conditional probabilities) we can create arcs between the nodes (8, HD), (D, HD), (D, Hb), (HD, CP), (Hb, CP), and (HD, BP). These arcs result in the network structure shown in Figure 5.13. r

Algorithm 5.3 guarantees a topology that does not contain any cycles. The proof for this is quite straightforward. If a cycle exists, then there must be at least one arc connecting the lower-ordered nodes to the higher-ordered nodes, and at least another arc connecting the higher-ordered nodes to the lower- ordered nodes. Since Algorithm 5.3 prevents any arc from connecting the

5 . 3

244 Chapter 5 Classification: Alternative Techniques

lower-ordered nodes to the higher-ordered nodes, there cannot be any cycles in the topology.

Nevertheless, the network topology may change if we apply a different or- dering scheme to the variables. Some topology may be inferior because it produces many arcs connecting between different pairs of nodes. In principle, we may have to examine all d! possible orderings to determine the most appro- priate topology, a task that can be computationally expensive. An alternative approach is to divide the variables into causal and effect variables, and then draw the arcs from each causal variable to its corresponding effect variables. This approach eases the task of building the Bayesian network structure.

Once the right topology has been found, the probability table associated with each node is determined. Estimating such probabilities is fairly straight- forward and is similar to the approach used by naiVe Bayes classifiers.

Example of Inferencing Using BBN

Suppose we are interested in using the BBN shown in Figure 5.13 to diagnose whether a person has heart disease. The following cases illustrate how the diagnosis can be made under different scenarios.

Case 1: No Prior Information

Without any prior information, we can determine whether the person is likely to have heart disease by computing the prior probabilities P(HD : Yes) and P(HD: No). To simplify the notation, let a € {Yes,No} denote the binary values of Exercise and B e {Healthy,Unhealthy} denote the binary values o f D i e t .

P ( H D : v e s ) : t t P ( H n : Y e s l E : ( t , D : P ) P ( E : a , D : 0 )

d 1 3

: t t p ( u o : y e s l , O : ( t , D : i l P ( E : a ) P ( D : g )

a R

0 . 2 5 x 0 . 7 x 0 . 2 5 + 0 . 4 5 x 0 . 7 x 0 . 7 5 + 0 . 5 5 x 0 . 3 x 0 . 2 5

+ 0 . 7 5 x 0 . 3 x 0 . 7 5

0 . 4 9 .

Since P(HD - no) - 1 - P(ttO : yes) : 0.51, the person has a slightly higher chance of not getting the disease.

5.3 Bayesian Classifiers 245

Case 2: High Blood Pressure

If the person has high blood pressure) we can make a diagnosis about heart disease by comparing the posterior probabilities, P(HD : YeslBP : High) against P(ttO : Nolnt : High). To do this, we must compute P(Be : High):

P ( n e : H i g h ) : f r l n r : H i g h l H D : 7 ) p ( H D : 7 )

: 0 . 8 5 x 0 . 4 9 + 0 . 2 x 0 . 5 1 : 0 . 5 1 8 5 .

where 7 € {Yes, No}. Therefore, the posterior probability the person has heart disease is

P(Ho : yeslBp : High) : P(ge : HighlHD : Yes)P(HD : Yes)

P(ne : High) 0 ' 8 5 x 0 . 4 9: f f i : o ' 8 0 3 3 '

Similarly, P(ttO : NolBe : High) - 1 - 0.8033 : 0.1967. Therefore, when a person has high blood pressure, it increases the risk of heart disease.

Case 3: High Blood Pressure, Healthy Diet, and Regular Exercise

Suppose we are told that the person exercises regularly and eats a healthy diet. How does the new information affect our diagnosis? With the new information, the posterior probability that the person has heart disease is

P ( H o :

P(BP

P ( n e : H i g h l D : H e a I t h V , E : Y e s )

x P ( H D : Y e s l D : H e a l t h y , E : Y e s )

P ( e e : H i g h l H D : Y e s ) P ( H D : Y e s l D : H e a l t h y , E : Y e s )

D , f l B e : H i g h l H D : ? ) P ( H D : . y l D : H e a l t h y , E : Y e s )

0 . 8 5 x 0 . 2 5 0 . 8 5 x 0 . 2 5 + 0 . 2 x 0 . 7 5

: 0 . 5 8 6 2 ,

while the probability that the person does not have heart disease is

P ( t t 0 : N o l e e : H i g h , D : H e a l t h y , E : y e s ) - 1 - 0 . 5 8 6 2 : 0 . 4 1 3 8 .

Y e s l B P : H l g h , D : H e a l - t h y , E : Y e s )

: H i g h l H D : Y e s , D : H e a l t h y , E : Y e s)l

246 Chapter 5 Classification: Alternative Techniques

The model therefore suggests that eating healthily and exercising regularly may reduce a person's risk of getting heart disease.

Characteristics of BBN

Following are some of the general characteristics of the BBN method:

1. BBN provides an approach for capturing the prior knowledge of a par- ticular domain using a graphical model. The network can also be used to encode causal dependencies among variables.

Constructing the network can be time consuming and requires a large amount of effort. However, once the structure of the network has been determined, adding a new variable is quite straightforward.

Bayesian networks are well suited to dealing with incomplete data. In- stances with missing attributes can be handled by summing or integrat- ing the probabilities over all possible values of the attribute.

Because the data is combined probabilistically with prior knowledge, the method is quite robust to model overfitting.

5.4 Artificial Neural Network (ANN)

The study of artificial neural networks (ANN) was inspired by attempts to simulate biological neural systems. The human brain consists primarily of nerve cells called neurons) linked together with other neurons via strands of fiber called axons. Axons are used to transmit nerve impulses from one neuron to another whenever the neurons are stimulated. A neuron is connected to the axons of other neurons via dendrites, which are extensions from the cell body of the neuron. The contact point between a dendrite and an axon is called a synapse. Neurologists have discovered that the human brain learns by changing the strength of the synaptic connection between neurons upon repeated stimulation by the same impulse.

Analogous to human brain structure, an ANN is composed of an inter- connected assembly of nodes and directed links. In this section, we will exam- ine a family of ANN models, starting with the simplest model called percep- tron, and show how the models can be trained to solve classification problems.

2 .

3 .

4 .

5.4 Artificial Neural Network (ANN) Z4T

5.4.1 Perceptron

Consider the diagram shown in Figure 5.14. The table on the left shows a data set containing three boolean variables (*t, ,r, 13) and an output variable, gr, that takes on the value -1 if at least two of the three inputs arezero, and +1 if at least two of the inputs are greater than zero.

Input nodes

(a) Data set. (b) perceptron.

Figure 5.14. Modeling a boolean function using a perceptron.

Figure 5.14(b) illustrates a simple neural network architecture known as a perceptron. The perceptron consists of two types of nodes: input nodes, which are used to represent the input attributes, and an output node, which is used to represent the model output. The nodes in a neural network architecture are commonly known as neurons or units. In a perceptron, each input node is connected via a weighted link to the output node. The weighted link is used to emulate the strength of synaptic connection between neurons. As in biological neural systems, training a perceptron model amounts to adapting the weights of the links until they fit the input-output relationships of the underlying data.

A perceptron computes its output value, Q, by performing a weighted sum on its inputs, subtracting a bias factor t from the sum, and then examining the sign of the result. The model shown in Figure 5.14(b) has three input nodes, each of which has an identical weight of 0.3 to the output node and a bias factor of f : 0.4. The output computed by the model is

X1 X2 x3 v 1 1 1 1 0 0 0 0

0 0 1 1 0 1 1 0

0 1 0 1 "l

0 1 0

- 1

1 1 1

-1 -1

1 -1

(a) Data set.

" I t , i f 0 . 3 1 1 * o . J r 2 f 0 . 3 1 3 - 0 . 4 > o ;

t i - "

[ - 1 , i f o . J q * 0 . J r 2 * 0 . 3 1 3 - 0 . 4 < 0 .

( 5 . 2 1 )

248 Chapter 5 Classification: Alternative Techniques

For example, if 11 : \, fi2 : L, rz : 0, then 0 : +1 because 0.32r * 0.312 + 0 . 3 1 3 - 0 . 4 i s p o s i t i v e . O n t h e o t h e r h a n d , i f r y : 0 , f r 2 : L , f r B : 0 , t h e n

A : -t because the weighted sum subtracted by the bias factor is negative. Note the difference between the input and output nodes of a perceptron.

An input node simply transmits the value it receives to the outgoing link with- out performing any transformation. The output node, on the other hand, is a mathematical device that computes the weighted sum of its inputs, subtracts the bias term, and then produces an output that depends on the sign of the resulting sum. More specifically, the output of a perceptron model can be expressed mathematicallv as follows:

0 : s i ' g n ( w d r d + w d - t . x d - t + . . . + w z r z I w p 1 - t ) , (5.22)

w h e r e u ) r t l r 2 t . . . , 1 1 ) d . a r e t h e w e i g h t s o f t h e i n p u t l i n k s a n d 1 1 , r 2 t . . . ) r d a r e the input attribute values. The sign function, which acts as an activation function for the output neuron, outputs a value *1 if its argument is positive

and -1 if its argument is negative. The perceptron model can be written in a more compact form as follows:

0 : s i ' g n l w a r a l w d , - L r d - r + . . . + w y r l l u t o r o f : s i g n ( w ' x ) , ( 5 . 2 3 )

where u)0: -t, frl: I, and w'x is the dot product between the weight vector w and the input attribute vector x.

Learning Perceptron Model

During the training phase of a perceptron model, the weight parameters w are adjusted until the outputs of the perceptron become consistent with the true outputs of training examples. A summary of the perceptron learning algorithm is given in Algorithm 5.4.

The key computation for this algorithm is the weight update formula given in Step 7 of the algorithm:

.\k+r) :.f, + \(ur - g[r))"ni, (5.24)

where tr(k) is the weight parameter associated with the i'h input link after the kth ileration, ) is a parameter known as the learning rate, and rii is the value of lhe jth attribute of the training example 4. The justification for the weight update formula is rather intuitive. Equation 5.24 shows that the new weight ?r(k+l) is a combination of the old weight tr(ft) and a term proportional

5 . 4 Artificial Neural Network (ANN) 249

Algorithm 5.4 Perceptron learning algorithm. t: Let D: {(xr, yt) | i : I,2,. . .,1[] be the set of training examples. 2: Initialize the weight vector with random values, w(0) 3: repeat 4: for each training example (xt,llt) € D do

5: Compute the predicted output yjk) 6: for each weight wi do

7: Update the weight, .\k+r) :.:*) + )(g, - g[r))*0,. 8: end for 9: end for

10: until stopping condition is met

to the prediction error, (g - i). If the prediction is correct, then the weight remains unchanged. Otherwise, it is modified in the following ways:

o If y : +1 and f - -1, then the prediction error is (a - 0) : 2. To

compensate for the error, we need to increase the value of the predicted output by increasing the weights of all links with positive inputs and decreasing the weights of all links with negative inputs.

o I f a t : - 1 a n d 0 : l I , t h e n ( g - 0 ) : - 2 . T o c o m p e n s a t e f o r t h e e r r o r , we need to decrease the value of the predicted output by decreasing the weights of all links with positive inputs and increasing the weights of all Iinks with negative inputs.

In the weight update formula, Iinks that contribute the most to the error term are the ones that require the largest adjustment. However, the weights should not be changed too drastically because the error term is computed only for the current training example. Otherwise, the adjustments made in earlier iterations will be undone. The learning rate ), a parameter whose value is between 0 and 1, can be used to control the amount of adjustments made in each iteration. If ,\ is close to 0, then the new weight is mostly influenced by the value of the old weight. On the other hand, if ) is close to 1, then the new weight is sensitive to the amount of adjustment performed in the current iteration. In some cases) an adaptive ) value can be used; initially, ,\ is moderately large during the first few iterations and then gradually decreases in subsequent iterations.

The perceptron model shown in Equation 5.23 is linear in its parameters w and attributes x. Because of this, the decision boundary of a perceptron, which is obtained by setting 0 : 0, is a linear hyperplane that separates the data into two classes, -1 and *1. Figure 5.15 shows the decision boundary

25O Chapter 5 Classification: Alternative Techniques

Figure 5.15. Perceptron decision boundary for the data given in Figure 5.14,

obtained by applying the perceptron learning algorithm to the data set given in Figure 5.14. The perceptron learning algorithm is guaranteed to converge to an optimal solution (as long as the learning rate is sufficiently small) for linearly separable classification problems. If the problem is not linearly separable, the algorithm fails to converge. Figure 5.16 shows an example of nonlinearly separable data given by the XOR function. Perceptron cannot find the right solution for this data because there is no linear hyperplane that can perfectly separate the training instances.

0 . 5 x1

1 . 5

1

Xe 0.5

0

- 0 . 5 L - 0 . 5 0.5

x1

Figure 5.16. XOR classification problem. No linear hyperplane can separate the two classes.

1 . 5

5.4 Artificial Neural Network (ANN) 251

5.4.2 Multilayer Artificial Neural Network

An artificial neural network has a more complex structure than that of a perceptron model. The additional complexities may arise in a number of ways:

1. The network may contain several intermediary layers between its input and output layers. Such intermediary layers are called hidden layers and the nodes embedded in these layers are called hidden nodes. The resulting structure is known as a multilayer neural network (see Fig- ure 5.17). In a feed-forward neural network, the nodes in one layer

Figure 5.17. Example of a multilayer feed{onrvard artificial neural network (ANN).

are connected only to the nodes in the next layer. The perceptron is a single-layer, feed-forward neural network because it has only one layer of nodes-the output layer that performs complex mathematical op- erations. fn a recurrent neural network, the links may connect nodes within the same layer or nodes from one layer to the previous layers.

2. The network may use types of activation functions other than the sign function. Examples of other activation functions include linear, sigmoid (logistic), and hyperbolic tangent functions, as shown in Figure 5.18. These activation functions allow the hidden and output nodes to produce output values that are nonlinear in their input parameters.

These additional complexities allow multilayer neural networks to model more complex relationships between the input and output variables. For ex-

252 Chapter 5 Classification: Alternative Techniques

Linear function Sigmoid tun

- 0 . 5 0 0 . s 1 - 1 - 0 5 0 0 . 5

Figure 5,18. Types of activation functions in artificial neural networks.

ample, consider the XOR problem described in the previous section. The in- stances can be classified using two hyperplanes that partition the input space into their respective classes, as shown in Figure 5.19(a). Because a percep-

tron can create only one hyperplane, it cannot find the optimal solution. This problem can be addressed using a two-layer, feed-forward neural network, as shown in Figure 5.19(b). Intuitively, we can think of each hidden node as a perceptron that tries to construct one of the two hyperplanes, while the out- put node simply combines the results of the perceptrons to yield the decision boundary shown in Figure 5.19(a).

To learn the weights of an ANN model, we need an efficient algorithm that converges to the right solution when a sufficient amount of training data is provided. One approach is to treat each hidden node or output node in the network as an independent perceptron unit and to apply the same weight update formula as Equation 5.24. Obviously, this approach will not work because we lack a priori, knowledge about the true outputs of the hidden nodes. This makes it difficult to determine the error term, (gr - f), associated

Sigmoid tunction

Sign function

- 1 . 5

5 . 4 Artificial Neural Network (ANN) 253

1 . 5

1

X z 0 5

0 - 0 . 5 0 . 5

x1

(a) Decision boundary. (b) Neural network topology.

Figure 5.19. A two-layer, feed{orward neural network for the XOR problem.

with each hidden node. A methodology for learning the weights of a neural network based on the gradient descent approach is presented next.

Learning the ANN Model

The goal of the ANN learning algorithm is to determine a set of weights w that minimize the total sum of souared errors:

1 'A{

E(w) : il,rr,-o)2. ; - l

( 5 . 2 5 )

Note that the sum of squared errors depends on w because the predicted class

f is a function of the weights assigned to the hidden and output nodes. Figure 5.20 shows an example of the error surface as a function of its two parameters, to1 and 'u2. This type of error surface is typically encountered when Qi is a linear function of its parameters, w. If we replace Q : w 'x into Equation 5.25, then the error function becomes quadratic in its parameters and a global minimum solution can be easily found.

In most cases) the output of an ANN is a nonlinear function of its param- eters because of the choice of its activation functions (e.g., sigmoid or tanh function). As a result, it is no longer straightforward to derive a solution for w that is guaranteed to be globally optimal. Greedy algorithms such as those based on the gradient descent method have been developed to efficiently solve the optimization problem. The weight update formula used by the gradient

254 Chapter 5 Classification: Alternative Techniques

E ( w 1 , w 2 )

1 . 8

w 1 0 ' l

Figure 5,20. Enor sudace E(w1,w2) for a two-parameter model.

descent method can be written as follows:

1 . 6

1 . 4

1 . 2

1 I

( 5 . 2 6 )

where ) is the Iearning rate. The second term states that the weight should be increased in a direction that reduces the overall error term. However, because the error function is nonlinear, it is possible that the gradient descent method may get trapped in a local minimum.

The gradient descent method can be used to learn the weights of the out- put and hidden nodes ofa neural network. For hidden nodes, the computation is not trivial because it is difficult to assess their error Lerm,0Ef 0t17, without knowing what their output values should be. A technique known as back- propagation has been developed to address this problem. There are two phases in each iteration of the algorithm: the forward phase and the backward phase. During the forward phase, the weights obtained from the previous iter- ation are used to compute the output value of each neuron in the network. The computation progresses in the forward direction; i.e., outputs of the neurons at level k are computed prior to computing the outputs at level /c + 1. Dur- ing the backward phase, the weight update formula is applied in the reverse direction. In other words, the weights at level k + 7 arc updated before the weights at level k are updated. This back-propagation approach allows us to use the errors for neurons at layer k + t to estimate the errors for neurons at Iaver k.

. O E ( w ) W ; + ' l I ; - A - - - ^r r

dw.i

5 . 4

Design Issues in ANN Learning

Before we train a neural network to design issues must be considered.

Artificial Neural Network (ANN) 255

Iearn a classification task, the following

1. The number of nodes in the input layer should be determined. Assign an input node to each numerical or binary input variable. If the input vari- able is categorical, we could either create one node for each categorical value or encode the k-ary variable using flog2 k-l input nodes.

2. The number of nodes in the output layer should be established. For a two-class problem, it is sufficient to use a single output node. For a k-class problem, there are k output nodes.

3. The network topology (e.g., the number of hidden layers and hidden nodes, and feed-forward or recurrent network architecture) must be se- lected. Note that the target function representation depends on the weights of the links, the number of hidden nodes and hidden layers, bi- ases in the nodes, and type of activation function. Finding the right topology is not an easy task. One way to do this is to start from a fully connected network with a sufficiently large number of nodes and hid- den layers, and then repeat the model-building procedure with a smaller number of nodes. This approach can be very time consuming. Alter- natively, instead of repeating the model-building procedure, we could remove some of the nodes and repeat the model evaluation procedure to select the right model complexity.

4. The weights and biases need to be initialized. Random assignments are usually acceptable.

5. Tlaining examples with missing values should be removed or replaced with most likely values.

5.4.3 Characteristics of ANN

Following is a summary of the general characteristics of an artificial neural network:

1. Multilayer neural networks with at least one hidden layer are univer- sal approximators; i.e., they can be used to approximate any target functions. Since an ANN has a very expressive hypothesis space, it is im- portant to choose the appropriate network topology for a given problem to avoid model overfitting.

256 Chapter 5 Classification: Alternative Techniques

ANN can handle redundant features because the weights are automat- ically learned during the training step. The weights for redundant fea- tures tend to be very small.

Neural networks are quite sensitive to the presence of noise in the train- ing data. One approach to handling noise is to use a validation set to determine the generalization error of the model. Another approach is to decrease the weight by some factor at each iteration.

The gradient descent method used for learning the weights of an ANN often converges to some local minimum. One way to escape from the local minimum is to add a momentum term to the weight update formula.

Training an ANN is a time consuming process, especially when the num- ber of hidden nodes is large. Nevertheless, test examples can be classified rapidly.

5.5 Support Vector Machine (SVM)

A classification technique that has received considerable attention is support vector machine (SVM). This technique has its roots in statistical learning the- ory and has shown promising empirical results in many practical applications, from handwritten digit recognition to text categorization. SVM also works very well with high-dimensional data and avoids the curse of dimensionality problem. Another unique aspect of this approach is that it represents the deci- sion boundary using a subset of the training examples, known as the support vectors.

To illustrate the basic idea behind SVM, we first introduce the concept of a maximal margin hyperplane and explain the rationale of choosing such a hyperplane. We then describe how a linear SVM can be trained to explicitly look for this type of hyperplane in linearly separable data. We conclude by showing how the SVM methodology can be extended to non-linearly separable data.

5.5.1 Maximum Margin Hyperplanes

Figure 5.21 shows a plot of a data set containing examples that belong to two different classes, represented as squares and circles. The data set is also linearly separable; i.e., we can find a hyperplane such that all the squares reside on one side of the hyperplane and all the circles reside on the other

2 .

. t .

4 .

tr i r .

D . D Support Vector Machine (SVM) 257

Figure 5.21. Possible decision boundaries for a linearly separable data set.

side. However, as shown in Figure 5.21, there are infinitely many such hyper- planes possible. Although their training errors are zerol there is no guarantee that the hyperplanes will perform equally well on previously unseen examples. The classifier must choose one of these hyperplanes to represent its decision boundary, based on how well they are expected to perform on test examples.

To get a clearer picture of how the different choices of hyperplanes affect the generalization errors, consider the two decision boundaries, ,B1 and 82, shown in Figure 5.22. Both decision boundaries can separate the training examples into their respective classes without committing any misclassification errors. Each decision boundary B; is associated with a pair of hyperplanes, denoted as bi1 and bi2, respectively. b61 is obtained by moving a parallel hyperplane away from the decision boundary until it touches the closest square(s), whereas b;2 is obtained by moving the hyperplane until it touches the closest circle(s). The distance between these two hyperplanes is known as the margin of the classifier. From the diagram shown in Figure 5.22,notice that the margin for .B1 is considerably larger than that for Bz. In this example, 81 turns out to be the maximum margin hyperplane of the training instances.

Rationale for Maximum Margin

Decision boundaries with large margins tend to have better generalization errors than those with small margins. Intuitively, if the margin is small, then

I I

I

T

I

T

T

I

T

T T

l r

T I

o o o

o o o

o o

o o o o

O O

o o o o

o o

258 Chapter 5 Classification: Alternative Technioues

brr margin for 81 bp

Figure 5.22. Margin of a decision boundary.

any slight perturbations to the decision boundary can have quite a significant impact on its classification. Classifiers that produce decision boundaries with small margins are therefore more susceptible to model overfitting and tend to generalize poorly on previously unseen examples.

A more formal explanation relating the margin of a linear classifier to its generalization error is given by a statistical learning principle known as struc- tural risk minimization (SRM). This principle provides an upper bound to the generalization error of a classifier (R) in terms of its training error (,R"), the number of training examples (l/), and the model complexity otherwise known as its capacity (h). More specifically, with a probability of 1 - q, the generalization error of the classifier can be at worst

( 5 . 2 7 )

where cp is a monotone increasing function of the capacity h. The preced- ing inequality may seem quite familiar to the readers because it resembles the equation given in Section 4.4.4 (on page 179) for the minimum descrip- tion length (MDL) principle. In this regard, SRM is another way to express generalization error as a tradeoff between training error and model complexity.

R 1 R . + r ( * ,

+ - - - - - - - - - - - - - - - - t brr margin for 81

o . o Support Vector Machine (SVM) 259

The capacity of a linear model is inversely related to its margin. Models with small margins have higher capacities because they are more flexible and can fit many training sets, unlike models with large margins. However, accord- ing to the SRM principle, as the capacity increases, the generalization error bound will also increase. Therefore, it is desirable to design linear classifiers that maximize the margins of their decision boundaries in order to ensure that their worst-case generalization errors are minimized. One such classifier is the linear SVM, which is explained in the next section.

5.5.2 Linear SVM: Separable Case

A linear SVM is a classifier that searches for a hyperplane with the largest margin, which is why it is often known as a maximal margin classifier. To understand how SVM learns such a boundary, we begin with some preliminary discussion about the decision boundary and margin of a linear classifier.

Linear Decision Boundary

Consider a binary classification problem consisting of l/ training examples. Each example is denoted by a tuple (*i,At) (i, : I,2,. . . ,ly'), where x1 : (*nt.,*t2,...,nu)r correspond.s to the attribute set for the i,th example. By convention, Iet At € {-1, 1} denote its class label. The decision boundary of a linear classifier can be written in the followins form:

w . x * b : 0 , ( 5 . 2 8 )

where w and b are parameters of the model. Figure 5.23 shows a two-dimensional training set consisting of squares and

circles. A decision boundary that bisects the training examples into their respective classes is illustrated with a solid line. Any example located along the decision boundary must satisfy Equation 5.28. For example, if xo and x6 are two points located on the decision boundary, then

\ n / . X o * b : 0 ,

w ' x b f b : 0 .

Subtracting the two equations will yield the following:

w . ( x a - x r ) : 0 ,

260 Chapter 5 Classification: Alternative Techniques

Figure 5.23. Decision boundary and margin of SVM.

where Xb - Xo is a vector parallel to the decision boundary and is directed from xo to x6. Since the dot product is zero, the direction for w must be perpendicular to the decision boundary, as shown in Figure 5.23.

For any square x" located above the decision boundary, we can show that

W ' X s l b : k , (5.2e)

where k > 0. Similarly, for any circle x. located below the decision boundary, we can show that

w . X c * b : k " ( 5 . 3 0 )

where k' < 0. If we label all the squares as class f 1 and all the circles as class -1, then we can predict the class label gr for any test example z in the following way:

( 5 . 3 1 )

Margin of a Linear Classifier

Consider the square and the circle that are closest to the decision boundary. Since the square is located above the decision boundary, it must satisfy Equa- tion 5.29 for some positive value k, whereas the circle must satisfy Equation

I l , i f w . z * b > 0 ; ' : t - t . i f w . z + b < 0 .

5.5 Support Vector Machine (SVM) 261

5.30 for some negative value kt. We can rescale the parameters w and b of the decision boundary so that the two parallel hyperplanes b;1 and bi2 can be expressed as follows:

b i 1 : w . x * b : 1 ,

b n ; w ' x * b - - 1 .

( 5 . 3 2 )

( 5 . 3 3 )

The margin of the decision boundary is given by the distance between these two hyperplanes. To compute the margin, Iet x1 be a data point located on br1 and x2 be a data point on bi2, as shown in Figure 5.23. Upon substituting these points into Equations 5.32 and 5.33, the margin d can be computed by subtracting the second equation from the first equation:

Learning a Linear SVM Model

The training phase of SVM involves estimating the parameters w and b of the decision boundary from the training data. The parameters must be chosen in such a way that the following two conditions are met:

w . x i + b > l i f y , ; : 1 ,

u / . x i + b < - 7 i f y i - - 1 . ( 5 . 3 5 )

These conditions impose the requirements that all training instances from class gr : 1 (i.e., the squares) must be located on or above the hyperplane w . x * b : L , w h i l e t h o s e i n s t a n c e s f r o m c l a s s U : - I ( i . e . , t h e c i r c l e s ) m u s t be located on or below the hyperplane w .x * b - -1. Both inequalities can be summarized in a more comoact form as follows:

g t ( w . x i * b ) ) 1 , ' i : 7 , 2 , . . . , N . ( 5 . 3 6 )

Although the preceding conditions are also applicable to any linear classi- fiers (including perceptrons), SVM imposes an additional requirement that the margin of its decision boundary must be maximal. Maximizing the margin, however, is equivalent to minimizing the following objective function:

w ' ( x r - x z ) : 2

l l w l l x d : 2 ,.

l l * l l ' ( 5 . 3 4 )

f r * r : l l w l l 2' 2 ( 5 . 3 7 )

262 Chapter 5 Classification: Alternative Techniques

Definition 5.1 (Linear SVM: Separable Case). The learning task in SVM can be formalized as the following constrained optimization problem:

Since the objective function is quadratic and the constraints are linear in the parameters w and b, this is known as a convex optimization problem, which can be solved using the standard Lagrange multiplier method. Fol- lowing is a brief sketch of the main ideas for solving the optimization problem. A more detailed discussion is given in Appendix E.

First, we must rewrite the objective function in a form that takes into account the constraints imposed on its solutions. The new objective function is known as the Lagrangian for the optimization problem:

, I l * l l ' mln -

subject to ,it* i * ,, - ,, ,i : r,2,. . . ,N.

r, : ill*ll'

l/ ^ \ - .- U + w :

) A i ? J t x i , ^Lt t - 1

1V ^ \ -- U + ) A ; t t ; : U .

,Lt

i,:1

- f ^ , ( r , t * ' x i * b ) - 1 ) , ( 5 . 3 8 )

(5.3e)

(5.40)

where the parameters ); are called the Lagrange multipliers. The first term in the Lagrangian is the same as the original objective function, while the second term captures the inequality constraints. To understand why the objective function must be modified, consider the original objective function given in Equation 5.37. It is easy to show that the function is minimized when w : 0, a null vector whose components are all zeros. Such a solution, however, violates the constraints given in Definition 5.1 because there is no feasible solution for b. The solutions for w and b are infeasible if they violate the inequality constraints; i.e., if Ailw-xi+b) - 1 < 0. The Lagrangian given in Equation 5.38 incorporates this constraint by subtracting the term from its original objective function. Assuming that .\; > 0, it is clear that any infeasible solution may only increase the value of the Lagrangian.

To minimize the Lagrangian, we must take the derivative of ,Lp with respect to w and b and set them to zero:

OL, 0w

oLo ab

5.5 Support Vector Machine (SVM) 263

Because the Lagrange multipliers are unknown, we still cannot solve for w and b. If Definition 5.1 contains only equality instead of inequality constraints, then we can use the ly' equations from equality constraints along with Equations 5.39 and 5.40 to find the feasible solutions for w, b, and ).;. Note that the Lagrange multipliers for equality constraints are free parameters that can take any values.

One way to handle the inequality constraints is to transform them into a set of equality constraints. This is possible as long as the Lagrange multipliers are restricted to be non-negative. Such transformation leads to the following constraints on the Lagrange multipliers, which are known as the Karush-Kuhn- T\rcker (KKT) conditions:

) ; ) 0 ,

\ u [ s n ( * ' x t * b ) - 1 ] : O .

( 5 . 4 1 )

( 5 . 4 2 )

At first glance, it may seem that there are as many Lagrange multipli- ers as there are training instances. It turns out that many of the Lagrange multipliers become zero after applying the constraint given in trquation 5.42. The constraint states that the Lagrange multiplier Aa must be zero unless the training instance x6 satisfies the equation At(w .xt I b) : t. Such training instance, with )i ) 0, lies along the hyperplanes bi1 or b.i2 and is known as a support vector. taining instances that do not reside along these hyperplanes have )6:0. Equations 5.39 and5.42 also suggest that the parameters w and b, which define the decision boundary, depend only on the support vectors.

Solving the preceding optimization problem is still quite a daunting task because it involves a large number of parameters: w, b, and )a. The problem can be simplified by transforming the Lagrangian into a function of the La- grange multipliers only (this is known as the dual problem). To do this, we first substitute Equations 5.39 and 5.40 into Equation 5.38. This will lead to the following dual formulation of the optimization problem:

,A|I

L n : 1 , ^ ' - * L s ; \ a t u i x i . x j .- z-/ 2 z-.J i:1. i,i

( 5 . 4 3 )

The key differences between the dual and primary Lagrangians are as fol- lows:

1. The dual Lagrangian involves only the Lagrange multipliers and the training data, while the primary Lagrangian involves the Lagrange mul- tipliers as well as parameters of the decision boundary. Nevertheless, the solutions for both optimization problems are equivalent.

264 Chapter 5 Classification: Alternative Techniques

2. The quadratic term in Equation 5.43 has a negative sign, which means that the original minimization problem involving the primary Lagrangian, Lp, has turned into a maximization problem involving the dual La- grangian, ,L2.

For large data sets, the dual optimization problem can be solved using numerical techniques such as quadratic programming, a topic that is beyond the scope of this book. Once the );'s are found, we can use Equations 5.39 and 5.42 to obtain the feasible solutions for w and b. The decision boundary can be expressed as follows:

/ N \ ( f \ y n * t ' * ) + b : 0 . \ - /

(5.44)

b is obtained by solving Equation 5.42 for the support vectors. Because the )1's are calculated numerically and can have numerical errors, the value computed for b may not be unique. Instead it depends on the support vector used in Equation 5.42. In practice, the average value for b is chosen to be the parameter of the decision boundary.

Example 5.5. Consider the two-dimensional data set shown in Figure 5.24, which contains eight training instances. Using quadratic programming, we can solve the optimization problem stated in Equation 5.43 to obtain the Lagrange multiplier .\6 for each training instance. The Lagrange multipliers are depicted in the last column of the table. Notice that only the fi.rst two instances have non-zero Lagrange multipliers. These instances correspond to the support vectors for this data set.

Let w : (1q,w2) and b denote the parameters of the decision boundary. Using Equation 5.39, we can solve for w1 and w2 in the following way:

w r : D \ n r o r o r : 6 5 . 5 6 2 L x 1 x 0 . 3 8 5 8 + 6 5 . 5 6 2 1 x - 1 x 0 ' 4 8 7 r : - 6 ' 6 4 .

z

\ - ,w 2 : ) \ ; A p n : 6 5 . 5 6 2 L x 1 x 0 . 4 6 8 7 + 6 5 . 5 6 2 1 x - 1 x 0 . 6 1 1 : - 9 ' 3 2 . 2

The bias term b can be computed using Equation 5.42 for each support vector:

6 ( t ) : 1 - w . x 1 : 1 - ( - 6 . 6 4 ) ( 0 . 3 8 5 3 ) - ( - 9 . 3 2 ) ( 0 . 4 6 8 7 ) : 7 . 9 3 0 0 .

6 ( z ) : - 1 - w . x 2 : - 1 - ( - 6 . 6 4 ) ( 0 . 4 8 7 1 ) - ( - 9 . 3 2 ) ( 0 . 6 1 1 ) : 7 . 9 2 8 9 .

Averaging these values, we obtain b : 7.93. The decision boundary corre- sponding to these parameters is shown in Figure 5.24. r

X1 X2 v LagrangeMultiplier 0.3858 0.4871 0 . 9 2 1 8 o.7382 0 . 1 7 6 3 o.4057 0.9355 0.2146

0.4687 0 . 6 1 1 0 . 4 1 0 3 0.8936 0.0579 0.3529 0 . 8 1 3 2 0.0099

1 -1 -1 - 1

1 1

-1 1

65.5261 65.5261

0 0 0 0 0 0

5.5 Support Vector Machine (SVM) 265

1

0.9

0 . 8

0.7

0 . 6

s 0.5

0 . 4

0 . 3

o.2

0 . 1

0

-6.64 x1 - 9.32 x2 + 7.93 = 0 tr

tr

tr

o o

n

o 0 0.2 o.4 0.6 0.8 1

X 1

Figure 5.24. Example of a linearly separable data set.

Once the parameters of the decision boundary are found, a test instance z is classified as follows:

/ N \

f ( r ) : s i e n ( w ' z + b ) : s ' i s n ( f ^ u r u " t z + b ) . \ d : l

It f (z): 1, then the test instance is classified as a positive class; otherwise, it is classified as a negative class.

266 Chapter 5 Classification: Alternative Techniques

5.5.3 Linear SVM: Nonseparable Case

Figure 5.25 shows a data set that is similar to Figure 5.22, except it has two new examples, P and Q. Although the decision boundary 81 misclassifies the new examples, while 82 classifies them correctly, this does not mean that .B2 is a better decision boundary than 81 because the new examples may correspond to noise in the training data. .B1 should still be preferred over 82 because it has a wider margin, and thus, is less susceptible to overfitting. However, the SVM formulation presented in the previous section constructs only decision boundaries that are mistake-free. This section examines how the formulation can be modified to learn a decision boundary that is tolerable to small training errors using a method known as the soft margin approach. More importantly, the method presented in this section allows SVM to construct a linear decision boundary even in situations where the classes are not linearly separable. To do this, the learning algorithm in SVM must consider the trade-off between the width of the margin and the number of training errors committed by the Iinear decision boundary.

f; ;;;;i" ro,, i,--'o',

Figure 5.25, Decision boundary of SVM for the nonseparable case.

margin tor e2'l

r r al'', r r l'\,,

l l f ' .

n

o \ , , o o

o o o

o o

! t .

['?

O O

O O

o o

o

5.5 Support Vector Machine (SVM) 267

N X

- 0 . 5 0 0 . 5 1 1 . 5 x 1

Figure 5,26. Slack variables for nonseparable data.

While the original objective function given in Equation 5.37 is still appli- cable, the decision boundary .B1 no longer satisfies all the constraints given in Equation 5.36. The inequality constraints must therefore be relaxed to ac- commodate the'nonlinearly separable data. This can be done by introducing positive-valued slack variables ({) into the constraints of the optimization problem, as shown in the following equations:

v / ' x i + b > 7 - € n

w . x i + b < - 1 + € , ( 5 . 4 5 )

where Vz : {6 > 0. To interpret the meaning of the slack variables {a, consider the diagram

shown in Figure 5.26. The circle P is one of the instances that violates the constraints given in Equation 5.35. Let w.x* b: -7 *{ denote a line that is parallel to the decision boundary and passes through the point P. It can be shown that the distance between this line and the hyperplane w'x * b: -L

is {/llwll. Thus, { provides an estimate of the error of the decision boundary on the training example P.

In principle, we can apply the same objective function as before and impose the conditions given in Equation 5.45 to find the decision boundary. However,

1 . 2

0.8

i f y 6 - I ,

i f y a : - 1 ,

w . x + b = 0

t r t r t r tr

t1 LI

tr

. w . x + b = - 1 + (

\

268 Chapter 5 Classification: Alternative Techniques

Figure 5.27. A decision boundary that has a wide margin but large training error.

since there are no constraints on the number of mistakes the decision boundary can make, the learning algorithm may find a decision boundary with a very wide margin but misclassifies many of the training examples, as shown in Figure 5.27. To avoid this problem, the objective function must be modified to penalize a decision boundary with large values of slack variables. The modified objective function is given by the following equation:

. l l w l l 2 N

, f ( * ) : # * c ( D , t o ) * , i,:L

where C and k are user-specified parameters representing the penalty of mis- classifying the training instances. For the remainder of this section, we assume k : I to simplify the problem. The parameter C can be chosen based on the model's performance on the validation set.

It follows that the Lagrangian for this constrained optimization problem can be written as follows:

. l /

L p : | l l w l l 2 + c t F , O r l " l l

' * L \ 1

i : 7

N ,^/ S - . . \ r . - ) S -- L A t | a t ( w . x i + b )

- 1 + € r i - \ u & , ( 5 . 4 6 ) i : I i : l

where the first two terms are the objective function to be minimized, the third term represents the inequality constraints associated with the slack variables,

o g \ o

o

o o

T \ \ \

l l I

\,I I

r \- ' . r

l l .

T I

, l l

a\,

t.,, I

I

I

I

P I

D . O Support Vector Machine (SVM) 269

and the last term is the result of the non-negativity requirements on the val-

ues of {,;'s. F\rrthermore, the inequality constraints can be transformed into

equality constraints using the following KKT conditions:

€ r ) 0 , ) r 2 0 , h ) 0 , l ; { s r ( * . x ; * b ) - 1 + € , } : 0 ,

l t t € t : 0 -

-^r s - .- ) A i U & i i : u 1 u j /-t

i : 7

( 5 . 4 7 )

(5.48)

( 5 . 4 e )

Note that the Lagrange multiplier ,\; given in Equation 5.48 is non-vanishing

only if the training instance resides along the lines w'x, * b : il or has

€r > O. On the other hand, the Lagrange multipliers;.ri given in Equation 5.49

arezero for any training instances that are misclassified (i.e., having {z > 0).

Setting the first-order derivative of ,L with respect to w, b, and {; to zero

would result in the following equations:

AL A u t r :

' '

AL 0b

AL otr,

.lr : t \&r.ru. ( 5 , 5 0 )

( 5 5 1 )

( 5 " 5 2 )

will pro-

( 5 . 5 3 )

N .lr

: - t \ r a r : Q a I ) n r , : 0 . i:t i:l

- C - \ t - t - t t - Q 4 \ t l l . t t : C .

Substituting Equations 5.50, 5.51, and 5.52 into the Lagrangian

duce the following dual Lagrangian:

1 . - - - - Lp : iD,xoxtoiaixi. xi + Cl€tz -

L,J z

- \ ^ o { r , ( \ ^ t o t * 0 . x i f b ) - 1 + € , }

- \ , - \ ; ) € '

N l

: Ir1 - : t Ar\ilrlixi' xi,1 / 2 , i : I i , i

which turns out to be identical to the dual Lagrangian for linearly separable

data (see Equation 5.40 on page 262). Nevertheless, the constraints imposed

27O Chapter 5 Classification: Alternative Techniques

on the Lagrange multipliers )1's are slightly different those in the linearly separable case. In the linearly separable case, the Lagrange multipliers must be non-negative, i.e., )e > 0. On the other hand, Equation 5.52 suggests that )z should not exceed C (since both pri and )a are non-negative). Therefore, the Lagrange multipliers for nonlinearly separable data are restricted to 0 (

\ r < C . The dual problem can then be solved numerically using quadratic pro-

gramming techniques to obtain the Lagrange multipliers .\a. These multipliers can be replaced into Equation 5.50 and the KKT conditions to obtain the parameters of the decision boundary.

5.5.4 Nonlinear SVM

The SVM formulations described in the previous sections construct a linear de- cision boundary to separate the training examples into their respective classes. This section presents a methodology for applying SVM to data sets that have nonlinear decision boundaries. The trick here is to transform the data from its original coordinate space in x into a new space O(x) so that a linear decision boundary can be used to separate the instances in the transformed space. Af- ter doing the transformation, we can apply the methodology presented in the previous sections to find a linear decision boundary in the transformed space.

Attribute TYansformation

To illustrate how attribute transformation can lead to a linear decision bound- ary, Figure 5.28(a) shows an example of a two-dimensional data set consisting of squares (classified as A

- 1) and circles (classified as A - -1). The data set

is generated in such a way that all the circles are clustered near the center of the diagram and all the squares are distributed farther away from the center. Instances of the data set can be classified using the following equation:

a ( r t , r z ) : { 1 ,

l f M ) 0 . 2 , ( 5 . 5 4 )

otherwise.

The decision boundary for the data can therefore be written as follows:

M - 0 . 2 ,

which can be further simplified into the following quadratic equation:

* ? - q + " 3 - r z : - 0 . 4 6 .

(a) Decision boundary in the original two-dimensional soace.

o . o Support Vector Machine (SVM) 27L

tr

d tr

tr

tr tr

&\'

x ? - x ,

(b) Decision boundary in the formed soace.

I

- { 1

- 0 1 5

Figure 5.28. Classifying data with a nonlinear decision boundary.

A nonlinear transformation O is needed to map the data from its original feature space into a new space where the decision boundary becomes linear. Suppose we choose the following transformation:

Q : ( q , r 2 ) - ( r ? , * Z , t [ 2 q , r t r 2 , t \ . I o . o o J

In the transformed space, we can find the parameters w : (t10, lrrt ..., w+) such that:

w4r21 t .s*f + uz{2q + wrrtr2t tr.rs : fl.

For illustration purposes, let us plot the graph of r/ - t2 versus rl - q for the previously given instances. Figure 5.28(b) shows that in the transformed space, all the circles are located in the lower right-hand side of the diagram. A Iinear decision boundary can therefore be constructed to separate the instances into their respective classes.

One potential problem with this approach is that it may suffer from the curse of dimensionality problem often associated with high-dimensional data. We will show how nonlinear SVM avoids this problem (using a method known as the kernel trick) later in this section.

Learning a Nonlinear SVM Model

Although the attribute transformation approach seems promising, it raises several implementation issues. First, it is not clear what type of mapping

272 Chapter 5 Classification: Alternative Techniques

function should be used to ensure that a linear decision boundary can be constructed in the transformed space. One possibility is to transform the data into an infinite dimensional space, but such a high-dimensional space may not be that easy to work with. Second, even if the appropriate mapping function is known, solving the constrained optimization problem in the high-dimensional feature space is a computationally expensive task.

To illustrate these issues and examine the ways they can be addressed, let us assume that there is a suitable function, O(x), to transform a given data set. After the transformation, we need to construct a linear decision boundary that will separate the instances into their respective classes. The linear decision boundary in the transformed space has the following form: w.O(*) * b: 0.

Definition 5.2 (Nonlinear SVM). The learning task for a nonlinear SVM can be formalized as the following optimization problem:

-,n ll:Yll' w 2

s u b j e c t t o A i ( w . A ( x 6 ) + b ) > 1 , i : I , 2 , . . . , N .

Note the similarity between the learning task of a nonlinear SVM to that of a linear SVM (see Definition 5.1 on page 262). The main difference is that, instead of using the original attributes x, the learning task is performed on the transformed attributes O(x). Following the approach taken in Sections 5.5.2 and 5.5.3 for linear SVM, we may derive the following dual Lagrangian for the constrained optimization problem:

( 5 . 5 6 ) z r J

Once the );'s are found using quadratic programming techniques, the param- eters w and b can be derived using the following equations:

L D : 1 ^ , - I f ) , ; A i s i y i e ( x z ) . a ( x r )

-: T

)'iY';Q(x)

r , ; { s , , ( I \ 1 y 1 a ( x ) . o ( * r ) + b ) - 1 } : 0 , J

l o . o / /

( 5 . 5 8 )

5.5 Support Vector Machine (SVM) 273

which are analogous to Equations 5.39 and 5.40 for linear SVM. Finally, a test

instance z canbe classified using the following equation:

f ( " ) : s i ' e n ( w ' o ( z ) + b ) ( 5 . 5 e )

Except for Equation 5.57, note that the rest of the computations (Equa-

tions 5.58 and 5.59) involve calculating the dot product (i.e., similarity) be-

tween pairs of vectors in the transformed space, O(*r) 'O("i). Such computa-

tion can be quite cumbersome and may suffer from the curse of dimensionality

problem. A breakthrough solution to this problem comes in the form ,rf a

method known as the kernel trick.

I(ernel TYick

The dot product is often regarded as a measure of similarity between two

input vectors. For example, the cosine similarity described in Section 2.4.5

on page 73 can be defined as the dot product between two vectors that are

normalized to unit length. Analogously, the dot product O(x6)'O(*;) can also

be regarded as a measure of similarity between two instances, x2 and x3, in

the transformed space. The kernel trick is a method for computing similarity in the transformed

space using the original attribute set. Consider the mapping function O given

in Equation 5.55. The dot product between two input vectors u and v in the

transformed space can be written as follows:

O ( u ) . O ( v ) : ( u ? , u \ , t / 2 u t , J i u 2 , L ) ' ( u ? , r | , r t r r , r t u r , t 1 :

"lul +

"|rl + 2upt * 2uzuz l7

: ( u . v + 1 ) 2 .

This analysis shows that the dot product in the transformed space can be

expressed in terms of a similarity function in the original space:

K ( u , v ) : O ( u ) . O ( " ) : ( u . v + 1 ) 2 ( 5 . 6 1 )

The similarity function, K, which is computed in the original attribute space'

is known as the kernel function. The kernel trick helps to address some

of the concelns about how to implement nonlinear SVM. First, we do not

have to know the exact form of the mapping function (D because the kernel

: sisn(i ^,r,*(x1) ' o(z) . ,

( 5 . 6 0 )

274 Chapter 5 Classification: Alternative Techniques

functions used in nonlinear SVM must satisfy a mathematical principle known as Mercerts theorem. This principle ensures that the kernel functions can always be expressed as the dot product between two input vectors in some high-dimensional space. The transformed space of the SVM kernels is called a reproducing kernel Hilbert space (RKHS). Second, computing the dot products using kernel functions is considerably cheaper than using the transformed attribute set O(x). Third, since the computations are performed in the original space, issues associated with the curse of dimensionality problem can be avoided.

Figure 5.29 shows the nonlinear decision boundary obtained by SVM using the polynomial kernel function given in Equation 5.61. A test instance x is classified according to the following equation:

rL . , \ - ,

f ( z ) : s i s n ( \ \ s i Q ( x i ) . a @ ) + b ) t : I

n . , \ . - .: s i g n ( \ ) , i g i K ( x i . z ) + b )

; - 1

n . z \ - ' , . ' ,

": s l g n \ ) k A i \ x i ' z + I ) ' + 0 ) , (5.62) 1,: I

where b is the parameter obtained using Equation 5.58. The decision boundary obtained by nonlinear SVM is quite close to the true decision boundary shown i n F i g u r e 5 . 2 8 ( a ) .

Mercerts Theorem

The main requirement for the kernel function used in nonlinear SVM is that there must exist a corresponding transformation such that the kernel function computed for a pair of vectors is equivalent to the dot product between the vectors in the transformed space. This requirement can be formally stated in the form of Mercer's theorem.

Theorem 5.1 (Mercer's Theorem). ,4 kernel functi,on K can be erpressed AS

K ( u , a ) : O ( z ) ' O ( r )

i,f and only i,f, for any functi,on g(r) such that I g(n)2dris fini,te, then

| * , * , s ) s @ ) g @ ) d n d a > - 0 .

1

0.9

0 . 8

o.7

0 . 6

x u,c

0.4

0.3

0.2

0 . 1

0

5.5 Support Vector Machine (SVM) 275

tr

tr tr

t r y

tr tr

tr

x l o.7

Figure 5.29. Decision boundary produced by a nonlinear SVM with polynomial kernel.

Kernel functions that satisfy Theorem 5.1 are called positive definite kernel functions. Examples of such functions are listed below:

K ( * , y ) : ( x . y + l ) P

K(*, y) : "-ll*-vll2

/ {2"2)

K ( * , y ) : t a n h ( k x ' y - d )

( 5 . 6 3 )

( 5 . 6 4 )

( 5 . 6 5 )

Example 5.6. Consider the polynomial kernel function given in Equation 5.63. Let 9(z) be a function that has a finite L2 rrorm, i.e., -[ g(x)2dx < a.

f

J 8 ' Y + l ) e e ( x ) s ( Y ) d x d Y

" P / \

: / t ( l ) t * . y ) i g ( * ) g ( y ) d . x d , y J 7 _ o \ x / '

: f

(1) I ,-8, (',J, ) [r""lo'(rzaz)o"('zas)o"

276 Chapter 5

D

: \ - \ - L L i : O a t , a z , . . .

Classification: Alternative Techniques

Because the result of the integration is non-negative, the polynomial kernel function therefore satisfies Mercer's theorem. r

5.5.5 Characteristics of SVM

SVM has many desirable qualities that make it one of the most widely used classification algorithms. Following is a summary of the general characteristics of SVM:

1. The SVM learning problem can be formulated as a convex optimization problem, in which efficient algorithms are available to find the global minimum of the objective function. Other classification methods, such as rule-based classifiers and artificial neural networks, employ a greedy- based strategy to search the hypothesis space. Such methods tend to find only locally optimum solutions.

2. SVM performs capacity control by maximizing the margin of the decision boundary. Nevertheless, the user must still provide other parameters such as the type of kernel function to use and the cost function C for introducing each slack variable.

3. SVM can be applied to categorical data by introducing dummy variables for each categorical attribute value present in the data. For example, if Marital Status has three values {Single, Married, Divorced}, we can introduce a binary variable for each of the attribute values.

4. The SVM formulation presented in this chapter is for binary class prob- lems. Some of the methods available to extend SVM to multiclass orob- lems are presented in Section 5.8.

5.6 Ensemble Methods

The classification techniques we have seen so far in this chapter, with the ex- ception of the nearest-neighbor method, predict the class labels of unknown examples using a single classifier induced from training data. This section presents techniques for improving classification accuracy by aggregating the predictions of multiple classifiers. These techniques are known as the ensem- ble or classifier combination methods. An ensemble method constructs a

( , ) ( - , : , ) [ l

' ? " 7 ' s ( r t ' r 2 ' ) d ' r 1 d ' " ] '

5.6 Ensemble Methods 277

set of base classifiers from training data and performs classification by taking a vote on the predictions made by each base classifier. This section explains why ensemble methods tend to perform better than any single classifier and presents techniques for constructing the classifier ensemble.

5.6.1 Rationale for Ensemble Method

The following example illustrates how an ensemble method can improve a classifier's performance.

Example 5.7. Consider an ensemble of twenty-five binary classifiers, each of which has an error rate of e : 0.35. The ensemble classifier predicts the class Iabel of a test example by taking a majority vote on the predictions made by the base classifiers. If the base classifiers are identical, then the ensemble will misclassify the same examples predicted incorrectly by the base classifiers. Thus, the error rate of the ensemble remains 0.35. On the other hand, if the base classifiers are independent-i.e., their errors are uncorrelated-then the ensemble makes a wrong prediction only if more than half of the base classifiers predict incorrectly. In this case, the error rate of the ensemble classifier is

O a 1 -

vensemDle - - e ) 2 5 - t : 0 ' 0 6 , ( 5 . 6 6 )

which is considerably lower than the error rate of the base classifiers. r

Figure 5.30 shows the error rate of an ensemble of twenty-five binary clas- sifiers (e".,""-bre) for different base classifier error rates (e). The diagonal line represents the case in which the base classifiers are identical, while the solid line represents the case in which the base classifiers are independent. Observe that the ensemble classifier performs worse than the base classifiers when e is larger than 0.5.

The preceding example illustrates two necessary conditions for an ensem- ble classifier to perform better than a single classifier: (1) the base classifiers should be independent ofeach other, and (2) the base classifiers should do bet- ter than a classifier that performs random guessing. In practice, it is difficult to ensure total independence among the base classifiers. Nevertheless, improve- ments in classification accuracies have been observed in ensemble methods in which the base classifiers are slightly correlated.

278 Chapter 5 Classification: Alternative Techniques

Base classifier error

Figure 5.30. Comparison between enors of base classifiers and errors of the ensemble classifier.

Step 1: Create Multiple

Data Sets

Step 2: Build Multiple

Classifiers

Step 3: Combine

Classifiers

Figure 5.31. A logicalview of the ensemble learning method.

5.6.2 Methods for Constructing an Ensemble Classifier

A logical view of the ensemble method is presented in Figure 5.31. The basic idea is to construct multiple classifiers from the original data and then aggre- gate their predictions when classifying unknown examples. The ensemble of classifiers can be constructed in many ways:

1

0.9

6 o o

a o o o

E o @ c

uJ

0 . 7

u . o

o.4

U J

0 . 2

0 . 1

1 .

5.6 Ensemble Methods 279

By manipulating the training set. In this approach, multiple train-

ing sets are created by resampling the original data according to some

sampling distribution. The sampling distribution determines how lili<ely

it is that an example will be selected for training, and it may vary from

one trial to another. A classifier is then built from each training set using

a particular learning algorithm. Bagging and boosting are two exam-

ples of ensemble methods that manipulate their training sets. These

methods are described in further detail in Sections 5.6.4 and 5.6.5.

By manipulating the input features. In this approach, a subset

of input features is chosen to form each training set. The subset can

be either chosen randomly or based on the recommendation of domain

experts. Some studies have shown that this approach works very well

with data sets that contain highly redundant features. Random forest,

which is described in Section 5.6.6, is an ensemble method that manip-

ulates its input features and uses decision trees as its base classifiers.

By manipulating the class labels. This method can be used when the

number of classes is sufficiently large. The training data is transformed

into a binary class problem by randomly partitioning the class la,bels

into two disjoint subsets, Ag and A1. TYaining examples whose class

label belongs to the subset As are assigned to class 0, while those that

belong to the subset Al are assigned to class 1. The relabeled examples

are then used to train a base classifier. By repeating the class-relabeling

and model-building steps multiple times, an ensemble of base classifiers

is obtained. When a test example is presented, each base classifiet Ci is

used to predict its class label. If the test example is predicted as class

0, then all the classes that belong to As will receive a vote. Conversely'

if it is predicted to be class 1, then all the classes that belong to A1

will receive a vote. The votes are tallied and the class that receives the

highest vote is assigned to the test example. An example of this approach

is the error-correcting output coding method described on page 307.

4.By manipulating the learning algorithm. Many learning algo-

rithms can be manipulated in such a way that applying the algoriithm

several times on the same training data may result in different models.

For example, an artificial neural network can produce different rnod-

els by changing its network topology or the initial weights of the links

between neurons. Similarly, an ensemble of decision trees can be con-

structed by injecting randomness into the tree-growing procedure. For

2 .

3 .

280 Chapter 5 Classification: Alternative Techniques

example, instead of choosing the best splitting attribute at each node, we can randomly choose one of the top k attributes for splitting.

The first three approaches are generic methods that are applicable to any classifiers, whereas the fourth approach depends on the type of classifier used. The base classifiers for most of these approaches can be generated sequentially (one after another) or in parallel (all at once). Algorithm 5.5 shows the steps needed to build an ensemble classifier in a sequential manner. The first step is to create a training set from the original data D. Depending on the type of ensemble method used, the training sets are either identical to or slight modifications of D. The size of the training set is often kept the same as the original data, but the distribution of examples may not be identicall i.e., some examples may appear multiple times in the training set, while others may not appear even once. A base classifier Ci is then constructed from each training set Da. Ensemble methods work better with unstable classifiers. i.e.. base classifiers that are sensitive to minor perturbations in the training set. Ex- amples of unstable classifiers include decision trees, rule-based classifiers, and artificial neural networks. As will be discussed in Section 5.6.3, the variability among training examples is one of the primary sources of errors in a classifier. By aggregating the base classifiers built from different training sets, this may help to reduce such types of errors.

Finally, a test example x is classified by combining the predictions made by the base classifiers C;(x):

C . ( * ) : V o t e ( C t ( x ) , C 2 ( x ) , . . . , C n ( * ) ) .

The class can be obtained by taking a majority vote on the individual predic- tions or by weighting each prediction with the accuracv of the base classifier.

Algorithm 5.5 General procedure for ensemble method. l: Let D denote the original training data, k denote the number of base classifiers,

and 7 be the test data. 2 : f o r i : l t o k d o 3: Create training set, D4 ftom D. 4: Build a base classifier Ci from Di. 5: end for 6: for each test record r € T d,o 7 : C * ( r ) : V o t e ( C 1 ( x ) , C z ( * ) , . . . , C r ( * ) ) 8: end for

5 . 6 Ensemble Methods 28'1.

5.6.3 Bias-Variance Decomposition

Bias-variance decomposition is a formal method for analyzing the prediction

error of a predictive model. The following example gives an intuitive explana-

tion for this method. Figure 5.32 shows the trajectories of a projectile launched at a particular

angle. Suppose the projectile hits the floor surface at some location r, aL a

distance d away from the target position t. Depending on the force applied

to the projectile, the observed distance may vary from one trial to another.

The observed distance can be decomposed into several components. The first

component, which is known as bias, measures the average distance between

the target position and the Iocation where the projectile hits the floor. The

amount of bias depends on the angle of the projectile launcher. The second

component, which is known as variance, measutes the deviation between r

and the average position 7 where the projectile hits the floor. The variance

can be explained as a result of changes in the amount of force applied to the

projectile. Finally, if the target is not stationary, then the observed distance

is also affected by changes in the location of the target. This is considered the

noise component associated with variability in the target position. Putting

these components together, the average distance can be expressed as:

dy,e(a,t) : BiasB* VarianceTf Noiset, ( 5 , 6 7 )

where / refers to the amount of force applied and 0 is the angle of the launcher.

The task of predicting the class label of a given example can be analyzed

using the same apploach. For a given classifier, some predictions may turn out

to be correct, while others may be completely off the mark. We can decompose

the expected error of a classifier as a sum of the three terms given in Equation

5.67, where expected error is the probability that the classifier misclassifies a

{----} <--> 'Variance' 'Noise'

\_ 'Bias'

Figure 5.32. Bias-variance decomposition.

282 Chapter 5 Classification: Alternative Techniques

given example. The remainder of this section examines the meaning of bias, variance, and noise in the context of classification.

A classifier is usually trained to minimize its training error. However, to be useful, the classifier must be abie to make an informed guess about the class labels of examples it has never seen before. This requires the classifier to generalize its decision boundary to regions where there are no training exam- ples available--a decision that depends on the design choice of the classifier. For example, a key design issue in decision tree induction is the amount of pruning needed to obtain a tree with low expected error. Figure 5.33 shows two decision trees, ft and 72, that are generated from the same training data, but have different complexities. 7z is obtained by pruning ?r until a tree with maximum depth of two is obtained . Ty, on the other hand, performs very little pruning on its decision tree. These design choices will introduce a bias into the classifier that is analogous to the bias of the projectile launcher described in the previous example. In general, the stronger the assumptions made by a classifier about the nature of its decision boundary, the larger the classi- fier's bias will be. 72 therefore has a larger bias because it makes stronger assumptions about its decision boundary (which is reflected by the size of the tree) compared to ?r. other design choices that may introduce a bias into a classifier include the network topology of an artificial neural network and the number of neighbors considered by a nearest-neighbor classifier.

The expected error of a classifier is also affected by variability in the train- ing data because different compositions of the training set may lead to differ- ent decision boundaries. This is analogous to the variance in r when different amounts of force are applied to the projectile. The last component of the ex- pected error is associated with the intrinsic noise in the target class. The target class for some domains can be non-deterministic; i.e., instances with the same attribute values can have different class labels. Such errors are unavoidable even when the true decision boundary is known.

The amount of bias and variance contributing to the expected error depend on the type of classifier used. Figure 5.34 compares the decision boundaries produced by a decision tree and a 1-nearest neighbor classifier. For each classifier, we plot the decision boundary obtained by "averaging" the models induced from 100 training sets, each containing 100 examples. The true deci- sion boundary from which the data is generated is also plotted using a dashed line. The difference between the true decision boundary and the "averaged" decision boundary reflects the bias of the classifier. After averaging the mod- els, observe that the difference between the true decision boundary and the decision boundary produced by the l-nearest neighbor classifier is smaller than

x 1 < - 1 . 2 4 x 1 < 1 1 . 0 0

5 . 6 Ensemble Methods 283

1 5

) < 2 < 9 . 2 5

O +

(a) Decision tree Tj -5 1 0 1 5

1 5

1 0

x 1 < 1 1 . 0 0

+ O

(b) Decision tree T2 -5 1 0 1 5

Figure 5.33. Two decision trees with different complexities induced from the same training data.

the observed difference for a decision tree classifier. This result suggests that

the bias of a l-nearest neighbor classifier is lower than the bias of a decision

tree classifier. On the other hand, the l-nearest neighbor classifier is more sensitive to

the composition of its training examples. If we examine the models induced

from different training sets, there is more variability in the decision boundary

of a l-nearest neighbor classifier than a decision tree classifier. Therefore, the

decision boundary of a decision tree classifier has a lower variance than the

l-nearest neighbor classifier.

5.6.4 Bagging

Bagging, which is also known as bootstrap aggregating, is a technique that

repeatedly samples (with replacement) from a data set according to a uniform probability distribution. Each bootstrap sample has the same size as the origi-

nal data. Because the sampling is done with replacement, some instances may

appear several times in the same training set, while others may be omitted

from the training set. On average, a bootstrap sample D; contains appnoxi-

d o o o o o o o o

o o o o o 3

o

o _ o f , ; o t f f i ' +

o o l * *

+ + +

+ + + *

+ * * * * * *

o

o o o

- * +' + + +

*4

+

x 1 < - 1 . 2 4

o o o id o o o o o o o o

o o o o o 3 ++

+ +

+

+ ^ o

o * * o o 8- o +

+ * l * + * +t + + *

t a J * *

*a

+

o o

o o o

284 Chapter 5 Classification: Alternative Technioues

-30 L -30 + 0 L

(a) Decision boundary for decision tree. (b) Decision boundary for l-nearest neighbor.

- 1 0

Figure 5.34. Bias of decision tree and 1-nearest neighbor classifiers.

+ . . , + +

: . . { o * ** *** * ** #..* #"3...

ii* .. *H o o

+ j i , / " " : " o - - - l )

,-tjf".= t ".. ?" ooo^

o r i f J e " " o " . p d g . ,

o o ^ o n " O o r c )

o, ' o

r + +

\*

o

T

' f + + , ' J+ r.,. T +

' ' I

+ T '

+

, r + + +

* J a o H

- ^ o o o o

O ' 9 O O

- O O

o o o . p d I a _ '

o . ' o ^ o ^t o o ( g

Algorithm 5.6 Bagging algorithm. 1: Let k be the number of bootstrap samples. 2 : f o r i , : I t o k d o 3: Create a bootstrap sample of size N, Dt. 4: Tlain a base classifier Ca on the bootstrap sample Di. 5: end for 6: C. (r) : argmax 1,6(Ct1r1 : y) .

c {d(') : 1 if its argument is true and 0 otherwise}.

mately 63% of the original training data because each sample has a probability 1 - (1 - 1/l/)r' of being selected in each D,i.. If l/ is sufficiently large, this probability converges to 1- Ll"- 0.632. The basic procedure for bagging is summarized in Algorithm 5.6. After training the k classifiers, a test instance is assigned to the class that receives the highest number of votes.

To illustrate how bagging works, consider the data set shown in Table 5.4. Let r denote a one-dimensional attribute and y denote the class label. Suppose we apply a classifier that induces only one-level binary decision trees, with a test condition r ( k, where k is a split point chosen to minimize the entropy of the leaf nodes. Such a tree is also known as a decision stump.

Without bagging, the best decision stump we can produce splits the records at either z < 0.35 or r I 0.75. Either way, the accuracy of the tree is at

r 0 . 1 0 . 2 0 . 3 0 . 4 0 . 5 0 . 6 0 . 7 0 . 8 0 . 9 I

a I 1 1 - 1 - 1 - 1 - 1 1 1 1

5.6 Ensemble Methods 285

Table 5.4. Example of data set used to construct an ensemble of bagging classifiers.

most 70%. Suppose we apply the bagging procedure on the data set using

ten bootstrap samples. The examples chosen for training in each bagging

round are shown in Figure 5.35. On the right-hand side of each table, we also

illustrate the decision boundary produced by the classifier. We classify the entire data set given in Table 5.4 by taking a majority

vote among the predictions made by each base classifier. The results of the predictions are shown in Figure 5.36. Since the class labels are either -1 or

*1, taking the majority vote is equivalent to summing up the predicted values

of gr and examining the sign of the resulting sum (refer to the second to last

row in Figure 5.36). Notice that the ensemble classifier perfectly classifies all

ten examples in the original data. The preceding example illustrates another advantage of using ensemble

methods in terms of enhancing the representation of the target function. Ilven

though each base classifier is a decision stump, combining the classifiers can

Iead to a decision tree of depth 2. Bagging improves generalization error by reducing the variance of the base

classifiers. The performance of bagging depends on the stability of the base

classifier. If a base classifier is unstable, bagging helps to reduce the errors

associated with random fluctuations in the training data. If a base classifier

is stable, i.e., robust to minor perturbations in the training set, then the

error of the ensemble is primarily caused by bias in the base classifier. In

this situation, bagging may not be able to improve the performance of the

base classifiers significantly. It may even degrade the classifier's performance

because the effective size of each training set is about 37% smaller than the

original data. Finally, since every sample has an equal probability of being selected, bag-

ging does not focus on any particular instance of the training data. It is

therefore less susceptible to model overfitting when applied to noisy data.

5.6.5 Boosting

Boosting is an iterative procedure used to adaptively change the distribution

of training examples so that the base classifiers will focus on examples that

are hard to classify. Unlike bagging, boosting assigns a weight to each training

R o u n d 1 : x 1 0 . 1 o.2 o.2 0.3 0.4 o.4 0.5 0.6 0.9 0.9

1v 1 1 1 - 1 - 1 - l 1 1 1

286 Chapter 5 Classification: Alternative Techniques

x < = 0 . 3 5 - = > y = 1 x > 0 . 3 5 = = > y = - 1

x < = 0 . 6 5 = = 1 y = | x > 0 . 6 5 - = > y = 1

x < = 0 . 3 5 = = y y = | x > 0 . 3 5 = = y y = - l

x < = 0 . 3 = = 1 y = l x v Q . t = = 1 y = - 1

x < = 0 . 3 5 = = > y = l x > 0 . 3 5 = = > y = - 1

x <= 0.75 ==> y - -1

x > 0 . 7 5 = = > y = 1

x <= 0.75 ==> y = -1

x > 0 . 7 5 = = > y = 1

x <= 0.75 ==> y = -1

x > 0 . 7 5 = = 2 y = l

x <= O.75 ==> y - -1

x > 0 . 7 5 = = > y = - t

x <= 0.05 ==> y = -1

x > 0 . 0 5 = = > y = 1

Figure 5.35. Example of bagging.

example and may adaptively change the weight at the end of each boosting round. The weights assigned to the training examples can be used in the following ways:

1. They can be used as a sampling distribution to draw a set of bootstrap samples from the original data.

2. They can be used by the base classifier to learn a model that is biased toward higher-weight examples.

R o u n d 2 : x 1 0 . 1 o.2 0.3 0.4 0.5 0.8 0.9 1 1 1

1v 1 1 - 1 -1 1 1 1 1 1

R o u n d 3 : x I 0 . 1 o.2 0.3 o.4 o.4 0.5 0.7 o.7 0.8 0.9 v 1 1 1 - 1 - 1 -1 1 1

Round 4: x 1 0 . 1 0 . 1 o.2 0.4 o.4 0.5 0.5 0.7 0.8 0.9

1v 1 1 -1 - 1 1 -1 1 1 1

Bagging Round 5:

x 0.1 0 . 1 0.2 0.5 0.6 0.6 0.6 1 1 v 1 1 1 1 - 1 I 1 1 1

Round 6: x o.2 0.4 0.5 0.6 0.7 o.7 o.7 0.8 0.9 1 v 1 1 1 1 1 1 1 1 1

R o u n d 7 : x 0 . 1 o.4 o.4 0.6 0.7 0.8 0.9 0.9 0.9 1 v 1 -1 1 1 1 1 1 1 1 1

Round 8: x 0.1 o.2 0.5 0.5 0.5 0.7 o.7 0.8 0.9 1 v 1 1 1 - 1 - 1 -t 1 1 1 1

Round x 0 . 1 0.3 0.4 o.4 0.6 o.7 o.7 0.8 1 1 v 1 1 1 1 1 1 1

R o u n d 1 0 : x 0 . 1 0 . 1 0 . 1 0 . 1 0.3 0.3 0.8 0.8 0.9 0.9 v 1 1 1 1 1 1 I 1 1

Ensemble Methods 287

Round x=0,1x=0.2x=0.3x=0.4 x=0,5 x=0,6x=0.7x=0.8 x=0.9x=1.0 1 1 1 1 1 1 1 1 1 -1 1

2 1 1 1 1 1 1 1 1 1

3 1 1 1 -1 1 1 1 1 I

4 1 1 1 1 1 1 1 -1 1 -1

5 1 1 1 1 1 -1 1 1 1

6 -1 - 1 -1 1 1 1 -1 1 1 ,l

'l -1 -1 1 1 1 1 1 1

I -1 -1 -1 1 1 1 -1 1 1 1

I 1 1 1 1 1 1 -1 1 1 1

1 0 1 1 1 'l t 1 1 1 1 1 Sum 2 2 2 -o -6 -6 -6 2 2 2

Sign 1 1 1 1 1 1 1 1 1

True Class 1 1 1 1 1 1 1 1 1 1

Figure 5.36. Example of combining classifiers constructed using the bagging approach.

This section describes an algorithm that uses weights of examples to de-

termine the sampling distribution of its training set. InitiallS the examples

are assigned equal weights, 1/N, so that they are equally likely to be chosen

for training. A sample is drawn according to the sampling distribution of the

training examples to obtain a new training set. Next, a classifier is induced

from the training set and used to classify all the examples in the original data.

The weights of the training examples are updated at the end of each boost-

ing round. Examples that are classified incorrectly will have their weights

increased, while those that are classified correctly will have their weights de-

creased. This forces the classifier to focus on examples that are difficult to

classify in subsequent iterations. The following table shows the examples chosen during each boosting round.

Boosting (Round 1): 7 3 2 8 7 I 4 1 0 6 3 Boosting (Round 2): 5 4 I 4 2 5 1 F7I 4 2 Boosting (Round 3): 4 4 8 1 0 4 5 4 6 3 4

Initially, all the examples are assigned the same weights. However, some ex-

amples may be chosen more than once, e.g., examples 3 and 7, because the

sampling is done with replacement. A classifier built from the data is then

used to classify all the examples. Suppose example 4 is difficult to classify.

The weight for this example will be increased in future iterations as it gets

misclassified repeatedly. Meanwhile, examples that were not chosen in the pre-

5 . 6

288 Chapter 5 Classification: Alternative Techniques

vious round, e.g., examples 1 and 5, also have a better chance of being selected in the next round since their predictions in the previous round were Iikely to be wrong. As the boosting rounds proceed, examples that are the hardest to classify tend to become even more prevalent. The final ensemble is obtained by aggregating the base classifiers obtained from each boosting round.

Over the years, several implementations of the boosting algorithm have been developed. These algorithms differ in terms of (1) how the weights of the training examples are updated at the end of each boosting round, and (2) how the predictions made by each classifier are combined. An implementation called AdaBoost is explored in the next section.

AdaBoost

L e t { ( x y , y i ) | j : 7 , 2 , . . . , l t r } d e n o t e a s e t o f N t r a i n i n g e x a m p l e s . I n t h e AdaBoost algorithm, the importance of a base classifier Ci depends on its error rate, which is defined as

( 5 . 6 8 )

where I (p) : 1 if the predicate p is true, and 0 otherwise. The importance of a classifier Ci is given by the following parameter,

1 , / 1 - e r \ a i : r t " \

* /

Note that a; has a large positive value if the error rate is close to 0 and a large negative value if the error rate is close to 1, as shown in Figure 5.37.

The a6 parameter is also used to update the weight of the training ex- amples. To illustrate,let w[i) denote the weight assigned to example (*t,A) during the jth boosting round. The weight update mechanism for AdaBoost is given by the equation:

# tf '1 r(co61t +r,)],

lexp-".r lexpo:

. U + r 1 : ' [ i )

*' z.i If Ci(x) : y,

i f C i @ ) I y i ' (5.6e)

where Zi isthe normalization factor used to ensure that !r.\l+t1 :1. The weight update formula given in Equation 5.69 increases the weights of incor- rectly classified examples and decreases the weights of those classified correctly.

5 . 6 Ensemble Methods 289

e l

E - 1

- 4

o.4 E

Figure 5.37, Plot of o as a function of training error e .

Instead of using a majority voting scheme, the prediction made by each classifier C7 is weighted according to oi. This approach allows AdaBoost to penalize models that have poor accuracy, e.g., those generated at the earlier boosting rounds. In addition, if any intermediate rounds produce an error rate higher than 50%, the weights are reverted back to their original uniform values, ut: IlN, and the resampling procedure is repeated. The AdaBoost algorithm is summarized in Algorithm 5.7.

Let us examine how the boosting approach works on the data set shown in Table 5.4. Initially, all the examples have identical weights. Afber three boosting rounds, the examples chosen for training are shown in Figure 5.38(a). The weights for each example are updated at the end of each boosting round using Equation 5.69.

Without boosting, the accuracy of the decision stump is, at best, 70%. With AdaBoost, the results of the predictions are given in Figure 5.39(b). The final prediction of the ensemble classifier is obtained by taking a weighted average of the predictions made by each base classifi.er, which is shown in the last row of Figure 5.39(b). Notice that AdaBoost perfectly classifies all the examples in the training data.

An important analytical result of boosting shows that the training error of the ensemble is bounded bv the following expression:

€ensemble ( 5 . 7 0 )

29O Chapter 5 Classification: Alternative Techniques

Algorithm 5.7 AdaBoost algorithm. 1: w : {r j : rlN I i : I,2,. . . ,,4f}. {Initialize the weights for all l/ examples.} 2: Let k be the number of boosting rounds. 3: for i,: I to k do 4'. Create training set Da by sampling (with replacement) from D according to w. 5: tain a base classifier Ci on Di. 6: Apply Ci to aIl examples in the original training set, D. 7: et : *lDrlu, 6(C{r) I Ai)l {Calculate the weighted error.} 8: if e; > 0.5 then 9 : w : { w j : l l N I j : t , 2 , . . . , N } . { R e s e t t h e w e i g h t s f o r a l l l / e x a m p l e s . }

10: Go back to Step 4. 11: end if 1 2 : o . i : + h * . 13: Update the weight of each example according to Equation 5.69. 14: end for 15: C.(x) : argmax ll:ro,5(C1(*) : g))

a

where e,; is the error rate of each base classifier i. If the error rate of the base classifier is less than 50%, we can write e; : 0.5 - li, where l; fir€asur€s how much better the classifier is than random guessing. The bound on the training error of the ensemble becomes

€ensemble ( T ( 5 . 7 r )

If m < 7x for all i's, then the training error of the ensemble decreases expo- nentially, which leads to the fast convergence of the algorithm. Nevertheless, because of its tendency to focus on training examples that are wrongly classi- fied, the boosting technique can be quite susceptible to overfitting.

5.6.6 Random Forests

Random forest is a class of ensemble methods specifically designed for decision tree classifiers. It combines the predictions made by multiple decision trees, where each tree is generated based on the values of an independent set of random vectors, as shown in Figure 5.40. The random vectors are generated from a fixed probability distribution, unlike the adaptive approach used in AdaBoost, where the probability distribution is varied to focus on examples that are hard to classify. Bagging using decision trees is a special case of random forests, where randomness is injected into the model-building process

U f - + " : ( e x p ( _ r . " r )

Boosting Round 1:

x 0.1 0.4 0.5 0.6 0.6 o.7 0.7 0.7 0.8 1 v 1 1 -1 -1 -1 -1 -1 -1 1 1

5.6 Ensemble Methods 29L

Round x=0.1 x=0.2 x=0,3 x=0,4 x=0.5 x=0.6 x=0.7 x=0,8 x=0.9 x=1,0 1 0 . 1 0 . 1 0 . 1 0 . 1 0 . 1 0 . 1 0 . 1 0 . 1 0 . 1 2 0 . 3 1 1 0.3't'l 0 . 3 t 1 0 . 0 1 0 . 0 1 0 . 0 1 0.01 0 . 0 1 0.01 0.01 J 0.029 0.029 0.029 0.228 0.228 0.228 0.228 0.009 0.009 0.009

(b) Weights of training records

Figure 5.38. Example of boosting.

by randomly choosing ly' samples, with replacement, from the original training set. Bagging also uses the same uniform probability distribution to generate its bootstrapped samples throughout the entire model-building process.

It was theoretically proven that the upper bound for generalization error of random forests converges to the following expression, when the number of trees is sufficiently large.

where p is the average correlation among the trees and s is a quantity that measures the "strength" of the tree classifiers. The strength of a set of classi- fiers refers to the average performance of the classifiers, where performance is measured probabilistically in terms of the classifier's margin:

G e n e r a l i z a t i o n e r r o r = - r y ,

m a r g i n , M ( X , Y ) : P l Y o - Y ) - r y # P ( Y o : Z ) ,' Z+Y

(5,.72)

( 5 . 7 3 )

where YB is the predicted class of X according to a classifier built from some random vector d. The higher the margin is, the more likely it is that the

Boosting Round 2:

x 0.1 0.1 0.2 0.2 0.2 0.2 0.3 0.3 0.3 0.3

v 1 1 1 1 1 1 1 1 1 1

Boosting Round 3:

x o.2 0.2 0.4 0.4 0.4 0.4 0.5 0.6 0.6 0.7 v 1 1 -1 1 -1 -1 -1 -1 -1 -1

(a) Training records chosen during boosting

ffi+ (c

292 Chapter 5 Classification: Alternative Techniques

Round Split Point Left Glass Right Class cx, 1 0.75 1 1 1 . 7 3 8

2 0.05 I 1 2.7784

.t 0.3 1 1 4 . 1 1 9 5

(a)

Round x=0.1 x=0.2 x=0.3 x--0.4 x=0.5 x=0.6 x=0.7 x=0,8 x=0.9 x=l.0 1 1 1 1 1 1 -1 1 1 I

2 1 1 1 1 1 I 1 1

.t 1 1 1 1 1 1 1 1 1 1

Sum 5 . 1 6 5 . 1 6 5 . 1 6 -3.08 -3.08 -3.08 -3.08 0.397 0.397 0.397 Sign 1 1 1 1 1 1 1 1 1

(b)

Figure 5,39. Example ol combining classifiers constructed using the AdaBoost approach.

Original Training data

Step 2: Use random

vector to build multiple decision trees

Step 3: Combine

decision trees

Figure 5.40. Random forests.

classifier correctly predicts a given example X. Equation5.72 is quite intuitive; as the trees become more correlated or the strength of the ensemble decreases, the generalization error bound tends to increase. Randomization helps to reduce the correlation among decision trees so that the generalization error of the ensemble can be improved.

+

5.6 Ensemble Methods 293

Each decision tree uses a random vector that is generated from some fixed probability distribution. A random vector can be incorporated into the t,ree- growing process in many ways. The first approach is to randomly select F input features to split at each node of the decision tree. As a result, instead of examining all the available features, the decision to split a node is determined from these selected f. features. The tree is then grown to its entirety without any pruning. This may help reduce the bias present in the resulting 1;ree. Once the trees have been constructed, the predictions are combined using a majority voting scheme. This approach is known as Forest-Rl, where RI refers to random input selection. To increase randomness, bagging can also be used to generate bootstrap samples for Forest-Rl The strength and correlation of random forests may depend on the size of F. If F is sufficiently small, then the trees tend to become less correlated. On the other hand, the strength of the tree classifier tends to improve with a larger number of features, F. As a tradeoff, the number of features is commonly chosen to be F - logz d + L, where d is the number of input features. Since only a subset of the features needs to be examined at each node, this approach helps to significantly re<luce the runtime of the algorithm.

If the number of original features d is too small, then it is difficult to choose an independent set of random features for building the decision trees. One way to increase the feature space is to create linear combinations of the input features. Specifically, at each node, a new feature is generated by randomly selecting .t of the input features. The input features are linearly combined using coefficients generated from a uniform distribution in the range of [-1, 1]. At each node, F of such randomly combined new features are generated, and the best of them is subsequently selected to split the node. This approach is known as Forest-RC.

A third approach for generating the random trees is to randomly select one of the f' best splits at each node of the decision tree. This approach may potentially generate trees that are more correlated than Forest-Rl and Fonest- RC, unless -F is sufficiently large. It also does not have the runtime savings of Forest-Rl and Forest-RC because the algorithm must examine all the splitting features at each node of the decision tree.

It has been shown empirically that the classification accuracies of random forests are quite comparable to the AdaBoost algorithm. It is also more robust to noise and runs much faster than the AdaBoost algorithm. The classification accuracies of various ensemble algorithms are compared in the next section.

294 Chapter 5 Classification: Alternative Techniques

Table 5.5. Comparing the accuracy of a decision tree classifier against three ensemble methods.

Data Set Number of (Attributes, Classes,

Records)

Decision Tlee (%)

Bagging ( % )

Boosting (%)

RF ( % )

Anneal Australia Auto Breast Cleve Credit Diabetes German Glass Heart Hepatitis Horse Ionosphere Iris Labor LedT Lymphography Pima Sonar Tic-tac-toe Vehicle Waveform Wine Zoo

( 3 9 , 6 , 8 9 8 ) ( 1 5 , 2 , 6 9 0 ) (26, 7, 205) ( 1 1 , 2 , 6 9 9 ) (14, 2, 303) (16, 2, 690) ( 9 , 2 , 7 6 9 )

(21, 2, 1000) (r0, 7, 2L4)

\ 1 4 , 2 , 2 7 0 ) (20,2, L55) (23, 2, 369) (35, 2, 351) ( 5 , 3 , 1 5 0 ) (17, 2, 57)

(8, 10, 3200) ( 1 9 , 4 , 1 4 8 ) ( 9 , 2 , 7 6 9 ) ( 6 r , 2 , 2 0 8 ) (10, 2, g5g) ( r 9 , 4 , 8 4 6 ) (22, 3, 5000) (14, 3, 178) ( r 7 , 7 , L o l )

92.09 85.51 8 1 . 9 5 95.r4 76.24 85.8 72.40 70.90 67.29 80.00 8 1 . 9 4 85.33 89.L7 94.67 78.95 73.34 77.03 74.35 78.85 83.72 7 r . 0 4 76.44 94.38 93.07

94.43 8 7 . 1 0 8 5 . 3 7 96.42 8 1 . 5 2 86.23 76.30 73.40 7 6 . r 7 8 1 . 4 8 8 1 . 2 9 8 5 . 8 7 92.02 94.67 84.2r 73.66 79.05 76.69 78.85 93.84 7 4 . r r 83.30 96.07 93.07

95.43 85.22 8 5 . 3 7 a7 2F,

8 2 . 1 8 86.09 7 3 . 1 8 73.00 a , < n

80.74 83.87 8r.25 93.73 94,00 89,47 73.34 8 5 . 1 4 73,44 84.62 98.54 78.25 83.90 97.75 95.05

85.80 84.39 96.14 8 2 . 1 8 8 5 . 8 r o . r o 74.5 78.04 8 3 . 3 3 83.23 85.33 93.45 93.33 84.21 73.06 82.43 77.60 d b . b d

95.82 74.94 84.04 97.75 97.03

.4395

5.6.7 Empirical Comparison among Ensemble Methods

Table 5.5 shows the empirical results obtained when comparing the perfor-

mance of a decision tree classifier against bagging, boosting, and random for- est. The base classifiers used in each ensemble method consist of fifty decision trees. The classification accuracies reported in this table are obtained from ten-fold cross-validation. Notice that the ensemble cla,ssifi.ers generally out- perform a single decision tree classifier on many of the data sets.

5.7 Class Imbalance Problem

Data sets with imbalanced class distributions are quite common in many real applications. For example, an automated inspection system that monitors products that come off a manufacturing assembly line may find that the num-

5.7 Class Imbalance Problem 295

ber of defective products is significantly fewer than that of non-defective prod- ucts. Similarly, in credit card fraud detection, fraudulent transactions are outnumbered by legitimate transactions. In both of these examples, there is a disproportionate number of instances that belong to different classes. The degree of imbalance varies from one application to another-a manufacturing plant operating under the six sigma principle may discover four defects in a million products shipped to their customers, while the amount of credit card fraud may be of the order of 1 in 100. Despite their infrequent occurrences, a correct classification of the rare class in these applications ofben has greater value than a correct classification of the majority class. However, because the class distribution is imbalanced, this presents a number of problems to existing classification algorithms.

The accuracy measure, which is used extensively to compare the perfor- mance of classifiers, may not be well suited for evaluating models derived from imbalanced data sets. For example, if lTo of the credit card transactions are fraudulent, then a model that predicts every transaction as legitimate has an accuracy of 99% even though it fails to detect any of the fraudulent activities. Additionally, measures that are used to guide the learning algorithm (e.g., in- formation gain for decision tree induction) may need to be modified to focus on the rare class.

Detecting instances of the rare class is akin to finding a needle in a haystack. Because their instances occur infrequently, models that describe the rare class tend to be highly specialized. For example, in a rule-based classifier, the rules extracted for the rare class typically involve a large number of attributes and cannot be easily simplified into more general rules with broader coverage (unlike the rules for the majority class). Such models are also susceptible to the presence of noise in training data. As a result, many of the existing classification algorithms may not effectively detect instances of the rare class.

This section presents some of the methods developed for handling the class imbalance problem. F irst, alternative metrics besides accuracy are introduced, along with a graphical method called RoC analysis. we then describe how cost-sensitive Iearning and sampling-based methods may be used to improve the detection of rare classes.

5.7.t Alternative Metrics

Since the accuracy measure treats every class as equally important, it rnay not be suitable for analyzing imbalanced data sets, where the rare class is considered more interesting than the majority class. For binary classification, the rare class is often denoted as the positive class, while the majority class is

296 Chapter 5 Classification: Alternative Techniques

Table 5.6. A confusion matrix for a binary classification problem in which the classes are not equally imoortant.

Predicted Class

-f

Actual

Class

+ /++ (TP) /+- (FN)

/-+ (FP) /-- (rN)

denoted as the negative class. A confusion matrix that summarizes the number of instances predicted correctly or incorrectly by a classification model is shown in Table 5.6.

The following terminology is often used when referring to the counts tab- ulated in a confusion matrix:

o TYue positive (TP) or fi1, which corresponds to the number of positive

examples correctly predicted by the classification model.

o False negative (FN) or fi-, which corresponds to the number of positive

examples wrongly predicted as negative by the classification model.

o False positive (FP) or /-1, which corresponds to the number of negative examples wrongly predicted as positive by the classification model.

o TYue negative (TN) or /--, which corresponds to the number of negative examples correctly predicted by the classification model.

The counts in a confusion matrix can also be expressed in terms of percentages.

The true positive rate (TPR) or sensitivity is defined as the fraction of positive examples predicted correctly by the model, i.e.,

T P R : T P I Q P + r ' N ) .

Similarly, the true negative rate (?l/R) or specificity is defined as the fraction of negative examples predicted correctly by the model, i.e.,

T N R : T N / ( T N + F p ) .

Finally, the false positive rate (FPR) is the fraction of negative examples predicted as a positive class, i.e.,

FPR: FPIQN + FP),

5.7 Class Imbalance Problem 297

while the false negative rate (F N R) is the fraction of positive examples predicted as a negative class, i.e.,

F N R : F N I ( T P + - F ' N ) .

Recall and precision are two widely used metrics employed in applica- tions where successful detection of one of the classes is considered more signif- icant than detection of the other classes. A formal definition of these met,rics is given below.

P r e c i s i o n , p : TP

T P + F P TP

Recall, r :

( 5 . 7 4 )

( 5 . 7 5 ) T P + F N

Precision determines the fraction of records that actually turns out to be positive in the group the classifier has declared as a positive class. The higher the precision is, the lower the number of false positive errors committed by the classifier. Recall measures the fraction of positive examples correctly predicted by the classifier. Classifiers with large recall have very few positive examples misclassified as the negative class. In fact, the value of recall is equivalent to the true positive rate.

It is often possible to construct baseline models that maximize one metric but not the other. For example, a model that declares every record to be the positive class will have a perfect recall, but very poor precision. Conversely, a model that assigns a positive class to every test record that matches one of the positive records in the training set has very high precision, but low rer:all. Building a model that maximizes both precision and recall is the key challenge of classification algorithms.

Precision and recall can be summarized into another metric known as the Fr meaSure.

( 5 . 7 6 )

In principle, .F'1 represents a harmonic mean between recall and precision, i.e.,

The harmonic mean of two numbers z and gr tends to be closer to the smaller of the two numbers. Hence, a high value of F1-measure ensures that both

2 r p 2 x T P - r + p 2 x T P + F P + f ' l ' r

, a l - 1 1 .

I I I

i - p

298 Chapter 5 Classification: Alternative Techniques

precision and recall are reasonably high. A comparison among harmonic, ge-

ometric, and arithmetic means is given in the next example.

Example 5.8. Consider two positive numbers a,: 1 and b: 5. Their arith-

m e t i c m e a n i s L r o : ( a + b ) 1 2 : 3 a n d t h e i r g e o m e t r i c m e a n i s F g : \ / o b : 2.236. Their harmonic mean is p,h : (2xlx5) 16 : L.667, which is closer to the

smaller value between o and b than the arithmetic and geometric means. r

More generally, the FB measure can be used to examine the tradeoff be-

tween recall and precision:

/ ^ t a \

D \lt- + tlrp , ^ qr + p - p

( P ' + r ) x T P (5.77)

@ 2 + t ) r p + p 2 F P + F r \ /

Both precision and recall are special cases of FB by setting 0 :0 and B : 66,

respectively. Low values of B make Fp closer to precision, and high values make it closer to recall.

A more general metric that captures .F-B as well as accuracy is the weighted accuracy measure, which is defined by the following equation:

Weighted &ccltro,c/: wtTP -f utTN

( 5 . 7 8 ) utTP * utzF P + unF N + u)4T N'

The relationship between weighted accuracy and other performance metrics is

summarized in the following table:

Measure lal 1.D2 wJ w4 Recall Precision FB Accuracy

1 1

9 2 + t 1

I 0 13' 1

0 1 1 1

0 0 0 1 I

5.7.2 The Receiver Operating Characteristic Curve

A receiver operating characteristic (ROC) curve is a graphical approach for

displaying the tradeoll between true positive rate and false positive rate of a

classifier. In an ROC curve, the true positive rate (TPR) is plotted along the g axis and the false positive rate (FPR) is shown on the r axis. Each point

along the curve corresponds to one of the models induced by the classifier. Figure 5.41 shows the ROC curves for a pair of classifiers, M1 and M2.

M z . '

4

M 1

[ , / I

,/, ,, ,/ .'

l '

5.7 Class Imbalance Problem 299

0 . 1 0 . 2 0 . 3 0 . 4 0 . 5 0 . 6 0 . 7 0 . 8 0 . 9 False Positive Rate

Figure 5.41. ROC curves for two different classifiers.

There are several critical points along an ROC curve that have well-known interpretations:

(TPR:O, FPR:0): Model predicts every instance to be a negative class. (TPR:l, FPR:I): Model predicts every instance to be a positive class. (TPR:l, FPR:O): The ideal model.

A good classification model should be located as close as possible to the up- per left corner of the diagram, while a model that makes random guesses should reside along the main diagonal, connecting the points (TPR:0,FPR:0) and (TPR: I,FPR:1). Random guessing means that a record is classi- fied as a positive class with a fixed probability p, irrespective of its attribute set. For example, consider a data set that contains na positive instances and n- negative instances. The random classifier is expected to correctly classify pna of the positive instances and to misclassify pn- of the negative instances. Therefore, the TPR of the classifier is (pn)lnt : pt while its FPRis (pn,)/p - p. Since theTPR and FPR are identical, the ROC curve for a random classifier always reside along the main diagonal.

An ROC curve is useful for comparing the relative performance among different classifiers. In Figure 5.4I, Ml is better than M2 when f'PE is less

o (d

E c)

o o(L c) 5

0.9

o.7

u . o

u . c

0.4

0 . 1

0

300 Chapter 5 Classification: Alternative Techniques

than 0.36, while Mz is superior when FPIB is greater than 0.36. Clearly, neither of these two classifiers dominates the other.

The area under the ROC curve (AUC) provides another approach for eval-

uating which model is better on average. If the model is perfect, then its area

under the ROC curve would equal 1. If the model simply performs random guessing, then its area under the ROC curve would equal 0.5. A model that

is strictly better than another would have a larger area under the ROC curve.

Generating an ROC curve

To draw an ROC curve, the classifier should be able to produce a continuous- valued output that can be used to rank its predictions, from the most likely

record to be classified as a positive class to the least likely record. These out- puts may correspond to the posterior probabilities generated by a Bayesian classifier or the numeric-valued outputs produced by an artificial neural net-

work. The following procedure can then be used to generate an ROC curve:

Assuming that the continuous-valued outputs are defined for the positive

class, sort the test records in increasing order of their output values.

Select the lowest ranked test record (i.e., the record with lowest output value). Assign the selected record and those ranked above it to the positive class. This approach is equivalent to classifying all the test records as positive class. Because all the positive examples are classified correctly and the negative examples are misclassified, TPR: FPR: I.

Select the next test record from the sorted list. Classify the selected record and those ranked above it as positive, while those ranked below it

as negative. Update the counts of TP and FP by examining the actual class label of the previously selected record. If the previously selected record is a positive class, the TP count is decremented and the FP count remains the same as before. If the previously selected record is a negative class, the FP count is decremented and TP cotnt remains the same as before.

Repeat Step 3 and update theTP and FP counts accordingly until the highest ranked test record is selected.

PIot the TPR against FPR of the classifier.

Figure 5.42 shows an example of how to compute the ROC curve. There are five positive examples and five negative examples in the test set. The class

1 .

2 .

3 .

4 .

5 .

5 4 4 o J 3 2 2 1 0

5 c 4 4 2 'I 0 0 0

TN 0 0 1 1 J 4 4 5 5

FN 0 1 1 2 2 2 3 3 4 5

TPR I 0 . 8 0 . 8 u . o 0 . 6 0 . 6 0 . 6 o.4 o.4 0 . 2 0 FPR 'I 1 0 . 8 0 . 8 0 6 o.4 o.2 o.2 0 0 0

5.7 Class Imbalance Problem 301-

Figure5.42. Constructing an ROC curve.

Figure 5,43. BOC curve for the data shown in Figure 5.42.

Iabels of the test records are shown in the first row of the table. The second row corresponds to the sorted output values for each record. For example, they may correspond to the posterior probabilities P(*lx) generated by a naive Bayes classifier. The next six rows contain the counts of.TP, FP,TN, and f.ly', along with their corresponding TPR and FPR. The table is then filled from left to right. Initially, all the records are predicted to be positive. Thus, TP : FP :5 and TPR : FPR : 1. Next, we assign the test record with the lowest output value as the negative class. Because the selected record is actually a positive example, the TP count reduces from 5 to 4 and the FP count is the same as before. The FPR and TPR are updated accordingly. This process is repeated until we reach the end of the list, where TPR : 0 and FPR:0. The ROC curve for this example is shown in Figure 5.43.

3O2 Chapter 5 Classification: Alternative Techniques

5.7.3 Cost-Sensitive Learning

A cost matrix encodes the penalty of classifying records from one class as another. LeL C(i,,j) denote the cost of predicting a record from class i as class j. With this notation, C(1, -) is the cost of committing a false negative error, while C(-, +) is the cost of generating a false alarm. A negative entry in the

cost matrix represents the reward for making correct classification. Given a collection of ly' test records, the overall cost of a model M is

C { M ) : T P x C ( + , + ) + F P x C ( - , + ) + F , ^ r x C ( * , - )

+ T,^/ x C(-, -). (5.79)

U n d e r t h e 0 / 1 c o s t m a t r i x , i . e . , C ( + , * ) : C ( - , - ) : 0 a n d C ( + , , - 1 :

C(-, +) : 1, it can be shown that the overall cost is equivalent to the number of misclassification errors.

Ct(M) : 0 x gP +7,^/) + 1 x (FP+f.lr) : N x Err, ( 5 . 8 0 )

where Err is the error rate of the classiher.

Example 5.9. Consider the cost matrix shown in Table 5.7: The cost of committing a false negative error is a hundred times larger than the cost of committing a false alarm. In other words, failure to detect any positive

example is just as bad as committing a hundred false alarms. Given the classification models with the confusion matrices shown in Table 5.8, the total cost for each model is

C t ( M ) : 1 5 0 x ( - 1 ) * 6 0 x 1 + 4 0 x 1 0 0 : 3 9 1 0 ,

Ct(Mz) : 250 x (-1) * 5 x 1 -f 45 x 100 : 4255.

Table 5,7. Cost matrix for Example 5.9.

Predicted Class Ulass : * Class : -

Actual Class

Class : f - l 1 0 0 UIaSS : - I (.)

5.7 Class Imbalance Problem 3OB

Table 5,8. Confusion matrix for two classification models. Model M1 Predicted Olass

UIASS + UIASS -

Actual Class

Class + 1 5 0 40 UIaSS - 60 250

Model M2 Predicted Class Ulass + Ulass -

Actual Class

Class + 250 45 Class - 200

Notice that despite improving both of its true positive and false positive counts, model Mz is still inferior since the improvement comes at the expense of in- creasing the more costly false negative errors. A standard accuracy measure would have preferred model M2 over M1. I

A cost-sensitive classification technique takes the cost matrix into consid- eration during model building and generates a model that has the lowest cost. For example, if false negative errors are the most costly, the learning algorithm will try to reduce these errors by extending its decision boundary toward the negative class, as shown in Figure 5.44. In this way, the generated model can cover more positive examples, although at the expense of generating additional false alarms.

Figure 5.44. Modifying the decision boundary (from 81 to F2)to reduce the false negative errors of a classifier.

There are various ways to incorporate cost information into classification algorithms. For example, in the context of decision tree induction, the cost

3O4 Chapter 5 Classification: Alternative Techniques

information can be used to: (1) choose the best attribute to use for splitting the data, (2) determine whether a subtree should be pruned, (3) manipulate the weights ofthe training records so that the learning algorithm converges to a decision tree that has the lowest cost, and (4) modify the decision rule at each leaf node. To illustrate the last approach, let p(ilt) denote the fraction of training records from class i that belong to the leaf node t. A typical decision rule for a binary classification problem assigns the positive class to node t if the following condition holds.

The preceding decision rule suggests that the class label of a leaf node depends on the majority class of the training records that reach the particular node. Note that this rule assumes that the misclassification costs are identical for both positive and negative examples. This decision rule is equivalent to the expression given in Equation 4.8 on page 165.

Instead of taking a majority vote, a cost-sensitive algorithm assigns the class label e to node t if it minimizes the following expression:

:=+ ==+

+

p(+lt)C(+, -)

+ p(+lt)C(+, -)

p(+lt) > p(-lt)

e 1 l t ) > ( 1 - e ( + l r ) ) zp(+lt) > 1 p(+lt) > 0.5. ( 5 . 8 1 )

( 5 . 8 2 )c (i,lt) -- D e(j lt) c (j, i.). J

In the case where C(+,+) : C(-,-) : 0, a leaf node f is assigned to the positive class if:

+ p ( + l t ) >

> ' p ( - l t ) C ( - , + ) > ( 1 - e ( + l r ) ) c ( - , + ) c ( - , + )

( 5 . 8 3 ) C ( - , + ) + C ( + , - ) '

This expression suggests that we can modify the threshold of the decision rule from 0.5 to C(-,+)lQe, +)+ C(+, -)) to obtain a cost-sensitive classifier. If C(-,+) < C(+,-), then the threshold will be less than 0.5. This result makes sense because the cost of making a false negative error is more expensive than that for generating a false alarm. Lowering the threshold will expand the decision boundary toward the negative class, as shown in Figure 5.44.

Class Imbalance Problem 305

X

(a) Without oversampling (b) With oversampling

Figute 5.45. lllustrating the effect of oversampling of the rare class.

5.7.4 Sampling-Based Approaches

Sampling is another widely used approach for handling the class imbalance problem. The idea of sampling is to modify the distribution of instances so that the rare class is well represented in the training set. Some of the available techniques for sampling include undersampling, oversampling, and a hybrid of both approaches. To illustrate these techniques, consider a data set that contains 100 positive examples and 1000 negative examples.

In the case of undersampling, a random sample of 100 negative examples is chosen to form the training set along with all the positive examples. One potential problem with this approach is that some of the useful negative exam- ples may not be chosen for training, therefore, resulting in a less than optimal model. A potential method to overcome this problem is to perform undersam- pling multiple times and to induce multiple classifiers similar to the ensemble Iearning approach. Focused undersampling methods may also be used, where the sampling procedure makes an informed choice with regard to the nega- tive examples that should be eliminated, e.g., those located far away from the decision boundary.

Oversampling replicates the positive examples until the training set has an equal number of positive and negative examples. Figure 5.45 illustrates the effect of oversampling on the construction of a decision boundary using a classi- fier such as a decision tree. Without oversampling, only the positive examples at the bottom right-hand side of Figure 5.45(a) are classified correctly. The positive example in the middle of the diagram is misclassified because there

D . J

N

X1

306 Chapter 5 Classification: Alternative Techniques

are not enough examples to justify the creation of a new decision boundary to separate the positive and negative instances. Oversampling provides the additional examples needed to ensure that the decision boundary surrounding the positive example is not pruned, as illustrated in Figure 5.45(b).

However, for noisy data, oversampling may cause model overfitting because some of the noise examples may be replicated many times. In principle, over* sampling does not add any new information into the training set. Replication of positive examples only prevents the learning algorithm from pruning certain parts of the model that describe regions that contain very few training exam- ples (i.e., the small disjuncts). The additional positive examples also tend to increase the computation time for model building.

The hybrid approach uses a combination of undersampling the majority class and oversampling the rare class to achieve uniform class distribution. Undersampling can be performed using random or focused subsampling. Over- sampling, on the other hand, can be done by replicating the existing positive

examples or generating new positive examples in the neighborhood of the ex- isting positive examples. In the latter approach, we must first determine the k-nearest neighbors for each existing positive example. A new positive ex- ample is then generated at some random point along the line segment that joins the positive example to one of its k-nearest neighbors. This process is repeated until the desired number of positive examples is reached. Unlike the data replication approach, the new examples allow us to extend the decision boundary for the positive class outward, similar to the approach shown in Fig' ure 5.44. Nevertheless, this approach may still be quite susceptible to model overfitting.

5.8 Multiclass Problem

Some of the classification techniques described in this chapter, such as support vector machines and AdaBoost, are originally designed for binary classification problems. Yet there are many real-world problems, such as character recogni- tion, face identification, and text classification, where the input data is divided into more than two categories. This section presents several approaches for extending the binary classifiers to handle multiclass problems. To illustrate t h e s e a p p r o a c h e s , l e t Y : { A r , y z , . . . , A K } b e t h e s e t o f c l a s s e s o f t h e i n p u t data.

The first approach decomposes the multiclass problem into K binary prob- lems. For each class At e Y , a binary problem is created where aII instances that belong to 96 are considered positive examples, while the remaining in-

Multiclass Problem 307

stances are considered negative examples. A binary classifier is then con- structed to separate instances of class y6 from the rest of the classes. This is known as the one-against-rest (1-r) approach.

The second approach, which is known as the one-against-one (1-1) ap- proach, constructs K(K - L)/2binary classifiers, where each classifier is used to distinguish between a pair of classes, (Ao,Ai). Instances that do not belong to either Ai or yj are ignored when constructing the binary classifier for (gi,yi). In both 1-r and 1-1 approaches, a test instance is classified by combining the predictions made by the binary classifiers. A voting scheme is typically em- ployed to combine the predictions, where the class that receives the highest number of votes is assigned to the test instance. In the 1-r approach, if an instance is classified as negative, then all classes except for the positive class receive a vote. This approach, however, may lead to ties among the different classes. Another possibility is to transform the outputs of the binary classifiers into probability estimates and then assign the test instance to the class that has the highest probability.

Example 5.10. Consider a multiclass problem where Y : {Ar,,A2,Az,A4}. Suppose a test instance is classified as (+, -, -, -) according to the 1-r ap- proach. In other words, it is classified as positive when 91 is used as the positive class and negative when 92, 93, and !4 are used as the positive class. Using a simple majority vote, notice that gfl receives the highest number of votes, which is four, while the remaining classes receive only three votes. The test instance is therefore classified as gr.

Suppose the test instance is classified as follows using the 1-1 approach:

Binary pair of classes

* i U t - i A z

l , a t - i

A z

l : a t - i U +

+ i A 2 - i u s

+" a2 - i

U +

*: at ' y 4

Classification -T- + + +

The first two rows in this table correspond to the pair of classes (AtAi) chosen to build the classifier and the last row represents the predicted class for the test instance. After combining the predictions, 91 and 94 each receive two votes, while 92 and gr3 each receives only one vote. The test instance is therefore classified as either At or A+, depending on the tie-breaking procedure. I

Error-Correcting Output Coding

A potential problem with the previous two approaches is that they are sensitive to the binary classification errors. For the 1-r approach given in Example 5.10,

5 . 8

308 Chapter 5 Classification: Alternative Techniques

if at least of one of the binary classifiers makes a mistake in its prediction, then the ensemble may end up declaring a tie between classes or making a wrong prediction. For example, suppose the test instance is classified as (+, -, *, -)

due to misclassification by the third classifier. In this case, it will be difficult to tell whether the instance should be classified as 91 or 93, unless the probability

associated with each class prediction is taken into account. The error-correcting output coding (ECOC) method provides a more ro-

bust way for handling multiclass problems. The method is inspired by an information-theoretic approach for sending messages across noisy channels. The idea behind this approach is to add redundancy into the transmitted message by means of a codeword, so that the receiver may detect errors in the received message and perhaps recover the original message if the number of erro s is small.

For multiclass learning, each class ga is represented by a unique bit string of length n known as its codeword. We then train n binary classifiers to predict

each bit of the codeword string. The predicted class of a test instance is given by the codeword whose Hamming distance is closest to the codeword produced

by the binary classifiers. Recall that the Hamming distance between a pair of bit strings is given by the number of bits that differ.

Example 5.11. Consider a multiclass problem where Y : {At,Az,ys,Aa}. Suppose we encode the classes using the following 7-bit codewords:

Class Codeword

At Ut

UI U+

1 0 0 0

1

0 0 1

1 0 1 0

1 0 1 1

1 1 0 0

1 1 0 I

1 1 I 0

Each bit of the codeword is used to train a binary classifier. If a test instance is classified as (0,1,1,1,1,1,1) by the binary classifiers, then the Hamming dis- tance between the codeword and gr1 is 1, while the Hamming distance to the remaining classes is 3. The test instance is therefore classified as 91. I

An interesting property of an error-correcting code is that if the minimum Hamming distance between any pair of codewords is d, then any t@

- L) 12)) errors in the output code can be corrected using its nearest codeword. In Example 5.11, because the minimum Hamming distance between any pair of codewords is 4, the ensemble may tolerate errors made by one of the seven

5.9 Bibliographic Notes 309

binary classifiers. If there is more than one classifier that makes a mistake, then the ensemble may not be able to compensate for the error.

An important issue is how to design the appropriate set of codewords for different classes. FYom coding theory, a vast number of algorithms have been developed for generating n-bit codewords with bounded Hamming distance. However, the discussion of these algorithms is beyond the scope of this book. It is worthwhile mentioning that there is a significant difference between the design of error-correcting codes for communication tasks compared to those used for multiclass learning. For communication, the codewords should max- imize the Hamming distance between the rows so that error correction can be performed. Multiclass learning, however, requires that the row-wise and column-wise distances of the codewords must be well separated. A larger column-wise distance ensures that the binary classifiers are mutually indepen- dent, which is an important requirement for ensemble learning methods.

5.9 Bibliographic Notes

Mitchell [208j provides an excellent coverage on many classification techniques from a machine learning perspective. Extensive coverage on classification can also be found in Duda et al. [180], Webb [219], Fukunaga [187], Bishop [159], Hastie et al. [192], Cherkassky and Mulier [167], Witten and F]ank [221], Hand et al. [190], Han and Kamber [189], and Dunham [181].

Direct methods for rule-based classifiers typically employ the sequential covering scheme for inducing classification rules. Holte's 1R [195] is the sim- plest form of a rule-based classifier because its rule set contains only a single rule. Despite its simplicity, Holte found that for some data sets that exhibit a strong one-to-one relationship between the attributes and the class label, lR performs just as well as other classifiers. Other examples of rule-based classifiers include IREP [184], RIPPER [170], CN2 1168, 1691, AQ [207], RISE

[176], and ITRULE [214]. Table 5.9 shows a comparison of the characteristics of four of these classifiers.

For rule-based classifiers, the rule antecedent can be generalized to include any propositional or first-order logical expression (e.g., Horn clauses). Read- ers who are interested in first-order logic rule-based classifiers may refer to references such as [208] or the vast literature on inductive logic programming

1209]. Quinlan [211] proposed the C4.5rules algorithm for extracting classifi- cation rules from decision trees. An indirect method for extracting rules from artificial neural networks was given by Andrews et al. in [157].

310 Chapter 5 Classification: Alternative Techniques

Table 5,9. Comparison of various rule-based classifiers, RIPPER u1\ z

(unordered) L ] N 2

(ordered)

AQR

ttule-growlng strategy

Generai-to- specific

L;eneral-to-

specific

General-tG. specific

General-to-specific (seeded by a

oositive examole)

Evaluation Metric

FOIL's Infb gain Laplace Itntropy ancl likelihood ratio

Number of true positives

Stopping condition for rule-erowinq

All examples belong to the

same class

No performance galn

No perfbrmance galn

Kules cover only positive class

Rule Pruning Keducect error Drunlnq

None None None

lnstance Elimination

Positive and negative

Positive only Positive only Positive and negatlve

btopprng condition for adding rules

tlrlor > bUTo oI

based on MDL

I\o perlormance garn

No perlormance garn

All positive examples are

covered

Kule uet Pruning

tfeplace or

modifv rules

:itatrstrcal tests

None None

Search stratesv Greedv Beam search Beam search Beam sea,rch

Cover and Hart [t72] presented an overview of the nearest-neighbor classi- fication method from a Bayesian perspective. Aha provided both theoretical and empirical evaluations for instance-based methods in [155]. PEBLS, which was developed by Cost and Salzberg [171], is a nearest-neighbor classification algorithm that can handle data sets containing nominal attributes. Each train- ing example in PEBLS is also assigned a weight factor that depends on the number of times the example helps make a correct prediction. Han et al. [188] developed a weight-adjusted nearest-neighbor algorithm, in which the feature weights are learned using a greedy, hill-climbing optimization algorithm.

Naive Bayes classifiers have been investigated by many authors, including Langley et al. [203], Ramoni and Sebastiani l2I2), Lewis [204], and Domingos andPazzani [178]. Although the independence assumption used in naive Bayes classifiers may seem rather unrealistic, the method has worked surpdsingly well for applications such as text classification. Bayesian belief networks provide a more flexible approach by allowing some of the attributes to be interdependent. An excellent tutorial on Bayesian belief networks is given by Heckerman in

[1e4]. Vapnik [2I7, 218] had written two authoritative books on Support Vector

Machines (SVM). Other useful resources on SVM and kernel methods include the books by Cristianini and Shawe-Taylor [173] and Scholkopf and Smola

5.9 Bibliographic Notes 311

[213]. There are several survey articles on SVM, including those written by Burges [164], Bennet et al. [158], Hearst 1193], and Mangasarian [205].

A survey of ensemble methods in machine learning was given by Diet- terich [174]. The bagging method was proposed by Breiman [161]. Reund and Schapire [186] developed the AdaBoost algorithm. Arcing, which stands for adaptive resampling and combining, is a variant of the boosting algorithm proposed by Breiman [162]. It uses the non-uniform weights assigned to train- ing examples to resample the data for building an ensemble of training sets. Unlike AdaBoost, the votes of the base classifiers are not weighted when de- termining the class label of test examples. The random forest method was introduced by Breiman in [163].

Related work on mining rare and imbalanced data sets can be found in the survey papers written by Chawla et al. [166] and Weiss 1220]. Sampling-based methods for mining imbalanced data sets have been investigated by many au- thors, such as Kubat and Matwin 1202), Japkowitz [196], and Drummond and Holte [179]. Joshi et al. [199] discussed the limitations of boosting algorithms for rare class modeling. Other algorithms developed for mining rare classes include SMOTE [165], PNrule [198], and CREDOS [200].

Various alternative metrics that are well-suited for class imbalanced prob- lems are available. The precision, recall, and F1-measure are widely used met- rics in information retrieval 1216]. ROC analysis was originally used in signal detection theory. Bradley [160] investigated the use of area under the ROC curve as a performance metric for machine learning algorithms. A method for comparing classifier performance using the convex hull of ROC curves was suggested by Provost and Fawcett in [210]. Ferri et al. lf85] developed a methodology for performing ROC analysis on decision tree classifiers. They had also proposed a methodology for incorporating area under the ROC curve (AUC) as the splitting criterion during the tree-growing process. Joshi [197] examined the performance of these measures from the perspective of analyzing rare classes.

A vast amount of literature on cost-sensitive learning can be found in the online proceedings of the ICML'2000 Workshop on cost-sensitive learn- ittg. The properties of a cost matrix had been studied by Elkan in [182]. Margineantu and Dietterich [206] examined various methods for incorporating cost information into the C4.5 learning algorithm, including wrapper meth- ods, class distribution-based methods, and loss-based methods. Other cost- sensitive learning methods that are algorithm-independent include AdaCost

[t83], Metacost [177], and costing [222].

3L2 Chapter 5 Classification: Alternative Techniques

Extensive literature is also available on the subject of multiclass learning. This includes the works of Hastie and Tibshirani [191], Allwein et al. 1156], Kong and Dietterich [201], and Tax and Duin [215]. The error-correcting output coding (ECOC) method was proposed by Dietterich and Bakiri [175]. They had also investigated techniques for designing codes that are suitable for solving multiclass problems.

Bibliography [155] D. W. Aha. A studg of instance-based algorithms for superuised learning tasks: mathe-

rnatical, empirical, and, psgchological eualuat'ions. PhD thesis, University of California, Irvine, 1990.

[156] E. L. Allwein, R. E. Schapire, and Y. Singer. Reducing Multiclass to Binary: A Unifying Approach to Margin Classifiers. Journal of Machine Learn'ing Research, I: 1 1 3 - 1 4 1 , 2 0 0 0 .

[157] R, Andrews, J. Diederich, and A. Tickle. A Survey and Critique of Techniques For Extracting Rules Flom Tbained Artificial Neural Networks. Knowledge Based, Sgstems, 8 ( 6 ) : 3 7 3 - 3 8 9 , 1 9 9 5 .

[158] K. Bennett and C. Campbell. Support Vector Machines: Hype or Hallelujah. SIGKDD Erp lorat'i, o n s, 2 (2) : L-13, 2000.

[159] C. M. Bishop. Neural Networks for Pattern Recognit'ion. Oxford University Press, O x f o r d . U . K . . 1 9 9 5 .

[160] A. P. Bradley. The use of the area under the ROC curve in the Evaluation of Machine Learning Algorithms. P attern Recogni,tion, 30(7) : 1 145-1 1 49, 1997.

[161] L. Breiman. Bagging Predictors. Mach,ine Lear"ning,24(.2):123 140, 1996.

[162] L. Breiman. Bias, Variance, and Arcing Classifiers. Technical Report 486, University of California, Berkeley, CA, 1996.

f163] L. Breiman. Random Forests. Mach'ine Learn'ing, 45(I):5-32,2001.

[164] C. J. C. Burges. A Ttrtorial on Support Vector Machines for Pattern Recognition. D ata Mining and, Knowled,g e Discox erg, 2(2) :t21-167, 1998.

[165] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artifi,cial Intelli,gence Research, 16:321- 357, 2002.

[166] N. V. Chawla, N. Japkowicz, and A. Kolcz. Editorial: Special Issue on Learning from Imbalanced Data Sets. SIGKDD Erplorat'ions, 6(1):1-6, 2004.

[167] V. Cherkassky and F. Mulier. Learn'ing frorn Data: Concepts, Theory, and Method,s. Wiley Interscience, 1998.

[168] P. Clark and R. Boswell. Rule Induction with CN2: Some Recent Improvements. In Machine Lear"ning: Proc. of the Sth European Conf. (EWSL-97/, pages 151-163, 1991.

[169] P. Clark and T. Niblett. The CN2 Induction Algorithm. Machine Learning, 3(4): 26L-283, 1989.

[170] W. W. Cohen. Fast Effective Rule Induction. In Proc. of the 12th Intl. Conf. on Mach'ine Leamting, pages 115-123, Tahoe City, CA, July 1995.

[171] S. Cost and S. Salzberg. A Weighted Nearest Neighbor Algorithm for Learning with Symbolic Features. Mach'ine Learn'ing, 10:57-78, 1993.

ll72l T. M. Cover and P. E. Hart. Nearest Neighbor Pattern Classification. Knowleilge Baseil Sgstems, 8(6):373-389, 1995.

Bibliography 313

[173] N. Cristianini and J. Shawe-Taylor. An Introduction to Support Vector Machines and Other Kernel-baseil Learning Method,s. Cambridge University Press, 2000.

f174] T. G. Dietterich. Ensemble Methods in Machine Learning. In Fi,rst IntI. Workshop on Multiple Classifi,er Sgsterns, Cagliari, Italy, 2000.

[175] T. G. Dietterich and G. Bakiri. Solving Multiclass Learning Problems via Error- Correcting Output Codes. Joum,al of Arti,ficial Intelligence Research,2:263-286, Lggl.

[176] P. Domingos. The RISE system: Conquering without separating. In Proc. of the 6th IEEE Intl. Conf. on Tools wi,th Artificial Intelligence, pages 704-707, New Orleans, LA, 1 9 9 4 .

lL77l P. Domingos. MetaCost: A General Method for Making Classifiers Cost-Sensitive. In Proc. of the Sth Intl. Conf. on Knowledge Discouery and. Data Mi,ning, pages 155-164, San Diego, CA, August 1999.

f1781 P. Domingos and M. Pazzani. On the Optimality of the Simple Bayesian Classifier under Zero-One Loss. Machine Leanri,ng,29(2-3):103 130, 1997.

[179] C. Drummond and R. C. Holte. C4.5, Class imbalance, and Cost sensitivity: Why under-sampling beats over-sampling. In ICML'2001 Workshop on Learning from Im- balanced Data Sets 1d Washington, DC, August 2003.

[180] R. O. Duda, P. E. Hart, and D. G. Stork. Pattern Classi,fication John Wiley & Sons, Inc., New York, 2nd edition, 2001.

f1811 M. H. Dunham. Data Mi.ni,ng: Introductory and Aduanced, Topics. Prentice Hall, 2002.

[182] C. Elkan. The Foundations of Cost-Sensitive Learning. ln Proc. of the 17th IntL Joint Conf. on Arti,ficial Intelligence, pages 973-978, Seattle, WA, August 2001.

[183] W. Fan, S. J. Stolfo, J. Zhang, and P. K. Chan. AdaCost: misclassification cost- sensitive boosting. In Proc. of the 16th Intl. Conf. on Mach,ine Learning, pages 97-105, Bled, Slovenia, June 1999.

[184] J. Fiirnkranz and G. Widmer. Incremental reduced error pruning. In Proc. of the 11th IntI. Conf . on Mach'ine Learning, pages 70-77, New Brunswick, NJ, July 1994.

[185] C. Ferri, P. Flach, and J. Hernandez-Orallo. Learning Decision TYees Using the Area Under the ROC Curve. In Proc. of the 19th IntL Conf. on Machine Leanting, pages 739-L46, Sydney, Australia, July 2002.

1186] Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and Sgstern Sciences,55(1):119- 1 3 9 , 1 9 9 7 .

[187] K. Fukunaga. Introd"uction to Stati,sti.cal Pattern Recognition. Academic Press, New York, 1990.

U88] E.-H. Han, G. Karypis, and V. Kumar. Text Categorization Using Weight Adjusted k-Nearest Neighbor Classification. In Proc. of the Sth Pacifi,c-Asia Conf. on Knowled,ge Discouerg and, Data Mining, Lyon, Fbance,2001.

f1891 J. Han and M. Kamber. Data Mining: Concepts and Techniques. Morgan Kaufmann Publishers, San FYancisco, 2001.

[190] D. J. Hand, H. Mannila, and P. Smyth. Pri,nci,ples of Data Mi,ning. MIT Press,2001.

[191] T. Hastie and R. Tibshirani. Classification by pairwise coupling. Annals of Statisti,cs, 26(2):451-a71, 1998.

f192] T. Hastie, R. Tibshirani, and J. H. Fbiedman. The Elements of Statistical Learning: Data Mining, Inference, PreiLiction. Springer, New York, 2001.

[193] M. Hearst. Ttends & Controversies: Support Vector Machines. IEEE Intetti,gent Systems, 13(4) : 18-28, 1998.

3L4 Chapter 5 Classification: Alternative Techniques

[194] D. Heckerman. Bayesian Networks for Data Mining. Data Mi,ni,ng and Knouledge D i s c o u e r g , 1 ( 1 ) : 7 9 L I 9 , L 9 9 7 .

1195] R. C. Holte. Very Simple Classification Rules Perform Well on Most Commonly Used Data sets. Machine Learn'ing, 11:63-91, 1993.

[196] N. Japkowicz. The Class Imbalance Problem: Significance and Strategies. In Proc. of the 2000 Intl. Conf. on Arti,ficial Intelli,gence: Special Tracle on Ind,uct'iue Learning, volume 1, pages lll-117, Las Vegas, NV, June 2000.

[197] M. V. Joshi. On Evaluating Performance of Classifiers for Rare Classes. In Proc. of the 2002 IEEE IntI. Conf. on Data Mining, Maebashi City, Japan, December 2002.

1198] M. V. Joshi, R. C. Agarwal, and V. Kumar. Mining Needles in a Haystack: Classifying Rare Classes via Two-Phase Rule Induction. In Proc. of 2001 ACM-SIGMOD IntI. Conf. on Managernent of Data, pages 91-102, Santa Barbara, CA, June 2001.

[199] M. V. Joshi, R. C. Agarwal, and V. Kumar. Predicting rare classes: can boosting make any weak learner strcing? In Proc. of the 8th IntI. Conf. on Knowledge Discouery and Data Mining, pages 297 306, Edmonton, Canada, JuIy 2002.

1200] M. V. Joshi and V. Kumar. CREDOS: Classification Using Ripple Down Structure (A Case for Rare Classes). ln Proc. of the SIAM IntI. Conf. on Data Min'ing, pages

32L-332, Orlando, FL, April 2004.

[201] E. B. Kong and T. G. Dietterich. Error-Correcting Output Coding Corrects Bias and Variance. In Proc. of the 12th IntI. Conf. on Machine Lear"ning, pages 313 321, Tahoe

City, CA, July 1995.

[202] M. Kubat and S. Matwin. Addressing the Curse of Imbalanced Ttaining Sets: One Sided Selection. In Proc. of the l/+th IntI. Conf. on Machine Lear"n'ing, pages 179-186, Nashville, TN, July 1997.

[203] P. Langley, W. Iba, and K. Thompson. An analysis of Bayesian classifiers. In Proc. oJ the 10th National Conf. on Artif,cial Intelligence, pages 223-228, 1992.

[204] D. D. Lewis. Naive Bayes at Forty: The Independence Assumption in Information Retrieval. In Proc. of the 10th European Conf. on Mach'i,ne Learning (ECML 1998), p a g e s 4 - 1 5 , 1 9 9 8 .

[205] O. Mangasarian. Data Mining via Support Vector Machines. Technical Report Tech- nical Report 01-05, Data Mining Institute, May 2001.

[206] D. D. Margineantu and T. G. Dietterich. Learning Decision Tlees for Loss Minimization in Multi-Class Problems. Technical Report 99-30-03, Oregon State University, 1999.

1207] R. S. Michalski, I. Mozetic, J. Hong, and N. Lavrac. The Multi-Purpose Incremental Learning System AQ15 and Its Testing Application to Three Medical Domains. In Proc.

of sth National Conf. on Arti,fici,al Intell'igence, Orlando, August 1986.

[208] T. Mitchell. Machine Learning. McGraw-Hill, Boston, MA, 1997.

f209] S. Muggleton. Found,ati,ons of Inducti,ue Log,ic Programm'ing. Prentice Hall, Englewood Cliffs. NJ, 1995.

[210] F. J. Provost and T. Fawcett. Analysis and Visualization of Classifier Performance: Comparison under Imprecise Class and Cost Distributions. In Proc. of the Srd, Intl. Conf. on Knowled,ge D'iscouery and Data M'in'ing, pages 43-48, Newport Beach, CA, August 1997.

[211] J. R. Quinlan. CI.S: Programs for Machi,ne Learn'ing. Morgan-Kaufmann Publishers, San Mateo. CA. 1993.

l2L2l M. Ramoni and P. Sebastiani. Robust Bayes classifierc. Arti.fi,cial Intelligence, I25: 209-226,200r.

5.10 Exercises 31-5

f213] B. Scholkopf and A. J. Smola. Learning with Kernels: Support Vector Machines,

Regularizati,on, Opt'i,mizat'ion, and, Begond. MIT Press, 2001.

[2I4) P. Smyth and R. M. Goodman. An Information Theoretic Approach to Rule Induction

from Databases. IEEE Trans. on Knowled,ge and" Data Engi'neering,4(4):301-316, 1992.

[215] D. M. J. Tax and R. P. W. Duin. Using Two-Class Classifiers for Multiclass Classi- fication. In Proc. of the 16th IntI. Conf. on Pattern Recogn'ition (ICPR 2002), pages

124-127, Quebec, Canada, August 2002.

f216] C. J. van Rijsbergen. Inforrnat'ion Retrieual. Butterworth-Heinemann, Newton, MA,

1 9 7 8 .

[217] V. Vapnik. The Nature of Statistical Learn'ing Theorg. Springer Verlag, New York,

1 9 9 5 .

[218] V. Vapnik. Statistical Learn'ing Theorg. John Wiley & Sons, New York, 1998.

[219] A. R. Webb. Statistical Pattern Recogni.tion. John Wiley & Sons, 2nd edition, 2002.

1220] G. M. Weiss. Mining with Rarity: A Unifying Flamework. SIGKDD Erplorations,6 ( l ) : 7 - L 9 , 2 0 0 4 .

l22l] I. H. Witten and E. FYank. Data Mining: Pract'ical Machine Learning Tools and

Techn'iques with Jaua Implementations. Morgan Kaufmann, 1999.

[222) B. Zadrozny, J. C. Langford, and N. Abe. Cost-Sensitive Learning by Cost- Proportionate Example Weighting. In Proc. of the 2003 IEEE IntI. Conf. on Data

M'i,n'ing, pages 435-442, Melbourne, FL, August 2003.

5.10 Exercises

1. Consider a binary classification problem with the following set of attributes and attribute values:

o Air Conditioner : {Working, Broken}

o Engine : {Good, Bad}

o Mileage : {High, Medium, Low}

o Rust : {yes, No}

Suppose a rule-based classifier produces the following rule set:

Mileage : HiSh _- Value : Low Mileage : Low ------+ Value : High Air Conditioner : Working, Engine : Good ----- Value : High Air Conditioner : Working, Engine : Bad ----+ Value : Low Air Conditioner : Brokel ---+ Value : Low

(a) Are the rules mutually exclustive?

316 Chapter 5 Classification: Alternative Techniques

Is the rule set exhaustive?

Is ordering needed for this set of rules?

Do you need a default class for the rule set?

2. The RIPPER algorithm (by Cohen [170]) is an extension of an earlier algorithm called IREP (by Fiirnkranz and Widmer 1184]). Both algorithms apply the reduced-error pruning method to determine whether a rule needs to be pruned. The reduced error pruning method uses a validation set to estimate the generalization error of a classifier. Consider the following pair of rules:

Rt: A ----- C R z : A n B - - - - - - + C

-R2 is obtained by adding a new conjunct, B, to the left-hand side of R1. For this question, you will be asked to determine whether -R2 is preferred over fi1 from the perspectives of rule-growing and rule-pruning. To determine whether a rule should be pruned, IREP computes the following measure:

n - L ( N - n ) urREp: -_fr1,, ,

where P is the total number of positive examples in the validation set, Iy' is the total number of negative examples in the validation set, p is the number of positive examples in the validation set covered by the rule, and n is the number of negative examples in the validation set covered by the rtIe. ulppp is actually similar to classification accuracy for the validation set. IREP favors rules that have higher values of u1pBp. On the other hand, RIPPER applies the following measure to determine whether a rule should be pruned:

u n t p p n n : ' - =- p + n

(a) Suppose -R1 is covered by 350 positive examples and 150 negative ex- amples, while R2 is covered by 300 positive examples and 50 negative examples. Compute the FOIL's information gain for the rule Rz with respect to R1.

(b) Consider a validation set that contains 500 positive examples and 500 negative examples. For R1, suppose the number of positive examples covered by the rule is 200, and the number of negative examples covered by the rule is 50. For rR2, suppose the number of positive examples covered by the rule is 100 and the number of negative examples is 5. Compute urREp for both rules. Which rule does IREP prefer?

(c) Compute uRrppEn for the previous problem. Which rule does RIPPER prefer?

(b)

(c)

(d)

4 .

5.10 Exercises 3L7

C4.5rules is an implementation of an indirect method for generating rules from a decision tree. RIPPER is an implementation of a direct method for generating rules directly from data.

(a) Discuss the strengths and weaknesses of both methods.

(b) Consider a data set that has a large difference in the class size (i.e., some classes are much bigger than others). Which method (between C4.5rules and RIPPER) is better in terms of finding high accuracy rules for the small classes?

Consider a training set that contains 100 positive examples and 400 negative examples. For each of the following candidate rules,

R1 A .-----+ * (covers 4 positive and I negative examples), Rz: B .-----+ * (covers 30 positive and 10 negative examples), R3: C ------+ * (covers 100 positive and 90 negative examples),

determine which is the best and worst candidate rule according to:

(a) Rule accuracy.

(b) FOIL's information gain.

(c) The likelihood ratio statistic.

(d) The Laplace measure.

(e) The m-estimate measure (with k :2 and p+ :0.2).

Figure 5.4 illustrates the coverage of the classification rules R1, ,R2, and R3. Determine which is the best and worst rule according to:

(a) The likelihood ratio statistic.

(b) The Laplace measure.

(c) The m-estimate measure (with k :2 and P+ :0.58).

(d) The rule accuracy after R1 has been discovered, where none of the exam- ples covered by R1 are discarded).

(e) The rule accuracy after rRl has been discovered, where only the positive

examples covered by iBl are discarded).

(f) The rule accuracy after Rl has been discovered, where both positive and negative examples covered by -Rl are discarded.

(a) Suppose the fraction of undergraduate students who smoke is 15% and the fraction of graduate students who smoke is 23%. If one-fifth of the college students are graduate students and the rest are undergraduates, what is the probability that a student who smokes is a graduate student?

5 .

6 .

Table 5.10. Data set for Exercise 7. Record A B C Class

1 2 e

t

o

7 8 9 1 0

0 0 0 0 0 1 1 1 1 I

0 0 I 1 0 0 0 0 1 0

0 1 1 I 1 1 1 I

1 1

+

+ +

+ +

318 Chapter 5 Classification: Alternative Techniques

(b) Given the information in part (a), is a randomly chosen college student more likely to be a graduate or undergraduate student?

(c) Repeat part (b) assuming that the student is a smoker.

(d) Suppose 30% of the graduate students live in a dorm but only l0% of the undergraduate students live in a dorm. If a student smokes and lives in the dorm, is he or she more likely to be a graduate or undergraduate student? You can assume independence between students who live in a dorm and those who smoke.

7. Consider the data set shown in Table 5.10

Estimate the conditional probabilities for P(Al-l_), P(Bi+), P(Cl+), P(Al-), P ( B l - ) , a n d P ( C l - ) .

Use the estimate of conditional probabilities given in the previous question to predict the class label for a test sample (A:0,8 - I,C :0) using the naive Bayes approach.

(c) Estimate the conditional probabilities using the m-estimate approach, w i t h p : I / 2 a n d m : 4 .

(d) Repeat part (b) using the conditional probabilities given in part (c).

(e) Compare the two methods for estimating probabilities. Which method is better and why?

8. Consider the data set shown in Table 5.11.

(a) Estimate the conditional probabilities for P(A : 1l+), P(B : 11a), P ( C : 1 l + ) , P ( . 4 : 1 l - ) , P ( B : 1 l - ) , a n d P ( C : 1 l - ) u s i n g t h e same approach as in the previous problem.

( a )

( b )

Table 5.11. Data set for Exercise 8. lnstance A B C Olass

I 2 J

4 5

t

7 8 q

1 0

0 1 0 1 1 0 1 0 0 1

0 0 1 0 0 0 I 0 1 1

1 1 0 0 I

1 0 0 0 I

+

+ +

+ +

5.10 Exercises 319

(b) Use the conditional probabilities in part (a) to predict the class label for

a test sample (A : l, B : l,C :1) using the naive Bayes approach.

( c ) C o m p a r e P ( A : 1 ) , P ( B : 1 ) , a n d P ( A : ! , 8 : 1 ) . S t a t e t h e r e l a t i o n - ships between A and B.

(d) Repeat the analysis in part (c) using P(A : l), P(B : 0), and P(A: 1 , , B : 0 ) .

( e ) C o m p a r e P ( A : I , B : I l C l a s s : * ) a g a i n s t P ( A : l l C l a s s : * ) a n d P(B : llClass: 1). Are the variables conditionally independent given

the class?

9. (a) Explain how naive Bayes performs on the data set shown in Figure 5.46.

(b) If each class is further divided such that there are four classes (AI, 42,

,B1, and B2), will naive Bayes perform better?

(c) How will a decision tree perform on this data set (for the two-class prob-

lem)? What if there are four classes?

10. Repeat the analysis shown in Example 5.3 for finding the location of a decision boundary using the following information:

(a) The prior probabilities are P(Crocodile):2 x P(lrrigator).

(b) The prior probabilities are P(Alligator):2 x P(Crocodile).

(c) The prior probabilities are the same, but their standard deviations are different; i.e., o(Crocodile) :4 and o(Alligator) : 2.

11. Figure 5.47 illustrates the Bayesian belief network for the data set shown in

Table 5.12. (Assume that all the attributes are binary).

(a) Draw the probability table for each node in the network.

Attributes

32O Chapter 5 Classification: Alternative Techniques

Figure 5.46, Data set for Exercise 9.

Figure 5.47, Bayesian belief network.

(b) Use the Bayesian network to compute P(Engine : Bad, Air Conditioner : Broken).

12. Given the Bayesian network shown in Figure 5.48, compute the following prob- abilities:

(a) P(B : good, F : empty, G : empty, 5 : yes).

(b) P(B - bad, F : empty? G : not empty, S : no).

(c) Given that the battery is bad, compute the probability that the car will start.

13. Consider the one-dimensional data set shown in Table 5.13.

Table 5,12. Data set for Exercise 11. Mileage Engine Air Conditioncr Number of Records

with Car Value:Hi Number of Records with Car Value:Lo

H Hi H i Hi Lo Lo Lo Lo

Good Good Bad Bad

G o o d Good Bad Bad

Working Broken

Working Broken

Working Broken

Working Broken

J

1 1 0 I

I 0

4

z K

A

0 1 2 z

5.1-0 Exercises 32L

P ( B = b a d ) = 0 . 1 P ( F = e m p t y ) = 0 . 2

(u)

P(S = no I B - good, F = not empty) = 0.1 P(S = no I 3 = good, F = emPtY) = 0.8 P(S = no I B = bad, F = not emPty) = 0.9 P(S = no I B = bad, F = emPty) = 1.0

Figure 5.48, Bayesian belief network for Exercise 12.

Classify the data point r : 5.0 according to its 1-, 3-, 5-, and 9-nearest neighbors (using majority vote).

Repeat the previous analysis using the distance-weighted voting approach described in Section 5.2.1.

( b )

14. The nearest-neighbor algorithm described in Section 5.2 can be extended to handle nominal attributes. A variant of the algorithm called PEBLS (Parallel Examplar-Based Learning System) by Cost and Salzberg ll7ll measures the distance between two values of a nominal attribute using the modified value difference metric (MVDM). Given a pair of nominal attribute values' V1 and

P(G = empty I B = good, F = not empV) = 0.1 P(G = empty I g = good, F = emPtV) = 0.8 P(G = empty I B = bad, F = not empty) = 0.2 P(G = empty I B = bad, F = empty) = 0.9

322 Chapter 5 Classification: Alternative Techniques

Table 5.13. Data set for Exercise 13. x U . D 3 . 0 4 . O 4 . 6 4 . 9 5 . 2 5 . 3 ( t 7 . 0 9 . 5 v + + -t- +

V2,lhe distance between them is defined as follows:

( 5 . 8 4 )

where nii is the number of examples from class i with attribute value Vi and n, is the number of examples with attribute value [.

Consider the training set for the loan classification problem shown in 5.9. Use the MVDM measure to compute the distance between every attribute values for the Home 0wner and Marital Status attributes.

For each of the Boolean functions given below, state whether the problem is linearly separable.

(a) A AND B AND C

(b) NOT A AND B

(") (,.+ oR B) AND (,4 oR C)

(d) (,4 xoR B) AND (A OR B)

(a) Demonstrate how the perceptron model can be used to represent the AND and OR functions between a pair of Boolean variables.

(b) Comment on the disadvantage of using linear functions as activation func- tions for multilayer neural networks.

You are asked to evaluate the performance of two classification models, M1 and M2. The test set you have chosen contains 26 binary attributes, labeled as ,4 throtgh Z.

Table 5.14 shows the posterior probabilities obtained by applying the models to the test set. (Only the posterior probabilities for the positive class are shown). As this is a two-class problem, P(-) : 1- P(+) and P(-lA, . . . , Z) : I - P(+lA, . . . , Z). Assume that we are mostly interested in detecting instances from the positive class.

(a) PIot the ROC curve for both M1 and M2. (You should plot them on the same graph.) Which model do you think is better? Explain your reasons.

(b) For model M1, suppose you choose the cutoffthreshold to be f :0.5. In other words, any test instances whose posterior probability is greater than C will be classified as a positive example. Compute the precision, recall, and F-measure for the model at this threshold value.

k 1

d . ( V t , v ) : t l L L - ' ' ' 1 , - | n r n z It - I I

Figure pair of

1 5 .

l o .

1 7 .

5.1-0 Exercises 323

Table 5.14. Posterior orobabilities for Exercise 17.

Instance Tlue Class P ( + \ A , . . . , Z , M r ) P ( + 1 , 4 , . . . , 2 , M 2 ) 1 2 3 4 5 6 7 8 q

10

+ +

+ +

+

0 . 0 . 6 9 0.44 0 . 5 5 0 . 6 7 0 . 4 7 0.08 0 . 1 5 0 . 4 5 0 . 3 5

0 . 6 1 0.03 0 . 6 8 0 . 3 1 0 . 4 5 0.09 0 . 3 8 0 . 0 5 0 . 0 1 0.04

1 8 .

(c) Repeat the analysis for part (c) using the same cutoff threshold on model

M2. Compare the F-measure results for both models. Which model is

better? Are the results consistent with what you expect from the ROC

curve?

(d) Repeat part (c) for model M1 using the threshold t : 0.1. Which thresh-

old do you prefer, t : 0.5 or f : 0.1? Are the results consistent with what

you expect from the ROC curve?

Following is a data set that contains two attributes, X and Y, and two class

labels. "+" and "-". Each attribute can take three different values: 0' 1, or 2.

The concept for the "+" class is Y : 1 and the concept for the "-" class is

X : 0 Y X : 2 .

(a) Build a decision tree on the data set. Does the tree capture ffts ((1"

3,nd tt-tt concepts?

X Y Number of Instances +

0 1 2 0 I 2 0 1 2

0 0 0 1

1 I 2 2 2

0 0 0

1 0 1 0 1 0 0 0 0

100 U

1 0 0 100

0 100 100

0 1 0 0

324 Chapter 5 Classification: Alternative Techniques

What are the accuracy, precision, recall, and F1-measure of the decision tree? (Note that precision, recall, and F1-measure are defined with respect to the "+" class.)

Build a new decision tree with the following cost function:

f 0 , i f i : i ; c ( i , j ) : 1 1 , i f i : * , j : - :

l ,*N'#B:ifii+fi#*tr , ir'i: -, j : +. (Hint: only the leaves of the old decision tree need to be changed.) Does the decision tree capture the "+" concept?

(d) What are the accuracy) precision, recall, and f'1-measure of the new deci- sion tree?

(a) Consider the cost matrix for a two-class problem. Let C(*, *) : C(-, -) : p , C ( + , - ) : C ( - , 1 ) : q , a n d q ) p . S h o w t h a t m i n i m i z i n g t h e c o s t function is equivalent to maximizing the classifier's accuracy.

(b) Show that a cost matrix is scale-invariant. For example, if the cost matrix is rescaled from C(i, j) ------ PC(i,j), where B is the scaling factor, the decision threshold (Equation 5.82) will remain unchanged.

(c) Show that a cost matrix is translation-invariant. In other words, adding a constant factor to all entries in the cost matrix will not affect the decision threshold (Equation 5.82).

Consider the task of building a classifier from random data, where the attribute values are generated randomly irrespective of the class labels. Assume the data set contains records from two classes, "+" and "-." Half of the data set is used for training while the remaining half is used for testing.

(a) Suppose there are an equal number of positive and negative records in the data and the decision tree classifier predicts every test record to be positive. What is the expected error rate of the classifier on the test data?

(b) Repeat the previous analysis assuming that the classifier predicts each test record to be positive class with probability 0.8 and negative class with probability 0.2.

(c) Suppose two-thirds of the data belong to the positive class and the re- maining one-third belong to the negative class. What is the expected error of a classifier that predicts every test record to be positive?

(d) Repeat the previous analysis assuming that the classifier predicts each test record to be positive class with probabllity 213 and negative class with orobabilitv 1/3.

(b)

(")

1 9 .

20.

5.10 Exercises 325

21. Derive the dual Lagrangian for the linear SVM with nonseparable data where the objective function is

r , - - - \ l l * l l ' , c r $ r . i 2" f ( w ) : , + u \ L e t ) .

22. Consider the XOR problem where there are four training points:

( 1 , 1 , - ) , ( 1 , 0 , + ) , ( 0 , 1 , + ) , ( 0 , 0 , - ) .

Tlansform the data into the following feature space:

iD : (1, Jirr, r/i*r, Jirrrr, "?,

*7).

Find the maximum margin linear decision boundary in the transformed space.

23. Given the data sets shown in Figures 5.49, explain how the decision tree, naive Bayes, and k-nearest neighbor classifiers would perform on these data sets.

326 Chapter 5 Classification: Alternative Techniques

Attrlbub

Noise Aflributes

l r l

t l I

1 l

Class A

l r r l

I

l l l l I

Clss B

Distinguishing Anribule6

(a) Synthetic data set 1.

(c) Synthetic data set 3.

(e) Synthetic data set 5.

( b ) S y n t h e t i c d a t a s e t 2 .

(d) Synthetic data set 4

(f) Synthetic data set 6.

Afrribuie6

Dislinguishing Anributes Noise Atribut€s

i:iii;:i;iii Class A " ' i i i

' t t

' ' , ,

l l t r

r r i i '

, , i , ' : ' i l r l

Class B

Attributes

Oistinguishing Attribute ret 1

Oislinguishing Attribute set 2 Noise Attribules

60% lilled with 1

40% filled with 1 C l a s s

40ol. filled wilh 1

60% filled with I

Clss

Class A Class B Class A Class B Class A

Class B Class A Class B Class A Class B

Class A Class B Cla$ A Class B Class A

Class B Class A Class B Class A Class B

Figure 5.49. Data set for Exercise 23.

Association Analysis: Basic Concepts and Algorithms

Many business enterprises accumulate large quantities of data from their day- to-day operations. For example, huge amounts of customer purchase data are collected daily at the checkout counters of grocery stores. Table 6.1 illustrates an example of such data, commonly known as market basket transactions. Each row in this table corresponds to a transaction, which contains a unique identifier labeled TID and a set of items bought by a given customer. Retail- ers are interested in analyzing the data to learn about the purchasing behavior of their customers. Such valuable information can be used to support a vari- ety of business-related applications such as marketing promotions, inventory management, and customer relationship management.

This chapter presents a methodology known as association analysis, which is useful for discovering interesting relationships hidden in large data sets. The uncovered relationships can be represented in the form of associa-

Table 6.1. An example of market basket transactions.

TID Items 1 2 3 4

tr

{Bread, Milk}

{Bread, Diapers, Beer, Eggs}

{Milk, Diapers, Beer, Cola}

{Bread, Milk, Diapers, Beer}

{Bread, Milk, Diapers, Cola}

328 Chapter 6 Association Analvsis

tion rules or sets of frequent items. For example, the following rule can be extracted from the data set shown in Table 6.1:

{liapers} --' {eeer}.

The rule suggests that a strong relationship exists between the sale of diapers and beer because many customers who buy diapers also buy beer. Retailers can use this type of rules to help them identify new opportunities for cross- selling their products to the customers.

Besides market basket data, association analysis is also applicable to other application domains such as bioinformatics, medical diagnosis, Web mining, and scientific data analysis. In the analysis of Earth science data, for example, the association patterns may reveal interesting connections among the ocean, land, and atmospheric processes. Such information may help Earth scientists develop a better understanding of how the different elements of the Earth system interact with each other. Even though the techniques presented here are generally applicable to a wider variety of data sets, for illustrative purposes) our discussion will focus mainly on market basket data.

There are two key issues that need to be addressed when applying associ- ation analysis to market basket data. First, discovering patterns from a large transaction data set can be computationally expensive. Second, some of the discovered patterns are potentially spurious because they may happen simply by chance. The remainder of this chapter is organized around these two is- sues. The first part of the chapter is devoted to explaining the basic concepts of association analysis and the algorithms used to efficiently mine such pat- terns. The second part of the chapter deals with the issue of evaluating the discovered patterns in order to prevent the generation of spurious results.

6.1 Problem Definition

This section reviews the basic terminology used in association analysis and presents a formal description of the task.

Binary Representation Market basket data can be represented in a binary format as shown in Table 6.2, where each row corresponds to a transaction and each column corresponds to an item. An item can be treated as a binary variable whose value is one if the item is present in a transaction and zero otherwise. Because the presence of an item in a transaction is often considered more important than its absence, an item is an asymmetric binary variable.

\ Problem Definition 329

Table 6.2. A binary 0/1 representation of market basket data,

TID Bread Milk Diapers Beer Eggs Cola I 2 3 4

5

1 1 0 1 1

-l

0 1 1 1

0 1 1 1 1

0 I

1 I 0

0 1 0 0 0

0 0 1 0 I

This representation is perhaps a very simplistic view of real market basket data

because it ignores certain important aspects of the data such as the quantity

of items sold or the price paid to purchase them. Methods for handling such

non-binary data will be explained in Chapter 7.

Itemset and Support Count Let I : {h,i2,...,i'a} be the set of all items

in a market basket data and T : {h,t2,..-,t1"} be the set of all transactions'

Each transaction ti contains a subset of items chosen from 1. In association

analysis, a collection of zero or more items is termed an itemset. If an itemset

contains /c items, it is called a k-itemset. For instance, {Beer, Diapers, Mj-Ik}

is an example of a 3-itemset. The null (or empty) set is an itemset that does

not contain any items. The transaction width is defined as the number of items present in a trans-

action. A transaction fy is said to contain an itemset X if X is a subset of

t7. For example, the second transaction shown in Table 6.2 contains the item-

set {Bread, Diapers} but not {Bread, Milk}. An important property of an

itemset is its support count, which refers to the number of transactions that

contain a particular itemset. Mathematically, the support count, o(X), for an

itemset X can be stated as follows:

o ( X ) : l { t , l x C t i , t , i

where the symbol | . I denote the number of elements in a set. In the data set

shown in Table 6.2, the support count for {Beer, Diapers, Milk} is equal to

two because there are only two transactions that contain all three items.

Association Rule An association rule is an implication expression of the

form X , Y, where X and Y are disjoint itemsets, i.e., X )Y : @. The

strength of an association rule can be measured in terms of its support and

confidence. Support determines how often a rule is applicable to a given

6 . L

330 Chapter 6 Association Analysis

data set, while confidence determines how frequently items in Y appear in transactions that contain X. The formal definitions of these metrics are

Support, s(X ------+ I/) :

Confidence, c(X --+ Y) :

o ( X u Y ) . .Atr

1

o ( X u Y )

"(x)

( 6 . 1 )

( 6 . 2 )

Example 6.1. Consider the rule {Uitt, Diapers} -----* {eeer}. Since the support count for {ltitt<, Diapers, Beer} is 2 and the total number of trans- actions is 5, the rule's support is 2f 5 :0.4. The rule's confidence is obtained by dividing the support count for {ttitt<, Diapers, Beer} by the support count for {Uitt<, Diapers}. Since there are 3 transactions that contain milk and di- apers, the confidence for this rule is 213: 0.67. I

Why Use Support and Confidence? Support is an important measure because a rule that has very low support may occur simply by chance. A low support rule is also likely to be uninteresting from a business perspective because it may not be profitable to promote items that customers seldom buy together (with the exception of the situation described in Section 6.8). For these reasons, support is often used to eliminate uninteresting rules. As will be shown in Section 6.2.1, support also has a desirable property that can be exploited for the efficient discovery of association rules.

Confidence, on the other hand, measures the reliability of the inference made by a rule. For a given rule X > Y, the higher the confidence, the more Iikely it is for Y to be present in transactions that contain X. Confidence also provides an estimate of the conditional probability of Y given X.

Association analysis results should be interpreted with caution. The infer- ence made by an association rule does not necessarily imply causality. Instead, it suggests a strong co-occurrence relationship between items in the antecedent and consequent of the rule. Causality, on the other hand, requires knowledge about the causal and effect attributes in the data and typically involves rela- tionships occurring over time (e.g., ozone depletion leads to global warming).

Formulation of Association Rule Mining Problem The association rule mining problem can be formally stated as follows:

Definition 6.1 (Association Rule Discovery). Given a set of transactions 7, find all the rules having support ) minsup and confidence ) minconf , where m'insup and m'inconf are the corresponding support and confidence thresholds.

Problem Definition 331

A brute-force approach for mining association rules is to compute the sup- port and confidence for every possible rule. This approach is prohibitively expensive because there are exponentially many rules that can be extracted from a data set. More specifically, the total number of possible rules extracted from a data set that contains d items is

R : J d - 2 d + r + I . ( 6 . 3 )

The proof for this equation is left as an exercise to the readers (see Exercise 5 on page 405). Even for the small data set shown in Table 6.1, this approach requires us to compute the support and confidence for 36 - 27 * 1 : 602 rules. More than 80% of the rules are discarded after applying mi,nsup : 20Vo and minconf : 5070, thus making most of the computations become wasted. To avoid performing needless computations, it would be useful to prune the rules early without having to compute their support and confidence values.

An initial step toward improving the performance of association rule min- ing algorithms is to decouple the support and confidence requirements. From Equation 6.2, notice that the support of a rule X -----+ Y depends only on the support of its corresponding itemset , X U Y. For example, the following rules have identical support because they involve items from the same itemset,

{ B e e r , D i a p e r s , M i I k } :

{Beer, Diapers} ----*

{t'li.ft}, {Beer, Milk} ------

{Diapers}, {Diapers, Milk}

-----r {eeer}, {eeer} ----* {Diapers, Milk},

{uirt} ------ {Beer,Diapers}, {oiapers}

------+ {Beer,Milk}.

If the itemset is infrequent, then all six candidate rules can be pruned imme- diately without our having to compute their confidence values.

Therefore, a common strategy adopted by many association rule mining algorithms is to decompose the problem into two major subtasks:

1. FYequent Itemset Generation, whose objective is to find all the item- sets that satisfy Lhe mi,nsup threshold. These itemsets are called frequent itemsets.

2. Rule Generation, whose objective is to extract all the high-confidence rules from the frequent itemsets found in the previous step. These rules are called strong rules.

The computational requirements for frequent itemset generation are gen- erally more expensive than those of rule generation. Efficient techniques for generating frequent itemsets and association rules are discussed in Sections 6.2 and 6.3, respectively.

6 . 1

332 Chapter 6 Association Analysis

Figure 6.1. An itemset lattice.

6.2 Frequent Itemset Generation

A lattice structure can be used to enumerate the list of all possible itemsets. Figure 6.1 shows an itemset lattice for 1: {a,b,c.,d,e}.In general, a data set that contains k items can potentially generate up to 2k - 7 frequent itemsets, excluding the null set. Because k can be very large in many practical appli- cations, the search space of itemsets that need to be explored is exponentially Iarge.

A brute-force approach for finding frequent itemsets is to determine the support count for every candidate itemset in the lattice structure. To do this, we need to compare each candidate against every transaction, an opera- tion that is shown in Figure 6.2. If the candidate is contained in a transaction, its support count will be incremented. For example, the support for {Bread, Milk) is incremented three times because the itemset is contained in transac- tions 1, 4, and 5. Such an approach can be very expensive because it requires O(N Mw) comparisons, where l/ is the number of transacti ons, M : 2k - | is the number of candidate itemsets, and tl is the maximum transaction width.

Frequent Itemset Generation 333

Figure 6.2. Counting the support of candidate itemsets.

There are several ways to reduce the computational complexity of frequent itemset generation.

1. Reduce the number of candidate itemsets (M). The Apri'ori' prin'

ciple, described in the next section, is an effective way to eliminate some

of the candidate itemsets without counting their support values.

2. Reduce the number of comparisons. Instead of matching each can-

didate itemset against every transaction, we can reduce the number of

comparisons by using more advanced data structures, either to store the

candidate itemsets or to compress the data set. We will discuss these

strategies in Sections 6.2.4 and 6.6.

6.2.L The Apriori Principle

This section describes how the support measure helps to reduce the number

of candidate itemsets explored during frequent itemset generation. The use of

support for pruning candidate itemsets is guided by the following principle.

Theorem 6.I (Apriori Principle). If an'itemset'is frequent, then all of its

subsets must also be frequent.

To illustrate the idea behind the Apri,ore principle, consider the itemset

Iattice shown in Figure 6.3. Suppose {c, d, e} is a frequent itemset. Clearly,

any transaction that contains {c,d,e} must also contain its subsets, {",d},

{ " , " } , { d , e } , { " } , { d } , a n d { e } . A s a r e s u l t , i f { c , d , e } i s f r e q u e n t , t h e n

all subsets of {c, d,e} (i.e., the shaded itemsets in this figure) must also be

frequent.

6 . 2

Mt N

Y

/

334 Chapter 6 Association Analysis

Figure 6,3. An illustration ol lhe Aprioriprinciple. lt {c,d,e} is frequent, then all subsets of this itemset are frequent.

Conversely, if an itemset such as {a, b} is infrequent, then all of its supersets must be infrequent too. As illustrated in Figure 6.4, the entire subgraph containing the supersets of {o, b} can be pruned immediately once {a, b} is found to be infrequent. This strategy of trimming the exponential search space based on the support measure is known as support-based pruning. Such a pruning strategy is made possible by a key property of the support measure, namely, that the support for an itemset never exceeds the support for its subsets. This property is also known as the anti-monotone property of the support measure.

Definition 6.2 (Monotonicity Property). Let 1 be a set of items, and J :21 be the pov/er set of 1. A measure / is monotone (or upward closed) if

YX,Y e J : (X eY) ------, f(X) < f(Y),

6 . 2 Frequent Itemset Generation 335

Pruned \r

Supersets \\--_

Figure 6.4. An illustration of support-based pruning. lt {a,b} is infrequent, then allsupersets of {a,b} are infrequent.

which means that if X is a subset of Y, then /(X) must not exceed /(f). On

the other hand, / is anti-monotone (or downward closed) if

v x , Y e J : ( X q Y ) - f ( Y ) < f ( x ) ,

which means that if X is a subset of Y, then /(Y) must not exceed /(X).

Any measure that possesses an anti-monotone property can be incorpo- rated directly into the mining algorithm to effectively prune the exponential search space of candidate itemsets, as will be shown in the next section.

6.2.2 Fbequent Itemset Generation in the Apri'ori Algorithm

Apri,ori, is the first association rule mining algorithm that pioneered the use

of support-based pruning to systematically control the exponential growth of candidate itemsets. Figure 6.5 provides a high-level illustration of the frequent itemset generation part of the Apriori algorithm for the transactions shown in

336 Chapter 6 Association Analysis

Candidate 1-ltemsets

Minimum support count = 3

Itemsets removed because of low support

Candidate 3-ltemsets

Itemset Count {Bread, Diapers, Milk} 3

Figure 6.5. llluskation of frequent itemset generation using the Apriori algorithm.

Table 6.1. We assume that the support threshold is60To, which is equivalent to a minimum support count equal to 3.

Initially, every item is considered as a candidate l-itemset. Afber count- ing their supports, the candidate itemsets {Co1a} and {Eggs} are discarded because they appear in fewer than three transactions. In the next iteration, candidate 2-itemsets are generated using only the frequent 1-itemsets because the Apri,orz principle ensures that all supersets of the infrequent 1-itemsets must be infrequent. Because there are only four frequent 1-itemsets, the num- ber of candidate 2-itemsets generated by the algorithm is ( f ) : O. Two of these six candidates, {Beer, Bread} and {Beer, Milk}, are subsequently found to be infrequent after computing their support values. The remain- ing four candidates are frequent, and thus will be used to generate candidate 3-itemsets. Without support-based. pruning, there are ( ! ) : 2O candidate 3-itemsets that can be formed using the six items given in this example. With the Apri,ori principle, we only need to keep candidate 3-itemsets whose subsets are frequent. The only candidate that has this property is {Bread, Diapers, M i r k ) .

The effectiveness of the Apri,ore pruning strategy can be shown by count- ing the number of candidate itemsets generated. A brute-force strategy of

6.2 Freouent Itemset Generation 337

enumerating all itemsets (up to size 3) as candidates will produce

candidates. With the Apri,ori principle, this number decreases to

candidates, which represents a 68% reduction in the number of candidate itemsets even in this simple example.

The pseudocode for the frequent itemset generation part of the Apri,ori, algorithm is shown in Algorithm 6.1. Let Cp denote the set of candidate k-itemsets and Fr denote the set of frequent k-itemsets:

o The algorithm initially makes a single pass over the data set to determine the support of each item. Upon completion of this step, the set of all frequent 1-itemsets, f'r, will be known (steps 1 and 2).

o Next, the algorithm will iteratively generate new candidate k-itemsets using the frequent (k - 1)-itemsets found in the previous iteration (step

5). Candidate generation is implemented using a function called apriori- gen, which is described in Section 6.2.3.

Algorithm 6.1 Flequent itemset generation of the Apriori, algorithm. I : k : 1 .

2: F1" - { i I i e I no({t}) > lr' x minsup}. {Find all frequent l-itemsets} 3: repeat 4 : k : k + 7 . 5: Cr : apriori-gen(Fr-1). {Generate candidate itemsets} 6: for each transaction t eT do 7: Ct : subset(Cn, t). {Identify all candidates that belong to l} 8: for each candidate itemset c € C1 do 9: o(c) : o(c) + 1. {Increment support count}

10: end for 11: end for 12: Fr: { cl c€Cp Ao(c) ) N x mi.nsup}. {Extract the frequent /c-itemsets} 13: until F* : A 14: Result : UF*.

(?) . (!) . (:) :6 + 15 t20:4r

( l ) . ( i ) . 1 : 6 * 6 * 1 : 1 3

338 Chapter 6 Association Analysis

To count the support of the candidates, the algorithm needs to make an additional pass over the data set (steps 6 10). The subset function is used to determine all the candidate itemsets in Ca that are contained in each transaction f. The implementation of this function is described in Section 6.2.4.

After counting their supports, the algorithm eliminates all candidate itemsets whose support counts are less than mi,nsup (step 12).

The algorithm terminates when there are no new frequent itemsets gen- e r a t e d , i . e . , F p : 0 ( s t e p 1 3 ) .

The frequent itemset generation part of the Apri,orz algorithm has two im- portant characteristics. First, it is a level-wise algorithm; i.e., it traverses the itemset lattice one level at a time, from frequent 1-itemsets to the maximum size of frequent itemsets. Second, it employs a generate-and-test strategy for finding frequent itemsets. At each iteration, new candidate itemsets are generated from the frequent itemsets found in the previous iteration. The support for each candidate is then counted and tested against the minsup threshold. The total number of iterations needed by the algorithm is k-.* a 1, where k-r* is the maximum size of the frequent itemsets.

6.2.3 Candidate Generation and Pruning

The apriori-gen function shown in Step 5 of Algorithm 6.1 generates candidate itemsets by performing the following two operations:

1. Candidate Generation. This operation generates new candidate k- itemsets based on the frequent (k - l)-itemsets found in the previous iteration.

2. Candidate Pruning. This operation eliminates some of the candidate k-itemsets using the support-based pruning strategy.

To illustrate the candidate pruning operation, consider a candidate /c-itemset, X : { h , i 2 , . . . , i p } . T h e a l g o r i t h m m u s t d e t e r m i n e w h e t h e r a l l o f i t s p r o p e r s u b s e t s , X - { l i } ( V l : 1 , 2 , . . . , k ) , a r e f r e q u e n t . I f o n e o f t h e m i s i n f r e - quent, then X is immediately pruned. This approach can effectively reduce the number of candidate itemsets considered during support counting. The complexity of this operation is O(k) for each candidate k-itemset. However, as will be shown later, we do not have to examine all k subsets of a given candidate itemset. If m of the k subsets were used to generate a candidate, we only need to check the remaining k - rn subsets during candidate pruning.

6.2 Frequent Itemset Generation 339

In principle, there are many ways to generate candidate itemsets. The fol- Iowing is a list of requirements for an effective candidate generation procedure:

1. It should avoid generating too many unnecessary candidates. A candi- date itemset is unnecessary if at least one of its subsets is infrequent. Such a candidate is guaranteed to be infrequent according to the anti- monotone property of support.

2. It must ensure that the candidate set is complete, i.e., no frequent item- sets are left out by the candidate generation procedure. To ensure com- pleteness, the set of candidate itemsets must subsume the set of all fre- quent itemsets, i.e., Vk : fi, C Cp.

3. It should not generate the same candidate itemset more than once. For example, the candidate itemset {a,b,c,d} can be generated in many ways-by merging {a,b, c} with {d}, {b, d} with {o,

"}, {c} with {a,b, d},

etc. Generation of duplicate candidates leads to wasted computations and thus should be avoided for efficiency reasons.

Next, we will briefly describe several candidate generation procedures, in- cluding the one used by the apriori-gen function.

Brute-Force Method The brute-force method considers every k-itemset as a potential candidate and then applies the candidate pruning step to remove any unnecessary candidates (see Figure 6.6). The number of candidate item- sets generated at level k is equal to ( i, ), where d is the total number of items. Although candidate generation is rather trivial, candidate pruning becomes extremely expensive because a large number of itemsets must be examined. Given that the amount of computations needed for each candidate is O(k), the overall complexity of this method k O(Dfl:, tt

" (1")) : O(d .2d-t).

Fr-r x F1 Method An alternative method for candidate generation is to extend each frequent (k - 1)-itemset with other frequent items. Figure 6.7 illustrates how a frequent 2-itemset such as {Beer, Diapers} can be aug- mented with a frequent item such as Bread to produce a candidate 3-itemset

{Beer, Diapers, Bread}. This method will produce O(lFn_tl x lF1l) candi- date,k-itemsets, where l4l ir the number of frequent j-itemsets. The overall complexity of this step is O(DnklFn_tllF l)

The procedure is complete because every frequent k-itemset is composed of a frequent (k - 1)-itemset and a frequent 1-itemset. Therefore, all frequent k-itemsets are part of the candidate k-itemsets generated by this procedure.

34O Chapter 6 Association Analysis

Candidate Generation

Itemset Itreer, trreao, uora

iBeer, Bread, Diapers) {Beer, Bread, Milk} {Beer, Bread, Eqqs} {Beer, Cola, Diapers} {Beer, Cola, Milk} itseer, uola, Eggs'

{Beer, Diaoers. Milk} {Beer, Diapers, Eggs} {Beer, Milk, Eqqs}

{Bread, Cola, Diapers} {Bread, Cola, Milk}

{ D r e a o , u o r a , t r q q s l

{Bread, Diapers, Milk} {Breao, Drapers, Eqqsl {Bread, Milk, Eqqs}

lola, Diapers, Milk)

{Cola, Diapers, Eqqs)

{Cola, Milk, Eggs} {Diapers, Milk, Eqqs)

Figure 6.6. A brute-force method for generating candidate 3-itemsets.

Freouent 2-itemset

Candidate Pruning

Frequent 1-itemset

Figure 6,7. Generating and pruning candidate k-itemsets by merging a frequent (k - l)-itemset with a frequent item. Note that some of the candidates are unnecessary because their subsets are infrequent,

This approach, howevet, does not prevent the same candidate itemset from being generated more than once. For instance, {Bread, Diapers, Mitk} can be generated by merging {Bread, Diapers} with {Milk}, {Bread, MiIk} with

{niapers}, or {Diapers, MiIk} with {Bread}. One way to avoid generating

Candidate Generation

Itemset {Beer, Diapers. Bread) {Beer, Diapers. Milk}

{Bread. Diapers, Milk}

{Bread, Milk, Beer}

Itemset {Bread, Diapers, Milk}

6.2 Requent Itemset Generation 34L

duplicate candidates is by ensuring that the items in each frequent itemset are kept sorted in their lexicographic order. Each frequent (/c-l)-itemset X is then extended with frequent items that are lexicographically larger than the items in X. For example, the itemset {Bread, Diapers} can be augmented with {Milk} since Mil-k is lexicographically larger than Bread and Diapers. However, we should not augment {Diapers, MiIk} with {Bread} nor {Bread, Milk} with

{Oi-apers} because they violate the lexicographic ordering condition. While this procedure is a substantial improvement over the brute-force

method, it can still produce a large number of unnecessary candidates. For example, the candidate itemset obtained by merging {Beer, Diapers} with

{Ui-ft} is unnecessary because one of its subsets, {Beer, Milk}, is infrequent. There are several heuristics available to reduce the number of unnecessary candidates. For example, note that, for every candidate k-itemset that survives the pruning step, every item in the candidate must be contained in at least k - L of the frequent (k - 1)-itemsets. Otherwise, the candidate is guaranteed to be infrequent. For example, {Beer, Diapers, Milk} is a viable candidate 3-itemset only if every item in the candidate, including Beer, is contained in at least two frequent 2-itemsets. Since there is only one frequent 2-itemset containing Beer, all candidate itemsets involving Beer must be infrequent.

Fr-r x F6-1 Method The candidate generation procedure in the apriori-gen function merges a pair of frequent (k - l)-itemsets only if their first k - 2 items a r e i d e n t i c a l . L e t A : { o t , a 2 t . . . , a k _ r } a n d B : { b t , b z , . . . , b n _ t } b e a p a i r of frequent (/c - l)-itemsets. A and B are merged if they satisfy the following conditions:

a i : b i ( f o r z : 1 , 2 , . . . , k - 2 ) a n d a 7 r - 1 I b n - t .

In Figure 6.8, the frequent itemsets {Bread, Diapers} and {Bread, Milk} are merged to form a candidate 3-itemset {Bread, Diapers, Milk}. The algorithm does not have to merge {Beer, Di-apers} with {Diapers, Milk} because the first item in both itemsets is different. Indeed, if {Beer, Diapers, Milk} is a viable candidate, it would have been obtained by merging {Beer, Diapers} with {Beer, MiIk} instead. This example illustrates both the completeness of the candidate generation procedure and the advantages of using lexicographic ordering to prevent duplicate candidates. However, because each candidate is obtained by merging a pair of frequent (k-1)-itemsets, an additional candidate pruning step is needed to ensure that the remaining k - 2 subsets of the candidate are frequent.

342 Chapter 6 Association Analysis

Frequent 2-itemset

Candidate Generation

Candidate Pruning

Frequent 2-itemsel

Figure 6.8, Generating and pruning candidate k-itemsets by merging pairs of frequent (k- l)-itemsets.

6.2.4 Support Counting

Support counting is the process of determining the frequency of occurrence for every candidate itemset that survives the candidate pruning step of the apriori-gen function. Support counting is implemented in steps 6 through 11 of Algorithm 6.1. One approach for doing this is to compare each transaction against every candidate itemset (see Figure 6.2) and to update the support counts of candidates contained in the transaction. This approach is computa- tionally expensive, especially when the numbers of transactions and candidate itemsets are large.

An alternative approach is to enumerate the itemsets contained in each transaction and use them to update the support counts oftheir respective can- didate itemsets. To illustrate, consider a transaction t that contains five items,

{I,2,3,5,6}. There are ( 3 ) : tO itemsets of size 3 contained in this transac-

tion. Some of the itemsets may correspond to the candidate 3-itemsets under investigation, in which case, their support counts are incremented. Other subsets of t that do not correspond to any candidates can be ignored.

Figure 6.9 shows a systematic way for enumerating the 3-itemsets contained in l. Assuming that each itemset keeps its items in increasing lexicographic order, an itemset can be enumerated by specifying the smallest item first, followed by the larger items. For instance, given t : {L,2,3,5,6}, all the 3- itemsets contained in f must begin with item 1, 2, or 3. It is not possible to construct a 3-itemset that begins with items 5 or 6 because there are only two

Frequent Itemset Generation 343

Level 2

Level 3 Subsets of 3 items

Figure 6.9. Enumerating subsets of three items from a transaction r.

items in f whose labels are greater than or equal to 5. The number of ways to specify the first item of a 3-itemset contained in t is illustrated by the Level 1 prefix structures depicted in Figure 6.9. For instance, 1 [2J 5 6 I represents a 3-itemset that begins with item 1, followed by two more items chosen from t h e s e t { 2 , 3 , 5 , 6 } .

After fixing the first item, the prefix structures at Level 2 represent the number of ways to select the second item. For example, 1 2 F 5 6l corresponds to itemsets that begin with prefix (1 2) and are followed by items 3, 5, or 6. Finally, the prefix structures at Level 3 represent the complete set of 3-itemsets contained in t. For example, the 3-itemsets that begin with prefix {1 2} are

{ 1 , 2 , 3 } , { 7 , 2 , 5 } , a n d { 1 , 2 , 6 } , w h i l e t h o s e t h a t b e g i n w i t h p r e f i x { 2 3 } a r e

{ 2 , 3 , 5 } a n d { 2 , 3 , 6 } . The prefix structures shown in Figure 6.9 demonstrate how itemsets con-

tained in a transaction can be systematically enumerated, i.e., by specifying their items one by one, from the leftmost item to the rightmost item. We still have to determine whether each enumerated 3-itemset corresponds to an existing candidate itemset. If it matches one of the candidates, then the sup- port count of the corresponding candidate is incremented. In the next section, we illustrate how this matching operation can be performed efficiently using a hash tree structure.

6 . 2

1 2

Transaction, t

1 2 3 5 6

Bread, Milk, Diapers, Cola

344 Chapter 6 Association Analysis

Leaf nodes containing candidate 2-itemsets

Transactions

Figure 6.10. Counting the support of itemsets using hash structure.

Support Counting Using a Hash Tlee

In the Apriori, algorithm, candidate itemsets are partitioned into different buckets and stored in a hash tree. During support counting, itemsets contained in each transaction are also hashed into their appropriate buckets. That way, instead of comparing each itemset in the transaction with every candidate itemset, it is matched only against candidate itemsets that belong to the same bucket, as shown in Figure 6.10.

Figure 6.11 shows an example of a hash tree structure. Each internal node of the tree uses the following hash function, h(p) : p mod 3, to determine which branch of the current node should be followed next. For example, items 1, 4, and 7 are hashed to the same branch (i.e., the leftmost branch) because they have the same remainder after dividing the number by 3. All candidate itemsets are stored at the leaf nodes of the hash tree. The hash tree shown in Figure 6.11 contains 15 candidate 3-itemsets, distributed across 9 leaf nodes.

Consider a transaction, f, : {1,2,3,5,6}. To update the support counts of the candidate itemsets, the hash tree must be traversed in such a way that all the leaf nodes containing candidate 3-itemsets belonging to f must be visited at least once. Recall that the 3-itemsets contained in t must begin with items 1, 2,or 3, as indicated by the Level 1prefix structures shown in Figure 6.9. Therefore, at the root node of the hash tree, the items 1, 2, and 3 of the transaction are hashed separately. Item 1 is hashed to the left child ofthe root node, item 2 is hashed to the middle child, and item 3 is hashed to the right child. At the next level of the tree, the transaction is hashed on the second

6 . 2 FYeouent Itemset Generation 345

Hash Function

-'-:,@ ,-.w

Figure 6.1 1. Hashing a transaction at the root node of a hash tree.

item listed in the Level 2 structures shown in Figure 6.9. For example, after hashing on item 1 at the root node, items 2, 3, and 5 of the transaction are hashed. Items 2 and 5 are hashed to the middle child, while item 3 is hashed to the right child, as shown in Figure 6.12. This process continues until the leaf nodes of the hash tree are reached. The candidate itemsets stored at the visited leaf nodes are compared against the transaction. If a candidate is a subset of the transaction, its support count is incremented. In this example, 5 out of the 9 leaf nodes are visited and 9 out of the 15 itemsets are compared against the transaction.

6.2.5 Computational Complexity

The computational complexity of the Apri,ori, algorithm can be affected by the following factors.

Support Threshold Lowering the support threshold often results in more itemsets being declared as frequent. This has an adverse effect on the com-

Transaction

F,*;]

346 Chapter 6 Association Analysis

1 + l 2 3 s 6 l ---

z +

3 +

Figure 6.12. Subset operation on the leftmost subtree of the root of a candidate hash tree.

putational complexity of the algorithm because more candidate itemsets must be generated and counted, as shown in Figure 6.13. The maximum size of frequent itemsets also tends to increase with lower support thresholds. As the maximum size of the frequent itemsets increases, the algorithm will need to make more passes over the data set.

Number of Items (Dimensionality) As the number of items increases, more space will be needed to store the support counts of items. If the number of frequent items also grows with the dimensionality of the data, the computation and I/O costs will increase because of the larger number of candidate itemsets generated by the algorithm.

Number of Tbansactions Since the Apri,ori, algorithm makes repeated passes over the data set, its run time increases with a larger number of trans- actions.

Average TYansaction Width For dense data sets, the average transaction width can be very large. This affects the complexity of the Apriori algorithm in two ways. First, the maximum size of frequent itemsets tends to increase as the

Transaction t;;;-:-:r l r z r c o l

-@

x105

i i l

\:, Y il

I t = - - * * - * -

6 . 2 Freouent Itemset Generation 347

4

3.5

f ; 3 E I z s a 6

d

E r . s 3 E z 1

0.5

0 1 0

Size of ltemsel

(a) Number of candidate itemsets.

, x 1 0 5

E o

f

E ; d E z

0 - + J - f - + - + - +

0 1 0 Size of ltemsel

(b) Number of frequent itemsets.

Figure 6.13. Effect of support threshold on the number of candidate and frequent itemsets.

average transaction width increases. As a result, more candidate itemsets must be examined during candidate generation and support counting, as illustrated in Figure 6.14. Second, as the transaction width increases, more itemsets

v t il i l i l i l i l i l i l i l

Y I i l l t : A \ y

' + # t * .

.F. v v

v

348 Chapter 6 Association Analysis

1 0 1 5

Size ol ltemset

(a) Number of candidate itemsets.

1 0 1 5 Size of ltemset

(b) Number of Frequent ltemsets.

Figure 6.14. Effect of average transaction width on the number of candidate and frequent itemsets.

are contained in the transaction. This will increase the number of hash tree traversals performed during support counting.

A detailed analysis of the time complexity for the Apri,ori, algorithm is presented next.

9

8

o _

o

o 6 6

() 6

E 3

z

1

0

1 0

8

E o

o

I

o 4

z

2

1

0

x1 05

I

4 * + - .

v

\b

; v

6.3 Rule Generation 349

Generation of frequent l-itemsets For each transaction, we need to up- date the support count for every item present in the transaction. Assuming that tr is the average transaction width, this operation requires O(l/tr) time, where .A[ is the total number of transactions.

Candidate generation To generate candidate k-itemsets, pairs of frequent (k - l)-itemsets are merged to determine whether they have at least k - 2 items in common. Each merging operation requires at most k - 2 equality comparisons. In the best-case scenario, every merging step produces a viable candidate k-itemset. In the worst-case scenario, the algorithm must merge ev- ery pair of frequent (k - 1)-itemsets found in the previous iteration. Therefore, the overall cost of merging frequent itemsets is

iU - z)lcnl ( cost or merging .irr - 2)lLn-r12. k : 2 k : 2

A hash tree is also constructed during candidate generation to store the can- didate itemsets. Because the maximum depth of the tree is k, the cost for populating the hash tree with candidate itemsets rs O(l[:2klcrl). During candidate pruning, we need to verify that the ,k - 2 subsets of every candidate k-itemset are frequent. Since the cost for looking up a candidate in a hash tree is O(k), the candidate pruning step require" O(D|:rk(k - z)lckl) time.

Support counting Each transaction of length ltl produces (lll) itemsets of size k. This is also the effective number of hash tree traversals performed for each transaction. The cost for support counting is O(,n/Dr(T)"r), where tr is the maximum transaction width and a7, is the cost for updating the support count of a candidate k-itemset in the hash tree.

6.3 Rule Generation

This section describes how to extract association rules efficiently from a given frequent itemset. Each frequent k-itemse t, Y , can produce up to 2k - 2 associa- tion rules, ignoring rules that have empty antecedents or consequents (0 ---- Y orY ------+ 0). An association rule can be extracted by partitioning the itemset Y into two non-empty subsets, X and Y -X, suchthat X ------+Y -X satisfies the confidence threshold. Note that all such rules must have already met the support threshold because they are generated from a frequent itemset.

350 Chapter 6 Association Analysis

Example 6.2. Let X : {I,2,3} be a frequent itemset. There are six candi- date association rules that can be generated from X: {1,2} - i3}, {1,3} -

{ 2 } , { 2 , 3 } . - - - - *

{ 1 } , { U - - - - -

{ 2 , 3 } , { 2 } . _ - - > { 1 , 3 } , a n d { 3 } - - - - *

{ 1 , 2 } . A s each of their support is identical to the support for X, the rules must satisfy the support threshold. r

Computing the confidence of an association rule does not require additional scans of the transaction data set. Consider the rule {1,2} - {3}, which is generated from the frequent itemset X : {1,2,3}. The confi.dence for this rule is o({1, 2,3}) lo({1, 2}). Because {1, 2, 3} is frequent, the anti-monotone prop-

erty of support ensures that {1,2} must be frequent, too. Since the support counts for both itemsets were already found during frequent itemset genera- tion, there is no need to read the entire data set again.

6.3.1 Confidence-Based Pruning

Unlike the support measure) confidence does not have any monotone property. For example, the confidence for X ------. Y can be larger, smaller, or equal to the c o n f i d e n c e f o r a n o t h e r r u l e * , t , w h e r e * g X a n d f e Y ( s e e E x e r c i s e

3 on page 405). Nevertheless, if we compare rules generated from the same frequent itemset Y, the following theorem holds for the confidence measure.

Theorem 6.2. If a rule X ------+ Y - X does not sati,sfy the confidence threshold, then any rule Xt --+ Y - Xt , where X' ,is a subset of X, must not sati,sfy the confidence threshold as well.

To prove this theorem, consider the following two rules: Xt ---', Y - Xt and X - + Y - X , w h e r e X t c X . T h e c o n f i d e n c e o f t h e r u l e s a r e o ( Y ) l o ( X / ) a n d o(V) lo(X), respectively. Since X/ is a subset of X , o(Xt) >

"(X). Therefore,

the former rule cannot have a higher confidence than the latter rule.

6.3.2 Rule Generation in Apriori Algorithm

The Apri,orz algorithm uses a level-wise approach for generating association rules, where each level corresponds to the number of items that belong to the rule consequent. Initially, all the high-confidence rules that have only one item in the rule consequent are extracted. These rules are then used to generate new candidate rules. For example, if {acd}------- {b} and {abd}

------ {c} are

high-confidence rules, then the candidate rule {a,d} ------,

{bc} is generated by merging the consequents of both rules. Figure 6.15 shows a lattice structure for the association rules generated from the frequent itemset {a,b,c,d}. If any node in the lattice has low confidence, then according to Theorem 6.2, the

Rule Generation 35L

a a P r u n e d - - . '' . . - R u l e s

Figure 6.15, Pruning of association rules using the confidence measure.

entire subgraph spanned by the node can be pruned immediately. Suppose the confidence lbr {bcd}

------' {o} is low. All the rules containing item a in

its consequent, including {cd} -----

{ab}, {bd} -----+

{ac}, {bc) ------ {od}, and

{d} - {abc} can be discarded. A pseudocode for the rule generation step is shown in Algorithms 6.2 and

6.3. Note the similarity between the ap-genrules procedure given in Algo- rithm 6.3 and the frequent itemset generation procedure given in Algorithm 6.1. The only difference is that, in rule generation, we do not have to make additional passes over the data set to compute the confidence ofthe candidate rules. Instead, we determine the confidence of each rule by using the support counts computed during frequent itemset generation.

Algorithm 6.2 Rule generation of the Apri,ori algorithm.

6 . 3

I I \ \ \ \

1: for each frequent k-itemset f p, k > 2 do 2 : H 1 : { i l i e f n } {l-item consequents of the rule.} 3: call ap-genrules(/6,fI1.) 4: end for

352 Chapter 6 Association Analysis

Algorithm 6.3 Procedure ap-genrules(fp, H*).

9 : 1 0 :

t: k : lfpl {size of frequent itemset.} Z, m: lH^l {size of rule consequent.} 3 : i f k > m l l t h e n 4: Hm+t : aPriori-gen(f1-). 5: for each h^+r € H*a1 do 6 : c o n f :

" ( f n ) l " U n - h ^ + t ) .

7: if conf ) m'inconf tl:.en 8: output the rule (f* - h*+t) ------+ hm*r.

else delete h*q1 from Hm+r.

1 1 : e n d i f 12: end for 13: call ap-genrules(/p, H-+r.) 14: end if

6.3.3 An Example: Congressional Voting Records

This section demonstrates the results of applying association analysis to the voting records of members of the United States House of Representatives. The data is obtained from the 1984 Congressional Voting Records Database, which is available at the UCI machine learning data repository. Each transaction contains information about the party affiIiation for a representative along with his or her voting record on 16 key issues. There are 435 transactions and 34 items in the data set. The set of items are listed in Table 6.3.

The Apri,ore algorithm is then applied to the data set with mi,nsup : 30To and minconf : 90T0. Some of the high-confidence rules extracted by the algorithm are shown in Table 6.4. The first two rules suggest that most of the members who voted yes for aid to El Salvador and no for budget resolution and MX missile are Republicans; while those who voted no for aid to EI Salvador and yes for budget resolution and MX missile are Democrats. These high- confidence rules show the key issues that divide members from both political parties. If mi,nconf is reduced, we may find rules that contain issues that cut across the party lines. For example, with mi.nconf : 40Vo, the rules suggest that corporation cutbacks is an issue that receives almost equal number of votes from both parties-52.3% of the members who voted no are Republicans, while the remaining 47.7% of them who voted no are Democrats.

6.4 Compact Representation of Frequent Itemsets 353

Table 6.3. List of binary attributes from the 1984 United States Congressional Voting Records. Source: The UCI machine learning repository.

1. Republican 2. Democrat 3. handicapped-infants -- yes 4. handicapped-infants : no 5. water project cost sharing : yes 6. water project cost sharing : pe

7. budget-resolution - yes 8. budget-resolution : no 9. physician L," 1r"s2s - yes 10. physician fee freeze : no 11. aid to EI Salvador : yes 12. aid to EI Salvador : no 13. religious groups in schools : yes 14. religious groups in schools : no 15. anti-satellite test f61 - 1les 16. anti-satellite test barr: no 17. aid to Nicaragua : yes

aid to Nicaragua : no MX-missile : yes MX-missile : no immigration : yes immigration : no synfuel corporation cutback : yes synfuel corporation cutback : no education spending : yes education spending : 1e right-to-sue : yes right-to-sue : no crrme - yes crime : no duty-free-exports : yes duty-free-exports : no export administration act : yes export administration act : no

1 8 . 1 9 . 20. 21. 2 2 . 23. , L

2 5 . 2 6 . 2 7 . 28. 29. 3 0 . 3 1 . 32. 3 3 . 34.

Table 6.4. Association rules extracted from the 1984 United States Congressional Voting Records.

Association Rule Confidence

{budget resolution : no, Mx-missile:no, aid to El Salvador : yes } ------+ {Republican}

9r.0%

{budget resolution : y€s, MX-missile:yes, aid to El Salvador : no } -----+ {Democrat}

97.5%

{crime: y€s, right-to-sue : y€s, physician fee freeze : yes} ------+ {Republican}

93.5To

{crime : no, right-to-sue : no, physician fee freeze : no} ------+ {Democrat}

l00Yo

6.4 Compact Representation of Flequent Itemsets

In practice, the number of frequent itemsets produced from a transaction data set can be very large. It is useful to identify a small representative set of itemsets from which all other frequent itemsets can be derived. Two such representations are presented in this section in the form of maximal and closed frequent itemset;s.

354 Chapter 6 Association Analysis

Figure 6.16, Maximal frequent itemset.

6.4.L Maximal Flequent Itemsets

Definition 6.3 (Maximal Flequent ltemset). A maximal frequent item- set is defined as a frequent itemset for which none of its immediate supersets are frequent.

To illustrate this concept, consider the itemset lattice shown in Figure 6.16. The itemsets in the lattice are divided into two groups: those that are frequent and those that are infrequent. A frequent itemset border, which is represented by a dashed line, is also illustrated in the diagram. Every itemset located above the border is frequent, while those located below the border (the shaded nodes) are infrequent. Among the itemsets residing near the border,

{o, d}, {a, c, e), and {b, c, d, e} are considered to be maximal frequent itemsets because their immediate supersets are infrequent. An itemset such as {o,d} is maximal frequent because all of its immediate supersets, {a,b,d}, {a,c,d}, and {a, d,e}, are infrequent. In contrast, {o,"} is non-maximal because one of its immediate supersets, {a, c, e}, is frequent.

Maximal frequent itemsets effectively provide a compact representation of frequent itemsets. In other words, they form the smallest set of itemsets from

Compact Representation of Frequent Itemsets 355

which all frequerrt itemsets can be derived. For example, the frequent itemsets shown in Figure 6.16 can be divided into two groups:

o Frequent il;emsets that begin with item a and that may contain items c, d, or e. This group includes itemsets such as {o), {o.,c), {a,d}, {a,e}, a n d { a , c , e } .

o Flequent it,emsets that begin with items b, c, d, or e. This group includes i t e m s e t s s u c h a s { b } , { b , c } , { c , d } , { b , c , d , e } , e t c .

Requent itemsel;s that belong in the first group are subsets of either {a,c,e} or {4, d}, while those that belong in the second group are subsets of {b, c, d, e}. Hence, the maximal frequent itemsets {a, c, e}, {o, d}, and {b, c, d, e} provide a compact representation of the frequent itemsets shown in Figure 6.16.

Maximal frequent itemsets provide a valuable representation for data sets that can produce very long, frequent itemsets, as there are exponentially many frequent itemsets in such data. Nevertheless, this approach is practical only if an efficient algorithm exists to explicitly find the maximal frequent itemsets without havingto enumerate all their subsets. We briefly describe one such approach in Secl;ion 6.5.

Despite providing a compact representation, maximal frequent itemsets do not contain the support information of their subsets. For example, the support of the maximal fiequent itemsets {a,c,e}, {o,d}, and {b,c,d,e} do not provide any hint about the support of their subsets. An additional pass over the data set is therefore needed to determine the support counts of the non-maximal frequent itemsets. In some cases, it might be desirable to have a minimal representation of frequent itemsets that preserves the support information. We illustrate such a representation in the next section.

6.4.2 Closed Flequent Itemsets

Closed itemsets provide a minimal representation of itemsets without losing their support infbrmation. A formal definition of a closed itemset is presented below.

Definition 6.4 (Closed Itemset). An itemset X is closed if none of its immediate supersets has exactly the same support count as X.

Put another way, X is not closed if at least one of its immediate supersets has the same support count as X. Examples of closed itemsets are shown in Figure 6.17. To better illustrate the support count of each itemset, we have associated each node (itemset) in the lattice with a list of its corresponding

6 . 4

1 aoc

1 , 2 , 4

abcd

oce

4 acde

de

356 Chapter 6 Association Analysis

Figure 6.17. An example of the closed frequent itemsets (with minimum support count equalto 40%).

transaction IDs. For example, since the node {b, c} is associated with transac- tion IDs 1,2, and 3, its support count is equal to three. Flom the transactions given in this diagram, notice that every transaction that contains b also con- tains c. Consequently, the support for {b} is identical to {b, c} and {b} should not be considered a closed itemset. Similarly, since c occurs in every transac- tion that contains both a and d, the itemset {o,d} is not closed. On the other hand, {b, c} is a closed itemset because it does not have the same support count as any of its supersets.

Definition 6.5 (Closed Flequent Itemset). An itemset is a closed fre- quent itemset if it is closed and its support is greater than or equal to m'insup.

In the previous example, assuming that the support threshold is 40%, {b,c} is a closed frequent itemset because its support is 60%. The rest of the closed frequent itemsets are indicated by the shaded nodes.

Algorithms are available to explicitly extract closed frequent itemsets from a given data set. Interested readers may refer to the bibliographic notes at the end of this chapter for further discussions of these algorithms. We can use the closed frequent itemsets to determine the support counts for the non-closed

6 . 4 Compact Representation of Requent Itemsets 357

Algorithm 6.4 Support counting using closed frequent itemsets. 1: Let C denote the set of closed frequent itemsets 2: Let k-.* denote the maximum size of closed frequent itemsets 3: F6-"* : {flf e C, lfl: k-.*} {Find all frequent itemsets of size k-.*.} 4: for k : k^u*- 1 downto 1 do 5: Fn : {f lf ( Fp',1, lf l: k} {Find all frequent itemsets of size k.} 6: for each f e F* do 7: if f f C then 8 : f . s u p p o r t : m a x { f t . s u r y o r t l f t e F n + r , f C f ' } 9: end if

10: end for 11: end for

frequent itemsets. For example, consider the frequent itemset {o,d} shown in Figure 6.17. Because the itemset is not closed, its support count must be identical to one of its immediate supersets. The key is to determine which superset (amon6; {a,b,d}, {a,c,d}, or {a,d,e}) has exactly the same support count as {a, d}. The Apri,ori, principle states that any transaction that contains the superset of .l-a, d) must also contain {o,d}. However, any transaction that contains {o,d} does not have to contain the supersets of {a,d}. For this reason, the support for {a, d} must be equal to the largest support among its supersets. Since {a, c, d} has a larger support than both {a,b,d} and {a,d,e}, the support for .[a, d] must be identical to the support for {4, c, d}. Using this methodology, an algorithm can be developed to compute the support for the non-closed frequent itemsets. The pseudocode for this algorithm is shown in Algorithm 6.4. The algorithm proceeds in a specific-to-general fashion, i.e., from the largest to the smallest frequent itemsets. This is because, in order to find the support for a non-closed frequent itemset; the support for all of its supersets must be known.

To illustrate the advairtage of using closed frequent itemsets, consider the data set shown in Table 6.5, which contains ten transactions and fifteen items. The items can be divided into three groups: (1) Group -4, which contains items ar through as; Q) Group B, which contains items b1 through b5i and (3) Group C, which contains items c1 through c5. Note that items within each group are perfectly associated with each other and they do not appear with items from another group. Assuming the support threshold is 20To, the total number of frequent itemsets is 3 x (25 - 1) : 93. However, there are only three c l o s e d f r e q u e n t i t e m s e t s i n t h e d a t a : ( { 4 1 , a 2 t a y t a 4 l a b } , { b t , b z , b z , b + , b 5 } , a n d

{"t,"z,cs,ca,c5}). It is often sufficient to present only the closed frequent itemsets to the analysts instead of the entire set of frequent itemsets.

Table 6.5. A transaction data set for minino closed itemsets.

TID A 7 A 2 A J A4 A s O 1 bz bs ba b5 C 1 C2 ca C4 Cb I 2 J

4 (

t)

7 e

rl

1 0

I I 1 0 0 0 0 0 0 0

1 1 I 0 0 0 0 0 0 0

I 1_ 1. 0 0 0 0 0 0 0

I 1_ l. 0 0 0 0 0 0 0

1 1 I 0 0 0 0 0 0 0

0 0 0 I I I 0 0 0 0

0 0 0 1 1 1 0 0 0 0

0 0 0 1 1 1 0 0 0 0

0 0 0 1 1 1 0 0 0 0

0 0 0 1 1 1 0 0 0 0

0 0 0 0 0 0 1 1 1 1

U 0 0 0 0 0 1 L 1 1

U 0 0 0 0 0 1 I 1 I

0 0 0 0 0 0 1 1 I 1

0 0 0 0 0 0 1 1 I 1

358 Chapter 6 Association Analysis

Figure 6.18. Relationships among f requent, maximal f requent, and closed frequent itemsets.

Closed frequent itemsets are useful for removing some of the redundant association rules. An association rule X ------ Y is redundant if there exists another rule X/ ------ Y' , where X is a subset of X/ and Y is a subset of Y/, such that the support and confidence for both rules are identical. In the example shown in Figure 6.L7, {b} is not a closed frequent itemset while {b, c} is closed. The association rule {b} - {d, e} is therefore redundant because it has the sarne support and confidence as {b, c} - {d,"}. Such redundant rules are not generated if closed frequent itemsets are used for rule generation.

Finally, note that all maximal frequent itemsets are closed because none of the maximal frequent itemsets can have the same support count as their immediate supersets. The relationships among frequent, maximal frequent, and closed frequent itemsets are shown in Figure 6.18.

6.5 Alternative Methods for Generating Frequent Itemsets 359

6.5 Alternative Methods for Generating Fhequent Itemsets

Apri,ori, is one of the earliest algorithms to have successfully addressed the combinatorial explosion of frequent itemset generation. It achieves this by ap- plying the Apii,ori principle to prune the exponential search space. Despite its significant performance improvement, the algorithm still incurs considerable I/O overhead since it requires making several passes over the transaction data set. In addition, as noted in Section 6.2.5, the performance of the Apri,ori algorithm may degrade significantly for dense data sets because of the increas- ing width of transactions. Several alternative methods have been developed to overcome these limitations and improve upon the efficiency of the Apri,ori, algorithm. The following is a high-level description of these methods.

Tlaversal of Itemset Lattice A search for frequent itemsets can be con- ceptually viewecl as a traversal on the itemset lattice shown in Figure 6.1. The search strategy employed by an algorithm dictates how the lattice struc- ture is traversed during the frequent itemset generation process. Some search strategies are better than others, depending on the configuration of frequent itemsets in the lattice. An overview of these strategies is presented next.

o General-t;o-Specific versus Specific-to-General: The Apriori, al- gorithm uses a general-to-specific search strategy, where pairs offrequent (k- l)-itemsets are merged to obtain candidate k-itemsets. This general- to-specific search strategy is effective, provided the maximum length of a frequent itemset is not too long. The configuration of frequent item- sets that works best with this strategy is shown in Figure 6.19(a), where the darker nodes represent infrequent itemsets. Alternatively, a specific- to-general search strategy looks for more specific frequent itemsets first, before finding the more general frequent itemsets. This strategy is use- ful to discover maximal frequent itemsets in dense transactions, where the frequent itemset border is located near the bottom of the lattice, as shown in Figure 6.19(b). The Apri,ori, principle can be applied to prune all subsets of maximal frequent itemsets. Specifically, if a candi- date k-itemset is maximal frequent, we do not have to examine any of its subsets of size k - l. However, if the candidate k-itemset is infrequent, we need to check all of its k - 1 subsets in the next iteration. Another approach is to combine both general-to-specific and specific-to-general search strategies. This bidirectional approach requires more space to

360 Chapter 6 Association Analysis

Frequent Itemset Border null

{ a 1 , a 2 , . . . , a n }

(a) General-to-specific (c) Bidirectional

Figure 6.19. General-to-specific, specific{o-general, and bidirectional search.

store the candidate itemsets, but it can help to rapidly identify the fre- quent itemset border, given the configuration shown in Figure 6.19(c).

Equivalence Classes: Another way to envision the traversal is to first partition the lattice into disjoint groups of nodes (or equivalence classes). A frequent itemset generation algorithm searches for frequent itemsets within a particular equivalence class first before moving to another equiv- alence class. As an example, the level-wise strategy used in the Apri,ori, algorithm can be considered to be partitioning the lattice on the basis of itemset sizes; i.e., the algorithm discovers all frequent l-itemsets first before proceeding to larger-sized itemsets. Equivalence classes can also be defined according to the prefix or suffix labels of an itemset. In this case, two itemsets belong to the same equivalence class if they share a common prefix or suffix of length k. In the prefix-based approach, the algorithm can search for frequent itemsets starting with the prefix a before looking for those starting with prefixes b) c) and so on. Both prefix-based and suffix-based equivalence classes can be demonstrated using the tree-like structure shown in Figure 6.20.

Breadth-First versus Depth-First: The Apriori, algorithm traverses the lattice in a breadth-first manner) as shown in Figure 6.2L(a). It first discovers all the frequent 1-itemsets, followed by the frequent 2-itemsets, and so on, until no new frequent itemsets are generated. The itemset

Frequent Itemset null

{a1,a2,...,an} \ ltemsel

Border

(b) Specific-to-general

6.5 Alternative Methods for Generating Flequent Itemsets 361

(a) Prefix tree.

Figure 6,20" Equivalence classes based on the prefix and suffix labels of itemsets.

(a) Breadth first (b) Depth first

Figure 6,21. Breadth{irst and depth-first traversals.

lattice can also be traversed in a depth-first manner, as shown in Figures 6.21(b) and 6.22. The algorithm can start from, say, node a, in Figure 6.22, and count its support to determine whether it is frequent. If so, the algorithm progressively expands the next level of nodes, i.e., ab, abc, and so on, until an infrequent node is reached, say, abcd. It then backtracks to another branch, say, abce, and continues the search from there.

The deprth-first approach is often used by algorithms designed to find maximal frequent itemsets. This approach allows the frequent itemset border to be detected more quickly than using a breadth-first approach. Once a merximal frequent itemset is found, substantial pruning can be

\ . - - /

362 Chapter 6 Association Analysis

bce bde cde r/// - - - - - - - - t '

abce abde

Figure 6.22. Generating candidate itemsets using the depth{irst approach.

performed on its subsets. For example, if the node bcde shown in Figure 6.22 is maximal frequent, then the algorithm does not have to visit the subtrees rooted at bd,, be, c, d, and e because they will not contain any maximal frequent itemsets. However, if abcis maximal frequent, only the nodes such as ac and bc arc not maximal frequent (but the subtrees of ac and bc may still contain maximal frequent itemsets). The depth-first approach also allows a different kind of pruning based on the support of itemsets. For example, suppose the support for {a,b,c} is identical to the support for {a, b}. The subtrees rooted at abd and abe can be skipped because they are guaranteed not to have any maximal frequent itemsets. The oroof of this is left as an exercise to the readers.

Representation of Transaction Data Set There are many ways to rep- resent a transaction data set. The choice of representation can affect the I/O costs incurred when computing the support of candidate itemsets. Figure 6.23 shows two different ways of representing market basket transactions. The rep- resentation on the left is called a horizontal data layout, which is adopted by many association rule mining algorithms, including Apri,ori,. Another pos- sibility is to store the list of transaction identifiers (TID-list) associated with each item. Such a representation is known as the vertical data layout. The support for each candidate itemset is obtained by intersecting the TlD-lists of its subset items. The length of the TlD-lists shrinks as we progress to larger

I I I I I qc

I \

)r tt

ade

\ ) l I t- ,

I I

o

de I

I

bcd

abcde

6.6 FP-Growth Algorithm 363

Horizontal Data Layout Vertical Data Layout

'l 1 2 2 1 4 2 3 4 3 5 5 4 5 o 6 7 8 I 7 I I 8 1 0 9

Figure 6.23, Horizontal and vertical data format.

sized itemsets. [fowever, one problem with this approach is that the initial

set of TlD-lists may be too large to fit into main memory, thus requiring

more sophisticated techniques to compress the TlD-lists. We describe another

effective approach to represent the data in the next section.

6.6 FP-Growth Algorithm

This section presents an alternative algorithm called FP-growth that takes

a radically different approach to discovering frequent itemsets. The algorithm

does not subscribe to the generate-and-test paradigm of Apriori^ Instead, it

encodes the data set using a compact data structure called an FP-tree and

extracts frequent itemsets directly from this structure. The details of this

approach are presented next.

6.6.1 FP-Tbee Representation

An FP-tree is a compressed representation of the input data. It is constructed

by reading the data set one transaction at a time and mapping each transaction

onto a path in the FP-tree. As different transactions can have several items

in common, their paths may overlap. The more the paths overlap with one

another, the more compression we can achieve using the FP-tree structure. If

the size of the F.P-tree is small enough to fit into main memory, this will allow

us to extract frequent itemsets directly from the structure in memory instead

of making repeated passes over the data stored on disk.

364 Chapter 6 Association Analysis

Transaction Data Set

(i) After reading TID=1 (ii) After reading TID=2

(iv) After reading TID=10

Figure 6.24, Construction of an FP-tree.

Figure 6.24 shows a data set that contains ten transactions and five items. The structures ofthe FP-tree after reading the first three transactions are also depicted in the diagram. Each node in the tree contains the label of an item along with a counter that shows the number of transactions mapped onto the given path. Initially, the FP-tree contains only the root node represented by the nulf symbol. The FP-tree is subsequently extended in the following way:

1. The data set is scanned once to determine the support count of each item. Infrequent items are discarded, while the frequent items are sorted in decreasing support counts. For the data set shown in Figure 6.24, a is the most frequent item, followed by b, c, d, and e.

b : 1

FP-GrowthAlgorithm 365

2. The algoril;hm makes a second pass over the data to construct the FP-

tree. After reading the first transaction, {o,b), the nodes labeled as a

and b are created. A path is then formed from nulI --+ a '--+ b to encode

the transaction. Every node along the path has a frequency count of 1.

3. After reading the second transaction, {b,cd}, a new set of nodes is cre-

ated for items b, c, arrd d. A path is then formed to represent the

transaction by connecting the nodes null ---+ b ---+ c -* d. Every node

along this path also has a frequency count equal to one. Although the first two transactions have an item in common, which is b, their paths

are disjoint because the transactions do not share a common prefix.

4. The third transaction, {a,cd,e}, shares a common prefix item (which

is a) with the first transaction. As a result, the path for the third

transaction, null ) a ---+ c ---+ d --+ e, overlaps with the path for the

first transaction, nuII ---+ a -- b. Because of their overlapping path, the frequency count for node o is incremented to two, while the frequency

counts for the newly created nodes, c, d, and e) are equal to one.

5. This process continues until every transaction has been mapped onto one

of the paths given in the FP-tree. The resulting FP-tree after reading all the transactions is shown at the bottom of Figure 6.24.

The size of an FP-tree is typically smaller than the size of the uncompressed data because ma,ny transactions in market basket data often share a few items

in common. In the best-case scenario, where all the transactions have the

same set of iterns, the FP-tree contains only a single branch of nodes. The

worst-case scenario happens when every transaction has a unique set of items.

As none of the transactions have any items in common, the size of the FP-tree is effectively the same as the size of the original data. However, the physical

storage requirement for the FP-tree is higher because it requires additional space to store pointers between nodes and counters for each item.

The size of an FP-tree also depends on how the items are ordered. If

the ordering scheme in the preceding example is reversed, i.e., from lowest

to highest support item, the resulting FP-tree is shown in Figure 6.25. The

tree appears to be denser because the branching factor at the root node has

increased from 2 to 5 and the number of nodes containing the high support items such as a and b has increased from 3 to 12. Nevertheless, ordering

by decreasing support counts does not always lead to the smallest tree. For

exanlple, suppose we augment the data set given in Figure 6.24 with 100

transactions that contain {e}, 80 transactions that contain {d}, 60 transactions

6 . 6

3 6 6 Chapter 6 Association Analysis

. . - . c:2d:2

" ' i t b : 1

"-a:l

Figure 6.25. An FP-tree representation for the data set shown in Figure 6.24 with a different item ordering scheme.

that contain {"}, and 40 transactions that contain {b}. Item e is now most frequent, followed by d, c, b, and a. With the augmented transactions, ordering by decreasing support counts will result in an FP-tree similar to Figure 6.25, while a scheme based on increasing support counts produces a smaller FP-tree similar to Figure 6.2a$v).

An FP-tree also contains a list of pointers connecting between nodes that have the same items. These pointers, represented as dashed lines in Figures 6.24 and 6.25, help to facilitate the rapid access of individual items in the tree. We explain how to use the FP-tree and its corresponding pointers for frequent itemset generation in the next section.

6.6.2 FYequent Itemset Generation in FP-Growth Algorithm

FP-growth is an algorithm that generates frequent itemsets from an FP-tree by exploring the tree in a bottom-up fashion. Given the example tree shown in Figure 6.24, the algorithm looks for frequent itemsets ending in e first, followed by d, c, b, and finally, a. This bottom-up strategy for finding frequent item- sets ending with a particular item is equivalent to the suffix-based approach described in Section 6.5. Since every transaction is mapped onto a path in the FP-tree, we can derive the frequent itemsets ending with a particular item, sayT e, by examining only the paths containing node e. These paths can be accessed rapidly using the pointers associated with node e. The extracted paths are shown in Figure 6.26(a). The details on how to process the paths to obtain frequent itemsets will be explained later.

6.6 FP-Growth Algorithm 367

(a) Paths containing node e

null

(b) Paths containing node d

) b : 2

c:3 (c) Paths containing node c (d) Paths containing node b (e) Paths containing node a

Figure 6.26. Decomposing the frequent itemset generation problem into multiple subproblems, where each subproblem involves finding frequent itemsets ending in e, d, c, b, and a.

Table 6.6. The list of frequent itemsets ordered by their corresponding suffixes.

Suffix Fbequent Itemsets e { e } , { d , e } , { a , d , e } , { c , e } , { a , e } d { d } , { c , d } , { b , c , d } , { a , c , d } , { b , d } , { a , b , d } , { a , d } c { c } , { b , c } , { a , b , c } , { a , c } b { b } , { a , b } a { a }

After finding the frequent itemsets ending in e, the algorithm proceeds to look for frequent itemsets ending in d by processing the paths associated with node d. The corresponding paths are shown in Figure 6.26(b). This process continues until all the paths associated with nodes c, b, and finally a) are processed. The paths for these items are shown in Figures 6.26(c), (d), and (e), while their corresponding frequent itemsets are summarized in Table 6.6.

FP-growth finds all the frequent itemsets ending with a particular suffix by employing a divide-and-conquer strategy to split the problem into smaller subproblems. For example, suppose we are interested in finding all frequent

c:3

d : 1

n u l l

b:5

a:8

null o ,\

a:8 c:2

368 Chapter 6 Association Analysis

d : 1 d : 1

(b) Conditional FP-tree for e

(c) Prefix paths ending in de (d) Conditional FP-tree for de

n u l l

c : 1 c : 1

(e) Prefix paths ending in ce (f) Prefix paths

Figure 6.27. Example of applying the FP-growth algorithm to find frequent itemsets ending in e.

itemsets ending in e. To do this, we must first check whether the itemset

{e} itself is frequent. If it is frequent, we consider the subproblem of finding frequent itemsets ending in de, followed by ce, be, and ae. In turn, each of these subproblems are further decomposed into smaller subproblems. By merging the solutions obtained from the subproblems, all the frequent itemsets ending in e can be found. This divide-and-conquer approach is the key strategy employed by the FP-growth algorithm.

For a more concrete example on how to solve the subproblems, consider the task of finding frequent itemsets ending with e.

1. The first step is to gather all the paths containing node e. These initial paths are called prefix paths and are shown in Figure 6.27(a).

2. Fbom the prefix paths shown in Figure 6.27(a), the support count for e is obtained by adding the support counts associated with node e. Assuming that the minimum support count is 2, {e} is declared a frequent itemset because its support count is 3.

.)

I d a:2 ending in ae

FP-GrowthAlgorithm 369

Because {e} is frequent, the algorithm has to solve the subproblems of finding frequent itemsets ending in de, ce, be, and ae. Before solving these subproblems, it must first convert the prefix paths into a con- ditional FP-tree, which is structurally similar to an FP-tree, except it is used to find frequent itemsets ending with a particular suffix. A conditional FP-tree is obtained in the following way:

(a) First, the support counts along the prefix paths must be updated because some ofthe counts include transactions that do not contain item e. For example, the rightmost path shown in Figure 6.27(a), nufl ------+ b:2 -------+ c:2 ------+ e:1, includes a transaction {b,c} that does not contain item e. The counts along the prefix path must therefore be adjusted to 1 to reflect the actual number of transac- tions containing {b, c, e}.

(b) The prefix paths are truncated by removing the nodes for e. These nodes can be removed because the support counts along the prefix pathsr have been updated to reflect only transactions that contain e and the subproblems of finding frequent itemsets ending in de, ce, be, anrd ae no longer need information about node e.

(c) After updating the support counts along the prefix paths, some of the items may no longer be frequent. For example, the node b appears only once and has a support count equal to 1, which means that there is only one transaction that contains both b and e. Item b can be safely ignored from subsequent analysis because all itemsets ending in be must be infrequent.

The conditional FP-tree for e is shown in Figure 6.27(b). The tree looks different than the original prefix paths because the frequency counts have been updated and the nodes b and e have been eliminated.

FP-growth uses the conditional FP-tree for e to solve the subproblems of finding frequent itemsets ending in de, ce, and ae. To find the frequent itemsets ending in de, the prefix paths for d are gathered from the con- ditional FP-tree for e (Figure 6.27(c)). By adding the frequency counts associated with node d, we obtain the support count for {d,e}. Since the support count is equal to 2, {d,e} is declared a frequent itemset. Next, the algorithm constructs the conditional FP-tree for de using the approach described in step 3. After updating the support counts and removing the infrequent item c, the conditional FP-tree for de is shown in Figure 6.27(d). Since the conditional FP-tree contains only one item,

6 . 6

3 .

A

37O Chapter 6 Association Analysis

o, whose support is equal to minsup, the algorithm extracts the fre- quent itemset {a,d,e} and moves on to the next subproblem, which is to generate frequent itemsets ending in ce. After processing the prefix paths for c, only {c, e} is found to be frequent. The algorithm proceeds to solve the next subprogram and found {a,e} to be the only frequent itemset remaining.

This example illustrates the divide-and-conquer approach used in the FP- growth algorithm. At each recursive step, a conditional FP-tree is constructed by updating the frequency counts along the prefix paths and removing all infrequent items. Because the subproblems are disjoint, FP-growth will not generate any duplicate itemsets. In addition, the counts associated with the nodes allow the algorithm to perform support counting while generating the common suffix itemsets.

FP-growth is an interesting algorithm because it illustrates how a compact representation of the transaction data set helps to efficiently generate frequent itemsets. In addition, for certain transaction data sets, FP-growth outperforms the standard Apriori, algorithm by several orders of magnitude. The run-time performance of FP-growth depends on the compaction factor of the data set. If the resulting conditional FP-trees are very bushy (in the worst case, a full prefix tree), then the performance of the algorithm degrades significantly because it has to generate a large number of subproblems and merge the results returned by each subproblem.

6.7 Evaluation of Association Patterns

Association analysis algorithms have the potential to generate a large number of patterns. For example, although the data set shown in Table 6.1 contains only six items, it can produce up to hundreds of association rules at certain support and confidence thresholds. As the size and dimensionality of real commercial databases can be very large, we could easily end up with thousands or even millions of patterns, many of which might not be interesting. Sifting through the patterns to identify the most interesting ones is not a trivial task because "one person's trash might be another person's treasure." It is therefore important to establish a set of well-accepted criteria for evaluating the quality of association patterns.

The first set of criteria can be established through statistical arguments. Patterns that involve a set of mutually independent items or cover very few transactions are considered uninteresting because they may capture spurious relationships in the data. Such patterns can be eliminated by applying an

Evaluation of Association Patterns 371

objective intenestingness measure that uses statistics derived from data to determine whether a pattern is interesting. Examples of objective interest- ingness measures include support, confidence, and correlation.

The second set of criteria can be established through subjective arguments. A pattern is considered subjectively uninteresting unless it reveals unexpected information about the data or provides useful knowledge that can lead to profitable actions. For example, the rrle {Butter} - {Bread} may not be interesting, despite having high support and confidence values, because the relationship represented by the rule may seem rather obvious. On the other hand, the rule {Di,apers} ------ {Beer} is interesting because the relationship is quite unexpected and may suggest a new cross-selling opportunity for retailers. Incorporating subjective knowledge into pattern evaluation is a difficult task because it requires a considerable amount of prior information from the domain experts.

The following are some of the approaches for incorporating subjective knowledge into the pattern discovery task.

Visualization This approach requires a user-friendly environment to keep the human user in the loop. It also allows the domain experts to interact with the data mining system by interpreting and verifying the discovered patterns.

Template-based approach This approach allows the users to constrain the type of patterns extracted by the mining algorithm. Instead of reporting all the extracted rules, only rules that satisfy a user-specified template are returned to the users.

Subjective interestingness measure A subjective measure can be defined based on domain information such as concept hierarchy (to be discussed in Section 7.3) or profit margin of items. The measure can then be used to filter patterns that are obvious and non-actionable.

Readers interested in subjective interestingness measures may refer to re- sources listed in the bibliography at the end of this chapter.

6.7.t Objective Measures of Interestingness

An objective measure is a data-driven approach for evaluating the quality

of association patterns. It is domain-independent and requires minimal in- put from the users, other than to specify a threshold for filtering low-quality patterns. An objective measure is usually computed based on the frequency

6 . 7

372 Chapter 6 Association Analysis

Tabfe 6.7. A 2-way contingency table for variables A and B.

B B

A

A

J t l

J 0 1

J L O

T ./ 00

fr+

fo+

J + O t/

counts tabulated in a contingency table. Table 6.7 shows an example of a contingency table for a pair of binary variables, ,4 and B. We use the notation A (B) to indicate that ,4 (B) is absent from a transaction. Each entry fii in this 2 x 2 table denotes a frequency count. For example, fi1 is the number of times A and B appear together in the same transaction, while /e1 is the num- ber of transactions that contain B but not -4. The row sum fi-. represents the support count for A, while the column sum /a1 represents the support count for B. Finally, even though our discussion focuses mainly on asymmet- ric binary variables, note that contingency tables are also applicable to other attribute types such as symmetric binary, nominal, and ordinal variables.

Limitations of the support-confidence Flamework Existing associa- tion rule mining formulation relies on the support and confidence measures,to eliminate uninteresting patterns. The drawback of support was previously de- scribed in Section 6.8, in which many potentially interesting patterns involving low support items might be eliminated by the support threshold. The dra6- back of confidence is more subtle and is best demonstrated with the following example.

Example 6.3. Suppose we are interested in analyzing the relationship be- tween people who drink tea and coffee. We may gather information about the beverage preferences among a group of people and summarize their responses into a table such as the one shown in Table 6.8.

Table 6.8. Beverage preferences among a group of 1000 people.

Cof f ee Cof f ee

Tea

Tea

1 5 0

650

50

150

200

800

800 200 1000

Evaluation of Association Patterns 373

The information given in this table can be used to evaluate the association rule {?ea,}

------, {Cof f ee}. At fi.rst glance, it may appear that people who drink

tea also tend to drink coffee because the rule's support (15%) and confidence (75%) values are reasonably high. This argument would have been acceptable except that the fraction of people who drink coffee, regardless of whether they drink tea, is 80%, while the fraction of tea drinkers who drink coffee is only 75%. Thus knowing that a person is a tea drinker actually decreases her probability of being a coffee drinker from 80% to 75Tol The rule {Tea)

-,

{Cof f ee} is therefore misleading despite its high confidence value. r

The pitfall of confidence can be traced to the fact that the measure ignores the support of the itemset in the rule consequent. Indeed, if the support of coffee drinkers is taken into account, we would not be surprised to find that many of the people who drink tea also drink coffee. What is more surprising is that the fraction of tea drinkers who drink coffee is actually less than the overall fraction of people who drink coffee, which points to an inverse relationship between tea drinkers and coffee drinkers.

Because of rbhe limitations in the support-confidence framework, various objective measures have been used to evaluate the quality of association pat-

terns. Below, we provide a brief description of these measures and explain some of their strengths and limitations.

Interest Factor The tea-coffee example shows that high-confidence rules can sometimes be misleading because the confidence measure ignores the sup- port of the itemset appearing in the rule consequent. One way to address this problem is by applying a metric known as lift:

6 . 7

which computes the ratio between the rule's confidence and the support of the itemset in the rule consequent. For binary variables, Iift is equivalent to another objective measure called interest factor, which is defined as follows:

I ( 4 , B ) : s ( A , B ) _ N"frr

f+f +t ( 6 . 5 )

s ( , 4 ) x s ( B )

Interest factor compares the frequency of a pattern against a baseline fre- quency computed under the statistical independence assumption. The baseline frequency for a pair of mutually independent variables is

fn f+ f+t ir " h+f+t f

: 1 f r t ? , o r e q u i v a l e n t l y , f n : t : ; : :

( 6 . 4 )

( 6 . 6 )

p p

q

q

880

50

b t ,

20

930

70

930 70 1000

Chapter Association Analysis

Table 6.9. Contingency tables for the word pairs ({p,q} and {r,s}.

This equation follows from the standard approach of using simple fractions as estimates for probabilities. The fraction fnlN is an estimate for the joint probability P(A,B), while fia/,n/ and fyf N are the estimates for P(A) and P(B), respectively. lt A and B are statistically independent, then P(A,B): P(A) x P(B), thus leading to the formula shown in Equation 6.6. Using Equations 6.5 and 6.6, we can interpret the measure as follows:

I ( A , B ) 1, if ,4 and B arc independent; 1, if A and B are positively correlated; l, if A and B are negatively correlated.

( 6 . 7 )

For the tea-coffee example shown in Table 6.8, 1: O.H3_8- :0.9375, thus sug-

gesting a slight negative correlation between tea drinkers and coffee drinkers.

Limitations of Interest Factor We illustrate the limitation of interest factor with an example from the text mining domain. In the text domain, it is reasonable to assume that the association between a pair of words depends on the number of documents that contain both words. For example, because of their stronger association, we expect the words data and mining to appear together more frequently than the words compiler and mining in a collection of computer science articles.

Table 6.9 shows the frequency of occurrences between two pairs of words,

{p,q} and {","}. Using the formula given in Equation 6.5, the interest factor for {p,q} is 1.02 and for {r, s} is 4.08. These results are somewhat troubling for the following reasons. Although p and q appear together in 88% of the documents, their interest factor is close to 1, which is the value when p and q are statistically independent. On the other hand, the interest factor for {r, s} is higher than {p, q} even though r and s seldom appear together in the same document. Confidence is perhaps the better choice in this situation because it considers the association between p and q (9a.6%) to be much stronger than that between r and s (28.6%).

{ .

r r

s

5

20

C U

50

880

70

930

70 930 r000

Evaluation of Association Patterns 375

Correlation Analysis Correlation analysis is a statistical-based technique for analyzing relationships between a pair of variables. For continuous vari- ables, correl-ation is defined using Pearson's correlation coefficient (see Equa- tion 2.10 on page 77). For binary variables, correlation can be measured using the d-coefficient. which is defined as

6 . 7

( 6 . 8 )

The value of correlation ranges from -1 (perfect negative correlation) to *1 (perfect positive correlation). If the variables are statistically independent, then @ : 0. For example, the correlation between the tea and coffee drinkers given in Table 6.8 is -0.0625.

Limitations of Correlation Analysis The drawback of using correlation can be seen from the word association example given in Table 6.9. Although

the words p and g appear together more often than r and s, their /-coefficients are identical, i.e., Q(p,q): Q(r,s) :0.232. This is because the @-coefficient gives equal importance to both co-presence and co-absence of items in a trans-

action. It is therefore more suitable for analyzing symmetric binary variables.

Another limitation of this measure is that it does not remain invariant when

there are proportional changes to the sample size. This issue will be discussed

in greater detail when we describe the properties of objective measures on page

377.

IS Measure .I^9 is an alternative measure that has been proposed for han-

dling asymmetric binary variables. The measure is defined as follows:

rs(A, B) : ( 6 . e )

Note that .LS is large when the interest factor and support of the pattern

are large. For example, the value of 1^9 for the word pairs {p, q} and {r, s}

shown in Table 6.9 are 0.946 and 0.286, respectively. Contrary to the results given by interest factor and the @-coefficient, the 15 measure suggests that

the association between {p, q} i. stronger than {r, s}, which agrees with what

we expect from word associations in documents. It is possible to show that 15 is mathematically equivalent to the cosine

measure for binary variables (see Equation2.T on page 75). In this regard, we

fnfoo - fotfn

q q

p

p

800

100

100

0

900

1 0 0

900 100 1000

376 Chapter 6 Association Analysis

Table 6.10. Example of a contingency table for items p and q.

consider A and B as a pair of bit vectors, A o B : s(A,B) the dot product between the vectors, and lAl : \f(A) the magnitude of vector A. Therefore:

( 6 . 1 0 )

The 1,S measure can also be expressed as the geometric mean between the confidence of association rules extracted from a pair of binary variables:

I S ( 4 , B ) : ( 6 . 1 1 )

Because the geometric mean between any two numbers is always closer to the smaller number, the 1^9 value of an itemset {p, q} is low whenever one of its rules, p ---+ Q or e + p, has low confidence.

Limitations of rs Measure The 1,9 value for a pair of independent item- sets, A and B. is

IS1"a"o(,4, B) : s ( A , B ) s ( A ) x s ( B )

since the value depends on s(A) and s(B), IS shares a similar problem as the confidence measure-that the value of the measure can be quite large, even for uncorrelated and negatively correlated patterns. For example, despite the large 1^9 value between items p and q given in Table 6.10 (0.889), it is still less than the expected value when the items are statistically independent (ISi'auo : 0.9).

I S ( A , a s : - $ : , l ' " * , : c o s i , n e ( A , B ) .- \ - - r - ' J 4 A ) A B ) l a l x I n l

c ( A - - B ) x c ( B - - A ) .

s(.4) x s ( A ) x s ( B )

Evaluation of Association Patterns 377

Alternative Objective Interestingness Measures

Besides the measures we have described so far, there are other alternative mea- sures proposed for analyzing relationships between pairs of binary variables. These measures can be divided into two categories, symmetric and asym- metric measures. A measure M is symmetric if M(A --'- B): M(B ------ A). For example, interest factor is a symmetric measure because its value is iden- tical for the rules A ------+ B and B ---- A. In contrast, confidence is an asymmetric measure since the confidence for A ---. B and B -'----+,4. may not be the same. Symmetric measures are generally used for evaluating itemsets, while asymmetric measures are more suitable for analyzing association rules. Tables 6.11 and 6.12 provide the definitions for some of these measures in terms of the frequency counts of a 2 x 2 contingency table.

Consistency among Objective Measures

Given the wide variety of measures available, it is reasonable to question

whether the measures can produce similar ordering results when applied to a set of association patterns. If the measures are consistent, then we can choose any one of them as our evaluation metric. Otherwise, it is important to understand what their differences are in order to determine which measure is more suitable for analyzing certain types of patterns.

Tabfe 6.11. Examples of symmetric objective measures for the itemset {A, B}.

Measure (Symbol) Definition

Correlation (@)

Odds ratio (a)

Kappa (rc)

Interest (1)

Cosine (1S)

Piatetsky-Shapiro (P,9)

Collective strength (S)

Jaccard (o

All-confidence (h)

N " f r r - " f r + f + r t;_-i--

v J r + J + 1 J 0 + J + o / . ? \ l l o a \

\ I t t I o o ) / ( , 1 ' t o " r o t /

N"fr r +Nloq lrr.ffl:/ot_lt_q--Nt=1+ f-+r=j;+j+"

/ ^ r r \ l l r r \ \ 1 \ J t l ) / U 1 + J + 1 / / r \ l/ /-.-.a'-- r-\

\ I r t ) / \ t / I t + J + r )

. f l t J L + J + 1 M

- --T-

f u f ( f r + + f + t

- i " [ # , # ]

6 . 7

Tabfe 6.12. Examples of asymmetric objective measures for the rule A ------+ B.

Measure (Symbol) Definition

Goodman-Kruskal (.\)

Mutual Information (M)

J-Measure (J)

Gini index (G)

Laplace (.L)

Conviction (V)

Certainty factor (F)

Added Value (AV)

(Di *** f 1x - rnarxf +*)/(N - max;, fup)

(D,; D, # to* #tl t t- D, + t"g #) # t " g # + f f r o g # # * (#)'+ (#)'l - (+)'

+#'t(#)'+(#)'l -(#)' ( / " + t ) 1 f f p + z ) (fr+f +o) l(w f,o)

(# - #)t0- +) J 7 1 _ J + 1

378 Chapter 6 Association Analysis

Table 6.13. Example of contingency tables.

Example . t 1 7 . t 1 0 J 0 1 foo E1 E2 Es Ea E5 E6 R.

Es Es Erc

8 12 3 8330 3954 2886 1500 4000 9481 4000 7450 6 1

83 2

3080 1363 2000 2000 298

2000 2483 2483

424 6 2 2 5

1320 500 1000 \ 2 7

2000 4 /l

1370 r046 296r 4431 6000 3000 94

2000 63

7452

Suppose the symmetric and asymmetric measures are applied to rank the ten contingency tables shown in Table 6.13. These contingency tables are cho- sen to illustrate the differences among the existing measures. The ordering produced by these measures are shown in Tables 6.14 and 6.15, respectively (with 1 as the most interesting and 10 as the least interesting table). Although some of the measures appear to be consistent with each other, there are certain measures that produce quite different ordering results. For example, the rank- ings given by the fcoefficient agree with those provided by rc and collective strength, but are somewhat different than the rankings produced by interest

o a K I IS PS ^9 C h n 1 E2 E3 Ea E5 E6 E7 E8 Es E r c

1 2 3 i

5 6 ,7

8 9 1 0

3 1 2 8 n I

9 6 . 1 0 4 5

I 2 A

3 0

5 ,7

8 I 1 0

tt 7 ^

3 2 5 I 8 1 0 1

2 3 5 F7 I

q

o 1 8 4

1 0

2 r

1 3 6 4 8 F7

I 1 0

1 2 3 4 6 E J

7 8 q

1 0

2 3 o

7 q

l)

1 8 4 1 0

2 3 8 5 9 7

1 1 I

4 1 0

6.7 Evaluation of Association Patterns 379

Table 6.14. Rankings of contingency tables using the symmetric measures given in Table 6.11.

Table 6.15. Rankings of contingency tables using the asymmetric measures given in Table 6.12.

factor and odds ratio. Furthermore, a contingency table such as -E1s is ranked lowest according to the @-coefficient, but highest according to interest factor.

Properties of Objective Measures

The results shown in Table 6.14 suggest that a significant number of the mea-

sures provide conflicting information about the quality of a pattern. To under- stand their differences, we need to examine the properties of these measures.

Inversion Property Consider the bit vectors shown in Figure 6.28. The

0/1 bit in each column vector indicates whether a transaction (row) contains a particular item (column). For example, the vector A indicates that item a

) M J G L V F AV E1 E2 E3 Ea E5 E6 E7 Eg Es E t o

I 2 tr

4 q

3 7 8 6 1 0

1 r

2 3 o F7

8 5 9 4 1 0

1 2 r

3 4 6 9 7

1 0 8

1 .) 2 4 6 r

8 ,7

I 1 0

4 tr J

2 q

8 F7

3 1 0 '|

h

2 1 6 3 D

4 .7

8 I 1 0

2 I o 3 tr d

n

7 8 q

1 0

r J

6 4 I 2 3 q

7 1 0 8

380 Chapter 6 Association

Figure 6,28. Effect of the inversion operation. The vectors C and E are inversions of vector A, while the vector D is an inversion of vectors B and F.

belongs to the first and last transactions, whereas the vector B indicates that item b is contained only in the fifth transaction. The vectors C and E are in fact related to the vector A-their bits have been inverted from 0's (absence) to l's (presence), and vice versa. Similarly, D is related to vectors B and F by inverting their bits. The process of flipping a bit vector is called inversion. If a measure is invariant under the inversion operation, then its value for the vector pair (C, D) should be identical to its value for (A, B). The inversion property of a measure can be tested as follows.

Definition 6.6 (Inversion Property). An objective measure M is invariant under the inversion operation if its value remains the same when exchanging the frequency counts fi1 with /66 and fis with /61.

Among the measures that remain invariant under this operation include the @-coefficient, odds ratio, n, and collective strength. These measures may not be suitable for analyzing asymmetric binary data. For example, the /- coefficient between C and D is identical to the @-coefficient between A and B, even though items c and d appear together more frequently than a and b. Fbrthermore, the d-coefficient between C and D is less than that between E and F even though items e and / appear together only once! We had previously raised this issue when discussing the limitations of the fcoefficient on page 375. For asymmetric binary data, measures that do not remain invariant under the inversion operation are preferred. Some of the non-invariant measures include interest factor, IS, PS, and the Jaccard coefficient.

E F

(c)

Analysis

C D

(b)

B

)

A

(

6,7 Evaluation of Association Patterns 381

NuIl Addition Property Suppose we are interested in analyzing the re-

lationship between a pair of words, such as data and rnining, in a set of

documents. If a collection of articles about ice fishing is added to the data set,

should the association between data and mining be affected? This process of

adding unrelated data (in this case, documents) to a given data set is known

as the null addition operation.

Deffnition 6.7 (Null Addition Property). An objective measure M is

invariant under the null addition operation if it is not affected by increasing

/es, while all other frequencies in the contingency table stay the same.

For applications such as document analysis or market basket analysis, the

measure is expected to remain invariant under the null addition operation.

Otherwise, the relationship between words may disappear simply by adding

enough documents that do not contain both words! Examples of measures

that satisfy this property include cosine (19) and Jaccard ({) measures, while

those that viqlate this property include interest factor, PS, odds ratio, and

the fcoefficient.

Scaling Property Table 6.16 shows the contingency tables for gender and

the grades achieved by students enrolled in a particular course in 1993 and

2004. The data in these tables showed that the number of male students has