Decision Support system

Business Intelligence and Analytics Systems for Decision Support TENTH EDITION

Ramesh Sharda • Dursun Delen • Efraim Turban

PEARSONA L W A Y S L E A R N I N G

T e n t h E d i t i o n

B u s i n e s s I n t e l l i g e n c e

a n d A n a l y t i c s :

S y s t e m s f o r D e c i s i o n S u p p o r t

Global Edition

Ramesh Sharda Oklahoma State University

Dursun Delen Oklahoma State University

Efraim Turban University o f Hawaii

With contributions by

J . E. A ronson The University o f Georgia

T ing-Peng Liang National Sun Yat-sen University

David King JDA Software Group, Inc.

PEARSON B o s t o n C o lu m b u s I n d ia n a p o l is N e w Y o r k S a n F r a n c i s c o U p p e r S a d d le R iv e r

A m s te r d a m C a p e T o w n D u b a i L o n d o n M a d rid M ila n M u n i c h P a r is M o n t r e a l T o r o n t o D e l h i M e x i c o C ity S a o P a u l o S y d n e y H o n g K o n g S e o u l S i n g a p o r e T a i p e i T o k y o

Editor in Chief: Stephanie Wall Executive Editor: Bob Horan P ublisher, Global Edition: Laura Dent Senior Acquisitions Editor, Global Edition: Steven

Jackson Program M anager Team Lead: Ashley Santora Program M anager: Denise Vaughn Marketing Manager, International: Kristin Schneider P ro ject M anager Team Lead: Judy Leale P ro ject M anager: Tom Benfatti Assistant P roject Editor, Global Edition: Paromita

Baneijee

Pearson Education Limited Edinburgh Gate Harlow Essex CM20 2JE England and Associated Companies throughout the world Visit us on the World Wide Web at: www.pearsonglobaleditions.com

© Pearson Education Limited 2014 The rights of Ramesh Sharda, Dursun Delen, and Efraim Turban to be identified as the authors of this work have been asserted by them in accordance with the Copyright, Designs, and Patents Act 1988. Authorized adaptation f r o m the United States edition, entitled Business Intelligence a n d Analytics: Systems f o r D ecision Support, l ( f h edition, ISBN978-0-133-05090-5, by Ramesh Sharda, Dursun Delen, a n d E fraim Turban, p u b lish ed by P earson E ducation © 2014. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmittedin any form or by any means, electronic, mechanical, photocopying, recording or otheiwise, withouteither the prior written permission o f the publisher or a license permitting restricted copying in the United Kingdom issued by the Copyright Licensing Agency Ltd, Saffron House, 6-10 Kirby Street, London EC1N 8TS. All trademarks used herein are the property o f their respective owners.The use of any trademark in this text does not vest in the author or publisher any trademark ownership rights in such trademarks, nor does the use of such trademarks imply any affiliation with or endorsement of this book by such owners. Microsoft and/or its respective suppliers make no representations about the suitability of the information contained in the documents and related graphics published as part of the services for any purpose. All such documents and related graphics are provided “as is” without warranty o f any kind. Microsoft and/or its respective suppliers hereby disclaim all warranties and conditions with regard to this information, including all warranties and conditions of merchantability, whether express, implied or statutory, fitness for a particular purpose, title and non-infringement. In no event shall Microsoft and/or its respective suppliers be liable for any special, indirect or consequential damages or any damages whatsoever resulting from loss of use, data or profits, whether in an action of contract, negligence or other tortious action, arising out o f or in connection with the use or performance o f information available from the sendees. The documents and related graphics contained herein could include technical inaccuracies or typographical errors. Changes are periodically added to the information herein. Microsoft and/or its respective suppliers may make improvements and/or changes in the product(s) and/or the program(s) described herein at any time. Partial screen shots may be viewed in hill within the software version specified. Microsoft® and Windows® are registered trademarks of the Microsoft Corporation in the U.S.A. and other countries. This book is not sponsored or endorsed by or affiliated with the Microsoft Corporation.

ISBN 10: 1-292-00920-9 ISBN 13: 978-1-292-00920-9 British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library

9 8 7 6 5 4 3 2 1 14 13 12 11 10 Typeset in ITC Garamond Std. Integra Software Solutions Printed and bound by Courier Kendalville in The United States o f America

Operations Specialist: Michelle Klein Senior Manufacturing Controller, Production: Trudy

Kimber Creative D irector: Jayne Conte Cover Im age Credit: © Robert Adrian Hillman Cover P rin ter: Courier Kendallville Cover D esig n er: Jodi Notowitz at Wicked Design Full-Service P ro je c t Managem ent: George Jacob,

Integra Software Solutions. T ex t Font: ITC Garamond Std

BRIEF CONTENTS

Preface 21 A b o u t t h e A u t h o r s 2 9

P A R T I D e c is io n M a k in g a n d A n a ly t ic s : A n O v e r v ie w 31 C h a p te r 1 A n O v e r v ie w o f Bu sin ess In te llig e n c e , A n a ly tic s ,

a n d D e cisio n S u p p o r t 32 C h a p te r 2 F o u n d a tio n s a n d T e c h n o lo g ie s f o r D e c is io n M a k in g 67

P A R T II D e s c r ip t iv e A n a l y t i c s 107 C h a p te r 3 D a ta W a r e h o u s in g 108 C h a p te r 4 Bu sin e ss R e p o rtin g , V is u a l A n a ly tic s , a n d Business

P e r fo r m a n c e M a n a g e m e n t 165

P A R T III P r e d ic t iv e A n a l y t i c s 2 1 5 C h a p te r 5 D a ta M in in g 216

C h a p te r 6 T e c h n iq u e s f o r P re d ic tiv e M o d e lin g 273 C h a p te r 7 T e x t A n a ly tic s , T e x t M in in g , a n d S e n t im e n t A n a ly s is 318 C h a p te r 8 W e b A n a ly tic s , W e b M in in g , a n d S o c ia l A n a ly tic s 368

P A R T IV P r e s c r ip t iv e A n a l y t i c s 421 C h a p te r 9 M o d e l- B a s e d D e c is io n M a k in g : O p tim iz a tio n a n d M u lti-

C rite ria S ystem s 422 C h a p te r 10 M o d e lin g a n d A n a lys is: H e u ris tic S e a rc h M e t h o d s a n d

S im u la t io n 465 C h a p te r 11 A u t o m a t e d D e cisio n System s a n d E x p e rt S ystem s 499 C h a p te r 12 K n o w le d g e M a n a g e m e n t a n d C o lla b o r a tiv e System s 537

P A R T V B ig D a ta a n d F u t u r e D ir e c t io n s f o r B u s in e s s A n a l y t i c s 571 C h a p te r 13 B ig D a ta a n d A n a ly tic s 572 C h a p te r 14 Bu sin ess A n a ly tic s : E m e rg in g T re n d s a n d F u tu re

Im p a c ts 622

Glossary 664 Index 678

CONTENTS

Preface 21 A b o u t the A u th o rs 29

P a r t I D e c is io n M a k in g a n d A n a l y t ic s : A n O v e r v ie w 31 C h a p te r 1 An O verview o f Business Intelligence, Analytics, and

Decision Support 32 1.1 O p e n in g V ig n e t t e : M a g p ie S e n s in g E m p lo y s A n a ly tic s

M a n a g e a V a c c in e S u p p ly C h a in E ff e c tiv e ly a n d S a fe ly

1.2 C h a n g in g Bu sin ess E n v ir o n m e n ts a n d C o m p u te r iz e d D e cis io n S u p p o r t 35 The Business Pressures-Responses-Support Model 35

1.3 M a n a g e r ia l D e c is io n M a k in g 37 The Nature of Managers' Work 37 The Decision-Making Process 38

1.4 In fo r m a tio n S ystem s S u p p o r t f o r D e cis io n M a k in g 3 1.5 A n E a r ly F r a m e w o r k f o r C o m p u te r iz e d D ecisio n

S u p p o r t 41 The Gorry and Scott-Morton Classical Framework 41 Computer Support for Structured Decisions 42

Computer Support for Unstructured Decisions 43 Computer Support for Semistructured Problems 43

1.6 T h e C o n c e p t o f D e c is io n S u p p o r t S ys te m s (D S S ) 43 DSS as an Umbrella Term 43 Evolution of DSS into Business Intelligence 44

1.7 A F r a m e w o r k f o r Bu sin e ss In te llig e n c e (B l) 44 Definitions of Bl 44 A Brief History of Bl 44 The Architecture of Bl 45 Styles of Bl 45 The Origins and Drivers of Bl 46 A Multimedia Exercise in Business Intelligence 46 ► APPLICATION CASE 1.1 Sabre Helps Its Clients Through

and Analytics 47 The DSS-BI Connection 48

1.8 Bu sin e ss A n a ly tic s O v e r v ie w 49 Descriptive Analytics 50 ► APPLICATION CASE 1.2 Eliminating Inefficiencies at Seattle

Children's Hospital 51 ► APPLICATION CASE 1.3 Analysis a t the Speed of Thought 53 Predictive Analytics 52

Contents 5

► APPLICATION CASE 1.4 Moneyball: Analytics in Sports and Movies 53 ► APPLICATION CASE 1.5 Analyzing Athletic Injuries 54 Prescriptive Analytics 54 ► APPLICATION CASE 1.6 Industrial and Commercial Bank of China

(ICBC) Employs Models to Reconfigure Its Branch Network 55 Analytics Applied to Different Domains 56 Analytics or Data Science? 56

1.9 B r ie f In tr o d u c tio n t o B ig D a ta A n a ly tic s 57 What Is Big Data? 57 ► APPLICATION CASE 1.7 Gilt Groupe's Flash Sales Streamlined by Big

Data Analytics 59 1.10 P la n o f t h e B o o k 59

Part I: Business Analytics: An Overview 59 Part II: Descriptive Analytics 60 Part III: Predictive Analytics 60 Part IV: Prescriptive Analytics 61 Part V: Big Data and Future Directions for Business Analytics 61

1.11 R eso u rce s, Links, a n d t h e T e r a d a ta U n iv e r s ity N e tw o r k C o n n e c tio n 61 Resources and Links 61 Vendors, Products, and Demos 61 Periodicals 61 The Teradata University Network Connection 62 The Book's Web Site 62 Ch apter H ig h lig hts 62 • K ey Terms 63 Q uestions fo r Discussion 63 • Exercises 63 ► END-OF-CHAPTER APPLICATION CASE Nationwide Insurance Used Bl

to Enhance Customer Service 64 References 65

C h a p te r 2 Foundations and Technologies for Decision M aking 67 2.1 O p e n in g V ig n e t t e : D e c is io n M o d e lin g a t H P U sin g

S p r e a d s h e e ts 68 2.2 D e cisio n M a k in g : In tr o d u c tio n a n d D e fin itio n s 70

Characteristics of Decision Making 70 A Working Definition of Decision Making 71 Decision-Making Disciplines 71 Decision Style and Decision Makers 71

2.3 P h a se s o f t h e D e c is io n - M a k in g Process 72 2.4 D e c is io n M a k in g : T h e In te llig e n c e P h a se 74

Problem (or Opportunity) Identification 75 ► APPLICATION CASE 2.1 Making Elevators Go Faster! 75

Problem Classification 76 Problem Decomposition 76 Problem Ownership 76

2.5 D e c is io n M a k in g : T h e D e sig n P h a s e 77

Models 77 Mathematical (Quantitative) Models 77 The Benefits of Models 77 Selection of a Principle of Choice 78 Normative Models 79 Suboptimization 79 Descriptive Models 80 Good Enough, or Satisficing 81 Developing (Generating) Alternatives 82 Measuring Outcomes 83

Risk 83 Scenarios 84 Possible Scenarios 84 Errors in Decision Making 84

2.6 D e c is io n M a k in g : T h e C h o ic e P h a s e 85 2.7 D e cisio n M a k in g : T h e Im p le m e n ta tio n P h a s e 85

2.8 H o w D e cisio n s A r e S u p p o r te d 86 Support for the Intelligence Phase 86 Support for the Design Phase 87 Support for the Choice Phase 88 Support for the Implementation Phase 88

2.9 D e c is io n S u p p o r t System s: C a p a b ilitie s 89 A DSS Application 89

2.10 DSS C la ss ific a tio n s 91 The AIS SIGDSS Classification for DSS 91

Other DSS Categories 93 Custom-Made. S'fste.ccvs, Vex'.us Read^-Wlade Systems 93

2.11 C o m p o n e n ts o f D e c is io n S u p p o r t S ys te m s 94 The Data Management Subsystem 95 The Model Management Subsystem 95 ► APPLICATION CASE 2.2 Station Casinos Wins by Building Customer

Relationships Using Its Data 96 ► APPLICATION CASE 2.3 SNAP DSS Helps OneNet Make

Telecommunications Rate Decisions 98

The User Interface Subsystem 98 The Knowledge-Based Management Subsystem 99 ► APPLICATION CASE 2.4 From a Game Winner to a Doctor! 100 C h ap ter H ig h lig hts 102 • K e y Terms 103 Questions fo r Discussion 103 • Exercises 104 ► END-OF-CHAPTER APPLICATION CASE Logistics Optimization in a

Major Shipping Company (CSAV) 104 References 105

P a r t II D e s c r ip t iv e A n a l y t i c s 1 07 C h a p te r 3 Data W arehousing 108

3.1 O p e n in g V ig n e t t e : Isle o f C a p ri C asin os Is W i n n in g w it h E n te rp ris e D a ta W a r e h o u s e 109

3.2 D a ta W a r e h o u s in g D e fin itio n s a n d C o n c e p ts 111 What Is a Data Warehouse? 111

A Historical Perspective to Data Warehousing 111 Characteristic of Data Warehousing 113 Data Marts 114 Operational Data Stores 114

Enterprise Data Warehouses (EDW) 115 Metadata 115 ► APPLICATION CASE 3.1 A Better Data Plan: Well-Established TELCOs

Leverage Data Warehousing and Analytics to Stay on Top in a Competitive Industry 115

3.3 D a ta W a r e h o u s in g Process O v e r v ie w 117 ► APPLICATION CASE 3.2 Data Warehousing Helps MultiCare Save

More Lives 118

3.4 D a ta W a r e h o u s in g A r c h ite c tu r e s 120 Alternative Data Warehousing Architectures 123 Which Architecture Is the Best? 126

3.5 D a ta In te g r a tio n a n d t h e E x tra c tio n , T r a n s fo r m a tio n , a n d L o a d (E T L ) Processes 127 Data Integration 128 ► APPLICATION CASE 3.3 BP Lubricants Achieves BIGS Success 128 Extraction, Transformation, and Load 130

3.6 D a ta W a r e h o u s e D e v e lo p m e n t 132 ► APPLICATION CASE 3.4 Things Go Better with Coke's Data

Warehouse 133

Data Warehouse Development Approaches 133 ► APPLICATION CASE 3.5 Starwood Hotels & Resorts Manages Hotel

Profitability with Data Warehousing 136

Additional Data Warehouse Development Considerations 137 Representation of Data in Data Warehouse 138 Analysis of Data in the Data Warehouse 139 OLAP Versus OLTP 140 0LAP Operations 140

3.7 D a ta W a r e h o u s in g Im p le m e n ta tio n Issues 143 ► APPLICATION CASE 3.6 EDW Helps Connect State Agencies in

Michigan 145

Massive Data Warehouses and Scalability 146

3.8 R eal-T im e D a ta W a r e h o u s in g 147 ► APPLICATION CASE 3.7 Egg Pic Fries the Competition in Near Real

Time 148

3.9 D a ta W a r e h o u s e A d m in is tr a tio n , S e c u r ity Issues, a n d Fut_-~ T re n d s 151 The Future of Data Warehousing 153

3.10 R eso u rce s, Links, a n d t h e T e r a d a ta U n iv e rs ity N e tw o r k C o n n e c tio n 156

Resources and Links 156 Cases 156 Vendors, Products, and Demos 157 Periodicals 157 Additional References 157 The Teradata University Network (TUN) Connection 157 Ch apter H ig h lig hts 158 • K e y Terms 158 Questions f o r Discussion 158 • Exercises 159 ► END-OF-CHAPTER APPLICATION CASE Continental Airlines Flies High

with Its Real-Time Data Warehouse 161 R eferences 162

C h a p te r 4 Business Reporting, Visual Analytics, and Business Perform ance M anagem ent 165

4.1 O p e n in g V ig n e tte :S e lf- S e rv ic e R e p o rtin g E n v ir o n m e n t S a ve s M illio n s f o r C o r p o r a te C u sto m e rs 166

4.2 Bu sin ess R e p o rtin g D e fin itio n s a n d C o n c e p ts 169 What Is a Business Report? 170 ► APPLICATION CASE 4.1 Delta Lloyd Group Ensures Accuracy and

Efficiency in Financial Reporting 171 Components of the Business Reporting System 173 ► APPLICATION CASE 4.2 Flood of Paper Ends at FEMA 174

4.3 D a ta a n d In fo r m a tio n V is u a liz a tio n 175 ► APPLICATION CASE 4.3 Tableau Saves Blastrac Thousands of Dollars

with Simplified Information Sharing 176 A Brief History of Data Visualization 177 ► APPLICATION CASE 4.4 TIBCO Spotfire Provides Dana-Farber Cancer

Institute with Unprecedented Insight into Cancer Vaccine Clinical Trials 179

4.4 D if f e r e n t T yp e s o f C h a rts a n d G r a p h s 180 Basic Charts and Graphs 180

Spedalized Charts and Graphs 181 4.5 T h e E m e rg e n c e o f D a ta V is u a liz a tio n a n d V is u a l

A n a ly tic s 184 Visual Analytics 186 High-Powered Visual Analytics Environments 188

4.6 P e r fo r m a n c e D a s h b o a rd s 190 ► APPLICATION CASE 4.5 Dallas Cowboys Score Big with Tableau and

Teknion 191

Dashboard Design 192 ► APPLICATION CASE 4.6 Saudi Telecom Company Excels with

Information Visualization 193

What to Look For in a Dashboard 194

Best Practices in Dashboard Design 195 Benchmark Key Performance Indicators with Industry Standards 195

Wrap the Dashboard Metrics with Contextual Metadata 195

Validate the Dashboard Design by a Usability Specialist 195

Prioritize and Rank Alerts/Exceptions Streamed to the Dashboard 195

Enrich Dashboard with Business Users' Comments 195

Present Information in Three Different Levels 196 Pick the Right Visual Construct Using Dashboard Design Principles 196

Provide for Guided Analytics 196 4.7 Bu sin ess P e r fo r m a n c e M a n a g e m e n t 196

Closed-Loop BPM Cycle 197 ► APPLICATION CASE 4.7 IBM Cognos Express Helps Mace for Faster

and Better Business Reporting 199

4.8 P e r fo r m a n c e M e a s u r e m e n t 200 Key Performance Indicator (KPI) 201

Performance Measurement System 202 4.9 B a la n c e d S c o re c a rd s 202

The Four Perspectives 203

The Meaning of Balance in BSC 204

Dashboards Versus Scorecards 204 4.10 Six S ig m a as a P e r fo r m a n c e M e a s u r e m e n t S y s te m 205

The DMAIC Performance Model 206

Balanced Scorecard Versus Six Sigma 206

Effective Performance Measurement 207 ► APPLICATION CASE 4.8 E xpedia.com 's Customer Satisfaction

Scorecard 208 Ch apter H ig h lig hts 209 • K ey Terms 210 Q uestions fo r Discussion 211 • Exercises 211 ► END-OF-CHAPTER APPLICATION CASE Smart Business Reporting

Helps Healthcare Providers Deliver Better Care 212 R eferences 214

P a r t III P r e d ic t iv e A n a l y t i c s 2 1 5 C h a p te r 5 Data M ining 216

5.1 O p e n in g V ig n e t t e : C a b e la 's R e e ls in M o r e C u s to m e rs w it h A d v a n c e d A n a ly tic s a n d D a ta M in in g 217

5.2 D a ta M in in g C o n c e p ts a n d A p p lic a tio n s 219 ► APPLICATION CASE 5.1 Smarter Insurance: Infinity P&C Improves

Customer Service and Combats Fraud with Predictive Analytics 221

1 0 Contents

Definitions, Characteristics, and Benefits 222 ► APPLICATION CASE 5.2 Harnessing Analytics to Combat Crime:

Predictive Analytics Helps Memphis Police Department Pinpoint Crime and Focus Police Resources 226

How Data Mining Works 227 Data Mining Versus Statistics 230

5.3 D a ta M in in g A p p lic a tio n s 231 ► APPLICATION CASE 5.3 A Mine on Terrorist Funding 233

5.4 D a ta M in in g Pro cess 234 Step 1: Business Understanding 235 Step 2: Data Understanding 235 Step 3: Data Preparation 236 Step 4: Model Building 238 ► APPLICATION CASE 5.4 Data Mining in Cancer Research 240 Step 5: Testing and Evaluation 241 Step 6: Deployment 241 Other Data Mining Standardized Processes and Methodologies 242

5.5 D a ta M in in g M e th o d s 244 Classification 244 Estimating the True Accuracy of Classification Models 245 Cluster Analysis for Data Mining 250 ► APPLICATION CASE 5.5 2degrees Gets a 1275 Percent Boost in Churn

Identification 251 Association Rule Mining 254

5.6 D a ta M in in g S o f t w a r e T o o ls 258 ► APPLICATION CASE 5.6 Data Mining Goes to Hollywood: Predicting

Financial Success of Movies 261 5.7 D a ta M in in g P riv a c y Issues, M y th s , a n d B lu n d e rs 264

Data Mining and Privacy Issues 264 ► APPLICATION CASE 5.7 Predicting Customer Buying Patterns— The

Target Story 265 Data Mining Myths and Blunders 266 C h ap ter H ig h lig h ts 267 • K e y Terms 268 Questions f o r Discussion 2 6 8 • Exercises 269 ► END-OF-CHAPTER APPLICATION CASE Macys.com Enhances Its

Customers' Shopping Experience with Analytics 271 R eferences 271

C h a p te r 6 Techniques fo r Predictive M odeling 273 6.1 O p e n in g V ig n e t t e : P r e d ic t iv e M o d e lin g H e lp s B e t t e r

U n d e rs ta n d a n d M a n a g e C o m p le x M e d ic a l P ro c e d u re s 274

6.2 Ba sic C o n c e p ts o f N e u r a l N e tw o r k s 277 Biological and Artificial Neural Networks 278 ► APPLICATION CASE 6.1 Neural Networks Are Helping to Save Lives in

the Mining Industry 280 Elements of ANN 281

C ontents 11

Network Information Processing 282 Neural Network Architectures 284 ► APPLICATION CASE 6.2 Predictive Modeling Is Powering the Power

Generators 286 6.3 D e v e lo p in g N e u ra l N e tw o r k - B a s e d S ystem s 288

The General ANN Learning Process 289 Backpropagation 290

6.4 Illu m in a tin g t h e B la c k B o x o f A N N w it h S e n s itiv ity A n a ly s is 292 ► APPLICATION CASE 6.3 Sensitivity Analysis Reveals Injury Severity

Factors in Traffic Accidents 294

6.5 S u p p o r t V e c t o r M a c h in e s 295 ► APPLICATION CASE 6.4 Managing Student Retention with Predictive

Modeling 296 Mathematical Formulation of SVMs 300 Primal Form 301 Dual Form 301 Soft Margin 301 Nonlinear Classification 302 Kernel Trick 302

6.6 A Process-Based A p p r o a c h t o t h e U se o f S V M 303 Support Vector Machines Versus Artificial Neural Networks 304

6.7 N e a r e s t N e ig h b o r M e t h o d f o r P re d ic tio n 305 Similarity Measure: The Distance Metric 306 Parameter Selection 307 ► APPLICATION CASE 6.5 Efficient Image Recognition and

Categorization with kNN 308 Ch ap ter H ighlights 310 • K ey Terms 310 Q uestions fo r Discussion 311 • Exercises 311 ► END-OF-CHAPTER APPLICATION CASE Coors Improves Beer Flavors

with Neural Networks 314 R eferences 315

C h a p te r 7 Text Analytics, Text M ining, and Sentim ent Analysis 318 7.1 O p e n in g V ig n e t t e : M a c h in e V e rsu s M e n o n Jeo pa rdy7: T h e

S to ry o f W a t s o n 319 7.2 T e x t A n a ly tic s a n d T e x t M in in g C o n c e p ts a n d

D e fin itio n s 321 ► APPLICATION CASE 7.1 Text Mining for Patent Analysis 325

7.3 N a tu r a l L a n g u a g e P ro ce ssin g 326 ► APPLICATION CASE 7.2 Text Mining Improves Hong Kong

Government's Ability to Anticipate and Address Public Complaints 328

7.4 T e x t M in in g A p p lic a tio n s 330 Marketing Applications 331 Security Applications 331 ► APPLICATION CASE 7.3 Mining for Lies 332

Biomedical Applications 334

1 2 Contents

Academic Applications 335 ► APPLICATION CASE 7.4 Text Mining and Sentiment Analysis Help

Improve Customer Service Performance 336 7.5 T e x t M in in g Pro cess 337

Task 1: Establish the Corpus 338

Task 2; Create the Term-Document Matrix 339

. Task 3: Extract the Knowledge 342 ► APPLICATION CASE 7.5 Research Literature Survey with Text

Mining 344 7.6 T e x t M in in g T o o ls 347

Commercial Software Tools 347

Free Software Tools 347 ► APPLICATION CASE 7.6 A Potpourri of Text Mining Case Synopses 348

7.7 S e n t im e n t A n a ly s is O v e r v ie w 349 ► APPLICATION CASE 7.7 Whirlpool Achieves Customer Loyalty and

Product Success with Text Analytics 351 7.8 S e n t im e n t A n a ly s is A p p lic a tio n s 353 7.9 S e n t im e n t A n a ly s is Process 355

Methods for Polarity Identification 356

Using a Lexicon 357

Using a Collection of Training Documents 358 Identifying Semantic Orientation of Sentences and Phrases 358

Identifying Semantic Orientation of Document 358

7.10 S e n t im e n t A n a ly s is a n d S p e e c h A n a ly tic s 359 How Is It Done? 359 ► APPLICATION CASE 7.8 Cutting Through the Confusion: Blue Cross

Blue Shield of North Carolina Uses Nexidia's Speech Analytics to Ease Member Experience in Healthcare 361

C h apter H ig h lig hts 363 • K e y Terms 363 Q uestions fo r Discussion 364 • Exercises 364 ► END-OF-CHAPTER APPLICATION CASE BBVA Seamlessly Monitors

and Improves Its Online Reputation 365 References 366

C h a p te r 8 W eb Analytics, W eb M ining, and Social Analytics 368 8.1 O p e n in g V ig n e t t e : S e c u r ity F irs t In s u ra n c e D e e p e n s

C o n n e c tio n w it h P o lic y h o ld e rs 369 8.2 W e b M in in g O v e r v ie w 371 8.3 W e b C o n te n t a n d W e b S tr u c tu r e M in in g 374

► APPLICATION CASE 8.1 Identifying Extremist Groups with Web Link and Content Analysis 376

8.4 S e a rc h E n g in e s 377 Anatomy of a Search Engine 377 1. Development Cycle 378 Web Crawler 378 Document Indexer 378

2. Response Cycle 379 Query Analyzer 379 Document Matcher/Ranker 379 How Does Google Do It? 381 ► APPLICATION CASE 8.2 IGN Increases Search Traffic by 1500 Percent 383

8.5 S e a rc h E n g in e O p tim iz a tio n 384 Methods for Search Engine Optimization 385 ► APPLICATION CASE 8.3 Understanding Why Customers Abandon

Shopping Carts Results in $10 Million Sales Increase 387

8.6 W e b U s a g e M in in g ( W e b A n a ly tic s ) 388 Web Analytics Technologies 389 ► APPLICATION CASE 8.4 Allegro Boosts Online Click-Through Rates by

500 Percent with Web Analysis 390 Web Analytics Metrics 392 Web Site Usability 392 Traffic Sources 393 Visitor Profiles 394 Conversion Statistics 394

8.7 W e b A n a ly tic s M a t u r it y M o d e l a n d W e b A n a ly tic s T o o ls 396 Web Analytics Tools 398 Putting It All Together— A Web Site Optimization Ecosystem 400 A Framework for Voice of the Customer Strategy 402

8.8 S o c ia l A n a ly tic s a n d S o c ia l N e t w o r k A n a ly s is 403 Social Network Analysis 404 Social Network Analysis Metrics 405 ► APPLICATION CASE 8.5 Social Network Analysis Helps

Telecommunication Firms 405 Connections 406 Distributions 406 Segmentation 407

8.9 S o c ia l M e d ia D e fin itio n s a n d C o n c e p ts 407 How Do People Use Social Media? 408 ► APPLICATION CASE 8.6 Measuring the Impact of Social Media at

Lollapalooza 409

8.10 S o c ia l M e d ia A n a ly tic s 410 Measuring the Social Media Impact 411 Best Practices in Social Media Analytics 411 ► APPLICATION CASE 8.7 eHarmony Uses Social Media to Help Take the

Mystery Out of Online Dating 413 Social Media Analytics Tools and Vendors 414 Ch apter H ig h lig hts 416 • K e y Terms 417 Q uestions fo r Discussion 417 • Exercises 418 ► END-OF-CHAPTER APPLICATION CASE Keeping Students on Track with

Web and Predictive Analytics 418 References 420

P a r t IV P r e s c r ip t iv e A n a l y t i c s 421 C h a p te r 9 Model-Based Decision M aking: O ptim ization and

M ulti-Criteria System s 422 9.1 O p e n in g V ig n e t t e : M id w e s t IS O S a ve s B illio n s b y B e t t e r

P la n n in g o f P o w e r P la n t O p e r a tio n s a n d C a p a c ity P la n n in g 423

9.2 D e c is io n S u p p o r t System s M o d e lin g 424 ► APPLICATION CASE 9.1 Optimal Transport for ExxonMobil

Downstream Through a DSS 425 Current Modeling Issues 426 ► APPLICATION CASE 9.2 Forecasting/Predictive Analytics Proves to Be

a Good Gamble for Harrah's Cherokee Casino and Hotel 427

9.3 S tru c tu re o f M a th e m a tic a l M o d e ls f o r D ecision S u p p o rt 429 The Components of Decision Support Mathematical Models 429 The Structure of Mathematical Models 431

9.4 C e r ta in ty , U n c e r ta in ty , a n d R isk 431 Decision Making Linder Certainty 432 Decision Making Under Uncertainty 432 Decision Making Under Risk (Risk Analysis) 432 ► APPLICATION CASE 9.3 American Airlines Uses

Should-Cost Modeling to Assess the Uncertainty of Bids for Shipment Routes 433

9.5 D e c is io n M o d e lin g w it h S p r e a d s h e e ts 434 ► APPLICATION CASE 9.4 Showcase Scheduling at Fred Astaire East

Side Dance Studio 434

9.6 M a t h e m a t ic a l P ro g ra m m in g O p tim iz a tio n 437 ► APPLICATION CASE 9.5 Spreadsheet Model Helps Assign Medical

Residents 437

Mathematical Programming 438 Linear Programming 438 Modeling in LP: An Example 439 Implementation 444

9.7 M u lt ip le G o a ls, S e n s itiv ity A n a ly s is , W h a t - lf A n alysis, a n d G o a l S e e k in g 446 Multiple Goals 446 Sensitivity Analysis 447 What-lf Analysis 448 Goal Seeking 448

9.8 D e cisio n A n a ly s is w it h D e c is io n T a b le s a n d D e cisio n T re e s 450 Decision Tables 450 Decision Trees 452

9.9 M u lti- C rite ria D e cis io n M a k in g W i t h P a irw is e C o m p a ris o n s 453 The Analytic Hierarchy Process 453

► APPLICATION CASE 9.6 U.S. HUD Saves the House by Using AHP for Selecting IT Projects 453

Tutorial on Applying Analytic Hierarchy Process Using Web-HIPRE 455 Ch apter H ig h lig hts 459 • K e y Terms 460 Q uestions fo r Discussion 460 • Exercises 460 ► END-OF-CHAPTER APPLICATION CASE Pre-Positioning of Emergency

Items for CARE International 463 References 464

Chapter 10 M odeling and Analysis: Heuristic Search M ethods and Sim ulation 465 10.1 O p e n in g V ig n e t t e : S ys te m D y n a m ic s A llo w s F lu o r

C o rp o ra tio n t o B e t t e r P la n f o r P ro je c t a n d C h a n g e M a n a g e m e n t 466

10.2 P ro b le m - S o lv in g S e a rc h M e th o d s 467 Analytical Techniques 468 Algorithms 468 Blind Searching 469

Heuristic Searching 469 ► APPLICATION CASE 10.1 Chilean Government Uses Heuristics to

Make Decisions on School Lunch Providers 469 10.3 G e n e tic A lg o r ith m s a n d D e v e lo p in g G A A p p lic a tio n s 471

Example: The Vector Game 471 Terminology of Genetic Algorithms 473 How Do Genetic Algorithms Work? 473 Limitations of Genetic Algorithms 475

Genetic Algorithm Applications 475 10.4 S im u la t io n 476

► APPLICATION CASE 10.2 Improving Maintenance Decision Making in the Finnish Air Force Through Simulation 476

► APPLICATION CASE 10.3 Simulating Effects of Hepatitis B Interventions 477

Major Characteristics of Simulation 478 Advantages of Simulation 479 Disadvantages of Simulation 480 The Methodology of Simulation 480 Simulation Types 481 Monte Carlo Simulation 482 Discrete Event Simulation 483

10.5 V is u a l In te ra c tiv e S im u la t io n 483 Conventional Simulation Inadequacies 483 Visual Interactive Simulation 483 Visual Interactive Models and DSS 484 ► APPLICATION CASE 10.4 Improving Job-Shop Scheduling Decisions

Through RFID: A Simulation-Based Assessment 484 Simulation Software 487

10.6 S ys te m D y n a m ic s M o d e lin g 488 10.7 A g e n t- B a s e d M o d e lin g 491

► APPLICATION CASE 10.5 Agent-Based Simulation Helps Analyze Spread of a Pandemic Outbreak 493

C h apter H ighlights 494 • K e y Terms 494 Q uestions fo r Discussion 495 • Exercises 495 ► END-OF-CHAPTER APPLICATION CASE HP Applies Management

Science Modeling to Optimize Its Supply Chain and Wins a Major Award 495

R eferences 497

C h a p te r 11 Autom ated Decision System s and Expert System s 499 11.1 O p e n in g V ig n e t t e : In te r C o n tin e n ta l H o te l G r o u p Uses

D e c is io n R u le s f o r O p tim a l H o te l R o o m R a te s 500 11.2 A u t o m a t e d D e c is io n S y s te m s 501

► APPLICATION CASE 11.1 Giant Food Stores Prices the Entire Store 502

11.3 T h e A r tific ia l In te llig e n c e F ie ld 505 11.4 Ba sic C o n c e p ts o f E x p e rt S y s te m s 507

Experts 507 Expertise 508 Features of ES 508 ► APPLICATION CASE 11.2 Expert System Helps in Identifying Sport

Talents 510 11.5 Applications of Expert Systems 510

► APPLICATION CASE 11.3 Expert System Aids in Identification of Chemical, Biological, and Radiological Agents 511

Classical Applications of ES 511 Newer Applications of ES 512 Areas for ES Applications 513

11.6 S tru c tu r e o f E x p e rt System s 514 Knowledge Acquisition Subsystem 514 Knowledge Base 515 Inference Engine 515 User Interface 515 Blackboard (Workplace) 515 Explanation Subsystem (Justifier) 516 Knowledge-Refining System 516 ► APPLICATION CASE 11.4 Diagnosing Heart Diseases by Signal

Processing 516

11.7 K n o w le d g e E n g in e e r in g 517 Knowledge Acquisition 518 Knowledge Verification and Validation 520 Knowledge Representation 520 Inferencing 521 Explanation and Justification 526

Contents 17

11.8 P ro b le m A r e a s S u ita b le f o r E x p e rt S ys te m s 527 11.9 D e v e lo p m e n t o f E x p e r t S ystem s 528

Defining the Nature and Scope of the Problem 529 Identifying Proper Experts 529 Acquiring Knowledge 529 Selecting the Building Tools 529 Coding the System 531 Evaluating the System 531 ► APPLICATION CASE 11.5 Clinical Decision Support System for Tendon Injuries 531

11.10 C o n c lu d in g R e m a rk s 532 Ch apter H ig h lig hts 533 • K e y Terms 533 Q uestions fo r Discussion 534 • Exercises 534 ► END-OF-CHAPTER APPLICATION CASE Tax Collections Optimization

for New York State 534 References 535

C h a p te r 12 Know ledge M anagem ent and Collaborative System s 537 12.1 O p e n in g V ig n e t t e : E x p e rtis e T ra n s fe r S ys te m t o T ra in

F u tu r e A r m y P e r s o n n e l 538 12.2 Introduction to Knowledge Management 542

Knowledge Management Concepts and Definitions 543

Knowledge 543 Explicit and Tacit Knowledge 545

12.3 A p p r o a c h e s t o K n o w le d g e M a n a g e m e n t 546 The Process Approach to Knowledge Management 547 The Practice Approach to Knowledge Management 547 Hybrid Approaches to Knowledge Management 548 Knowledge Repositories 548

12.4 In fo r m a tio n T e c h n o lo g y (IT ) in K n o w le d g e M a n a g e m e n t 550 The KMS Cycle 550 Components of KMS 551 Technologies That Support Knowledge Management 551

12.5 M a k in g D e cisio n s in G ro u p s : C h a ra c te ris tic s, Process, B e n e fits , a n d D y s fu n c tio n s 553 Characteristics of Groupwork 553 The Group Decision-Making Process 554 The Benefits and Limitations of Groupwork 554

12.6 S u p p o r tin g G r o u p w o r k w it h C o m p u te r iz e d S ys te m s 556 An Overview of Group Support Systems (GSS) 556 Groupware 557 Time/Place Framework 557

12.7 T o o ls f o r In d ire c t S u p p o r t o f D e c is io n M a k in g 558 Groupware Tools 558

Contents

Groupware 560 Collaborative Workflow 560 Web 2.0 560 Wikis 561 Collaborative Networks 561

12.8 D ire c t C o m p u te r iz e d S u p p o r t f o r D e c is io n M a k in g : Fro m G r o u p D e cis io n S u p p o r t S ys te m s t o G r o u p S u p p o rt S ys te m s 562 Group Decision Support Systems (GDSS) 562 Group Support Systems 563 How GDSS (or GSS) Improve Groupwork 563 Facilities for GDSS 564 Ch apter H ig h lig hts 565 • K ey Terms 566 Questions f o r Discussion 566 • Exercises 566 ► END-OF-CHAPTER APPLICATION CASE Solving Crimes by Sharing

Digital Forensic Knowledge 567 References 569

P a r t V B ig D a ta a n d F u t u r e D ir e c t io n s f o r B u s in e s s A n a l y t i c s 571

C h a p te r 13 Big Data and A nalytics 572 13.1 O p e n in g V ig n e t t e : B ig D a ta M e e ts B ig S c ie n c e a t C E R N 573 13.2 D e fin itio n o f B ig D a ta 576

The Vs That Define Big Data 577 ► APPLICATION CASE 13.1 Big Data Analytics Helps Luxottica Improve

Its Marketing Effectiveness 580 13.3 F u n d a m e n ta ls o f B ig D a ta A n a ly tic s 581

Business Problems Addressed by Big Data Analytics 584 ► APPLICATION CASE 13.2 Top 5 Investment Bank Achieves Single

Source of Truth 585 13.4 B ig D a ta T e c h n o lo g ie s 586

MapReduce 587 Why Use MapReduce? 588 Hadoop 588 How Does Hadoop Work? 588 Hadoop Technical Components 589 Hadoop: The Pros and Cons 590 NoSQL 592 ► APPLICATION CASE 13.3 eBay's Big Data Solution 593

13.5 D a ta S c ie n tis t 595 Where Do Data Scientists Come From? 595 ► APPLICATION CASE 13.4 Big Data and Analytics in Politics 598

13.6 B ig D a ta a n d D a ta W a r e h o u s in g 599 Use Case(s) for Hadoop 600 Use Case(s) for Data Warehousing 601

The Gray Areas (Any One of the Two Would Do the Job) 602 Coexistence of Hadoop and Data Warehouse 602

13.7 B ig D a ta V e n d o r s 604 ► APPLICATION CASE 13.5 Dublin City Council Is Leveraging Big Data

to Reduce Traffic Congestion 605 ► APPLICATION CASE 13.6 Creditreform Boosts Credit Rating Quality

with Big Data Visual Analytics 610 13.8 B ig D a ta a n d S tre a m A n a ly tic s 611

Stream Analytics Versus Perpetual Analytics 612 Critical Event Processing 612 Data Stream Mining 613

13.9 A p p lic a tio n s o f S tre a m A n a ly tic s 614 e-Commerce 614 Telecommunications 614 ► APPLICATION CASE 13.7 Turning Machine-Generated Streaming Data

into Valuable Business Insights 615 Law Enforcement and Cyber Security 616 Power Industry 617 Financial Services 617 Health Sciences 617 Government 617 Ch apter H ig h lig hts 618 • K e y Terms 618 Q uestions fo r Discussion 618 • Exercises 619 ► END-OF-CHAPTER APPLICATION CASE Discovery Health Turns Big

Data into Better Healthcare 619 R eferences 621

C h a p te r 14 Business Analytics: Em erging Trends and Future Impacts 622 14.1 O p e n in g V ig n e t t e : O k la h o m a G a s a n d E le c tr ic E m p lo y s

A n a ly tic s t o P r o m o te S m a r t E n e r g y U se 623 14.2 Lo c a tio n - B a s e d A n a ly tic s f o r O rg a n iz a tio n s 624

Geospatial Analytics 624 ► APPLICATION CASE 14.1 Great Clips Employs Spatial Analytics to

Shave Time in Location Decisions 626 A Multimedia Exercise in Analytics Employing Geospatial Analytics 627 Real-Time Location Intelligence 628 ► APPLICATION CASE 14.2 Quiznos Targets Customers for Its

Sandwiches 629 14.3 A n a ly tic s A p p lic a tio n s f o r C o n su m e rs 630

► APPLICATION CASE 14.3 A Life Coach in Your Pocket 631 14.4 R e c o m m e n d a tio n E n g in e s 633 14.5 W e b 2.0 a n d O n lin e S o c ia l N e t w o r k in g 634

Representative Characteristics of Web 2.0 635 Social Networking 635 A Definition and Basic Information 636 Implications of Business and Enterprise Social Networks 636

14.6 C lo u d C o m p u tin g a n d B l 637 Service-Oriented DSS 638 Data-as-a-Service (DaaS) 638 Information-as-a-Service (Information on Demand) (laaS) 641 Analytics-as-a-Service (AaaS) 641

14.7 Im p a c ts o f A n a ly tic s in O rg a n iz a tio n s : A n O v e r v ie w 643 New Organizational Units 643 Restructuring Business Processes and Virtual Teams 644 The Impacts of ADS Systems 644 Job Satisfaction 644 Job Stress and Anxiety 644 Analytics' Impact on Managers' Activities and Their Performance 645

14.8 Issues o f L e g a lity , P riva c y, a n d Eth ics 646 Legal Issues 646 Privacy 647 Recent Technology Issues in Privacy and Analytics 648 Ethics in Decision Making and Support 649

14.9 A n O v e r v ie w o f t h e A n a ly tic s Eco s ys te m 650 Analytics Industry Clusters 650 Data Infrastructure Providers 650 Data Warehouse Industry 651 Middleware Industry 652 Data Aggregators/Distributors 652 Analytics-Focused Software Developers 652 Reporting/Analytics 652 Predictive Analytics 653 Prescriptive Analytics 653 Application Developers or System Integrators: Industry Specific or General 654 Analytics User Organizations 655 Analytics Industry Analysts and Influences 657 Academic Providers and Certification Agencies 658 C h ap ter H ig h lig hts 659 • K e y Terms 659 Questions f o r Discussion 659 • Exercises 660 ► END-OF-CHAPTER APPLICATION CASE Southern States Cooperative

Optimizes Its Catalog Campaign 660 References 662

Glossary 664 In d e x 678

Overview of Business Intelligence, Analytics, and Decision Support

LEARNING OBJECTIVES

■ U n d erstand to d ay ’s tu rb u len t b u sin ess e n v iro n m en t an d d e s c rib e h o w org an izatio n s survive an d e v e n e x c e l in su ch a n e n v iro n m en t (so lv in g p ro b lem s a n d e x p lo itin g op p ortu n ities)

H U n d erstand th e n e e d fo r com p u terized su p p o rt o f m anagerial d e c is io n m aking

* U n d erstand a n early fram ew o rk for m an ag erial d e c is io n m aking

Th e b u sin e ss en v iro n m en t (clim a te ) is co n stan tly ch a n g in g , a n d it is b e c o m in g m o re and m o re co m p le x . O rgan izatio n s, private an d p u b lic, a re u n d e r p re ssu re s that fo rce th e m to re sp o n d q u ick ly to c h a n g in g co n d itio n s an d to b e in n ov ativ e in the w a y th ey o p era te. S u c h activities re q u ire o rg an izatio n s to b e a g ile an d to m a k e fre q u e n t a n d q u ick strateg ic, tactical, a n d o p era tio n a l d e cisio n s, s o m e o f w h ich are very co m p le x . M akin g s u ch d e cis io n s m ay re q u ire co n s id e ra b le am o u n ts o f relev an t data, in form ation, a n d k n o w le d g e. P ro ce s sin g th e s e , in th e fram ew o rk o f th e n e e d e d d e cisio n s, m u st b e d o n e q u ick ly , freq u en tly in real tim e, a n d u su ally req u ires s o m e co m p u te riz e d supp ort.

T h is b o o k is a b o u t u sin g b u s in e s s a n a ly tics as c o m p u te riz e d s u p p o rt fo r m a n a g e ­ rial d e c is io n m a k in g . It c o n c e n tr a te s o n b o t h th e th e o r e tic a l a n d c o n c e p tu a l fo u n d a ­ tio n s o f d e c is io n su p p o rt, as w e ll a s o n th e c o m m e rc ia l to o ls and te c h n iq u e s th a t are av a ila b le . T h is in tro d u cto ry c h a p te r p ro v id e s m o re d etails o f th e s e to p ic s a s w e ll a s a n o v e rv ie w o f th e b o o k . T h is c h a p te r h a s th e fo llo w in g s e c tio n s :

1 .1 O p e n in g V ig n e tte : M a g p ie S e n s in g E m p lo y s A n a ly tic s to M a n a g e a V a c c in e S u p p ly C h a in E ffe c tiv e ly a n d S a fe ly 3 3

1 .2 C h a n g in g B u s in e s s E n v ir o n m e n ts a n d C o m p u te riz e d D e c i s io n S u p p o rt 35

■ L earn th e c o n c e p tu a l fo u n d atio n s o f th e d e c is io n su p p o rt sy stem s (D S S 1) m e th o d o lo g y

■ D e s c r ib e th e b u sin e ss in te llig e n ce (B I) m e th o d o lo g y a n d c o n c e p ts and relate th e m to D SS

■ U nderstand th e various typ es o f analytics

■ List th e m a jo r to o ls o f co m p u terized d e c is io n su p p ort

‘The acronym DSS is treated as both singular and plural throughout this book. Similarly, other acronyms, such as MIS and GSS, designate both plural and singular forms. This is also true of the word analytics.

Chapter 1 • An O verview o f B u sin ess Intelligence, Analytics, a n d D ecision Support 3 3

1 .3 M a n a g e ria l D e c is io n M a k in g 3 7 1 .4 In fo r m a tio n S y s te m s S u p p o r t fo r D e c is io n M a k in g 3 9 1 .5 A n E a rly F ra m e w o rk fo r C o m p u te riz e d D e c is io n S u p p o rt 41 1 .6 T h e C o n c e p t o f D e c i s io n S u p p o rt S y s te m s (D S S ) 4 3 1 .7 A F ra m e w o rk fo r B u s in e s s I n te llig e n c e ( B I ) 4 4 1 .8 B u s in e s s A n a ly tic s O v e r v ie w 4 9 1 .9 B r i e f I n tr o d u c tio n to B ig D a ta A n a ly tics 5 7

1 .1 0 P la n o f th e B o o k 5 9 1 .1 1 R e s o u r c e s , L in k s, a n d th e T e r a d a ta U n iv ersity N e tw o rk C o n n e c tio n 61

1.1 OPENING VIGNETTE: Magpie Sensing Employs Analytics to Manage a Vaccine Supply Chain Effectively and Safely

C old ch a in in h e a lth ca re is d efin e d as th e te m p e ra tu re -co n tro lle d su p p ly ch a in involving a system o f tran sp ortin g and s to rin g v a c c in e s an d p h arm aceu tical drugs. It co n sists o f th ree m a jo r co m p o n e n ts — tran sp ort and sto rag e e q u ip m en t, trained p e rs o n n e l, an d e fficie n t m a n a g e m e n t p ro ced u res. T h e m ajority o f th e v a c c in e s in th e c o ld c h a in a re typ ically m a in ­ tain ed a t a tem p e ratu re o f 3 5 - 4 6 d eg ree s F ah re n h e it [ 2 - 8 d e g re e s C entigrade]. M aintaining cold ch a in integrity is e x trem ely im portant fo r h e a lth ca re p ro d u ct m anu factu rers.

E sp ecially fo r th e v a c c in e s , im p ro p er sto rag e a n d h an d lin g p ra ctice s th at co m p ro m ise v a ccin e viability p ro v e a co stly , tim e -co n su m in g affair. V a ccin e s m u st b e sto red p ro p e rly fro m m an u factu re until th e y a re a v ailab le fo r u se. Any e x tre m e te m p e ratu re s o f h e a t o r c o ld will re d u ce v a c c in e p o te n c y ; s u ch v a c c in e s , i f ad m inistered, m ight n o t y ield e ffe ctiv e results o r co u ld ca u se ad v e rse effects.

E ffectiv ely m aintainin g th e tem p eratu res o f sto rag e units th ro u g h o u t th e h e a lth ca re supp ly ch a in in re a l tim e— i.e ., b e g in n in g fro m th e gath erin g o f th e re so u rces, m a n u fa c­ turing, d istrib ution, a n d d isp e n sin g o f th e p rodu cts— is th e m o st e ffe ctiv e so lu tio n d esire d in th e c o ld ch ain . A lso, th e lo ca tio n -ta g g e d re al-tim e en v iro n m en tal data a b o u t the sto rag e units h e lp s in m o n ito rin g th e c o ld ch a in fo r s p o ile d p ro d u cts. T h e ch a in o f cu sto d y ca n b e easily id en tified to a ssig n p ro d u ct liability.

A stud y c o n d u cte d b y th e C enters fo r D ise a se C ontrol a n d P rev e n tio n (C D C ) lo o k e d a t the h an d lin g o f c o ld ch a in v a c c in e s b y 4 5 h e a lth ca re p ro vid ers a ro u n d U n ited States an d reported that th ree -q u arters o f th e p ro vid ers e x p e rie n c e d s erio u s c o ld c h a in v iolation s.

A WAY TOWARD A PO SSIBLE SOLUTION

M agpie S en sin g , a start-up p r o je c t u n d e r E bers Sm ith a n d D o u g las A sso ciate d LLC, p ro ­ vid es a suite o f c o ld ch a in m o n ito rin g an d an alysis te c h n o lo g ie s fo r th e h e a lth ca re in d u s­ try. It is a sh ip p a b le , w ire le s s tem p eratu re an d hum idity m o n ito r th a t p ro v id es real-tim e, lo ca tio n -a w a re tra ck in g o f c o ld ch a in p ro d u cts during sh ip m en t. M agpie S e n sin g ’s s o lu ­ tio n s re ly o n rich an alytics algorithm s th at le v e ra g e th e data g ath e re d fro m th e m o n ito r­ ing d e v ice s to im p ro ve th e e ffic ie n c y o f c o ld ch ain p ro c e s s e s a n d p re d ict c o ld sto rag e p ro b lem s b e fo r e th e y o ccu r.

M agp ie se n sin g a p p lie s all th ree ty p e s o f analytical te ch n iq u e s — d escrip tiv e, p re d ic­ tive, a n d p rescrip tiv e an aly tics— to tu rn th e raw data retu rn ed fro m th e m o n ito rin g d ev ice s into a c tio n a b le re co m m e n d a tio n s and w arnings.

T h e p ro p e rtie s o f th e c o ld s to ra g e sy stem , w h ic h in clu d e th e s e t p o in t o f th e sto rag e sy stem ’s th erm o stat, th e ty p ica l ran g e o f te m p e ratu re v a lu e s in th e s to ra g e system , a n d

3 4 Part I • D ecisio n M aking and Analytics: An Overview

the d uty c y c le o f th e sy stem ’s co m p re sso r, a re m o n ito re d an d re p o rte d in real tim e. This in form ation h e lp s train ed p e rs o n n e l to e n su re th at th e sto rag e unit is p ro p erly co n fig u re d to sto re a p articu lar p ro d u ct. All th e tem p eratu re in fo rm atio n is d isp layed o n a W e b d a sh ­ b o a rd th at s h o w s a g rap h o f th e tem p e ratu re in sid e th e s p e c ific s to ra g e unit.

B a s e d o n in form ation d eriv ed fro m th e m o n ito rin g d ev ice s, M ag p ie ’s p red ictiv e a n a ­ lytic algorithm s c a n d eterm in e th e s e t p o in t o f th e s to ra g e u n it’s th erm o stat a n d alert the sy stem s users i f th e s y stem is in co rre ctly co n fig u re d , d e p e n d in g u p o n th e vario u s typ es o f p ro d u cts stored . T h is o ffe rs a so lu tio n to th e u sers o f c o n s u m e r refrigerators w h e re th e th e rm o stat is n o t tem p e ratu re grad ed . M ag p ie ’s sy stem a lso s en d s alerts a b o u t p o s ­ sib le tem p eratu re v iolation s b a s e d o n th e s to ra g e u n it’s av e ra g e tem p eratu re and s u b s e ­ q u e n t c o m p re ss o r c y c le ru ns, w h ich m ay d rop th e te m p e ra tu re b e lo w th e fre e z in g point. M agpie s p red ictiv e analytics fu rth er re p o rt p o ss ib le h u m an errors, s u ch as failu re to shut th e sto ra g e unit d oors o r th e p re s e n c e o f an in c o m p le te se a l, b y analy zing th e te m p e ra ­ ture trend an d alertin g u sers via W e b in te rface , te x t m e s s a g e , o r au d ib le alert b e fo re th e tem p eratu re b o u n d s are actu ally violated . In a sim ilar w ay, a c o m p re ss o r o r a p o w e r failure c a n b e d etecte d ; th e e stim ated tim e b e fo re th e s to ra g e unit re a ch e s a n u n safe tem ­ p eratu re a lso is re p o rted , w h ich p re p a re s th e users to lo o k fo r b a c k u p so lu tio n s s u ch as u sin g dry ic e to resto re p o w er.

In ad dition to p red ictiv e analy tics, M agpie S e n s in g ’s an alytics sy stem s c a n p ro v id e p re scrip tiv e re co m m en d a tio n s fo r im p ro v in g th e c o ld sto rag e p ro c e s s e s an d b u sin ess d e cisio n m akin g. P rescriptive an aly tics h e lp u sers dial in th e op tim al tem p eratu re setting, w h ich h e lp s to a ch ie v e th e right b a la n c e b e tw e e n fre e z in g a n d s p o ila g e risk; this, in turn, p ro v id e s a cu sh io n -tim e to re a c t to th e situ ation b e fo re th e p ro d u cts sp o il. Its prescrip tiv e an alytics a lso g a th e r usefu l m eta-in fo rm atio n o n c o ld s to ra g e units, in clu d in g th e tim es o f d ay th a t a re b u s ie s t a n d p e rio d s w h e re th e sy stem ’s d o o rs a re o p e n e d , w h ic h c a n b e u se d to p ro v id e ad d itional d esig n p lan s an d institu tional p o lic ie s that e n su re that th e sy stem is b e in g p ro p e rly m ain tain ed an d n o t o v eru sed .

F u rth erm ore, p rescrip tive an alytics c a n b e u se d t o g u id e e q u ip m e n t p u rch a s e d e ci­ sio n s b y co n stan tly analyzing th e p e rfo rm a n ce o f cu rre n t s to ra g e units. B a sed o n th e s to ra g e sy stem ’s e fficie n cy , d ecisio n s o n d istributing th e p ro d u cts a cro ss a v ailab le storag e u nits c a n b e m ad e b a s e d o n th e p ro d u ct’s sensitivity.

U sing M ag p ie S e n sin g ’s c o ld ch a in analytics, ad d ition al m an u factu rin g tim e an d e x p e n d itu re c a n b e elim in ated b y e n su rin g that p ro d u ct safe ty c a n b e se cu re d th rou g h o u t th e su p p ly ch a in an d e ffe ctiv e p ro d u cts c a n b e ad m in istered to th e patients. C o m p lian ce w ith state and fed eral safe ty regu lation s c a n b e b etter a ch ie v e d th ro u g h au to m atic data g ath erin g an d re p o rtin g a b o u t th e p ro d u cts involved in th e co ld ch ain .

QUESTIONS FO R THE OPENING VIGNETTE

1 . W h at in form ation is pro v id ed b y th e d escrip tiv e an aly tics e m p lo y e d at M agpie Sensing?

2 . W h a t ty p e o f su p p o rt is pro v id ed b y th e p red ictiv e an aly tics e m p lo y e d at M agpie Sensing?

3 . H o w d o e s p rescrip tiv e analytics h e lp in b u sin ess d e c is io n m aking?

4 . In w h a t w ays c a n a c tio n a b le in form ation b e re p o r te d in re al tim e to c o n c e rn e d u sers o f th e system ?

5 . In w h at o th e r situ ation s m ight re al-tim e m o n ito rin g a p p lica tio n s b e need ed ?

WHAT WE CAN LEARN FROM THIS VIGNETTE

I his v ig n ette illustrates h o w data fro m a b u sin e ss p r o c e s s c a n b e u se d to g e n era te insights at various levels. First, th e grap h ical analysis o f th e data (te rm ed reportin g an aly tics) allow s

Chapter 1 • An O verview o f B usiness Intelligen ce, Analytics, and D ecision Support

users to g e t a g o o d fe e l fo r th e situation. T h e n , additional analysis u sin g data m ining .e ch n iq u es c a n b e u s e d to e stim ate w h at future b eh a v io r w o u ld b e like. T h is is th e d om ain o f pred ictive analytics. S u ch analysis c a n th e n b e ta k e n to cre ate sp e cific re co m m en d atio n s ror op erators. T h is is a n e x a m p le o f w h at w e call prescriptive analytics. Finally, this o p e n ­ ing vignette a lso suggests th at innovative ap p licatio n s o f analytics ca n c re a te n e w b u sin ess ventures. Id entifying op p ortu nities fo r ap p licatio n s o f analytics an d assisting w ith d ecisio n m aking in s p e cific d om ains is an e m e rg in g en trep ren eu rial opportunity.

Sources: Magpiesensing.com, "Magpie Sensing Cold Chain Analytics and Monitoring," magpiesensing.com/ wp-content:/upload.s/2013/01/ColdChainAnalyticsMagpieSensing-W hitepaper.pdf (accessed July 2013); Centers for Disease Control and Prevention, Vaccine Storage and Handling, http://www.cdc.gov/vaccines/pubs/ pinkbook/vac-storage.htmI*storage (accessed July 2013); A. Zaleski, “Magpie Analytics System Tracks Cold- -iafti Products to Keep Vaccines, Reagents Fresh ’ (2012). technicallybaltimore.com/profiles/startups/magpie- analytics-system-tracks-cold-chain-products-to-keep-vaccines-reagents-fresh (accessed February 2013).

1.2 CH AN G IN G BU SIN ESS ENVIRO NM ENTS A N D COM PUTERIZED D ECISIO N SUPPORT

T h e o p e n in g v ig n e tte illustrates h o w a co m p a n y can e m p lo y te c h n o lo g ie s to m a k e s e n se o f data a n d m a k e b etter d ecisio n s. C o m p an ie s are m o vin g aggressively to co m p u terized supp ort o f th e ir o p era tio n s. T o u n d erstan d w h y c o m p a n ie s are e m b ra cin g co m p u te r­ ized supp ort, in clu d in g b u s in e s s in te llig e n ce , w e d e v e lo p e d a m o d el c a lle d th e B u sin ess P ressu res-R espon ses-S u pport M odel, w h ich is sh o w n in Figure 1.1.

The Business Pressures-Responses-Support M odel T h e B u s in e s s P r e s s u r e s - R e s p o n s e s -S u p p o r t M o d el, as its n a m e in d ica te s , h a s th re e c o m ­ p o n e n ts: b u s in e s s p re s s u re s that resu lt fro m to d a y ’s b u s in e s s clim a te, re s p o n s e s (a c tio n s ta k e n ) b y c o m p a n ie s to c o u n te r th e p re ssu re s (o r to ta k e ad v a n ta g e o f th e o p p o rtu n itie s av ailab le in th e e n v iro n m e n t), an d c o m p u te riz e d su p p o rt th a t fa cilita te s th e m o n ito rin g o f th e e n v iro n m e n t an d e n h a n c e s th e r e s p o n s e a c tio n s ta k e n b y o rg a n iz a tio n s.

Decisions and Su p p o rt

FIGURE 1.1 The Business Pressures-Responses-Support Model.

3 6 P a r t i • D ecision M aking and Analytics: An Overview

5SS5 i ^ s S * = ^ K SS sssa&rss* •“*" T ^ iE is s E * * * * / . T h e s e ca te g o rie s a r i s ^ S b ^ " 0f i K * “ *

S w i S a ^ s s s Ltd (Krivda, 2 0 0 8 ), fo r e x a m p le tu rned to B l T ^ 6 preSSUreS' V o d a fo n e N ew Z ealand

E m p lo y strategic planning. • U se n e w and in n ov ativ e b u sin e ss m od els.

R estructu re b u sin ess p ro ce sse s. P articip ate in b u sin ess alliances.

• Im p ro v e co rp o ra te in form ation system s. • Im p ro v e p artn ersh ip relatio nsh ip s.

J A B L E 1.1 B usiness Environm ent Factors That Create Pressures on O rganizations

Factor __________ Description ~ "---- “ — —

Markets Strong competition Expanding global markets Booming electronic markets on the Internet Innovative marketing methods Opportunities for outsourcing with IT support Need for real-time, on-demand transactions

Consumer demands Desire for customization

Desire for quality, diversity of products, and speed of delivery Customers getting powerful and less loyal

Technology More innovationS/ new productSi and new servjcfis

Increasing obsolescence rate Increasing information overload Social networking, W eb 2.0 and beyond

Growing government regulations and deregulation Workforce more diversified, older, and composed of more women Prime concerns of homeland security and terrorist attacks Necessity of Sarbanes-Oxley Act and other reporting-related legislation Increasing social responsibility of companies Greater emphasis on sustainability

Societal

• E n co u ra g e in n o v atio n a n d creativity. • Im p rove cu sto m e r serv ice an d re latio n sh ip s. • E m p loy s o c ia l m e d ia a n d m o b ile platform s for e -c o m m e r c e and b ey o n d . • M o ve to m a k e -to -o rd e r p ro d u ctio n an d o n -d e m a n d m an u factu rin g a n d serv ices. • U se n e w IT to im p ro v e co m m u n ica tio n , data a c c e s s (d isco v e ry o f in fo rm atio n ), an d

c o lla b o ra tio n . • R e sp o n d q u ick ly to c o m p etito rs’ a ctio n s (e .g ., in pricing, p ro m o tio n s, n e w p ro d u cts

an d se rv ice s). • A u tom ate m an y task s o f w h ite -co lla r e m p lo y e e s. • A u tom ate ce rta in d e c is io n p ro ce s s e s, e sp ecia lly th o se d ealin g w ith cu sto m ers. • Im p ro v e d e c is io n m ak in g b y e m p lo y in g analytics.

Many, if n o t all, o f th e s e actio n s req u ire s o m e co m p u te rize d su p p ort. T h e s e a n d o th e r re sp o n se a c tio n s are fre q u e n tly facilitated b y co m p u te riz e d d e c is io n su p p o rt (D S S ).

C L O S IN G T H E S T R A T E G Y G A P O n e o f th e m a jo r o b je c tiv e s o f co m p u te riz e d d e c is io n su p p o rt is to fa cilita te c lo s in g th e gap b e tw e e n th e cu rre n t p e rfo rm a n c e o f an o rg an i­ z a tio n a n d its d e sire d p e rfo rm a n ce , as e x p r e s s e d in its m issio n , o b je c tiv e s , a n d goals, an d th e strate g y to a c h ie v e th e m . In o rd e r to u n d e rstan d w h y c o m p u te riz e d su p p o rt is n e e d e d a n d h o w it is p ro v id e d , e sp e c ia lly fo r d e cis io n -m a k in g su p p o rt, le t’s lo o k at m an ag erial d e c is io n m akin g.

SECTION 1 . 2 REVIEW QUESTIONS

1 . List th e c o m p o n e n ts o f an d e x p la in th e B u s in e s s P r e s s u r e s -R e s p o n s e s -S u p p o r t

M odel. 2 . W hat are s o m e o f th e m a jo r facto rs in to d ay ’s b u sin e ss environm ent?

3 . W h at a re s o m e o f th e m a jo r re s p o n s e activities that o rg an ization s take?

1.3 M A N A G E R IA L D ECISIO N M AKING M a n a g e m e n t is a p r o c e s s b y w h ic h o r g a n iz a tio n a l g o a ls a re a c h ie v e d b y u sin g re s o u r c e s . T h e r e s o u r c e s a re c o n s id e r e d in p u ts, a n d atta in m e n t o f g o a ls is v ie w e d as th e o u tp u t o f th e p r o c e s s . T h e d e g r e e o f s u c c e s s o f th e o r g a n iz a tio n a n d th e m a n a g e r is o fte n m e a s u re d b y th e ratio o f o u tp u ts to in p u ts. T h is ra tio is a n in d ic a tio n o f th e o r g a n iz a tio n ’s p ro d u ctiv ity , w h ic h is a r e fle c tio n o f t h e o r g a n iz a tio n a l a n d m a n a g e r ia l p er fo r m a n c e .

T h e le v e l o f pro d u ctiv ity o r th e s u c c e s s o f m a n a g e m e n t d e p e n d s o n th e p e rfo r­ m a n c e o f m a n a g e ria l fu n ctio n s, s u c h a s p la n n in g , o rg a n iz in g , d ire ctin g , a n d c o n tro l­ ling. T o p e rfo rm th e ir fu n c tio n s , m a n a g e rs e n g a g e in a c o n tin u o u s p r o c e s s o f m ak in g d e c is io n s . M a k in g a d e c is io n m e a n s s e le c tin g th e b e s t a ltern a tiv e fro m tw o o r m o re

so lu tio n s.

The Nature o f M anagers' W ork M intzberg’s (2 0 0 8 ) c la ss ic study o f to p m an ag e rs a n d sev eral re p lica te d stu d ies suggest that m a n a g e rs p e rfo rm 10 m a jo r ro le s that c a n b e classified into th ree m a jo r ca te g o iie s . in terperson al, in fo rm a tio n a l, an d d e c is io n a l (.s e e l a b l e 1.2).

T o p e rfo rm th e s e ro le s, m an ag e rs n e e d in fo rm atio n th at is d eliv e red efficie n tly an d in a tim ely m a n n e r to p e rso n a l co m p u te rs (P C s) o n th eir d esk to p s and to m o b ile d ev ices. T h is in form ation is d eliv e red b y n etw o rk s, g e n erally v ia W e b te ch n o lo g ie s.

In ad d itio n to o b ta in in g inform ation n e c e s s a ry to b e tte r p erfo rm th e ir ro le s, m an ag ­ ers u se c o m p u te rs d irectly to su p p o rt and im p ro ve d e c is io n m akin g, w h ic h is a k e y task

C hapter 1 • An O verview o f B u sin ess Intelligence, Analytics, and D ecision Support 3 7

D ecisio n M aking and Analytics: An Overview

T A B L E 1.2 M intzberg's 10 M anagerial Roles

Role Description Interpersonal Figurehead Is symbolic head; obliged to perform a number of routine duties of a

legal or social nature Leader Is responsible for the motivation and activation of subordinates;

responsible for staffing, training, and associated duties Liaison Maintains self-developed network of outside contacts and informers

who provide favors and information Informational Monitor Seeks and receives a wide variety of special information (much of it

current) to develop a thorough understanding of the organization and environment; emerges as the nerve center of the organization's internal and external information

Disseminator Transmits information received from outsiders or from subordinates to members of the organization; some of this information is factual, and some involves interpretation and integration

Spokesperson Transmits information to outsiders about the organization's plans, policies, actions, results, and so forth; serves as an expert on the organization's industry

Decisional Entrepreneur Searches the organization and its environment for opportunities and

initiates improvement projects to bring about change; supervises design of certain projects

Disturbance handler Is responsible for corrective action when the organization faces important, unexpected disturbances

Resource allocator Is responsible for the allocation of organizational resources of all kinds; in effect, is responsible for the making or approval of all significant organizational decisions

Negotiator Is responsible for representing the organization at major negotiations

Sources: Compiled from H. A. Mintzberg, The Nature o f M an agerial Work. Prentice Hall, Englewood Cliffs, NJ, 1980; and H. A. Mintzberg, The Rise a n d Fall o f Strategic P lanning. The Free Press, New York, 1993.

that is p art o f m o st o f th e s e ro le s. M any m an ag erial activities in all ro les rev o lv e around d e cisio n m akin g. M an agers, esp ecia lly th ose a t h ig h m a n a g e r ia l levels, a r e p rim a rily d e c i­ sio n m akers. W e re v iew th e d ecisio n -m a k in g p ro c e s s n e x t b u t w ill stud y it in m o re detail in th e n e x t ch ap ter.

The Decision-Making Process F o r years, m an ag ers co n sid e re d d e cisio n m ak in g p u re ly a n art— a talen t a cq u ire d o v e r a lo n g p e rio d th ro u g h e x p e rie n c e (i.e ., learn in g b y trial-an d -error) an d b y u sin g intuition. M an ag e m e n t w a s c o n s id e re d a n art b e c a u s e a v ariety o f individual styles c o u ld b e u sed in a p p ro a ch in g a n d s u cce ssfu lly so lv in g th e sa m e ty p e s o f m an ag erial p ro b lem s. T h e s e styles w e r e o fte n b a s e d o n creativity, ju d g m en t, intu ition, an d e x p e r ie n c e rath er than o n sy stem atic q u an titative m e th o d s g ro u n d e d in a s c ie n tific ap p ro a ch . H o w ev er, re ce n t re se a rch su g g e sts that c o m p a n ie s w ith to p m an ag e rs w h o a re m o re fo c u s e d o n p ersisten t w o rk (a lm o st d u lln ess) te n d to o u tp e rfo rm th o se w ith lead e rs w h o s e m a in strengths are in terp erso n al co m m u n icatio n skills (K a p la n e t al., 2 0 0 8 ; B ro o k s, 2 0 0 9 ). It is m o re im por­ tant to e m p h a s iz e m e th o d ical, thoughtful, an aly tical d e c is io n m ak in g rath er th an flashi­ n e s s a n d in terp erso n al co m m u n ica tio n skills.

Chapter 1 • An O verview o f B u sin ess Intelligence, Analytics, and D ecisio n Support 3 9

M an agers u su ally m a k e d e cis io n s b y fo llo w in g a fo u r-step p r o c e s s (w e le a rn m o re a b o u t th e s e in C h ap ter 2):

1 . D e fin e th e p ro b le m (i.e ., a d e c is io n situ atio n that m ay d ea l w ith s o m e d ifficulty o r w ith a n o p p ortu n ity ).

2. C o n stru ct a m o d el th at d e s c rib e s th e real-w orld p ro b lem . 3- Id en tify p o ss ib le so lu tio n s to th e m o d e le d p ro b lem a n d e v alu ate th e solu tions. 4 . C o m p are , c h o o s e , and re co m m en d a p o ten tial solu tion to th e p ro b lem .

T o fo llo w this p ro c e s s , o n e m u st m a k e sure th at su fficien t altern ativ e so lu tio n s are b e in g co n s id e re d , that th e c o n s e q u e n c e s o f u sin g th e s e alternativ es c a n b e re a so n a b ly p red icted , a n d that co m p a riso n s are d o n e p ro p erly. H ow ever, th e en v iro n m en tal facto rs listed in T a b le 1.1 m a k e s u ch a n evalu ation p ro c e s s d ifficult fo r th e fo llo w in g reason s:

• T e c h n o lo g y , inform ation sy stem s, ad v an ce d s e a rch e n g in e s , an d g lo b a liz a tio n result in m o re a n d m o re alternatives fro m w h ich to c h o o s e .

• G o v e rn m e n t reg u latio n s a n d th e n e e d fo r co m p lia n ce , p o litical instability an d te r­ rorism , co m p etitio n , and ch a n g in g c o n s u m e r d em an d s p ro d u ce m o re u n certainty, m a k in g it m o re d ifficu lt to p re d ict c o n s e q u e n c e s and th e future.

• O th e r fa c to rs are th e n e e d to m a k e rap id d ecisio n s, th e fre q u e n t an d u n p re d ictab le c h a n g e s that m ak e trial-an d -error learn in g difficult, and th e p o ten tial co s ts o f m aking m istak es.

• T h e s e e n v iro n m en ts a re g ro w in g m o re c o m p le x e v e ry day. T h e re fo re , m ak in g d e ci­ sio n s to d a y is in d e e d a c o m p le x task.

B e c a u s e o f th e s e tren d s an d ch a n g e s, it is n early im p o ssib le to rely o n a trial-and- e rro r a p p r o a c h to m an ag e m e n t, e sp e cia lly fo r d e cis io n s fo r w h ich th e fa c to rs sh o w n in T a b le 1.1 a re stro n g in flu e n ces. M anagers m u st b e m o re so p h isticated ; th e y m ust u s e th e n e w to o ls a n d te c h n iq u e s o f th eir field s. M ost o f th o se to o ls an d te c h n iq u e s are d iscu ssed in this b o o k . U sing th e m to su p p ort d e c is io n m ak in g c a n b e e x tre m e ly rew ard in g in m ak in g e ffe ctiv e d e cisio n s. In th e fo llo w in g s e ctio n , w e lo o k a t w h y w e n e e d co m p u te r su p p ort a n d h o w it is provided.

SECTION 1 . 3 REVIEW QUESTIONS

1 . D escrib e th e th ree m ajor m anagerial roles, and list som e o f the sp ecific activities in each.

2. W h y h a v e s o m e argu ed that m a n a g e m e n t is th e sam e as d e cisio n m aking? 3. D e s c r ib e th e fo u r ste p s m an ag ers ta k e in m ak in g a d ecisio n .

1.4 INFORM ATION S Y S T E M S SUPPORT FOR D ECISIO N M AKIN G From trad itional u s e s in p ayro ll a n d b o o k k e e p in g fu n ctio n s, co m p u te riz e d system s hav e p en e trate d c o m p le x m an agerial areas ran gin g fro m th e d esig n an d m an a g e m e n t o f au to ­ m ated fa cto rie s to th e a p p lica tio n o f analytical m eth o d s fo r th e e v a lu a tio n o f p ro p o s e d m ergers a n d acq u isitio n s. N early all e x e c u tiv e s k n o w that in form ation te c h n o lo g y is vital to th eir b u s in e s s an d e x te n siv e ly u se in form ation te ch n o lo g ie s.

C o m p u te r a p p lica tio n s h av e m o v e d fro m tran saction p ro ce s s in g a n d m o n ito rin g activities to p ro b le m analysis an d so lu tio n a p p licatio n s, an d m u ch o f th e activity is d o n e w ith W e b -b a s e d te ch n o lo g ie s, in m a n y c a s e s a c c e ss e d th ro u g h m o b ile d ev ice s. Analytics an d B I to o ls s u ch as data w areh o u sin g , d ata m ining, o n lin e analy tical p ro c e s s in g (O LAP), dashboards, an d th e u se o f th e W e b for d e cisio n su p p o rt a re th e co r n e r sto n e s o f to d ay ’s m o d e rn m an ag e m e n t. M anagers m ust h av e h ig h -sp e ed , n e tw o rk e d in form ation sys­ tem s (w ire lin e o r w ire le s s ) to assist th e m w ith th e ir m o st im p ortan t task : m ak in g d e ci­ sions. B e s id e s th e o b v io u s gro w th in h ard w are, so ftw are, a n d n e tw o rk c a p a c itie s , so m e

4 0 Part I • D ecisio n M aking and Analytics: An Overview

d ev elo p m en ts h a v e clearly co n trib u ted to facilitatin g gro w th o f d e c is io n su p p o rt and analytics in a n u m b e r o f w ay s, in clu d in g th e fo llow in g :

• G ro u p co m m u n ica tio n a n d collaboration. M any d ecisio ns are m ade to d ay b y groups w h o se m em b ers m ay b e in different locations. G roups ca n collab orate and com m u nicate readily b y using W e b -b a sed tools a s w e ll as th e ubiqu itous sm artphones. C ollaboration is esp ecially im portant alo n g th e supp ly chain, w h ere partners— all the w ay from vend ors to custom ers— m ust sh are inform ation. A ssem bling a group o f d ecisio n m akers, esp ecially experts, in o n e p la ce c a n b e costly. Infom iation system s c a n im prove th e collab oratio n p ro cess o f a gro u p an d e n a b le its m em b ers to b e at dif­ ferent locations (saving travel costs). W e will study so m e applications in Chapter 12.

• Im p ro v e d d a ta m a n a g e m e n t. M any d e cis io n s in v o lv e c o m p le x co m p u tatio n s. D ata fo r th e s e c a n b e sto red in d ifferen t d a ta b a s e s an y w h ere in th e o rg an ization a n d e v e n p o ss ib ly a t W e b sites ou tsid e th e org an izatio n . T h e data m ay in clu d e text, sou n d , grap h ics, a n d v id eo , an d th e y c a n b e in d ifferent lan g u ag es. It m ay b e n e c e s ­ sary to transm it data q u ick ly fro m d istant lo ca tio n s. System s to d ay c a n se a rch , store, an d tran sm it n e e d e d d ata q u ick ly , e co n o m ica lly , s e cu re ly , an d transp arently.

• M a n a g in g g i a n t d a ta w a reh o u ses a n d B ig D a ta . Large d ata w a reh o u se s, like th e o n e s o p e ra te d b y W alm art, co n ta in te ra b y tes an d e v e n p e ta b y tes o f d ata. Sp ecial m e th o d s, in clu d in g p arallel com p u tin g , are a v ailab le to o rg an ize , se a rch , a n d m ine th e d ata. T h e co sts related to d ata w a reh o u sin g are d eclin in g . T e c h n o lo g ie s th at fall u n d e r th e b ro ad ca te g o ry o f B ig D ata h av e e n a b le d m assiv e data c o m in g fro m a variety o f so u rce s and in m an y d ifferen t fo rm s, w h ich allo w s a v e ry d ifferen t v iew in to organ izatio n al p e rfo rm a n ce th at w as n o t p o ss ib le in th e past.

• A n a ly tic a l su p p o rt. W ith m o re data a n d a n aly sis te c h n o lo g ie s , m o re altern a­ tiv e s c a n b e e v alu ated , fo reca sts c a n b e im p ro v e d , risk an alysis c a n b e p e rfo rm e d q u ick ly , a n d th e v iew s o f e x p e rts (s o m e o f w h o m m ay b e in re m o te lo c a tio n s ) ca n b e c o lle c te d q u ick ly an d a t a re d u c e d co st. E x p e rtis e c a n e v e n b e d eriv ed d irectly fro m a n aly tical sy stem s. W ith s u ch to o ls , d e c is io n m a k e rs ca n p e rfo rm c o m p le x sim u latio n s, c h e c k m a n y p o ss ib le sce n a rio s, a n d a s s e s s d iv erse im p acts q u ic k ly and e c o n o m ic a lly . T h is, o f co u rse , is th e fo c u s o f s e v e r a l ch a p te rs in th e b o o k .

• O vercom ing cognitive lim its in p ro c e s s in g a n d sto rin g in fo rm a tio n . A ccording to Sim on (1 9 7 7 ), th e hu m an mind has only a lim ited ability to p ro cess and store infor­ mation. P e o p le som etim es find it difficult to recall an d use inform ation in a n error-free fash ion due to their cognitive limits. T h e term cogn itive lim its indicates that a n indi­ vidual’s problem -solving capability is limited w h e n a w id e range o f diverse inform ation and k n ow led g e is required. Com puterized system s e n a b le p e o p le to o v erco m e their cognitive limits b y q uickly a ccessin g a n d p ro cessin g vast am ounts o f stored inform ation (s e e C hapter 2).

• K n o w led g e m a n a g e m e n t . O rg an izatio n s h a v e g ath e re d vast s to res o f in form a­ tio n a b o u t th e ir o w n o p e ra tio n s, cu sto m e rs, in te rn a l p ro c e d u re s , e m p lo y e e in te ra c­ tion s, a n d s o fo rth th ro u g h th e u n stru ctu red a n d stru ctu red c o m m u n ica tio n s tak in g p la c e am o n g th e v ario u s s ta k e h o ld e rs. K n o w le d g e m a n a g e m e n t sy stem s (KM S, C h ap te r 12) h av e b e c o m e s o u rc e s o f fo rm al an d in form al su p p o rt fo r d e c is io n m ak in g to m a n a g e rs, a lth o u g h s o m e tim e s th e y m ay n o t e v e n b e ca lle d KMS.

• A n y w h ere, a n y tim e su p p o rt. U sing w ire le s s te c h n o lo g y , m an ag e rs c a n a c c e s s in form ation an y tim e an d fro m an y p la ce , an aly ze and in terp ret it, an d co m m u n icate w ith th o s e involved . T h is p e rh a p s is th e b ig g e st c h a n g e that h a s o ccu rre d in th e last fe w years. T h e s p e e d at w h ic h in form ation n e e d s to b e p ro c e s s e d a n d co n v e rte d in to d ecisio n s h a s truly c h a n g e d e x p e c ta tio n s fo r b o th co n su m e rs a n d b u sin esses.

T h e s e an d o th e r capabilities h av e b e e n driving th e u s e o f com p u terized d ecisio n supp ort s in ce th e late 1960s, b u t esp ecially sin ce th e m id -1990s. T h e grow th o f m o b ile tech n o lo g ies,

Chapter 1 • An O verview o f B usiness Intelligence, Analytics, and D ecisio n Support 41

social m ed ia platform s, and analytical to o ls has e n a b led a m u ch h ig h er lev el o f inform ation system s su p p ort fo r m anagers. In th e n e x t sectio n s w e study a historical classification o f d e cisio n su p p ort tasks. T h is lead s us to b e introd u ced to d ecisio n su p p ort system s. W e will th e n study a n o v erview o f te ch n o lo g ie s that hav e b e e n b road ly referred to as b u sin ess intel­ lig e n ce . F ro m th ere w e w ill b ro ad en o u r h orizons to introd u ce v arious types o f analytics.

SECTION 1 .4 REVIEW QUESTIONS

1 . W h a t are s o m e o f th e k e y sy stem -o rie n te d trend s that h av e fo s te re d IS-su p p orted d e c is io n m ak in g to a n e w level?

2 . List s o m e cap ab ilitie s o f in form ation sy stem s that c a n facilitate m an ag erial d e cisio n m akin g.

3 . H o w c a n a co m p u te r h e lp o v e rco m e the co g n itiv e lim its o f hum ans?

1.5 A N E A R L Y F R A M E W O R K FO R C O M PU T ER IZ ED D EC ISIO N SU PPO R T

An e a rly fram ew o rk fo r co m p u teriz ed d e c is io n su p p o rt in clu d es sev eral m a jo r co n c e p ts th at are u s e d in fo rth co m in g se ctio n s an d ch ap ters o f this b o o k . G o rry a n d Scott-M orton cre a te d a n d u s e d this fram ew o rk in th e early 19 7 0 s, an d th e fram ew o rk th e n e v o lv ed into a n e w te c h n o lo g y ca lle d DSS.

The G orry and Scott-M orton Classical Fram ework G orry an d Scott-M o rton (1 9 7 1 ) p ro p o s e d a fram ew o rk th at is a 3 -b y -3 m atrix, as s h o w n in Figure 1.2 . T h e tw o d im en sio n s are th e d eg re e o f stru ctu red n ess a n d th e ty p e s o f con trol.

Type of Control

Type of Decision Operational

Control M anagerial

Control S tra te g ic Planning

S tru c tu re d Accounts receivable Accounts payable Order entry

Budget analysis Short-term forecasting Personnel reports Make-or-buy

Financial management Investment portfolio W arehouse location Distribution systems

Production scheduling Inventory control

Se m istru ctu re d

Credit evaluation Budget preparation Plant layout Project scheduling Reward system design Inventory

categorization

Building a new plant M ergers & acquisitions New product planning Compensation planning Quality assurance HR policies Inventory planning

8

U n stru ctu re d Buying software Approving loans Operating a help desk Selecting a cover for

a magazine

Negotiating Recruiting an executive Buying hardware Lobbying

R S . D planning New tech development Social responsibility

planning

FIGURE 1.2 Decision Support Frameworks.

4 2 Part I • D ec isio n M aking and Analytics: An O verview

D E G R EE OF STRU C TU RED N ESS T h e left sid e o f F ig u re 1 .2 is b a se d o n S im o n ’s (1 9 7 7 ) idea that d ecisio n -m a k in g p ro c e s s e s fall alo n g a co n tin u u m that ra n g e s fro m hig hly structured (s o m e tim e s called p rog ram m ed ) to h ig h ly u n stru ctu red (i.e ., n on p rog ram m ed ) d ecisio n s. Stru ctu red p ro c e s s e s a re ro u tin e and typ ically re p e titiv e p ro b lem s fo r w h ich standard so lu tio n m e th o d s exist. U n stm ctu red p ro cesses are fuzzy, c o m p le x p ro b lem s fo r w h ich th e re a re n o cu t-and -d ried so lu tio n m ethod s.

An u n stru ctu red problem is o n e w h e re th e articu latio n o f th e p ro b le m o r th e so lu ­ tio n a p p ro a ch m ay b e un stru ctu red in itself. In a stru ctu red problem , th e p ro ced u res fo r o b ta in in g th e b e s t (o r a t le a s t a g o o d e n o u g h ) so lu tio n a re k n o w n . W h e th e r th e p ro b ­ lem involves find ing an ap p rop riate inventory le v e l o r ch o o s in g a n op tim al in v estm en t strategy, th e o b je c tiv e s a re clearly d efin ed . C o m m o n o b je c tiv e s are c o s t m inim ization and p ro fit m axim ization.

Sem istructured problem s fall b e tw e e n stru ctu red a n d u n stru ctu red p ro b le m s, h av ­ ing s o m e stru ctu red e le m e n ts a n d so m e u n stru ctu red e le m e n ts. K e e n a n d Sco tt-M o ito n ( 1 9 7 8 ) m e n tio n e d trad ing b o n d s, settin g m arketin g b u d g ets fo r c o n s u m e r p ro d u cts, and p erfo rm in g cap ital a cq u isitio n analysis as sem istru ctu red p ro b lem s.

T Y P E S OF CONTROL T h e s e c o n d h a lf o f th e G o r ry an d S c o tt-M o rto n fra m ew o rk (r e fe r to F ig u re 1 .2 ) is b a s e d o n A n th o n y ’s ( 1 9 6 5 ) ta x o n o m y , w h ic h d e fin e s th re e b r o a d c a te g o r ie s th at e n c o m p a s s all m a n a g e ria l a c tiv itie s : stra teg ic p la n n in g , w h ic h in v o lv e s d e fin in g lo n g -r a n g e g o a ls a n d p o lic ie s fo r r e s o u r c e a llo c a tio n ; m a n a g e­ m en t co n tro l, th e a c q u is itio n a n d e ffic ie n t u s e o f r e s o u r c e s in th e a c c o m p lis h m e n t o f o r g a n iz a tio n a l g o a ls ; a n d o p e r a tio n a l co n tro l, th e e ffic ie n t an d e ffe c tiv e e x e c u t io n o f s p e c ific task s.

THE D EC ISIO N SU PPO RT M A T R IX A n th on y ’s an d S im o n ’s tax o n o m ie s are c o m b in e d in the n in e -c e ll d e c is io n su p p ort m atrix s h o w n in Figure 1.2 . T h e initial p u rp o s e o f this m atrix w a s to su g g e st d ifferen t typ es o f co m p u te rize d su p p o rt to d ifferent c e lls in th e matrix. G orry an d Scott-M o rton su gg ested , fo r e x a m p le , th a t fo r sem istru ctu red d ecisio n s and u n stru ctu red d ecisio n s, co n v e n tio n a l m a n a g e m e n t in form ation sy stem s (M IS) a n d m an­ a g e m e n t s c ie n c e (M S) to o ls are in su fficien t. H u m an in te lle ct a n d a d ifferen t a p p ro a ch to co m p u te r te c h n o lo g ie s a re n e cessa ry . T h e y p ro p o s e d th e u se o f a su p p ortiv e inform ation system , w h ic h th ey ca lle d a DSS.

N o te th a t th e m o re stru ctu re d a n d o p e r a tio n a l c o n tr o l-o r ie n te d ta s k s ( s u c h as th o s e in c e lls 1, 2, a n d 4 ) a re u su a lly p e rfo rm e d b y lo w e r-le v e l m a n a g e rs , w h e re a s th e ta s k s in c e lls 6 , 8 , an d 9 a re th e re s p o n s ib ility o f to p e x e c u tiv e s o r h ig h ly train e d s p e c ia lis ts .

Com puter Support fo r Structured Decisions C o m p u ters h av e historically su p p o rte d stru ctured an d s o m e sem istru ctu red d ecisio n s, e sp e cia lly th o s e th at involve o p era tio n a l an d m an ag erial co n tro l, s in c e th e 1960s. O p e ra tio n a l a n d m anag erial co n tro l d e cis io n s are m a d e in all fu n ctio n al areas, e sp ecia lly in fin a n ce an d p ro d u ctio n (i.e ., o p e ra tio n s) m an ag em en t.

S ta ic tu r e d p ro b le m s, w h ich a re e n c o u n te re d re p e a te d ly , h av e a h ig h le v e l o f stru c­ tu re. It is th e re fo r e p o s s ib le to ab stract, an aly ze, a n d classify th e m into s p e c ific c a te g o ­ ries. F o r e x a m p le , a m a k e -o r-b u y d e c is io n is o n e ca te g o ry . O th e r e x a m p le s o f c a te g o rie s a re cap ital b u d g etin g , a llo ca tio n o f re s o u r c e s , d istrib u tio n , p ro cu re m e n t, p lan n in g , and in v e n to ry co n tro l d e cisio n s. F o r e a c h c a te g o ry o f d e c is io n , a n e asy -to -a p p ly p re scrib e d m o d e l an d so lu tio n a p p r o a c h h a v e b e e n d e v e lo p e d , g e n e ra lly as q u antitative form ulas. T h e re fo re , it is p o ss ib le to u se a s c ie n tific a p p ro a c h fo r au to m atin g p o rtio n s o f m a n a g e ­ rial d e c is io n m akin g.

Chapter 1 • A n O verview o f B u sin ess Intelligen ce, Analytics, and D ecisio n Support 4 3

Com puter Support fo r Unstructured Decisions U nstructured p ro b le m s c a n b e o n ly partially su p p o rte d b y standard co m p u te riz e d q u a n ­ titative m e th o d s. It is u su ally n e ce s s a ry to d e v e lo p cu sto m ized solu tion s. H o w e v e r, su ch so lu tion s m ay b e n e fit fro m data a n d in form ation g e n era ted fro m c o rp o ra te o r e xte rn al data so u rce s. In tu itio n an d ju d g m en t m ay p lay a larg e ro le in th e s e ty p es o f d e cisio n s, as m ay co m p u te riz e d co m m u n icatio n a n d co lla b o ra tio n te ch n o lo g ie s, as w e ll a s k n o w le d g e m an ag e m e n t ( s e e C h ap ter 12).

Com puter Support fo r Sem istructured Problem s Solving sem istru ctu red p ro b lem s m ay in v o lv e a co m b in a tio n o f stand ard so lu tio n p ro­ ce d u re s and h u m a n ju d gm en t. M an ag em en t s c ie n c e c a n p ro v id e m o d els fo r th e p o rtio n o f a d e cisio n -m a k in g p ro b le m that is structured. F o r th e unstructured p o rtio n , a D SS c a n im prove th e q u ality o f th e in form ation o n w h ich th e d e cisio n is b a s e d b y p roviding, fo r e x a m p le , n o t o n ly a sin gle so lu tio n b u t a lso a ran g e o f alternative so lu tio n s, a lo n g w ith th eir p o ten tia l im pacts. T h e s e cap ab ilitie s h e lp m an ag e rs to b e tte r u n d e rstan d th e natu re o f p ro b lem s a n d , thu s, to m a k e b e tte r d ecisio n s.

SECTION 1 .5 REVIEW QUESTIONS

1 . W h at are stru ctu red , un structured , and sem istru ctu red d ecisio n s? P ro v id e tw o e x a m ­ p le s o f e a c h .

2. D e fin e o p era tio n a l con trol, m a n a g eria l con trol, an d strateg ic p la n n in g . P rov id e tw o e x a m p le s o f e a ch .

3. W h at are th e n in e c e lls o f th e d e cisio n fram ew ork? E x p la in w h at e a c h is for. 4 . H o w c a n co m p u te rs pro v id e su p p o rt fo r m ak in g stru ctu red decisions? 5. H o w c a n co m p u te rs pro v id e su p p o rt to sem istru ctu red a n d un stru ctu red decisions?

1.6 THE CON CEPT OF DECISIO N SUPPORT SYST EM S (D SS) In th e e a rly 1 9 7 0 s , Scott-M o rton first articu lated th e m a jo r c o n c e p ts o f D SS. H e d efin ed decision su p p o rt system s (DSS) as “in teractiv e co m p u te r-b a se d sy stem s, w h ich h elp d e cisio n m a k e rs u tilize d a ta an d m od els to so lv e unstructured p ro b le m s ” (G o rry and Scott-M orton , 1 9 7 1 ). T h e fo llo w in g is a n o th e r cla ssic D SS d efin ition , p ro v id e d b y K e en an d Sco tt-M o rto n (1 9 7 8 ):

D e c is io n su p p o rt system s c o u p le th e in tellectu al re so u rce s o f individ uals w ith th e cap a b ilitie s o f th e co m p u te r to im p ro ve th e qu ality o f d e cisio n s. It is a c o m p u te r-b a s e d su p p o rt sy stem fo r m an a g e m e n t d e c is io n m ak ers w h o deal w ith sem istru ctu red p ro b lem s.

N ote th a t th e term d ecisio n su pport system , lik e m an ag em en t in fo rm a tio n system (M IS) an d o th e r te rm s in th e field o f IT , is a c o n te n t-fre e e x p r e s s io n ( i.e ., it m e a n s d ifferent things to d iffe ren t p e o p le ). T h e re fo re , th e re is n o u n iv ersally a c c e p te d d efin itio n o f D SS. (W e p re s e n t ad d itio n al d efin itio n s in C h ap ter 2 .) Actually, D SS c a n b e v ie w e d a s a c o n ­ cep tu a l m eth od olog y — th at is, a b ro a d , u m b rella term . H o w ev er, s o m e v ie w D SS as a nar­ ro w er, s p e c ific d e c is io n su p p o rt ap p licatio n .

DSS as an Um brella Term T h e term DSS c a n b e u se d as a n u m brella term to d e s c rib e any co m p u te riz e d sy stem that su p p orts d e c is io n m ak in g in a n organization. An o rg an izatio n m ay h a v e a k n o w le d g e

4 4 Part I * D ec isio n M aking and Analytics: An O verview

m a n a g e m e n t sy stem to g u id e all its p e rs o n n e l in th eir p ro b le m solving. A n o th e r o rg an iza­ tio n m ay h av e sep arate su p p o rt system s fo r m arketing , fin a n ce , an d a cco u n tin g ; a su p ­ p ly ch a in m a n a g e m e n t (SC M ) sy stem fo r p ro d u ctio n ; a n d sev eral ru le -b a s e d sy stem s fo r p ro d u ct re p a ir d iag n o stics an d h e lp d esk s. D SS e n c o m p a s s e s th e m all.

Evolutio n of DSS into Business intelligence I n th e e a rly d ay s o f D SS, m a n a g e rs le t th e ir s ta ff d o s o m e s u p p o rtiv e a n a ly sis b y u sin g D SS to o ls . As PC te c h n o lo g y a d v a n c e d , a n e w g e n e r a tio n o f m a n a g e rs e v o lv e d — o n e th a t w a s c o m fo r ta b le w ith co m p u tin g a n d k n e w th a t te c h n o lo g y c a n d ir e c tly h e lp m a k e in te llig e n t b u s in e s s d e c is io n s fa ste r. N ew to o ls s u c h a s OLAP, d ata w a re h o u s in g , data m in in g , a n d in te llig e n t s y stem s, d e liv e re d v ia W e b te c h n o lo g y , a d d e d p ro m ise d ca p a b ilitie s a n d e a sy a c c e s s to to o ls , m o d e ls, a n d d a ta fo r c o m p u te r-a id e d d e c is io n m a k in g . T h e s e to o ls sta rte d to a p p e a r u n d e r th e n a m e s B I and b u sin ess a n a ly tic s in th e m id -1 9 9 0 s . W e in tro d u c e th e s e c o n c e p t s n e x t, a n d re la te th e D SS an d B I c o n c e p ts in th e fo llo w in g s e c tio n s .

SECTION 1 .6 REVIEW QUESTIONS

1 . P rov id e tw o d efin itio n s o f DSS. 2 . D e s c r ib e DSS a s an u m brella term .

1.7 A FRAM EW ORK FOR BU SIN ESS IN TELLIG EN C E (BI) T h e d e c is io n su p p o rt c o n c e p ts p re se n te d in S e ctio n s 1.5 an d 1.6 h av e b e e n im p lem en ted in crem en tally , u n d e r d ifferen t n am e s, b y m an y v e n d o rs th a t h av e c re a te d to o ls an d m e th ­ o d o lo g ie s fo r d e c is io n su p p ort. As th e e n te rp rise-w id e sy stem s grew , m an ag ers w e re a b le to a c c e s s u ser-frien d ly reports that e n a b le d th e m to m a k e d e cis io n s q u ick ly . T h e s e sy stem s, w h ich w e re g e n erally ca lle d ex ecu tiv e in fo rm a tio n system s (E IS ), th e n b e g a n to o ffe r ad d itional v isualization , alerts, an d p e rfo rm a n ce m e a su re m e n t cap ab ilitie s. B y 2 0 0 6 , th e m a jo r co m m erc ia l p ro d u cts an d s erv ices a p p e a re d u n d er th e u m brella term bu sin ess in tellig en ce (B I).

D efinitions o f BI B usiness intelligence (B I) is a n u m brella term th at c o m b in e s arch itectu res, to o ls, data­ b a se s , analy tical to o ls, a p p licatio n s, an d m e th o d o lo g ie s. It is, lik e D SS, a co n ten t-fre e e x p re s s io n , s o it m e a n s d ifferen t things to d ifferen t p e o p le . P art o f th e c o n fu sio n a b o u t B I lies in th e flurry o f a cro n y m s an d b u zzw o rd s th at a re a s s o cia te d w ith it (e .g ., b u sin e ss p e rfo rm a n ce m an a g e m e n t [BPM]). B I ’s m a jo r o b je c tiv e is to e n a b le in teractiv e a c c e ss (s o m e tim e s in re al tim e ) to data, to e n a b le m an ip u latio n o f d ata, an d to give b u sin ess m an ag ers and analysts th e ability to co n d u c t a p p ro p riate analyses. B y analyzing historical a n d cu rre n t d ata, situations, an d p e rfo rm a n ce s, d e c is io n m ak ers g e t v a lu a b le insights that e n a b le th e m to m ak e m o re in fo rm ed an d b e tte r d e c is io n s . T h e p ro c e s s o f B I is b ase d o n th e tran sform ation o f data to in form ation , th e n to d e cisio n s, an d finally to action s.

A Brief H istory o f BI T h e term B I w as c o in e d b y th e G artn er G ro u p in th e m id -1 9 9 0 s. H o w ev er, th e c o n c e p t is m u ch o ld er; it h a s its ro ots in th e MIS re p o rtin g sy stem s o f th e 1970s. D u ring th at p erio d , rep ortin g sy stem s w e re static, tw o d im en sio n al, a n d had n o analytical cap ab ilities. In the e a rly 1 9 8 0 s, th e c o n c e p t o f ex ecu tiv e in fo rm a tio n system s (E IS ) em e rg ed . T h is c o n c e p t e x p a n d e d th e co m p u te rize d su p p o rt to to p -lev el m a n a g e rs an d e x e cu tiv e s . S o m e o f th e

Chapter 1 • An O verview o f B usiness Intelligen ce, Analytics, and D ecision Support

R G U R E 1.3 Evolution of Business Intelligence (Bl).

cap ab ilities in tro d u ced w e re d y n am ic m u ltid im en sion al (a d h o c o r o n -d e m a n d ) reporting, fo reca stin g an d p re d ictio n , tre n d analysis, drill-d ow n to d etails, statu s a c c e ss , a n d criti­ c a l s u c c e s s facto rs. T h e s e fe a tu res a p p e a re d in d o z e n s o f co m m e rcia l p ro d u cts until the m id -1990s. T h e n th e sa m e cap ab ilitie s a n d s o m e n e w o n e s a p p e a re d u n d e r th e n a m e B I. 7 odav. a g o o d B l-b a s e d e n te rp rise inform ation sy stem co n ta in s all th e in fo rm atio n e x e c u ­ t e s n e ed . S o , th e orig in al c o n c e p t o f EIS w a s tran sform ed in to B I. B y 2 0 0 5 , B I system s s e in e d to in clu d e a r tific ia l in tellig en ce cap ab ilitie s as w e ll as p o w erfu l an aly tical ca p a b ili­ ties. Figu re 1 .3 illustrates th e vario u s to o ls and te c h n iq u e s that m ay b e in clu d e d in a B I svstem . It illustrates th e e v o lu tio n o f B I as w ell. T h e to o ls s h o w n in F ig u re 1.3 p ro v id e the cap ab ilities o f B I. T h e m o st so p h istica te d B I p ro d u cts in clu d e m o s t o f th e s e cap ab ilities; oth ers s p e cia liz e in o n ly s o m e o f th em . W e w ill stud y sev eral o f th e s e ca p a b ilitie s in m o re detail in C h ap ters 5 th ro u g h 9.

The Architecture o f BI A B I system has four m ajor com ponents: a d a ta w arehou se, w ith its so u rce data; business an alytics, a co llectio n o f tools fo r manipulating, m ining, and analyzing th e data in th e data w areho u se; business p erfo rm a n ce m an agem en t (BPM) fo r m onitoring a n d analyzing perfor­ m ance; and a u ser in terface (e .g ., a dashboard). T h e relationship am ong th e se com p on en ts is illustrated in Figure 1.4. W e will discuss th ese com p on en ts in detail in Chapters 3 through 9-

Styles o f BS T h e a rch ite ctu re o f B I d e p e n d s o n its a p p licatio n s. M icroStrategy C o rp . d istinguishes five styles o f B I a n d o ffers s p e c ia l to o ls fo r e a c h . T h e fiv e styles are re p o rt d elivery a n d alert­ ing; e n te rp rise re p o rtin g (u sin g d ash b o ard s and s c o re ca rd s ); c u b e an aly sis (a ls o k n o w n as s lice -a n d -d ice an alysis); ad h o c q u e ries; a n d statistics a n d data m ining.

4 6 Part I • D ec isio n M aking and Analytics: An Overview

FIGURE 1.4 A High-Level Architecture of Bl. Source: Based on W. Eckerson, Smart Companies in the 21st Century: The Secrets o f Creating Successful Business Intelligent Solutions. The Data Warehousing Institute, Seattle, WA, 2003, p. 32, Illustration 5.

The Origins and D rivers o f Bl W h e re did m o d e rn a p p ro a c h e s to data w a reh o u sin g (D W ) and B l c o m e from? W h at are th e ir ro ots, and h o w d o th o s e ro o ts a ffe c t t h e w a y o rg an izatio n s are m an agin g th e s e initia­ tives today? T o d a y ’s in v estm en ts in in form ation te c h n o lo g y a re u n d e r in cre ase d scrutiny in term s o f th e ir b o tto m -lin e im p act and p o ten tial. T h e sam e is tru e o f D W an d th e B l ap p lica tio n s that m a k e th e s e initiatives p o ssib le.

O rg an izatio n s are b e in g c o m p e lle d to cap tu re, un d erstan d , a n d h a rn e ss th eir data to su p p o rt d e c is io n m ak in g in ord e r to im p ro ve b u sin e ss o p era tio n s. L egislation an d regu lation (e .g ., th e S a rb a n e s -O x le y A ct o f 2 0 0 2 ) n o w req u ire b u sin e ss lead e rs to d o cu ­ m e n t th e ir b u sin e ss p ro c e s s e s a n d to sig n o f f o n th e leg itim acy o f th e in form ation th ey rely o n an d re p o rt to stak e h o ld e rs. M oreover, b u s in e s s c y c le tim es are n o w e xtrem ely co m p re sse d ; faster, m o re in form ed , and b e tte r d e c is io n m ak in g is th e re fo re a com p etitiv e im perative. M an agers n e e d the right in form ation at th e right tim e an d in th e right p la c e . T h is is th e m antra fo r m o d e rn a p p ro a ch e s to B l.

O rganizatio ns have to w o rk smart. P aying carefu l atten tion to th e m an ag e m e n t o f B l initiatives is a n e cessa ry a s p e c t o f d o in g b u sin ess. It is n o surprise, th en , that o ig an izatio n s are in creasin g ly ch am p io n in g B l. Y o u will h e a r ab ou t m o re B l s u c c e s s e s an d th e funda­ m entals o f th o se s u c c e s s e s in C hapters 3 th ro u g h 9. E x a m p le s o f m any ap p licatio n s o f B l are provid ed in T a b le 1.3. A p plication C ase 1.1 illustrates o n e s u ch a p p licatio n o f B l that has h e lp ed m any airlines, as w e ll as th e co m p a n ie s o ffe rin g s u ch serv ices to th e airlines.

A M ultim edia Exercise in Business Intelligence Terad ata University N etw ork (TU N ) inclu des so m e v id e o s alo n g th e lin es o f the televi­ sio n sh o w CSI to illustrate co n c e p ts o f analytics in d ifferent industries. T h e s e are called “B S I V id eo s (B u sin e ss Scen ario Investigations).” N o t o n ly th e se are entertaining, but th ey also p rovide th e class w ith so m e q u estio ns fo r d iscu ssion. F or starters, p lease g o to teradatauniversitynetwork.com/teach-and-leam/library-item/?LibraryItemId=889 W atch th e vid eo that a p p ea rs o n Y o u T u b e . Essentially, you have to assu m e th e ro le o f a cu stom er service ce n te r professional. An in com in g flight is running late, and several pas­ sen gers are likely to m iss th eir co n n ectin g flights. T h e re are seats o n o n e ou tg o in g flight that c a n acco m m o d ate tw o o f th e fo u r p assen gers. W h ich tw o passen gers sh ou ld b e given

C hapter 1 • An Overview o f B u sin ess Intelligen ce, Analytics, and D ecisio n Support 4 7

T A B L E 1.3 Business V alu e of BI A nalytical Applications u -11 i 11111 i | i 111§ 11;

Analytic Application Business Q uestion Business V alue

Customer segmentation W hat market segments do my customers fall into, and what are their characteristics?

Personalize customer relationships for higher satisfaction and retention.

Propensity to buy Which customers are most likely to respond to my promotion?

Target customers based on their need to increase their loyalty to your product line.

Also, increase campaign profitability by focusing on the most likely to buy.

Customer profitability W hat is the lifetime profitability of my customer?

Make individual business interaction decisions based on the overall profitability of customers.

Fraud detection How can I tell which transactions are likely to be fraudulent?

Quickly determine fraud and take immediate action to minimize cost.

Customer attrition Which customer is at risk of leaving? Prevent loss of high-value customers and let go of lower-value customers.

Channel optimization W hat is the best channel to reach my cus­ tomer in each segment?

Interact with customers based on their preference and your need to manage cost.

Source: A. Ziama and J. Kasher, D ata M ining Prim er f o r the D ata W arehousing Professional. Teradata, Dayton, OH, 2004.

Application Case 1.1 Sabre Helps Its Clients Through Dashboards and Analytics Sabre is o n e o f the w orld leaders in th e travel indus­ try, providing b o th b u sin ess-to -consu m er services as well as b u sin ess-to -bu siness services. It serves travel­ ers, travel agents, corporations, and travel suppliers through its fo u r m ain com p an ies: Travelocity, Sabre Travel N etw ork, Sab re Airline Solutions, and Sabre Hospitality Solutions. T h e cu rrent volatile global e c o ­ no m ic e nvironm ent p o se s significant com petitive chal­ leng es to th e airline industry. T o stay ah e a d o f the com p etition, Sabre Airline Solutions recog n ized that airline e x e cu tiv es n eed ed en h a n ce d to o ls fo r m anag­ ing their b u sin ess d ecisio ns b y elim inating the tradi­ tional, m anual, tim e-consu m ing p ro cess o f co llect­ ing an d aggregating financial and o th er inform ation n eed ed fo r action ab le initiatives. This e n a b les real-tim e d ecisio n .support at airlines throughout th e w orld that m axim ize their (and, in turn, S a b re ’s) return o n infor­ m ation b y driving insights, action ab le intelligence, and value fo r cu stom ers fro m the grow ing data.

Sab re d ev elo p e d a n Enterprise Travel D ata W are h o u se (E T D W ) using Terad ata to hold its m as­ siv e reservations data. E TD W is up dated in near-real tim e w ith b a tch es that run e v ery 15 m inutes, gathering

data fro m all o f S ab re ’s b u sin esses. S ab re u se s its ETD W to cre a te Sab re E xecu tiv e D ashbo ard s that p ro ­ vid e n e a r-re a l-tim e e x e cu tiv e insights using a C ognos 8 B I platform w ith O racle D ata Integrator an d O ra cle G old en gate te ch n o lo g y infrastructure. T h e E xecutive D ash bo ard s o ffe r their clie n t airlines’ top-lev el m an­ agers and d e cisio n m ak ers a tim ely, autom ated , user- friendly solu tion, aggregating critical perfo rm an ce m etrics in a su ccin ct w ay an d providing at a glance a 3 6 0 -d e g ree v ie w o f the overall h ealth o f th e airline. At o n e airline, Sab re’s E xecu tiv e D ash b o ard s provide sen io r m an ag e m e n t w ith a daily and intra-day sn a p ­ sh o t o f k e y p erfo rm an ce indicators in a sin gle appli­ cation, re p lacin g th e o n ce -a -w e e k , 8-h o u r p ro cess o f generating th e sam e rep o rt fro m various data sou rces. T h e use o f d ash bo ard s is n o t lim ited to th e external cu stom ers; S ab re also u se s th e m fo r th eir assessm en t o f internal o p eratio n al perform ance.

T h e dashboards h elp Sab re's cu stom ers to have a clea r understanding o f th e data through d ie visual displays that in corp o rate interactive drill-dow n cap a­ bilities. It rep laces flat presentations and allow s for m o re fo cu se d review o f th e data w ith less effort and

('C on tin u ed )

4 8 Part I • D ecisio n M aking and Analytics: An Overview

Application Case 1.1 (Continued) tim e. This facilitates team d ialog b y m aking the data/ m etrics p ertainin g to sales p erform ance, including ticketing, seats sold a n d flow n, op erational perfor­ m a n ce s u ch as data o n flight m o v em en t and track­ ing, cu stom er reservations, inventory, and revenue acro ss a n airline’s multiple distribution channels, avail­ a b le to m any stakeholders. T h e dashboard system s provide scalab le infrastructure, graphical u se r interface (G U I) supp ort, data integration, an d data aggregation that e m p o w e r airline execu tiv es to b e m o re proactive in taking action s that lead to positive im pacts o n the overall health o f th eir airline.

W ith its ETD W , Sab re co u ld also d ev elo p o th er W e b -b a s e d analytical an d reporting solutions that lev ­ era g e data to g ain cu sto m e r insights through analysis Of cu sto m e r p ro files an d th eir s ales interactions to ca l­ cu late cu sto m e r value. T h is e n a b les b etter cu stom er seg m en tatio n a n d insights fo r v alu e-ad d ed services.

Q u e s t i o n s f o r D i s c u s s i o n

1. W h at is trad itional reporting? H o w is it u se d in organizations?

2. H o w c a n an aly tics b e u sed to tran sform tradi­ tion al reporting?

3- H o w c a n in teractiv e rep ortin g assist o rg an iza­ tio n s in d e c is io n m aking?

W h a t W e C a n L e a m f r o m T h is A p p lic a tio n C a se

T h is A p p lic a tio n C a s e s h o w s th at o rg a n iz a tio n s th a t e a r lie r u s e d re p o r tin g o n ly fo r tra c k in g th e ir in te rn a l b u s in e s s activ itie s a n d m e e tin g c o m p lia n c e r e q u ire m e n ts s e t o u t b y th e g o v e r n m e n t a re n o w m o v in g to w ard g e n e r a tin g a c tio n a b le in te llig e n c e fro m th e ir tra n s a c tio n a l b u s in e s s data. R e p o rtin g h a s b e c o m e b r o a d e r a s o r g a n iz a tio n s a re n o w try­ in g to a n a ly z e a rc h iv e d tra n sa c tio n a l d ata to u n d e r­ sta n d u n d e rly in g h id d e n tre n d s a n d p a tte rn s that w o u ld e n a b le th e m to m a k e b e tt e r d e c is io n s b y g a in in g in sig h ts in to p ro b le m a tic a re a s a n d re s o lv ­ in g th e m to p u rs u e cu rre n t an d fu tu re m a rk e t o p p o rtu n itie s . R e p o rtin g h a s a d v a n c e d to in te ra c ­ tiv e o n lin e r e p o r ts th a t e n a b le u s e rs t o pu ll and q u ic k ly b u ild c u s to m re p o r ts as re q u ire d a n d e v e n p re s e n t th e re p o r ts a id e d b y v is u a liz a tio n to o ls th a t h a v e th e a b ility to c o n n e c t to th e d a ta b a s e , p ro v id in g th e c a p a b ilitie s o f d ig g in g d e e p into su m m a riz ed d ata.

Source: Teradata.com, “Sabre Airline Solutions/'teradata.eom/t/ case-stu d ies/ Sabre-A ir!ine-Solution s-E B 6281 (accessed February 2013).

priority? Y o u are given inform ation a b o u t cu sto m ers’ profiles an d relationsh ip w ith th e air­ line. Y o u r d ecisio n s m ight ch a n g e as you learn m o re a b o u t th o se cu stom ers’ profiles.

W atch th e v id eo , p au se it as ap p ro p riate, an d a n s w e r th e q u e stio n s o n w h ich p as­ se n g e rs sh o u ld b e g iv e n priority. I h e n re su m e th e v id e o to g e t m o re in form ation . After th e v id e o is c o m p le te , y o u c a n s e e th e slid es re la te d to this v id eo and h o w th e analysis w a s p re p a re d o n a slid e s e t at teradatauniversitynetwork.com/templates/Download. aspx?ContentItemId=891. P lease n o te that access to this content requires initial registration.

T h is m u ltim ed ia e x cu rs io n p ro v id es a n e x a m p le o f h o w a d d itional in form ation m ad e a v ailab le th ro u g h a n e n te rp rise data w a re h o u s e c a n assist in d e cisio n m aking.

The DSS-BI Connection B y n o w , y o u sh ou ld b e a b le to s e e s o m e o f th e sim ilarities a n d d iffe re n ce s b e tw e e n D SS a n d B I. First, th e ir arch ite ctu res are v e ry sim ilar b e c a u s e B I e v o lv e d fro m D SS. H ow ever, B I im p lies th e u s e o f a data w a re h o u s e , w h e re a s D SS m ay o r m ay n o t h a v e s u ch a feature. B I is, th e re fo re , m o re ap p rop riate fo r larg e o rg a n iz a tio n s (b e c a u s e data w a re h o u s e s are e x p e n s iv e to b u ild a n d m ain tain ), b u t D SS ca n b e a p p ro p riate to a n y typ e o f organization.

S e c o n d , m o st D SS are co n stru cte d to d irectly su p p o rt s p e c ific d ecisio n m aking. B I sy stem s, in g e n era l, a re g e a re d to pro v id e a ccu ra te an d tim ely in form ation, an d th e y sup­ p o rt d e c is io n su p p o rt in directly. T h is situ ation is ch a n g in g , h o w e v e r, as m o re a n d m o re d e c is io n su p p o rt to o ls a re b e in g ad d ed to B I so ftw a re p a ck a g e s.

Chapter 1 • An O verview o f B u sin ess Intelligence, Analytics, and D ecisio n Support 4 9

Third , B I h a s an e x e c u tiv e a n d strategy o rie n tatio n , e sp e cia lly in its B P M a n d d ash­ b oard co m p o n e n ts . D SS, in con trast, is o rie n te d to w ard analysts.

Fou rth, m o st B I sy stem s a re co n stru cte d w ith co m m e rcia lly a v ailab le to o ls a n d c o m ­ p o n en ts th at a re fitted to th e n e e d s o f o rgan ization s. In b u ild in g D SS, th e in te rest m ay b e in co n stru ctin g so lu tio n s to very u n stru ctu red p ro b lem s. In s u c h situ ation s, m o re p ro ­ gram m ing (e .g ., u sin g to o ls s u ch as E x c e l) m ay b e n e e d e d to cu sto m ize th e solu tions.

Fifth, D SS m e th o d o lo g ie s an d e v e n s o m e to o ls w e re d e v e lo p e d m o stly in th e a c a ­ d em ic w o rld . B I m e th o d o lo g ie s an d to o ls w e re d e v e lo p e d m o stly b y so ftw are co m p a n ie s. (S e e Z am an, 2 0 0 5 , fo r in form ation o n h o w B I h a s e v o lv e d .)

Sixth, m a n y o f th e to o ls that B I u se s a re a lso co n sid e re d D SS to o ls. F o r e x a m p le , data m in in g a n d p red ictiv e analysis are c o r e to o ls in b o th areas.

A lthough s o m e p e o p le e q u a te D SS w ith B I, th e s e sy stem s a re not, a t p re sen t, th e sam e. It is in te restin g to n o te that s o m e p e o p le b e lie v e that D SS is a p art o f B I — o n e o f its analytical to o ls . O th ers th in k that B I is a s p e c ia l c a s e o f D SS th at d eals m o stly w ith rep o rt­ ing, co m m u n ica tio n , an d co lla b o ra tio n (a fo rm o f d ata-o rien ted D SS). A n o th e r e x p la n a ­ tio n (W atso n , 2 0 0 5 ) is that B I is a result o f a co n tin u o u s re v o lu tio n and , a s s u ch , D SS is o n e o f B I ’s original e le m e n ts. In this b o o k , w e sep arate D SS fro m B I. H o w e v e r, w e p o in t to th e D S S -B I c o n n e c tio n freq u en tly . Further, a s n o te d in th e n e x t s e c tio n onw ard , in m any c irc lc s B I h a s b e e n su b su m e d b y th e n e w term a n a ly tics o r d a ta scien ce.

SECTION 1 .7 REVIEW QUESTIONS

1 . D e fin e BI. 2 . List an d d e s c r ib e th e m ajo r c o m p o n e n ts o f B I. 3. W h at a re th e m a jo r sim ilarities a n d d iffe ren ce s o f D SS a n d BI?

1,8 B U SIN ESS A N A L Y T IC S OVERVIEW T h e w o rd “an alytics” h a s re p la ced th e previou s individual co m p o n e n ts o f com p u terized d ecisio n su p p o rt te ch n o lo g ie s that h av e b e e n available u n d er v arious lab e ls in th e past. In d eed , m an y p ractitioners and a cad em ics n o w u se th e w o rd a n a ly tics in p la ce o f B I. Although m an y au th ors an d con su ltan ts h av e d efin ed it slightly differently, o n e ca n v ie w analytics as th e p ro cess o f d ev elo p in g a ctio n a b le d ecisio n s o r re co m m en d a tio n fo r action s b a se d u p o n insights g e n era ted fro m h istorical data. T h e Institute fo r O p e ra tio n s R esearch and M an ag em en t S c ie n c e (IN FO RM S) has cre a te d a m a jo r initiative to org an ize and p ro ­ m o te analytics. A cco rd in g to INFORMS, analytics re p re sen ts th e co m b in a tio n o f co m p u te r te ch n o lo g y , m a n a g e m e n t s c ie n c e te ch n iq u es, and statistics to solve real p ro b lem s. O f co u rse , m an y o th e r organ ization s h av e p ro p o se d th eir o w n interpretations a n d m otivation fo r analytics. F o r e x a m p le , SAS Institute In c. p ro p o s e d eig h t lev els o f an alytics th at b eg in w ith stand ard ized reports fro m a co m p u te r system . T h e s e rep o rts essen tially p rovide a s e n se o f w h a t is h a p p e n in g w ith an organization. A dditional te c h n o lo g ie s h av e e n a b le d us to cre a te m o re cu sto m ized reports that c a n b e g e n era ted o n a n ad h o c b asis. T h e n e x t e x te n sio n o f rep ortin g tak es us to o n lin e analytical p ro cessin g (O L A P )-ty p e q u eries that allow a u se r t o dig d e e p e r an d d eterm in e th e sp e cific so u rce o f c o n c e rn o r o p p o rtu n i­ ties. T e c h n o lo g ie s available to d ay c a n also autom atically issu e alerts fo r a d e c is io n m aker w h e n p e rfo rm a n ce issu es w arrant s u ch alerts. At a co n su m e r lev el w e s e e s u ch alerts fo r w e ath e r o r o th e r issues. B u t sim ilar alerts c a n a lso b e g e n era ted in s p e cific settin gs w h e n sales fall a b o v e o r b e lo w a certain lev el w ith in a certain tim e p eriod o r w h e n th e in v en toiy fo r a s p e cific p ro d u ct is running low . All o f th e se ap p licatio n s are m ad e p o ss ib le through analysis a n d q u e rie s o n data b e in g c o lle cte d b y a n organization. T h e n e x t le v e l o f analysis m ight entail statistical analysis to b e tte r un derstand pattern s. T h e s e c a n th e n b e ta k e n a step fu rther to d e v e lo p fo reca sts o r m o d e ls fo r p red ictin g h o w cu stom ers m ig h t re sp o n d to

5 0 Part I * D ecisio n M aking and Analytics: An O verview

FIGURE 1.5

L T a t o o d â w mSf ° r ° n80ing Service/Product « « * * * * ■ an organization o th e r r !c h ^ ! ! h,ap p em n « an d w h at is lUcely » h ap p e n , it c a n also em p loy

e r te ch n iq u es to m a k e th e b e s t d ecisio n s u n d e r th e circu m stan ces. T h e s e eigh t lev els o f

:2Z“LZ'X" “ - • * ' T h is id ea o f lo o k in g at all th e data to u n d erstan d w h a t is h a p p e n in g w h at will

h a p p e n , a n d h o w to m a k e th e b e s t o f it h a s a ls o b e e n e n ca p s u la te d b y IN FORM S in p ro p o sin g th ree lev els o f analytics. T h e s e th re e lev els a r e identified (in fo rm s.o r^ Community/Analytics) as d escrip tiv e, p red ictiv e, an d p re scrip tiv e . Fig u re 1.5 p r e s e n t

o grap h ical v iew s o f th e se th ree lev els o f analytics. O n e v ie w su gg ests that th e s e th ree

t T a ” T h e lderPendC nt ST ( 1 3 Iad der) ^ ° n e ^ ° f:analytlCS a p p lica tio n leads an o th er. T h e in te rco n n e cte d circle s v ie w su ggests th at th e re is actu ally s o m e ov erlap

* * * ? ° f analytlCS, In CaSe’ th e in te rc o n n e c te d n atu re o f d ifferent ty p e s o f an aly tics a p p lica tio n s is evident. W e n e x t in tro d u ce th e s e th re e lev els o f analytics.

D escriptive A nalytics

D escriptive or rep o rtin g an aly tics re fe rs to k n o w in g w h a t is h a p p e n in g in th e o r g a n iz a tio n a n d u n d e rs ta n d in g s o m e u n d e rly in g tre n d s a n d c a u s e s o f s u c h o c c u r ­ r e n c e s . T h is in v o lv e s , first o f all, c o n s o lid a tio n o f d ata s o u r c e s a n d a v ailab ility o f

P re d ictiv e Statistical Analysis and

Data Mining

R e p o rtin g Visualization

Periodic, ad hoc Reporting Trend Analysis

P r e s c r ip tiv e Management Science Models and Solution

Management Science Models and

Solution P red ictiu e

Statistical Analysis and

Data Mining R e p o rtin g

Visualization Periodic,

ad hoc Reporting Trend Analysis

Three Types of Analytics.

Chapter 1 • An O verview o f B usiness Intelligence, Analytics, and D ecisio n Support 51

a ll re le v a n t d a ta in a fo rm th at e n a b le s a p p r o p ria te re p o r tin g a n d a n a ly sis. U su ally d ev e lo p m e n t o f this d ata in fra stru ctu re is p a rt o f d ata w a r e h o u s e s , w h ic h w e stu d y in C h apter 3 . F ro m th is d ata in fra stru ctu re w e c a n d e v e lo p a p p r o p ria te re p o r ts , q u e rie s , -i.erts, a n d tre n d s u s in g v a rio u s re p o r tin g to o ls a n d te c h n iq u e s . W e s tu d y th e s e in C h ap ter 4.

A sig n ifican t te c h n o lo g y that h a s b e c o m e a k e y p la y er in this a re a is visualization . Using th e la test visu alization to o ls in th e m a rk e tp la ce , w e c a n n o w d e v e lo p p o w erfu l insights in to th e o p era tio n s o f o u r org anization. A p p licatio n C ase s 1 .2 an d 1 .3 highlight fo m e s u ch a p p lica tio n s in th e h e a lth ca re d om ain. C o lo r ren d erin g s o f s u ch ap p licatio n s are av ailab le o n th e c o m p a n io n W e b site an d a lso o n T a b le a u ’s W e b site. C h ap ter 4 co v ers v isu alization in m o re detail.

Application Case 1.2 Eliminating Inefficiencies at Seattle Children's Hospital Seattle C h ild ren ’s w as th e sev en th h ig h est ranked ch ild ren ’s h o sp ital in 2 0 1 1 , a cco rd in g to U.S. News & W orld R eport. F o r a n y org an izatio n th at is c o m ­ m itted to sav in g liv es, id entifying an d rem ov in g the in efficie n cie s fro m sy stem s and p ro c e s s e s s o that m o re re s o u rc e s b e c o m e a v ailab le to c a te r to p atien t care b e c o m e v e ry im portant. At Seattle C h ild ren’s, m an ag e m e n t is co n tin u o u sly lo o k in g fo r n e w w ay s to im p ro ve th e quality, safety , an d p ro c e s s e s from th e tim e a p a tie n t is ad m itted to th e tim e th e y are d ischarged . T o this end , th e y sp e n d a lo t o f tim e in analyzing th e d ata a s s o cia te d w ith th e p atie n t visits.

T o q u ick ly a i m p atien t and hospital data into insights, Seattle C hildren’s im plem en ted T a b le a u Softw are’s b u sin e ss in te llig e n ce ap p licatio n . It p ro­ vides a b ro w se r b a se d o n e asy -to -u se analytics to th e stakehold ers; this m ak e s it intuitive fo r individuals to create visualization s and to u n d erstan d w h at th e data has to offer. T h e data analysts, b u sin ess m anagers, an d financial analysts as w e ll a s clinician s, d octo rs, an d re sea rch ers a re all u sin g d escriptive analytics to solve d ifferen t p ro b lem s in a m u ch faster w ay. T h ey are d ev elo p in g visual system s o n th e ir ow n, resulting in d ash b o ard s an d sco re card s th at help in d efin in g th e standards, th e cu rren t p erfo rm an ce ach iev ed m e a su re d against th e standards, an d h o w th ese sy stem s w ill g ro w into the future. T h ro u g h th e use o f m o n th ly a n d daily d ashbo ard s, day-to-day d ecisio n m ak in g at Seattle C hildren’s has im proved significantly.

Seattle C h ild ren ’s m e a su re s p atien t w ait-tim es an d an aly zes th e m w ith th e h e lp o f visualization s to d isco v e r th e ro o t ca u se s an d con trib u tin g facto rs

fo r p atie n t w aiting. T h e y fo u n d that early d elays c a s c a d e d during th e day. T h e y fo c u s e d o n o n -tim e ap p o in tm en ts o f p a tie n t s e rv ice s as o n e o f th e s o lu ­ tio n s to im p ro vin g p atie n t ov erall w aitin g tim e an d in cre asin g th e availability o f b ed s. Seattle C hildren's s a v e d a b o u t $3 m illion fro m th e su p p ly ch ain , and w ith th e h e lp o f to o ls lik e T a b le a u , th e y a re find ­ ing n e w w ays to in c re a s e savings w h ile treatin g as m an y p a tie n ts as p o ss ib le b y m ak in g th e existin g p ro c e s s e s m o re efficien t.

Q u e s t i o n s f o r D i s c u s s i o n

1. W h o a re th e u se rs o f th e tool? 2. W h a t is a d ashboard? 3. H o w d o e s v isu alization h e lp in d e c is io n m aking? 4 . W h at a re th e sig n ifican t results a ch ie v e d b y th e

u se o f T ab leau ?

W h a t W e C a n L e a r n f r o m T h is A p p lic a tio n C a se

T h is A p p licatio n C a se s h o w s th at re p o rtin g analyt­ ics in volvin g v isu alization s s u c h as d ash b o ard s can o ffe r m a jo r insights in to e x istin g data an d s h o w h o w a variety o f u sers in d iffe ren t d om ain s an d d ep art­ m e n ts c a n co n trib u te to w ard p ro c e s s and q u a l­ ity im p ro v em en ts in a n organization. Fu rth erm ore, e x p lo rin g th e d ata visually c a n h e lp in identifying th e ro o t c a u se s o f p ro b le m s an d pro v id e a b a sis fo r w o rk in g to w ard p o s s ib le solu tions.

Source: Tableausoftware.com, “Eliminating Waste at Seattle Children’s, ” tableausoftware.com/ eliminating-waste-at-seattle- childrens (accessed Febmaiy 2013).

5 2 Part I • D ec isio n M aking and Analytics: An Overview

Application Case 1.3 Analysis at the Speed of Thought K aleid a H ealth , th e largest h e a lth ca re p ro v id e r in w e ste rn N ew Y o rk , h a s m o re th a n 1 0 ,0 0 0 e m p lo y ­ e e s , fiv e h o sp itals, a n u m b e r o f clin ics an d nu rsing h o m es, a n d a v isitin g-n u rse a sso cia tio n th at d eals w ith m illion s o f p atie n t reco rd s. K a leid a ’s traditional rep ortin g to o ls w e re in a d e q u a te to h a n d le th e g ro w ­ ing data, an d th e y w e re fa c e d w ith the c h a lle n g e o f fin d in g a b u sin e ss in te llig e n ce to o l th a t c o u ld h an d le large d ata s e ts effo rtlessly, q u ick ly , a n d w ith a m u ch d e e p e r an aly tic capability.

At K aleid a, m an y o f th e calcu latio n s are n o w d o n e in T a b le a u , prim arily pu lling th e d ata from O ra cle d a ta b a se s in to E x c e l an d im p orting th e d ata into T a b le a u . F o r m an y o f th e m o n th ly a n a ­ ly tic re p o rts, data is d irectly e x tra cted in to T ab leau fro m th e d ata w a re h o u s e ; m an y o f th e data q u eries a re sav e d a n d reru n, resultin g in tim e savings w h e n d ealin g w ith m illions o f re co rd s— e a c h h av in g m o re th an 4 0 field s p e r reco rd . B e s id e s sp e e d , K aleid a a lso u s e s T a b le a u to m e rg e d ifferen t ta b le s fo r g e n ­ e ratin g extracts.

U sing T a b le a u , K aleid a c a n an alyze e m e rg en c y ro o m data t o d eterm in e th e n u m b e r o f p atien ts w h o visit m o re th a n 10 tim es a year. T h e data o fte n reveal that p e o p le fre q u e n tly u se e m e rg e n c y ro o m and a m b u la n ce s e rv ice s in ap p ro p riately fo r sto m a ch ­ a c h e s , h e a d a c h e s , and fevers. K aleid a c a n m an ag e re s o u rce utilization s— th e u se a n d c o s t o f su p p lies— w h ich will u ltim ately le a d t o e fficie n cy a n d stand ard ­ izatio n o f su p p lie s m a n a g e m e n t a cro ss th e system .

K a leid a n o w h a s its o w n b u sin ess in te llig e n ce d ep artm en t an d u se s T a b le a u to co m p a re its e lf to

o th e r h osp itals a c ro ss th e cou ntry. C o m p ariso n s are m ad e o n v ario u s asp e cts, s u ch as len g th o f p atien t stay, h o sp ital p ra ctice s, m ark e t sh are, an d p artn er­ sh ip s w ith d o cto rs.

Q u e s t i o n s f o r D i s c u s s i o n

1. W h a t are th e d esire d fu n ctio n alitie s o f a rep o rt­ ing tool?

2. W h at ad v an tag e s w e re d eriv ed b y u sin g a rep o rt­ ing to o l in th e case?

W h a t W e C a n L e a r n f r o m T h is A p p lic a tio n C a se

C o rre ct s e le c tio n o f a rep ortin g to o l is e x trem ely im portant, e sp e c ia lly if a n org an izatio n w an ts to d eriv e v alu e fro m reporting. T h e g e n e ra te d reports an d visu a lizatio n s sh o u ld b e e asily d isce rn ib le; they sh o u ld h e lp p e o p le in d ifferen t s e c to rs m a k e s e n se o u t o f th e re p o rts, identify th e p ro b lem a tic areas, an d co n trib u te to w ard im p ro vin g them . M any future org an izatio n s w ill req u ire re p o rtin g an alytic to o ls that are fast an d c a p a b le o f h an d lin g h u ge am o u nts o f d ata e fficie n tly to g e n e ra te d esired rep o rts w ith­ o u t th e n e e d fo r third-party co n su ltan ts an d serv ice providers. A tru ly u se fu l re p o rtin g to o l c a n e x e m p t org an izatio n s fro m u n n e ce s s a ry exp en d itu re.

Source: TabIeau.software.com, “Kaleida Health Finds Efficiencies, Stays Competitive,” tableausoftware.com/leam/stories/user- experience-speed-thought-kaleida-health (accessed February 2013).

Predictive A nalytics Predictive analytics aim s to d eterm in e w h at is lik e ly to h a p p e n in th e future. T h is analy­ sis is b a s e d o n statistical te c h n iq u e s as w e ll a s o th e r m o re re ce n tly d e v e lo p e d te ch n iq u e s th at fall u n d e r th e g e n eral ca te g o ry o f d ata m ining . T h e g o al o f t h e s e te ch n iq u e s is to b e a b le to p re d ict i f th e cu sto m e r is lik ely to sw itch t o a co m p e tito r ( “ch u rn ”), w h a t th e cu s­ to m e r is lik e ly to b u y n e x t a n d h o w m u ch , w h at p ro m o tio n a cu sto m e r w o u ld re sp o n d to, o r w h e th e r this cu sto m e r is a cred itw orth y risk. A n u m b e r o f te ch n iq u e s a re u s e d in d e v e lo p in g p red ictiv e an aly tical ap p licatio n s, in clu d in g vario u s classificatio n algorithm s. F o r e x a m p le , as d e s crib e d in C h ap ters 5 an d 6 , w e c a n u s e classificatio n te c h n iq u e s su ch a s d e cisio n tre e m o d els a n d n eu ral n e tw o rk s to p re d ict h o w w e ll a m o tio n p ictu re will d o a t th e b o x o ffice . W e c a n a ls o u se clu sterin g algorithm s fo r s eg m en tin g cu sto m ers in to d ifferen t clu sters to b e a b le to targ et s p e c ific p ro m o tio n s to them . Finally, w e c a n

C hapter 1 • An O verview o f B u sin ess Intelligen ce, Analytics, and D ecisio n Support 5 3

use a s s o cia tio n m in in g te c h n iq u e s to e stim ate relatio n sh ip s b e tw e e n d iffe ren t p u rch asin g b ehav io rs. T h a t is, i f a cu sto m e r b u y s o n e p ro d u ct, w h a t e ls e is th e cu sto m e r lik e ly to pu r­ chase? S u c h a n aly sis c a n assist a retailer in re co m m e n d in g o r p ro m o tin g re la ted produ cts. F o r e x a m p le , an y p ro d u ct s e a rch o n A m a z o n .co m results in the re taile r a lso suggestin g other sim ilar p ro d u cts th at m ay in te rest a cu sto m er. W e will study th e s e te c h n iq u e s and th eir a p p lica tio n s in C h ap ters 6 throu gh 9- A p p lication C ases 1.4 an d 1 .5 hig h lig h t so m e sim ilar a p p licatio n s. A p p lication C ase 1 .4 in tro d u ces a m o v ie y o u m ay h av e h e ard of: M oneyball. It is p e rh a p s o n e o f th e b e s t e x a m p le s o f ap p licatio n s o f p re d ictiv e analysis in sports.

Application Case 1.4 M oneyball: Analytics in Sports and Movies M oneyball, a b io g ra p h ica l, sp orts, dram a film , w as re le a se d in 2011 and d irecte d b y B e n n e tt M iller. T h e film w a s b a s e d o n M ich ael Lew is’s b o o k , M oneyball. T h e m o v ie g a v e a d eta iled a c c o u n t o f th e O ak lan d A thletics b a s e b a ll te a m during th e 2 0 0 2 s e a s o n and the O a k la n d g e n eral m an ag e r’s e ffo rts to a sse m b le a com p etitiv e team .

T h e O ak lan d A thletics su ffered a b ig lo ss to th e Newr Y o rk : Y a n k e e s in 20 0 1 p o stse a so n . As a result, O ak la n d lo s t m a n y o f its star p layers to fre e a g e n c y and e n d e d u p w ith a w e a k te a m w ith u n fav o rab le financial p ro sp ects. T h e g e n eral m an ag e r’s efforts to re a sse m b le a co m p etitiv e te a m w e r e d en ie d b e c a u s e O ak lan d h a d lim ited payroll. T h e sco u ts for th e O ak lan d A thletics fo llo w e d th e o ld b a se b a ll cu sto m o f m ak in g s u b je ctiv e d e cis io n s w h en s e le ctin g th e team m e m b e rs. T h e g e n e ra l m an ag e r th e n m e t a you ng, co m p u te r w h iz w ith an e c o n o m ic s d eg ree from Y a le . T h e g e n e ra l m an ag e r d e c id e d to ap p o in t him as th e n e w assistan t g e n e ra l m an ager.

T h e assistan t g e n eral m an ag e r h a d a d e e p p as­ sio n fo r b a s e b a ll an d h ad th e e x p e rtis e to c m n c h the n u m b e rs fo r th e g am e. H is lo v e fo r th e g a m e m ad e him d e v e lo p a rad ical w a y o f un d erstand ing b a se b a ll statistics. H e w a s a d iscip le o f B ill J a m e s , a m arginal fig u re w h o o ffe red ratio n alized te ch n iq u e s io an aly ze b a se b a ll. Ja m e s lo o k e d at b a s e b a ll statis­ tics in a d iffe ren t w ay , cru n ch in g th e n u m b e rs pu rely o n facts a n d e lim in atin g subjectivity. Ja m e s p io ­ n e e re d th e n on trad ition al analysis m e th o d calle d th e S ab erm etric ap p ro a ch , w h ich d eriv ed fro m SABR— S o cie ty fo r A m erican B a s e b a ll R esearch .

T h e a s s is ta n t g e n e r a l m a n a g e r fo llo w e d th e S a b e rm e tric a p p r o a c h b y b u ild in g a p re d ic tio n

m o d e l to h e lp th e O a k la n d A th le tics s e l e c t p la y ­ e rs b a s e d o n th e ir “o n - b a s e p e r c e n ta g e ” ( O B P ) , a s ta tistic th a t m e a s u re d h o w o fte n a b a tte r r e a c h e d b a s e fo r a n y r e a s o n o t h e r th a n fie ld in g e rro r, fie ld ­ e r ’s c h o ic e , d ro p p e d / u n ca u g h t third s trik e , fie ld e r ’s o b s tru c tio n , o r c a tc h e r ’s in te r fe r e n c e . R a th e r th an re ly in g o n th e s c o u t ’s e x p e r ie n c e a n d in tu itio n , th e a s s is ta n t g e n e r a l m a n a g e r s e le c te d p la y e rs b a s e d a lm o s t e x c lu s iv e ly o n O B P .

S p o iler A lert: T h e n e w te a m b e a t all od d s, w o n 2 0 co n s e cu tiv e g a m e s , an d s e t a n A m erican L eagu e record .

Q u e s t i o n s f o r D i s c u s s i o n

1. H o w is p re d ictiv e an alytics a p p lie d in M oneyball? 2. W h at is th e d iffe re n ce b e tw e e n o b je c tiv e and

s u b je ctiv e a p p r o a c h e s in d e c is io n m aking?

W h a t W e C a n L e a m f r o m T h is A p p lic a tio n C a se

A n aly tics fin d s its u s e in a v arie ty o f ind u stries. It h e lp s o rg a n iz a tio n s re th in k th e ir trad itio n al p r o b ­ le m -s o lv in g a b ilitie s, w h ic h are m o st o f te n s u b je c ­ tiv e, re ly in g o n th e sa m e o ld p r o c e s s e s to find a so lu tio n . A n aly tics ta k e s th e rad ical a p p r o a c h o f u s in g h isto rica l d ata to fin d fa c t-b a s e d so lu tio n s th a t w ill re m a in a p p ro p ria te fo r m ak in g e v e n future d e c is io n s .

Source: Wikipedia, “On-Base Percentage,” en.wikipedia.org/ wiki/On_base_percentage (accessed January 2013); Wikipedia, “Saberrnetricsm,” wikipedia.org/wiki/Saberm etfics (accessed January 2013).

Part I • D e c is io n M aking and Analytics: An Overview

Application Case 1.5 Analyzing Athletic Injuries A ny a th le tic activ ity is p r o n e to in ju rie s. I f t h e in ju ­ ries a re n o t h a n d le d p ro p e rly , th e n th e te a m suf­ fers. U sin g a n a ly tic s to u n d e rstan d in ju rie s c a n h e lp in d eriv in g v a lu a b le in sig h ts th a t w o u ld e n a b le th e c o a c h e s a n d te a m d o c to rs to m a n a g e th e te a m c o m p o s itio n , u n d e rsta n d p la y e r p ro file s, a n d ulti­ m a te ly a id in b e tte r d e c is io n m ak in g c o n c e rn in g w h ic h p la y e r s m ig h t b e a v a ila b le to p la y at an y g iv e n tim e.

In a n e x p lo ra to ry study, O k la h o m a State U niversity a n a ly z ed A m erican fo o tb a ll-re la ted sport in ju ries b y u sin g re p o rtin g and p red ictiv e analytics. T h e p ro je c t fo llo w e d th e CRISP-D M m e th o d o l­ o g y to u n d erstan d th e p ro b le m o f m ak in g re c o m ­ m e n d atio n s o n m an ag in g inju ries, un derstand ing th e vario u s d ata e le m e n ts c o lle c te d a b o u t injuries, c le a n in g th e d ata, d ev elo p in g visu alization s to draw v arious in fe re n c e s , b u ild in g p red ictiv e m o d e ls to an aly ze th e injury- h e a lin g tim e p erio d , a n d draw ing s e q u e n c e ru le s to p re d ict th e re latio n sh ip am o n g th e in ju ries a n d th e v ario u s b o d y part parts afflicted w ith injuries.

T h e inju ry data s e t c o n s is te d o f m o re than 5 6 0 fo o tb a ll in ju ry re co rd s, w h ich w e re ca te g o riz e d in to in ju ry -sp e cific v ariab les— b o d y part/site/later- ality, a ctio n ta k e n , sev erity, injury ty p e, injury start an d h e a lin g d ates— and p layer/ sp ort-sp ecific varia­ b le s — p lay er ID , p o sitio n p lay ed , activity, o n s e t, an d g a m e lo c a tio n . H e alin g tim e w as ca lcu la ted fo r e a c h re co rd , w h ic h w a s cla ssifie d in to d ifferen t sets o f tim e p e rio d s: 0 - 1 m o n th , 1 - 2 m o n th s, 2 - 4 m onths, 4 - 6 m o n th s, a n d 6 - 2 4 m onths.

V ario u s visu alization s w e re built to draw in fe re n c e s fro m injury d ata s e t in form ation d ep ict­ ing th e h e a lin g tim e p e rio d a s s o cia te d w ith p lay ers’ p o sitio n s, sev erity o f inju ries a n d th e h e a lin g tim e p e rio d , tre a tm e n t o ffe re d an d th e a s s o cia te d h e a lin g tim e p erio d , m a jo r inju ries afflictin g b o d y parts, an d s o forth.

N eural n e tw o rk m o d els w e re b u ilt to p re ­ d ict e a c h o f th e h e a lin g ca te g o rie s using IB M SPSS M o d eler. S o m e o f th e p re d icto r v ariab les w e re cu r­ rent statu s o f injury, severity, b o d y part, b o d y site, ty p e o f injury, activity, e v e n t lo ca tio n , a c tio n tak e n , a n d p o sitio n p lay ed . T h e s u c c e s s o f classify in g th e h e a lin g ca te g o ry w a s q u ite g o o d : A ccu racy w a s 79-6 p e rce n t. B a s e d o n th e analy sis, m an y b u sin e ss re c ­ o m m en d a tio n s w e re su g g e ste d , in clu d in g e m p lo y ­ ing m o re sp e cia lists’ input fro m in ju iy o n s e t instead o f lettin g th e training ro o m sta ff s c r e e n th e in ju red p layers; training p lay ers a t d efen siv e p o sitio n s to a v o id b e in g in ju red ; a n d h o ld in g p ractice to th o r­ ou g h ly s a fe ty -c h e c k m ech an ism s.

Q u e s t i o n s f o r D i s c u s s i o n

1. W hat ty p e s o f an aly tics are ap p lie d in th e injury analysis?

2. H o w d o v isu alization s aid in u n d erstan d in g th e d ata and d eliv erin g insights in to th e data?

3. W h at is a cla ssifica tio n problem ? 4. W h at c a n b e d eriv ed b y p erfo rm in g s e q u e n c e

analysis?

W h a t W e C an L e a r n f r o m T h is A p p lic a tio n C a se

F o r an y an alytics p ro je ct, it is alw ays im portant to u n d erstan d th e b u sin e ss d o m ain a n d th e cu r­ re n t state o f th e b u sin e ss p ro b le m th ro u g h e x te n ­ siv e analysis o f th e only re so u rce — h istorical data. V isu alizatio n s o fte n pro v id e a g reat to o l fo r g aining th e initial insights in to d ata, w h ich c a n b e further refin ed b a s e d o n e x p e r t o p in io n s to id en tify th e re la ­ tive im p o rtan ce o f th e d ata e le m e n ts related to th e p ro b lem . V isu alizatio n s a lso aid in g e n eratin g ideas fo r o b s c u re b u s in e s s p ro b le m s, w h ic h c a n b e pu r­ su e d in b u ild in g p red ictiv e m o d els th at co u ld h e lp o rg an ization s in d e c is io n m aking.

Prescriptive A nalytics T h e third ca te g o ry o f a n alytics is term ed p rescrip tive analytics. T h e g o a l o f p rescriptive an aly tics is to re co g n iz e w h a t is g o in g o n as w e ll as th e lik e ly fo re c a s t a n d m ak e d ecisio n s to a c h ie v e th e b e s t p e rfo rm a n ce p o ssib le . T h is g ro u p o f te c h n iq u e s h a s historically b e e n stu d ied u n d er th e u m b rella o f o p era tio n s re se a rch o r m an a g e m e n t s c ie n c e s an d h a s g e n ­ erally b e e n aim ed a t op tim izing th e p e rfo rm a n ce o f a system . T h e g o a l h e re is to p rovide

Chapter 1 • An O verview o f B u sin ess Intelligence, Analytics, and D ecision Support 55

2 decision o r a re co m m e n d a tio n fo r a s p e cific a ctio n . T h e s e re co m m en d a tio n s c a n b e in i h e form s o f a s p e c ific yes/no d e cisio n fo r a p ro b le m , a s p e c ific am o u n t (say , p ric e fo r a so e cific item o r airfare to c h a rg e ), o r a c o m p le te s e t o f p ro d u ctio n plans. T h e d ecisio n s m ay b e p re se n te d t o a d e c is io n m a k e r in a report o r m ay d irectly b e u s e d in a n au tom ated decision ru les sy stem (e .g ., in airline p ricin g sy stem s). T h u s, th e s e ty p e s o f an aly tics ca n u o b e te rm e d d ecision o r norm ative analytics. A p p lication C ase 1.6 gives a n e x a m p le r f s u ch p rescrip tiv e an alytic a p p licatio n s. W e w ill le a rn a b o u t s o m e o f th e s e te c h n iq u e s irid sev eral ad d ition al a p p lica tio n s in C h ap ters 10 th ro u g h 12.

Application Case 1.6 Industrial and Commercial Bank of China (ICBC) Employs Models to Reconfigure Its Branch Network T h e Industrial an d C o m m ercial B a n k o f China (IC B C ) has m o re th an 1.6,000 b ra n c h e s and serves o v er 2 3 0 m illion individual cu stom ers an d 3 .6 mil­ lion co rp o ra te clien ts. Its daily financial tran saction s total a b o u t $ 1 8 0 m illion. It is also th e largest p u b ­ licly trad ed b a n k in th e w o rld in term s o f m arket capitalization, d ep o sit v o lu m e, and profitability. T o stay com p etitiv e an d in cre a se profitability, IC B C w as faced w ith th e c h a lle n g e to q u ick ly ad ap t to th e fast- p a ce d e c o n o m ic grow th, u rbanization, an d in crease in p ersonal w e a lth o f the C h in ese. C h an ges h a d to b e im plem ented in o v e r 3 0 0 cities w ith high variability in cu stom er b e h a v io r an d financial status. O bv io u sly, th e natu re o f th e ch a lle n g e s in s u ch a h u g e e c o n o m y m ean t th at a larg e-scale op tim ization so lu tio n h ad to b e d e v e lo p e d to lo ca te b ra n c h e s in th e right places, with right serv ices, to serve th e right cu stom ers.

W ith their existing m ethod , ICBC u sed to d ecid e w h ere to o p e n n e w b ran ch es through a scoring m odel in w hich d ifferent variables w ith varying w eight w ere u sed as inputs. S o m e o f th e variables w e re cu stom er flow , n u m ber o f residential hou seh old s, and nu m ber o f com petitors in th e in ten d ed geograp h ic region. This m ethod w a s d eficien t in determ ining th e cu stom er dis­ tribution o f a g eo g rap h ic area. T h e existing m ethod w as also u n a b le to optim ize th e distribution o f b an k bran ch es in th e b ran ch netw ork. W ith supp ort from IBM, a b ran ch reconfigu ration (B R ) to o l w as devel­ op ed . Inputs fo r th e B R system are in th ree parts:

a. G e o g ra p h ic data w ith 8 3 d ifferen t categ o ries b. D e m o g ra p h ic and e c o n o m ic d ata w ith 22 dif­

fe re n t ca te g o rie s c . B ra n c h tran sactio n s a n d p e rfo rm a n ce d ata that

co n s is te d o f m o re th a n 6 0 m illio n tran sactio n re co rd s e a c h day

T h e s e th ree inputs h e lp e d g e n era te a ccu rate cu s­ to m er d istribution fo r e a c h a re a and, h e n c e , h e lp e d th e b a n k o p tim ize its b ra n ch n etw o rk . T h e B R sy stem co n siste d o f a m ark e t p o ten tial calcu latio n m o d el, a b ra n ch n e tw o rk o p tim ization m o d el, a n d a b ra n ch site evalu ation m o d e l. In th e m ark e t p o ten tial m odel, th e cu sto m e r v o lu m e an d v a lu e is m ea su red b a se d o n input data a n d e x p e rt k n o w le d g e. F o r in stan ce, e x p e rt k n o w le d g e w o u ld h e lp d eterm in e if p e r­ so n al in co m e sh ou ld b e w e ig h te d m o re th an g ro ss d o m e stic p ro d u ct (G D P ). T h e g e o g ra p h ic are a s are also d em arcated into cells, and th e p re fe re n c e o f o n e ce ll o v e r th e o th e r is d eterm ined . In th e b ra n ch n e t­ w o rk op tim ization m o d el, m ix e d in te g er p rogram ­ m ing is u sed to lo c a te b ra n ch e s in can d id ate cells so that th e y co v e r th e largest m arket p o ten tial areas. In th e b ra n ch site e v alu atio n m o d el, th e value fo r estab lish in g b a n k b ra n c h e s at sp e cific lo ca tio n s is d eterm ined .

S in c e 2 0 0 6 , th e d e v e lo p m e n t o f th e B R has b e e n im p ro v e d th r o u g h a n itera tiv e p r o c e s s . IC B C ’s b r a n c h re c o n fig u r a tio n to o l h a s in c r e a s e d d e p o s its b y $ 2 1 .2 b illio n s in c e its in c e p tio n . T h is in c re a s e in d e p o s it is b e c a u s e th e b a n k c a n n o w re a c h m o re c u s to m e rs w ith th e rig h t s e r v ic e s b y u s e o f its o p tim iz a tio n to o l. In a s p e c ific e x a m p le , w h e n B R w a s im p le m e n te d in S u z h o u in 2 0 1 0 , d e p o s its in c re a s e d to $ 1 3 -6 7 b illio n fro m an in itial le v e l o f $ 7 .5 6 b illio n in 2 0 0 7 . H e n c e , th e B R to o l a s s is te d in a n in c r e a s e o f d e p o s its to th e tu n e o f $ 6 .1 1 b illio n b e tw e e n 2 0 0 7 a n d 2 0 1 0 . T h is p r o je c t w a s s e le c te d a s a fin a lis t in th e E d e lm a n C o m p e titio n 2 0 1 1 , w h ic h is ru n b y IN FO R M S t o p r o m o te actu al a p p lic a tio n s o f m a n a g e m e n t s c ie n c e / o p e ra tio n s re s e a r c h m o d e ls.

0C on tin u ed )

Part I • D ec isio n M aking and Analytics: An O verview

Application Case 1.6 (Continued) Q u e s t i o n s f o r D i s c u s s i o n

1. H o w c a n an aly tical te c h n iq u e s h e lp o rg an iza­ tio n s to re tain co m p etitiv e advantage?

2. H o w c a n d escrip tiv e a n d p red ictiv e analytics h e lp in p u rsu in g p rescrip tiv e analytics?

3. W h a t k in d s o f p rescrip tive an aly tic te ch n iq u e s a re e m p lo y e d in th e c a s e saidy?

4. A re th e p re scrip tiv e m o d e ls o n c e built g o o d forever?

W h a t W e C a n L e a r n f r o m T h is A p p lic a tio n C a se

M any o r g a n iz a tio n s in th e w o rld a re n o w e m b r a c ­ in g a n a ly tic a l te c h n iq u e s to s ta y c o m p e titiv e a n d a c h ie v e g ro w th . M any o rg a n iz a tio n s p ro v id e

c o n s u ltin g s o lu tio n s to th e b u s in e s s e s in e m p lo y ­ in g p re s c rip tiv e a n a ly tica l s o lu tio n s . It is e q u a lly im p o rta n t to h a v e p ro a c tiv e d e c is io n m a k e rs in th e o rg a n iz a tio n s w h o a re a w a re o f th e c h a n g in g e c o ­ n o m ic e n v ir o n m e n t as w e ll as th e a d v a n c e m e n ts in th e fie ld o f a n a ly tic s t o e n s u re th a t a p p ro p ria te m o d e ls a re e m p lo y e d . T h is c a s e s h o w s a n e x a m p le o f g e o g r a p h ic m a rk e t s e g m e n ta tio n a n d c u s to m e r b e h a v io ra l s e g m e n ta tio n te c h n iq u e s to is o la te th e p ro fita b ility o f c u s to m e rs a n d e m p lo y o p tim iz a tio n te c h n iq u e s to lo c a t e th e b r a n c h e s th a t d e liv e r high p ro fita b ility in e a c h g e o g r a p h ic s e g m e n t.

Source: X. Wang et al., “Branch Reconfiguration Practice Through Operations Research in Industrial and Commercial Bank of China,” Interfaces, January/February 2012, Vol. 42, No. 1, pp. 33-44: DOI: 10.1287/inte.1110.0614.

A nalytics Applied to D ifferent Domains A p p lication s o f an alytics in vario u s industry se cto rs h a v e sp a w n ed m an y re la ted a re a s o r at le a s t b uzzw ord s. It is alm o st fa s h io n a b le to a tta ch th e w o rd an a ly tics to any sp e cific indu stry o r typ e o f data. B e s id e s th e g e n e ra l c a te g o ry o f te x t analytics— aim ed at getting v alu e o u t o f te x t (to b e stu d ied in C h ap ter 6 )— o r W e b analytics— analyzing W e b data stream s (C h ap ter 7 )— m an y industry- o r p ro b le m -s p e c ific an alytics professions/stream s have c o m e u p . E x a m p le s o f s u ch are a s are m arketin g analytics, retail analytics, frau d ana­ lytics, tran sp ortation analytics, h e a lth analytics, sp o rts analytics, talen t analytics, b e h a v ­ ioral analytics, an d s o fo rth. F o r e x a m p le , A p p licatio n C ase 1.1 co u ld a lso b e term ed as a c a s e stud y in airlin e analytics. A p p lication C ases 1 .2 an d 1.3 w o u ld b e lo n g to h ealth analytics; A p p lication C ases 1 .4 an d 1.5 to sports an aly tics; A p p lication C ase 1 .6 to b an k analytics; a n d A p p lication C ase 1.7 to retail analy tics. T h e E n d -o f-C h ap ter A p p lication C ase co u ld b e term ed in su ra n ce analytics. Literally, a n y sy stem atic an alysis o f d ata in a s p e cific s e c to r is b e in g la b e le d as “(fill-in -b la n k s)” A nalytics. A lthough this m ay result in o v ersellin g th e c o n c e p ts o f a nalytics, th e b e n e fit is th a t m o re p e o p le in s p e c ific industries are aw are o f th e p o w e r and p o ten tial o f analytics. It a lso p ro v id e s a fo cu s to p ro fe ssio n als d ev elo p in g a n d ap p ly in g th e c o n c e p ts o f an alytics in a vertical s ecto r. A lthough m an y o f the te ch n iq u e s to d ev elo p a n alytics ap p lica tio n s m ay b e co m m o n , th e re a re u n iq u e issu es w ith in e a c h v e rtical s e g m e n t th at in flu e n ce h o w th e data m ay b e c o lle c te d , p ro cesse d , an aly zed , a n d th e ap p licatio n s im p lem en ted . T h u s, th e d ifferen tiatio n o f an alytics b a se d o n a vertical fo c u s is g o o d fo r th e ov erall gro w th o f th e d iscip line.

A nalytics or Data Science? E v e n as th e c o n c e p t o f analytics is g ettin g p o p u la r am o n g industry an d a c a d e m ic circles, an o th e r term h a s alread y b e e n in tro d u ced an d is b e c o m in g p o p u lar. T h e n e w term is d a ta scien ce. T h u s th e p ractition ers o f d ata s c ie n c e are d ata scien tists. Mr. D . J . Patil o f L in ked ln is s o m e tim e s cred ited w ith cre a tin g th e te rm d a ta sc ien c e. T h e r e h av e b e e n s o m e attem pts to d e s c rib e th e d iffe re n ce s b e tw e e n d ata analysts a n d data scien tists (e .g ., s e e this stud y at em c.com /collateral/ab ou t/new s/em c-d ata-scien ce-stu d y-w p.p d f). O n e v iew is that

Chapter 1 • An O verview o f B u sin ess Intelligence, Analytics, and D ecision Support 57

ia ia an aly st is ju s t a n o th e r term fo r p ro fe ssio n a ls w h o w e re d o in g b u sin e ss in te llig e n ce in iie form o f data co m p ila tio n , cle a n in g , rep ortin g, and p e rh a p s s o m e v isu alization . T h e ir s i l l sets in clu d e d E x c e l, s o m e SQ L k n o w le d g e , and rep ortin g. A re a d e r o f S e c tio n 1.8 srould re co g n iz e th a t a s d escrip tive o r rep ortin g analy tics. In con trast, a d ata s cie n tist is

resp o n sib le fo r p re d ictiv e analysis, statistical analy sis, and m o re a d v a n ce d an aly tical to o ls and algorithm s. T h e y m ay h av e a d e e p e r k n o w le d g e o f algorithm s and m ay re co g n iz e *b em u n d e r v ario u s la b e ls — d ata m ining, k n o w le d g e d isco v er)7, m a ch in e learn in g , and

forth. S o m e o f th e s e p ro fe ssio n a ls m ay also n e e d d e e p e r p ro g ram m in g k n o w le d g e to be a b le to w rite c o d e fo r d ata c le a n in g an d analysis in cu rre n t W e b -o rie n te d lan g u ag es 5-ich as Ja v a a n d P y th on . Again, o u r re a d ers sh o u ld re co g n iz e th e s e as fallin g u n d er the pred ictive a n d p rescrip tiv e an aly tics u m brella. O u r v ie w is that th e d istin ction b e tw e e n analytics an d d ata s c ie n c e is m o re o f a d e g re e o f te ch n ica l k n o w le d g e an d skill sets than d ie fu n ction s. It m ay a lso b e m o re o f a d istin ction a cro ss d iscip lin es. C o m p u te r s c ie n c e , statistics, a n d a p p lie d m ath em atics p rogram s a p p e a r to p re fe r th e data s c ie n c e lab el, reserving th e an aly tics la b e l fo r m o re b u s in ess-o rie n te d p ro fessio n als. As a n o th e r e x a m p le

this, ap p lie d p h y s ics p ro fe ssio n a ls h a v e p ro p o s e d u sin g n etw ork sc ien c e a s th e term for d escrib in g an alytics that relate to a g ro u p o f p e o p le — so cia l n e tw o rk s, su p p ly ch ain n etw o rks, an d s o forth. S e e barabasilab .n eu .ed u/n etw ork scien ceb ook /d ow n lPD F . fcdnl fo r a n ev o lv in g te x tb o o k o n this to p ic.

Aside fro m a clea r d ifferen ce in th e skill sets o f p ro fession als w h o o n ly h av e to d o iescriptive/reporting analytics versus th o se w h o e n g ag e in all three types o f analytics, the distinction is fu zzy b e tw e e n th e tw o lab els, at b est. W e o b serv e that grad uates o f ou r analytics program s ten d to b e re sp o n sib le fo r tasks m o re in line w ith data s c ie n c e p ro fes­ sionals (a s d efin ed b y so m e circles) than ju st reporting analytics. T h is b o o k is clearly aim ed i : introducing th e capabilities an d fu nctionality o f all analytics (w h ich in clu d es data sci­ e n ce), n o t ju st rep ortin g analytics. From n o w o n , w e will u se th ese term s interchangeably.

SECTION 1 .8 REVIEW QUESTIONS

1 . D efin e an aly tics. 2 . W hat is d escriptive analytics? W h at various to o ls are em p loyed in descriptive analytics?

3. H o w is d escrip tiv e an alytics d ifferen t fro m trad itional reporting? 4 . W h a t is a d a ta w a r e h o u s e ? H o w c a n d ata w a r e h o u s in g te c h n o lo g y h e lp in e n a ­

b lin g a n aly tics?

5. W h at is p re d ictiv e analytics? H o w c a n o rg an ization s e m p lo y p red ictiv e analytics? 6. W h a t is p re s c rip tiv e an aly tics? W h a t k in d s o f p r o b le m s c a n b e so lv e d b y p re s c rip ­

tive analytics?

7. D e fin e m o d e lin g fro m th e an alytics p e rsp ectiv e. 8 . Is it a g o o d id e a to fo llo w a h iera rch y o f d escrip tiv e an d p re d ictiv e an aly tics b e fo re

ap p lyin g p rescrip tiv e analytics? 9. H o w c a n an aly tics aid in o b je c tiv e d e c is io n m aking?

1.9 B R IE F IN TRO D U CTIO N TO B IG D A T A A N A LY T IC S W hat Is Big D ata? O u r b ra in s w o r k e x tr e m e ly q u ic k ly a n d a re e ffic ie n t a n d v e rsa tile in p r o c e s s in g la rg e am o u n ts o f all k in d s o f d ata: im a g es, te x t, so u n d s, sm e lls, a n d v id eo . W e p r o c e s s all d ifferen t fo rm s o f d a ta re la tiv ely easily . C o m p u ters, o n th e o th e r h a n d , a re still fin d in g it hard to k e e p u p w ith th e p a c e at w h ic h d ata is g e n e r a te d — le t a lo n e a n a ly z e it q u ick ly . W e h av e th e p r o b le m o f B ig D ata. S o w h a t is B ig D ata? S im p ly pu t, it is d ata th a t c a n n o t

M aking and Analytics: An Overview

b e s to r e d in a sin g le s to ra g e u n it. B ig D ata ty p ically re fe rs to d ata th a t is arrivin g in m a n y d iffe re n t fo rm s, b e th e y stru ctu red , u n stru ctu re d , o r in a stream . M ajor s o u rc e s o f s u ch d ata a re clic k s tre a m s fro m W e b sites, p o s tin g s o n s o c ia l m e d ia site s s u ch as F a c e b o o k , o r d ata fro m traffic, s e n s o rs , o r w e a th e r. A W e b s e a r c h e n g in e lik e G o o g le n e e d s to s e a r c h a n d in d e x b illio n s o f W e b p a g e s in o r d e r to g iv e y o u re le v a n t s e a r c h re su lts in a fra c tio n o f a s e c o n d . A lth o u g h this is n o t d o n e in re a l tim e , g e n e ra tin g an in d e x o f all th e W e b p a g e s o n th e In te rn e t is n o t a n e a s y task . L u ckily fo r G o o g le , it w a s a b le to s o lv e th is p ro b le m . A m o n g o th e r to o ls , it h a s e m p lo y e d B ig D ata a n aly tical te c h n iq u e s .

T h e r e are tw o a sp e cts to m an ag in g d ata o n this s c a le : sto rin g a n d p ro cessin g . I f w e co u ld p u rch a s e a n e x tre m e ly e x p e n s iv e s to ra g e s o lu tio n to sto re all th e data a t o n e p la ce o n o n e u n it, m ak in g this u n it fau lt to le ra n t w o u ld in v o lv e m a jo r e x p e n s e . An in g en iou s s o lu tio n w a s p ro p o s e d that involved sto rin g this data in ch u n k s o n d ifferen t m a ch in e s c o n n e c te d b y a n e tw o rk , pu tting a c o p y o r tw o o f th is ch u n k in d ifferen t lo ca tio n s o n th e n e tw o rk , b o th lo g ically an d p hy sically. It w as o rig in ally u se d at G o o g le (th e n calle d G oogle F ile System) and la te r d e v e lo p e d a n d re le a s e d as a n A p a ch e p ro je c t as th e H ad oo p D istributed File System (H D FS).

H o w ev er, sto rin g this d ata is o n ly h a lf the p ro b le m . D ata is w o rth le ss i f it d o e s n o t p ro v id e b u sin ess v alu e , an d fo r it to pro v id e b u s in e s s v alu e, it h a s to b e analyzed. H o w are s u ch vast am o u n ts o f data analyzed? P assin g all co m p u tatio n to o n e p o w erfu l co m p u te r d o e s n o t w o rk ; this s c a le w o u ld cre a te a h u g e o v e rh e a d o n s u ch a p o w e r­ fu l co m p u ter. A n o th er in g en io u s so lu tio n w a s p ro p o se d : P u sh co m p u ta tio n to th e data, in stead o f p u sh in g d ata to a co m p u tin g n o d e . T h is w a s a n e w parad igm , a n d it g a v e rise to a w h o le n e w w a y o f p ro c e s s in g d ata. T h is is w h at w e k n o w to d ay as th e M ap R ed u ce p ro g ram m in g parad igm , w h ich m ad e p ro ce ssin g B ig D a ta a reality. M ap R ed u ce w a s origi­ nally d e v e lo p e d at G o o g le , an d a s u b s e q u e n t v e rsio n w a s re le a s e d b y th e A p a ch e p ro je ct c a lle d H ad o o p M ap R ed uce.

T o d ay , w h e n w e talk a b o u t storing, p ro cessin g , o r analyzing B ig D ata, H D FS and M ap R ed u ce a re in v o lv ed at s o m e lev el. O th e r relev an t stand ard s an d so ftw are so lu tion s h av e b e e n p ro p o se d . A lthough th e m a jo r to o lk it is av a ila b le as o p e n so u rce , sev eral c o m p a n ie s h av e b e e n la u n ch e d to pro v id e train in g o r s p e cia liz e d analytical hard w are o r so ftw are s erv ices in this s p a c e . S o m e e x a m p le s are H orton W o rk s, C lou d era, a n d Terad ata Aster.

O v e r th e p a s t fe w y e a rs , w h a t w a s c a lle d B ig D a ta c h a n g e d m o re an d m o re as B ig D ata a p p lic a tio n s a p p e a re d . T h e n e e d to p r o c e s s data c o m in g in at a rap id rate ad d e d v e lo c ity to th e e q u a tio n . O n e e x a m p le o f fa st d ata p r o c e s s in g is a lg o rith m ic trad ing. It is th e u s e o f e le c tr o n ic p latfo rm s b a s e d o n a lg o rith m s fo r trad in g s h a re s o n th e fin an cial m ark et, w h ic h o p e ra te s in th e o r d e r o f m ic ro s e c o n d s . T h e n e e d to p r o c e s s d iffe re n t k in d s o f d ata a d d e d v arie ty to th e e q u a tio n . A n o th e r e x a m p le o f th e w id e v arie ty o f d ata is s e n tim e n t analy sis, w h ic h u s e s v ario u s fo rm s o f d a ta fro m s o c ia l m ed ia p latfo rm s a n d c u s to m e r re s p o n s e s to g a u g e s en tim en ts. T o d a y B ig D a ta is a s s o c ia te d w ith a lm o st a n y k in d o f la rg e d ata th a t h a s th e c h a ra c te ris tic s o f v o lu m e , v e lo city , a n d variety. A p p lica tio n C ase 1 .7 illu strates o n e e x a m p le o f B ig D a ta a n aly tics. W e w ill stu d y B ig D ata c h a ra c te ris tic s in m o re d etail in C h ap ters 3 a n d 13.

SECTION 1 .9 REVIEW QUESTIONS

1 . W h at is B ig D ata analytics?

2 . W h at are th e so u rce s o f B ig Data?

3 . W h at are th e ch aracteristics o f B ig Data? 4 . W h at p ro ce s s in g te c h n iq u e is ap p lie d to p ro c e s s B i ta?

C hapter 1 * An Overview o f B u sin ess Intelligence, Analytics, and D ecisio n Support 59

Application Case 1.7 G ilt Groupe's Flash Sales Streamlined by Big Data Analytics Gilt G ro u p e is a n o n lin e d estination offering flash sjdes for m ajor brand s b y selling their cloth in g and accessories. It o ffers its m em b ers ex clu siv e discounts on high-end clo th in g an d o th er apparel. After regis- lering w ith Gilt, cu sto m ers are s e n t e-m ails containing i variety o f offers. Custom ers are given a 3 6 -4 8 hou r w ind ow to m ake p u rch ases using th e se offers. T h e re are a b o u t 30 d ifferent sales e a c h day. W h ile a typical d ep artm en t store turns o v er its inventory tw o o r three rimes a year, Gilt d o e s it eigh t to 10 tim es a year. Thus, th ey hav e to m a n a g e th eir inventory extrem ely w ell : r th ey cou ld in cu r extrem ely h ig h inventory costs. In ord er to d o this, analytics softw are d ev elo p e d at Gilt k e e p s track o f every cu sto m e r click — ranging from w h at brand s th e cu stom ers click on , w h at colors ih e y c h o o s e , w h a t styles th e y p ick, an d w h at they en d up buying. T h e n Gilt tries to p red ict w h at th ese cu stom ers are m o re likely to b u y and stocks inven- :o iy accord in g to th e se pred iction s. Custom ers are <ent cu stom ized alerts to sa le offers d ep en d in g o n th e rjg g e s tio n s b y th e analytics softw are.

T h at, h o w e v e r, is n o t th e w h o le p ro ce ss. T h e softw are also m o n ito rs w h at offers th e cu stom ers c h o o s e fro m th e re co m m e n d e d offers to m ak e m o re accu rate p re d ictio n s an d to in cre a se th e e ffectiv en ess o f its p e rso n a liz e d reco m m en d atio n s. S o m e cu sto m ­ ers d o n o t c h e c k e-m ail that often . G ilt’s analytics

softw are k e e p s track o f re s p o n s e s to offers an d sen ds th e sa m e o ffe r 3 days la ter to th o se cu stom ers w h o h a v en ’t resp on d ed . G ilt a lso k e e p s track o f w h at cu sto m ers are saying in g e n eral a b o u t Gilt’s p ro d ­ u cts b y analyzing T w itte r fe e d s to an alyze sen tim ent. Gilt’s re co m m en d atio n so ftw are is b a se d o n T erad ata Aster’s te ch n o lo g y so lu tio n that in clu d es B ig D ata analytics te ch n o lo g ie s.

Q u e s t i o n s f o r D i s c u s s i o n

1. W h a t m a k e s this c a s e study a n e x a m p le o f B ig D ata analytics?

2. W h a t ty p e s o f d e c is io n s d o e s G ilt G ro u p e h av e to m ake?

W h a t W e C a n L e a m F r o m t h i s A p p lic a tio n C a se

T h e re is co n tin u ou s g ro w th in th e am o u n t o f struc­ tu red an d unstructured data, and m any organiza­ tio n s are n o w tapping th e s e data to m ak e action ab le d ecisio n s. B ig D ata analytics is n o w e n a b led b y the ad van cem en ts in te ch n o lo g ie s that aid in storag e and p ro cessin g o f vast am o u n ts o f rapidly grow ing data.

Source-. Asterdata.com, “Gilt Groupe Speaks on Digital Marketing Optimization," asterdata.com/gilt_groupe_video.php (accessed February 2013).

1.10 PLAN OF THE BOOK T h e previou s s e c tio n s h a v e g iv e n y o u a n u n d erstan d in g o f th e n e e d fo r u sin g in form a­ n t te c h n o lo g y in d e c is io n m ak in g ; a n IT -o rie n te d v ie w o f vario u s ty p e s o f d e c is io n s ; i o d th e e v o lu tio n o f d e c is io n su p p o rt sy stem s in to b u s in e s s in te llig e n c e , a n d n o w in to m a ly tics. In th e la st tw o s e c tio n s w e h a v e s e e n a n o v erv iew o f v ario u s ty p e s o f an aly t­ i c an d th e ir a p p lica tio n s. N ow w e a re re ad y fo r a m o re d eta iled m a n a g e ria l e x c u rs io n e e o th e se to p ic s , a lo n g w ith s o m e p o ten tia lly d e e p h a n d s -o n e x p e r ie n c e in s o m e o f the Technical to p ics. T h e 1 4 c h a p te rs o f this b o o k are o rg an ize d in to fiv e parts, as s h o w n in

fig u r e 1.6.

=art l: Business A nalytics: An O verview fc. Chapter 1, w e p ro v id e d a n in trod u ctio n , d efin ition s, a n d a n o v erv iew o f d e c is io n sup- r c t t sy stem s, b u s in e s s in te llig e n ce , an d analytics, in clu d in g B ig D ata analytics. C h ap te r 2

rvers th e b a sic p h a s e s o f th e d ecisio n -m a k in g p ro c e s s a n d in trod u ces d e c is io n su p p ort

553cems in m o re d etail.

6 0 Part I • D ec isio n Making and Analytics: An Overview

P a r t I D e cisio n M a k in g a n d A n a ly tic s : A n O verview

C h a p t e r 1 An Overview of Business

Intelligence, Analytics, and Decision Support

C h a p t e r 2 Foundations and Technologies for

Decision Making

1

P a r t III P a r t IV P re d ictiv e A n a ly t ic s P r e s c r ip t iv e A n a ly t ic s

P a r t II D e s c r ip t iv e A nalytic

C h a p t e r 3 Data Warehousing

C h a p t e r 4 Business Reporting. Visual

Analytics, and Business Performance Management

C h a p t e r 5 Data Mining

C h a p t e r 6 Techniques for Predictive

Modeling

C h a p t e r 7 Text Analytics, Text Mining, and

Sentiment Analysis

C h a p t e r S W eb Analytics, W eb Mining, and

Social Analytics

C h a p t e r 9 Model-Based Decision Making: Optimization and Multi-Criteria

Systems

C h a p t e r 1 0 Modeling and Analysis:

Heuristic Search Methods and Simulation

C h a p t e r 1 1 Automated Decision Systems and

Expert Systems

C h a p t e r 1 2 Knowledge Management and

Collaborative System s

P a r t V B ig D a ta a n d F u t u r e D ire c tio n s

f o r B u s in e s s A n a ly tic s

C h a p t e r 1 3 Big Data and Analytics

C h a p t e r 1 4 Business Analytics: Emerging Trends and Future Impacts

P a r t V I Online S u p p le m e n ts

FIGURE 1.6 Plan of the Book.

Part II: Descriptive Analytics P art II b e g in s w ith a n in trod u ctio n to d ata w a re h o u sin g issu es, ap p licatio n s, and te c h n o lo ­ g ie s in C h ap ter 3. D ata re p re se n t th e fu nd am ental b a c k b o n e o f an y d e c is io n su p p o rt and analytics ap p licatio n . C h ap ter 4 d e s crib e s b u s in e s s rep ortin g, v isu alization te ch n o lo g ie s, a n d ap p licatio n s. It a lso in clu d es a b rie f o v erv iew o f b u sin e ss p e rfo rm a n ce m an ag em en t te ch n iq u e s a n d ap p licatio n s, a to p ic th a t h a s b e e n a k e y part o f traditional B I.

Part III: Predictive A nalytics P art III co m p rise s a large part o f th e b o o k . It b e g in s w ith a n in trod u ctio n to p red ictive analytics a p p lica tio n s in C h ap ter 5- It in clu d es m a n y o f th e co m m o n a p p lica tio n te ch ­ n iq u es: cla ssificatio n , clu sterin g, a s s o cia tio n m ining , a n d so fo rth. C h ap ter 6 in clu d es a te ch n ica l d escrip tio n o f s e le c te d d ata m in in g te ch n iq u e s , e sp e cia lly n eu ral n e tw o rk m o d ­ e ls. C h ap ter 7 fo c u s e s o n te x t m in in g a p p lica tio n s. Sim ilarly, C h ap ter 8 fo c u s e s o n W e b analytics, in clu d in g so cia l m ed ia analytics, s en tim en t analysis, a n d o th e r re la ted to p ics.

Chapter 1 * An Overview o f B u sin ess Intelligence, Analytics, and D ec isio n Support 61

Part IV: Prescrip tive Analytics Part IV in tro d u ces d e c is io n an alytic te ch n iq u e s, w h ic h a re a lso calle d p rescrip tiv e analyt­ ics. S p ecifically , C h ap te r 9 co v ers s e le c te d m o d els that m ay b e im p lem e n te d in sp read ­ sh e et e n v iro n m en ts. It a lso c o v e rs a p o p u la r m u lti-o b jectiv e d e cisio n te c h n iq u e analytic

hierarch y p ro ce s s e s. C h ap ter 10 th e n in tro d u ces o th er m o d e l-b a s e d d ecisio n -m a k in g te c h n iq u e s , e sp e ­

cially heu ristic m o d e ls an d sim ulation. C h ap ter 11 in tro d u ces au to m ate d d e c is io n system s inclu ding e x p e rt system s. T h is p a n c o n c lu d e s w ith a b r ie f d iscu ssion o f k n o w le d g e m a n a g e m e n t a n d g ro u p su p p o rt system s in C h ap ter 12.

Part V: Big Data and Future Directions fo r Business A nalytics Part V b e g in s w ith a m o re d eta iled co v e ra g e o f B ig D ata and analytics in C h ap te r 13-

C h a p te r 1 4 a tte m p ts to in te g ra te all th e m ate rial c o v e r e d in th is b o o k and c o n c lu d e s w ith a d is c u s s io n o f e m e rg in g tre n d s, s u c h a s h o w th e u b iq u ity o f w ir e ­ le s s an d G P S d e v ic e s an d o th e r s e n s o r s is re s u ltin g in th e c r e a tio n o f m a ssiv e n e w d a ta b a s e s a n d u n iq u e a p p lic a tio n s . A n e w b r e e d o f d ata m in in g an d B I c o m p a n ie s is e m e rg in g to a n a ly z e th e s e n e w d a ta b a s e s a n d c r e a te a m u c h b e tte r a n d d e e p e r u n d e r­ stan d in g o f c u s to m e r s ’ b e h a v io rs a n d m o v e m e n ts . T h e c h a p te r a ls o c o v e r s c lo u d -b a s e d an a ly tics, r e c o m m e n d a tio n sy stem s, a n d a b r ie f d is c u ss io n o f secu rity / p riv acy d im e n ­ sio n s o f a n a ly tic s . It c o n c lu d e s th e b o o k b y a ls o p re s e n tin g a d is c u s s io n o f th e a n a ly tics e c o s y s te m . A n u n d e rs ta n d in g o f th e e c o s y s te m a n d th e v a rio u s p la y e rs in th e a n a ly tics in d u stry h ig h lig h ts th e v a rio u s c a r e e r o p p o rtu n itie s fo r stu d e n ts an d p ra c titio n e rs o f

an aly tics.

1.11 RESO U RCES, LIN K S, A N D THE T ER A D A T A U N IVERSITY NETW ORK CONNECTION

T h e u s e o f this c h a p te r an d m o st o th er ch ap ters in this b o o k c a n b e e n h a n c e d b y th e to o ls d e s crib e d in th e fo llo w in g sectio n s.

Resources and Links W e re co m m e n d th e fo llo w in g m a jo r re so u rce s an d links:

• T h e D ata W a re h o u sin g Institute (tdw i.org) • In fo rm a tio n M an ag em en t (rnform ation-m anagem ent.com ) • D SS R e so u rc e s (d ssresou rces.co m ) • M icroso ft E n terp rise C o n sortiu m ( e n t e r p r i s e . w a l t o n c o l l e g e . u a r k . e d u / m e c . a s p )

Vendors, Products, and Demos M ost v e n d o rs p ro v id e so ftw are d em o s o f th e ir p ro d u cts an d ap p licatio n s. In fo rm ation a b o u t p ro d u cts, arch ite ctu re, a n d so ftw are is a v a ilab le at d ssresou rces.co m .

Periodicals W e re co m m en d th e fo llo w in g p eriod icals:

• D ecision S u pport System s • CIO In sight (cioinsight.com ) • T echn ology E v alu ation (technologyevaluation.com ) • B a selin e M ag azin e (baselinem ag.com )

6 2 Part I • D ecisio n M aking and Analytics: An Overview

The Teradata U niversity N etw ork Connection T h is b o o k is tightly c o n n e c te d w ith th e fr e e r e s o u r c e s p ro v id e d b y T e ra d a ta U n iv ersity N e tw o rk (T U N ; s e e te ra d a ta u n iv e rsity n e tw o rk .co m ). T h e TU N p o rta l is d ivid ed in to tw o m a jo r p a rts: o n e fo r stu d e n ts a n d o n e fo r fa cu lty . T h is b o o k is c o n n e c te d to th e TU N p o rta l v ia a s p e c ia l s e c tio n a t th e e n d o f e a c h c h a p te r. T h a t s e c tio n in clu d e s a p p r o p ria te lin k s fo r th e s p e c ific c h a p te r, p o in tin g to re le v a n t re s o u r c e s . In ad d ition, w e p ro v id e h a n d s -o n e x e r c is e s , u sin g s o ftw a re a n d o th e r m a te ria l (e .g ., c a s e s ) avail­ a b le a t TUN.

The Book's W eb Site T h is b o o k ’s W e b site, pearson glob aled ition s.com /tu rb an , co n ta in s su p p le m e n tal te x ­ tual m aterial o rg an ize d as W e b ch ap ters th at co r re s p o n d to th e p rin ted b o o k ’s chapters. T h e to p ics o f th e s e ch ap ters are listed in th e o n lin e ch a p te r ta b le o f co n ten ts. O th e r c o n ­ te n t is a lso a v ailab le o n a n in d e p e n d e n t W e b site (d ssbibook.com ).2

Chapter Highlights

• T h e b u s in e s s e n v iro n m en t is b e c o m in g c o m p le x an d is rap id ly ch a n g in g , m ak in g d e c is io n m aking m o re difficult.

• B u s in e s s e s m u st re sp o n d an d ad ap t to th e ch a n g ­ ing e n v iro n m en t rap id ly b y m ak in g faste r an d b e tte r d ecisio n s.

• T h e tim e fram e fo r m ak in g d ecisio n s is shrinkin g, w h e re a s th e g lo b a l n atu re o f d e cisio n m ak in g is e x p a n d in g , n e cessitatin g th e d ev e lo p m e n t an d u s e o f co m p u te rize d DSS.

• C o m p u te rize d su p p o rt fo r m an ag ers is o ften e sse n tia l fo r th e survival o f a n organization.

• An e a rly d e cisio n su p p ort fram ew ork divides d e c is io n situations into n in e categ o ries, d ep en d in g o n th e d e g re e o f stru ctured ness an d m anagerial activities. E a ch c a te g o iy is su p p orted differently.

• Structured repetitive d ecisio n s are su p p orted b y standard quantitative analysis m ethod s, su ch a s MS, MIS, an d ru le-b ased autom ated d ecisio n support.

• D SS u s e data, m o d els, and s o m e tim e s k n o w le d g e m an a g e m e n t to find so lu tio n s fo r sem istm ctu re d a n d s o m e u n stru ctu red p ro b lem s.

• B I m e th o d s utilize a cen tral rep ository calle d a d ata w a re h o u s e th at e n a b le s e fficie n t data m ining, OLAP, BPM , and d ata visualization.

• B I a rch ite ctu re in clu d e s a d ata w a reh o u se , b u si­ n e ss an aly tics to o ls u s e d b y e n d u sers, a n d a user in te rface (s u c h as a d ash b o ard ).

• M any o rg an izatio n s e m p lo y d escrip tiv e analytics to re p la c e th e ir traditional flat rep ortin g w ith inter­ activ e re p o rtin g that p ro v id es insights, trend s, and p attern s in th e tran saction al data.

• P red ictiv e an aly tics e n a b le o rg an ization s to estab ­ lish p re d ictiv e ru les th at drive th e b u sin ess ou t­ c o m e s th ro u g h h istorical data analysis o f the existin g b e h a v io r o f th e cu stom ers.

• P rescrip tive an alytics h e lp in b u ild in g m o d e ls that involve fo re ca s tin g a n d o p tim ization te ch n iq u es b a s e d o n th e p rin cip le s o f o p era tio n s research an d m an a g e m e n t s c ie n c e to h e lp o rg an ization s to m a k e b e tte r d ecisio n s.

• B ig D ata an aly tics fo c u s e s o n un structured , large d ata sets th a t m ay a lso in clu d e vastly different typ es o f d ata fo r analysis.

• A nalytics as a field is a lso k n o w n b y industry- s p e cific a p p lica tio n n a m e s s u ch as sp orts analytics. It is a lso k n o w n b y o th e r related n a m e s s u ch as d ata s c ie n c e o r n e tw o rk s cie n ce .

1 As this book went to press, we verified that all the cited Web sites were active and valid. However, URLs are dynamic. Web sites to which we refer in the text sometimes change or are discontinued because companies change names, are bought or sold, merge, or fail. Sometimes Web sites are down for maintenance, repair, or redesign. Many organizations have dropped the initial “www” designation for their sites, but some still use it. If you have a problem connecting to a Web site that we mention, please be patient and simply run a Web search to try to identify the possible new site. Most times, you can quickly find the new site through one of the popular search engines. We apologize in advance for this inconvenience.

Chapter 1 • An O verview o f B u sin ess Intelligen ce, Analytics, and D ecisio n Support 6 3

Key Terms b u sin ess in te llig e n ce

( B l ) d ash bo ard d ata m in in g

d e c is io n (o r n orm ativ e) analytics

d e c is io n su p p o rt system (D S S )

d escrip tiv e ( o r rep ortin g) analytics

p red ictiv e an alytics prescrip tiv e an alytics

sem istru ctu red p ro b le m

stru ctu red p ro b le m unstructured p ro b lem

Questions for Discussion 1 . D istinguish betw een strategic and tactical planning? 2 . W hat is data mining and why is it classified under predictive

analytics? Search the W eb for an exam ple o f data mining in an organization o f you r ch o ice and illustrate th e w ay it is currently in use.

3 . Prescriptive analytics is considered to b e a step further ahead o f predictive analysis and substantially different

from it. Provide an exam ple o f ea ch and outline their differences.

4 . Provide a definition o f B l. 5 . D efine m anagerial decision making. Discuss this con cep t in

the context o f the four-step approach to d ecision making.

Exercises T e ra d a ta U n iv e rs ity N e tw o r k (T U N ) a n d O th e r H an d s-O n E x e r c i s e s 1. G o to te ra d a ta u n iv e r s ity n e tw o r k .c o m . Using the reg­

istration you r instructor provides, log o n and learn the con tent o f th e site. Y o u will receiv e assignm ents related to this site. Prepare a list o f 2 0 item s in th e site that you think cou ld b e b en eficial to you.

2 . Enter th e TUN site and sele c t “cases, projects and assign­ m en ts.” T h e n select the c a se study: “Harrah’s High Payoff from C ustom er Inform ation.” Answ er th e follow ing ques­ tions ab o u t this case: a . W hat inform ation d oes th e data m ining generate? b . H ow is this inform ation helpful to m anagem ent in

d ecision making? (B e specific.) c . List the types o f data that are mined. d . Is this a DSS o r B l application? Why?

? . G o to te ra d a ta u n iv e rsity n e tw o rk .c o m and find the paper titled “Data Warehousing Supports Corporate Strategy at First American Corporation” (by Watson, W ixom, and Goodhue). Read th e paper and answer the following questions: a . W hat w ere the drivers for the DW/BI p roject in the

com pany? b . W hat strategic advantages w ere realized? c . W hat operational and tactical advantages w ere achieved? d . W hat w ere the critical su ccess factors (CSF) for the

im plementation? -i, G o to a n a ly tic s-m a g a z in e .o rg / is su e s/ d ig ita l-e d itio n s

and find the January/February 2012 edition titled “Special Issue: T h e Future o f Healthcare.” Read the article “Predictive

Analytics— Saving Lives and Lowering Medical Bills.” Answer th e following questions: a . W hat is th e problem that is b ein g addressed b y apply­

ing predictive analytics? b . W hat is the FICO M edication A dh erence Score? c . H ow is a prediction m od el trained to predict the FICO

M edication A dherence Score? D id the prediction m od el classify FICO M edication A dh erence Score?

d. Z oom in o n Figure 4 and exp lain w hat kind o f tech­ niqu e is applied o n th e generated results.

e . List so m e o f th e action able d ecisions that w ere based o n th e results o f th e predictions.

5 . Visit h ttp ://w w w .ib m .c o m /a n a ly tic s /u s /e n /w h a t-is - s m a rte r -a n a ly tics /b ig -d a ta -a n a ly sis .h tm l. Read the s e c ­ tion “Gain actionable insights from big data analysis,” and w atch th e video “See how analytics can help organizations in all industries use big data to achieve significant outcom es” under Case Studies to answer the following questions: a . E xplain b ig d a t a and its im portance in d ecision m ak­

ing w ith exam ples. b . A ppraise the m axim “w ithout analytics, b ig data is just

n o ise .” c . In w h ic h sectors and areas might big data analytics b e

m ost useful? G ive exam ples. d . W hat are the suggested solutions to m anaging risks? e . Review and analyze a c a se study from IBM ’s W eb site

and exp lain h ow b ig data analytics provided solutions. 6 . B usiness analytics and com puterized data processing

support m anagers and d ecision m aking. K eep in g current

6 4 Part I • D ecisio n Making and Analytics: An O verview

bu sin ess environm en t challeng es in mind, along with M intzberg’s 10 m anagerial roles ( s e e T a b l e 1 .2 ), identify five su ch roles in com panies/governm ent’s press release and com m unications.

7 - G o to oracle.co m , a leading com p an y in B I. M ake a map o f their W eb site illustrating their products and com m uni­ cation styles w ith available resou rces fo r business.

8 . Search th e W eb fo r a com pany that uses the four m ajor com p on en ts o f a standard B I system.

9 . In the com p an y identified in th e previous question, illus­ trate their m ain prod ucts and style o f B I and discuss the m ain tools used. R efer to th e tools m en tioned in this chapter.

End-of-Chapter Application Case

Nationwide Insurance Used BI to Enhance Customer Service Nationwide Mutual Insurance Company, headquartered in Columbus, O hio, is o n e o f the largest insurance and financial services com panies, with $23 billion in revenues and m ore than $160 billion in statutory assets. It offers a com prehensive range o f products through its family o f 100-plus com panies with insurance products fo r auto, m otorcycle, boat, life, hom eow n­ ers, and farms. It also offers financial products and services including annuities, mortgages, mutual funds, pensions, and investment m anagement.

Nationwide strives to achieve greater efficiency in all operations by m anaging its ex p en ses along with its ability to grow its revenue. It recognizes the u se o f its strategic asset o f information com bin ed with analytics to ou tp ace com petitors in strategic and operational decision making even in com plex and unpredictable environments.

Historically, Nationwide’s business units w orked inde­ pendently and with a lot o f autonom y. This led to duplication o f efforts, widely dissimilar data processing environments, and extrem e data redundancy, resulting in higher expen ses. The situation got com plicated w hen Nationwide pursued any merg­ ers o r acquisitions.

Nationwide, using enterprise data w arehouse technology from Teradata, set ou t to create, from the ground up, a single, authoritative environment for clean, consistent, and com plete data that can b e effectively used for best-practice analytics to m ake strategic and tactical business decisions in the areas o f custom er growth, retention, product profitability, cost contain­ m ent, and productivity improvements. Nationwide transformed its siloed business units, which w ere supported by stove-piped data environments, into integrated units by using cutting-edge analytics that w ork with clear, consolidated data from all o f its business units. T h e Teradata data w arehouse at Nationwide has grown from 400 gigabytes to more than 100 terabytes and supports 85 percent o f Nationwide’s business with m ore than 2,500 users.

I n t e g r a t e d C u s to m e r K n o w le d g e N ationw ide’s C u stom er K n ow led g e Store (C K S) initiative d ev elo p ed a cu stom er-cen tric d atabase that integrated cu s­ tom er, prod uct, a n d extern ally acq u ired data from m ore

th an 4 8 sou rces into a sin g le cu stom er data m art to deliver a holistic v iew o f cu stom ers. This data m art w as c o u p led with T erad ata’s cu stom er relatio n sh ip m an agem en t ap p lication to c reate and m an age effe ctiv e cu stom er m arketing cam paigns that u s e behavioral analysis o f cu stom er interactions to drive cu stom er m an agem en t action s (CMAs) for target segm ents. N ationw ide ad ded m o re sop h isticated cu stom er analytics that lo o k e d at cu stom er p ortfolios and th e effectiv en ess o f various m arketing cam p aign s. T h is data analysis help ed N ationw ide to initiate p roactive cu stom er com m u nications around cu stom er lifetim e ev en ts like m arriage, birth o f child, or h o m e p u rch ase a n d had significant im p act o n im prov­ ing cu stom er satisfaction. Also, b y integrating cu stom er co n ta ct history, p rod u ct o w nership , and p ay m ent inform a­ tion , N ationw ide’s beh av ioral analytics team s further created prioritized m odels th a t cou ld identify w h ic h sp ecific cu s­ to m er in teraction w as im p ortan t for a custom er at an y given time. This resulted in o n e p ercen ta g e p oint im provem en t in cu stom er reten tio n rates and sig nificant im provem ent in cu stom er enth u siasm sco res. N ationw ide also ach ieved 3 p ercen t ann ual grow th in increm ental s a le s by using CKS. T h e re are o th er uses o f the cu stom er d atabase. In o n e o f th e initiatives, by integrating cu stom er te le p h o n e data from m ultiple system s into CKS, th e relation ship m an agers at N ationw ide tiy to b e proactives in con tactin g custom ers in ad van ce o f a p ossib le w ea th er catastrop h e, su ch as a hur­ rican e o r flood , to p rov id e th e prim aiy p o licy h o ld er infor­ m ation and exp lain th e claim s p ro c esses. T h e s e and o th er analytic insights n o w drive N ationw ide to provide extrem ely person al cu stom er service.

F in a n c ia l O p e r a tio n s A similar perform ance p ay off from integrated information was also noted in financial operations. Nationwide’s decentralized m anagem ent style resulted in a fragmented financial report­ ing environm ent that included m ore than 14 general ledgers, 20 charts o f accounts, 17 separate data repositories, 12 different reporting tools, and hundreds o f thousands o f spreadsheets. Th ere w as n o com m o n central view o f th e business, w hich resulted in labor-intensive slow and inaccurate reporting.

Chapter 1 * An O verview o f B u sin ess In tellig en ce, Analytics, and D ecision Support 6 5

About 75 p ercen t o f th e effort was spent o n acquiring, clean­ ing, and consolidating and validating the data, and very7 little time was spent o n meaningful analysis o f the data.

T h e Financial P erform an ce M anagem ent initiative im plem ented a n e w operating ap p roach that w o rk ed o n a single data and tech n o log y architecture with a com m on set o f system s standardizing th e p rocess o f reporting. It enabled Nationwide to o p erate analytical cen ters o f e x c e lle n c e with w orld-class planning, capital m anagem ent, risk assessm ent, ttA other d ecisio n support capabilities that delivered timely, accurate, and efficien t accounting, reporting, and analytical services.

T h e data fro m m ore than 2 0 0 o p eratio n al system s was sent to the en terp rise-w id e data w areh ou se and th e n distrib­ uted to various ap p lications and analytics. This resulted in a 5 0 p e rc e n t im p rovem en t in the m onthly closin g p ro cess w ith clo sin g intervals red u ced from 14 days to 7 days.

P o s t m e r g e r D a t a I n t e g r a t i o n N ationw id e’s G o al State R ate M anag em ent initiative e n a ­ bled th e c o m p a n y to m erge A llied In su ra n c e ’s au to m o b ile c o iic v system in to its existin g system . B o th N ationw id e and Allied s o u rc e system s w e re cu stom -bu ilt ap p lication s that ■did not sh a re a n y c o m m o n v alu es o r p ro c e ss data in th e v.rnp m an ner. N ationw id e’s IT d ep artm en t d ecid ed to bring i l l th e data fro m s o u rc e system s in to a cen tralized data « ^ re h o u se , o rg an ized in a n in tegrated fash io n that resulted i a standard d im en sio n a l rep orting and h e lp e d N ationw ide

p erform in g w h a t-if an aly ses. T h e data analysis team could identify p rev iou sly u n k n o w n p o ten tia l d ifferen ces m th e data en v iro n m en t w h e r e prem ium s rates w e re c a l­ culated d ifferently b e tw e e n N ationw id e and A llied sides. Correcting all o f th e s e b e n e fite d N ationw id e’s p o licy h o ld ­ ers b e c a u se th e y w e re safeg u ard ed from e x p e rie n c in g w ide prem ium rate sw in gs.

E n h a n c e d R e p o r t i n g ,*2C«3nwide’s legacy reporting system , w hich catered to the c c e c s o f property and casualty business units, to o k w eek s D com pile and deliver th e n eed ed reports to th e agents. X ero n w id e determ ined that it n eed ed better access to sales n-k-T policy inform ation to reach its sales targets. It ch o se a

single data w areh ou se ap p roach and, after careful assessm ent o f th e need s o f s a le s m anagem ent and individual agents, selected a bu sin ess intelligence platform that w ould integrate d ynam ic enterp rise dashboards into its reporting systems, m aking it easy fo r th e agents and associates to view policy inform ation at a g la n ce. T h e n e w reporting s y s tem , dubbed R evenue C on n ection , also enabled users to analyze th e infor­ m ation w ith a lo t o f interactive and drill-dow n-to-details cap a­ bilities at various lev els that elim inated the n e e d to gen erate custom ad h o c reports. Revenue C on nection virtually elim i­ nated requ ests for manual policy audits, resulting in huge savings in tim e and m on ey for the business and tech nology team s. T h e reports w ere produced in 4 to 45 secon d s, rather than days o r w eek s, and productivity in som e units improved b y 2 0 to 3 0 percent.

Q u e s t i o n s f o r D i s c u s s i o n

1 . W hy did Nationw ide n eed an enterprise-w ide data w arehouse?

2 . H ow did integrated data drive th e business value? 3 . W h at fo r m s o f a n a ly tics a r e em ployed af Nationwide? 4 . W ith integrated data available in an enterprise data

w areh ou se, w hat other applications cou ld Nationwide potentially develop?

W h a t W e C an L e a r n f r o m T h is A p p lic a tio n C a se T h e p ro p er u se o f integrated inform ation in org aniza­ tion s c a n h e lp a c h iev e b e tte r b u sin ess o u tc o m e s . Many org an ization s n o w rely o n data w areh ou sin g tech n o lo g ies to perform th e o n lin e analytical p ro c e ss e s o n th e data to d erive v a lu a b le insig hts. T h e insights a re used to d evelop pred ictive m o d els that further e n a b le th e grow th o f th e org an ization s by m o re p re c is e ly a ssessin g cu stom er needs. In creasin gly , o rg an izatio n s a re m oving tow ard deriving v alu e from an aly tical ap p licatio n s in real tim e w ith the h e lp o f integrated data from real-tim e data w areh ou sin g te ch n o lo g ie s .

Source: Teradata.com, “Nationwide, Delivering an On Your Side Experience," t e r a d a t a . c o m / W o r k A r e a / l i n k i t . a s p x P L i n k I d e n t i f i e r= id & Ite m ID = l4 7 l4 (accessed February 2013).

References Anehony, R. N. (1965). P la n n in g a n d C o n tro l Systems: A

F r a m e w o r k f o r A n alysis. Cambridge, MA: Harvard University Graduate Sch ool o f Business.

Aaerdata.com . “Gilt G roupe Speaks on Digital Marketing Optimization. ” w w w .a s te r d a ta .c o m /g ilt_ g ro u p e _ v id e o . p h p (accessed February 2013).

Barabasilab.n eu .ed u . “Network S cien ce." b a ra b a s ila b .n e u . e d u /n e tw o r k s c ie n c e b o o k /d o w n lP D F .h tm l (a c cessed February 2013).

B rooks, D . (2 0 0 9 , M ay 18). “In Praise o f D u llness.” N ew Y ork T im es, n y t i m e s .c o t n / 2 0 0 9 / 0 5 / 1 9 / o p i n i o n / 1 9 b r o o k s . h tm l (a c cessed February 2013).

Centers for D isease Control and Prevention, V accines for Children Program , “M odule 6 o f the VFC O perations G uide. ” c d c .g o v / v a c c in e s / p u b s / p in k b o o k / v a c -s to r - a g e .h t m l # s to r a g e (a c cessed Jan u ary 2013).

E ck erson, W. (2 0 0 3 ). S m a r t C o m p a n ie s in t h e 2 1 s t C en tury: T h e S e c re ts o f C r e a t in g S u c c e s s fu l B u s in e s s In te llig en t S o lu tion s. Seattle, WA: T h e D ata W arehousing Institute.

E m c.com . “D ata S cie n c e Revealed: A D ata-Driven G lim pse into th e B u rg eon in g N ew Field .” e m c .c o m / c o lla te ra l/ a b o u t/ n e w s / e m c -d a ta -s c ie n c e -s tu d y -w p .p d f (a c cessed

February 2013). Gorry, G . A., and M. S. Scott-M orton. (1 9 7 1 ). “A Fram ew ork

for M anagem ent Inform ation System s.” S lo a n M a n a g e m e n t R ev iew , Vol. 13, No. 1, pp. 5 5 -7 0 .

INFORMS. “Analytics Section O verview .” in fo r m s .o r g / C o m m u n ity / A n a ly tic s (accessed February 2013).

K een, P. G. W ., and M. S. Scott-M orton. (1 9 7 8 ). D e c is io n S u p p o r t System s: A n O r g a n iz a t io n a l P ers p ectiv e. Reading, MA: Addison-W esley.

Krivda, C. D. (20 0 8 , M arch). “D ialing Up G row th in a Mature Market.” T e r a d a t a M a g a z in e , pp. 1-3-

Magpiesensing.com . “MagpieSensing Cold Chain Analyt­ ics and Monitoring.” m a g p ie se n s in g .c o m / w p -c o n te n t/ uploads/2013/01/ColdChainAnalyticsMagpieSensing- W h i t e p a p e r .p d f (accessed Janu ary 2013).

Mintzberg, H. A. (1 9 8 0 ). T h e N a tu r e o f M a n a g e r ia l W ork. E n glew ood Cliffs, NJ: P rentice Hall.

Mintzberg, H. A. (1 9 9 3 ). T h e R ise a n d F a l l o f S trateg ic P la n n in g . N ew Y o rk : T h e Free Press.

Sim on, H. (1 9 7 7 ). T h e N ew S c ie n c e o f M a n a g e m e n t D ecisio n . E n glew ood Cliffs, NJ: Prentice Hall.

6 6 Part I • D ec isio n M aking and Analytics: An O verview

T ableau softw are.com . “Eliminating W aste at Seattle Chil­ dren’s .” t a b le a u s o ftw a r e .c o m / e lim in a tin g -w a s te -a t- s e a t t le - c h il d r e n s (a c ce s se d February 2013).

Tableausoftw are.com . “K aleida Health Finds Efficiencies, Stays Com petitive.” ta b le a u s o ftw a r e .c o m / le a m / s to r ie s / u s e r - e x p e r ie n c e -s p e e d -th o u g h t-k a le id a -h e a lth (accessed

February 2013). T eradata.com . “Nationw ide, Delivering an O n Y ou r Side

E xp erien ce. te r a d a ta .c o m / c a s e -s tu d ie s / d eliv e r in g -o n - y o u r -s id e -e x p e r ie n c e (a c cessed February 2013).

Teradata.com . “Sabre Airline Solu tions.” tera d a ta .eo m / t/ c a s e - s t u d i e s / S a b r e - A i r l i n e - S o l u t i o n s - E B 6 2 8 1

(a c cessed February 2 013). W ang, X ., et al. (20 1 2 , January/February). “B ranch Reconfigu­

ration P ractice T hrou gh O perations R esearch in Industrial and Com m ercial B a n k o f China.” I n t e r fa c e s , Vol. 42, No. 1,

pp. 3 3 -^ 4 . W atson, H. (2005, W inter). “Sorting Out W hat’s New in Decision

Support.” B u sin ess In te llig e n c e Jo u r n a l. W ikipedia. “O n B a se P ercen tag e.” e n .w ik ip e d ia .o rg / w ik i/

O n _ b a s e _ p e r c e n t a g e (a c ce s se d Janu ary 2013). W ikip ed ia. “S a b e rm e tric s.” e n .w ik ip e d ia .o r g / w ik i/

S a b e r m e t r i c s (a c c e s s e d Ja n u a ry 2 0 1 3 ). Zaleski, A. (2 0 1 2 ). “M agpie Analytics System Tracks Cold-

Chain Products to K eep V accin es, Reagents Fresh.” T e c h n i c a l ly B a lt i m o r e .c o m (a c cessed February 2013).

Zaman, M. (2009, April). “Business Intelligence: Its Ins and Outs.’' te c lin o lo g y e v a lu a tio n .c o m (accessed Febniary 2013).

Ziam a, A., and J . K ash er. (2 0 0 4 ). “D ata M ining Prim er for th e Data W a reh o u sin g P ro fe ss io n a l.” D ay to n , OH: T e r a d a t a .

Foundations and Technolog for Decision Making

LEARNING OBJECTIVES

■ U n d erstan d th e c o n c e p tu a l fo u n d atio n s o f d e c is io n m aking

■ U n d erstand S im o n ’s fo u r p h a s e s o f d e c is io n m akin g: in te llig e n ce , d esign, c h o ic e , a n d im p lem en tatio n

■ U n d erstan d th e esse n tial d efinition o f D SS

■ U nderstand im portant D SS classifications

* L earn h o w D SS su p p o rt fo r d e cisio n m ak in g c a n b e p ro v id e d in p ra ctice

* U n d erstan d D SS c o m p o n e n ts and h o w th ey integrate

Ou r m a jo r fo cu s in this b o o k is th e su p p o rt o f d e c is io n m ak in g throu gh c o m p u te r-b a s e d in form ation sy stem s. T h e p u rp o se o f this c h a p te r is to d e scrib e th e c o n c e p tu a l fo u n d atio n s o f d e cisio n m ak in g an d h o w d e c is io n su p p o rt is p rovided. T h is ch a p te r in clu d e s th e fo llo w in g sectio n s:

2 .1 O p e n in g V ig n e tte : D e c is io n M o d e lin g a t H P U s in g S p r e a d s h e e ts 6 8 2 .2 D e c i s io n M ak in g : In tr o d u c tio n a n d D e fin itio n s 7 0 2 .3 P h a s e s o f th e D e c is io n -M a k in g P r o c e s s 7 2 2 . 4 D e c i s io n M ak in g : T h e I n te llig e n c e P h a s e 7 4 2 .5 D e c i s io n M ak in g : T h e D e s ig n P h a s e 7 7 2 .6 D e c i s io n M ak in g : T h e C h o ic e P h a s e 8 5 2 .7 D e c i s io n M ak in g : T h e I m p le m e n ta tio n P h a s e 8 5 2 . 8 H o w D e c is io n s A re S u p p o rte d 8 6 2 . 9 D e c i s io n S u p p o rt S y s te m s : C a p a b ilitie s 8 9

2 .1 0 D S S C la s s ific a tio n s 91 2 .1 1 C o m p o n e n ts o f D e c is io n S u p p o r t S y s te m s 9 4

6 8 Part I • D ec isio n M aking and Analytics: An O verview

2.1 OPENING VIGNETTE: Decision Modeling at HP Using Spreadsheets

HP is a m a jo r m an u factu rer o f co m p u ters, p rinters, a n d m an y industrial p ro d u cts. Its vast p ro d u ct lin e lead s to m an y d e c is io n p ro b lem s. O la v s o n an d Fry ( 2 0 0 8 ) h av e w o rk e d o n m an y s p re a d sh ee t m o d e ls fo r assisting d e c is io n m a k e rs at HP a n d h a v e id en tified sev eral le s s o n s fro m b o th th e ir s u c c e s s e s an d th e ir failu res w h e n it co m e s to co n stru ctin g and ap p ly in g sp re a d sh e e t-b a s e d to o ls. T h e y d efin e a to o l as ‘ a re u sab le , analytical so lu tio n d esig n e d to b e h an d e d o f f to n o n te ch n ica l e n d u s e rs to assist th e m in solv in g a re p e a ted

b u sin e ss p ro b le m .” W h e n trying to s o lv e a p ro b lem , H P d e v e lo p e rs co n s id e r th e th ree p h a se s in d ev el­

o p in g a m o d e l. T h e first p h a se is p ro b le m fram ing, w h e re th e y c o n s id e r th e fo llo w in g q u e stio n s in o rd e r to d e v e lo p th e b e s t so lu tio n fo r th e p ro blem :

• W ill an alytics so lv e th e problem ? • C an a n e x istin g so lu tio n b e leveraged ? • Is a to o l need ed ?

T h e first q u e s tio n is im portant b e c a u s e th e p ro b le m m ay n o t b e o f a n an alytic natu re, a n d th e re fo re , a s p re a d sh e e t to o l m ay n o t b e o f m u c h h e lp in th e lo n g ru n w ith o u t fixing th e n o n an aly tical p art o f th e p ro b le m first. F o r e x a m p le , m an y in v en tory -related issu es a rise b e c a u s e o f th e in h e re n t d iffe ren ce s b e tw e e n th e g o a ls o f m arketin g a n d su p p ly c h a in g ro u p s. M arketing lik e s to h av e th e m ax im u m variety in th e p ro d u ct lin e , w h ere a s su p p ly ch a in m a n a g e m e n t fo c u s e s o n red u cin g th e in v en tory co sts. T h is d iffe re n ce is par­ tially ou tsid e th e s c o p e o f an y m o d e l. C o m in g u p w ith n o n m o d e lin g so lu tio n s is im p or­ tan t as w ell. I f th e p ro b le m arises d u e to “m isalig n m en t” o f in ce n tiv e s o r u n c le a r lin es o f authority o r p lan s, n o m o d e l c a n h e lp . T h u s, it is im p ortan t to id entify th e ro o t issue.

T h e s e c o n d q u e s tio n is im portant b e c a u s e so m e tim e s a n existin g to o l m ay so lv e a p ro b le m th a t th e n sav e s tim e a n d m o n ey . S o m etim e s m odifying a n e x istin g to o l m ay solve th e p ro b lem , ag ain saving s o m e tim e an d m o n e y , b u t s o m e tim e s a cu sto m to o l is n e c e s ­ sary to so lv e th e p ro b lem . T h is is clearly w o rth w h ile to e x p lo re .

• T h e third q u e s tio n is im p o rtan t b e c a u s e s o m e tim e s a n e w c o m p u te r-b a s e d sy stem is n o t re q u ire d to s o lv e th e p ro b le m . T h e d e v e lo p e rs h a v e fo u n d th a t th e y o fte n u se an a ly tica lly d eriv ed d e c is io n g u id e lin e s in s te a d o f a to o l. T h is s o lu tio n re q u ire s le s s tim e fo r d e v e lo p m e n t a n d training, h a s lo w e r m a in te n a n c e re q u ire m e n ts, a n d a ls o p ro v id e s s im p le r a n d m o re intuitive resu lts. T h a t is, a fte r th e y h a v e e x p lo r e d th e p ro b le m d e e p e r, th e d e v e lo p e rs m ay d ete rm in e th a t it is b e tte r to p re s e n t d e c is io n ru le s th a t c a n b e e a s ­ ily im p le m e n te d a s g u id e lin e s fo r d e c is io n m a k in g ra th e r th a n a s k in g th e m a n a g e rs to ru n s o m e ty p e o f a c o m p u te r m o d e l. T h is resu lts in e a s ie r train in g , b e tte r u n d e rstan d in g o f th e ru le s b e in g p ro p o s e d , an d in c re a s e d a c c e p ta n c e . It a lso ty p ically le a d s t o lo w e r d e v e lo p m e n t c o s ts a n d re d u c e d tim e fo r d e p lo y m e n t.

I f a m o d e l h a s to b e built, the d e v e lo p e rs m o v e o n to th e s e c o n d p h a s e th e actu al d esig n an d d ev e lo p m e n t o f th e to o ls. A dhering to five g u id elin es te n d s to in c re a se th e p ro b a b ility that th e n e w to o l w ill b e s u cce ssfu l. T h e first g u id elin e is to d e v e lo p a p ro to ­ typ e as q u ick ly as p o ssib le . T h is allow s th e d e v e lo p e rs to te st th e d esign s, d em on strate vario u s fe a tu res a n d id eas fo r th e n e w to o ls, g e t e a rly fe e d b a c k fro m th e e n d u sers to s e e w h a t w o rk s fo r th e m a n d w h a t n e e d s to b e ch a n g e d , a n d te st ad o p tio n . D ev e lo p in g a p ro to ty p e a lso p rev en ts th e d e v e lo p e rs fro m ov erb u ild in g th e to o l an d y e t a llo w s th em to co n stru ct m o re s c a la b le an d stan d ard ized so ftw a re a p p lica tio n s later. A dditionally, b y d ev elo p in g a p ro toty p e, d e v e lo p e rs c a n sto p th e p r o c e s s o n c e th e to o l is “g o o d e n o u g h ,” rath er th a n b u ild in g a stan d ard ized s o lu tio n th a t w o u ld ta k e lo n g e r to b u ild a n d b e m o re

e x p e n s iv e .

C hapter 2 • Foundations and T ech n o lo g ies for D ecision Making

T h e s e c o n d g u id e lin e is t o “b u ild in sig h t, n o t b la c k b o x e s .” T h e H P s p re a d s h e e t a o d e l d e v e lo p e r s b e lie v e th a t th is is im p o rta n t, b e c a u s e o fte n ju s t e n te r in g s o m e d ata

2n d r e c e iv in g a c a lc u la te d o u tp u t is n o t e n o u g h . T h e u se rs n e e d to b e a b le to th in k o f a ltern a tiv e s c e n a r io s , a n d th e to o l d o e s n o t s u p p o rt this i f it is a '‘b la c k b o x ” that p ro v id e s o n ly o n e re c o m m e n d a tio n . T h e y a rg u e th a t a to o l is b e s t o n ly i f it p ro v id e s in fo rm a tio n to h e lp m a k e a n d s u p p o rt d e c is io n s ra th e r th a n ju st g iv e th e a n s w e rs . T h e y lis o b e lie v e th a t a n in te ra c tiv e to o l h e lp s th e u sers to u n d e rs ta n d th e p r o b le m b e tte r, —e r e fo r e le a d in g to m o re in fo r m e d d e c is io n s .

T h e third g u id elin e is to “re m o v e u n n e e d e d co m p le x ity b e fo re h a n d o ff.” T h is is im portant, b e c a u s e a s a to o l b e c o m e s m o re c o m p le x it re q u ires m o re train in g an d e x p e r- ^se. m o re data, and m o re re calib ratio n s. T h e risk o f b u g s and m isu se a ls o in creases. Som etim es it is b e s t to stud y th e p ro b lem , b e g in m o d e lin g a n d analysis, a n d th e n start shaping th e p ro g ram in to a sim p le -to -u se to o l fo r th e e n d user.

T h e fo u rth g u id elin e is to “p artn er w ith e n d u se rs in d isco v ery and d e s ig n .” B y w o rk ­ ing w ith th e e n d u sers th e d e v e lo p e rs g e t a b e tte r fe e l o f th e p ro b le m and a b etter idea o f w h at th e e n d u se rs w ant. It a lso in cre a se s th e e n d u s e rs ’ ab ility to u se an aly tic to ols. T h e e n d u sers a ls o g ain a b e tte r u n d erstan d in g o f th e p ro b le m an d h o w it is so lv e d u sin g th e n e w to ol. A dditionally, in clu d in g th e en d users in th e d ev e lo p m e n t p r o c e s s e n h a n c e s d i l d e cisio n m a k e rs ’ an aly tical k n o w le d g e and cap ab ilities. B y w o rk in g to g e th e r, th e ir k n o w le d g e an d sk ills c o m p le m e n t e a c h o th er in th e fin a l solu tion.

T h e fifth g u id elin e is to “d e v e lo p a n O p e ra tio n s R e se a rch (O R ) c h a m p io n .” B y involv­ ing e n d u se rs in th e d ev e lo p m e n t p ro c e s s , th e d ev elo p e rs cre a te ch a m p io n s fo r th e n e w to o ls w h o th e n g o b a c k to th e ir d ep artm en ts o r co m p a n ie s an d e n c o u ra g e th e ir c o w o rk ­ ers to a c c e p t an d u s e th em . T h e ch a m p io n s are th e n th e e x p e rts o n th e to o ls in th eir areas and ca n th e n h e lp th o se b e in g in tro d u ced to th e n e w to o ls. H aving ch a m p io n s in cre a se s th e p o ssibility th a t th e to o ls w ill b e a d o p te d in to th e b u s in e s s e s su ccessfu lly .

T h e fin al s ta g e is the h an d o ff, w h e n th e final to o ls that pro v id e c o m p le te so lu tio n s a re given to th e b u sin esses. W h e n p lan n in g th e h an d o ff, it is im p o rtan t to a n s w e r th e fo l­ low ing q u e stio n s:

• W h o w ill u s e th e tool? • W h o o w n s th e d ecisio n s that th e to o l will support? • W h o e ls e m u st b e involved? • W h o is re s p o n s ib le fo r m a in te n a n ce a n d e n h a n c e m e n t o f th e tool? • W h e n will th e to o l b e used? • H o w w ill th e u se o f th e to o l fit in w ith o th e r p ro cesse s? • D o e s it c h a n g e th e p ro cesses? • D o e s it g e n e r a te in p u t in to th o se p ro cesse s? • H o w w ill th e to o l im p a ct b u s in e s s p erfo rm an ce? • A re th e e x istin g m etrics su fficien t to rew ard this a s p e c t o f p erfo rm an ce? • H o w sh o u ld th e m etrics and in ce n tiv e s b e c h a n g e d to m ax im ize im p a ct to th e b u si­

n e s s fro m th e to o l an d process?

B y k e e p in g th e s e le s s o n s in m ind , d e v e lo p e rs a n d p ro p o n e n ts o f c o m p u te riz e d d e c i­ s io n su p p o rt in g e n era l a n d sp re a d sh e e t-b a s e d m o d e ls in p articu lar are lik e ly to e n jo y g re a te r su cce ss.

QUESTIONS F O R TH E OPENING VIGNETTE

1 . W h at are s o m e o f th e k e y q u estio n s to b e ask ed in su p p o itin g d ecisio n m aking throu gh DSS?

2 . W h a t g u id elin es c a n b e le a rn e d fro m this v ig n e tte a b o u t d ev elo p in g DSS?

3. W h at le s s o n s sh o u ld b e k e p t in m in d fo r su cce ssfu l m o d e l im plem entation ?

7 0 Part I • D ecisio n Making and Analytics: An O verview

WHAT W E CAN LEARN FROM THIS VIGNETTE

T h is v ig n e tte re la tes to p ro vid ing d e cisio n su p p o rt in a larg e org anization:

• B e f o r e b u ild in g a m o d e l, d e c is io n m a k e rs sh o u ld d ev elo p a g o o d u n d erstan d in g o f th e p ro b le m that n e e d s to b e ad d ressed .

• A m o d el m ay n o t b e n e ce s s a ry to ad dress th e p ro b lem . • B e fo re developing a n e w tool, d ecision m akers shou ld e xp lo re reu se o f existing tools. • T h e g o al o f m o d e l b u ild in g is to g ain b etter in sig h t in to th e p ro b lem , n o t ju st to

g e n e r a te m o re nu m bers. • Im p le m en tatio n p lan s sh o u ld b e d e v e lo p e d a lo n g w ith th e m odel.

Source: Based on T. Olavson and C. Fry, “Spreadsheet Decision-Support Tools: Lessons Learned at Hewlett- Packard,” Interfaces, Vol. 38, No. 4, July/August 2008, pp. 300-310.

2.2 DECISIO N M AKING: INTRODUCTION A N D DEFINITIONS W e are a b o u t to e x a m in e h o w d e c is io n m ak in g is p ra c tic e d an d s o m e o f th e un d erlying th e o r ie s and m o d e ls o f d e c is io n m ak in g . Y o u will a ls o learn a b o u t th e v ario u s traits o f d e c is io n m ak e rs, in clu d in g w h a t ch a ra cte riz e s a g o o d d e c is io n m ak er. K n o w in g this ca n h e lp y o u to u n d erstan d th e ty p e s o f d e c is io n s u p p o rt to o ls th at m an ag e rs c a n u s e to m a k e m o re e ffe c tiv e d e cisio n s. In th e fo llo w in g s e c tio n s , w e d iscu ss v ario u s a s p e c ts o f d e c is io n m akin g.

Characteristics of Decision M aking In ad dition to th e ch aracteristics p re s e n te d in th e o p e n in g v ignette, decision making m ay involve th e fo llow ing:

• G ro u p th in k (i.e ., gro u p m e m b ers a c c e p t th e so lu tio n w ith o u t th in k in g fo r th em ­ s e lv e s ) c a n lea d to b a d d ecisio n s.

• D e c is io n m ak ers are in te reste d in ev alu atin g w h a t-if scen ario s. • E x p e rim e n tatio n w ith a re al sy stem (e .g ., d e v e lo p a s ch e d u le , try it, an d s e e h ow

w e ll it w o rk s ) m ay result in failure. • E x p e rim e n tatio n w ith a re al sy stem is p o ss ib le o n ly fo r o n e s e t o f co n d itio n s a t a

tim e a n d c a n b e disastrous. • C h an g es in th e d e cisio n -m a k in g en v iro n m en t m ay o c c u r co n tin u o u sly , lead in g to

invalid ating assu m p tio n s a b o u t a situ atio n (e .g ., d eliv eries aro u n d h o lid ay tim es m ay in cre a se , requ irin g a d ifferen t v ie w o f th e p ro b le m ).

• C h an ges in the d ecisio n -m a k in g en v iro n m en t m a y a ffect d e c is io n q u ality b y im pos­ in g tim e p ressu re o n the d e c is io n m aker.

• C o lle ctin g in form ation a n d analy zing a p ro b le m ta k e s tim e an d c a n b e e x p e n s iv e . It is d ifficu lt to d eterm in e w h e n to sto p and m a k e a d ecisio n .

• T h e r e m ay n o t b e s u fficie n t in form ation to m a k e a n in tellig en t d ecisio n . • T o o m u ch in fo rm atio n m ay b e a v ailab le (i.e ., in fo rm atio n o v erlo ad ).

T o d eterm in e h o w real d e cisio n m ak ers m a k e d ecisio n s, w e m ust first u n d erstan d the p ro cess and th e im portant issu es in v o lv ed in d e c is io n m aking. T h e n w e c a n understand ap p rop riate m e th o d o lo g ie s fo r assisting d e cisio n m a k e rs a n d th e co n trib u tio n s inform ation system s c a n m ak e. O nly th e n c a n w e d e v e lo p D SS to h e lp d e c is io n m akers.

T h is c h a p te r is o r g a n iz e d b a s e d o n th e th re e k e y w o rd s th at fo rm th e te rm DSS: d e c i s i o n , s u p p o r t , a n d s y s t e m s . A d e c is io n m a k e r s h o u ld n o t sim p ly ap p ly IT to o ls b lin d ly . R ath er, th e d e c is io n m a k e r g e ts s u p p o rt th ro u g h a ra tio n a l a p p r o a c h that

Chapter 2 • Foundations and T ech n o lo g ies for D ecisio n Making

sim p lifies re a lity a n d p ro v id e s a re la tiv e ly q u ic k a n d in e x p e n s iv e m e a n s o f co n s id e rin g v arious a ltern a tiv e c o u rs e s o f a c tio n to arrive a t th e b e s t ( o r at le a s t a v e ry g o o d ) s o lu ­ tio n t o th e p ro b le m .

A W orking D efinition of Decision M aking D ecision m akin g is a p ro c e s s o f ch o o s in g a m o n g m o o r m o re altern ativ e co u rse s o f actio n fo r th e p u rp o s e o f attaining o n e o r m o re g oals. A cco rd in g to Sim o n ( 1 9 7 7 ) , m an a­ g erial d e c is io n m ak in g is sy n o n y m o u s w ith th e en tire m a n a g e m e n t p ro c e s s . C o n sid er the im p ortan t m an ag erial fu n ctio n o f p lan n in g . P lan n in g in v olv es a serie s o f d ecisio n s: W hat sh o u ld b e d o n e? W hen? W here? W hy? How ? B y w hom ? M an ag ers s e t g o a ls , o r plan; h e n ce , p lan n in g im p lie s d e cisio n m akin g. O th e r m an ag erial fu n ctio n s, s u ch as organizing and co n tro llin g , a ls o involve d e c is io n m aking.

Decision-Making Disciplines D e cis io n m ak in g is d irectly in flu e n ced b y sev eral m a jo r d iscip lin es, s o m e o f w h ic h are b eh av io ral a n d s o m e o f w h ic h are scie n tific in natu re. W e m u st b e aw are o f h o w th eir p h ilo so p h ies c a n a ffe c t o u r ability to m ak e d e cis io n s an d pro v id e su p p ort. B eh av io ral d iscip lin es in c lu d e an th ro p o lo g y , law , p h ilo so p h y , p o litical s c ie n c e , p sy ch o lo g y , so cial p sy ch olog y , a n d so cio lo g y . S cien tific d iscip lin e s in clu d e co m p u te r s c ie n c e , d e cisio n analysis, e c o n o m ic s , e n g in e e rin g , th e h ard s c ie n c e s (e .g ., b io lo g y , ch em istry, p hy sics), m an ag e m e n t s cie n ce / o p e ra tio n s re s e a rch , m ath em atics, and statistics.

An im p ortan t ch a ra cteristic o f m an a g e m e n t su p p o rt system s (M SS) is th e ir e m p h a ­ sis o n th e e ffe c tiv e n e s s, o r “g o o d n e s s ,” o f th e d e c is io n p ro d u ce d rath er th a n o n the com p u tatio n al e ffic ie n c y o f o b ta in in g it; this is u su ally a m a jo r c o n c e r n o f a tra n sa ctio n p ro cessin g sy stem . M ost W e b -b a s e d D SS are fo c u s e d o n im p ro vin g d e c is io n e ffe ctiv e n e ss. E fficie n cy m ay b e a by-p rod u ct.

Decision Style and Decision M akers In th e fo llo w in g s e c tio n s , w e e x a m in e th e n o tio n o f d e c is io n style and s p e c ific a sp e cts a b o u t d e c is io n m akers.

DECISION STYLE D e cisio n sty le is th e m a n n er b y w h ich d e cisio n m ak ers th in k an d react to p ro b lem s. T h is in clu d es th e w ay th e y p e rc e iv e a p ro b lem , th eir co g n itiv e re sp o n se s, in d h o w v a lu e s a n d b e lie fs v a ry fro m individual to individual a n d fro m situ atio n to situation. As a resu lt, p e o p le m a k e d e cis io n s in d ifferen t w ay s. A lthough th e re is a g en eral p ro cess o f d e c is io n m ak in g , it is fa r from linear. P e o p le d o n o t fo llo w th e sam e steps of th e p ro c e s s in th e sa m e s e q u e n c e , n o r d o th ey u se all th e step s. F u rth e rm o re, the em phasis, tim e a llo tm e n t, a n d priorities g iv en to e a c h s te p v ary sig nificantly, n o t only from o n e p e rs o n to an o th er, b u t a lso fro m o n e situ ation to th e n e x t. T h e m a n n e r in w h ich m anagers m a k e d e c is io n s (a n d th e w a y th e y in te ract w ith o th er p e o p le ) d e s c rib e s th eir d ecisio n style. B e c a u s e d e c is io n styles d e p e n d o n the facto rs d e s crib e d e arlier, th e re are m any d e c is io n sty les. P erso n ality te m p e ra m e n t tests are o fte n u se d to d ete rm in e d e cisio n sqdes. B e c a u s e th e re are m an y s u ch tests, it is im p ortan t to try to e q u a te th e m in d eter­ m ining d e c is io n style. H o w ev er, th e v arious tests m e a su re so m e w h a t d iffe ren t a s p e cts o f personality, s o th e y c a n n o t b e e q u ated .

R e se a rch e rs h a v e id en tified a n u m b e r o f d ecisio n -m a k in g styles. T h e s e in clu d e h e u ­ ristic and an aly tic styles. O n e c a n a lso d istinguish b e tw e e n a u to cra tic v e rsu s d em o cra tic styles. A n o th er sty le is co n su ltativ e (w ith individuals o r g ro u p s). O f c o u rs e , th e re are m any co m b in a tio n s an d v ariation s o f styles. F o r e x a m p le , a p e rs o n c a n b e an aly tic an d au tocratic, o r co n su ltativ e (w ith individuals) a n d heuristic.

72 Part I • D ecisio n M aking and Analytics: An Overview

F o r a co m p u te riz e d sy stem to s u cce ssfu lly s u p p o rt a m a n a g e r, it sh o u ld fit th e d e c is io n situ atio n as w e ll as th e d e c is io n sty le. T h e re fo re , th e sy stem s h o u ld b e fle x ib le an d a d a p ta b le to d iffe ren t u sers. T h e ab ility to a s k w h a t-if an d g o a l-s e e k in g q u e s tio n s p ro v id e s fle x ib ility in this d irectio n . A W e b -b a s e d in te rfa c e u sin g g ra p h ics is a d esira b le fe a tu re in su p p o rtin g ce rta in d e c is io n styles. I f a D SS is to s u p p o rt v arying styles, skills, and k n o w le d g e , it sh o u ld n o t attem p t to e n fo r c e a s p e c ific p ro c e s s . R ather, it sh o u ld h e lp d e c is io n m a k e rs use a n d d e v e lo p th e ir o w n sty les, s k ills, an d k n o w le d g e.

D ifferen t d e c is io n sty les re q u ire d ifferen t typ es o f su p p ort. A m a jo r fa cto r th at d eter­ m in es th e ty p e o f su p p o rt requ ired is w h e th e r th e d e c is io n m a k e r is a n individual o r a group. Individual d e c is io n m akers n e e d a c c e s s to d ata and to e x p e rts w h o ca n p rovide a d v ice, w h e re a s gro u p s ad d itionally n e e d co lla b o ra tio n to ols. W e b -b a s e d D SS c a n p ro ­ v id e su p p o rt to b oth .

A lo t o f in form ation is a v ailab le o n th e W e b a b o u t co g n itiv e styles an d d ecisio n styles (e .g ., s e e B irk m an In tern atio n al, In c., birkm an.com ; K eirsey T e m p e ra m e n t Sorter and K e irse y T e m p e ra m e n t T h e o ry -II, keirsey.com ). M any p erson ality/ tem p eram en t tests are av ailab le to h e lp m an ag ers id entify th e ir o w n sty les an d th o s e o f th e ir em p lo y e es. Id entify ing a n individual’s style c a n h e lp e sta b lish th e m o st e ffe ctiv e co m m u n icatio n p attern s and id eal ta sk s fo r w h ich th e p e rs o n is suited.

D E C IS IO N M A K E R S D ecisio n s are o ften m ad e b y individuals, esp ecially a t low er m anage­ rial levels and in sm all organizations. T h e re m ay b e conflicting o b jectiv es e v e n fo r a sole d ecisio n m aker. F o r exam p le, w h e n m aking an investm ent decision, a n individual investor m ay co n sid er th e rate o f return o n th e investm ent, liquidity, and safety as objectives. Finally, d ecisio ns m ay b e fully autom ated (b u t only after a h u m an d ecisio n m aker d ecid es t o d o so!).

T h is d iscu ssio n o f d e cisio n m aking fo cu se s in large part o n a n individual d ecisio n m aker. M ost m ajo r d ecisio n s in m ed iu m -sized an d la rg e organ ization s are m ad e b y groups. O bv io u sly , th e re are o ften con flictin g o b je ctiv e s in a g ro u p d ecisio n -m ak in g setting. G roups c a n b e o f v ariab le size and m ay in clu d e p e o p le fro m d ifferent d ep artm ents o r fro m differ­ e n t org anizations. C ollaborating individuals m ay h av e d ifferent co g n itiv e styles, personality types, and d e cisio n styles. S o m e clash , w h ere a s o th ers are m utually e n h an cin g . C o nsensu s ca n b e a difficult p o litical p ro b lem . T h e re fo re , th e p ro c e s s o f d e c is io n m ak in g b y a group c a n b e very co m p licate d . C o m p u terized su p p ort c a n greatly e n h a n c e gro u p d ecisio n m aking. C o m p u ter su p p o rt can b e provid ed at a b ro a d lev el, e n a b lin g m e m b ers o f w h o le dep artm ents, divisions, o r e v e n entire organ ization s to co lla b o ra te on lin e. S u ch supp ort has e v o lv ed o v e r the p a st fe w y ears into en terp rise inform ation system s (E IS) and includes gro u p su p p ort system s (G S S ), e n terp rise re so u rce m a n a g e m e n t (ERM )/enterprise resou rce p lan n in g (E R P ), su p p ly c h a in m a n a g e m e n t (SC M ), k n o w le d g e m an ag e m e n t system s (KM S), and cu sto m e r relatio n sh ip m an ag e m e n t (CRM) system s.

SECTION 2 .2 REVIEW QUESTIONS

1 . W h a t are th e v arious a s p e cts o f d e cisio n m aking? 2 . Id en tify sim ilarities an d d iffe ren ce s b e tw e e n individ ual an d g ro u p d e cisio n m aking. 3 - D e fin e d ecisio n style an d d e s c rib e w h y it is im p o rtan t to c o n s id e r in th e d e c is io n ­

m ak in g p ro ce ss. 4 . W h a t are th e b e n e fits o f m ath em atical m odels?

2.3 PH A SES OF THE D ECISIO N -M A KIN G PR O CESS It is ad visable to fo llo w a system atic d ecisio n -m ak in g pro cess. Sim on (1 9 7 7 ) said that this involves th ree m ajo r p h ases: in tellig en ce, d esign, an d c h o ic e . H e later ad d ed a fourth p hase, im plem entation. M onitoring c a n b e co n sid ered a fifth p h ase— a fo rm o f fe e d b ack . H ow ever.

Chapter 2 • Foundations and T ech n o lo g ies for D ecisio n Making

S u c c e s s V

Organization objectives Search and scanning procedures

| Simplification Data collection Assumptions Problem identification

Problem ownership Problem classification Problem statement

Validation of the model

Verification, testing of proposed solution

Implementation of solution

Problem State m en t

Formulate a model Se t criteria for choice Search for alternatives Predict and measure outcomes

Solution to the model Sensitivity analysis Selection of the best [good]

alternative^) Plan for implementation

Failure

FIGURE 2.1 The Decisior-Making/Modeling Process.

w e v iew m onitoring a s th e intelligence p h a s e ap p lied to the im plem entation p h a se. Sim on 's m odel is th e m o st c o n c is e and y e t co m p le te characterization o f rational d ecisio n m aking. A con cep tu al picture o f th e d ecisio n -m akin g p ro cess is sh o w n in Figure 2.1.

T h e re is a c o n tin u o u s flo w o f activity fro m in te llig e n ce to d esig n to c h o ic e (s e e the b o ld lin es in Figure 2 .1 ), b u t a t a n y p h a s e , th e re m ay b e a return to a p rev iou s p h a se fe e d b a c k ). M odeling is a n e sse n tia l part o f this p ro ce ss. T h e see m in g ly c h a o tic n a tu re o f

fo llow in g a h a p h a z a rd p ath fro m p ro b le m d isco v e ry to so lu tio n v ia d e c is io n m ak in g can b e e x p la in e d b y th e s e fe e d b a c k lo o p s.

T h e d ecisio n -m akin g p ro cess starts w ith the intelligence phase; in this p h ase , the decision m aker ex a m in e s reality an d identifies and d efines th e problem . Problem ow nership s established as w ell. In th e design phase, a m o d el that represents th e system is constructed . This is d o n e b y m aking assum ptions that sim plify reality and w riting d ow n the relationships im o n g all the variables. T h e m o d el is th e n validated, an d criteria are determ ined in a princi­ ple o f c h o ic e fo r evaluation o f the alternative cou rses o f a ction that are identified. O ften, the process o f m o d el d ev elop m en t identifies alternative solutions an d v ice versa.

T h e ch o ice p h ase in clu d e s s e le c tio n o f a p ro p o s e d so lu tio n to th e m o d e l (n o t necessarily to th e p ro b le m it re p re sen ts). T h is so lu tio n is te sted to d eterm in e its viability. W h en th e p ro p o s e d so lu tio n se e m s re a s o n a b le , w e are re ad y fo r th e last p h a se : im p le­ m e n tation o f th e d e c is io n (n o t n e cessa rily o f a sy stem ). S u cc essfu l im p lem e n ta tio n results m solving th e re al p ro b le m . F ailu re lead s to a return to a n e a rlie r p h a s e o f th e p ro c e s s . In fact, w e c a n retu rn to a n e a rlie r p h a s e during an y o f th e latter th re e p h a se s. T h e d e c is io n ­ m aking situ ation s d e s c rib e d in th e o p e n in g v ig n e tte fo llo w S im o n ’s fo u r-p h a se m o d e l, as do alm o st all o th e r d e cisio n -m a k in g situations. W e b im p acts o n th e fo u r p h a se s, a n d vice versa, a re s h o w n in T a b le 2.1.

7 4 Part I • D ec isio n M aking and Analytics: An Overview

T A B L E 2.1 Sim on's Four Phases o f Decision M aking a n d th e W e b

Phase W e b Impacts Im pacts on th e W e b

Intelligence Access to information to identify problems and opportunities from internal and external data sources

Access to analytics methods to identify opportunities

Collaboration through group support systems (GSS) and knowledge management systems (KMS)

Identification of opportunities for e-commerce, W eb infrastructure, hardware and software tools, etc.

Intelligent agents, which reduce the burden of information overload

Smart search engines

Design Access to data, models, and solution methods

Use of online analytical processing (OLAP), data mining, and data warehouses

Collaboration through GSS and KMS Similar solutions available from KMS

Brainstorming methods (e.g., GSS) to collaborate in W eb infrastructure design

Models and solutions of W eb infrastructure issues

Choice Access to methods to evaluate the impacts of proposed solutions

Decision support system (DSS) tools, which examine and establish crite-a from models to determine Web, intranet, and extranet infrastructure

DSS tools, which determine how to route messages

Implementation Web-based collaboration tools (e.g., GSS) and KMS, which can assist in implementing decisions

Tools, which monitor the performance of e-commerce and other sites, including intranets, extranets, and the Internet

Decisions implemented on browse' and server design and access, which ultimately determined ho* j to set up the various com ponent " that have evolved into the Internet

N ote that th e re are m an y o th e r d ecisio n -m a k in g p ro c e s s e s. N otable am o n g t h e n 3 th e K e p n e r-T re g o e m e th o d (K e p n e r a n d T re g o e , 1 9 9 8 ), w h ich h a s b e e n a d o p te d b y man firm s b e c a u s e its to o ls a re read ily a v ailab le fro m K e p n e r-T re g o e , In c. (kepner-tregc»±. com). W e h av e fo u n d th at th e s e altern ativ e m o d e ls, in clu d in g th e K e p n e r-T re g o e m e ::.a read ily m a p in to S im o n ’s fo u r-p h a se m o d el.

W e n e x t tu rn to a d eta iled d iscu ssio n o f th e fo u r p h a s e s identified b y Sim on.

SECTION 2 . 3 REVIEW QUESTIONS

1 . List and b rie fly d e s c rib e S im o n ’s fo u r p h a s e s o f d e c is io n m aking.

2. W h a t are th e im p acts o f th e W e b 011 th e p h a se s o f d e cisio n m aking?

2.4 D ECISIO N M AKING: THE IN TELLIG EN C E PH ASE In te llig e n ce in d e c is io n m ak in g in v olv es sca n n in g th e e n v iro n m en t, e ith e r in te rm itte n t o r co n tin u o u sly . It in clu d e s sev era l activities a im ed a t id entifying p ro b le m situ a tio n ' 1 op p o rtu n ities. It m ay a ls o in clu d e m o n ito rin g th e results o f th e im p lem e n tatio n pha>r o a d ecisio n -m a k in g p ro cess.

Chapter 2 • Foundations and T ech n o lo g ies for D ecisio n Making 7 5

Problem (or O pportunity) Identification TEir intelligence p h a se b eg in s w ith th e identification o f organizational g oals and ob jectives e a c r d to an issue o f c o n c e rn (e .g ., inventory m anagem ent, jo b selectio n , lack o f o r in correct

W e r o re se n ce ) a n d determ ination o f w h eth er th ey are b ein g m et. P rob lem s o ccu r b e c a u se o f f a s a s f a c t i o n w ith th e status q u o. D issatisfaction is th e result o f a d ifferen ce b e tw e e n w hat

sien-cie desire (o r e x p e c t) an d w h at is occurring. In this first p h ase, a d ecisio n m ak e r attem pts i determine w h eth er a p ro b lem exists, identify its sym ptom s, d eterm ine its m agnitude, and

E m i l y d efine it. O ften, w h at is d escribed as a p ro b lem (e .g ., exce ssiv e co sts) m ay b e iv 2 sym ptom (i.e., m easu re) o f a p ro b lem (e .g ., im proper inventory levels). B e c a u s e real-

E ad d problem s are usu ally com plicated b y m an y interrelated factors, it is som etim es difficult D ifd n g u is h b e tw e e n th e sym ptom s and th e real problem . N ew opportunities an d prob-

t - s certainly m ay b e u n co v e re d w h ile investigating th e cau ses o f sym ptom s. F or exam p le, pciieation C ase 2.1 d escrib e s a classic story o f recognizin g th e correct problem .

T h e e x is te n c e o f a p ro b le m ca n b e d eterm in ed b y m o n ito rin g an d an aly zin g th e e s s m iz a tio n ’s p ro d u ctivity lev el. T h e m e a su re m e n t o f produ ctivity a n d th e co n stru ctio n r f a m o d el are b a se d o n re al d ata. T h e c o lle c tio n o f data an d th e estim atio n o f fu ture data

s - e am o n g th e m o st d ifficu lt s te p s in th e analysis. T h e fo llo w in g are s o m e issu es th a t m ay d uring d ata c o lle c tio n an d estim atio n an d thu s p lag u e d e cisio n m akers:

• D ata are n o t a v ailab le. As a result, th e m o d el is m ad e w ith, a n d re lie s o n , p o ten tially

in accu rate estim ates. • O b tain in g d ata m ay b e e x p e n siv e . • D ata m ay n o t b e a ccu ra te o r p re c ise e n o u g h . • D ata e stim a tio n is o fte n su b jectiv e. • D ata m ay b e in secu re. • Im p o rtan t data th at in flu e n ce th e results m ay b e qualitative (so ft). • T h e re m ay b e to o m an y d ata (i.e ., in form ation o v erlo ad ).

Application Case 2.1 Making Elevators Go Faster! This story h a s b e e n re p o rted in n u m e ro u s p la ce s and h a s alm o st b e c o m e a c la ss ic e x a m p le to e x p la in d ie n e e d fo r p ro b le m id entification. A c k o ff (a s cited in L arson, 1 9 8 7 ) d e s c rib e d th e p ro b le m o f m an agin g com p laints a b o u t slo w elev ato rs in a tall h otel tow er. After trying m a n y so lu tio n s fo r re d u cin g th e c o m ­ plaint: stag g erin g e le v ato rs to g o to d ifferen t floors, adding o p erato rs, a n d s o o n , th e m a n a g e m e n t d e te r­ m ined that th e re a l p ro b le m w a s n o t a b o u t th e a c tu a l w aiting tim e b u t rath er th e p e r c e iv e d w aitin g tim e. So th e so lu tio n w a s to install fu ll-len gth m irrors o n e le v a to r d o o rs o n e a c h flo o r. As H e s se a n d W o o ls e y (1 9 7 5 ) p u t it, “th e w o m e n w o u ld lo o k a t th e m se lv es in th e m irrors a n d m a k e ad ju stm ents, w h ile th e m e n w ould lo o k a t th e w o m e n , a n d b e fo r e th ey k n e w it, th e e le v a to r w as th e r e .” B y red u cin g th e p e rce iv ed w aiting tim e, th e p ro b le m w e n t aw ay. B a k e r and

C am eron ( 1 9 9 6 ) give sev eral o th e r e x a m p le s o f d is­ tractio n s, in clu d in g lig h tin g, d isplays, an d s o o n , that org an izatio n s u s e to re d u c e p e rc e iv e d w aiting tim e. I f th e real p ro b le m is id e n tified a s p e r c e iv e d w aiting tim e, it c a n m a k e a b ig d iffe re n ce in th e p ro p o s e d so lu tio n s an d th eir c o s ts . F o r e x a m p le , fu ll-len gth m irrors p ro b a b ly c o s t a w h o le lo t less th a n ad ding a n elevator!

Sources: Based on J. Baker and M. Cameron, “The Effects of the Service Environment on Affect and Consumer Perception of Waiting Time: An Integrative Review and Research Propositions,” Jo u rn a l o f th e A cadem y o f M arketin g S cience, Vol. 24, September 1996, pp. 338-349; R. Hesse and G . Woolsey, A pplied M anagem ent Scien ce: A Q uick a n d Dirty A pproach, SRA Inc., Chicago, 1975; R. C. Larson. “Perspectives on Queues: Social Justice and the Psychology of Queuing,” O peration s R esearch, Vol. 35, No. 6, November/December 1987, pp. 895-905.

• O u tc o m e s ( o r re s u lts ) m a y o c c u r o v e r a n e x te n d e d p e rio d . As a re su lt, re v ­ e n u e s , e x p e n s e s , a n d p ro fits w ill b e r e c o r d e d a t d iffe re n t p o in ts in tim e . T o o v e r c o m e th is d ifficu lty , a p re s e n t-v a lu e a p p r o a c h c a n b e u s e d i f th e re su lts are

q u a n tifia b le . • It is a ssu m e d th at future d ata w ill b e sim ilar to h istorical data. I f this is n o t th e ca se ,

th e natu re o f th e c h a n g e h a s to b e p re d icte d a n d in clu d e d in th e analysis.

W h e n th e p relim in ary in v estigation is co m p le te d , it is p o ss ib le to d eterm in e w h e th e r a p ro b le m really exists, w h e re it is lo ca te d , an d h o w sig n ifican t it is. A k e y issu e is w h e th e r a n in fo rm atio n sy stem is rep ortin g a p ro b le m o r o n ly th e sym ptom s o f a p ro b lem . For ex a m p le , if rep o rts in d icate th at s a le s are d o w n , th e re is a p ro b lem , b u t th e situation, n o d o u b t, is’ sy m p to m atic o f th e p ro b lem . It is critical to k n o w th e real p ro b lem . Som etim es it m ay b e a p ro b le m o f p e rce p tio n , in cen tiv e m ism atch , o r organ ization al p ro c e s s e s rath er

th a n a p o o r d e c is io n m odel.

Problem Classification P ro b le m classificatio n is th e co n c e p tu a liz a tio n o f a p ro b lem in a n attem p t to p la c e it in a d e fin a b le categ ory , p o ssib ly lead in g to a stan d ard so lu tio n ap p ro a ch . An im portant a p p ro a ch classifies p ro b lem s a cco rd in g to th e d e g r e e o f stru ctu red n ess e v id e n t in them . T h is ran g es fro m totally stru ctured ( i.e ., p ro g ra m m e d ) to to tally u n sta ictu re d ( i.e ., u n p ro­ gram m ed ), as d e s crib e d in C h ap ter 1.

Problem Decomposition M any c o m p le x p ro b le m s c a n b e divided into su b p ro b le m s. So lv in g th e sim p ler su b p ro b ­ lem s m ay h e lp in solv in g a c o m p le x p ro b lem . A lso, see m in g ly p o o rly stru ctured p ro b lem s s o m e tim e s h a v e h ighly stru ctured s u b p ro b lem s. J u s t as a sem istru ctu red p ro b le m results w h e n s o m e p h a se s o f d e c is io n m ak in g are stru ctu red w h e re a s o th e r p h a se s are un struc­ tured, so w h e n so m e s u b p ro b lem s o f a d ecisio n -m a k in g p ro b le m are stru ctured w ith o th ers un structured , th e p ro b le m its e lf is sem istru ctu red . As a D SS is d e v e lo p e d an d th e d e c is io n m a k e r an d d ev e lo p m e n t sta ff learn m o re a b o u t th e p ro b le m , it gains structure. D e c o m p o sitio n a lso facilitates co m m u n ica tio n a m o n g d e c is io n m akers. D e co m p o sitio n is o n e o f th e m o st im portant a s p e cts o f th e an aly tical h iera rch y p ro ce ss. (AHP is d iscu sse d in C h ap ter 1 1, w h ich h e lp s d e c is io n m a k e rs in co rp o ra te b o th qualitative an d quantitative fa cto rs into th eir d ecisio n -m ak in g m o d e ls.)

Problem Ownership In th e in te llig e n c e p h a s e , it is im p o rta n t to e s ta b lis h p r o b le m o w n e rs h ip . A p ro b le m e x is ts in a n o rg a n iz a tio n o n ly if s o m e o n e o r s o m e g ro u p ta k e s o n th e re s p o n s ib ility o a tta c k in g it a n d i f t h e o r g a n iz a tio n h a s th e ab ility to s o lv e it. T h e a s s ig n m e n t o f a u th o r­ ity to s o lv e th e p r o b le m is c a lle d p ro b lem ow n ersh ip . F o r e x a m p le , a m a n a g e r m ay f e e l th a t h e o r s h e h a s a p r o b le m b e c a u s e in te r e s t ra te s a re to o h ig h . B e c a u s e in te re st ra te le v e ls a r e d e te rm in e d at th e n a tio n a l a n d in te rn a tio n a l le v e ls , a n d m o st m a n a g e rs c a n d o n o th in g a b o u t th e m , h ig h in te re s t rate s a r e th e p r o b le m o f th e g o v e r n m e n t, n o t a p r o b le m fo r a s p e c ific c o m p a n y to s o lv e . T h e p r o b le m c o m p a n ie s a c tu a lly fa c e is h o w to o p e r a te in a h ig h -in te r e s t-r a te e n v iro n m e n t. F o r a n in d iv id u al c o m p a n y , th e in te re s t ra te le v e l sh o u ld b e h a n d le d as a n u n c o n tr o lla b le (e n v ir o n m e n ta l) fa c to r to b e

p re d ic te d . , . W h e n p ro b le m o w n e rsh ip is n o t e sta b lish e d , e ith e r s o m e o n e is n o t d o in g his or

h e r jo b o r th e p ro b le m at h an d h a s y e t to b e id e n tified as b e lo n g in g to a n y o n e. It is th e n im portant fo r s o m e o n e to e ith e r v o lu n te e r to o w n it o r assig n it to so m e o n e .

T h e in te llig e n ce p h a s e e n d s w ith a fo rm al p ro b le m statem en t.

7 6 Part I • D ec isio n M aking and Analytics: An O verview

Chapter 2 * Foundations and T ech n o lo g ies for D ecisio n M aking 7 7

f ZCTION 2 . 4 R EV IEW QUESTIONS

1 . W hat is th e d iffe re n ce b e tw e e n a p ro b le m a n d its sym ptom s?

2 . W h y is it im p o rtan t to classify a problem ? 3 . W hat is m e a n t b y p r o b le m decom p osition ? -L W hy is e sta b lish in g p ro b le m o w n e rsh ip s o im portant in th e d e cis io n -m a k in g process?

2.5 DECISIO N M AKIN G : THE D ESIG N PH ASE The d esig n p h a s e in v olv es find ing o r d ev elo p in g an d a n aly zin g p o ss ib le co u rs e s o f a ctio n These in clu d e u n d erstan d in g th e p ro b le m a n d testin g so lu tio n s fo r fe a sib ility A m o d el rr th e d e cisio n -m a k in g p ro b le m is co n stru cte d , te sted , an d validated . Let u s first d efin e

i m odel.

Models1 A m ajor characteristic o f a D SS and m any B l to o ls (n o ta b ly th o se o f b u sin ess an aly tics) is the inclu sion o f a t le a st o n e m o d el. T h e b a sic idea is to p erfo rm th e D SS analysis o n a m o d el o f reality rath er th an o n the real system . A m o d el is a sim plified rep resen tatio n o r ab strac­ tio n o f reality. It is usually sim plified b e ca u se reality is to o co m p le x to d escrib e e xactly an d b ecau se m u ch o f th e com p lex ity is actu ally irrelevant in solving a sp e cific p ro blem .

Mathem atical (Quantitative) Models T h e com p lexity o f relationships in m an y organizational system s is d escrib e d m athem ati­ cally. M ost D SS an alyses are p erfo rm ed num erically w ith m ath em atical o r o th e r quantitative

m odels.

The Benefits of Models We u se m o d els fo r th e fo llo w in g re aso n s:

• M anipu lating a m o d e l (ch a n g in g d e c is io n v ariab les o r th e e n v iro n m e n t) is m u ch e a sie r th a n m an ip u latin g a real system . E x p e rim e n tatio n is e a s ie r an d d o e s n o t in terfere w ith th e o rg an izatio n ’s daily o p eratio n s.

• M odels e n a b le th e c o m p re ss io n o f tim e. Y e a r s o f o p era tio n s c a n b e sim u lated in

m in u tes o r s e c o n d s o f co m p u te r tim e. • T h e c o s t o f m o d e lin g analysis is m u ch lo w e r th a n th e c o s t o f a sim ilar e x p e rim e n t

c o n d u cte d o n a re al system . • T h e c o s t o f m ak in g m istak es during a trial-an d -error e x p e rim e n t is m u ch lo w e r

w h e n m o d e ls are u s e d th a n w ith real system s. • T h e b u s in e s s e n v iro n m en t in v olv es co n s id e ra b le u n certainty. W ith m o d elin g , a

m an ag e r c a n estimate th e risks resultin g fro m s p e c ific actions. • M ath em atical m o d e ls e n a b le th e an alysis o f a very large, s o m e tim e s in fin ite, n u m b e r

o f p o ss ib le solu tion s. E v e n in s im p le p ro b le m s, m an ag e rs o fte n h av e a larg e n u m b e r

o f a ltern ativ es fro m w h ich to c h o o s e . • M o d els e n h a n c e an d re in fo rce learn in g and training. • M o d els a n d so lu tio n m e th o d s are read ily available.

M o d e lin g in v o lv e s co n c e p tu a liz in g a p ro b le m an d a b stra ctin g it t o q u an titativ e and/or q u a lita tiv e fo rm ( s e e C h ap te r 9 ) . F o r a m a th em a tica l m o d e l, th e v a r ia b le s are

-Caution- Many students and professionals view models strictly as those of “data modeling’ in the context of S te r n s a n a " ! and design. Here, we consider analytical models such as those of linear programmmg, s t a ­ tion, and forecasting.

7 8 P art I • D e c is io n M aking and Analytics-. An O verview

id e n tified , a n d th e ir m u tu al re la tio n sh ip s a re e sta b lis h e d . S im p lifica tio n s a re m ad e , w h e n e v e r n e c e s s a ry , th ro u g h a ssu m p tio n s. F o r e x a m p le , a re la tio n sh ip b e tw e e n tw o v a r ia b le s m ay b e a ssu m e d to b e lin e a r e v e n th o u g h in re a lity th e r e m ay b e s o m e n o n ­ lin e a r e ffe c ts . A p r o p e r b a la n c e b e tw e e n th e le v e l o f m o d e l s im p lifica tio n a n d th e re p ­ r e s e n ta tio n o f re ality m u st b e o b ta in e d b e c a u s e o f th e c o s t—b e n e fit tra d e -o ff. A s im p le r m o d e l lead s to lo w e r d e v e lo p m e n t c o s ts , e a s ie r m a n ip u la tio n , a n d a fa s te r s o lu tio n b u t is le s s re p re s e n ta tiv e o f th e re a l p ro b le m a n d c a n p r o d u c e in a c c u ra te resu lts. H o w e v e r, a sim p le r m o d e l g e n e ra lly re q u ire s fe w e r d ata, o r th e d ata a re a g g re g a te d an d e a s ie r

to o b ta in . T h e p ro c e s s o f m o d e lin g is a co m b in a tio n o f art a n d s c ie n c e . As a s c ie n c e , th e re

a re m an y stan d ard m o d el cla ss e s av a ila b le, and , w ith p ra ctice , a n an aly st c a n d e te im in e w h ic h o n e is a p p lic a b le to a g iv e n situ ation . As an art, creativity an d fin e s s e a re requ ired w h e n d eterm in in g w h at sim p lifying a ssu m p tio n s c a n w o rk , h o w to c o m b in e ap p ro p ri­ a te fe a tu re s o f th e m o d e l cla ss e s , an d h o w to in te g rate m o d e ls to o b ta in valid solu tion s. M o d e ls h av e decision variables that d e s c rib e th e altern ativ es fro m a m o n g w h ic h a m a n a g e r m u st c h o o s e (e .g ., h o w m an y ca rs to d e liv e r to a s p e c ific re n tal ag e n cy , h o w to ad vertise at s p e c ific tim e s, w h ic h W e b serv e r to b u y o r le a s e ), a resu lt v a ria b le o r a s e t o f resu lt v a ria b le s ( e .g ., p rofit, re v e n u e , s a le s ) th a t d e s c r ib e s th e o b je c tiv e o r g o a l o f th e d e cis io n -m a k in g p ro b le m , and u n c o n tro lla b le v a ria b le s o r p a ra m ete rs (e .g ., e c o n o m ic c o n d itio n s ) th a t d e s c rib e th e e n v iro n m en t. T h e p r o c e s s o f m o d e lin g in v o lv e s d eterm in ­ ing th e (u su ally m ath em atical, s o m e tim e s s y m b o lic ) re la tio n sh ip s am o n g th e v ariab le s. T h e s e to p ic s a re d iscu sse d in C h ap ter 9-

Selection o f a Principle of Choice A p rinciple o f ch o ice is a c rite rio n th at d e s c r ib e s th e a c c e p ta b ility o f a so lu tio n a p p ro a c h . In a m o d e l, it is a re su lt v aria b le . S e le c tin g a p rin cip le o f c h o ic e is n o t part o f th e c h o ic e p h a s e b u t in v o lv e s h o w a p e rs o n e sta b lis h e s d e cis io n -m a k in g o b je c tiv e (s ) an d in c o rp o ra te s th e o b je c tiv e (s ) in to th e m o d e l(s ). A re w e w illin g to a ssu m e high risk, o r d o w e p re fe r a lo w -risk a p p ro ach ? A re w e a tte m p tin g to o p tim ize o r satisfice? It is a ls o im p o rtan t to r e c o g n iz e th e d iffe re n ce b e tw e e n a crite rio n a n d a co n strain t ( s e e T e c h n o lo g y In sig h ts 2 .1 ). A m o n g th e m an y p rin cip le s o f c h o ic e , n o rm ativ e and d escrip tiv e a re o f p rim e im p o rtan ce .

T E C H N O L O G Y IN SIG H T S 2 . 1 T h e D if f e r e n c e B e t w e e n a C r ite r io n a n d a C o n s tr a in t

Many people new to the formal study o f decision making inadvertently confuse the concepts o f criterion and constraint. Often, this is because a criterion may imply a constraint, either implicit or explicit, thereby adding to the confusion. For example, there may b e a distance criterion that the decision maker does not want to travel too far from home. However, there is an implicit constraint that the alternatives from which he selects must be within a certain distance from his home. This constraint effectively says that if the distance from home is greater than a certain amount, then the alternative is not feasible— or, rather, the distance to an alternative must be less than or equal to a certain number (this would b e a formal relationship in some models; in the model in this case, it reduces the search, considering fewer alternatives). This is similar to what happens in some cases when selecting a university, where schools beyond a single d ays diiv- ing distance would not be considered by most people, and, in fact, the utility function (criterion value) o f distance can start out low close to home, peak at about 70 miles (about 100 km)— say, the distance betw een Atlanta (home) and Athens, Georgia— and sharply drop off thereafter.

C hapter 2 • Foundations and T ech n o lo g ies for D ecisio n Making 7 9

Normative Models Normative m odels a re m o d e ls in w h ich th e c h o s e n alternative is d em o n stra b ly th e b e s t o f all p o ssib le altern ativ es. T o find it, th e d e c is io n m a k e r sh o u ld e x a m in e all th e altern a­ t e s a n d p ro v e th a t th e o n e s e le c te d is in d e e d th e b e s t, w h ic h is w h a t th e p e rs o n w o u ld no n n ally w an t. T h is p ro c e s s is b a sica lly optim ization. T h is is ty p ically th e g o a l o f w h at w e call p rescrip tiv e analytics (P a rt IV ). In o p era tio n a l term s, o p tim ization c a n b e a ch iev ed

in o n e o f th re e w ays:

1 . G e t th e h ig h est lev el o f g o al attain m en t fro m a g iv e n s e t o f re s o u rce s . F o r e x a m p le , w h ich altern ativ e w ill y ield th e m ax im u m p ro fit fro m an in v e stm e n t o f $ 1 0 m illion?

2 . Fin d th e altern ativ e w ith th e h ig h est ratio o f g o al attain m en t to c o s t (e .g ., profit p e r d ollar in v e ste d ) o r m ax im ize productivity.

3 . Fin d th e altern ativ e w ith th e lo w e st c o s t (o r sm allest am o u n t o f o th e r r e s o u r c e s ) that w ill m e e t a n a c c e p ta b le lev el o f goals. F o r e x a m p le , if y o u r ta sk is to s e le c t hard w are fo r a n in tran et w ith a m inim um b an d w id th , w h ic h altern ativ e w ill a c c o m p lis h this g o al a t th e le a st cost?

N orm ative d e c is io n th e o ry is b a s e d o n th e fo llo w in g assu m p tio n s o f rational

d ecision m akers:

• H um ans are e c o n o m ic b e in g s w h o s e o b je c tiv e is to m ax im ize th e a tta in m e n t o f goals; th a t is, th e d e c is io n m ak er is rational. (M ore o f a g o o d th in g [re v en u e, fun] is b etter th a n le ss; le s s o f a b a d th in g [cost, pain] is b e tte r than m o re .)

• F o r a d e cisio n -m a k in g situation, all v ia b le alternative co u rs e s o f a ctio n an d th eir c o n s e q u e n c e s , o r at le a st th e p ro b ab ility a n d th e v a lu e s o f th e c o n s e q u e n c e s , are

k now n. • D e c is io n m a k e rs h av e a n o rd e r o r p re fe re n c e that e n a b le s th e m to ran k th e d esir­

ability o f all c o n s e q u e n c e s o f th e analysis (b e s t to w o rst).

Are d ecisio n m akers really rational? T h o u g h th ere m ay b e m ajor anom alies in th e pre­ sumed rationality o f financial an d e c o n o m ic behavior, w e tak e th e view that th e y cou ld b e caused b y in co m p e te n ce , lack o f kn ow led g e, multiple g oals b ein g fram ed inadequately, m is­ understanding o f a d ecisio n m aker’s true e x p e cte d utility, an d tim e-pressure im pacts. T h ere axe o th er anom alies, o ften cau sed b y tim e pressure. F o r exam p le, Stew art (2 0 0 2 ) described i. nu m ber o f research ers w orking w ith intuitive d ecisio n m aking. I h e idea o f thinking with vour gut” is obviou sly a heuristic ap p ro ach to d ecisio n making. It w orks w ell fo r firefighters m d military person n el o n th e battlefield. O n e critical asp ect o f d ecisio n m aking in this m ode is that m any scen ario s hav e b e e n thought through in ad vance. Even w h e n a situation is new . i c a n quickly b e m atch ed to a n existing o n e on-the-fly, an d a re aso n ab le solu tion can b e obtained (through p attern recognition). Luce e t al. (2 0 0 4 ) d escribed h o w em o tio n s affect d ecision m aking, an d Pauly (2 0 0 4 ) d iscu ssed in con sisten cies in d ecisio n m aking.

W e b e lie v e th at irrationality is ca u se d b y th e facto rs liste d p reviou sly. F o r e x a m ­ p le, T v ersk y e t al. (1 9 9 0 ) investigated th e p h e n o m e n o n o f p re fe re n c e rev ersal, w h ich is a k n o w n p ro b le m in ap p ly in g th e AHP to p ro b lem s. A lso, s o m e criterio n o r p re fe re n c e m ay b e om itted fro m th e analy sis. R atn er e t al. (1 9 9 9 ) in vestigated h o w v arie ty c a n ca u se individuals to c h o o s e less-p re fe rre d o p tio n s, e v e n th o u g h th e y will e n jo y th e m less. B u t w e m ain tain th at v ariety cle a rly has valu e, is p art o f a d e c is io n m a k e r’s utility, a n d is a criterion and/or co n stra in t th at sh o u ld b e c o n s id e re d in d e c is io n m aking.

Suboptimization B y d efin ition , op tim izatio n re q u ires a d e c is io n m a k e r to co n s id e r th e im p act o f e a c h alter­ native c o u rs e o f a c tio n o n th e e n tire org an izatio n b e c a u s e a d e c is io n m ad e in o n e area m ay h av e sig n ifican t e ffe c ts (p o sitiv e o r n e g a tiv e ) o n o th e r areas. C o nsid er, fo r e x a m p le , a

8 0 Part I • D ec isio n M aking and Analytics: An Overview

m ark etin g d ep artm en t that im p lem e n ts a n e le c tro n ic c o m m e rc e (e -c o m m e r c e ) site. W ithin hou rs, o rd ers far e x c e e d p ro d u ctio n cap acity . T h e p ro d u ctio n d ep artm en t, w h ic h plans its o w n sch e d u le , ca n n o t m e e t d em an d . It m ay g e a r up fo r a s high d em an d as p o ssi­ b le . Id eally a n d in d e p e n d en tly , th e d ep artm en t sh o u ld p ro d u ce o n ly a fe w p ro d u cts in ex tre m e ly larg e q u an tities to m inim ize m an u factu rin g co sts. H ow ever, s u c h a p la n m ight result in large, co stly in v en to ries and m ark etin g d ifficu lties ca u se d b y th e la c k o f a variety o f p ro d u cts, e sp e c ia lly i f c u sto m ers start to c a n c e l o rd ers th a t are n o t m e t in a tim ely w ay. T h is situ ation illustrates th e s eq u en tial n atu re o f d e c is io n m aking.

A sy stem s p o in t o f v ie w a ss e s s e s th e im p act o f e v ery d e c is io n o n th e en tire sys­ tem . T h u s, th e m ark etin g d ep artm en t sh o u ld m a k e its plans in co n ju n ctio n w ith o th er d ep artm en ts. H o w ev er, s u ch a n a p p ro a ch m ay re q u ire a co m p lica te d , e x p e n s iv e , tim e- c o n su m in g analysis. In p ra ctice , th e MSS b u ild e r m a y c lo s e th e sy stem w ithin n arrow b o u n d arie s, co n sid erin g o n ly th e part o f th e o rg an izatio n u n d e r study (th e m ark etin g and/ o r p ro d u ctio n d ep artm ent, in this c a s e ). B y sim plifying, th e m o d e l th e n d o e s n o t in co rp o ­ rate ce rta in co m p lica te d relatio n sh ip s th a t d e s c rib e in te ra ctio n s w ith an d am o n g th e o th er d ep artm en ts. T h e o th e r d ep artm en ts c a n b e a g g re g a te d in to sim p le m o d e l co m p o n e n ts. S u c h a n a p p ro a ch is ca lle d suboptim ization.

I f a su b o p tim al d e c is io n is m ad e in o n e p art o f th e org an izatio n w ith o u t co n sid erin g th e d etails o f th e re st o f th e o rg an ization , th e n a n o p tim al so lu tio n fro m th e p o in t o f v iew o f th a t part m ay b e in ferio r fo r th e w h o le . H o w e v e r, su b o p tim izatio n m ay still b e a very p ractical a p p ro a ch to d e cisio n m akin g, a n d m an y p ro b le m s are first a p p ro a c h e d fro m this p e rsp e ctiv e. It is p o ss ib le to re a c h ten tative c o n c lu sio n s (a n d g e n erally u s a b le resu lts) b y an aly zin g o n ly a p o rtio n o f a system , w ith o u t g e ttin g b o g g e d d o w n in to o m an y details. A fter a so lu tio n is p ro p o se d , its p o ten tia l e ffe c ts o n th e rem ain in g d ep artm en ts o f th e o rg an izatio n c a n b e tested . I f n o sig n ifican t n e g a tiv e e ffe cts are fo u n d , th e so lu tio n ca n b e im p lem en ted .

Su b op tim izatio n m ay a lso ap p ly w h e n sim p lifying assu m p tio n s are u s e d in m o d ­ e lin g a s p e cific p ro b lem . T h e re m ay b e to o m an y d etails o r to o m an y d ata to in co rp o rate in to a s p e c ific d ecisio n -m a k in g situation, an d s o n o t all o f th e m are u s e d in th e m o d el. I f th e so lu tio n to th e m o d e l s e e m s re a s o n a b le , it m ay b e v alid fo r th e p ro b le m a n d thus b e ad o p ted . F o r e x a m p le , in a p ro d u ctio n d ep artm e n t, parts are o fte n p artitio n ed into A/B/C inventory cate g o rie s. G e n e rally , A item s (e .g ., larg e g e ars, w h o le a s s e m b lie s ) are e x p e n s iv e (say , $ 3 ,0 0 0 o r m o re e a c h ), built to o rd e r in sm all b a tc h e s , an d in ven toried in lo w q u an tities; C item s (e .g ., nuts, b o lts, scre w s) a re very in e x p e n s iv e (say , less th a n $2) a n d o rd e re d a n d u se d in very larg e q u antities; an d B item s fall in b e tw e e n . All A item s ca n b e h a n d led b y a d etailed s ch e d u lin g m o d e l an d p h ysically m o n ito re d clo s e ly b y m an­ ag e m e n t; B item s are g e n erally so m e w h a t a g g reg ated , th eir g ro u p in g s are sch e d u le d , and m a n a g e m e n t rev iew s th e s e parts less freq u en tly ; a n d C item s are n o t sch e d u le d b u t are sim p ly a cq u ire d o r built b a se d o n a p o licy d e fin e d b y m a n a g e m e n t w ith a sim p le e c o ­ n o m ic ord e r qu antity (E O Q ) o rd erin g sy stem that a ssu m e s co n s ta n t an n u al d em an d . T h e p o licy m ig h t b e re v iew e d o n c e a year. T h is situ atio n a p p lie s w h e n d eterm in in g all criteria o r m o d e lin g th e en tire p ro b le m b e c o m e s p ro h ib itiv ely tim e -co n su m in g o r e x p e n s iv e .

Su b op tim izatio n m ay a ls o in v o lv e sim p ly b o u n d in g th e s e a rch fo r a n op tim u m (e .g ., b y a h e u ristic) b y co n sid e rin g fe w e r criteria o r alternativ es o r b y elim in atin g large p o rtio n s o f th e p ro b le m fro m evalu ation. I f it tak es to o lo n g to s o lv e a p ro b le m , a g o o d - e n o u g h so lu tio n fo u n d alread y m ay b e u s e d a n d th e o p tim ization e ffo rt term inated.

Descriptive Models D escriptive m odels d e s c rib e th in gs as th e y a re o r as th e y are b e lie v e d to b e . T h e s e m o d e ls are typ ically m ath em atically b a se d . D escrip tiv e m o d e ls are e x tre m e ly u sefu l in D SS fo r in vestigatin g th e c o n s e q u e n c e s o f vario u s alternative co u rs e s o f a c tio n u n d er

Chapter 2 • Foundations and T ech n o lo g ies for D ecisio n Making 81

d ifferent co n fig u ra tio n s o f inputs and p ro ce s s e s. H o w ev er, b e c a u s e a d e scrip tiv e analysis ch e c k s th e p e rfo rm a n c e o f th e sy stem fo r a g iv e n s e t o f alternativ es (ra th er th an fo r all alternatives), th e re is n o g u aran te e th a t a n altern ativ e s e le c te d w ith th e aid o f d escrip tive analysis is o p tim al. In m any c a s e s , it is o n ly satisfactory.

Sim ulation is p ro b a b ly t h e m o st co m m o n d escrip tiv e m o d e lin g m eth o d . S im u latio n is th e im itatio n o f reality a n d h a s b e e n a p p lie d to m an y a re a s o f d e c is io n m aking. C o m p u ter an d v id e o g a m e s a re a fo rm o f sim u lation: A n artificial reality is cre a te d , and th e g am e p la y er liv es w ith in it. Virtual reality is a lso a fo rm o f sim u latio n b e c a u s e th e en vi­ ro n m en t is sim u lated , n o t real. A co m m o n u s e o f sim u latio n is in m an u factu rin g. Again, co n sid er th e p ro d u ctio n d ep artm e n t o f a firm w ith co m p lica tio n s ca u se d b y th e m arketin g d ep artm ent. T h e ch aracteristics o f e a c h m a ch in e in a jo b s h o p a lo n g th e su p p ly ch a in c a n b e d e s c rib e d m ath em atically. R elation sh ip s c a n b e e sta b lis h e d b a s e d o n h o w e a c h m a ch in e p h y sically runs and re la tes to o th ers. G iv e n a trial s c h e d u le o f b a tc h e s o f parts, ir is p o ss ib le to m e a su re h o w b a tc h e s flo w th ro u g h th e system and to u s e th e statistics from e a c h m a ch in e . A lternative sch e d u le s m ay th e n b e tried a n d th e statistics re co rd e d until a re a s o n a b le s ch e d u le is fo u n d . M arketing c a n e x a m in e a c c e s s a n d p u rch a s e pat­ terns o n its W e b site. S im u lation c a n b e u se d to d eterm in e h o w to stru cture a W e b site fo r im proved p e rfo rm a n ce an d to e stim ate future p u rch ases. B o th d ep artm en ts c a n th e re fo re u se prim arily e x p e rim e n ta l m o d e lin g m ethod s.

C lasses o f d escrip tiv e m o d els in clu d e th e fo llow ing:

• C o m p lex in v e n to ry d ecisio n s • E n v iro n m en tal im p act analysis • Fin an cial p lan n in g • In fo rm atio n flow • M arkov an alysis (p red ictio n s) • S ce n a rio analysis • Sim ulation (altern ativ e typ es) • T e c h n o lo g ic a l fo recastin g • W aitin g -lin e (q u e u in g ) m a n a g e m e n t

A n u m b e r o f n o n m ath e m atical d escrip tiv e m o d e ls are av ailab le fo r d e c is io n m ak ­ ing. O n e is th e co g n itiv e m ap (s e e E d en a n d A ck erm an n , 2 0 0 2 ; an d Je n k in s , 2 0 0 2 ). A co g n itiv e m ap c a n h e lp a d e c is io n m a k e r sk e tc h o u t th e im portant q u alitativ e facto rs an d ih eir cau sal re la tio n sh ip s in a m essy d ecisio n -m a k in g situation. T h is h e lp s th e d e cisio n m aker (o r d ecisio n -m a k in g g ro u p ) fo c u s o n w h at is re le v a n t an d w h a t is not, and th e m ap e v o lv e s as m o re is learn ed a b o u t th e p ro b lem . T h e m ap c a n h e lp th e d e c is io n m ak er understand issu es b etter, fo c u s b ette r, and re a c h clo su re . O n e in terestin g so ftw a re to o l fo r co g n itiv e m a p p in g is D e c is io n E x p lo rer fro m B a n x ia Softw are Ltd. (b anxia.com ; try th e d em o ).

A n o th er d escrip tiv e d ecisio n -m ak in g m o d e l is th e u s e o f narratives t o d e s c rib e a d ecisio n -m ak in g situation. A n arrativ e is a sto ry th a t h e lp s a d e c is io n m a k e r u n c o v e r th e im portant a s p e cts o f th e situ ation a n d lead s to b e tte r u n d erstan d in g a n d fram ing . T h is is extrem ely e ffe c tiv e w h e n a g ro u p is m ak in g a d e cis io n , an d it c a n lea d to a m o re c o m ­ m on v ie w p o in t, a lso c a lle d a fr a m e . Ju rie s in co u rt trials typ ically u s e n arrativ e-b ased a p p ro ach e s in re a ch in g verd icts ( s e e A llan, Fram e, an d T u rn ey , 2 0 0 3 ; B e a c h , 2 0 0 5 ; an d D en n in g , 2 0 0 0 ).

Good Enough, or Satisficing A ccord ing to S im o n ( 1 9 7 7 ) , m o st h u m an d e c is io n m ak in g , w h e th e r o rgan izatio n al o r indi­ vidual, in v o lv es a w illin g n ess to settle fo r a satisfacto ry so lu tio n , “so m e th in g less th a n th e b e s t.” W h e n satisficing, th e d e c is io n m a k e r sets up a n asp iration, a g o a l, o r a d esired

8 2 Part I • D ec isio n M aking and Analytics: An Overview

lev el o f p e rfo rm a n ce a n d th e n s e a r c h e s th e altern ativ es un til o n e is fo u n d th at a ch iev es this lev el. T h e u su al re a s o n s fo r satisficin g are tim e p re ssu re s (e .g ., d ecisio n s m ay lo se v a lu e o v e r tim e ), th e ability to a ch ie v e o p tim ization (e .g ., solv in g s o m e m o d els cou ld ta k e a really lo n g tim e, a n d re co g n itio n th a t th e m argin al b e n e fit o f a b e tte r so lu tio n is n o t w o rth th e m argin al c o s t to o b tain it (e .g ., in s e a r c h in g th e In tern et, y o u ca n lo o k at on ly s o m any W e b sites b e fo re y o u ru n o u t o f tim e a n d e n e rg y ). In s u c h a situation, the d e c is io n m a k e r is b eh a v in g rationally, th o u g h in re ality h e o r s h e is satisficing. Essentially, satisficin g is a fo rm o f su b op tim izatio n . T h e re m ay b e a b e s t so lu tio n , a n o p tim u m , b u t it w o u ld b e difficult, i f n o t im p o ssib le, to attain it. W ith a n orm ative m o d el, to o m u ch co m ­ pu tatio n m ay b e involved ; w ith a d escrip tiv e m o d e l, it m ay n o t b e p o ss ib le to e v alu ate all th e sets o f alternatives.

R e la te d to s a tisficin g is S im o n ’s id e a o f b o u n d e d ration ality . H u m an s h a v e a lim ite d c a p a c ity fo r ra tio n a l th in k in g ; th e y g e n e r a lly c o n s tr u c t a n d a n a ly z e a sim ­ p lifie d m o d e l o f a re a l situ a tio n b y c o n s id e r in g fe w e r a lte rn a tiv e s, crite ria , and/or co n s tr a in ts th a n a c tu a lly e x is t. T h e ir b e h a v io r w ith r e s p e c t to th e s im p lifie d m o d e l m ay b e ra tio n a l. H o w e v e r, th e ra tio n a l s o lu tio n fo r th e s im p lifie d m o d e l m ay n o t b e ra tio n a l fo r th e re a l-w o rld p ro b le m . R a tio n a lity is b o u n d e d n o t o n ly b y lim ita tio n s o n h u m a n p r o c e s s in g c a p a c itie s , b u t a ls o b y in d iv id u a l d iffe r e n c e s , s u c h a s a g e , e d u c a ­ tio n , k n o w le d g e , an d attitu d e s. B o u n d e d ra tio n a lity is a ls o w h y m a n y m o d e ls are d e s crip tiv e ra th e r th a n n o rm a tiv e . T h is m a y a ls o e x p la in w h y s o m a n y g o o d m a n a g e rs re ly o n in tu itio n , a n im p o rta n t a s p e c t o f g o o d d e c is io n m a k in g ( s e e Stew art, 2 0 0 2 ; and P au ly, 2 0 0 4 ).

B e c a u s e ratio n ality a n d th e u s e o f n o rm a tiv e m o d e ls le a d to g o o d d e c is io n s , it is natu ral to a s k w h y s o m a n y b a d d e c is io n s a re m a d e in p ra c tic e . In tu itio n is a critical fa c to r th at d e c is io n m a k e rs u s e in so lv in g u n stru ctu re d a n d s e m istru ctu red p ro b le m s . T h e b e s t d e c is io n m a k e rs re c o g n iz e th e tr a d e -o ff b e tw e e n th e m arg in al c o s t o f o b ta in ­ ing fu rth e r in fo rm a tio n a n d a n aly sis v e rsu s th e b e n e fit o f m a k in g a b e tte r d e c is io n . B u t s o m e tim e s d e c is io n s m u st b e m a d e q u ick ly , an d , id e a lly , th e in tu itio n o f a s e a s o n e d , e x c e lle n t d e c is io n m a k e r is c a lle d fo r. W h e n a d e q u a te p la n n in g , fu n d in g , o r in fo rm a­ tio n is n o t a v a ila b le , o r w h e n a d e c is io n m a k e r is in e x p e r ie n c e d o r ill tra in e d , d isaste r c a n strik e.

Developing (Generating) Alternatives A significant part o f th e m o d el-bu ild in g p ro ce s s is g e n eratin g alternatives. In op tim ization m o d els (s u c h as lin ear program m ing), th e alternatives m ay b e g e n e ra te d au tom atically b y the m o d el. In m o st d ecisio n situations, h ow ev er, it is n e ce s s a ry to g e n era te alternatives m anually. T h is c a n b e a len gth y p ro ce s s that in v olv es search in g a n d creativity, p erh ap s utilizing e le ctro n ic b rainsto rm ing in a GSS. It ta k e s tim e and co s ts m o n ey . Issu e s su ch as w h e n to sto p g en eratin g alternatives c a n b e very im portant. T o o m an y alternatives c a n b e detrim ental to th e p ro cess o f d e cisio n m aking. A d e c is io n m a k e r m ay su ffer fro m inform a­ tio n overload .

G e n e ra tin g alternativ es is h e av ily d e p e n d e n t o n th e availability an d c o s t o f inform a­ tio n a n d re q u ires e x p e rtis e in th e p ro b le m are a. T h is is th e least form al a s p e c t o f problem solving. A lternatives c a n b e g e n e r a te d a n d e v a lu a te d u sin g h eu ristics. T h e g e n e ra tio n o f alternatives fro m e ith er individuals o r g ro u p s c a n b e s u p p o rte d b y e le c tro n ic b rain sto rm ­ ing softw are in a W e b -b a s e d GSS.

N ote th at th e s e a rch fo r a lternativ es u su ally o c c u rs after th e criteria fo r e valu atin g the alternativ es are d eterm in ed . T h is s e q u e n c e c a n e a se th e s e a r c h fo r a lternativ es and re d u ce th e e ffo rt in v o lv ed in evalu atin g th em , b u t identifying p o ten tial alternatives c a n so m e tim e s aid in id entify ing criteria.

Chapter 2 • Foundations and T ech n o lo g ies for D ecisio n Making

T h e o u tc o m e o f e v ery p ro p o s e d altern ativ e m u st b e estab lish ed . D e p e n d in g o n w h e th e r th e d e cisio n -m a k in g p ro b le m is classified a s o n e o l certainty, risk, o r u n certainty, d ifferent m o d e lin g a p p ro a ch e s m ay b e u se d (s e e D ru m m ond , 2 0 0 1 ; an d K o ller, 2 0 0 0 ). T h e se a re d iscu sse d in C h ap ter 9.

Measuring Outcomes T h e v a lu e o f a n altern ativ e is e v a lu a te d in te rm s o f g o a l attain m en t. S o m e tim e s an o u tc o m e is e x p r e s s e d d irectly in te rm s o f a g o a l. F o r e x a m p le , p ro fit is a n o u tc o m e , p ro fit m a x im iz a tio n is a g o a l, a n d b o th a re e x p r e s s e d in d o lla r term s. An o u tc o m e s u ch as cu sto m e r s a tisfa ctio n m a y b e m e a su re d b y th e n u m b e r o f co m p la in ts, b y th e lev el o f loyalty to a p ro d u ct, o r b y ratin gs fo u n d th ro u g h su rveys. Id ea lly , a d e c is io n m a k e r w o u ld w a n t to d e a l w ith a sin g le g o a l, b u t in p ra ctice , it is n o t u n u su a l to h a v e m u ltip le g o als (s e e B a rb a -R o m e ro , 2 0 0 1 ; an d K o k sa la n an d Z io n ts, 2 0 0 1 ). W h e n g ro u p s m ak e d e cisio n s, e a c h g ro u p p articip an t m ay h av e a d iffe re n t ag e n d a . F o r e x a m p le , e x e c u tiv e s m ight w a n t to m a x im iz e p ro fit, m ark etin g m ight w a n t to m a x im iz e m a rk e t p e n e tra tio n , o p era tio n s m ig h t w a n t to m in im ize c o s ts , an d s to c k h o ld e rs m ight w a n t to m a x im iz e the b o tto m lin e . T y p ica lly , th e s e g o a ls co n flict, s o s p e c ia l m u ltip le-criteria m e th o d o lo g ie s h av e b e e n d e v e lo p e d to h a n d le th is. O n e s u c h m e th o d is th e AHP. W e w ill stud y AHP in C h ap te r 9-

Risk All d e cis io n s are m a d e in a n in h eren tly u n stab le e n v iro n m en t. T h is is d u e to th e m any u n p re d icta b le e v e n ts in b o th th e e c o n o m ic a n d p h ysical en v iron m en ts. S o m e risk (m e a s ­ ured as p ro b ab ility ) m ay b e d u e to internal organ izatio n al e v e n ts, s u ch a s a valu ed e m p lo y e e q u ittin g o r b e c o m in g ill, w h e re a s o th ers m ay b e d u e to natu ral d isasters, s u ch as a h u rrican e. A side fro m th e h u m an toll, o n e e c o n o m ic a s p e c t o f H u rricane Katrina w as th at th e p rice o f a g allo n o f g a so lin e d o u b le d o v ern ig h t d u e to u n certain ty in th e port cap ab ilities, re fin in g, an d p ip e lin e s o f th e so u th e rn U n ited States. W h a t c a n a d e cisio n m ak e r d o in th e f a c e o f s u c h instability?

In g e n e r a l, p e o p le h a v e a te n d e n c y to m e a s u re u n c e rta in ty an d risk b a d ly . Purd y ( 2 0 0 5 ) said th a t p e o p le te n d to b e o v e r c o n fid e n t a n d h a v e a n illu s io n o f c o n tr o l in d e c is io n m a k in g . T h e re s u lts o f e x p e rim e n ts b y A d am G o o d ie a t th e U n iv ersity o f G e o rg ia in d ic a te th a t m o st p e o p le a re o v e r c o n fid e n t m o s t o f th e tim e (G o o d ie , 2 0 0 4 ). T h is m ay e x p la in w h y p e o p le o fte n f e e l th a t o n e m o re p u ll o f a s lo t m a c h in e w ill d e fin ite ly p a y o ff.

H ow ever, m e th o d o lo g ie s fo r h an d lin g e x tre m e u n certain ty d o exist. F o r e x a m p le , Y a k o v ( 2 0 0 1 ) d e s c rib e d a w a y to m a k e g o o d d ecisio n s b a s e d o n very little in fo rm atio n , using a n in fo rm a tio n g a p th e o ry an d m e th o d o lo g y ap p ro a ch . A side fro m estim atin g th e p o ten tial utility o r valu e o f a p articu lar d e c is io n ’s o u tc o m e , th e b e s t d e c is io n m a k e rs are c a p a b le o f accu ra te ly estim atin g th e risk a s s o cia te d w ith th e o u tc o m e s th at resu lt fro m m ak in g e a c h d e cis io n . T h u s, o n e im portant ta sk o f a d e c is io n m a k e r is to attrib u te a lev el o f risk to th e o u tc o m e a sso cia ted w ith e a c h p o ten tial alternative b e in g co n s id e re d . So m e d ecisio n s m ay le a d to u n a c c e p ta b le risks in term s o f s u c c e s s a n d c a n th e re fo r e b e dis­ card e d o r d is co u n te d im m ediately.

In s o m e c a s e s , s o m e d e c is io n s a re a ssu m e d to b e m a d e u n d e r c o n d itio n s o f c e r ­ tain ty sim p ly b e c a u s e th e e n v iro n m e n t is a ssu m e d to b e s ta b le . O th e r d e c is io n s are m ad e u n d e r c o n d itio n s o f u n certa in ty , w h e re risk is u n k n o w n . Still, a g o o d d e c is io n m a k e r c a n m a k e w o rk in g e stim a tes o f risk. A lso, th e p r o c e s s o f d e v e lo p in g BI/DSS in v o lv e s le a rn in g m o re a b o u t th e situ atio n , w h ic h le a d s to a m o re a c c u r a te a s s e s s m e n t o f t h e risks.

8 4 Part I • D ec isio n Making and Analytics: An O verview

Scenarios A scen ario is a s ta te m e n t o f a ssu m p tio n s a b o u t th e o p e ra tin g e n v iro n m e n t o f a p articu ­ lar s y stem a t a g iv e n tim e ; th a t is, it is a narrative d e s crip tio n o f th e d ecisio n -situ a tio n settin g. A s c e n a rio d e s c rib e s th e d e c is io n an d u n c o n tro lla b le v a ria b le s a n d p a ia m e te rs fo r a s p e c ific m o d e lin g situ ation . It m a y a lso p ro v id e th e p ro c e d u re s a n d co n strain ts for

th e m o d e lin g . . S ce n a rio s originated in th e theater, a n d th e te rm w a s b o rro w e d for w ar g am in g and

larg e-scale sim ulations. Sce n ario p lan n in g and an aly sis is a D SS to o l that can cap tu re a w h o le range o f possibilities. A m an ag er c a n co n stru ct a serie s o f sce n a rio s (i.e ., w hat-ir c a s e s ), p erfo rm co m p u terized analyses, a n d learn m o re a b o u t th e system a n d d ecisio n ­ m ak in g p ro b le m w h ile analyzing it. Ideally, th e m a n a g e r can identify an e x c e lle n t, p o ssibly op tim al, solu tion to th e m o d el o f th e p ro blem .

S ce n a rio s a re e sp e cia lly h e lp fu l in sim u latio ns a n d w h a t-if an aly ses. In b o th ca ses, w e c h a n g e sce n a rio s a n d e x a m in e th e results. F o r e x a m p le , w e c a n c h a n g e th e an ticip ated d em an d fo r h osp italizatio n (a n input v ariab le fo r p la n n in g ), thu s creatin g a n e w scen ario . T h e n w e c a n m e asu re th e a n ticip ated ca sh flo w o f th e h osp ital fo r e a c h scen ario .

S ce n a rio s p lay a n im portant ro le in d e c is io n m ak in g b e c a u s e they:

• H elp id en tify o p p o rtu n ities and p ro b le m areas • P rov id e flex ibility in p lan n in g • Id en tify th e lead in g e d g e s o f c h a n g e s that m an a g e m e n t sh o u ld m o n ito r • H e lp validate m ajo r m o d e lin g assu m p tio n s • A llow th e d e c is io n m a k e r to e x p lo r e th e b e h a v io r o f a sy stem th ro u g h a m o d el • H elp to c h e c k th e sensitivity o f p ro p o s e d so lu tio n s to c h a n g e s in th e en viron m en t,

as d e s crib e d b y th e scen a rio

Possible Scenarios T h e re m ay b e th o u san d s o f p o ss ib le s ce n a rio s fo r e v e iy d e c is io n situation. H o w ev er, th e

fo llo w in g are e sp e cia lly u sefu l in p ractice:

• T h e w o rst p o ss ib le scen a rio • T h e b e s t p o ss ib le scen a rio • T h e m o st lik e ly scen a rio • T h e a v e ra g e s ce n a rio

T h e sce n a rio d eterm in e s th e c o n te x t o f t h e analysis t o b e p erfo rm ed .

Errors in Decision Making T h e m o d e l is a critical c o m p o n e n t in th e d ecisio n -m a k in g p ro c e s s , b u t a d e c is io n m aker m ay m a k e a n u m b e r o f errors in its d ev e lo p m e n t a n d u se. V alidating th e m o d e l b e fo re it is u s e d is critical. G ath e rin g th e right am o u n t o f in fo rm atio n , w ith th e right lev el o f p re ci­ s io n a n d a ccu ra cy , to in co rp o ra te in to th e d e cisio n -m a k in g p ro c e s s is a lso critical. Saw yer ( 1 9 9 9 ) d e s crib e d “th e s e v e n d ead ly sin s o f d e c is io n m a k in g ,” m o st o f w h ic h are b eh av io r

o r in fo rm atio n related .

SECTION 2 . 5 REVIEW QUESTIONS

1 . D e fin e op tim ization and co n trast it w ith su boptim ization . 2 . C o m p are th e n orm ative a n d d escrip tive a p p r o a c h e s to d e c is io n m aking. 3 . D e fin e r a tio n a l d ec isio n m akin g . W h a t d o e s it really m e a n to b e a rational d ecisio n

m aker? 4 . W h y d o p e o p le e x h ib it b o u n d e d rationality w h e n solv in g problem s?

Chapter 2 • Foundations and T ech n o lo g ies for D ecisio n Making 8 5

6 S o m e “e rro rs” in d e c is io n m ak in g c a n b e attributed to th e n o tio n o f d e c is io n m aking fro m th e gut. E x p la in w h at is m e a n t b y this a n d h o w s u ch erro rs ca n h a p p e n .

5 . D e f in e sce n a rio . H o w is a s c e n a r io u s e d in d e c i s io n m a k in g ?

2.6 D ECISIO N M AKIN G : THE CHOICE PH ASE I n o ic e is th e critical a c t o f d e c is io n m aking. T h e c h o ic e p h a s e is th e o n e in w h ic h th e actual d e cisio n a n d th e co m m itm e n t to fo llo w a ce rta in co u rse o f a c tio n are m ad e . T h e -vDundary b e tw e e n th e d esig n a n d c h o ic e p h a s e s is o fte n u n cle a r b e c a u s e ce rta in activi­ tie s c a n b e p e rfo rm e d during b o th o f th e m an d b e c a u s e th e d e c is io n m a k e r c a n return freq u en tly fro m c h o ic e activities to d esig n activities (e .g ., g e n e ra te n e w altern ativ es w h ile perform ing a n e v a lu a tio n o f e x istin g o n e s ). T h e c h o ic e p h a s e in clu d es th e s e a r c h for, e v alu atio n o f, a n d re co m m e n d a tio n o f a n a p p ro p riate so lu tio n to a m o d el. A s o lu tio n to a m o d el is a s p e c ific s e t o f valu es fo r the d e c is io n v ariab les in a s e le c te d alternativ e. C h o ice s ca n b e e v alu ate d a s to th e ir viability a n d profitability.

N ote th at so lv in g a m o d e l is n o t th e sam e a s solv in g th e p ro b le m th e m o d e l re p re sen ts. T h e solu tion to th e m o d e l y ield s a re co m m e n d e d so lu tio n to th e p ro b lem . T h e p ro b le m is co n sid ered so lv e d o n ly i f th e re co m m e n d e d so lu tio n is s u cce ssfu lly im p lem en ted .

Solving a d ecisio n -m a k in g m o d e l involves s e a rch in g fo r a n a p p ro p ria te co u rse o f actio n . S e a rch a p p ro a ch e s in clu d e a n a ly tic a l te ch n iq u e s ( i.e ., s o l v i n g a form u la) a lg o rith m s ( i.e ., s te p -b y -ste p p ro ce d u re s ), h eu ristics (i.e ., ru les o f th u m b ), a n d b lin d sea rch es (i.e ., sh o o tin g in th e dark, ideally in a lo g ica l w a y ). T h e s e a p p r o a c h e s are

exam in ed in C h ap ter 9- . , E a c h a lte rn a tiv e m u st b e e v a lu a te d . I f a n altern ativ e h a s m u ltip le g o a ls , th e y m u st

all b e e x a m in e d a n d b a la n c e d a g a in st e a c h o th e r. Sensitivity analysis is u s e d to d e te r­ m in e th e r o b u s tn e s s o f a n y g iv e n altern ativ e; slig h t c h a n g e s in th e p a ra m e te rs sh o u ld ideally le a d to slig h t o r n o c h a n g e s in th e a ltern a tiv e ch o s e n . W hat-if an alysis is used to e x p lo r e m a jo r c h a n g e s in th e p a ra m e te rs . G o a l s e e k in g h e lp s a m a n a g e r d eter­ m in e v a lu e s o f th e d e c is io n v a r ia b le s to m e e t a s p e c ific o b je c tiv e . All this is d is cu ss e d

in C h ap ter 9.

SECTION 2 . 6 REVIEW QUESTIONS

1 . E x p la in th e d iffe re n ce b e tw e e n a p rin cip le o f c h o ic e an d th e actu al c h o ic e p h a se o f

d e cisio n m akin g. 2 . W h y d o s o m e p e o p le cla im that th e c h o ic e p h a s e is th e p o in t in tim e w h e n a d e cisio n

is really m ade? 3. H o w c a n sensitivity analysis h e lp in th e c h o ic e phase?

2.7 D ECISIO N M AKIN G : THE IM PLEM ENTATION PHASE In The P rince, M achiavelli astutely n o te d so m e 5 0 0 y ears a g o that th ere w a s ‘ n o th in g m o re difficult to carry ou t, n o r m o re d oubtful o f su cce ss, n o r m o re dangerou s to h an d le than to initiate a n e w ord e r o f things.” T h e im p lem entation o f a p ro p o se d solu tion to a p ro b le m is, in effe ct th e initiation o f a n e w ord er o f things o r th e introd u ction o f ch an g e. A nd ch an ge m ust b e 'm a n a g e d . U ser ex p e cta tio n s m ust b e m an ag ed as part o f ch an g e m anagem ent.

T h e d e fin itio n o f im p lem en tation is s o m e w h a t c o m p lic a te d b e c a u s e im p le m e n ta tio n is a lo n g in v o lv e d p ro ce s s w ith v a g u e b o u n d a rie s. Sim p listically, th e im plem entation p h a s e in v olv es p u ttin g a re c o m m e n d e d s o lu tio n to w o rk , n o t n e c e s s a rily im p lem e n tin g a co m p u te r sy stem . M any g e n e r ic im p le m e n ta tio n issu es, s u ch a s re s is ta n c e to change^ d e g r e e o f s u p p o rt o f to p m a n a g e m e n t, an d u s e r training , are im p ortan t in d e a lin g w ith

8 6 Part I • D ec isio n Making and Analytics: An O verview

in fo rm a tio n sy stem s u p p o rte d d e c is io n m ak in g . In d e e d , m a n y p re v io u s te c h n o lo g y - re la ted w a v e s (e .g ., b u sin e ss p r o c e s s re e n g in e e rin g (B P R ), k n o w le d g e m an ag e m e n t, e tc .) h a v e fa c e d m ix e d resu lts m ain ly b e c a u s e o f c h a n g e m a n a g e m e n t ch a lle n g e s and issu es. M a n a g e m e n t o f c h a n g e is alm o st a n e n tire d is c ip lin e in itself, s o w e r e c o g n iz e its im p o rta n ce an d e n c o u ra g e th e re a d ers to fo c u s o n it in d e p e n d en tly . Im p le m en ta tio n also in clu d e s a th o ro u g h u n d e rstan d in g o f p r o je c t m a n a g e m e n t. Im p o rta n ce o f p r o je c t m a n ­ a g e m e n t g o e s far b e y o n d a n a ly tics, so th e la st fe w y e a rs h av e w itn e sse d a m a jo r g ro w th in ce rtifica tio n p ro gram s fo r p r o je c t m an ag ers. A v e ry p o p u la r ce rtifica tio n n o w is P ro ject M an ag e m e n t P ro fe s s io n a l (P M P ). S e e p m i . o r g fo r m o re details.

Im p le m en ta tio n m u st a lso involve c o lle c tin g an d analyzing d ata to learn fro m th e prev iou s d ecisio n s an d im p ro v e th e n e x t d ecisio n . A lth o u g h analysis o f d ata is usu ally co n d u cte d to id entify th e p ro b le m and/or t h e solu tio n , an aly tics sh o u ld a lso b e e m p lo y e d in th e fe e d b a c k p ro ce ss. T h is is e sp e cia lly t a i e fo r an y p u b lic p o lic y d e cisio n s. W e n e e d to b e su re that th e d ata b e in g u s e d fo r p ro b le m id e n tificatio n is valid. S o m etim e s p e o p le find this o u t o n ly a fte r th e im p lem e n tatio n p h ase.

T h e d ecisio n -m a k in g p ro c e s s , th o u g h c o n d u cte d b y p e o p le , c a n b e im p ro v ed w ith co m p u te r su p p ort, w h ich is th e s u b je c t o f th e n e x t sectio n .

SECTION 2 . 7 REVIEW QUESTIONS

1 . D e fin e im plem entation. 2 . H o w ca n D SS su p p o rt th e im p lem e n tatio n o f a d ecisio n ?

2.8 H O W D E C IS IO N S A R E SU PPO R T ED In C h a p te r 1, w e d is cu ss e d th e n e e d fo r co m p u te riz e d d e c is io n s u p p o rt an d b riefly d e s c rib e d s o m e d e c is io n aids. H e re w e re la te s p e c ific te c h n o lo g ie s to th e d e c is io n ­ m a k in g p r o c e s s ( s e e F ig u re 2 .2 ). D a ta b a se s , d ata m arts, a n d e s p e c ia lly d ata w a r e h o u s e s are im p o rta n t te c h n o lo g ie s in su p p o rtin g all p h a s e s o f d e c is io n m a k in g . T h e y p ro v id e th e d ata that drive d e c is io n m ak in g .

Support for the Intelligence Phase T h e prim ary requ irem en t o f d ecisio n supp ort for th e in tellig en ce p h ase is th e ability to scan e xte rn al and internal inform ation so u rces fo r op p ortu nities an d p ro blem s a n d to interpret w h at th e scan n in g discovers. W e b to o ls an d so u rce s are e x trem ely usefu l fo r environm ental

P h ase ANN MIS

Data Mining, OLAP E S, E R P

E S S , E S , SCM CRM, ERP, K V S Management Science ANN

D S S E S

E S S , E S KMS, E R P

CRM SCM

FIG U R E 2 .2 DSS Support.

Chapter 2 • Foundations and T ech n o lo g ies fo r D ec isio n Making

scanning. W e b b ro w se rs p rovide u sefu l front en d s fo r a variety o f to ols, fro m OLAP to data m ining and data w areh o u ses. D ata so u rces c a n b e internal o r external. Internal so u rce s may b e a cce ssib le via a co rp o ra te intranet. E xternal so u rces are m any an d varied.

D e c is io n su p p ort/ B I te c h n o lo g ie s c a n b e v e ry h elp fu l. F o r e x a m p le , a d ata w a r e ­ h o u s e c a n su p p o rt th e in te llig e n c e p h a s e b y co n tin u o u sly m o n ito rin g b o th in te rn al a n d e x te rn a l in fo rm atio n , lo o k in g fo r e a rly sig n s o f p ro b le m s and o p p o rtu n itie s th ro u g h a W e b -b a s e d e n te rp ris e in fo rm a tio n p o rtal (a ls o ca lle d a d a s h b o a rd ). Sim ilarly, (a u to m a tic) d ata (a n d W e b ) m in in g (w h ic h m ay in clu d e e x p e rt sy stem s [ES], CRM, g e n e tic alg o rith m s, n eu ral n e tw o rk s, a n d o th e r an aly tics s y ste m s) a n d (m a n u a l) OLAP a ls o su p p o rt th e in te l­ lig e n ce p h a s e b y id e n tify in g re la tio n sh ip s am o n g activ ities an d o th e r facto rs. G e o g ra p h ic inform ation sy stem s (G IS ) ca n b e u tilized e ith e r a s s ta n d -a lo n e sy stem s o r in te g ra te d w ith th e se sy stem s s o th a t a d e c is io n m a k e r c a n d ete rm in e o p p o rtu n itie s and p ro b le m s in a spatial s e n s e . T h e s e re la tio n sh ip s c a n b e e x p lo ite d fo r co m p etitiv e ad v an tag e ( e .g ., CRM identifies cla ss e s o f cu sto m e rs to a p p r o a c h w ith s p e c ific p ro d u cts an d s e r v ic e s ). A KMS c a n b e u s e d to id en tify sim ilar p a st situ atio n s an d h o w th e y w e re h an d led . G S S c a n b e u se d to s h a re in fo rm a tio n and fo r b rain sto rm in g . As s e e n in C h ap te r 14, e v e n c e ll p h o n e and G P S d ata c a n b e ca p tu re d to c re a te a m icro -v ie w o f cu sto m e rs an d th e ir h ab its.

A n oth er a s p e c t o f id entifying in tern al p ro b lem s and cap ab ilitie s in v olv es m o n ito rin g d ie cu rren t statu s o f o p era tio n s. W h e n so m e th in g g o e s w ro n g , it c a n b e id en tified q uickly and th e p ro b le m c a n b e so lv ed . T o o ls s u ch as b u sin e ss activity m o n ito rin g (B A M ), b u si­ n ess p ro ce s s m an a g e m e n t (B P M ), an d p ro d u ct life -cy cle m an a g e m e n t (PLM) p ro v id e su ch cap ab ility to d e c is io n m ak ers. B o th ro u tin e and a d h o c rep o rts c a n aid in th e in te llig e n ce p h ase. F o r e x a m p le , regu lar rep o rts c a n b e d esig n ed to assist in th e p ro b lem -fin d in g activity b y co m p a rin g e x p e c ta tio n s w ith cu rre n t an d p ro je c te d p e rfo rm an ce . W e b -b a s e d OLAP to o ls are e x c e lle n t at this task. So a re visu alization to o ls a n d e le c tro n ic d o cu m en t m an ag em en t system s.

E xp e rt sy stem s (E S ), in con trast, c a n re n d er a d v ice reg ard ing th e n atu re o f a p ro b ­ lem , its cla ssifica tio n , its serio u sn e ss, an d th e like. ES c a n ad vise o n th e su itab ility o f a solution a p p r o a c h a n d th e lik e lih o o d o f s u cce ssfu lly solv in g th e p ro b lem . O n e o f th e primary are a s o f ES s u c c e s s is in terp retin g in form ation a n d d iag n o sin g p ro b le m s. T h is capability c a n b e e x p lo ite d in th e in te llig e n ce p h a s e . E v e n in tellig en t ag en ts c a n b e u sed 10 identify op p o rtu n ities.

M u ch o f th e in form ation u s e d in s e e k in g new' o p p o rtu n ities is q u alitativ e, o r soft. This in d icates a h ig h lev el o f u n stru ctu red n ess in th e p ro b le m s, thu s m ak in g D SS q u ite useful in th e in te llig e n ce p h ase.

T h e In te rn e t a n d a d v a n ce d d atab ase te c h n o lo g ie s h a v e c re a te d a glut o f d ata and inform ation a v ailab le to d e c is io n m ak ers— s o m u ch that it c a n d etract fro m th e quality and s p e e d o f d e c is io n m aking. It is im portant to re co g n iz e s o m e issu es in u sin g d ata and analytics to o ls fo r d e c is io n m akin g. First, to p arap h rase b a s e b a ll g reat V in Scully, “data should b e u se d th e w a y a d run k u s e s a lam p p o st. F o r su p p ort, n o t fo r illu m in atio n .” It ^ e sp ecia lly true w h e n th e fo cu s is o n u n d erstan d in g th e p ro b lem . W e sh o u ld re co g n iz e tftat n o t all th e data th a t m ay h elp un derstand th e p ro b le m is a v ailab le. T o q u o te Einstein, ■“Not everyth in g th a t co u n ts c a n b e co u n te d , a n d n o t everyth in g that c a n b e co u n te d cou nts.” T h e re m ig h t b e o th e r issu es th at h av e to b e re co g n iz e d as w ell.

Support fo r the Design Phase H i e d esig n p h a s e in v olv es g e n era tin g alternative co u rs e s o f a ctio n , d iscu ssin g th e criteria far c h o ic e s a n d th e ir relativ e im p o rtan ce , and fo re ca stin g th e future c o n s e q u e n c e s o f ■«ang v arious altern ativ es. S ev eral o f th e s e activities ca n u s e stand ard m o d els p ro v id e d b y a D SS (e .g ., fin an cial an d fo reca stin g m o d e ls, a v ailab le as a p p le ts). A lternatives fo r stru c- c_red p ro b lem s c a n b e g e n e ra te d th ro u g h th e u s e o f e ith e r stand ard o r s p e c ia l m o d els.

H o w e v e r th e g e n era tio n o f a lternativ es fo r c o m p le x p ro b le m s req u ires e x p e rtis e that c a n b e p ro v id e d o n ly b y a hu m an, b rain sto rm in g softw are, o r a n ES. OLAP an d d ata m m m g softw are are q u ite u sefu l in identifying re latio n sh ip s th a t c a n b e u se d m m od els. M ost DSS h av e quantitative analysis cap ab ilitie s, and a n internal ES c a n assist w ‘th qUal‘ta” e " od s as w e ll as w ith th e e x p e rtis e re q u ired in s e le ctin g q u an titative analysis an d fo re ca stin g m o d e ls A KMS sh o u ld ce rtain ly b e co n su lte d to d eterm in e w h e th e r s u ch a p ro b le m has b e e n e n c o u n te re d b e fo r e o r w h e th e r th e re are e x p e rts o n h an d w h o ca n u n d erstan d in g a n d an sw ers. CRM sy stem s, re v e n u e m a n a g e m e n t sy stem s, ERP, and K , system s softw are a r e u sefu l in that th e y p ro v id e m o d e ls o f b u s i n e s s p ro c e s s e s th a t can ^ e s assu m p tio n s an d scen a rio s. I f a p ro b le m re q u ires b rain sto rm in g to h e lp id en tify im portant issu es an d o p tio n s, a G SS m ay p ro v e h elp fu l. T o o ls th a t pro v id e co g n itiv e m apping; can a lso h e lp C o h e n e t al. (2 0 0 1 ) d e s crib e d sev eral W e b -b a s e d to o ls that p rovide d e cisio n su p p ort, m ainly in the d esig n p h a s e , b y providing m o d e ls an d re p o rtin g o f alternative results E a c h o f th e ir ca s e s has sa v e d m illions o f d ollars an n u ally b y utilizing th e s e to ols. Su ch D SS are h e lp in g e n g in e e rs in p ro d u ct d esig n as w e ll as d e c is io n m ak ers solving

b u sin e ss p ro b lem s.

Support for the Choice Phase In ad dition to providing m o d els that rapidly identify a b e s t o r g o o d -en o u g h alternative a D SS c a n supp ort th e c h o ic e p h ase throu gh w h at-if an d go al-seek m g analyses. D ifferent scen ario s can b e tested fo r th e sele cte d o p tio n to re in fo rce th e final decision. Again, a KMS h elp s identify sim ilar past e x p e rien ces; CRM, ERP, an d SCM system s are u se d to test th e im pacts o f d ecisio n s in establishing their value, lead ing to an intelligent ch o ice . An ES can b e u sed to assess th e desirability o f ce rta in solutions as w e ll as to re co m m en d an ap p ro p n - ate solution. If a group m ak e s a d ecisio n , a G SS c a n p rovide supp ort to lead to consensus.

Support fo r the Im plem entation Phase T h is is w h e re “m ak in g th e d e c is io n h a p p e n ” o ccu rs. T h e D SS b e n e fits pro v id ed during im p lem e n ta tio n m ay b e as im p ortan t as o r e v e n m o re im p o rtan t th a n th o se in th e e a r i p h a se s. D SS c a n b e u s e d in im p lem e n tatio n activities s u ch a s d e c is io n co m m u n icatio n ,

ex p la n a tio n , a n d ju stification . f Im p le m en ta tio n -p h a se D SS b e n e fits a re partly d u e to the vivid ness an d d etail o f

an aly ses and rep orts. F o r e x a m p le , o n e c h ie f e x e c u tiv e o ffice r (C E O ) gives e m p lo y e e s and e xte rn al p arties n o t o n ly th e a g g re g ate fin an cial g o a ls and c a s h n e e d s fo r th e n e a r term b u t a lso th e calcu latio n s, interm ed iate results, an d statistics u sed m d eterm in g th e ag g re g ate figures. In ad d ition to co m m u n icatin g th e fin an cial g o als u n am b ig u o u sly th e C E O sign als o th er m e ssa g es. E m p lo y e e s k n o w th a t th e CEO has th o u g ro“ 8 . assu m p tio n s b e h in d th e fin an cial g o als an d is serio u s a b o u t th eir im p o rtan ce an d attain­ ability B a n k e rs a n d d irecto rs are sh o w n th at th e C E O w as p e rso n ally i n v o k e d m a n a ­ lyzing c a s h n e e d s and is a w a re o f an d re s p o n s ib le fo r th e im p lications o f th e financing re q u e sts p re p a re d b y th e fin an ce d ep artm en t. E a c h o f th e se m e ssa g es im p ro ves d e cisio n

im p lem e n ta tio n in s o m e w ay . . , , As m e n tio n e d e arlie r, re p o rtin g sy stem s an d o th e r to o ls variou sly la b e le d a s B

BPM KMS EIS, ERP, CRM, an d SCM a re all u sefu l in tra ck in g h o w w ell an im p lem en tatio is w o rk in g . G SS is u sefu l fo r a te a m to c o lla b o ra te in estab lish in g im p lem e n tatio n e ffe c ­ tiv en ess. F o r e x a m p le , a d e c is io n m ight b e m a d e to g e t rid o f u n p ro fitab le cu sto m e is . An e ffe ctiv e CRM c a n id entify cla ss e s o f cu sto m ers to g e t rid o f, id entify th e im p act o f d oing

so a n d th e n v erify that it really w o rk e d th a t w ay. All p h ases o f the decision-m aking p ro cess c a n b e su p p orted b y im proved com m u nica­

tion through collaborative com puting via G SS and KMS. Com puterized system s ca n facilitate com m unication b y help ing p e o p le e xp lain and justify th e ir suggestions and opinions.

D ec isio n Making and Analytics: An Overview

Chapter 2 • Foundations and T ech n o lo g ies fo r D ecision Making

D e cis io n im p lem e n ta tio n c a n also b e s u p p o rte d b y ES. An ES c a n b e u s e d a s a n advi­ sory sy stem reg ard in g im p lem e n tatio n p ro b lem s (s u c h as h an d lin g re sista n ce to c h a n g e ). Finally, a n ES c a n p ro v id e training that m ay s m o o th th e c o u rs e o f im p lem en tatio n .

Im pacts alo n g th e v alu e ch ain , th o u g h rep o rted b y an EIS throu gh a W e b -b a s e d enterprise in form ation portal, are typically id entified b y BAM, BPM , SCM, a n d ERP system s. CRM system s rep ort a n d up d ate internal reco rd s, b ase d o n th e im p acts o f th e im p lem en ta- oon. T h e s e inputs a re th e n u s e d to identify n e w p ro b lem s an d o p p ortu nities— a return to d ie in tellig en ce p h ase.

SECTION 2 . 8 REVIEW QUESTIONS

1. D escrib e h ow D SS/BI tech n olog ies an d tools c a n aid in e a ch p h ase o f d ecisio n making.

2. D e s c r ib e h o w n e w te c h n o lo g ie s c a n pro v id e d ecisio n -m a k in g support.

Now th at w e h av e stu d ied h o w te c h n o lo g y c a n assist in d e cisio n m akin g, w e study so m e details o f d e c is io n su p p o rt sy stem s (D S S ) in th e n e x t tw o s ectio n s.

2,9 D EC ISIO N SU PPO R T S Y S T E M S : C A P A B IL IT IE S The early d efin itio n s o f a D SS id en tified it as a sy stem in te n d e d to su p p o rt m an ag erial d ecisio n m ak ers in sem istru ctu red an d unstructured d e c is io n situations. D SS w e re m e an t

: b e ad ju n cts to d e c is io n m ak e rs, e x te n d in g th eir c ap ab ilitie s b u t n o t re p la cin g th e ir ju d g­ m ent. T h e y w e r e a im e d a t d ecisio n s that re q u ired ju d g m e n t o r a t d e cis io n s th at co u ld n o t r e co m p le te ly su p p o rte d b y algorithm s. N ot sp e cifica lly stated b u t im plied in th e early d efinitions w as th e n o tio n that th e system w o u ld b e co m p u te r b a se d , w o u ld o p e ra te inter-

o n lin e, a n d p re fe rab ly w o u ld h a v e grap h ical ou tp u t ca p a b ilitie s, n o w sim p lified ~rz. b row sers an d m o b ile d ev ices.

- DSS Application D SS is typically b u ilt to su p p o rt th e so lu tio n o f a ce rta in p ro b le m o r to e v alu ate an

icp o rtu n ity . T h is is a k e y d iffe ren ce b e tw e e n D SS and B I ap p licatio n s. In a v e ry strict ease, business intelligence (B I ) system s m o n ito r situ ation s and id entify p ro b le m s and/

o p p ortu n ities, u sin g analytic m eth o d s. R e p o rtin g plays a m a jo r ro le in B I; th e u s e r g en erally m ust id entify w h e th e r a p articu lar situ ation w arrants atten tion , a n d th e n analyti-

i m eth o d s c a n b e ap p lied . A gain, alth o u g h m o d e ls an d data a c c e s s (g e n e ra lly throu gh d iia w a re h o u se ) a re in clu d e d in B I, D SS typically h av e th e ir o w n d a ta b a se s a n d are

E '-d o p e d to s o lv e a s p e c ific p ro b le m o r s e t o f p ro b lem s. T h e y a re th e re fo r e calle d 3SS applications.

Fo rm ally , a D S S is a n a p p r o a c h (o r m e th o d o lo g y ) fo r s u p p o rtin g d e c is io n m ak in g . k ^ 5 e s a n in te ra c tiv e , fle x ib le , a d a p ta b le c o m p u te r -b a s e d in fo rm a tio n s y s te m (C B IS ) s p e c ia lly d e v e lo p e d fo r s u p p o rtin g th e s o lu tio n to a s p e c ific u n stru c tu re d m a n a g e - m=-- p ro b le m . It u s e s d ata, p ro v id e s a n e a s y u s e r in te rfa c e , a n d c a n in c o r p o r a te th e a o sio n m a k e r’s o w n in sig h ts. In ad d itio n , a D SS in c lu d e s m o d e ls a n d is d e v e lo p e d

possibly b y e n d u s e r s ) th r o u g h a n in te ra c tiv e a n d itera tiv e p r o c e s s . It c a n s u p p o r t all o f d e c is io n m a k in g a n d m ay in c lu d e a k n o w le d g e c o m p o n e n t. F in a lly , a D SS

i b e u s e d b y a s in g le u s e r o r c a n b e W e b b a s e d fo r u s e b y m a n y p e o p le a t sev eral ( h c E l O O S .

B e c a u s e th e re is n o c o n s e n s u s o n e x a ctly w h at a D SS is, th ere is o b v io u sly n o a g re e- y - r o n th e stan d ard ch aracteristics an d cap ab ilitie s o f D SS. T h e cap ab ilitie s in F ig u re 2.3

in s t i t u t e a n id eal set, s o m e m e m b ers o f w h ic h are d e s crib e d in th e d efin itio n s o f D SS j o d illustrated in th e a p p licatio n ca ses.

T h e k e y ch a ra cteristics a n d cap ab ilitie s o f D SS (a s sh o w n in Figu re 2 .3 ) are:

9 0 Part I • D e c is io n Making and Analytics: An Overview

Support individuals

and groups

Modeling and analysis Interdependent

or sequential decisionsD ecision S u p p o rt

S y s t e m s (D SS}Ease of development by end users

Humans control the process Support variety

of decision processes and styles

Effectiveness and efficiency Interactive,

ease of use

Data access

Support intelligence

design, choice, and implementation

Adaptable and flexible

Stand-alone, integration, and

Web-based

Semistructured or unstructured

problems Support

managers at all levels

FIGURE 2.3 Key Characteristics and Capabilities of DSS.

1 . Su p p ort fo r d e c is io n m ak e rs, m ainly in sem istru ctu red a n d u n stru ctu red situations, b y b rin g in g to g e th e r h u m an ju d g m en t and co m p u te rize d in form ation. S u c h p ro b ­ lem s ca n n o t b e so lv e d (o r c a n n o t b e so lv e d c o n v e n ie n tly ) b y o th e r co m p u terized system s o r th ro u g h u s e o f stan d ard q u antitative m e th o d s o r to ols. G e n e rally , th e se p ro b lem s g ain structure as th e D SS is d e v e lo p e d . E v e n s o m e stru ctu red p ro b lem s

have b e e n so lv e d b y DSS. 2 . Su p p ort fo r all m an ag erial lev els, ran g in g fro m to p e x e c u tiv e s to lin e m anagers. 3 Su p p ort fo r individuals a s w e ll a s g ro u p s. L ess-stru ctu red p ro b lem s o fte n req u ire th e

in v olv em en t o f individuals fro m d iffe ren t d ep artm e n ts a n d org an izatio n al lev els or e v e n fro m d ifferent organizations. D SS su p p o rt virtual te am s th ro u g h co llab o rativ e W e b to o ls. D SS h av e b e e n d e v e lo p e d to su p p o rt individual a n d g ro u p w o rk , a s w ell a s to su p p o rt individ ual d e c is io n m ak in g an d g ro u p s o f d e c is io n m ak ers w o rk in g

so m e w h a t in d ep en d en tly. 4 . Su p p ort fo r in te rd e p e n d e n t and/or se q u e n tia l d e cisio n s. T h e d e cis io n s m ay b e m ade

o n c e , sev eral tim es, o r rep eated ly. 5 . Su p p ort in all p h a s e s o f th e d ecisio n -m a k in g p ro ce ss: in te llig e n ce , d esig n , ch o ic e ,

and im p lem en tation . 6 . S u p p o rt fo r a variety o f d ecisio n -m a k in g p ro c e s s e s an d styles. 7 . T h e d e c is io n m a k e r shou ld b e reactiv e, a b le to c o n fro n t ch a n g in g co n d itio n s q u ic k y,

and a b le to ad ap t th e D S S to m e e t th e s e ch a n g e s. D SS are fle x ib le , s o u se rs c a n add, d e le te , c o m b in e , ch a n g e , o r rearran g e b a s ic e le m e n ts . T h e y are a ls o fle x ib le in that th e y c a n b e read ily m o d ified to s o lv e o th er, sim ilar p ro b lem s.

C hapter 2 • Foundations a n d T ech n o lo g ies for D ecisio n Making

8 . U ser-frien d lin ess, stro n g g rap h ical cap ab ilitie s, an d a natu ral la n g u a g e interactive h u m a n -m a c h in e in te rface c a n greatly in c re a se th e e ffe ctiv e n e s s o f D SS. M o st n e w D SS ap p lica tio n s u s e W e b -b a s e d in te rfa ce s o r m o b ile p latfo rm in te rface s.

9 . Im p ro v e m e n t o f th e e ffe ctiv e n e s s o f d e c is io n m ak in g (e .g ., a ccu ra cy , tim elin ess, q u ality) ra th er th a n its e ffic ie n c y (e .g ., th e c o s t o f m ak in g d e c is io n s ). W h e n D SS are d e p lo y e d , d e cisio n m aking o fte n ta k e s lo n g e r, b u t th e d ecisio n s a re b etter.

10 . T h e d e c is io n m a k e r h a s co m p le te co n tro l o v e r all step s o f th e d ecisio n -m a k in g p ro c e s s in solv in g a p ro b lem . A D SS s p e cifica lly aim s to su p p ort, n o t to re p la ce , th e d e c is io n m ak er.

1 1 . E n d u se rs are a b le to d ev elo p and m o d ify s im p le system s b y th e m se lv es. Larger system s c a n b e b u ilt w ith a ssistan ce fro m in form ation sy stem (IS ) sp ecialists. S p re a d s h e e t p a c k a g e s h av e b e e n utilized in d ev elo p in g sim p le r sy stem s. OLAP and d ata m in in g so ftw are, in c o n ju n c tio n w ith data w a reh o u se s, e n a b le users to build fairly la rg e , c o m p le x DSS.

12 . M odels a re g e n era lly utilized to an aly ze d ecisio n -m a k in g situations. T h e m o d ­ e lin g cap a b ility e n a b le s e x p e rim e n ta tio n w ith d ifferen t strateg ies u n d e r d ifferen t c o n fig u ra tio n s.

1 3 . A cce ss is p ro v id e d to a variety o f d ata so u rce s, fo rm ats, an d ty p es, in clu d in g GIS, m u ltim ed ia, a n d o b je c t-o rie n te d data.

14 . T h e D SS can b e e m p lo y e d as a stand -alone to o l u se d b y an individual d ecisio n m ak er in o n e lo ca tio n o r distributed throughout a n organization and in sev eral organizations a lo n g th e supp ly ch ain . It c a n b e integrated w ith o th er D SS and/or ap p licatio n s, and it c a n b e distributed internally an d externally, u sin g netw orkin g an d W e b tech n olog ies.

T h e s e k e y D SS ch aracteristics and cap ab ilitie s allo w d e c is io n m a k e rs to m ake b etter, m o re c o n s is te n t d e cis io n s in a tim ely m ann er, and th e y are p ro v id e d b y the m ajor DSS co m p o n e n ts , w h ic h w e w ill d e scrib e a fte r d iscu ssin g vario u s w a y s o f classifying D SS (n e x t).

SECTION 2 . 9 REVIEW QUESTIONS

1 . List th e k e y ch aracteristics a n d cap a b ilitie s o f D SS. 2 . D e s c r ib e h o w p ro vid ing su p p ort to a w o rk g ro u p is d ifferen t fro m p ro v id in g su p p ort

to g ro u p w o rk . E x p la in w h y it is im portant to d ifferen tiate th e s e c o n c e p ts.

3 . W h at k in d s o f D SS c a n en d u se rs d ev elo p in sp read sh eets? 4. W h y is it s o im p ortan t to in clu d e a m o d e l in a DSS?

2.10 D S S C L A SS IF IC A T IO N S D SS ap p licatio n s h av e b e e n classified in sev eral d ifferent w ays (s e e P o w e r, 2 0 0 2 ; P o w er and Sharda, 2 0 0 9 ). T h e d esign p ro cess, as w e ll as th e o p eratio n and im p lem entation o f DSS, d ep en d s in m any cases o n th e typ e o f D SS involved. H ow ever, re m e m b er that n o t every DSS fits n eatly into o n e category. M ost fit into th e classification provid ed b y th e A ssociation fo r Info rm ation System s Sp ecial In terest G ro u p o n D e cis io n Su p p ort System s (AIS SIG D SS). W e discuss this classification b u t a lso point o u t a fe w o th er attem pts at classifying DSS.

The AIS SIGD SS Classification for DSS T h e AIS SIG D SS (ais.site-ym .com /group/SIG D SS) h a s a d o p te d a c o n c is e classificatio n s c h e m e fo r D SS th at w a s p ro p o s e d b y P o w e r (2 0 0 2 ). It in clu d e s th e fo llo w in g cate g o rie s:

• C o m m u n icatio n s-d riv en a n d g ro u p D SS (G SS) • D ata-d riv e n D SS

9 2 Part I • D ecisio n M aking and Analytics: An O verview

• D o cu m e n t-d riv e n D SS • K n o w led g e -d riv en D SS, data m ining , an d m a n a g e m e n t ES ap p licatio n s • M o d el-d riv en D SS

T h e r e m ay a ls o b e hybrid s th at co m b in e tw o o r m o re ca te g o rie s. T h e s e are calle d c o m p o u n d DSS. W e d iscu ss th e m a jo r ca te g o rie s n ext.

C O M M U N IC A T IO N S -D R IV E N A N D G R O U P D S S C o m m u n ica tio n s -d riv e n a n d g ro u p D SS (G S S ) in c lu d e D SS th a t u se c o m p u te r, co lla b o r a tio n , a n d c o m m u n ic a tio n te c h n o lo g ie s to s u p p o rt g ro u p s in ta s k s th a t m ay o r m ay n o t in c lu d e d e c is io n m a k in g . E ssen tially, all D SS th a t s u p p o rt an y k in d o f g ro u p w o rk fa ll in to this ca te g o ry . T h e y in clu d e th o se th at s u p p o rt m e e tin g s , d e s ig n co lla b o r a tio n , a n d e v e n su p p ly ch a in m a n a g e m e n t. K n o w le d g e m a n a g e m e n t sy stem s (K M S) that a re d e v e lo p e d a ro u n d co m m u n itie s that p ra c tic e c o lla b o r a tiv e w o rk a ls o fall in to th is ca te g o ry . W e d iscu ss th e s e in m o re d etail in la te r ch a p te rs.

D A T A -D R IV E N D S S D ata-d riv en D SS a re prim arily in v o lv e d w ith d ata a n d p ro cessin g th e m in to in form ation a n d p re sen tin g th e in fo rm atio n to a d e c is io n m ak er. M any DSS d e v e lo p e d in OLAP an d rep ortin g an alytics softw are sy stem s fall in to this cate g o ry . T h e re is m inim al em p h a sis o n th e u se o f m ath em atical m o d e ls.

In this ty p e o f D SS, th e d a ta b a s e o r g a n iz a tio n , o f te n in a d ata w a r e h o u s e , p lays a m a jo r r o le in th e D SS stru ctu re. E arly g e n e r a tio n s o f d a ta b a s e -o r ie n te d D SS m ain ly u s e d th e r e la tio n a l d a ta b a s e c o n fig u ra tio n . T h e in fo r m a tio n h a n d le d b y re la tio n a l d a ta b a s e s te n d s to b e v o lu m in o u s, d e s crip tiv e , a n d rigid ly stru ctu red . A d a ta b a s e - o r ie n te d D SS fe a tu re s s tro n g re p o rt g e n e r a tio n a n d q u e ry c a p a b ilitie s . In d e e d , this is p rim arily th e c u rre n t a p p lic a tio n o f th e to o ls m a rk e d u n d e r th e B I u m b re lla o r u n d e r th e la b e l o f re p o rtin g / b u sin e ss an a ly tics. T h e c h a p te r s o n d ata w a r e h o u s in g an d b u s in e s s p e rfo rm a n c e m a n a g e m e n t (B P M ) d e s c r ib e s e v e r a l e x a m p le s o f th is c a te g o ry o f D SS.

D O C U M EN T-D R IV EN D S S D o cu m en t-d riv en D SS re ly o n k n o w le d g e co d in g , analysis, search , and retrieval fo r d ecisio n supp ort. T h e y essen tially in clu d e all D SS that are text b a se d . M ost KMS fall in to this categ ory . T h e s e D SS a ls o h av e m inim al em p h asis o n utiliz­ ing m ath em atical m o d els. F o r e x a m p le , a system th at w e built fo r th e U.S. Army s D efen se A m m unitions C en ter falls in this category . T h e m ain o b je c tiv e o f d ocu m en t-d riven D SS is to provid e su p p ort fo r d e cisio n m ak in g u sin g d o cu m en ts in v arious form s: oral, w ritten, an d m ultim edia.

K N O W LE D G E -D R IV E N D S S , D A T A M IN IN G , A N D M A N A G E M E N T E X P E R T S Y S T E M S A P P L IC A T IO N S T h e s e D SS involve th e a p p licatio n o f k n o w le d g e te ch n o lo g ie s to address sp e cific d e c is io n supp ort n e ed s. Essentially, all artificial in te llig e n c e -b a s e d D SS fall into this categ ory . W h e n s y m b o lic storag e is utilized in a D SS, it is g en erally in this category. ANN a n d ES a re in clu d e d h e re . B e c a u s e th e b en efits o f th e s e intelligent DSS o r know ledge- b a s e d DSS can b e larg e, organ ization s h av e inv ested in them . T h e s e D SS are utilized in the creation o f a u to m a te d d ecision -m akin g systems, as d e s c rib e d in C h ap ter 12. T h e b a sic idea is that rules are u se d to au to m ate th e d ecisio n -m ak in g p ro ce ss. T h e s e ru les are b asically e ith er a n ES o r stru ctured like o n e . T h is is im portant w h e n d ecisio n s m ust b e m ad e quickly, as in m an y e -c o m m e r c e situations.

M O D E L-D R IV E N D S S T h e m a jo r e m p h a s e s o f D SS th a t a re prim arily d e v e lo p e d around o n e o r m o re (la rg e -sca le / co m p le x ) op tim izatio n o r sim u latio n m o d e ls ty p ically inclu de sig n ifican t activities in m o d e l fo rm u lation , m o d e l m a in te n a n ce , m o d el m an ag em en t

Chapter 2 • Foundations and T ech n o lo g ies for D ecisio n Making

in distributed co m p u tin g e n v iro n m en ts, a n d w h a t-if analyses. M any la rg e-sc a le a p p lic a ­ tions fall in to this ca te g o ry . N otable e x a m p le s in clu d e th o s e u sed b y P ro c te r & G a m b le (Farasyn e t al., 2 0 0 8 ), H P (O la v s o n an d F iy , 2 0 0 8 ), an d m an y oth ers.

T h e fo c u s o f s u c h system s is o n u sin g th e m o d e l(s ) to o p tim ize o n e o r m o re o b je c ­ tives (e .g ., p ro fit). T h e m o st co m m o n e n d -u se r to o l fo r D SS d ev e lo p m e n t is M icroso ft E xcel. E x c e l in clu d e s d o z e n s o f statistical p a ck a g e s, a lin e a r p rogram m ing p a c k a g e (S o lv er), an d m a n y fin an cial a n d m a n a g e m e n t s c ie n c e m o d e ls. W e w ill study th e s e in m o re d etail in C h ap te r 9- T h e s e D SS ty p ically c a n b e g ro u p e d u n d e r th e n e w la b e l o f prescriptive an alytics.

CO M PO U N D D S S A co m p o u n d , o r h yb rid , D SS in clu d e s tw o o r m o re o f th e m a jo r ca t­ e g o rie s d e s c rib e d e a rlie r. O fte n , a n ES c a n b e n e fit b y u tilizin g s o m e o p tim izatio n , an d clearly a d ata-d riv e n D SS c a n fe e d a la rg e -sc a le o p tim iz a tio n m o d e l. S o m e tim e s d o c u ­ m e n ts a re critical in u n d erstan d in g h o w to in te rp re t th e results o f visu alizin g d ata fro m a d ata-d riven D SS.

A n e m e rg in g e x a m p le o f a c o m p o u n d D SS is a p ro d u ct o ffe re d b y W o lfram A lp h a (w o lfram alp h a.co m ). It c o m p ile s k n o w le d g e fro m o u ts id e d a ta b a s e s , m o d e ls , a lg o ­ rithm s, d o c u m e n ts , a n d s o o n to p ro v id e a n s w e rs to s p e c ific q u e s tio n s . F o r e x a m p le , it c a n fin d an d a n a ly z e c u rre n t d ata fo r a s to c k a n d c o m p a r e it w ith o th e r s to c k s . It c a n a lso te ll y o u h o w m a n y c a lo rie s y o u w ill b u rn w h e n p e rfo rm in g a s p e c ific e x e r c i s e o r th e s id e e ffe c ts o f a p a rticu la r m e d ic in e . A lth o u g h it is in e a rly s ta g e s as a c o lle c tio n o f k n o w le d g e c o m p o n e n ts fro m m a n y d iffe re n t a re a s, it is a g o o d e x a m p le o f a c o m p o u n d D SS in g e ttin g its k n o w le d g e fro m m a n y d iv erse s o u rc e s a n d atte m p tin g t o s y n th e s iz e it.

Other DSS Categories M any o th e r p ro p o s a ls h a v e b e e n m ad e to classify D SS. P erh a p s th e first form al attem p t w a s b y A lter (1 9 8 0 ) . S e v eral o th e r im p o rtan t ca te g o rie s o f D SS in clu d e ( 1 ) in stitu tion al and ad h o c D SS; ( 2 ) p e rso n a l, gro u p , a n d organ izatio n al su p p ort; ( 3 ) individual su p p ort system versu s G SS; a n d ( 4 ) cu sto m -m a d e sy stem s v ersu s re ad y -m ad e sy stem s. W e d iscu ss s o m e o f th e s e n ext.

IN S T IT U T IO N A L A N D A D H O C D S S Institutional DSS (s e e D o n o v a n an d M ad n ick , 1 9 7 7 ) d ea l w ith d ecisio n s o f a recu rrin g natu re. A ty p ical e x a m p le is a p o rtfo lio m a n a g e m e n t system (P M S), w h ic h h a s b e e n u se d b y sev eral larg e b a n k s fo r su p p o rtin g in v e stm e n t d ecisio n s. An institu tionalized D SS c a n b e d e v e lo p e d a n d re fin ed as it e v o lv e s o v e r a n u m b e r o f y e ars, b e c a u s e th e D SS is u s e d rep eated ly to so lv e id en tical o r sim ilar p ro b ­ lem s. It is im p ortan t to re m e m b e r that a n institutional D SS m ay n o t b e u s e d b y e v e ry o n e in a n o rg an ization ; it is th e recu rrin g n a tu re o f th e d ecisio n -m a k in g p ro b lem th a t d eter­ m in es w h e th e r a D SS is institutional v ersu s ad h o c .

Ad h o c DSS d e a l w ith s p e c ific p ro b lem s that a re u su ally n e ith er an ticip ated n o r re cu r­ ring. A d h o c d e cis io n s o fte n in v o lv e strateg ic p lan n in g issu es an d s o m e tim e s m a n a g e m e n t co n tro l p ro b lem s. Ju s tify in g a D SS that w ill b e u s e d o n ly o n c e o r tw ice is a m a jo r issu e in D SS d ev elo p m en t. C o u n tless a d h o c D SS ap p licatio n s h a v e e v o lv e d in to institu tional DSS. E ith er th e p ro b le m recu rs a n d th e sy stem is re u s e d o r oth ers in th e o rg an izatio n have sim ilar n e e d s th a t c a n b e h a n d led b y th e fo rm erly a d h o c D SS.

Custom-Made System s Versus Ready-Made Systems M any D SS a re c u s to m m a d e fo r individ ual u se rs and o rg a n iz a tio n s. H o w e v e r, a c o m ­ p a ra b le p r o b le m m a y e x is t in sim ilar o rg a n iz a tio n s. F o r e x a m p le , h o s p ita ls , b a n k s, a n d u n iv e rs itie s s h a re m a n y sim ilar p ro b le m s . Sim ilarly, c e rta in n o n ro u tin e p ro b le m s in a fu n ctio n a l a re a ( e .g ., fin a n c e , a c c o u n tin g ) c a n re p e a t th e m s e lv e s in t h e s a m e fu n c tio n a l

9 4 Part I • D ec isio n M aking and Analytics: An O verview

a re a o f d iffe re n t a re a s o r o rg a n iz a tio n s. T h e r e fo r e , it m a k e s s e n s e to b u ild g e n e r ic D SS th a t c a n b e u s e d (s o m e tim e s w ith m o d ific a tio n s ) in se v e ra l o rg a n iz a tio n s. S u c h D SS a re c a lle d r e a d y -m a d e an d a re s o ld b y vario u s v e n d o rs (e .g ., C o g n o s , M icroStrategy , T e ra d a ta ). E sse n tia lly , th e d a ta b a s e , m o d e ls, in te rfa c e , a n d o th e r su p p o rt fe a tu re s are b u ilt in: Ju s t a d d a n o r g a n iz a tio n ’s d ata an d lo g o . T h e m a jo r OLAP a n d a n a ly tics v e n d o rs p ro v id e D SS te m p la te s fo r a v arie ty o f fu n c tio n a l a re a s , in clu d in g fin a n c e , re a l e sta te, m ark e tin g , a n d a c c o u n tin g . T h e n u m b e r o f re a d y -m a d e D SS c o n tin u e s to in c re a se b e c a u s e o f th e ir flex ib ility a n d lo w c o s t. T h e y a r e ty p ica lly d e v e lo p e d u sin g In te rn e t te c h n o lo g ie s fo r d a ta b a s e a c c e s s a n d c o m m u n ic a tio n s , and W e b b ro w s e rs fo r in te rfa ce s. T h e y a ls o re a d ily in c o rp o ra te OLAP a n d o th e r e a s y -to -u s e D SS g e n e ra to rs.

O n e c o m p lic a tio n in te rm in o lo g y resu lts w h e n a n o rg a n iz a tio n d e v e lo p s a n in stitu tion al sy stem b u t, b e c a u s e o f its stru ctu re, u s e s it in a n ad h o c m an n er. An o rg an i­ z a tio n c a n b u ild a larg e d ata w a r e h o u s e b u t th e n u s e OLAP to o ls to q u e ry it an d p erfo rm ad h o c an aly sis to s o lv e n o n re cu rrin g p ro b le m s. T h e D SS e x h ib its th e traits o f ad h o c an d institu tional sy stem s a n d a lso o f c u sto m a n d re a d y -m a d e sy stem s. S ev eral ER P, CRM, k n o w le d g e m a n a g e m e n t (K M ), a n d SCM c o m p a n ie s o ffe r D SS a p p lica tio n s o n lin e . T h e s e k in d s o f sy stem s c a n b e v ie w e d as re a d y -m a d e, a lth o u g h typ ically th e y re q u ire m o d ifica­ tio n s (s o m e tim e s m a jo r) b e fo r e th e y c a n b e u s e d effe ctiv e ly .

SECTION 2 .1 0 REVIEW QUESTIONS

1 . List th e D SS classificatio n s o f th e A1S SIG D SS.

2 . D e fin e d ocu m en t-d riv en DSS. 3 . List th e cap ab ilitie s o f institu tional D SS an d a d h o c DSS. 4 . D e fin e th e term rea d y -m a d e DSS.

2.11 C O M PO N EN T S OF D E C IS IO N SU P PO R T S Y S T E M S A D SS a p p lica tio n c a n b e c o m p o s e d o f a d ata m a n a g e m e n t su b sy stem , a m o d e l m an­ a g e m e n t su b sy stem , a user in te rface su b sy stem , an d a k n o w le d g e -b a s e d m an ag em en t su b system . W e sh o w th e s e in Figure 2.4.

FIG U R E 2 .4 Schematic V iew o f DSS.

Chapter 2 • Foundations and T ech n o lo g ies for D ecision M aking 9 5

FIGURE 2.5 Structure of the Data Management Subsystem.

The Data M anagem ent Subsystem T h e data m an a g e m e n t su b sy stem in clu d es a d a ta b a se that co n ta in s re le v a n t d ata fo r th e situ atio n an d is m a n a g e d b y so ftw are c a lle d th e database m an agem en t system (DBMS).2 T h e d ata m a n a g e m e n t su b sy stem c a n b e in te rc o n n e c te d w ith th e co rp o ra te data w areh ou se, a re p o sito ry fo r co rp o ra te re le v a n t d ecisio n -m a k in g d ata. U sually, th e data are s to red o r a c c e s s e d via a d atab ase W e b server. T h e data m a n a g e m e n t su b sy stem is c o m p o s e d o f th e fo llo w in g elem en ts:

• D SS d atab ase • D a ta b a se m a n a g e m e n t system • D ata d irecto ry • Q u e ry facility

T h e s e e le m e n ts a re s h o w n sch e m a tica lly in F ig u re 2.5 (in th e sh a d e d a r e a ). T h e figu re ils o s h o w s th e in te ra ctio n o f th e d ata m a n a g e m e n t su b sy stem w ith th e o t h e r parts o f th e DSS, a s w e ll as its in te ra ctio n w ith se v e ra l d ata s o u rc e s. M any o f th e B l o r d escrip tiv e analytics a p p lica tio n s d eriv e th e ir stren g th fro m th e d ata m a n a g e m e n t sid e o f th e su b sy s­ tem s. A p p licatio n C ase 2 .2 p ro v id e s a n e x a m p le o f a D SS that fo c u s e s o n d ata.

The Model M anagem ent Subsystem T h e m o d e l m a n a g e m e n t su b sy stem is th e c o m p o n e n t th a t in clu d es fin an cial, statistical, m a n a g e m e n t s c ie n c e , o r o th e r quantitative m o d e ls th a t pro v id e th e sy stem ’s analytical cap ab ilitie s a n d ap p ro p riate so ftw are m an ag e m e n t. M o d elin g la n g u a g e s fo r b u ild in g c u s ­ to m m o d e ls a re a lso in clu d ed . T h is so ftw are is o fte n ca lle d a m odel b ase m anagem ent

2DBMS is used as both singular and plural (system and system s), as are many other acronyms in this text.

9 6 Part I • D ec isio n Making and Analytics: An O verview

Application Case 2.2 Station Casinos Wins by Building Customer Relationships Using Its Data Station C asin o s is a m a jo r p ro vid er o f g am in g for Las V e g a s -a r e a resid en ts. It o w n s a b o u t 2 0 p ro p e r­ tie s in N evada a n d o th e r states, em p lo y s o v e r 12 ,0 0 0 p e o p le , a n d h a s re v e n u e o f o v e r $1 b illion.

Station C asin o s w a n te d to d e v e lo p an in -d ep th v ie w o f e a c h cu stom er/ gu est w h o v isited C asino Statio n p ro p e rtie s. T h is w o u ld p erm it th e m to b e t­ te r u n d erstan d cu sto m e r trend s a s w e ll as e n h a n c e th e ir o n e -to -o n e m ark etin g fo r e a c h g u est. T h e c o m ­ p a n y e m p lo y e d th e T erad ata w a re h o u s e to d ev elo p th e “T o ta l G u e s t W o rth ” solu tio n . T h e p ro je c t u se d u se d A p rim o R e latio n sh ip M anager, In fo rm atica, and C o g n o s to ca p tu re , an aly ze, an d s e g m e n t cu stom ers. A lm o st 5 0 0 d iffe ren t data s o u rc e s w e re in teg rated to d e v e lo p th e fu ll v ie w o f a cu sto m er. As a result, the co m p a n y w as a b le to realize th e fo llo w in g b en e fits:

• Custom er seg m en ts w e re e xp an d ed from 14 (originally) to 160 segm ents s o as to b e a b le to target m o re sp ecific p rom otion s to e a ch segm ent.

• A 4 p e rc e n t to 6 p e r c e n t in cre a se in m onthly s lo t profit.

• Slot p ro m otion co sts w e re red u ced b y $1 million (from $13 m illion p e r m onth) b y b etter targeting th e cu stom er segm ents.

• A 14 p e rc e n t im p ro v e m en t in g u e st retention. • In c re a s e d n e w -m e m b e r a cq u isitio n b y 160

p ercen t. • R e d u ctio n in d ata e rro r rates fro m as high as

8 0 p e rc e n t to le s s th a n 1 p ercen t. • R edu ced th e tim e to analyze a cam p aign’s e ffe c­

tiveness fro m alm ost 2 w e e k s to ju st a few hours.

Q u e s t i o n s f o r D i s c u s s i o n

1. W h y is this d e c is io n su p p o rt sy stem classified as a d ata-fo cu sed DSS?

2. W h at w e re s o m e o f th e b e n e fits fro m im p lem en t­ ing this solution?

Source: Teradata.com, “No Limits: Station Casinos Breaks the Mold on Customer Relationships,” teradata.com/case-studies/ S t a t i o n - C a s i n o s - N o - L i m i t s - S t a t i o n - C a s i n o s - B r e a k s - t h e - M o l d - o n -C u s to m e r-R e la tio n s h ip s -E x ec u tiv e -S u m m a ry -e b 6 4 lO (accessed February 2013)-

system (MBMS). T h is c o m p o n e n t c a n b e c o n n e c te d to co rp o ra te o r e x te rn a l storage o f m o d els. M o d el so lu tio n m e th o d s an d m a n a g e m e n t sy stem s a re im p lem e n te d in W e b d ev e lo p m e n t system s (s u c h as Ja v a ) to ru n o n a p p lica tio n serv ers. T h e m o d e l m a n a g e ­ m e n t su b sy stem o f a D SS is c o m p o s e d o f th e fo llo w in g elem en ts;

• M o d el b a se • M BMS • M o d elin g lan g u ag e • M odel d irecto ry • M o d el e x e c u tio n , in teg ratio n , and co m m a n d p ro c e s s o r

T h e s e e le m e n ts and th e ir in te rfa ce s w ith o th e r D SS c o m p o n e n ts are s h o w n in Fig u re 2.6. At a h igh er lev el than build ing b lo ck s, it is im portant to co n sid er the d ifferent typ es o f

m o d els and solu tion m eth o d s n e e d e d in th e DSS. O fte n at th e start o f d ev elop m en t, th e re is so m e s e n se o f th e m o d el typ es to b e incorp o rated , b u t this m ay ch a n g e as m o re is learn ed ab o u t th e d e cisio n p ro blem . So m e D SS d ev elo p m en t system s inclu de a w id e variety o f co m ­ p o n en ts (e .g ., Analytica fro m Lumina D e cisio n System s), w h ere a s others h av e a sin gle o n e (e .g ., Lindo). O ften, th e results o f o n e ty p e o f m o d el c o m p o n e n t (e .g ., fo recastin g ) are u sed as inp u t to an o th er (e .g ., p ro d u ction sched u ling ). In s o m e ca ses, a m o d elin g language is a co m p o n e n t that g e n erates input to a solver, w h ere a s in o th e r ca ses, th e tw o are com bin ed .

B e c a u s e D SS d eal w ith sem istru caired o r un structured p ro b lem s, it is o ften necessary to cu stom ize m o d els, using program m ing to o ls an d languages. So m e e x a m p le s o f th e se are .NET Fram ew ork lang u ages, C++, an d Ja v a . OLAP softw are m ay a lso b e u se d to w o rk w ith m o d els in data analysis. E v e n lan gu ages fo r sim ulation s u ch as A rena and statistical p a ck ­ ages s u ch as th o se o f SPSS o ffe r m o d elin g to o ls d e v e lo p e d through th e u s e o f a proprietary

C hapter 2 • Foundations and T ech n o lo g ies for D ecision Making

FIGURE 2.6 Structure of the Model Management Subsystem.

program m ing lang u age. For sm all an d m ed iu m -sized D SS o r fo r less co m p le x o n e s , a spread­ sheet (e .g ., E x c e l) is usually used. W e w ill u se E x ce l fo r m any k e y e x a m p le s in this b o o k . A pplication C ase 2 .3 d escrib e s a sp re ad sh ee t-b ased DSS. H ow ever, using a sp read sh eet for m o d eling a p ro b le m o f an y significant size p resen ts p ro blem s w ith d o cu m en tatio n and error diagnosis. It is v e iy difficult to d eterm ine o r understand nested , c o m p le x relationships in sp read sh eets crea ted b y s o m e o n e else. This m akes it difficult to m odify a m o d el built b y s o m e o n e e lse . A related issu e is the in cre ase d lik elih o o d o f errors cre e p in g in to th e form u­ las. W ith all th e eq u atio n s ap p earin g in th e fo rm o f cell re fe re n ces, it is ch allen g in g to figure ou t w h ere a n e rro r m ight b e . T h e s e issu es w e re ad dressed in a n early g e n eratio n o f DSS d ev elo p m en t softw are that w as available o n m ainfram e com p u ters in th e 1980s. O n e su ch product w as calle d Interactive Financial P lanning System (IFP S). Its d ev elo p e r. Dr. G erald W agner, th e n re le a se d a d esk to p softw are called P lanners Lab. P lanners Lab includes th e follow ing co m p o n e n ts: (1 ) a n easy -to-u se algeb raically orien ted m od el-bu ild ing language and (2 ) a n e asy -to -u se state-of-the-art o p tio n fo r visualizing m o d el output, s u ch as answ ers to w h at-if a n d g o a l s e e k q u estio ns to analyze results o f ch a n g e s in assum ptions. T h e co m ­ b in ation o f th e s e co m p o n e n ts e n a b le s b u sin ess m anagers and analysts to b u ild , review , and ch allen g e th e assum p tions th at underlie d ecisio n -m akin g scen arios.

P lan n ers L ab m a k e s it p o ss ib le fo r th e d e c is io n m ak ers to “p lay ” w ith assu m p tions to re flect altern ativ e v iew s o f th e fu ture. E v ery P lan n e rs Lab m o d e l is a n a s s e m b la g e o f a ssu m p tio n s a b o u t th e future. A ssu m ptions m ay c o m e fro m d a tab ase s o f h isto rical p e r­ fo rm a n ce , m a rk e t re s e a rch , a n d th e d e c is io n m a k e rs’ m in d s, to n a m e a fe w so u rce s. M ost a ssu m p tio n s a b o u t th e fu ture c o m e fro m th e d e c is io n m a k e rs’ a ccu m u la te d e x p e rie n c e s in th e fo rm o f o p in io n s.

T h e resu ltin g c o lle c tio n o f e q u a tio n s is a P lan n e rs Lab m o d e l th at tells a r e a d a b le story f o r a p a r t ic u la r scen ario . P lan n e rs Lab lets d e c is io n m ak ers d e s c rib e th e ir plans in th eir o w n w o rd s and w ith th e ir o w n assu m p tio n s. T h e p ro d u ct’s ra iso n d ’etre is that a sim ulator sh o u ld facilitate a co n v e rsa tio n w ith th e d e c is io n m a k e r in th e p ro c e s s o f

Part I • D ec isio n M aking and Analytics: An Overview

Application Case 2.3 S N A P DSS Helps O neN et M ak e Telecom m unications R ate Decisions T e le co m m u n ica tio n s n e tw o rk serv ices to ed u cation al institutions a n d g o v ern m e n t entities a re typically pro v id ed b y a m ix o f private and p u b lic organiza­ tions. M any states in th e U n ited States h av e o n e o r m o re state a g e n c ie s that a re re sp o n sib le fo r providing n e tw o rk serv ices to s c h o o ls , co lle g e s, an d o th er state ag e n cie s. O n e e x a m p le o f s u ch a n a g e n c y is O n eN et in O k lah o m a. O n e N e t is a d ivision o f th e O k lah o m a State R eg en ts fo r H ig h er E d u cation an d o p e ra te d in co o p e ra tio n w ith th e O ffice o f State Fin an ce.

U sually a g e n c ie s s u ch as O n e N et o p e ra te as a n e n te rp rise -ty p e fund. T h e y m ust re c o v e r th eir co s ts th ro u g h b illin g th e ir c lie n ts and/or b y justifying ap p ro p riatio n s d irectly fro m th e state legislatu res. T h is c o s t re c o v e ry sh o u ld o c c u r th ro u g h a p ricing m e ch a n ism th a t is e fficie n t, sim p le to im p lem en t, a n d e q u ita b le. T h is p ricing m o d e l ty p ically n e e d s to re co g n iz e m an y facto rs: c o n v e rg e n c e o f v o ic e , data, a n d v id eo traffic o n th e sa m e infrastructure; diver­ sity o f u s e r b a s e in term s o f e d u ca tio n a l institutions, sta te a g e n c ie s , a n d s o o n ; diversity o f ap p licatio n s in u s e b y state clie n ts, fro m e-m ail to v id e o c o n fe r­ e n c e s , IP te le p h o n in g , an d d istan ce learn in g; re co v ­ ery o f cu rre n t c o s ts , a s w ell a s p lan n in g fo r up grad es

a n d fu ture d ev elo p m en ts; a n d lev erag e o f th e sh ared infrastru ctu re to e n a b le fu rth er e c o n o m ic d e v e lo p ­ m e n t a n d co lla b o ra tiv e w o rk a cro ss th e state that lead s to in n ov ativ e u s e s o f O neN et.

T h e s e c o n sid e ra tio n s le d to th e d e v e lo p m e n t o f a sp re a d sh e e t-b a s e d m o d e l. T h e system , SNAP-DSS, o r Serv ice N etw o rk A p p licatio n a n d P ricin g (SNAP)- b ase d D SS, w a s d e v e lo p e d in M icroso ft E x c e l 2 0 0 7 an d u se d th e V BA p ro g ram m in g langu age.

T h e SNAP-DSS o ffers O n e N et th e ability to s ele ct th e rate card op tio n s th a t b e s t fit th e preferred p ric­ ing strategies b y providing a real-tim e, user-friendly, graphical u ser in terface (G U I). In addition, th e SNAP- D SS n o t o n ly illustrates th e in flu ence o f the ch an g e s in th e p ricing factors o n e a c h rate card o p tio n , but also allow s th e u ser to an alyze various rate card op tio ns in d ifferent scen ario s u sin g different param eters. This m o d el has b e e n u sed b y O n eN et financial p lanners to gain insights into th e ir cu stom ers an d analyze m any w h at-if s cen ario s o f d ifferent rate p lan op tions.

Source: Based on J. Chongwatpol and R. Sharda, “SNAP: A DSS to Analyze Network Service Pricing for State Networks,” D ecision Support Systems, Vol. 50, No. 1, December 2010, pp. 347-359.

d escrib in g b u sin e ss assu m p tio ns. All assu m p tio n s are d e s c rib e d in English e q u a tio n s (o r th e u ser’s native lan g u ag e).

T h e b e s t w a y to learn h o w to use P lan n ers L ab is to la u n ch th e so ftw are an d fo llo w th e tutorials. T h e softw are ca n b e d o w n lo a d e d at p l a n n e r s l a b . c o m .

The User Interface Subsystem T h e u s e r c o m m u n ic a te s w ith a n d co m m a n d s th e D S S th ro u g h th e u s e r i n t e r f a c e s u b ­ sy stem . T h e u s e r is c o n s id e re d p art o f th e sy stem . R e s e a rc h e rs a ss e rt th a t s o m e o f th e u n iq u e c o n trib u tio n s o f D SS a re d eriv ed fro m th e in te n siv e in te ra c tio n b e tw e e n th e c o m p u te r an d th e d e c is io n m ak e r. T h e W e b b r o w s e r p ro v id e s a fam iliar, co n s is te n t g ra p h ica l u s e r in te rfa c e (G U I) stru ctu re fo r m o st D SS. F o r lo c a lly u s e d D SS, a s p re a d ­ s h e e t a ls o p ro v id e s a fam iliar u s e r in te rfa c e . A d ifficu lt u s e r in te rfa ce is o n e o f th e m a jo r re a s o n s m a n a g e rs d o n o t u s e c o m p u te rs a n d q u a n tita tiv e a n a ly se s a s m u c h as th e y c o u ld , g iv e n th e a v ailab ility o f th e s e te c h n o lo g ie s . T h e W e b b r o w s e r h a s b e e n re c o g n iz e d a s a n e ffe c tiv e D SS G U I b e c a u s e it is fle x ib le , u s e r frien d ly , a n d a g a te w a y to a lm o st all s o u r c e s o f n e c e s s a ry in fo rm a tio n a n d d ata. E sse n tia lly , W e b b ro w s e rs h a v e le d to th e d e v e lo p m e n t o f p o rtals and d a s h b o a rd s , w h ic h fro n t e n d m a n y D SS.

E xp losive gro w th in p o rtab le d ev ices inclu ding sm artp h on es an d tablets has ch an g e d th e D SS u ser interfaces as w ell. T h e s e d ev ices allow e ith er h and w ritten input o r typed input fro m internal o r e xte rn al k ey board s. So m e D SS u ser in te rface s utilize natural-language input

Chapter 2 • Foundations and T ech n o lo g ies fo r D ecision Making 9 9

- r .. te x t in a h u m an lan g u ag e) s o that th e users c a n easily e x p ress th e m se lv es in a m ean - mgful way. B e c a u s e o f th e fuzzy natu re o f hu m an langu age, it is fairly difficult to d ev elop software to interpret it. H ow ever, th e se p ack ag e s in cre ase in a ccu racy e v e ry y ear, and they will ultim ately lead to accu rate input, output, and language translators.

C ell p h o n e inputs th ro u g h SMS are b e c o m in g m o re co m m o n fo r a t le a s t s o m e c o n ­ su m er D SS-ty p e ap p licatio n s. F o r e x a m p le , o n e c a n s en d a n SMS re q u e s t fo r s e a rch on any to p ic to G O O G L (4 6 6 4 5 ) . It is m o st u sefu l in lo catin g n e a rb y b u s in e s s e s , ad d resses, o r p h o n e n u m b e rs, b u t it ca n a lso b e u se d fo r m an y o th e r d e c is io n su p p o rt tasks. F or e xam p le, u se rs c a n fin d d efin ition s o f w o rd s b y e n te rin g th e w o rd “d e fin e ” fo llo w e d b y a w ord, s u ch a s “d efin e e x te n u a te .” S o m e o f th e o th e r cap ab ilitie s in clu d e:

• T ran slatio n s: “T ra n sla te th a n k s in S p a n ish .” • P rice lo o k u p s : “P rice 3 2 G B iP h o n e .” • C alcu lator: A lthough y o u w o u ld p ro b a b ly ju st w an t to u s e y o u r p h o n e ’s built-in

c a lcu la to r fu n ctio n , y o u c a n s en d a m ath e x p r e s s io n as a n SMS fo r a n answ er. • C u rren cy co n v e rsio n s: “10 u sd in e u r o s .” • Sports s co re s and gam e times: Ju s t en ter the nam e o f a team ( “NYC G iants”), and G oo g le

SMS w ill sen d the m ost re ce n t g am e’s sco re an d th e date and tim e o f th e next match.

This typ e o f SM S-based s e a rch cap ab ility is a lso av ailab le fo r o th e r s e a r c h e n g in e s , inclu d ­ ing Y a h o o ! a n d M icroso ft’s n e w s e a rch e n g in e B in g .

W ith th e e m e r g e n c e o f s m a rtp h o n e s s u c h as A p p le ’s iP h o n e a n d A n d ro id sm art­ p h o n e s fro m m a n y v e n d o rs , m a n y c o m p a n ie s a re d e v e lo p in g a p p lic a tio n s (c o m m o n ly c a lle d apps) to p ro v id e p u rc h a s in g -d e c is io n su p p o rt. F o r e x a m p le , A m a z o n .c o m ’s ap p a llo w s a u s e r t o ta k e a p ictu re o f a n y ite m in a s to r e (o r w h e re v e r) an d s e n d it to A m azon, co m . A m a z o n .c o m ’s g ra p h ics-u n d e rsta n d in g alg o rith m trie s to m a tch th e im a g e to a re al p ro d u ct in its d a ta b a s e s a n d s e n d s th e u s e r a p a g e sim ilar to A m a z o n .c o m ’s p ro d u ct in fo p a g e s , allo w in g u se rs to p e rfo rm p ric e c o m p a riso n s in re al tim e . T h o u s a n d s o f o th e r a p p s h a v e b e e n d e v e lo p e d that p ro v id e c o n s u m e rs su p p o rt fo r d e c is io n m ak in g o n fin d in g a n d s e le c tin g stores/ restau ran ts/ serv ice p ro v id ers o n th e b a s is o f lo c a tio n , re c o m m e n d a tio n s fro m o th e rs, a n d e s p e c ia lly fro m y o u r o w n so cia l circle s .

V o ic e in p u t fo r th e s e d e v ice s an d PCs is co m m o n an d fairly a ccu ra te (b u t n o t p e r­ fe ct). W h e n v o ic e in p u t w ith acco m p a n y in g s p e e c h -re c o g n itio n so ftw a re (a n d read ily av ailab le te x t-to -s p e e c h so ftw a re ) is u se d , v e rb a l instruction s w ith a c c o m p a n ie d a ctio n s and o u tp u ts c a n b e in v o k ed . T h e s e are read ily av ailab le fo r D SS an d are in co rp o ra te d into th e p o rtab le d e v ice s d e s crib e d ea rlier. An e x a m p le o f v o ic e inputs th a t c a n b e u se d fo r a g e n e ra l-p u rp o se D SS is A p p le ’s Siri a p p licatio n an d G o o g le ’s G o o g le N o w serv ice. F o r e x a m p le , a u s e r c a n g iv e h e r zip c o d e an d say “pizza d eliv ery .” T h e s e d e v ic e s p ro v id e th e s ea rch resu lts a n d ca n e v e n p la c e a call to a b u sin ess.

R e c e n t e ffo rts in b u sin e ss p ro c e s s m an a g e m e n t (B P M ) h av e le d t o inputs directly fro m p h y sica l d ev ice s fo r an alysis via D SS. F o r e x a m p le , ra d io -fre q u e n cy id en tificatio n (R F ID ) c h ip s c a n re co rd data fro m sen so rs in railcars o r in -p ro ce s s p ro d u cts in a factory. D ata from th e s e s e n so rs (e .g ., re co rd in g a n ite m ’s statu s) c a n b e d o w n lo a d e d at k e y lo c a ­ tio n s a n d im m ed iately tran sm itted to a d a ta b a se o r d ata w a re h o u se , w h e r e th e y c a n b e an aly zed a n d d e cis io n s c a n b e m ad e c o n c e rn in g th e statu s o f th e item s b e in g m o n ito red . W alm art a n d B e s t B u y a re d e v e lo p in g this te c h n o lo g y in th e ir SCM, an d s u ch sen sor netw orks a re a lso b e in g u s e d e ffe ctiv e ly b y o th e r firms.

The Knowledge-Based M anagem ent Subsystem T h e k n o w led g e-b a sed m an agem en t subsystem can support an y o f th e o th e r subsystem s or act as a n in d ep en d en t com p on en t. It provides intelligence to augm ent th e d ecisio n m ak­ e r’s ow n. It c a n b e in tercon n ected w ith th e organization’s kn ow led g e rep ository (p art o f

a kn ow led g e m an agem en t system [K M S D , w h ich is som etim es called t h e o r g a n i z a t i o n a l k n o w le d g e b a s e . K now led ge m ay b e provided via W e b servers. M any artificial m telligence m ethods hav e b e e n im p lem ented in W e b d ev elo p m en t system s su ch as Ja v a an d are easy to integrate into th e other D S S com p on en ts. O n e o f th e m o s t w i d e l y pu blicized k n ow led g e- b a se d D SS is IBM ’s W atson com p u ter system . It is d escribed in A pplication C ase 2 ̂ .

W e c o n c lu d e th e s e c tio n s o n th e th re e m a jo r D SS co m p o n e n ts ^ o n s o m e re c e n t te c h n o lo g y an d m e th o d o lo g y d ev elo p m en ts that a ffect D SS and. t e : - s io n m aking. T e c h n o lo g y Insigh ts 2 .2 su m m arizes s o m e e m e rg in g d e v e lo p m e n ts in u s e r

0 Part I • D ecisio n M aking and Analytics: An Overview

Application Case 2.4 From a G am e W in n e r to a Doctor! T h e te le v isio n s h o w Je o p a r d y ! in sp ire d a n IBM re sea rch te a m to b u ild a s u p e rco m p u ter n a m e d W atso n that s u cce ssfu lly to o k o n th e c h a lle n g e o f playing Jeo p a r d y ! an d b e a t th e o th e r h u m an c o m ­ p etitors. S in c e th e n , W a tso n h a s e v o lv e d in to a q u e stio n -a n sw e rin g co m p u tin g platform that is n o w b e in g u se d co m m e rcia lly in th e m e d ica l field an d is e x p e c te d to fin d its u s e in m a n y o th e r areas.

W a tso n is a co g n itiv e sy stem b u ilt o n clu s­ te rs o f p o w erfu l p ro c e s s o rs su p p o rte d b y IB M ’s D ee p Q A ® so ftw are . W a tso n em p lo y s a co m b in a ­ tio n o f te c h n iq u e s lik e natu ral-lan g u ag e p ro cessin g , h y p o th e sis g e n e r a tio n an d e v alu atio n , a n d e v id e n ce - b a s e d learn in g t o o v e r c o m e th e con strain ts im p o se d b y p ro g ram m atic co m p u tin g . T h is e n a b le s W a tso n to w o rk o n m assiv e am o u n ts o f real-w orld , unstruc-

tu red B ig D ata efficien tly. In th e m ed ical field, it is estim ated th at the

am o u n t o f m e d ica l inform ation d o u b les every 5 years. T h is m assiv e grow th limits a p h y s ic ia n s d ecisio n -m a k in g ability in d iagn o sis an d treatm ent o f illness u sin g a n e v id e n c e -b a se d a p p ro a ch . W ith the a d v an ce m e n ts b e in g m ad e in th e m ed ical field every­ d ay, p h ysician s d o n o t h av e e n o u g h tim e to read e v ery jo u rn al that c a n h e lp th e m in k e e p in g up -to- d ate w ith th e latest ad v an cem en ts. P atien t histories and e le ctro n ic m e d ica l record s co n tain lots o f data. If this in form ation ca n b e an aly zed in co m b in a tio n w ith v a st am o u nts o f e xistin g m e d ica l k n o w le d g e, m any u sefu l clu e s c a n b e pro v id ed to th e p h ysician s to h e lp th e m id en tify d iag n o stic and treatm ent o p tio n s. W atso n , d u b b e d Dr. W atso n , w ith its ad v an ced m a ch in e learn in g cap ab ilities, n o w find s a n e w role as a co m p u te r c o m p a n io n th at assists p h ysician s b y p roviding relev an t real-tim e inform ation fo r critical d e cisio n m ak in g in ch o o s in g th e right d iagn o stic and tre atm e n t p ro ced u res. (A lso s e e the o p e n in g vignette

fo r C h ap ter 7 .)

M em orial Sloan-K ettering C an cer Center (M SKCC), New' Y o rk , a n d W ellPoint, a m ajo r insur­ a n c e provider, h av e b e g u n using W atson as a treat­ m e n t advisor in o n co lo g y diagnosis. W atso n learn ed th e p ro cess o f d iag n o sis an d treatm ent through its natural-language p ro cessin g capabilities, w h ich e n a ­ b le d it t o lev erag e th e u nstructured data w ith a n e n o r­ m o u s am o u nt o f clin ical e x p ertise data, m o lecu lar an d g e n o m ic data fro m existing c a n c e r ca se histo­ ries, journal articles, p h y sician s’ n o tes, and guidelines an d b e s t p ractices fro m th e N ational C om prehensiv e C an cer N etw ork. It w as th e n trained b y o n co lo g ists to apply th e k n o w le d g e g a in e d in com p arin g an individ­ u al patient’s m ed ical inform ation against a w id e vari­ e ty o f treatm ent guid elines, p u b lish ed research , and o th er insights to p rovide individualized, co n fid e n ce- sco re d recom m en d ation s to the physicians.

At MSKCC, W atso n facilitates ev id en ce-b ased supp ort fo r every su ggestio n it m akes w h ile analyz­ ing a n individual ca se b y bringing ou t the facts from m ed ical literature that p o in t to a particular sugges­ tion. It also provides a platform fo r th e physicians to lo o k a t th e c a s e fro m m ultiple d irections b y doing fur­ th e r analysis relevan t to th e individual case. Its v o ice recogn ition capabilities allow physicians to sp e a k to W atson, en ablin g it to b e a p erfect assistant that helps physicians in critical ev id en ce-b ased d ecisio n making.

W e llP o in t a lso train e d W a ts o n w ith a vast h is­ to ry o f m e d ica l c a s e s a n d n o w re lie s o n W a tso n ’s h y p o th e sis g e n e ra tio n an d e v id e n c e -b a s e d learn in g to g e n e ra te re co m m e n d a tio n s in providing approval fo r m e d ical treatm en ts b a s e d o n th e clin ical and p a tie n t data. W a tso n a lso assists th e in su ran ce p ro ­ v id ers in d etectin g frau d u len t claim s an d p ro tectin g p h y sician s fro m m a lp ra ctice claim s.

W a tso n p ro v id e s a n e x c e lle n t e x a m p le o f a k n o w le d g e -b a s e d D SS that em p lo y s m ultiple ad v a n ce d te ch n o lo g ie s.

Chapter 2 • Foundations and T ech n o lo g ies fo r D ecision M aking 101

Q u e s t i o n s f o r D i s c u s s i o n

1. W h at is a co g n itiv e system ? H o w can it assist in real-tim e d e c is io n m aking?

2 . W h a t is e v id e n c e -b a se d d e c is io n m aking?

3. W h at is th e ro le played b y W atson in th e d iscu ssion?

4 . D o e s W a ts o n elim in ate th e n e e d fo r h u m an d e c i­ s io n m aking?

W h a t W e C an L e a r n f r o m T h is A p p lic a tio n C a se

A d vancem ents in te ch n o lo g y n o w e n a b le the build­ ing o f p o w erfu l, cogn itive com p u tin g platform s co m ­ b in e d w ith co m p le x analytics. T h e s e system s are

im pacting th e d ecisio n -m ak in g p ro cess radically b y shifting th em fro m a n o p in io n -b ased p ro cess to a m o re real-tim e, e v id e n ce -b a se d p ro cess, th e re b y turn­ ing available inform ation in tellig ence into action ab le w isd om that c a n b e readily e m p lo y ed acro ss m any industrial sectors.

Sources-. Ibm.com, “IBM Watson: Ushering In a New Era of Computing," www-03-ibm.com/innovation/us/watson (accessed February 2013); Ibm.com, “IBM Watson Helps Fight Cancer with Evidence-Based Diagnosis and Treatment Suggestions,” www- 03.ibm.com/innovation/us/watson/pdf/MSK_Case_Study_ IM C l4794.p d f (accessed February 2013); Ibm.com, “IBM Watson Enables More Effective Healthcare Preapproval Decisions Using Evidence-Based Learning,” www-03.ibm.com/innovation/us/ watson/pdf/W ellPoint_Case_Study_IMC1 4 7 9 2 .pdf (accessed Febmary 2013).

T E C H N O L O G Y IN S IG H T S 2 .2

Next Generation of Input Devices

T h e last few years have s e e n exciting developm ents in user interfaces. Perhaps the m ost com ­ m on exam p le o f the new u ser interfaces is the iPhone’s multi-touch interface that allows a user to zoom , p an, and scroll through a screen just with the use o f a finger. T h e su ccess o f iPhone has spaw ned developm ents o f similar u ser interfaces from m any other providers including Blackberry, HTC, LG, M otorola (a part o f G oog le), Microsoft, Nokia, Samsung, and others. Mobile platform has b e co m e the m ajor access m echanism for all d ecision support applications.

In th e last fe w years, gam in g d ev ices have evolved significantly to b e a b le to rec eiv e and p ro cess g estu re-b ased inputs. In 2 0 0 7 , N intendo introd uced th e W ii g am e platform , w h ic h is a b le to p ro c e ss m otion s and gestures. M icrosoft’s K in ect is ab le to reco g n iz e im age m ovem ents and u se th a t to d iscern inputs. T h e n e x t gen eratio n o f th ese te ch n o lo g ie s is in th e fo rm o f m ind-readin g platform s. A com p an y called Em otiv ( e n .w ik ip e d ia .o r g / w ik i/ E m o tiv ) m ade big n ew s in early 2 0 0 8 w ith a prom ise to d eliver a g am e con troller that a u s e r w ou ld b e ab le to con tro l b y think in g ab o u t it. T h e s e tech n o log ies a re to b e b a sed o n e le c tr o e n c e p h a lo g r a ­ p h y (E E G ), th e tech n iq u e o f read in g and p ro cessin g the electrical activity a t th e sca lp level a s a result o f s p e cific thou ghts in th e brain. T h e tech n ical details are av ailable o n W ikipedia ( e n .w ik ip e d i a .o r g / w ik i / E l e c t r o e n c e p h a l o g r a p h y ) and th e W eb . A lthough EEG has not y e t b e e n k n o w n to b e used as a DSS u ser in terface (a t least to th e au th o rs), its p o ten tial is significant fo r m an y o th e r D SS-type app lications. M any o th e r co m p a n ies a re d ev elo p in g similar t e c h n o lo g ie s .

I t i s a ls o p o s s ib le t o s p e c u la te o n o t h e r d e v e lo p m e n ts o n t h e h o r iz o n . O n e m a jo r g r o w th a r e a is l ik e ly to b e in w e a r a b le d e v ic e s . G o o g le ’s w e a r a b le g la s s e s th a t a r e la b ele d “augm ented reality” glasses will lik ely em erg e as a n ew u ser interface for d ecision support in b oth con su m er a n d c o r p o r a t e d e c is io n settin g s. S im ilarly , A p p le i s s u p p o s e d t o b e w o r k in g o n iO S -b a s e d w rist - w atch-type com puters. T h e s e d evices w ill significantly im pact how w e interact w ith a system and use th e system for d ecision support. So it is a safe b e t that u ser interfaces are g o in g to chang e significantly in th e next few years. T h eir first u se w ill probably b e in gam ing and con su m er applications, but the business and DSS applications w o n ’t b e far behind.

Sources: Various Wikipedia sites and the company Web sites provided in the feature.

1 0 2 Part I • D e c is io n M aking and Analytics: An Overview

in te rface s. M any d e v e lo p m e n ts in D S S c o m p o n e n ts a re th e resu lt o f n e w d ev elo p m en ts in hard w are and so ftw are co m p u te r te ch n o lo g y , d ata w a re h o u s in g , d ata m ining, OLAP, W e b te c h n o lo g ie s, in teg ratio n o f te c h n o lo g ie s, an d D SS ap p lica tio n to vario u s and new fu n ctio n al areas. T h e r e is a lso a cle a r lin k b e tw e e n hard w are an d so ftw are cap ab ilities a n d im p ro v em en ts in D SS. H ard w are co n tin u e s to sh rin k in size w h ile in cre asin g in s p e e d a n d o th e r cap ab ilitie s. T h e size s o f d a ta b a se s an d d ata w a re h o u s e s h av e in cre a se d dra­ m atically. D ata w a re h o u s e s n o w p ro v id e hu n d red s o f p e ta b y tes o f sa le s d ata fo r retail org an izatio n s an d c o n te n t fo r m a jo r n e w s n etw o rks.

W e e x p e c t to s e e m o re sea m le ss in tegratio n o f D SS c o m p o n e n ts as th ey a d o p t W e b te c h n o lo g ie s, e sp ecia lly XML. T h e s e W e b -b a s e d te c h n o lo g ie s h av e b e c o m e th e c e n te r o f activity in d ev elo p in g D SS. W e b -b a s e d D SS h av e re d u ce d te c h n o lo g ic a l b arriers an d hav e m ad e it e a s ie r a n d less co stly to m a k e d ecisio n -re le v a n t in form ation an d m od el-d riv en D SS a v ailab le to m an ag ers an d sta ff u sers in g e o g ra p h ica lly d istrib uted lo ca tio n s, e s p e ­ cially th ro u g h m o b ile d ev ices.

D SS b e c o m in g m o re e m b e d d ed in o th e r system s. Similarly, a m ajor area to e x p e ct im provem en ts in D SS is in G SS in supp orting co llab o ratio n at th e enterprise level. T h is is true e v e n in the ed u cation al arena. Almost e v e iy n e w a re a o f inform ation system s involves so m e lev el o f d ecisio n-m aking support. T hu s, D SS, e ith er directly o r indirectly, h a s im pacts o n CRM, SCM, ERP, KM, PLM, BAM, BPM , a n d o th e r E IS. As th e se system s evolve, th e active d ecisio n -m akin g c o m p o n e n t that utilizes m ath em atical, statistical, o r e v e n descriptive m o d els in cre ase s in size a n d capability, althou gh it m ay b e b u ried d e e p w ithin th e system .

Finally, d ifferen t typ es o f D SS co m p o n e n ts are b e in g integrated m o re freq u ently. F or e x a m p le , G IS are read ily in teg rated w ith o th er, m o re trad itional, D SS c o m p o n e n ts an d to o ls fo r im p ro ved d e c is io n m akin g. ,

B y d efin itio n , a D SS m ust in clu d e th e th ree m a jo r c o m p o n e n ts — D BM S, M BM S, an d u s e r in terface. T h e k n o w le d g e -b a s e d m a n a g e m e n t su b sy stem is o p tio n al, b u t it ca n p ro ­ v id e m a n y b e n e fits b y p ro vid ing in te llig e n ce in an d to th e th re e m a jo r co m p o n e n ts. As in a n y o th e r M IS, th e u s e r m ay b e c o n s id e re d a c o m p o n e n t o f D SS.

Chapter Highlights

• M anagerial d e c is io n m ak in g is sy n o n y m o u s w ith th e w h o le p r o c e s s o f m an ag em en t.

• H u m an d e c is io n styles n e e d to b e re c o g n iz e d in d esig n in g sy stem s.

• Ind ivid u al a n d g ro u p d e cisio n m ak in g c a n b o th b e su p p o rte d b y system s.

• P ro b le m so lv in g is a lso op p o rtu n ity evalu ation. • A m o d e l is a sim p lified re p re s e n ta tio n o r ab strac­

tio n o f reality. • D e c is io n m a k in g in v olv es fo u r m ajo r p h ase s:

in te llig e n ce , d esig n , c h o ic e , and im p lem en tation . • I n th e in te llig e n c e p h a s e , th e p r o b le m ( o p p o r ­

tu n ity ) is id e n tifie d , c la s s ifie d , a n d d e c o m ­ p o s e d ( i f n e e d e d ), a n d p r o b le m o w n e rs h ip is e sta b lis h e d .

• In th e d e s ig n p h a s e , a m o d e l o f th e sy stem is built, criteria fo r s e le c tio n are a g re ed o n , altern a­ tives are g e n e ra te d , results are p red icted , a n d a d e c is io n m e th o d o lo g y is created .

• In th e c h o ic e p h a s e , alternatives a re com p ared , an d a search fo r th e b e s t (o r a g o o d -en o u g h ) solu tion is launched . M any search te ch n iq u es are available.

• In im p lem en tin g alternatives, a d e cisio n m ak e r s h o u ld c o n s id e r m ultiple g o als an d sensitivity- analysis issues.

• Satisficing is a w illin g n ess to settle fo r a satis­ fa cto ry so lu tio n . In e ffe ct, satisficin g is su b op ti- m izin g. B o u n d e d rationality results in d e c is io n m ak ers satisficing.

• C o m p u ter sy stem s c a n su p p o rt all p h a s e s o f d e c i­ s io n m ak in g b y au tom atin g m an y o f th e requ ired tasks o r b y ap p ly in g artificial in te llig e n ce .

• A D SS is d e s ig n e d to su p p o rt c o m p le x m a n a g e ­ rial p ro b le m s th a t o th e r co m p u te rize d te ch n iq u es c a n n o t. D SS is u ser o rie n te d , and it u s e s d ata and m o d els.

• D SS are generally d ev elop ed to solve specific m anagerial p ro blem s, w h ere a s B I system s typically

Chapter 2 • Foundations and T ech n o lo g ies for D ecisio n M aking 1 0 3

report status, and , w h e n a p ro b lem is discovered, their analysis to o ls are utilized b y d ecisio n m akers.

• D S S c a n pro v id e su p p o rt in all p h a s e s o f th e d eci- sio n -m a k in g p ro c e s s an d to all m an ag erial lev els fo r individ uals, g ro u p s, a n d organizations.

• D S S is a u se r-o rie n te d to o l. M an y a p p lic a ­ tio n s c a n b e d e v e lo p e d b y e n d u s e rs , o fte n in s p re a d sh e e ts .

• D SS c a n im p ro ve th e e ffe ctiv e n e s s o f d ecisio n m a k in g , d e c r e a s e th e n e e d fo r training, im prove m an a g e m e n t co n tro l, facilitate com m u n icatio n , sav e e ffo rt b y th e u sers, re d u ce co sts, and allo w fo r m o re o b je c tiv e d e c is io n m aking.

• T h e AIS SIG D SS cla ssifica tio n o f D SS in clu d es c o m m u n icatio n s-d riv e n a n d g ro u p D SS (G S S ), d ata-d riven D SS, d o cu m en t-d riv e n D SS, k n o w l­ e d g e -d riv e n D SS, data m in in g an d m an ag e m e n t E S a p p lica tio n s, an d m o d el-d riv en D SS. Several o th e r classificatio n s m ap in to this o n e .

• S ev eral u se fu l cla ssifica tio n s o f D SS are b a s e d o n w h y th e y a re d e v e lo p e d (in stitu tio n al versu s a d h o c ) , w h at lev el w ith in th e o rg a n iz a tio n they s u p p o rt (p e rs o n a l, g ro u p , o r o rg a n iz a tio n a l), w h e th e r th e y su p p o rt ind ivid u al w o rk o r gro u p w o rk (in d iv id u al D SS v ersu s G S S ), a n d h o w th ey a re d e v e lo p e d (c u s to m v e rsu s re ad y -m ad e).

• T h e m a jo r co m p o n e n ts o f a D SS are a d atabase an d its m an ag em en t, a m o d el b a se an d its m an ­ ag em en t, an d a user-friendly interface. An intelli­ g en t (k n o w led g e -b a se d ) co m p o n e n t c a n also b e in clu d ed . T h e u ser is also co n sid ere d to b e a co m ­ p o n e n t o f a DSS.

• D ata w areh o u ses, data m ining, and OLAP hav e m ad e it p o ssib le to d ev elo p D SS q u ickly and easily.

• T h e d ata m a n a g e m e n t su b sy stem u su ally in clu d es a D SS d a ta b a se , a D BM S, a d ata d irectory, an d a q uery facility.

• T h e m o d e l b a s e in clu d es stand ard m o d e ls an d m o d e ls s p e cifica lly w ritten fo r the D SS.

• C ustom -m ade m o d els ca n b e w ritten in program ­ m ing languages, in sp ecial m odeling languages, and in W e b -b a sed d ev elop m en t system s (e .g ., Jav a, th e .N ET Fram ew ork).

• T h e u s e r interface (o r d ialog) is o f utm ost im por­ tan ce. It is m anaged b y softw are that provides the n e e d e d capabilities. W e b b row sers an d sm art­ phones/tablets co m m o n ly provide a friendly, co n ­ sistent D SS GUI.

• T h e u s e r in te rfa ce cap a b ilitie s o f D SS h av e m o v ed in to sm all, p o rta b le d ev ice s, in clu d in g sm art­ p h o n e s , tab lets, an d s o forth.

Key Terms

ad h o c D SS algorithm an aly tical te c h n iq u e s b u sin e ss in te llig e n ce

(B I ) c h o ic e p h a s e d ata w a re h o u s e d atab ase m a n a g e m e n t

sy stem (D B M S )

d e c is io n m aking d e c is io n style d e c is io n variab le d escrip tiv e m o d el d e s ig n p h ase D SS a p p licatio n e ffe ctiv e n ess e fficie n cy im plem entation p hase

institu tional D SS in te llig e n ce p h a s e m o d e l b a s e m a n a g e m e n t

system (M BM S) n orm ative m o d e l op tim izatio n organ izatio n al

k n o w le d g e b a se p rin cip le o f c h o ic e

p ro b le m ow n ersh ip p ro b le m solv in g satisficing scen a rio sensitivity analysis sim ulation su b op tim ization u s e r in terface w h a t-if analysis

Questions for Discussion 1 . Discuss the need to have decision support systems betw een

behavioral and scientific methods. 2 . D iscuss th e com plexity in including behavioral patterns

into com puterized systems. 3 . Explain w h y d ecision m akers using th e sam e d ecision­

m aking p ro cess m ight end up w ith tw o different decisions. 4 . P resen t sim ilarities and d iffe re n c e s b e tw e e n Sim on ’s and

K e p n e r-T re g o e m eth od s.

5 . Apply Sim on’s four-phase model to the elevator application case (see Application Case 2.1).

6 . Exam ine w hy the d ecom position o f a p roblem is o n e o f the m ost im portant steps in d ecision making.

7. Appraise th e maxim: “All m odels are w rong, som e are useful.”

8 . D evelop a n exam p le illustrating satisficing and explain w hy this m eth od is suboptim al yet very useful.

1 0 4 Part I * D ecision Making and Analytics: An Overview

9 . E xam in e w h y c h a n g e is d ifficult to im plem ent. 1 0 . “T h e m ore data th e b e tte r.” Assess the n eed to have qual­

ity data to feed d ecision-m aking systems. 1 1 . D iscuss th e d ifferen ce betw een data and information. 1 2 . Identify the type o f DSS you r organization/university is

using and discuss th e n eed to m ake it evolve towards another DSS.

1 3 . E xam ine th e difficulties to im plem ent a n ew DSS over legacy systems.

1 4 . Exam ine w hy d ecision-m aking p ro cesses n eed to b e w ell understood in ord er to create a sou nd data m anage­ m en t subsystem .

Exercises T e r a d a t a U n iv e r s ity N e tw o r k T U N ) a n d O t h e r H a n d s -O n E x e r c i s e s

1 . C h oose a c a se at TUN o r use the c a se that you r instruc­ tor ch o o ses. D escrib e in detail w hat d ecisions w ere to b e m ade in th e c a se a n d w hat p rocess w as actually followed. B e sure to describe h o w technology' assisted o r hindered th e d ecision-m aking p rocess and w hat th e d ecision’s im pacts w ere.

2 . Most com panies and organizations have dow nloadable d em os o r trial versions o f their softw are products o n the W eb so that you c a n cop y and try them ou t o n your ow n com puter. O thers h av e o n lin e dem os. Find o n e that pro­ vid es d ecision support, try it out, and write a short report ab o u t it. Include details about th e intended p u rpose o f th e softw are, h o w it w orks, and h o w it supports d ecision making.

3 . Com m ent o n Sim on’s (1 9 7 7 ) p hilosoph y that m anagerial d ecision m aking is synonym ous w ith the w h o le p rocess o f m anagem ent. D o es this m ake sense? Explain. U se a real-w orld exam p le in y o u r explanation.

4 . Consider a situation in w h ich you have a p reference about w h ere you g o to c olleg e: Y o u w ant to b e not too far aw ay from h o m e and not to o clo se. W hy might this situation arise? Explain h o w this situation fits w ith rational decision-m aking behavior.

5 . Explore te r a d a t a u n iv e r s i t y n e tw o r k .c o m . In a report, d escribe at least three interesting DSS applications and three interesting DSS areas (e.g ., CRM, SCM) that you have d iscovered there.

6 . Explain th e c o n ce p t o f bounded-rationality and how d ecisio n support system s attem pt to ov erco m e that lim itation.

End-of-Chapter Application Case Logistics O ptim ization in a M a jo r Shipping Com pany (C SAV )

I n t r o d u c t i o n Comparna Sud Americana de V apores (CSAV) is a shipping com pany headquartered in Chile, South America, and is the sixth largest shipping com pany in the world. Its operations in over 100 countries worldwide are managed from seven regional offices. CSAV operates 700,000 containers valued at $2 billion. Less than 10 p ercent o f th ese containers are ow ned b y CSAV. T h e rest are acquired from other third-party com ­ panies o n lease. At the heart o f CSAV’s business operations is their container fleet, w hich is only second to vessel fuel in terms o f cost. As part o f their strategic planning, the com pany recognized that addressing the problem o f empty container logistics w ould help red u ce operational cost. In a typical cycle o f a cargo container, a shipper first acquires a n empty con­ tainer from a con tain er depot. T h e container is then loaded onto a ta ic k and sen t to the merchant, w h o then fills it w ith his products. Finally, th e con tain er is sent by truck to the ship for

onward transport to the destination. Typically, there are trans­ shipments along the way where a container m ay b e moved from o n e vessel to another until it gets to its destination. At the destination, th e container is transported to the consignee. After emptying the container, it is sent to the nearest CSAV depot, w here m aintenance is d o n e o n the container.

T h e r e w e r e fo u r m a i n c h a lle n g e s r e c o g n iz e d b y C SA V to its e m p ty c o n t a i n e r lo g is tic s p r o b le m :

• I m b a la n c e . Som e geographic regions are net exporters w hile others are n e t ivmporters. Places like China are net exporters; h en ce, there are always shortages o f con­ tainers. North America is a net importer; it always has a surplus o f containers. This creates an im balance o f con­ tainers as a result o f uneven flow o f containers.

• U n c e r ta in ty . Factors like dem and, date o f return o f em pty con tain ers, travel tim es, and the ship’s capacity

Chapter 2 • Foundations a n d T ech n o lo g ies for D ecision Making 1 0 5

fo r em pty con tain ers create uncertainty in the location and availability o f containers.

♦ Inform ation h a n d lin g a n d sh a rin g. H uge loads o f data n e e d to b e p rocessed every day. CSAV p rocesses 4 0 0 ,0 0 0 con tain er transactions every day. T im ely d eci­ sion s b a sed o n accu rate inform ation had to b e g en er­ ated in order to help reduce safety stocks o f empty containers.

• Coordination o f interrelated decisions worldwide. P reviou sly, d ec isio n s w ere m ad e at th e lo c a l level. C onsequently, in order to alleviate the em pty container problem , d ecisions regarding m ovem ent o f em pty c o n ­ tainers a t various locations had to b e coordinated.

M e t h o d o lo g y /S o lu tio n CSAV d ev elo p ed an integrated system called Empty Container Logistics Optim ization (E C O ) using m oving average, trended and seasonal tim e series, and sales fo rce forecast (CFM) m eth­ ods. T h e ECO system com prises a forecasting m odel, inven­ tory m odel, m ulti-com m odity (MC) netw ork flow m odel, and a W eb interface. T h e forecasting m odel draws data from the regional o ffices, p rocesses it, and feed s th e resultant informa­ tion to the inventory m odel. Som e o f the inform ation th e fore­ casting m odel gen erates are th e sp a ce in the vessel for empty containers and con tain er dem and. T h e forecasting m odule also helps red u ce forecast error and, h en ce, allow s CSAV’s d ep ot to m aintain low er safety stocks. T h e inventory m odel calculates the safety stocks and feeds it to the MC Network Flow m odel. T h e MC N etw ork Flow m odel is the core o f the ECO system . It provides inform ation for optimal d ecisions to b e m ade regarding inventory levels, con tain er reposition­ ing flow s, and th e leasin g and return o f em pty containers. T h e objective function is to minim ize em pty con tain er logis­ tics cost, w h ich is m ostly a result o f leasing, repositioning, storage, loading, and discharge operations.

R e s u l t s /B e n e f i t s T he ECO system activities in all regional centers are w ell coor­ dinated w hile still maintaining flexibility and creativity in their operations. T h e system resulted in a 50 p ercen t reduction in inventory stock. T h e generation o f intelligent inform ation from historical transactional data help ed increase efficiency o f operation. F o r instance, th e em pty tim e p er con tain er cycle d ecreased from a high o f 4 7 .2 days in 2 0 0 9 to only 27.3 days the follow ing year, resulting in an increase o f 60 p ercen t o f the average em pty container turnover. Also, con tain er cycles

increased from a reco rd low o f 3.8 cycles in 2009 to 4.8 cycles in 2010. M oreover, w hen the ECO system w as im plem ented in 2010, th e e x c e ss c o s t p er full voyage b eca m e §35 ch eap er than the average cost fo r the period betw een 2006 and 2009. This resulted in cost savings o f $101 million o n all voyages in 2010. It w as estim ated that ECO ’s direct contribution to this cost reduction was ab o u t 8 0 p ercen t ($81 million). CSAV projected that ECO will h elp generate $200 million profits over the next 2 years sin ce its im plem entation in 2010.

C a s e Q u e s t i o n s 1 . Explain w h y solving the em pty con tain er logistics

problem con tributes to cost savings for CSAV. 2 . W h at a re so m e o f th e qu alitative b en efits o f th e op tim i­

zation m o d el fo r th e em p ty c o n ta in e r m ovem ents? 3 . W hat are so m e o f the k ey benefits o f the forecasting

m od el in th e ECO system im plem ented by CSAV? 4 . Perform an o n lin e search to determ ine how' other ship­

ping com p an ies handle th e em pty con tain er problem . D o you think the ECO system w ould directly b enefit th o se com panies?

5 . B esid es shipping logistics, can you think o f any other dom ain w h ere such a system w ould b e useful in reduc­ ing cost?

W h a t W e C a n L e a r n f r o m T h is E n d -o f- C h a p te r A p p lic a tio n C ase T h e em pty con tain er p roblem is faced by m ost shipping com panies. T h e problem is partly caused by an im balance in th e dem and o f em pty containers betw een different g e o ­ graphic areas. CSAV used an optim ization system to solve the em p ty c o n ta in e r p r o b le m . T h e c a s e d e m o n s tr a te s a situ­ ation w here a bu sin ess problem is solved n o t ju st b y on e m eth od o r m od el, but by a com bination o f different opera­ tions research and analytics m ethods. For instance, w e realize that th e optim ization m od el used by CSAV consisted o f differ­ en t subm odels su ch as the forecasting and inventory m odels. T h e shipping industry is only o n e sec to r am ong a myriad o f sectors w here optim ization m odels are u sed to d ecrease the cost o f business operations. T h e lessons learned in this case cou ld b e ex p lo red in o th er dom ains such as m anufacturing and supply chain.

Source: R. Epstein et al., "A Strategic Empty Container Logistics Optimization in a Major Shipping Company,” In terfaces, Vol. 42, No. 1, January-February 2012, pp. 5—16.

References Allan, N., R. Fram e, and I. Turney. (2003). “Trust and Narrative:

E xperiences o f Sustainability.” T h e C o rp o ra te C itiz en , Vol. 3, No. 2.

Alter, S. L. (1 9 8 0 ). D ec isio n S u p p ort S ystem s: C u rren t P r a c tic e s a n d C o n tin u in g C h a llen g es. Reading, MA: Addison-W esley.

Baker, J ., and M. Cameron. (1996, Septem ber). "The Effects o f the Service Environment o n Affect and Consumer Perception o f Waiting Tim e: An Integrative Review a n d R esea rch Propositions/’ J o u r n a l o f th e A c a d em y o f M arketin g S cien ce, Vol. 24, pp. 3 3 8 -3 4 9 .

1 0 6 P a r t i • D ecisio n M aking and Analytics: An Overview

Barba-R om ero, S. (2 0 0 1 , July/August). “T h e Spanish G overn­ m ent U ses a D iscrete Multicriteria DSS to D eterm ine D ata Processing A cquisitions.” I n te r fa c e s , Vol. 31, No. 4, pp. 1 2 3 -1 3 1 .

B ea c h , L. R. (2 0 0 5 ). T h e P sy c h o lo g y o f D e c is io n M a k in g : P e o p le in O r g a n iz a tio n s , 2nd ed. T h o u san d O aks, CA: Sage.

B irkm an International, In c., b ir k m a n .c o m ; K eirsey T em ­ p eram ent Sorter and K eirsey Tem peram ent Theory-II, k e ir s e y .c o m .

Chongw atpol, J ., and R. Sharda. (2 0 1 0 , D ecem b er). “SNAP: A D SS to A nalyze Netw ork Service Pricing for State Netw orks.” D e c is io n S u p p o rt S ystem s, Vol. 50, No. 1, pp. 3 4 7 -3 5 9 .

C ohen, M .-D., C. B . Charles, and A. L. Medaglia. (2 0 0 1 , March/ April). “D ecision Support w ith W eb-E n abled Softw are.” In te r fa c e s , Vol. 31 , No. 2, pp. 1 0 9 -1 2 9 .

D enning, S. (2 0 0 0 ). T h e S p r in g b o a r d : H ow S to ry tellin g Ig n ites A c tio n in K n o w led g e -E r a O rg a n iz a tio n s. Burlington, MA: Butterw orth-H einem ann.

D onovan, J . J . , and S. E. M adnick. (1 9 7 7 ). “Institutional and Ad H oc DSS and T h eir Effective U se.” D a ta B a s e , Vol. 8, No. 3, pp. 7 9 -8 8 .

Drum m ond, H. (2 0 0 1 ). T h e A rt o f D e c is io n M a k in g : M irro rs o f Im a g in a tio n , M asks o f F a te . N ew Y ork : Wiley.

Eden, C„ and F. Ackerm ann. (2 0 0 2 ). “Em ergent Strategizing.” In A. H uff and M. Je n k in s (ed s.). M ap p in g S tra teg ic T h in k in g . Thou sand O aks, CA: Sage Publications.

Epstein, R., et al. (20 1 2 , January/February). “A Strategic Empty Container Logistics O ptim ization in a M ajor Shipping C om pany.” In te r fa c e s , Vol. 42 , No. 1, pp. 5 -1 6 .

Farasyn, I., K. Perkoz, a n d W . V an de V elde. (2008, July/ August). “Spread sh eet M odels for Inventory Target Setting at Procter and G am ble.” In te r fa c e s , Vol. 38, No. 4, pp. 2 4 1 -2 5 0 .

G ood ie, A. (2004, Fall). “G ood ie Studies Pathological G am blers’ R isk-Taking B eh av ior.” T h e In d e p e n d e n t V a ria b le. Athens, GA: T h e University o f G eorgia, Institute o f Behavioral R esearch , ib r .u g a .e d u / p u b lic a tio n s / f a l l 2 0 0 4 . p d f (a c ce s se d February 2013).

H esse, R ., a n d G . W o o lsey . (1 9 7 5 ). A p p lie d M a n a g e ­ m e n t S c ie n c e : A Q u ic k a n d D irty A p p r o a c h . Chicago: SRA Inc.

Ibm .com . “IBM W atson: U shering In a N ew Era o f Com puting.” w w w -0 3 .ib m .c o m / in n o v a tio n / u s / w a ts o n (a c cessed

February 2 013). Ibm .com . “IBM W atson H elps Fight C ancer w ith Evidence-

B ased D iagnosis and Treatm ent Suggestions.” w w w -0 3 . i b m . c o m / i n n o v a t i o n / u s / w a t s o n / p d f / M S K _ C a s e _ S t u d y _ I M C l 4 7 9 4 .p d f (a c cessed February 2013).

Ib m .c o m . “IBM W a ts o n E n a b les M ore E ffective H ealth care P reap p rov al D e c is io n s U sing E v id en ce -B a se d Learn­ ing. ” w w w -0 3 .ib m .c o m / in n o v a tio n / u s / w a ts o n / p d f/

W e l l P o i n t _ C a s e _ S t u d y _ I M C l 4 7 9 2 .p d f (a c c e s s e d Feb ru ary 2 0 1 3 ).

Je n k in s, M. (2 0 0 2 ). “Cognitive M apping.” In D. Partington (ed .). E s s e n tia l S k ills f o r M a n a g e m e n t R e s e a r c h . Thou sand O aks, CA: Sage Publications.

K epner, C., and B . T regoe. (1 9 9 8 ). T h e N ew R a tio n a l M a n a g er. Princeton, NJ: K epner-Tregoe.

Koksalan, M., and S. Zionts (ed s.). (2 0 0 1 ). M u ltip le C riter ia D e c is io n M a k in g in t h e N ew M illen n iu m . H eidelberg: Springer-Verlag.

Koller, G. R. (2 0 0 0 ). R isk M o d elin g f o r D eter m in in g V a lu e a n d D e c is io n M a k in g . B o c a Raton, FL: CRC Press.

Larson, R. C. (1 9 8 7 , N ovem ber/ D ecem ber). “P ersp ectiv es o n Q u eu es: So cial Ju s tic e a n d th e P sy ch o lo g y o f Q u e u e in g .” O p e r a tio n s R e s e a r c h , V ol. 35 , No. 6 , pp. 8 9 5 - 9 0 5 .

Luce, M. F., J . W . P ayne, a n d J . R. Bettm an. (2 0 0 4 ). “T h e Em otional Nature o f D ec isio n Trad e-offs.” In S. J . Hoch, H. C. Kunreuther, and R. E. Gunther (ed s.). W h arto n o n M a k in g D ecisio n s. N ew Y o rk : Wiley.

O lavson, T ., and C. Fry. (2 0 0 8 , July/August). “Spreadsheet D ecision-Su pport T o o ls: Lessons Learned at H ewlett- P ackard.” In te r fa c e s , V ol. 38 , No. 4, pp. 3 0 0 -3 1 0 .

Pauly, M. V. (2004). “Split Personality: Inconsistencies in Private and Public D ecision s.” In S. J . H och , H. C. Kunreuther, and R. E. G unther (e d s .). W h arto n o n M a k in g D ecisio n s. New Y ork: Wiley.

Pow er, D. J . (2002). D e c is io n M ak in g S u p p ort System s: A ch iev em en ts, T ren d s a n d C h a llen g es. Hershey, PA: Idea G roup Publishing.

Pow er, D. J ., and R. Sharda. (2009). “D ecision s Support System s.” In S.Y. N of (ed .), S p r in g er H a n d b o o k o f A u to m a tio n . N ew York: Springer.

Purdy, J . (20 0 5 , Sum m er). “D ecisio n s, D elusions, & D eb a cles.” UGA R e s e a r c h M a g a z in e .

Ratner, R. K., B . E. Kahn, and D. K ahnem an. (19 9 9 , Ju n e ). “C hoosing Less-Preferred E xp erien ces for the Sake o f Variety r f o u m a l o f C o n su m er R es ea rc h , Vol. 26, No. 1.

Sawyer, D. C. (1 9 9 9 ). G ettin g It R ig h t: A v o id in g th e H ig h C ost o f W ron g D ecisio n s. B o c a Raton, FL: St. Lucie Press.

Sim on, H. (1 9 7 7 ). T h e N ew S c ie n c e o f M a n a g e m e n t D ec isio n . Englew ood Cliffs, NJ: P rentice Hall.

Stewart, T . A. (2002, N ovem ber). “How to T h in k w ith Y our G ut.” B u sin ess 2 .0 .

Terad ata.com . “No Limits: Station Casinos B reaks the Mold o n Custom er R elationships.” te r a d a ta .c o m / c a s e -s tu d ie s / S t a t i o n - C a s i n o s - N o - L i m i t s - S t a t i o n - C a s i n o s - B r e a k s - t h e - M o l d - o n - C u s t o m e r - R e l a t i o n s h i p s - E x e c u t i v e - S u m m a r y - e b 6 4 l 0 (a c ce s se d February 2013).

Tversky, A., P. Slovic, and D. K ahnem an. (1 9 9 0 , March). “T h e C auses o f P referen ce Reversal.” A m e r ic a n E c o n o m ic R ev iew , Vol. 80, No. 1.

Yakov, B.-H. (2001). In fo rm a tio n G a p D ec isio n T h eory : D ecision s U n d er S ev ere U n cen ain ty . N ew York: Academic Press.

Descriptive Analytics

LEARNING OBJECTIVES FO R PART II

■ Learn th e ro le o f d escrip tive analytics (D A ) in solving b u sin e ss p ro b lem s

■ Learn th e b a sic definitions, co n cep ts, and architectures o f d ata w arehousing (D W )

■ L earn th e ro le o f d ata w a re h o u s e s in m anagerial d e cisio n su p p o rt

■ Learn th e cap a b ilitie s o f b u sin ess re p o rtin g an d visualization as e n a b le r s o f DA

* L earn th e im p o rta n ce o f in form ation v isu alization in m a n a g e ria l d e cisio n su p p ort

■ L earn th e fo u n d atio n s o f th e e m e rg in g field o f visual an alytics

■ L earn the cap a b ilitie s an d lim itations o f d ash b o a rd s and sco re ca rd s

■ L earn th e fu n d am en tals o f b u sin ess p e rfo rm a n ce m a n a g e m e n t (B P M )

Descriptive analytics, often referred to as business intelligence, uses data and models to answer the “what happened?” and “why did it happen?” questions in business settings. It is perhaps ” 9 most fundamental echelon in the three-step analytics continuum upon which predictive and prescriptive analytics capabilities are built. A s you will see in th© following chapters, the key enablers of descriptive analytics include data warehousing, business reporting, decision dashboard/ scorecards, and visual analytics.

Data Warehousing

LEARNING OBJECTIVES

* U n derstand the b a sic d efin ition s and * E x p la in th e ro le o f data w a re h o u s e s in d e c is io n su p p ortc o n c e p ts o f data w a reh o u se s

■ U n d erstand data w areh o u sin g * E x p la in data in tegratio n a n d th e ex tra ctio n , tran sform atio n , and load (E T L ) p ro c e s s e s

arch ite ctu res

■ D e s c r ib e th e p ro c e s s e s u s e d in d e v e lo p in g an d m an agin g data ■ D e s c r ib e real-tim e (a ctiv e ) data

w a reh o u sin gw a reh o u se s

■ E x p la in d ata w areh o u sin g o p eratio n s 11 U n d erstan d d ata w a reh o u se ad m inistratio n a n d secu rity issu es

T h e c o n c e p t o f d ata w a re h o u s in g h a s b e e n a ro u n d sin c e th e late 1 9 8 0 s. T h is ch ap ter pro v id es th e fo u n d atio n fo r a n im portant ty p e o f d atab ase, c a lle d a d a ta w a re­house, w h ic h is prim arily u s e d fo r d e c is io n su p p o rt an d p ro v id es im p ro v ed analyti­ ca l cap ab ilitie s. W e d iscu ss d ata w areh o u sin g in th e fo llo w in g sectio n s:

3 .1 O p e n in g V ig n e tte : Is le o f C ap ri C a s in o s Is W in n in g w ith E n te r p r is e D a ta W a r e h o u s e 1 0 9

3 .2 D a ta W a r e h o u s in g D e fin itio n s a n d C o n c e p ts 111 3 . 3 D a ta W a r e h o u s in g P r o c e s s O v e r v ie w 1 1 7 3 . 4 D a ta W a r e h o u s in g A r c h ite c tu r e s 1 2 0 3 . 5 D a ta In te g r a tio n a n d th e E x tr a c tio n , T r a n s fo r m a tio n , a n d L o a d (E T L )

P r o c e s s e s 1 2 7 3 . 6 D a ta W a r e h o u s e D e v e lo p m e n t 1 3 2 3 . 7 D a ta W a r e h o u s in g Im p le m e n ta tio n Is s u e s 1 4 3 3 . 8 R e a l-T im e D a ta W a r e h o u s in g 1 4 7 3 . 9 D a ta W a r e h o u s e A d m in istra tio n , S e c u rity I s s u e s , and F u tu re T r e n d s 151

3 . 1 0 R e s o u r c e s , L in k s, a n d th e T e r a d a ta U n iv e rs ity N e tw o rk C o n n e c tio n 1 5 6

C hapter 3 • D ata W arehousing 1 0 9

3.1 OPENING VIGNETTE: Isle of Capri Casinos Is Winning with Enterprise Data Warehouse

Isle o f C ap ri is a u n iq u e a n d in n o v ativ e p la y er in th e gam in g industry. A fter e n te rin g th e m arket in B ilo x i, M ississippi, in 1 9 9 2 , Is le h a s g ro w n in to o n e o f th e co u n try s largest pu blicly trad ed g am in g co m p a n ie s , m ostly b y e stab lish in g p ro p e rtie s in th e s o u th e a ste rn U nited States a n d in th e co u n try ’s heartland . Isle o f Capri C asin o s, In c., is cu rren tly o p erat­ ing 18 ca s in o s in s e v e n states, serv in g n e arly 2 m illio n visitors e a c h year.

CHALLENGE

Even th o u gh th ey s e e m to h av e a differentiating e d g e , co m p ared to others in th e highly com petitive gam ing industry, Isle is n o t entirely un iq ue. Like any gam ing co m p an y , Is le s su ccess d ep en d s largely o n its relationsh ip w ith its cu stom ers— its ability to cre ate a gam ing, entertainm ent, an d hospitality atm o sp h ere that anticip ates cu stom ers’ n e e d s an d e x ce e d s iheir exp ectatio n s. M eetin g s u ch a g oal is im p ossible w itho u t tw o im portant com p on en ts: 2 com p an y culture that is laser-focu sed o n m aking th e cu sto m e r e x p e rie n c e an e n jo y a b le o n e , and a data a n d te ch n o lo g y architecture that e n a b les Isle to constan tly d e e p e n its u n d er­ standing o f its cu stom ers, a s w ell as the v ario u s w ays cu stom er n e ed s c a n b e efficiently m et.

SOLUTION

After a n initial d ata w a r e h o u s e im p lem e n tatio n w a s d erailed in 2 0 0 5 , in part b y H u rrican e Katrina, Isle d e c id e d to r e b o o t th e p ro je c t w ith en tirely n e w c o m p o n e n ts a n d T e ra d a ta a s th e c o r e so lu tio n an d k e y p artn er, a lo n g w ith IBM C o g n o s fo r B u s in e s s In te llig e n ce . Shortly a fte r th a t c h o ic e w a s m ad e, Is le b ro u g h t o n a m a n a g e m e n t te a m th a t clearly u n d ersto od h o w th e T erad ata a n d C o g n o s so lu tio n co u ld e n a b le k e y d e c is io n m ak ers Throughout th e o p e ra tio n t o e asily fram e th e ir o w n initial q u e ries, a s w e ll as tim ely fo llo w - up q u estio n s, thu s o p e n in g up a w e a lth o f p o ssib ilities to e n h a n c e th e b u sin ess.

RESULTS

T h an ks to its s u cce s s fu l im p lem e n ta tio n o f a c o m p re h e n siv e data w a reh o u sin g an d b u si­ n ess in te llig e n ce so lu tio n , Isle h a s a ch ie v e d so m e d e e p ly satisfying results. T h e co m p a n y has d ram atically a c c e le ra te d a n d e x p a n d e d th e p ro c e s s o f in form ation g ath erin g and dispersal, p ro d u cin g a b o u t 150 rep o rts o n a daily b asis, 100 w e e k ly , and 50 m o n th ly , in addition to ad h o c q u e rie s , co m p le te d w ith in m in u tes, all d ay e v ery day. P rior to a n e n te r­ prise d ata w a r e h o u s e (E D W ) fro m T erad ata, Isle p ro d u ce d a b o u t 5 m o n th ly rep o rts p e r property, b u t b e c a u s e th e y to o k a w e e k o r m o re to p ro d u ce , p ro p e rtie s co u ld n o t b e g in to an alyze m o n th ly activity un til th e s e c o n d w e e k o f th e fo llo w in g m o n th . M o reov er, n o n e o f th e rep o rts an aly zed an y th in g less th a n a n en tire m o n th a t a tim e; today, re p o rts u sin g u p -to -th e m inute d ata o n s p e c ific cu sto m e r seg m en ts a t p articu lar p ro p e rtie s are av ailable, o ften th e sa m e d ay, e n a b lin g th e co m p a n y to re a ct m u ch m o re q u ick ly to a w id e range

o f cu sto m e r n eed s. Is le h a s c u t th e tim e in h a lf n e e d e d to co n stru ct its c o r e m o n th ly d irect-m ail ca m ­

paigns a n d c a n g e n e r a te le s s in v o lv ed cam p aig n s p ractically o n th e sp o t. In ad d itio n to m oving faster, Is le h a s h o n e d th e p ro c e s s o f s eg m en ta tio n an d n o w ca n cro ss -re fe re n c e a w id e ra n g e o f attrib u tes, s u ch as o v erall cu sto m e r v alu e, gam in g b e h a v io rs, a n d h o tel p re fe re n ce s. T h is e n a b le s th e m to p ro d u ce m o re targ e te d cam p aig n s a im e d a t particu lar cu sto m e r seg m en ts a n d p articu lar b eh av io rs.

Is le a lso h a s e n a b le d its m an a g e m e n t and e m p lo y e e s to fu rther d e e p e n th e ir u n d e r­ stand in g o f cu sto m e r b e h a v io rs b y c o n n e c tin g data fro m its h o te l sy stem s a n d data from

1 1 0 Part II • D escriptive Analytics

its cu sto m e r-track in g system s— an d to a ct o n th a t u n d erstan d in g throu gh im proved m ark etin g ca m p a ig n s an d h e ig h te n e d lev els o f cu sto m e r se rv ice . F o r e x a m p le , th e addi­ tio n o f h o te l d ata o ffe re d n e w insights a b o u t th e in c re a se d g am in g lo c a l p atro n s d o w h en th e y stay a t a h o tel. T h is, in turn, e n a b le d n e w in ce n tiv e p rogram s (s u c h as a fre e h otel nig h t) that h av e p le a s e d lo ca ls an d in cre a se d Is le ’s cu sto m e r loyalty.

T h e h o te l data a lso has e n h a n c e d Is le ’s cu sto m e r h o stin g program . B y autom atically n otifyin g h o sts w h e n a hig h -v alu e g u e s t arrives at a h o tel, h o s ts h av e fo rg e d d e e p e r rela­ tio n sh ip s w ith th e ir m o st im portant clien ts. “T h is is b y far th e b e s t to o l w e ’v e h a d sin ce I’v e b e e n a t th e c o m p a n y ,” w ro te o n e o f th e hosts.

Is le o f C ap ri c a n n o w d o m o re a c c u r a te p ro p e rty -to -p ro p e rty c o m p a ris o n s and a n a ly se s , la rg e ly b e c a u s e T e raclata c o n s o lid a te d d is p a ra te d ata h o u s e d a t individ ual p ro p e rtie s a n d ce n tra liz e d it in o n e lo c a tio n . O n e re su lt: A c e n tra liz e d in tra n e t site p o s ts d aily fig u res fo r e a c h in d iv id u al p ro p e rty , s o th e y c a n c o m p a r e s u c h th in g s as p e rfo rm a n c e o f re v e n u e fro m s lo t m a c h in e s a n d ta b le g a m e s , a s w e ll a s co m p lim e n ta ry re d e m p tio n v a lu e s . In ad d itio n , th e IB M C o g n o s B u s in e s s I n te llig e n c e to o l e n a b le s a d d itio n a l c o m p a ris o n s , s u c h as d ire ct-m a il r e d e m p tio n v a lu e s , s p e c ific d irect-m ail p ro g ra m r e s p o n s e ra te s , d ir e c t- m a il-in c e n te d g a m in g r e v e n u e , h o te l-in c e n te d gam in g re v e n u e , n o n c o m p lim e n ta ry (c a s h ) re v e n u e fro m h o t e l r o o m re s e r v a tio n s , a n d h o te l r o o m o c c u p a n c y . O n e c le a r b e n e fit is th a t it h o ld s in d iv id u al p ro p e rtie s a c c o u n ta b le fo r c o n s ta n tly ra isin g th e b ar.

B e g in n in g w ith a n im p ortan t c h a n g e in m ark etin g strategy th at sh ifted th e fo cu s to cu sto m e r days, tim e a n d a g ain th e Teradata/IBM C o g n o s BI im p lem e n tatio n has d em ­ on strated th e v alu e o f e x te n d in g th e p o w e r o f d ata th ro u g h o u t Is le ’s en te rp rise. This in clu d e s im m ed iate an alysis o f re s p o n s e rates to m a rk e tin g cam p aig n s and the ad dition o f p ro fit an d lo ss d ata that has s u cce ssfu lly c o n n e c te d cu sto m e r v alu e a n d total p ro p erty v alu e. O n e e x a m p le o f th e p o w e r o f this in teg ratio n : B y jo in in g cu sto m e r v alu e a n d total p ro p e rty valu e, Isle g ain s a b e tte r u n d erstan d in g o f its retail cu sto m ers— a p o p u latio n in visib le to th e m b e fo re — e n a b lin g th e m to m o re e ffe ctiv e ly target m ark etin g effo rts, su ch as ra d io ads.

P e rh a p s m o st sig n ifica n tly , I s le h a s b e g u n to a d d s lo t m a c h in e d ata to th e m ix. T h e m o st im p o rta n t a n d im m e d ia te im p a c t w ill b e th e w a y in w h ic h c u s to m e r v alu e w ill in fo rm p u rc h a s in g o f n e w m a c h in e s a n d p r o d u c t p la c e m e n t o n th e c u s to m e r flo o r. D o w n th e ro a d , th e a d d itio n o f th is d ata a ls o m ig h t p o s itio n Is le to ta k e a d v an tag e o f s e r v e r -b a s e d g a m in g , w h e r e s lo t m a c h in e s o n th e c a s in o flo o r w ill e ss e n tia lly b e c o m p u te r te rm in a ls th at e n a b le th e c a s in o to s w itc h a g a m e to a n e w o n e in a m a tter o f s e c o n d s .

In short, a s Isle co n stru cts its so lu tio n s fo r regu larly fu n n e lin g slo t m a ch in e data into th e w a re h o u se , its ability to u se d ata to re-im ag in e th e flo o r a n d fo rg e e v e r d e e p e r and m o re lastin g re latio n sh ip s w ill e x c e e d any thing it m ig h t h av e e x p e c te d w h e n it e m b a rk ed o n this p ro ject.

QUESTIONS FO R TH E OPENING VIGNETTE

1 . W h y is it im p ortan t fo r Is le to have an EDW?

2 . W h at w e re th e b u sin e ss ch a lle n g e s o r o p p o rtu n itie s th at Is le w a s facing? 3 . W h at w as th e p ro c e s s Is le fo llo w e d to realize EDW ? C o m m e n t o n th e p o ten tial

ch a lle n g e s Isle m ight h a v e h ad g o in g throu gh th e p ro ce s s o f E D W d ev elo p m en t.

4 . W h at w e re th e b en e fits o f im p lem e n tin g a n E D W a t Isle? C an y o u th in k o f o th e r p o ten tial b e n e fits th at w e re n o t listed in th e case?

5. W h y d o you th in k large e n terp rises like Is le in th e gam in g indu stry c a n s u c c e e d w ith o u t having a c a p a b le d ata w areh o u se / b u sin e ss in te llig e n ce infrastructure?

Chapter 3 • Data W arehou sing 111

WHAT WE CAN LEARN FROM THIS VIGNETTE

T h e o p e n in g v ig n e tte illu stra te s th e s tra te g ic v a lu e o f im p le m e n tin g a n e n te rp ris e d ata w a r e h o u s e , a lo n g w ith its s u p p o rtin g B l m e th o d s . Is le o f C ap ri C a s in o s w a s a b le to le v e ra g e its d ata a s s e ts s p r e a d th r o u g h o u t th e e n te rp ris e to b e u s e d b y k n o w le d g e w o rk e rs (w h e r e v e r a n d w h e n e v e r th e y a re n e e d e d ) to m a k e a c c u ra te a n d tim e ly d e c i­ s io n s. T h e d ata w a r e h o u s e in te g ra te d v a rio u s d a ta b a s e s th ro u g h o u t th e o r g a n iz a tio n in to a s in g le , in -h o u s e e n te rp ris e u n it to g e n e r a te a s in g le v e r s io n o f th e tru th fo r th e co m p a n y , p u ttin g all d e c is io n m a k e rs , fro m p la n n in g to m a rk e tin g , o n th e s a m e p a g e . F u rth e rm o re, b y re g u la rly fu n n e lin g s lo t m a c h in e d ata in to th e w a r e h o u s e , c o m b in e d w ith c u s to m e r -s p e c if ic rich d ata th a t c o m e s fro m v a rie ty o f s o u r c e s , Is le s ig n ifica n tly im p ro v e d its a b ility to d is c o v e r p a tte rn s to re -im a g in e/ re in v e n t th e g a m in g f lo o r o p e r a ­ tio n s a n d fo rg e e v e r d e e p e r a n d m o re lastin g re la tio n s h ip s w ith its cu s to m e rs . T h e k e y le s s o n h e r e is th a t a n e n te rp ris e -le v e l d ata w a r e h o u s e c o m b in e d w ith a s tra te g y fo r its u se in d e c is io n s u p p o rt c a n re su lt in s ig n ific a n t b e n e fits (fin a n c ia l a n d o th e r w is e ) fo r an o rg a n iz a tio n .

Sources: Teradata, Customer Success Stones, teradata.com/t/case-studies/Isle-of-Capri-Casinos-Executive- Sum m ary-EB6277 (accessed February' 2013); www-01.ibm.com/software/analytics/cognos.

3.2 D A T A W A R E H O U S IN G D EFIN IT IO N S A N D CO NCEPTS Using real-tim e d ata w a reh o u sin g in co n ju n ctio n w ith D SS and B l to o ls is a n im p o rtan t w ay to co n d u ct b u s in e s s p ro ce s s e s. T h e o p e n in g v ig n e tte d em o n strates a s c e n a rio in w h ic h a real-tim e a ctiv e d ata w a re h o u s e su p p o rte d d e cisio n m ak in g b y an aly zin g larg e am o u n ts o f data from vario u s s o u rce s to pro v id e rap id results to su p p o rt critical p ro c e s s e s. T h e sin g le v ersio n o f th e truth s to red in th e data w a r e h o u s e and pro v id ed in a n e asily d ig e stib le fo rm ex p an d s th e b o u n d a rie s o f Is le o f C ap ri’s in n ov ativ e b u sin ess p ro ce s s e s. W ith real-tim e data flow s, Is le c a n v ie w th e cu rren t state o f its b u sin ess an d q u ick ly id entify p ro b lem s, w h ich is th e first an d fo rem o st step tow ard solv in g th e m analytically.

D e c is io n m a k e rs re q u ire c o n c is e , d e p e n d a b le in form ation a b o u t cu rren t o p eratio n s, trends, and ch a n g e s . D ata are o fte n frag m en ted in distinct o p era tio n a l sy stem s, s o m an ag ­ ers o fte n m a k e d e c is io n s w ith partial in form ation, a t b est. D ata w a reh o u sin g cu ts throu gh this o b s ta c le b y a c c e ss in g , integratin g, an d o rgan izin g k e y op eratio n al data in a fo rm that is co n siste n t, re lia b le , tim ely, an d read ily a v ailab le, w h e re v e r a n d w h e n e v e r n e e d e d .

W h at Is a D a ta W a r e h o u s e ?

In sim p le term s, a data w areh ou se (DW) is a p o o l o f data p ro d u ce d to su p p o rt d e cisio n m akin g; it is a lso a re p o sito ry o f cu rre n t an d h istorical data o f p o ten tia l in te re st to m an­ agers th ro u g h o u t th e o rg an ization . D ata are usu ally stru ctu red to b e a v ailab le in a fo rm read y fo r an aly tical p ro c e s s in g activities (i.e ., o n lin e analytical p ro c e s s in g [OLAP], data m ining, q u ery in g , rep ortin g, a n d o th er d e c is io n su p p o rt a p p licatio n s). A d ata w areh o u se is a su b je ct-o rie n te d , integ rated , tim e-variant, n o n v o latile c o lle c tio n o f data in su p p o rt o f m an ag e m e n t’s d e cisio n -m a k in g p ro ce ss.

A H is to ric a l P e rs p e c tiv e t o D a ta W a re h o u s in g

E ven th o u g h d ata w a reh o u sin g is a relativ ely n e w term in in form ation te c h n o lo g y , its ro ots ca n b e tra ce d w a y b a c k in tim e, e v e n b e fo r e co m p u te rs w e re w id ely u se d . In the early 19 0 0 s, p e o p le w e r e u sin g data (th o u g h m o stly via m an u al m e th o d s ) t o fo rm u late trend s to h e lp b u s in e s s u sers m a k e in fo rm ed d e cisio n s, w h ich is th e m o st p rev ailin g p u r­ p o se o f d ata w areh o u sin g .

T h e m o tiv atio n s th at le d to d e v e lo p in g data w a reh o u sin g te c h n o lo g ie s g o b a c k to th e 1 9 7 0 s, w h e n th e co m p u tin g w o d d w as d o m in a te d b y th e m ainfram es. R eal b u sin ess d a ta -p ro ce ssin g ap p licatio n s, th e o n e s ru n o n th e co rp o ra te m ainfram es, h a d co m p licated file stru ctures u sin g ea rly -g en e ra tio n d a ta b a s e s (n o t th e ta b le -o rie n te d relatio n al d atabases m o st a p p lica tio n s u se to d a y ) in w h ic h th e y s to re d d ata. A lthough th e s e a p p lica tio n s did a d e c e n t jo b o f p erfo rm in g ro u tin e tran sactio n al d a ta -p ro ce ssin g fu n ctio n s, the data cre ­ a te d a s a resu lt o f th e s e fu n ctio n s (s u c h a s in form ation a b o u t cu sto m ers, th e p rodu cts th e y o rd ered , a n d h o w m u ch m o n e y th ey s p e n t) w as lo c k e d aw ay in th e d ep th s o f the files a n d d ata b ases. W h e n ag g re g ate d in fo rm atio n s u ch as sale s tren d s b y re g io n an d by p ro d u ct ty p e w as n e e d e d , o n e h a d to fo rm ally re q u e st it fro m th e d a ta -p ro ce ssin g d ep art­ m en t, w h e re it w a s put o n a w aitin g list w ith a c o u p le h u n d red o th er rep o rt requ ests (H am m e rg re n a n d S im o n , 2 0 0 9 ). E v e n th o u g h th e n e e d fo r in form ation a n d th e d ata that co u ld b e u s e d to g e n e ra te it e xisted , th e d a ta b a se te c h n o lo g y w as n o t th e re to satisfy it. Fig u re 3 .1 sh o w s a tim elin e w h e re s o m e o f th e sig n ifican t e v e n ts th a t le d to th e d ev elo p ­ m e n t o f d ata w a reh o u sin g are sh ow n .

L ater in this d e c a d e , co m m e rcia l h ard w are a n d softw are c o m p a n ie s b e g a n to e m erg e w ith so lu tio n s to this p ro b lem . B e tw e e n 1 9 7 6 an d 1 9 7 9 , th e c o n c e p t fo r a n e w com p an y , T e rad ata, g re w o u t o f re s e a r c h at th e C aliforn ia Institute o f T e c h n o lo g y (C a lte ch ), driven fro m d iscu ssio n s w ith C itiban k ’s a d v a n ce d te c h n o lo g y gro u p . F o u n d ers w o rk e d to d esign a d atab ase m a n a g e m e n t system fo r p arallel p ro ce s s in g w ith m u ltiple m icro p ro ce sso rs, targ eted s p e cifica lly fo r d e c is io n su p p ort. T e ra d a ta w a s in co rp o ra te d o n J u ly 13, 1 9 7 9 , and started in a g arag e in B re n tw o o d , C alifornia. T h e n a m e T erad ata w as c h o s e n to sy m b olize the ab ility to m a n a g e te ra b y tes (trillion s o f b y te s ) o f data.

T h e 1 9 8 0 s w e re th e d e ca d e o f p e rso n a l co m p u te rs an d m in icom p u ters. B e f o r e any­ o n e k n e w it, re al co m p u te r ap p licatio n s w e r e n o lo n g e r o n ly o n m ainfram es; th e y w ere all o v e r th e p la ce — ev e ry w h e re y o u lo o k e d in an o rgan ization . T h a t led to a p o rten tou s p ro b le m ca lle d islan d s o f d a ta . T h e s o lu tio n to this p ro b le m le d to a n e w ty p e o f so ft­ w are, ca lle d a distribu ted d a t a b a s e m a n a g e m e n t system, w h ich w o u ld m ag ically pu ll the re q u e sted data fro m d a tab ase s acro ss th e org an izatio n , bring all th e d ata b a c k to th e sam e p la ce , and th e n co n so lid a te it, s o n it, an d d o w h a tev er e ls e w a s n e ce s s a ry to an sw e r the u s e r’s q u e stio n . A lthough th e c o n c e p t w a s a g o o d o n e and early results fro m re sea rch w e re p ro m ising, th e results w e re p la in an d sim p le : T h e y ju st d id n’t w o rk efficie n tly in the re al w o rld , a n d th e island s-of-d ata p ro b lem still existed .

1 1 2 Part II • D escriptive Analytics

■/ Mainframe computers ✓ Simple data entry ■/ Routine reporting V Primitive database structures / Teradata incorporated______

v' Centralized data storage ^ Big Data analytics s Data warehousing was born ^ Social media analytics ■/ Inmon, Building the Data Warehouse •/ Text and Web analytics ■/ Kimball, The Data Warehouse Toolkit S Hadoop, MapReduce, NoSQL •/ EDW architecture design__________ ^ In-memory, in-database_____

- 1 9 7 0 s - 1 9 8 0 s - - 1 9 9 0 s - - 2 0 0 0 s 2 0 1 0 s -

•/ Mini/personal computers (PCs] V Business applications for PC s ■/ Distributer DBM S / Relational DBMS S Teradata ships commercial DBs s Business Data W arehouse coined

s Exponentially growing data Web data ✓ Consolidation of D W /B I industry v' Data warehouse appliances emerged s Business intelligence popularized •/ Data mining and predictive modeling v'- Open source software v' SaaS, PaaS, Cloud computing

FIG U R E 3.1 A List o f Events That Led to Data W arehousing Development.

Chapter 3 * Data W arehousing 1 1 3

M ean w h ile, T e ra d a ta b e g a n sh ip p in g co m m e rcia l p ro d u cts to s o lv e this p r o b ­ lem. W ells F arg o B a n k re ce iv e d th e first T erad ata te s t sy stem in 1 9 8 3 , a p arallel RD BM S ■ relatio nal d atab ase m a n a g e m e n t sy stem ) fo r d e c is io n su p p ort— th e w o rld ’s first. B y 1984, Teradata re le a se d a p ro d u ctio n v e rsio n o f th e ir p ro d u ct, and in 1 9 8 6 , F ortu n e m ag azin e nam ed T e ra d a ta P ro d u ct o f t h e Y ea r. T erad ata, still in e x is te n c e today, b u ilt th e first data w areh o u sin g a p p lia n ce — a c o m b in a tio n o f h ard w are an d softw are to so lv e th e d ata w a r e ­ hou sing n e e d s o f m an y. O th e r c o m p a n ie s b e g a n to fo rm u late th e ir strategies, as w ell.

D u ring this d e c a d e sev eral o th e r e v e n ts h a p p e n e d , co lle ctiv e ly m ak in g it th e d e ca d e : : data w areh o u sin g in n o v atio n . F o r in stan ce, Ralph K im ball fo u n d ed R ed B rick Sy stem s

m 1986. R ed B ric k b e g a n to e m e rg e as a v isio n a iy so ftw are co m p a n y b y d iscu ssin g h o w io im prove d ata a c c e s s ; in 1 9 8 8 , Barry D ev lin an d P au l M urphy o f IB M Irelan d in tro d u ced the term business d a t a w a reh o u se as a k e y c o m p o n e n t o f b u sin e ss in form ation system s.

In th e 1 9 9 0 s a n e w a p p ro a ch to solving th e islan d s-of-d ata p ro b le m su rfaced . I f th e 1980s a p p ro a ch o f re a ch in g o u t a n d a c c e ss in g d ata d irectly fro m th e file s and d a tab ase s didn’t w o rk , th e 1990s p h ilo s o p h y in v o lv ed g o in g b a c k to th e 1 970s m e th o d , in w h ic h data fro m th o se p la c e s w a s c o p ie d to a n o th e r lo ca tio n — o n ly d o in g it right this tim e; h e n c e , data w a re h o u sin g w a s b o rn . In 1 9 9 3 , B ill In m o n w ro te th e sem in al b o o k B u ild in g tb e D ata W arehouse. M any p e o p le re c o g n iz e B ill as th e fath er o f d ata w areh o u sin g . Additional p u b licatio n s e m e rg ed , in clu d in g th e 1 9 9 6 b o o k b y R alp h K im ball, The D ata W arehouse Toolkit, w h ic h d iscu sse d g e n e ra l-p u rp o se d im en sio n al d esig n te c h n iq u e s to im prove th e d ata a rch ite ctu re fo r q u e ry -ce n te re d d e c is io n su p p ort system s.

In th e 2 0 0 0 s , in th e w o rld o f d ata w areh o u sin g , b o th pop u larity a n d th e am o u n t o f data co n tin u ed to g ro w . T h e v e n d o r com m u n ity a n d o p tio n s h av e b e g u n to co n so lid a te. Ln 2 0 0 6 , M icroso ft a c q u ire d ProC larity, ju m p in g in to th e data w a reh o u sin g m ark et. In 2007, O ra cle p u rch a se d H yp erion , SAP a cq u ire d B u s in e s s O b je c ts , an d IB M m e rg e d w ith C ognos. T h e data w a re h o u s in g lead e rs o f th e 1 990s h a v e b e e n sw allo w e d b y s o m e o f th e larg est p ro vid ers o f in form ation sy stem so lu tio n s in th e w o rld . D uring this tim e, o th er innov ations h av e e m e rg ed , in clu d in g data w a re h o u s e a p p lia n ce s fro m v e n d o rs s u c h as N etezza (a cq u ire d b y IB M ), G re e n p lu m (a c q u ire d b y EM C), D A TA llegro (a cq u ire d b y M icroso ft), and p e rfo rm a n ce m a n a g e m e n t a p p lia n ce s that e n a b le real-tim e p e rfo rm a n ce m onitoring. T h e s e in n o v ativ e so lu tio n s pro v id ed c o s t savin gs b e c a u s e th e y w e re plug- co m p atib le to leg a cy d ata w a re h o u s e solu tions.

In th e 2 0 1 0 s th e b ig b u z z has b e e n B ig D ata. M any b e lie v e that B ig D ata is g o in g to m ake a n im p a ct o n d ata w a reh o u sin g as w e know- it. E ith er th e y w ill find a w ay to c o e x ­ ist (w h ich se e m s to b e th e m o st lik e ly c a s e , a t least fo r sev eral y e ars) o r B ig D ata (a n d -Jie te ch n o lo g ie s th at c o m e w ith it) w ill m a k e trad itional data w a reh o u sin g o b s o le te . T h e te ch n o lo g ie s that ca m e w ith B ig D ata in clu d e H a d o o p , M ap R ed u ce, N oSQ L, H ive, a n d so forth. M aybe w e will s e e a n e w term co in e d in th e w o rld o f d ata that c o m b in e s th e n e e d s and cap ab ilities o f trad itional data w a reh o u sin g an d th e B ig D ata p h e n o m e n o n .

C h a ra c te ris tic s o f D a ta W a re h o u s in g

A co m m o n w ay o f in trod u cin g data w a reh o u sin g is to re fe r to its fu n d am en tal ch a ra cte r­ istics (s e e In m o n , 2 0 0 5 ):

• S u b je c t o r ie n t e d . D ata are organized b y detailed su b ject, su ch as sales, products, or custom ers, containing o n ly inform ation relevan t fo r d ecisio n support. Su b ject orienta­ tion e n ab les u sers to d eterm ine n o t o n ly h o w their b u sin ess is perform ing, b u t w h y . A data w areh o u se differs from a n o p eration al d atabase in th at m ost o p eration al d atabases h av e a product orientation and are tu n ed to hand le transactions that update th e dam- b ase. S u b ject orientation provides a m o re com p reh en siv e v iew o f th e organization.

• I n t e g r a t e d . In teg ratio n is clo s e ly re lated to s u b je c t orien tatio n . D ata w a re h o u s e s m u st p la c e d ata fro m d ifferen t s o u rc e s in to a co n siste n t form at. T o d o s o , th e y m ust

Descriptive Analytics

d eal w ith n am in g co n flicts an d d is cre p a n cie s a m o n g units o f m easu re. A d ata w are­ h o u s e is p re su m ed to b e to tally in tegrated .

• T im e v a r ia n t (tim e series) . A w a re h o u s e m aintains historical data. T h e data d o n o t n e cessa rily pro v id e cu rrent status (e x c e p t in real-tim e system s). T h e y d etect trend s, d ev iation s, an d lo n g -term relatio n sh ip s fo r fo recastin g an d com p ariso n s, lead­ ing to d e cisio n m akin g. E very d ata w a re h o u s e h a s a tem poral quality. T im e is th e o n e im portant d im en sio n th at all data w a re h o u se s m ust supp ort. D ata fo r analysis from m ultiple so u rce s co n tain s m ultiple tim e p o in ts (e .g ., daily, w e e k ly , m o n th ly view s). N onvolatile. After data are e n te re d into a data w areh o u se , users ca n n o t ch a n g e or up d ate th e data. O b s o le te data are d iscard ed , and ch a n g e s are re co rd ed a s n e w data.

T h e s e ch aracteristics e n a b le data w a r e h o u s e s to b e tu n ed alm o st e x clu siv e ly fo r data a c c e ss . S o m e ad d itional ch aracteristics m a y in clu d e th e fo llow ing:

• Web b a sed . D ata w a re h o u s e s a re typ ically d esig n e d to pro v id e a n efficien t co m p u tin g en v iro n m en t fo r W e b -b a s e d ap p licatio n s.

• R e la tio n a l/m u ltid im e n s io n a l. A d ata w a re h o u s e u s e s e ith er a relatio n al struc­ ture o r a m u ltid im ensional stru cture. A re ce n t survey o n m u ltid im ensional structures c a n b e fo u n d in R o m e ro an d A b e llo (2 0 0 9 ).

• C lien t/serv er. A data w a re h o u s e u se s th e client/server arch ite ctu re to p rovide e a sy a c c e s s fo r en d users.

• R e a l tim e. N e w e r d ata w a re h o u s e s p ro v id e real-tim e, o r activ e, d a ta -a cce s s and analysis cap ab ilitie s (s e e B a s u , 2 0 0 3 ; a n d B o n d e an d K u ck u k , 2 0 0 4 ).

• I n c l u d e m eta d a ta . A d ata w a r e h o u s e co n ta in s m etad ata (d ata a b o u t d ata) ab ou t h o w th e data a re org an ized an d h o w to e ffe ctiv e ly u s e them .

W h e re a s a data w a r e h o u s e is a re p o sito ry o f data, data w a reh o u sin g is literally the en tire p ro c e s s (s e e W atso n , 2 0 0 2 ). D ata w a re h o u sin g is a d iscip lin e th at results in appli­ ca tio n s that p ro v id e d e c is io n su p p o rt cap ab ility , allow s re ad y a c c e s s to b u sin e ss infor­ m ation , an d cre a te s b u sin ess insight. T h e th re e m ain typ es o f data w a re h o u se s a re data m arts, o p eratio n al d ata s to res (O D S ), an d e n te rp rise data w a re h o u se s (E D W ). In ad dition to d iscu ssin g th e s e th ree typ es o f w a re h o u s e s n e x t, w e a lso d iscu ss m etadata.

D a ta M arts

W h e re a s a data w a re h o u s e c o m b in e s d a ta b a se s a c ro ss a n e n tire en terp rise, a data m art is usu ally sm alle r a n d fo c u s e s o n a p a rticu la r s u b je c t o r d ep artm en t. A data m art is a s u b s e t o f a d ata w a re h o u se , ty p ically c o n s is tin g o f a sin gle s u b je c t a re a (e .g ., m arketing, o p e ra tio n s). A d ata m art c a n b e e ith er d e p e n d e n t o r in d e p e n d en t. A d ependent data m a rt is a s u b s e t th at is c re a te d d irectly fro m th e data w a re h o u se . It h a s th e ad vantages o f u sin g a co n siste n t d ata m o d el and p ro v id in g q u ality data. D e p e n d e n t data m arts sup­ p o rt th e c o n c e p t o f a sin g le e n te rp rise-w id e data m o d e l, but th e data w a reh o u se m u st b e co n stru cte d first. A d ep e n d e n t d ata m art e n s u re s that th e e n d user is view in g th e sam e v e rsio n o f th e d ata th at is a c c e s s e d b y all o th e r data w a re h o u s e u sers. T h e high c o s t o f d ata w a reh o u se s lim its th e ir u s e to larg e co m p a n ie s . As a n alternativ e, m any firm s u s e a lo w er-co st, sca le d -d o w n v e rsio n o f a data w a re h o u s e re fe rre d to as a n in d ep en d en t d a ta m art. An independent data m a rt is a sm all w a r e h o u s e d e sig n e d fo r a strateg ic b u sin ess u n it (S B U ) o r a d ep artm en t, b u t its s o u rc e is n o t a n EDW.

O p e ra tio n a l D a ta S to re s

An operational data store (ODS) provides a fairly recen t form o f cu stom er inform ation file (C IF). This type o f d atabase is often u s e d as a n interim staging area fo r a data w are­ h ou se. U nlike th e static contents o f a data w areh o u se, th e con ten ts o f an O D S are updated throughout th e cou rse o f b u sin ess op eration s. An O D S is u sed fo r short-term decisions

Chapter 3 • Data W arehousing 1 1 5

involving m ission-critical applications rather than for th e m ed iu m - and long-term d ecisio ns associated w ith an EDW . An O D S is similar to short-term m em ory in that it stores o n ly very recent inform ation. In com parison, a data w areh o u se is like lo n g -te n n m em ory b e c a u s e it stores p erm anent inform ation. An O D S consolid ates data fro m multiple sou rce system s and provides a n e ar-re al-tim e , integrated view o f volatile, current data. T h e e x ch a n g e transfer, and load (ETL) p ro cesse s (d iscu ssed later in this ch ap ter) fo r a n O D S are identical to th o se for a data w areho u se. Finally, oper m arts (s e e Im hoff, 2 0 0 1 ) are cre a te d w h en op erational data n e e d s to b e analyzed m u ltidim ensional^. T h e data for an o p e r m art c o m e from a n ODS.

E n te rp ris e D a ta W a re h o u s e s (EDW) \n en terp rise data w areh ou se (EDW) is a la rg e-sc a le d ata w a re h o u s e th at is u se d acro ss th e e n te rp rise fo r d e cisio n su p p ort. It is th e ty p e o f d ata w a re h o u s e th at Is le o f Capri d e v e lo p e d , as d e s c rib e d in th e o p e n in g v ignette. T h e larg e-scale n atu re provides integration o f d ata fro m m an y s o u rc e s in to a stand ard fo rm at fo r e ffectiv e B I an d d e cisio n su p p ort a p p licatio n s. E D W are u s e d to p rovide data for m a n y typ es o f ^ i n c l u d i n g CRM supp ly ch a in m a n a g e m e n t (SC M ), b u sin e ss p e rfo rm a n ce m a n a g e m e n t (B P M ), b u si­ ness activity m o n ito rin g (B A M ), p ro d u ct life -cy cle m a n a g e m e n t (PLM ), rev en u e m a n a g e ­ m en t an d so m e tim e s e v e n k n o w le d g e m a n a g e m e n t system s (K M S). A p p licatio n C a se 3.1 sh o w s th e variety o f b e n e fits th at te le co m m u n ica tio n c o m p a n ie s lev erag e fro m im p le ­ m enting d ata w a r e h o u s e d riven an alytics solu tions.

M etad ata Metadata are d ata a b o u t data (e .g ., s e e Sen , 2 0 0 4 ; and Z h ao, 2 0 0 5 ). M etadata d e s c rib e the stru cture o f an d s o m e m e an in g a b o u t d ata, th e re b y con trib u tin g to th e n e ffe ctiv e o r

Application Case 3.1 A Better Data Plan: Well-Established TELCOs Leverage Data Warehousing and Analytics to Stay on Top in a Competitive Industry M o bile serv ice p ro v id ers ( i.e ., T e le co m m u n ica tio n C o m p an ies, o r TE L C O s in sh o rt) that h e lp e d trigger th e e x p lo s iv e g ro w th o f th e industry in th e m id- to later 1 9 9 0 s h av e lo n g re a p e d th e b e n e fits o f b e in g first to m arket. B u t to stay co m p etitiv e , th e s e co m p a n ie s m ust co n tin u o u sly re fin e e v ery th in g fro m cu sto m e r serv ice to p la n pricin g. In fa ct, v e te ra n carriers fa ce m an y o f th e sa m e ch a lle n g e s th a t u p -an d -co m in g carriers d o : re ta in in g cu sto m e rs, d e cre a sin g costs, fin e-tu n in g p ricin g m o d e ls, im p ro vin g cu sto m e r sat­ isfactio n , a cq u irin g n e w cu sto m e rs a n d u n d erstan d ­ ing th e ro le o f so cia l m ed ia in cu sto m e r loyalty

H ighly ta rg e te d data an alytics p lay a n ev er- m o re-critical ro le in h e lp in g carriers s e cu re o r im p ro v e th e ir stan d in g in an in creasin g ly co m p e ti­ tive m ark e tp lace . H e re ’s h o w s o m e o f th e w o r ld s lead in g p ro v id ers are cre a tin g a stro n g future b a se d o n so lid b u s in e s s an d cu sto m e r in tellig en ce.

C u s to m e r R e te n tio n

It’s n o s e c r e t that th e s p e e d an d s u c c e s s w ith w h ich a p ro vid er h an d les s e rv ic e re q u e sts d irectly affects cu sto m e r satisfactio n a n d , in turn, th e p ro p e n sity to ch u rn . B u t g ettin g d o w n to w h ic h facto rs h av e th e g re ate st im p act is a ch a lle n g e .

“I f w e co u ld tra c e th e s te p s involved w ith e a c h p ro c e s s , w e co u ld u n d e rsta n d p o in ts o f failu re an d a c c e le ra tio n ,” n o te s R o x a n n e G arcia, m a n a g e r o f th e C o m m ercial O p e ra tio n s C en ter fo r T e le fo n ic a d e A rgentina. “W e co u ld m e a su re w o rk flo w s b o th w ith in an d a cro ss fu n ctio n s, an ticip ate rath er th an re a c t to p e rfo rm a n ce ind icato rs, a n d im p ro v e th e ov erall satisfactio n w ith o n b o a rd in g n e w cu sto m e rs.”

T h e co m p an y ’s so lu tio n w a s its traceability p ro ­ je ct, w h ich b e g a n w ith 10 d ashboard s in 2009- It has sin ce realized U S$2.4 m illion in annu alized revenu es

( C o n tin u ed )

1 1 6 Part II • D escriptive Analytics

Application Case 3.1 (Continued) an d c o s t savings, sh o rten ed cu sto m e r provisioning tim es an d re d u ce d cu stom er d efection s b y 30% .

C o s t R e d u c tio n

Staying a h e a d o f th e g am e in a n y indu stry d ep en d s, in large p art, o n k e e p in g co s ts in lin e . F o r F ra n c e ’s B o u y g u e s T e le c o m , c o s t re d u ctio n ca m e in th e fo rm o f a u to m atio n . A ladin, th e co m p a n y ’s T e rad ata-b ase d m ark etin g o p e ra tio n s m an a g e m e n t system , au to ­ m ates m arketing/ com m u nications co lla tera l p ro d u c­ tion . It d eliv ered m o re th an U S$1 m illio n in savings in a s in g le y e a r w h ile tripling e m a il cam p aig n and c o n te n t p ro d u ction .

“T h e g o al is to b e m o re produ ctive an d re sp o n ­ siv e, t o sim plify team w ork, [and] to standardize and p ro tect o u r e x p e rtise ,” n o tes C atherine C orrado, the co m p a n y ’s p ro je ct lead a n d retail com m u nications m anager. “[Aladin lets] te a m m e m b ers fo cu s o n value- ad d ed w o rk b y red ucing low -value tasks. T h e end result is m o re quality an d m o re creative [output].”

An u n in ten d ed b u t very' w e lco m e b en efit o f A ladin is th at o th er d ep artm ents h av e b e e n inspired to b e g in d ep loying sim ilar p ro je cts fo r everything from call c e n te r supp ort to product/offer lau n ch p ro cesses.

C u s to m e r A c q u is itio n

W ith m ark et p e n e tra tio n n e a r o r a b o v e 100% in m a n y co u n tries, th an k s to co n su m e rs w h o o w n m u ltip le d ev ice s, th e issu e o f n e w cu sto m e r a cq u isi­ tio n is n o sm all ch a lle n g e . P ak istan ’s larg est carrier, M o b ilin k , a lso fa c e s th e difficu lty o f o p e ra tin g in a m a rk e t w h e re 9 8 % o f u sers h av e a p re-p aid p lan that re q u ire s regu lar p u rch a ses o f ad d itional m inutes.

“T o p p in g up, in particular, k e e p s th e revenues strong an d is critical to ou r co m p an y ’s grow th,” says U m er Afzal, sen io r m anager, B I. “Previously w e lack ed th e ability to e n h a n ce this asp ect o f increm en­ tal grow th. O u r sales inform ation m o d el gave us that ability b e c a u se it h elp ed th e distribution team plan sales tactics b a se d o n sm arter data-driven strategies that k e e p ou r suppliers [o f SIM cards, scratch cards an d ele ctro n ic to p -u p capability] fully stocked .

As a result, M obilink has n o t only grow n su b ­ scrib e r recharges b y 2% b u t also e x p a n d e d newr cus­ to m er acquisition b y 4% a n d im proved the profitability

o f th o se sales b y 4%.

S o c ia l N e tw o r k in g

T h e e x p a n d in g u s e o f s o c ia l n e tw o rk s is c h a n g ­ in g h o w m a n y o rg a n iz a tio n s a p p r o a c h ev e ry th in g fro m c u s to m e r s e rv ic e to s a le s a n d m ark e tin g . M ore carrie rs a r e tu rn in g th e ir a tte n tio n to s o c ia l n e t­ w o rk s to b e tt e r u n d e rstan d and in flu e n c e cu sto m e r

b e h a v io r. M o b ilin k h a s in itia te d a s o c ia l n e tw o rk a n a ly ­

sis p r o je c t th a t w ill e n a b le th e c o m p a n y to e x p lo r e th e c o n c e p t o f viral m ark e tin g an d id en tify k e y in flu e n ce rs w h o c a n a c t a s b ra n d a m b a ssa d o rs to c r o s s -s e ll p ro d u cts. V e lc o m is lo o k in g fo r sim ilar k e y in flu e n c e r s a s w e ll a s lo w -v a lu e cu sto m e rs w h o s e s o c ia l v a lu e c a n b e le v e ra g e d to im p ro v e e x is tin g re la tio n sh ip s . M e an w h ile, S w is s co m is lo o k in g t o c o m b in e th e s o c ia l n e tw o rk a s p e c t o f cu sto m e r b e h a v io r w ith th e re s t o f its a n aly sis o v e r th e n e x t se v e ra l m onths.

R is e t o t h e C h a lle n g e

W h ile e a c h m ark e t p re s e n ts its o w n u n iq u e c h a l­ le n g e s , m o st m o b ile carrie rs s p e n d a g re a t d e a l o f tim e a n d r e s o u r c e s cre a tin g , d e p lo y in g a n d re fin in g p la n s to a d d re ss e a c h o f th e c h a lle n g e s o u tlin ed h e re . T h e g o o d n e w s is th a t ju st a s th e in d u stry an d m o b ile te c h n o lo g y h a v e e x p a n d e d an d im p ro v ed o v e r th e y e a rs , s o a ls o h a v e th e d ata a n a ly tics s o lu ­ tio n s th a t h a v e b e e n c re a te d to m e e t th e s e c h a l­ le n g e s h e a d on .

Sou n d data analysis u se s existin g cu stom er, b u sin ess an d m arket in telligence to pred ict and influ­ e n c e fu ture b eh av io rs an d ou tco m es. T h e e n d result is a sm arter, m o re agile an d m o re su ccessfu l ap p roach to gaining m ark et sh are and im proving profitability.

Q u e s t i o n s f o r D i s c u s s i o n

1. W h a t a re th e m ain ch a lle n g e s fo r TELCOs? 2. H o w c a n d ata w a re h o u s in g an d data analytics

h e lp TELC O s in o v e rco m in g th e ir ch allen ges?

3. W h y d o y o u th in k T ELC O s a re w e ll su ited to tak e full a d v an tag e o f data analytics?

Source: Teradata Magazine, Case Study by Colleen Marble, “A Better D ata Plan: Well-Established Telcos Leverage Analytics to Stay o n Top in a Competitive Industry" h tt p :/ / w w w . tera d a ta m a g a z in e .co m / v l3 n 0 1 / F ea tu re s/ A -B etter-D ata- Plan/ (accessed September 2013).

Chapter 3 • Data W arehousing 1 1 7

in effectiv e u se. M eh ra ( 2 0 0 5 ) in d icated th at few org an izatio n s really u n d e rstan d m etad ata, and fe w e r u n d e rsta n d h o w to d esig n a n d im p lem en t a m etad ata strategy. M etad ata are generally d efin e d in term s o f u sa g e as te ch n ica l o r b u sin e ss m etad ata. P attern is an o th e r w ay to v iew m etad ata. A cco rd in g to th e p a ttern view , w e c a n d ifferen tiate b e tw e e n syn­ tactic m etad ata (i.e ., data d escrib in g th e sy n ta x o f d ata), structural m etad ata (i e ., data describing th e stru ctu re o f th e d ata), a n d s em an tic m etad ata ( i.e ., d ata d e s crib in g the

m ean in g o f th e d ata in a s p e c ific d om ain ). W e n e xt e xp la in traditional m etadata patterns an d insights into h o w to im plem ent

an effective m etadata strategy via a holistic ap p roach to enterprise m etadata integration. T h e ap p roach inclu des ontology and m etadata registries; enterprise inform ation integration Eli); extraction, transform ation, and load ( E H ) ; and service-oriented architectures (SOA).

Effectiveness, extensibility, reusability, interoperability, efficiency a n d perform ance, evolution, entitlement, flexibility, segregation, user interface, versioning, versatility, and lo w m aintenance cost are som e o f th e k e y requirem ents for building a successful m etadata-driven enterprise.

A cco rd in g t o K assam (2 0 0 2 ) , b u sin ess m etad ata co m p rise in form ation th a t in cre a se s ou r u n d erstan d in g o f trad itional (i.e ., stru ctu red ) data. T h e prim ary p u rp o s e o f m etad ata should b e to p ro v id e co n te x t to th e re p o rted data; that is, it p ro v id es e n rich in g in form a- tio n that lea d s to th e cre atio n o f k n o w le d g e. B u s in e s s m etad ata, th o u g h d ifficu lt to p ro ­ vide efficien tly, re le a s e m o re o f th e p o ten tial o f stru ctured data. T h e c o n te x t n e e d n ot b e th e sa m e fo r all u sers. In m a n y w ay s, m etad ata assist in th e c o n v e rs io n o f data and inform ation into k n o w le d g e. M etad ata fo rm a fo u n d atio n fo r a m e ta b u sin ess arch itectu re (s e e B e ll, 2 0 0 1 ). T a n n e n b a u m ( 2 0 0 2 ) d e s crib e d h o w to identify m etad ata req u irem en ts. V adu va an d V etterli (2 0 0 1 ) p ro v id e d a n o v erv iew o f m etad ata m a n a g e m e n t fo r data w a re ­ h o u sin g Z h ao (2 0 0 5 ) d e s crib e d five lev els o f m etad ata m a n a g e m e n t m aturity: ( 1 ) ad h o c, ( 2 ) d isco v e red , ( 3 ) m an ag e d , ( 4 ) op tim ized , an d ( 5 ) au tom ated . T h e s e lev els h e lp in understanding w h e r e a n o rg an izatio n is in term s o f h o w an d h o w w e ll it u se s its m etad ata.

T h e d esig n , cre a tio n , a n d u s e o f m etad ata— d escrip tiv e o r su m m ary data a b o u t data— an d its a cco m p a n y in g stand ard s m ay in v o lv e e th ical issu es. T h e r e are eth ical con sid eratio n s in v o lv e d in th e c o lle c tio n a n d o w n e rsh ip o f th e in form ation co n ta in e d in m etad ata, in clu d in g p rivacy and in tellectu al p ro p erty issu es that a rise in th e d esign , co lle ctio n , an d d isse m in a tio n stag es (fo r m o re, s e e B ro d y , 2 0 0 3 ).

SECTION 3 -2 REVIEW QUESTIONS

1 . W h a t is a d ata w areho u se? 2 . H o w d o e s a d ata w a re h o u s e d iffer fro m a d atabase?

3 . W h at is a n O D S? 4 . D ifferen tiate a m o n g a data mart, an O D S , an d a n EDW . 5. E x p la in th e im p o rta n ce o f m etad ata.

3.3 D A T A W A R E H O U S IN G P R O C E S S O V E R V IE W O rg an izatio n s, p riv ate an d p u b lic, co n tin u o u sly c o lle c t d ata, in fo rm atio n , a n d k n o w le d g e at a n in cre a sin g ly a c c e le ra te d rate a n d sto re th e m in co m p u te riz e d sy stem s. M aintainin g and u sin g th e s e data a n d in fo rm a tio n b e c o m e s e x tr e m e ly c o m p le x , e s p e c ia lly as scalab ility iss u e s arise. In ad d itio n , th e n u m b e r o f u se rs n e e d in g to a c c e s s th e in fo rm a­ tio n c o n tin u e s to in c re a se a s a resu lt o f im proved reliab ility a n d av ailability o f n e tw o rk a c c e ss e s p e c ia lly th e In te rn e t. W o rk in g w ith m u ltip le d a ta b a se s, e ith e r in te g ra te d in a data w a r e h o u s e o r n o t, h a s b e c o m e a n e x tr e m e ly d ifficu lt ta sk req u irin g co n s id e ra b le e x p e rtis e , b u t it c a n p ro v id e im m e n se b e n e fits far e x c e e d in g its c o s t. As a n illustrative e x a m p le ,’ F ig u re 3 .2 s h o w s b u s in e s s b e n e fits o f th e e n te rp rise d ata w a r e h o u s e b u ilt by T erad ata fo r a m a jo r a u to m o b ile m an u factu rer.

1 1 8 Part II • D escriptive Analytics

E n te rp r is e D a ta W a re h o u s e One management and analytical platform

for product configuration, warranty, ___ and diagnostic readout data ______

I T A r c h i t e c t u r e S t a n d a r d iz a t io n

One strate g ic platform for busin ess intelligence and

com pliance reporting

FIGURE 3.2 Data-Driven Decision Making— Business Benefits of an Enterprise Data Warehouse.

Application Case 3.2 Data Warehousing Helps MultiCare Save More Lives In th e sp rin g o f 2 0 1 2 , lead e rsh ip at M ultiCare H ealth System (M ultiC are)— a T a c o m a , W a s h in g to n -b a s e d h e alth system — realized th e results o f a 12-m on th jo u rn e y to re d u ce sep tice m ia.

T h e e ffo rt w a s s u p p o rte d b y the sy stem ’s to p lea d e rsh ip , w h o p articip ated in a d ata-d riven a p p r o a c h to prioritize care im p ro v e m en t b a s e d o n a n a n aly sis o f re so u rce s c o n s u m e d a n d v ariation in c a re o u tco m e s . R ed u cin g se p tice m ia (m ortality rates) w a s a to p priority fo r M ultiCare as a resu lt o f th ree h o sp ita ls p erfo rm in g b e lo w , an d o n e that w a s p e r­ fo rm in g w e ll b e lo w , n a tio n a l m ortality averages.

In S e p te m b e r 2 0 1 0 , M ultiCare im p lem e n te d H ealth Catalyst’s A daptive D ata W a re h o u s e , a h e a lth c a re -s p e c ific data m o d e l, and s u b s e q u e n t clin ­ ica l an d p ro ce s s im p ro v e m en t s erv ices to m e asu re a n d e ffe c t c a re th ro u g h organ izatio n al and p ro ce s s im p ro v em en ts. T w o m a jo r facto rs co n trib u ted to the rap id re d u ctio n in s ep tice m ia m ortality.

C lin ic a l D a t a t o D riv e I m p r o v e m e n t

T h e A d aptive D ata W a re h o u se™ o rg an ize d a n d sim ­ p lified d ata from m u ltip le data s o u rce s a cro ss the co n tin u u m o f ca re . It b e c a m e the sin gle s o u rc e o f truth re q u isite to s e e c a re im p ro v e m en t o p p o rtu n i­ ties an d to m e a su re ch a n g e . It a lso p ro v ed to b e a n im p ortan t m e a n s to un ify clin ical, IT , an d financial

lead e rs a n d to drive acco u n tab ility for p e rfo rm an ce im p ro v em en t.

B e c a u s e it p ro v ed d ifficult to d efin e se p sis d u e to th e c o m p le x co m o rb id ity facto rs lead in g to s e p ­ ticem ia , M ultiC are p artn ered w ith H ealth Catalyst to re fin e th e clin ical d efin itio n o f sep sis. H ealth Catalyst’s d ata w o rk a llo w e d M ultiCare to e x p lo re a ro u n d th e b o u n d a rie s o f the d efin itio n a n d to ulti­ m ately settle o n a n algorithm th at d efin e d a s ep tic p atient. T h e iterative w o rk resulted in in cre a se d c o n ­ fid e n c e in th e s e v e re sep sis co h o rt.

S y s te m -W id e C r itic a l C a r e C o lla b o ra tiv e

T h e e sta b lish m e n t an d co lla b o ra tiv e e ffo rts o f p e r­ m an e n t, in teg rated te am s co n sistin g o f clin ician s, te ch n o lo g is ts , analysts, a n d qu ality p e rs o n n e l w e re e ss e n tia l fo r a c ce le ra tin g M ultiC are’s e ffo rts to re d u ce s e p tic e m ia m ortality. T o g e th e r th e co lla b o ra ­ tive a d d re sse d th re e k e y b o d ie s o f w ork— standard o f c a re d efin itio n , e a rly id en tificatio n , an d e fficie n t d elivery o f d e fin e d -ca re standard.

S t a n d a r d o f C a r e : S e v e r e S e p s is O r d e r S e t

T h e Critical Care C o llab o rativ e stream lin ed sev eral s e p sis o r d e r sets fro m acro ss th e organ ization into o n e sy stem -w id e standard fo r th e c a re o f sev erely

C hapter 3 • D ata W arehou sing 119

sep tic p atients. A dult p atie n ts p re sen tin g w ith sep sis re ce iv e th e sam e c a re , n o m atter a t w h ic h MultiCare h osp ital th e y p resen t.

E a r l y I d e n t i f i c a t i o n : M o d ifie d E a r ly W a r n in g S y s te m (M E W S )

M ultiCare d e v e lo p e d a m o d ified early w arning sys­ te m (M EW S) d a sh b o a rd th at lev era g e d th e co h o rt d efin ition an d th e clin ical EMR to q u ick ly identify p atien ts w h o w e re tren d in g tow ard a su d d en d o w n ­ turn. H osp ital s ta ff con stan tly m o n ito r M EW S, w h ich serv e s as a n e a rly d e te c tio n to o l fo r careg iv ers to p ro v id e p re e m p tiv e interventions.

E f f ic ie n t D e liv e r y : C o d e S e p s is ( “T im e I s T i s s u e ”)

T h e final k e y p ie c e o f clin ical w o rk u n d e rta k e n b y th e C o llab o rativ e w a s to e n su re tim ely im p lem e n ta­ tio n o f th e d e fin e d stand ard o f c a re to p atien ts w h o are m o re e fficie n tly identified . T h a t m o d el alread y ex ists in h e a lth c a re and is k n o w n as th e ‘‘c o d e ” p ro ­ c e s s . Sim ilar to o th e r “c o d e ” p ro c e s s e s (c o d e traum a,

c o d e n e u ro , c o d e STE M I), c o d e sep sis a t M ultiCare is d esig n e d to b rin g to g e th e r e ss e n tia l careg iv ers in o rd e r to efficie n tly d e liv e r tim e-sen sitiv e, life-sav ing treatm en ts to th e p atie n t p re sen tin g w ith sev ere

sep sis. I n ju s t 1 2 m o n th s , M u ltiC are w a s a b le to

re d u c e s e p tic e m ia m o rtality rate s b y a n a v e ra g e o f 2 2 p e rc e n t, le a d in g to m o re th a n $ 1 .3 m illio n in v a lid a te d c o s t sav in g s d u rin g th a t sa m e p e rio d . T h e s e p s is c o s t re d u ctio n s a n d q u a lity o f c a r e im p ro v e ­ m e n ts h a v e raised th e e x p e c ta tio n th a t sim ilar re su lts c a n b e re a liz e d in o th e r a re a s o f M ultiC are, in clu d in g h e a rt fa ilu re , e m e rg e n c y d ep a rtm e n t p e rfo rm a n c e , a n d in p a tie n t th rou gh p u t.

Q u e s t i o n s f o r D i s c u s s i o n 1. W h a t d o y o u th in k is th e ro le o f data w a re h o u s ­

ing in h e a lth ca re system s? 2. H o w did M ultiCare u se d ata w a reh o u sin g to

im p ro ve h e alth ou tco m es?

Source: healthcatalyst.com /success_stories/m ulticare-2 (ac­ cessed February 2013)-

M anv o rg an izatio n s n e e d to cre a te data w a re h o u s e s — m assiv e data s to re s o f tim e- series d ata fo r d e c is io n su p p ort. D ata are im p orted fro m v arious e x te rn a l a n d internal re so u rces a n d are c le a n s e d a n d org an ized in a m a n n er c o n s is te n t w ith th e o rg a n iz a tio n s n eed s. After th e d ata a re p o p u lated in th e d ata w a re h o u s e , data marts ca n b e lo a d e d lo r a sp e cific area o r d ep artm en t. A lternatively, d ata m arts c a n b e cre a te d first, as n e e d e d , and th e n in teg rated in to a n ED W . O ften , th o u g h , data m arts are n o t d e v e lo p e d , b u t data are sim ply lo a d e d o n to PCs o r left in th e ir original state fo r d irect m an ip u latio n u sin g B I to ols.

In Figu re 3 -3 , w e s h o w th e d ata w a re h o u s e c o n c e p t. T h e fo llo w in g are th e m ajo r

co m p o n e n ts o f th e d ata w a reh o u sin g p ro cess:

• D a ta s o u rc e s . D a ta a re so u rce d fro m m u ltiple in d e p e n d e n t o p e ra tio n a l leg acy sy stem s a n d p o ssib ly fro m e x te rn a l d ata providers (s u c h as th e U.S. C e n su s). D ata m ay a lso c o m e fro m a n O LTP o r ERP system . W e b d ata in th e fo rm o f W e b lo g s m ay

also fe e d a d ata w a reh o u se . • D a ta e x tra c tio n a n d tra n s fo rm a tio n . D ata a re e x tra cted an d p ro p e rly tran s­

fo rm e d u sin g cu stom -w ritten o r co m m e rcia l so ftw are ca lle d ETL. • D a ta lo a d in g . D ata a re lo a d e d into a staging a re a, w h e re th e y a re tran sform ed

an d cle a n s e d . T h e d ata are th e n re ad y to lo a d in to th e d ata w a re h o u s e and/or data

m arts. „ , . C o m p reh en siv e d a ta b a se. Essentially, this is th e E D W to su p p o rt all d e cisio n

an aly sis b y p ro v id in g re le v a n t su m m arized and d eta iled in fo rm atio n originating

fro m m an y d iffe ren t so u rce s. • M eta d a ta . M etad ata are m ain tain ed s o that th e y c a n b e a s s e s s e d b y I T p e rs o n n e

an d u sers. M etad ata in clu d e softw are p ro g ram s a b o u t d ata an d ru les fo r organizing data su m m aries th at are e a sy to in d e x a n d se a rch , e sp e cia lly w ith W e b to o ls.

1 2 0 P ari II • D escriptive Analytics

No data marts option Applications

(Visualization)Data Sources'

Select

Transform

Integrate

M id d lew a re tools. M i d d l e w a r e o t h f t ^ m T y e m p lo y a m an- u sers s u c h as analysts m ay O bjeT K to a ^ e s s d a tt T ta o e are m any a g e d q u e ry en v iron m en t, s u c h as B us' - > ^ w ith d ata s to red in the fro n t-e n d ap p lica tio n s th at b u s in e s - ̂re p o rtin g to o ls, a n d d ata visualiza- d ata rep o sito ries, inclu d ing d ata m m m g, OLAP, rep ortin g

tio n to ols.

— A c c e s s ■ n

Data m art L [M arketingjJ

m 4

business reports

Data mart CD JJ2 ll̂ lData/textmining k

Data mart (Finance)-^]

a. <

H I

□LAR Dashboard, Web

\ rrr~f" A

H Data m artL1 yll 1 Custom-built j applications F IG U R E 3 .3 A D ata W a re h o u se F ra m e w o rk a n d V ie w s.

SECTION 3 .3 REVIEW QUESTIONS

1 . D e s c r ib e th e data w a reh o u sin g p ro ce ss. 2 D e s c r ib e th e m ajo r co m p o n e n ts o f a d ata w a re h o u se .

3 . Id entify a n d d iscu ss th e ro le o f m id d lew are to ols.

3 4 DATA WAREHOUSING ARCHITECTURES

T h e re are se v e ra l b a sic in form ation hou sin g . G e n e ra lly sp e ak in g , th e s e arc i ^ a rch ite ctu res are th e m o st co m m o n n-tier arch itectu res, o f w h ic h tw o -tie r a q { (s e e F igu res 3 .4 an d 3 .5 ), b u t so m e tim e s th e re is sim p ly o n e tier, i n yp

FIG U R E 3 .4 Architecture of a Three-Tier Data W arehouse.

C hapter 3 • Data W arehousing 121

( f t

̂ j v _ )

Tier 1: Tier 2: Client workstation Application and

database server

F IG U R E 3 .5 A rc h ite c tu re o f a T w o -T ie r D ata W are ho use .

architectures are k n o w n to b e c a p a b le o f serving th e n e e d s o f la rg e-sca le , p e rfo rm a n ce - d em and ing in form ation system s su ch as data w a re h o u s e s . R eferrin g to th e u se o f n -tiered architectures fo r data w a reh o u sin g , H offer e t al. (2 0 0 7 ) d istin gu ish ed am o n g th e se arch i­ tectu res b y d ividing th e data w a re h o u s e in to th re e parts:

1 . T h e d ata w a r e h o u s e itself, w h ich co n ta in s th e data an d a sso cia ted softw are 2 . D ata a cq u isitio n (b a c k -e n d ) so ftw are, w h ich e xtracts data fro m le g a cy sy stem s an d

e x te rn a l so u rce s, co n so lid a te s an d su m m arizes th em , an d load s th e m into th e data w a reh o u se

3 . C lient (fro n t-e n d ) so ftw are, w h ic h allow s u sers to a c c e s s and an alyze data fro m th e w a re h o u s e ( a D SS/BI/business a n alytics [BA] e n g in e )

In a th re e -tie r a rch ite ctu re, o p e ra tio n a l system s c o n ta in th e data a n d th e so ftw are fo r -data acq u isitio n in o n e tier (i.e ., th e serv er), th e d ata w a re h o u s e is an o th e r tier, a n d th e third tier in clu d es th e DSS/BI/BA e n g in e ( i.e ., the a p p lica tio n serv e r) a n d th e c lie n t (s e e Figure 3 .4 ). D ata fro m th e w a re h o u s e are p ro c e s s e d tw ice an d d ep o site d in a n ad d itional m ultid im ensional d a ta b a s e , org an ized fo r e a sy m u ltid im en sion al an alysis an d p re s e n ta ­ tion, o r re p lica te d in d ata m arts. T h e ad v an tag e o f th e th ree -tier arch ite ctu re is its s e p a ra ­ tion o f th e fu n ctio n s o f th e data w a re h o u se , w h ich e lim in ate s re s o u rce co n strain ts an d m akes it p o ss ib le to e a sily c re a te data marts.

In a tw o -tie r arch itectu re, th e D SS e n g in e p h ysically ru ns o n th e sa m e hard w are platform as th e data w a r e h o u s e ( s e e Figure 3-5). T h e re fo re , it is m o re e c o n o m ic a l than d ie th ree -tier stru cture. T h e tw o -tie r arch ite ctu re c a n h av e p e rfo rm a n ce p ro b lem s fo r large data w a re h o u se s th at w o rk w ith d ata-in ten siv e ap p lica tio n s fo r d e cisio n supp ort.

M u ch o f th e c o m m o n w isd o m assu m e s a n ab so lu tist ap p ro a ch , m ain tain in g that o n e so lu tio n is b e tte r th an th e o th er, d esp ite th e o rg an izatio n ’s circu m sta n ces a n d u n iq u e r.eeds. T o fu rth er c o m p lic a te th e s e arch itectu ral d e cisio n s, m an y co n su ltan ts a n d softw are ven d ors fo cu s o n o n e p o rtio n o f th e arch itectu re, th e re fo re lim iting th e ir ca p a city and m otivation to assist a n o rg an izatio n th ro u g h the o p tio n s b a s e d o n its n e ed s. B u t th ese asp ects are b e in g q u e s tio n e d a n d an aly zed . F o r e x a m p le , B a ll (2 0 0 5 ) p ro v id e d d e ci­ sion criteria fo r o rg an izatio n s that p lan to im p lem e n t a B I a p p lica tio n and h a v e alread y determ ined th e ir n e e d fo r m u ltid im ensional data m arts b u t n e e d h e lp d eterm in in g th e appropriate tiered arch itectu re. H is criteria rev o lv e aro u n d fo re ca stin g n e e d s fo r s p a c e and s p e e d o f a c c e s s ( s e e B all, 2 0 0 5 , fo r details).

D ata w a reh o u sin g a n d th e In te rn e t are tw o k e y te c h n o lo g ie s th at o ffe r im p ortan t so lu tion s fo r m a n a g in g co rp o ra te data. T h e in teg ratio n o f th e s e tw o te c h n o lo g ie s p ro ­ duces W e b -b a s e d d ata w a reh o u sin g . In Figu re 3 .6 , w e sh o w th e arch ite ctu re o f W e b - Dised d ata w a reh o u sin g . T h e arch ite ctu re is th ree tiered and in clu d es th e PC clie n t, W e b server, an d ap p lica tio n serv er. O n th e clie n t sid e, th e u s e r n e e d s a n In te rn e t c o n n e c tio n and a W e b b ro w se r (p r e fe ra b ly Ja v a e n a b le d ) throu gh th e fam iliar grap h ical u s e r in te r­ fa ce (G U I). T h e Internet/intranet/extranet is th e co m m u n ica tio n m ed iu m b e tw e e n clien t

1 2 2 Part II • D escriptive Analytics

FIGURE 3.6 Architecture of Web-Based Data Warehousing.

an d servers. O n th e se rv e r sid e, a W e b s e r v e r is u se d to m a n a g e th e in flo w an d ou tflow o f in form ation b e tw e e n c lie n t a n d server. It is b a c k e d b y b o th a d ata w a r e h o u s e a n d an ap p licatio n server. W e b -b a s e d data w a re h o u s in g o ffers sev eral co m p ellin g ad vantages, in clu d in g e a s e o f a c c e s s , p latfo rm in d e p e n d e n c e , an d lo w e r cost.

T h e V ang u ard G ro u p m o v e d to a W e b -b a s e d , th re e -tie r a rch ite ctu re fo r its en terp rise arch ite ctu re to in teg rate all its data an d p ro v id e cu sto m ers w ith th e sa m e v iew s o f data as internal u se rs (D ra g o o n , 2 0 0 3 ). L ikew ise, H ilton m igrated all its in d e p e n d en t client/ serv er sy stem s to a th ree -tier d ata w a re h o u s e , u sin g a W e b d esig n e n te rp rise system . This c h a n g e in v o lv ed a n in v e stm e n t o f $3-8 m illion (e x clu d in g la b o r) an d a ffe c te d 1 ,5 0 0 users. It in cre a se d p ro ce s s in g e ffic ie n c y (s p e e d ) b y a fa cto r o f six . W h e n it w a s d ep lo y e d , H ilton e x p e c te d to sav e $ 4 .5 to $5 m illion annu ally. Finally, H ilto n e x p e rim e n te d w ith D ell’s clu s­ terin g ( i.e ., p arallel co m p u tin g ) te ch n o lo g y to e n h a n c e scalab ility a n d s p e e d (s e e A nthes, 2 0 0 3 ).

W e b arch ite ctu res fo r data w a reh o u sin g are sim ilar in stru cture to o th e r d ata w are­ h o u sin g a rch itectu res, requ irin g a d esig n c h o ic e fo r h o u sin g th e W e b d ata w areh o u se w ith th e tran sactio n serv e r o r as a se p a ra te se rv e r(s). P a g e-lo a d in g s p e e d is a n im portant co n sid era tio n in d esig n in g W e b -b a s e d a p p lica tio n s; th e re fo re, serv e r ca p a city m u st b e p la n n e d carefully.

Several issu es m u st b e co n sid e re d w h e n d e cid in g w h ic h arch ite ctu re to u se . Am ong th e m are th e fo llow ing:

• W hich d a ta b a s e m a n a g e m e n t system (D BM S) s h o u ld b e u s e d ? M ost data w a re h o u s e s a re built u sin g relatio nal d a ta b a s e m a n a g e m e n t system s (R D B M S). O racle (O ra c le C o rp o ratio n , o r a c le .c o m ), SQ L Serv er (M icro so ft C o rp o ratio n , m icro so ft. c o m /s q l), an d D B 2 (IB M C o rp o ratio n , h tt p ://w w w -0 1 .i b m .c o m /s o f tw a r e /d a t a / d b 2 /) are th e o n e s m o st c o m m o n ly u se d . E a ch o f th e s e p ro d u cts su p p orts b oth client/server an d W e b -b a s e d a rch itectu res.

• Will p a r a l l e l p r o c e s s in g a n d / o r p a r t i t i o n i n g b e u s e d ? P arallel p ro cessin g e n a b le s m u ltip le CPUs to p ro c e s s d ata w a re h o u s e q u e iy re q u e sts sim u ltaneou sly a n d p ro v id e s scalability. D ata w a r e h o u s e d esig n ers n e e d to d e cid e w h e th e r th e data­ b a s e ta b le s w ill b e p a rtitio n ed (i.e ., sp lit in to sm alle r ta b le s ) fo r a c c e s s e ffic ie n c y and w h a t th e criteria w ill b e . T h is is a n im p o rtan t co n sid e ra tio n th at is n e ce s s ita te d by

C hapter 3 * D ata W arehousing 1 2 3

the larg e am o u n ts o f d ata co n ta in e d in a typ ical data w a re h o u se . A r e c e n t su rv ey o n p arallel and d istrib u ted d ata w a re h o u s e s c a n b e fo u n d in Furtad o (2 0 0 9 ). T erad ata (teradata.com ) h a s s u cce ssfu lly a d o p te d a n d o fte n co m m e n d e d o n its n o v e l im p le ­ m e n ta tio n o f this ap p ro ach .

• Will d a ta m ig r a t io n tools b e u s e d to lo a d the d a ta w a reh o u se? M oving d ata fro m a n e x istin g sy stem into a data w a r e h o u s e is a te d io u s a n d la b o rio u s task. D e p e n d in g o n th e diversity a n d th e lo c a tio n o f th e data assets, m igration m a y b e a relativ ely s im p le p ro ce d u re o r (in co n tra st) a m o n th s-lo n g p ro je ct. T h e results o f a th o ro u g h a s s e s s m e n t o f th e existin g d ata a ssets sh ou ld b e u se d to d eterm in e w h e th e r to u s e m igration to o ls and , if so , w h a t cap a b ilitie s to s e e k in th o se co m ­ m ercial to ols.

• What tools w ill b e u s e d to s u p p o rt d a ta re trie v a l a n d a n a ly sis? O fte n it is n e ce s s a ry to u s e s p e cia liz e d to o ls to p e rio d ically lo ca te , a c c e s s , an aly ze, extract, transform , an d lo a d n e cessa ry data in to a d ata w a re h o u se . A d e c is io n h a s to b e m ad e o n ( 1 ) d e v e lo p in g th e m igration to o ls in -h o u se , ( 2 ) p u rch a sin g th e m fro m a third-party p ro v id er, o r (3 ) u sin g th e o n e s pro v id ed w ith th e data w a reh o u se system . O verly co m p le x , real-tim e m igrations w arran t s p e cia liz e d third -part ETL to o ls.

A lt e r n a t iv e D a ta W a r e h o u s in g A rc h ite c tu re s

At th e h ig h est lev el, d ata w a r e h o u s e arch itectu re d esig n v ie w p o in ts c a n b e ca te g o riz e d into en te rp rise-w id e d ata w a re h o u s e (E D W ) d esig n an d d ata m art (D M ) d esig n (G o lfa relli an d Rizzi, 2 0 0 9 ). In F ig u re 3 .7 (p arts a - e ) . w e s h o w s o m e alternativ es to th e b a sic arch i­ tectu ral d esig n ty p e s th a t a re n e ith e r p u re E D W n o r p u re DM , b u t in b e tw e e n o r b e y o n d th e trad itional arch ite ctu ral stru ctures. N otable n e w o n e s in clu d e h u b -a n d -sp o k e and fed erated arch ite ctu res. T h e five arch ite ctu res s h o w n in Figu re 3-7 (p arts a - e ) a re p ro ­ p o se d b y A riyachand ra and W a tso n (2 0 0 5 , 2 0 0 6 a , a n d 2 0 0 6 b ). P reviously, in a n e x te n s iv e study, S e n an d Sinha (2 0 0 5 ) id en tified 15 d iffe ren t d ata w areh o u sin g m e th o d o lo g ie s. T h e so u rces o f th e s e m e th o d o lo g ie s are classified in to th re e b ro a d ca te g o rie s: c o r e -te c h n o lo g y vend ors, infrastructure v en d o rs, a n d in fo rm atio n -m o d elin g co m p an ie s.

a. I n d e p e n d e n t d a t a m arts. T h is is argu ably th e sim p le st and th e lea st co stly arch i­ te ctu re altern ativ e. T h e data m arts are d e v e lo p e d to o p e ra te in d e p e n d en tly o f e a c h an o th e r to serv e th e n e e d s o f individual organ izatio n al units. B e c a u s e o f th e ir in d e­ p e n d e n c e , th e y m a y h av e in co n sisten t d ata d efin ition s a n d d ifferent d im en sio n s an d m easu res, m a k in g it d ifficult to an alyze d ata a cro ss th e d ata m arts (i.e ., it is difficult, i f n o t im p o ssib le, to g e t to th e “o n e v e rsio n o f th e truth”).

b. D a ta m a r t b u s a rc h ite c tu re . T h is arch ite ctu re is a v ia b le altern ativ e to th e in d e­ p e n d e n t d ata m arts w h e re th e individual m arts are lin k e d to e a c h o th e r v ia so m e kin d o f m id d lew are. B e c a u s e th e d ata a re lin k e d am o n g th e individual m arts, th e re is a b e tte r c h a n c e o f m ain tain in g d ata c o n s is te n cy a cro ss th e e n terp rise (a t le a s t at th e m etad ata le v e l). E v e n th o u g h it allo w s fo r c o m p le x d ata q u e ries a c ro ss data m arts, th e p e rfo rm a n ce o f th e s e ty p e s o f analysis m ay n o t b e a t a satisfacto ry lev el.

c. H u b -a n d -sp o k e a rc h ite c tu re . T h is is p e rh a p s the m o st fa m o u s d ata w a re h o u s ­ ing arch ite ctu re tod ay . H e re th e atten tio n is fo c u s e d o n b u ild in g a s c a la b le an d m ain tain ab le infrastru ctu re (o fte n d e v e lo p e d in a n iterative w ay , s u b je c t a re a b y s u b je ct a re a ) th a t in clu d es a ce n tralize d d ata w a r e h o u s e and sev era l d e p e n d e n t data marts (e a c h fo r a n organ izatio n al u n it). T h is arch ite ctu re allow s fo r e a sy cu sto m iza­ tio n o f u s e r in te rfa ce s and reports. O n th e n eg ativ e sid e, this arch ite ctu re la c k s th e h olistic e n te rp rise view , a n d m a y le a d to data re d u n d a n cy an d d ata laten cy .

d. C en tra liz ed d a t a w a reh o u se. T h e cen tralized d ata w a re h o u s e a rch ite ctu re is sim ilar to th e h u b -a n d -sp o k e arch ite ctu re e x c e p t that th e re are n o d e p e n d e n t data marts; in stead , th e re is a gig an tic e n te rp rise d ata w a r e h o u s e th at serv e s th e n e ed s

Part II • D escriptive Analytics

(a) Independent Data M arts Architecture

End-user a cce ss and applications

Independent data marts (atomic/summarized data)

Staging area

Source systems

(b) Data Mart Bu s Architecture with Linked Dimensional Data Marts

Dimensionalized ^ S l data m arts linked by

conformed dimensions (atomic/summarized data]

End-user a cce ss and applications

Staging area

Source systems

[c] Hub-and-Spoke Architecture (Corporate Information Factory)

End-user a cce ss and applications

Normalized relational warehouse (atomic data)

Staging area

Source systems

Dependent data marts (sum m arized/some atomic data)

(d) Centralized Data Warehouse Architecture

End-user a cce ss and applications

Normalized relational warehouse [atomic/some

summarized data] Staging

area Source systems

(e) Federated Architecture

Data mapping/metadata End-user

a cce ss and applications

Existing data warehouses Data marts and legacy systems

Logical/physical integration of common data elements

FIGURE 3.7 Alternative Data Warehouse Architectures. Source: Adapted from T. Ariyachandra and H. Watson, "Which Data Warehouse Architecture Is Most Successful?" Business Intelligence Journal, Vol. 11, No. 1, First Quarter, 2006, pp. 4-6.

o f all o rgan ization al units. T h is ce n tralize d a p p r o a c h p ro v id e s u sers w ith a c c e s s to all d ata in th e d ata w a re h o u s e in stead o f lim iting th e m to data m arts. In addition, it re d u ce s th e am o u n t o f d ata th e te ch n ica l te a m has to tran sfer o r ch a n g e , th e re ­ fo re sim plifying d ata m a n a g e m e n t an d adm inistration. I f d esig n e d and im p lem ented p ro p erly, this arch ite ctu re p ro v id e s a tim ely a n d h o listic v ie w o f th e e n te rp rise to

Chapter 3 * D ata W arehou sing 1 2 5

Transactional Users

Transactional Data

Data Transformation

Operational Data Store [ODS]

"Enterprise” Data Warehouse

Data Replication

Data Marts

Decision U sers

Stra te g ic Tactica l Reporting D ata Event-driven/ U s e r s U s e rs O L A P U s e rs M iners Closed Loop

FIGURE 3.8 Teradata Corporation's Enterprise Data Warehouse. Source: Teradata Corporation (teradata.com). Used with cermission.

w h o m e v e r, w h e n e v e r, and w h e re v e r th e y m ay b e w ith in th e org an izatio n . T h e ce n tra l d ata w a re h o u s e s arch itectu re, w h ic h is a d v o cated m ain ly b y T erad ata Corp., ad vises u sin g d ata w a re h o u s e s w itho u t a n y data m arts ( s e e Figure 3 .8 ).

e. F e d e r a t e d d a t a w a r e h o u s e . T h e fed erated a p p ro a ch is a c o n c e s s io n to th e natu ­ ral fo rc e s th at u n d e rm in e th e b e s t p lan s fo r d ev elo p in g a p e rfe ct system . It u ses all p o ss ib le m e a n s to in teg rate an aly tical re so u rce s fro m m u ltiple so u rce s to m e e t ch an g in g n e e d s o r b u sin e ss co n d itio n s. E ssentially, the fe d era ted a p p ro a ch involves integrating d isp arate sy stem s. In a fe d e ra te d arch ite ctu re, e x istin g d e c is io n su p p o rt stru ctures are left in p la ce , a n d d ata are a c c e s s e d fro m th o s e so u rce s as n e e d e d . T h e fe d erated a p p r o a c h is su p p o rte d b y m id d lew are v e n d o rs that p ro p o s e distributed q u e ry an d jo in ca p a b ilitie s. T h e s e e x te n s ib le M arkup L anguage (X M L )-b a s e d tools o ffe r u se rs a g lo b a l v ie w o f d istrib uted d ata so u rce s, inclu d ing d ata w a re h o u s e s , data m arts, W e b sites, d o cu m en ts, an d o p era tio n a l sy stem s. W h e n u sers c h o o s e q u ery o b je c ts fro m this v ie w an d p ress th e su b m it b u tto n , th e to o l au tom atically q u e ries th e distributed s o u rc e s, jo in s th e results, and p re sen ts th e m to th e u ser. B e c a u s e o f p e rfo rm a n ce an d data qu ality issu es, m o st e x p e rts a g re e th a t fe d era ted a p p r o a c h e s w o rk w e ll to s u p p le m e n t data w a reh o u se s, n o t re p la c e th e m (s e e E ck e rso n , 2 0 0 5 ).

A riyachand ra a n d W a ts o n (2 0 0 5 ) id en tified 10 facto rs that p o ten tially a ffe c t th e ireh ite ctu re s e le c tio n d ecisio n :

1 . In fo rm ation in te rd e p e n d e n c e b e tw e e n org anizational units 2 . U p p er m a n a g e m e n t’s in form ation n e e d s 3 . U rg en cy o f n e e d fo r a data w a re h o u s e

4 . N ature o f e n d -u se r tasks 5 . C onstraints o n re so u rces 6. 7. 8 .

1 2 6 Part II • D escriptive Analytics

Strategic v ie w o f th e d ata w a re h o u s e p rio r to im p lem en tatio n Com patibility w ith existin g system s P e rc e iv e d ab ility o f th e in -h o u s e IT staff

9 . T e c h n ic a l issu es 1 0 . Social/p olitical facto rs

T h e s e facto rs are sim ilar to m any s u c c e s s facto rs d escrib e d in th e literatu re for in fo rm a tio n sy stem s p ro je c ts B S S m d B l p ro tects. T e c h n ic a l issu es, fe y o n d ^ p ro v M j in s te c h n o lo g y th a t is fe a s ib ly re ad y fo r u s e , are im portant, b u t o fte n n o t as m P orta as b e h a v io ra l issu es, s u ch a s m e e tin g u p p e r m an ag e m e n t's in form ation n e e d s an d use in v o lv em en t in th e d e v e lo p m e n t p ro ce s s (a social/ political facto r). E a ch d ^ ar* ° US“ « arch itectu re has s p e cific ap p lica tio n s fo r w h ic h it is m o st (a n d l e a s t ) e ffe ctiv e an d thus provides m axim al b e n e fits to th e org an izatio n . H o w e v e r o v e r a lL th e data m art structu^ se e m s to b e th e le a st e ffe ctiv e in p ra ctice . S e e A riyachandra and W atso n (2 0 0 6 a )

ad d itional details.

W h ic h A r c h it e c t u r e Is t h e B e s t?

Ever sin c e data w a re h o u sin g b e c a m e a critical p art o f m o d e m o f w h ic h data w a re h o u s e arch itectu re is th e b e s t has b e e n a to p ic o f regular dlscu Sion T h e tw o gurus o f th e d ata w a reh o u sin g field, B ill In m o n an d R alp h K im ball, are at the h e art o f this d iscu ssion . In m o n a d v o ca tes th e h u b -a n d -sp o k e arch ite ctu re (e .g . , th C o rp o ra te In fo rm atio n F a c t o ^ ) , w h e re a s K im b all p ro m o te s the data m art b u s a rch itectu e w ith co n fo rm e d d im en sio n s. O th e r arch ite ctu res are p o ss ib le , b u t fu n d am en tally d ifferent a p p r o a c h e s, a n d e a c h has stro n g ad v o cates. T o sh e d light ori t a s c o n t r o v e r s i a l q u e stio n , A riyachand ra an d W a ts o n (2 0 0 6 b ) c o n d u cte d a n e m p .n e a study^ T o c o lle c t th e d ata, th ey u s e d a W e b -b a s e d survey targ eted at individuals in v o lv e d m data w a re h o u s e im p lem en tation s. T h e ir survey in clu d e d questions th e re sp o n d e n t's co m p a n y , th e co m p a n y ’s data w a re h o u se , a n d th e s u c c e s s o f th e data

W arelInUt^ ^ 4 5 4 resp on d en ts provid ed u sa b le inform ation. Surveyed co m p an ie s ranged fro m sm all (less th an $ 1 0 m illion in re v e n u e ) to large (in e x c e s s o f »10 b illion ). M ost o t the co m p an ie s w e re lo cated i n th e U n ited States (6 0 % ) and re p resen ted a v a n ety o f industnes w ith th e financial services industry (1 5 % ) providing th e m o st resp o n ses. T h e P red o“ “ “ architecture w as th e h u b -an d -sp o k e architecture (39% X P 6 % ) the centralized architecture (1 7 % ), in d e p e n d en t data marts (1 2 % ), a n d th e fed erated architecture (4% ). T h e m o st co m m o n platform fo r h ostin g the data w areh o u se s w as O rac (41% ) fo llow ed b y M icroso ft (1 9 % ), an d IB M (18% ). T h e average (m e a n ) gross rev en u e var­ ie d from $3.7 b illion fo r in d ep en d en t data m arts to $6 b illion fo r th e fed erated architecture.

T h e y u se d fo u r m e a su re s to a s s e s s th e s u c c e s s o f th e arch itectu res: (1 ) inform ation q u ality ( 2 ) sy stem quality, ( 3 ) individual im pacts, a n d ( 4 ) o rgan ization al im pacts. e q u e stio n s u s e d a sev e n -p o in t sca le , w ith th e h ig h er s c o re ind icating a m o re su cce ssfu l arch ite ctu re. T a b le 3 .1 sh o w s th e av erag e s c o re s fo r th e m e a su re s acro ss

As th e results o f th e study in d icate, in d e p e n d e n t data m arts s co re d th e lo w e st o n all m e asu re s. T h is find ing con firm s th e co n v e n tio n a l w isd o m th a t in d e p e n d en t d ata m a s are a p o o r arch itectu ral solu tio n . N ext lo w e s t o n all m easu res w as th e fe d era ted arch itec ture Firm s so m e tim e s have: d isp arate d e c is io n su p p o rt p latfo rm s resultin g f r o m m ergers an d acq u isitio n s, an d th e y m ay c h o o s e a fe d era ted ap p ro a ch , a t le a st i n the short: r u n T h e find ings su g g est that th e fe d era ted arch ite ctu re is n o t a n op tim al lo n g -te rm solu tion. W h at is interesting, h o w e v e r, is th e sim ilarity o f th e av erag es fo r the b u s, h u b -an d -sp o k e , an d ce n tra liz e d arch itectu res. T h e d iffe re n ce s are su fficien tly sm all that n o claim s can

C h a p te r s * Data W arehou sing 1 2 7

T A B L E 3.1 A v e ra g e Assessm ent Scores fo r th e Success o f th e Architectures

Independent Data M arts

Bus Architecture

Hub-and Spoke Architecture

Centralized Architecture (N o D ependent Data M arts)

Federated Architecture

Information Quality 4.42 5.16 5.35 5.23 4.73 System Quality 4.59 5.60 5.56 5.41 4.69 Individual Impacts 5.08 5.80 5.62 5.64 5.15 Organizational Impacts 4.66 5.34 5.24 5.30 4.77

m ad e fo r a p articu lar arch ite ctu re ’s su p eriority o v e r th e o th ers, at le a s t b a s e d o n a sim p le co m p ariso n o f th e s e s u c c e s s m easu res.

T h e y a ls o c o lle c te d d ata o n th e d o m ain (e .g ., v arying fro m a su b u n it to co m p a n y - w id e) a n d th e siz e (i.e ., am o u n t o f data sto re d ) o f th e w a re h o u s e s . T h e y fo u n d th a t the h u b -a n d -sp o k e a rch ite ctu re is typ ically u s e d w ith m o re en te rp rise-w id e im p lem e n tatio n s and larg er w a re h o u s e s . T h e y a lso in vestigated th e c o s t and tim e re q u ired to im p lem e n t th e d ifferen t arch ite ctu res. O v erall, th e h u b -a n d -sp o k e arch ite ctu re w a s th e m o st e x p e n ­ sive an d tim e -co n su m in g to im p lem en t.

SECTION 3 . 4 REVIEW QUESTIONS

1 . W h a t are th e k e y sim ilarities a n d d iffe ren ce s b e tw e e n a tw o -tie red a rch ite ctu re an d a th ree -tiered architectu re?

2. H o w h a s th e W e b in flu e n ce d data w a r e h o u s e design? 3 . List th e altern ativ e data w a reh o u sin g arch itectu res d iscu sse d in this sectio n . 4 . W h at issu es sh o u ld b e co n sid e re d w h e n d e cid in g w h ich arch ite ctu re to u se in d ev el­

o p in g a d ata w a reh o u se ? List th e 10 m o st im portant factors.

5 . W h ich d ata w a re h o u s in g arch ite ctu re is th e best? Why?

3.5 D A T A IN TEG RA TIO N A N D THE EXTRACTIO N , T R A N SFO R M A T IO N , A N D LO AD (ET L) P R O C E S S E S

G lo b al co m p etitiv e p re ssu re s, d em an d fo r return o n in v estm en t (R O I), m a n a g e m e n t and investor inquiry, an d g o v ern m e n t reg u latio n s are fo rcin g b u sin e ss m anagers to reth in k h o w th e y in teg rate an d m a n a g e th e ir b u sin esses. A d e c is io n m a k e r typically n e e d s a c c e s s ro m u ltiple so u rce s o f d ata that m u st b e integrated . B e fo re d ata w a re h o u s e s , data marts, and B I so ftw are, p ro v id in g a c c e s s to data so u rce s w as a m ajor, la b o rio u s p ro c e s s . E ven w ith m o d e rn W e b -b a s e d d ata m a n a g e m e n t to o ls, re co g n iz in g w h at d ata to a c c e s s and p roviding th e m to th e d e c is io n m a k e r is a non trivial ta sk that req u ires d a ta b a se sp ecialists. As data w a re h o u s e s g ro w in size, th e issu es o f integratin g data g ro w a s w ell.

T h e b u sin e ss a n aly sis n e e d s co n tin u e to e v o lv e . M ergers an d a cq u isitio n s, re g u la ­ tory req u irem en ts, an d th e in trod u ctio n o f n e w ch a n n e ls c a n drive c h a n g e s in B I re q u ire ­ m ents. In ad d ition to h isto rical, cle a n s e d , co n so lid a ted , and p o in t-in -tim e data, b u sin e ss u sers in creasin g ly d em an d a c c e s s to real-tim e, u n stru ctu red , and/or re m o te d ata. And everything m ust b e in te g rate d w ith th e c o n te n ts o f a n existin g d ata w a re h o u se . M o re o v e r, a cce ss v ia PDA s a n d th ro u g h s p e e c h re co g n itio n an d synthesis is b e c o m in g m o re c o m ­ m o n p la ce , fu rth er co m p lica tin g in teg ratio n issu es (E d w ard s, 2 0 0 3 ). M any in teg ratio n p ro ­ jects in v o lv e e n te rp rise-w id e sy stem s. O ro v ic ( 2 0 0 3 ) p ro v id e d a c h e c k lis t o f w h at w o rk s and w h at d o e s n o t w o rk w h e n attem p tin g s u ch a p ro je ct. P rop erly in tegratin g d ata fro m

1 2 8 Part II • D escriptive Analytics

v ario u s d a ta b a se s a n d o th e r d isp arate so u rce s is difficult. B u t w h e n it is n o t d o n e p ro p ­ erly, it c a n lea d to d isaster in en te rp rise-w id e sy stem s s u ch as CRM, ERP, and su p p ly chain p ro je cts (N ash , 2 0 0 2 ).

D a ta In te g r a t io n

D ata in tegration co m p rise s th ree m a jo r p r o c e s s e s that, w h e n co rrectly im p lem en ted , p erm it d ata to b e a c c e s s e d a n d m ad e a c c e s s ib le to a n array o f ETL an d an alysis to o ls and th e d ata w a reh o u sin g en v iron m en t: d ata a c c e s s ( i .e ., th e ability to a c c e s s an d e x tra ct data fro m a n y data s o u rc e ), d ata fe d era tio n (i.e ., th e in te g ratio n o f b u sin e ss v iew s a cro ss m ul­ tip le data sto re s ), an d c h a n g e ca p tu re (b a s e d o n th e id en tificatio n , cap tu re, an d delivery o f th e c h a n g e s m a d e to e n te rp rise data so u rc e s). S e e A p p lication C ase 3-3 fo r an ex am p le o f h o w B P L ubricant b e n e fits fro m im p lem e n tin g a data w a re h o u s e that integrates data

Application Case 3.3 BP Lubricants Achieves BIGS Success B P L u b rican ts e sta b lish e d th e B IG S p ro g ram fo llo w ­ ing re c e n t m e rg e r activity to d eliver g lo b a lly c o n ­ sisten t a n d tran sp aren t m a n a g e m e n t in form ation . As w ell a s tim ely b u sin e ss in te llig e n ce , B IG S provides d etailed , c o n s is te n t v iew s o f p e rfo rm a n ce a cro ss fu n ctio n s s u c h as fin a n c e , m arketing, sales, and su p ­ p ly an d lo g istics.

B P is o n e o f th e w o rld ’s largest oil and p e t­ ro ch em icals groups. Part o f th e B P p ic group, BP Lubricants is an estab lish ed lead er in th e g lobal autom otiv e lubricants m arket. P erh ap s b est k n o w n fo r its Castrol brand o f oils, th e b u sin ess op erates in o v er 1 0 0 cou ntries and em p loy s 1 0 ,0 0 0 p eo p le. Strategically, B P Lubricants is co n cen tratin g o n fur­ th e r im proving its cu stom er fo cu s and in creasin g its e ffe ctiv e n ess in autom otive m arkets. F o llo w in g re ce n t m e rg e r activity, the co m p a n y is un dergoing transfor­ m ation to b e c o m e m o re e ffectiv e and ag ile an d to seize op p ortu nities fo r rapid grow th.

C h allen ge

Fo llo w in g re ce n t m erg e r activity, B P Lubricants w an ted to im prove th e co n sisten cy , transp arency, and accessib ility o f m an ag em en t inform ation an d business in telligen ce. In ord er to d o so , it n e e d e d to integrate data h eld in disparate so u rce system s, w ithout the d elay o f introd u cing a standardized ERP system .

S o l u t i o n

B P L u b rican ts im p lem e n te d th e p ilo t fo r its B u sin ess In te llig e n c e a n d G lo b a l Standards (B IG S ) p ro gram , a

strateg ic initiative fo r m a n a g e m e n t in form ation and b u sin ess in te llig e n ce . At th e h e a it o f B IG S is Kalido, a n ad aptive e n te rp rise d ata w a re h o u s in g so lu tio n fo r prep arin g, im p lem e n tin g , o p eratin g , an d m an agin g d ata w a re h o u s e s .

K a lid o ’s fe d e r a te d e n te rp ris e d ata w a r e h o u s ­ in g s o lu tio n s u p p o rte d th e p ilo t p ro g ra m ’s c o m ­ p l e x d ata in te g ra tio n a n d d iv erse re p o rtin g re q u ire ­ m e n ts. T o a d a p t to th e p ro g ra m ’s e v o lv in g re p o rtin g re q u ire m e n ts, th e so ftw a re a lso e n a b le d th e u n d e r­ ly in g in fo rm a tio n a rc h ite c tu re to b e e a sily m o d i­ fie d a t h ig h s p e e d w h ile p re serv in g all in fo rm atio n . T h e sy stem in te g ra te s a n d s to res in fo rm a tio n fro m m u ltip le s o u r c e sy stem s to p ro v id e c o n s o lid a te d v ie w s for:

• M a r k e t i n g . C u stom er p ro c e e d s and mar­ g in s fo r m ark e t seg m en ts w ith drill d o w n to in v o ice -le v e l d etail

• S a le s . S a le s in v o ice rep ortin g au gm ented w ith b o th d etailed tariff co sts and actual p ay m en ts

• F i n a n c e . G lo b a lly stand ard profit and loss, b a la n c e s h e e t, an d ca sh flo w statem en ts— w ith audit ability; cu sto m e r d e b t m an a g e m e n t sup­ p ly and lo gistics; co n so lid ated v ie w o f order an d m o v e m e n t p ro ce s s in g a cro ss m u ltiple ERP platform s

B e n e fits

B y im p ro v in g th e v isib ility o f c o n s is te n t, tim ely d ata, B IG S p ro v id e s th e in fo rm a tio n n e e d e d to

C hapter 3 * Data W arehou sing 129

a ssist th e b u s in e s s in id e n tify in g a m u ltitu d e o f b u s in e s s o p p o rtu n itie s to m a x im iz e m arg in s and/or m a n a g e a s s o c ia te d c o s ts . T y p ica l re s p o n s e s to th e b e n e fits o f c o n s is te n t d ata re su ltin g fro m th e B IG S p ilo t in clu d e :

• Im p ro v ed c o n s is te n c y a n d tran sp aren cy o f b u sin e ss data

• E asier, faster, a n d m o re fle x ib le rep orting • A cco m m o d a tio n o f b o th g lo b al an d lo c a l

stand ard s • Fast, c o s t-e ffe c tiv e , and fle x ib le im p lem e n ta­

tio n cy cle • M inim al d isru p tion o f e x istin g b u sin e ss p ro ­

c e s s e s an d th e d ay-to -d ay b u sin e ss

• Id en tifies d ata q u ality issu es an d e n c o u ra g e s th e ir re so lu tio n

• Im p ro v e d ab ility t o re s p o n d in telligen tly to n e w b u sin e ss o p p o rtu n itie s

Q u e s t io n s f o r D is c u s s io n

1. W h a t is B IG S a t B P Lubricants? 2. W h at w e re th e c h a lle n g e s, th e p ro p o s e d s o lu ­

tion , an d th e o b ta in e d results w ith BIG S?

Sources: Kalido, “BP Lubricants Achieves BIGS, Key IT Solutions," http:/ /w w w .kalid o.com / custom er-stories/bp-p lc.htm (accessed on August 2013). Kalido, “BP Lubricants Achieves BIGS Success," kalido.com/collateral/Documents/English-US/ CS-BP%20BIGS.pdf (accessed August 2013); and BP Lubricant homepage, bp.com/lubricanthome.do (accessed August 2013).

fro m m an y so u rce s. S o m e v e n d o rs, s u ch as SAS Institute, In c., h a v e d e v e lo p e d strong data in teg ratio n to ols. T h e SAS e n te rp rise d ata in teg ratio n serv e r i n c l u d e s c u sto m e r data in tegratio n to o ls that im p ro v e d ata qu ality in th e in tegratio n p ro c e s s . T h e O ra cle B u s in e s s In te llig e n ce Suite assists in in tegratin g d ata as w ell.

A m ajo r p u rp o s e o f a d ata w a r e h o u s e is to integrate d ata fro m m u ltiple system s. V arious in tegratio n te c h n o lo g ie s e n a b le data an d m etad ata integration:

• E n terp rise a p p lica tio n in teg ratio n (E A I) • S e rv ice-o rien ted arch ite ctu re (SO A ) • E n terp rise in fo rm atio n in teg ratio n (E li) • E xtractio n , tran sform atio n , a n d lo a d (ETL)

E n t e r p r i s e a p p l i c a t i o n i n t e g r a t i o n ( E A I ) p ro v id es a v e h ic le fo r p u sh in g data fro m so u rce system s in to th e data w a reh o u se . It in v olv es integratin g a p p lica tio n fu n ctio n ­ ality an d is fo c u s e d o n sharing fu n ction ality (rath er th a n d ata) a cro ss sy stem s, th e re b y e n a b lin g flex ibility an d reu se. T rad itionally, EAI so lu tio n s h av e fo c u s e d o n e n a b lin g a p p licatio n re u se at th e a p p lica tio n p ro g ram m in g in te rface (A P I) lev el. R ecen tly , EAI a cco m p lish e d b y u s in g SO A co a rse -g ra in e d s e r v ic e s (a c o lle c tio n o f b u sin e ss o r fu n ctio n s) th at a re w e ll d efin e d an d d o cu m en ted . U sing W e b s erv ices is a sp e cia liz e d w ay o f im p lem e n tin g a n SO A . EAI c a n b e u s e d to facilitate d ata acq u isitio n d irectly into a n e a r-re a l-tim e d a ta w a r e h o u s e o r to d eliver d e c is io n s to th e O L TF sy stem s. T h e r e a re m an y d ifferen t a p p r o a c h e s to an d to o ls fo r EAI im p lem en tation .

E n t e r p r i s e i n f o r m a t i o n i n t e g r a t i o n ( E l i ) is a n ev o lv in g to o l s p a ce that p ro m ise s real-tim e data in te g ra tio n fro m a variety o f so u rce s, s u ch as relatio n al d a ta b a se s, W e b serv ices and m u ltid im en sion al d atab ases. It is a m e ch a n ism fo r pu lling data fro m so u rce system s to satisfy a re q u e s t fo r in form ation. E li to o ls u s e p re d e fin e d m etad ata to p o p u la te view s th at m a k e in teg rated data a p p e a r relatio n al to e n d u s e r s XML m ay b e th e m o st im portant a s p e c t o f E li b e c a u s e XML a llo w s data to b e tag g e d e ith e r a t c re a tio n tim e o r later. T h e s e tags c a n b e e x te n d e d a n d m o d ified to a c c o m m o d a te a lm o st a n y a re a o f

k n o w le d g e (s e e K ay, 2 0 0 5 )- . . • , P h y sical d ata in te g ratio n h a s co n v e n tio n ally b e e n th e m am m e ch a n ism fo r cre atin g

an in teg rated v ie w w ith data w a re h o u se s a n d data m arts. W ith th e ad v en t o f E li to o ls (s e e Kay, 2 0 0 5 ), n e w virtu al d ata in teg ratio n p attern s are fe a sib le . M anglik an d M eh ra (2 0 0 5 )

1 3 0 Part II • D escriptive Analytics

d iscu sse d th e b e n e fits a n d con strain ts o f n e w d ata in teg ratio n p attern s th at c a n e x p a n d trad itional p h y sica l m e th o d o lo g ie s to p re s e n t a c o m p re h e n siv e v ie w fo r th e en terp rise.

W e n e x t turn to th e a p p ro a ch fo r lo ad in g d ata in to th e w a re h o u se : ETL.

E x tra c tio n , T r a n s fo r m a t io n , a n d Load

At th e heart o f the tech n ical side o f th e data w areh o u sin g p ro cess is extraction, tran s­ form ation, and load (ETL). ETL tech n o lo g ies, w h ich h av e existed fo r so m e tim e, are instrum ental in the p ro cess an d u s e o f d ata w areh o u ses. T h e ETL p ro ce s s is an integral co m p o n e n t in an y data-centric p ro ject. IT m anag ers a re o ften fa c e d w ith ch alle n g e s b e ca u se the ETL p ro cess typically co n su m es 7 0 p e rce n t o f th e tim e in a d ata-centric project.

T h e ETL p ro c e s s co n sists o f e x tra ctio n ( i.e ., re a d in g d ata fro m o n e o r m o re d ata­ b a s e s ), tran sform atio n (i.e ., co n v e rtin g th e e x tra cte d data fro m its p rev iou s fo rm into the fo rm in w h ich it n e e d s to b e s o that it c a n b e p la c e d in to a data w a re h o u s e o r sim ply an o th e r d a ta b a se ), a n d lo a d ( i.e ., putting th e data in to th e d ata w a re h o u s e ). T ran sfo rm atio n o ccu rs b y u sin g ru les o r lo o k u p tab le s o r b y co m b in in g th e data w ith o th e r data. T h e th ree d a ta b a se fu n ctio n s a re in teg rated in to o n e to o l to p u ll d ata o u t o f o n e o r m o re d a tab ase s an d p la c e th e m in to an o th er, co n so lid a te d d atab ase o r a data w a reh o u se .

ETL to o ls a lso tran sp ort data b e tw e e n s o u rc e s a n d targets, d o cu m en t how' data e le m e n ts (e .g ., m etad ata) c h a n g e as th e y m o v e b e tw e e n s o u rc e an d target, e x c h a n g e m etad ata w ith o th er a p p lica tio n s a s n e e d e d , a n d ad m in ister all ru ntim e p ro c e s s e s and o p eratio n s (e .g ., sch ed u lin g , e rro r m an ag e m e n t, au d it logs, statistics). ETL is e xtrem ely im portant fo r d ata in teg ratio n as w e ll as for data w a reh o u sin g . T h e p u rp o s e o f th e ETL p ro c e s s is to lo ad th e w a re h o u s e w ith in teg rated a n d c le a n s e d data. T h e d ata u se d in ETL p ro c e s s e s c a n c o m e fro m an y so u rce : a m ain fram e a p p lica tio n , a n ERP a p p licatio n , a CRM to o l, a flat file, a n E x ce l sp re a d sh ee t, o r e v e n a m e s s a g e q u e u e . In Fig u re 3 .9 , w e o u tline th e ETL p ro cess.

T h e p ro c e s s o f m igrating data to a d ata w a r e h o u s e in v olv es th e e x tra ctio n o f data fro m all relev an t so u rce s. D ata so u rce s m ay co n sist o f file s e x tra cte d fro m O LTP d atabases, sp re ad sh ee ts, p e rso n a l d a ta b a se s (e .g ., M icroso ft A c c e s s ), o r e x te rn a l files. T yp ically, all the in p u t files a re w ritten to a s e t o f stag in g tab les, w h ich a re d e sig n e d to facilitate the lo a d p ro ce ss. A data w a reh o u se co n ta in s n u m e ro u s b u sin e ss ru les that d e fin e s u c h things as h o w th e d ata w ill b e u sed , su m m arization ru les, stan d ard ization o f e n co d e d attributes, and ca lcu la tio n rules. Any d ata qu ality issu es p e rta in in g to th e s o u rc e file s n e e d to b e c o rre c te d b e fo re th e data are lo a d e d into th e d ata w a re h o u s e . O n e o f th e b e n e fits o f a

FIGU RE 3.9 The ETL Process.

Chapter 3 * Di

w e ll-d e sig n e d d ata w a r e h o u s e is th at th e s e ru les c a n b e s to red in a m etad ata re p o sito ry and ap p lie d to th e d ata w a r e h o u s e centrally. T h is d iffers fro m a n OLTP ap p ro a ch , w h ic h typically has d ata a n d b u sin e ss ru les s ca tte re d th ro u g h o u t th e system . T h e p ro c e s s o f loading d ata in to a d ata w a re h o u s e c a n b e p e rfo rm e d e ith er throu gh d ata tran sform atio n -ools th at pro v id e a G U I to aid in th e d ev e lo p m e n t and m a in te n a n ce o f b u s in e s s rules o r throu gh m o re trad itio nal m e th o d s, s u c h a s d ev elo p in g p rogram s o r utilities t o load th e data w a re h o u se , u s in g p ro gram m in g lan g u ag es s u c h as PL/SQL, C++, Ja v a , o r .NET F ram ew ork lan g u ag es. T h is d e c is io n is n o t easy fo r o rgan ization s. S ev eral issu es atte ct w h eth er a n o rg an izatio n w ill p u rch a se d ata tran sform atio n to o ls o r b u ild th e tian sfo rm a-

iio n p ro ce s s itself:

• D ata tran sfo rm atio n to o ls are e x p e n siv e . • D ata tran sfo rm atio n to o ls m ay h av e a lo n g learn in g cu rve. • It is d ifficult to m e a s u re h o w th e IT o rg an izatio n is d o in g until it h a s le a rn e d to u se

th e d ata tran sform atio n to ols.

In th e lo n g ru n, a tran sfo rm atio n -to o l a p p ro a ch sh ou ld sim p lify th e m a in te n a n ce o f i n o rg an ization ’s d ata w a re h o u se . T ran sfo rm atio n to o ls c a n a lso b e e ffe ctiv e m d etectin g m d scru b b in g ( i.e ., re m o v in g an y an o m a lies in th e d ata). OLAP an d data m ining to o ls rely on h o w w e ll th e data are transform ed .

As an e x a m p le o f e ffe ctiv e ETL, M otorola, In c., u s e s ETL t o fe e d its d ata w a reh o u se s. M otorola co lle c ts in fo rm atio n fro m 3 0 d ifferen t p ro cu re m e n t system s and s e n d s it to irs g lo b a l SCM d ata w a r e h o u s e fo r analysis o f ag g reg ate c o m p a n y sp e n d in g (s e e Son g im ,

i 2004). u ■ • . j S o lo m o n (2 0 0 5 ) classified ETL te ch n o lo g ie s into fo u r ca te g o rie s: s o p h istica ted , e n a-

b ie r. sim ple, and rudim entary. It is g e n erally a ck n o w le d g e d that to o ls in th e so p h isticated category will result in th e ETL p ro ce s s b e in g b e tte r d o cu m en ted a n d m o re accu ra te ly asanaged a s th e d ata w a r e h o u s e p ro je ct evolv es. ............................

E ven th o u g h it is p o ss ib le fo r p ro g ram m ers to d ev elo p softw are fo r ETL, it is sim p le r Ld use a n existin g E TL to o l. T h e fo llo w in g are s o m e o f the im p ortan t criteria in s e le ctin g

a n ETL to o l (s e e B ro w n , 2 0 0 4 ):

• Ability to read fro m a n d w rite to a n unlim ited n u m b e r o f data s o u rc e arch ite ctu res • A u tom atic cap tu rin g and d elivery o f m etad ata • A h istory o f co n fo rm in g to o p e n standards • An e a sy -to -u se in te rfa ce fo r th e d e v e lo p e r a n d th e fu n ctio n al u se r

P e rfo rm in g e x te n s iv e ETL m ay b e a s ig n o f p o o rly m a n a g e d d ata a n d a ^ nd am ental la c k o f a c o h e r e n t d ata m a n a g e m e n t strateg y . K a ra c s o n y ( 2 0 0 6 ) in d i­

c a te d th a t th e re is a d ir e c t c o r r e la tio n b e tw e e n th e e x te n t o f re d u n d a n t data a n d th e - u m b e r o f ETL p r o c e s s e s . W h e n d ata a r e m a n a g e d c o r re c tly a s a n e n te rp ris e a sse t, E TL e ffo rts a re sig n ific a n tly r e d u c e d , a n d re d u n d a n t d ata a re c o m p le te ly e lim in a te d .

Ibis lea d s to h u g e sav in g s in m a in te n a n c e a n d g r e a te r e ffic ie n c y in n e w d e v e lo p ­ m en t w h ile a ls o im p ro v in g d ata q u ality . P o o r ly d e s ig n e d E TL p r o c e s s e s a re c o s tly to C i n t a i n , c h a n g e , a n d u p d a te . C o n s e q u e n tly , it is c ru c ia l to m a k e th e p r o p e r c h o ic e s n :e rm s o f th e te c h n o lo g y a n d to o ls to u s e fo r d e v e lo p in g an d m a in ta in in g th e ETL

^ A n u m b e r o f p a ck a g e d ETL to o ls are available. D a ta b a se v e n d o rs cu rren tly o f fe r ETL capabilities that b o th e n h a n c e a n d c o m p e te w ith in d e p e n d e n t ETL to o ls . SAS a c k n o w l­ edges th e im p o rta n ce o f data q u ality a n d o ffe rs th e industry’s first fully in te g rate d solu -

n that m e rg e s ETL a n d data quality to tran sform d ata in to strateg ic v a lu a b le assets, fcher ETL so ftw are p ro v id ers in clu d e M icrosoft, O ra cle , IB M , In fo rm atica, E m b arcad ero ,

■ r l T ib co . F o r ad d ition al in form ation o n ETL, s e e G o lfarelli an d Rizzi (2 0 0 9 ), K a rak so n y

2006), a n d S o n g in i ( 2 0 0 4 ) .

1 3 2 Part II • D escriptive Analytics

SECTION 3 .5 REVIEW QUESTIONS

1 . D e s c r ib e d ata integration. 2 . D e s c r ib e th e th re e step s o f th e ETL p ro cess. 3 . W h y is th e ETL p ro ce s s s o im p ortan t fo r d ata w a re h o u s in g efforts?

3.6 D A T A W A R E H O U S E D E V E L O P M E N T A d ata w a reh o u sin g p ro je ct is a m a jo r u n d e rtak in g fo r an y o rg an ization a n d is m ore co m p lica te d th a n a sim p le, m ain fram e s e le c tio n an d im p lem e n tatio n p ro je c t b e c a u s e it co m p rise s and in flu e n ces m any d ep artm en ts a n d m an y in p u t an d ou tp u t in te rface s an d it can b e p art o f a CRM b u sin e ss strategy. A d ata w a r e h o u s e p ro v id es sev eral b e n e fits that c a n b e classified as d irect a n d indirect. D ir e c t b e n e fits in clu d e th e ioWowmg-.

• E n d u sers c a n p e rfo rm e x te n s iv e analysis in nu m erou s w ays. • A c o n s o lid a te d v ie w o f c o rp o ra te d ata ( i .e ., a sin g le v ersio n o f th e tru th ) is p o ssible. • B e tte r and m o re tim ely in form ation is p o s s ib le . A d ata w a r e h o u s e p erm its inform a­

tio n p ro ce s s in g to b e re lie v e d fro m co stly o p era tio n a l system s o n to lo w -co st serv­ ers; th e re fo re, m an y m o re e n d -u s e r in fo rm atio n re q u e sts ca n b e p ro c e s s e d m o re

q u ick ly . • E n h a n c e d s y s te m p e r fo r m a n c e c a n re su lt. A d a ta w a r e h o u s e fre e s p ro d u c tio n

p r o c e s s in g b e c a u s e s o m e o p e ra tio n a l s y s te m re p o r tin g re q u ire m e n ts a re m o v e d

t o D S S . • Data access is sim p lified

In d irect b e n e fits resu lt fro m e n d u sers u sin g th e s e d irect b e n e fits . O n th e w h o le , th e s e b e n e fits e n h a n c e b u sin ess k n o w le d g e, p re s e n t co m p etitiv e ad van tag e, im p ro ve cu s­ to m er serv ice an d satisfactio n , facilitate d e c is io n m ak in g , a n d h e lp in refo rm in g b u sin ess p ro ce s s e s; th e re fo re, th ey a re th e stro n g est co n trib u tio n s to com p etitiv e ad vantage. (F o r a d iscu ssio n o f h o w to c re a te a co m p etitiv e ad v an tag e th ro u g h d ata w areh o u sin g , s e e P arzin g er and F ro lick , 2 0 0 1 .) F o r a d eta iled d iscu ssio n o f h o w o rg an izatio n s c a n ob tain e x c e p tio n a l lev els o f p ay o ffs, s e e W a tso n e t al. (2 0 0 2 ). G iv e n the p o ten tial b en e fits th at a d ata w a re h o u s e c a n pro v id e and th e su b stantial investm ents in tim e a n d m o n e y that su ch a p ro je ct re q u ires, it is critical th at a n o rg a n iz a tio n stru cture its data w a re h o u s e p ro je ct to m ax im ize th e c h a n c e s o f s u cce s s . In ad d ition, th e o rg an izatio n m ust, ob v iou sly , take co s ts in to co n sid era tio n . K e lly (2 0 0 1 ) d e s c rib e d a R O I a p p ro a ch th at co n sid ers b en e fits in th e ca te g o rie s o f k e e p e rs (i.e ., m o n e y sav e d b y im proving trad itional d e cisio n supp ort fu n ctio n s); gath e re rs (i.e ., m o n e y sav e d d u e to au to m ated c o lle c tio n and d issem in atio n o f in fo rm atio n ); a n d u sers (i.e ., m o n ey saved o r g a in ed fro m d e cis io n s m ad e u sin g th e d ata w a re h o u s e ). C o sts in clu d e th o s e related t o h ard w are, softw are, n e tw o rk band w id th, internal d ev elo p m en t, internal su p p ort, training, an d e x te rn a l con su ltin g . The n e t p re ­ s e n t v a lu e (N PV ) is ca lcu la ted o v e r th e e x p e c te d life o f th e data w a re h o u se . B e c a u s e th e b e n e fits a re b ro k e n d o w n ap p ro xim ate ly a s 2 0 p e rc e n t fo r k e e p e rs , 3 0 p e rce n t fo r g ath e re rs, and 5 0 p e rc e n t fo r u se rs, K elly in d icate d th a t u se rs shou ld b e involved in the d e v e lo p m e n t p ro c e s s , a s u c c e s s fa cto r typ ically m e n tio n e d as critical fo r system s that

im ply c h a n g e in a n org anization. A p p lication C ase 3 .4 p ro v id es a n e x a m p le o f a d ata w a reh o u se that w a s d e v e lo p e d

an d d elivered in te n se co m p etitiv e ad van tage fo r th e H oku riku (Ja p a n ) C o ca-C ola Bottling C o m p any. T h e sy stem w as s o su cce ssfu l th at p la n s are u n d e w a y to e x p a n d it to e n c o m ­ p ass th e m o re th a n 1 m illio n C o ca-C o la v e n d in g m a ch in e s in Ja p a n .

C learly d efin in g th e b u s in e s s o b je c tiv e , g a th e rin g p r o je c t su p p o rt fro m m a n a g e ­ m e n t e n d u s e rs , settin g r e a s o n a b le tim e fra m es an d b u d g ets, an d m a n a g in g e x p e c ta tio n s a re critica l to a s u c c e s s fu l d ata w a re h o u s in g p ro je ct. A data w a re h o u sin g strate g y is a

C hapter 3 * Data W arehou sing 1 3 3

Application Case 3.4 Things Go Better with Coke's Data Warehouse In th e fa ce o f co m p e titiv e p re ssu re s a n d co n su m e r d em an d , h o w d o e s a s u cce s s fu l b o ttlin g co m p a n y e n su re that its v e n d in g m a ch in e s are p rofitable? T h e an s w e r fo r H oku riku C o ca-C o la B o ttlin g C o m p an y (H C C B C ) is a d ata w a re h o u s e and analytical so ft­ w a re im p lem e n te d b y T erad ata Corp. H C CBC built th e system in re s p o n s e to a data w a reh o u sin g sy stem d e v e lo p e d b y its rival, M ikuni. T h e d ata w a reh o u se co lle cts n ot o n ly h istorical d ata b u t a lso n e a r -r e a l­ tim e d ata fro m e a c h v e n d in g m a ch in e (v ie w e d as a s to re ) that c o u ld b e transm itted v ia w ire le s s c o n ­ n e c tio n to h e ad q u arte rs. T h e initial p h a s e o f the p ro je c t w a s d e p lo y e d in 2 0 0 1 . T h e d ata w a re h o u s e a p p ro a ch p ro v id e s d etailed p ro d u ct in form ation, su ch as tim e a n d d a te o f e a c h sale, w h e n a p ro d ­ u ct sells ou t, w h e th e r s o m e o n e w a s sh o rt-ch a n g e d , and w h e th e r th e m a ch in e is m alfu n ctio n in g . In e a c h c a s e , a n alert is triggered , an d th e v e n d in g m a ch in e im m ed iately re p o rts it to th e d ata ce n te r o v e r a w ire ­ less tran sm issio n system . (N o te that C o ca-C o la in th e U n ited States h a s u se d m o d e m s to link v en d in g m a ch in e s to d istributors fo r o v e r a d e c a d e .)

In 2 0 0 2 , H C C BC c o n d u cte d a p ilo t te st an d put all its N ag an o v e n d in g m a ch in e s o n a w ire le ss n e t­ w o rk to g a th e r n e a r-re a l-tim e p o in t o f sa le (P O S ) data fro m e a c h o n e . T h e results w e re astou nd ing b e c a u s e th e y accu ra te ly fo re ca s te d d em an d an d id en tified p ro b le m s q u ick ly . T o tal sale s im m ed iately

in cre a se d 10 p e rce n t. In ad d ition, d u e to th e m o re a ccu ra te m a ch in e serv icin g , ov ertim e and o th e r co sts d e cre a s e d 4 6 p e rce n t. I n ad d ition, e a c h sa le sp e rso n w as a b le to serv ice up to 4 2 p e rce n t m o re v en d in g m ach in e s.

T h e te st w a s s o s u cce s s fu l th at p lan n in g b eg a n to e x p a n d it to e n c o m p a s s th e e n tire e n terp rise (6 0 ,0 0 0 m a ch in e s ), u sin g a n a ctiv e data w a reh o u se . Eventually, th e data w a re h o u sin g so lu tio n w ill id e­ ally e x p a n d a cro ss c o r p o ra te b o u n d arie s in to th e e n tire C o ca-C o la B o ttle rs n e tw o rk s o that th e m o re th an 1 m illio n v e n d in g m a ch in e s in Ja p a n w ill b e n e tw o rk e d , lead in g to im m e n se c o s t savings and h ig h e r rev e n u e .

Q u e s t io n s f o r D is c u s s io n

1. H o w d id C o ca-C o la in Ja p a n u s e d ata w a re h o u s ­ ing to im p ro ve its b u s in e s s p ro cesses?

2. W h at w e re th e resu lts o f th eir e n terp rise active data w a re h o u s e im p lem en tation ?

Sources: Adapted from K. D. Schwartz, “Decisions at the Touch of a Button,” T eradata M agazine, teradata.com/t/page/117774/ ind ex.h tm l (accessed June 2009); K. D. Schwartz, “Decisions at the Touch of a Button," DSS R esources, March 2004, pp. 28-31, d s i5 r e s o u r c e s .c o m / c a s e s / c o c a - c o la ja p a n / in d e x .h t m l (accessed April 2006); and Teradata Corp., “Coca-Cola Japan Puts the Fizz Back in Vending Machine Sales," teradata.eom/t/ page/118866/index.httnI (accessed June 2009).

b lu ep rin t fo r th e s u c c e s s fu l in tro d u ctio n o f th e d ata w a re h o u s e . T h e strateg y sh o u ld d e s c rib e w h e re th e c o m p a n y w an ts to g o , w h y it w a n ts to g o th e re , a n d w h a t it w ill d o w h e n it g e ts th e re . It n e e d s to ta k e in to c o n s id e ra tio n th e o rg a n iz a tio n ’s v isio n , stru ctu re, a n d cu ltu re. S e e M atn ey ( 2 0 0 3 ) fo r th e s te p s that c a n h e lp in d e v e lo p in g a fle x ib le and e fficie n t su p p o rt strategy. W h e n th e p la n a n d su p p o rt fo r a d ata w a r e h o u s e a re e sta b ­ lish e d , th e o rg a n iz a tio n n e e d s to e x a m in e d ata w a r e h o u s e v e n d o rs. ( S e e T a b le 3 .2 fo r a sa m p le list o f v e n d o rs ; a ls o s e e T h e D ata W a re h o u sin g In stitu te [twdi.org] a n d DM Review [inform ation-m anagem ent.com ].) M any v e n d o rs p ro v id e so ftw a re d e m o s o f th e ir d ata w a re h o u s in g a n d B I produ cts.

D a ta W a re h o u s e D e v e lo p m e n t A p p ro a c h e s

M any org an izatio n s n e e d to c re a te th e data w a re h o u se s u sed fo r d e c is io n su p p ort. T w o co m p e tin g a p p r o a c h e s a re e m p lo y e d . T h e first a p p ro a ch is that o f Bill In m o n , w h o is o ften ca lle d ‘ th e fa th e r o f data w a re h o u s in g .” In m o n su p p o rts a to p -d o w n d ev e lo p m e n t ap p ro ach that ad ap ts trad itio nal re latio n al d a ta b a se to o ls to th e d ev e lo p m e n t n e e d s o f an

134 P art II • Descriptive Analytics

T A B L E 3.2 Sam ple List o f Data W areh o u sin g Vendors

V e n d o r Product O fferings A comprehensive set of business intelligence and data visuali­

zation software (now owned by SAP) Comprehensive set of data warehouse (DW ) tools and products

Business Objects (businessobjects.com)

Computer Associates (cai.com) DataMirror (datamirror.com) Data Advantage Group (dataadvantagegroup.com )

Dell (dell.com) Embarcadero Technologies (embarcadero.com) Greenplum (greenplum.com)

Harte-Hanks (harte-hanks.com)

HP (hp.com) Hummingbird Ltd. (hummingbird.com, now is a

subsidiary of Open Text.) Hyperion Solutions (hyperion.com , now an Oracle

company) IBM InfoSphere (w w w - 01.ibm.com/software/data/

infosphere/) Informatica (informatica.com) Microsoft (microsoft.com) Netezza

Oracle (including PeopleSoft and Siebel) (oracle.com)

SAS Institute (sas.com) Siemens (siemens.com) Sybase (sybase.com) Teradata (teradata.com )

D W administration, management, and performance products

Metadata software D W servers D W administration, management, and performance products Data warehousing and data appliance solution provider (now

owned by EMC) Customer relationship management (CRM) products and services

D W servers D W engines and exploration warehouses

Comprehensive set of D W tools, products, and applications

Data integration, DW, master data management, big data products

D W administration, management, and performance products

D W tools and products D W software and hardware (D W appliance) provider (now

owned by IBM) DW, ERP, and CRM tools, products, and applications

D W tools, products, and applications

D W servers Comprehensive set of D W tools and applications D W tools, D W appliances, D W consultancy, and applications

en te rp rise-w id e d ata w a re h o u s e , a lso k n o w n as th e E D W a p p ro a ch . T h e s e c o n d a p p ro a ch is th at o f R alp h K im b all, w h o p ro p o s e s a b o tto m -u p a p p ro a ch that em p lo y s d im en sio n al m o d elin g , a lso k n o w n as th e d ata m art ap p ro a ch .

K n o w in g h o w th e s e tw o m o d els a re a lik e an d h o w th e y d iffer h e lp s us u n d erstan d th e b a sic d ata w a re h o u s e c o n c e p ts (e .g ., s e e B re slin , 2 0 0 4 ). T a b le 3 3 co m p a re s th e tw o a p p r o a c h e s. W e d e s c rib e th e s e a p p ro a c h e s in d etail n ext.

T H E INM O N M O D EL: T H E ED W A P P R O A C H In m o n ’S a p p r o a c h e m p h a siz e s to p -d o w n d ev elo p m en t, e m p lo y in g e sta b lish e d d a ta b a se d ev e lo p m e n t m e th o d o lo g ie s a n d to o ls, s u c h as e n tity-relation sh ip d iagram s (E R D ), a n d an ad ju stm en t o f th e sp iral d ev elo p m en t ap p ro a ch . T h e E D W a p p ro a ch d o e s n o t p re clu d e th e c re a tio n o f d ata m arts. T h e E D W is th e id eal in this a p p ro a ch b e c a u s e it p ro v id e s a co n siste n t and c o m p re h e n siv e v ie w o f th e en te rp rise. M urtaza ( 1 9 9 8 ) p re se n te d a fram ew o rk fo r d ev elo p in g EDW .

TH E K IM B A L L M O D EL: T H E D A T A M A R T A P P R O A C H K im b all’s data m art strategy is a “plan b ig, b u ild sm all” ap p ro a ch . A data m art is a s u b je ct-o rie n te d o r d ep artm e n t-o rien te d data w a reh o u se . It is a s c a le d -d o w n v e rs io n o f a d ata w a re h o u s e th at fo c u s e s o n th e req u ests

Chapter 3 • Data W arehousing 1 3 5

T A B L E 3.3 Contrasts B e tw e e n th e Data M a rt and E D W D evelopm ent Approaches

Effort Data M a rt Approach E D W Approach

Scope One subject area Several subject areas

Development time Months Years

Development cost $10,000 to $100,000+ $1,000,0004-

Development difficulty Low to medium High

Data prerequisite for sharing Common (within business area) Common (across enterprise)

Sources Only some operational and external systems Many operational and external systems

Size Megabytes to several gigabytes Gigabytes to petabytes

Time horizon Near-current and historical data Historical data

Data transformations Low to medium High

Update frequency Hourly, daily, weekly Weekly, monthly

T e c h n o lo g y Hardware Workstations and departmental servers Enterprise servers and mainframe computers

Operating system Windows and Linux Unix, Z/OS, OS/390

Databases Workgroup or standard database servers Enterprise database servers

Usage Number of simultaneous 10s 100s to 1,000s

users User types Business area analysts and managers Enterprise analysts and senior executives

Business spotlight Optimizing activities within Cross-functional optimization and decision the business area making

■urces: Adapted from J. Van den Hoven, "Data Marts: Plan Big, Build Small," in IS M anagem ent H an dbook. Sth ed.. CRC Press, B oca taton, FL, 2003; and T. Ariyachandra and H. Watson, “Which Data W arehouse Architecture Is Most Successful?” B usiness In telligen ce 'm m al, Vol. 11, No. 1, First Quarter 2006, pp. 4 -6 .

o f a sp e cific d ep artm e n t, s u ch as m ark etin g o r sales. T h is m o d e l a p p lie s d im en sio n al data m odeling, w h ich starts w ith ta b le s. K im ball a d v o cated a d ev e lo p m e n t m e th o d o lo g y that t r a i l s a b o tto m -u p a p p ro a c h , w h ich in th e c a s e o f data w a re h o u s e s m e a n s b u ild in g o n e

data m art a t a tim e.

WHICH M O D E L IS B E S T ? T h e re is n o o n e -siz e-fits-a ll strategy to data w areh o u sin g . An e n e r p r is e ’s d ata w a re h o u s in g strategy c a n e v o lv e fro m a sim p le data m art to a c o m p le x tfaro w a reh o u se in re s p o n s e to u s e r d em an d s, th e e n te rp rise ’s b u sin e ss re q u irem en ts, an d f e e e n te rp rise ’s m aturity in m an ag in g its d ata re so u rces. F o r m an y en te rp rises, a d ata mart b freq u en tly a c o n v e n ie n t first step to acq u irin g e x p e r ie n c e in co n stru ctin g a n d m a n a g ­ ing a data w a re h o u s e w h ile p re sen tin g b u sin e ss u se rs w ith th e b e n e fits o f b e tte r a c c e ss

: their data; in ad d ition, a data m art co m m o n ly in d icate s th e b u sin e ss valu e o f data m ̂ rehousing. U ltim ately, e n g in e e rin g a n E D W that co n so lid a te s o ld d ata m arts a n d data

• ^rehouses is th e id e a l so lu tio n ( s e e A p p lication C ase 3-5). H o w ev er, th e d ev e lo p m e n t of individual data m arts c a n o fte n pro v id e m an y b e n e fits alo n g th e w a y tow ard d e v e lo p - w g a n E D W , e sp e cia lly if th e o rg an ization is u n a b le o r u n w illin g to in v est in a la rg e-sc a le

reject. D ata m arts c a n a lso d em o n strate feasib ility an d s u c c e s s in p ro vid ing b en e fits. I K s co u ld p o ten tially lea d to a n in v estm en t in a n EDW . T a b le 3 .4 sum m arizes th e m ost essential ch aracteristic d iffe re n ce s b e tw e e n th e tw o m od els.

1 3 6 Part II • D escriptive Analytics

Application Case 3.5 Starwood Hotels & Resorts Manages Hotel Profitability with Data Warehousing S tarw o o d H o tels & R eso rts W o rld w id e, In c ., is o n e o f th e lea d in g h o tel a n d leisu re co m p a n ie s in th e w orld w ith 1 ,1 1 2 p ro p e rtie s in n e arly 1 0 0 co u n trie s and 1 5 4 ,0 0 0 e m p lo y e e s at its o w n e d an d m an ag ed p ro p ­ erties. S ta rw o o d is a fully in teg rated o w n e r, o p erato r an d fra n ch iso r o f h o tels, resorts, an d re s id e n c e s w ith th e fo llo w in g in tern ation ally re n o w n e d b rand s: St. R e g is® , T h e Luxury C o lle ctio n ® , W ® , W e stin ® , Le M erid ien ® , S h e ra to n ® , F o u r P o in ts® b y Sh eraton , A loft® , a n d E lem entSM . T h e C o m p an y b o a sts o n e o f th e indu stry’s lea d in g loyalty pro gram s, S tarw oo d P referred G u est (S P G ), allo w in g m e m b ers to earn a n d re d e e m p o in ts fo r r o o m stay s, r o o m u p g rad es, an d fligh ts, w ith n o b la c k o u t d ates. S ta rw o o d a lso o w n s S ta rw o o d V a ca tio n O w n e rsh ip In c ., a p re ­ m ier p ro v id e r o f w o rld -class v a ca tio n e x p e rie n c e s th r o u g h villa-style resorts a n d privileged a c c e s s to S ta rw o o d b rand s.

C h a lle n g e

S tarw oo d H o tels h a s significantly in cre a se d th e n u m ­ b e r o f h o tels it o p e ra te s o v e r th e p a st fe w y ears th ro u g h g lo b al co rp o ra te e x p a n sio n , particularly in th e Asia/Pacific reg io n . T h is has resu lted in a dra­ m atic rise in th e n e e d for b u sin ess critical inform a­ tio n a b o u t Starw oo d ’s h o tels and cu stom ers. All S tarw oo d h o te ls glo b ally u s e a sin gle en terp rise data w a re h o u s e to retriev e inform ation critical to efficien t h o te l m a n ag e m e n t, s u ch a n that regard ing rev e n u e , ce n tra l reservations, an d rate p lan reports. In addi­ tion , S tarw o o d H o tels’ m a n a g e m e n t runs im portant daily o p eratin g rep orts fro m th e d ata w a reh o u se fo r a w id e range o f b u sin ess fu n ction s. Starw oo d ’s e n te r­ p rise d ata w a reh o u se sp an s alm ost all areas w ithin th e co m p a n y , s o it is esse n tial n o t o n ly fo r central- reserv atio n and co n su m p tio n in form ation, b u t a lso to S tarw o o d ’s loyalty program , w h ich relies o n all gu est in form ation , sales in form ation, co rp o rate sa le s infor­ m ation , cu sto m e r serv ice, a n d o th e r data th at m an ­ agers, analysts, a n d e x e cu tiv e s d ep en d o n to m ake o p eratio n al d ecisio n s.

T h e co m p an y is com m itted to k n o w in g an d ser­ v icin g its guests, yet, “as data grow th and d em ands g re w to o great fo r th e co m p an y ’s leg acy system , it w as falling sh ort in delivering th e inform ation h otel m anagers and adm inistrators requ ired o n a daily

basis, s in ce cen tral reservation system (C RS) reports cou ld tak e a s lo n g as 18 h ou rs,” said R ichard Chung, Starw oo d H o tels’ d irector o f data integration. Chung ad d ed that h o te l m anag ers w o u ld re ce iv e th e tran­ sie n t p a ce rep o rt—-which presen ts m arket-segm ented inform ation o n reservations— 5 hou rs later than it w as n e ed e d . S u ch delays p rev en ted m anagers from adjusting rates appropriately, w h ich co u ld result in lost revenue.

S o lu tio n a n d R e s u lts

A fter re v iew in g sev eral v e n d o r o fferin g s, Starw oo d H o tels s e le c te d O ra cle E x ad ata D a ta b a se M achine X 2 -2 H C Full R a ck an d O ra cle E xad ata D a ta b a se M ach in e X 2 -2 HP Full R ack, ru nning o n O ra c le Linux. “W ith th e im p lem e n ta tio n o f E xad ata, Starw oo d H o tels c a n c o m p le te e x tra ct, tran sform , a n d load (E TL ) o p e ra tio n s fo r o p era tio n a l reports in 4 to 6 h ou rs, a s o p p o s e d to 18 to 2 4 hou rs p reviou sly , a s ix -fo ld im p ro v e m en t,” C h u n g said. R eal-tim e fe e d s, w h ic h w e r e n o t p o ss ib le b e fo re , n o w a llo w tran sac­ tio n s to b e p o sted im m ed iately to th e data w a re ­ h o u se , an d u sers c a n a c c e s s th e c h a n g e s in 5 to 10 m in u tes in s te a d o f 2 4 h o u rs, m ak in g th e p ro c e s s up to 2 8 8 tim e s faster.

A cce lera ted data a c c e s s allow s all Starw ood p ro p e rtie s t o g e t t h e s a m e , u p -to -d ate d ata n e e d e d fo r th e ir rep orts, globally. P reviously, h o te l m anagers in s o m e a re a s co u ld n o t d o sam e-d ay o r next-d ay analyses. T h e r e w e re so m e lo catio n s that g o t fresh data an d o th e rs th at g o t o ld e r data. H otel m anagers, w o rld w id e, n o w h av e u p -to-d ate data fo r th e ir h otels, in creasin g e fficie n cy a n d profitability, im proving cu s­ to m er serv ice b y m ak in g su re ro o m s are available for p rem ier cu stom ers, a n d im proving th e co m p a n y s ability to m a n a g e ro o m o c c u p a n c y rates. Additional rep ortin g to o ls , s u ch as th o se u s e d fo r CRM and sales and cate rin g , a lso b e n e fite d fro m th e im proved p ro ­ ce ssin g . O th e r critical rep orting has b e n e fite d a s w ell. M arketing cam p aig n m an ag e m e n t is a lso m o re effi­ c ie n t n ow th at m an ag ers c a n analyze results in days o r w e e k s in stead o f m onths.

“O ra c le E xad ata D a ta b a se M ach in e e n a b le s u s to m o v e forw ard w ith a n en v iro n m en t that p ro ­ v id es ou r h o te l m an ag ers an d co rp o ra te e x e cu tiv e s w ith n e a r -r e a l4 im e in form ation to m a k e optim al

Chapter 3 • Data W arehousing 1 3 7

b u sin e ss d e c is io n s and pro v id e ideal am en ities for ou r g u ests." — G o rd o n Light, B u s in e s s R elation sh ip M anager, S tarw o o d H otels & R eso rts W orld w id e, In c.

Q u e s t io n s f o r D is c u s s io n

1. H ow b ig a n d c o m p le x a re th e b u sin ess o p e ra ­ tio n s o f S tarw o o d H o tels & Resorts?

2. H o w did S tarw o o d H o tels & R eso rts u s e data w a reh o u sin g fo r b e tte r profitability?

3. W h a t w e re th e c h a lle n g e s, th e p ro p o s e d s o lu ­ tion , an d th e o b ta in e d results?

Source: Oracle customer success story, www.oracle.com/us/ corp orate/cu stom ers/custom ersearch /starw ood -hotels-1- exad ata-sl-1855106.h tm l; Starw ood H otels and R esorts, starw oodhotels.com (accessed Ju ly 2 0 1 3 ).

Additional Data W arehouse Development Considerations S o m e o r g a n iz a tio n s w a n t to c o m p le te ly o u ts o u r c e th e ir d ata w a r e h o u s in g e ffo rts . T h e y sim p ly d o n o t w a n t to d e a l w ith s o ftw a re a n d h a rd w a re a c q u isitio n s , a n d th e y d o n o t w a n t to m a n a g e th e ir in fo rm a tio n sy stem s. O n e a lte rn a tiv e is to u s e h o s te d data w a r e h o u s e s . In th is s c e n a r io , a n o th e r firm — id e a lly , o n e th a t h a s a lo t o f e x p e r ie n c e

T A B L E 3.4 Essential Differences B e tw e e n Inm on's and Kim ball's Approaches

Characteristic Inmon Kimball

M e th o d o lo g y a n d A r c h ite c tu re

Overall approach Top-down Bottom-up Architecture structure Enterprise-wide (atomic) data

warehouse "feeds" departmental databases

Data marts model a single business process, and enterprise consistency is achieved through a data bus and conformed dimensions

Complexity of the method Quite complex Fairly simple Comparison with established

development methodologies Derived from the spiral methodology Four-step process; a departure from relational

database management system (RDBMS) methods

Discussion of physical design Fairly thorough Fairly light

Data M o d e lin g

Data orientation Subject or data driven Process oriented Tools Traditional (entity-relationship diagrams

[ERD], data flow diagrams [DFD]) Dimensional modeling; a departure from

relational modeling End-user accessibility Low High

P h ilo s o p h y

Primary audience IT professionals End users Place in the organization Integral part of the corporate

information factory Transformer and retainer of operational data

Objective Deliver a sound technical solution based on proven database methods and technologies

Deliver a solution that makes it easy for end users to directly query the data and still get reasonable response times

Sources: Adapted from M. Breslin, “Data Warehousing Battle o f the Giants: Comparing the Basics o f Kimball and Inmon Models," B usiness In tellig en ce Jo u r n a l Vol. 9, No. 1, W inter 2004, pp. 6 -2 0 ; and T. Ariyachandra and FT. Watson, “W hich Data W arehouse Architecture Is Most Successful?” B u sin ess In tellig en ce Jo u rn a l, Vol. 11, No. 1, First Quarter 2006.

1 3 8 Part I I • D escriptive Analytics

T E C H N O L O G Y IN S IG H T S 3 . 1 H o s t e d D a ta W a r e h o u s e s

A h osted data w areh ou se has nearly the sam e, if n o t m ore, functionality as an on-site data w are­ h ouse, but it d oes not con su m e com p u ter resources o n client prem ises. A hosted data w arehou se offers th e benefits o f BI minus the cost o f com p u ter upgrades, netw ork upgrades, software licenses, in-hou se developm ent, and in-hou se support and m aintenance.

A hosted data w areh ou se offers th e follow ing benefits:

• R e q u ire s m in im a l in v e s tm e n t in in fra stru ctu re • Frees up capacity o n in-hou se system s • F r e e s u p c a s h flo w • M akes pow erful solutions affordable • E n a b le s p o w e r fu l so lu tio n s th a t p ro v id e fo r g ro w th • O ffers b etter quality equipm ent and softw are • P ro v id e s fa s te r c o n n e c tio n s • E nables users to access data from rem ote locations • Allows a com pany to focu s o n c o re business • M eets storage need s for large volum es o f data

D espite its benefits, a h osted data w areh ou se is not necessarily a g o o d fit for every organi­ zation. Large com p an ies w ith revenue upwards o f $ 5 0 0 m illion cou ld lose m oney if they already have underused internal infrastructure and IT staff. Furtherm ore, com p an ies that s e e the para­ digm shift o f outsourcing applications as loss o f con trol o f their data are n o t likely to u se a business intelligence service provider (BISP). Finally, the m ost significant and com m on argument against im plem enting a h osted data w areh ou se is that it may b e unw ise to outsource sensitive applications for reasons o f security and privacy.

Sources: Compiled from M. Thornton and M. Lampa, “Hosted Data Warehouse,’'Jou rn al o f D ata W arehousing, Vol. 7. No. 2, 2002, pp. 2 7 -3 4 ; and M. Thornton, “What About Security? T he Most Common, but Unwarranted, Objection to Hosted Data Warehouses,” DM Review, Vol. 12, No. 3, March 18, 2002, pp. 30-43.

a n d e x p e r tis e — d e v e lo p s an d m a in ta in s th e d ata w a r e h o u s e . H o w e v e r, th e re are s e c u rity an d p riv a cy c o n c e r n s w ith this a p p r o a c h . S e e T e c h n o lo g y In sig h ts 3 .1 fo r s o m e d etails.

Representation o f Data in Data W arehouse A ty p ical d ata w a re h o u s e stru cture is sh o w n in Figure 3-3. M any v ariation s o f d ata w a re ­ h o u s e arch itectu re are p o ss ib le ( s e e F igu re 3 -7 ). No m atter w h at th e arch ite ctu re w as, th e d esig n o f data re p re sen ta tio n in th e data w a r e h o u s e h a s alw ays b e e n b a s e d o n th e c o n c e p t o f d im en sion al m o d elin g. Dim ensional modeling is a re triev al-b ase d system that su p p orts h ig h -v o lu m e q u e ry a c c e s s . R e p re se n ta tio n a n d sto rag e o f d ata in a data w a re h o u s e sh o u ld b e d esig n e d in s u ch a w ay th at n o t o n ly a cco m m o d a te s b u t also b o o s ts th e p ro ce ssin g o f c o m p le x m u ltid im en sion al q u e rie s. O fte n , th e star sch e m a an d th e sn o w fla k es s ch e m a a re th e m e a n s b y w h ic h d im en sio n al m o d e lin g is im p lem e n te d in d ata w areh o u se s.

T h e star schem a (s o m e tim e s r e fe re n c e d as star jo in s ch e m a ) is th e m o st co m m o n ly u se d a n d the sim p lest sty le o f d im en sio n al m o d e lin g . A star s ch e m a co n ta in s a cen tral fact ta b le su rro u n d ed b y a n d c o n n e c te d to sev eral dimension tables (A d am son, 2 0 0 9 ). T h e fact ta b le co n ta in s a large n u m b e r o f ro w s th a t c o rre s p o n d to o b s e rv e d facts a n d e xte rn al links (i.e ., fo reig n k e y s). A fa ct ta b le co n ta in s th e d escrip tiv e attributes n e e d e d to p erfo rm d e c is io n analysis and q u e ry rep ortin g, a n d fo re ig n k e y s are u s e d to lin k to d im en sio n

Chapter 3 • D ata W arehousing 1 3 9

tab les. T h e d e c is io n analysis attributes c o n s is t o f p e rfo rm a n ce m e asu re s, o p era tio n a l m e t­ rics, ag g reg ated m e a su re s (e .g ., s a le s v o lu m es, cu sto m e r re te n tio n rates, profit m argins, p ro d u ctio n co sts, cra p rates, a n d s o fo rth ), an d all th e o th er m etrics n e e d e d to a n a ly z e the org an ization ’s p e rfo rm a n ce . In o th e r w ord s, th e fa ct ta b le prim arily ad d resses w h at the data w a re h o u s e su p p o rts fo r d e c is io n analysis.

Surrou nd ing th e cen tral fa ct tab le s (a n d lin k e d via fo re ig n k e y s ) are d im en sio n ta b les. T h e d im e n sio n tab le s co n ta in classificatio n an d a g g reg atio n in fo rm atio n a b o u t the central fa ct row s. D im e n s io n ta b le s c o n ta in attributes th at d e s c rib e th e d ata co n ta in e d w ithin th e fa ct tab le ; th e y ad d ress h o w data w ill b e an aly zed and sum m arized . D im en sio n tab le s h a v e a o n e -to -m a n y relatio n sh ip w ith ro w s in th e cen tral fa ct tab le. In q u eryin g, the d im en sio n s are used to s lice a n d d ice th e n u m erical v a lu e s in th e fact ta b le to address the re q u irem en ts o f a n ad h o c in form ation n e ed . T h e star s ch e m a is d esig n e d to provide fast q u e ry -re s p o n se tim e, sim plicity, a n d e a s e o f m ain te n a n ce fo r re a d -o n ly d atab ase structures. A sim p le star s ch e m a is sh o w n in Figure 3 .1 0 a . T h e star s ch e m a is co n sid e re d a s p e c ia l c a s e o f th e sn o w fla k e sch e m a.

T h e snow flake schem a is a lo g ica l arran g em en t o f ta b le s in a m u ltid im en sion al d atab ase in s u c h a w a y that th e en tity-relation sh ip d iagram re se m b le s a sn o w fla k e in sh ap e . C lo se ly r e la ted to th e star sch e m a , th e sn o w fla k e s ch e m a is re p re s e n te d b y ce n tra l­ ized fact ta b le s (u su ally o n ly o n e ) that a re c o n n e c te d to m u ltiple d im en sion s. In th e s n o w ­ flak e sch e m a , h o w e v e r, d im en sio n s are no rm alized in to m u ltip le related ta b le s w h e re a s the star s c h e m a ’s d im en sio n s are d en o rm alize d w ith e a c h d im en sio n b e in g re p re sen ted b y a sin g le ta b le . A sim p le sn o w fla k e s ch e m a is sh o w n in Fig u re 3.1 0 b .

Analysis o f Data in the Data Warehouse O n c e th e d ata is p ro p e rly sto red in a data w a re h o u s e , it c a n b e u s e d in vario u s w ay s to su p p ort o rg an izatio n al d e cisio n m akin g. OLAP (o n lin e an aly tical p ro ce s s in g ) is argu ably the m o st c o m m o n ly u s e d d ata analysis te ch n iq u e in data w a reh o u se s, a n d it h a s b e e n g ro w in g in p o p u larity d u e to th e e x p o n e n tia l in c re a se in data v o lu m e s an d th e re co g n i­ tio n o f th e b u sin ess v a lu e o f d ata-d riven analytics. Sim ply, OLAP is an a p p ro a ch t o q u ick ly an sw e r ad h o c q u e s tio n s b y e x e c u tin g m u ltid im ensional analy tical q u e ries a g ain st org an i­ zation al d ata re p o sito rie s (i.e ., d ata w areh o u se s, data m arts).

FIG U R E 3 .1 0 (a) The Star Schema, and (b) the Sno w flake Schema.

1 4 0 Part II • D escriptive Analytics

OLAP Versus OLTP OLTP (o n lin e tran sactio n p ro ce s s in g sy stem ) is a te rm u s e d fo r a tra n sa ctio n system , w h ich is prim arily re s p o n s ib le fo r cap tu rin g a n d storin g data related to d ay -to -d ay b u si­ n e ss fu n ction s su ch as ERP, CRM, SCM, p o in t o f sa le , an d s o forth. T h e O LTP system ad d resses a critical b u sin e ss n e ed , a u tom atin g d aily b u sin e ss tran sactio n s an d running real-tim e rep o rts a n d ro u tin e analyses. B u t th e s e sy stem s are n o t d esig n e d fo r a d h o c an aly sis a n d c o m p le x q u e ries that d eal w ith a n u m b e r o f data item s. OLAP, o n th e o th er h an d , is d esig n e d to ad d ress this n e e d b y p ro v id in g ad h o c analysis o f organ ization al data m u c h m o re e ffe ctiv e ly an d efficien tly. OLAP an d O LTP rely h eavily o n e a c h o th er: OLAP u s e s th e d ata cap tu res b y O LTP, an d O LTP a u to m a te s th e b u sin ess p ro c e s s e s that are m a n a g e d b y d ecisio n s s u p p o rte d b y OLAP. T a b le 3-5 p ro v id es a m ulti-criteria co m p a riso n b e tw e e n O LTP an d OLAP.

OLAP Operations T h e m ain o p era tio n a l structure in OLAP is b a s e d o n a c o n c e p t calle d cube. A c u b e in OLAP is a m ultid im ensional data structure (actu al o r virtual) th at allow s fast analysis o f data. It c a n a lso b e d efin e d as th e cap ab ility o f efficie n tly m anipu lating an d analyzing data fro m m ultiple p ersp ectiv es. T h e arran gem en t o f d ata into c u b e s aim s to o v e rco m e a limita­ tio n o f relatio nal d atabases: R elational d atab ases a re n o t w ell su ited fo r n e a r instantaneou s analysis o f large am o u nts o f data. Instead , th ey a re b e tte r su ited fo r m anipu lating record s (ad d in g, d eletin g , a n d up dating d ata) that re p re se n t a series o f tran saction s. A lthough m an y report-w riting to o ls e x is t fo r relatio nal d atab ases, th e se to o ls are slo w w h e n a multi­ d im en sion al q u ery that e n c o m p a ss e s m any d atab ase tab le s n e e d s to b e e x e cu ted .

U sing OLAP, a n analyst c a n n av igate th ro u g h th e d atab ase a n d s c r e e n fo r a p ar­ ticu lar su b s e t o f th e data (a n d its p ro g re ssio n o v e r tim e ) b y ch an g in g th e d ata’s o rien ta ­ tio n s and d efin in g analytical calcu latio n s. T h e s e ty p e s o f u ser-in itiated nav igation o f data th ro u g h th e s p e cifica tio n o f slice s (v ia ro tatio n s) an d drill down/up (v ia ag g reg atio n and d isag g re g atio n ) is s o m e tim e s ca lle d “s lice an d d ic e .” C o m m o n ly u s e d OLAP op eratio n s in clu d e s lic e and d ice , drill d o w n , roll up, an d p ivot.

• S lic e . A s lic e is a su b s e t o f a m u ltid im en sio n al array (u su ally a tw o-d im en sional re p re se n ta tio n ) co rre sp o n d in g to a sin gle v a lu e s e t fo r o n e (o r m o re ) o f th e d im en ­ sio n s n o t in th e su b set. A sim p le slicin g o p e ra tio n o n a th ree -d im en sio n al c u b e is s h o w n in Fig u re 3.11.

T A B L E 3.5 A Comparison B e tw e e n OLTP and O LAP

Criteria OLTP O LAP Purpose To carry out day-to-day business functions To support decision making and provide answers

to business and management queries Data source Transaction database (a normalized data

repository primarily focused on efficiency and consistency)

Data warehouse or data mart (a nonnormalized data repository primarily focused on accuracy and completeness)

Reporting Routine, periodic, narrowly focused reports Ad hoc, multidimensional, broadly focused reports and queries

Resource requirements

Execution speed

Ordinary relational databases

Fast (recording of business transactions and routine reports)

Slow (resource intensive, complex, large-scale queries)

Multiprocessor, large-capacity, specialized databases

Chapter 3 * D ata W arehou sing 141

A three-dimensional □LAP cube with slicing operations

Sales volumes of a specific product on variable time and region

Cells are filled with num bers representing

sales volumes CD

Sales volumes of a specific region on variable time and products

Sales volumes of a specific time on variable region and products

F IG U R E 3 .1 1 S licin g O p e ra tio n s o n a S im p le T h re e -D im e n sio n a l D ata Cube.

• D ic e . T h e d ic e o p e ra tio n is a s lice o n m o re th a n tw o d im en sio n s o f a d ata cu b e. • D r i l l D o w n /U p D rilling d o w n o r u p is a s p e cific OLAP te c h n iq u e w h e re b y th e

u s e r n av igates a m o n g lev els o f d ata ran gin g fro m th e m o st su m m arized (u p ) to th e m o st d eta iled (d o w n ).

• R o ll- u p . A ro ll-u p in v olv es co m p u tin g all o f th e data re latio n sh ip s fo r o n e o r m ore d im en sio n s. T o d o this, a com p u tatio n al relatio n sh ip o r fo rm u la m ight b e d efin ed .

• P iv o t : A p iv o t is a m e a n s o f ch a n g in g th e d im en sio n al o rie n ta tio n o f a re p o rt or ad h o c q u e ry -p a g e display.

V A R IA T IO N S OF O L A P OLAP h a s a fe w v ariation s; am o n g th e m ROLAP, MOLAP, and HOLAP are th e m o st co m m o n o n e s.

ROLAP stand s fo r R elation al O n lin e A nalytical P ro ce ssin g . ROLAP is a n altern ativ e to th e MOLAP (M ultid im ension al OLAP) te ch n o lo g y . A lthough b o th ROLAP a n d MOLAP analytic to o ls a re d e s ig n e d t o a llo w analysis o f data th ro u g h th e u se o f a m u ltid im en sion al d ata m o d e l, ROLAP d iffers significantly in that it d o e s n o t re q u ire th e p re co m p u ta tio n . nd sto rag e o f in fo rm atio n . In stead , ROLAP to o ls a c c e s s th e data in a re latio n al d atab ase and g e n era te SQ L q u e rie s to ca lcu la te in form ation at the ap p ro p riate lev el w h e n a n en d user re q u e sts it. W ith ROLAP, it is p o ss ib le to c re a te ad ditional d a ta b a se ta b le s (su m m ary tab le s o r ag g re g a tio n s) th at sum m arize th e d ata at an y d esire d c o m b in a tio n o f d im en sion s. W hile ROLAP u se s a relatio n al d a ta b a se so u rce , g e n era lly th e d atab ase m u st b e carefu lly d esigned fo r ROLAP u se. A d a ta b a se that w a s d esig n ed fo r O LTP will n o t fu n ctio n w e ll as a ROLAP d atab ase. T h e re fo re , ROLAP still in v olv es creatin g a n ad d ition al c o p y o f th e data.

1 4 2 Part II • D escriptive Analytics

MOLAP is an alternativ e to th e ROLAP te ch n o lo g y . MOLAP differs fro m ROLAP significantly in th a t it re q u ires th e p re co m p u ta tio n an d sto rag e o f in form ation in the c u b e — th e o p e ra tio n k n o w n as p re p ro cessin g . MOLAP s to res this d ata in a n op tim ized m u ltid im en sion al array sto ra g e, rath er th an in a relatio nal d atab ase (w h ic h is o ften the c a s e fo r ROLAP).

T h e u n d e sira b le tra d e -o ff b e tw e e n ROLAP a n d MOLAP w ith reg ard s to th e addi­ tio n al ETL (e x tra ct, transform , an d lo a d ) c o s t a n d slo w q u e ry p e rfo rm a n ce h a s le d to in qu iries fo r b e tte r a p p ro a ch e s w h e re th e p ro s a n d c o n s o f th e s e tw o a p p r o a c h e s are op tim ized . T h e s e in qu iries resu lted in HOLAP (H yb rid O n lin e A nalytical P ro ce ssin g ), w h ich is a c o m b in a tio n o f ROLAP and MOLAP. HOLAP a llo w s storing part o f th e data in a MOLAP sto re an d an o th e r part o f th e data in a ROLAP store. T h e d eg re e o f co n tro l that th e c u b e d e sig n e r h a s o v e r this p artitioning v a rie s fro m p ro d u ct to p ro d u ct. T e c h n o lo g y Insigh ts 3-2 p ro v id e s an o p p ortu n ity fo r c o n d u ctin g a sim p le h a n d s-o n analysis w ith the M icroStrategy B I to ol.

T E C H N O L O G Y IN SIG H T S 3 . 2 H a n d s -O n D a ta W a r e h o u s in g w ith M ic r o S tr a te g y

MicroStrategy is the leading independent provider o f business intelligence, data warehousing performance management, and business reporting solutions. The other big players in this market were recently acquired by large IT firms: Hyperion was acquired by Oracle; Cognos was acquired by IBM; and Business Objects was acquired by SAP. Despite these recent acquisitions, the busi­ ness intelligence and data warehousing market remains active, vibrant, and full o f opportunities.

Following is a step-by-step approach to using MicroStrategy software to analyze a hypo­ thetical business situation. A more comprehensive version o f this hands-on exercise can be found at the TUN Web site. According to this hypothetical scenario, you (the vice president of sales at a global telecommunications company) are planning a business visit to the European region. Before, meeting with the regional salespeople on Monday, yon want to know the sale representatives' activities for the last quarter (Quarter 4 of 2004). You are to create such an ad hoc report using MicroStrategy’s Web access. In order to create this and many other OLAP reports, you will need the access code for the T e r a d a ta U n iv e r s ity N e tw o r k .c o m Web site. It is free o f charge for educational use and only your professor will be able to get die necessary access code for you to utilize not only MicroStrategy software but also a large collection o f other business intelligence resources at this site.

Once you are in TeradataLiniversityNetwork, you need to go to ' APPLY & DO ” and select “MicroStrategy BI” from the “Software” section. On the “MicroStrategy/BI” Web page, follow these steps:

1 . Click on the link for “MicroStrategy Application Modules.” This will lead you to a page that shows a list o f previously built MicroStrategy applications.

2 . Select the “Sales Force Analysis Module.” This module is designed to provide you with in- depth insight into the entire sales process. This insight in turn allows you to increase lead conversions, optimize product lines, take advantage o f your organization’s most successful sales practices, and improve your sales organization’s effectiveness.

3 . In the “Sales Force Analysis Module” site you will see three sections: View, Create, and Tolls. In the View section, click on the link for “Shared Reports.” This link will take you to a place where a number o f previously created sales reports are listed for everybody’s use.

4 . In the “Shared Reports” page, click on the folder named “Pipeline Analysis.” Pipeline Analysis reports provide insight into all open opportunities and deals in the sales pipeline. These reports measure the current status o f the sales pipeline, detect changing trends and key events, and identify key open opportunities. You want to review what is in the pipe­ line for each sales rep, as well as whether or not they hit their sales quota last quarter.

5- In the “Pipeline Analysis" page, click on the report named “Current Pipeline vs. Quota by Sales Region and District.” This report presents the current pipeline status for each sales

Chapter 3 • Data W arehou sing 1 4 3

district within a sales region. It also projects whether target quotas can b e achieved for the current quarter.

6. In the “Current Pipeline vs. Quota by Sales Region and District” page, select (with single click) “2004 Q4” as the report parameter, indicating that you want to see how the repre­ sentatives performed against their quotas for the last quarter.

7- Run the report by clicking on the “Run Report” button at the bottom o f the page. This will lead you to a sales report page where the values for each Metric are calculated for all three European sales regions. In this interactive report, you can easily change the region from Europe to United States or Canada using the pull-down combo box, or you can drill-in one o f the three European regions by simply clicking on the appropriate region’s heading to see more detailed analysis o f the selected region.

SECTION 3 . 6 R EV IEW QUESTIONS

1 . List th e b e n e fits o f d ata w areh o u se s. 2 . List sev eral criteria fo r s e le c tin g a d ata w a re h o u s e v en d o r, an d d e scrib e w h y th e y are

im p oitant. 3 . W h at is OLAP a n d h o w d o e s it d iffer fro m OLTP? 4 . W h at is a cu b e ? W h at d o drill d o w n , ro ll u p , an d s lic e an d d ice m ean?

5 . W h a t a re ROLAP, MOLAP, an d HOLAP? H o w d o th e y differ fro m OLAP?

3.7 D A TA W AREHOUSING IM PLEM ENTATION ISSU ES Im p lem en tin g a data w a r e h o u s e is g en erally a m assiv e e ffo rt th at m u st b e p la n n e d and e x e c u te d a c c o r d in g to e sta b lish e d m eth o d s. H o w ev er, th e p ro je ct life c y c le has m any facets, an d n o sin g le p e rs o n c a n b e an e x p e rt in e a c h are a. H ere w e d iscu ss s p e c ific id eas an d issu es as th e y relate to data w areh o u sin g .

P e o p le w a n t to k n o w h o w su cce ssfu l th e ir B I an d d ata w a re h o u sin g initiatives are in co m p a riso n to th o se o f o th e r co m p a n ie s. A riyachand ra a n d W a tso n (2 0 0 6 a ) p ro­ p o se d s o m e b e n ch m a rk s fo r B I and data w a reh o u sin g s u cce s s . W a tso n e t al. (1 9 9 9 ) re se a rch e d data w a re h o u s e failu res. T h e ir results s h o w e d th a t p e o p le d e fin e a “failu re” in d ifferen t w ays, a n d this w as co n firm ed b y A riyachand ra and W a tso n (2 0 0 6 a ) . T h e D ata W are h o u sin g In stitu te (td w i.o r g ) h a s d e v e lo p e d a data w a reh o u sin g m atu rity m o d el th at an e n te rp rise c a n ap p ly in ord e r to b e n c h m a rk its e v o lu tio n . T h e m o d e l o ffers a fast m e a n s to g au g e w h e re th e o rg an izatio n ’s data w a reh o u sin g initiative is n o w an d w h e re it n e e d s to g o n e x t. T h e m aturity m o d el co n sists o f s ix stages: p ren atal, in fan t, child, te en ag er, adult, an d sag e. B u s in e s s v a lu e rises as th e data w a r e h o u s e p ro g re ss e s throu gh e a c h s u c c e e d in g stag e. T h e sta g e s a re id entified b y a n u m b e r o f c h aracteristics, inclu d ing s c o p e , an alytic stru ctu re, e x e c u tiv e p e rce p tio n s , ty p e s o f analytics, stew ard sh ip , fu nd ing, te ch n o lo g y p latfo rm , c h a n g e m an ag e m e n t, an d ad m inistration. S e e E ck e rso n e t al. (2 0 0 9 ) a n d E c k e r s o n (2 0 0 3 ) fo r m o re details.

D ata w a r e h o u s e p ro je cts h av e m an y risks. M o st o f th em are a lso fo u n d in o th e r IT p ro jects, b u t d ata w a reh o u sin g risks are m o re s erio u s b e c a u s e data w a r e h o u s e s are e x p e n ­ sive, tim e -a n d -re s o u rce d em an d in g , la rg e -sc a le p ro je cts. E a ch risk sh ou ld b e a ss e s s e d at th e in ce p tio n o f th e p ro je ct. W h e n d e v e lo p in g a su cce ssfu l data w a re h o u se , it is im portant to carefu lly c o n s id e r various risks a n d av o id th e fo llo w in g issues:

• S t a r t i n g w it h t h e w r o n g s p o n s o r s h i p c h a i n . Y o u n e e d a n e x e c u tiv e sp o n so r w h o h a s in flu e n c e o v e r th e n e ce s s a ry re s o u rc e s to su p p o rt an d in v e st in th e data w a r e h o u s e . Y o u a lso n e e d a n e x e c u tiv e p ro je c t driver, s o m e o n e w h o h a s e a rn e d

1 4 4 Part I I • D escriptive Analytics

th e re s p e c t o f o th e r e x e c u tiv e s , h a s a h e a lth y s k ep ticism a b o u t te c h n o lo g y , a n d is d ecisiv e b u t fle x ib le . Y o u a lso n e e d a n IS/IT m an ag e r to h e a d u p th e p ro ject.

• S ettin g exp ecta tio n s th a t y o u c a n n o t m eet. Y o u d o n o t w an t to frustrate e x e c ­ u tiv es at th e m o m en t o f truth. E very d ata w a reh o u sin g p ro je c t h a s tw o p h ases: P h a s e 1 is th e sellin g p h ase , in w h ic h y o u internally m ark et th e p ro je ct b y selling th e b e n e fits to th o se w h o h av e a c c e s s to n e e d e d re so u rces. P h a se 2 is th e stru ggle to m e e t th e e x p e c ta tio n s d e s c rib e d in P h a s e 1. F o r a m e re $1 to $7 m illion, h o p efu lly, y o u c a n deliver.

• E n g a g i n g i n p o litica lly n a iv e b eh a v io r. D o n o t sim ply state that a d ata w a re ­ h o u s e w ill h e lp m an ag e rs m a k e b e tte r d e c is io n s . T h is m ay im p ly that y o u fe e l they h av e b e e n m ak in g b ad d e cis io n s until n o w . Sell th e id e a that th e y w ill b e a b le to get th e in form ation th ey n e e d to h e lp in d e c is io n m aking.

• L o a d in g the w a re h o u s e with in fo r m a t io n j u s t b e c a u s e it is a v a ila b le. D o n o t let th e d ata w a re h o u s e b e c o m e a d ata landfill. T h is w o u ld u n n ece ssarily slo w th e u se o f th e system . T h e re is a tre n d to w ard real-tim e co m p u tin g a n d analysis. D ata w a re h o u s e s m u st b e shu t d o w n to lo a d d ata in a tim ely way.

• B e lie v in g th a t d a ta w a r e h o u s in g d a ta b a s e d e s ig n is th e s a m e a s tra n s a c ­ tio n a l d a ta b a se d e s ig n . I n g e n era l, i t is n o t. T h e goal o f d ata w a re h o u sin g is to a c c e s s ag g re g ate s ra th er th a n a sin g le o r a fe w re co rd s, a s in tran sactio n -p ro cessin g sy stem s. C o n te n t is a lso d ifferen t, as is e v id e n t in h o w data are organ ized . D BM S te n d to b e n o n red u n d an t, n orm alized , a n d relatio n al, w h ere a s data w a re h o u s e s are red u nd ant, n o t n orm alized , a n d m u ltid im ensional.

9 C h o o sin g a d a ta w a re h o u s e m a n a g e r who is tech n o lo gy o rie n t e d r a t h e r t h a n u s e r o rien ted . O n e k e y to d ata w a r e h o u s e s u c c e s s is to u n d erstan d th at th e u sers m u st g e t w h a t th e y n e e d , n o t a d v a n ce d te ch n o lo g y for te c h n o lo g y ’s sak e.

• F o c u s in g o n tra d itio n a l i n t e r n a l re c o rd -o rie n te d d a ta a n d i g n o r i n g the v a lu e o f e x t e r n a l d a ta a n d o f text, im a ges, a n d , p e r h a p s , s o u n d a n d video. D ata c o m e in m an y form ats a n d m ust b e m ad e a c c e s s ib le to th e right p e o ­ p le at th e right tim e an d in th e right form at. T h e y m ust b e ca ta lo g e d properly.

• D e liv e rin g d a ta with o v e rla p p in g a n d c o n f u s i n g d e fin itio n s . D ata c le a n s ­ ing is a critical a s p e c t o f data w areh o u sin g . It in clu d es re co n cilin g co n flictin g data d efin itio n s a n d form ats o rg an ization -w id e. P olitically, this m ay b e d ifficu lt b e c a u s e it in v olv es ch a n g e , typically a t th e e x e c u tiv e lev el.

• B e lie v in g p r o m is e s o f p e r fo r m a n c e , capacity, a n d scalability . D ata w are­ h o u s e s g e n era lly re q u ire m o re ca p a city an d s p e e d th a n is originally b u d g e te d for. P lan a h e a d to s c a le up.

• B eliev in g that y o u r p ro b lem s a r e o v er w h en the d a ta w a reh o u se is u p a n d r u n n i n g . DSS/BI projects tend to e v o lv e continually. E ach d ep loym ent is a n iteration o f the prototyping process. T h e re will alw ays b e a n e e d to add m o re and different data sets to th e data w areho u se, as w ell as additional analytic to o ls fo r existing and addi­ tional g roups o f d ecisio n m akers. High en erg y a n d annual b ud gets m ust b e p lan n ed for b e ca u se su cce ss b reed s su ccess. Data w areh o u sin g is a continu ou s process.

• F o c u s in g o n a d h o c d a ta m i n i n g a n d p e r i o d i c r e p o r t in g in s t e a d o f a lerts. T h e natural p ro g re ssio n o f in form ation in a d ata w a re h o u s e is ( D e xtract th e data fro m le g a cy sy stem s, c le a n s e th e m , an d fe e d th e m to th e w a re h o u se ; (2 ) su p p ort ad h o c re p o rtin g un til you learn w h at p e o p le w an t; a n d ( 3 ) co n v e rt th e ad h o c reports into regularly sch e d u le d re p o rts. T h is p ro c e s s o f learn in g w h a t p e o p le w a n t in o rd e r to pro v id e it se e m s natu ral, b u t it is n o t op tim al o r e v e n p ractical. M an agers are b u sy an d n e e d tim e to re a d reports. A lert system s are b e tte r than p e rio d ic rep orting system s a n d c a n m a k e a data w a re h o u s e m issio n critical. Alert system s m o n ito r th e d ata flo w in g in to th e w a re h o u s e a n d in form all k e y p e o p le w h o h av e a n e e d to k n o w as s o o n as a critical ev e n t o ccu rs.

C hapter 3 • Data W arehousing 1 4 5

In m an y o rg an izatio n s, a d ata w a re h o u s e will b e su cce ssfu l o n ly i f th e re is strong s e n io r m a n a g e m e n t s u p p o rt fo r its d ev elo p m en t a n d i f th e re is a p ro je ct c h a m p io n w h o is high up in th e o rg an izatio n al chart. A lthough this w o u ld lik e ly b e tru e lo r a n y large- sca le IT p ro je ct, it is e sp e c ia lly im p ortan t fo r a data w a re h o u s e realization. T h e su cce ssfu l im p lem en tatio n o f a d ata w a r e h o u s e results in th e e stab lish m e n t o f a n arch ite ctu ral fram e­ w o rk th at m ay allowr fo r d e c is io n analysis th ro u g h o u t an org an izatio n and in s o m e ca ses a lso p ro v id es co m p re h e n siv e SCM b y granting a c c e s s to in form ation o n a n o rg an izatio n s cu stom ers an d su p p liers. T h e im p lem e n tatio n o f W e b -b a s e d data w a re h o u s e s (s o m e tim e s calle d W ebhousing) h a s facilitated e a s e o f a c c e s s to v ast am o u n ts o f d ata, b u t it is dif­ ficult to d eterm in e th e hard b e n e fits a s s o cia te d w ith a data w a re h o u se . H ard b e n e fits are d efin ed as b e n e fits t o a n o rg an izatio n th at c a n b e e x p r e s s e d in m o n etary term s. M any organizations h a v e lim ited IT re so u rce s a n d m u st prioritize p ro jects. M an agem en t su p p ort and a stro n g p ro je c t c h a m p io n c a n h e lp e n su re th at a data w a reh o u se p ro je ct w ill re ce iv e th e re so u rces n e c e s s a ry fo r su cce ssfu l im p lem en tation . D ata w a re h o u s e r e s o u r c e s c a n b e a sig n ifican t co st, in s o m e c a s e s requ irin g h ig h -e n d p ro c e s s o rs an d large in c re a se s in d irect-access sto rag e d e v ice s (D A SD ). W e b -b a s e d data w a re h o u s e s m ay also h a v e sp e cial secu rity req u irem en ts to e n su re th at o n ly au th orized u se rs h av e a c c e s s to th e data.

U ser p a rticip a tio n in th e d ev e lo p m e n t o f data an d a c c e s s m o d e lin g is a critical su c­ cess fa c to r in d ata w a re h o u s e d ev elo p m en t. D uring data m o d elin g , e x p e rtis e is req u ired ro d eterm in e w h at d ata are n e e d e d , d efin e b u sin e ss rules a sso cia ted w ith th e d ata, and d ecid e w h at ag g re g atio n s and o th e r ca lcu la tio n s m ay b e n e ce ssa ry . A ccess m o d e lin g is n e e d e d t o d eterm in e h o w data a re to b e retriev ed fro m a data w a re h o u se , an d it assists in ± e p h y sical d efin itio n o f th e w a re h o u s e b y h e lp in g to d e fin e w h ich d ata re q u ire in d e x ­ ing. It m ay a lso in d ica te w h e th e r d e p e n d e n t data m arts are n e e d e d to facilitate in fo rm a­ tio n retrieval. T h e te a m skills n e e d e d to d ev elo p an d im p lem e n t a data w a r e h o u s e in clu d e in -d ep th k n o w le d g e o f th e d atab ase te ch n o lo g y an d d ev e lo p m e n t to o ls u sed . S o u rc e sys­ tem s a n d d e v e lo p m e n t te c h n o lo g y , as m e n tio n e d p reviou sly, re fe re n c e th e m an y inputs and th e p ro c e s s e s u s e d to lo a d an d m ain tain a data w a reh o u se .

A p p lication C a se 3 .6 p re se n ts a n e x c e lle n t e x a m p le fo r a larg e-scale im p lem e n tatio n o f an in teg rated d ata w a re h o u s e b y a state g ov ern m en t.

Application Case 3.6 E D W Helps C onnect S ta te Agencies in M ichigan T h ro u g h cu sto m e r serv ice, re s o u rce op tim ization, and th e in n o v ativ e u se o f in form ation an d te c h ­ n ology, th e M ich igan D ep artm e n t o f T e ch n o lo g y , M an agem en t & B u d g e t (D T M B ) im p acts every area o f g o v ern m e n t. N early 1 0 ,0 0 0 u sers in five m ajo r d ep artm en ts, 2 0 a g e n c ie s , an d m o re th an 100 b u re au s rely o n th e E D W to d o th e ir jo b s m o re e ffectiv ely an d b e tte r serv e M ich ig an resid en ts. T h e E D W a c h ie v e s $1 m illio n p e r b u sin e ss d ay in fin a n ­ cial b en e fits.

T h e E D W h e lp e d M ichigan a c h ie v e $ 2 0 0 m illion in annu al fin an cial b e n e fits w ithin th e D ep artm e n t o f C om m unity H e a lth a lo n e , p lu s an o th e r $ 7 5 m illion

p e r y e a r w ith in th e D ep a rtm e n t o f H u m an Serv ices (D H S). T h e s e savings in c lu d e p ro g ram integrity b e n ­ efits, c o s t a v o id a n ce d u e to im p ro ved o u tco m e s, sa n ctio n av o id a n ce , o p era tio n a l e ffic ie n c ie s , and th e re c o v e ry o f in ap p ro p riate p ay m en ts w ith in its M ed icaid program .

T h e M ich ig an D H S d ata w a re h o u s e (D W ) p ro ­ vid es u n iq u e an d in n ov ativ e in form ation critical to th e e ffic ie n t o p e ra tio n o f th e ag e n cy fro m b o th a strateg ic a n d tactical le v e l. O v e r th e last 10 years, th e D W h a s y ield ed a 15:1 c o s t-e ffe c tiv e n e s s ratio. C o n so lid ate d in fo rm atio n fro m th e D W n o w c o n ­ tributes to n e arly e v e ry fu n ctio n o f D H S, inclu d ing

( C o n tin u ed )

1 4 6 Part II • D escriptive Analytics

Application Case 3.6 (Continued) a ccu ra te d eliv ery o f a n d a cco u n tin g fo r b en e fits d eliv e red to a lm o st 2.5 m illion D H S p u b lic assis­ ta n c e clients.

M ich igan h a s b e e n am bitiou s in its attem pts to so lv e real-life p ro b lem s through th e innovative shar­ ing and c o m p re h e n siv e an aly ses o f data. Its ap p ro a ch to B I/D W has alw ays b e e n “e n te rp rise ” (state w id e) in nature, rath er th an h av in g sep arate BI/D W platform s fo r e a c h b u sin e ss a re a o r state ag e n cy . B y rem o v ­ ing barriers to sharing en terp rise data acro ss b u sin ess units, M ich ig an h a s lev erag e d m assiv e am o u nts o f data to c re a te innov ativ e ap p ro a ch e s to th e u se o f BI/DW , d eliv erin g efficien t, reliab le e n terp rise so lu ­ tio n s using m u ltip le ch an n els.

1. W h y w o u ld a sta te invest in a large a n d e x p e n ­ siv e IT infrastru ctu re (s u c h as an ED W )?

2. W h a t are th e s iz e an d co m p le x ity o f E D W u sed b y state a g e n c ie s in M ichigan?

3. W h a t w e re th e c h a lle n g e s, th e p ro p o se d s o lu ­ tio n , and th e o b ta in e d results o f th e EDW?

Source: Compiled from TDWI Best Practices Awards 2012 Winner, Enterprise Data Warehousing, Government and Non-Profit: Category, “Michigan Departments of Technology, Management & Budget (DTMB), Community Health (DCH), and Human Services (DHS),’’ featured in TDWI What Works, Vol. 34, p. 22; and m ichigan.m ichigan.gov.

Q u e s t i o n s f o r D i s c u s s i o n

M a s s iv e D a ta W a re h o u s e s a n d S c a la b ilit y

In ad dition to flexibility , a d ata w a re h o u s e n e e d s to su p p o rt scalability. T h e m ain issues p ertain in g to scalab ility are th e am o u n t o f data in th e w a re h o u s e , h o w q u ick ly th e w are­ h o u s e is e x p e c te d to gro w , th e n u m b e r o f co n c u rre n t u sers, a n d th e co m p le x ity o f user q u eries. A d ata w a re h o u s e m ust s c a le b o th h o rizo n ta lly a n d vertically. T h e w a re h o u s e will gro w a s a fu n ctio n o f d ata gro w th and th e n e e d to e x p a n d th e w a r e h o u s e to su p p o rt n e w b u s in e s s fu nctionality. D ata g ro w th m ay b e a result o f th e ad d ition o f cu rren t c y c le data (e .g ., this m o n th ’s resu lts) and/or h istorical data.

H icks (2 0 0 1 ) d escrib e d h u g e d a tab ase s a n d d ata w a reh o u se s. W alm art is continu ally in cre asin g th e siz e o f its m assiv e d ata w a re h o u se . W alm art is b e lie v e d to u s e a w a reh o u se w ith hu n d red s o f terab y tes o f data to study s a le s tre n d s, track in v e n to iy , and p erfo rm o th e r tasks. IB M re ce n tly p u b liciz e d its 50 -te ra b y te w a r e h o u s e b e n ch m a rk (IB M , 2 0 0 9 ). T h e U .S. D ep artm e n t o f D e fe n se is u sin g a 5 -p e ta b y te data w a re h o u s e an d re p o sito ry to h o ld m ed ical re co rd s fo r 9 m illio n m ilitary p e rso n n e l. B e c a u s e o f th e sto rag e req u ired to arch iv e its n e w s fo o ta g e , CNN a lso h a s a p e ta b y te -siz e d data w a reh o u se .

G iv e n th at th e siz e o f data w a re h o u s e s is e x p a n d in g at a n e x p o n e n tia l rate, sca la b il­ ity is a n im portant issu e. G o o d scalab ility m e a n s th a t q u e rie s an d o th e r d a ta -a cce ss fu nc­ tio n s w ill g ro w (id eally ) linearly w ith th e siz e o f th e w a re h o u se . S e e R o s e n b e rg (2 0 0 6 ) for a p p ro a ch e s to im p ro ve q u e ry p e rfo rm a n ce . In p ra ctice , s p e cia liz e d m e th o d s h a v e b e e n d ev elo p e d to c re a te s c a la b le d ata w a re h o u s e s . Scalab ility is d ifficult w h e n m an ag in g hun­ d reds o f terab y tes o r m o re. T e ra b y tes o f data h av e c o n s id e ra b le inertia, o c c u p y a lo t o f ph ysical s p a c e , and re q u ire p o w erfu l co m p u ters. S o m e firm s u s e p arallel p ro cessin g , and oth ers u se c le v e r in d e x in g a n d s e a r c h s c h e m e s to m a n a g e th e ir data. S o m e sp read their data acro ss d ifferen t p h y sical d ata stores. As m o re d ata w a re h o u s e s a p p ro a ch th e p etab y te size, b e tte r an d b e tte r so lu tio n s to scalab ility co n tin u e to b e d ev elo p e d .

H all (2 0 0 2 ) a lso ad d resse d scalab ility issu es. A T & T is a n industry le a d e r in d ep lo y ­ ing an d u sin g m assiv e d ata w a reh o u se s. W ith its 26 -te ra b y te d ata w a re h o u se , AT& T ca n d e te c t frau d ulen t u s e o f callin g card s an d in v estig ate calls re la ted to k id n ap p in g s and o th er crim es. It c a n a lso c o m p u te m illion s o f call-in v o te s fro m te le v isio n v ie w e rs s e le c t­ ing th e n e x t A m erican Idol.

Chapter 3 * Data W arehousing 1 4 7

F o r a sa m p le o f s u c c e s s fu l d ata w a re h o u s in g im p lem e n ta tio n s, s e e E d w ard s (2 0 0 3 ). J u k ic an d L ang ( 2 0 0 4 ) e x a m in e d th e tren d s a n d s p e c ific issu es related to th e u se o f o ff­ sh o re re s o u rc e s in th e d e v e lo p m e n t an d su p p o rt o f data w a re h o u s in g a n d B I ap p lica ­ tio n s. D av iso n (2 0 0 3 ) in d ica te d that IT -re lated o ffs h o re o u tso u rcin g h a d b e e n g ro w in g at 2 0 to 25 p e r c e n t p e r y e a r. W h e n c o n s id e rin g o ffsh o rin g data w a r e h o u s in g p ro je cts, carefu l c o n s id e ra tio n m ust b e g iv e n to cu ltu re and secu rity (fo r d etails, s e e J u k ic an d Lang, 2 0 0 4 ).

SECTION 3 .7 REVIEW QUESTIONS

X. W h at are th e m a jo r D W im p lem e n tatio n ta sk s th at c a n b e p erfo rm ed in parallel?

2 . List a n d d iscu ss th e m o st p ro n o u n c e d D W im p lem e n tatio n g u id elines.

3 . W h e n d e v e lo p in g a su cce ssfu l data w a re h o u s e , w h a t are th e m o st im p ortan t risks and issu es to c o n s id e r and p o ten tially avoid?

4 . W h a t is scalability? Howr d o e s it ap p ly to DW?

3.8 REAL-TIM E D A TA W AREHOUSING D ata w a re h o u sin g an d B I to o ls trad itionally fo cu s o n assisting m an ag ers in m ak in g stra­ te g ic an d tactical d e cisio n s. In c re a s e d d ata v o lu m e s a n d acce le ra tin g u p d a te s p e e d s are fu n d am en tally c h a n g in g th e ro le o f th e data w a re h o u s e in m o d e m b u sin ess. F o r m any b u sin esses, m ak in g fast and c o n s is te n t d ecisio n s a cro ss th e e n te rp rise re q u ire s m o re than a traditional d ata w a r e h o u s e o r data mart. T rad itional d ata w a re h o u s e s a re n o t b u si­ n ess critical. D ata a re c o m m o n ly up dated o n a w e e k ly b asis, a n d this d o e s n o t allow fo r re sp o n d in g to tra n sactio n s in n e a r-re a l-tim e .

M ore d ata, co m in g in faste r an d requ irin g im m ed iate co n v e rs io n in to d ecisio n s, m ean s that o rg an izatio n s a re co n fro n tin g th e n e e d fo r real-tim e d ata w a reh o u sin g . This is b e c a u s e d e c is io n su p p o rt h a s b e c o m e o p eratio n al, integrated B I req u ires clo s e d -lo o p analytics, a n d y e sterd a y ’s O D S w ill n o t su p p ort existin g requ irem en ts.

In 2 0 0 3 , w ith th e ad v en t o f real-tim e data w areh o u sin g , th e re w a s a shift tow ard u sin g th e se te c h n o lo g ie s fo r o p eratio n al d ecisio n s. R eal-tim e d a ta w a re h o u s in g (RD W ), a ls o k n o w n as a ctiv e d a ta w a re h o u s in g (ADW ), is th e p ro c e s s o f load in g a n d p ro vid ing d ata via the data w a re h o u s e as th ey b e c o m e available. It e v o lv ed fro m th e E D W c o n c e p t. T h e activ e traits o f a n RDW/ADW s u p p le m e n t an d e x p a n d traditional data w a re h o u s e fu n ctio n s in to th e re alm o f tactical d e c is io n m aking. P e o p le th rou gh o u t th e org an izatio n w h o in teract d irectly w ith cu sto m ers and su p p liers will b e e m p o w ered w ith in fo rm a tio n -b a sed d e c is io n m ak in g at th eir fingertips. E v e n fu rth er le v e ra g e results w h e n a n A D W p ro v id e s in fo rm atio n d irectly to cu sto m e rs a n d su p p liers. T h e re a c h and im pact o f in fo rm atio n a c c e s s fo r d e c is io n m ak in g c a n p o sitiv e ly a ffe c t alm o st all asp ects o f c u sto m e r se rv ice , SCM, lo gistics, a n d b ey o n d . E -b u sin ess h a s b e c o m e a m a jo r catalyst in th e d em an d fo r activ e data w a reh o u sin g ( s e e A rm strong, 2 0 0 0 ). F o r e x a m p le , o n lin e retailer O v e rs to c k .c o m , In c. (o v e r s to c k .c o m ) c o n n e c te d data u sers to a real-tim e data w a reh o u se . At E g g p ic, th e w o rld ’s larg est p u rely o n lin e b a n k , a cu sto m e r d ata w a reh o u se is re fre sh e d in n e a r-re a l-tim e . S e e A p p lication C ase 3.7.

As b u s in e s s n e e d s e v o lv e , so d o th e re q u irem en ts o f th e data w a re h o u s e . At this b a sic lev el, a d ata w a re h o u s e sim p ly reports w h at h a p p e n e d . At th e n e x t lev el, so m e analysis o ccu rs. As th e sy stem e v o lv e s, it p ro v id es p re d ictio n cap ab ilitie s, w h ich lead to th e n e x t le v e l o f op eratio n alizatio n . At its h ig h est e v o lu tio n , th e A D W is c a p a b le o f = a k in g e v en ts h a p p e n (e .g ., activities s u ch as cre a tin g sale s a n d m ark etin g ca m p a ig n s o r identifying a n d e x p lo itin g o p p o rtu n itie s). S e e Figu re 3 .1 2 fo r a g rap h ic d escrip tio n o f this ■rolutionary p ro c e s s . A re c e n t survey o n m an ag in g ev o lu tio n o f d ata w a re h o u s e s c a n b e

fro n d in W re m b e l (2 0 0 9 ).

1 4 8 Part II • D escriptive Analytics

Application Case 3.7 Egg Pic Fries th e C om petition in N ea r Real Time E g g p ic , n o w a p art o f Y o rk sh ire B u ild in g S o cie ty (e g g .c o m ) is th e w o rld ’s larg est o n lin e b a n k . It p ro ­ v id es b an k in g , in s u ra n ce , in v estm en ts, an d m o rt­ g a g e s to m o re th a n 3-6 m illio n cu sto m e rs th ro u g h its In te rn e t site. In 1 9 9 8 , E gg s e le c te d Sun M icrosystem s to c re a te a re lia b le , s c a la b le , s e c u re infrastru ctu re to su p p o rt its m o re th a n 2.5 m illion d aily tran saction s. In 2 0 0 1 , th e s y s te m w a s u p g rad ed to elim in ate la te n c y p ro b le m s . T h is n e w cu sto m e r d ata w a r e ­ h o u s e (C D W ) u s e d Su n , O ra c le , a n d SAS so ftw are p ro d u cts. T h e in itial d ata w a r e h o u s e h a d a b o u t 10 te ra b y te s o f d ata a n d u se d a 16-C PU server. T h e sy s­ tem p ro v id e s n e a r -r e a l-tim e d ata a c c e s s . It p ro v id es d ata w a r e h o u s e a n d d ata m in in g s e rv ice s to in ter­ n a l u s e rs , an d it p ro v id e s a re q u isite s e t o f cu s­ to m e r d ata to th e cu sto m e rs th e m se lv es. H undreds o f s a le s an d m ark e tin g ca m p a ig n s a re co n stru cte d

u sin g n e a r -r e a l-tim e data (w ith in sev eral m in u tes). A n d b e tte r, th e s y s te m e n a b le s fa ste r d e c is io n m ak ­ ing a b o u t s p e c ific c u sto m e rs a n d cu sto m e r classes.

Q u e s t i o n s f o r D i s c u s s i o n

1. W h y kind o f b u s in e s s is E gg p ic in? W h a t is th e co m p etitiv e lan d scap e?

2. H o w d id E gg p ic u s e n e a r-re a l-tim e data w are­ h o u sin g fo r co m p etitiv e advantage?

Sources: Compiled from “Egg's Customer Data Warehouse Hits the Mark," DM Review, Vol. 15, No. 10, October 2005, pp. 24-28; Sun Microsystems, ‘‘Egg Banks on Sun to Hit the Mark with Customers,” September 19, 2005, sun.com/smi/Press/sunflash/2005-09/ s u n fla s h .2 0 0 5 0 9 1 9 .1 .xm l (accessed April 2006); and ZD Net UK, "Sun Case Study: Egg’s Customer Data Warehouse," w hitepapers. z d n e t.c o .u k / 0 ,3 9 0 2 5 9 4 5 , 6 0 1 5 9 4 0 1 p - 3 9 0 0 0 4 4 9 q ,0 0 .h t m (accessed June 2009).

a, Es Q u£ra T3f5O

Real-Time Decisioning Applications

OPERATIONALIZING W H A T IS

happening now?

Enterprise Decisioning Management

ACTIVATING MAKE it happen!

Continuous Update and Time- Sensitive Queries

Become Important

Primarily Batch and

Som e Ad Hoc Reports

Event-Based Triggering Takes Hold

H Batch ES Ad Hoc H Analytics □ Continuous Update/Short Queries ■ Event-Based Triggering

D a ta S o p h istic a tio n

FIGURE 3 .1 2 E n te rp rise D ecisio n Ev o lu tio n . Source: Courtesy o f Teradata Corporation. Used with permission.

Chapter 3 • Data W arehousing 1 4 9

Actiue A c c e s s Front-Line operational decisions or services supported by N R T access; Service Level Agreements of 5 seconds or less

A ctiue Load Intra-day data acquisition; Mini-batch to near-real-time (NRT) trickle data feeds measured in minutes or seconds

Actiue Eve nts Proactive monitoring of business activity initiating intelligent actions based on rules and context; to system s or users supporting an operational business process

Actiue Workload M anagement Dynamically manage system resources for optimum performance and resource utilization supporting a mixed-workload environment

Actiue En te rp rise Integration Integration into the Enterprise Architecture for delivery of intelligent decisioning services

Actiue Auailability Business Continuity to support the requirements of the business (up to 7 x 2 4 ]Many

Product M arketina

FIGURE 3.13 The Teradata Active EDW. Source: Courtesy of Teradata Corporation. Used with permission.

T erad ata C o rp o ratio n p ro v id es th e b a s e lin e re q u irem en ts to su p p ort a n EDW'. It also p rovides th e n e w traits o f activ e d ata w a reh o u sin g req u ired to d eliv e r data fre sh n e ss, p e r­ fo rm an ce, a n d availability an d to e n a b le e n te rp rise d e c is io n m a n a g e m e n t (s e e Figu re 3 .1 3 fo r a n e x a m p le ).

A n A D W o ffers a n integrated inform ation re p o sito ry to drive strateg ic and tactical d ecisio n su p p o rt w ithin a n organization. W ith real-tim e data w a reh o u sin g , in stead o f extractin g o p e ra tio n a l d ata fro m a n OLTP sy stem in nightly b a tc h e s in to a n O D S, data are a sse m b le d fro m OLTP sy stem s as and w h e n e v e n ts h a p p e n and are m o v e d at o n c e into th e data w a re h o u se . T h is p erm its th e in stan t u p d atin g o f th e data w a r e h o u s e and th e elim in atio n o f a n O D S. At this p o in t, tactical an d strateg ic q u e ries c a n b e m a d e against th e RDW to u s e im m e d iate as w e ll as h istorical data.

A cco rd in g to B a su ( 2 0 0 3 ) , th e m o st d istinctive d iffe re n ce b e tw e e n a trad itional data w areh o u se a n d an R D W is th e shift in th e d ata acq u isitio n paradigm . S o m e o f th e b u si­ ness ca s e s a n d e n te rp rise re q u irem en ts that led to th e n e e d fo r d ata in re a l tim e in clu d e die fo llow ing:

• A b u sin e ss o fte n ca n n o t afford to w ait a w h o le d ay fo r its o p era tio n a l data to load into th e d ata w a r e h o u s e fo r analysis.

• Until n o w , data w a re h o u s e s h a v e ca p tu re d sn a p sh o ts o f an o rg a n iz a tio n ’s fixed states in s te a d o f in crem en tal real-tim e d ata sh o w in g e v ery state c h a n g e an d alm ost a n a lo g o u s p attern s o v e r tim e.

• W ith a traditional h u b -a n d -sp o k e arch itectu re, k e e p in g th e m etad ata in sy n c is dif­ ficult. It is a lso co stly to d ev elo p , m aintain, an d s e c u re m an y sy stem s as o p p o s e d to o n e h u g e d ata w a re h o u s e s o th at data are cen tralized fo r BI/BA to ols.

• In c a s e s o f h u g e nightly b a tc h lo ad s, th e n e ce s s a ry ETL setu p and p ro c e s s in g p o w e r fo r larg e n igh tly d ata w a r e h o u s e lo ad in g m ight b e very h igh , an d th e p ro c e s s e s m ight ta k e to o lon g. An EAI w ith real-tim e d ata c o lle c tio n c a n re d u c e o r elim inate the n igh tly b a tch p ro ce sse s.

1 5 0 Part II • D escriptive Analytics

D e sp ite th e b e n e fits o f a n RD W , d e v e lo p in g o n e c a n cre a te its o w n s e t o f issues. T h e s e p ro b lem s relate to arch ite ctu re, d ata m o d elin g , p h y sica l d atab ase d esign , storag e an d scalab ility, a n d m aintainability. In ad dition, d e p e n d in g o n e x a c tly w h e n d ata are a c c e ss e d , e v e n d o w n to th e m icro se co n d , d ifferen t v e rsio n s o f th e truth m ay b e e xtracted an d cre ate d , w h ic h c a n co n fu s e te a m m e m b ers. F o r d etails, re fe r to B asu ( 2 0 0 3 ) and T err (2 0 0 4 ).

R eal-tim e so lu tio n s p re s e n t a re m ark ab le s e t o f c h a lle n g e s to B I activities. A lthough it is n o t id eal fo r all solu tion s, real-tim e data w a reh o u sin g m ay b e su cce ssfu l i f th e organ i­ z a tio n d e v e lo p s a so u n d m e th o d o lo g y to h a n d le p ro je c t risks, in co rp o ra te p ro p e r plan­ nin g , an d fo cu s o n quality assu ran ce activities. U n d erstan d in g th e co m m o n ch a lle n g e s a n d ap p ly in g b e s t p ra ctice s c a n re d u c e th e e x te n t o f th e p ro b le m s th at are o fte n a part o f im p lem en tin g c o m p le x d ata w areh o u sin g system s th a t in co rp o ra te BI/BA m ethods. D etails a n d real im p lem e n tatio n s are d iscu sse d b y B u rd e tt an d Sin gh ( 2 0 0 4 ) a n d W ilk (2 0 0 3 ). A lso s e e A k b ay ( 2 0 0 6 ) an d E ricso n (2 0 0 6 ).

S e e T e c h n o lo g y Insigh ts 3-3 fo r s o m e d etails o n h o w th e real-tim e c o n c e p t evolved . T h e flig h t m an a g e m e n t d ash b o ard a p p lica tio n a t C o n tin en tal A irlines (s e e th e E nd -of- C h ap ter A p p lication C a se ) illustrates th e p o w e r o f re al-tim e B I in a c c e ss in g a d ata w are­ h o u s e fo r u se in fa c e -to -fa c e cu sto m e r in te ractio n situ ation s. T h e o p era tio n s s ta ff u s e s the real-tim e sy stem to identify issu es in th e C o n tin en tal flight n etw o rk . As an o th e r exam p le. UPS in v e sted $ 6 0 0 m illion s o it cou ld u se real-tim e data a n d p ro ce s s e s. T h e investm ent w a s e x p e c te d to cu t 1 0 0 m illio n d elivery m iles and sav e 14 m illion g allo n s o f fu el an n u ­ ally b y m an ag in g its real-tim e p a c k a g e -flo w te c h n o lo g ie s (s e e M alykhina, 2 0 0 3 ). T a b le 3 .6 c o m p a re s trad itional a n d a ctiv e d ata w a reh o u sin g en v iro n m en ts.

R eal-tim e d a t a w arehousing, n ea r-r e a l-tim e d a ta w arehou sin g, z ero -la ten cy w are­ housing, a n d activ e d a t a w areh ou sin g are d iffe ren t n a m e s u s e d in p ractice to d escrib e th e sa m e c o n c e p t. G o n z a le s ( 2 0 0 5 ) p re sen ted d ifferen t d efin itio n s fo r A D W . A cco rd in g to G o n z a le s, AD W is o n ly o n e o p tio n th a t p ro v id es b le n d e d tactical a n d strateg ic d ata o n d em and. T h e arch ite ctu re to b u ild an A D W is v e ry sim ilar to th e co rp o ra te inform ation facto ry arch ite ctu re d e v e lo p e d b y B ill In m o n . T h e o n ly d iffe re n c e b e tw e e n a co rp o rate in form ation facto ry an d a n A D W is th e im p lem e n tatio n o f b o th data sto res in a single

T E C H N O L O G Y IN SIG H T S 3 . 3 T h e R e a l-T im e R e a litie s o f A ctiv e D a ta W a r e h o u s in g

By 2003, the role o f data warehousing in practice was growing rapidly. Real-time systems, though a novelty, were the latest buzz, along with the major complications o f providing data and infor­ mation instantaneously to those who need them. Many experts, including Peter Coffee, eW eek's technology editor, believe that real-time systems must feed a real-time decision-making process. Stephen Brobst, CTO o f the Teradata division o f NCR, indicated that active data warehousing is a process o f evolution in how an enterprise uses data. A ctiv e means that the data warehouse is also used as an operational and tactical tool. Brobst provided a five-stage model that fits Coffee’s experience (2003) o f how organizations “grow” in their data utilization (see Brobst et al., 2005). These stages (and the questions they purport to answer) are reporting (What happened?), analysis (Why did it happen?), prediction (What will happen?), operationalizing (What is happening?), and active warehousing (What do I want to happen?). The last stage, active warehousing, is where the greatest benefits may be obtained. Many organizations are enhancing centralized data warehouses to serve both operational and strategic decision making.

Sources.- Adapted from P. Coffee, “'Active’ Warehousing,” eWeek, Vol. 20, No. 25, June 23, 2003, p. 36; and Teradata Corp., “Active Data Warehousing,” t e r a d a t a .c o m / a c t i v e - d a t a - w a r e h o u s i n g / (accessed August 2013).

Chapter 3 • Data Warehousing 151

T A B L E 3.6 Comparison Be tw ee n Traditional and A ctive Data W arehousing Environm ents

Traditional Data W a re h o u s e Environm ent A ctive Data W areh o u se Environm ent

Strategic decisions only Results sometimes hard to measure Daily, weekly, monthly data currency

Strategic and tactical decisions Results measured with operations

acceptable; summaries often appropriate Moderate user concurrency

Only comprehensive detailed data available within minutes is acceptable

High number (1,000 or more) of users accessing and querying the system simultaneously

Highly restrictive reporting used to confirm or check existing processes and patterns; often uses predeveloped summary tables or data marts

Flexible ad hoc reporting, as well as machine-assisted modeling (e.g., data mining) to discover new hypotheses and relationships

Power users, knowledge workers, internal users

Operational staffs, call centers, external users

Sources: Adapted from P. Coffee, “’Active' Warehousing,” eW eek, Vol. 20, No. 25, June 23, 2003, p. 36; and Teradata Corp., “Active Data Warehousing," t e r a d a t a . c o m / a c t i v e - d a t a - w a r e h o u s i n g / (accessed August 2013).

en viron m en t. H o w e v e r, a n SO A b a s e d o n XML a n d W e b s erv ices p ro v id e s a n o th e r o p tio n for b le n d in g ta ctica l an d strateg ic data o n d em and.

O n e critical issu e in real-tim e data w a reh o u sin g is th at n o t all data sh o u ld b e up dated con tin u ou sly. T h is m ay certain ly ca u se p ro b lem s w h e n r e p o n s are g e n e ra te d in real tim e, b e c a u s e o n e p e rs o n ’s results m ay n o t m atch a n o th e r p e rs o n ’s. F o r e x a m p le , a co m p a n y u sing B u s in e s s O b je c ts W e b In te llig e n ce n o tice d a sig n ifican t p ro b le m w ith real-tim e in tellig en ce. R e al-tim e rep o rts p ro d u ce d a t slightly d ifferen t tim es differ (s e e P ete rso n , 2 0 0 3 ). A lso, it m ay n o t b e n e ce s s a ry to up d ate certain d ata co n tin u o u sly ( e .g ., co u rse

g rad es th a t are 3 o r m o re y e ars old). R e al-tim e re q u irem en ts c h a n g e th e w a y w e v iew th e d esig n o f d atab ase s, d ata w a r e ­

h o u ses, OLAP, a n d data m in in g to o ls b e c a u s e th ey are literally u p d ated co n cu rre n tly w h ile q u e ries a re activ e. B u t th e substantial b u sin e ss v alu e in d o in g so h a s b e e n d e m o n ­ strated, s o it is c ru c ia l th a t o rg an izatio n s ad o p t th e s e m e th o d s in th e ir b u s in e s s p ro cesse s. C areful p lan n in g is critical in s u ch im p lem en tation s.

SECTION 3 . 8 REVIEW QUESTIONS

1 . W h at is a n RDW? 2 . List th e b e n e fits o f a n RDW. 3. W h at are th e m a jo r d iffe ren ce s b e tw e e n a trad itional data w a re h o u s e a n d a n RDW?

4. List s o m e o f th e drivers fo r RDW .

3.9 DATA WAREHOUSE ADMINISTRATION, SECURITY ISSUES, AND FUTURE TRENDS

D ata w a re h o u se s pro v id e a d istin ct co m p etitiv e e d g e to en te rp rises th at e ffe ctiv e ly c re ­ ate an d u se th e m . D u e to its h u g e siz e and its in trinsic natu re, a data w a r e h o u s e req u ires esp ecially stro n g m o n ito rin g in ord e r to sustain satisfactory e ffic ie n c y an d productivity. T h e su cce ssfu l ad m inistratio n and m an ag e m e n t o f a data w a reh o u se e n ta ils skills an d p ro ficie n cy th at g o p a st w h at is req u ired o f a trad itional d a ta b a se ad m in istrator (D B A ).

1 5 2 Part II • D escriptive Analytics

A data w areh ou se ad m in istrator (DWA) sh ou ld b e fam iliar w ith h ig h -p e rfo rm an ce so ftw are, h ard w are, an d n e tw o rk in g te ch n o lo g ie s. H e o r s h e sh o u ld a lso p o ss e s s solid b u sin ess insight. B e c a u s e data w a re h o u s e s fe e d B l sy stem s an d D SS that h e lp m an ag ­ e rs w ith th e ir d ecisio n -m a k in g activities, th e D W A sh o u ld b e fam iliar w ith th e d e cisio n ­ m ak in g p ro c e s s e s s o a s to su itab ly d esig n a n d m aintain th e data w a re h o u s e stru cture. It is particu larly sig n ifican t fo r a DW A to k e e p th e existin g re q u irem en ts a n d ca p a b ilitie s o f th e d ata w a r e h o u s e sta b le w h ile sim u ltan eou sly p ro vid ing flex ib ility fo r rapid im p ro vem en ts. Finally, a D W A m u st-p o sse ss e x c e lle n t co m m u n icatio n s sk ills. S e e B e n a n d e r e t al. (2 0 0 0 ) fo r a d e scrip tio n o f th e k e y d iffe re n ce s b e tw e e n a D BA and a DWA.

Security a n d privacy o f inform ation are m ain an d sign ifican t co n c e rn s fo r a data w are­ h o u s e p ro fession al. T h e U .S. go v ern m e n t h a s p assed regu lation s (e .g ., th e G ram m -L each- Bliley privacy an d safeguard s ru les, th e H ealth In su ran ce Portability and A cco u ntability A ct o f 1 9 9 6 [HIPAA]), instituting o b ligato ry req u irem en ts in th e m a n a g e m e n t o f cu stom er inform ation. H e n ce , co m p a n ie s m ust create secu rity p ro ce d u re s that a re e ffe ctiv e y e t flex ­ ib le to co n fo rm to nu m erou s privacy regulations. A cco rd in g to E lso n an d LeC lerc (2 0 0 5 ), e ffe ctiv e secu rity in a data w a reh o u se sh ou ld fo cu s o n fo u r m ain areas:

1 . E stablish in g e ffe ctiv e co rp o ra te an d secu rity p o lic ie s an d p ro ced u res. An e ffe ctiv e secu rity p o lic y sh ou ld start a t th e to p , w ith e x e c u tiv e m an ag e m e n t, a n d sh ou ld b e co m m u n ica te d to all individuals w ithin th e organization.

2 . Im p le m en tin g lo g ical secu rity p ro ced u res a n d te c h n iq u e s to restrict a c c e ss . This in clu d e s u s e r au th en ticatio n , a c c e s s co n tro ls, an d e n cry p tio n te ch n o lo g y .

3 . Lim iting p h ysical a c c e s s to th e d ata c e n te r e n v iron m en t. 4 . E stablish in g a n e ffe ctiv e in tern al c o n tro l review' p r o c e s s w ith an e m p h a sis o n secu rity

a n d privacy.

S e e T e c h n o lo g y Insigh ts 3 .4 fo r a d e scrip tio n o f A m b e o ’s im p ortan t so ftw are to o l th a t m o n ito rs secu rity an d p rivacy o f data w a re h o u s e s . Finally, k e e p in m ind th at a c c e s s ­ ing a data w a re h o u s e via a m o b ile d e v ice sh ou ld alw ays b e p erfo rm ed cau tiou sly. In this in sta n ce, d ata sh o u ld o n ly b e a c c e s s e d as read -on ly.

In th e n e a r term , d ata w a reh o u sin g d e v e lo p m e n ts will b e d eterm in ed b y n o tic e ­ a b le facto rs (e .g ., d ata v o lu m es, in cre a se d in to lera n ce fo r laten cy , th e diversity and c o m ­ p lex ity o f data ty p e s ) an d less n o tic e a b le facto rs (e .g ., u n m e t e n d -u s e r re q u irem en ts fo r

TECHNOLOGY IN SIG H TS 3 .4 Am beo D elivers Proven D ata-Access A uditing Solu tion

Since 1997, Ambeo (ambeo.com; now Embarcadero Technologies, Inc.) has deployed technol­ ogy that provides performance management, data usage tracking, data privacy auditing, and monitoring to Fortune 1000 companies. These firms have some o f the largest database environ­ ments in existence. Ambeo data-access auditing solutions play a major role in an enterprise information security infrastructure.

The Ambeo technology is a relatively easy solution that records everything that happens in the databases, with low or zero overhead. In addition, it provides data-access auditing that identifies exactly who is looking at data, when they are looking, and what they are doing with the data. This real-time monitoring helps quickly and effectively identify security breaches.

Sources: Adapted from “Ambeo Delivers Proven Data Access Auditing Solution,” D atabase Trends a n d Applications, Vol. 19, No. 7, July 2005; and Ambeo, “Keeping Data Private (and Knowing It): Moving Beyond Conventional Safeguards to Ensure Data Privacy," am-beo.com/why_am beo_white_papers.html (accessed May 2009).

Chapter 3 * Data W arehousing 1 5 3

L fcboard s, b a la n c e d s co re ca rd s, m aster data m a n a g e m e n t, in form ation q u ality ). G iv en Ifcse drivers, M o se le y ( 2 0 0 9 ) an d A gosta ( 2 0 0 6 ) su g g e ste d that d ata w a re h o u sin g trends

H Lean tow ard sim plicity, value, an d p e rfo rm a n ce .

Future o f Data Warehousing C field o f data w a reh o u sin g has b e e n a vibrant area in in form ation te c h n o lo g y in the

Ie co u p le o f d e c a d e s , an d th e e v id e n c e in th e BI/BA a n d B ig D ata w o rld sh o w s that im p ortan ce o f th e field w ill o n ly g e t e v e n m o re interesting . F o llo w in g are s o m e o f the

cmly p o p u larized c o n c e p ts a n d te c h n o lo g ie s that w ill p lay a sig n ifican t ro le in d efin in g k future o f data w areh o u sin g .

Sourcing (m e c h a n is m s fo r a c q u is itio n o f d a ta fro m d iv e r se a n d d is p e r s e d s o u r c e s ): • Web, s o c ia l m ed ia , a n d B ig D a ta . T h e re c e n t u p su rg e in th e u se o f th e W e b

fo r p e rs o n a l as w ell as b u sin e ss p u rp o se s c o u p le d w ith th e tre m e n d o u s in terest in ‘ so cial m e d ia cre a te s o p p o rtu n ities fo r analysts to tap into very rich d ata so u rce s.

B e c a u s e o f th e s h e e r v o lu m e, v e lo city , an d variety o f th e data, a n e w term , B ig D ata, h as b e e n c o in e d to n a m e th e p h e n o m e n o n . T a k in g ad v an tag e o f B ig D ata requ ires d ev elo p m en t o f n e w an d d ram atically im p ro ved BI/BA te c h n o lo g ie s, w h ic h w ill result in a re v o lu tio n ize d data w areh o u sin g w orld.

• Open s o u rc e softw are. U se o f o p en sou rce softw are tools is in creasin g at an u n p reced ented level in w arehousing, b u sin ess intelligence, an d data integration. T h ere are g o o d reaso n s for th e up sw ing o f o p e n so u rce softw are u sed in data w areho u s­ ing (R ussom , 2 009): (1 ) T h e recession has driven up interest in low -cost o p e n source software; (2 ) o p e n so u rce tools are com in g into a n e w level o f maturity, an d (3 ) o p e n source softw are augm ents traditional enterprise softw are w ithout rep lacin g it.

* S a a S (s o ftw a re a s a serv ice) , “T h e E x te n d e d ASP M o d e l.” SaaS is a cre ativ e w ay o f d e p lo y in g in fo rm a tio n sy stem a p p lic a tio n s w h e r e th e p ro v id e r lic e n s e s its a p p lic a tio n s to c u sto m e rs fo r u se as a s e r v ic e o n d e m a n d (u s u a lly o v e r th e In te rn e t). S a a S s o ftw a re v e n d o rs m ay h o s t th e a p p lic a tio n o n th e ir o w n serv ers

I o r u p lo a d th e a p p lic a tio n to th e c o n s u m e r site. In e s s e n c e , SaaS is th e n e w an d im p ro v ed v e r s io n o f th e ASP m o d e l. F o r d ata w a r e h o u s e cu sto m e rs , fin d in g SaaS- b a s e d s o ftw a re a p p lic a tio n s a n d r e s o u r c e s that m e e t s p e c ific n e e d s a n d re q u ire ­ m e n ts c a n b e c h a lle n g in g . As th e s e so ftw a re o ffe rin g s b e c o m e m o re a g ile , the a p p e a l a n d th e a ctu a l u se o f SaaS as th e c h o ic e o f data w a re h o u s in g p latfo rm w ill a ls o in c re a se .

■ C loud c o m p u tin g . C lou d co m p u tin g is p e rh a p s th e n e w e s t an d th e m o st in n o ­ vative p latfo rm c h o ic e to c o m e a lo n g in y ears. N um erous hardwra re a n d softw are

r' re so u rces a r e p o o le d and virtualized, s o that th e y c a n b e fre e ly a llo c a te d to ap p li­ cation s a n d so ftw are platform s as re s o u rc e s are n e ed e d . T h is e n a b le s inform ation fv stem a p p lica tio n s to d ynam ically s c a le up as w o rk lo ad s in cre a se . A lth o u gh clo u d com p u ting an d sim ilar virtu alization te ch n iq u e s are fairly w e ll e sta b lis h e d fo r o p e ra ­ tional a p p lica tio n s to d ay , th e y are ju st n o w starting to b e u se d as data w a re h o u s e platform s o f c h o ic e . T h e d ynam ic a llo ca tio n o f a clo u d is particu larly u se fu l w h e n th e d ata v o lu m e o f th e w a re h o u s e v arie s u n p red ictab ly , m ak in g ca p a city p lan n in g difficult.

Infrastructure (a r c h ite c tu ra l— h a rd w a re a n d s o ftw a re — e n h a n c e m e n ts ): * C o lu m n a r (a n ew w ay to sto re a n d a ccess d a ta in th e d a ta b a se). A co lu m n -

o rie n te d d a ta b a se m a n a g e m e n t sy stem (a ls o c o m m o n ly ca lle d a c o lu m n a r d a t a ­ b a se) is a sy stem that sto res data tab le s as se c tio n s o f co lu m n s o f data rath er than as ro w s o f data (w h ich is th e w a y m o st relatio n al d atab ase m a n a g e m e n t system s do it). T h a t is, th e s e co lu m n ar d a tab ase s sto re data by co lu m n s in stead o f row s

1 5 4 Part II • D escriptive Analytics

(a ll valu es o f a sin gle co lu m n are s to red co n s e c u tiv e ly o n d isk m e m o ry ). S u ch a structure gives a m u ch fin e r grain o f co n tro l to th e relatio n al d atab ase m a n a g e m e n t system . It c a n a c c e s s o n ly th e co lu m n s re q u ired fo r th e q u ery as o p p o s e d to b e in g fo rce d to a c c e s s all co lu m n s o f th e row . It p e rfo rm s sign ifican tly b e tte r fo r q u e ries that n e e d a sm all p e rce n ta g e o f th e co lu m n s in th e tab le s th ey are in b u t perform s sig n ifican tly w o rs e w h e n y o u n e e d m o st o f th e co lu m n s d u e to th e o v e rh e a d in attach in g all o f th e co lu m n s to g e th e r to fo rm th e result sets. C o m p arison s b e tw e e n ro w -o rie n te d an d co lu m n -o rie n te d d ata layouts are typ ically c o n c e rn e d w ith th e e ffic ie n c y o f hard -d isk a c c e s s fo r a g iv e n w o rk lo a d (w h ic h h a p p e n s to b e o n e o f th e m o st tim e -co n su m in g o p era tio n s in a co m p u te r). B a s e d o n th e task a t hand , o n e m ay b e sig nificantly ad v an tag e o u s o v e r th e oth er. C o lu m n -o rien ted o rg an iza­ tio n s are m o re e fficie n t w h e n (1 ) a n ag g reg ate n e e d s to b e co m p u te d o v e r m any ro w s b u t o n ly fo r a n o ta b ly sm aller s u b s e t o f all co lu m n s o f d ata, b e c a u s e read in g th at sm aller su b se t o f data c a n b e fa ste r th an re a d in g all data, and ( 2 ) n e w v a lu e s o f a co lu m n are su p p lied fo r all ro w s at o n c e , b e c a u s e th at co lu m n data c a n b e w ritten efficie n tly an d re p la ce old co lu m n data w ith o u t to u ch in g an y o th e r co lu m n s fo r the ro w s. R o w -o rien ted o rg an ization s a re m o re e ffic ie n t w h e n ( 1 ) m an y co lu m n s o f a sin gle ro w are re q u ired at th e sam e tim e, a n d w h e n ro w siz e is relativ ely sm all, as th e en tire ro w c a n b e retriev ed w ith a sin gle d isk s e e k , an d ( 2 ) w riting a n e w ro w if all o f th e co lu m n data is su p p lie d a t th e sa m e tim e , as th e en tire row c a n b e w ritten w ith a sin gle d isk s e e k . A dditionally, sin c e th e d ata s to red in a co lu m n is o f u n iform ty p e, it len d s itself b e tte r fo r co m p re ssio n . T h a t is, sig n ifican t sto rag e siz e op tim iza­ tio n is av ailab le in co lu m n -o rie n te d data that is n o t a v ailab le in ro w -o rie n te d data. S u c h o p tim al c o m p re ss io n o f d ata re d u ce s s to ra g e size, m ak in g it m o re e c o n o m i­ cally ju stifiab le to p u rsu e in -m em o ry o r so lid sta te sto rag e alternatives.

* R eal-tim e d a ta w a re h o u s in g . R e al-tim e d ata w areh o u sin g im p lies th at the re fre sh c y c le o f a n e x istin g d ata w a re h o u s e u p d ates th e data m o re fre q u e n tly (alm o st at th e sa m e tim e a s th e d ata b e c o m e s a v ailab le a t o p e ra tio n a l d a ta b a se s). T h e s e real-tim e data w a re h o u s e sy stem s c a n a ch ie v e n e a r-re a l-tim e u p d ate o f data, w h e re th e data la te n cy typ ically is in th e ran g e fro m m in u tes to hou rs. As th e laten cy gets sm aller, th e c o s t o f data up d ate s e e m s to in c re a se e x p o n e n tia lly . Fu tu re ad v an ce ­ m e n ts in m an y te ch n o lo g ica l fronts (ran g in g fro m au tom atic data acq u isitio n to intel­ lig en t softw are a g e n ts) are n e e d e d to m a k e re al-tim e d ata w a re h o u s in g a reality w ith a n a ffo rd ab le p rice tag.

• D a ta w a reh o u se a p p lia n c e s (all-in-one so lu tio n s to DW). A data w a reh o u se a p p lia n ce co n sists o f a n in teg rated s e t o f serv ers, sto rag e, o p eratin g sy stem (s), d ata­ b a s e m an a g e m e n t sy stem s, an d softw are sp e cifica lly p rein stalled an d p reop tim ized fo r d ata w areh o u sin g . In p ra ctice , data w a r e h o u s e a p p lia n ce s p ro v id e so lu tion s fo r th e m id -to -b ig d ata w a re h o u s e m ark et, o ffe rin g lo w -c o s t p e rfo rm a n ce o n data v o lu m e s in th e te ra b y te to p e ta b y te ran ge. In o rd e r to im p ro ve p e rfo rm a n ce , m ost d ata w a re h o u s e a p p lia n ce v e n d o rs u se m assiv ely p arallel p ro ce s s in g architectu res. E v e n th o u g h m o st d a ta b a se an d d ata w a r e h o u s e v e n d o rs pro v id e a p p lia n ce s n o w a­ days, m an y b e lie v e th a t T erad ata w as th e first to pro v id e a co m m e rcia l d ata w a re ­ h o u s e a p p lia n ce p ro d u ct. W h a t is o fte n o b s e rv e d n o w is th e e m e rg e n c e o f data w a re h o u s e b u n d le s, w h e re v e n d o rs co m b in e th e ir hard w are a n d d a ta b a se softw are as a d ata w a re h o u s e platform . From a b e n e fits stan d p o in t, d ata w a re h o u s e ap p li­ a n c e s h av e significantly lo w total c o s t o f o w n e rsh ip , w h ich in clu d es initial p u rch ase co sts, o n g o in g m a in te n a n ce c o s ts , and th e c o s t o f ch a n g in g ca p a city as th e data grow s. T h e re s o u rce c o s t fo r m o n ito rin g a n d tu n in g th e d ata w a re h o u s e m a k e s up a larg e p art o f th e to tal c o s t o f o w n ersh ip , o fte n as m u ch a s 8 0 p e rce n t. D W appli­ a n c e s re d u ce ad m inistration fo r d ay-to -d ay o p e ra tio n s, setu p , an d integration. Sin ce data w a re h o u s e a p p lia n ce s pro v id e a sin g le -v e n d o r solu tio n , th ey te n d to b etter

o p tim iz e th e h a r d w a r e a n d s o ftw a r e w ith in th e a p p lia n c e . S u c h a u n ifie d in te g r a tio n m a x im iz e s th e c h a n c e s o f s u c c e s s f u l in te g r a tio n a n d te s tin g o f th e D B M S s to r a g e a n d o p e r a tin g s y s te m b y a v o id in g s o m e o f th e c o m p a tib ility is s u e s th a t a r is e fr o m m u lti-v e n d o r s o lu tio n s . A d a ta w a r e h o u s e a p p li a n c e a ls o p r o v id e s a s in g le p o in t o f c o n t a c t fo r p r o b l e m r e s o lu tio n a n d a m u c h s im p le r u p g r a d e p a th f o r b o t h s o ftw a r e a n d h a r d w a r e .

• D a t a m a n a g e m e n t t e c h n o l o g i e s a n d p r a c t i c e s . S o m e o f t h e m o s t p r e s s in g n e e d s fo r a n e x t- g e n e r a t i o n d a ta w a r e h o u s e p la tfo r m in v o lv e t e c h n o lo g ie s a n d p r a c t i c e s th a t w e g e n e r a lly d o n ’t th in k o f a s p a rt o f th e p la tfo rm . I n p a rtic u la r, m a n y u s e r s n e e d to u p d a te th e d ata m a n a g e m e n t t o o ls th a t p r o c e s s d a ta f o r u s e th r o u g h d a ta w a r e h o u s in g . T h e fu tu re h o ld s s tr o n g g r o w th f o r m a s te r d a ta m a n ­ a g e m e n t (M D M ). T h is r e la tiv e ly n e w , b u t e x t r e m e ly im p o rta n t, c o n c e p t is g a in in g p o p u la r ity fo r m a n y r e a s o n s , in c lu d in g th e fo llo w in g : ( 1 ) T ig h te r in te g r a tio n w ith o p e r a tio n a l s y s te m s d e m a n d s M D M ; ( 2 ) m o s t d a ta w a r e h o u s e s still la c k M D M a n d d a ta q u a lity fu n c tio n s ; a n d ( 3 ) r e g u la to r y a n d f in a n c ia l r e p o r ts m u s t b e p e r f e c tly c le a n a n d a c c u r a te .

• I n - d a t a b a s e p r o c e s s i n g t e c h n o lo g y ( p u t t in g t h e a l g o r i t h m s w h e r e t h e d a t a is). I n - d a ta b a s e p r o c e s s i n g ( a ls o c a lle d i n - d a t a b a s e a n a ly t ic s ' ) r e fe r s to th e in te g r a tio n o f t h e a lg o r ith m ic e x t e n t o f d a ta a n a ly tic s in to d a ta w a r e h o u s e . B y d o in g s o , th e d a ta a n d th e a n a ly tic s th a t w o r k o f f th e d a ta liv e w ith in th e s a m e e n v i r o n ­ m e n t. H a v in g th e tw o in c lo s e p r o x im ity in c r e a s e s th e e f f ic i e n c y o f th e c o m p u t a ­ tio n a lly in te n s iv e a n a ly tic s p r o c e d u r e s . T o d a y , m a n y la r g e d a ta b a s e -d r iv e n d e c is io n s u p p o r t s y s te m s , s u c h a s t h o s e u s e d f o r c r e d it c a r d fra u d d e t e c t io n a n d in v e s tm e n t risk m a n a g e m e n t, u s e th is t e c h n o lo g y b e c a u s e it p r o v id e s s ig n ific a n t p e r f o r m a n c e im p r o v e m e n ts o v e r tr a d itio n a l m e th o d s in a d e c i s io n e n v ir o n m e n t w h e r e tim e is o f th e e s s e n c e . I n - d a ta b a s e p r o c e s s in g is a c o m p l e x e n d e a v o r c o m p a r e d to th e tr a d itio n a l w a y o f c o n d u c tin g a n a ly tic s , w h e r e th e d ata is m o v e d o u t o f th e d a ta ­ b a s e ( o f t e n in a flat file fo r m a t th a t c o n s is ts o f r o w s a n d c o lu m n s ) in to a s e p a ­ ra te a n a ly tic s e n v ir o n m e n t ( s u c h as SA S E n te r p r is e M o d e le r , S ta tis tic a D a ta M in e r, o r IB M S P S S M o d e le r ) f o r p r o c e s s in g . I n - d a t a b a s e p r o c e s s i n g m a k e s m o r e s e n s e fo r h ig h -th r o u g h p u t, r e a l-tim e a p p lic a tio n e n v ir o n m e n ts , in c lu d in g fra u d d e t e c ­ tio n , c r e d it s c o r in g , risk m a n a g e m e n t, tr a n s a c tio n p r o c e s s in g , p r ic in g a n d m a r g in a n a ly s is, u s a g e - b a s e d m ic r o -s e g m e n tin g , b e h a v io r a l a d ta rg e tin g , a n d r e c o m m e n d a ­ tio n e n g in e s , s u c h as t h o s e u s e d b y c u s t o m e r s e r v ic e o r g a n iz a tio n s to d e te r m in e n e x t- b e s t a c tio n s . I n - d a ta b a s e p r o c e s s in g is p e r fo r m e d a n d p r o m o t e d a s a fe a tu r e b y m a n y o f th e m a jo r d a ta w a r e h o u s i n g v e n d o r s , in c lu d in g T e r a d a ta (in te g r a tin g SAS a n a ly tic s c a p a b ilitie s in to th e d a ta w a r e h o u s e a p p li a n c e s ) , IB M N e te z z a , EM C G r e e n p lu m , a n d S y b a s e , a m o n g o th e r s .

• I n - m e m o r y s t o r a g e t e c h n o lo g y ( m o v i n g t h e d a t a i n t h e m e m o r y f o r f a s t e r p r o c e s s i n g ) . C o n v e n tio n a l d a ta b a s e s y s te m s , s u c h a s r e la tio n a l d a ta b a s e m a n ­ a g e m e n t s y s te m s , ty p ic a lly u s e p h y s ic a l h a r d d riv e s to s to r e d a ta f o r a n e x t e n d e d p e r io d o f tim e . W h e n a d a ta -r e la te d p r o c e s s is r e q u e s t e d b y a n a p p lic a tio n , th e d a ta b a s e m a n a g e m e n t s y s te m lo a d s th e d a ta ( o r p a rts o f th e d a ta ) in to th e m a in m e m o r y , p r o c e s s e s it, a n d r e s p o n d s b a c k to th e a p p lic a tio n . A lth o u g h d a ta ( o r p a rts o f th e d a ta ) is te m p o r a r ily c a c h e d in t h e m a in m e m o r y in a d a ta b a s e m a n a g e m e n t sy s te m , th e p rim a r y s to r a g e lo c a t i o n r e m a in s a m a g n e tic h a r d d is k . I n c o n tr a s t, a n in - m e m o r y d a ta b a s e s y s te m k e e p s th e d a ta p e r m a n e n tly in th e m a in m e m o iy . W h e n a d a ta -r e la te d p r o c e s s is r e q u e s t e d b y a n a p p lic a tio n , th e d a ta b a s e m a n a g e m e n t s y s te m d ir e c tly a c c e s s e s th e d a ta , w h ic h is a lr e a d y in th e m a in m e m o r y , p r o c e s s e s it, a n d r e s p o n d s b a c k to th e r e q u e s t i n g a p p lic a tio n . T h is d ir e c t a c c e s s to d a ta in m a in m e m o r y m a k e s th e p r o c e s s in g o f d a ta o r d e r s m u c h fa s te r th a n th e tr a d itio n a l m e th o d . T h e m a in b e n e f it o f in -m e m o r y t e c h n o l o g y ( m a y b e t h e o n ly b e n e f it o f it) is

Chapter 3 • Data W arehousing 1 5 5

1 5 6 Part II • D escriptive Analytics

th e in cre d ib le s p e e d at w h ic h it a c c e s s e s th e data. T h e d isad van tag es in clu d e c o s t o f p ay in g fo r a very larg e m ain m e m o ry (e v e n th o u g h it is g ettin g ch e a p e r, it still costs a g re a t d eal to hav e a large e n o u g h m ain m e m o ry th at ca n h o ld all o f com p an y 's d a ta ) and th e n e ed fo r so p h istica ted d ata re c o v e ry strategies (s in c e m ain m em ory is v o latile a n d ca n b e w ip e d o u t accid en tally ).

• New d a ta b a se m a n a g e m e n t systems. A data w areh o u se platform consists o f sev­ eral b asic com p on en ts, o f w h ich the m o st critical is th e d atabase m an ag em en t system (D BM S). T h is is only natural, given th e fact that D BM S is th e co m p o n e n t o f th e platform w h ere th e m ost w o rk m ust b e d o n e to im p lem ent a data m o d el and optim ize it for query perform ance. T h erefore, the D BM S is w h ere m an y n ext-gen eratio n innovations are e x p e c te d to happen.

• A d v a n c e d a n a ly tics. U sers c a n c h o o s e d iffe ren t an alytic m e th o d s as th e y m ove b e y o n d b a s ic O L A P -based m e th o d s a n d in to a d v a n ce d analy tics. S o m e u se rs c h o o s e ad v an ce d an alytic m e th o d s b a s e d o n d ata m in in g , p red ictiv e analytics, statistics, artificial in te llig e n ce , and s o on . Still, th e m ajority o f u se rs s e e m to b e ch o o s in g SQL- b a s e d m eth o d s. E ith er SQ L -based o r n o t, a d v a n ce d an alytics s e e m to b e am o n g the m o st im portant p ro m ise s o f n e x t-g e n e ra tio n d ata w areh o u sin g .

T h e future o f data w a reh o u sin g s e e m s to b e full o f p ro m ise s a n d significant ch a lle n g e s. As th e w o rld o f b u s in e s s b e c o m e s m o re g lo b a l an d c o m p le x , th e n e e d for b u sin e ss in te llig e n ce a n d d ata w a reh o u sin g to o ls w ill a ls o b e c o m e m o re p ro m in en t. T h e fast-im p roving in form ation te c h n o lo g y to o ls an d te c h n iq u e s s e e m to b e m o vin g in the right d irectio n to ad dress th e n e e d s o f future b u sin e ss in te llig e n ce system s.

SECTION 3 . 9 REVIEW QUESTIONS

1 . W h a t step s c a n a n o rg an izatio n ta k e to e n su re th e se cu rity an d con fid en tiality o f cu s­ to m e r data in its data w areh o u se?

2. W h a t sk ills sh o u ld a D W A p o ssess? Why? 3 . W h at re c e n t te c h n o lo g ie s m ay s h a p e th e future o f d ata w areho u sing? Why?

3.10 R ESO U R C ES, LIN K S, A N D THE T E R A D A T A U N IV E R S IT Y N E T W O R K CO NNECTION

T h e u s e o f this ch a p te r and m o st o th e r ch ap ters in this b o o k c a n b e e n h a n c e d b y th e tools d escrib e d in th e fo llo w in g sectio n s.

Resources and Links W e re c o m m e n d lo o k in g at th e fo llo w in g re so u rce s a n d lin ks fo r fu rth er re ad in g and exp lan atio n s:

• T h e D ata W a re h o u se Institute (tdw i.org) • DM Review (inform ation-m anagem ent.com ) • D SS R e so u rce s (d ssresou rces.co m )

Cases All m ajor MSS v end ors (e .g ., MicroStrategy, M icrosoft, O racle, IBM, H yperion, C ognos, Exsys, Fair Isaac, SAP, Inform ation B uilders) provide interesting cu sto m e r su ccess stories. A cadem ic- oriented ca ses are available at th e Harvard B u sin ess S ch o o l C ase C ollection (harvardbu sinessonline.hbsp.harvard.edu), B u sin ess P erform an ce Im provem ent R eso u rce (bpir. com), IG I G lob al D issem in ator o f K n o w led ge (igi-global.com), Ivy League Publishing (ivylp.com), ICFAI C enter fo r M anag em ent R esearch (icm r.icfai.org/casestudies/

Chapter 3 • D ata W arehou sing 1 5 7

icm r_ ca se _ stu d ie s.h tm ), K now led geStorm (k n o w le d g e sto rm .co m ), an d o th er sites. For additional c a s e resou rces, s e e Teradata University N etw ork (te ra d a ta u n iv e rsity n e tw o rk . co m ). F o r data w areho u sing cases, w e specifically recom m en d the follow ing fro m the Teradata University N etw ork (te ra d a ta u n iv e rsity n e tw o rk .co m ): “C ontinental Airlines Flies High w ith R eal-Tim e B u sin ess In tellig en ce,” “D ata W areh ou se G o v ern an ce at B lu e Cross and B lu e Shield o f North Carolina/’ “3M M oves to a Custom er F o cu s U sing a G lob al Data W areh ou se,” “D ata W arehou sing Supports C orporate Strategy at First A m erican C orporation,” “Harrah’s H igh P ay o ff fro m Custom er Inform ation,” an d “W hirlp ool.” W e a lso recom m end th e D ata W areh ou sin g Failures Assignm ent, w h ich consists o f eight short c a s e s o n data w arehousing failures.

Vendors, Products, and Demos A co m p re h e n siv e list o f v e n d o rs, p ro d u cts, an d d em o s is a v ailab le a t DM Review (d m re v ie w .co m ). V en d o rs are liste d in T a b le 3-2. A lso s e e te ch n o lo g y e v a lu a tio n .co m .

Periodicals W e r e c o m m e n d th e fo llo w in g p eriod icals:

• B a s e lin e (b a s e lin e m a g .c o m ) • B u sin ess In telligen ce J o u r n a l (td w i.o rg ) •C IO (c io .c o m ) • CIO Insight (cio in s ig h t.c o m ) • C om puterw orld (c o m p u te rw o rld .c o m ) • D ecision Support Systems (elsevier.com) • DM R eview (d m re v ie w .co m ) • eW eek (e w e e k .c o m ) • In fo W eek (in fo w e e k .co m ) • In foW orld (in fo w o rld .co m ) • In tern etW eek (in te m e tw e e k .c o m ) • M a n a g em en t In form ation Systems Q uarterly (MIS Q uarterly; m isq .o rg ) 3 Technology E valu ation (te c h n o lo g y e v a lu a tio n .c o m ) • T era d a ta M a g a z in e (te r a d a ta .c o m )

Additional References F o r ad d ition al in form ation o n data w areh o u sin g , s e e th e fo llow in g :

• C. Im h o ff, N. G a lem m o , and J . G . G eig er. (2 0 0 3 ). M astering D ata W arehou se Design: R ela tio n a l a n d D im en sio n a l T echniques. N ew Y o rk : W iley.

• D . M arco and M. Je n n in g s . (2 0 0 4 ). Universal M eta D a ta Models. N e w Y o rk : W iley. • J . W an g . (2 0 0 5 ) . E n cy clop ed ia o f D a ta W arehousing a n d M ining. H e rsh e y , PA: Idea

G ro u p P u b lish in g.

F o r m o re o n d a ta b a se s, th e stru cture o n w h ich d ata w a re h o u s e s a re d e v e lo p e d , s e e th e fo llo w in g :

• R. T . W a tso n . (2 0 0 6 ). D ata M anagem ent, 5th e d ., N ew Y o rk : W iley.

The Teradata University Network (TUN) Connection TUN (te ra d a ta u n iv e rs ity n e tw o rk .c o m ) pro v id es a w ealth o f in fo rm a tio n an d ca ses o n data w a reh o u sin g . O n e o f th e b e s t is th e C o n tin en tal A irlines c a s e , w h ic h w e req u ire y o u to so lv e in a la ter e x e rc ise . O th e r re co m m e n d e d c a s e s a re m e n tio n e d e a rlie r in this

1 5 8 Part II • D escriptive Analytics

ch ap ter. At TU N , i f y o u c lic k th e C o u rses tab an d s e le c t D ata W are h o u sin g , y o u w ill s e e lin k s to m a n y re le v a n t articles, assig n m en ts, b o o k ch a p te rs, c o u rs e W e b sites, P o w e rP o in t p re sen tatio n s, p ro je cts, re se a rch rep orts, syllabi, an d W e b sem in ars. Y o u w ill a lso find lin k s to a ctiv e data w a reh o u sin g so ftw are d em o n stratio n s. Finally, y o u w ill s e e lin ks to T erad ata (te r a d a ta .c o m ), w h e re y o u c a n fin d ad d itional in form ation , in clu d in g e x c e l­ le n t d ata w a reh o u sin g s u c c e s s stories, w h ite p ap ers, W e b -b a s e d c o u rs e s, an d th e o n lin e v e rs io n o f T era d a ta M agazin e.

Chapter Highlights

• A d ata w a r e h o u s e is a s p e cia lly co n stru cte d data re p o sito ry w h e r e d ata are o rg an ize d s o that th ey c a n b e e asily a c c e s s e d b y e n d u sers fo r sev eral ap p licatio n s.

• D ata marts co n ta in data on o n e to p ic (e .g ., m arket­ ing ). A data m art c a n b e a rep lication o f a su b set o f data in th e d ata w areh o u se . D ata marts are a less e x p e n siv e solu tion that c a n b e re p la ced b y o r c a n su p p lem en t a data w areh o u se. D ata marts c a n b e in d e p e n d en t o f o r d ep en d e n t o n a data w areh o u se.

• An O D S is a ty p e o f cu stom er-in fo rm ation -file d atab ase th at is o fte n u s e d as a staging a re a fo r a d ata w a reh o u se .

• D ata in te g ratio n co m p rise s th ree m a jo r p ro ­ c e ss e s : d ata a c c e s s , d ata fe d era tio n , an d ch an g e

cap tu re. W h e n th e s e th ree p ro c e s s e s a re co rrectly im p lem en ted , d ata c a n b e a c c e s s e d a n d m ad e a c c e s s ib le to a n array o f ETL an d analysis to o ls an d d ata w a re h o u sin g en viron m en ts.

• ETL te c h n o lo g ie s p u ll d ata fro m m an y so u rce s, c le a n s e th e m , an d lo ad th e m in to a d ata w a re ­ h o u se . ETL is a n integral p ro c e s s in a n y d ata- ce n tric p ro ject.

• R e al-tim e o r a ctiv e data w areh o u sin g su p p le ­ m en ts and e x p a n d s trad itional data w areh o u sin g , m o vin g in to th e re a lm o f o p era tio n a l an d tacti­ ca l d e c is io n m ak in g b y lo ad in g data in re al tim e an d p ro vid ing d ata to u sers fo r activ e d e cisio n m aking.

• T h e security an d privacy o f data an d inform ation are critical issues fo r a data w areh o u se professional.

Key Terms

a ctiv e data w a reh o u sin g (A D W )

c u b e d ata in tegratio n d ata mart d ata w a re h o u s e (D W ) d ata w a reh o u se

ad m inistrator (D W A )

d e p e n d e n t data mart d im en sio n al m o d elin g d im en sio n ta b le drill d ow n en terp rise a p p licatio n

in teg ratio n (EAI) e n te rp rise data

w a r e h o u s e (E D W )

e n te rp rise in form ation in tegratio n (E li)

e x tractio n , tran sform ation, an d lo a d (ETL)

in d e p e n d e n t d ata m art m etad ata OLTP

o p e r mart o p e ra tio n a l d ata sto re

(O D S ) real-tim e data

w a reh o u sin g (R D W ) sn o w fla k e sch e m a star s ch e m a

Questions for Discussion

1 . Identify and d escrib e th e core characteristics o f a DW. 2 . List and critically exam in e at least three potential sou rces

o f data that cou ld b e used w ithin a data w arehou se. 3 . W hat are th e d ifferen ces betw een d ep en d en t and ind e­

p en d en t data marts?

4 . W hat are th e structural differences betw een tw o-tier and three-tier architectures? W hich o n e o f the tw o m odels is m ore suitable for analytical operations perform ed o n a very large data set?

5 . W hat is the role o f ODS? H ow d oes it differ from an EDW?

C hapter 3 • Data W arehousing 1 5 9

6 . Highlight th e utility o f the three types o f m etadata em phasized in th e chapter.

7. In w h at circu m stan ces w ou ld a relational d atabase be m ore suitable than a m ultidim ensional database?

8 . D iscuss security con cern s involved in building a data w arehou se.

9 . Investigate curren t data w areh ou se d evelopm ent im ple­ m entation through offshoring. W rite a report about it. In class, d ebate th e issue in term s o f th e benefits and costs, as w ell a s social factors.

Exercises T e r a d a ta U n iv e rs ity a n d O th e r H a n d s -O n E x e r c i s e s

1 . Consider th e c a s e describing th e d evelopm ent and appli­ cation o f a data w areh ou se for C oca-C ola Ja p a n (a sum­ mary appears in Application Case 3-4), available at the DSS R esources W eb site, h t t p :/ / d s s r e s o u r c e s .c o m / c a s e s /c o c a - c o l a j a p a n /. Read the case and answ er the n ine question s for further analysis and discussion.

2 . R ead th e B a ll (2 0 0 5 ) article and rank-ord er th e criteria (ideally for a real organization). In a report, exp lain how important e a c h criterion is and why.

3 . Explain w h e n you should im plem ent a tw o- o r three­ tiered architecture w h e n con siderin g d eveloping a data w arehou se.

4 . Read th e full Continental Airlines c a se (sum m a­ rized in th e End-of-Chapter Application Case) at te ra d a ta u n iv e r s ity n e tw o r k .c o m and answ er the questions.

5 . At te ra d a ta u n iv e r s ity n e tw o r k .c o m , read and answ er the qu estion s to th e c a se “Harrah’s High P ayoff from C ustom er Inform ation.” Relate Harrah’s results to how airlines and o th er casinos u se their custom er data.

6 . At te ra d a ta u n iv e r s ity n e tw o r k .c o m . read and answ er th e qu estion s o f the assignm ent “D ata W arehousing Failures.” B e c a u s e eight cases are d escribed in that assignm ent, th e class m ay b e divided into eight groups, w ith o n e c a se assigned p er group. In addition, read Ariyachandra and W atson (2 0 0 6 a ), and for ea ch case identify h o w the failure occu rred as related to not focu s­ ing o n o n e o r m ore o f th e referen ce’s success factor(s).

7 . At te r a d a ta u n iv e r s ity n e tw o r k .c o m , read and answer th e qu estion s w ith the assignm ent “Ad-Vent Tech nolog y: U sing the MicroStrategy Sales Analytic M odel.” T h e M icroStrategy softw are is a ccessib le from the TUN site. Also, you m ight w ant to use Barbara W ixom ’s Pow erPoint presentation ab o u t the MicroStrategy softw are ( “D em o Slides for MicroStrategy Tutorial Script”), w hich is also available at th e TUN site.

8 . At te r a d a ta u n iv e r s ity n e tw o r k .c o m , w atch th e W eb sem inars titled “Real-Tim e Data W arehousing: T h e N ext G en eration o f D ecisio n Support D ata M anagem ent” and “Building th e Real-Tim e Enterprise.” Read the article “Terad ata’s R eal-Tim e Enterprise R eferen ce Architecture: A Blu eprint for th e Future o f IT .” also available at this site. D escrib e h o w real-tim e con cep ts and tech nolog ies

w ork and how they c a n b e u sed to ex ten d existing data w areh ou sing and B I architectures to support day-to-day d ecisio n m aking. W rite a report indicating h ow real-tim e data w areh ou sing is sp ecifically providing com petitive advantage for organizations. D escrib e in detail th e dif­ ficulties in su ch im plem entations and operations and d escrib e how th ey are b ein g addressed in practice.

9 . At te r a d a ta u n iv e r s ity n e tw o r k .c o m , watch th e W eb sem inars “Data Integration Renaissance: N ew Drivers and Em erging A pproaches,” “In Search o f a Single V ersion o f the Truth: Strategies for Consolidating Analytic Silos,” and “Data Integration: Using ETL, EAI, and E li T ools to Create an Integrated E nterprise.” Also read the “Data Integration’ research report. Com pare and contrast the presentations. W hat is the m ost important issue described in th ese sem i­ nars? W hat is th e best way to handle th e strategies and challenges o f consolidating data marts and spreadsheets into a unified data w arehousing architecture? Perform a W eb search to identify th e latest developm ents in the field. Com pare th e presentation to the material in the text and th e new material that you found.

1 0 . Consider the future o f data warehousing. Perform a W eb search o n this topic. Also, read these tw o articles: L. Agosta, “Data W arehousing in a Flat World: Trends for 2006,” DM D ir ec t N ew sletter, March 31, 2006; and J . G. Geiger, “CIFe: Evolving with th e Tim es,” D M R eview , N ovem ber 2005, pp. 3 8 -4 1 . Com pare and contrast your findings.

1 1 . A ccess te r a d a ta u n iv e r s ity n e tw o r k .c o m . Identify the latest articles, research reports, and cases o n data w are­ housing. D escrib e recen t developm ents in the field. Include in you r report h ow data w arehousing is used in B I and DSS.

Team Assignments and Role-Playing Projects 1 . Franklin Lyons has b e e n a Ju n ior Advisor with an IT con­

sultancy for alm ost tw o years. His com pany w on a bid and tasked him to advise XCOMP— a medium-sized fast grow­ ing call centre provider with an increasing international presence— on developing an EDW suitable for their analytical needs. T h e client is currently facing increasing con cern s over data transparency as a result o f th e expan­ sion in its divisional organizational structure and would like a DW to help the m anagem ent gain a holistic view o f the enterprise for decision m aking purposes. XCOMP

1 6 0 Part II • D escriptive Analytics

currently uses a CRM system to monitor custom er interac­ tion, Excel for operational reporting and an ERP system to manage the com pany’s HR and Finance requirements. After analyzing the circumstances, Franklin still has the following questions:

a . W hat type o f data w areh ou se architecture should be recom m en d ed b ased o n th e custom er’s n eed s and why?

b. W hat potential ch allen g es should th e..com p an y be aw are of?

c . Should the com p an y con sid er using a h osted data w arehouse? W hat are th e potential im plications o f such an approach?

H elp Franklin answ er th ese questions. 2 . Ju lia G onzales has recently b e e n em ployed as a data

w areh ou se m anager, o r DWM, within P op log - a large logistics com pany. As part o f h er n ew responsibilities, she has b e e n asked to review th e current D W solu­ tion and identify areas o f potential im provem ent, sin ce the CEO is n o t satisfied w ith th e quality o f the insights d erived from their existing system s, as w ell as th e long lead tim e a sso ciated with their data w arehousing process. Ju lia has identified that the com p an y utilizes a tw o-tiered data w areh ou sing with lim ited m etadata and a variety o f data sou rces b ein g used, dep en ding o n w h ich bran ch o f th e com p an y th ey originate from. B efo re b ein g ab le to subm it a report with her findings and suggestions, Ju lia w ould still like to find out:

a . W hat elem en ts o f the current system are likely to slow th e p ro cess down? Why?

b. How cou ld the d ecision support functionality o f the data w areh ou sing b e improved?

c . W hat factors w ould sh e have to take into con sid era­ tion b efore suggesting the im plem entation o f a new data w areh ou sin g design?

3 . D oug O ’B rien is a DWA for P h on e Sales - a large m obile d evice retailer. T h e com pany has b e e n effectively using their data w areh ou sing for over 6 years; how ever, the board o f directors has d ecided that th e tech nolog y might n e e d a n upgrade to ensure its sustainability. D oug has b een ask ed to w rite a report identifying current data w areh ou sing trends and relevant tech n o log ies o n the m arket and m ak e suggestions for their potential use w ithin P h on e Sales. Although h e is partially familiar with current grow ing tech n olog ies, D oug still has th e follow ­ ing questions:

a . W hich tech n o log ies influ ence th e d evelopm ent o f data w areh ou sing as a business area?

b. W hat are th e effects o f th ese technologies? c. How d o th ese tech n o log ies affect data w arehousing

security? 4 . G o through th e list o f data w areh ou sing security m eas­

ures su gg ested in th e chapter and discuss the potential

n egative effects o f their neg lect. Identify exam p les o f data security failure in recen t events.

5 . A ccess teradata.com and read th e w hite papers “M easuring Data W arehou se ROI” and “Realizing ROI: Projecting and Harvesting th e B usiness Value o f an Enterprise D ata W arehou se.” Also, w atch the W eb-based cou rse “T h e ROI Factor: How Leading Practitioners D eal w ith th e Tough Issu e o f M easuring DW ROI.” D escrib e the m ost im portant issues described in them . Com pare th ese issues to the su ccess factors d escribed in Ariyachandra and Watson (2 0 0 6 a ).

6 . Read the article b y K. Liddell Aveiy and Hugh J. W atson, “Training Data W arehouse End Users,’" B u sin ess In te llig en c e J o u r n a l , Vol. 9, No. 4, Fall 2004, pp. 4 0 -5 1 (w hich is available at teradatauniversitynetwork.com). Consider the different classes o f end users, describe their difficulties, and discuss the benefits o f appropriate train­ ing for each group. Have each m em ber o f the group take o n o n e o f the roles and have a discussion about how an appropriate type o f data warehousing training would be good for each o f you.

Internet Exercises 1 . Search the Internet to find information about data ware­

housing. Identify som e newsgroups that have an interest in this concept. Explore ABI/INFORM in your library, e-library, and G oogle for recent articles o n the topic. B eg in with tdwi.org, technologyevaluation.com, and the major vendors: teradata.com, sas.com, oracle.com, and ncr. com. Also ch eck cio.com, information-management. com, dssresources.com, and db2mag.com.

2 . Survey som e ETL to o ls and vendors. Start with fairisaac. com and egain.com. Also consu lt information- management. com .

3. C ontact som e data w areh ou se vendors and obtain infor­ m ation about th eir products. G ive special attention to vendors that p rovide tools for m ultiple purposes, su ch as Cognos, Softw are A&G, SAS Institute, and O racle. Free on lin e d em os are available from som e o f th ese vendors. D ow nload a d em o o r tw o and try them . Write a report d escribing you r exp erien ce.

4 . Explore teradata.com for d evelopm ents and success stories about data w arehousing. W rite a report about w hat you have discovered.

5. E xplore teradata.com for w hite papers and W eb- b ased cou rses o n data w arehousing. Read th e former and w atch the latter. (D ivide th e class so that all the sou rces are cov ered .) W rite w hat you have d iscovered in a report.

6 . Find recen t ca se s o f successful data w areh ou sing appli­ cations. G o to data w areh ou se vendors’ sites and look for cases o r su c ce ss stories. Select o n e and write a brief sum m ary to presen t to you r class.

Chapter 3 * D ata W arehousing 161

End-of-Chapter Application Case

Continental Airlines Flies High with Its Real-Time Data Warehouse As business intelligence (B I) becom es a critical com ponent o f daily operations, real-time data w arehouses that provide end users with rapid updates and alerts generated from transactional systems are increasingly being deployed. Real-time data w are­ housing and B I, supporting its aggressive G o Forward business plan, have helped Continental Airlines alter its industry status from “w orst to first” and then from “first to favorite.” Continental airlines (n ow a part o f United Airlines) is a leader in real-time DW and BI. In 2004, Continental w on the Data W arehousing Institute’s B est Practices and Leadership Award. Even though it has b e e n a w hile since Continental Airlines deployed its hugely successful real-time D W and B I infrastructure, it is still regarded as o n e o f the b est exam ples and a seminal success story for real-time active data warehousing.

P r o b l e m ( s ) Continental Airlines w as founded in 1934, with a single-engine Lockheed aircraft in the Southwestern United States. As o f 2006, Continental w as the fifth largest airline in th e United States and th e seventh largest in the world. Continental had the broadest global route netw ork o f any U.S. airline, with more than 2,300 daily departures to m ore than 2 27 destinations.

B a ck in 1994, Continental w as in deep financial trouble. It had filed for Chapter 11 bankruptcy protection tw ice and was heading for its third, and probably final, bankruptcy. Ticket sales w ere hurting b ecau se perform ance on factors that are important to custom ers w as dismal, including a lo w percent­ age o f on-tim e departures, frequ ent baggage arrival problem s, and to o m any custom ers turned away due to overbooking.

S o lu tio n T h e revival o f Continental began in 1994, w h en Gordon B eth u n e b eca m e CEO and initiated th e G o Forward plan, w hich con sisted o f four interrelated parts to b e im plem ented simultaneously. Beth une targeted the need to improve cus­ tom er-valued perform ance m easures by better understanding custom er n eed s as w ell as custom er perceptions o f th e value o f services that w ere and cou ld b e offered. Financial m anage­ m ent practices w ere also targeted for a significant overhaul. As early as 1998, th e airline had separate databases for marketing and operations, all hosted and m anaged b y outside vendors. Processing queries and instigating marketing programs to its high-value custom ers w ere tim e-consum ing and ineffective. In additional, inform ation that the w orkforce need ed to m ake quick d ecisions w as simply not available. In 1999, Continental ch o se to integrate its marketing, IT, revenue, and operational data sources into a single, in-house, EDW . T h e data w are­ hou se provided a variety o f early, m ajor benefits.

As s o o n as Continental returned to profitability and ranked first in th e airline industry in m any perform ance m et­ rics, B eth u n e a n d his m anagem ent team raised th e b a r by escalating the vision. Instead o f just perform ing best, they

w anted Continental to be their custom ers’ favorite airline. T h e G o Forw ard p lan establish ed m ore action able w ays to m ove from first to favorite am ong custom ers. T ech n o lo g y becam e increasingly critical for supporting th ese new initiatives. In th e early days, having ac ce s s to historical, integrated informa­ tion w as sufficient. This prod uced substantial strategic value. But it b ecam e increasingly imperative for th e data w areh ou se to provide real-tim e, action able inform ation to support enter­ prise-w ide tactical d ecisio n m aking and business p rocesses.

Luckily, the w arehouse team had expected and arranged for the real-time shift. From the very beginning, the team had created an architecture to handle real-time data feeds into the warehouse, extracts o f data from legacy systems into the w are­ house, and tactical queries to the warehouse that required almost immediate response times. In 2001, real-time data becam e avail­ able from the w arehouse, and the amount stored grew rapidly. Continental m oves real-time data (ranging from to-the-minute to hourly) about customers, reservations, check-ins, operations, and flights from its m ain operational systems to the warehouse. Continental’s real-time applications include the following:

• R evenue m an agem en t and accounting • Custom er relationship m anagem ent (CRM) • Crew operations and payroll • Security and fraud • Flight operations

R e s u lts In the first year alone, after the data w arehouse project was deployed. Continental identified and eliminated over $7 million in fraud and red uced costs by $41 million. With a $30 million investment in hardware and software over 6 years, Continental has reached over $500 million in increased revenues and cost savings in marketing, fraud detection, dem and forecasting and tracking, and improved data center management. T h e single, integrated, trusted view o f the business (i.e., the single version o f die truth) has led to better, faster decision making.

B ecau se o f its tremendous success, Continental’s DW implementation has b een recognized as an excellent exam ple for real-time B I, b ased o n its scalable and extensible architec­ ture, practical decisions o n w hat data are captured in real time, strong relationships with end users, a small and highly com pe­ tent data w arehouse staff, sensible weighing o f strategic and tac­ tical decision support requirements, understanding o f the syn­ ergies betw een d ecision support and operations, and changed business processes that use real-time data.

Q u e s t i o n s f o r t h e E n d - o f - C h a p t e r A p p l i c a t i o n C a s e

1 . D escrib e th e b enefits o f im plem enting die Continental G o Foiw ard strategy.

2 . Explain w hy it is im portant fo r an airline to u se a real­ tim e data w arehou se.

162 Part II • D escriptive Analytics

3 . Identify th e m ajor d ifferences betw een th e traditional data w areh ou se a n d a real-tim e data w areh ou se, as w as im plem ented at Continental.

4 . W hat strategic advantage c a n Continental derive from th e real-tim e system as o p p o sed to a traditional infor­ m ation system?

Sources: Adapted from H. Wixom, J . Hoffer, R. Anderson-Lehman, and A. Reynolds, “Real-Time Business Intelligence: Best Practices at Continental Airlines,” In form ation Systems M anagem ent Jou rn al, Winter 2006, pp. 7 -1 8 ; R. Anderson-Lehman, H. Watson, B. Wixom, and J . Hoffer, “Continental Airlines Flies High with Real-Time Business

Intelligence,” MIS Q uarterly Executive, Vol. 3, No. 4, December 2004, pp. 163-176 (available at teradatauniversitynetwork.com); H. Watson, “Real Time: T he Next Generation o f Decision-Support Data Management,” B u sin ess In telligen ce Jo u rn al, Vol. 10, No. 3, 2005, pp. 4 -6 : M. Edwards, “2003 Best Practices Awards Winners: Innovators in Business Intelligence and Data Warehousing,” B usiness In tellig en ce Jo u rn a l, Fall 2003, pp. 5 7 -6 4 ; R. Westervelt, “Continental Airlines Builds Real-Time Data W arehouse,” August 20, 2003, searchoracle.techtarget.com ; R. Clayton, “Enterprise Business Performance Management: Business Intelligence + Data Warehouse = Optimal Business Performance,” T eradata M agazine, September 2005, and T he Data Warehousing Institute, “2003 Best Practices Summaries: Enterprise Data W arehouse,” 2003-

References Adam son, C. (2 0 0 9 ). T h e S ta r S c h e m a H a n d b o o k : T he

C o m p le te R e fe r e n c e to D im e n s i o n a l D a t a W a r e h o u s e D esig n . H oboken , NJ: Wiley.

Adelman, S., and L. M oss. (2 0 0 1 , W inter). “D ata W arehou se Risks.” J o u r n a l o f D a l a W a r e h o u s in g , Vol. 6, No. 1.

Agosta, L. (2006, Jan u ary). “T h e D ata Strategy Adviser: T h e Y ear Ahead— Data W arehou sing Trends 20 0 6 .” DM R eview , Vol. 16, No. 1.

Akbay, S. (2 0 0 6 , Q uarter 1). “Data W arehousing in Real T im e.” B u s in e s s I n t e llig e n c e J o u r n a l , Vol. 11, No. 1.

Am beo. (2 0 0 5 , Ju ly ). “A m beo D elivers P roven Data Access Auditing Solution.” D a t a b a s e T ren d s a n d A p p lica tio n s, Vol. 19, No. 7.

Anthes, G. H. (2 0 0 3 , Ju n e 30). “Hilton C h ecks into N ew Suite.” C o m p u terw o r ld , V ol. 37 , No. 26.

Ariyachandra, T ., a n d H. W atson. (2 0 0 5 ). “Key Factors in Selecting a Data W arehou se A rchitecture.” B u s in e s s In t e llig e n c e J o u r n a l , Vol. 10, No. 3-

Ariyachandra, T ., and H. Watson. (2006a, January). “Benchmarks for B I and Data W arehousing Success.” DM Review, Vol. 16, No. 1.

Ariyachandra, T ., and H. W atson. (2 0 0 6 b ). “W hich Data W arehou se A rchitecture Is M ost Successful?” B u s in e s s I n t e llig e n c e J o u r n a l , Vol. 11, No. 1.

Armstrong, R. (20 0 0 , Q uarter 3). “E-nalysis for th e E-busin ess.” T e r a d a t a M a g a z i n e O n lin e, teradata.com.

Ball, S. K. (20 0 5 , N ovem ber 14). “D o You N eed a Data W arehou se Layer in Your B u sin ess In telligence Architecture?” dataw arehouse.ittoolbox.com /docum ents/industry- articles/do-you-need-a-data-warehouse-layer-in-your- business-intelligencearchitecture-2729 (accessed Ju n e 2009).

Barquin, R., A. Paller, and H. Edelstein. (1997). “T e n Mistakes to Avoid for Data W arehousing Managers.” In R. Barquin and H. Edelstein (ed s.). B u ild in g , Using, a n d M a n a g in g th e D a t a W a r eh o u se . U pper Saddle River, NJ: Prentice Hall.

Basu, R. (2 0 0 3 , N ovem ber). “Challenges o f Real-Tim e Data W arehou sing .” D M R eview .

B ell, L. D. (2 0 0 1 , Spring). “M etaBu siness Meta D ata fo r the M asses: A dm inistering K now ledge Sharing fo r Y o u r Data W areh o u s e . ” J o u r n a l o f D a t a W a r e h o u s in g , Vol. 6, N o. 3.

Benander, A., B. Ben and er, A. Fadlalla, and G . Jam es. (2000, Winter). “Data W arehou se Administration and Management.” In fo r m a t io n System s M a n a g e m e n t, Vol. 17, No. 1.

B o n d e, A., and M. K ucku k. (2004, April). “Real World B usiness Intelligence: T h e Im plem entation P erspective.” D M R eview , Vol. 14, No. 4.

B reslin, M. (2004, W inter). “D ata W arehousing Battle o f th e Giants: Com paring th e B asics o f Kimball and Inm on M odels.” B u s in e s s I n t e llig e n c e J o u r n a l , Vol. 9, No. 1.

B ro b st, S., E. Levy, and C. Muzilla. (2 0 0 5 , Spring). “E n terp rise A p p lication In tegration and Enterprise In form ation In teg ratio n ." B u s in e s s I n t e ll ig e n c e J o u r n a l , Vol. 10 , No. 3.

Brody, R. (2 0 0 3 , Sum m er). “Inform ation Ethics in the D esign and U se o f M etadata.” IEEE T e c h n o lo g y a n d S o ciety M a g a z in e , Vol. 22, No. 3-

B row n, M. (2 0 0 4 , May 9 - 1 2 ) . “8 Characteristics o f a Successful D ata W areh ou se.” P r o c e e d in g s o f t h e T w en ty-N inth A n n u a l SAS Users G r o u p I n t e r n a t i o n a l C o n f e r e n c e (SUGI 29). Montreal, Canada.

Burdett, J ., and S. Singh. (2 0 0 4 ). “Challenges and Lessons Learned from Real-Tim e Data W arehou sing.” B u sin ess I n t e llig e n c e J o u r n a l , Vol. 9, No. 4.

C offee, P. (2003, Ju n e 2 3 ). “’Active’ W arehousing.” eW ee k , Vol. 20, No. 25.

Cooper, B . L., H. J . W atson, B . H. W ixom , and D. L. Goodhue. (1 9 9 9 , August 1 5 -1 9 ). “Data W arehousing Supports Corporate Strategy a t First Am erican Corporation.” SIM International C on feren ce, Atlanta.

C ooper, B . L., H. J . W atson , B . H. W ixom , and D. L. G oodhu e. (2 0 0 0 ). “Data W arehou sing Supports Corporate Strategy at First A m erican C orporation.” MIS Q u a rterly , Vol. 24, No. 4, pp. 5 4 7 -5 6 7 .

Dasu, T ., and T . Jo h n s o n . (2 0 0 3 ). E x p lo r a to r y D a t a M in in g a n d D a t a C le a n in g . N ew Y ork: Wiley.

Davison, D. (2003, N ovem ber 14). “T o p 10 Risks o f O ffshore O utsourcing.” META G roup R esearch Report, now Gartner, In c., Stamford, CT.

Devlin, B . (2003, Q uarter 2). “Solving the Data W arehou se Puzzle.” D B 2 M a g a z in e .

D ragoon, A. (2 0 0 3 , Ju ly 1). “All for O n e V iew .” CIO.

Chapter 3 • Data W arehou sing 1 6 3

E ck erson, W . (2 0 0 3 , Fall). “T h e Evolution o f ETL.” B u s in e s s In t e llig e n c e J o u r n a l , Vol. 8, No. 4.

E ck erson, W . (2 0 0 5 , April 1). “Data W arehou se Builders Advocate for D ifferent A rchitectures.” A p p lic a tio n D e v e lo p m e n t T ren ds.

Eckerson, W., R. H ack ath om , M. McGivern, C. Tw ogood , and G. W atson. (2 0 0 9 ). “D ata W arehousing A ppliances.” B u s in e s s I n t e ll ig e n c e J o u r n a l , Vol. 14, No. 1, pp. 4 0 -4 8 .

Edw ards, M. (2 0 0 3 , Fall). “2 0 0 3 B e s t P ractices Awards W inners: In n o v ato rs in B u sin ess In te llig e n c e and D ata W a re h o u sin g .” B u s i n e s s I n t e lli g e n c e J o u r n a l , Vol. 8 , N o. 4.

“E gg’s C ustom er D ata W areh ou se Hits the Mark.” (2005, O cto b er). D M R ev iew , Vol. 15, No. 10, pp. 2 4 -2 8 .

E lson, R., and R. LeClerc. (2 0 0 5 ). “Security and Privacy C oncerns in die Data W areh ou se Environm ent.” B u s in e s s In t e llig e n c e J o u r n a l , Vol. 10, No. 3.

Ericson, J . (20 0 6 , M arch). “Real-Tim e Realities.” B I R eview . Furtado, P. (2009)- “A Survey o f Parallel and Distributed Data

W arehou ses.” I n t e r n a t i o n a l J o u r n a l o f D a t a W a r eh o u sin g a n d M in in g , Vol. 5, No. 2, pp. 5 7 -7 8 .

Golfarelli, M., and Rizzi, S. (2 0 0 9 ). D a t a W a r e h o u s e D esig n : M o d e m P r in c ip le s a n d M e th o d o lo g ie s . San Francisco: McGraw-Hill O sb o rn e Media.

G onzales, M. (2 0 0 5 , Q uarter 1). “Active Data W arehou ses Are Ju s t O n e A pproach for C om bining Strategic and T ech nical D ata.” D B 2 M a g a z in e .

Hall, M. (2 0 0 2 , April 15)- “Seed ing for Data G row th.” C o m p u terw o r ld , Vol. 36, No. 16.

Hammergren, T . C., and A. R. Simon. (2009). D a t a W a r eh o u sin g f o r D u m m ies, 2nd ed. H oboken, NJ: Wiley.

Hicks, M. (20 0 1 , N ovem ber 26). “G etting Pricing Ju st Right." e W ee k , Vol. 18, No. 46.

Hoffer, J . A., M. B . Prescott, and F. R. M cFadden. (2007). M o d e m D a t a b a s e M a n a g e m e n t, 8th ed. U pper Saddle River, NJ: P rentice Hall.

Hwang, M., and H. Xu. (2005, Fall). “A Survey o f Data W arehousing S u ccess Issues.” B u s in e s s In te llig e n c e J o u r n a l, Vol. 10, No. 4.

IBM. (2 0 0 9 ). 5 0 T b D a t a W a r e h o u s e B e n c h m a r k o n IB M System Z . Arm onk, NY: IBM Redbooks.

Im hoff, C. (20 0 1 , May). “P ow er Up Y o u r Enterprise Portal.” E -B u sin es s A d v ic e.

Inm on, W. H. (2 0 0 5 ). B u ild in g t h e D a t a W a r eh o u se , 4th ed. N ew Y ork : W iley.

Inm on, W. H. (2 0 0 6 , Janu ary). “Inform ation M anagem ent: H ow D o Y o u T u n e a D ata W arehouse?” D M R eview , Vol. 16, No. 1.

Ju k ic , N., and C. Lang. (2 0 0 4 , Sum m er). “U sing O ffshore R esources to D e v e lo p and Support D ata W arehou sing A pplications.” B u s i n e s s I n t e ll ig e n c e J o u r n a l , Vol. 9, No. 3-

Kalido. “B P Lubricants Achieves B IG S S u ccess.” k a l i d o .c o m / c o l l a t e r a l / D o c u m e n t s /E n g l i s h - U S /C S - B P % 2 0 B I G S . p d f (accessed August 2009).

Karacsony, K. (2 0 0 6 , January). “ETL Is a Sym ptom o f the Problem , n o t th e Solu tion.” D M R eview , Vol. 16, No. 1.

Kassam, S. (20 0 2 , April 16). “Freed om o f Inform ation.” In te llig e n t E n terp rise, Vol. 5, No. 7.

Kay, R. (2 0 0 5 , S ep tem b er 19). “E li.” C o m p u terw o r ld , Vol. 39, No. 38.

Kelly, C. (2 0 0 1 , Ju n e 14). “Calculating Data W arehousing R O I.” S e a rc h S Q L S e rv e r.c o m Tips.

M alykhina, E. (20 0 3 , Ja n u a ry 3). “T h e Real-Tim e Im perative.” I n fo r m a t io n W e e k , Issu e 1020.

Manglik, A., and V. Mehra. (2005, Winter). “Extending Enterprise B I Capabilities: N ew Patterns for Data Integration.” B u sin ess In te llig e n c e J o u r n a l , Vol. 10, No. 1.

Martins, C. (2005, D ec em b e r 13)- “HP to Consolidate Data Marts into Single W areh o u se.” C o m p u terw o r ld .

Matney, D. (20 0 3 , Spring). “End-User Support Strategy.” B u s in e s s I n t e llig e n c e J o u r n a l , Vol. 8, No. 3-

M cCloskey, D. W . (2 0 0 2 ). C h o o s in g V en d o rs a n d P r o d u c ts to M a x im iz e D a t a W a r e h o u s in g S u cces s. New York: Auerbach Publications.

Mehra, V. (2 0 0 5 , Sum m er). “Building a Metadata-Driven Enterprise: A H olistic A pproach.” B u s in e s s In te llig e n c e J o u r n a l Vol. 10, No. 3-

M oseley, M. (2 0 0 9 ). “Eliminating Data W arehou se Pressures w ith Master D ata Services and SOA.” B u s in e s s I n t e llig e n c e J o u r n a l , Vol. 14, No. 2, pp. 3 3 -4 3 .

Murtaza, A. (1998, Fall). “A Framework for Developing Enterprise Data W arehouses.” In fo r m a t io n System s M a n a g e m e n t, Vol. 15, No. 4.

Nash, K. S. (2002, Ju ly ). “Chem ical R eaction.” B a s e lin e . Orovic, V. (2003, Ju n e ). “T o D o & Not to D o .” e A I J o u r n a l. Parzinger, M. J ., and M. N. Frolick. (2 0 0 1 , Ju ly ). “Creating

Com petitive Advantage Throu gh D ata W arehousing.” In fo r m a t i o n Strategy, Vol. 17, No. 4.

Peterson, T . (20 0 3 , April 21). “Getting Real About Real Tim e.” C o m p u terw o r ld , Vol. 37, No. 16.

Raden, N. (2003, Ju n e 30). “Real Tim e: G et Real, Part II.” In te llig e n t E n terp rise.

R eeves, L. (2 0 0 9 ). M a n a g e r ’s G u id e to D a t a W a r eh o u sin g . H oboken , NJ: W iley.

Rom ero, O ., and A. Abello. (2009). “A Survey o f Multidimensional Modeling M ethodologies.” I n t e r n a t io n a l J o u r n a l o f D a ta W a r eh o u sin g a n d M in in g, Vol. 5, No. 2, pp. 1-24.

R osenberg, A. (2 0 0 6 , Quarter 1). “Im proving Query P erform ance in D ata W arehou ses.” B u s in e s s In te llig e n c e J o u r n a l , Vol. 11, No. 1.

Russom , P. (2 0 0 9 ). Next G en eration Data W arehou se Platforms. TDW I B est P ractices Report, available at w w w . t d w i.o r g (a c ce s se d Janu ary 2010).

Sam m on, D., and P . Finnegan. (2000, Fall). “T h e T en Com m andm ents o f Data W arehou sing.” D a t a b a s e f o r A d v a n c e s in I n f o r m a t i o n System s, Vol. 31, No. 4.

Sapir, D. (2005, May). “Data Integration: A Tutorial.” DM R eview , Vol. 15, No. 5.

Saunders, T. (2 0 0 9 ). “C ooking up a D ata W areh ou se.” B u s in e s s I n t e llig e n c e J o u r n a l , Vol. 14, No. 2, pp. 1 6 -2 3 .

Schwartz, K. D. “D ecision s at the T ou ch o f a Button.” T e r a d a ta M a g a z in e , (a c ce s se d Ju n e 2009).

1 6 4 Part II • D escriptive Analytics

Schwartz, K. D. (2 0 0 4 , M arch). “D ecision s at the T o u c h o f a B u tton .” DSS R es o u r c e s , pp. 2 8 -3 1 - dssresources.com / cases/coca-colajapan/index.html (a c ce s se d April 2006).

Sen, A. (2004, April). “Metadata M anagem ent: Past, Present and Future.” D e c is io n S u p p o r t System s, Vol. 37, No. 1.

Sen, A., and P. Sinha. (2 0 0 5 ). “A Com parison o f Data W arehou sing M ethodologies.” C o m m u n ic a t io n s o f th e ACM, Vol. 48 , No. 3-

Solom on, M. (2 0 0 5 , W inter). “Ensuring a Successful Data W arehouse Initiative.” I n f o r m a t i o n System s M a n a g e m e n t J o u r n a l .

Songini, M. L. (2 0 0 4 , February 2). “ETL Q uickstudy.” C o m p u terw o r ld , V ol. 38 , No. 5.

Sun M icrosystems. (20 0 5 , Sep tem ber 19). “Egg B an k s on Sun to Hit th e Mark w ith Custom ers." sun.com /sm i/ P r e ss /s u n f la s h /2 0 0 5 -0 9 /su n fla sh .2 0 0 5 0 9 1 9 .1 .xml (a c cessed April 20 0 6 ; n o lo n g er available online).

Tannenbaum , A. (2002, Spring). “Identifying Meta Data Requirem ents. ” J o u r n a l o f D a t a W areh ou sin g , Vol. 7 , No. 3-

T ennant, R. (20 0 2 , May 15). “T h e Im portance o f B eing Granular.” L ib r a r y J o u r n a l , V ol. 127, No. 9.

Terad ata Corp. “A Large U S-B ased Insurance Com pany M asters Its F in a n ce D ata.” (a c cessed Ju ly 2009).

Teradata Corp. “A ctive Data W arehou sing.” teradata.com / active-data-warehousing/ (a c cessed April 2006).

Teradata Corp. “C oca-C ola Ja p a n Puts th e Fizz B a ck in V ending M ach in e Sales." (accessed Ju n e 2009).

Teradata. “Enterprise Data W arehouse Delivers Cost Savings and Process Efficiencies." teradata.com/t/resources/case- studies/NCR-Corporation-eb4455 (accessed Ju n e 2009).

Terr, S. (2004, February). “Real-Tim e Data W arehousing: H ardware and Softw are.” D M R eview , Vol. 14, No. 3.

T horn ton , M. (2 0 0 2 , March 18). “W hat A bout Security? T h e M ost Com m on, bu t Unwarranted, O b jectio n to Hosted D ata W areh ou ses.” D M R eview , Vol. 12, No. 3, pp. 3 0-43-

Thornton, M., and M . Lampa. (2002). “H osted Data W arehou se.” J o u r n a l o f D a t a W a r eh o u sin g , Vol. 7, No. 2, pp. 2 7 -3 4 .

Turban, E ., D. Leidner, E. McLean, and J . W etherbe. (2006). I n f o r m a t io n T e c h n o lo g y f o r M a n a g e m e n t, 5th ed. New Y ork : Wiley.

Vaduva, A., and T . Vetterli. (2 0 0 1 , Septem ber). “Metadata M anagem ent for D ata W arehousing: An O verview .” I n t e r n a t i o n a l J o u r n a l o f C o o p e r a tiv e I n fo r m a t i o n System s, Vol. 10, No. 3.

V an d en H oven, J . (1 9 9 8 ). “Data Marts: Plan Big, Build Sm all.” I n f o r m a t io n S y stem s M a n a g e m e n t, Vol. 15, No. 1.

W atson , H. J . ( 2 0 0 2 ) . “R e c en t D ev elo p m en ts in Data W a re h o u sin g .” C o m m u n ic a t i o n s o f t h e ACM, V ol. 8, No. 1.

W atson, H. J . , D. L. G ood h u e, and B . H. W ixom . (2002). “T h e B en efits o f D ata W arehou sing: W hy Som e Organizations R ealize E xcep tional Payoffs.” I n f o r m a t io n & M a n a g e m e n t, Vol. 39.

W atson , H ., J . G erard , L. G o n z a le z , M. H ayw ood , and D. F e n to n . (1 9 9 9 ). “D ata W a re h o u se Failu res: C ase Studies a n d F in d in g s.” J o u r n a l o f D a t a W a r e h o u s in g , V ol. 4, No. 1.

W eir, R. (2 0 0 2 , W inter). “B es t P ractices for Im plem enting a D ata W a reh o u se.” J o u r n a l o f D a t a W a r eh o u sin g , Vol. 7, No. 1.

W ilk , L. ( 2 0 0 3 , S p rin g ). “D ata W a re h o u sin g a n d R eal- T im e C o m p u tin g .” B u s i n e s s I n t e l l i g e n c e J o u r n a l , V o l. 8 , No. 3.

W ixom , B ., and H. W atson. (2001, March). “An Empirical Investigation o f th e Factors Affecting Data W arehousing S u ccess.” M IS Q u a rter ly , Vol. 25, No. 1.

W rem bel, R. (2009)- “A Survey o f M anaging th e Evolution o f D ata W areh ou ses.” I n t e r n a t i o n a l J o u r n a l o f D a t a W a r e h o u s in g a n d M in in g , Vol. 5, No. 2, pp. 24—56.

ZD Net UK. “Sun Case Study: Egg’s C ustom er D ata W areh ou se.” w h ite p a p e rs.z d n e t.co .u k /0 ,3 9 0 2 5 9 4 5 ,6 0 1 5 9 4 0 1 p - 3 9 0 0 0 4 4 9 q ,0 0 .h tm (a c cessed Ju n e 2009).

Zhao, X . (2005, O cto b e r 7 ). “M eta D ata M anagem ent Maturity M odel. ” D M D ir e c t N ew sletter.

C H A P T E R

1 : 1 - v ! • 5"- f f O | i

t f f n r a i ■ w m m M M

Business Reporting, Visual Analytics, and Business

Performance Management

LEARNING OBJECTIVES

■ D efin e b u sin e ss rep ortin g and u n d erstan d its h istorical ev o lu tio n

■ R e co g n iz e th e n e e d fo r a n d th e p o w e r o f b u sin e ss rep ortin g

■ U n d erstand th e im p o rtan ce o f data/ in form ation visualization

■ L earn d iffe ren t ty p e s o f visualization te ch n iq u es

■ A p p re ciate th e v a lu e that visual analytics b rin gs to BI/BA

■ K n o w th e cap ab ilitie s a n d lim itations o f d a sh b o ard s

* U n derstand th e n atu re o f b u sin e ss p e rfo rm a n ce m a n a g e m e n t (B P M )

* L earn th e c lo s e d -lo o p BPM m e th o d o lo g y

■ D e s crib e th e b a sic e le m e n ts o f th e b a la n c e d s co re ca rd

A rep o rt is a co m m u n ica tio n artifact p re p a re d w ith th e s p e c ific in te n tio n o f relaying in fo rm atio n in a p re s e n ta b le form . I f it c o n c e rn s b u sin ess m atters, th e n it is ca lle d a b u s i n e s s r e p o r t . B u s in e s s re p o rtin g is a n esse n tial p art o f th e b u sin ess in te llig e n ce m o v e m e n t to w ard im provin g m an ag erial d e cisio n m akin g. N ow ad ays, th e se reports a re m o re visually o rie n te d , o fte n u sin g co lo rs an d grap h ical ic o n s th a t co lle ctiv e ly lo o k lik e a d a s h b o a rd to e n h a n c e th e in form ation co n ten t. B u sin ess re p o rtin g an d b u sin ess p e rfo rm a n ce m a n a g e m e n t (B P M ) are b o th e n a b le rs o f b u sin e ss in te llig e n ce an d analytics. As a d e c is io n su p p o rt to o l, BPM is m o re th an ju s t a re p o rtin g te ch n o lo g y . It is an integrated s e t o f p ro c e s s e s, m e th o d o lo g ie s, m etrics, an d a p p lica tio n s d e sig n e d to drive ih e overall fin a n cia l an d o p e ra tio n a l p e rfo rm a n ce o f a n en te rp rise. It h e lp s en terp rises translate th e ir strateg ies a n d o b je c tiv e s in to p lan s, m o n ito r p e rfo rm a n ce ag ain st th o se plans, an aly ze v ariation s b e tw e e n actu al results a n d p la n n e d results, a n d ad ju st th e ir o b je ctiv e s a n d a c tio n s in re s p o n s e to this analysis.

T h is ch a p te r starts w ith e x a m in in g th e n e e d fo r an d th e p o w e r o f b u s in e s s rep o rt­ ing W ith th e e m e rg e n c e o f analytics, b u sin e ss rep ortin g e v o lv e d into d ash b o ard s an d V2? ja l analytics, w h ich , co m p a re d to trad itional d escrip tive rep ortin g , is m u c h m o re p re - ■ a civ e an d p rescrip tiv e. C o v erag e o f d ash b o ard s an d visual an aly tics is fo llo w e d b y a

c o m p re h e n siv e in trod u ctio n to BPM . As y o u will s e e an d a p p recia te , B P M a n d visual an alytics h av e a sy m b io tic relatio n sh ip (o v e r sco re ca rd s and d a sh b o a rd s) w h e re they

b e n e fit fro m e a c h o th e r’s strengths.

4 .1 O p e n in g V ig n e tte : S e lf- S e r v ic e R e p o r tin g E n v iro n m e n t S a v e s M illio n s fo r C o rp o ra te C u s to m e rs 1 6 6

4 . 2 B u s in e s s R e p o rtin g D e f in itio n s a n d C o n c e p ts 1 6 9

4 . 3 D a ta a n d In fo r m a tio n V is u a liz a tio n 1 7 5

4 . 4 D if fe r e n t T y p e s o f C h a rts a n d G r a p h s 1 8 0 4 . 5 T h e E m e r g e n c e o f D a ta V is u a liz a tio n an d V is u a l A n a ly tics 1 8 4

4 . 6 P e r f o r m a n c e D a s h b o a r d s 1 9 0 4 . 7 B u s in e s s P e r f o r m a n c e M a n a g e m e n t 1 9 6

4 . 8 P e r fo r m a n c e M e a s u re m e n t 2 0 0

4 . 9 B a la n c e d S c o r e c a r d s 2 0 2 4 . 1 0 S ix S ig m a a s a P e r f o r m a n c e M e a s u r e m e n t S y ste m 2 0 5

1 6 6 P art II • Descriptive Analytics

4.1 OPENING VIGNETTE: Self-Service Reporting Environment Saves Millions for Corporate Customers

H e ad q u arte red in O m ah a, N ebrask a, T ra v e l a n d T ran sp o rt, In c., is th e six th largest travel m a n a g e m e n t co m p a n y in th e U n ite d States, w ith m o re than 7 0 0 e m p lo y e e -o w n e rs lo cated n atio n w id e. T h e co m p a n y h a s e x te n s iv e e x p e r ie n c e in m u ltip le v erticals, inclu d ing travel m an ag e m e n t, loyalty so lu tio n s p ro g ram s, m e e tin g a n d in cen tiv e p lan n in g , a n d leisu re

travel services.

CHALLENGE In th e field o f e m p lo y e e travel serv ice s, th e ab ility to e ffe ctiv e ly co m m u n ica te a value p ro p o sitio n to e x istin g a n d p o ten tial cu sto m e rs is critical to w in n in g and retaining b u sin ess. W ith travel arran g em en ts o fte n m ad e o n a n a d h o c b a sis, cu sto m ers rind it difficult to an alyze co s ts o r instate op tim al p u rch a se ag re em en ts. T rav el and T ran sp o rt w a n te d to o v e rc o m e th e s e ch a lle n g e s b y im p lem en tin g a n integ rated re p o rtin g and analysis sy stem to e n h a n c e re latio n sh ip s w ith existin g clie n ts, w h ile p ro vid ing th e kind o f valu e -ad d ed s erv ices that w o u ld attract n e w p ro sp ects.

SOLUTION T rav el a n d T ran sp o rt im p lem e n te d In fo rm atio n B u ild e rs’ W e b F O C U S b u sin e ss in te llig e n ce ( B I ) platform (c a lle d e T T e k R ev iew ) a s th e fo u n d atio n o f a d y n am ic cu sto m e r self- serv ice B I e n v iron m en t. T h is d ash b o ard -d riv en e x p e n s e -m a n a g e m e n t a p p lica tio n h e lp s m o re th a n 8 0 0 e x te rn a l clie n ts lik e R o b e rt W . B aird & C o ., M etLife, and A m erican Fam i y In su ran ce to p lan , track , an aly ze, a n d b u d g e t th e ir travel e x p e n s e s m o re efficien tly and to b e n ch m a rk th e m ag ain st sim ilar c o m p a n ie s , savin g th e m m illions o f dollars. M ore than 2 0 0 in tern al e m p lo y e e s , in clu d in g cu sto m e r sen d e e sp ecialists, a lso h a v e a c c e s s to th e system , using it to g e n e ra te m o re p re cise fo reca sts fo r clien ts an d to stream lin e a n d a c c e l­ erate o th er k e y su p p o rt p ro c e s s e s s u ch a s q uarterly review s.

T h a n k s to W eb F O C U S, T rav el a n d T ra n sp o rt d o e s n ’t ju st tell its clie n ts h o w m u ch th e y are savin g b y u sin g its s erv ices— it sh o w s them . T h is h a s h e lp e d th e co m p a n y to d ifferen tiate itself in a m ark e t d efin e d b y a g g re ssiv e co m p etitio n . A dditionally, W ebP O C U S

Chapter 4 • B u sin ess Reporting, V isual Analytics, and B u sin ess P erform an ce M anagem ent 167

elim inates m an u al re p o rt co m p ila tio n for c lie n t serv ice sp e cialists, sav in g th e co m p a n y

d o s e to $ 2 0 0 ,0 0 0 in lo s t tim e e a c h year.

AN INTUITIVE, GRAPHICAL WAY TO MANAGE TRAVEL DATA

Using stu n n in g g ra p h ics c re a te d w ith W e b F O C U S an d A d o b e F lex , th e b u sin e ss in telli­ g e n c e sy stem p ro v id e s a c c e s s to th o u san d s o f rep o rts th at s h o w individual clien t m etrics, b en ch m a rk e d in form ation against ag g re g ate d m ark et d ata, a n d e v e n ad h o c re p o rts that users c a n s p e cify as n e e d e d . “F o r m o st o f o u r co rp o ra te cu sto m ers, w e tho rou g hly m a n a g e th e ir travel fro m p la n n in g a n d reserv atio n s to billing, fulfillm ent, an d o n g o in g analysis, say s M ike K u basik, s e n io r v ic e p re sid e n t a n d CIO at T rav el an d T ran sp o rt. “W e b F O C U S as im portant to o u r b u s in e s s . It h e lp s o u r cu sto m ers m o n ito r e m p lo y e e s p e n d in g , b o o k trav el w ith p re fe rre d v e n d o rs, an d n e g o tia te co rp o ra te p u rch asin g a g re em en ts th a t c a n

save th e m m illion s o f d ollars p e r y e a r.” Clients lo v e it, a n d it’s giving T rav el an d T ran sp o rt a com p etitiv e e d g e in a cro w d ed

m arketp lace. “I u s e T rav el and T ran sp o rt’s e T T e k R e v ie w to au tom atically e-m ail reports throu gho u t th e c o m p a n y fo r a variety o f re a so n s, s u c h as m o n ito rin g travel tren d s and com p an y e xp e n d itu re s and assisting w ith airlin e e x p e n s e re co n cilia tio n and a llo catio n s, savs Cathy M o u lton, v ic e p re sid e n t an d trav el m an ag e r at R o b ert W . B a ird & C o., a pro m in en t fin an cial s e rv ice s co m p an y . W h at s h e lo v e s a b o u t th e W e b F O C U S -e n a b le d W eb portal is that it m a k e s all o f th e c o m p a n y ’s travel in form ation av ailab le in just a few d ic k s . “I h av e th e data a t m y fin g ertip s,” s h e adds. “I d o n ’t h av e to w ait fo r s o m e o n e to g o in a n d d o it fo r m e. I c a n s e t u p th e rep o rts o n m y o w n . T h e n w e c a n g o to th e h o tels an d p referred v e n d o rs a rm e d w ith d eta iled in form ation that g iv e s u s lev erag e to n e g o tia te

o u r rates.” R o b ert W . B a ird & C o . isn ’t th e o n ly firm b e n e fitin g fro m this a d v a n ce d a c c e s s to

reporting. M any o f T ra v e l an d T ran sp o rt’s o th e r clie n ts are a lso h ap p y w ith th e te c h n o l­ o g y “W ith T rav el a n d T ra n sp o rt’s state-o f-th e-art rep ortin g te ch n o lo g y , M etLife is ab le © m e a s u r e its travel p ro g ram th ro u g h data analysis, stand ard rep ortin g, and th e ability to create ad h o c re p o rts d y n am ically,” says T o m M o lesk y, d irecto r o f trav el s e rv ice s at MetLife. “M etrics d eriv ed fro m a c tio n a b le d ata pro v id e d irection and drive u s tow ard ou i =oals T h is is k e y to h e lp in g u s n e g o tia te w ith ou r su p p liers, e n fo rc e o u r travel p o licy , an d save o u r co m p a n y m o n ey . T rav el and T ra n sp o rt’s lea d in g -e d g e p ro d u ct h a s h e lp e d ” 5 to m e e t and , in s o m e c a s e s , e x c e e d o u r travel goals.

READY FOR TAKEOFF , T rav ei an d T ra n sp o rt u s e d W eb F O C U S to cre a te an o n lin e sy stem th a t allow s clien ts

s a cce ss in form ation d irectly, s o th e y w o n ’t h av e to rely o n th e IT d ep artm en t to run reports fo r th em . Its o b je c tiv e w a s to give cu sto m ers o n lin e to o ls to m o n ito r co rp o ra te t e v e l e x p e n d itu re s th ro u g h o u t th e ir co m p a n ie s . B y giving clie n ts a c c e s s to th e right daia, T rav el and T ra n sp o rt c a n h e lp m a k e sure its cu sto m ers are g ettin g th e b e s t p ricing f r o m airlin es, h o tels, c a r ren tal co m p a n ie s , an d o th e r v en d o rs. “W e n e e d e d m o re than fust pretty re p o rts,” K u b a sik re calls, lo o k in g b a c k o n th e early p h a s e s o f th e B I p ro je ct. “W e w an ted to b u ild a rep ortin g en v iro n m en t th a t w as p o w erfu l e n o u g h to h and le tran saction -in ten siv e o p e ra tio n s, y e t sim p le e n o u g h to d ep lo y o v e r th e W e b .” It w as a ■sinning form u la. C lients a n d cu sto m e r serv ice sp ecialists co n tin u e to u s e e T l e k R ev iew L create fo recasts fo r th e co m in g y e ar a n d to target s p e cific are a s o f b u s in e s s travel

I expend itures. T h e s e u sers c a n c h o o s e fro m d o z e n s o f m a n a g e m e n t reports. P o p u la r -sn o rts in clu d e travel sum m ary, airlin e c o m p lia n c e , h o tel analysis, an d ca r analysis.

T rav el m an ag e rs a t a b o u t 7 0 0 c o rp o ra tio n s u se th e s e rep o rts to a n aly ze c o rp o ra te trave I - e n d i n g o n a daily, w e e k ly , m onthly, quarterly, an d an n u al b asis. A b ou t 160 standard

-sp o rts an d m o re th a n 3 ,0 0 0 cu sto m re p o rts a re cu rren tly s e t up in e T T e k R eview ,

1 6 8 Part II * D escriptive Analytics

in clu d in g e v ery th in g fro m n o n c o m p lia n c e re p o rts th at re v e a l w h y a n e m p lo y e e did n o : o b ta in th e lo w e s t airfare fo r a particu lar flight to e x e c u tiv e o v erv iew s that sum m arize sp e n d in g pattern s. M o st rep o rts a re p a ra m ete r d riv e n w ith In fo rm ation B u ild e rs’ unique g u id ed ad h o c rep ortin g te ch n o lo g y .

PE E R REVIEW SYSTEM KEEPS EXPENSES ON TRACK

U sers c a n a lso run rep o rts th at co m p a re th eir o w n trav el m etrics w ith aggregated, travel d ata fro m o th e r T rav el an d T ra n sp o rt clien ts. T h is b en ch m a rk in g serv ice lets th e m gauge w h e th e r th e ir e x p e n d itu re s, p re fe rre d rates, a n d o th e r m etrics are in lin e w ith th o se ot o th e r c o m p a n ie s o f a sim ilar siz e o r w ith in th e s a m e industry. B y p o o lin g th e data, Travel a n d T ran sp o rt h e lp s p ro te c t individual clie n ts ’ in fo rm atio n w h ile a lso e n a b lin g its entire cu sto m e r b a se to a ch ie v e lo w e r rates b y giving th e m lev erag e fo r th e ir n eg otiation s.

R ep orts can b e ru n in teractiv ely o r in b a tc h m o d e , w ith results d isp lay ed o n the s cre e n , s to red in a library, sav e d to a P D F file , lo a d e d in to a n E x c e l sp re a d sh ee t, or se n t as a n A ctive R e p o rt th at p erm its ad d ition al analysis. “O u r clie n ts lo v e th e visual m e tap h o rs p ro v id e d b y In fo rm ation B u ild e rs’ g rap h ical d isplays, in clu d in g A d o b e Flex an d W eb F O C U S A ctive P D F file s ,” e x p la in s S te v e C ords, IT m a n a g e r at T rav el and T ra n sp o rt and te a m le a d e r fo r th e e T T e k R e v ie w p ro je ct. “M ost su m m ary reports have d rill-d ow n cap ab ility to a d eta iled rep ort. All re p o rts c a n b e ru n fo r a p articu lar hierarchy stru cture, a n d m o re th a n o n e h iera rch y c a n b e s e le c te d .”

O f co u rse , u sers n e v e r s e e th e c o d e that m a k e s all o f this p o ssib le . T h e y op erate in a n intuitive d ash b o ard en v iro n m en t w ith d ro p -d o w n m en u s an d d rillable grap h s, all a c c e s s ib le th ro u g h a b ro w se r-b a s e d in te rfa ce th a t req u ires n o clie n t-sid e softw are. This arch ite ctu re m a k e s it e a sy a n d c o s t-e ffe c tiv e fo r u se rs to ta p in to e T T e k R ev iew fro m any lo ca tio n . C o llectiv ely , cu sto m ers run a n e stim a ted 5 0 ,0 0 0 rep o rts p e r m o n th . A b ou t 2 0 ,0 0 0 o f th o se reports are au tom atically g e n e ra te d an d distributed via W eb F O C U S ReportC aster.

AN EFFICIENT ARCHITECTURE THAT YIELDS SOARING RESULTS

T rav el a n d T ran sp o rt cap tu res trav el in fo rm atio n fro m reserv ation sy stem s k n o w n as G lo b a l D istrib u tion Sy stem s (G D S ) via a p ro p rietary b a c k -o ffic e sy stem th at re sid e s in a D B 2 d atab a se o n a n IB M iSeries co m p u ter. T h e y u s e SQ L tab le s to sto re u s e r IDs a n d p assw o rd s, a n d u se o th e r d a tab ase s to s to r e th e in form ation. “T h e d a ta b a se c a n b e so rted a cco rd in g to a s p e cific h ierarch y to m a tch th e b re a k d o w n o f rep o rts re q u ired b y e a c h c o m p a n y ,” co n tin u e s Cords. “I f th e y w a n t to s e e ju st m arketin g and acco u n tin g in form ation , w e c a n d eliv e r it. I f th e y w an t to s e e th e p articu lar lev el o f d etail re flectin g a g iv e n c o s t ce n te r, w e ca n d eliv e r that, t o o .”

B e c a u s e all data is se cu re ly s to red fo r th ree years, clie n ts c a n g e n era te trend reports to co m p a re cu rren t trav el to p rev iou s years. T h e y c a n a lso u s e th e BT sy stem to m o n ito r w h e re e m p lo y e e s are trav eling a t a n y p o in t in tim e. T h e rep o rts are s o e a sy to u s e that C ord s a n d his te a m h av e started re p la cin g ou td ated p ro c e s s e s w ith n e w au tom ated o n e s u sin g th e sa m e W eb F O C U S te ch n o lo g y . T h e c o m p a n y a lso u se s W eb F O C U S to stream line th e ir q u arterly re v iew p ro ce ss. In th e past, clie n t serv ice m an ag e rs h a d to m an u ally cre ate th e s e q u arterly rep o rts b y agg reg atin g d ata fro m a variety o f clien ts. T h e 8 0 -p a g e report to o k o n e w e e k to cre a te at th e e n d o f e v ery q u arter.

T rav el and T ran sp o rt h a s co m p le te ly au to m ated th e q uarterly review system using W eb F O C U S s o th e m an ag ers ca n s e le c t th e p a g e s, p e rce n ta g e s, a n d sp e cific data they w a n t to inclu d e. T h is gives th e m m o re tim e to d o fu rther analysis a n d m ak e b etter u se o f th e inform ation. Cords estim ates th at th e tim e savings add up to a b o u t $ 2 0 0 ,0 0 0 e v ery year fo r this p ro je ct a lo n e . “M etrics d eriv ed fro m a c tio n a b le d ata are k e y to h e lp in g u s n eg o tiate w ith o u r supp liers, e n fo rc e o u r trav el p o licy , a n d save o u r co m p a n y m o n e y ,” co n tin u es Cords. “D u ring th e re ce ssio n , th e travel industry w as hit particularly hard , b u t T rav el and

T ran sp o rt m an ag ed to ad d n e w m u ltim illion d ollar a cco u n ts e v e n in th e w o rst o f tim es. W e attribute a lo t o f this gro w th to th e cu ttin g-ed g e rep orting te c h n o lo g y w e o ffe r to clie n ts.”

QUESTIONS FO R TH E OPENING VIGNETTE

1 . W h at d o e s Travel an d T ran sp o rt, In c ., do? 2 . D e s c r ib e th e co m p le x ity an d th e com p etitiv e n atu re o f th e b u sin e ss e n v iro n m en t in

w h ic h Travel an d T ran sp o rt, In c ., fu n ction s.

3 . W h at w e re th e m ain b u sin ess ch allen ges?

4 . W h at w as th e solution? H o w w as it im plem ented ? 5 . W h y d o y o u th in k a m ulti-vendor, m u lti-tool so lu tio n w a s im plem ented ?

6 . List an d co m m e n t o n a t le a s t th ree m a in b e n e fits o f th e im p le m e n te d system . Can you th in k o f o th e r p o ten tial b e n e fits that are n o t m e n tio n e d in th e case?

WHAT W E CAN LEARN FROM THIS VIGNETTE

T ry in g to survive (a n d thrive) in a h ighly co m p etitiv e industry. T rav el a n d T ransp o rt, I n c ., w a s aw are o f th e n e e d to c re a te a n d e ffe ctiv e ly co m m u n ica te a v alu e p ro p o sitio n to its e x istin g an d p o ten tial cu sto m ers. As is th e c a s e in m a n y indu stries, in th e travel b u s in e s s , s u c c e s s o r m e re survival d e p e n d s o n co n tin u o u sly w in n in g n e w cu stom ers w h ile retain in g th e existin g o n e s . T h e k e y w as to pro v id e v alu e -ad d ed s erv ices to th e c lie n t s o th at th ey c a n efficie n tly a n aly ze co sts an d o th e r o p tio n s to q u ick ly instate o p tim al p u rch a s e a g re em en ts. U sing W eb F O C U S (a n in te g rate d rep ortin g a n d inform ation visu alizatio n e n v iro n m en t b y In fo rm atio n B u ild e rs), T rav el a n d T ra n sp o rt em po^veied th e ir clie n ts to a c c e s s in form ation w h e n e v e r a n d w h e re v e r th e y n e e d it. In fo rm atio n is th e p o w e r th at d e c is io n m ak ers n e e d th e m o st to m a k e b e tte r a n d fa s te r d ecisio n s. W h e n e c o n o m ic co n d itio n s are tight, e v ery m an ag erial d e cisio n — e v e ry b u sin e ss tran sactio n co u n ts . T rav el and T ra n sp o rt u se d a variety o f re p u ta b le vend ors/ p rod u cts (hard w are a n d so ftw are ) to c re a te a cu ttin g -ed g e rep ortin g te c h n o lo g y s o th at th e ir clie n ts c a n m ak e b e tte r, faste r d e cis io n s to im p ro ve th e ir fin an cial w e ll-b e in g .

Source: Information Builders, Customer Success Story, i n f o r m a t i o n b u i l d e r s . c o m / a p p l i c a t i o n s / t r a v e l - a n d - tran sp ort (accessed February 2013).

Chapter 4 • Business Reporting, Visual Analytics, and B u sin ess P erform ance M anagem ent 1 6 9

4.2 B U S IN E S S REPO R T IN G D EFIN IT IO N S A N D CO N CEPTS D e c is io n m ak ers are in n e e d o f in form ation to m a k e a cc u ra te and tim ely d ecisio n s. In fo rm a tio n is e sse n tia lly th e co n tex tu a liz a tio n o f data. In fo rm a tio n is o ften provid ed in th e fo rm o f a w ritten r e p o r t (d igital o r o n p a p e r), alth o u g h it c a n a lso b e pro v id ed orally. S im p ly put, a re p o rt is an y co m m u n ica tio n artifact p re p a re d w ith th e s p e c ific in te n tio n o f co n v e y in g in form ation in a p re s e n ta b le fo rm to w h o e v e r n e e d s it, w h e n e v e r a n d w h ere v e r th e y m ay n e e d it. It is usu ally a d o cu m e n t th a t co n ta in s in form ation (u su ally d riven from d ata and p e rso n a l e x p e r ie n c e s ) o rg an ize d in a narrative, g ra p h ic, and/or tab u lar form , p rep ared p erio d ically (recu rrin g ) o r o n a n as-re q u ired (a d h o c ) b asis, referring to sp e cific tim e p e rio d s, e v e n ts, o c c u rre n c e s , o r su b jects.

In b u s in e s s settin g s, ty p e s o f rep o rts in clu d e m e m o s , m in u tes, lab rep orts, sales re p o rts, p ro g re ss rep orts, ju stificatio n rep orts, c o m p lia n ce rep orts, an n u al re p o rts, and p o licie s an d p ro ced u res. R ep orts ca n fulfill m any d ifferen t (b u t o fte n re lated ) Ju nctions. H e re are a fe w o f th e m o st p rev ailin g o n e s:

• T o e n s u re that all d ep artm en ts a rc fu n ctio n in g p ro p e rly • T o p ro v id e in form ation

1 7 0 Part II • D escriptive Analytics

• T o pro v id e th e results o f a n analysis • T o p e rsu a d e o th ers to act • T o c re a te a n organ ization al m em ory (a s p a n o f a k n o w le d g e m a n a g e m e n t sy stem )

R ep orts c a n b e len gth y at tim es. F o r th o s e rep orts, th e re u su ally is a n e x e cu tiv e su m m ary fo r th o se w h o d o n o t h a v e th e tim e and in te rest to g o th ro u g h it all. T h e sum m ary (o r ab stract, o r m o re co m m o n ly c a lle d e x e c u tiv e b rie f) sh o u ld b e crafted carefu lly, e x p re ssin g o n ly th e im portant p o in ts in a v e ry c o n c is e an d p re c ise m an n er, and lasting n o m o re th an a p a g e o r tw o.

In ad d ition to b u sin e ss re p o rts, e x a m p le s o f o th e r typ es o f rep o rts in clu d e crim e s c e n e rep orts, p o lic e rep orts, cre d it rep orts, s c ie n tific rep orts, re co m m e n d a tio n reports, w h ite p ap e rs, a n n u al rep orts, au d itor’s re p o rts, w o rk p la ce rep orts, c e n s u s rep orts, trip rep orts, p ro g re ss reports, investigative rep orts, b u d g et rep orts, p o licy rep orts, d em o g rap h ic rep orts, cre d it rep orts, ap p raisal re p o rts, in s p e c tio n rep orts, an d m ilitary rep orts, am o n g o th ers. In this ch a p te r w e are p articu larly in te reste d in b u sin e ss reports.

W h a t Is a B u s in e s s R e p o rt?

A b u sin e ss re p o rt is a w ritten d o cu m e n t th a t co n ta in s in form ation reg ard in g b u sin ess m atters. B u s in e s s re p o rtin g (a ls o ca lle d e n te rp rise rep o rtin g ) is a n e sse n tia l part o f the larg er drive tow ard im p ro ved m an ag erial d e c is io n m ak in g an d organ izatio n al k n o w le d g e m an ag e m e n t. T h e fo u n d a tio n o f th e se re p o rts is vario u s so u rce s o f data c o m in g from b o th in sid e an d o u tsid e th e o rgan ization . C reatio n o f th e s e rep o rts in v olv es ETL (e x tract, tran sform , and lo a d ) p ro ced u res in c o o rd in a tio n w ith a d ata w a reh o u se a n d th e n u sin g o n e o r m o re rep ortin g to ols. W h ile rep o rts c a n b e d istrib uted in p rin t fo rm o r via e-m ail, th e y a re ty p ically a c c e s s e d v ia a co rp o ra te intranet.

D u e to th e e x p a n sio n o f in fo rm atio n te c h n o lo g y co u p le d w ith th e n e e d fo r im proved co m p etitiv e n e ss in b u sin esses, th e re has b e e n a n in cre a se in the u se o f co m p u tin g p o w e r to p ro d u ce u n ified reports that jo in d ifferent v ie w s o f th e e n terp rise in o n e p la c e . Usually, this rep ortin g p ro c e s s in v olv es q u eryin g stru ctu red data s o u rc e s, m o st o f w h ich are created b y u sin g d ifferen t lo g ical d ata m o d e ls and d ata d ictio n aries to p ro d u ce a h u m an -read ab le, easily d ig estib le report. T h e s e typ es o f b u s in e s s rep o rts a llo w m an ag ers and co w o rk e rs to stay in fo rm ed and involved , re v iew o p tio n s an d alternativ es, an d m a k e in form ed d ecisio n s. Figu re 4.1 sh o w s th e c o n tin u o u s c y c le o f data acq u isitio n —» in form ation g e n era tio n d e cisio n m ak in g —> b u sin e ss p ro c e s s m an ag em en t. P erh ap s th e m o st critical task in this cy clic p ro ce s s is th e re p o rtin g (i.e ., in form ation g e n e ra tio n )— co n v ertin g data from d ifferen t so u rce s into a c tio n a b le in form ation.

T h e k e y to an y su cce ssfu l rep o rt is clarity, brevity, co m p le te n e ss , a n d co rrectn ess. In term s o f c o n te n t an d form at, th e re are o n ly a fe w ca te g o rie s o f b u s in e s s rep ort: infor­ m al, fo rm al, an d short. In fo rm al reports are u su ally u p to 10 p a g e s lo n g ; a re ro u tin e and in tern al; fo llo w a letter o r m e m o form at; an d u s e p e rso n a l p ro n o u n s and co n traction s. Form al reports are 10 to 1 0 0 p a g e s lo n g ; d o n o t u s e p e rso n al p ro n o u n s o r co n traction s; in clu d e a title p ag e , ta b le o f co n te n ts, an d a n e x e c u tiv e sum m ary; are b a se d o n d e e p re se a rch o r a n analytic study; and are d istrib u ted to e xte rn al o r internal p e o p le w ith a n e e d -to -k n o w d esig n ation . Short rep o rts a re to inform p e o p le a b o u t e v e n ts o r system status c h a n g e s an d are o ften p e rio d ic, in v estigative, c o m p lia n ce , an d situational focused.

T h e n atu re o f th e re p o rt a lso c h a n g e s sig n ifican tly b a s e d o n w h o m th e re p o rt is c re a te d for. M o st o f th e re se a rch in e ffe ctiv e rep ortin g is d ed icated to internal reports that in form stak e h o ld e rs an d d e c is io n m a k e rs w ithin th e o rgan ization . T h e r e are also e x te rn a l rep o rts b e tw e e n b u sin e sse s a n d th e g o v ern m e n t (e .g ., fo r ta x p u rp o se s o r for regular filings to th e S e cu rities a n d E x c h a n g e C o m m ission ). T h e s e form al rep o rts are m o stly stand ard ized a n d p e rio d ically file d e ith e r n atio n ally o r in ternationally. Standard B u s in e s s R ep ortin g , w h ic h is a c o lle c tio n o f in tern atio n al p ro g ram s instigated b y a

Chapter 4 • B u s i n e s s Reporting, Visual Analytics, and B u sin ess P erform an ce M anagem ent 171

i ransactional R e c o rd s Exception Event

S-.mbol Count Description

1 Machine

Failure

FIGURE 4.1 The Role of Information Reporting in Managerial Decision Making.

Action (decision)

(reporting)

Decision M ake r

B u s in e s s F un ction s

D ata |

n u m ber o f g o v ern m e n ts, aim s to re d u ce th e regulatory b u rd e n fo r b u sin ess b y sim plifying an d stand ard izing re p o rtin g req u irem en ts. T h e id ea is to m a k e b u sin e ss th e e p ic e n te r w h e n it co m e s to m an ag in g b u s in ess-to -g o v ern m en t rep ortin g ob lig atio n s. B u s in e s s e s co n d u ct th e ir o w n fin an cial ad m inistration; th e facts th e y re co rd and d e cis io n s th e y m ake 5h ou ld drive th eir rep o rtin g . T h e g o v ern m e n t sh o u ld b e a b le to re c e iv e and p ro ce s s tfus in form ation w ith o u t im p osin g u n d u e con strain ts o n h o w b u s in e s s e s ad m inister m e ir fin an ces. A p p lication C ase 4 .1 illustrates a n e x c e lle n t e x a m p le fo r o v e rco m in g th e

ch alle n g e s o f fin an cial reporting.

Application Case 4.1 Delta Lloyd Group Ensures Accuracy and Efficiency in Financial Reporting

to € 3 .9 billion and investm ents under m anagem entD elta Lloyd G ro u p is a financial services provider based in th e N etherlands. It offers insurance, p e n ­ sions, investing, an d b an k in g services to its private and corporate clients through its th ree strong brands: Delta Lloyd, OHRA, and ABN AiMRO Insurance. Since its founding in 1807, th e com p an y has grow n in th e N etherlands, G erm any, and Belgium , and n ow em ploys around 5 ,4 0 0 p erm anent staff. Its 2011 full-year financial reports sh o w €5-5 b illion in gross written prem ium s, w ith sharehold ers funds am ounting

w o rth nearly € 7 4 billion.

C h a lle n g e s

S in ce D elta Lloyd G ro u p is p u b licly listed o n the N Y SE E u ro n e xt A m sterdam , it is o b lig e d to p ro d u ce an n u al a n d half-y ear re p o rts. V ario u s su b sid iaries in D elta Lloyd G ro u p m u st a ls o p ro d u ce rep o rts to fulfill lo c a l legal req u irem en ts: fo r e x a m p le , b a n k in g and

0C on tin u ed )

1 7 2 Part II • D escriptive Analytics

Application Case 4.1 (Continued) in su ra n ce rep o rts a re o b lig a to ry in th e N etherlands. In ad d itio n , D elta Lloyd G ro u p m ust pro v id e reports to m e e t in tern ation al req u irem en ts, s u ch a s the IFR S (In tern atio n al F in an cial R ep orting Standards) fo r a c c o u n tin g and th e EU S o lv e n cy I D irectiv e for in su ra n ce co m p a n ie s. T h e data fo r th e s e rep o rts is g a th e re d b y th e g ro u p ’s fin a n ce d ep artm en t, w h ich is d iv id ed in to sm all te am s in sev eral lo catio n s, and th e n co n v e rte d in to XML s o th at it c a n b e p u b lish e d o n th e c o rp o ra te W e b site.

I m p o r t a n c e o f A c c u r a c y

T h e m o st ch a lle n g in g p art o f th e re p o rtin g p ro ce s s is th e “last m ile ”— th e sta g e a t w h ic h th e co n so lid a ted figu res a r e cited , form atted , an d d e s crib e d to fo rm th e fin al te x t o f the rep ort. D elta Lloyd G ro u p w as u sin g M icro so ft E x c e l fo r th e last-m ile stag e o f th e re p o rtin g p ro c e s s . T o m inim ize th e risk o f errors, th e fin a n c e team n e e d e d to m anu ally c h e c k all th e d ata in its rep o rts fo r accu racy . T h e s e m anu al c h e c k s w e re v e ry tim e -co n su m in g . Arnold H onig, te a m le a d e r fo r rep ortin g at D elta Lloyd G ro u p , co m m e n ts: “A ccu racy is e sse n tia l in financial rep o rtin g , s in c e errors co u ld lead to p en alties, rep u tatio n al d am ag e, and e v e n a n eg ativ e im p act o n th e c o m p a n y ’s s to c k p rice . W e n e e d e d a n e w so lu tio n th a t w'ould a u to m ate s o m e o f th e last m ile p r o c e s s e s a n d re d u ce th e risk o f m anu al e rro r.”

S o lu tio n

T h e g ro u p d e cid e d to im p lem en t IB M C o gnos Fin an cial Statem en t R ep ortin g (FSR ). T h e im p lem en ­ tation o f th e softw are w a s co m p le te d in ju st 6 w e e k s during th e late sum m er. T h is rapid im p lem en ta tio n g a v e th e fin a n c e d ep artm en t e n o u g h tim e to p re p are a trial d raft o f th e an n u al rep o rt in FSR, b a se d on figu res fro m th e third fin an cial q u arter. T h e s u c c e s s ­ ful cre a tio n o f this draft g a v e D elta Lloyd G roup e n o u g h c o n fid e n c e to u se C o g n o s FSR fo r th e final v e rsio n o f th e an n u al rep ort, w h ich w as p u b lish ed shortly a fte r th e e n d o f th e year.

R e s u lts

E m p lo y e es are d eligh ted w ith th e IBM C o g n os FSR solution. D elta Lloyd G ro u p has divided th e annual

rep o rt into ch a p ters, and e a c h m e m b e r o f th e report­ ing te a m is re sp o n sib le fo r o n e chapter. Arnold H onig says, “S in ce e m p lo y e e s c a n w o rk o n d ocu m en ts sim ultaneously, th e y ca n sh are th e h u g e w o rk load involved in rep o rt g eneration. B e fo re , th e reporting p ro cess w as in efficien t, b e c a u s e o n ly o n e p erson co u ld w o rk o n th e rep o rt at a tim e .”

S in c e t h e w o rk lo a d c a n b e d ivid ed u p , sta ff can co m p le te th e re p o rt w ith less ov ertim e. Arnold H on ig co m m e n ts, “P rev io u sly , e m p lo y e e s w e re pu tting in 2 w e e k s o f o v ertim e during th e 8 w e e k s req u ired to g e n era te a rep ort. T h is y e a r, th e 10 m e m b e rs o f sta ff : in v o lv ed in th e re p o rt g e n era tio n p ro c e s s w o rk e d 25 p e rc e n t le s s ov ertim e, e v e n th o u g h th e y w e r e still g etting used t o th e new' so ftw are. T h is is a b ig w in fo r D elta L loyd G ro u p a n d its staff.” T h e g ro u p is e x p e c tin g fu rth e r re d u ctio n s in e m p lo y e e ov ertim e I in th e future as sta ff b e c o m e s m o re fam iliar with th e softw are.

A c c u r a te R e p o r ts

T h e IB M C o g n o s FSR s o lu tio n a u to m a te s k e y sta g e s in th e re p o rt-w ritin g p r o c e s s b y p o p u la tin g th e fin al re p o rt w ith a ccu ra te , u p -to -d a te fin a n c ia l d ata. W h e re v e r th e te x t o f th e re p o rt n e e d s to m e n tio n a s p e c ific fin a n c ia l fig u re, th e fin a n c e te a m sim p ly in serts a “v a r ia b le ”— a tag th a t is lin k e d to a n u n d e r­ lyin g data s o u r c e . W h e re v e r th e v a ria b le a p p e a rs in th e d o c u m e n t, FSR w ill pu ll th e fig u re th ro u g h fro m th e s o u r c e in to th e re p o rt. I f th e v a lu e o f th e fig u re n e e d s t o b e ch a n g e d , th e te a m c a n sim p ly u p d a te it in th e s o u rc e , a n d th e n e w v a lu e w ill a u to m a tica lly flo w th ro u g h in to th e te x t, m a in ta in ­ ing a c c u ra c y a n d c o n s is te n c y o f d ata th ro u g h o u t th e rep ort.

A rnold H o n ig co m m e n ts, “T h e ability to u p d ate figu res au tom atically a cro ss th e w h o le report re d u ce s th e s c o p e fo r m an u al e rro r in h e re n t in s p re a d sh e e t-b a s e d p ro c e s s e s an d activities. S in ce : w e h av e full c o n tro l o f o u r rep ortin g p ro ce s s e s, w e ca n p ro d u c e b e tte r qu ality re p o rts m o re effi­ cien tly an d re d u c e o u r b u s in e s s risk .” IBM C o g n os FSR a lso p ro v id e s a c o m p a riso n featu re, w h ich hig hlights an y c h a n g e s m a d e to reports. T h is featu re m a k e s it q u ic k e r an d e a s ie r fo r users to re v ie w n e w v e rsio n s o f d o cu m e n ts an d e n su re th e a ccu ra cy o f th e ir reports.

Chapter 4 • B u sin ess Reporting, Visual Analytics, and B u sin ess P erform an ce M anagem ent 1 7 3

A d h e r in g t o I n d u s t r y R e g u la tio n s

in the future, D elta Lloyd G roup is p lanning to e xte n d a s u se o f IBM C o g n os FSR to g e n erate internal m an­ agem ent reports. It will also help D elta Lloyd G roup to m eet industry regulatory standards, w h ich are b eco m in g stricter. Arnold H on ig com m ents, “T h e EU Solvency II D irective will c o m e into effe ct s o o n , and ou r Solvency II rep o rts will n e e d to b e tagged w ith rX ten sib le B u sin ess Reporting L anguage [XBRL]. B y im plem enting IB M C o g n os FSR, w h ich fully supports XBRL tagging, w e hav e e q u ip p ed ou rselves to m eet b oth cu rrent an d future regulatory requ irem en ts.”

Q u e s t i o n s f o r D i s c u s s i o n

1. H o w d id D elta L loyd G ro u p im p ro v e a ccu ra cy a n d e ffic ie n c y in fin an cial reporting?

2. W h at w e re th e c h a lle n g e s, th e p ro p o s e d solu tio n , a n d th e o b ta in e d results?

3. W h y is it im p ortan t fo r D elta Lloyd G ro u p to c o m p ly w ith ind u stry regulations?

Source: IBM, Customer Success Story, “Delta Lloyd Group Ensures Accuracy in Financial Reporting,” public.dhe.ibm.com / com m on/ ssi/ ecm / en/ ytc0356ln len /Y TC 0356lN L EN .P D F (accessed Febmary 2013); and www.deltalloydgroep.com .

E v e n th o u g h th e re are a w id e variety o f b u sin ess re p o rts, th e o n e s th at are o ften _5ed fo r m a n a g e ria l p u rp o se s c a n b e g ro u p e d in to th re e m a jo r ca te g o rie s (H ill, 2 0 1 3 ).

VETRIC M A N A G E M E N T REPO RTS In m an y org an izatio n s, b u sin ess p e rfo rm a n ce is m n a g e d th ro u g h o u tc o m e -o rie n te d m etrics. F o r e x te rn a l g ro u p s, th e s e are serv ice-le v e l i^ re e m e n ts (SLAs). F o r internal m an ag e m e n t, th e y are k e y p e rfo rm a n ce in d icato rs (K P Is). 7;.p ically, th e re a re e n te rp rise-w id e a g re e d targets to b e tra ck e d o v e r a p e rio d o f tim e. They m ay b e u s e d as p art o f o th e r m a n a g e m e n t strategies s u c h as Six Sigm a o r T o tal Q uality M an ag e m e n t (T Q M ).

DASHBOARD-TYPE REPO RTS A p o p u la r id e a in b u sin ess rep ortin g in re c e n t y e ars has ~cen to p re sen t a ra n g e o f d ifferen t p e rfo rm a n ce in d icato rs o n o n e p ag e , lik e a d ash ­ board in a car. T y p ically , d ash b o ard v e n d o rs w o u ld pro v id e a s e t o f p re d e fin e d reports w ith static e le m e n ts a n d fix e d stru cture, b u t a lso allo w fo r cu sto m izatio n o f th e d ash b o ard v-. idgets, v iew s, a n d s e t targets fo r v arious m etrics. It’s co m m o n to h a v e c o lo r -c o d e d traf­ fic lights d e fin e d fo r p e rfo rm a n ce (re d , o ran g e, g re e n ) to draw m a n a g e m e n t a tte n tio n to particular areas. M o re details o n d ash b o ard s are g iv en later in this ch ap ter.

BA LA N C ED SC O REC A RD -TYPE REPO RTS Th is is a m e th o d d e v e lo p e d b y K a p la n and N orton that attem p ts to p re s e n t a n in teg rated v ie w o f s u c c e s s in a n org an izatio n . In addi- :io n to fin an cial p e rfo rm a n ce , b a la n c e d s c o re c a rd -ty p e reports a ls o in clu d e cu sto m er, bu sin ess p ro c e s s , a n d learn in g an d g ro w th p e rsp e ctiv es. M ore d etails o n b a la n c e d s c o r e ­ card s are pro v id ed later in this ch ap ter.

Components of the Business Reporting System A lthough e a c h b u s in e s s re p o rtin g sy stem h a s its u n iq u e c h a ra cte ris tics, th e r e s e e m s to b e a g e n e r ic p a tte rn th a t is c o m m o n a c ro ss o rg a n iz a tio n s an d te c h n o lo g y a rch ite ctu re s . T h in k o f th is g e n e r ic p a ttern as h a v in g th e b u s in e s s u s e r o n o n e e n d o f th e re p o rtin g co n tin u u m an d th e data s o u rc e s o n th e o th e r e n d . B a s e d o n th e n e e d s a n d re q u ire m e n ts o f th e b u s in e s s u s e r, th e d ata is ca p tu re d , sto re d , c o n s o lid a te d , a n d c o n v e rte d to d esire d re p o rts u s in g a s e t o f p re d e fin e d b u s in e s s ru les. T o b e s u c c e s s fu l, s u c h a sy stem n e e d s a n o v e r a rc h in g a s s u ra n c e p r o c e s s that c o v e r s th e e n tire v a lu e c h a in and m o v e s b a c k a n d fo rth , e n s u rin g that re p o rtin g re q u ire m e n ts a n d in fo rm a tio n d eliv e ry

1 7 4 Pari II • D escriptive Analytics

a re p ro p e rly a lig n e d (H ill, 2 0 0 8 ). F o llo w in g a r e th e m o st c o m m o n c o m p o n e n ts o f a b u s in e s s re p o rtin g sy stem .

• OLTP (o n lin e tra n s a c tio n p r o c e s s in g ) . A sy stem th at m e a su re s s o m e asp e ct o f th e real w o rld as ev en ts (e .g ., tra n sa ctio n s) an d re co rd s th e m in to en terp rise d atab ases. E x a m p le s in clu d e ERP sy stem s, P O S sy stem s, W e b servers, RFID read ers, h and held in v en tory read ers, card re ad ers, a n d s o forth.

• D a ta sup p ly . A sy stem th at ta k e s re c o rd e d events/ transactions and delivers th em reliab ly to th e rep ortin g system . T h e data a c c e s s c a n b e p u sh o r pull, d ep en d in g o n w h e th e r o r n o t it is re s p o n s ib le fo r initiating th e d elivery p ro ce ss. It c a n a lso b e p o lle d (o r b a tc h e d ) i f th e d ata are tran sferred p eriod ically , o r trigg ered (o r o nline ) i f data are tran sferred in c a s e o f a s p e cific ev en t.

• ETL (extra ct, tra n s fo rm , a n d lo a d ). T h is is th e in term ed iate step w h e re th e se re co rd ed transactions/events are c h e c k e d fo r quality, put into th e ap p rop riate form at, an d in serted in to th e d esire d data form at.

• D a ta storage. T h is is th e sto rag e area fo r th e data an d m etad ata. It co u ld b e a flat file o r a sp re a d sh ee t, b u t it is usu ally a relatio nal d atab ase m a n a g e m e n t sy stem (R D B M S) s e t u p as a data m art, d ata w a re h o u s e , o r o p era tio n a l data sto re (O D S ); it o fte n em p lo y s o n lin e analytical p ro ce s s in g (O LA P) fu n ctio n s lik e c u b e s.

• B u s in e s s logic. T h e e x p licit s te p s fo r h o w th e re co rd e d tran saction s/events are to b e co n v e rte d into m etrics, s co re ca rd s, an d d ash bo ard s.

• P u b lic a tio n . T h e sy stem th at builds th e v arious rep o rts a n d h o sts th e m (fo r u se rs) o r d issem in ates th e m (to u se rs). T h e s e system s m ay a lso pro v id e n otification , an n o tatio n , co lla b o ra tio n , an d o th e r serv ices.

• A s s u r a n c e . A g o o d b u sin e ss rep ortin g sy stem is e x p e c te d to o ffe r a quality s erv ice to its u sers. T h is in clu d e s d eterm in in g i f a n d w h e n th e right inform ation is to b e d eliv e red to th e right p e o p le in th e right way/format.

A p p licatio n C ase 4 .2 is a n e x c e lle n t e x a m p le to illustrate the p o w e r an d th e util­ ity o f au tom ated rep ort g e n era tio n fo r a larg e (a n d , at a tim e o f natu ral crisis, so m ew h at c h a o tic ) org an izatio n lik e FEMA.

Application Case 4.2 Flood of Paper Ends at FEMA Staff at th e F e d eral E m e rg e n cy M an ag em en t A g en cy (FEM A), a U .S. fed eral a g e n c y that co o rd in ates d isaster r e s p o n s e w h e n th e P resid en t d e cla re s a n atio n al d isaster, alw ays g o t tw o flo o d s at o n ce . First, w a te r co v e re d th e land. N ext, a flo o d o f p ap er, re q u ired to ad m inister th e N ational F lo o d In su ran ce P rogram (N F IP ), co v e re d th e ir d esks— pallets an d p alle ts o f g re en -strip ed reports p o u re d o f f a m ainfram e p rin te r a n d in to th e ir o ffice s. Individual rep o rts w e re so m e tim e s 18 in c h e s th ick , w ith a n u g g et o f in form ation a b o u t in su ran ce claim s, p rem ium s, o r p aym en ts b u ried in th e m s o m e w h ere .

B ill B a rto n and M ike M iles d o n ’t claim to b e a b le to d o an y th in g a b o u t th e w e a th e r, b u t th e

p ro je c t m a n a g e r a n d c o m p u te r scie n tist, resp ectiv ely , fro m C o m p u ter S c ie n c e s C o rp o ratio n (C SC ) h av e u s e d W eb F O C U S so ftw are fro m In fo rm ation B u ild ers to tu rn b a c k th e flo o d o f p a p e r g e n era ted b y th e NFIP. T h e p ro g ram allo w s th e g ov ern m en t to w o rk to g e th e r w ith n atio n al in su ra n ce co m p a n ie s to c o lle c t flo o d in su ra n ce p rem ium s a n d p a y claim s fo r flo o d in g in co m m u n ities that ad o p t flo o d co n tro l m e asu re s. As a result o f CSC’s w o rk , FEMA s ta ff no lo n g e r le a f th ro u g h p a p e r rep o rts to fin d th e data th e y n e ed . In ste a d , th e y b ro w se in su ra n ce data p o ste d o n N F IP ’s B u reau N et intranet site, s e le c t just th e in form ation th ey w an t to s e e , a n d g e t a n o n ­ s c re e n re p o rt o r d o w n lo a d th e data a s a sp read sh eet.

C hapter 4 • Business Reporting, Visual Analytics, and B u sin ess P erform an ce M anagem ent 1 7 5

And th at is o n ly th e start o f th e savin gs that W eb F O C U S h a s p ro vid ed . T h e n u m b e r o f tim es that NFIP staff ask s CSC fo r sp e cia l rep orts h a s d rop p ed in h alf, b e c a u s e NFIP sta ff c a n g e n era te m an y o f th e sp e cial rep o rts th ey n e e d w ith o u t callin g o n a p io - g ram m er to d e v e lo p th em . T h e n th e re is th e c o s t o f crea tin g B u re a u N et in th e first p la ce . B a ito n esti­ m ates th at u sin g co n v e n tio n a l W e b an d d atab ase so ftw are to e x p o rt d ata fro m FEMA’s m ainfram e, store it in a n e w d a ta b a s e , a n d lin k that to a W e b serv er w o u ld h av e c o s t a b o u t 1 0 0 tim es a s m u ch m o re th an $ 5 0 0 ,0 0 0 — an d ta k e n a b o u t tw o years to co m p le te , c o m p a re d w ith th e fe w m o n th s M iles sp e n t o n th e W e b F O C U S solu tion.

W h e n T ro p ical Storm Allison, a hu g e slug o f sod d en, sw irling cloud s, m oved o u t o f th e G u lf o f M exico o n to the T e x a s and Louisiana coastlin e in Ju n e 2001, it killed 34 p e o p le , m ost from drow ning; dam ­ aged o r d estroyed 16,000 h om es an d b u sin esses; and displaced m o re than 1 0 ,0 0 0 fam ilies. President G eorge W . B u sh d eclared 2 8 T e x a s cou n ties disaster areas, and FEMA m o ved in to help . This w as the first serious test fo r BureauN et, an d it delivered. T h is first com p re­ hensive u se o f B ureau N et resulted in FEMA field staff readily accessin g w h at they n e e d e d and w h en they

n e e d e d it, and ask in g fo r m an y n e w types o f reports. Fortunately, Miles an d W ebFO C U S w e re u p to the task. In so m e ca ses, B a ito n says, “FEMA w o u ld ask fo r a n e w typ e o f report o n e day, an d Miles w o u ld hav e it o n B ureau N et th e n ext day, thanks to th e s p e ed w ith w h ich h e cou ld create n e w reports in W ebF O C U S.”

T h e su d d en d em a n d o n th e sy stem had little im p act o n its p e rfo rm a n ce , n o te s B arton . "It h an d led th e d em an d ju st fin e ,” h e says. "W e h a d n o p ro b ­ lem s w ith it at all.” “A nd it m ad e a h u g e d iffe ren ce to FEMA an d th e jo b th e y h a d to do. T h e y had n e v e r had that lev el o f a c c e s s b e fo r e , n e v e r h ad b e e n a b le to ju st c lic k o n th eir d e s k to p and g e n e ra te su ch d eta iled and s p e cific re p o rts .”

Q u e s t i o n s f o r D i s c u s s i o n

1. W h a t is FEMA a n d w h a t d o e s it do? 2. W h a t are th e m ain c h a lle n g e s th a t FEMA faces? 3. H o w did FEMA im p ro v e its in e fficie n t rep orting

practices?

Sources: Information Builders, Customer Success Story-. Useful Information Flows at Disaster Response Agency,” informationbuilders.coni/applications/fema (accessed January 2013); and fema.gov.

SECTION 4 . 2 REVIEW QUESTIONS

1 . W h at is a report? W h a t are th ey u s e d for? 2 . W h at is a b u s in e s s report? W h a t are th e m ain ch aracteristics o f a g o o d b u sin e ss

report? 3 . D e s c r ib e th e c y c lic p ro ce s s o f m a n a g e m e n t a n d co m m e n t o n th e ro le o f b u sin e ss

reports. 4 . List a n d d e s c rib e th e th ree m a jo r ca te g o rie s o f b u sin e ss reports. 5. W hat are th e m a in co m p o n e n ts o f a b u sin e ss rep ortin g system ?

4.3 D A T A A N D IN FO R M A T IO N V IS U A LIZ A T IO N Data visualization (o r m o re ap p rop riately , in form ation visu alization ) h a s b e e n d efin e d i s . "th e u s e o f visu al re p re sen ta tio n s to e x p lo re , m a k e s e n s e o f, an d co m m u n ica te data Few, 2 0 0 8 ). A lth o u g h th e n a m e th at is co m m o n ly u s e d is d a t a v isu alization , usu ally

T,h a t is m e a n t b y th is is in form ation v isualization . S in ce in form ation is th e aggrega- ;:o n sum m arizations, a n d co n tex tu a liz a tio n o f data (raw fa cts ), w h a t is p o rtray ed in visualization s is th e in form ation a n d n o t th e data. H o w ev er, s in c e th e tw o te rm s d a ta i isu alization an d in fo rm a tio n v isu alization a re u se d in te rch a n g e a b ly a n d syn on ym ou sly, m this ch a p te r w e w ill fo llo w suit.

D ata v isu alization is clo se ly related to th e fields o f in form ation grap h ics, in form ation -isualization, s c ie n tific v isu alization , an d statistical grap h ics. U ntil re ce n tly , th e m ajo r

1 7 6 Part II • D escriptive Analytics

fo rm s o f data v isu alization a v a ilab le in b o th b u s in e s s in te llig e n ce a p p lica tio n s have in clu d e d ch arts an d g rap h s, as w e ll as the o th e r typ es o f visual e le m e n ts u s e d to create sco re ca rd s and d ash b o ard s. A p p licatio n C ase 4 .3 sh o w s h o w visual rep ortin g to o ls car. h elp facilitate co st-e ffe c tiv e b u sin e ss in form ation cre a tio n s an d sharing.

Application Case 4.3 Tableau Saves Blastrac Thousands of Dollars with Simplified Information Sharing B lastrac, a se lf-p ro cla im e d g lo b a l le a d e r in p o rtab le su rfa ce p re p a ra tio n te c h n o lo g ie s an d e q u ip m en t (e .g ., s h o t b lastin g , grinding, p o lish in g , scarifying, scrap in g , m illing, an d cu ttin g e q u ip m e n t), d e p e n d e d o n th e c re a tio n and distribution o f rep o rts a cro ss th e o rg an izatio n to m ak e b u sin e ss d ecisio n s. H ow ever, th e c o m p a n y did n o t have a co n siste n t rep orting m e th o d in p la c e and , c o n se q u e n tly , p rep aratio n o f re p o rts fo r th e co m p a n y ’s v arious n e e d s (s a le s data, w o rk in g ca p ita l, in ven tory, p u rch a se analysis, e tc .) w a s te d io u s. B la s tra c’s analysts e a c h s p e n t n early o n e w h o le d ay p e r w e e k (a to tal o f 2 0 to 3 0 h ou rs) e x tractin g d ata fro m th e m u ltiple e n te rp rise re so u rce p lan n in g (E R P ) sy stem s, lo ad in g it in to sev eral E x ce l sp re a d sh e e ts , creatin g filtering cap ab ilities an d esta b lish in g p re d efin e d p iv o t tab les.

N ot o n ly w e re th e s e m assiv e sp re ad sh ee ts o fte n in a ccu ra te an d co n siste n tly hard to u n d er­ stan d , b u t a lso th e y w e r e virtu ally u s e le s s fo r th e sa le s te a m , w h ich co u ld n ’t w o rk w ith th e co m p le x form at. In ad dition, e a c h c o n s u m e r o f th e reports h a d d iffe re n t need s.

B la s tra c V ice P resid en t a n d C IO D a n Murray b e g a n lo o k in g fo r a s o lu tio n to th e c o m p a n y ’s rep ort­ in g tro u b le s. H e q u ick ly rilled o u t th e ro llou t o f a sin gle ER P system , a m ultim illion-d ollar p ro p ositio n . H e a lso e lim in a te d th e p o ssibility o f a n en te rp rise- w id e b u sin e ss in te llig e n ce ( B I ) platform d ep lo y m en t b e c a u s e o f co st— q u o te s fro m five d ifferen t v e n ­ dors ran g ed fro m $ 1 3 0 ,0 0 0 to o v e r $ 5 0 0 ,0 0 0 . W h at M urray n e e d e d w as a so lu tio n that w as affo rd ab le, co u ld d e p lo y q u ick ly w ith o u t disru pting cu rren t sys­ tem s, an d w a s a b le to re p re sen t data co n siste n tly regard less o f th e m u ltiple cu rre n cie s B la stra c o p e r­ ates in.

T h e S o lu tio n a n d t h e R e s u lts

W o rk in g w ith IT s erv ices co n su ltan t firm , Interw orks, In c ., o u t o f O k la h o m a , M urray an d te a m fin e sse d

th e d ata s o u rc e s. M urray th e n d e p lo y e d tw o data visu alization to o ls from T a b le a u Softw are: T a b le a u D e s k to p , a visu al data analysis so lu tio n that allow ed B la stra c an aly sts to q u ick ly an d e asily cre a te intui­ tive and visually co m p ellin g rep orts, a n d T ab leau R ead er, a fr e e a p p lica tio n that e n a b le d ev e ry o n e a cro ss th e c o m p a n y to directly in teract w ith the rep orts, filtering, sorting, e xtractin g , an d printing data a s it fit th e ir n e e d s — an d a t a total c o s t o f less th an o n e-th ird th e lo w e s t co m p e tin g B I q u o te.

W ith o n ly o n e h o u r p e r w e e k n o w req u ired to cre a te rep o rts— a 95 p e rce n t in cre a se in p rodu ctiv­ ity— an d u p d a tes to th e s e reports h a p p e n in g auto­ m atically th ro u g h T a b le a u , M urray and h is te a m are a b le to p ro activ ely identify m a jo r b u sin ess events re fle cte d in c o m p a n y data— s u ch as a n e x c e p tio n ­ ally large s a le — in stead o f re a ctin g to in com in g q u e stio n s fro m e m p lo y e e s a s th e y h ad b e e n fo rced to d o previou sly.

“P rio r to d ep lo y in g T a b le a u , I s p e n t co u n tless h o u rs cu sto m iz in g an d cre a tin g n e w rep o rts b ase d o n individ ual re q u e sts, w h ic h w as n o t e ffic ie n t or pro d u ctiv e fo r m e ,” said Murray. “W ith T ab leau , w e c re a te o n e rep o rt fo r e a c h b u s in e s s are a, and, w ith very little training, th e y c a n e x p lo r e th e data th em selv es. B y d ep lo y in g T a b le a u , I n o t only saved th o u sa n d s o f d ollars an d e n d le ss m o nths o f d ep lo y m en t, b u t T’m a ls o n o w a b le to c re a te a p ro d u ct th at is infinitely m o re v a lu a b le fo r p e o p le acro ss th e o rg an ization .

Q u e s t i o n s f o r D i s c u s s i o n

1. H o w did B la stra c a ch ie v e sig n ifican t c o s t savin- g in re p o rtin g and in form ation sharing?

2. W h a t w e re th e ch a lle n g e , th e proposed, solu tio n , and th e o b ta in e d results?

Sources: tableausoftware.com/leam/stories/spotlight-blastric; blastrac.com/about-us; and interworks.com .

T o b e tte r u n d e rstan d th e cu rre n t an d future trend s in th e field o f data v isualization , it h elp s to b e g in w ith s o m e h istorical co n tex t.

A Brief History o f Data Visualization D esp ite th e fa c t th at p re d e ce s s o rs to data visu alization d ate b a c k to th e s e c o n d cen tu ry AD. m ost d ev elo p m en ts h av e o ccu rre d in th e last tw o and a h a lf ce n tu rie s, p red om inantly during th e last 3 0 y e a rs (F e w , 2 0 0 7 ). A lthough visu alization h a s n o t b e e n w id ely re co g n ize d as a d iscip lin e until fairly re ce n tly , to d a y ’s m o st p o p u la r visual fo rm s date b a ck a few cen tu ries. G e o g ra p h ica l e x p lo ratio n , m ath em atics, an d p o p u larized h istory spurred th e cre a tio n o f e a rly m aps, g rap h s, an d tim elin es as far b a c k as th e 1 6 0 0 s , but W illiam Playfair is w id ely cre d ite d as th e in v e n to r o f th e m o d e rn chart, h av in g cre a te d th e first w id ely d istrib u ted lin e an d b a r charts in his C o m m ercial an d P olitical A tlas o f I " 8 6 an d w h a t is g e n era lly co n sid e re d to b e th e first p ie chart in his Statistical B reviary , p u blished in 180 1 ( s e e Figure 4 .2 ).

P erh a p s th e m o st n o ta b le in n o v ato r o f in form ation g rap h ics d uring this p e rio d w as C harles J o s e p h M inard, w h o g rap h ically p ortrayed th e lo sses su ffered b y N a p o le o n s army m the Russian ca m p a ig n o f 181 2 (s e e Figu re 4 .3 ). B e g in n in g a t the P o lish -R u ssia n b o rd er, d ie th ick b a n d s h o w s th e siz e o f th e arm y a t e a c h p o sitio n . T h e p ath o f N a p o le o n ’s retreat fro m M o sco w in th e b itterly c o ld w in te r is d e p icte d b y th e dark lo w e r b a n d , w h ich ^ tied to te m p e ra tu re a n d tim e sca le s. P o p u la r visu alization e x p e rt, au thor, a n d critic

Chapter 4 • Business Reporting, Visual Analytics, and B u sin ess P erform ance M anagem ent 1 7 7

T he Bottom Inw is divided in to years, the R irfh t ha n d lin e into JLM A c t v * * Ig r W ? e it* v r a ir ... ..... -............................. ....................................... ..

: GURE 4.2 The First Pie Chart Created by William Playfair in 1801. Source: en.wikipedia.org.

1 7 8 Part II • D escriptive Analytics

FIGURE 4.3 Decimation of Napoleon's Army During the 1812 Russian Campaign. Source: en.wikipedia.org.

E dw ard T u fte says th a t this “m ay w ell b e th e b e s t statistical g rap h ic e v e r d raw n .” In this g rap h ic M inard m an ag ed to sim u ltan eou sly re p re s e n t sev eral data d im en sio n s (th e size o f th e arm y, d irectio n o f m o v em en t, g e o g ra p h ic lo ca tio n s , o u tsid e tem p eratu re, e tc .) in a n artistic a n d inform ative m an n er. M any m o re g re a t visu alization s w e re c re a te d in the 1 8 0 0 s, and m o st o f th e m a re ch ro n ic le d in T u fte ’s W e b site ( e d w a r d t u f t e .c o m ) and h i' v isu alization b o o k s .

T h e 1 9 0 0 s saw th e rise o f a m o re fo rm al, em p irical attitu de to w ard visualization, w h ich te n d e d to fo cu s o n a s p e cts su ch as co lo r, v a lu e sca le s, and lab e lin g . In th e mid- 1 9 0 0 s, carto g rap h e r and th e o rist Ja c q u e s B ertin p u b lish e d his S e m io lo g ie G raphique. w h ic h s o m e say serv e s as th e th e o re tica l fo u n d a tio n o f m o d e m in form ation visualization. W h ile m o st o f h is pattern s are e ith e r ou td ated b y m o re re c e n t re se a rch o r com p letely in a p p lica b le to digital m ed ia, m an y are still very relevan t.

In th e 2 0 0 0 s th e In te rn e t h a s e m e rg ed as a n e w m ed iu m fo r visualization and b ro u g h t w ith it a w h o le lo t o f n e w trick s and ca p a b ilitie s. N ot o n ly h a s th e w orldw ide, d igital distribution o f b o th d ata an d v isu alization m a d e th e m m o re a c c e s s ib le to a b ro ad er a u d ie n ce (raisin g visual lite ra cy a lo n g th e w ay ), b u t it h a s a lso spurred th e d esig n o f new fo rm s that in co rp o ra te in teractio n , anim ation, g ra p h ics-re n d e rin g te c h n o lo g y u n iq u e to s c re e n m ed ia, and real-tim e d ata fe e d s to cre a te im m ersiv e e n v iro n m en ts fo r com m u n i­ catin g and co n su m in g data.

C o m p anies a n d individuals are, seem in gly all o f a sudden, interested in data; that interest has, in turn, sp arked a n e e d fo r visual to o ls that h elp th em understand it. C h eap hard­ w are sen so rs an d d o-it-yo u rself fram ew orks fo r b uild ing you r o w n system are driving dow n th e co sts o f collectin g and p ro cessin g data. C ountless o th er applications, softw are tools, a n d low -level c o d e libraries are springing up to h e lp p e o p le co lle ct, organize, m anipulate, visualize, an d understand data fro m practically an y so u rce . T h e In ternet has also serv ed as a fantastic distribution ch a n n e l fo r visualizations; a d iv erse com m unity o f designers, program ­ m ers, cartographers, tinkerers, and data w o n k s h a s assem b led to dissem inate all sorts o f n e w ideas an d tools fo r w orking w ith data in b o th visual and nonvisual forms.

C hapter 4 • B u sin ess Reporting, Visual Analytics, and B u sin ess P erform an ce M anagem ent 1 7 9

G o o g le M aps has a ls o sin g le -h an d e d ly d em o cra tiz e d b o th th e in te rfa ce co n v e n tio n s click to p an , d o u b le -c lic k to z o o m ) a n d th e te ch n o lo g y (2 5 6 -p ix e l sq u are m ap tiles w ith

rre d icta b le file n a m e s ) fo r d isp layin g in teractiv e g e o g ra p h y o n lin e , to th e e x te n t that m ost p e o p le ju st k n o w w h a t to d o w h e n th e y ’r e p re s e n te d w ith a m ap o n lin e. F lash has served w ell as a c ro ss -b ro w s e r p latfo rm o n w h ic h to d esig n an d d e v e lo p rich, beau tifu l In tern et a p p lica tio n s in co rp o ra tin g interactiv e data visu alization an d m ap s; n o w , n e w b row ser-n ativ e te c h n o lo g ie s s u ch a s ca n v a s a n d SV G (s o m e tim e s co lle ctiv e ly in clu d e d under th e u m b rella o f H TM L5) a re e m e rg in g to c h a lle n g e Flash 's su p rem acy a n d e x te n d th e re a ch o f d y n am ic visu a lizatio n in te rfa ce s to m o b ile d ev ices.

T h e future o f data/inform ation visualization is very hard to pred ict. W e c a n o n ly extrapolate from w h at h a s alread y b e e n invented: m o re th ree -d im en sio n al visualization, m ore im m ersiv e e x p e rie n c e w ith m u ltid im ensional data in a virtual reality en v iron m en t, and h o lo g rap h ic visu alization o f inform ation. T h e re is a pretty g o o d c h a n c e that w e w ill s e e som ething that w e h av e n e v e r s e e n in th e inform ation visualization realm in v en ted b e fo re die en d o f this d e ca d e . A p p lication C ase 4 .4 sh o w s h o w D an a -F arb e r C an cer Institute u s e d inform ation visualization to b etter un derstand th e c a n c e r v a ccin e clinical trials.

Application Case 4.4 TIBCO Spotfire Provides Dana-Farber Cancer Institute with Unprecedented Insight into Cancer Vaccine Clinical Trials W h e n K aren M aloney, b u sin ess d ev elop m en t m anager o f the C an cer V accin e C en ter (C VC ) at D ana-Farb er C an cer Institute in B o sto n , d ecid ed to investigate the com petitive lan d scap e o f th e ca n ce r v accin e field, sh e lo o k e d to a strategic p lanning an d m arketing MBA class at B a b s o n C o llege in W ellesley, M assachusetts, fo r h e lp w ith th e re se a rch p ro ject. T h e re s h e m et X iao h o n g Cao, w h o s e b ioinform atics b ack g rou n d led to th e d ecisio n to fo cu s o n clinical v accin e trials as representative o f p otential com petition. T h is b e ca m e D ana-Farber CVC’s first organized attem pt to assess in-depth th e ca n ce r v accin e m arket.

C ao fo c u s e d o n th e an alysis o f 6 4 5 clin i­ cal trials related to c a n c e r v accin e s. T h e data w as e x tra cted in XML fro m th e C l i n i c a l T r i a l s .g o v W e b site, and in clu d e d c a te g o rie s s u ch as “Sum m ary o f P u rp o s e ,” “T rial S p o n s o r,” “P h a se o f th e T ria l,” “R e cm itin g S tatu s,” an d “L o ca tio n .” A dditional sta­ tistics o n c a n c e r ty p es, in clu d in g in cid e n ce a n d sur­ vival rates, w e re retriev ed fro m th e N ational C an cer Institute Su rveillan ce d ata.

C h a lle n g e a n d S o lu tio n

Although inform ation from clinical v accin e trials is organized fairly w ell in to categories an d ca n b e d ow n­ loaded, th ere is great in con sisten cy and redundancy

inherent in th e data registry. T o gain a g o o d under­ standing o f th e lan d scap e, b o th a n overview and an in-d ep th analytic capability w e re requ ired sim ul­ taneously. It w o u ld hav e b e e n very difficult, n o t to m en tion incredibly tim e-consu m ing, to analyze infor­ m ation from th e multiple d ata so u rces separately, in ord er to understand th e relationships underlying the data o r identify trends a n d patterns using spread ­ sh eets. And to attem pt to u s e a traditional business intelligence tool w o u ld h av e requ ired significant IT resources. C ao p ro p osed using th e T IB C O Spotfire D XP (Spotfire) com putational an d visual analysis tool for data exp loratio n an d discovery.

R e s u lts

W ith th e h e lp o f C ao a n d Sp o tfire softw are, D an a- F a rb e r’s CVC d e v e lo p e d a first-of-its-kind analysis a p p ro a ch to rapid ly e x tr a c t c o m p le x data sp e cifi­ cally fo r c a n c e r v a ccin e s fro m th e m a jo r clin ical trial rep ository . Su m m arizatio n and visu alization o f th e s e d ata re p re s e n ts' a c o s t-e ffe c tiv e m e a n s o f m ak in g in fo rm ed d e c is io n s a b o u t future c a n c e r v a c c in e clin ical trials. T h e find ings a re h e lp in g the CVC a t D a n a -F a rb e r u n d erstan d its co m p e titio n and th e d ise a se s th e y a re w o rk in g o n to h elp s h a p e its strategy in th e m ark etp lace.

0C on tin u ed )

1 8 0 Part II • D escriptive Analytics

Application Case 4.4 (Continued) Sp o tfire so ftw a re ’s visual a n d com p u tatio n al

analysis a p p r o a c h p ro v id es th e CVC at D an a-F arb er an d th e re s e a r c h co m m u n ity a t larg e w ith a b e t­ te r u n d erstan d in g o f th e c a n c e r v a ccin e clin ica l trials la n d s c a p e and e n a b le s rapid insight in to th e h o tsp o ts o f c a n c e r v a c c in e activity, as w e ll as into th e id e n tificatio n o f n e g le c te d can ce rs.

“T h e w h o le fie ld o f m e d ic a l r e s e a r c h is g o in g th r o u g h a n e n o r m o u s tra n sfo rm a tio n , in p art d riv e n b y in fo r m a tio n te c h n o lo g y ,” a d d s B ru s ic. “U sin g a to o l lik e S p o tfire fo r a n a ly s is is a p ro m ­ isin g a re a in th is fie ld b e c a u s e it h e lp s in te g ra te in fo r m a tio n fro m m u ltip le s o u r c e s , a s k s p e c ific q u e s tio n s , a n d ra p id ly e x tr a c t n e w k n o w le d g e

fro m th e d a ta th a t w a s p re v io u s ly n o t e a sily a tta in a b le .”

Q u e s t i o n s f o r D i s c u s s i o n

1. H o w did D a n a -F a rb e r C an cer Institute u se TIB C O Sp o tfire to e n h a n c e inform ation rep orting and visualization?

2. W h at w e re th e ch a lle n g e , th e p ro p o s e d solution, an d th e o b ta in e d results?

Sources: TIBCO Spotfire. Customer Success Story. “TIBCO Spotfire Provides Dana-Farber Cancer Institute with Unprecedented Insight into Cancer Vaccine Clinical Trials," spotfire.tibco.com/-/media/ content-center/case-studies/dana-farber.ashx (accessed March 2013); and Dana-Farber Cancer Institute, dana-farber.org.

SECTION 4 . 3 REVIEW QUESTIONS

1 . W h a t is d ata visualization? W h y is it need ed ?

2 . W h a t are th e h istorical ro o ts o f data visualization? 3 . C arefully an alyze C h arles Jo s e p h M inard’s g rap h ical portrayal o f N a p o le o n ’s m arch

Id en tify an d co m m e n t o n all o f th e in form ation d im en sio n s cap tu red in this an cien : diagram .

4 . W h o is E dw ard Tufte? W h y d o y o u th in k w e sh o u ld k n o w a b o u t h is work?

5. W hat d o y o u th in k th e “n e x t b ig th in g ” is in d ata visualization?

4.4 D IFFER EN T T Y P E S OF CH ARTS A N D G R A P H S O fte n e n d u sers o f b u sin e ss analytics system s a r e n o t su re w h at ty p e o f ch art o r graph to u se fo r a s p e cific p u rp o se . S o m e ch arts and/or grap hs a re b e tte r at an sw erin g certain ty p e s o f q u estio n s. W h at fo llo w s is a sh o rt d escrip tio n o f th e ty p e s o f charts and/or g rap h s co m m o n ly fo u n d in m o st b u sin ess an aly tics to o ls an d w h at ty p e s o f q u e stio n that th e y are b e tte r at answ ering/analyzing.

Basic Charts and Graphs W h a t fo llo w s are th e b a sic charts and graphs th a t are c o m m o n ly u s e d fo r inform ation visualization.

LIN E C H A R T Line ch arts a re p e rh a p s th e m o st freq u en tly u se d grap h ical visuals for tim e -serie s data. Line charts (o r lin e g rap h s) s h o w th e re latio n sh ip b e tw e e n tw o variables: th e y m o st o fte n are u se d to tra ck c h a n g e s o r tre n d s o v e r tim e (h av in g o n e o f th e vari­ a b le s s e t to tim e o n th e x -a x is ). Line charts se q u e n tia lly c o n n e c t individual d ata points to h e lp in fe r ch a n g in g trend s o v e r a p e rio d o f tim e. Line charts are o fte n u se d to show tim e -d e p e n d e n t c h a n g e s in th e v a lu e s o f s o m e m e a su re s u ch as c h a n g e s o n a sp ecific s to c k p rice o v e r a 5 -y e a r p e rio d o r c h a n g e s in th e n u m b e r o f d aily cu sto m e r serv ice calls o v e r a m onth.

Chapter 4 • Business R eporting, V isual Analytics, and B u sin ess P erform ance M anagem ent 181

BAR C H A R T B a r ch a rts are am o n g th e m o st b a sic visuals u s e d fo r data re p resen tatio n . B a r charts are e ffe ctiv e w h e n you h av e n o m in al data o r n u m erical d ata that splits n icely into d ifferen t c a te g o rie s s o y o u c a n q u ick ly s e e co m p arativ e results an d tren d s w ithin you r data. B a r charts a r c o fte n u s e d to co m p a re d ata acro ss m u ltiple ca te g o rie s s u c h as p e rce n t ad vertising s p e n d in g b y d ep artm en ts o r b y p ro d u ct cate g o rie s. B a i charts c a n b e vertically o r h o rizo n tally o rien ted . T h e y c a n a lso b e s ta ck e d o n to p o f e a c h o th e r to sh o w m ultiple d im en sio n s in a sin g le chart.

PIE C H A R T P ie ch arts are visu ally a p p e alin g , a s th e n a m e im p lies, p ie-lo o k in g charts. B e c a u s e th e y are s o visu ally attractive, th e y are o fte n in co rrectly used . P ie ch arts sh o u ld on ly b e u sed to illustrate relativ e p ro p o rtio n s o f a s p e cific m e asu re . F o r in sta n ce, th ey c a n b e u s e d to s h o w relativ e p e rc e n ta g e o f ad vertising b u d g et sp e n t o n d ifferen t p ro d u ct lines o r th e y c a n s h o w relativ e p ro p o rtio n s o f m a jo rs d ecla re d b y c o lle g e stu d en ts in their so p h o m o re y ear. I f th e n u m b e r o f ca te g o rie s to s h o w are m o re th an ju st a fe w (s a y , m o re than 4 ), o n e should, se rio u sly c o n s id e r u sin g a b a r ch a rt in stead o f a p ie chart.

S C A T TER P LO T Sca tte r p lots are o fte n u se d to e x p lo r e relatio n sh ip s b e tw e e n tw o or th ree v ariab les (in 2 D o r 2D visuals). S in ce th e y a re visual e x p lo ra tio n to o ls , having m ore th an th re e v a ria b le s, translating in to m o re th an th ree d im en sion s, is n o t easily ach iev a b le. Scatter p lo ts are a n e ffe ctiv e w a y to e x p lo r e th e e x is te n c e o f tren d s, c o n c e n ­ trations, an d ou tliers. F o r in sta n ce, in a tw o -v ariab le (tw o -a x is) grap h , a sca tte r p lo t can b e u s e d to illustrate th e co -rela tio n sh ip b e tw e e n a g e an d w e ig h t o f h eart d ise a se patients o r it c a n illustrate th e re latio n sh ip b e tw e e n n u m b e r o f cu sto m e r c a re re p re sen tativ es an d n u m ber o f o p e n cu sto m e r serv ice claim s. O fte n , a tren d lin e is su p e rim p o se d o n a tw o- d im en sion al scatter p lo t to illustrate th e natu re o f th e relatio nsh ip .

BU B BLE C H A R T B u b b le charts a re o fte n e n h a n c e d v e rsio n s o f sca tte r plots. T h e b u b b le chart is n o t a n e w visu alizatio n ty p e; in stead , it sh ou ld b e v ie w e d as a te ch n iq u e to e n rich data illustrated in s ca tte r p lo ts (o r e v e n g e o g ra p h ic m ap s). B y v arying th e s iz e and/ o r c o lo r o f th e circle s , o n e c a n ad d ad d itional d ata d im en sio n s, offerin g m o re e n rich e d m eaning a b o u t th e data. F o r in sta n ce, it c a n b e u se d to s h o w a co m p etitiv e v ie w o f co lle g e -lev el class a tte n d a n ce b y m a jo r an d b y tim e o f th e d ay o r it c a n b e u se d to sh o w profit m argin b y p ro d u ct ty p e an d b y g e o g ra p h ic reg io n .

Specialized Charts and Graphs T h e grap h s a n d ch arts th at w e re v iew in this s e c tio n a re e ith er d eriv ed fro m th e b a sic charts as s p e c ia l c a s e s o r th ey are relativ ely n e w an d s p e cific to a p ro b le m ty p e and/or an a p p lica tio n area.

H ISTO G RA M G ra p h ica lly sp e ak in g , a h istogram lo o k s ju st lik e a b ar chart. T h e d iffe re n ce b e tw e e n h istogram s a n d g e n e ric b a r ch arts is th e in form ation that is p o rtrayed in them . Histogram s a re u sed to s h o w th e fre q u e n c y d istrib u tion o f a v ariab le , o r sev era l v ariab les. In a histogram , th e x -axis is o ften u se d to s h o w th e ca te g o rie s o r ran g e s, and th e y -ax is is u sed to s h o w th e m easures/ values/ frequ encies. H istogram s sh o w th e d istributional shape o f th e data. T h a t w ay, o n e c a n visually e x a m in e i f th e d ata is d istrib uted n orm ally , e x p o n e n tially, a n d s o o n . F o r in sta n ce, o n e c a n u se a h istogram to illustrate th e e x a m p erfo rm an ce o f a c la ss , w h e re d istribution o f th e g rad es as w e ll a s co m p arativ e analysis o f individual results c a n b e sh o w n ; o r o n e c a n u s e a histogram to s h o w a g e distribution

o f th eir cu sto m e r b a s e .

1 8 2 Part II • D escriptive Analytics

G A N T T C H A R T G an tt charts are a sp e cia l c a s e o f h orizo n tal b a r charts that are u sed to p o rtray p ro je ct tim elin es, p ro je c t tasks/activity d urations, and ov erlap am o n g st th e tasks/ activities. B y sh o w in g start a n d e n d dates/tim es o f tasks/activities a n d th e ov erlap p ing relatio n sh ip s, G an tt ch arts m a k e a n in v alu ab le aid for m a n a g e m e n t an d co n tro l o f p ro je cts. F o r in sta n ce, G an tt ch arts a re o fte n u s e d to s h o w p ro je c t tim elin e, talk overlaps, relativ e ta sk co m p le tio n s (a partial b a r illustrating th e co m p le tio n p e rc e n ta g e in sid e a b a r th a t sh o w s th e actu al ta sk d u ratio n ), re s o u rc e s assig n e d to e a c h task, m ileston es, an d d eliverables.

PER T C H A R T P ER T ch arts (a ls o ca lle d n e tw o rk d iagram s) are d e v e lo p e d primarily to sim p lify th e p lan n in g a n d s ch e d u lin g o f large a n d c o m p le x p ro je cts. A P E R T ch a n sh o w s p r e c e d e n c e re latio n sh ip s am o n g th e p r o je c t activities/tasks. It is co m p o se d o f n o d e s (re p r e s e n te d as circle s o r re cta n g les) a n d e d g e s (re p re s e n te d w ith d irected arro w s). B a s e d o n th e s e le c te d P ER T c h a n co n v e n tio n , e ith e r n o d e s o r th e e d g e s may­ b e u se d to re p re se n t th e p ro je ct activities/tasks (a ctiv ity -o n -n o d e versu s activity-on-arrow re p re sen ta tio n sch e m a ).

G E O G R A P H IC M A P W h en th e d ata s e t in clu d es any k ind o f lo ca tio n d ata (e .g ., physical ad d resses, p o stal c o d e s , state n a m e s o r a b b re v ia tio n s, cou n try n am es, latitude/longitude, o r s o m e ty p e o f cu sto m g e o g ra p h ic e n c o d in g ), it is b e tte r an d m o re inform ative to s e e th e data o n a m ap. M aps u su ally are u s e d in co n ju n c tio n w ith o th e r c h a n s a n d graphs, as o p p o s e d to b y th em selv es. F o r in sta n ce, o n e c a n use m ap s to sh o w distribution o f cu sto m e r serv ice req u ests b y p ro d u ct typ e (d e p ic te d in p ie ch a rts) b y g e o g ra p h ic lo catio n s. O fte n a large variety o f in form ation (e .g ., a g e distribution, in c o m e distribution, ed u ca tio n , e c o n o m ic g ro w th , p o p u latio n ch a n g e s , e tc .) c a n b e p o rtrayed in a g eo g rap h ic m ap to h e lp d e cid e w h e re to o p e n a n e w restau ran t o r a n e w serv ice station. T h e s e types o f system s are o fte n ca lle d g e o g ra p h ic in fo rm atio n system s (G IS ).

B U LLE T B u lle t grap h s a re o fte n u se d to s h o w p ro g ress tow ard a g o a l. A b u lle t graph is essen tially a variation o f a b a r ch art. O fte n th e y are u s e d in p la c e o f g au g es, m eters, a n d th erm o m eters in d ash b o ard s to m o re intuitively co n v e y th e m e a n in g w ith in a m u ch sm aller sp a ce . B u lle t g rap h s co m p a re a prim ary m e asu re (e .g ., ye ar-to -d ate re v e n u e ) to o n e o r m o re o th e r m easu res (e .g ., an n u al re v e n u e target) and p re sen t this in th e co n te x t o f d efin e d p e rfo rm a n ce m etrics (e .g ., sale s q u o ta ). A b u lle t g rap h ca n intuitively illustrate h o w th e prim ary m e asu re is p erfo rm in g a g ain st ov erall g o als (e .g ., h o w c lo s e a sales re p re sen tativ e is to a ch iev in g his/her an n u al q u o ta ).

H E A T M A P H eat m ap s are g reat visuals to illustrate th e c o m p a riso n o f co n tin u o u s values a cro ss tw o ca te g o rie s u sin g co lo r. T h e g o al is to h e lp th e u s e r q u ick ly s e e w h e re th e in te rsectio n o f th e ca te g o rie s is stro n g est a n d w e a k e s t in term s o f n u m erical valu es o f th e m e asu re b e in g an aly zed . F o r in sta n ce, h e a t m a p s c a n b e u sed to s h o w seg m en tatio n analysis o f th e target m ark et w h e re th e m e a s u re (c o lo r grad ien t w o u ld b e th e p u rch ase a m o u n t) an d th e d im en sio n s w o u ld b e a g e an d in c o m e distribution.

H IG H LIG H T T A B L E H ighlight ta b le s a re in te n d e d to tak e h e a t m ap s o n e ste p further. In ad d ition to sh o w in g h o w data in te rsects b y u s in g co lo r, highlig ht ta b le s add a n u m ber o n to p to p ro v id e ad d itional detail. T h a t is, it is a tw o -d im en sio n al ta b le w ith cells p o p u la te d w ith n u m e rical v a lu e s a n d g rad ients o f co lo rs. F o r in sta n ce, o n e c a n sh ow s a le s re p re sen tativ e p e rfo rm a n ce b y p ro d u ct ty p e and b y s a le s v o lu m e.

T R E E M A P T r e e m a p s d isp la y h ie r a rc h ic a l (tr e e -s tru c tu re d ) d ata a s a s e t o f n e sted re c ta n g le s . E a c h b r a n c h o f th e tr e e is g iv e n a r e c ta n g le , w h ic h is th e n tile d w ith

sm alle r r e c ta n g le s r e p re s e n tin g s u b -b r a n c h e s . A l e a f n o d e ’s r e c ta n g le h a s a n a r e a p ro ­ p o rtio n a l to a s p e c ifie d d im e n s io n o n th e d a ta . O fte n th e l e a f n o d e s a re c o lo r e d to sh o w a s e p a ra te d im e n s io n o f th e d ata. W h e n th e c o lo r an d s iz e d im e n s io n s are co rrela te d in s o m e w a y w ith th e tre e stru ctu re, o n e c a n o fte n e a sily s e e p attern s th at w o u ld b e d ifficu lt t o s p o t in o th e r w a y s , s u ch a s i f a c e r ta in c o lo r is p a rticu la rly —lev a n t A s e c o n d a d v a n ta g e o f tre e m a p s is th at, b y c o n s tr u c tio n , th e y m a k e e ffic ie n t L se o f s p a c e . As a re s u lt, th e y c a n le g ib ly d isp la y th o u s a n d s o f ite m s o n th e s c r e e n

sim u ltan e o u sly . E v e n th o u gh th e s e charts a n d g rap h s c o v e r a m a jo r p art o f w h a t is co m m o n ly u sed

p i inform ation v isu alization , th ey b y n o m ean s c o v e r it all. N ow adays, o n e c a n find m any o th e r sp e cia liz e d g ra p h s a n d charts that serv e a s p e c ific p u rp o se . F u rth erm ore, cu rren t r a i d s are to com b in e/ h y b rid ize a n d an im ate th e s e charts fo r b e tte r lo o k in g a n d m o re intuitive visualization o f to d ay ’s c o m p le x a n d v o latile data so u rce s. F o r in s ta n ce , th e 'te r a c tiv e , an im ated , b u b b le ch arts av ailab le at th e G ap m in d e r W e b site ( g a p m in d e r .

o r g ) p ro v id e a n intriguing w ay o f e x p lo rin g w o rld h ealth , w ealth , an d p o p u la tio n data from a m u ltid im en sion al p e rsp e ctiv e. Fig u re 4 .4 d ep icts th e sorts o f displays av ailab le iz the site. In this grap h , p o p u la tio n size, life e x p e cta n cy , an d p e r cap ita in c o m e at the co n tin e n t le v e l are sh o w n ; also g iv en is a tim e-varying an im atio n th at s h o w s h o w th e se

variables c h a n g e d o v e r tim e.

C hapter 4 • Business Reporting, Visual Analytics, and B u sin ess P erform an ce M anagem ent 1 8 3

1000001 ooe 2 0 3 0 4 0 0 3 ' 10 3 0 0 2 0 000 0 0 0 0

Income per person (GDP/capita, PPP$ inflation-adjusted)

SG U B E 4.4 A Gapminder Chart That Shows Wealth and Health of Nations. Source: gapminder.org.

SECTION 4 . 4 REVIEW QUESTIONS

1 . W h y d o y o u th in k th e re are large n u m b e rs o f d iffe ren t ty p es o f ch arts an d graphs-

2 . W h at are th e m ain d iffe ren ce s am o n g lin e , b ar, an d p ie charts? W h e n shou ld y t * c h o o s e to u se o n e o v e r th e other?

3 . W h y ,w o u ld y o u u se a g e o g ra p h ic map? W h a t o th e r typ es o f charts c a n b e c o m b ir .r il w ith a g e o g ra p h ic map?

4 . Find tw o m o re charts that are n o t cov ered in this s ectio n , and co m m en t o n th eir usabil:7 I

4.5 THE E M E R G E N C E OF D A T A V IS U A L IZ A T IO N A N D V IS U A L A N A LY T IC S

As S eth G rim es ( 2 0 0 9 ) h a s n o te d , th e re is a “g ro w in g p a le tte ” o f data visualization te ch n iq u e s a n d to o ls that e n a b le th e u sers o f b u sin e ss analytics a n d b u sin e ss intelligence sy stem s to b e tte r “co m m u n ica te re latio n sh ip s, ad d h istorical co n te x t, u n c o v e r hidden c o rrela tio n s a n d te ll p ersu asive sto rie s that clarify an d call to a c tio n .” T h e latest Magic Q u ad ran t o n B u s in e s s In te llig e n ce and A nalytics P latform s re le a s e d b y G artn er in Febru ary 2 0 1 3 fu rth er em p h a siz es th e im p ortan ce o f visu alizatio n in b u sin e ss in te llig e n ce . As the ch art sh o w s, m o st o f th e so lu tio n providers in th e L ea d ers q u ad ran t are e ith e r relatively re ce n tly fo u n d e d in form ation v isu alization c o m p a n ie s (e .g ., T a b le a u Softw are, Q lik T e c L T ib c o Sp o tfire) o r are w e ll-e sta b lish ed , large analytics co m p a n ie s (e .g ., SAS, IBM. M icroso ft, SAP, M icroStrategy) th a t are in cre asin g ly fo cu sin g th e ir e ffo rts in inform ation visu alization a n d visual analytics. D etails o n th e G a rtn er’s latest M agic Q u ad ran t are given in T e c h n o lo g y Insigh ts 4 .1 .

1 8 4 Part II • D escriptive Analytics

T E C H N O L O G Y IN SIG H T S 4 . 1 G a r t n e r M a g ic Q u a d r a n t f o r B u s i n e s s I n te llig e n c e a n d A n a ly tic s P l a t f o r m s

Gartner, Inc., the creator o f Magic Quadrants, is a leading information technology research and advisory company. Founded in 1979, Gartner has 5,300 associates, including 1,280 research ana­ lysts and consultants, and numerous clients in 85 countries.

Magic Quadrant is a research method designed and implemented by Gartner to monitor and evaluate the progress and positions o f companies in a specific, technology-based market. By applying a graphical treatment and a uniform set o f evaluation criteria, Magic Quadrant helps users to understand how technology providers are positioned within a market.

Gartner changed the name o f this Magic Quadrant from “Business Intelligence Platforms to “Business Intelligence and Analytics Platforms” in 2012 to emphasize the growing importance of analytics capabilities to the information systems that organizations are now building. Gartner defines the business intelligence and analytics platform market as a software platfoim that delivers 15 capabilities across three categories: integration, information delivery, and analysis. These capabilities enable organizations to build precise systems of classification and measure­ ment to support decision making and improve performance.

Figure 4.5 illustrates the latest Magic Quadrant for Business Intelligence and Analytics platforms. Magic Quadrant places providers in four groups (niche players, challengers, visionaries, and leaders) along two dimensions: completeness o f vision Oc-axis) and ability to execute (j-ax is). As the quadrant clearly shows, most o f the well-known BI/BA providers are positioned in the “leaders” category while many o f the lesser known, relatively new, emerging providers are positioned in the “niche players” category'.

Right now, most o f the activity in the business intelligence and analytics platform market is from organizations that are trying to mam re their visualization capabilities and to move from descriptive to diagnostic (i.e., predictive and prescriptive) analytics. The vendors in the market have overwhelmingly concentrated on meeting this user demand. If there were a single market

Chapter 4 • B u sin ess Reporting, Visual Analytics, and Business P erform an ce M anagem ent 1 8 5

Challengers Leaders

Tableau Software Microsoft

^ __ QlikTech • — Oracle e IBM

•— S A S m MicroStrategy A Tibco Spotfire

Information Builders

■ S A P Proqnoz # Bifam • .

Board International Actuate

# Panorama Software T Alterys • |

Jaspersoft \ s a|jent Management Company Pentaho *

Targit 9 Arcplan •

GoodData

Niche players Visionaries ---------------------- 1 Completeness of vision [-------------- ►

A s of February 2 0 1 3

FIGURE 4.5 Magic Quadrant for Business Intelligence and Analytics Platforms. Source: gartner.com.

them e in 20 1 2 , it w o u ld b e that data discovery/visualization b ecam e a m ainstream arch itec­ ture For years’, data discovery/visualization vendors— su c h as Q lik T ech, Salient M anagem ent Company, T ableau Softw are, and T ib c o Spotfire— received m ore positive feed b a ck than vendors offering OLAP c u b e and sem antic-layer-based architectures. In 2012, th e m arket resp on t ed:

• MicroStrategy significantly improved Visual Insight. • SAP launched V isual Intelligence. • SAS launched Visual Analytics. • M icrosoft b o lstered Pow erPivot with P ow er View. • IBM launched C ognos Insight. • O racle acquired E ndeca. • Actuate acqu ired Quiterian.

T h is em phasis o n data discovery/visualization from m ost o f the leaders in the m arket— w hich are now prom oting tools w ith business-user-friendlv data integration, cou p led with em bed ded storage a n d com puting layers (typically in-m em ory/colum nar) and unfettered drilling— accelerates th e trend tow ard decentralization and u ser em pow erm ent o f B l and analytics, and greatly en a b le s organizations’ ability to perform diagnostic analytics.

Source: Gartner Magic Quadrant, released o n February 5, 2013. gartner.com (accessed February 2013).

In b u sin ess in te llig e n ce an d analytics, th e k e y ch a lle n g e s fo r v isu alization h av e e v o lv e d aro u n d th e intuitive re p re se n ta tio n o f large, c o m p le x data sets w ith m ultiple dimensions and m e asu re s. F o r th e m o st part, th e ty p ical charts, grap h s, an d o th e r visual d em ents u s e d in th e s e a p p lica tio n s usually involve tw o d im en sio n s, so m e tim e s three, m d fairly sm all s u b s e ts o f data sets. In co n trast, th e d ata in th e s e sy stem s re s id e in a

1 8 6 Part II • D escriptive Analytics

d ata w a re h o u se . At a m inim um , th e s e w a re h o u s e s in v o lv e a ran g e o f d im en sio n s (e.g.. p ro d u ct, lo ca tio n , organ izatio n al stru cture, tim e ), a ra n g e o f m e asu re s, and m illions or c e lls o f data. In a n e ffo rt to ad d ress th e s e c h a lle n g e s, a n u m b e r o f re sea rch ers have d e v e lo p e d a variety o f n e w visu alization te ch n iq u e s.

Visual Analytics Visual an aly tics is a re ce n tly c o in e d term th at is o fte n u se d lo o s e ly to m e a n n o th in g m ore th an inform ation visualization. W h at is m e a n t b y v i s u a l a n a l y t i c s is th e com bination o f visu alization a n d p red ictiv e analytics. W h ile inform ation v isu alization is aim ed ai an sw erin g “w h at h a p p e n e d ” an d “w h a t is h a p p e n in g ” and is clo s e ly a s s o cia te d with b u sin e ss in te llig e n ce (ro u tin e rep orts, s co re ca rd s , a n d d a sh b o a rd s), visual analytics if a im ed at a n sw e rin g “w h y is it h a p p e n in g ,” “w h a t is m o re lik e ly to h a p p e n ,” an d is usually a s s o cia te d w ith b u sin e ss an alytics (fo re ca stin g , seg m en tatio n , co rre la tio n analysis). M an y o f th e in form ation visualization v e n d o rs a re ad d in g th e cap ab ilitie s to call them ­ se lv e s visual an aly tics so lu tio n p rovid ers. O n e o f th e top, lo n g -tim e an alytics solution providers, SAS Institute, is a p p ro a ch in g it fro m a n o th e r d irection . T h e y are em bed d in g th e ir an alytics cap ab ilitie s in to a h ig h -p e rfo rm a n ce d ata v isu alization en v iro n m en t tha: th e y call visu al analytics.

V isual o r n o t visual, au to m ated o r m anu al, o n lin e o r p a p e r b a se d , b u sin e ss reporting is n o t m u ch d ifferen t th an telling a story. T e c h n o lo g y Insigh ts 4 .2 p ro v id es a different, u n o rth o d o x v ie w p o in t to b e tte r b u sin e ss reporting.

T E C H N O L O G Y IN S IG H T S 4 . 2 T e llin g G r e a t S t o r i e s w ith D a ta a n d V isu a liz a tio n

Everyone w h o has data to analyze has stories to tell, w hether it’s diagnosing th e reasons for m anufacturing defects, selling a n ew idea in a w ay that captures the im agination o f you r target au d ience, o r inform ing colleagu es about a particular custom er service im provem ent program. And w h en it’s telling the story behind a big strategic c h o ic e so that you and you r senior m an agem en t team can m ake a solid d ecision, providing a fact-based stoiy can b e especially challenging. In all cases, it’s a big jo b . Y o u w ant to b e interesting and m em orable; y o u know you need to k e e p it sim ple for you r busy execu tives and colleagues. Y et you also kn ow you have to b e factual, detail oriented, and data driven, esp ecially in today’s m etric-centric world.

It’s tem pting to presen t ju st the data and facts, but w hen colleagu es and sen ior m anage­ m ent are overw helm ed by data and facts w ithout con tex t, you lose. W e have all exp erienced presentations with large slide d ecks, on ly to find that th e au d ien ce is so overw helm ed with data that they d o n ’t k n o w w hat to think, o r they are so com p letely tuned out, they take aw'ay only a fraction o f th e k ey points.

Start engaging your execu tive team and explaining you r strategies and results more pow erfully by approaching your assignm ent as a story. Y ou will n eed th e “w hat” o f your story (th e facts and data) bu t you also n e e d the “w ho?,” the “how?,” the “why?," and the often missed “s o what?” It’s th ese stoiy elem ents that will m ake you r data relevan t and tangible for your audience. Creating a g o o d story ca n aid you and sen io r m an agem en t in focu sing o n what is important.

W h y S to r y ?

Stories bring life to data and facts. T h ey c a n help y o u m ake sen se and order out o f a disparate c ollectio n o f facts. T h ey m ak e it ea sie r to rem em ber k e y points and can paint a vivid picture o f w hat the future can lo o k like. Stories also create interactivity— p eo p le put them selves into stories and ca n relate to th e situation.

Chapter 4 • B u sin ess Reporting, Visual Analytics, and B u sin ess P erform an ce M anagem ent 1 8 7

Cultures have long used storytelling to pass o n kn ow led ge and content. In som e cultures, storytelling is critical to their identity. For exam p le, in New Zealand, som e o f the Maori p eo p le tattoo their faces w ith m o k u s . A m o k u is a facial tattoo con tain in g a story about ancestors the family tribe. A m an m ay h av e a tattoo design o n his fa c e that show s features o f a ham m erhead to highlight un iqu e qualities about his lineage. T h e design h e c h o o se s signifies w hat is part o f his “true s e lf ’ and his ancestral hom e.

Likewise, w hen w e are trying to understand a story, th e storyteller navigates to finding the “tn ie n orth .” I f sen ior m an agem en t is look in g to discuss how they w ill respond to a com petitive chang e, a good story can m ake sen se and ord er ou t o f a lot o f noise. For exam p le, you m ay have facts and data from tw o studies, o n e including results from a n advertising study and o n e from a product satisfaction study. D evelop ing a story for what you m easured across b o th studies can h elp p eo p le s e e th e w h o le w h ere th ere w ere disparate parts. F o r rallying your distributors around a new product, y o u c a n em ploy a story to give vision to what th e future ca n lo o k like. Most importantly, storytelling is interactive— typically the presenter uses w ords and pictures that audience m em bers c a n pu t them selves into. As a result, they b eco m e m ore eng aged and better understand the information.

So W hat Is a Good Story? M ost p eo p le ca n easily rattle o ff their favorite film or b o o k . O r they rem em ber a funny story that a colleagu e recently shared. W hy d o p eo p le rem em ber these stories? B ec a u se they contain certain characteristics. First, a g o o d story has great characters. In som e cases, th e read er or view er has a vicarious e x p erien c e w here they b eco m e involved with the character. T h e ch arac­ ter th en has to b e faced w ith a ch allen g e that is difficult but believable. Th ere must b e hurdles that th e ch aracter overcom es. And finally, the ou tcom e o r prognosis is clear by th e en d o f the story. T h e situation m ay n o t b e resolved— but the story has a clear endpoint.

Think o f Y o u r Analysis as a Story— Use a Story Structure W hen crafting a data-rich story, the first objective is to find the story. W ho are the characters? W hat is the drama o r challenge? W hat hurdles have to b e overcom e? And at the end o f you r story, w hat d o you w ant your au d ien ce to d o as a result?

O n c e you k n o w th e core story, craft your o th er story elem ents: d efine your characters, understand th e ch allen g e, identify th e hurdles, and crystallize the o u tco m e o r d ecisio n question. M ake su re y o u are c le a r w ith w hat y o u w ant p e o p le to d o as a result. This will sh ap e h o w your a u d ien ce w ill recall you r story. W ith th e story elem en ts in p lace, w rite out the storyboard, w hich rep resen ts th e structure and form o f your story. A lthough it’s tem pting to skip this step, it is b e tte r first to understand th e story you are telling and th en to fo cu s on th e p resentation structure and form . O n c e th e storyboard is in p lace, th e o th er elem en ts will fall into p lace. T h e storyboard will h elp you to think ab o u t th e b est analogies o r m etap h ors, to clearly s e t up ch allen g e o r opportunity, and to finally s e e the flow and transitions n eed ed . T h e storyboard also h elp s y o u fo cu s o n k ey visuals (graphs, charts, and grap hics) that y o u n eed your execu tives to recall.

In summary, d on’t b e afraid to u se data to tell great stories. B ein g factual, detail oriented, and data driven is critical in today’s m etric-centric world bu t it d oes not have to m ean b ein g b o r­ ing and lengthy. In fact, b y finding th e real stories in your data and follow ing th e b e st practices, you ca n get p eo p le to fo cu s o n your m essage— and thus o n w hat’s important. H ere are th o se b est practices:

1 . T h in k o f your analysis as a story— use a story structure. 2 . B e authentic— your story will flow. 3 . B e visual— think o f you rself as a film editor. 4 . M ake it easy for y o u r au d ien ce and you. 5 . Invite and direct discussion.

Sotirce: Elissa Fink and Susan J. Moore, “Five Best Practices for Telling Great Stories with Data,” 2012, white raper by Tableau Software, Inc., t a b l e a u s o f t w a r e . c o m / w h i t e p a p e r s / t e l l i n g - s t o r i e s - w i t h - d a t a (accessed February 2013).

1 8 8 Part II • D escriptive Analytics

High-Powered Visual Analytics Environm ents D u e to th e in creasin g d em an d fo r visual analytics c o u p le d w ith fast-grow ing data volum es, th ere is an e x p o n e n tia l m o v e m e n t tow ard investing in h ig h ly e fficie n t visualization system s. W ith th eir latest m o v e in to visual analytics, th e statistical softw are giant SAS Institute is n o w am o n g th e o n e s w h o are lead in g this w av e. T h e ir n e w p ro d u ct, SAS V isual Analytics, is a very h i g h - p e r f o r m a n c e , in -m em ory solu tion fo r e x p lo rin g m assive am o u n ts o f data in a v e ry sh o rt tim e (alm o st in stan tan eou sly). It e m p o w e rs users to sp o t pattern s, identify op p ortu nities fo r fu rther analysis, and co n v e y visual results via W e b reports o r a m o bile platform s u ch as tab lets a n d sm artp h on es. Figure 4 .6 s h o w s th e high-lev el arch itectu re of the SAS V isual A nalytics platform . O n o n e e n d o f th e arch itectu re, th e re are universal Data B u ild e r and A dm inistrator cap ab ilities, lead ing into E xp lo rer, R ep ort D esig n er, and M obile B I m o d u les, co lle ctiv e ly providing a n e n d -to -en d visu al analytics solution.

S o m e o f th e k e y b e n e fits p ro p o s e d b y SAS an aly tics are:

8 E m p o w e r all users w ith data e x p lo ra tio n te c h n iq u e s and ap p ro a ch a b le analytics to drive im proved d e cisio n m aking. SAS V isual A nalytics e n a b le s d ifferen t typ es o f users to co n d u ct fast, th o rou g h e xp lo ratio n s o n all a v ailab le data. Subsettin g o r sam pling o f data is n ot requ ired . E asy-to-u se, interactiv e W e b in terfaces b ro a d e n th e audi­ e n c e fo r analytics, e n a b lin g ev e ry o n e to g lea n n e w insights. U sers c a n lo o k a t m ore o p tio n s, m a k e m o re p re c ise d ecisio n s, an d drive s u cce s s e v e n faster th a n b efo re .

• A n sw er c o m p le x q u e stio n s faster, e n h a n cin g th e co n trib u tio n s fro m y o u r analytic talent. SAS V isual A nalytics au g m ents th e data d isco v e ry an d e x p lo ra tio n p ro ce s s b y pro vid ing e x tre m e ly fast results to e n a b le b etter, m o re fo c u s e d analysis. Analytically savvy u sers ca n identify a re a s o f op p o rtu n ity o r c o n c e r n fro m v ast am o u n ts o f data so fu rth er in vestig ation c a n ta k e p la c e quickly.

• Im p ro v e in form ation sh arin g an d co lla b o ra tio n . Large n u m bers o f users, including th o se w ith lim ited analytical skills, c a n q u ick ly v ie w an d in te ract w ith rep o rts and charts via th e W e b , A d o b e P D F files, and iPad m o b ile d ev ice s, w h ile IT m aintains co n tro l o f th e u n d erlying data a n d secu rity. SAS V isual A nalytics p ro v id es the right in form ation to the right p e rs o n at th e right tim e to im prove productivity and organ izatio n al k n o w le d g e.

FIG U R E 4 .6 A n O verview of SAS Visual Analytics Architecture. Source: SAS.com.

Chapter 4 • B u sin ess Reporting, Visual Analytics, and B u sin ess P erform an ce M anagem ent 1 8 9

F IG U R E 4 .7 A S c re e n sh o t fro m S A S V isu a l A n a ly tic s. Source; SA S.co m .

• L ib erate IT b y g ivin g u sers a n e w w a y to a c c e s s th e in form ation th e y n e e d . F ree IT fro m th e co n s ta n t b arrag e o f d em an d s fro m u sers w h o n e e d a c c e s s to d ifferen t am o u nts o f d ata, d ifferen t d ata v ie w s, ad h o c re p o rts, and o n e -o ff re q u e sts for inform ation. SAS V isual A nalytics e n a b le s IT to e asily load an d p re p are d ata fo r m ultiple u sers. O n c e d ata is lo a d e d an d av ailable, users ca n d ynam ically e x p lo r e data, c re a te re p o rts, a n d sh are in fo rm atio n o n th eir ow n.

• P rov id e ro o m to g ro w at a self-d e te rm in e d p a c e . SAS V isu al A nalytics p ro v id e s th e o p tio n o f u sin g co m m o d ity hard w are o r d a ta b a se a p p lia n ces fro m EMC G re e n p lu m and T e rad a ta. It is d esig n ed from th e ground up fo r p e rfo rm a n ce o p tim izatio n an d scalab ility to m e e t th e n e e d s o f an y size organization.

Figure 4 .7 s h o w s a s c re e n s h o t o f a n SAS A nalytics platform w h e re tim e-series p x eca stin g an d c o n fid e n c e intervals aro u n d th e fo re c a s t are d ep icte d . A w e a lth o f in for­ m ation o n SAS V isual A n aly tics, alo n g w ith a c c e s s to th e to o l itself fo r te a ch in g a n d learn - L p u rp o se s, c a n b e fo u n d at terad atau n iv ersityn etw o rk .com .

SECTION 4 . 5 REVIEW QUESTIONS

1. W hat a re th e re a s o n s fo r th e re c e n t e m e rg e n c e o f visual analytics? 2 . Look a t G artn er’s M agic Q u ad ran t fo r B u sin ess In te llig e n ce and A nalytics Platform s.

W h at d o you see? D iscu ss a n d ju stify y o u r o b serv ation s. 3 . W h at is th e d iffe re n ce b e tw e e n in form ation visu alization an d visual analytics?

-L W hy sh o u ld storytellin g b e a p art o f y o u r re p o rtin g and data visualization?

5 . W h at is a h ig h -p o w e re d visu al an alytics environm ent? W h y d o w e n e e d it?

1 9 0 Part II • D escriptive Analytics

4.6 P E R F O R M A N C E D A S H B O A R D S P erfo rm a n ce d ash b o ard s a re co m m o n c o m p o n e n ts o f m o st, i f n o t all, p e rfo rm a n ce m an­ a g e m e n t system s, p e rfo rm a n ce m e a su re m e n t sy stem s, BPM softw are su ites, a n d B I plat­ form s. D a s h b o a r d s pro v id e visual displays o f im p ortan t in form ation th at is co n so lid ated an d arran g ed o n a sin gle s c r e e n s o that in fo rm atio n ca n b e d ig e ste d at a sin g le glance an d e asily drilled in an d fu rth er e x p lo red . A ty p ical d ash b o ard is s h o w n in Figure 4.8. T h is particu lar e x e c u tiv e d ash b o ard displays a v arie ty o f K P Is fo r a h y p o th e tical softw are co m p a n y ca lle d S o n a tica (s e llin g au d io to o ls ). T h is e x e c u tiv e d ash b o ard sh o w s a high- lev el v ie w o f th e d ifferen t fu n ctio n al g ro u p s su rrou n d in g th e p ro d u cts, starting fro m a g e n e ra l o v erv iew to th e m arketin g effo rts, sa le s, fin a n ce , an d su p p o rt d ep artm ents. All o f this is in te n d e d to g iv e e x e c u tiv e d e cisio n m a k e rs a q u ick a n d a ccu ra te id ea o f w h at is g o in g o n w ith in th e o rgan ization . O n th e left sid e o f th e d ash b o rd , w e ca n s e e (in a tim e- se rie s fa sh io n ) th e q uarterly c h a n g e s in re v e n u e s, e x p e n s e s , a n d m argins, as w e ll as the c o m p a riso n o f th o se figures to p re v io u s y e a rs ’ m o n th ly nu m b ers. O n th e u p p er-right side w e s e e tw o dials w ith c o lo r-c o d e d reg io n s sh o w in g th e am o u n t o f m o n th ly e x p e n s e s for su p p o rt s erv ices (d ial o n th e left) a n d th e am o u n t o f o th e r e x p e n s e s (d ial o n th e right).

E x e c u tiv e D a s h b o a rd

S p e c i f y a d a t e r a n g e : jJ u n e , 2 0 0 9 j:^ » jJu ly , 2 0 1 0 | | ^

Sonatica '' s'fPl,. Turn it up.

□ Margin Margin (p re v io u s yea r) n Monthly Expense A verage M onthly Expense H igh Monthly Expense lo w

n 0 9 A u g 0 9 O ct 0 9 D e c 09 10 A p r 10

FIG U R E 4 .8 A Sample Executive Dashboard. Source: dundas.com.

Chapter 4 ♦ Business Reporting, Visual Analytics, and B u sin ess P erform ance M anagem ent 191

the co lo r co d in g in d icate s, w h ile th e m o n th ly su p p o rt e x p e n s e s a re w ell w ithin th e norm al ran g e s, th e o th e r e x p e n s e s a re in th e red (o r d ark er) re g io n , in d icatin g e x c e s s iv e " i l u e s . T h e g e o g ra p h ic m ap o n th e b o tto m right sh o w s th e d istrib u tion o f sa le s a t the country lev el th rou g h p u t th e w o rld . B e h in d th e s e g rap h ical ico n s th e re a re v arie ty o f m athem atical fu n ctio n s aggregatin g n u m erio u s d ata p o in ts to th eir h ig h est lev el o f m e a n - mgul figures. B y c lick in g o n th e s e grap h ical ico n s , th e c o n s u m e r o f this in fo rm atio n can drill d ow n to m o re gran u lar lev els o f in form ation a n d data.

D ash b o ard s are u s e d in a w id e variety o f b u s in e s s e s fo r a w id e variety o f re a so n s. For in stan ce, in A p p licatio n C ase 4 .5 , y o u will find th e su m m ary o f a su cce ssfu l im p le­ m entation o f in fo rm atio n d ash b o a rd s b y th e D allas C o w b o y s fo o tb a ll team .

Application Case 4.5 Dallas Cowboys Score Big with Tableau and Teknion F o u n d ed in I 9 6 0 , th e D allas C o w b o y s are a p ro ­ fe ssio n al A m erican fo o tb a ll team h ead q u artered in Irving, T e x a s . T h e te a m h a s a larg e n atio n al fo llow ing, w h ich is p e rh a p s b e s t re p re sen ted b y the NFL re co rd fo r n u m b e r o f co n s e c u tiv e g am e s at sold-ou t stadium s.

C h a lle n g e

Bill P riakos, C O O o f th e D allas C o w b o y s M erch an ­ dising D ivision, an d h is te a m n e e d e d m o re visibility into th eir data s o th e y co u ld run it m o re profitably. M icrosoft w a s s e le c te d as th e b a se lin e platform fo r this up grad e as w e ll as a n u m b e r o f o th er sales, log is­ tics, an d e -c o m m e r c e ap p licatio n s. T h e C o w b oys e x p e c te d that this n e w inform ation arch itectu re w ould pro v id e th e n e e d e d an aly tics an d reporting. U nfortunately, this w a s n o t th e ca s e , an d th e sea rch b e g a n fo r a ro b u st d ash bo ard in g , analytics, an d reporting to o l to fill this gap.

S o lu tio n a n d R e s u lts

T a b le a u an d T e k n io n to g e th e r pro v id ed real-tim e reporting and d ash b o ard cap ab ilitie s th at e x c e e d e d the C o w b o y s’ req u irem en ts. Sy stem atically an d m eth o d ically th e T e k n io n te a m w o rk e d sid e b y side w ith d ata o w n e rs an d data u sers w ithin th e D allas C o w b oy s to d e liv e r all req u ired fu nctionality, o n tim e a n d u n d e r b u d g et. “Early in th e p ro c e s s , w e w e re a b le to g e t a c le a r u n d erstan d in g o f w h at it w o u ld ta k e to ru n a m o re p ro fitab le o p e ra tio n fo r the C o w b o y s,” sa id T e k n io n V ice P resid en t B ill Luisi. “T h is p ro c e s s step is a k e y step in T e k n io n ’s a p p ro ach w ith a n y clie n t, and it alw ays pay s hu ge d ividends a s th e im p lem e n tatio n p lan p ro g re sse s.”

A dded Luisi, “O f c o u rs e , T a b le a u w o rk e d very c lo s e ly w ith u s and th e C o w b o y s during th e entire p ro je ct. T o g e th e r, w e m a d e su re that th e C o w b oy s c o u ld a ch ie v e th e ir re p o rtin g an d analy tical g o a ls in re co rd tim e .”

N ow , fo r th e first tim e, th e D allas C o w b oy s are a b le to m o n ito r th e ir co m p le te m erch an d isin g activities fro m m an u factu re to e n d cu sto m e r a n d s e e n o t o n ly w h at is h a p p e n in g a cro ss th e life c y cle , b u t drill d ow n e v e n fu rth er in to w h y it is h ap p en in g.

T o d ay , this B I s o lu tio n is u se d to re p o rt and an aly ze th e b u sin ess activ ities o f th e M erchand ising D iv ision, w h ich is re s p o n s ib le fo r all o f th e D allas C o w b o y s’ b ra n d sales. Industry' estim ates say that th e C o w b o y s g e n e ra te 2 0 p e rce n t o f all NFL m er­ ch a n d ise sales, w h ic h re fle cts th e fa ct th e y are th e m o st re co g n iz e d sports fra n ch is e in th e w orld.

A cco rd in g to E ric Lai, a Com puterW orld rep orter, T o n y R o m o a n d th e rest o f th e D allas C o w b o y s m ay h a v e b e e n o n ly av erag e o n th e fo o t­ b a ll field in the last fe w y e a rs, b u t o f f th e field , e sp ecia lly in th e m e rch an d isin g a re n a , th e y rem ain A m erica’s team .

Q u e s t i o n s f o r D i s c u s s i o n

1. H o w did th e D allas C o w b o y s u se inform ation visualization?

2. W hat w e re the c h a lle n g e , th e p ro p o se d solu tion, an d th e o b ta in e d results?

Sources: Tableau, Case Study, tableausoftware.com/learn/ stories/tableau-and-teknion-exceed-cow boys-requirem ents (accessed February 2013); and E. Lai, “B I Visualization Tool Helps Dallas Cowboys Sell More Tony Romo Jerseys,” ComputerW orld, October 8, 2009-

1 9 2 Part II • D escriptive Analytics

Dashboard Design D a s h b o a rd s a re n o t a n e w c o n c e p t. T h e ir ro o ts c a n b e tra c e d a t le a s t to th e E IS o f the 1 9 8 0 s . T o d a y , d a s h b o a rd s a r e u b iq u ito u s . F o r e x a m p le , a fe w y e a rs b a c k , F o rie s te i R e s e a r c h e stim a ted th a t o v e r 4 0 p e r c e n t o f th e la rg e s t 2 ,0 0 0 c o m p a n ie s in th e w orld u s e th e te c h n o lo g y (A n te a n d M cG reg o r, 2 0 0 6 ) . S in c e th e n , o n e c a n s a fe ly assum e th a t th is n u m b e r h a s g o n e up q u ite sig n ifica n tly . In fa c t, n o w a d a y s it w o u ld b e rather u n u su a l to s e e a la rg e c o m p a n y u s in g a B I s y s te m th a t d o e s n o t e m p lo y s o m e s o rt o; p e r fo r m a n c e d a s h b o a rd s . T h e D a s h b o a rd Sp y W e b s ite ( d a s h b o a r d s p y . c o m / a b o u t p ro v id e s fu rth e r e v id e n c e o f th e ir u b iq u ity . T h e s ite c o n ta in s d e s c rip tio n s a n d s c ie e n - s h o ts o f th o u s a n d s o f B I d a s h b o a rd s , s c o r e c a r d s , a n d B I in te rfa c e s u s e d b y b u s in e s s e s o f all s iz e s a n d in d u stries, n o n p ro fits , and g o v e r n m e n t a g e n c ie s .

A cco rd in g to E c k e r s o n (2 0 0 6 ), a w e ll-k n o w n e x p e rt o n B I in g e n eral a n d dash­ b o a rd s in p articu lar, th e m o st d istinctive fe a tu re o f a d ash b o ard is its th re e layers o f

inform ation:

1 . M o n it o r in g . G rap h ical, a b stra cte d data to m o n ito r k e y p e rfo rm a n ce m etrics. 2 . A n a ly s is . Su m m arized d im en sio n al data to an aly ze th e ro o t c a u se o f problem s. 3 . M a n a g e m e n t . D e ta ile d o p era tio n a l d ata th at id entify w h at actio n s to tak e to

re so lv e a p ro b lem .

B e c a u s e o f th e s e la y ers, d a s h b o a rd s p a c k a lo t o f in fo rm a tio n in to a sin gle s c r e e n . A cco rd in g to F e w ( 2 0 0 5 ) , “T h e fu n d a m e n ta l c h a lle n g e o f d a sh b o a rd d esig n is to d isp la y a ll th e re q u ire d in fo rm a tio n o n a sin g le s c r e e n , cle a rly a n d w itho u t d istra ctio n , in a m a n n e r th a t c a n b e a s s im ila te d q u ic k ly .” T o s p e e d a s s im ila tio n o f th e n u m b e rs , th e n u m b e rs n e e d to b e p la c e d in c o n te x t. T h is c a n b e d o n e b y c o m ­ p a rin g th e n u m b e rs o f in te re s t to o th e r b a s e lin e o r ta rg e t n u m b e rs , b y in d icatin g w h e th e r th e n u m b e rs are g o o d o r b a d , b y d e n o tin g w h e th e r a tre n d is b e tte r o r w o rse , a n d b y u s in g s p e c ia liz e d d isp lay w id g e ts o r c o m p o n e n ts to s e t th e c o m p a ra tiv e and

e v a lu a tiv e c o n te x t. S o m e o f th e c o m m o n c o m p a ris o n s th a t a re ty p ically m a d e in b u sin e ss

in te llig e n c e sy stem s in c lu d e c o m p a riso n s a g a in s t p a s t v a lu e s , fo r e c a s te d v alu es, ta rg e te d v a lu e s , b e n c h m a r k o r a v e ra g e v a lu e s , m u ltip le in s ta n c e s o f th e s a m e m e asu re , an d th e v a lu e s o f o th e r m e a s u re s ( e .g ., r e v e n u e s v e rsu s c o s ts ). In Figu re 4 .8 , th e v a rio u s K P Is a re s e t in c o n t e x t b y c o m p a rin g th e m w ith ta rg e te d v a lu e s, th e re v e n u e fig u re is s e t in c o n t e x t b y co m p a rin g it w ith m a rk e tin g c o s ts , a n d th e fig u re s for th e v a rio u s s ta g e s o f th e s a le s p ip e lin e are s e t in c o n t e x t b y co m p a rin g o n e stage

w ith a n o th e r . E v e n w ith co m p a ra tiv e m e a s u re s , it is im p o rta n t to s p e c ific a lly p o in t out

w h e th e r a p a rtic u la r n u m b e r is g o o d o r b a d a n d w h e th e r it is tre n d in g in th e right d ir e c tio n . W ith o u t th e s e s o rts o f e v a lu a tiv e d e s ig n a tio n s , it c a n b e tim e -c o n s u m in g to d e te rm in e th e statu s o f a p a rticu la r n u m b e r o r re su lt. T y p ic a lly , e ith e r s p e c ia liz e d v isu a l o b je c t s ( e .g ., tra ffic lig h ts ) o r v isu a l a ttrib u te s ( e .g ., c o lo r c o d in g ) a re u se d to s e t th e e v a lu a tiv e c o n t e x t. A gain , fo r th e d a s h b o a rd in F ig u re 4 .8 , c o lo r c o d in g ( o r v ary in g g ray to n e s ) is u s e d w ith th e g a u g e s to d e s ig n a te w h e th e r th e KP1 is g o o d o r b a d , a n d g r e e n u p a rro w s a r e u s e d w ith th e v a r io u s s ta g e s o f th e s a le s p ip e lin e to in d ic a te w h e th e r th e re su lts fo r th o s e s ta g e s a re tre n d in g u p o r d o w n a n d w h e th e r up o r d o w n is g o o d o r b a d . A lth o u g h n o t u s e d in this p a rtic u la r e x a m p le , a d d itio n al c o lo r s — red a n d o r a n g e , fo r in s ta n c e — c o u ld b e u s e d to r e p r e s e n t o t h e r s ta te s o n th e v a r io u s g a u g e s . A n in te re s tin g a n d in fo rm a tiv e d a s h b o a rd -d r iv e n re p o rtin g s o lu tio n b u ilt s p e c ific a lly fo r a v e ry la rg e te le c o m m u n ic a tio n c o m p a n y is fe a tu re d in

A p p lic a tio n C a s e 4 .6 .

Chapter 4 • B usiness Reporting, Visual Analytics, and B u sin ess P erform an ce M anagem ent 193

Application Case 4.6 Saudi Telecom Company Excels with Information Visualization Sup plying In te rn e t an d m o b ile serv ices to o v er 160 m illion cu sto m e rs a cro ss th e M iddle East, Saudi T e le c o m C o m p an y (ST C ) is o n e o f th e larg­ e st p ro vid ers in th e reg io n , e x te n d in g a s far as Africa a n d So u th A sia. W ith m illions o f cu stom ers c o n ta ctin g STC d aily fo r billing, p ay m en t, n etw o rk u sage, a n d su p p ort, all o f this in form ation has to b e m o n ito red so m e w h e re . L o cate d in the h ead q u arters o f STC is a d ata c e n te r th at featu res a s o c c e r field­ sized w all o f m o n ito rs all d isp lay in g in form ation regard ing n e tw o rk statistics, s e rv ic e analy tics, an d cu sto m e r calls.

T h e P r o b le m

W h e n y o u h av e a c re s o f in form ation in front o f y ou , prioritizing a n d co n tex tu a liz in g th e data are p aram ou n t in u n d erstan d in g it. STC n e e d e d to id e n ­ tify th e relevan t m etrics, p ro p e rly visu alize them , and pro v id e th e m t o th e right p e o p le , o fte n w ith tim e-sen sitive in form ation . “T h e e x e c u tiv e s d id n ’t h av e th e ab ility to s e e k e y p e rfo rm a n ce in d icato rs” said W aleed Al E shaiw y , m a n a g e r o f th e data ce n te r at STC. “T h e y w o u ld h a v e to c o n ta c t th e te ch n ica l team s to g e t status re p o rts. B y th at tim e, it w o u ld o ften b e to o late a n d w e w o u ld b e reactin g to p ro b ­ lem s rath er th a n p re v e n tin g th e m .”

T h e S o lu tio n

A fter carefu lly e v alu atin g sev eral v e n d o rs, STC m ad e d ie d e cisio n to g o w ith D u n d as b e c a u s e o f its rich data visu alization alternativ es. D u n d as b u sin ess in te llig e n ce co n su lta n ts w o rk e d o n -site in S T C s h ead qu arters in R iyadh to re fin e th e te le co m m u ­ nication d ash b o ard s s o th ey fu n ctio n e d properly. 'E v e n i f s o m e o n e w e r e to s h o w y o u w h a t w as in d ie d atab ase, lin e b y lin e , w ith o u t visualizing it, it w ould b e d ifficu lt t o k n o w w h a t w as g o in g o n ,” said W a lee d , w h o w o r k e d c lo s e ly w ith D u n d as c o n ­ sultants. T h e s u c c e s s th at STC e x p e rie n c e d led to en g ag em en t o n a n e n te rp rise-w id e , m ission -critical p ro ject to tran sform th e ir data c e n te r an d cre a te a m ore p ro activ e m o n ito rin g e n v iro n m en t. T h is p ro je ct cu lm inated w ith th e m o n ito rin g sy stem s in STC ’s

data c e n te r fin ally tran sform in g fro m reactiv e to p ro ­ active. Figure 4 .9 s h o w s a sam p le d ash b o ard fo r call c e n te r m an agem en t.

T h e B e n e f its

“D u n d a s’ in fo rm a tio n v isu a liz a tio n to o ls a llo w e d u s to s e e tren d s a n d c o r r e c t issu es b e fo r e th e y b e c a m e p r o b le m s ,” said Mr. E sh aiw y . H e ad d ed , “W e d e c r e a s e d th e a m o u n t o f s e r v ic e tick e ts b y 5 5 p e r c e n t th e y e a r th a t w e started u sin g th e in fo rm a tio n v isu a liz a tio n to o ls and d ash b o ard s. T h e availab ility o f th e s y s te m in c re a se d , w h ich m e a n t cu sto m e r s a tis fa c tio n le v e ls in c re a se d , w h ic h le d to a n in c re a se d c u s to m e r b a s e , w h ic h o f c o u rs e le a d to in c re a s e d r e v e n u e s .” W ith n e w , cu sto m K P Is b e c o m in g visu ally a v a ila b le to th e STC team , D u n d a s’ d a s h b o a rd s cu rre n tly o c c u p y n e a rly a q u a rte r o f th e s o c c e r fie ld -s iz e d m o n ito r w all. “E v ery th in g is o n m y s c r e e n , a n d I c a n drill d o w n an d fin d w h a te v e r I n e e d to k n o w ,” e x p la in e d W a le e d . H e a d d e d , “B e c a u s e o f th e d esig n and stru ctu re o f th e d a s h b o a rd s , w e c a n v e ry q u ick ly re c o g n iz e th e ro o t c a u s e o f th e p ro b le m s an d tak e ap p ro p ria te a c tio n .” A cco rd in g to Mr. E shaiw y, D u n d a s is a s u c c e s s : “T h e a d o p tio n ra te s are e x c e lle n t, it’s e a sy to u s e , a n d it’s o n e o f th e m o st s u c c e s s fu l p ro je c ts th a t w e h a v e im p lem e n te d . E v e n v isito rs w h o s to p b y m y o ffic e are g ra b b e d right a w a y b y th e lo o k o f th e d a s h b o a rd !”

Q u e s t i o n s f o r D i s c u s s i o n

1. W h y d o y o u th in k te le co m m u n ica tio n s c o m p a ­ n ie s are am o n g th e p rim e u se rs o f inform ation visu alization tools?

2. H o w did Saudi T e le c o m u se inform ation visualization?

3- W hat w e re th eir c h a lle n g e s, th e p ro p o s e d so lu ­ tio n , an d th e o b ta in e d results?

Source: Dundas, Customer Success Story, “Saudi Telecom Company Used Dundas’ Information Visualization Solution," dundas.com/wp-content/ uploads/ Saudi-Telecom-Company- C ase-Stu dyl.pdf (accessed February 2013).

0C on tin u ed )

1 9 4 Part II • D escriptive Analytics

Application Case 4.6 (Continued)

CALL CENTER DASHBOARD D u n d a s D a t a V i s u a l i z a t i o n I n c .

• r * , w tsuikww

AGENT

M «tkW ev«i He<n»one Grange*

FCRw(Nt«* Q T Oz. g 80% 92

0 2 4 6 3 i 10 A H T (tnins) 14 W 18

Aibus CMmbledore S -jv w s Snape

Miranda July Syivia Piath Billy CoHinj

Pablo Netuda

Ku«Vonnegot Richatd Btaulfcjan

C M Sagan

ColsnRrth OavWSHrigley

TF R S C ALLS RATE 30

.1 17 32 92

?2 16 28 89

f 12 29 81 11 31 81

" » IS 24 81

I 17 27 76 i 20 22

74

i 15 25 74

8 11 24 70

3 7 17 22 68

I 16 21 68 i 3 15 18 67

?J 10 15 65 4s 11 16 62 I s 14 61

(RATE ABANDON RATE

F IG U R E 4 .9 A Sam p le D ash b o a rd fo r C all C e n te r M a n a g e m e n t. Source: du n d a s.co m .

W hat to Look fo r in a Dashboard A lth o u gh p e rfo rm a n ce d ash b o ard s and o th e r in form ation v isu alization fram ew o rk s dif­ fe r in th e ir p u rp o se , th e y all sh a re s o m e c o m m o n d esig n ch aracteristics. First, th e y all fit w ith in th e larger b u sin e ss in te llig e n ce and/or p e rfo rm a n ce m e a su re m e n t s y s te rn This m e a n s th a t th e ir u n d erlyin g arch itectu re is th e B I o r p e rfo rm a n ce m an a g e m e n t arch itec- tu re o f th e larger sy stem . S e c o n d , all w e ll-d e sig n e d d ash b o ard a n d o th e r inform ation visu alization s p o ss e s s th e fo llo w in g ch a ra cteristics (N o vell, 2009)-

• T h e y u se visual co m p o n e n ts (e .g ., charts, p e rfo n n a n ce b ars, sparklines, gauges, m eters, stoplights) to highlight, a t a g lan ce, th e data an d e x ce p tio n s that require action.

• T h e y a re tran sp aren t to th e u ser, m e a n in g th at th e y re q u ire m inim al training a n d are

e x tre m e ly e a sy to u se. • T h e y co m b in e data fro m a variety o f sy stem s in to a sin g le , sum m arized , un itied

v ie w o f th e b u sin ess. • T h e y e n a b le d rill-d ow n o r d rill-through to un d erlying d ata so u rce s o r reports,

providing m o re d etail a b o u t th e u n d erly in g co m p arativ e an d evalu ative co n tex t. • T h e y p re s e n t a d yn am ic, re al-w o rld v ie w w ith tim ely data re fre sh e s, e n a b lin g the

e n d u ser to stay u p to d ate w ith an y re c e n t ch a n g e s in th e b u sin ess. • T h e y req u ire little, if any, cu sto m ized co d in g to im p lem en t, d e p lo y , a n d m aintain.

Chapter 4 • B u sin ess Reporting, Visual Analytics, and B usiness P erform an ce M anagem ent 1 9 5

B est P ra ctic e s in D a s h b o a r d D e s ig n

T h e re al e sta te say in g “lo c a tio n , lo ca tio n , lo c a tio n ” m a k e s it o b v io u s th at th e m o st im p o r­ tan t attribute fo r a p ie c e o f re al e state p ro p erty is w h e re it is lo cate d . F o r d ash b o ard s, it is “d ata, data, d a ta .” An o f te n o v e rlo o k e d a s p e ct, d ata is o n e o f th e m o st im p ortan t th in gs to c o n s id e r in d esig n in g d ash b o ard s (C aro ten u to , 2 0 0 7 ). E v e n if a d a s h b o a r d s a p p e a i- an c e lo o k s p ro fe ssio n a l, is a e sth etica lly p leasin g , an d in clu d es g rap h s a n d ta b le s cre a te d acco rd in g to a c c e p te d v isu a l d esig n stand ard s, it is a lso im p ortan t to ask a b o u t th e d ata: Is it reliable? Is it tim ely? Is a n y d ata m issing? Is it co n siste n t a cro ss all d ashboards? H e re are so m e o f th e e x p e rie n c e s -d riv e n b e s t p ra ctice s in d ash b o ard d esig n (R ad ha, 2 0 0 8 ).

B e n ch m a rk K e y P e rfo rm a n c e In d ic a to r s w it h In d u s tr y S ta n d a rd s

Many cu sto m e rs, a t s o m e p o in t in tim e, w an t to k n o w if th e m etrics th e y are m easu ring are th e right m etrics to m o n ito r. At tim es, m an y cu sto m ers h a v e fo u n d that th e m etrics th ey are track in g are n o t th e right o n e s to track. D o in g a gap a sse ssm e n t w ith industry b en ch m a rk s alig n s y o u w ith industry b e s t p ractices.

W rap t h e D a s h b o a rd M e tric s w it h C o n t e x t u a l M e ta d a ta

O fte n w h e n a re p o rt o r a visual d ash bo ard / sco recard is p re se n te d to b u sin e ss u se rs, m any q u e stio n s rem ain u n an sw e red . T h e fo llo w in g are s o m e exam p les:

• W h e re d id y o u s o u rc e this data? • W h ile lo ad in g th e data w a re h o u s e , w h a t p e rce n ta g e o f th e d ata g o t rejected /

e n c o u n te re d data q u ality problem s? • Is th e d ash b o ard p re sen tin g “fre sh ” in fo rm atio n o r “sta le ” inform ation? • W h e n w a s th e d ata w a r e h o u s e last refreshed? • W h e n is it g o in g to b e re fre sh e d next? • W e re a n y h ig h -v alu e tran sactio n s that w o u ld s k e w th e ov erall trend s re je c te d a s a

part o f th e lo a d in g process?

V a lid a t e th e D a s h b o a rd D e s ig n b y a U s a b ilit y S p e c ia lis t

In m o st d a sh b o a rd e n v iro n m en ts, th e d ash b o ard is d esig n e d b y a to o l sp e cia lis t w ith o u t giving co n sid era tio n to u sab ility p rin cip les. E ven th o u g h it’s a w e ll-e n g in e e re d data w areh o u se that c a n p e rfo rm w e ll, m an y b u sin e ss u sers d o n o t u s e th e d ash b o ard b e c a u s e it is p e rc e iv e d as n o t b e in g u s e r friend ly, lead in g to p o o r a d o p tio n o f th e infrastructure and c h a n g e m a n a g e m e n t issu es. U p fron t v alid ation o f th e d ash b o ard d esig n b y a u sab ility

sp ecialist c a n m itigate th is risk.

P rio ritiz e a n d R an k A le r ts /E x c e p tio n s S tre a m e d t o t h e D a sh b o a rd

B e c a u s e th e re are to n s o f raw data, it is im p ortan t to h av e a m e ch a n ism b y w h ic h im portant exce p tio n s/ b eh a v io rs are p ro activ ely p u sh e d to th e in form ation co n su m e rs. A b u sin ess ru le c a n b e co d ified , w h ic h d e te c ts th e alert p attern o f interest. It c a n b e cod ed in to a program , u sin g d a ta b a se -sto red p ro ced u res, w h ic h c a n craw l th ro u g h th e fact ta b le s an d d e te ct pattern s th at n e e d th e im m ed iate a tte n tio n o f th e b u sin e ss user. This w ay, in fo rm atio n fin d s th e b u sin e ss u s e r as o p p o s e d to th e b u s in e s s u s e r p o llin g th e fact ta b le s fo r o c c u rre n c e o f critical pattern s.

Enrich D a s h b o a rd w it h B u s in e s s U sers' C o m m e n ts

W h en th e sa m e d a sh b o a rd in form ation is p re se n te d to m u ltiple b u sin e ss users, a sm all lext b o x c a n b e p ro v id e d to cap tu re th e co m m e n ts fro m a n e n d -u s e r p e rsp e ctiv e. T h is can

1 9 6 Part II • D e s c rip tiv e Analytics „ w r.f n e r- o f t e n b e t a g g e d t o t h e d a s h b o a r d a n d p u t t h e in fo r m a tio n m c o n t e x t , a

s p e c tiv e to t h e s tru c tu re d K P Is b e m g r e n d e r e d .

P resent I n f o r m a t i o n in T h re e D iff e re n t LevelsPresent in ro rm a iiu ii ■!« ---------------- f ,

I n f o r m a t io n c a n b e p r e s e n t e d in t h r e e fh | s e lf - s e r v ic e c u b e i n fo r m a tio n ; t h e v is u a l d a s h b o a r d le: , & K p Is c a n b e p r e s e n te d ,

t e v e l. W h e n a u s e r w J t is n o t. w h ic h w o u ld give a s e n s e o f w h a t is g o i g

Pick th e Right V isu a l C onstruct U s in g D ash bo ard D e sig n P r i n c ip l e s

i n p resen tin g in form ation in a ^ ^ ^ “ “ ^ ^ ^ " n s ^ s c a T t e r plot charts, so m e w ith tim e -serie s lin e grap h s, a n , P Q n c e th e d ashboard s u s e k . So m etim es m erely re n d erin g it as sim p le a b k s s e ffe ^ ^ ^ ^

« £ - —

P ro v id e for G u id e d A n a ly t ic s .

In a ty p ical W n e s T u s 'e r in ord er

■ -

3 . S l T d ^ s X 't h e t e e fayers o f ^ ^ ^ f X r t “ o n visuals? ! 4 . W h a t a r e t h e c o m m o n c h a r a c t e n s t ic s f o r d a s h b o a r d s

5 . W h a t a r e t h e b e s t p r a c t i c e s in d a s h b o a r d d e s ig n .

4 7 B U S IN E S S P E R F O R M A N C E M A N A G E M E N T

th e b u s in e s s an d trad e literatu re, \ n u m b e r o f n a m e s , in clu d in g c o r p o ra te P * e n te rpriSe m a n a g e m e n t (SE M ). C P M w a s p e rfo rm a n ce m a n a g e m e n t CEP ) , a n * r tn e r .c o m " ) . EPM is a te rm a s s o c ia te d w ith c o in e d b y th e m ark e t a n aly st firm G a r t n ( g ^ is th e te rm that SAP ( s a p .c o m ) O r a c l e ’s ( o r a c l e .c o m ) o f f e r in g y e b e c a u s e it is t h e e a r lie s t, I u s e s , i n th is c h a p t e r , B P M is p r e f e r r e d o v e r t h e o t h e r « e r ^ ^ ^ ^ s in g l e _s o lu tio n

t h e m o s t g e n e r a lly u s e d , a n d m a n a g e m e n t ( B P M ) r e fe r s t o t h e b u s i n e s s 1 p r o v id e r . T h e t e r m b u s i n e s s p e r f o o n ^ S ^ e n t e r p r is e s to m e a s u r e ,

*— three key components ]

s. SKfL“ SSTd,“ ®«s. ‘,,d p e r f o r m a n c e a g a in s t t h o s e g o a l s o p e r a tio n a l p la n n in g , c o n s o lid a tio n

3 ' r d° : X P m « “ s , a n d m o m t o n n g o f k e y p e r f o r m a n c e in d ic a to r s

(K P Is), lin k e d to organ izatio n al strategy

Chapter 4 • B u sin ess Reporting, Visual Analytics, and B u sin ess P erform an ce M anagem ent 1 9 7

FIGURE 4.10 Closed-Loop BPM Cycle Source: Business Intelligence, 2e.

Closed-Loop BPM Cycle M aybe th e m o st s ig n ifica n t d iffe ren tia to r o f B P M fro m an y o th e r B I to o ls an d p ra c tic e s is its strateg y fo c u s . BPM e n c o m p a s s e s a c lo s e d -lo o p s e t o f p r o c e s s e s that lin k strategy to e x e c u tio n in o r d e r to o p tim iz e b u s in e s s p e rfo rm a n ce (s e e Fig u re 4 .1 0 ). T h e lo o p im p lies th a t o p tim u m p e rfo rm a n c e is a c h ie v e d b y settin g g o a ls and o b je c tiv e s ( i .e ., strat- e g iz e ), e sta b lish in g initiatives a n d p lan s to a ch ie v e th o se g o a ls ( i.e ., p la n ), m o n ito rin g actu al p e rfo rm a n ce a g a in st th e g o a ls and o b je c tiv e s (i.e ., m o n ito r), an d tak in g co r re c tiv e actio n ( i.e ., a c t a n d ad ju st). T h e c o n tin u o u s an d re p e titiv e n atu re o f th e c y c le im p lies that th e co m p le tio n o f a n itera tio n lead s to a n e w and im p ro v ed o n e (su p p o rtin g c o n ­ tinues p r o c e s s im p ro v e m e n t e ffo rts). In th e fo llo w in g s e c tio n th e s e fo u r p r o c e s s e s are d escrib ed .

1 . S t r a t e g iz e : W h ere d o w e w a n t to g o ? Strategy, in g e n e ra l term s, is a h igh - level p la n o f a ctio n , e n c o m p a ss in g a lo n g p e rio d o f tim e (o fte n sev eral y e a rs ) to a c h ie v e a d efin ed g oal. It is e sp e c ia lly n e c e s s a ry in a situ ation w h e re th e re are nu m erou s con strain ts (d riven b y m ark e t co n d itio n s, re s o u r c e availabilities, an d legal/political a lteratio n s) to d eal w ith o n th e w a y to a ch ie v in g th e g oal. In a b u sin e ss setting, strategy is th e art a n d th e s c ie n c e o f craftin g d e c is io n s th a t h e lp b u s in e s s e s a ch ie v e th e ir goals. M ore s p e cifica lly , it is th e p ro c e s s o f id entifying an d stating th e o rg an izatio n ’s m issio n , vision, an d o b je c tiv e s , and d e v e lo p in g p la n s (a t d ifferen t lev els o f granularity— strategic, tactical, a n d o p e ra ­ tion al) to a ch ie v e th e s e o b je ctiv e s.

B u s in e s s strate g ies a re norm ally p la n n e d a n d cre a te d b y a te a m o f co rp o ra te e x e cu tiv es (o fte n le d b y th e C E O ), a p p ro v e d an d au th o rized b y th e b o a rd o f d irecto rs, and th e n im p le m e n te d b y th e c o m p a n y ’s m an a g e m e n t te a m u n d er th e su p e rv isio n o f th e

1 9 8 Part II • D escriptive Analytics

s e n io r e x e cu tiv es. B u s in e s s strategy p ro v id e s a n o v erall d irectio n to th e e n terp rise and is th e first an d fo re m o st im p ortan t p ro c e s s in th e B P M m eth o d o lo g y .

2. P la n : H ow do w e g e t th ere? W h e n o p e ra tio n a l m an ag ers k n o w a n d u n der­ stan d th e w h at (i.e ., th e organ izatio n al o b je c tiv e s a n d g o a ls), th e y w ill b e a b le to com e u p w ith th e h ow ( i.e ., d eta iled o p era tio n a l and fin a n cia l p lan s). O p e ra tio n a l an d financial p lan s a n s w e r tw o q u e stio n s: W h a t ta ctics and initiatives w ill b e p u rsu ed to m e e t th e p e r­ fo rm a n ce targ ets e sta b lish e d b y th e strategic plan? W h a t a re th e e x p e c te d fin an cial results

o f e x e cu tin g th e tactics? An o p e ra tio n a l p la n tran slates a n o rg a n iz a tio n ’s strateg ic o b je c tiv e s and g o a ls into

a s e t o f w e ll-d e fin ed ta ctics a n d initiatives, re s o u r c e req u irem en ts, and e x p e c te d results fo r s o m e fu tu re tim e p eriod , usually, b u t n o t alw ay s, a y ear. In e s s e n c e , a n op eration al p lan is lik e a p ro je c t p la n th at is d esig n e d to e n s u re th at a n org an izatio n s strategy is realized . M o st o p eratio n al p lan s e n c o m p a ss a p o rtfo lio o f ta ctics an d initiatives. T h e key to s u cce s s fu l o p era tio n a l p lan n in g is in tegratio n. Strategy drives tactics, an d ta ctics drive results. B asically , th e ta ctics a n d initiatives d efin e d in a n o p era tio n a l p lan n e e d to b e d irectly lin k e d to k e y o b je c tiv e s and targ ets in th e strateg ic plan. I f th e re is n o linkage b e tw e e n a n individual ta ctic a n d o n e o r m o re strate g ic o b je c tiv e s o r targets, m an ag em en t sh o u ld q u e stio n w h e th e r th e tactic an d its a s s o cia te d initiatives are really n e e d e d at all. T h e BPM m e th o d o lo g ie s d iscu sse d la te r in this ch a p te r are d esig n e d to e n su re th at th ese

lin k ag e s exist. T h e fin an cial p la n n in g a n d b u d g etin g p r o c e s s h a s a lo g ica l stru ctu re that typically

starts w ith th o s e ta ctics that g e n e r a te s o m e fo rm o f re v e n u e o r in co m e . In o rg an ization s th a t sell g o o d s o r serv ice s, th e ability to g e n e ra te re v e n u e is b a s e d o n e ith e r th e ability to d irectly p ro d u ce g o o d s a n d s e rv ice s o r a cq u ire th e righ t am o u n t o f g o o d s and s e rv ice s to sell. A fter a re v e n u e figu re h a s b e e n e sta b lish e d , th e a s s o c ia te d c o s ts o f d eliv erin g th at lev el o f re v e n u e c a n b e g e n e ra te d . Q u ite o fte n , this en tails in p u t from s e v e ra l d ep artm e n ts o r tactics. T h is m e a n s th e p r o c e s s h a s t o b e co lla b o ra tiv e a n d that d e p e n d e n c ie s b e tw e e n fu n ctio n s n e e d to b e cle a rly co m m u n ica te d a n d u n d e rsto o d . In ad d itio n to th e co lla b o ra tiv e input, th e o rg a n iz a tio n a lso n e e d s to ad d vario u s o v erh ea d c o s ts , as w e ll as th e co s ts o f th e cap ital requ ired . T h is in fo rm atio n , o n c e co n so lid ated , s h o w s th e c o s t b y ta c tic as w e ll a s th e c a s h a n d R in d in g re q u irem en ts to p u t th e plan in to o p era tio n .

3. M o n ito r/A n a ly ze: H ow a r e we d o in g ? W h e n th e o p eratio n a l a n d finan­ cial p lan s are un derw ay, it is im perative th at th e p e rfo rm a n ce o f th e organ ization b e m o n ito red . A co m p re h e n siv e fram ew ork fo r m o n ito rin g p e rfo rm an ce sh o u ld ad d ress tw o k e y issues: w h at to m o n ito r a n d h o w t o m onitor. B e c a u s e it is im p o ssib le to lo o k a t e very­ thing, a n org an izatio n n e e d s to fo cu s o n m o n ito rin g sp e cific issu es. After th e organization h a s id entified th e in d icato rs o r m easu res to lo o k at, it n e e d s to d ev elo p a strategy fo r m o n ­ itoring th o se facto rs a n d resp o n d in g effectiv ely . T h e s e m e a su re s are m o st o fte n ca lle d k e y p e rfo rm a n ce ind icators (o r KPI, in short). An o v erv iew o f th e p ro ce s s o f d eterm ining KPI is g iv en later in this ch ap ter. A related to p ic to th e s e le c tio n o f th e optim al s e t o f K P Is is th e b a la n c e d s co re ca rd m eth o d , w h ic h w ill a lso b e co v e re d in d etail la te r in this chapter.

4 . A ct a n d A d ju st: What do w e n e e d to d o d iffe r e n t ly ? W h e th e r a co m p a n y is in terested in g ro w in g its b u sin ess o r sim ply im p ro vin g its o p era tio n s, virtually all strategies d e p e n d o n n e w p ro je cts— crea tin g n e w p ro d u cts, e n terin g n e w m arkets, acq u irin g new c u sto m e rs o r b u sin esses, o r stream lining so m e p ro c e s s e s. M ost co m p a n ie s a p p ro a ch th ese n e w p ro je cts w ith a spirit o f op tim ism rath er th an ob jectivity, ign orin g th e fa ct th at m ost n e w p ro je cts and v en tu res fail. W h a t is th e c h a n c e o f failure? O bv io u sly , it d ep en d s o n the typ e o f p ro je c t (Sly w o tzky a n d W e b e r, 2 0 0 7 ). H o lly w o o d m o v ies h av e aro u n d a 6 0 p e r­ c e n t c h a n c e o f failure. T h e sa m e is true fo r m e rg e rs and acq u isition s. Large IT p ro je cts fail

Chapter 4 • B u sin ess Reporting, Visual Analytics, and B u sin ess P erform ance M anagem ent 1 9 9

at the rate o f 7 0 p e rc e n t. F o r n e w fo o d p ro d u cts, th e failure rate is 8 0 p e rce n t. F o r n e w p h arm aceu tical p ro d u cts, it is e v e n higher, aro u n d 9 0 p e rce n t. O verall, th e rate o f failure fo r m o st n e w p ro je cts o r v en tu res runs b e tw e e n 6 0 a n d 8 0 p e rce n t. G iv e n th e s e n u m b e rs, th e an sw e r to th e q u e s tio n o f “w h at d o w e n e e d to d o differently?” b e c o m e s a vital issue.

A p p lication C a se 4 .7 s h o w s h o w a large co n stru ctio n and co n s u lta n cy co m p a n y im p lem en ted a n in te g rate d rep ortin g sy stem to b e tte r track th eir fin an cials an d o th e r im portant K P Is a cro ss its in tern atio n al b ra n ch e s.

Application Case 4.7 IBM Cognos Express Helps Mace for Faster and Better Business Reporting H ead q u artered in th e UK, M a ce is an in tern ation al co n su lta n cy a n d co n stru ctio n c o m p a n y that offers hig hly integrated s e rv ice s a cro ss th e fu ll pro p erty and infrastru ctu re life cy cle . It em p lo y s 3 ,7 0 0 p e o ­ ple in m o re th an 6 0 co u n trie s w o rld w id e, an d is involved in s o m e o f th e w o rld ’s h ig h est-p ro file p ro j­ ects, s u ch a s th e co n stru ctio n o f L o n d o n ’s “Sh ard ”, th e tallest b u ild in g in W e stern E urope.

M any o f M a c e ’s in tern ation al p ro je c ts are co n tra cte d o n a fix e d -p ric e b asis, a n d th eir su c­ c e ss d e p e n d s o n th e c o m p a n y ’s ab ility to control co s ts a n d m ain tain profitability. Until re ce n tly , th e o n ly w a y fo r s e n io r m an ag ers to g a in a fu ll u n d er­ stan d in g o f in tern atio n al o p era tio n s wras a m onthly rep ort, b a s e d o n a c o m p le x s p re a d sh e e t w h ich drew' data fro m n u m e ro u s d ifferen t a cco u n tin g sy stem s in su b sid iaries a ro u n d th e w orld.

B re n d a n Kitley, F in a n c e System s M an ag er at M ace, co m m e n ts: “T h e sp re a d sh e e t o n w h ich w e b a se d o u r in tern atio n al re p o rts h a d a b o u t 4 0 tab s and hu nd red s o f cro ss-lin k s, w h ich m e a n t it w as very e a sy to in tro d u ce errors, a n d th e la ck o f a stan ­ dardized a p p ro a ch w as affectin g a ccu ra cy and c o n ­ sistency. W e w a n te d to find a m o re ro b u st p ro ce s s fo r fin an cial re p o rtin g .”

F in d in g t h e R i g h t P a r t n e r

M ace w a s alread y u sin g IBM C o g n o s T M 1® softw are fo r its d o m e stic b u sin e ss in th e UK, an d w a s k e e n to find a sim ilar s o lu tio n fo r the in tern atio n al b u sin ess.

“W e d e cid e d to u s e IB M C o g n o s E x p re ss as the fo u n d atio n o f o u r in tern ation al rep ortin g plat­ fo rm ,” says B r e n d a n K itley. “W e w e re im p re sse d b y its ab ility to give u s m a n y o f th e sa m e cap ab ilities as o u r TM1 so lu tio n , b u t at a p rice -p o in t that w as m o re a ffo rd a b le fo r a n org an izatio n th e siz e o f ou r

in tern atio n al d ivision. W e a lso lik e d th e w e b inter­ fa c e , w h ich w e k n e w o u r in tern atio n al u se rs w o u ld b e a b le to a c c e s s e a sily .”

“W e e n g a g e d w ith B a rra ch d , a n IB M B u sin ess Partner, to su p p o rt us d u rin g th e p ro je ct, an d th ey d id a v e ry p ro fe ssio n a l jo b . T h e y h e lp e d u s n e g o ­ tiate a re a s o n a b le p rice fo r th e so ftw are lice n se s, d eliv ered e x c e lle n t te c h n ic a l su p p o rt, and pro v id ed u s w ith a c c e s s to th e righ t IB M e x p e rts w h e n e v e r w e n e e d e d th e m .”

R a p id I m p le m e n ta tio n

T h e im p lem e n ta tio n w a s c o m p le te d w ith in six m o n th s, d esp ite all th e co m p le x itie s o f im porting data fro m m u ltip le a c c o u n tin g system s an d h an d lin g e x c h a n g e ra te ca lcu la tio n s fo r o p era tio n s in o v e r 4 0 co u n tries. M ace is u sin g all fo u r m o d u le s o f IBM C o g n o s E xp ress: X c e le r a to r an d P la n n e r fo r m o d e l­ in g a n d p lanning; R e p o rte r fo r b u sin e ss in te llig e n ce ; a n d A dvisor fo r sev era l im p ortan t d ash b o ard s.

“W e have b e e n really im pressed b y th e ease o f d evelopm ent w ith C og nos Express,” com m ents B re n d an Kitley. “Setting u p new' applications an d cu bes is relatively q u ick and sim ple, and w e fe e l that w e ’ve only scratched the su rface o f w h at w e can achieve. We have a lot o f C ognos skills an d e x p e rien ce in-house, s o w e are k e e n to build a w id er rang e o f functional­ ities into C ognos E xpress as w e m ove forward.”

F a s t e r , M o r e S o p h i s t i c a t e d R e p o r t i n g

T h e first m a jo r p ro je c t w ith C o g n o s E x p re ss w as to re p lica te th e d efau lt re p o rts that u s e d to b e p ro ­ d u c e d b y th e old s p re a d sh e e t-b a s e d p ro c e s s . T h is w a s a ch ie v e d relativ ely q u ick ly , s o th e te a m w as a b le to m o v e o n to a s e c o n d p h a se o f d ev elo p in g m o re so p h istica ted an d d eta ile d reports.

0C on tin u ed )

2 0 0 Part II * D escriptive Analytics

A D D l i c a t i o n Case 4.7 (Continued) “T h e rep o rts w e h a v e n o w are m u ch m o re

u sefu l b e c a u s e th ey a llo w us to drill d o w n fro m th e g ro u p lev el th ro u g h all ou r in tern ation al su b sid iar­ ie s to th e individual co s t-c e n te rs , and e v e n to th e p ro je cts th e m se lv es,” e xp lain s B re n d an Kitley. “T h e ability to g e t an accu rate picture o f financial perroi- m a n ce in e a c h p ro je ct e m p o w ers o u r m anag ers to

m ak e b e tte r d ecisio n s.” “M oreover, sin ce th e reporting p ro cess is n o w

largely a u tom ated , w e c a n create reports m o re q uickly and w ith less effort - w h ich m ean s w e c a n gen erate th e m m o re frequently. In stead o f a o n e -m o n th lead tim e fo r reporting, w e c a n d o a full profitability an al­ y sis in h a lf th e tim e, a n d g iv e o u r m an ag e rs m o re tim ely a c c e s s to th e in form ation th ey n e e d .”

M o v in g T o w a r d s a S in g le P la tf o r m

W ith th e s u cce s s o f th e in tern ation al rep orting p ro j­ e ct, M a ce is w o rk in g to u n ite all its U K and interna­ tio n al su b sid iaries into this sin gle financial reporting a n d b u d g etin g system . W ith a co m m o n platform for all rep ortin g p ro ce sse s, th e c o m p a n y ’s cen tral fin an ce te a m w ill b e a b le to sp e n d less tim e an d effo rt o n

m aintaining an d cu stom izing th e p ro ce s s e s and m o re o n actu ally analyzing th e fig u res them selv es.

B re n d an K itley co n clu d es: “W ith b etter vis­ ibility and m o re tim ely a c c e s s to m o re d etailed and accu rate inform ation, w e are in a b e tte r p o sition to m onitor p erfo rm an ce and m aintain profitability w h ile ensuring th at o u r p ro jects are d elivered o n tim e and w ithin bud get. B y continu in g to w o rk w ith IBM and Barrach d to d ev elo p ou r C ognos E xpress solution, w e e x p e c t to u n lo c k e v e n greater b en e fits in term s o t standardization an d financial control.

Q u e s t i o n s f o r D i s c u s s i o n 1. W h at w a s th e rep ortin g c h a lle n g e M ace w as fa c­

ing? D o y o u th in k this is a n u n u su al ch a lle n g e

s p e cific to M ace? 2. W h at w a s th e a p p ro a ch fo r a p o ten tial solution? 3. W h at w e re th e results o b ta in e d in th e sh o rt term ,

an d w h at w e re th e future plans?

so u rce: IBM, Customer Success Story, “Mace gains insight in to , the performance o f international projects’- ibm.com/software/success/cssdb.nsf/CS/STRD-99ALB

(accessed Septem ber 2013)-

S E C T I O N 4 . 7 R E V I E W Q U E S T I O N S

1 . W h a t is b u sin e ss p e rfo rm a n ce m an ag em en t? H o w d o e s it relate to BI?

2 . W h at are th e th ree k e y c o m p o n e n ts o f a B P M system?

3 . List an d b riefly d e scrib e th e fo u r p h a s e s o f th e B P M cycle. 4 . W h y is strategy th e m o st im p ortan t p art o f a BPM im plem entation .

4 .8 P E R F O R M A N C E M E A S U R E M E N T U n d erly in g BPM is a p e rfo rm a n ce m e a s u re m e n t system . A cco rd in g to Sim o n s (2 0 0 2 ),

p e rf o rm a n c e m e a s u re m e n t sy stem s:

Assist m an ag ers in tracking th e im p lem en tation s o f b u sin ess strategy b y Com paq ing actu alTesults against sttateg ic goals an d ob jectiv es. A p e rfo rm an ce m easu re­ m e n t system typically com p rises system atic m eth o d s o f setting b u sin ess g o a s to g eth e r w ith p eriod ic fe e d b a c k reports th a t indicate p ro g ress against g o al .

All m e a su re m e n t is a b o u t co m p ariso n s. R a w n u m b ers are o f M e * . » T ™

Chapter 4 * B u sin ess Reporting, Visual Analytics, and B u sin ess P erform ance M anagem ent

co m p a n y w as 8 0 p ercen t? O bviously, th at p articu lar sa le s p e rs o n n e e d s to p ic k u p the p ace . As S im o n s’ d efin itio n su gg ests, in p e rfo rm a n ce m easu rem en t, th e k e y co m p a riso n s revolve aro u n d strateg ies, g o als, and o b je ctiv e s . O p e ra tio n a l m etrics th at are u s e d to m easu re p e rfo rm a n ce are usu ally ca lle d k e y p e rfo rm a n ce in d icato rs (K P Is).

Key Perform ance Indicator (KPI) T h ere is a d iffe re n c e b e tw e e n a “run o f th e m ill” m etric a n d a “strategically a lig n e d ” m e t­ ric. T h e te rm k e y p e rf o rm a n c e in d ic a to r (K P I ) is o fte n u s e d to d e n o te th e latter. A KPI re p re sen ts a strate g ic o b je c tiv e a n d m e a su re s p e rfo rm a n ce ag ain st a goal. A cco rd in g to E ck e rso n (2 0 0 9 ), K P Is a re m u ltid im en sion al. L o o se ly translated, this m e a n s th at K P Is h av e a variety o f d istin gu ish in g featu res, including:

• Strategy. K P Is e m b o d y a strateg ic o b je ctiv e . • Targets. K P Is m e a su re p e rfo rm a n ce ag ain st s p e cific targets. T arg e ts a re d efin ed

in strategy, p lan n in g , o r b u d g etin g se ssio n s an d c a n ta k e d ifferen t fo rm s (e .g ., a c h ie v e m e n t targets, re d u ctio n targets, a b so lu te targets).

• R a n g e s . T arg e ts h av e p e rfo rm a n ce ran g es (e .g ., a b o v e , on , o r b e lo w targ et). • E n c o d in g s . R an g es are e n c o d e d in so ftw are, e n a b lin g th e visual d isp lay o f

p e rfo rm a n ce (e .g ., g re e n , y e llo w , red ). E n co d in g s c a n b e b a s e d o n p e rc e n ta g e s o r m o re c o m p le x rules.

• T im e f r a m e s . T arg e ts a re assig n e d tim e fram es b y w h ic h th ey m u st b e a cco m p lis h e d . A tim e fram e is o fte n d ivid ed in to sm aller intervals to p rovide p e rfo rm a n ce m ilep o sts.

• B e n c h m a r k s . T arg e ts a re m e a su re d ag ain st a b a s e lin e o r b e n ch m a rk . T h e p rev iou s y e a r’s results o fte n serve as a b e n ch m a rk , b u t arbitrary n u m b ers o r e x te rn a l b e n ch m a rk s m ay a lso b e used .

A d istin ction is s o m e tim e s m ad e b e tw e e n K PIs that a re “o u tc o m e s ” a n d th o s e that are “d rivers.” O u tc o m e KPIs— so m e tim e s k n o w n a s lag g in g in d icators— m e a s u re the ou tp u t o f p ast activity (e .g ., re v e n u e s). T h e y are o fte n fin an cial in natu re, b u t n o t alw ays. D river KPIs— s o m e tim e s k n o w n as lea d in g in d ica to rs o r v alu e drivers— m e a su re activities that h av e a sig n ifican t im p act o n o u tc o m e KPIs (e .g ., sale s lead s).

In s o m e c irc le s , d river K P Is are s o m e tim e s ca lle d o p era tio n a l KPIs, w h ich is a bit o f an o x y m o ro n (H atch , 2 0 0 8 ). M ost o rg an ization s c o lle c t a w id e ran g e o f o p era tio n a l m etrics. As th e n a m e im p lies, th e s e m etrics d ea l w ith th e o p era tio n a l activities an d p e rfo rm a n ce o f a co m p an y . T h e fo llo w in g list o f e x a m p le s illustrates th e variety o f o p e ra tio n a l areas co v ered by th e se m etrics:

• C u sto m er p e r f o r m a n c e . M etrics fo r cu sto m e r satisfactio n , s p e e d an d a ccu ra cy o f issu e re so lu tio n , and cu sto m e r reten tion .

• S erv ice p e r f o r m a n c e . M etrics fo r s erv ice-ca ll re so lu tio n rates, s e rv ic e ren ew al rates, s e rv ice -le v e l a g re em en ts, d elivery p e rfo rm a n ce , an d retu rn rates.

• S a les o p era tio n s. N ew p ip e lin e acco u n ts, s a le s m e e tin g s s ecu re d , c o n v e rs io n o f in qu iries to lea d s, a n d a v e ra g e call clo su re tim e.

• S a les p la n /fo r e c a s t . M etrics fo r p rice -to -p u rch a se accu racy , p u rch a se o rd er-to - fulfillm ent ratio , qu antity e arn e d , fo reca st-to -p la n ratio, an d to tal c lo s e d co n tracts.

W h e th e r a n o p e ra tio n a l m etric is strateg ic o r n o t d e p e n d s o n th e c o m p a n y an d its use o f th e m e a su re . I n m an y in sta n ces, th e s e m etrics re p re se n t critical drivers o f strate­ g ic o u tco m e s. F o r in sta n ce, H atch (2 0 0 8 ) recalls th e c a s e o f a m id -tier w in e d istributor that w as b e in g s q u e e z e d u p stream b y th e co n so lid a tio n o f s u p p liers an d d o w n stream b y th e co n so lid a tio n o f retailers. In re s p o n s e , it d e c id e d to fo c u s o n fo u r o p e ra tio n a l m e a ­ sures: o n -hand / on-tim e in v e n to ry availability, ou tstan d in g “o p e n ” ord e r v alu e , n e t-n e w

2 0 2 Part II • D escriptive Analytics

acco u n ts, a n d p ro m o tio n co s ts an d retu rn o n m ark etin g in vestm en t. T h e n e t result o f its e ffo rts w a s a 12 p e rc e n t in c re a se in re v e n u e s in 1 year. O bv io u sly , th e s e o p eratio n al m etrics w e re k e y drivers. H o w ev er, as d escrib e d in th e fo llo w in g s e ctio n , in m an y ca ses, c o m p a n ie s sim p ly m e asu re w h at is c o n v e n ie n t w ith m inim al co n sid era tio n a s to w h y th e d ata a re b e in g c o lle cte d . T h e result is a sig n ifican t w a ste o f tim e, effo rt, a n d m o n ey .

Performance M easurem ent System T h e re is a d iffe re n ce b e tw e e n a p e rfo rm a n ce m e a su re m e n t sy stem a n d a p e rfo rm an ce m a n a g e m e n t system . T h e latter e n c o m p a ss e s th e fo rm er. T h a t is, an y p e rfo rm an ce m an a g e m e n t sy stem has a p e rfo rm a n ce m e a su re m e n t system , b u t n o t th e o th e r w ay aro u n d . I f y o u w e re to ask , m o st c o m p a n ie s to d ay w o u ld cla im that th e y h av e a p e r­ fo rm a n ce m e a su re m e n t sy stem b u t n o t n e c e s s a rily a p e rfo rm a n ce m an a g e m e n t system , e v e n th o u g h a p e rfo rm a n ce m e a su re m e n t system has very little, if any, u se w ith o u t th e o v erarch in g stru cture o f th e p e rfo rm a n ce m a n a g e m e n t system .

T h e m o st p o p u la r p e rfo rm a n ce m e a su re m e n t sy stem s in u se a re s o m e varian t o f K ap lan a n d N orton’s b a la n c e d s co re ca rd (B S C ). V arious surveys an d b en ch m a rk in g stu d ies in d icate that a n y w h ere fro m 5 0 to o v e r 9 0 p e rc e n t o f all co m p a n ie s h a v e im p le­ m e n te d s o m e fo rm o f B SC a t o n e tim e o r an o th er. A lth o u gh th e re s e e m s to b e so m e c o n fu s io n a b o u t w h at con stitu tes “b a la n c e ,” th e re is n o d o u b t a b o u t th e originators o f th e B S C (K a p lan & N orton, 1 9 9 6 ): “C entral to th e B S C m e th o d o lo g y is a h o listic v isio n o f a m e a su re m e n t sy stem tied to th e strateg ic d ire ctio n o f th e o rg an ization . It is b a s e d o n a fo u r-p ersp ectiv e v ie w o f th e w o rld , w ith fin an cial m e a su re s su p p o rte d b y c u sto m e r, inter­ n al, an d learn in g a n d gro w th m e trics.”

SECTION 4 . 8 REVIEW QUESTIONS

1 . W h a t is a p e rfo rm a n ce m an a g e m e n t system ? W h y d o w e n e e d one?

2 . W h at a re th e m o st distinguishing fe a tu res o f KPIs? 3 . List a n d b rie fly d efin e fo u r o f th e m o st c o m m o n ly c ite d o p era tio n a l areas fo r KPIs. 4 . W h at is a p e rfo rm a n ce m e a su re m e n t system ? H o w d o e s it w ork?

4.9 B A L A N C E D SC O R EC A R D S P ro b a b ly th e b e s t-k n o w n a n d m o st w id ely u s e d p e rfo rm a n ce m a n a g e m e n t sy stem is th e b a la n c e d s c o re c a rd (B S C ). K ap lan an d N o rto n first articu lated this m e th o d o lo g y in th eir H arv ard B u sin ess R eview article, “T h e B a la n c e d Sco recard : M easu res T h a t D rive P erfo rm a n ce,” w h ic h a p p e a re d in 1 9 9 2 . A fe w y e a rs later, in 1 9 9 6 , th e s e s a m e authors p ro d u ce d a g ro u n d b reak in g b o o k — The B a la n c e d S corecard : T ran slatin g Strategy in to A ction — th at d o cu m e n te d h o w co m p a n ie s w e re u sin g th e B S C n o t o n ly to s u p p le m e n t th e ir fin an cial m easu res w ith n o n fin a n cia l m e asu re s, b u t a lso to co m m u n ica te a n d im p le­ m e n t th e ir strategies. O v e r th e past fe w y ears, B SC h a s b e c o m e a g e n e r ic te rm th at is u sed to re p re s e n t virtually e v ery ty p e o f s co re ca rd ap p lica tio n a n d im p lem en tatio n , regard­ less o f w h e th e r it is b a la n ce d o r strategic. In re s p o n s e to this b astard ization o f th e term , K ap lan a n d N orton re le a se d a n e w b o o k in 2 0 0 0 , The S trategy-F ocu sed O rg an ization : H ow B a la n c e d S co rec a rd C om pan ies T hrive in th e N ew B u sin ess E n viron m en t. T h is b o o k w a s d esig n e d to re em p h a siz e th e strateg ic n atu re o f th e B SC m eth o d o lo g y . T h is w as fo llo w e d a fe w y e ars later, in 2 0 0 4 , b y Strategy M aps: C onverting In ta n g ib le Assets in to T an gible O utcom es, w h ic h d e s crib e s a d etailed p ro c e s s fo r lin kin g strategic o b je c tiv e s to o p e ra tio n a l ta ctics a n d initiatives. Finally, th e ir la te s t b o o k , The E x ecu tion P rem iu m , p u b ­ lish ed in 2 0 0 8 , fo c u s e s o n th e strategy gap — lin k in g strategy fo rm u latio n an d p lan n in g w ith o p era tio n a l e x e cu tio n .

C hapter 4 • B u sin ess Reporting, Visual Analytics, and B u sin ess P erform an ce M anagem ent 2 0 3

The Four Perspectives T h e b a la n c e d s c o re c a rd su gg ests th a t w e v ie w th e o rg an izatio n fro m fo u r p e rsp e ctiv e s— custom er, fin an cial, in tern al b u sin e ss p ro c e s s e s, learn in g an d grow th— an d d e v e lo p o b je c - tives, m e asu re s, targ ets, a n d initiatives relativ e to e a c h o f th e s e p e rsp e ctiv es. Figu re 4.11 show s th e s e fo u r o b je c tiv e s an d th e ir in terrelation sh ip w ith th e o rg an izatio n ’s v isio n and strategy.

THE C U STO M ER P E R S P E C T IV E R e c e n t m a n a g e m e n t p h ilo s o p h ie s h av e sh o w n a n in c re a s ­ ing realization o f th e im p o rta n ce o f cu sto m e r fo cu s an d cu sto m e r satisfactio n in any b u sin ess. T h e s e a re lea d in g ind icato rs: I f cu sto m ers a re n o t satisfied , th e y w ill ev en tu ally rind o th e r su p p liers th a t w ill m e e t th e ir n e e d s . P o o r p e rfo rm a n ce fro m this p e rsp e ctiv e is ± u s a lead in g in d ica to r o f future d e clin e , e v e n th o u g h th e cu rre n t fin an cial p ictu re m ay lo o k g o o d . In d e v e lo p in g m etrics fo r satisfactio n , cu sto m ers sh ou ld b e an aly zed in term s of kinds o f cu sto m e rs a n d th e k in d s o f p ro c e s s e s fo r w h ich w e are providing a p ro d u ct o r s erv ice to th o s e cu sto m e r groups.

THE F IN A N C IA L P E R S P E C T IV E K ap lan a n d N orton d o n ot d isregard th e trad itio nal n e e d fo r financial data. T im e ly a n d a ccu ra te fu n d in g data w ill alw ays b e a priority, an d m an ag ­ ers w ill d o w h a te v e r is n e c e s s a ry to pro v id e it. In fact, o fte n th e re is m o re th an e n o u g h handling an d p ro c e s s in g o f fin an cial data. W ith th e im p lem e n tatio n o f a co rp o ra te ic ta b a s e , it is h o p e d th at m o re o f th e p ro ce s s in g c a n b e ce n tra liz e d an d au to m ated . B u t ± e p o in t is that th e cu rren t em p h a sis o n fin an cials lead s to th e "u n b alan ce d " situ ation w ith regard to o th e r p e rsp e ctiv es. T h e r e is p e rh a p s a n e e d to in clu d e ad d itional fin a n cia l- related data, s u c h as risk a sse ssm e n t an d c o s t- b e n e fit d ata, in this category.

THE LE A R N IN G A N D GROW TH P E R S P E C T IV E T h is p e rsp e ctiv e aim s to a n s w e r th e q uestion, “T o a c h ie v e o u r v isio n , h o w w ill w e su stain o u r ab ility to c h a n g e an d im prove?” k inclu des e m p lo y e e training, k n o w le d g e m an ag e m e n t, an d co rp o ra te cultural ch a ra cte r­ istics re lated to b o th individ ual an d co rp o ra te -le v e l im p ro vem en t. In the cu rren t clim ate :e rapid te c h n o lo g ic a l c h a n g e , it is b e c o m in g n e c e s s a ry fo r k n o w le d g e w o rk e rs to b e in a continu ou s learn in g a n d g ro w in g m o d e . M etrics c a n b e p u t in to p la c e to g u id e m an ag ers

1 GURE 4.11 Four Perspectives in Balanced Scorecard Methodology.

2 0 4 Part II • D escriptive Analytics

in fo cu sin g training fu nd s w h e re th ey c a n h e lp th e m o st. In an y ca se , learn in g a n d growth co n stitu te th e esse n tial fo u n d atio n fo r th e s u c c e s s o f an y k n o w le d g e -w o rk e r organization. K ap lan a n d N orton e m p h a siz e th at “learn in g ” is m o re th a n "training”; it a lso inclu des th in gs lik e m e n to rs an d tu tors w ith in th e org an izatio n , as w e ll as that e a s e o f co m m u n ica­ tio n am o n g w o rk e rs that allow s th e m to read ily g e t h e lp o n a p ro b le m w h e n it is need ed .

TH E IN T E R N A L B U S IN E S S P R O C E S S P E R S P E C T IV E T h is p e rsp e ctiv e fo c u s e s o n th e im por­ ta n c e o f b u sin e ss p ro ce s s e s. M etrics b a s e d o n this p e rsp e ctiv e allo w th e m an ag ers to k n o w h o w w e ll th e ir internal b u sin ess p ro c e s s e s a n d fu n ction s are ru nning, an d w h eth er th e o u tc o m e s o f th e s e p ro c e s s e s (i.e ., p ro d u cts a n d s e rv ice s ) m e e t an d e x c e e d the cu sto m e r re q u irem en ts (th e m ission).

The Meaning o f Balance in BSC Fro m a high-lev el view point, the balanced scorecard (BSC) is b o th a perform ance m easu rem en t and a m an ag em en t m eth o d o lo g y that h elp s translate a n organization’s finan­ cial, cu stom er, internal p ro cess, and learning an d grow th o b je ctiv e s an d targets into a set o f actio n ab le initiatives. As a m easu rem en t m eth o d o lo g y, B SC is d esig n ed to o v erco m e the lim itations o f system s that are financially fo cu sed . It d o e s this b y translating a n organization’s v isio n an d strategy into a s e t o f interrelated financial a n d n on fin ancial ob je ctiv e s, m easures, targets, and initiatives. T h e n on fin an cial o b je ctiv e s fall into o n e o f th ree perspectives:

• C ustom er. T h is o b je c tiv e d efin e s h o w th e org an izatio n sh o u ld a p p e a r to its cu sto m e rs if it is to a cco m p lis h its vision.

• I n t e r n a l b u s in e s s p ro cess. T h is o b je c tiv e s p e cifie s th e p ro c e s s e s th e organiza­ tio n m u st e x c e l at in ord e r to satisfy its s h a re h o ld ers an d cu stom ers.

• L e a r n i n g a n d grow th. T h is o b je c tiv e in d icate s how' a n o rg an ization c a n im prove its ability to c h a n g e a n d im p ro ve in o rd e r t o a ch ie v e its vision.

B a sica lly , n o n fin a n cia l o b je c tiv e s form a s im p le cau sal ch a in w ith “learn in g and g ro w th ” driving “internal b u sin e ss p r o c e s s ” c h a n g e , w h ic h p ro d u ce s “cu sto m e r” ou t­ co m e s th at are re s p o n s ib le fo r re a ch in g a c o m p a n y ’s “fin an cial” o b je ctiv e s . A sim ple ch a in o f this sort is e x e m p lifie d in Fig u re 4 .1 2 , w h e re a strateg y m ap a n d b a la n ce d s co re ca rd fo r a fictitious co m p a n y are d isplayed . Fro m th e strategy m ap , w e c a n s e e that th e organ ization has fo u r o b je c tiv e s a cro ss th e fo u r B SC p e rsp e ctiv es. L ike o th e r strategy m aps, this o n e b e g in s a t th e to p w ith a fin an cial o b je c tiv e (i.e ., in cre a se n e t in c o m e ). This o b je c tiv e is driven b y a cu sto m e r o b je c tiv e (i.e ., in c re a se cu sto m e r re te n tio n ). In turn, the cu sto m e r o b je c tiv e is th e resu lt o f a n internal p r o c e s s o b je c tiv e (i.e ., im p ro ve call ce n te r p e rfo rm a n ce ). T h e m ap co n tin u e s d o w n to th e b o tto m o f th e h ierarch y , w h e re th e learn­ ing o b je c tiv e is fo u n d (e .g ., re d u ce e m p lo y e e tu rn ov er).

In B S C , th e te rm b a la n c e arises b e c a u s e th e c o m b in e d s e t o f m e a su re s is su p p o sed to e n c o m p a ss in d icato rs that are:

• Fin an cial a n d n on fin an cial • Lead ing and lagging • In te rn al an d e xte rn al • Q uantitative an d qualitative • Short term a n d lo n g term

Dashboards Versus Scorecards In th e trad e jo u rn als, th e term s d a s h b o a rd an d sc o r e c a r d a re u s e d alm ost interchangeably, e v e n though BPM/BI ven d ors usu ally o ffe r sep arate d ash b o ard an d s co re ca rd applications. A lthough d ash bo ard s and sco re card s h av e m u ch in co m m o n , th ere are d iffe ren ce s b e tw e e n

Chapter 4 • B u sin ess Reporting, Visual Analytics, and B u sin ess P erform ance M anagem ent 2 0 5

S tr a t e g y Map: Linked Objectives

Balanced Sco re card : M easu res and T a r g e ts

S tr a te g ic Initiatives; A ction Plans

Financial { Increase N et \ V Income )

Net income growth

Increase 2 5 %

Custom er f Increase f Custom er ) \ R e t e n t io n ,/

Maintenance retention rate

Increase 15 %

Change licensing and maintenance contracts

P ro ce ss f Improve C a l i \ ( Center ) ^ P e r f o r m a n c e ^

Issue turnaround

time

Improve 3 0 %

Standardized call center processes

Learning and

Growth

f Reduce N, ( Employee )

Turnover

Voluntary turnover

rate

Reduce 2 5 %

Salary and bonus upgrade

FIGURE 4.12 Strategy Map and Balanced Scorecard. Source: Business Intelligence, 2e.

d ie tw o. O n th e o n e hand , e xe cu tiv es, m anagers, an d sta ff u se sco re card s to m onitor strategic alignm ent an d s u cce s s w ith strategic o b je ctiv e s an d targets. As n o ted , th e b est- k n ow n e x a m p le is th e BSC. O n th e o th er h an d , d ash bo ard s are u se d at the o p era tio n a l an d Tactical lev els. M anagers, supervisors, and op erato rs u se op eratio n al d ash bo ard s to m o n ito r detailed o p eratio n al p e rfo rm a n ce o n a w e e k ly , daily, o r e v e n hourly basis. F o r e x a m p le , op erational d ash b o ard s m ight b e u se d to m o n ito r p ro d u ction quality. In th e sa m e vein, m anagers and staff u s e tactical d ash bo ard s to m o n ito r tactical initiatives. F o r e x a m p le , :actical d ashboard s m ight b e u se d to m o n ito r a m arketing cam p aig n o r sale s p erfo rm an ce.

SECTION 4 . 9 REV IEW QUESTIONS

1 . W h at is a b a la n c e d s co re ca rd (B SC )? W h e re did it c o m e from? 2 . W h at a re th e fo u r p e rsp e ctiv e s that B SC su g g ests u s to u se to v ie w o rg an ization al

perfo rm an ce? 3. W h y d o w e n e e d to d efin e sep arate o b je ctiv e s , m e asu re s, targets, an d initiatives fo r

e a c h o f th e s e fo u r B S C persp ectiv es? 4 . W h at is th e m e a n in g o f an d m o tiv atio n fo r b a la n c e in BSC? 5. W h at are th e d iffe re n c e s an d co m m o n a litie s b e tw e e n d ash b o ard s an d scorecard s?

4.10 S IX S IG M A A S A PERFO RM AN CE M EASUREM ENT SYSTEM Since its in c e p tio n in th e m id -1 9 8 0 s, Six Sigm a h a s e n jo y e d w id esp rea d a d o p tio n b y com p an ies th ro u g h o u t th e w o rld . F o r th e m o st part, it h a s n o t b e e n u s e d as a p e rfo rm a n ce m easu rem en t a n d m a n a g e m e n t m eth o d o lo g y . In stead , m o st co m p a n ie s u se it as a p ro cess im p ro vem en t m e th o d o lo g y th at e n a b le s th e m to scru tin ize th e ir p ro c e s s e s, p in p o in t problem s, and ap p ly re m e d ie s. In re c e n t years, s o m e co m p a n ie s , s u ch as M o toro la, hav e re co g n ize d th e v a lu e o f u sin g Six Sigm a fo r strateg ic p u rp o se s. In th e s e in sta n ce s, Six Sigma p ro v id es th e m e a n s to m e a su re a n d m o n ito r k e y p ro c e s s e s re la ted to a co m p a n y 's profitability a n d to a c c e le ra te im p ro v e m en t in ov erall b u sin e ss p e rfo rm a n ce . B e c a u s e o f its fo cu s o n b u s in e s s p ro ce s s e s, Six Sigm a a lso p ro v id e s a straightforw ard w a y t o ad d ress p erfo rm an ce p ro b le m s after th e y are id en tified o r d etected .

2 0 6 Part II • D escriptive Analytics

Sigm a, a , is a le tte r in th e G r e e k a lp h a b e t th a t statistician s u s e to m e a s u re th e v ariab ility in a p ro c e s s . In th e q u a lity a re n a , v a ria b ility is s y n o n y m o u s w ith th e n u m b e r o f d e fe c ts . G e n e ra lly , c o m p a n ie s h a v e a c c e p te d a g re a t d ea l o f v ariab ility in their b u s in e s s p ro c e s s e s . In n u m e ric term s, th e n o rm h a s b e e n 6 ,2 0 0 to 6 7 ,0 0 0 d e fe c ts per m illio n o p p o rtu n itie s (D P M O ). F o r in s ta n c e , i f a n in s u ra n c e c o m p a n y h a n d le s 1 m illion cla im s, th e n u n d e r n o rm a l o p e ra tin g p ro c e d u re s 6 ,2 0 0 to 6 7 ,0 0 0 o f th o s e cla im s w o u ld b e d e fe c tiv e ( e .g ., m ish a n d le d , h a v e errors in th e fo rm s). T h is le v e l o f variab ility re p re s e n ts a th r e e - to fo u r-sig m a le v e l o f p e rfo rm a n c e . T o a c h ie v e a S ix Sigm a le v e l o f p e rfo rm a n c e , th e c o m p a n y w o u ld h a v e to re d u c e th e n u m b e r o f d e fe c ts to n o m o re th a n 3 .4 D PM O . T h e r e fo r e , Six Sigm a is a p e rfo rm a n c e m a n a g e m e n t m e th o d o lo g y a im e d a t re d u cin g th e n u m b e r o f d e fe c ts in a b u s in e s s p r o c e s s to a s c lo s e to zero D P M O as p o ss ib le .

The DMAIC Performance Model Six Sigm a rests o n a sim p le p e rfo rm a n ce im p ro v e m en t m o d e l k n o w n as DMAIC. Like B PM , DMAIC is a c lo s e d -lo o p b u sin e ss im p ro v e m en t m o d e l, an d it e n c o m p a ss e s the ste p s o f d efining, m easu rin g, an alyzin g, im p ro vin g, a n d con trollin g a p ro c e s s . T h e steps ca n b e d escrib e d as fo llow s:

1 . D e fin e . D e fin e th e g o als, o b je ctiv e s , an d b o u n d a rie s o f th e im p ro v em en t activ­ ity. At th e to p lev el, th e g o a ls a re th e strate g ic o b je c tiv e s o f th e co m p an y . At lo w e r lev els— d ep artm en t o r p ro je c t lev els— th e g o a ls are fo c u s e d o n s p e c ific o p eratio n al p ro cesse s.

2 . M e a s u r e . M easu re th e e x istin g system . E sta b lish quantitative m e a su re s th a t will y ield statistically valid d ata. T h e data c a n b e u s e d to m o n ito r p ro g re ss to w ard the g o als d efin e d in th e p re v io u s step .

3 . A n a ly z e . A n alyze th e sy stem to id entify w a y s to elim in ate th e gap b e tw e e n th e cu rre n t p e rfo rm a n ce o f th e system o r p ro c e s s a n d th e d esire d g oal.

4 . Im p r o v e . In itiate a ctio n s to elim in ate th e g a p b y find ing w ay s to d o things b etter, c h e a p e r, o r faster. U s e p ro je c t m a n a g e m e n t a n d o th e r p lan n in g to o ls to im p lem en t th e n e w a p p ro ach .

5 . C o n tr o l. In stitu tion alize th e im p ro ved sy stem b y m o d ifyin g c o m p e n sa tio n an d in cen tiv e system s, p o licie s, p ro ced u res, m an u factu rin g re s o u r c e p lan n in g , b ud gets, o p e ra tio n instru ction s, o r o th e r m a n a g e m e n t system s.

F o r n e w p ro c e s s e s, th e m o d e l that is u s e d is ca lle d DMADV (d e fin e , m easu re, an aly ze, d esign , a n d v erify ). T rad itionally, DMAIC an d DM A DV h av e b e e n u se d prim arily w ith o p era tio n a l issu es. H ow ever, n o th in g p re clu d e s th e a p p lica tio n o f th e s e m e th o d o lo ­ g ies to strategic issu es s u c h a s co m p a n y profitability. In re c e n t years, th e re h a s b e e n a fo cu s o n c o m b in in g th e Six Sigm a m e th o d o lo g y w ith o th e r su cce ssfu l m e th o d o lo g ie s. F or in sta n ce, th e m e th o d o lo g y k n o w n as L ean M an u factu rin g, L ean P rod u ction , o r sim p ly as L ean has b e e n c o m b in e d w ith S ix Sigm a in o r d e r to im p ro ve its im p a ct in p e rfo rm an ce m an ag em en t.

Balanced Scorecard Versus Six Sigma W h ile m an y h av e c o m b in e d Six Sigm a a n d B a la n c e d S co re ca rd fo r a m o re h olistic solu tio n , s o m e fo c u s e d o n fav orin g o n e v e rsu s th e oth er. G u p ta ( 2 0 0 6 ) in h is b o o k titled S ix S igm a B u sin ess S co rec a rd p ro v id e s a g o o d su m m ary o f th e d iffe re n ce s b e tw e e n the b a la n c e d s co re ca rd an d Six Sigm a m e th o d o lo g ie s ( s e e T a b le 4 .1 ). In a nu tshell, th e m ain d iffe re n ce is th a t B SC is fo c u s e d o n im p ro vin g overall strategy, w h e re a s S ix Sigm a is fo c u s e d o n im proving p ro ce s s e s.

Chapter 4 • B u sin ess Reporting, Visual Analytics, and B u sin ess P erform an ce M anagem ent 2 0 7

T A B L E 4,1 Com parison o f Balanced Scorecard and Six Sigm a

Balanced Scorecard Six Sigm a

Strategic management system Performance measurement system

Relates to the longer-term view of the Provides snapshot of business's performance and business identifies measures that drive performance

toward profitability

Designed to develop balanced set of Designed to identify a set of measurements that measures impact profitability

Identifies measurements around vision Establishes accountability for leadership for wellness and values and profitability

Critical management processes are to Includes all business processes— management clarify vision/strategy, communicate, and operational plan, set targets, align strategic initiatives, and enhance feedback

Balances customer and internal Balances management and employees' roles; operations without a clearly defined balances costs and revenue of heavy processes leadership role

Emphasizes targets for each Emphasizes aggressive rate of improvement for measurement each measurement, irrespective of target

Emphasizes learning of executives Emphasizes learning and innovation at all levels based on the feedback based on the process feedback; enlists all employ­

ees' participation

Focuses on growth Focuses on maximizing profitability

Heavy on strategic content Heavy on execution for profitability

Management system consisting of Measurement system based on process measures management

Source: P. Gupta, Six Sigm a B usiness S corecard, 2nd ed., McGraw-Hill Professional, New York, 2006.

Effective Performance Measurem ent A nu m ber o f b o o k s provid e re cip es fo r determ ining w h e th e r a co lle ctio n o f p erfo rm an ce —easu res is g o o d o r bad. A m on g th e b a sic ingredien ts o f a g o o d c o lle ctio n are th e follow ing:

• M easu res sh o u ld fo c u s o n k e y facto rs. • M easu res sh o u ld b e a m ix o f p ast, p re sen t, a n d future. • M easu res sh o u ld b a la n c e th e n e e d s o f sh are h o ld ers, e m p lo y e e s, p artn ers, su p p liers,

an d o th e r stak eh o ld ers. • M easu res sh o u ld start a t th e to p a n d flo w d o w n to th e b otto m . • M easures n e e d to h a v e targets th a t are b a s e d o n re se a rch an d reality ra th er th an

arbitrary.

As th e s e c tio n o n K P Is n o te s , alth o u g h all o f th e s e ch aracteristics a re im portant, d ie real k e y to a n e ffe ctiv e p e rfo rm a n ce m e a su re m e n t sy stem is to h a v e a g o o d strategy. M easures n e e d to b e d eriv ed fro m th e co rp o ra te and b u sin e ss u n it strateg ies a n d fro m a n analysis o f th e k e y b u sin e ss p ro c e s s e s re q u ired to a ch ie v e th o se strateg ies. O f co u rse , this is e a sie r said th a n d o n e . I f it w e re sim p le, m o st org an izatio n s w o u ld alread y have e le c tiv e p e rfo rm a n c e m e a su re m e n t system s in p la c e , b u t th e y d o not.

A p p lication C a se 4 .8 , w h ic h d e s crib e s th e W e b -b a s e d KPI s co re ca rd sy stem at E x p e d ia .c o m , o ffe rs insights in to th e d ifficu lties o f d efin in g b o th o u tc o m e a n d driver KPIs an d th e im p o rta n ce o f align in g d ep artm en tal K P Is to o v erall c o m p a n y o b je ctiv e s .

2 0 8 Part II • D escriptive Analytics

Application Case 4.8 Expedia .com ' s Custom er Sa tisfactio n Scorecard E x p e d ia , I n c ., is th e p a re n t c o m p a n y to s o m e o f th e w o rld ’s le a d in g trav el c o m p a n ie s , p ro v id in g trav el p ro d u cts a n d s e r v ic e s to le is u re a n d c o r p o ra te trav­ e le r s in th e U n ited S tate s a n d aro u n d th e w o rld . It o w n s an d o p e ra te s a d iv ersified p o rtfo lio o f w e ll-r e c o g n iz e d b ra n d s, in clu d in g E xp ed ia.co m , H o tels.co m , H otw ire.com , T rip A d v isor, E g e n c ia , C la ssic V a c a tio n s , a n d a ran g e o f o th e r d o m e stic a n d in te rn a tio n a l b u s in e s s e s . T h e c o m p a n y ’s travel o ffe rin g s c o n s is t o f airlin e fligh ts, h o te l stays, car re n tals, d e s tin a tio n s e r v ic e s , cru is e s , a n d p a c k a g e trav el p ro v id e d b y v ario u s a irlin e s, lo d g in g p ro p ­ e rtie s , c a r ren tal co m p a n ie s , d e s tin a tio n s e rv ic e p ro v id e rs, c ru is e lin e s, an d o th e r trav el p ro d u ct a n d s e r v ic e c o m p a n ie s o n a sta n d -a lo n e an d p a c k ­ a g e b a sis . It a ls o fa cilita te s th e b o o k in g o f h o te l ro o m s, a irlin e sea ts, c a r re n ta ls , a n d d estin a tio n s e r v ic e s fro m its trav el su p p liers. It a cts as a n a g e n t in th e tra n sa ctio n , p a s s in g re s e rv a tio n s b o o k e d b y its tra v e le rs to th e re le v a n t airlin e , h o te l, ca r ren tal c o m p a n y , o r cru ise lin e . T o g e th e r, th e s e p o p u la r b ra n d s a n d in n o v ativ e b u s in e s s e s m a k e E x p e d ia th e la rg e s t o n lin e tra v el a g e n c y in th e w o rld , th e third la rg e s t trav el c o m p a n y in th e U n ite d States, an d th e fo u rth la rg est trav el c o m p a n y in th e w o rld . Its m is sio n is to b e c o m e th e la rg est an d m o st p ro fita b le s e lle r o f trav el in th e w o rld , b y h e lp in g e v e r y o n e e v e ry w h e re p la n a n d p u rc h a s e ev e ry th in g in travel.

P r o b l e m

C u stom er satisfactio n is k e y to E x p e d ia ’s overall m is­ sio n , strategy, a n d s u c c e s s . B e c a u s e E xp ed ia.com is a n o n lin e b u s in e s s , th e cu sto m e r’s sh o p p in g e x p e ­ r ie n ce is critical to E x p e d ia ’s re v e n u e s. T h e o n lin e s h o p p in g e x p e rie n c e c a n m a k e o r b re a k a n o n lin e b u sin ess. It is a lso im p ortan t that th e cu sto m e r’s sh o p p in g e x p e rie n c e is m irrored b y a g o o d trip e x p e rie n c e . B e c a u s e the cu sto m e r e x p e rie n c e is critical, all cu sto m e r issu es n e e d to b e track ed , m o n ito re d , an d resolv ed as q u ick ly a s p o ssib le. U n fortu nately, a fe w y ears b a c k , E x p e d ia la ck e d visibility in to th e “v o ic e o f th e cu sto m e r.” It h a d n o u n ifo rm w a y o f m easuring satisfactio n , o f analyz­ in g th e d rivers o f satisfactio n , o r o f d eterm in in g th e

im p act o f satisfactio n o n th e co m p a n y ’s profitability o r o v erall b u s in e s s o b je ctiv e s.

S o lu tio n

E x p e d ia ’s p r o b le m w a s n o t la c k o f d ata. T h e c u s ­ to m e r s a tis fa c tio n g ro u p a t E x p e d ia k n e w th a t it h a d lo ts o f d a ta . In all, th e r e w e r e 2 0 d isp arate d a ta b a s e s w ith 2 0 d iffe re n t o w n e rs. O riginally , th e g ro u p c h a r g e d o n e o f its b u s in e s s an aly sts w ith th e ta sk o f p u llin g to g e th e r a n d a g g re g a tin g th e d ata fro m th e s e v a rio u s s o u rc e s in to a n u m ­ b e r o f k e y m e a s u re s fo r sa tisfa ctio n . T h e b u s in e s s a n a ly st s p e n t 2 to 3 w e e k s e v e ry m o n th p u llin g a n d a g g re g a tin g th e d ata, leav in g virtu ally n o tim e fo r a n aly sis. E ventu ally, th e g ro u p re a liz e d that it w a s n ’t e n o u g h to a g g re g a te th e d ata. T h e data n e e d e d to b e v ie w e d in th e c o n t e x t o f stra te g ic g o a ls , a n d in d iv id u als h a d to ta k e o w n e rs h ip o f th e results.

T o ta c k le th e p ro b lem , th e g ro u p d e c id e d it n e e d e d a re fin ed vision. It b e g a n w ith a d etailed analysis o f th e fu n d am en tal drivers o f th e d ep art­ m e n t’s p e rfo rm a n ce a n d th e lin k b e tw e e n this p e rfo rm a n ce a n d E x p e d ia ’s overall goals. N ext, the g ro u p c o n v e rte d th e s e drivers an d lin k s in to a s c o r e ­ card. T h is p r o c e s s involved th re e step s:

1. D e c id in g how to m e a s u r e sa tisfa ctio n . T h is re q u ire d th e g ro u p to d eterm in e w h ich m easu res in th e 2 0 d a tab ase s w o u ld b e u se ­ fu l fo r d em o n stratin g a cu sto m e r’s lev el o f satisfactio n . T h is b e c a m e th e b asis fo r the sco re ca rd s an d KPIs.

2 . S e ttin g t h e r ig h t p e r fo r m a n c e ta r g e ts . T h is re q u ired th e g ro u p to d eterm in e w h e th e r KPI targ ets h ad sh ort-term o r lo n g -term pay­ offs. J u s t b e c a u s e a cu sto m e r w a s satisfied w ith his o r h e r o n lin e e x p e r ie n c e did n o t m e a n that th e cu sto m e r w a s satisfied w ith th e v e n d o r p ro v id in g th e trav el serv ice.

3 . P u ttin g d a t a in t o c o n te x t. T h e g ro u p had to tie th e data to o n g o in g cu sto m e r satisfaction p ro jects.

T h e vario u s real-tim e data so u rce s are fed in to a m ain d a ta b a s e (c a lle d the D e c is io n Su p p ort

Chapter 4 • B u sin ess Reporting, Visual Analytics, and B u sin ess P erform ance M anagem ent 2 0 9

F acto ry ). In th e c a s e o f th e cu sto m e r satisfactio n g ro u p , th e se in clu d e cu sto m e r su rveys, CRM system s, in teractiv e v o ic e re s p o n s e sy stem s, and o th e r cu sto m e r-se rv ice sy stem s. T h e d ata in th e D SS Facto ry are lo a d e d o n a daily b a sis into sev eral data m arts an d m u ltid im en sion al c u b e s. U sers c a n a c c e s s th e data in a v arie ty o f w ays th at are relevan t to th eir particu lar b u s in e s s n eed s.

B e n e f its

Ultim ately, th e cu sto m e r sa tisfa ctio n g ro u p c a m e u p w ith 10 to 1 2 o b je c tiv e s th a t lin k e d d irectly to E x p e d ia ’s c o r p o ra te initiatives. T h e s e o b je ctiv e s w e re , in turn, lin k e d to m o re th a n 2 0 0 KPIs w ithin th e cu sto m e r sa tisfa ctio n gro u p . K P I o w n e rs ca n b u ild , m an ag e , an d c o n s u m e th e ir o w n sco re card s, an d m an ag e rs a n d e x e c u tiv e s h av e a tran sp ar­ e n t v ie w o f h o w w e ll a c tio n s are alig n in g w ith the strategy. T h e s c o re c a rd a lso p ro v id es th e cu sto m e r sa tisfa ctio n g ro u p w ith th e ab ility to drill d ow n in to th e d ata u n d erly in g a n y o f th e trend s o r pat­ te rn s o b se rv e d . In th e past, all o f this w o u ld have ta k e n w e e k s o r m o n th s to d o , if it w a s d o n e a t all. W ith th e s c o re c a rd , th e C u stom er S e rv ice g ro u p can im m ed iately s e e h o w w ell it is d o in g w ith re s p e c t to th e K PIs, w h ich , in tu rn , a re re fle cte d in th e g ro u p ’s o b je c tiv e s a n d th e co m p a n y ’s o b je ctiv e s.

A s a n a d d ed b e n e fit, th e data in th e system sup­ p o rt n o t only the cu sto m e r satisfactio n g roup, b u t also o th e r b u sin ess units in; th e com p an y . F o r e x a m p le , a fro ntline m an a g e r c a n an aly ze airline e xp en d itu res o n a m ark et-b y -m ark et b asis to evalu ate n eg o ti­ ate d co n tra ct p e rfo rm a n ce o r d eterm in e th e savings p o ten tial fo r co n so lid a tin g sp en d in g w ith a sin gle carrier. A trav el m a n a g e r c a n lev erag e th e b u sin ess in te llig e n ce to d isco v e r a re a s w ith high v o lu m e s o f u n u se d tick e ts o r o fflin e b o o k in g s a n d d ev ise strate­ g ies to ad ju st b e h a v io r an d in cre a se overall savings.

Q u e s t i o n s f o r D i s c u s s i o n

1. W h o a re th e cu sto m e rs fo r E x p e d ia.co m ? W hy is cu sto m e r sa tisfa ctio n a v e ry im portant p art o f th e ir business?

2. H o w d id E x p e d ia .c o m im p ro ve cu sto m e r satis­ fa ctio n w ith scorecard s?

3. W h at w e re th e c h a lle n g e s, th e p ro p o s e d s o lu ­ tion , a n d th e o b ta in e d results?

Sources: Based o n Microsoft, “Expedia: Scorecard Solution Helps Online Travel Company Measure the Road to Greatness," download.m icrosoft.com/documents/customer evidence/22483_Expedia_Case Study.doc (accessed January 2013); and R. Smith, "Expedia-5 Team Blog: Technology,” April 5, 2007, expedia-team5.blogspot.com (accessed September 2010).

SECTION 4 . 1 0 REVIEW QUESTIONS

1 . W h a t is S ix Sigm a? H o w is it u se d as a p e rfo rm a n ce m e a su re m e n t system?

2 . W h a t is DMAIC? List and b riefly d e s c rib e th e s te p s in v o lv ed in DMAIC. 3 . C o m p are B SC a n d S ix Sigm a as tw o co m p e tin g p e rfo rm a n ce m e a su re m e n t system s

4 . W h at are th e in g red ien ts fo r a n e ffe ctiv e p e rfo rm a n ce m a n a g e m e n t system ?

Chapter Highlights • A rep o rt is a n y co m m u n ica tio n artifact p re p are d

w ith th e s p e c ific in te n tio n o f co n v e y in g inform a­ tio n in a p re s e n ta b le form .

• A b u sin e ss rep o rt is a w ritten d o cu m en t that co n ta in s in form ation regard ing b u sin e ss m atters.

• T h e k e y to an y su cce ssfu l b u sin ess re p o rt is clarity, b rev ity, c o m p le te n e ss , an d co rrectn ess.

• D ata visualization is th e u s e o f visual re p re sen ­ tation s to e x p lo r e , m ak e s e n s e o f, an d co m m u ­ n icate data.

• P erh ap s th e m o st n o ta b le inform ation g raphic o f th e p ast w a s d ev elo p e d b y C h arles J . Minard, w h o g rap hically portrayed th e lo sses su ffered by- N a p o le o n ’s arm y in th e R ussian cam p aig n o f 1812.

• B a s ic ch art ty p e s in clu d e lin e , b ar, a n d p ie chart. • S p e cia liz ed ch a rts are o fte n d eriv ed fro m th e

b a s ic charts as e x c e p tio n a l cases. • D ata v isu alizatio n te ch n iq u e s and to o ls m a k e th e

u sers o f b u s in e s s an aly tics an d b u sin e ss intelli­ g e n c e sy stem s b e tte r in fo rm atio n co n su m e rs.

2 1 0 Part II • D escriptive Analytics

• V isual an aly tics is th e c o m b in a tio n o f visualiza­ tio n a n d p re d ictiv e analytics.

• In c re a s in g d em an d fo r visual analytics co u p le d w ith fast-g ro w in g d ata v o lu m e s le d to e x p o n e n ­ tial g ro w th in h ig h ly e ffic ie n t visu alization sys­ tem s investm ent.

• D a sh b o a rd s pro v id e visual displays o f im portant in form ation th at is co n so lid a ted and arran ged o n a sin gle s c re e n so th at inform ation c a n b e d igested a t a sin g le g la n c e a n d e asily drilled in an d further e x p lo red .

• B P M re fe rs to th e p ro c e s s e s, m e th o d o lo g ie s, m et­ rics, a n d te ch n o lo g ie s u s e d b y e n terp rises to m e a ­ su re, m o nito r, and m a n a g e b u sin ess p erfo rm an ce.

• BPM is a n outgrow th o f B I, a n d it incorporates m any o f its tech n o lo g ies, applications, and techniqu es.

• T h e p rim ary d iffe re n ce b e tw e e n B I a n d BPM is th at B P M is alw ays strategy driven.

• BPM e n c o m p a ss e s a c lo s e d -lo o p s e t o f p ro ce s s e s th a t lin k strategy to e x e c u tio n in o rd e r to o p ti­ m iz e b u s in e s s p e rfo rm a n ce .

• T h e k e y p ro c e s s e s in BPM a re strategize, plan, m o n ito r, act, an d adjust.

• Strategy an sw ers th e q u e s tio n “W h e re d o w e w an t to g o in th e future?”

• D e c a d e s o f re s e a r c h hig h lig h t th e g a p b e tw e e n strategy a n d e x e cu tio n .

• T h e g a p b e tw e e n strategy a n d e x e c u tio n is fo u n d in th e b ro a d areas o f co m m u n icatio n , alignm ent, fo c u s , an d re so u rces.

• O p e ra tio n a l and tactical p lan s ad d ress th e q u e s­ tio n “H o w d o w e g e t to th e future?”

• An o rg a n iz a tio n ’s strateg ic o b je c tiv e s and key m etrics sh o u ld serv e as to p -d o w n drivers fo r th e a llo ca tio n o f th e o rg an ization ’s ta n g ib le an d in ta n g ib le assets.

• M o n itorin g ad d resses th e q u e stio n o f “H o w are w e doing?”

• T h e o v e ra ll im p act o f th e p lan n in g a n d rep orting p ra ctice s o f th e av erag e co m p a n y is th at m a n a g e ­ m e n t h a s little tim e to re v iew results fro m a stra­ te g ic p e rs p e ctiv e , d e c id e w h a t sh o u ld b e d o n e d ifferen tly, a n d a c t o n th e rev ised plans.

T h e d ra w b a ck s o f u sin g fin an cial d ata as th e co re o f a p e rfo rm a n ce m e a su re m e n t sy stem a re w ell k now n.

• P e rfo rm a n ce m e a su re s n e e d to b e d eriv ed from th e c o rp o ra te an d b u s in e s s u n it strateg ies and fro m a n a n aly sis o f th e k e y b u sin e ss p ro ce s s e s re q u ired to a ch ie v e th o s e strategies.

• P ro b a b ly th e b e s t-k n o w n an d m o st w id ely u sed p e rfo rm a n ce m an a g e m e n t sy stem is th e BSC.

• C entral t o th e B S C m e th o d o lo g y is a holistic v isio n o f a m e a su re m e n t system tied to th e strate­ g ic d ire ctio n o f th e organization.

• As a m e a su re m e n t m e th o d o lo g y , B SC is d esign ed to o v e r c o m e th e lim itations o f system s that are fin an cially fo cu sed .

• As a strateg ic m a n a g e m e n t m eth o d o lo g y , BSC e n a b le s a n org an izatio n to align its actio n s w ith its ov erall strategies.

• In BSC, strateg y m ap s pro v id e a w ay to form ally re p re se n t a n o rg an izatio n ’s strateg ic o b je ctiv e s a n d th e ca u sa l c o n n e c tio n s am o n g them .

• M ost c o m p a n ie s u s e Six Sigm a a s a p ro cess im p ro v e m en t m e th o d o lo g y th a t e n a b le s th e m to scru tin ize th e ir p ro c e s s e s, p in p o in t p ro b lem s, an d a p p ly re m e d ie s.

• Six Sigm a is a p e rfo rm a n c e m an a g e m e n t m eth o d ­ o lo g y a im e d at red u cin g th e n u m b e r o f d e fe cts in a b u s in e s s p ro ce s s to as c lo s e to z e ro D PM O as p o ssib le.

• Six Sigm a u se s DMAIC, a c lo s e d -lo o p b u sin ess im p ro v e m en t m o d e l th at in v olv es th e s te p s o f d efining, m easu rin g , an alyzing, im provin g, an d co n tro llin g a p ro cess.

• Su b stan tial p e rfo rm a n ce b e n e fits c a n b e g a in ed b y in tegratin g B SC a n d S ix Sigm a.

• T h e m a jo r B P M ap p licatio n s in clu d e strategy m an ag e m e n t; b u d g etin g , p lan n in g , a n d fo re ca st­ ing; fin a n cia l co n so lid a tio n ; profitability analysis a n d o p tim ization ; a n d fin an cial, statutory, an d m a n a g e m e n t reporting.

• O v e r th e p a st 3 to 4 years, th e b ig g e st c h a n g e in th e BPM m ark et has b e e n th e co n so lid a tio n o f th e BPM v en d o rs.

K ey Terms

b u sin ess rep o rt b a la n ce d sco re card (B SC ) b u sin e ss p e rfo rm a n ce

m a n a g e m e n t (B P M ) d ash bo ard s

d ata visualization DMAIC h ig h -p e rfo rm an ce k e y p e rfo rm a n ce in d icato r (K P I) learn in g

p e rfo rm a n ce m e asu re m e n t sy stem s

rep o rt Six Sigm a v isu al analytics

Chapter 4 • B u sin ess Reporting, Visual Analytics, and B u sin ess P erform ance M anagem ent 2 11

Questions for Discussion 1 . What are the best practices in business reporting? How

can w e make our reports stand out? 2 . Why has information visualization become a centerpiece

in the business intelligence and analytics business? Is there a difference between information visualization and visual analytics?

3 . Do you think performance dashboards are here to stay? Or are they about to b e outdated? What do you think will be the next big wave in business intelligence and analytics?

4 . SAP uses the term strategic enterprise management (SEM), Cognos uses the term corporate perform ance management (CPM), and Hyperion uses the term business perform ance m anagement (BPM). Are they referring to the same basic ideas? Provide evidence to support your answer.

5. BPM encompasses five basic processes: strategize, plan, monitor, act, and adjust. Select one o f these processes and discuss the types o f software tools and applications that are available to support it. Figure 4.10 provides some hints. Also, refer to Bain & Company’s list o f man­ agement tools for assistance (bain.com/management_ tools/hom e .asp).

6. Select a public company of interest. Using the compa­ ny’s 2013 annual report, create three strategic financial objectives for 2014. For each objective, specify a strate­ gic goal or target. The goals should b e consistent with the company’s 2013 financial performance.

7 . Netflix’s strategy o f moving to online video downloads has been widely discussed in a number o f articles that

Exercises Teradata University and Other Hands-On Exercises 1. Download Tableau (tableausoftware.com). Using the

Visualization_MFG_Sample data set (available as an Excel file on this b ook ’s Web site), answer the following questions: a. What is the relationship betw een gross b ox office

revenue and other movie-related parameters given in the data set?

b. How does this relationship vary across different years? Prepare a professional-looking written report that is enhanced with screenshots o f your graphical findings.

2. Go to teradatauniversitynetwork.com. Select the “Articles” content type. Browse down the list o f articles and locate one titled “Business/Corporate Performance Management: Changing Vendor Landscape and New Market Targets.” Based on the article, answer the follow­ ing questions: a. What is the basic focus o f the article? b. What are the major “take aways” from die article? c. In the article, which organizational function or role is

most intimately involved in CPM?

can be found online. What are the basic objectives of Netflix’s strategy now? What are some o f the major assumptions underlying the strategy? Given what you know about discovery-driven planning, do these assump­ tions seem reasonable?

8 . In recent years, the Beyond Budgeting Round Table (BBRT; bbrt.org) has called into question traditional budgeting practices. A number o f articles on the Web discuss the BBRT’s position. In the BBRT’s view, what is wrong with today’s budgeting practices? What does the BBRT recommend as a substitute?

9 . Distinguish between performance management and performance measurement.

10. Create a measure for some strategic objective o f interest (you can use one o f the objectives formu­ lated in discussion question 6). For the selected mea­ sure, complete the measurement template found in Table W4.2.1 in the online file for this chapter.

1 1 . Using the four perspectives o f the BSC, create a strat­ egy for a hypothetical company. Express the strategy as a series o f strategic objectives. Produce a strategy map depicting the linkages among the objectives.

12. Compare and contrast the DMAIC model with the closed- loop processes o f BPM.

13. Select two companies that you are familiar with. What terms do they use to describe their BPM initiatives and software suites? Compare and contrast their offerings in terms o f BPM applications and functionality.

d. Which applications are covered by CPM? e. How are these applications similar to or different

from the applications covered by Gartner’s CPM? f. What is GRC, and what is its link to corporate

performance? g. What are some o f the major acquisitions that occurred

in the CPM marketplace over the last couple o f years? h. Select two o f the companies discussed by the article

(not SAP, Oracle, or IBM). What are the CPM strate­ gies o f each o f the companies? What do the authors think about these strategies?

3. Go to teradatauniversitynetwork.com. Select the “Case Studies” content type. Browse down the list o f cases and locate one titled “Real-Time Dashboards at Western Digital.” Based on the article, answer the following questions: a. What is VIS? b. In what ways is the architecture o f VIS similar to or

different from the architecture o f BPM? c. What are the similarities and differences between the

closed-loop processes o f BPM and the processes in the OODA decision cycle?

2 1 2 Part II • D escriptive Analytics

d. What types o f dashboards are in the system? Are they operational or tactical, or are they actually scorecards? Explain.

e. What are the basic benefits provided by Western Digital’s VIS and dashboards?

f. What sorts o f advice can you provide to a com­ pany that is getting ready to create its own VIS and dashboards?

4 . Go to Stephen Few’s blog “The Perceptual Edge” (perceptualedge.com). G o to the section o f “Examples.” In this section, he provides critiques o f various dashboard examples. Read a handful o f these examples. Now go to dundas.com. Select the “Gallery” section o f the site. O nce there, click the “Digital Dashboard” selection. You will be shown a variety o f different dashboard demos. Run a couple o f the demos. a. What sorts o f information and metrics are shown on

the demos? What sorts o f actions can you take? b. Using some o f the basic concepts from Few’s

critiques, describe some o f the good design points and bad design points o f the demos.

5. Download an information visualization tool, such as Tableau, QlikView, or Spotfire. If your school does not have an educational agreement with these companies, then a trial version would b e sufficient for this exercise. Use your ow n data (if you have any) or use one o f the data sets that comes with the tool (they usually have one or more data sets for demonstration purposes). Study the data, com e up with a couple o f business problems, and use data and visualization to analyze, visualize, and potentially solve those problems.

6 . Go to teradatauniversitynetwork.com. Find the “Tableau Software Project.” Read the description, execute the tasks, and answer the questions.

7. Go to teradatauniversitynetwork.com. Find the assign­ ment for SAS Visual Analytics. Using the information and

End-of-Chapter Application Case

Sm art Business Reporting Helps H ealthcare Providers

Premier, which serves more than 2,600 U.S. hospitals and 84,000-plus other healthcare sites, exists to help its members improve the cost and quality o f the care they provide the com­ munities they serve. Premier also assists its members to prepare for and stay ahead o f health reform, including accountable care and other new models o f care delivery and reimbursement.

C h a lle n g e As Premier executives looked to execute this vision, they recognized that the company’s existing technical infrastructure could not support the new model. Over the years, Premier had developed a series o f “siloed” applications, making it difficult for members to connect different data sources and metrics and see die “big picture” o f how to drive healthcare transformation.

step-by-step instructions provided in the assignment, exe­ cute the analysis on the SAS Visual Analytics tool (which is a Web-enabled system that does not require any local installation). Answer the questions posed in the assignment.

8 . Develop a prototype dashboard to display the financial results o f a public company. The prototype can be on paper, on Excel, or on a commercial tool. Use data from the 2012 annual plans o f two public companies to illus­ trate the features o f your dashboard.

Team Assignments and Role-Playing Projects 1. Virtually every BPM/CPM vendor provides case studies

on their Web sites. As a team, select two o f these ven­ dors (you can get their names from the Gartner or AMR lists). Select two case studies from each o f these sites. For each, summarize the problem the customer was trying to address, the applications or solutions implemented, and the benefits the customer received from the system.

2 . Go to the Dashboard Spy Web site map for executive dash­ boards (enterprise-dashboard.com/sitemap). This site provides a number o f examples of executive dashboards. As a team, select a particular industry (e.g., healthcare, banking, airlines). Locate a handful o f example dashboards for that industry. Describe the types o f metrics found on the dashboards. What types o f displays are used to provide the information? Using what you know about dashboard design, provide a paper prototype o f a dashboard for this information.

3. Go to teradatauniversitynetwork.com. From there, go to University o f Arkansas data sources. Choose one o f the large data sets, and download a large number of records (this may require you to write an SQL statement that creates the variables that you want to include in the data set). Come up with at least 10 questions that can be addressed with information visualization. Using your favor­ ite data visualization tool, analyze the data and prepare a detail report that includes screenshots and other visuals.

D eliver B e tte r Care

These platforms and associated software systems also lacked the scalability required to support the massive transaction volumes that were needed. At the same time, as Premier inte­ grates data in new ways, it needs to ensure that the historic high level o f data privacy and security is maintained. Moving forward with new technology, Premier had to confirm that it can isolate each healthcare organization’s information to continue to meet patient privacy requirements and prevent unauthorized access to sensitive information.

S o lu tio n — B r i d g i n g t h e I n f o r m a tio n G ap Premier’s “re-platforming” effort represents groundbreaking work to enable the sharing and analysis o f data from its thou­ sands o f member organizations. The new data architecture

C hapter 4 • Business Reporting, Visual Analytics, and B u sin ess P erform ance M anagem ent 2 1 3

and infrastructure uses IBM software and hardware to deliver trusted information in the right context at the right time to users based on their roles. Using the new platform, Premier members will b e able to use the portal to access the integiated system for various clinical, business, and compliance-related applications. From a clinical aspect, they will have access to best practices from leading hospitals and healthcare experts across the nation and can match patient care protocols with clinical outcomes to improve patient care.

Applications on the new platform will am the gamut from retrospective analysis o f patient populations focused on identifying how to reduce readmissions and hospital-acquired conditions to near—real-time identification o f patients receiv­ ing sub-therapeutic doses o f an antibiotic. Business users within the alliance will b e able to compare the effectiveness o f care locally and with national benchmarks, which will help :hem improve resource utilization, minimizing waste both in healthcare delivery and in administrative costs. Additionally, rhis integrated data will help healthcare organizations con­ tract with payers in support o f integrated, accountable care.

Premier’s commitment to improving healthcare extends beyond its member organizations. As part o f its work, it teamed, with IBM to create an integrated set o f data models and templates that would help other organizations establish a comprehensive data warehouse o f clinical, operational, and outcomes information. This data model, called the IBM Healthcare Provider Data Warehouse (HCPDW), can help healthcare organizations provide their staff with accurate and timely information to support the delivery o f evidence-based, patient-centric, and accountable care.

J o u r n e y t o S m a r t e r D e c is io n s Fundamental to helping Premier turn its vision into reality is an Information Agenda strategy that transforms information into a strategic asset that can b e leveraged across applications, processes, and decisions. “In its simplest form, Premier’s platform brings together information from all areas o f the healthcare system, aggregates it, normalizes it, and bench­ marks it, so it impacts performance while the patient is still in the hospital or the physician’s office,’’ says Figlioli, senior vice president o f healthcare informatics at the Premier healthcare alliance. “We wanted a flexible, nimble partner because this is not a cookie-cutter kind o f project,” says Figlioli. “Premier and IBM brought to the table an approach that was best of breed and included a cultural and partnering dimension that was fundamentally different from other vendors.’’

The organization’s IT division is building its new infra­ structure from the ground up. This includes replacing its exist­ in g x 8 6 servers from a variety o f hardware vendors with IBM POWER7 processor-based systems to gain greater performance 22 a lower cost. In fact, an early pilot showed up to a 50 per- tent increase in processing power with a reduction in costs. Additionally, the company is moving its core data warehouse sd IBM DB2 pureScale, which is highly scalable to support the growing amount o f data that Premier is collecting from its members. As part o f Premier’s platform, DB2 pureScale will

help doctors gain the information they need to avoid patient infections that are common in hospitals, and will help phar­ macists ensure safe and effective medication use.

Data from facility admission, discharge, and transfer (ADT) systems along with departmental systems, such as pharmacy, microbiology, and lab information systems, will be sent to Premier’s core data warehouse as HL7 messages, with near-real-time processing of this data occurring at a rate o f 3,000 transactions per second. With the high perfor­ mance that DB2 data software provides, Premier members can quickly learn o f emerging healthcare issues, such as an increased incidence o f MRSA (a highly drug-resistant version o f staphylococcus aureus bacteria) in a particular area.

Data from IBM DB2 database software will b e loaded into the IBM Netezza data warehouse appliance to enable members to conduct advanced analytics faster and easier than was previously possible. IBM Cognos Business Intelligence will b e used to help members identify and analyze opportu­ nities and trends across their organizations.

IBM InfoSphere software is used to acquire, transform, and create a single, trusted view o f each constituent or entity. The data is then integrated and validated, and clinical or business rules management is applied through WebSphere ILOG software. These rules can help automatically notify clinicians of critical issues, such as the appropriate dosing o f anti-coagulation medication. IBM Tivoli software provides security and service management. Application develop­ ment is built upon Rational® software and a common user experience and collaboration are provided through IBM Connections software.

B u s i n e s s B e n e f its Potential benefits for saving lives, helping people enjoy healthier lives, and reducing healthcare costs are enormous. In one Premier project, 157 participating hospitals saved an estimated 24,800 lives while reducing healthcare spending by $2.85 billion. The new system helps providers better iden­ tify which treatments will enable their patients to live longer, healthier lives. It also supports Premier members’ work to address healthcare reform and other legislative requirements.

“When I think about my children, I think about what it will mean for them to live in a society that has solved the complexities of the healthcare system, so that no matter where they live, no matter what they do, no matter what condition they have, they can have the best possible care,” says Figlioli.

Over the next 5 years, Premier plans to provide its members with many new applications in support o f health­ care reform and other legislative requirements. As capabili­ ties are added, the SaaS model will enable the platform to support its commitment to keep all 2,600 hospital members on the same page, and even expand its user community. With its new approach, Premier IT staff can develop, test, and launch new applications from a central location to pro­ vide users with updates concurrently. This is a lower-cost way to give Premier members an analytics solution with a shorter time to value.

2 1 4 Part II • D escriptive Analytics

Q u e s t i o n s f o r t h e E n d - o f - C h a p t e r

A p p l i c a t i o n C a s e

1 . What is Premier? What does it do? 2 . What w ere the main challenges for Premier to achieve

its vision? 3. What was the solution provided by IBM and other

partners?

References Allen, S. (2010). “Data Visualization.” i n t e r a c t i o n d e s i g n .

s v a . e d u (accessed March 2013). Ante, S., and J. McGregor. (2006, February 13). “Giving the

Boss the Big Picture.” B u sin ess W e ek , b u s i n e s s w e e k . c o m / m a g a z i n e / c o n t e n t / 0 6 _ 0 7 / b 3 9 7 1 0 8 3 - h t m

(accessed January 2010). Colbert, J . (2009, June). “Captain Jack and the BPM Market:

Performance Management in Turbulent Times.” B PM M a g a z in e , b m p m a g . n e t / m a g / c a p t a i n _ j a c k _ b p m (acces­ sed January 2010).

Eckerson, W . (2009, January). “Performance Management Strategies: How to Create and Deploy Effective Metrics.” TDWT B e s t P r a c t ic e s R ep o rt, t d w i . o r g / r e s e a r c h / d i s p l a y . a s p x ? I D = 9 3 9 0 (accessed January 2010).

Eckerson, W. (2006). P e r f o r m a n c e D a s h b o a r d s . Hoboken, N J : Wiley.

Few, S. (2005, Winter). “Dashboard Design: Beyond Meters, Gauges, and Traffic Lights.” B u s in e s s I n t e llig e n c e J o u r n a l , Vol. 10, No. 1.

Few, S. (2007). “Data Visualization: Past, Present and Future.” p e r c e p t u a l e d g e . c o m / a r t i c l e s / W h i t e p a p e r s / D a t a _

V i s u a l i z a t i o n . p d f (accessed March 2013). Few, S. (2008). “Data Visualization and Analysis— B l Blind

Spots.” V isu a l P e r c e p t u a l E d g e, p e r c e p t u a l e d g e . c o m / b l o g / ? p = 3 6 7 (accessed January 2010).

Grimes, S. (2009, May 2). “Seeing Connections: Visualizations Makes Sense o f Data.” In te llig e n t E n terp rise, i . c m p n e t . c o m / i n t e l l i g e n t e n t e r p r i s e / n e x t - e r a - b u s i n e s s - in t e llig e n c e / I n t e llig e n t _ E n t e r p r is e _ N e x t _ E r a _ B I _ V i s u a l i z a t i o n . p d f (accessed January 2010).

Gupta, P. (2006). S ix S ig m a B u s in e s s S c o r e c a r d , 2nd ed. New York: McGraw-Hill Professional.

Hammer, M. (2003). A g e n d a : What E v ery 'B u sin ess M ust D o to D o m i n a t e t h e D e c a d e . Pittsburgh, PA: Three Rivers Press.

Hatch, D. (2008, January). “Operational B l: Getting ‘Real Time’ about Performance.” In te llig en t E n terp rise. i n t e l l i g e n t e n t e r p r i s e . c o m / s h o w A r t i c l e . j h t m l ?

a r t i c l e I D = 2 0 5 9 2 0 2 3 3 (accessed January 2010). Hill, G. (2008). “A Guide to Enterprise Reporting.” g h i l l .

c u s t o m e r . n e t s p a c e . n e t . a u / r e p o r t i n g / c o m p o n e n t s .

h t m l (accessed February 2013). Kaplan, R., and D. Norton. (2008). T h e E x e c u tio n P re m iu m .

Boston, MA: Harvard Business School Press. Kaplan, R., and D. Norton. (2004). S trateg y M ap s: C o n v er tin g

I n t a n g ib le A ssets in t o T a n g ib le O u tco m es. Boston, MA: Harvard Business School Press.

4 . What w ere the results? Can you think o f other benefit coming from such an integrated system?

Source: IBM, Customer Success Story, “Premier Healthcare Alliance Making a Quantum Leap Toward More Integrated Care anc Improved Provider Performance,” ibm.com /smarterplanet/us en/leadership/prem ier/assets/pdf/IBM _Prem ier.pdf (accesses February 2013).

Kaplan, R., and D. Norton. (2000). T h e S tra teg y -F o cu sed O r g a n iz a t io n : H o w B a l a n c e d S c o r e c a r d C o m p a n ie s Thrive i n t h e N ew B u s in e s s E n v ir o n m e n t. Boston, MA: Harvard Business School Press.

Kaplan, R., and D. Norton. (1996). T h e B a l a n c e d S c o r e c a r d : T r a n s la tin g S trateg y in to A c tio n . Boston, MA: Harvard University Press.

Kaplan, R., and D. Norton. (1992, January-February). “The Balanced Scorecard— Measures That Drive Performance. H a r v a r d B u s in e s s R ev iew , pp. 71-79-

Knowledge@W. P. Carey. (2009, March 2). “High-Rolling Casinos Hit a Losing Streak.” k n o w l e d g e . w p c a r e y . a s u . e d u / a r t i c l e . c f m ? a r t i c l e i d = 1 7 5 2 # (accessed January 2010).

Microsoft. (2006, April 12). “Expedia: Scorecard Solution Helps Online Travel Company Measure the Road to Greatness. m i c r o s o f t . c o m / c a s e s t u d i e s / C a s e _ S t u d y _ D e t a i l .

a s p x ? C a s e S t u d y I D = 4 9 0 7 6 (accessed January 2010). Norton, D. (2007). “Strategy Execution— A Competency

That Creates Competitive Advantage.” The Palladium Group. t h e p a l l a d i u m g r o u p . c o m / K n o w l e d g e O b j e c t R e p o s ito ry / N o rto n _ S tra te g y E x e ccre a te s co m p e titiv e a d v W P . p d f (accessed December 2012).

Novell. (2009, April). “Executive Dashboards Elements of Success.” Novell white paper, n o v e l l .c o m /r c /d o c r e p o s i t o r y / p u b li c / 3 7 / b a s e d o c u m e n t .2 0 0 9 - 0 3 - 2 3 . 4 8 7 1 8 2 3 0 l 4 / E x e c u t i v e D a s h b o a r d s _ E l e m e n t s _ o f _ S u c c e s s _ W h ite _ P a p e r _ e n .p d f (accessed January 2013).

Simons, R. (2002). P e r f o r m a n c e M e a s u r e m e n t a n d C o n tro l S ystem s f o r I m p le m e n tin g Strategy. Upper Saddle River, NJ: Prentice Hall.

Six Sigma Institute. (2009). “Lean Enterprise.” s ix s ig m a in s tit u te .c o m /le a n /in d e x _ le a n .s h tm l (accessed August 2009).

Slywotzky, A., and K. Weber. (2007). T h e Upside: T he 7 S tr a te g ie s f o r T u rn in g B ig T h rea ts in t o G row th B r e a k th r o u g h s . New York: Crown Publishing.

Smith, R. (2007, April 5). “Expedia-5 Team Blog: Technology.” e x p e d ia -te a m 5 .b lo g s p o t.c o m (accessed January 2012).

Tufte, E. (2013). “Presenting o f Data and Information.” A resource for Web site designers by Edward Tufte and Dariane Hunt, e d w a r d t u f t e . c o m (accessed February 2013).

Watson, H., and L. Volonino. (2001, January). “Harrah’s High Payoff from Customer Information.” T h e D a t a W a r e h o u s in g In stitu te In d u s tr y S tu d y 2 0 0 0 H a r n e s s in g C u s to m e r I n f o r m a t io n f o r S tr a te g ic A d v a n ta g e : T e c h n ic a l C h a lle n g e s a n d B u s in e s s S o lu tio n s, t e r r y . u g a . e d u / ~ h w a t s o n / H a r r a h s . d o c (accessed January 2013).

Predictive Analytics

LEARNING OBJECTIVES FOR PART III

■ L earn th e ro le o f p re d ictiv e an aly tics (PA) an d d ata m in in g (D M ) in solv in g b u sin ess p ro blem s

* L earn th e p ro c e s s e s a n d m e th o d s fo r co n d u ctin g d ata m in in g p ro je cts

■ Learn th e role an d capabilities o f predictive m od­ eling techniqu es, including artificial neural n et­ w orks (ANN) and supp ort v ecto r m ach in es (SVM)

■ Learn th e c o n te m p o ra ry v ariation s to data m ining, s u ch a s te x t m in in g and W eb m ining

■ G a in fam iliarity w ith th e p ro c e s s , m e th o d s, and a p p lica tio n s o f te x t an aly tics a n d te x t m ining

* Learn th e tax o n o m y o f W eb m in in g so lu tio n s— W eb c o n te n t m ining, W eb u sa g e m ining, and W e b stru cture m ining

* G a in fam iliarity w ith th e p ro c e s s , m e th o d s, a n d ap p licatio n s o f W eb an aly tics an d W e b m ining

Data Mining

LEARNING OBJECTIVES

* D e fin e d ata m in in g as a n e n a b lin g te ch n o lo g y fo r b u sin ess analytics

■ U n d erstand th e o b je c tiv e s a n d b en e fits o f data m ining

■ B e c o m e fam iliar w ith th e w id e ran g e o f ap p lica tio n s o f data m ining

■ L earn th e stand ard ized d ata m ining p ro ce s s e s

a U n d erstand th e s te p s involved in data p re p ro c e s s in g fo r data m ining

* L earn d ifferen t m eth o d s an d algorithm s o f d ata m ining

H B u ild a w aren e ss o f th e e x istin g data m in in g so ftw are to o ls

■ U n d erstan d th e privacy issu es, pitfalls, a n d m yth s o f d ata m ining

Ge n era lly sp e a k in g , data m in in g is a w a y t o d e v e lo p in te llig e n ce (i.e ., a ctio n a b le in fo rm atio n o r k n o w le d g e ) fro m data th a t a n org an ization c o lle cts , organizes, an d stores. A w id e ran g e o f data m ining te c h n iq u e s are b e in g u s e d b y orga­ n izatio n s to g ain a b e tte r u n d erstan d in g o f th e ir cu sto m e rs a n d th e ir o w n op eratio n s a n d to so lv e c o m p le x organ izatio n al p ro b lem s. In this ch a p ter, w e stud y d ata m ining as a n e n a b lin g te ch n o lo g y fo r b u sin e ss analytics, learn a b o u t th e stand ard p ro c e s s e s o f co n d u ctin g d ata m in in g p ro je cts, u n d erstan d an d b u ild e x p e rtis e in th e u s e o f m a jo r data m in in g te ch n iq u e s , d ev elo p a w a ren e ss o f th e e x is tin g softw are to o ls, a n d e x p lo r e privacy issu es, co m m o n m yths, and pitfalls th at are o ften a s s o c ia te d w ith d ata m ining.

5 .1 O p e n in g V ig n e tte : C a b e la ’s R e e ls in M o re C u s to m e rs w ith A d v a n c e d A n a ly tics a n d D a ta M in in g 2 1 7

5 .2 D a ta M in in g C o n c e p ts a n d A p p lic a tio n s 2 1 9 5 -3 D a ta M in in g A p p lic a tio n s 2 3 1 5 . 4 D a ta M in in g P r o c e s s 2 3 4 5 .5 D a ta M in in g M e th o d s 2 4 4 5 .6 D a ta M in in g S o ftw a r e T o o ls 2 5 8 5 -7 D a ta M in in g P riv a c y Is s u e s , M yths, a n d B lu n d e r s 2 6 4

2 1 6

C hapter 5 * Data Mining 2 1 7

5.1 OPENING VIGNETTE: Cabela's Reels in More Customers with Advanced Analytics and Data Mining

A d v anced an aly tics, s u ch a s d ata m ining, h a s b e c o m e a n in tegral part o f m a n y retailers1 d ecisio n -m ak in g p ro c e s s e s . Utilizing large a n d in fo rm atio n -rich tran sactio n al and cu sto m e r d ata (th a t th e y c o lle c t o n a daily b a sis ) to o p tim ize th e ir b u sin e ss p ro c e s s e s is n ot a c h o ic e fo r la rg e -sc a le retailers an y m o re, b u t a n e c e s s ity to stay co m p etitiv e . C a b e la ’s is o n e o f t h o se retailers w h o u n d erstan d s th e v alu e p ro p o s itio n an d strives to fu lly utilize th eir data assets.

BACKGROUND

Started aro u n d a k itc h e n ta b le in C h ap p ell, N ebraska, in 1961, C a b e la ’s h a s g ro w n to b e c o m e th e larg est d irect m ark eter, a n d a lea d in g sp e cialty retailer, o f h u n tin g , fishing, cam p in g , a n d re la te d o u td o o r m e rch a n d ise w ith $ 2 .3 b illio n in sales. C red ited larg ely to its in form ation te c h n o lo g y a n d an alytics p ro je c t initiatives, C a b e la ’s has b e c o m e o n e o f th e v e ry fe w truly o m n i-ch a n n e l retailers (a n ad v an ce d fo rm o f m u lti-ch an n el re ta ile r w h o co n cen trate o n a s e a m le s s a p p ro a ch to th e c o n s u m e r e x p e rie n c e th ro u g h all available sh op p in g c h a n n e ls, in clu d in g b rick s-an d -m ortar, telev isio n , catalo g , an d e -c o m m e r c e — d irou gh co m p u te rs a n d m o b ile d ev ice s).

E ssentially, C a b e la ’s w a n te d to h av e a sin g le v ie w o f th e cu sto m ers a c ro ss m ultiple ch a n n e ls to b e tte r fo c u s its m ark etin g efforts a n d drive in cre a se d sales. F o r m o re th an a d e ca d e , C a b e la ’s h a s re lie d o n SAS statistics a n d data m in in g to o ls to h e lp a n a ly z e the data it g ath e rs fro m sa le s tran sactio n s, m ark et re sea rch , an d d em o g ra p h ic d ata a s s o ­ ciated w ith its la rg e d atab ase o f cu stom ers. “U sing SAS d ata m ining to o ls, w e create pred ictive m o d e ls to o p tim ize cu sto m e r s e le c tio n fo r all cu sto m e r con tacts. C a b e la ’s u se s m e se p re d ictio n s c o r e s to m ax im ize m arketin g s p e n d a cro ss ch a n n e ls an d w ith in e a ch custom er's p e rs o n a l c o n ta c t strategy. T h e s e e ffo rts h av e a llo w e d C a b e la ’s to co n tin u e its grow th in a p ro fita b le m an n e r,” says C o rey B erg stro m , d irecto r o f m ark e tin g re se a rch m d analytics fo r C a b e la ’s. “W e ’r e n o t talking sin gle-d igit grow th. O v e r sev eral years, it’s dou ble-digit g ro w th .”

USING THE BEST O F THE BREED (SAS AND TERADATA) FO R ANALYTICS

By d ism antling th e in form ation silo s e x istin g in d ifferen t b ra n ch e s, C a b e la ’s w a s a b le to create w h a t T illo ts o n (m a n a g e r o f cu sto m e r an alytics at C a b e la ’s ) calls “a h o listic v iew of th e cu sto m e r.” “U sin g SAS a n d T erad ata, o u r statisticians w e re a b le to c re a te th e first com p lete p ictu re o f th e cu sto m ers and co m p a n y activities. T h e flex ib ility o f SAS in taking ia ta fro m m u ltip le s o u rc e s, w ith o u t h e lp fro m IT, is critical.”

As th e v o lu m e a n d co m p le x ity o f d ata in cre a se s, s o d o e s th e tim e s p e n t o n p rep aring and analy zing it. F a s te r a n d b e tte r analysis results c o m e s fro m tim ely an d th ro u g h m o d e l­ ing o f large data s o u rc e s. F o r that, an in teg ratio n o f d ata and m o d e l b u ild in g algorithm s is n e ed e d . T o h e lp org an izatio n s m e e t th e ir n e e d s fo r s u ch integrated so lu tio n s, SAS recently jo in e d fo r c e s w ith T erad ata ( o n e o f th e lea d in g p ro vid ers o f d ata w a reh o u sin g e lu t io n s ) to c re a te to o ls an d te ch n iq u e s aim ed a t im p ro vin g s p e e d an d a c c u ra c y fo r p re- i c t i v e an d e x p la n a to ry m od els.

P rior to th e in te g ratio n o f SAS an d T e rad ata, d ata fo r m o d e lin g a n d s c o rin g cu s- ::m e r s w a s s to re d in a d ata m art. T h is p ro ce s s re q u ired a large am o u n t o f tim e to c o n ­ f e c t , b rin g in g to g e th e r d isp arate d ata so u rce s and k e e p in g statisticians fro m w o rk in g □n analytics. O n a v e ra g e , th e statisticians s p e n t 1 to 2 w e e k s p e r m o n th ju st b u ild in g th e data. N ow , w ith th e in teg ratio n o f th e tw o sy stem s, statisticians c a n le v e ra g e th e p o w e r o f SA S using th e T e ra d a ta w a re h o u s e as o n e s o u rc e o f in form ation rath er th a n th e m u ltiple

2 1 8 Part III • Predictive Analytics

s o u rc e s th at e x is te d b e fo re . T h is c h a n g e h a s p ro v id e d th e o p p ortu n ity to b u ild m o d els fa ste r and w ith less d ata la te n cy u p o n e x e cu tio n .

“W ith th e SAS [and] T erad ata in teg ratio n w e h a v e a lo t m o re flexibility. W e c a n u se m o re data a n d b u ild m o re m o d e ls to e x e c u te fa s te r,” says D e a n W y n k o o p , m an ag e r o f d ata m a n a g e m e n t fo r C ab e la’s.

T h e in tegratio n e n a b le d C a b e la ’s to b rin g its d ata c lo s e to its an aly tic fu n ctio n s in s e c o n d s v ersu s d ays o r w e e k s . It c a n a lso m o re e a sily find th e high est-v alu e cu stom ers in th e b e s t lo ca tio n s m o st lik e ly to b u y via th e b e s t ch a n n e ls. T h e in teg rated solu tion re d u ce s th e n e e d to c o p y data fro m o n e sy stem t o a n o th e r b e fo re analy zing th e m ost lik e ly in d icato rs, allo w in g C ab e la’s to ru n re lated q u e ries and flaggin g p o ten tia lly ideal n e w p ro sp ects b e fo re th e co m p etitio n d o e s . A nalytics h e lp s C a b e la ’s to

• Im p ro v e the r e t u r n o n its d ir e c t m a r k e t in g inv estm en t. In ste a d o f costly m ass m ailings to e v ery zip c o d e in a 1 2 0 -m ile radius o f a sto re, C ab e la s u s e s p re ­ d ictive m o d e lin g to fo c u s its m arketin g e ffo rts w ithin th e g e o g ra p h ie s o f cu stom ­ ers m o st lik ely to g e n e ra te th e g re ate st p o ss ib le in cre m en tal sa le s, resultin g in a 6 0 p e rc e n t in c re a se in re s p o n s e rates.

• S elect o p tim a l s ite lo ca tio n s. “P e o p le u s e d to c o m e to u s w ith s u g g e s tio n s o n w h e r e th e y ’d lik e o u r n e x t s to r e to b e b u ilt,” s a y s S arah J a e g e r , m a rk e tin g s ta tisticia n . “As w e m o v e fo rw ard , w e p r o a c tiv e ly le v e ra g e d ata to m a k e re ta il site s e le c t io n s .”

• U n d e rs ta n d the v a lu e o f cu sto m ers a c ro s s a ll c h a n n e ls . W ith d eta iled cu s­ to m er activity a cro ss sto re, W e b site, a n d c a ta lo g p u rch a ses, SAS h e lp s C ab e la s b u ild p red iction , clu sterin g , a n d a s s o cia tio n m o d e ls that rate cu sto m ers o n a five- star system . T h is sy stem h e lp s e n h a n c e th e cu sto m e r e x p e r ie n c e , o ffe rin g cu sto m e r serv ice rep s a c le a r u n d erstan d in g o f th at cu sto m e r’s v alu e to b e tte r p e rso n alize th e ir in te ractio n s. “W e treat all cu sto m e rs w e ll, b u t w e c a n d e v e lo p strateg ies to treat h ig h er-v alu e cu sto m ers a little b e tte r,” say s J o s h C o x, m arketin g statistician.

• D e s ig n p ro m o tio n a l o ffe r s that best e n h a n c e sa les a n d p ro fita b ility . W ith insights g a in ed fro m SAS A nalytics, C a b e la ’s h a s learn ed th at w h ile p ro m o tio n s g e n ­ erate o n ly m arginal ad ditional cu sto m e r sp e n d in g o v e r th e lo n g hau l, th e y d o bring cu sto m ers into th e ir sto res o r to th e In te rn e t fo r ca ta lo g p u rch ases.

• T a ilo r d ir e c t m a r k e t in g o ffe r s to c u s to m e r p r e f e r e n c e s . C a b e la ’s c a n identify th e cu sto m e r’s fav orite c h a n n e l and s e le ctiv e ly s en d related m arketin g m ateri­ als. “D o e s th e cu sto m e r lik e th e 1 0 0 -p a g e ca ta lo g s o r th e 1 ,5 0 0 -p a g e catalogs?” B erg stro m says. “T h e cu sto m e r te lls us this th ro u g h h is p ast in te ractio n s s o w e c a n s en d th e c a ta lo g that m a tch e s h is o r h e r n e e d s . SAS g iv e s C a b e la ’s th e p o w e r to c o n c e iv a b ly p e rso n a liz e a u n iq u e m ark etin g m e ssa g e, flyer, o r c a ta lo g to e v ery cu s­ to m er. T h e o n ly lim itation is th e c re a tio n o f e a c h p ie c e ,” B erg stro m says.

T h e in teg rated an alytics so lu tio n (SAS A nalytics w ith th e T erad ata in -d atabase so lu tio n ) a llo w e d C a b e la ’s to p e rso n a liz e ca ta lo g o fferin g s; s e le c t n e w sto re lo ca tio n s an d e stim ate th eir first-year sales; c h o o s e u p -sell o ffe rin g s that in c re a se p rofits; a n d s c h e d ­ u le p ro m o tio n s to drive sales. B y d o in g so , th e c o m p a n y has e x p e rie n c e d d o u b le-d ig it grow th. “O u r statisticians in th e past s p e n t 7 5 p e r c e n t o f th eir tim e ju st trying to m an ag e d ata. N ow th e y h a v e m o re tim e fo r analyzing th e data w ith SAS. A nd w e h a v e b e c o m e m o re fle x ib le in th e m ark etp lace. T h a t is ju st p r ic e le s s .” W y n k o o p says.

C a b e la ’s is cu rren tly w o rk in g o n analyzing th e click stre am pattern s o f cu stom ers sh o p p in g o n lin e . Its g o al is to p u t th e p e rfe c t o ffe r in fro n t o f th e cu sto m e r b a se d o n h istorical pattern s o f sim ilar sh o p p e rs. “It is b e in g te s te d an d it w o rk s— w e ju st n e e d to p ro d u ctio n alize it,” B erg stro m says. “T h is w o u ld n o t b e p o ss ib le w itho u t th e in -d atab ase p ro ce s s in g cap ab ilitie s o f SAS, to g e th e r w ith T e ra d a ta ,” W y n k o o p says.

C h a p te r5 * D ata Mining 2 1 9

QUESTIONS F O R THE OPENING VIGNETTE

1 . W h y sh o u ld retailers, e sp e cia lly o m n i-ch a n n e l retailers, pay e x tra a tte n tio n to ad v a n ce d an aly tics an d d ata mining?

2 . W h at are th e to p c h a lle n g e s fo r m u lti-ch an n el retails? C an y o u th in k o f o th e r indu stry se g m e n ts that fa ce sim ilar problem s?

3- W h at are th e s o u rce s o f d ata th a t retailers s u c h as C a b e la ’s u s e fo r th e ir d ata m ining projects?

4 . W h at d o e s it m e a n to hav e a “sin g le v ie w o f th e cu sto m e r”? H o w c a n it b e accom p lish ed ?

5 . W hat ty p e o f analytics h e lp did C a b e la ’s g e t fro m th e ir efforts? C an you th in k o f an y o th e r p o te n tia l b e n e fits o f an alytics fo r la rg e-sc a le retailers lik e C a b e la ’s?

6. W h at w as th e re a so n fo r C a b e la ’s to b rin g to g eth e r SAS and T erad ata, th e tw o lead in g v e n d o rs in an aly tics m arketplace?

7 . W h at is in -d a ta b a se analytics, and w h y w o u ld y o u n e e d it?

WHAT W E CAN LEARN FROM THIS VIGNETTE

T h e re ta il in d u stry is a m o n g s t th e m o s t c h a lle n g in g b e c a u s e o f th e c h a n g e th at th ey hav e to d e a l w ith c o n s ta n tly . U n d e rs ta n d in g c u s to m e r n e e d s an d w a n ts , lik e s an d d islik e s, is a n o n g o in g c h a lle n g e . O n e s w h o a re a b le to c r e a te a n in tim ate re la tio n sh ip th ro u g h a “h o lis tic v ie w o f th e c u s to m e r ” w ill b e th e b e n e fic ia r ie s o f th is s e e m in g ly c h a o tic e n v iro n m e n t. I n th e m id st o f th e s e c h a lle n g e s , w h a t w o rk s in fa v o r o f th e s e retailers is th e a v a ila b ility o f th e te c h n o lo g ie s to c o lle c t a n d a n a ly z e d ata a b o u t th e ir cu sto m e rs. A p p ly in g a d v a n c e d a n a ly tics to o ls ( i .e ., k n o w le d g e d is c o v e ry te c h n iq u e s ) io th e s e d ata s o u r c e s p ro v id e th e m w ith th e in s ig h t th a t th e y n e e d fo r b e tt e r d e c is io n m akin g. T h e r e fo r e th e reta il in d u stry h a s b e c o m e o n e o f th e le a d in g u s e rs o f th e n e w fa c e o f a n a ly tics . D ata m in in g is th e p rim e c a n d id a te fo r b e tte r m a n a g e m e n t o f this d a ta -rich , k n o w le d g e - p o o r b u s in e s s e n v iro n m e n t. T h e stu d y d e s c r ib e d in th e o p e n in g v ig n e tte c le a r ly illu stra tes th e p o w e r o f a n a ly tics a n d d ata m in in g to c r e a te a h o listic v ie w o f th e c u s to m e r fo r b e tt e r c u s to m e r re la tio n sh ip m a n a g e m e n t. In th is c h a p te r, you w ill s e e a w id e v a rie ty o f d ata m in in g a p p lic a tio n s s o lv in g c o m p le x p ro b le m s in a v arie ty o f in d u stries w h e r e th e d ata is u s e d to le v e ra g e c o m p e titiv e b u s in e s s ad v an tag e.

Sources: SAS, Customer Case Studies, sas.com /success/cabelas.htm l; and Retail Information Systems News, April 3, 2012, http://risnews.edgl.com /retail-best-practices/W hy-Cabela-s-Has-Emerged-as-the- T op-Omni-Channel-Retailer79 4 7 0 .

5.2 D A TA M INING CO N CEPTS A N D A P PLICA TIO N S In a n in te rv ie w w ith C om p u terw orld m a g a z in e in Ja n u a r y 1 9 9 9 , D r. A rn o P e n z ia s (N o b e l lau reate a n d fo rm e r c h i e f s c ie n tis t o f B e ll L a b s) id e n tifie d d ata m in in g fro m o rg a n iz a ­ tio n al d a ta b a s e s a s a k e y a p p lic a tio n fo r co r p o ra tio n s o f th e n e a r fu tu re. I n re s p o n s e to C om p u terw orld s a g e -o ld q u e s tio n o f “W h a t w ill b e th e k ille r a p p lic a tio n s in th e co rp oration ?” D r. P e n z ia s re p lie d : “D ata m in in g .” H e th e n a d d e d , “D a ta m in in g w ill b e c o m e m u c h m o r e im p o rta n t a n d c o m p a n ie s w ill th ro w aw ay n o th in g a b o u t th e ir c u s ­ to m ers b e c a u s e it w ill b e s o v a lu a b le . I f y o u ’re n o t d o in g th is, y o u ’r e o u t o f b u s in e s s .” Similarly, in a n a rtic le in H a rv a rd B u sin ess R eview , T h o m a s D a v e n p o rt ( 2 0 0 6 ) arg u ed m at th e la te s t s tra te g ic w e a p o n fo r c o m p a n ie s is a n a ly tica l d e c is io n m a k in g , p ro v id in g

2 2 0 Part III • Predictive Analytics

e x a m p le s o f c o m p a n ie s s u c h as A m a z o n .c o m , C ap ital O n e , M arriott In te rn a tio n a l, an d o th e rs th a t h a v e u s e d a n a ly tics to b e tte r u n d e rs ta n d th e ir c u sto m e rs an d o p tim iz e th eir e x te n d e d su p p ly c h a in s to m a x im iz e th e ir re tu rn s o n in v e stm e n t w h ile p ro v id in g th e b e s t cu sto m e r s e r v ic e . T h is le v e l o f s u c c e s s is h ig h ly d e p e n d e n t o n a c o m p a n y u n d e r­ s ta n d in g its cu sto m e rs , v e n d o rs, b u s in e s s p r o c e s s e s , an d th e e x te n d e d su p p ly c h a in v e ry w ell.

A large p o itio n o f “u n d erstan d in g th e cu sto m e r” c a n c o m e from analyzing th e vast am o u n t o f data that a c o m p a n y c o lle cts . T h e c o s t o f sto rin g an d p ro ce s s in g d ata has d e c r e a s e d dram atically in th e r e c e n t past, a n d , as a result, th e a m o u n t o f data sto red in e le c tro n ic fo rm h a s g ro w n at a n e x p lo s iv e rate. W ith th e c re a tio n o f large d atab ases, the p o ssib ility o f a nalyzing th e d ata s to re d in th e m h a s em e rg ed . T h e term d a ta m in in g w as originally u se d to d e s c rib e th e p ro c e s s th ro u g h w h ic h p rev iou sly u n k n o w n pattern s in d ata w e r e d isco v ered . T h is d efin itio n has s in c e b e e n s tre tch e d b e y o n d th o s e limits by s o m e so ftw are v e n d o rs to in clu d e m o st fo rm s o f data analysis in o rd e r to in c re a se sales w ith th e p o p u larity o f th e data m in in g la b e l. In this ch a p ter, w e a c c e p t the o rigin al d efini­ tio n o f data m ining.

A lth o u g h th e term d a ta m in in g is re lativ ely n e w , th e id e as b e h in d it a re not. M any o f the te ch n iq u e s u s e d in data m in in g h av e th e ir ro o ts in traditional statistical analysis a n d artificial in te llig e n ce w o rk d o n e sin c e th e early part o f th e 1980s. W hy, th e n , has it su d d en ly g a in ed th e atten tio n o f th e b u sin e ss w orld? F o llo w in g a re s o m e o f m o st p ro ­ n o u n c e d reason s:

• M ore in te n se co m p e titio n a t th e g lo b a l s c a le d riven b y cu sto m e rs’ e v e r-ch an g in g n e e d s a n d w a n ts in a n in cre asin g ly satu rated m ark etp lace.

• G e n e ra l re co g n itio n o f th e u n ta p p e d v a lu e h id d en in large d ata so u rce s. • C o n so lid a tio n and in teg ratio n o f d a ta b a se reco rd s, w h ich e n a b le s a sin gle v ie w o f

cu sto m e rs, v e n d o rs, tran saction s, etc. • C o n so lid atio n o f d a ta b a se s and o th e r d ata re p o sito rie s in to a sin gle lo ca tio n in th e

fo rm o f a data w a reh o u se . • T h e e x p o n e n tia l in c re a se in d ata p ro c e s s in g a n d sto rag e te ch n o lo g ie s. • Sign ifican t re d u ctio n in th e c o s t o f hard w are and so ftw are fo r data sto rag e and

p ro cessin g . • M o vem en t tow ard th e d e-m assificatio n (c o n v e rs io n o f in form ation re so u rce s into

n o n p h y sical fo rm ) o f b u sin e ss p ractices.

D a ta g e n e r a te d b y th e I n te r n e t is in c r e a s in g ra p id ly in b o th v o lu m e a n d c o m p le x ity . L arge a m o u n ts o f g e n o m ic d a ta a r e b e in g g e n e r a te d a n d a c c u m u la te d all o v e r th e w o rld . D is c ip lin e s s u c h a s a s tr o n o m y a n d n u c le a r p h y s ic s c r e a te h u g e q u a n titie s o f d ata o n a re g u la r b a sis . M e d ic a l a n d p h a r m a c e u tic a l r e s e a r c h e r s c o n ­ s ta n tly g e n e r a te a n d s to r e d a ta th a t c a n th e n b e u s e d in d ata m in in g a p p lic a tio n s to id e n tify b e tt e r w a y s to a c c u r a te ly d ia g n o s e a n d tre a t illn e s s e s a n d to d is c o v e r n e w a n d im p ro v e d d ru gs.

O n th e co m m e rcia l sid e, p e rh a p s th e m o st co m m o n u s e o f d ata m in in g h a s b e e n in th e fin a n ce , retail, a n d h e a lth ca re secto rs. D ata m in in g is u sed to d e te c t and re d u ce frau d u len t activities, e sp e cia lly in in su ra n ce cla im s a n d cre d it card u s e (C h an e t al., 1 9 9 9 ); to id entify cu sto m e r b u y in g p attern s (H o ffm an , 1 9 9 9 ); to re cla im p ro fitab le cu sto m ers (H o ffm an , 1 9 9 8 ); to identify trading rules fro m h istorical data; a n d to aid in in cre ase d p rofitability u sin g m a rk e t-b a s k e t analysis. D ata m ining is alread y w id ely u s e d to b e t­ te r target clien ts, an d w ith th e w id esp rea d d e v e lo p m e n t o f e -c o m m e r c e , this c a n only b e c o m e m o re im perative w ith tim e. S e e A p p licatio n C ase 5.1 fo r in form ation o n h o w Infinity P&C has u s e d p red ictiv e an alytics a n d data m in in g to im p ro ve cu sto m e r serv ice, c o m b a t fraud, a n d in cre a se profit.

Chapter 5 * Data Mining 2 2 1

Application Case 5.1 Smarter Insurance: Infinity P&C Improves Customer Service and Combats Fraud with Predictive Analytics

Infinity P ro p e rty & C asu alty C o rp o ratio n , a p ro vid er o f n o n stan d ard p e rs o n a l au to m o b ile in su ra n ce w ith a n e m p h a sis o n h ig h er-risk d rivers, d e p e n d s o n its ability to identify frau d ulen t claim s fo r su stain ed profitability. As a result o f im p lem en tin g analytics to o ls (fro m IBM SP SS), Infinity P&C h a s d o u b le d th e a ccu ra cy o f its fraud id en tificatio n , co n trib u tin g to a retu rn o n in v e stm e n t o f 4 0 3 p e rce n t p e r a N ucleus R e se a rch study. A n d th e b e n e fits d o n ’t sto p there: A cco rd in g to B ill D ib b le , s e n io r v ice p resid en t in Claim s O p e ra tio n s at Infinity P&C, th e u se o f p re ­ d ictive an alytics in serv in g th e c o m p a n y ’s legitim ate claim an ts is o f e q u a l o r e v e n g re a te r im p o rtan ce.

L o w -H a n g in g F r u i t

Initially, D ib b le fo cu se d th e p o w e r o f predictive a n a ­ lytics (i.e ., data m ining ) to assist th e co m p an y ’s S p ecial Investigative Unit (SIU ). “In th e early days o f SIU, adjusters w o u ld u s e lam in ated card s w ith ‘red flags’ to indicate p otential fraud. T ak in g th o se ‘red flags’ and d ev elop in g rules s e e m e d lik e an area o f low -hanging fruit w h ere w e co u ld q u ickly dem onstrate th e b en e fit o f o u r investm ent in predictive analytics.”

D ib b le th e n lev era g e d a su cce ssfu l a p p ro a ch fro m an o th e r p art o f th e b u sin ess. “W e re co g n iz e d h o w im portant cre d it w a s in th e underw riting aren a, and I th o u g h t, ‘Let’s s c o r e o u r claim s in th e sa m e w ay , to give u s a n in d icato r o f p o ten tial frau d .’ T h e larger th e n u m b e r w e a tta ch to a c a s e , th e m o re ap t w e are to h av e a frau d situation. L o w er n u m b e r, g e t th e claim p aid .” D ib b le n o te s th at fraud re p re sen ts a $ 2 0 b illio n e x p o s u re to th e in su ra n ce industry an d in ce rta in v e n u e s c o u ld b e a n e le m e n t in aro u n d 40 p e rce n t o f claim s. “A k e y b e n e fit o f th e IBM SPSS system is its a b ility to co n tin u ally an alyze an d s c o re th e se claim s, w h ic h h e lp s e n su re th at w e g e t th e claim to th e right ad ju ster at th e right tim e ,” h e says.

Adds T o n y Sm arrelli, v ic e p resid en t o f National O perations: “Ind ustry reports estim ate o n e o u t o f five claim s is p u re fraud— eith er op p ortu nity fraud, w h ere s o m e o n e e x a g g e ra te s a n injury o r v e h icle d am ag e, o r the h ard -co re crim inal rings th at w o rk w ith u n eth ical clinics and atto rn eys. R ather than putting all five

cu stom ers throu g h a n investigatory p ro cess, SPSS h e lp s u s ‘fast-track ’ fo u r o f th e m an d c lo s e th eir cases w ithin a m atter o f days. T h is results in m u ch h ap p ie r cu stom ers, con trib u tes to a m o re e fficie n t w o rkflow w ith im p ro ved c y c le tim es, an d im proves retention d u e to a n overall b etter claim s e x p e rie n c e .”

An U n expected B e n e fit D ib b le sa w s u b ro g a tio n , th e p ro c e s s o f c o lle ctin g d a m a g e s fro m th e at-fau lt driver’s in su ra n ce c o m ­ p an y , as a n o th e r p ie c e o f lo w -h an g in g fruit— an d h e w a s right. In th e first m o n th o f u sin g SPSS, Infinity P& C s a w re co rd re c o v e ry o n p aid co llisio n claim s, ad d in g a b o u t $1 m illion d irectly to th e co m p a n y ’s b o tto m lin e an d virtu ally e lim in atin g th e third-party c o lle c tio n fe e s o f m o re th a n $ 7 0 ,0 0 0 p e r m o n th that th e c o m p a n y w a s u s e d to paying. W h at’s m o re, e a c h o f th e fo llo w in g 4 o r 5 m o n th s w a s e v e n b etter th a n th e p rev iou s o n e . "I n e v e r th o u g h t w e w ould re c o v e r th e m o n e y th at w e ’v e re c o v e re d w ith SPSS in th e su b ro g a tio n a r e a ,” h e says. “T h a t w a s a real su rp rise to us. It b ro u g h t a lo t o f atten tion to SPSS w ith in th e co m p a n y , a n d to th e v alu e o f p red ictive an alytics in g e n e r a l.”

T h e ru le s-b a se d IB M SPSS so lu tio n is w ell su ited to Infinity P&C’s b u sin ess. F o r e x a m p le , in states that h av e n o -fa u lt b e n e fits , a n in su ra n ce co m ­ p an y ca n re c o v e r co m m e rcia l v e h icle s o r v e h icle s o v e r a certain g ro ss v e h ic le w eig h t. “W e c a n p u t a rule in IBM SPSS that i f m ed ical e x p e n s e s are paid o n a claim involving th is typ e o f v e h ic le , it is im m e­ d iately re fe rre d to th e su b ro g a tio n d ep a rtm e n t,” e x p la in s D ib b le . “T h is is a re al-tim e ability that k e e p s u s fro m m issin g p o ten tially v a lu a b le s u b ro ­ g a tio n op p o rtu n itie s, w h ic h u s e d to h a p p e n a lot w h e n w e re lie d s o le ly o n ad ju ster intu ition.”

T h e ru le s a re ju st as im portant o n th e fraud in v estigation sid e. C o n tin u e s D ib b le : “I f w e s e e a n a c c id e n t that h a p p e n e d a ro u n d 1 : 0 0 a .m . and in v o lv e d a g as-g u zzlin g GM C S u b u rban , w e n e e d to start lo o k in g fo r fraud. So w e dig a little d ee p e r: Is th is g u y u p sid e -d o w n o n h is lo an , s u ch that h e o w e s m o re m o n e y th a n th e c a r is w orth? D id

( C o n t i n u e d )

2 2 2 Part III • Predictive Analytics

Application Case 5.1 (Continued) th e a c c id e n t h a p p e n in a re m o te sp ot, su ggestin g that it m ay h av e b e e n staged? D o e s th e individual m o v e fre q u e n tly o r list m u ltip le ad dresses? As th e s e e le m e n ts a re ad d ed to th e e q u a tio n , th e s c o r e k e e p s b uild in g, a n d th e c a s e is m o re a n d m o re likely to b e re fe rre d t o o n e o f o u r SIU inv estig ato rs.”

W ith SPSS, In fin ity P&C h a s re d u ce d SIU re fe r­ ral tim e fro m a n av e ra g e o f 4 5 - 6 0 days to ap p ro x i­ m ate ly 1 - 3 days, w h ich m e a n s th at investigato rs c a n g e t to w o rk o n th e c a s e b e fo re m e m o rie s an d stories start to c h a n g e , ren tal and s to ra g e ch a rg e s m ount, a n d th e lik e lih o o d o f g etting a n attorney in volved in cre a se s. T h e co m p a n y is a lso creatin g a b etter cla im fo r th e SIU to in vestigate; a h ig h er s c o re co r­ re la tes to a h ig h e r p ro b ab ility o f fraud.

M a k in g U s S m a r t e r

SPSS ru le s start to s c o re th e claim im m ediately o n first n o tic e o f lo ss (FN O L) w h e n th e claim an t rep o rts th e a ccid e n t. “W e h av e co m p le te ly revised o u r FN O L sc re e n s to c o lle c t m o re data p o in ts ,” says D ib b le . “SPSS h a s m ad e u s m u ch sm arter in ask in g q u e s tio n s .” Currently SPSS c o lle c ts data m ainly from th e c o m p a n y ’s claim s an d p o lic y sy stem s; a future initiative to le v e ra g e th e p ro d u ct’s te x t m ining c a p a ­ b ilities w ill m ak e th e in form ation in claim s n o tes av a ila b le as w ell.

H av in g p ro v e n its v a lu e in su b ro g a tio n and SIU , th e SPSS so lu tio n is p o ise d fo r e x p a n s io n w ithin Infinity P&C. “O n e o f o u r k e y o b je c tiv e s m o vin g for­ w ard w ill b e w h a t w e call ‘right scrip tin g ,’ w h ere

w e ca n script th e ap p ro p riate q u e stio n s fo r ca ll c e n ­ ter ag e n ts b a s e d o n th e an sw ers th ey g e t fro m the cla im a n t,” s a y s D ib b le . “W e ’ll a ls o b e institu ting a p ro ce s s to fla g claim s w ith h ig h litigation potential. B y rev iew in g p a s t litigation claim s, w e c a n identify p red ictiv e traits a n d h an d le th o se c a s e s o n a priority b a sis .” D e c is io n m an ag e m e n t, cu sto m e r retention, p ricing analy sis, a n d d ash b o ard s a re a lso p otential future ap p lica tio n s o f SPSS te ch n o lo g y .

B u t at th e e n d o f th e day, e x c e lle n t cu sto m e r s erv ice re m ain s th e d riving fo rc e b e h in d Infinity P&C’s u s e o f p red ictiv e analytics. C o n clu d es D ib b le : “My g o al is to p ay th e legitim ate cu sto m e r very q u ick ly a n d g e t him o n h is w ay. P e o p le w h o are m o re e c o n o m ic a lly ch a lle n g e d n e e d th e ir ca r; they typ ically d o n ’t h av e a sp a re v e h icle . T h is is th e car th e y u se to g o b a c k and forth to w o rk , so I want, to g e t th em o u t an d o n th e road w itho u t d elay. IBM SP SS m ak e s th is p o s s ib le .”

Q u e s t i o n s f o r D i s c u s s i o n 1. H ow d id Infinity P&C im p ro ve cu sto m e r service

w ith d ata mining? 2. W h at w e r e t h e c h a lle n g e s, th e p ro p o s e d solu tion,

and th e o b ta in e d results? 3. W h a t w a s th e ir im p lem e n tatio n strategy? W h y is

it im p ortan t to p ro d u ce results a s early as p o s­ s ib le in d ata m in in g studies?

Source: public.dhe.ibm.com/common/ssi/ecm/en/ytc03l60 usen/YTC03l60U SEN .PD F (accessed January 2013)-

D e f in it io n s , C h a r a c te r is t ic s , a n d B e n e fits

Sim ply d efin e d , data m ining is a term u s e d t o d e s c rib e d isco v e rin g o r “m in in g” k n o w l­ e d g e fro m large am o u n ts o f data. W h e n c o n s id e re d b y a n alo g y , o n e c a n e asily realize th at th e term d a t a m i n i n g is a m isn o m er; th a t is, m ining o f g o ld from w ith in ro c k s o r dirt is re fe rre d to as “g o ld ” m in in g rath er th a n “r o c k ” o r “dirt” m ining. T h e re fo re , data m in in g p e rh a p s sh ou ld h a v e b e e n n a m e d “k n o w le d g e m in in g ” o r “k n o w le d g e d iscov ­ e ry .” D e sp ite th e m ism atch b e tw e e n th e te rm and its m ean in g , d a t a m i n i n g has b e c o m e th e c h o ic e o f th e com m u nity . M any o th e r n a m e s th at are a s s o cia te d w ith d ata m ining in clu d e k n o w l e d g e e x t r a c t i o n , p a t t e r n a n a l y s i s , d a t a a r c h a e o l o g y , i n f o r m a t i o n h a r v e s t -

in g , p a t t e r n s e a r c h i n g , a n d d a t a d r e d g i n g . T e c h n ic a lly sp e a k in g , d ata m in in g is a p r o c e s s that u s e s statistical, m ath em atical,

an d artificial in te llig e n ce te c h n iq u e s to e x tra ct a n d identify u sefu l in fo rm atio n and s u b s e ­ q u e n t k n o w le d g e (o r p attern s) fro m large sets o f d ata. T h e s e pattern s c a n b e in th e form

Chapter 5 • Data M ining 223

o f b u sin e ss ru les, affinities, co rrelatio n s, trend s, o r p re d ictio n m o d e ls (s e e N em ati and B a rk o , 2 0 0 1 ). M o st literatu re d e fin e s d ata m ining as “th e nontrivial p ro c e s s o f identifying valid, n o v e l, p o ten tia lly usefu l, a n d u ltim ately u n d e rstan d ab le p attern s in data s to re d in stru ctured d a ta b a s e s ,” w h e re th e data are o rg an ize d in re co rd s stru ctured b y ca te g o rica l, ordinal, a n d co n tin u o u s v a ria b le s (Fay yad e t al., 1 9 9 6 ). In this d efin ition , th e m e a n in g s o f th e k e y term s are as fo llo w s:

• P rocess im p lies th a t d ata m in in g co m p rise s m any iterative steps. • N on trivial m e a n s th a t s o m e e x p e rim e n ta tio n -ty p e s e a rc h o r in fe re n c e is involved ;

th at is, it is n o t a s straightforw ard as a co m p u ta tio n o f p re d efin e d q uantities. • V alid m e a n s th a t th e d isco v e red pattern s sh o u ld h o ld tru e o n n e w d ata w ith

su fficien t d e g r e e o f certainty. • N ovel m e a n s th a t th e pattern s are n o t previou sly k n o w n to th e u s e r w ith in th e

c o n te x t o f th e sy stem b e in g analyzed. • P oten tially u sefu l m e a n s th a t t h e d isco v e red p attern s sh o u ld lea d to s o m e b e n e fit to

th e u s e r o r task. • U ltim ately u n d ersta n d a b le m e a n s th a t th e p attern sh o u ld m a k e b u sin e ss s e n s e that

lea d s to th e u s e r saying “m m m ! It m a k e s s e n s e ; w h y d id n ’t I th in k o f th at” i f n o t im m ed iately, a t le a s t a fte r s o m e p o st p ro cessin g .

D ata m in in g is n o t a n e w d iscip lin e, b u t ra th er a n e w d efin ition fo r th e u se o f m any d iscip lin es. D ata m in in g is tightly p o sitio n e d a t th e in te rse ctio n o f m an y d iscip lin es, inclu d ing statistics, artificial in te llig e n ce , m a ch in e learn in g , m a n a g e m e n t s c ie n c e , in for­ m ation system s, a n d d a ta b a se s ( s e e Figure 5-1). U sing a d v a n ce s in all o f th e s e d iscip lin es, data m in in g strives t o m a k e p ro g re ss in extractin g u sefu l in form ation an d k n o w le d g e from larg e d atab ases. It is a n e m erg in g field that has attracted m u ch atten tio n in a very s h o rt tim e.

T h e fo llo w in g a re th e m a jo r ch aracteristics an d o b je c tiv e s o f data m ining:

• D ata are o fte n b u ried d e e p w ith in v e ry large d atab ases, w h ic h so m e tim e s co n ta in d ata fro m sev eral y ears. In m an y c a s e s , th e d ata are c le a n s e d a n d c o n s o lid a te d into a d ata w a re h o u s e . D ata m ay b e p re se n te d in a variety o f form ats (s e e T e c h n o lo g y Insigh ts 5 .1 fo r a b r ie f ta x o n o m y o f d ata).

Pattern Recognition

Machine Learning

Mathematical Modeling

Management Science and Information System s

FIG U RE 5.1 Data M ining as a Biend o f Multiple Disciplines.

2 2 4 Part III • Predictive Analytics

• T h e d ata m in in g e n v iro n m en t is usu ally a client/server arch ite ctu re o r a W e b -b a s e d in form ation system s arch itectu re.

• S o p h istica te d n e w to o ls , in clu d in g a d v a n ce d v isu alization to o ls, h e lp to rem ov e th e in form ation o r e b u rie d in c o rp o ra te file s o r arch ival p u b lic reco rd s. Finding it in v olv es m assagin g an d syn ch ro n izin g th e data to g e t th e right results. Cutting- e d g e data m in ers are a lso e x p lo rin g th e u se fu ln e ss o f so ft d ata (i.e ., unstructured te x t s to red in s u ch p la c e s as Lotus N otes d atab ases, te x t file s o n th e In tern et, or en te rp rise-w id e intranets).

• T h e m in e r is o fte n a n e n d u ser, e m p o w e r e d b y d ata drills an d o th e r p o w e r q uery to o ls to ask ad h o c q u e stio n s a n d o b ta in a n sw e rs q u ick ly , w ith little o r n o p rogram ­ m in g skill.

• Striking it rich o ften in v olv es fin d in g an u n e x p e c te d result an d re q u ires e n d u se rs to think cre ativ e ly th ro u g h o u t th e p ro c e s s , in clu d in g th e in terp retation o f th e findings.

• D ata m in in g to o ls are read ily c o m b in e d w ith sp re a d sh ee ts a n d o th e r softw are d ev e lo p m e n t to o ls. T h u s, th e m in ed data c a n b e a n aly zed a n d d e p lo y e d quickly an d easily.

• B e c a u s e o f th e large am o u n ts o f d ata a n d m assive s e a rc h effo rts, it is so m etim es n e c e s s a ry to u s e p arallel p ro c e s s in g fo r d ata m ining.

A co m p a n y th at e ffe ctiv e ly lev era g e s data m in in g to o ls an d te c h n o lo g ie s c a n acq u ire a n d m ain tain a strateg ic co m p etitiv e ad vantage. D a ta m in in g o ffers org an izatio n s a n indis­ p e n s a b le d e c is io n -e n h a n cin g e n v iro n m en t to e x p lo it n e w o p p o rtu n ities b y transform ing data into a strategic w e a p o n . S e e N em ati a n d B a r k o ( 2 0 0 1 ) fo r a m o re d etailed d iscu ssion o n th e strategic: b e n e fits o f d ata m ining.

TECHNOLOGY IN SIGHTS 5 .1 A Sim ple Taxonom y o f Data

D a t a refers to a collection o f facts usually obtained as the result o f experiences, observations, or experiments. Data may consist of numbers, letters, words, images, voice recordings, and so on as measurements o f a set o f variables. Data are often viewed as the lowest level o f abstraction from which information and then knowledge is derived.

At the highest level o f abstraction, one can classify data as structured and unstmctured (o r semistructured). Unstructured/semistructured data is composed o f any combination o f tex­ tual. imagery, voice, and Web content. Unstructured/semistructured data will b e covered in more detailed in the text mining and Web mining chapters (see Chapters 7 and 8). Structured data is what data mining algorithms use, and can b e classified as categorical or numeric. The categorical data can b e subdivided into nominal or ordinal data, whereas numeric data can be subdivided into interval or ratio. Figure 5.2 shows a simple taxonomy o f data.

• C ateg o rical d ata represent the labels o f multiple classes used to divide a variable into specific groups. Examples o f categorical variables include race, sex, age group, and educational level. Although the latter two variables may also be considered in a numeri­ cal manner by using exact values for age and highest grade completed, it is often more informative to categorize such variables into a relatively small number o f ordered classes. The categorical data may also b e called discrete data, implying that it represents a finite number o f values with no continuum betw een them. Even if the values used for the categorical (or discrete) variables are numeric, these numbers are nothing more than sym­ bols and do not imply the possibility o f calculating fractional values.

• N om inal d ata contain measurements o f simple codes assigned to objects as labels, which are not measurements. For example, the variable m a r it a l sta tu s can be generally cat­ egorized as (1 ) single, (2) married, and (3) divorced. Nominal data can b e represented with binomial values having two possible values (e.g., yes/no, true/false, good/bad), or multinomial values having three or more possible values (e.g., brown/green/blue, white/ black/Latino/Asian, single/married/divorced).

Chapter 5 • D ata Mining 2 2 5

FIGURE 5.2 A Simple Taxonomy of Data in Data Mining.

• Ordinal data contain codes assigned to objects or events as labels that also represent the rank order among them. For example, the variable credit score can be generally categorized as (1) low, (2) medium, or (3 ) high. Similar ordered relationships can b e seen in variables such as age group (i.e., child, young, middle-aged, elderly) and educational level (i.e., high school, college, graduate school). Some data mining algorithms, such as ordin al multiple logistic regression, take into account this additional rank-order informa­ tion to build a better classification model.

• Numeric data represent the numeric values o f specific variables. Examples o f numerically valued variables include age, number o f children, total household income (in U.S. dollars), travel distance (in miles), and temperature (in Fahrenheit degrees). Numeric values rep­ resenting a variable can b e integer (taking only whole numbers) or real (taking also the fractional number). The numeric data may also be called continuous data, implying that the variable contains continuous measures on a specific scale that allows insertion o f interim values. Unlike a discrete variable, which represents finite, countable data, a continuous variable represents scalable measurements, and it is possible for the data to contain an infinite number o f fractional values.

• Interval data are variables that can b e measured on interval scales. A common example o f interval scale measurement is temperature on the Celsius scale. In this particular scale, the unit o f measurement is 1/100 o f the difference betw een the melting temperature and the boiling temperature o f water in atmospheric pressure; that is, there is not an absolute zero value.

• Ratio data include measurement variables commonly found in the physical sciences and engineering. Mass, length, time, plane angle, energy, and electric charge are examples o f physical measures that are ratio scales. The scale type takes its name from the fact that measurement is the estimation o f the ratio between a magnitude o f a continuous quantity and a unit magniaide o f the same kind. Informally, the distinguishing feature o f a ratio scale is the possession o f a nonarbitrary zero value. For example, the Kelvin temperature scale has a nonarbitrary zero point o f absolute zero, which is equal to -2 7 3 .1 5 degrees Celsius. This zero point is nonarbitrary, because the particles that comprise matter at this temperature have zero kinetic energy.

Other data types, including textual, spatial, imageiy, and voice, need to b e converted into some form o f categorical or numeric representation before they can be processed by data mining algorithms. Data can also b e classified as static or dynamic (i.e., temporal or time-series).

Some data mining methods and algorithms are very selective about the type o f data that they can handle. Providing them with incompatible data types may lead to incorrect models or (more often) halt the model development process. For example, some data mining methods

2 2 6 Part III • Predictive Analytics

need all o f the variables (both input as well as output) represented as numerically valued variables (e.g., neural networks, support vector machines, logistic regression). The nominal or ordinal variables are converted into numeric representations using some type o f 1-oJ-N pseudo variables (e.g., a categorical variable with three unique values can be transformed into three pseudo variables with binary values— 1 or 0). Because this process may increase the number o f variables, one should b e cautious about the effect o f such representations, especially for the categorical variables that have large numbers o f unique values.

Similarly, some data mining methods, such as ID3 (a classic decision tree algorithm) and rough sets (a relatively new rule induction algorithm), need all o f the variables represented as categorically valued variables. Early versions o f these methods required the user to discretize numeric variables into categorical representations before they could be processed by the algo­ rithm. The good news is that most implementations o f these algorithms in widely available software tools accept a mix o f numeric and nominal variables and internally make the necessary conversions before processing the data.

A p p lication C ase 5 .2 illustrates a n in terestin g a p p lica tio n o f data m in in g w h ere p red ictiv e m o d e ls are u se d b y a p o lic e d ep artm e n t to identify crim e h o tsp o ts an d b etter u tilize lim ited crim e-figh tin g re so u rces.

Application Case 5.2 Harnessing Analytics to Combat Crime: Predictive Analytics Helps Memphis Police Department Pinpoint Crime and Focus Police Resources W h e n Larry G o d w in to o k o v e r as d irecto r o f th e M em phis P o lic e D ep artm en t (M P D ) in 2 0 0 4 , crim e a cro ss th e m e tro area w as surging, an d city lead ers w e re g ro w in g im patient. “T h e m ay o r to ld m e I w an t this crim e p ro b le m fix e d ,” re ca lls G od w in , a 3 8 -y e a r v e te ra n o f th e MPD. B u t th e n e w d ire cto r u n d er­ s to o d th at a b u sin ess-as-u su al a p p ro a ch to crim e fighting w o u ld n o lo n g e r b e g o o d e n o u g h . Early o n in h is te n u re , G o d w in co n v e n e d a m e e tin g o f to p la w e n fo r c e m e n t e x p e rts to fo rm u late a fresh strategy to tu rn th e tid e in th e city’s crim e war. A m on g th e p articip ants in this m ini-su m m it w as Dr. R ich ard Ja n ik o w s k i, a p ro fe ss o r o f crim in o lo g y at the U n iv ersity o f M em ph is, w h o sp e cia liz e d in using p red ictiv e an alytics to b e tte r u n d erstan d pattern s.

F i g h t i n g C rim e w i t h A n a ly tic s

Ja n ik o w s k i p r o p o s e d th e id e a o f m in in g M P D ’s crim e d ata b a n k s to h e lp z e ro in o n w h e r e a n d w h e n crim in a ls w e re h ittin g h a rd e s t an d th e n “fo c u s p o lic e r e s o u r c e s in te llig e n tly b y pu tting th e m in th e righ t p la c e , o n th e right d ay, a t th e right tim e .” B y d o in g s o , h e said , “y o u ’ll e ith e r d e te r c rim in a l activity o r y o u ’re g o in g to c a tc h p e o ­ p le ." T h e id e a m a d e s e n s e t o G o d w in a n d in sh ort

o rd e r th e M PD a n d th e U n iv ersity o f M e m p h is— a lo n g w ith P r o je c t S a fe N e ig h b o rh o o d s — te a m e d up in a p ilo t p ro g ra m th a t la ter b e c a m e k n o w n a s O p e ra tio n B lu e CRUSH, o r C rim e R e d u ctio n U tilizin g S tatistical H istory.

T h e d ata-d riven p ilo t wras w ildly su ccessfu l. D uring o n e 2 -h o u r o p era tio n , o ffice rs a rrested m o re crim inals th a n th e y norm ally a p p re h e n d o v e r an e n tire w e e k e n d . B u t fo r B lu e CRUSH to b e s u c c e s s ­ ful o n a city w id e s ca le , th e M PD w o u ld n e e d to align its re so u rce s an d o p era tio n s to ta k e full ad van tage o f th e p o w e r o f p red ictiv e analytics. I f d o n e right, a city -w id e ro llo u t o f B lu e CRUSH h ad th e p o te n ­ tial to sav e m o n e y th ro u g h e fficie n t d ep lo y m en ts— a b ig plus in a city fa cin g s erio u s b u d g et p ressu res— e v e n a s th e in te llig e n ce -b a s e d a p p ro a ch w o u ld h elp drive d o w n o v erall crim e rates. Shortly after, all p re ­ cin cts e m b r a c e d B lu e CRUSH, an d p red ictiv e analyt­ ics has b e c o m e o n e o f th e m o st p o te n t w e a p o n s in M PD ’s crim e -fig h tin g arsen al. At th e h e a rt o f the sy stem is a v e rsatile statistical analysis to o l— IBM SPSS M o d eler— th at e n a b le s o ffice rs to u n lo c k the in te llig e n ce h id d e n in th e d ep artm en t’s h u g e digi­ tal library o f c rim e re co rd s and p o lic e rep o rts g oin g b a c k n e arly a d e ca d e .

Chapter 5 • Data Mining 2 2 7

S a f e r S tr e e ts

All in d ica tio n s a re th a t B lu e CRUSH an d its in te llig e n c e -d riv e n c rim e fighting te c h n iq u e s are pu tting a s e rio u s d e n t in M em p h is a re a crim e. S in ce th e p ro g ra m w a s la u n c h e d , th e n u m b e r o f Part O n e crim e s— a c a te g o r y o f s e rio u s o ffe n s e s in clu d ­ in g h o m ic id e , ra p e , ag g ra v a ted assau lt, a u to theft, a n d larcen y — h a s p lu m m eted , d ro p p in g 2 7 p e r c e n t fro m 2 0 0 6 to 2 0 1 0 . In te llig e n t p o sitio n in g o f r e s o u r c e s h a s b e e n a m a jo r fa c to r in th e d e c lin e , h e lp in g to d e te r crim in al activity b y h av in g m o re o ffic e rs p a tro llin g th e rig h t a re a at th e right tim e o n th e righ t day.

M ore in te llig e n t d ep lo y m en ts a lso lea d s to faster re a c tio n tim e, sin c e o ffice rs a re lik e ly t o b e b e tte r p o sitio n e d to re sp o n d to a n u n fold in g crim e. In ad dition, M P D ’s o rg an ize d crim e u n its are using d ata fro m th e p red ictiv e an alytics so lu tio n to run sp e cia l d etails th a t lea d to su cce ssfu l m u lti-agen cy drug b u sts a n d o th e r crim inal ro u n d u p s. N ot sur­ prisingly, arrest rates h a v e b e e n stead ily im proving a cro ss th e M em p h is are a, w h ic h h a s a p o p u latio n o f 6 8 0 ,0 0 0 .

T o d ay , th e M PD is co n tin u in g to e x p lo r e n e w w ay s to e x p lo it statistical analysis in its crim e-figh tin g m ission . O f co u rse , p re d ictiv e an alytics a n d data m in in g is ju st o n e p art o f M PD ’s ov erall strategy fo r k e e p in g M em phis re sid e n ts safe. E ffectiv e liaisons w ith co m m u n ity g ro u p s an d b u sin e sse s, stro n g part­ n e rsh ip s w ith re g io n a l a n d fe d era l la w e n fo rc e m e n t a g e n c ie s , and in te llig e n t org an izatio n al a n d o p e ra ­ tion al stru ctures all p la y a part in co n tin u in g M PD’s s u c c e s s story. “At th e e n d o f th e day, e v e ry b o d y w an ts to re d u ce c rim e ,” say s G od w in . “E verybod y w an ts a sa fe co m m u n ity b e c a u s e w ith o u t it, y o u d o n ’t h a v e an y th in g .”

Q u e s t i o n s f o r D i s c u s s i o n

1. H o w did th e M em p h is P o lic e D ep artm e n t u se d ata m in in g to b e tte r co m b a t crime?

2. W h a t w e re th e c h a lle n g e s, th e p ro p o s e d solu tio n , an d th e o b ta in e d results?

Source: IBM Customer Story, “Harnessing Analytics to Combat Crime public.dhe.ibm.com/common/ssi/ecm / en/ im cl454l usen/IM C !454lU SEN .P D F

How Data M ining Works Using e x istin g an d relev an t data, data m in in g b u ild s m o d e ls to identify p attern s am o n g d ie attributes p re s e n te d in th e data set. M odels are th e m ath em atical re p re sen tatio n s

sim ple lin e ar re latio n sh ip s and/or c o m p le x h ig hly n o n lin e a r re latio n sh ip s) th a t identify ih e pattern s a m o n g th e attributes o f th e o b je c ts (e .g ., cu sto m e rs) d escrib e d in th e d ata set. Som e o f th e s e p attern s are e x p la n a to ry (e x p la in in g th e in terrelation sh ip s an d affinities am ong th e attrib u tes), w h e re a s oth ers a re p red ictive (fo re te llin g future v a lu e s o f certain attributes). In g e n era l, d ata m in in g s e e k s to id entify fo u r m a jo r ty p e s o f pattern s:

1 . A s s o c i a t i o n s fin d th e co m m o n ly co -o ccu rrin g gro u p ings o f things, su ch a s b e e r and d iap ers g o in g to g e th e r in m a rk e t-b a sk e t analysis.

2 . P red iction s te ll th e n atu re o f future o c c u rre n c e s o f ce rta in e v e n ts b a s e d o n w h a t has h a p p e n e d in th e past, su ch as p red ictin g th e w in n e r o f the S u p er B o w l o r fo re ca st­ ing th e a b so lu te tem p e ratu re o f a particu lar day.

3 . Clusters id entify natu ral gro u p in g s o f th in gs b a se d o n th e ir k n o w n ch aracteristics, su ch as a ssig n in g cu sto m ers in d ifferen t seg m en ts b a s e d o n th eir d em o g ra p h ics and p a st p u rch a s e b eh av io rs.

4 . S eq u en tial relation sh ip s d isco v e r tim e -o rd e re d e v e n ts, s u ch as p re d ictin g th at an existin g b a n k in g cu sto m e r w h o alread y h a s a ch e c k in g a c c o u n t w ill o p e n a savings a c c o u n t fo llo w e d b y a n in v estm en t a c c o u n t w ith in a year.

T h e s e ty p e s o f p attern s h av e b e e n m an u ally e x tra cte d fro m d ata b y h u m a n s for centuries, b u t th e in cre a sin g v o lu m e o f d ata in m o d e rn tim es h a s c re a te d a n e e d fo r m o re am o m atic a p p ro a ch e s. As d ata sets h av e g ro w n in siz e an d co m p le x ity , d ire ct m anual js ta analysis h a s in cre asin g ly b e e n a u g m en ted w ith ind irect, au tom atic data p ro cessin g

2 2 8 Part III • Predictive Analytics

to o ls th a t u s e s o p h istica ted m e th o d o lo g ie s, m e th o d s, an d algorithm s. T h e m anifestation o f su ch e v o lu tio n o f au tom ated an d sem iau to m ated m e a n s o f p ro cessin g large d ata sets is n o w c o m m o n ly re fe rre d to as d a ta m ining.

G e n e ra lly sp e a k in g , d ata m in in g task s c a n b e classified in to th ree m ain c a te g o ­ ries: p re d ictio n , a sso cia tio n , an d clu sterin g . B a s e d o n th e w a y in w h ich th e pattern s are e x tra cted fro m th e h isto rical d ata, th e learn in g algorithm s o f d ata m in in g m eth o d s can b e classified as e ith er su p erv ised o r u n su p erv ised . W ith su p erv ised learn in g algorithm s, th e training d ata in clu d es b o th th e d escrip tiv e attributes (i.e ., in d e p e n d en t v a ria b le s or d e c is io n v a ria b le s) a s w e ll a s th e class attrib u te (i.e ., o u tp u t v ariab le o r result v ariab le). In con trast, w ith u n su p e rv ised learn in g th e train in g d ata in clu d es o n ly th e d escrip tive attributes. Figu re 5-3 sh o w s a sim p le ta x o n o m y fo r data m in in g task s, alo n g w ith th e learn in g m eth o d s, and p o p u la r algorithm s fo r e a c h o f th e data m in in g tasks.

PREDICTION Prediction is c o m m o n ly re fe rre d to as th e a ct o f telling a b o u t th e future. It differs fro m sim p le g u e ssin g b y tak in g in to a c c o u n t th e e x p e rie n c e s , o p in io n s, and o th er re le v a n t in form ation in co n d u ctin g th e task o f fo retellin g . A term th a t is co m m o n ly a sso ci­ ated w ith p re d ictio n is fo rec a stin g . E ven th o u g h m any b e lie v e that th e s e tw o term s are sy n o n y m o u s, th e re is a s u b tle b u t critical d iffe re n ce b e tw e e n th e tw o. W h e re a s p red iction is largely e x p e rie n c e and o p in io n b a se d , fo re ca s tin g is data a n d m o d e l b a se d . T h a t is, in o rd e r o f in cre a sin g reliability, o n e m ight list th e relev an t term s as gu essin g, p red ictin g , a n d fo reca stin g , resp ectiv ely . In d ata m in in g term in o lo g y , p red ictio n an d fo r e c a s tin g are

Learning Method Popular A lgo rithm s

Supervised □ossification and Regression Trees, ANN, SVM, Genetic Algorithms

Supervised Decision Trees, A N N /M LR SVM, Rough Sets, Genetic Algorithms

Supervised Linear/Nonlinear Regression, Regression Trees, A N N /M LR SVM

Unsupervised Apriori, OneR, ZeroR, Eclat

Unsupervised Expectation Maximization Apriori Algorithm, Graph-Based Matching

Unsupervised Apriori Algorithm, FP-Growth technique

Unsupervised K-means, AN N /SD M

Unsupervised K-means, Expectation Maximization (EM)

FIG U R E 5 .3 A Sim ple Taxonom y fo r Data Mining Tasks.

Chapter 5 * D ata Mining 2 2 9

u sed sy n o n y m o u sly , an d th e term p red ictio n is u s e d as th e co m m o n re p re se n ta tio n o f th e a ct D e p e n d in g o n th e n atu re o f w h a t is b e in g pred icted , p re d ictio n ca n b e n a m e d m o re sp ecifically as cla ssifica tio n (w h e r e th e p red icted thing, s u c h as to m o rro w ’s fo re ca s t, is a class la b e l s u c h as “rain y ” o r “su n n y ”) o r re g re ssio n (w h e r e th e p re d icted thing, su ch as to m o rro w ’s tem p eratu re, is a re al n u m b er, s u ch as “6 5 ° F ”).

C L A S S IF IC A T IO N C l a s s i f i c a t i o n , o r su p erv ised ind u ction , is p e rh a p s th e m o st co m m o n o f all data m in in g task s. T h e o b je c tiv e o f classificatio n is to an aly ze th e h istorical data sto red in a d a ta b a se a n d au tom atically g e n era te a m o d el th at ca n p re d ict future b e h a v io r. T h is in d u ce d m o d e l co n sists o f g e n eralizatio n s o v e r th e re co rd s o f a training data set, w h ich h e lp d istin g u ish p re d efin e d classes. T h e h o p e is th a t th e m o d e l ca n th e n b e used to p re d ict th e cla ss e s o f o th er u n classifie d re co rd s and , m o re im portantly, to accu ra te ly

p red ict actu al fu ture ev en ts. C o m m o n classificatio n to o ls in clu d e n eu ral n etw o rk s a n d d e cisio n tre e s (from

m ach in e lea rn in g ), lo g istic re g re ssio n an d d iscrim in an t analysis (fro m trad itional statistics), and e m e rg in g to o ls s u ch as ro u gh sets, su p p o rt v e c to r m a ch in e s, an d g e n e tic algorithm s. Statistics-based cla ssifica tio n te ch n iq u e s (e .g ., lo g istic re g re ssio n an d d iscrim in an t analy ­ sis) h av e re c e iv e d th e ir sh are o f criticism — that th ey m ak e u n realistic a ssu m p tio n s a b o u t th e data, s u ch as in d e p e n d e n c e a n d norm ality— w h ic h lim it th e ir u s e in classificatio n -ty p e

data m in in g p ro je cts. . N eural n e tw o rk s (s e e C h ap ter 6 fo r a m o re d etailed c o v e ra g e o f this p o p u ­

lar m a ch in e -lea rn in g algorithm ) involve th e d ev e lo p m e n t o f m ath em atical stru ctures < so m ew h at re s e m b lin g th e b io lo g ic a l n e u ral n e tw o rk s in th e h u m an b ra in ) th at h av e th e cap ab ility to learn fro m past e x p e r ie n c e s p re se n te d in th e form o f w ell-stru ctu re d data sets. T h e y te n d to b e m o re e ffe ctiv e w h e n th e n u m b e r o f v ariab les involved is ra th er large and th e re la tio n sh ip s am o n g th e m a re c o m p le x an d im p re cise . N eural n e tw o rk s have d isad vantages a s w e ll as ad vantag es. F o r e x a m p le , it is u su ally v e ry d ifficult t o provide a g o o d ratio n ale fo r th e p re d ictio n s m ad e b y a n eu ral n etw o rk . A lso, n eu ral n etw o rks -end to n e e d co n s id e ra b le training. U n fortunately, th e tim e n e e d e d fo r train in g ten d s to in crease e x p o n e n tia lly as th e v o lu m e o f data in cre a se s, an d , in g e n eral, n eu ral n etw o rk s cannot b e train ed o n very large d atab ases. T h e s e a n d o th er facto rs h av e lim ited th e ap p li­ cability o f n eu ral n e tw o rk s in d ata-rich d om ains.

D e cis io n tre e s classify d ata in to a fin ite n u m b e r o f c la s s e s b a se d o n th e v a lu e s o f th e input v ariab les. D e c is io n tre e s are essen tially a h ierarch y o f if-th e n state m e n ts an d a re thu s significantly faster th a n n eu ral n etw o rk s. T h e y are m o st ap p ro p riate fo r ca te g o rica l and interval data. T h e re fo re , in co rp o ratin g c o n tin u o u s v ariab les into a d e cisio n tre e fram e­ w o rk re q u ires d iscretiz a tio n , th a t is, co n v e rtin g c o n tin u o u s valu ed n u m erical v a ria b le s to ranges an d c a te g o rie s. .

A related c a te g o ry o f classificatio n to o ls is ru le ind u ction . U n lik e w ith a d ecisio n 3 ee w ith rule in d u ctio n th e if-th e n statem en ts are in d u ce d fro m th e training d ata d irectly, a n d 'th e y n e e d n o t b e h iera rch ica l in natu re. O th er, m o re re c e n t te c h n iq u e s s u c h as SVM, rough sets, a n d g e n e tic algorithm s a re grad ually find ing th e ir w ay into th e arsenal o f

classification algorithm s.

C LU STERIN G C l u s t e r i n g p artitions a c o lle c tio n o f th in gs (e .g ., o b je c ts , e v e n ts, etc., - r e s e n te d in a stru ctu red data s e t) in to seg m en ts (o r natu ral gro u p in g s) w h o s e m e m b ers share sim ilar ch aracteristics. U n lik e classificatio n , in clu sterin g th e class la b e ls are u n k n o w n . As th e s e le c te d algorith m g o e s throu g h th e d ata set, id entifying th e co m m o n ­ a ltie s o f th in gs b a s e d o n th e ir ch aracteristics, th e clu sters a re e sta b lish e d . B e c a u s e the ;:u ste rs are d eterm in e d u sin g a h eu ristic-typ e algorith m , a n d b e c a u s e d iffe ren t algorithm s = a y e n d u p w ith d iffe ren t sets o f clu ste rs fo r th e sa m e d ata set, b e fo re th e results of clustering te c h n iq u e s are p u t to actu al u se it m ay b e n e ce s s a ry fo r a n e x p e rt t o interpret,

2 3 0 Part III • Predictive Analytics

and p o ten tially m o d ify, th e su g g e ste d clusters. A fter re a s o n a b le clu sters h av e b e e n id enti­ fied , th e y c a n b e u s e d to classify an d in terp ret n e w d ata.

N ot surprisingly, clu sterin g te c h n iq u e s in clu d e op tim ization . T h e g o al o f clu sterin g is to cre a te g ro u p s s o th at th e m e m b ers w ith in e a c h g ro u p h a v e m axim u m sim ilarity and th e m e m b ers a cro ss g ro u p s h av e m inim um sim ilarity. T h e m o st co m m o n ly u s e d clu ster­ ing te c h n iq u e s in clu d e &-means (fro m statistics) a n d self-o rg an izin g m a p s (fro m m ach in e lea rn in g ), w h ic h is a u n iq u e n eu ral n e tw o rk a rch ite ctu re d e v e lo p e d b y K o h o n e n (1 9 8 2 ).

Firm s o fte n e ffe ctiv e ly u s e th e ir data m ining sy stem s to p erfo rm m ark e t se g m e n ta ­ tio n w ith clu ste r analysis. C luster analysis is a m e a n s o f identifying cla ss e s o f item s s o that item s in a clu ste r h av e m o re in co m m o n w ith e a c h o th e r th a n w ith item s in o th e r clusters. It c a n b e u s e d in s eg m en tin g cu sto m ers an d d irectin g ap p rop riate m ark etin g p ro d u cts to th e seg m en ts at th e right tim e in th e right form at a t th e right p rice . C luster analysis is also u s e d to id entify natu ral gro u p in g s o f e v en ts o r o b je c ts s o that a co m m o n s e t o f c h aracter­ istics o f th e s e g ro u p s c a n b e id en tified to d e s c rib e them .

A S S O C IA T IO N S Associations, o r a s s o c ia tio n r u le le a r n in g i n d a t a m in in g , is a p o p u lar and w e ll-re s e a rch e d te c h n iq u e fo r d isco v e rin g in terestin g relatio n sh ip s am o n g v ariab les in larg e d atab ases. T h a n k s to au tom ated d ata-g ath erin g te ch n o lo g ie s s u ch as b a r c o d e scan n e rs, th e u se o f a s s o cia tio n ru les fo r d isco v e rin g regularities am o n g p ro d u cts in la rg e-sc a le tran sactio n s re co rd e d b y p o in t-o f-sa le sy stem s in su p erm ark ets h a s b e c o m e a co m m o n k n o w le d g e -d isco v e ry task in th e retail industry. In th e c o n te x t o f th e retail industry, a sso cia tio n ru le m ining is o fte n calle d m a rk e t-b a s k e t a n a ly s is .

T w o c o m m o n ly u s e d d erivatives o f a s s o cia tio n rule m in in g are link analysis an d sequence mining. W ith lin k analysis, th e lin k a g e am o n g m an y o b je c ts o f inter­ e st is d isco v e red au tom atically , s u c h as th e lin k b e tw e e n W e b p a g e s an d referential relatio n sh ip s am o n g g ro u p s o f a ca d e m ic p u b lica tio n au thors. W ith s e q u e n c e m ining, relatio n sh ip s are e x am in e d in term s o f th e ir o rd e r o f o c c u rre n c e to identify a sso ciatio n s o v e r tim e. A lgorithm s u se d in a s s o cia tio n ru le m in in g in clu d e th e p o p u la r Apriori (w h ere fre q u e n t item sets are id en tified ) and FP -G row th, O n e R , Z eroR , a n d Eclat.

V IS U A L IZ A T IO N A N D T IM E -S E R IE S F O R E C A S T IN G T w o te ch n iq u e s o fte n a sso cia ted w ith d ata m in in g a re v is u a liz a t io n a n d tim e -s e rie s fo re c a s tin g . V isu alizatio n c a n b e u se d in c o n ­ ju n ctio n w ith o th er d ata m ining te ch n iq u e s to gain a c le a re r u n d erstan d in g o f un derlying re latio n sh ip s. As th e im p o rtan ce to visu alization h a s in cre ase d in re c e n t y e ars, a n e w term , v is u a l a n a ly tic s , h a s em e rg ed . T h e id ea is to co m b in e an alytics a n d v isu alization in a sin gle e n v iro n m en t fo r e a sie r and fa ste r k n o w le d g e cre a tio n . V isual analytics is co v e re d in d etail in C h ap ter 4 . In tim e -serie s fo recastin g , th e data co n sists o f v alu es o f th e sam e v ariab le that is ca p tu re d an d s to red o v e r tim e in re g u lar intervals. T h e s e data are then u s e d to d e v e lo p fo reca stin g m o d els to e x tra p o la te th e future v a lu e s o f th e sam e v ariab le.

Data Mining Versus Statistics D ata m in in g a n d statistics h av e a lo t in co m m o n . T h e y b o th lo o k fo r relatio n sh ip s w ith in data. M ost call statistics th e fo u n d a tio n o f data m ining . T h e m ain d iffe re n ce b e tw e e n th e tw o is th at statistics starts w ith a w e ll-d e fin ed p ro p o sitio n an d h y p o th esis w h ile data m in in g starts w ith a lo o s e ly d efin e d d isco v e ry statem en t. Statistics co lle c ts a sam p le data (i.e ., prim ary d ata) to te st th e h y p o th e sis, w h ile d ata m in in g a n d analytics u s e all o f the e x istin g data (i.e ., o fte n o b serv a tio n al, s e co n d a ry d ata) to d isco v e r n o v e l pattern s and relatio n sh ip s. A n o th er d iffe re n ce co m e s fro m th e s iz e o f d ata that th e y u se. D ata m ining lo o k s fo r d ata sets th at a re as “b ig ” as p o ss ib le w h ile statistics lo o k s fo r right siz e o f data (if th e d ata is larg er th a n w h a t is need ed / req u ired fo r th e statistical analysis, a sam p le o f the data is u se d ). T h e m e a n in g o f “larg e d ata” is ra th er d ifferen t b e tw e e n statistics and

Chapter 5 • Data Mining 2 3 1

data m ining: A lth o u gh a fe w h u n d red to a th o u sa n d data p o in ts are larg e e n o u g h to a statistician, sev eral m illio n to a fe w b illio n d ata p o in ts are co n sid e re d larg e fo r data m in­ ing studies.

SECTION 5 .2 REVIEW QUESTIONS

1 . D e fin e d a ta m in in g. W h y are th e re m a n y d ifferent n a m e s and d efin itio n s for d ata mining?

2 . W h at r e c e n t fa cto rs h av e in cre a se d th e pop u larity o f data mining? 3 . H o w w o u ld data m in in g algorithm s d ea l w ith qualitative data lik e u n stru ctu red texts

fro m interview s? 4 . W h at are s o m e m a jo r d ata m in in g m eth o d s an d algorithm s? 5. W h at are th e k e y d iffe ren ce s b e tw e e n the m a jo r data m in in g m ethods?

5.3 D A T A M IN IN G A P PLIC A T IO N S D ata m in in g has b e c o m e a p o p u lar to o l in ad d ressin g m any c o m p le x b u s in e s s e s p ro b ­ lem s and op p o rtu n ities. It h a s b e e n p ro v en to b e v e ry su cce ssfu l and h elp fu l in m any areas, s o m e o f w h ic h are sh o w n b y th e fo llo w in g re p re sen tativ e e x a m p le s. T h e g o al o f m any o f th e s e b u sin e ss d ata m in in g ap p licatio n s is to s o lv e a p re ssin g p ro b le m o r to e x p lo re a n e m e rg in g b u s in e s s o p p ortu n ity in ord e r to c re a te a s u sta in a b le com p etitiv e advantage.

• C u sto m er r e la tio n s h ip m a n a g e m e n t. C u stom er re latio n sh ip m a n a g e m e n t (C R M ) is th e e x te n s io n o f trad itio n al m ark etin g . T h e g o a l o f CRM is to cre a te o n e -o n -o n e re la tio n sh ip s w ith cu sto m ers b y d ev elo p in g an in tim ate u n d e r­ stan d in g o f th e ir n e e d s and w an ts. As b u s in e s s e s b u ild re la tio n sh ip s w ith th e ir cu sto m e rs o v e r tim e throu g h a variety o f in te ra ctio n s (e .g ., p ro d u ct in q u irie s, s a le s, serv ice re q u e sts, w arranty calls, p ro d u ct rev iew s, s o c ia l m ed ia c o n n e c tio n s ), they accu m u la te tre m e n d o u s am o u n ts o f d ata. W h e n c o m b in e d w ith d e m o g ra p h ic an d s o c io e c o n o m ic attrib u tes, this in fo rm atio n -rich d ata ca n b e u se d to ( 1 ) id entify m o st lik e ly resp on d ers/ bu yers o f n e w p ro d u cts/ services ( i.e ., cu sto m e r p ro filin g ); ( 2 ) u n d e rstan d th e ro o t c a u se s o f cu sto m e r attrition in o rd e r to im p ro v e cu sto m e r re te n tio n ( i .e ., ch u rn an aly sis); ( 3 ) d is co v e r tim e-varian t a s s o c ia tio n s b e tw e e n p ro d u cts a n d s e rv ice s to m a x im iz e s a le s a n d cu sto m e r v alu e ; an d ( 4 ) id entify the m o st p ro fita b le cu sto m e rs an d th e ir p re fe re n tia l n e e d s to stre n g th e n re la tio n sh ip s an d to m a x im iz e sales.

• B a n k i n g . D ata m in in g ca n h e lp b an k s w ith th e fo llow in g : ( 1 ) a u to m atin g th e lo a n a p p lica tio n p ro c e s s b y a ccu rate ly p red ictin g th e m o st p ro b a b le defaulters; ( 2 ) d e te ctin g frau d ulen t cre d it card and o n lin e -b a n k in g tran saction s; ( 3 ) identifying w ay s to m a x im iz e cu sto m e r v a lu e b y sellin g th e m p ro d u cts an d s e rv ice s th at they a re m o st lik e ly to buy; an d (4 ) o p tim izin g th e c a s h return b y a ccu rate ly fo reca stin g th e c a s h flo w o n b a n k in g en tities (e .g ., ATM m a ch in e s, b a n k in g b ra n ch e s).

• R e t a ilin g a n d logistics. In th e retailing industry, d ata m in in g c a n b e u se d to ( 1 ) p re d ict a cc u ra te s a le s v o lu m e s a t s p e c ific retail lo ca tio n s in o rd e r to d eterm in e c o r re c t in v e n to ry lev els; ( 2 ) id en tify sale s re la tio n sh ip s b e tw e e n d iffe re n t p rodu cts (w ith m a rk e t-b a s k e t a n aly sis) to im p ro v e th e sto re lay o u t a n d o p tim iz e sa le s p ro ­ m o tio n s; ( 3 ) fo re c a s t c o n s u m p tio n le v e ls o f d ifferen t p ro d u ct ty p e s (b a s e d on s e a s o n a l a n d e n v iro n m en tal c o n d itio n s ) to o p tim ize lo g istics an d h e n c e m ax im ize sa le s; a n d ( 4 ) d is co v e r in terestin g p attern s in the m o v e m e n t o f p ro d u cts (e s p e c ia lly fo r th e p ro d u cts th a t h a v e a lim ited s h e lf life b e c a u s e th e y a re p ro n e t o e x p iratio n , perish ab ility , a n d co n ta m in a tio n ) in a su p p ly ch ain b y an aly zin g s e n so ry and R FID data.

2 3 2 Part III • Predictive Analytics

• M a n u f a c t u r i n g a n d p r o d u c t io n . M anu factu rers c a n u s e d ata m in in g to ( 1 ) p re d ict m ach in e ry failu res b e fo r e th e y o c c u r th ro u g h th e u s e o f s e n so ry data (e n a b lin g w h a t is ca lle d c o n d itio n -b a sed m ain ten an ce)-, (2 ) id entify an om alies an d co m m o n a litie s in p ro d u ctio n sy stem s to o p tim iz e m anu factu ring cap acity ; and ( 3 ) d is co v e r n o v e l pattern s to identify an d im p ro v e p ro d u ct quality.

• B r o k e r a g e a n d s e c u r it ie s t r a d in g . B ro k e rs an d trad ers u se d ata m in in g to ( 1 ) p re d ict w h e n and h o w m u ch ce rta in b o n d p rices w ill ch a n g e ; ( 2 ) fo re c a s t the ran g e an d d irectio n o f s to c k flu ctu ations; ( 3 ) a s s e s s th e e ffe c t o f p articu lar issu es and e v e n ts o n ov erall m ark e t m o v e m e n ts; an d ( 4 ) id entify an d p re v e n t fraudulent activities in secu ritie s trading.

• I n s u r a n c e . T h e in su ran ce indu stry u s e s d ata m in in g te ch n iq u e s to ( 1 ) fo recast claim am o u n ts fo r p ro p erty a n d m e d ica l c o v e r a g e co s ts fo r b e tte r b u sin e ss p lan ­ n in g ; ( 2 ) d eterm in e op tim al rate p lan s b a s e d o n th e an alysis o f claim s and cu stom er d ata; ( 3 ) p re d ict w h ich cu sto m ers a re m o re lik e ly to b u y n e w p o lic ie s w ith sp e cial featu res; and ( 4 ) identify and p re v e n t in c o rre c t claim p ay m en ts an d frau d ulent activities.

• C o m p u ter h a r d w a r e a n d so ftw a re. D ata m in in g ca n b e u s e d to ( 1 ) p re d ict disk drive failu res w e ll b e fo r e th e y actu ally o ccu r; ( 2 ) id entify an d filter u n w an te d W e b c o n te n t an d e-m ail m e ssa g es; ( 3 ) d e te c t a n d p re v e n t co m p u te r n e tw o rk secu rity b rid g es; and ( 4 ) identify p o ten tially u n se cu re so ftw are produ cts.

• G o v ern m en t a n d d e fe n s e . D ata m in in g a ls o h a s a n u m b e r o f m ilitary ap p lica­ tions. It c a n b e u s e d to (1 ) fo re ca st th e c o s t o f m o v in g m ilitary p e rs o n n e l and e q u ip m en t; ( 2 ) p re d ict a n ad versary’s m o v e s an d h e n c e d ev elo p m o re su cce ssfu l strateg ies fo r m ilitary e n g a g e m e n ts; ( 3 ) p re d ict re s o u rce co n s u m p tio n fo r b etter p lan n in g and b u d g etin g ; an d ( 4 ) id en tify cla ss e s o f u n iq u e e x p e rie n c e s , strategies, an d le s s o n s le a rn e d fro m m ilitary o p era tio n s fo r b e tte r k n o w le d g e sh arin g throu gh­ o u t th e organization.

• T ra v el in d u s t r y (a irlin e s , hotels/resorts, r e n t a l c a r c o m p a n ie s ). D ata m ining has a variety o f u s e s in th e travel industry. It is s u cce ssfu lly u se d to ( 1 ) p re d ict sales o f d ifferen t s e rv ice s (s e a t ty p e s in airp lan es, r o o m typ es in hotels/resorts, c a r types in ren tal ca r c o m p a n ie s ) in o rd e r to o p tim ally p rice s erv ices to m ax im ize rev en u es as a fu n ctio n o f tim e-varying tran saction s (c o m m o n ly refe rre d to as y ie ld m a n a g e­ ment)-, ( 2 ) fo re c a s t d em an d at d ifferen t lo ca tio n s to b e tte r allo ca te lim ited organ i­ zation al re so u rces; (3 ) id entify th e m o st p ro fitab le cu sto m ers a n d pro v id e th em w ith p e rso n a liz e d s erv ices to m ain tain th eir re p e a t b u sin ess; and ( 4 ) retain valu ab le e m p lo y e e s b y identifying an d actin g o n th e ro o t ca u se s fo r attrition.

• H e a lth c a re . D ata m in in g h a s a n u m b e r o f h e a lth c a re a p p licatio n s. It c a n b e u se d to ( 1 ) id entify p e o p le w ith o u t h e a lth in s u ra n c e an d th e fa cto rs u n d erlying this u n d e sired p h e n o m e n o n ; (2 ) id entify n o v e l c o s t- b e n e f it re latio n sh ip s b e tw e e n d ifferen t treatm en ts to d e v e lo p m o re e ffe c tiv e strateg ies; (3 ) fo re c a s t th e lev el an d th e tim e o f d em an d at d iffe ren t s e rv ic e lo ca tio n s to op tim ally a llo c a te o rg a ­ n iz a tio n a l re s o u r c e s ; a n d ( 4 ) u n d erstan d th e u n d erly in g re a s o n s fo r cu sto m e r and e m p lo y e e attrition.

• M e d ic in e . U se o f data m in in g in m e d icin e sh o u ld b e v ie w e d as a n in valu ab le c o m p le m e n t to trad itional m ed ical re sea rch , w h ic h is m ain ly clin ical and b io lo g ica l in natu re. D ata m in in g an a ly se s c a n (1 ) id entify n o v e l pattern s to im p ro ve surviv­ ability o f p atien ts w ith ca n ce r; ( 2 ) p re d ict s u c c e s s rates o f o rg a n tran sp lantation p atien ts to d ev elo p b e tte r d o n o r-o rg a n m a tch in g p o licie s; ( 3 ) identify th e fu n ctio n s o f d ifferen t g e n e s in th e h u m an ch ro m o s o m e (k n o w n as g e n o m ic s ); and ( 4 ) d is­ c o v e r th e re latio n sh ip s b e tw e e n sym p to m s an d illn esse s (a s w ell as illn esses and s u cce s s fu l treatm en ts) to h e lp m e d ica l p ro fe ssio n a ls m a k e in fo rm ed an d co rre ct d e cis io n s in a tim ely m ann er.

C hapter 5 * Data M ining 2 3 3

• E n t e r t a in m e n t in d u s tr y . D ata m in in g is s u cce ssfu lly u s e d b y th e e n te rta in m e n t industry to ( 1 ) a n a ly z e v ie w e r d ata to d e c id e w h at p ro gram s to s h o w d uring p rim e tim e an d h o w to m ax im ize returns b y k n o w in g w h e re to in sert ad vertisem en ts; (2 ) p red ict th e fin an cial s u c c e s s o f m o v ies b e fo re th e y are p ro d u ce d to m a k e in v estm en t d e c is io n s an d to o p tim ize th e retu rns; ( 3 ) fo re ca st th e d em an d at d ifferen t lo c a tio n s a n d d ifferen t tim e s t o b e tte r s ch e d u le en te rtain m e n t e v e n ts an d to op tim ally a llo ca te re so u rces; an d (4 ) d ev elo p op tim al p ricin g p o lic ie s to m ax im ize

re v en u es. , f • H o m e la n d s e c u r it y a n d law e n fo r c e m e n t . D ata m in in g h a s a n u m b e r o l

h o m e la n d s e c u rity a n d la w e n fo r c e m e n t a p p lic a tio n s . D a ta m in in g is o fte n u s e d to ( 1 ) id e n tify p a tte rn s o f te rro ris t b e h a v io rs ( s e e A p p lic a tio n C a se 5 .3 fo r a n e x a m p le o f th e u s e o f d ata m in in g to tr a c k fu n d in g o f te rro ris ts ’ a c tiv itie s ); ( 2 ) d is c o v e r c rim e p a tte rn s ( e .g ., lo c a tio n s , tim in g s, c rim in a l b e h a v io r s , a n d o th e r re la te d a ttr ib u te s ) to h e lp s o lv e crim in a l c a s e s in a tim e ly m a n n e r; ( 3 ) p re d ic t a n d e lim in a te p o te n tia l b io lo g ic a l a n d c h e m ic a l a tta c k s to th e n a tio n s critic a in fra stru ctu re b y an a ly z in g s p e c ia l-p u r p o s e s e n s o r y d a ta ; a n d (4 ) id e n tify an d s to p m a lic io u s a tta c k s o n critica l in fo r m a tio n in fra s tru ctu re s (o f t e n c a lle d in fo r ­ m a tio n w a rfa re).

. Sports. D a ta m in in g w a s u s e d to im p ro v e th e p e rfo rm a n c e o f N ational B a s k e tb a ll A s s o cia tio n (N B A ) te am s in th e U n ite d States. M ajor L eag u e B a s e b a ll te a m s are in to p re d ictiv e a n a ly tics an d d ata m in in g to op tim ally u tilize th e ir lim ited r e s o u r c e s fo r a w in n in g s e a s o n (s e e M o n ey b a ll article in C h ap te r 1). In fa ct, m o st, i f n o t all, o f th e p ro fe ss io n a l sp orts e m p lo y d ata c ru n ch e rs and u s e d ata m in in g to in c re a s e th e ir c h a n c e s o f w in n in g . D ata m in in g a p p lica tio n s are n o t lim ited to p ro fe ss io n a l sp o rts. I n re c e n tly p u b lis h e d a rticle, D e le n e t al. (2 0 1 2 ) d e v e lo p e d m o d e ls t o p re ­ d ict NCAA B o w l G a m e o u tc o m e s u sin g a w id e ran g e o f v a ria b le s a b o u t th e tw o o p p o s in g te a m s ’ p re v io u s g a m e statistics. W rig h t ( 2 0 1 2 ) u s e d a variety' o f p re d ic ­ to rs fo r e x a m in a tio n o f th e NCAA m e n ’s b a sk e tb a ll c h a m p io n sh ip b ra c k e t (a .k .a .

M arch M ad n e ss).

Application Case 5.3 A Mine on Terrorist Funding T h e terrorist a tta ck o n th e W o rld Ir a d e C e n te r o n S e p te m b e r 11, 2 0 0 1 , u n d e rlin e d th e im p o rtan ce o f o p e n so u rce in te llig e n ce . T h e USA PA TR IO T A ct and th e cre a tio n o f th e U .S. D ep artm en t o f H om elan d Secu rity (D H S) h e ra ld e d the p o ten tial a p p lica tio n o f inform ation te c h n o lo g y an d d ata m ining te ch n iq u e s to d e te c t m o n ey lau n d erin g an d o th e r fo rm s o f ter­ rorist fin an cin g . L aw e n fo rc e m e n t a g e n c ie s hav e b e e n fo cu sin g o n m o n e y lau n d erin g activities via norm al tran sactio n s th ro u g h b a n k s and o th e r fin an ­ cial serv ice o rgan ization s.

Law e n fo r c e m e n t a g e n c ie s a re n o w fo cu sin g o n in tern ation al trad e p ricin g a s a terro rism fu nd ing t o o l In tern atio n al trad e h a s b e e n u se d b y m o n ey lau n d erers to m o v e m o n ey silen tly o u t o f a cou ntry

w ith o u t attracting g o v e rn m e n t atten tion . T h is tran s­ fe r is a ch ie v e d b y o v erv alu in g im p orts an d u n der­ v alu in g e xp o rts. F o r e x a m p le , a d o m e stic im p orter a n d fo re ig n e x p o rte r c o u ld fo rm a p artn ersh ip and o v erv alu e im ports, th e re b y tran sferring m o n e y from th e h o m e cou ntry, resu ltin g in crim e s re la ted to cu s­ to m s fraud, in c o m e ta x e v asio n , an d m o n ey laun­ dering. T h e fo reig n e x p o rte r co u ld b e a m e m b e r o f a terro rist org anization.

D ata m in in g te c h n iq u e s fo c u s o n an alysis o f data o n im p ort a n d e x p o r t tran sactio n s fro m th e U.S. D ep a rtm e n t o f C o m m e rce an d co m m e rce -re la te d en tities. Im p o rt p rice s th a t e x c e e d th e u p p e r q u ar- tile im p ort p rice s an d e x p o rt p rice s that are lo w e r th an th e lo w e r q u artile e x p o rt p rice s are tracked .

( C on tin u ed )

2 3 4 Part III • P redictive Analytics

Application Case 5.3 (Continued) T h e fo cu s is o n a b n o rm al tran sfer p rice s b e tw e e n c o rp o ra tio n s th a t m ay result in sh iftin g ta x a b le in c o m e a n d ta x e s ou t o f th e U n ited States. An o b serv e d p ric e d ev iatio n m ay b e related to in co m e ta x av o id an ce/ ev asio n , m o n e y lau n d erin g, o r terror­ ist fin an cin g. T h e o b s e rv e d p rice d ev iatio n m ay also b e d u e to an e rro r in th e U .S. trad e d atabase.

D a ta m in in g w ill resu lt in e ffic ie n t e v a lu a ­ tio n o f data, w h ic h , in turn, w ill aid in th e fight a g ain st terro rism . T h e a p p lic a tio n o f in form ation te c h n o lo g y a n d d ata m in in g te c h n iq u e s to fin an cial tran sactio n s c a n c o n trib u te to b e tte r in te llig e n ce in form ation .

1. H o w ca n data m in in g b e u s e d to fight terrorism ? C o m m e n t o n w h a t e ls e c a n b e d o n e b ey o n d w h at is c o v e r e d in this sh o rt a p p lica tio n case.

2. D o y o u th in k th at, alth ou gh data m in in g is e ss e n ­ tial fo r fighting terrorist ce lls, it a lso jeo p ard izes individuals’ rights to privacy?

Sources: J. S. Zdanowic, “Detecting Money Laundering and Terrorist Financing via Data Mining," C om m un ication s o f the ACM, Vol. 47, No. 5, May 2004, p. 53; and R. J. Bolton, “Statistical Fraud Detection: A Review,” S tatistical S cience, Vol. 17, No. 5, January 2002, p. 235.

Q u e s t i o n s f o r D i s c u s s i o n

SECTION 5 .3 REVIEW QUESTIONS

1 . W h at are th e m a jo r ap p lica tio n a re a s fo r d ata m ining?

2 . Id en tify a t le a st fiv e s p e cific ap p lica tio n s o f data m in in g an d list fiv e co m m o n ch a ra c­ teristics o f th e s e ap p licatio n s.

3 . W h at d o y o u th in k is th e m o st p ro m in en t a p p lica tio n a re a fo r d ata m ining? Why?

4 . Can y o u th in k o f o th e r a p p licatio n are a s fo r data m in in g n o t d iscu sse d in this section? E xplain.

5.4 D A T A M IN IN G PR O C ESS In o rd e r to sy stem atically carry o u t data m in in g p ro je c ts , a g en eral p ro c e s s is usually fo llo w ed . B a s e d o n b e s t p ractice s, data m ining re s e a rch e rs an d p ractition ers h a v e p ro­ p o s e d sev eral p ro c e s s e s (w o rk flo w s o r sim p le ste p -b y -ste p a p p r o a c h e s) to m a x im iz e the c h a n c e s o f s u c c e s s in co n d u ctin g data m in in g p ro je cts. T h e s e e ffo rts h av e led to sev eral stand ard ized p ro ce s s e s, s o m e o f w h ic h (a fe w o f th e m o st p o p u la r o n e s ) a re d escrib e d in this sectio n .

O n e s u ch stand ard ized p ro c e s s , argu ably th e m o st p o p u la r o n e , C ross-Industry Stand ard P ro ce s s fo r D ata M ining— CRISP-DM— w a s p ro p o s e d in th e m id -1 9 9 0 s b y a E u ro p e a n co n so rtiu m o f c o m p a n ie s to serv e as a n o n p ro p rie ta ry stand ard m e th o d o lo g y fo r d ata m in in g (CRISP-D M , 2 0 1 3 ). Figure 5-4 illustrates this p ro p o s e d p ro c e s s , w h ich is a s e q u e n c e o f six ste p s that starts w ith a g o o d u n d erstan d in g o f th e b u sin e ss an d the n e e d fo r th e data m in in g p ro je c t (i.e ., th e ap p lica tio n d o m ain ) a n d e n d s w ith th e d ep lo y ­ m e n t o f th e so lu tio n th at satisfied th e s p e cific b u s in e s s n e ed . E ven th o u g h th e s e steps a re se q u e n tia l in n atu re, th e re is u su ally a great d e a l o f b a ck tra ck in g . B e c a u s e th e data m ining is d riven b y e x p e rie n c e an d e x p e rim e n ta tio n , d ep en d in g o n th e p ro b le m situation an d th e k n o w le d g e/ e x p erie n ce o f th e analyst, th e w h o le p ro c e s s c a n b e very iterative (i.e ., o n e sh o u ld e x p e c t to g o b a c k an d forth th ro u g h th e s te p s q u ite a fe w tim e s) and tim e-co n su m in g . B e c a u s e later s te p s are built o n th e o u tco m e o f the fo rm e r o n e s , o n e sh o u ld p a y e xtra a tte n tio n to th e e a rlie r s te p s in o rd e r n o t to p u t th e w h o le stud y o n an in c o rre c t p ath fro m th e on set.

Chapter 5 • D ata Mining 2 3 5

FIGURE 5.4 The Six-Step CRISP-DM Data Mining Process.

Step 1: Business Understanding T h e k e y e le m e n t o f a n y data m in in g stud y is to k n o w w h a t th e study is fo r. A nsw ering su ch a q u e s tio n b e g in s w ith a th o ro u g h u n d erstan d in g o f th e m an ag erial n e e d fo r n e w k n o w le d g e an d a n e x p licit sp e c ific a tio n o f th e b u sin ess o b je c tiv e reg ard in g th e study to b e co n d u cte d . S p e cific g o als s u c h as “W h a t are th e co m m o n ch a ra cteristics o f th e cu sto m ers w e h a v e lo st to o u r co m p etito rs recently?” o r “W h at are ty p ical p ro file s o f ou r cu stom ers, a n d h o w m u ch v alu e d o e s e a c h o f th e m pro v id e to us?” a re n e e d e d . T h e n a p ro je ct p lan fo r find ing s u ch k n o w le d g e is d e v e lo p e d that s p e cifie s th e p e o p le re s p o n ­ sib le fo r c o lle c tin g th e d ata, analy zing th e data, an d rep ortin g th e findings. At this early stage, a b u d g et to su p p o rt th e study sh ou ld a lso b e e sta b lish e d , at least a t a h ig h lev el w ith ro u g h nu m b ers.

Step 2: Data Understanding A data m in in g stud y is s p e c ific to ad d ressin g a w e ll-d e fin e d b u sin ess ta sk , a n d differ­ en t b u sin ess ta s k s re q u ire d ifferen t sets o f data. F o llo w in g th e b u sin e ss u n d erstand ing, th e m ain activity o f th e data m ining p ro c e s s is to identify th e relev an t d ata fro m m any available d a ta b a se s. S o m e k e y p o in ts m u st b e co n sid e re d in th e d ata id e n tifica tio n and sele ctio n p h a s e . First an d fo rem o st, th e an aly st sh o u ld b e c le a r a n d c o n c is e a b o u t th e d escrip tio n o f th e data m in in g ta sk s o that th e m o st relev an t d ata c a n b e identified. F o r e x a m p le , a retail d ata m ining p ro je c t m ay s e e k to identify sp e n d in g b eh a v io rs o f fem ale s h o p p e rs w h o p u rch ase s e a s o n a l c lo th e s b a se d o n th e ir d em o g ra p h ics, cred it card tran sactio n s, a n d s o c io e c o n o m ic attributes. F u ith e rm o re , th e an aly st sh o u ld build

2 3 6 Part III • Predictive Analytics

a n intim ate u n d erstan d in g o f th e data s o u rc e s (e .g ., w h e re th e relev an t data are stored an d in w h at fo rm ; w h at th e p ro c e s s o f c o lle c tin g th e data is— au to m ate d versus m anual; w h o th e c o lle c to rs o f th e data a re an d h o w o fte n th e d ata a re u p d ated ) a n d th e v ariab les (e .g ., W h at are th e m o st re le v a n t variables? Are th e re a n y sy n o n y m o u s and/or h o m ­ o n y m o u s variables? A re the v a ria b le s in d e p e n d e n t o f e a c h o th e r— d o th ey stand as a co m p le te in form ation s o u rc e w ith o u t o v erlap p in g o r co n flictin g inform ation?).

In o rd e r to b e tte r u n d erstan d th e data, th e a n aly st o fte n u s e s a v ariety o f statistical an d g rap hical te ch n iq u e s , su ch a s sim p le statistical sum m aries o f e a c h v ariab le (e .g ., fo r n u m e ric v ariab les th e a v erag e, m inim um /m axim um , m ed ian , a n d standard d ev iatio n are a m o n g th e ca lcu la ted m easu res, w h e re a s fo r c a te g o rica l v a ria b le s th e m o d e an d fre q u e n cy ta b le s are ca lcu lated ), co rrela tio n analysis, s ca tte r p lo ts, histogram s, and b o x plots. A care ­ ful id en tificatio n a n d s e le c tio n o f d ata so u rce s an d th e m o st relev an t v ariab les c a n m ak e it e a sie r fo r d ata m ining algorithm s to q u ick ly d is co v e r u sefu l k n o w le d g e patterns.

D ata so u rce s fo r data s e le c tio n c a n vary. N orm ally, d ata so u rce s fo r b u sin ess ap p licatio n s in clu d e d em o g ra p h ic d ata (s u c h as in c o m e , e d u ca tio n , n u m b e r o f h o u s e ­ h o ld s, and a g e ), s o cio g ra p h ic d ata (s u c h as h o b b y , clu b m em b ersh ip , an d e n tertain m en t), tran saction al d ata (s a le s reco rd , cre d it card sp e n d in g , issu ed c h e c k s ), an d s o on .

D ata c a n b e ca te g o riz e d a s quantitative a n d qualitative. Q uantitative d ata is m easu red u sin g n u m e ric valu es. It c a n b e d iscre te (s u c h as in teg ers) o r c o n tin u o u s (su ch as re al n u m b e rs). Q ualitative d ata, a lso k n o w n a s ca te g o rica l data, co n ta in s b o th n om inal an d ordinal d ata. N om inal d ata h a s fin ite n o n o rd e re d v a lu e s (e .g ., g e n d e r data, w h ich h a s tw o valu es: m ale an d fe m a le ). O rd inal d ata h a s fin ite o rd e re d v alu e s. F o r e x a m p le , cu sto m e r cred it ratings a re c o n s id e re d ord in al d ata b e c a u s e th e ratings c a n b e e x ce lle n t,

fair, an d b ad . Q uantitative d ata ca n b e read ily re p re s e n te d b y s o m e s o n o f p ro b ab ility distri­

b u tion . A p ro b ab ility distribution d e s crib e s h o w th e d ata is d isp e rse d a n d sh ap e d . F or in sta n ce, norm ally distributed d ata is sy m m etric a n d is co m m o n ly refe rre d to as b e in g a b e ll-sh a p e d cu rve. Q ualitativ e data m ay b e c o d e d to n u m b ers an d th e n d e s crib e d b y fre q u e n cy d istributions. O n c e th e relev an t data are s e le c te d a cco rd in g to th e data m ining b u sin ess o b je c tiv e , data p re p ro ce s s in g sh o u ld b e pu rsued .

Step 3: Data Preparation T h e p u rp o s e o f data p re p aratio n (o r m o re c o m m o n ly ca lle d d a ta p rep rocessin g ) is to ta k e th e d ata id en tified in th e p rev iou s ste p a n d p re p a re it fo r an aly sis b y d ata m ining m eth o d s. C o m p ared to th e o th e r step s in CRISP-D M , d ata p re p ro ce s s in g c o n s u m e s th e m o st tim e a n d effo rt; m o st b e lie v e th at this step a c c o u n ts fo r ro u gh ly 8 0 p e rc e n t o f th e total tim e s p e n t o n a d ata m in in g p ro je ct. T h e re a s o n fo r s u ch a n e n o rm o u s e ffo rt sp e n t o n this step is th e fa ct th at re al-w o rld data is g e n era lly in co m p le te (la c k in g attribute v alu e s, la ck in g ce rta in attributes o f in terest, o r co n ta in in g o n ly ag g re g ate d ata), n o isy (co n ta in in g errors o r ou tliers), and in co n sisten t (c o n ta in in g d is cre p a n cie s in c o d e s or n a m e s). Figu re 5.5 s h o w s th e fo u r m ain s te p s n e e d e d to co n v e rt th e raw real-w orld data in to m in a b le data sets.

In th e first p h ase o f data p rep ro cessin g , th e relev an t d ata is c o lle cte d fro m th e identified so u rce s (a cco m p lish e d in th e previous step — D ata Understanding— o f th e CRISP-DM p ro ­ ce ss ), the n e cessary record s an d variables are s e le c te d (b a s e d o n an intim ate understanding o f th e data, th e u n n ece ssary sectio n s are filtered ou t), an d th e record s co m in g from mul­ tiple data so u rces are integrated (ag ain, u sin g th e intim ate u nderstand ing o f th e data, the synonym s a n d hom on ym s are to b e hand led properly).

In th e s e c o n d p h a s e o f d ata p re p ro ce ssin g , th e data is c le a n e d (th is s te p is also k n o w n as d ata scru b b in g ). In this step , th e v a lu e s in th e data s e t a re id e n tified a n d d ealt w ith. In s o m e c a s e s , m issin g valu es a re a n an o m a ly in th e d ata set, in w h ic h c a s e th ey

Chapter 5 • D ata Mining 2 3 7

Data Consolidation

Data Cleaning -{ '

Data Transformation - L '

Data Reduction ■L Well-Formed

Data

• Collect data • Select data • Integrate data

• Impute missing values • Reduce noise in data • Eliminate inconsistencies

• Normalize data • Discretize/aggregate data • Construct new attributes

• Reduce number of variables • Reduce number of cases • Balance skewed data

FIGURE 5.5 Data Preprocessing Steps.

n e e d to b e im p u ted (fille d w ith a m o st p ro b a b le v a lu e ) o r ig n ored ; in o th e r c a s e s , th e m issing v alu es a re a natu ral p art o f th e d ata s e t (e .g ., th e h o u seh o ld in co m e field is o ften left u n a n sw e red b y p e o p le w h o are in th e to p in c o m e tier). In this s te p , th e analyst sh o u ld a lso id en tify n o isy valu es in th e data (i.e ., th e o u tliers) and s m o o th th e m out. A dditionally, in c o n s iste n c ie s (u n u su al valu es w ith in a v aria b le ) in th e d ata sh o u ld b e h an d led u sin g d o m a in k n o w le d g e and/or e x p e rt o p in ion .

In th e third p h a se o f data p re p ro cessin g , th e data is tran sform ed fo r b e tte r p ro c e s s ­ ing. F o r in sta n ce, in m a n y c a s e s th e d ata is no rm alized b e tw e e n a ce rta in m in im u m and m axim u m fo r all v a ria b le s in ord e r to m itigate th e p o ten tial b ia s o f o n e v a ria b le (h av in g large n u m e ric v a lu e s, s u ch as fo r h o u s e h o ld in c o m e ) d om inating o th er v a ria b le s (s u ch as n u m b er o f d ep en d en ts o r y e a r s in service, w h ich m ay p o ten tially b e m o re im portant) having sm aller values. A nother tran sform atio n that ta k e s p la ce is d iscretizatio n and/or aggreg ation . In s o m e c a s e s , th e n u m e ric v ariab les a re co n v e rte d to ca te g o rica l valu es (e .g ., low , m ed iu m , h ig h ); in o th e r c a s e s a n om in al v ariab le s u n iq u e v a lu e ran g e is re d u ce d to a sm a lle r s e t u sin g c o n c e p t h ierarch ie s (e .g ., as o p p o s e d to u sin g th e individual states w ith 5 0 d iffe ren t v alu e s, o n e m ay c h o o s e to u s e sev eral reg io n s fo r a v ariab le that sh ow s lo c a tio n ) in o rd e r to h av e a d ata s e t th at is m o re a m e n a b le to co m p u te r p ro c e s s ­ ing. Still, in o th e r c a s e s o n e m ig h t c h o o s e to c re a te newT v ariab les b a se d o n th e existin g o n e s in o rd e r t o m agnify th e in form ation fo u n d in a c o lle c tio n o f v a ria b le s in th e data set. F o r in s ta n ce , in a n o rg an tran sp lan tation d ata s e t o n e m ig h t c h o o s e t o u s e a sin gle variab le s h o w in g th e b lo o d -ty p e m atch (1 : m atch , 0: n o -m a tc h ) as o p p o s e d to sep arate m u ltinom inal v a lu e s fo r th e b lo o d ty p e o f b o th th e d o n o r an d th e re cip ie n t. Su ch sim ­ plification m ay in c re a se th e in form ation c o n te n t w h ile red u cin g th e co m p le x ity o f th e

relatio n sh ip s in th e data.

2 3 8 Part III • Predictive Analytics

T h e final p h a se o f d ata p re p ro ce ssin g is d ata red u ctio n . E v e n th o u g h data m iners lik e to h av e large d ata sets, to o m u ch d ata is a lso a p ro b lem . In th e sim p le st se n se , o n e c a n visu alize th e data co m m o n ly u s e d in d ata m in in g p ro je cts as a flat file co n sistin g o f tw o d im en sio n s: v ariab les (th e n u m b e r o f co lu m n s ) a n d cases/ record s (th e n u m b e r o f ro w s). In s o m e c a s e s (e .g ., im ag e p ro c e s s in g an d g e n o m e p ro je c ts w ith c o m p le x m icroar­ ray d ata), th e n u m b e r o f v ariab les c a n b e rath er la rg e , an d th e analyst m u st re d u ce the n u m b e r to a m a n a g e a b le size. B e c a u s e th e v a ria b le s are tre a te d as d ifferen t d im en sion s that d e s c rib e th e p h e n o m e n o n fro m d iffe ren t p e rsp e ctiv es, in d ata m in in g this p ro cess is c o m m o n ly ca lle d d im en sio n a l red u ction . E ven th o u g h th e re is n o t a sin gle b e s t w ay to a c c o m p lis h this task, o n e c a n u s e th e find ings fro m p rev iou sly p u b lis h e d literature; co n s u lt d o m ain e x p e rts ; run ap p ro p riate statistical te sts (e .g ., p rin cip al c o m p o n e n t analy­ sis o r in d e p e n d en t co m p o n e n t an alysis); and , m o re p referab ly, u se a co m b in a tio n o f th e s e te c h n iq u e s to s u cce ssfu lly re d u ce th e d im en sio n s in th e data into a m o re m a n a g e ­ a b le a n d m o st relev an t subset.

W ith r e s p e c t to th e o th e r d im e n s io n ( i .e ., th e n u m b e r o f c a s e s ) , s o m e d ata sets m ay in c lu d e m illio n s o r b illio n s o f r e c o rd s . E v e n th o u g h c o m p u tin g p o w e r is in c r e a s ­ in g e x p o n e n tia lly , p r o c e s s in g s u c h a la rg e n u m b e r o f re c o rd s m a y n o t b e p ra cti­ c a l o r fe a s ib le . In s u c h c a s e s , o n e m a y n e e d t o s a m p le a s u b s e t o f th e d ata fo r a n a ly sis. T h e u n d e rly in g a s s u m p tio n o f s a m p lin g is th at th e s u b s e t o f th e d ata w ill c o n ta in all re le v a n t p a tte rn s o f th e c o m p le te d ata s e t. In a h o m o g e n o u s d a ta s e t, s u ch a n a s s u m p tio n m ay h o ld w e ll, b u t re a l-w o rld d ata is h a rd ly e v e r h o m o g e n o u s . T h e a n a ly st sh o u ld b e e x tr e m e ly c a re fu l in s e le c tin g a s u b s e t o f th e d ata th at r e fle c ts th e e s s e n c e o f th e c o m p le te d ata s e t a n d is n o t s p e c ific to a s u b g ro u p o r s u b c a te g o ry . T h e d ata is u su a lly s o rte d o n s o m e v a r ia b le , a n d ta k in g a s e c tio n o f th e d ata fro m th e to p o r b o tto m m a y le a d to a b ia s e d d ata s e t o n s p e c ific v a lu e s o f th e in d e x e d v a r ia b le ; th e r e fo r e , o n e s h o u ld a lw ay s try to ra n d o m ly s e le c t th e re c o rd s o n th e sa m ­ p le s e t. F o r s k e w e d d a ta , straig h tfo rw ard ra n d o m s a m p lin g m a y n o t b e s u ffic ie n t, and s tra tifie d s a m p lin g (a p ro p o rtio n a l r e p r e s e n ta tio n o f d iffe re n t s u b g ro u p s in th e data is re p re s e n te d in th e s a m p le d ata s e t ) m ay b e re q u ire d . S p e a k in g o f s k e w e d d a ta : It is a g o o d p r a c t ic e to b a la n c e th e h ig h ly s k e w e d d a ta b y e ith e r o v e r sa m p lin g th e less r e p re s e n te d o r u n d e rs a m p lin g th e m o re r e p r e s e n te d c la s s e s . R e s e a r c h h a s s h o w n that b a la n c e d d ata s e ts te n d t o p r o d u c e b e tt e r p r e d ic tio n m o d e ls th a n u n b a la n c e d o n e s (W ils o n a n d S h ard a, 1 9 9 4 ).

T h e e s s e n c e o f data p re p ro ce s s in g is su m m arized in T a b le 5.1 , w h ic h m ap s th e m ain p h a s e s (a lo n g w ith th e ir p ro b le m d escrip tio n s) to a re p re sen tativ e list o f ta sk s and algorithm s.

Step 4: Model Building In this ste p , v arious m o d e lin g te c h n iq u e s are s e le c te d a n d ap p lie d to a n alread y p re ­ p ared d ata s e t in o rd e r to ad d ress th e s p e cific b u s in e s s n e ed . T h e m o d e l-b u ild in g step a lso e n c o m p a ss e s th e a sse ssm e n t an d co m p arativ e analysis o f th e vario u s m o d els built. B e c a u s e th e re is n o un iversally k n o w n best m e th o d o r algorithm fo r a d ata m ining task, o n e sh o u ld u s e a variety o f v ia b le m o d e l ty p es a lo n g w ith a w e ll-d e fin e d e x p e rim e n ta ­ tio n an d assessm e n t strategy to id entify th e “b e s t" m e th o d fo r a given p u rp o se . E v e n fo r a sin g le m e th o d o r algorithm , a n u m b e r o f p a ra m ete rs n e e d to b e calib rate d to o b tain op tim al results. S o m e m eth o d s m ay h av e s p e cific re q u irem en ts o n th e w a y that th e data is to b e form atted ; thu s, ste p p in g b a c k to th e data p rep aratio n s te p is o fte n n ecessary. A p p licatio n C ase 5-4 p re sen ts a re s e a r c h study w h e re a n u m b e r o f m o d e l ty p e s a re d ev el­ o p e d an d c o m p a re d to e a c h other.

D e p e n d in g o n th e b u s in e s s n e e d , th e d ata m in in g ta s k c a n b e o f a p r e d ic tio n (e it h e r c la s s ific a tio n o r re g r e s s io n ), a n a s s o c ia tio n , o r a c lu s te rin g ty p e. E a c h o f th e s e

Chapter 5 • Data Mining 2 3 9

T A B L E 5.1 A Sum m ary o f D ata Preprocessing Tasks and Po tential M ethods

M ain Task Subtasks Popular M ethods

Data consolidation Access and collect the data

Select and filter the data

SQL queries, software agents, W eb services.

Domain expertise, SQL queries, statistical tests.

Integrate and unify the data SQL queries, domain expertise, ontology-driven data mapping.

Data cleaning Handle missing values in the data Fill-in missing values (imputations) with most appropriate

values (mean, median, min/max, mode, etc.); recode the missing values with a constant such as ML ; remove the record of the missing value; do nothing.

Identify and reduce noise in the data Identify the outliers in data with simple statistical techniques (such as averages and standard deviations) or with cluster analysis; once identified either remove the outliers or smooth them by using binning, regression, or simple averages.

Find and eliminate erroneous data Identify the erroneous values in data (other than outliers), such as odd values, inconsistent class labels, odd distributions; once identified, use domain expertise to correct the values or remove the records holding the erroneous values.

Data transformation Normalize the data Reduce the range of values in each numerically valued variable to a standard range (e.g., 0 to 1 or -1 to +1) by using a variety of normalization or scaling techniques.

Discretize or aggregate the data If needed, convert the numeric variables into discrete representations using range or frequency-based binning techniques; for categorical variables reduce the number of values by applying proper concept hierarchies.

Construct new attributes Derive new and more informative variables from the exist­ ing ones using a wide range of mathematical functions (as simple as addition and multiplication or as complex as a hybrid combination of log transformations).

Data reduction Reduce number of attributes Principal component analysis, independent component

analysis, Chi-square testing, correlation analysis, and decision tree induction.

Reduce number of records Random sampling, stratified sampling, expert- knowledge-driven purposeful sampling.

Balance skewed data Oversample the less represented or undersample the more represented classes.

d ata m in in g ta s k s c a n u se a v a rie ty o f d ata m in in g m e th o d s a n d a lg o rith m s . S o m e o f th e s e d ata m in in g m e th o d s w e r e e x p la in e d e a rlie r in th is c h a p te r, an d s o m e o f th e m o s t p o p u la r a lg o rith m s, in clu d in g d e c is io n t r e e s fo r c la s s ific a tio n , fe-m eans fo r clu s te rin g , a n d th e A p rio ri a lg o rith m fo r a s s o c ia tio n ru le m in in g , a re d e s c r ib e d la te r

in th is c h a p te r.

2 4 0 Part III • Predictive Analytics

Application Case 5.4 Data Mining in Cancer Research A cco rd in g to th e A m erican C a n ce r S o cie ty , h a lf o f all m e n a n d o n e -th ird o f all w o m e n in th e U n ited States w ill d e v e lo p c a n c e r during th e ir lifetim es; ap p ro xi­ m ately 1.5 m illion n e w c a n c e r c a s e s w ill b e d iag­ n o s e d in 2013- C an cer is th e s e c o n d m o st co m m o n c a u s e o f d e a th in th e U n ited States an d in th e w orld, e x c e e d e d o n ly b y card io v ascu lar d ise a se. T h is y ear, o v e r 5 0 0 ,0 0 0 A m erican s a re e x p e c te d to d ie o f c a n c e r— m o re th an 1 ,3 0 0 p e o p le a day— acco u n tin g fo r n e arly 1 o f e v ery 4 deaths.

C a n c e r is a g ro u p o f d is e a s e s g e n e r a lly c h a r­ a c te r iz e d b y u n c o n tro lle d g ro w th a n d s p re a d o f a b n o rm a l c e lls . I f th e g ro w th and /or s p re a d is n o t c o n tr o lle d , it c a n re su lt in d e a th . E v e n th o u g h th e e x a c t r e a s o n s a re n o t k n o w n , c a n c e r is b e lie v e d to b e c a u s e d b y b o th e x te r n a l fa c to rs ( e .g ., to b a c c o , in fe c tio u s o rg a n is m s , c h e m ic a ls , a n d ra d ia tio n ) and in te rn a l fa c to rs ( e .g ., in h e rite d m u tatio n s, h o r m o n e s , im m u n e c o n d itio n s , an d m u ta tio n s th a t o c c u r fro m m e ta b o lis m ). T h e s e c a u sa l fa c to rs m ay a c t to g e th e r o r in s e q u e n c e to in itia te o r p ro m o te c a r c in o g e n e s is . C a n c e r is tre a te d w ith su rgery , ra d ia tio n , c h e m o th e r a p y , h o r m o n e th e ra p y , b io lo g ic a l th e ra p y , a n d ta rg e te d th e ra p y . Survival sta tistics v a ry g re a tly b y c a n c e r ty p e a n d s ta g e at d ia g n o s is .

T h e 5 -y e ar relativ e survival rate fo r all c a n ­ ce rs is im provin g, and d e clin e in c a n c e r m ortality h as r e a c h e d 2 0 p e rc e n t in 2 0 1 3 , tran slating to th e a v o id a n ce o f a b o u t 1 .2 m illion d eath s fro m c a n c e r s in c e 1 9 9 1 . T h a t’s m o re th a n 4 0 0 liv es sav e d p e r day! T h e im p ro v e m en t in survival re flects p ro g ress in d ia g n o sin g certain c a n ce rs a t an e a rlie r stag e and im p ro v e m en ts in treatm en t. Further im p ro vem en ts are n e e d e d to p rev en t an d treat ca n ce r.

E v e n th o u g h c a n c e r re s e a r c h has trad ition­ ally b e e n clin ical an d b io lo g ica l in natu re, in re c e n t y e a rs d ata-d riven an aly tic stu d ies h av e b e c o m e a co m m o n co m p le m e n t. In m e d ica l d o m ain s w h ere d ata- an d an alytics-d riven re se a rch h av e b e e n ap p lie d su cce ssfu lly , n o v e l re se a rch d irectio n s h av e b e e n id en tified to fu rth er ad v a n ce th e clin ical an d b io lo g ic a l studies. U sing vario u s ty p es o f data, inclu d ing m o lecu lar, clin ical, literatu re-based , an d clin ical-trial d ata, a lo n g w ith s u ita b le data m in in g : to o ls and te ch n iq u e s , re sea rch ers h av e b e e n a b le to

identify n o v e l pattern s, p av in g th e ro ad tow ard a c a n c e r-fre e s o ciety .

In o n e stu d y, D e le n ( 2 0 0 9 ) u s e d th r e e p o p u ­ lar d ata m in in g te c h n iq u e s (d e c is io n tre e s , artificial n e u ra l n e tw o rk s , a n d s u p p o rt v e c to r m a c h in e s ) in c o n ju n c tio n w ith lo g is tic re g re ss io n to d e v e lo p p r e d ic tio n m o d e ls fo r p ro s ta te c a n c e r survivability. T h e d ata s e t c o n ta in e d a ro u n d 1 2 0 ,0 0 0 re c o rd s and 7 7 v a r ia b le s . A M o l d c ro ss -v a lid a tio n m e th o d o l­ o g y w a s u s e d in m o d e l b u ild in g , e v a lu a tio n , an d co m p a riso n . T h e re su lts s h o w e d th at s u p p o rt v e c ­ to r m o d e ls a re th e m o st a c c u ra te p re d icto r (w ith a te s t s e t a c c u r a c y o f 9 2 .8 5 % ) fo r th is d o m a in , fo l­ lo w e d b y a rtificia l n e u ra l n e tw o rk s a n d d e c is io n tre e s. F u rth e rm o re , u s in g a se n sitiv ity -a n a ly s is - b a s e d e v a lu a tio n m e th o d , th e stu d y a ls o re v e a le d n o v e l p a tte rn s re la te d to p ro g n o s tic fa c to rs o f p ro sta te c a n c e r .

In a re la ted study, D e le n e t al. ( 2 0 0 4 ) u s e d tw o data m in in g algorithm s (artificial n eu ral n etw o rk s a n d d e cisio n tre e s) an d lo g istic re g re ssio n to d ev elo p p re d ictio n m o d e ls fo r b rea st c a n c e r survival u sin g a larg e data s e t (m o re th an 2 0 0 ,0 0 0 ca ses). U sing a 10-fo ld cro ss-v alid atio n m e th o d to m e a ­ su re th e u n b ia se d e stim ate o f the p red ictio n m o d els fo r p e rfo rm a n ce c o m p a riso n p u rp o se s, th e results in d icate d th at th e d e c is io n tree (C 5 alg orith m ) w as th e b e s t p re d icto r, w ith 9 3 .6 p e rce n t a ccu ra cy o n th e h o ld o u t sa m p le (w h ic h w a s th e b e s t p red ic­ tio n a ccu ra cy re p o rte d in th e literatu re); fo llo w ed by artificial n e u ra l n etw o rk s, w ith 9 1 -2 p e rce n t a ccu ra cy ; a n d lo g istic re g re ssio n , w ith 89-2 p e rce n t accu racy . F u rth e r an alysis o f p re d ictio n m o d els re v e a le d p rioritized im p o rta n ce o f th e p ro g n o stic facto rs, w h ich c a n th e n b e u se d as b asis fo r further clin ical a n d b io lo g ic a l re se a rch studies.

T h e s e e x a m p le s (a m o n g m an y oth ers in th e m ed ical literatu re) sh o w that ad v an ce d data m ining te c h n iq u e s c a n b e u se d to d ev elo p m o d els th at p o ss e s s a h ig h d eg re e o f pred ictive as w ell a s e xp lan ato ry p o w er. A lthough d ata m ining m eth o d s are ca p a b le o f extractin g patterns an d relationships hid d en d e e p in large an d c o m p le x m ed ical data­ b a se s , w ith o u t th e co o p e ra tio n a n d fe e d b a c k from th e m ed ical e x p e rts th e ir results are n o t o f m u ch use. T h e patterns fo u n d via data m ining m eth o d s shou ld

Chapter 5 • Data Mining 241

b e e v alu ated b y m ed ical p ro fessio n als w h o hav e y ears o f e x p e r ie n c e in th e p ro b le m d om ain to d ecid e w h e th e r th e y are log ical, a ctio n ab le , an d n o v e l to w arrant n e w re sea rch directions. In short, data m in­ ing is n o t m e an t to re p la ce m ed ical p ro fessio n als a n d re sea rch ers, b u t to co m p le m e n t th eir invaluable efforts to pro v id e data-driven n e w re sea rch directions and to ultim ately save m o re h u m an lives.

Q u e s t i o n s f o r D i s c u s s i o n

1. H o w c a n data m in in g b e u sed fo r u ltim ately cu r­ ing illn e s s e s lik e cancer?

2. W h at d o y o u th in k a re th e p ro m ise s an d m ajo r c h a lle n g e s fo r d ata m iners in co n trib u tin g to m ed ical a n d b io lo g ica l re se a rch end eavors?

Sources: D. Delen, “Analysis o f Cancer Data: A Data Mining Approach,” Expert Systems, Vol. 26, No. 1, 2009, pp. 100-112; J. Thongkam, G. Xu. Y. Zhang, and F. Huang, “Toward Breast Cancer Survivability Prediction Models Through Improving Training Space,” E xpert Systems w ith A pplications, Vol. 36, No. 10, 2009, pp- 12200-12209; D. Delen, G. Walker, and A. Kadam, “Predicting Breast Cancer Survivability: A Comparison o f Three Data Mining Methods," A rtificial In tellig en ce in M edicine, Vol. 34, No. 2, 2005, pp. 113-127.

Step 5: Testing and Evaluation In ste p 5, th e d e v e lo p e d m o d e ls a re a s s e s s e d a n d e v a lu a te d fo r th e ir a c c u r a c y a n d g e n era lity . T h is s te p a s s e s s e s th e d e g r e e to w h ic h th e s e le c te d m o d e l ( o r m o d e ls ) m e e ts th e b u s in e s s o b je c tiv e s an d , i f s o , to w h a t e x te n t ( i .e ., d o m o re m o d e ls n e e d to b e d e v e lo p e d a n d a s s e s s e d ). A n o th e r o p tio n is to te s t th e d e v e lo p e d m o d e l(s ) in a re a l-w o rld s c e n a r io i f tim e a n d b u d g e t co n s tra in ts p erm it. E v e n th o u g h th e o u tc o m e o f th e d e v e lo p e d m o d e ls is e x p e c te d to r e la te to th e o rig in a l b u s in e s s o b je c tiv e s , o th e r fin d in g s th a t a re n o t n e c e s s a rily re la te d to th e o rig in a l b u s in e s s o b je c tiv e s b u t th a t m ig h t a ls o u n v e il a d d itio n al in fo rm a tio n o r h in ts fo r fu tu re d ir e c tio n s o f te n are d isco v e re d .

T h e te stin g an d ev alu atio n s te p is a critical a n d ch a lle n g in g task. N o valu e is ad ded b y the d ata m in in g task un til th e b u sin e ss v alu e o b ta in e d fro m d isco v e red k n o w le d g e o atterns is id en tified an d re co g n iz e d . D eterm in in g th e b u sin ess v a lu e fro m d isco v e red k n o w le d g e p attern s is so m e w h a t sim ilar to p layin g w ith pu zzles. T h e e xtracted k n o w le d g e p attern s are p ie c e s o f th e p u zzle that n e e d to b e put to g e th e r in th e co n te x t o f the s p e c ific b u sin ess p u rp o se . T h e s u c c e s s o f this id en tificatio n o p e ra tio n d ep en d s o n th e in te ra ctio n am o n g data analysts, b u sin e ss analysts, an d d e cisio n m a k e rs (s u ch as b u sin ess m a n a g e rs). B e c a u s e data analysts m ay n o t h av e th e full u n d erstan d in g o f th e data m ining o b je c tiv e s an d w h a t th ey m e a n to th e b u sin e ss an d th e b u s in e s s analysts an d d e c is io n m ak ers m ay n o t h av e th e te ch n ica l k n o w le d g e to in te rp re t th e results o f so p h isticated m ath em atical solu tion s, in teractio n am o n g th e m is n e ce ssa ry . In o rd e r to properly in te rp re t k n o w le d g e p attern s, it is o fte n n e c e s s a ry to u se a v arie ty o f tab u lation and visu alization te c h n iq u e s (e .g ., p iv o t ta b le s, cro ss-ta b u la tio n o f fin d in g s, p ie charts, histogram s, b o x p lo ts, scatter p lots).

Step 6: Deployment D ev e lo p m en t an d a sse ssm e n t o f th e m o d e ls is n o t th e en d o f th e d ata m ining p ro ject. Even i f th e p u rp o s e o f th e m o d e l is to h av e a sim p le e x p lo ra tio n o f th e d ata, th e k n o w l­ ed ge g a in ed fro m s u ch e x p lo ra tio n w ill n e e d to b e org an ized a n d p re s e n te d in a w ay _hat th e e n d u se r c a n un derstand an d b e n e fit from . D ep en d in g o n th e req u irem en ts, i h f d ep lo y m en t p h a se c a n b e a s sim p le as g e n eratin g a rep o rt o r a s c o m p le x as im ple- o e n tin g a re p e a ta b le d ata m in in g p ro ce s s a cro ss th e en te rp rise. In m a n y c a s e s , it is th e custom er, n o t th e data analyst, w h o carries o u t th e d ep lo y m en t ste p s. H o w e v e r, e v e n if

2 4 2 Part III • Predictive Analytics

th e analyst w ill n o t carry o u t th e d e p lo y m e n t effo rt, it is im portant fo r th e cu sto m e r to u n d erstan d u p fro n t w h a t a ctio n s n e e d to b e ca rrie d o u t in o rd e r to actu ally m a k e u s e o f th e c re a te d m od els.

T h e d ep lo y m en t step m ay a lso in clu d e m a in te n a n c e activities fo r th e d ep loy ed m o d els. B e c a u s e e v ery th in g a b o u t th e b u sin e ss is co n stan tly ch a n g in g , th e data th at re flect th e b u sin e ss activities a lso are ch an g in g . O v e r tim e , th e m o d els (a n d th e pattern s e m b e d ­ d e d w ith in th e m ) b u ilt o n th e o ld d ata m ay b e c o m e o b s o le te , irrelevant, o r m isleading. T h e re fo re , m o n ito rin g a n d m ain te n a n ce o f th e m o d e ls are im portant i f th e data m ining results are to b e c o m e a p art o f th e d ay -to -d ay b u sin e ss and its en viron m en t. A care ­ ful p rep aratio n o f a m a in te n a n ce strategy h e lp s to av o id u n n ece ssa rily lo n g p erio d s o f in co rre ct u sa g e o f data m in in g results. In o rd e r to m o n ito r th e d ep lo y m en t o f th e data m in in g resu lt(s), th e p ro je c t n e e d s a d etailed p la n o n th e m o n ito rin g p ro ce ss, w h ic h m ay n o t b e a trivial ta sk fo r c o m p le x d ata m ining m o d e ls.

Other Data Mining Standardized Processes and Methodologies In ord e r to b e ap p lie d su ccessfu lly , a data m in in g stud y m ust b e v ie w e d a s a p ro cess th at fo llo w s a stand ard ized m e th o d o lo g y rath er th a n as a s e t o f a u tom ated softw are tools a n d te ch n iq u e s. In ad dition to CRISP-DM , th e re is an o th e r w e ll-k n o w n m eth o d o lo g y d e v e lo p e d b y th e SAS Institute, ca lle d SEMMA ( 2 0 0 9 ) . T h e acro n y m SEMMA stand s for “sam p le, e x p lo re , m odify, m o d el, a n d a s s e s s .”

B e g in n in g w ith a sta tistica lly re p re s e n ta tiv e s a m p le o f th e d ata, SEMMA m a k e s it e a s y to a p p ly e x p lo r a to ry s ta tistica l a n d v is u a liz a tio n te c h n iq u e s , s e le c t a n d tra n s­ fo rm th e m o st s ig n ific a n t p re d ic tiv e v a r ia b le s , m o d e l th e v a r ia b le s to p re d ic t o u tc o m e s , a n d c o n fir m a m o d e l’s a c c u ra c y . A p icto ria l re p re s e n ta tio n o f SEMMA is g iv e n in F ig u re 5 .6 .

B y a ssessin g th e o u tc o m e o f e a c h s ta g e in th e SEMMA p ro c e s s , th e m o d e l d e v e lo p e r c a n d eterm in e h o w to m o d e l n e w q u e s tio n s ra ise d b y th e p re v io u s results, a n d thu s p r o c e e d b a c k to th e e x p lo r a tio n p h a s e fo r ad d ition al re fin e m e n t o f th e data; th a t is, a s w ith CRISP-D M , SEMMA is d riven b y a h ig h ly iterativ e e x p e rim e n ta tio n c y cle .

FIG U R E 5 .6 SEM MA Data M ining Process.

Chapter 5 • Data Mining 2 4 3

T h e m ain d iffe re n c e b e tw e e n CRISP-D M a n d SEMMA is th a t CRISP-D M ta k e s a m o re co m p re h e n siv e a p p r o a c h — in clu d in g u n d erstan d in g o f th e b u sin e ss a n d th e relev an t data— to d ata m in in g p ro je cts, w h e re a s SEMMA im p licitly a ssu m e s th a t th e d ata m in in g p ro je c t’s g o a ls a n d o b je c tiv e s a lo n g w ith th e a p p ro p ria te d ata s o u rc e s h a v e b e e n id e n ti­

fie d and u n d e rsto o d . S o m e p ractitio n ers co m m o n ly u se th e te rm k n o w l e d g e d i s c o v e r y i n d a t a b a s e s

(KDD) as a sy n o n y m fo r d ata m ining. Fayy ad et al. (1 9 9 6 ) d efin e d kn ow led g e discovery in d a ta b a ses as a p ro c e s s o f u sin g data m in in g m eth o d s to find u sefu l in fo rm atio n and patterns in th e data, as o p p o s e d to d ata m ining, w h ic h in v o lv es using algorith m s to identify p attern s in data d eriv ed throu gh th e K D D p ro ce ss. K D D is a co m p re h e n siv e p ro ce s s th at e n c o m p a ss e s data m ining. T h e in p u t to th e K D D p ro ce s s co n sists o f organ i­ zational data. T h e e n terp rise d ata w a re h o u s e e n a b le s K D D to b e im p lem e n te d e fficien tly b e c a u se it p ro v id e s a sin gle s o u rc e fo r data to b e m ined . D u n h am (2 0 0 3 ) su m m aiized th e K D D p r o c e s s as co n sistin g o f th e fo llo w in g step s: data s e le c tio n , d ata p re p ro c e s s ­ ing, data tran sform atio n , d ata m ining, an d interpretation/evaluation. Figure 5 .7 s h o w s th e p o llin g results fo r th e q u e stio n “W h at m ain m e th o d o lo g y a re y o u u sin g fo r d ata mining? ’ (c o n d u cte d b y k d n u g g e t s . c o m in A ugust 2 0 0 7 ).

S E C T I O N 5 4 R E V I E W Q U E S T I O N S

1 . W h a t are th e m a jo r data m in in g p ro cesses? 2 . W h y d o y o u th in k th e early p h a s e s (u n d erstan d in g o f th e b u sin e ss an d u n d erstan d ­

ing o f th e d ata) tak e th e lo n g e st in d ata m ining projects?

3 . List ancl b rie fly d efin e th e p h a s e s in th e CRISP-DM p ro cess. 4 . W h a t are th e m ain d ata p re p ro ce s s in g steps? B riefly d e s c rib e e a c h s te p a n d p rovide

relev an t e x a m p le s. 5 . In th e CRISP-D M p ro c e s s , w h at d o e s ‘d ata re d u ctio n ’ m ean?

CRISP-DM

My own

SEM M A

KDD Pro cess

My organization’s

None

Domain-specific methodology

Other methodology [not domain specific]

10 20 30 40 50 60 70

F IG U R E 5 .7 R a n k in g o f D ata M in in g M e th o d o lo gie s/P ro cesses. Source: Used w ith perm ission —om k d n u g g e ts.c o m .

2 4 4 Part III • Predictive Analytics

5.5 D A T A M IN IN G M ETH O D S A variety o f m e th o d s are a v ailab le fo r p erfo rm in g d ata m in in g stud ies, in clu d in g classi­ ficatio n , re g re ssio n , clu sterin g, an d a sso cia tio n . M o st data m in in g so ftw are to o ls em p lo y m o re th an o n e te ch n iq u e (o r algorithm ) fo r e a c h o f th e s e m eth o d s. T h is s e c tio n d escrib e s th e m o st p o p u la r data m in in g m e th o d s an d e x p la in s th e ir re p re sen tativ e te ch n iq u e s.

Classification Classification is perhaps th e m ost frequently used data m ining m etho d fo r real-w orld prob­ lem s. As a popular m em b er o f th e m achine-learn ing fam ily o f techniqu es, classification learns patterns from past data (a set o f inform ation— traits, variables, f e a t u r e s - o n charac­ teristics o f th e previously lab eled item s, ob jects, o r events) in order to p lace new instances (w ith u n kn ow n lab els) into their respective gro u p s o r classes. F or exam p le, o n e cou ld use classification to predict w h eth er th e w eath er o n a particular day will b e “sunny,” rainy> o r “cloud y.” Popu lar classification tasks inclu de cred it approval (i.e ., g o o d o r b a d credit risk), store location (e .g ., goo d , m oderate, b ad ), target m arketing (e .g ., likely cu stom er no h o p e ), fraud d etection (i.e., y e s, n o ), and telecom m u n ication (e .g ., likely to turn to another p h o n e com pany, yes/no). I f w h at is b e in g pred icted is a class la b e l (e .g ., “sunny, rainy, or “cloudy”) , th e pred iction pro blem is called a classification, w h ereas if it is a n u m en c value (e .g ., tem perature su ch as 6 8°F ), th e pred iction p ro b lem is called a r e g r e s s i o n .

’ E ven th o u g h clu sterin g (an o th e r p o p u lar d ata m ining m e th o d ) c a n also b e u sed to d eterm in e gro u p s (o r class m e m b ersh ip s) o f things, th e re is a significant d ifferen ce b e tw e e n th e tw o. C lassification learn s th e fu n ctio n b e tw e e n th e ch aracteristics o f things (i e in d ep en d en t v ariab le s) an d th e ir m em b ersh ip (i.e ., ou tp u t v ariab le ) th ro u g h a super­ v ised learn in g p ro cess w h e re b o th ty p e s (in p u t a n d ou tpu t) o f v ariab les are p re sen ted to th e algorithm ; in clustering, th e m em b ersh ip o f th e o b je c ts is le a rn e d throu gh a n un su ­ p ervised learn in g p ro ce s s w h e re o n ly th e input variab les are p re sen ted to th e algorithm . U n like classificatio n, clu sterin g d o e s n o t h av e a supervising (o r con tro llin g ) m ech an ism th at e n fo rce s the learn in g p ro ce ss; instead , clu sterin g algorithm s u s e o n e o r m o re heuristics (e .g ., m ultid im ensional d istan ce m e asu re ) to d isco v e r natural gro u p ings o f o b je cts.

T h e m o st co m m o n tw o -ste p m e th o d o lo g y o f classificatio n -ty p e p re d ictio n in v o ves m o d e l dev elopm en t/training an d m o d e l testin g/d eploym ent. In th e m o d el d ev elo p m en t p h ase , a c o lle c tio n o f inp u t d ata, in clu d in g th e actu al class la b e ls, is u se d . A fter a m o e h a s b e e n trained , th e m o d e l is te s te d against th e h o ld o u t sam p le fo r a ccu ra cy a ssessm en t a n d ev en tu ally d ep lo y e d fo r actu al u se w h e re it is to p re d ict cla ss e s o f n e w d ata in stan ces (w h e r e th e class la b e l is u n k n o w n ). S ev eral fa c to rs a re co n sid e re d in a ssessin g th e m o d el,

in clu d in g th e fo llow ing: • P r e d ic t iv e a c c u r a c y . T h e m o d e l’s ab ility to co rrectly p red ict th e class la b e l o f n e w

o r p rev iou sly u n s e e n d ata. P red ictio n a c c u ra c y is th e m o st co m m o n ly u s e d a ssess­ m e n t facto r fo r classificatio n m o d e ls. T o c o m p u te this m easu re, actu al class lab e ls o f a te s t data s e t a re m a tch e d ag ain st th e class la b e ls p re d icte d b y th e m o d el. T h e a ccu ra cy c a n th e n b e co m p u te d as th e a c c u r a c y rate, w h ic h is th e p e rc e n ta g e o f te st d ata s e t sam p les co rrectly classified b y th e m o d e l (m o re o n this to p ic is p ro ­

v id ed later in th e ch a p ter). , ( , • S p e e d . T h e co m p u tatio n al co s ts in v o lv ed in g e n era tin g and u sin g th e m o d e l, w h ere

fa ste r is d e e m e d to b e b etter. • R o b u s tn e s s . T h e m o d e l’s ab ility to m a k e re a so n a b ly a ccu ra te p re d ictio n s, g iven

n o isy d ata o r d ata w ith m issin g and e rro n e o u s v alu es. • S c a la b ilit y . T h e ab ility to co n stru ct a p re d ictio n m o d e l efficien tly g iv en a rathei

larg e am o u n t o f data. • I n t e r p r e t a b ilit y . T h e lev el o f u n d erstan d in g an d insight pro v id ed b y th e m o d e l

(e .g ., h o w and/or w h a t th e m o d e l c o n c lu d e s o n ce rta in p red iction s).

C hapter 5 • Data Mining 2 4 5

True Class

Positive Negative

0> True False CD Positive Positive CL Count (TP) Count (FP]

0> False True ra03 Negative Negativeuj Z Count (FN] Count [TN]

FIGURE 5.8 A Simple Confusion Matrix for Tabulation of Two-Class Classification Results.

Estimating the True Accuracy o f Classification Models In classificatio n p ro b le m s, th e prim ary s o u rc e fo r a c c u ra c y estim atio n is th e co n fu sion m atrix (a ls o c a lle d a c la ssific a tio n m atrix o r a con tin g en cy tab le). Figure 5 .8 s h o w s a co n fu sio n m atrix fo r a tw o -class classificatio n p ro b lem . T h e n u m b e rs alo n g th e d iago n al from th e u p p er le ft to th e lo w e r right re p re se n t co r re c t d ecisio n s, an d th e n u m b e rs o u t­ sid e this d iag o n al re p re s e n t th e errors.

T a b le 5 .2 p ro v id e s e q u atio n s fo r co m m o n a ccu ra cy m etrics fo r cla ssifica tio n m o d els. W h e n th e cla ssifica tio n p ro b le m is n o t b in ary , th e c o n fu s io n m atrix g e ts b ig g e r

(a squ are m atrix w ith th e siz e o f th e u n iq u e n u m b e r o f c lass la b e ls ), and a c c u ra c y m etrics b e c o m e lim ited to p e r class a c c u ra c y rates an d th e ov era ll cla ssifier a ccu ra cy .

( T rue C lassification ) t ( T rue C lassification R ate)/ = —

^ {F alse C lassification ) f /= i

n ^ ( True C lassification ){

(O verall C la ssifier A ccu racy ), = ^ o m l N um ber1g C ases

Estim ating th e a ccu ra cy o f a classificatio n m o d el (o r classifier) in d u ced b y a su p e r­ vised learn in g a lg o rith m is im p ortan t fo r th e fo llo w in g tw o re a so n s: First, it c a n b e u se d :o estim ate its fu ture p re d ictio n a ccu ra cy , w h ic h co u ld im ply th e lev el o f c o n fid e n c e o n e should h av e in th e classifier’s ou tp u t in th e p red ictio n system . S e co n d , it c a n b e u s e d fo r .-no o sin g a cla ssifier fro m a g iv en s e t (id en tifyin g the ‘‘b e s t’’ cla ssifica tio n m o d e l am o n g m e m any train ed ). T h e fo llo w in g a re am o n g th e m o st p o p u la r estim atio n m e th o d o lo g ie s js e d fo r classificatio n -ty p e d ata m in in g m od els.

SIM P LE S P L IT T h e s i m p l e s p l i t ( o r h o ld o u t o r te s t s a m p le e s tim a tio n ) p a rtitio n s ibe d ata in to tw o m u tu ally e x c lu s iv e s u b s e ts c a lle d a tra in in g set a n d a test s e t (o r h o ld o u t set). It i s c o m m o n to d e s ig n a te tw o -th ird s o f th e d a ta as th e tra in in g s e t and die re m a in in g o n e -th ir d as th e te s t s e t. T h e tra in in g s e t is u s e d b y th e in d u c e r (m o d e l b u ild er), a n d th e b u ilt c la s s ifie r is th e n te s te d o n th e te s t s e t. An e x c e p t i o n t o this Eule o c c u rs w h e n th e c la s s ifie r is a n a rtificia l n e u ra l n e tw o rk . In th is c a s e , th e d ata _>■ p a rtitio n e d in to th r e e m u tu ally e x c lu s iv e s u b s e ts : tra in in g , v a lid a tio n , a n d te stin g .

2 4 6 Part III • Predictive Analytics

TABLE 5.2 Common Accuracy M etrics fo r Classification M odels M etric Description

True Positive Rate = TP

TP + FN

True Negative Rate = TN

TN + FP

Accuracy =

Precision =

Recall =

TP + TN TP + TN + FP + FN

TP TP + FP

TP TP + FN

The ratio of correctly classified positives divided by the total positive count (i.e., hit rate or recall)

The ratio of correctly classified negatives divided by the total negative count (i.e., false alarm rate)

The ratio of correctly classified instances (positives and negatives) divided by the total number of instances

The ratio of correctly classified positives divided by the sum of correctly classified positives and incorrectly classified positives

Ratio of correctly classified positives divided by the sum of correctly classified positives and incorrectly classified negatives

T h e v a lid a tio n s e t is u s e d d u rin g m o d e l b u ild in g to p r e v e n t o v e rfittin g (m o r e o n a rti­ fic ia l n e u ra l n e tw o rk s c a n b e fo u n d in C h a p te r 6 ) . F ig u re 5-9 s h o w s th e s im p le sp lit m e th o d o lo g y .

T h e m ain criticism o f this m e th o d is th at it m a k e s th e assu m p tio n that th e d ata in th e tw o su b sets are o f th e sa m e k in d (i.e ., hav e th e e x a c t sa m e p ro p e rtie s). B e c a u s e this is a sim p le ran d o m partitioning, in m o st realistic d ata sets w h e re th e data a re s k e w e d on th e classificatio n variab le, s u ch an assu m p tio n m ay n o t h o ld true. In o rd e r to im p ro ve th is situation, stratified sam p lin g is su g g ested , w h e re th e strata b e c o m e th e o u tp u t v ariab le. E v e n th o u g h this is a n im p ro v e m en t o v e r th e s im p le split, it still h a s a b ia s a sso cia ted fro m th e sin gle ran d o m partitioning.

fc-FOLD C R O S S -V A L ID A T IO N In o rd e r to m in im ize th e b ia s a s s o cia te d w ith th e rand om sam p lin g o f th e training a n d h o ld o u t data sa m p le s in co m p a rin g th e p re d ictiv e a ccu ra cy o f tw o o r m o re m e th o d s, o n e ca n u s e a m e th o d o lo g y ca lle d fe -fo ld c r o s s - v a li d a t io n . In M o ld cro ss-v alid atio n , a lso calle d rotation estim ation , th e c o m p le te data s e t is rand om ly sp lit in to k. m utually e x clu s iv e s u b sets o f a p p ro x im a te ly e q u a l size. T h e classificatio n m o d e l is train ed a n d te s te d k tim es. E a c h tim e it is train ed o n all b u t o n e fo ld and then te sted o n th e re m ain in g sin g le fo ld . T h e cro ss-v alid atio n estim ate o f th e ov erall a ccu ra cy

Chapter 5 * Data M ining 2 4 7

o f a m o d e l is ca lcu la ted b y sim p ly averagin g th e k individual a ccu ra cy m e a su re s, as sh o w n in th e fo llo w in g e q u atio n :

= \ i A‘ w h e re CVA stan d s fo r cro ss-v alid atio n accu racy , k is th e n u m b e r o f fo ld s u se d , and A is th e a ccu ra cy m e a s u re (e .g ., hit-rate, sensitivity, s p e cificity ) o f e a c h fold.

A D D IT IO N A L C L A S S IF IC A T IO N A S S E S S M E N T M E T H O D O LO G IES O th e r p o p u la r assess­ m en t m e th o d o lo g ie s in clu d e th e fo llow in g :

• L e a v e -o n e -o u t. T h e le a v e -o n e -o u t m e th o d is sim ilar to th e M o ld cro ss-v alid atio n w h e re th e k tak es th e v a lu e o f 1; that is, e v ery d ata p o in t is u se d fo r testin g o n c e o n as m a n y m o d e ls d e v e lo p e d as th ere are n u m b e r o f data p o in ts. T h is is a tim e- co n su m in g m eth o d o lo g y , b u t s o m e tim e s fo r sm all data sets it is a v ia b le o p tio n .

• B o o t s t r a p p in g . W ith bootstrapping, a fix e d n u m b e r o f in stan ces fro m th e origi­ n al data is sam p led (w ith re p la ce m e n t) fo r training an d th e rest o f th e data s e t is u s e d fo r testing. T h is p ro ce s s is re p e a te d as m any tim es as d esired.

• J a c k k n i f i n g . Similar to the leav e-on e-ou t m ethodology, w ith jackkn ifin g th e accu racy is calcu lated b y leaving o n e sam ple o u t a t e a ch iteration o f th e estim ation process.

• A r e a u n d e r t h e R O C c u r v e . T h e area under th e ROC curve is a grap h ical assess­ m e n t te ch n iq u e w h ere th e true positive rate is p lo tte d o n th e jy-axis and false positive rate is p lo tte d o n the x -a x is. T h e area u n d er th e ROC cu rve d eterm in es th e accu racy m e asu re o f a classifier: A value o f 1 in d icates a p e rfe ct classifier w h e re a s 0 .5 indicates n o b e tte r th an rand om ch a n ce ; in reality, th e values w ould ran g e b e tw e e n th e tw o extrem e c a s e s . For e x a m p le , in Figure 5 .1 0 A has a b e tte r classificatio n p erfo rm an ce than B , w h ile C is n o t an y b etter than th e ran d o m ch a n c e o f flip ping a coin.

:: 3URE 5 .1 0 A Sam ple ROC Curve.

2 4 8 Part III • Predictive Analytics

• D ecisio n tree a n a ly sis. D e c is io n tre e an alysis ( a m a ch in e -lea rn in g technique.) is argu ably th e m o st p o p u la r classificatio n te c h n iq u e in th e d ata m in in g arena. A d eta iled d escrip tio n o f this te ch n iq u e is g iv e n in th e fo llo w in g sectio n .

• Statistical a n a ly sis. Statistical te ch n iq u e s w e re th e prim ary classificatio n algo­ rithm fo r m an y y e a rs until th e e m e rg e n c e o f m a ch in e -le a rn in g te ch n iq u e s. Statistical c lassificatio n te c h n iq u e s in clu d e lo g istic re g re ssio n an d d iscrim in an t analysis, b oth o f w h ic h m ak e th e assu m p tio n s th at th e re latio n sh ip s b e tw e e n th e input a n d output v ariab les a re lin e ar in n atu re, th e d ata is n o rm ally distributed , an d th e v a ria b le s are n o t co rrela te d a n d are in d e p e n d e n t o f e a c h o th e r. T h e q u e s tio n a b le n atu re o f th ese assu m p tio n s h a s le d to th e sh ift to w ard m a ch in e -lea rn in g te ch n iq u e s.

• N e u r a l netw orks. T h e s e are am o n g th e m o st p o p u la r m a ch in e -lea rn in g te ch ­ n iq u e s that c a n b e u s e d fo r classificatio n -ty p e p ro b lem s. A d eta iled d escrip tio n o f this te ch n iq u e is p re s e n te d in C h ap ter 6.

• C ase-based r e a s o n in g . T h is a p p ro a ch u s e s h isto rical c a s e s to re co g n iz e co m m o n ­ alities in ord e r to assig n a n e w c a s e into th e m o st p ro b a b le categ ory .

• B a y e s ia n cla ssifiers. T h is a p p ro a ch u s e s p ro b ab ility th e o ry to b u ild classification m o d e ls b a s e d o n th e p a st o c c u rre n c e s th at a re c a p a b le o f p la cin g a n e w in stan ce into a m o st p ro b a b le class (o r categ o ry ).

• G enetic a lgo rith m s. T h is a p p ro a ch u se s th e a n a lo g y o f natu ral e v o lu tio n to build d ir e ctcd -s e a rch -b a s e d m e ch an ism s to classify data sam p les.

• R o u g h sets. T h is m e th o d ta k e s into a c c o u n t th e partial m e m b ersh ip o f class labels to p re d e fin e d ca te g o rie s in b u ild in g m o d e ls (c o lle c tio n o f ru les) fo r classificatio n

p ro b lem s.

A co m p le te d escrip tio n o f all o f th e s e classificatio n te c h n iq u e s is b e y o n d th e s c o p e o f this b o o k ; thus, o n ly sev era l o f th e m o st p o p u la r o n e s are p re se n te d here.

DECISION TREES B e f o r e d escrib in g th e details o f decision trees, w e n e e d to discuss s o m e sim p le term in olog y . First, d e c is io n tre e s in clu d e m an y input v a ria b le s th at may h av e a n im p act o n th e classificatio n o f d iffe ren t p attern s. T h e s e input v ariab les a re usually ca lle d attribu tes. F o r e x a m p le , i f w e w e re to b u ild a m o d el to classify lo a n risks o n the b a sis o f ju st tw o ch aracteristics— in co m e a n d a cre d it rating— th e s e tw o ch aracteristics w o u ld b e th e attributes a n d th e resu ltin g o u tp u t w o u ld b e th e class la b e l (e .g ., low , m ed iu m , o r h ig h risk ). S e c o n d , a tre e co n sists o f b ra n c h e s a n d n o d e s. A b ra n c h re p re sen ts th e o u tco m e o f a te s t to classify a p attern (o n th e b a sis o f a te s t) u sin g o n e o f th e attri­ b u te s. A l e a f n o d e a t th e e n d re p re sen ts th e fin a l class c h o ic e fo r a p attern (a ch a in o f b ra n c h e s fro m the ro o t n o d e to th e le a f n o d e , w h ic h c a n b e re p re s e n te d a s a c o m p le x

if-th e n statem en t). T h e b a sic id ea b e h in d a d e c is io n tree is th at it recu rsiv ely divid es a training s e t until

e a c h d iv ision co n sists en tirely o r prim arily o f e x a m p le s fro m o n e class. E a c h n o n le a f n o d e o f th e tre e co n ta in s a sp lit p o in t, w h ic h is a te s t o n o n e o r m o re attributes a n d d eter­ m in e s h o w th e d ata a re to b e d ivid ed further. D e c is io n tre e algorithm s, in g e n era l, b u ild a n initial tre e fro m th e training d ata s u c h that e a c h le a f n o d e is p u re, a n d th e y th e n prune th e tree to in cre a se its g e n eralizatio n , an d , h e n c e , th e p re d ictio n a ccu ra cy o n te st data.

In th e g ro w th p h a s e , th e tre e is b u ilt b y re cu rsiv e ly dividing th e d ata until e a c h divi­ s io n is e ith e r p u re (i.e ., co n ta in s m e m b e rs o f th e s a m e c la ss ) o r relativ ely sm all. T h e b a sic id ea is to a s k q u e stio n s w h o s e an sw ers w o u ld p ro v id e th e m o st in form ation , sim ilar to w h a t w e m ay d o w h e n p layin g th e g a m e “T w e n ty Q u e s tio n s .”

T h e split u se d to partition the data d ep end s o n th e typ e o f th e attribute used in th e split. F o r a con tin u ou s attribute A, splits are o f th e form value(A ) < x , w h e re x is so m e optimal

C L A S S IF IC A T IO N T E C H N IQ U E S A n u m b e r o f t e c h n i q u e s ( o r a lg o r ith m s ) a r e u s e d fo r

c la s s ific a tio n m o d e lin g , in c lu d in g t h e fo llo w in g :

Chapter 5 • Data Mining 2 4 9

split value o f A F o r exam p le, th e split b a se d o n in co m e cou ld b e “In co m e < 5 0 0 0 0 .” For the categorical attribute A, splits are o f th e fo rm v alue(A ) b elo n g s to x , w h ere x is a su b set o f A. As a n exam p le, th e split co u ld b e o n th e b asis o f g ender; “M ale versus F e m a le .”

A g e n e ra l alg o rith m fo r b u ild in g a d e c is io n tre e is as fo llow s:

1 . C reate a ro o t n o d e a n d assig n all o f th e training data to it. 2 . S e le c t th e best splitting attribute. 3 . Add a b ra n c h to th e ro o t n o d e fo r e a c h v a lu e o f th e split. Split th e d ata in to m u tu­

ally e x clu s iv e (n o n o v e rla p p in g ) s u b sets a lo n g th e lin e s o f th e s p e c ific sp lit and m o d e to th e b ra n ch e s.

4 . R e p e a t th e ste p s 2 an d 3 fo r e a c h an d e v ery le a f n o d e until th e sto p p in g crite rio n is re a ch e d ( e .g ., th e n o d e is d om in ated b y a sin g le class la b e l).

M any d iffe ren t algorithm s h a v e b e e n p ro p o s e d fo r cre a tin g d e cisio n tre e s. T h e s e algorithm s d iffe r prim arily in term s o f th e w a y in w h ic h th e y d eterm in e th e splitting attri­ b u te (a n d its sp lit v a lu e s), th e o rd e r o f splitting th e attributes (sp litting th e s a m e attribute on ly o n c e o r m a n y tim e s), th e n u m b e r o f sp lits a t e a c h n o d e (b in a ry v ersu s ternary), th e sto p p in g criteria, an d th e p ru n in g o f th e tre e (p r e - v ersu s p o stp ru n in g ). S o m e o f th e m o st w e ll-k n o w n algorithm s a re ID 3 (fo llo w e d b y C 4.5 a n d C 5 as th e im p ro v e d v e rsio n s o f ID 3 ) fro m m a ch in e learn in g , classificatio n a n d re g re ssio n tre e s (CART) fro m statistics, an d th e ch i-sq u are d au to m atic in te ractio n d e te cto r (C H A ID ) fro m p attern re co g n itio n .

W h e n b u ild in g a d e c is io n tre e , th e g o a l a t e a c h n o d e is to d eterm in e th e attribute an d th e sp lit p o in t o f that attribute th at b e s t divid es th e training re co rd s in o rd e r to purify th e cla ss re p re se n ta tio n at th a t n o d e . T o e v alu ate th e g o o d n e s s o f th e split, s o m e split­ ting in d ice s h a v e b e e n p ro p o se d . T w o o f th e m o st co m m o n o n e s are th e G in i in d e x an d inform ation g ain . T h e G in i in d e x is u s e d in CART an d SPRIN T (S c a la b le P aR allelizab le In d u ction o f D e c is io n T re e s ) algorithm s. V e rsio n s o f in form ation g ain are u s e d in ID 3 (a n d its n e w e r v e rs io n s , C 4.5 an d C 5).

T h e Gini index h a s b e e n u s e d in e c o n o m ic s to m e a su re th e diversity o f a p o p u la ­ tion. T h e sam e c o n c e p t c a n b e u se d to d eterm in e th e purity o f a s p e cific cla ss as a result o f a d e c is io n to b ra n ch a lo n g a p articu lar attribute o r variab le. T h e b e s t split is th e o n e that in cre a se s th e purity o f th e sets resultin g fro m a p ro p o s e d split. Let us b rie fly l o o k into a sim p le ca lcu la tio n o f G ini in d ex:

I f a d ata s e t S co n ta in s e x a m p le s fro m n cla sse s, th e G ini in d e x is d e fin e d as

n g in iiS ) = 1 -

j = i

w h ere p j is a relativ e fre q u e n c y o f class j in 5. I f a d ata s e t S is split into tw o su b sets, 5 , an d S2, w ith siz e s iVj a n d N2, resp ectiv ely , th e G ini in d e x o f th e sp lit data co n tain s exam p le s fro m n cla sse s, a n d th e G ini in d e x is d e fin e d as

N ig in i Sput (S ) = — g in i (SO + — g im (S 2)

T h e attribute/split c o m b in a tio n that provides th e sm allest g in isplit(S) is c h o s e n to split th e node. In s u ch a d eterm in ation , o n e sh ou ld e n u m erate all p o ss ib le splitting p o in ts fo r e a c h

attribute. Inform ation gain is th e splitting m e ch a n ism u s e d in ID 3 , w h ich is p e rh a p s th e

m ost w id ely k n o w n d e c is io n tre e algorithm . It w a s d e v e lo p e d b y R oss Q u in la n in 1986, m d sin c e th e n h e h a s e v o lv e d this algorithm in to th e C 4.5 an d C 5 alg orith m s. T h e b a sic idea b e h in d ID 3 (a n d its v arian ts) is to u se a c o n c e p t c a lle d en tropy in p la c e o f th e G ini ji d e x . Entropy m e a su re s th e e x te n t o f u n certain ty o r ra n d o m n ess in a d ata set. I f all th e -atfl in a s u b s e t b e lo n g to ju st o n e cla ss, th e re is n o u n certain ty o r ra n d o m n e ss in that

2 5 0 Part III • Predictive Analytics

data set, so th e en tro p y is zero . T h e o b je c tiv e o f this a p p r o a c h is to b u ild s u b tre e s s o that th e e n tro p y o f e a c h fin al su b se t is z e ro (o r c lo s e to z e ro ). Let us a lso lo o k at th e calcu la­ tio n o f th e in form ation gain.

A ssum e that th e re a re tw o cla sse s, P (p o s itiv e ) and N (n e g a tiv e ). Let th e s e t o f e x a m p le s S c o n ta in p co u n ts o f class P a n d n co u n ts o f class N. T h e am o u n t o f inform a­ tio n n e e d e d to d e cid e if a n arbitrary e x a m p le in S b e lo n g s to P o r N is d efin e d as

I(p , n ) - - P p + n

p n n log? — ;-------------------- l o g , --------

p + n p + n p +

A ssum e that u sin g attribute A a s e t S w ill b e p artition ed into sets (51; S2, k, S J. I f S, c o n ­ tains p i e x a m p le s o f P a n d nt e x a m p le s o f N, th e en tro p y , o r th e e x p e c te d inform ation n e e d e d to classify o b je c ts in all su b tre es, Sh is

Pi K p b n i>

i= i p + n

T h e n , th e in form ation th a t w o u ld b e g a in e d b y b ra n c h in g o n attribute A w o u ld b e

G ain (A ) = I ( p , n ) - E(A )

T h e s e ca lcu la tio n s are re p e a te d fo r e a c h a n d e v ery attrib u te, an d th e o n e w ith th e highest in form ation g a in is s e le c te d as th e splitting attribute. T h e b a s ic id eas b e h in d th e se split­ tin g in d ice s are rath er sim ilar to e a c h o th er b u t th e s p e cific algorith m ic d etails vaiy. A d eta iled d efin ition o f th e ID 3 algorithm and its splitting m e ch a n ism c a n b e found in Q u in lan (1 9 8 6 ).

A p p lication C ase 5-5 illustrates h o w sig n ifican t th e g ains m ay b e i f th e right data m in in g te c h n iq u e s are u s e d fo r a w e ll-d e fin e d b u s in e s s p ro b lem .

Cluster Analysis fo r Data Mining C luster analysis is a n e sse n tia l data m in in g m e th o d fo r classifying item s, e v e n ts, o r c o n c e p ts into co m m o n gro u p in g s calle d clusters. T h e m eth o d is co m m o n ly u se d in b io l­ ogy, m e d icin e , g e n e tics , so cial n e tw o rk analysis, a n th ro p o lo g y , a rch a e o lo g y , astronom y, ch a ra cte r re co g n itio n , a n d e v e n in MIS d ev elo p m en t. As d ata m in in g h a s in cre ase d in p opularity, th e un d erlying te ch n iq u e s have b e e n ap p lie d to b u sin ess, e sp e cia lly to m arketing. C luster analysis has b e e n u s e d e xte n siv e ly fo r fraud d e te c tio n (b o th cred it ca rd a n d e -c o m m e r c e frau d ) a n d m ark et seg m e n ta tio n o f cu sto m ers in co n tem p o rary CRM sy stem s. M ore ap p licatio n s in b u sin e ss co n tin u e to b e d e v e lo p e d as th e strength o f clu ste r an alysis is re co g n iz e d an d used .

C luster analysis is a n e x p lo ra to ry data an alysis to o l fo r solv in g classificatio n p ro b lem s. T h e o b je c tiv e is to so rt c a s e s (e .g ., p e o p le , things, e v e n ts) into g ro u p s, o r clusters, s o th a t th e d e g re e o f a sso cia tio n is stro n g am o n g m e m b ers o f th e sa m e clu s­ ter a n d w e a k am o n g m e m b ers o f d ifferen t clusters. E a ch clu ste r d e s crib e s the class to w h ich its m e m b ers b e lo n g . An o b v io u s o n e -d im e n s io n a l e x a m p le o f c lu ste r analysis is to e sta b lish s c o re ran g es into w h ich to a ssig n class g rad es fo r a c o lle g e class. T h is is sim ilar to th e clu ste r analysis p ro b le m th at th e U.S. T re a su ry fa c e d w h e n estab lish in g n e w tax b ra ck ets in th e 1980s. A fictio n al e x a m p le o f clu ste rin g o c c u rs in J . K. R ow lin g ’s H arry P otter b ooks,. T h e Sorting H at d eterm in es to w h ic h H o u se (e .g ., d orm ito ry) to assig n first- y e a r stu d ents at th e H ogw arts S ch o o l. A n o th er e x a m p le involves d eterm in in g h o w to s ea t gu ests a t a w ed d in g . As fa r as d ata m in in g g o e s , th e im p o rta n ce o f clu ste r analysis is that it m ay rev eal asso ciatio n s an d stru ctures in d ata th at w e re n o t previou sly a p p a ren t b u t are s e n sib le an d u sefu l o n c e fo u n d .

Chapter 5 • D ata Mining 2 51

Application Case 5.5 2degrees G ets a 1275 Percen t Bo o st in Churn Id entification 2 d eg rees is N ew Z ealan d ’s fastest grow ing m o b ile te le ­ com m u n ication s co m p a n y - In less th an 3 years, they h ave tran sform ed th e lan d scap e o f N ew Z ealand ’s m o b ile teleco m m u n icatio n s m arket. E ntering very m u ch as th e ch a lle n g e r and battling w ith incu m b ents en tre n ch e d in th e m ark et fo r o v e r 18 years, 2d egrees has w o n o v er 5 8 0 ,0 0 0 cu stom ers and has revenu es o f m o re th an $ 1 0 0 m illion in ju st th eir third y e ar o f o p eration . Last y e ar’s gro w th w as 376 1 p ercent.

S itu a tio n

2d egrees’ inform ation solutions m anager, Peter McCallum, explains that predictive analytics had b e e n o n th e radar at the com p an y fo r som e time. “At 2d egrees th e re are a lot o f analytically aw are p e o ­ ple, from th e CEO dow n. O n c e w e got to the point in ou r business that w e w e re interested in deploying ad vanced predictive analytics techniqu es, w e started to lo o k at w h at w as available in th e m arketplace.” It soo n b e c a m e clear that although o n p ap er th ere w e re sev­ eral options, th e reality w as that th e co st o f deploying the w ell-kn ow n solutions m ade it very difficult to build a business case, particularly given that th e b enefits to the business w ere as y e t unproven.

After ca re fu l e v alu ation , 2 d e g re e s d e cid e d u p o n a su ite o f an aly tics so lu tio n s fro m 11 Ants con sistin g o f C u stom er R e sp o n s e A nalyzer, C u stom er Churn A nalyzer, a n d M o d el B u ild er. “O n e o f th e b e a u tie s o f th e 11 Ants A nalytics so lu tio n w a s th a t it a llo w e d u s to g e t up a n d ru nning q u ick ly an d very e c o n o m i­ cally. W e c o u ld te s t th e w ater an d d eterm in e w h at th e R O I w a s lik e ly to b e fo r p red ictiv e analytics, m ak in g it a lo t e a s ie r to b u ild a b u sin e ss c a s e for future an aly tics p ro je c ts .” P e te r M cC allum said.

W h e n ask ed w h y th ey c h o s e 11 Ants Analytics’ solutions, P eter said, “O n e o f th e b eau ties o f the 11 Ants Analytics solu tion w as that it allow ed us to get up and running q u ickly a n d very econom ically. W e co u ld test th e w a ter an d d eterm ine w h at th e ROI w as likely to b e fo r predictive analytics, m aking it a lot easier to b u ild a b u sin ess c a s e fo r future analytics projects. Y e t w e d id n’t really h av e to s acrifice anything in term s o f functionality— in fact, th e ch u m m odels w e ’ve built hav e p erfo rm ed excep tio n ally w e ll.”

11 Ants A nalytics d ire cto r o f b u sin e ss d e v e lo p ­ m en t, T o m Fuyala, co m m e n ts : “W e are d ed ica te d to g ettin g org an izatio n s u p a n d ru nning w ith p red ictive a n alytics faster, w ith o u t co m p ro m isin g th e quality o f th e results. W ith o th e r so lu tio n s y o u m u st [use] trial an d e rro r th ro u g h m u ltiple alg orithm s m an u ­ ally, b u t w ith 11 Ants A nalytics so lu tio n s th e entire op tim izatio n an d m a n a g e m e n t o f th e algorithm s is au to m ated , allo w in g th o u san d s to b e trialed in a fe w m inutes. T h e b e n e fits o f this a p p ro a ch are e v id e n c e d in th e re al-w o rld re su lts.”

P e te r is a lso im p re sse d b y th e e a s e o f use. “T h e sim plicity w a s a b ig d eal to us. N ot h av in g to h a v e th e statistical k n o w le d g e in -h o u se w as d efi­ n itely a sellin g p o in t. C o m p an y cu ltu re w a s a ls o a b ig fa cto r in o u r d e c is io n m akin g. 11 Ants A nalytics fe lt lik e a g o o d fit. T h e y ’v e b e e n very re sp o n siv e an d h av e b e e n g re a t t o w o rk w ith. T h e tu rnaround o n s o m e o f th e c u sto m re q u e sts w e h av e m a d e has b e e n fa n tastic.”

P e te r a lso likes th e fa ct th at m o d e ls c a n b e b u ilt w ith th e d e s k to p m o d e lin g to o ls an d then d e p lo y e d against th e e n te rp rise cu sto m e r d atab ase w ith 11 Ants P red ictor. “O n c e th e m o d el has b e e n b u ilt, it is e a sy to d e p lo y it in l lA n t s P red icto r to ru n against O ra cle a n d s c o re ou r en tire cu sto m e r b a s e very q u ickly. T h e s p e e d w ith w h ic h 11 Ants P red icto r c a n re -s c o r e hu nd red s o f th o u san d s o f cu sto m e rs is fantastic. W e p re sen tly re -s co re ou r cu sto m e r b a s e m o n th ly , b u t it is s o e a sy th at w e co u ld b e re -s co rin g d aily i f w e w a n te d .”

B e n e f its

2d e g re e s p u t 11 Ants A nalytics so lu tio n s to w o rk q u ick ly w ith very satisfy ing results. T h e initial p ro j­ e c t w a s to fo cu s o n a n a ll-to o -c o m m o n p ro b le m in th e m o b ile te le co m m u n ica tio n s industry; cu sto m e r ch u rn (cu sto m e rs lea v in g ). F o r this th ey d ep lo y e d 11 Ants C u stom er C h u rn Analyzer.

2d e g re e s w a s in te reste d in id entifying cu sto m ­ ers m o st at risk o f ch u rn in g b y analy zing d ata su ch a s tim e o n n e tw o rk , d ay s s in c e last to p -u p , activ ation c h a n n e l, w h e th e r th e cu sto m e r p o rted th e ir n u m b e r o r n o t, cu sto m e r p la n , a n d o u tb o u n d callin g b e h a v ­ iors o v e r th e p re c e d in g 9 0 days.

( C on tin u ed .)

2 5 2 Part III • Predictive Analytics

Application Case 5.5 (Continued) A carefu lly co n tro lle d ex p e rim e n t w as a m o v er

a p eriod o f 3 m o n th s, an d th e results w e re tabulated and an alyzed . T h e results w e re e x ce lle n t: Custom ers id entified a s ch u rn ers b y 11 Ants C u stom er C h u m A n alyzer w e re a g am e-ch an g in g 127 5 p e rc e n t m o re likely to b e ch u rn ers th an cu stom ers c h o s e n at ran­ d om . T h is c a n a lso b e e x p re s s e d as a n in cre a se in lift o f 1 2 .7 5 a t 5 p e rce n t (th e 5% o f th e total p o p u latio n id entified a s m o st lik ely t o ch u rn b y th e m o d el). At 10 p e rce n t, lift w a s 7 .2 8 . O th e r b en e fits in clu d ed th e v arious insights th at 11 Ants C u stom er C h u m A nalyzer p rovided , fo r in stan ce, validating things that sta ff h ad intuitively felt, s u ch as tim e o n n e tw o rk ’s stro n g re la ­ tion sh ip w ith churn, an d highlighting areas w h ere p ro d u ct e n h a n c e m e n t w o u ld b e b en eficial.

A rm ed w ith th e in form ation o f w h ic h cu sto m ­ ers w e re m o st a t risk o f d efectin g , 2 d e g re e s co u ld n o w fo cu s re te n tio n e ffo rts o n th o se id en tified as m o st a t risk , th e re b y g etting su b stantially h ig h er retu rn o n in v e stm e n t o n re te n tio n m ark etin g e x p e n ­ diture. T h e b o tto m lin e is significantly b e tte r results fo r fe w e r d ollars sp ent.

2 d e g r e e s h e a d o f cu sto m ers, M att H o b b s, p ro­ v id es a p e rs p e c tiv e o n w h y this is n o t ju st im p or­ tant to 2 d e g r e e s b u t a lso to th e ir cu sto m ers: “Churn p red ictio n is a v alu ab le to o l fo r cu sto m e r m ark et­ ing a n d w e are e x c ite d a b o u t th e cap ab ilitie s 11 Ants A nalytics p ro v id e s to identify cu sto m e rs w h o display in d icatio n s o f ch u rn in g b eh av io r. T h is is b e n e ficia l to b o th 2 d e g r e e s an d to o u r cu sto m e rs .”

• T o cu sto m e rs g o th e b e n e fits o f id entificatio n ( if you a re n o t lik e ly to chu rn, y o u a re n ot b e in g co n sta n tly a n n o y e d b y m e ssa g es ask in g y o u to stay ) an d ap p ro p riate n ess (cu sto m e rs r e c e iv e o ffe rs th at actu ally are ap p ro p riate to th e ir u sa g e — m inu tes fo r s o m e o n e w h o lik e s to talk, te x ts fo r s o m e o n e w h o lik es to text, e tc.).

• T o 2 d e g r e e s g o th e b e n e fits o f targ etin g (b y id entifying a sm alle r g ro u p o f at-risk cu sto m ­ e rs, re te n tio n o ffe rs c a n b e rich e r b e c a u s e o f th e re d u ctio n in th e n u m b e r o f p e o p le w h o m ay r e c e iv e it b u t n o t n e e d it) and a p p ro p ria te n ess.

B y align in g th e s e b e n e fits fo r b o th 2 d e g re e s a n d the cu sto m e r, th e o u tc o m e s 2 d e g re e s a re e x p e rie n cin g a re vastly im p ro ved .

Q u e s t i o n s f o r D i s c u s s i o n

1. W h at d o e s 2 d e g r e e s do? W h y is it im portant for 2 d e g re e s to accu ra te ly identify churn?

2. W h a t w e re th e ch a lle n g e s, th e p ro p o s e d so lu ­ tion , a n d th e o b ta in e d results?

3. H o w c a n d ata m in in g h e lp in identifying cu s­ to m er churn? H o w d o s o m e co m p a n ie s d o it w ith o u t u sin g data m in in g to o ls a n d te ch n iq u es?

Source: HAntsAnalytics Customer Story’, “1275% Boost in Chum Identification at 2degrees,” Xlantsanalytics.com/ casestudies/2degrees_casestudy.aspx (accessed January 2013)-

C luster analysis results m ay b e u se d to:

• Id en tify a classificatio n s c h e m e (e .g ., ty p es o f cu sto m e rs) • Suggest statistical m o d els to d e scrib e p o p u la tio n s • In d ica te m le s fo r assigning n e w ca s e s to c la s s e s fo r id en tificatio n , targeting, and

d ia g n o stic p u rp o se s • P rov id e m easu res o f d efin ition , size, an d c h a n g e in w h at w e re p rev iou sly b road

co n c e p ts • Find typ ical c a s e s to la b e l an d re p re s e n t cla ss e s • D e c r e a s e the s iz e an d com p le x ity o f the p ro b le m s p a c e fo r oth er data m ining m ethods • Id entify ou tliers in a s p e c ific d om ain (e .g ., rare -e v e n t d ete ctio n )

D E T E R M IN IN G TH E O P T IM A L N U M BER O F C L U S T E R S C lustering algorithm s usually req u ire o n e to s p e cify th e n u m b e r o f clu sters to find . I f this n u m b e r is n o t k n o w n from p rior k n o w le d g e, it sh ou ld b e c h o s e n in s o m e w ay. U nfortunately, th e re is n o op tim al w a y o f calcu latin g w h at this n u m b e r is su p p o s e d to b e . T h e re fo re , sev eral d ifferent

Chapter 5 • D ata Mining 2 5 3

heuristic m e th o d s h a v e b e e n p ro p o se d . T h e fo llo w in g are am o n g th e m o st co m m o n ly

re fe re n ce d o n es;

• L o o k a t th e p e r c e n ta g e o f v a ria n ce e x p la in e d a s a fu n c tio n o f th e n u m b e r o f clu s­ ters; that is, c h o o s e a n u m b e r o f clu ste rs s o th a t ad d in g a n o th e r c lu s te r w o u ld n o t g iv e m u ch b e tte r m o d e lin g o f th e d ata. S p e cifica lly , i f o n e g rap h s th e p e rc e n ta g e o f v a ria n ce e x p la in e d b y th e clu ste rs, th e re is a p o in t a t w h ic h th e m a rg in a l gain w ill d ro p (g iv in g a n a n g le in th e g ra p h ), in d ica tin g th e n u m b e r o f clu s te rs to b e c h o s e n .

• Se t th e n u m b e r o f clu sters to (rc/2)1/2, w h e re n is th e n u m b e r o f d ata points. • U se th e A k aik e In fo rm a tio n C riterion (A IC), w h ich is a m e a su re o f th e g o o d n e s s o f

fit (b a s e d o n th e c o n c e p t o f e n tro p y ) to d eterm in e th e n u m b e r o f clusters. • U s e B a y e s ia n in form ation crite rio n (B IC ), w h ich is a m o d e l-s e le c tio n criterio n

(b a s e d o n m ax im u m lik e lih o o d estim atio n ) to d eterm in e th e n u m b e r o f clusters.

A N A L Y S IS M E T H O D S Cluster an aly sis m ay b e b a s e d o n o n e o r m o re o f th e fo llo w in g g e n eral m ethod s:

• Statistical m e th o d s (in clu d in g b o th h ierarch ical an d n o n h ie ra rch ica l), s u ch as &-means, &-m odes, an d s o o n

• N eural n e tw o rk s (w ith th e arch ite ctu re ca lle d self-org an izin g m ap , o r SO M ) • Fuzzy lo g ic (e .g ., fu zzy c-m ea n s algorithm ) • G e n e tic algorithm s

E a ch o f th e s e m e th o d s gen erally w o rk s w ith o n e o f tw o g e n era l m eth o d cla sse s:

• D ivisive. W ith divisive cla sse s, all item s start in o n e clu ste r a n d are b r o k e n apart. • Ag g lo m era tiv e. W ith ag g lo m erativ e cla sse s, all item s start in individ ual clusters,

a n d th e clu ste rs are jo in e d to g eth er.

M o st clu ste r an aly sis m e th o d s involve th e u se o f a d i s t a n c e m e a s u r e t o calcu late th e clo s e n e s s b e tw e e n pairs o f item s. P o p u la r d istan ce m easu res in clu d e E u clid ian dis- i^nce (th e ord in ary d ista n ce b e tw e e n tw o p o in ts th a t o n e w o u ld m e asu re w ith a ru ler) and M anh attan d is ta n ce (a ls o calle d th e re ctilin e a r d istan ce , o r ta x ica b d istan ce , b e tw e e n TA-o p o in ts). O fte n , th e y a re b a s e d o n true d istan ce s that a re m easu red , b u t this n e e d not b e so , as is typ ically th e c a s e in IS d ev elo p m en t. W e ig h ted av e rag e s m ay b e u se d to establish th e s e d istan ce s. F o r e x a m p le , in a n IS d ev e lo p m e n t p ro je ct, individual m o d u le s o f th e sy stem m ay b e related b y th e sim ilarity b e tw e e n th e ir inputs, outputs, p ro c e s s e s, i n d th e s p e cific d ata u sed . T h e s e facto rs are th e n a g g reg ated , p airw ise b y item , into a single d istan ce m e asu re .

< -M EA N S C L U S T E R IN G A LG O R IT H M T h e &-means algorithm (w h e r e k stand s fo r th e pre­ determ ined n u m b e r o f clu ste rs) is argu ably th e m o st r e fe re n c e d clu sterin g algorithm , k has its ro o ts in trad itio nal statistical analysis. As th e n a m e im p lies, th e a lg o rith m assigns e a c h data p o in t (cu sto m e r, e v e n t, o b je c t, e tc .) to th e clu ste r w h o s e c e n te r (a ls o calle d cen troid ) is th e n e a re st. T h e c e n te r is ca lcu la ted as th e av e ra g e o f all th e p o in ts in th e duster; th a t is, its co o rd in a te s are the arithm etic m e a n fo r e a c h d im en sio n s e p a ra te ly o v er £1 th e p o in ts in th e cluster. T h e algorith m step s are listed b e lo w a n d sh o w n g rap h ically

s i Figure 5.11:

I n i t i a l i z a t i o n s t e p : C h o o s e th e n u m b e r o f clu sters ( i.e ., th e v alu e o f k). S tep X: R and om ly g e n e ra te k ra n d o m p o in ts as initial clu ste r ce n te rs. S tep 2 : A ssign e a c h p o in t to th e n e a re st clu ste r ce n te r.

Step 3 : R e co m p u te th e n e w clu ste r ce n te rs.

Part ITT • Predictive Analytics

3

FIGURE 5.11 A Graphical Illustration of the Steps in k-Means Algorithm.

R epetition step: R e p e a t step s 2 an d 3 until s o m e c o n v e rg e n c e criterio n is m e t (usually th a t th e assig n m en t o f p o in ts to clu sters b e c o m e s stab le).

Association Rule Mining A sso ciatio n rule m in in g (a ls o k n o w n as a ffin ity an aly sis o r m a rket-b asketan aly sis') is a p o p u la r data m in in g m eth o d that is co m m o n ly u s e d a s a n e x a m p le to e x p la in w h a t data m in in g is and w h at it c a n d o to a te ch n o lo g ica lly less savvy a u d ie n ce M ost o f you might h av e h e a rd th e fam ou s (o r in fam o u s, d e p e n d in g o n h o w y o u lo o k a t it) re latio n s ip d isco v e red b e tw e e n th e sa le s o f b e e r and d iap e rs a t g ro cery stores. As th e story g o e s , a large su p erm ark et ch a in (m a y b e W alm art, m a y b e n o t; th e re is n o c o n s e n s u s o n w h i c h su p erm ark et ch a in it w a s ) d id a n analysis o f c u sto m e rs’ b u y in g h ab its and fo u n d a stat cally sig n ifican t co rre la tio n b e tw e e n p u rc h a s e s o f b e e r a n d p u rch a ses o f d iap ers. It was th e o riz e d th a t th e re a s o n fo r this w a s that fath ers (p resu m a b ly y o u n g m e n ) w e re stop p in g o f f a t th e su p erm a rk et to b u y d iap ers for th e ir b a b ie s (e s p e c ia lly o n T h u rsd ay s), an d sin ce th ey c o u ld n o lo n g e r g o to th e sp orts b a r as o fte n , w o u ld buy b e e r as w e ll. As a result o this find ing, th e su p e rm ark et ch a in is a lle g e d t o h av e p la c e d th e d iap ers n e x t to th e b e e r,

resultin g in in cre a se d sa le s o f b o th . . I n e ss e n c e , a s s o cia tio n rule m ining aim s to find in terestin g relatio n sh ip s (affin ities)

b e tw e e n v ariab les (ite m s) in larg e d atab ase s. B e c a u s e o f its su cce ssfu l a p p lica tio n to retail b u sin e ss p ro b le m s, it is co m m o n ly ca lle d m arket-b asket an aly sis. T h e m am idea in m a rk e t-b a sk e t analysis is to id entify s tro n g relatio n sh ip s am o n g A ffe re n t p rodu cts (o r serv ices) that are u su ally p u rch a se d to g e th e r (s h o w up m th e sa m e b a s k e t t ° 8 e * e r . e ith er a p h y sical b a s k e t at a g ro ce ry sto re o r a virtual b a sk e t at a n e -c o m m e r c e W site). F o r e x a m p le , 65 p e rce n t o f th o se w h o b u y c o m p re h e n siv e a u to m o b ile in su ra n ce also b u y h e alth in su ra n ce ; 8 0 p e rce n t o f th o s e w h o b u y b o o k s o n lin e a lso b u y m usi online- 6 0 p e rc e n t o f th o s e w h o h av e high b lo o d p re ssu re an d are o v erw e ig h t h a v e high ch o le s te r o l; a n d 7 0 p e rc e n t o f th e cu stom ers: w h o b u y la p to p co m p u te r a n d virus p ro te c-

tio n softw are a lso b u y e x te n d e d serv ice p lan . , T h e input to m a rk e t-b a sk e t analysis is s im p le p o in t-o f-sa le tran sactio n data, w h e re

n u m b e r o f p ro d u cts and/or s erv ices p u rch a se d to g e th e r (ju st like th e c o n te n t o f a p u rch ase re c e ip t) are ta b u lated u n d e r a sin gle tra n sa ctio n in stan ce. T h e o u tco m e o f th e a™ y * ‘s in valu ab le in form ation th at ca n b e u s e d to b e tte r u n d erstan d cu sto m e r-p u rch a se b eh a v io r in o rd e r to m ax im ize th e profit fro m b u sin ess tran saction s. A b u sin e ss ca n tak e ad vantage o f s u ch k n o w le d g e b y (1 ) putting th e item s n e x t to e a c h o th e r to m a k e it m o re c o n v - n ien t fo r th e cu sto m ers to p ick th e m up to g e th e r a n d n o t fo rg et to b u y o n e w h en b u y g

Chapter 5 • D ata Mining 2 5 5

ih e oth ers (in cre a s in g s a le s v o lu m e ); ( 2 ) p ro m o tin g th e item s as a p a c k a g e (d o n o t put o n e o n sa le i f th e o th e r(s ) are o n s a le ); a n d ( 3 ) p la cin g th e m apart fro m e a c h o th e r s o that the cu sto m e r h a s to w a lk th e aisles to s e a rch fo r it, and b y d o in g s o p o ten tia lly s e e in g and b u yin g o th e r item s.

A pplications o f m arket-basket analysis inclu de cross-m arketing, cross-selling, store design, catalog design, e -co m m erce site design, optim ization o f on lin e advertising, product pricing, an d sales/prom otion configuration. In e sse n ce , m arket-basket analysis helps businesses infer cu sto m e r n e ed s and p re fe re n ce s from their pu rchase patterns. O utside the business realm , associatio n rules are successfully used to d iscov er relationships b etw een symptoms an d illnesses, d iagnosis an d patient characteristics an d treatm ents (w h ich can b e u sed in m ed ical D SS), an d g en es an d their fu nctions (w h ich c a n b e u sed in g en om ics projects), am o n g oth ers. H ere are a few co m m o n areas an d uses fo r association rule mining:

• S ales tran saction s: C o m b in ation s o f retail p ro d u cts p u rch a sed to g eth e r c a n b e u sed to im p ro ve p ro d u ct p la c e m e n t o n th e sa le s flo o r (p la c in g p ro d u cts th at g o to g eth e r in c lo s e p ro xim ity ) and p ro m o tio n a l p ricin g o f p ro d u cts (n o t h av in g p ro m o tio n o n b o th p ro d u cts th at are o fte n p u rch ased to g eth er).

• C redit c a r d tran saction s: Item s p u rch a sed w ith a cre d it card pro v id e insig ht into o th er p ro d u cts th e cu sto m e r is lik e ly to p u rch ase o r frau d ulen t u s e o f cred it card nu m ber.

• B a n k in g serv ices: T h e se q u e n tia l pattern s o f s erv ices u s e d b y cu sto m e rs (c h e c k in g a c c o u n t fo llo w e d b y sav in g a c c o u n t) c a n b e u se d to id en tify o th e r s e rv ice s they m ay b e in te reste d in (in v e stm e n t a cco u n t).

• In su ra n c e serv ice p rod u cts: B u n d le s o f in su ra n ce p ro d u cts b o u g h t b y cu stom ers (c a r in su ra n ce fo llo w e d b y h o m e in su ra n ce ) c a n b e u se d to p ro p o s e ad ditional in su ra n ce p ro d u cts (life in su ra n ce ); or, u n u su al c o m b in a tio n s o f in su ra n ce claim s ca n b e a sig n o f fraud.

• T elecom m u n ication services: C o m m o n ly p u rch a sed g ro u p s o f o p tio n s (e .g ., call w aiting, c a lle r ID , th ree-w ay callin g , e tc .) h e lp b e tte r stru cture p ro d u ct b u n d le s to m ax im ize re v e n u e ; th e sa m e is a lso a p p lica b le to m u lti-ch an n el te le c o m providers w ith p h o n e , T V , and In te rn e t serv ice offerings.

• M ed ical record s: C ertain co m b in atio n s o f co n d itio n s c a n in d icate in c re a se d risk o f various co m p lica tio n s; or, certain treatm ent p ro ced u res at ce rta in m e d ica l facilities c a n b e tied to ce rta in ty p e s o f in fectio n .

A g o o d q u e stio n to ask w ith re s p e c t to th e patterns/ relationship s th at a sso cia tio n rule m ining c a n d is co v e r is “A re all a sso cia tio n ru les in terestin g a n d useful?” In ord e r to n e w e r s u ch a q u e s tio n , a s s o cia tio n ru le m ining u s e s tw o co m m o n m etrics: s u p p o r t , and c o n f id e n c e an d l i f t . B e f o r e d efin in g th e s e term s, let’s g e t a little te ch n ica l b y sh o w in g what a n a s s o cia tio n rule lo o k s like:

X = > Y[Supp(% ), C on f(% )]

{L aptop C o m p u ter, Antivirus Softw are] => {E x ten d e d Se rv ice P lan} [30% , 70%]

Here, X (p ro d u cts and/or serv ice; ca lle d th e left-h a n d sid e, LHS, o r th e a n te c e d e n t) is ^'Sociated w ith Y (products and/or serv ice; ca lle d th e rig h t-h a n d sid e, RHS, o r co n se­ quen t). S is th e su p p ort, a n d C is th e c o n fid e n c e fo r this p articu lar ru le. H e re a re the fim ple fo rm u las fo r Supp, C o n f an d Lift.

n u m b er o f baskets th a t co n ta in s both X a n d Y Support = S u p p o c m SO = ------------------ to ta l n u m b er o f baskets

2 5 6 Part III • Predictive Analytics

Supp(X => Y) C o n fid en ce = C o n fix =# Y) - ' su p p iM

■SI V > V>

cohjxx =» y) __ sao u soc=>y)_ Lifttx=> 1 0 = confQt^M sao-sa') m is e r y

SQO

T h e su p p ort * § o f a c o lle c tio n o f p ro d u cts is th e m e asu re o f h o w o fte n th e s e products and/or serv ices ( i e . LHS + RHS = L aptop C o m p u ter, Antivirus Softw are, and E xten d e d Serv ice P la n ) a p p e a r to g e th e r in th e sam e tran sactio n , that is, th e p ro p o rtio n o f trans­ a c t s i r X dPata s e t that c o n ta in all o f th e p ro d u cts and/or serv ices m e n tio n e d m a s p e cific rule In this e x a m p le , 30 p e rce n t o f all tran sactio n s in th e h y p o th e tical store

d atab ase h a d all th ree p ro d u cts p re s e n t in a sin g le sales is th e m e a su re o f h o w o fte n th e p ro d u cts and/or s erv ices o n th e RHS (c o n s e q u e n t; g to g e th e r w ith the p ro d u cts and/or serv ices o n th e LHS (a n te c e d e n t), that is, th e p ro p or- tio n o f tran sactio n s that in clu d e LHS w h ile a lso in clu d in g th e RHS. In o th e r w o rd s it th e co n d itio n al p ro b ab ility o f find ing th e RHS o f th e ru le p re s e n t m tran sactio n s w h ere th e LHS o f th e rule alread y exists. T h e lift v alu e o f an asso cia tio n ru le is th e ratic. o f the c o r X S Of th e ru le a n d V e x p e c te d c o n fid e n c e o f th e m le . T h e e ^ e d ^ o r r f id n e e . o f a rule is d efin e d as th e p ro d u ct o f the su p p ort v a lu e s o f th e LHS a n d th e .

b Y t h s " ^ S S a r e a v a ila b le fo r d is c o v e rin g a s s o c ia tio n ru le s. S o m e w e ll-

k n o w n alg o rith m s in c lu d e A p rio ri, E c la t, a n d E P - G r o w t h .T h e s e a g h a lf th e io b w h ic h is t o id e n tify th e S e q u e n t ite ffls e ts m th e da a b a s e . O n c e th e fr e q u e n t ite m se ts a re id e n tifie d , th e y n e e d to b e c o n v e r te d in to ru le s w ith a n te c e d ­ e n t a n d c o n s e q u e n t p a rts. D e te r m in a tio n o f th e ru le s fro m f r e q u e n t .te m s e fe t a stra ig h tfo rw a rd m a tch in g p r o c e s s , b u t th e p r « e s s m ay b e tm e -c o ffs u m in & w ith d a r g tr a n s a c tio n d a ta b a s e s . E v e n th o u g h th e re c a n b e m a n y item s o n each, s e c t.o n , o f t h e ru le in p r a c tic e th e c o n s e q u e n t p a rt u s u a lly C ofltain s a s in g le item . In th e fo lio ■ g s e c t i o n , o n e o f th e m o st p o p u la r a lg o rith m s fo r id e n tific a tio n o f fr e q u e n t ite m se ts IS

e x p la in e d .

A P R IO R I A L G O R IT H M T h e A p r i o r i a l g o r i t h m is th e m o st c o m m o n ly u s e d a lg o rith m to d is c o v e r a s s o c ia t i o n r u le s . G i v e n a s e t o f i t e m s e t s ( e . g . , s e ts e a c h listin g in d ivid u al item s p u rc h a s e d ), th e a lg o rith m atte m p ts to fin d s u b s e ts that c o m m o n to at le a s t a m in im u m n u m b e r o f th e ite m se ts ( i .e ., c o m p lie s w ith a — m s u p p o r t ) . A p rio ri u s e s a b o t t o m - u p a p p r o a c h , w h e r e f r e q u e n t s u b s e t s a re■ e x t e n o n e ite m a t a tim e (a m e th o d k n o w n as c a n d id a te g en era tio n . fr e q u e n t s u b s e ts in c re a s e s fro m o n e -ite m s u b s e ts to tw o -ite m su b se ts, subsets e tc ) a n d groups o f candidates at e a c h leyel am WSted against th e d ata for m in im u m su p p o rt. T h e alg o rith m te rm in a te s w h e n n o fu rth e r s u c c e s s fu l e x te n s io

“ e f° A s 1 ,n illu strativ e e x a m p le , c o n s id e r th e fo llo w in g . A g r o < w tra n s a c tio n s b y SK U (s to c k -k e e p in g u n it) a n d th u s k n o w s w h ic h item s a i e typ i y p u rc h a s e d to g e th e r . T h e d a ta b a s e o f tra n s a c tio n s , a lo n g w ith th e s u b s e q u e n t s te p s i id e n tify in g th e fre q u e n t ite m se ts , , s h o w n in F ig u re 5 T 2 . E a c h SKU m th e — e - tio n d a ta b a s e c o r r e s p o n d s t o a p ro d u ct, s u c h a s 1 = b u tte r, 2 .= b r e a d 3 - w ater,^ a n d s o o n . T h e first s te p in A p rio ri is to c o u n t u p th e fre q u e n c ie , . ., PP o f e a c h item (o n e - ite m ite m se ts ). F o r th is o v e r ly sim p lifie d e x a m p le let u s s e t J .h m in im u m su p p o rt to 3 (o r 5 0 % ; m e a n in g a n ite m s e t is c o n s id e r e d to b e a fre q u e n t

Chapter 5 * Data Mining 2 5 7

Raw Transactio n D ata One-Item Ite m se ts Two-Item Ite m se ts Fhree-ltem Ite m sets

Transaction No

SKUs (Item No) —

Itemset (SKUs)

Support — Itemset (SKUs)

Support — itemset (SKUs)

Support

1001 1, 2, 3, 4 1 3 1, 2 3 1, 2, 4 3

1002 2, 3, 4 2 6 1, 3 2 2, 3, 4 3

1003 2, 3 3 4 1, 4 3

1004 1, 2, 4 4 5 2, 3 4

1005 1, 2, 3, 4 2, 4 5

1006 2, 4 3, 4 3

FIGURE 5.12 Identification of Frequent Itemsets in Apriori Algorithm.

ite m se t i f it s h o w s u p in a t le a s t 3 o u t o f 6 tra n sa c tio n s in th e d a ta b a s e ). B e c a u s e all o f th e o n e -ite m ite m s e ts h a v e a t le a s t 3 in th e s u p p o rt c o lu m n , th e y a re a ll c o n s id e r e d fre q u e n t ite m se ts . H o w e v e r, h a d a n y o f th e o n e -ite m ite m se ts n o t b e e n fr e q u e n t, th e y w o u ld n o t h a v e b e e n in c lu d e d a s a p o s s ib le m e m b e r o f p o s s ib le tw o -ite m p airs. In this w a y , A p rio ri p r u n e s th e tre e o f a ll p o s s ib le ite m se ts . As F ig u re 5 .1 2 s h o w s , u s in g o n e -ite m ite m s e ts , a ll p o s s ib le tw o -ite m ite m se ts a re g e n e r a te d , a n d th e tra n s a c tio n d a ta b a s e is u s e d t o c a lc u la te th e ir s u p p o rt v a lu e s . B e c a u s e th e tw o -ite m ite m s e t {1, 3 h a s a s u p p o rt le s s th a n 3 , it s h o u ld n o t b e in c lu d e d in th e fr e q u e n t ite m s e ts th a t w ill b e u s e d to g e n e r a te th e n e x t-le v e l ite m se ts (th r e e -ite m ite m se ts ). T h e a lg o rith m s e e m s d e c e iv in g ly s im p le , b u t o n ly fo r sm all d ata s e ts. In m u c h la rg e r d ata s e ts , e s p e c ia lly th o se w ith h u g e a m o u n ts o f item s p re s e n t in lo w q u a n titie s a n d sm all a m o u n ts o item s p re s e n t in b ig q u a n titie s , th e s e a r c h a n d c a lc u la tio n b e c o m e a co m p u ta tio n a lly

in te n siv e p ro c e s s .

SECTION 5 .5 REVIEW QUESTIONS

1 . Id entify a t le a s t th ree o f th e m ain d ata m in in g m eth o d s. 2 . In th e final s te p o f d ata p ro cessin g , h o w w o u ld data red u ction facilitate d e cisio n

analysis? G iv e a n e x a m p le . 3 . List an d b rie fly d e fin e a t le a s t tw o classificatio n te ch n iq u es. 4 . W h at a re s o m e o f th e criteria fo r co m p a rin g an d s e le ctin g th e b e s t classificatio n

tech n iq u e? 5 . B riefly d e s c rib e th e g e n e ra l algorith m u s e d in d e c is io n trees.

6. D e fin e G in i in d ex . W h at d o e s it m easure? 7 . G ive e x a m p le s o f situ ation s in w h ic h d u s te r analysis w o u ld b e a n ap p ro p riate data

m in in g te ch n iq u e . 8 . W h at is th e m a jo r d iffe re n ce b e tw e e n clu ste r analysis an d classification?

9 . W h at are s o m e o f th e m e th o d s fo r clu ste r analysis? 1 0 . G ive e x a m p le s o f situ ation s in w h ic h a s s o cia tio n w o u ld b e a n a p p ro p ria te data m in­

ing te c h n iq u e .

2 5 8 Part III • Predictive Analytics

5.6 D A T A M IN IN G S O F T W A R E TO OLS M an y so ftw a re v e n d o rs p ro v id e p o w e rfu l d ata m in in g to o ls . E x a m p le s o f th e s e v e n ­ d o rs in clu d e IB M (IB M SP SS M o d eler, fo rm e rly k n o w n a s SP SS PASW M o d e le r and C le m e n tin e ), SAS (E n te rp rise M in er), S tatSo ft (S ta tistica D ata M iner), KXEN (Infinite In s ig h t), Salfo rd (CART, MARS, T re e N e t, R a n d o m F o re st), A n g o ss (K n o w le d g e S T U D IO . K n o w le d g e S e e k e r), an d M e g ap u ter (P o ly A n aly st). N o tice a b ly b u t n o t surprisingly, th e m o st p o p u la r d ata m in in g to o ls a re d e v e lo p e d b y th e w e ll-e s ta b lis h e d statistical so ftw a re c o m p a n ie s (SP SS, SAS, a n d StatSo ft)— larg ely b e c a u s e statistics is th e fo u n d a­ tio n o f d ata m in in g , an d th e s e c o m p a n ie s h a v e th e m e a n s to c o s t-e ffe c tiv e ly d ev elo p th e m in to fu ll-s ca le d ata m in in g sy stem s. M o st o f th e b u s in e s s in te llig e n c e to o l vend ors (e .g ., IB M C o g n o s, O ra cle H y p e rio n , SAP B u s in e s s O b je c ts , M icroStrategy , T e ra d a ta , an M icro so ft) a ls o h a v e s o m e le v e l o f d ata m in in g c a p a b ilitie s in te g ra te d in to th e ir softw are o ffe rin g s. T h e s e B I to o ls a re still p rim arily fo c u s e d o n m u ltid im en sio n a l m o d e lin g and d ata v isu a lizatio n a n d a re n o t c o n s id e re d to b e d irect co m p e tito rs o f th e d ata m ining

to o l v e n d o rs. In a d d itio n to th e s e c o m m e rc ia l to o ls , se v e ra l o p e n s o u rc e and /or tre e data

m in in g s o ftw a re to o ls a re a v a ila b le o n lin e . P r o b a b ly th e m o st p o p u la r fre e (a n d o p e n s o u r c e ) d ata m in in g to o l is W eka, w h ic h is d e v e lo p e d b y a n u m b e r o f re s e a r c h e rs fro m th e U n iv ersity o f W a ik a to in N ew Z e a la n d (th e to o l ca n b e d o w n lo a d e d from cs .w a ik a to .a c.n z /m l/w e k a ). W e k a in clu d e s a la rg e n u m b e r o f alg o rith m s fo r d iffer­ e n t d ata m in in g ta s k s a n d h a s a n in tu itiv e u s e r in te rfa c e . A n o th e r re c e n tly r e le a s e d , free (f o r n o n c o m m e rc ia l u s e ) d ata m in in g to o l is R apidM iner (d e v e lo p e d b y R ap id -I; it can b e d o w n lo a d e d fro m rap id -i.com ). Its g ra p h ic a lly e n h a n c e d u s e r in te rfa c e , e m p lo y ­ m e n t o f a ra th er la rg e n u m b e r o f alg o rith m s, a n d in c o rp o ra tio n o f a v arie ty o f data v is u a liz a tio n fe a tu re s s e t it ap art fro m th e re s t o f th e fre e to o ls . A n o th e r fr e e an d o p e n so u rc e d ata m in in g to o l w ith a n a p p e a lin g g ra p h ic a l u s e r in te rfa ce is KN IM E (w h ich c a n b e d o w n lo a d e d fro m k n im e.o rg ). T h e m a in d iffe re n c e b e tw e e n c o m m e rc ia l to o ls, s u ch a s E n te rp ris e M in er, IB M SP SS M o d e le r, a n d S tatistica, a n d fre e to o ls , s u ch as W e k a , R apid M iner, an d KNIM E, is c o m p u ta tio n a l e ffic ie n c y . T h e s a m e d ata m ining ta s k in v o lv in g a la rg e d ata s e t m ay ta k e a w h o le lo t lo n g e r to c o m p le te w ith th e free so ftw a re , an d fo r s o m e a lg o rith m s m ay n o t e v e n c o m p le te ( i .e ., cra sh in g d u e to th e in e ffic ie n t u s e o f c o m p u te r m e m o ry ). T a b le 5 .3 lists a fe w o f th e m a jo r p ro d u cts an d

th e ir W e b sites. , A s u ite o f b u s in e s s in te llig e n c e c a p a b ilitie s th a t h a s b e c o m e in c re a s in g ly m o re

p o p u la r fo r d ata m in in g p r o je c ts is M icro so ft SQL Server, w h e r e d ata an d th e m o d e ls a re s to r e d in th e s a m e re la tio n a l d a ta b a s e e n v iro n m e n t, m a k in g m o d e l m an- a g e m e n t a c o n s id e ra b ly e a s ie r task . T h e M icro so ft E n te rp rise C on sortiu m serv e s as th e w o rld w id e s o u r c e fo r a c c e s s to M ic ro s o ft’s SQ L S e rv e r 2 0 1 2 s o ftw a re s u ite fo r a c a d e m ic p u rp o s e s — te a c h in g a n d re s e a r c h . T h e co n s o rtiu m h a s b e e n e s ta b lis h e d to e n a b le u n iv e rs itie s a ro u n d th e w o rld to a c c e s s e n te rp ris e te c h n o lo g y w ith o u t h av in g to m a in ta in th e n e c e s s a r y h a rd w a re a n d s o ftw a re o n th e ir o w n c a m p u s . T h e c o n s o r ­ tiu m p ro v id e s a w id e ra n g e o f b u s in e s s in te llig e n c e d e v e lo p m e n t to o ls ( e .g ., data m in in g , c u b e b u ild in g , b u s in e s s re p o r tin g ) a s w e ll a s a n u m b e r o f la rg e , re a lis tic data se ts fro m S a m ’s C lu b , D illa rd ’s, a n d T y s o n F o o d s . T h e M icro so ft E n te rp ris e C o n so rtiu m is fre e o f c h a rg e a n d c a n o n ly b e u s e d fo r a c a d e m ic p u rp o s e s . T h e Sam M. W a lto n C o lle g e o f B u s in e s s a t th e U n iv e rsity o f A rk a n sa s h o s ts th e e n te rp ris e s y s te m a n d a llo w s c o n s o rtiu m m e m b e r s a n d th e ir s tu d e n ts to a c c e s s th e s e r e s o u r c e s b y u s in g a s im p le re m o te d e s k to p c o n n e c tio n . T h e d e ta ils a b o u t b e c o m in g a p a rt o f th e c o n s o r ­ tiu m a lo n g w ith e a s y -to -fo llo w tu to rials a n d e x a m p le s c a n b e fo u n d a t e n te rp rise . w alto n co lle g e .u a rk .e d u .

T A B L E 5 . 3 S e le c te d D a t a M in in g S o ftw a re

Chapter 5 * D ata Mining 2 5 9

Product Name W e b Site (URL)

IBM SPSS Modeler ibm.com/software/analytics/spss/products/modeler/

SAS Enterprise Miner sas.com/technologies/bi/analytics/index.html

Statistica statsoft.com/products/dataminer.htm

Intelligent Miner ibm .com /software/data/im iner

PolyAnalyst m egaputer.com /polyanalyst.php

C A R T MARS, TreeNet, RandomForest salford-systems.com

Insightful Miner insightful.com

XLMiner xlm iner.net

KXEN (Knowledge extraction ENgines) kxen.com

GhostMiner fqs.pl/ghostm iner

Microsoft SQL Server Data Mining microsoft.com/sqlserver/2012/data-mining.aspx

Knowledge Miner know ledgem iner.net

Teradata Warehouse Miner ncr.com /products/software/teradata_m ining.htm

Oracle Data Mining (ODM) otn.oracle.com/products/bi/9idmining.html

Fair Isaac Business Science fairisaac.com/edm

DeltaMaster bissantz.de

iData Analyzer infoacum en.com

Orange Data Mining Tool ailab.si/orange

Zementis Predictive Analytics zementis.com

I n M a y 2 0 1 2 , k d n u ggets.com c o n d u c t e d th e th ir te e n th a n n u a l S o f tw a r e P o ll o n th e fo llo w in g q u e s t io n : “W h a t A n a ly tic s , D a ta M in in g , a n d B i g D a ta s o ftw a r e h a v e y o u u s e d in t h e p a s t 1 2 m o n th s f o r a r e a l p r o je c t ( n o t ju s t e v a lu a tio n )? ” H e r e a r e s o m e o f t h e

in te r e s tin g fin d in g s th a t c a m e o u t o f t h e p o ll:

• F o r t h e firs t tim e ( i n t h e la st 1 3 y e a r s o f p o l li n g o n th e s a m e q u e s t io n ) , t h e n u m b e r o f u s e r s o f fr e e / o p e n s o u r c e s o ftw a r e e x c e e d e d t h e n u m b e r o f u s e r s o f c o m m e r c ia l

s o ftw a r e . • A m o n g v o t e r s 2 8 p e r c e n t u s e d c o m m e r c ia l s o ftw a r e b u t n o t f r e e s o ftw a r e , 3 0 p e r c e n t

u s e d f r e e s o f t w a r e b u t n o t c o m m e r c ia l, a n d 4 1 p e r c e n t u s e d b o th . • T h e u s a g e o f B ig D a ta t o o ls g r e w fiv e fo ld : 1 5 p e r c e n t u s e d t h e m in 2 0 1 2 , v e r s u s

a b o u t 3 p e r c e n t in 2 0 1 1 . • R , R a p id M in e r , a n d K N IM E a r e t h e m o s t p o p u la r f r e e / o p e n s o u r c e t o o ls , w h ile

S ta tS o ft’s S ta tis tic a , SA S’s E n te r p r is e M in e r, a n d IB M ’s S P S S M o d e le r a r e t h e m o s t

p o p u la r d a ta m in in g to o ls . • A m o n g t h o s e w h o w r o t e t h e ir o w n a n a ly tic s c o d e in lo w e r - le v e l la n g u a g e s , R , SQ L,

J a v a , a n d P y t h o n w e r e t h e m o s t p o p u la r .

2 6 0 Part III • Predictive Analytics

T o r e d u c e b ia s t h r o u g h m u ltip le v o tin g , in th is p o ll k d n u ggets.com u s e d e -m a il v e r ific a tio n , w h ic h r e d u c e d t h e to ta l n u m b e r o f v o t e s c o m p a r e d to 2 0 1 1 , b u t m a d e re s u lts m o r e r e p r e s e n ta tiv e . T h e r e s u lts f o r d a ta m in in g s o ftw a r e t o o ls a r e s h o w n in F ig u r e 5 .1 3 , w h i le t h e r e s u lts f o r B i g D a ta s o ftw a r e t o o ls u s e d , a n d th e p la tfo rm / la n g u a g e u s e d fo r y o u r o w n c o d e , is s h o w n in F ig u r e 5 .1 4 .

A p p lic a tio n C a s e 5 .6 is a b o u t a r e s e a r c h s tu d y w h e r e a n u m b e r o f s o ftw a r e t o o ls a n d d a ta m in in g t e c h n i q u e s a r e u s e d to b u ild m o d e ls t o p r e d ic t fin a n c ia l s u c c e s s ( b o x - o f f i c e r e c e ip t s ) o f H o lly w o o d m o v i e s w h il e t h e y a r e n o t h i n g m o r e t h a n id e a s .

R Excel

Rapid-I RapidMinen K N IM E

W e k a /P e n ta ho

StatSoft Statistica S A S

Rapid-I RapidAnalytics M A T L A B

IBM S P S S Statistics

IBM S P S S M odeler

S A S Enterprise M iner Orange

M icrosoft S Q L Serve r O ther free software

TIB C O S p o tfire / S + / M in e r

Tableau Oracle Data M iner

O ther com m ercial software J M P

M athematica M in e r3 D

IB M Cognos

Stata

Zem entis KXEN

Bayesia

C 4 .5 / C 5 .0 / S e e 5 Revolution Computing

Salford S P M / C A R T / M A R S / T r e e N e t / R F X L S T A T

S A P (BusinessO bjects/Sybase /H a na] Angoss

Rapidlnsight/Veera

Terad ata M iner 11 A n ts Analytics

W o rd S ta t

Predixion Software

FIGURE 5.13 Popular Data Mining Software Tools (Poll Results). Source: Used w ith permission of kdnuggets.com.

Chapter 5 * D ata Mining 2 61

3 6

Big Data software tools/platform s used for your analytics projects

Apache H a d o o p / H b a s e /P ig /H ive

A m a zo n W e b Services (A W S )

N o S Q L databases

O th e r Big Data software

O the r Hadoop-based tools

1 1 3 6 7

3 3

21

10

0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0

Platform s/languages used for your own analytics code

FIGURE 5.14 Popular Big Data Software Tools and Platforms/Languages Used. Source: Results o f a poll conducted by kdnuggets.com.

Application Case 5.6 D ata M ining G oes to H ollyw ood: Predicting Financial Success o f M o vies

P r e d ic tin g b o x - o f f i c e r e c e ip t s ( i .e ., fin a n c ia l s u c c e s s ) o f a p a r tic u la r m o t i o n p ic tu r e is a n in te r e s tin g a n d c h a l le n g in g p r o b l e m . A c c o r d in g t o s o m e d o m a in e x p e r ts , t h e m o v i e in d u s try is t h e “la n d o f h u n c h e s a n d w ild g u e s s e s ” d u e to t h e d iffic u lty a s s o c ia t e d w ith f o r e c a s t in g p r o d u c t d e m a n d , m a k in g th e m o v ie b u s in e s s in H o lly w o o d a risk y e n d e a v o r . I n s u p p o r t o f s u c h o b s e r v a tio n s , J a c k V a le n ti ( t h e lo n g tim e p r e s id e n t a n d C E O o f t h e M o tio n P ic tu r e A s s o c ia tio n o f A m e r ic a ) o n c e m e n t io n e d th a t “ . . . n o o n e c a n te ll y o u h o w a m o v ie is g o i n g to d o in th e m a r k e t p l a c e . .. n o t u n til t h e film o p e n s in d a r k e n e d

th e a tr e a n d s p a r k s fly u p b e t w e e n t h e s c r e e n a n d t h e a u d i e n c e ." E n te r ta in m e n t in d u s try tr a d e jo u r n a ls a n d m a g a z in e s h a v e b e e n fu ll o f e x a m p l e s , s ta te ­ m e n ts , a n d e x p e r ie n c e s th a t s u p p o r t s u c h a c la im .

L ik e m a n y o t h e r re s e a r c h e r s w h o h a v e a tte m p te d to s h e d lig h t o n th is c h a lle n g in g re a l-w o rld p ro b le m , R a m e s h S h ard a a n d D u rs u n D e l e n h a v e b e e n e x p lo r ­ in g t h e u s e o f d a ta m in in g t o p r e d ic t th e fin a n c ia l p e r ­ f o r m a n c e o f a m o tio n p ic a i r e a t th e b o x o ffic e b e f o r e it e v e n e n te r s p r o d u c tio n (w h ile th e m o v ie is n o th in g m o r e th a n a c o n c e p t u a l id e a ). I n th e ir h ig h ly p u b li­ c iz e d p re d ic tio n m o d e ls , th e y c o n v e r t th e fo r e c a s tin g

( C o n t i n u e d )

2 6 2 Part III • P redictive Analytics

Application Case 5.6 (Continued) ( o r r e g r e s s io n ) p r o b le m in to a c la s s ific a tio n p r o b le m ; th a t is, ra th e r th a n fo r e c a s tin g th e p o in t e stim a te o f b o x - o f f ic e r e c e ip ts , th e y c la s sify a m o v ie b a s e d o n its b o x - o f f i c e r e c e ip ts in o n e o f n in e c a te g o r ie s , ra n g in g fro m “f lo p ” t o “b lo c k b u s te r .” m a k in g th e p r o b le m a m u ltin o m ia l c la s s ific a tio n p r o b le m . T a b l e 5 .4 illus­ tra tes t h e d e fin itio n o f th e n in e c la s s e s in te rm s o f th e r a n g e o f b o x - o f f i c e re c e ip ts .

D a ta

D a ta w a s c o l l e c t e d fr o m v a rie ty o f m o v ie -r e la te d d a t a b a s e s ( e . g . , S h o w B iz , IM D b , IM S D b , A llM o v ie, e t c . ) a n d c o n s o li d a t e d in to a s in g le d a ta s e t. T h e d a ta s e t f o r t h e m o s t r e c e n t ly d e v e l o p e d m o d e ls c o n t a in e d 2 , 6 3 2 m o v ie s r e le a s e d b e t w e e n 1 9 9 8 a n d 2 0 0 6 . A s u m m a r y o f t h e i n d e p e n d e n t v a r ia b le s a lo n g w ith t h e ir s p e c i f ic a t i o n s is p r o v id e d in T a b l e 5-5 ■ F o r m o r e d e s c r ip tiv e d e ta ils a n d ju s tific a tio n f o r in c lu ­ s io n o f t h e s e i n d e p e n d e n t v a r ia b le s , t h e r e a d e r is r e f e r r e d t o S h a r d a a n d D e l e n ( 2 0 0 7 ) .

M e th o d o lo g y

U s in g a v a r ie ty o f d a ta m in in g m e th o d s , in c lu d ­ i n g n e u r a l n e t w o r k s , d e c i s i o n tr e e s , s u p p o r t v e c ­ to r m a c h in e s , a n d th r e e t y p e s o f e n s e m b le s , S h a r d a

a n d D e l e n d e v e l o p e d t h e p r e d ic t io n m o d e ls . T h e d a ta fro m 1 9 9 8 t o 2 0 0 5 w e r e u s e d a s tra in in g d a ta to b u ild t h e p r e d ic t io n m o d e ls , a n d t h e d ata f r o m 2 0 0 6 w a s u s e d a s t h e t e s t d a ta to a s s e s s a n d c o m ­ p a r e t h e m o d e ls ’ p r e d ic t io n a c c u r a c y . F ig u r e 5 .1 5 s h o w s a s c r e e n s h o t o f IB M S P S S M o d e le r (fo r m e r ly C le m e n tin e d a ta m in in g t o o l ) d e p ic tin g t h e p r o c e s s m a p e m p lo y e d f o r t h e p r e d ic tio n p r o b le m . T h e u p p e r -le ft s id e o f t h e p r o c e s s m a p s h o w s t h e m o d e l d e v e l o p m e n t p r o c e s s , a n d t h e lo w e r -r ig h t c o r n e r o f t h e p r o c e s s m a p s h o w s t h e m o d e l a s s e s s m e n t ( i .e ., te s tin g o r s c o r i n g ) p r o c e s s ( m o r e d e ta ils o n IB M S P S S M o d e le r to o l a n d its u s a g e c a n b e fo u n d o n t h e b o o k ’s W e b s ite ).

R e s u lts

T a b l e 5 . 6 p r o v id e s th e p r e d ic t io n r e s u lts o f all th r e e d a ta m in in g m e t h o d s a s w e l l a s t h e r e s u lts o f t h e th r e e d iffe r e n t e n s e m b l e s . T h e first p e r f o r m a n c e m e a s u r e is t h e p e r c e n t c o r r e c t c la s s ific a tio n ra te , w h i c h is c a ll e d b i n g o . A ls o r e p o r t e d in t h e ta b le is th e 1 - A w a y c o r r e c t c la s s ific a tio n r a te ( i .e ., w ith in o n e c a te g o r y ) . T h e re s u lts i n d ic a te th a t SV M p e r ­ f o r m e d t h e b e s t a m o n g t h e in d iv id u a l p r e d ic tio n m o d e ls , f o llo w e d b y A N N ; t h e w o r s t o f t h e t h r e e

T A B L E 5.4 M o v ie Classification Based on Receipts

Class No._______________________ 1 2 3 4 5 6 7 8 9

Range (in millions of dollars) <1 >1 > 1 0 > 2 0 > 4 0 > 6 5 > 100 >150 >200 (Flop) < 1 0 < 20 < 40 < 6 5 < 100 < 150 <200 (Blockbuster)

T A B L E 5.5 Sum m ary o f Independent Variables

Independent Variable Num ber o f Values Possible Values

M PA A Rating 5 G, PG, PG-13, R, NR

Competition 3 High, Medium, Low

Star value 3 High, Medium, Low

Genre 10 Sci-Fi, Historic Epic Drama, Modern Drama, Politically Related, Thriller, Horror, Comedy, Cartoon, Action, Documentary

Special effects 3 High, Medium, Low

Sequel 1 Yes, No

Number of screens 1 Positive integer

C hapter 5 • Data Mining 2 6 3

FIGURE 5.15 Process Flow Screenshot for the Box-Office Prediction System. Source: Used w ith permission from IBM SPSS.

w a s t h e C A R T d e c is i o n tr e e a lg o rith m . I n g e n e r a l, t h e e n s e m b l e m o d e ls p e r f o n n e d b e t t e r th a n t h e in d i­ v id u a l p r e d i c t i o n s m o d e ls , o f w h ic h t h e fu s io n a l g o ­ rith m p e r f o r m e d t h e b e s t . W h a t is p r o b a b ly m o r e im p o r ta n t t o d e c i s io n m a k e r s , a n d s ta n d in g o u t in t h e r e s u lts t a b le , is t h e s ig n ific a n tly l o w s ta n d a r d

d e v ia tio n o b t a in e d fr o m t h e e n s e m b l e s c o m p a r e d to t h e in d iv id u a l m o d e ls .

C o n c lu s io n

T h e r e s e a rc h e rs c la im th a t th e s e p re d ic tio n resu lts a re b e tte r th a n a n y re p o rte d in th e p u b lis h e d literatu re fo r

T A B L E 5.6 Tabulated Prediction Results fo r Individual and Ensem ble M odels

Prediction M odels

Individual M odels Ensem ble M odels Random Boosted Fusion

Perform ance M easure SV M ANN C&RT Forest Tree (A verage)

C ount (Bin g o ) 192 182 140 189 187 194

Count (1-Away) 104 120 126 121 104 120

Accuracy { % Bingo) 55.49% 52.60% 40.46% 54.62% 54.05% 56.07%

Accuracy ( % 1-Away) 85.55% 87.28% 76.88% 8 9.60% 84.10% 90.75%

Standard d eviation 0.93 0.87 1.05 0.76 0.84 0.63

( C o n t i n u e d )

2 6 4 Part III • Predictive Analytics

Application Case 5.6 (Continued) th is p r o b le m d o m a in . B e y o n d th e attractiv e a c c u ra c y o f th e ir p re d ic tio n resu lts o f d ie b o x -o ffic e re c e ip ts, th e s e m o d e ls c o u ld a ls o b e u s e d to fu rth e r a n a ly z e (a n d p o te n tia lly o p tim iz e ) th e d e c is io n v a ria b le s in o rd e r to m a x im iz e th e fin a n c ia l retu rn . S p e cifica lly , th e p a ra m e te r s u s e d fo r m o d e lin g c o u ld b e altere d u sin g th e a lre a d y tra in ed p re d ic tio n m o d e ls in o rd e r to b e tte r u n d e rs ta n d th e im p a c t o f d iffe re n t p a ra m e ters o n th e e n d resu lts. D u rin g th is p ro c e s s , w h ic h is c o m m o n ly re fe rre d to a s s e n s it iv ity a n a l y s i s , th e d e c is io n m a k e r o f a g iv e n e n te rta in m e n t firm c o u ld fin d o u t, w ith a fairly h ig h a c c u r a c y le v e l, h o w m u c h v a lu e a s p e c ific a c to r ( o r a s p e c ific r e le a s e d a te, o r th e a d d itio n o f m o r e te c h n ic a l e ffe c ts , e t c .) b rin g s to th e fin a n c ia l s u c c e s s o f a film , m a k in g t h e u n d e rly in g s y s te m a n in v a lu a b le d e c is io n aid.

Sources: R. Sharda and D. Delen, “Predicting Box-O ffice Success o f Motion Pictures with Neural Networks,” Expert Systems with A pplications, Vol. 30, 2006, pp. 243-254; D. Delen, R. Sharda, and P. Kumar, “Movie Forecast Guru: A W eb-based DSS for Hollywood Managers,” D ecision Support Systems, Vol. 43, No. 4, 2007, pp. 1151-1170.

Q u e s t i o n s f o r D i s c u s s i o n

3 . H o w d o y o u th in k H o lly w o o d p e r fo r m e d , a n d p e r h a p s is still p e r fo r m in g , th is ta s k w ith o u t th e h e l p o f d a ta m in in g t o o ls a n d t e c h n iq u e s ?

2 . H o w c a n d a ta m in in g b e u s e d to p r e d ic t th e fin a n c ia l s u c c e s s o f m o v ie s b e f o r e t h e s ta rt o f th e ir p r o d u c t i o n p ro c e s s ?

1. W h y is it im p o r ta n t f o r H o lly w o o d p r o fe s s io n a ls t o p r e d i c t t h e fin a n c ia l s u c c e s s o f m o v ie s?

SECTION 5 .6 REVIEW QUESTIONS

1 . W h a t a r e t h e m o s t p o p u la r c o m m e r c ia l d a ta m in in g to o ls ?

2 . W h y d o y o u th in k th e m o s t p o p u la r t o o ls a r e d e v e l o p e d b y s ta tis tic s c o m p a n ie s ? 3 . W h a t a r e t h e m o s t p o p u la r f r e e d a ta m in in g to o ls ?

4 . W h a t a r e t h e m a in d iffe r e n c e s b e t w e e n c o m m e r c ia l a n d fr e e d ata m in in g s o ftw a r e tools?

5- W h a t w o u ld b e y o u r t o p fiv e s e l e c t i o n c r ite r ia f o r a d a ta m in in g to o l? E x p la in .

5.7 D ATA M IN IN G P R IV A C Y IS S U E S , M YTH S, A N D B L U N D E R S Data Mining and Privacy Issues D a ta th a t is c o l l e c t e d , s to r e d , a n d a n a ly z e d i n d a ta m in in g o f t e n c o n ta in s in fo r m a tio n a b o u t r e a l p e o p le . S u c h in fo r m a tio n m a y i n c lu d e id e n tific a tio n d a ta ( n a m e , a d d r e s s , S o c ia l S e c u r ity n u m b e r , d riv e r’s li c e n s e n u m b e r , e m p l o y e e n u m b e r , e t c .) , d e m o g r a p h ic d a ta ( e .g ., a g e , s e x , e th n ic ity , m a rita l s ta tu s , n u m b e r o f c h ild r e n , e t c .) , f in a n c ia l d a ta ( e .g ., s a la r y , g r o s s fa m ily i n c o m e , c h e c k i n g o r s a v in g s a c c o u n t b a l a n c e , h o m e o w n e r s h ip , m o r tg a g e o r lo a n a c c o u n t s p e c i f ic s , c r e d it c a r d lim its a n d b a la n c e s , in v e s tm e n t a c c o u n t s p e c if ic s , e t c . ) , p u r c h a s e h is to r y ( i .e ., w h a t is b o u g h t f r o m w h e r e a n d w h e n e ith e r fro m t h e v e n d o r ’s t r a n s a c tio n r e c o r d s o r fr o m c r e d it c a r d t r a n s a c tio n s p e c i f ic s ) , a n d o t h e r p e r ­ s o n a l d a ta ( e .g ., a n n iv e r s a r y , p r e g n a n c y , illn e s s , lo s s in t h e fa m ily , b a n k r u p tc y filin g s, e t c .) . M o s t o f t h e s e d a ta c a n b e a c c e s s e d t h r o u g h s o m e th ird -p a rty d a ta p ro v id e r s . T h e m a in q u e s t io n h e r e is t h e p r iv a c y o f t h e p e r s o n to w h o m t h e d a ta b e lo n g s . I n o r d e r to m a in ta in t h e p r iv a c y a n d p r o t e c t io n o f in d iv id u a ls ’ r ig h ts , d a ta m in in g p r o fe s s io n a ls h a v e e th ic a l ( a n d o f t e n le g a l) o b lig a tio n s . O n e w a y to a c c o m p li s h th is is t h e p r o c e s s o f d e -id e n tific a tio n o f t h e c u s t o m e r r e c o r d s p r io r to a p p ly in g d a ta m in in g a p p lic a tio n s , s o t h a t t h e r e c o r d s c a n n o t b e tr a c e d to a n in d iv id u a l. M a n y p u b lic ly a v a ila b le d a ta s o u r c e s ( e .g ., C D C d a ta , S E E R d a ta , U N O S d a ta , e t c . ) a r e a lr e a d y d e -id e n tifie d . P r io r to a c c e s s i n g t h e s e d a ta s o u r c e s , u s e r s a r e o f t e n a s k e d to c o n s e n t th a t u n d e r n o c ir c u m s ta n c e s w ill th e y try to id e n tify t h e in d iv id u a ls b e h i n d t h o s e fig u r e s .

C hapter 5 • D ata Mining 2 6 5

T h e r e h a v e b e e n a n u m b e r o f i n s ta n c e s in t h e r e c e n t p a s t w h e r e c o m p a n ie s s h a r e d th e ir c u s t o m e r d a ta w ith o t h e r s w ith o u t s e e k in g t h e e x p l ic it c o n s e n t o f t h e i r c u s to m e r s . F o r in s ta n c e , a s m o s t o f y o u m ig h t r e c a ll, in 2 0 0 3 , J e t B l u e A ir lin e s p r o v id e d m o r e th a n a m illio n p a s s e n g e r r e c o r d s o f th e ir c u s to m e r s t o T o r c h C o n c e p ts , a U .S . g o v e r n m e n t c o n tr a c to r . T o r c h t h e n s u b s e q u e n t l y a u g m e n t e d t h e p a s s e n g e r d a ta w ith a d d itio n a l in fo r ­ m a tio n s u c h a s fa m ily s iz e a n d S o c ia l S e c u r ity n u m b e r s — in fo r m a tio n p u r c h a s e d fr o m a d a ta b r o k e r c a ll e d A c x io m . T h e c o n s o li d a t e d p e r s o n a l d a ta b a s e w a s in t e n d e d t o b e u s e d fo r a d a ta m in in g p r o je c t in o r d e r to d e v e l o p p o te n tia l te r r o r is t p r o file s . A ll o f th is w a s d o n e w it h o u t n o t i f ic a t io n o r c o n s e n t o f p a s s e n g e r s . W h e n n e w s o f th e a c tiv itie s g o t o u t, h o w e v e r , d o z e n s o f p r iv a c y la w s u its w e r e file d a g a in s t J e t B l u e , T o r c h , a n d A c x io m , a n d s e v e r a l U .S . s e n a t o r s c a ll e d f o r a n in v e s tig a tio n in to th e in c id e n t (W a ld , 2 0 0 4 ) . S im ila r, b u t n o t a s d r a m a tic , p r iv a c y -r e la te d n e w s h a s c o m e o u t in t h e r e c e n t p a s t a b o u t t h e p o p u la r s o c ia l n e t w o r k c o m p a n i e s , w h i c h a lle g e d ly w e r e s e llin g c u s t o m e r - s p e c i f ic d a ta to o th e r

c o m p a n i e s f o r p e r s o n a liz e d ta r g e t m a r k e tin g . T h e r e w a s a n o t h e r p e c u l ia r s to r y a b o u t p r iv a c y c o n c e r n s th a t m a d e it in to t h e h e a d ­

l in e s in 2 0 1 2 . I n th is in s ta n c e , t h e c o m p a n y d id n o t e v e n u s e a n y p riv a te a n d / o r p e r s o n a l d a ta . L e g a lly s p e a k i n g , th e r e w a s n o v io la tio n o f a n y la w s . It w a s a b o u t T a r g e t a n d is

s u m m a r iz e d in A p p lic a tio n C a s e 5 .7 .

Application Case 5.7 Predicting Custom er Buying Patte rn s— The T arg et Story

I n e a r ly 2 0 1 2 , a n in fa m o u s s to r y a p p e a r e d c o n c e r n ­ i n g T a r g e t ’s p r a c t ic e o f p r e d ic tiv e a n a ly tic s . T h e Story w a s a b o u t a t e e n a g e r g irl w h o w a s b e in g s e n t a d v e r tis in g fly e r s a n d c o u p o n s b y T a r g e t fo r th e k in d s o f th in g s t h a t a n e w m o t h e r - t o - b e w o u ld b u y f r o m a s to r e lik e T a r g e t. T h e s to r y g o e s lik e th is : A n a n g r y m a n w e n t in to a T a r g e t o u ts id e o f M in n e a p o lis , d e m a n d in g to ta lk to a m a n a g e r : “M y d a u g h te r g o t th is in t h e m a il!” h e s a id . “S h e ’s still in h ig h s c h o o l , a n d y o u ’r e s e n d in g h e r c o u p o n s fo r b a b y c l o t h e s a n d c rib s ? A re y o u try in g t o e n c o u r a g e h e r to g e t p r e g n a n t? ” T h e m a n a g e r d id n ’t h a v e a n y id e a w h a t t h e m a n w a s ta lk in g a b o u t. H e l o o k e d a t t h e m a ile r . S u r e e n o u g h , it w a s a d d r e s s e d to th e m a n ’s d a u g h te r a n d c o n t a in e d a d v e r tis e m e n ts fo r m a te r n ity c lo t h i n g , n u r s e r y fu r n itu re , a n d p ic tu r e s o f s m ilin g in fa n ts . T h e m a n a g e r a p o l o g i z e d a n d th e n c a lle d a f e w d a y s la t e r to a p o lo g i z e a g a in . O n th e p h o n e , t h o u g h , t h e fa th e r w a s s o m e w h a t a b a s h e d . “I h a d a ta lk w ith m y d a u g h te r ,” h e s a id . “I t tu r n s o u t t h e r e ’s b e e n s o m e a c tiv itie s in m y h o u s e I h a v e n ’t b e e n c o m p l e te l y a w a r e o f. S h e ’s d u e in A u g u st. I

o w e y o u a n a p o l o g y .” A s it tu r n s o u t, T a r g e t fig u r e d o u t a t e e n girl

w a s p r e g n a n t b e f o r e h e r f a th e r d id ! H e r e is h o w t h e y d id it. T a r g e t a s s ig n s e v e r y c u s to m e r a G u e s t

ID n u m b e r (t ie d t o th e ir c r e d it c a rd , n a m e , o r e -m a il a d d r e s s ) t h a t b e c o m e s a p la c e h o l d e r th a t k e e p s a h is to r y o f e v e r y th in g th e y h a v e b o u g h t. T a r g e t a u g ­ m e n ts th is d a ta w it h a n y d e m o g r a p h ic in fo r m a tio n th a t t h e y h a d c o l l e c t e d fr o m t h e m o r b o u g h t fro m o t h e r in fo r m a tio n s o u r c e s . U s in g th is in fo rm a tio n , T a r g e t l o o k e d a t h is to r ic a l b u y in g d a ta f o r a ll th e fe m a le s w h o h a d s ig n e d u p fo r T a r g e t b a b y r e g ­ is trie s in th e p a s t. T h e y a n a ly z e d t h e d a ta f r o m a ll d ir e c tio n s , a n d s o o n e n o u g h s o m e u s e fu l p a tte rn s e m e r g e d . F o r e x a m p le , l o tio n s a n d s p e c i a l v ita m in s w e r e a m o n g t h e p r o d u c ts w ith in te r e s tin g p u r c h a s e p a tte r n s . L o ts o f p e o p l e b u y lo tio n , b u t th e y h a v e n o t i c e d w a s th a t w o m e n o n t h e b a b y re g is try w e r e b u y in g la rg e r q u a n titie s o f u n s c e n t e d lo ti o n a r o u n d th e b e g i n n in g o f t h e ir s e c o n d trim e s te r . A n o th e r a n a ­ ly st n o t e d th a t s o m e t im e in t h e first 2 0 w e e k s , p r e g ­ n a n t w o m e n l o a d e d u p o n s u p p le m e n ts lik e c a lc iu m , m a g n e s iu m , a n d z in c . M a n y s h o p p e r s p u r c h a s e s o a p a n d c o t t o n b a lls , b u t w h e n s o m e o n e s u d d e n ly starts b u y in g lo ts o f s c e n t - f r e e s o a p a n d e x tr a -b ig b a g s o f c o t t o n b a lls , i n a d d itio n to h a n d s a n itiz e rs a n d w a s h c lo th s , it s ig n a ls th a t th e y c o u l d b e g e ttin g c lo s e to th e ir d e liv e r y d a te . At t h e e n d , th e y w e r e a b le t o id e n tify a b o u t 2 5 p r o d u c ts th a t, w h e n a n a ly z e d to g e th e r , a l lo w e d t h e m t o a s s ig n e a c h s h o p p e r a

(■C o n t i n u e d )

2 6 6 Part III • Predictive Analytics

Application Case 5.7 (Continued) “p r e g n a n c y p r e d ic t io n ” s c o r e . M o r e im p o rta n t, th e y c o u l d a l s o e s tim a te a w o m a n ’s d u e d a te t o w ith in a s m a ll w in d o w , s o T a r g e t c o u ld s e n d c o u p o n s tim e d t o v e r y s p e c i f ic s ta g e s o f h e r p r e g n a n c y .

I f y o u l o o k a t th is p r a c tic e fr o m a l e g a l p e r s p e c - tiv e , y o u w o u l d c o n c lu d e th a t T a r g e t d id n o t u s e a n y i n fo r m a tio n th a t v io la te s c u s to m e r p riv a c y ; ra th e r, t h e y u s e d tr a n s a c tio n a l d a ta th a t m o s t e v e r y o th e r re ta il c h a in is c o l le c t i n g a n d sto r in g ( a n d p e r h a p s a n a ly z in g ) a b o u t th e ir c u s to m e r s . W h a t w a s d is tu rb ­ in g in th is s c e n a r io w a s p e r h a p s th e ta r g e te d c o n c e p t: p r e g n a n c y . T h e r e a r e c e r ta in e v e n ts o r c o n c e p t s th a t s h o u ld b e o f f lim its o r tr e a te d e x tr e m e ly c a u tio u s ly , s u c h a s te rm in a l d is e a s e , d iv o r c e , a n d b a n k ru p tc y .

1. W h a t d o y o u th in k a b o u t d a ta m in in g a n d its im p lic a tio n s c o n c e r n in g p riv acy ? W h a t is th e th r e s h o ld b e t w e e n k n o w le d g e d is c o v e r y a n d p r iv a c y in fr in g e m e n t?

2 . D id T a r g e t g o t o o far? D id t h e y d o a n y th in g ille g a l? W h a t d o y o u th in k t h e y s h o u ld h a v e d o n e ? W h a t d o y o u th in k th e y s h o u ld d o n o w ( q u it t h e s e ty p e s o f p r a c tic e s )?

Sources: K. Hill “How Target Figured Out a Teen Girl Was Pregnant Before Her Father Did,” Forbes, February 13, 2012; and R. Nolan, “Behind the Cover Story: How Much Does Target Know'?” NYTim es.com , February 21, 2012.

Q u e s t i o n s f o r D i s c u s s i o n

D a ta M in in g M y th s a n d B lu n d e rs

D a ta m in in g is a p o w e r fu l a n a ly tic a l t o o l th a t e n a b l e s b u s i n e s s e x e c u t i v e s t o a d v a n c e f r o m d e s c r ib in g t h e n a tu r e o f t h e p a s t to p r e d ic tin g t h e fu tu re . It h e lp s m a r k e te r s fin d p a tte r n s th a t u n lo c k t h e m y s te r ie s o f c u s t o m e r b e h a v i o r . T h e r e s u lts o f d a ta m in in g c a n b e u s e d to i n c r e a s e r e v e n u e , r e d u c e e x p e n s e s , id e n tify fra u d , a n d lo c a t e b u s i n e s s o p p o r tu n i­ tie s , o f f e r in g a w h o l e n e w r e a lm o f c o m p e titiv e a d v a n ta g e . A s a n e v o lv in g a n d m a tu rin g fie ld , d a ta m in in g is o f te n a s s o c ia t e d w ith a n u m b e r o f m y th s , in c lu d in g t h e fo llo w in g

(Z a im a , 2 0 0 3 ) :

M yth R eality

Data mining provides instant, crystal-ball-like Data mining is a multistep process that requires predictions. deliberate, proactive design and use.

Data mining is not yet viable for business The current state-of-the-art is ready to go for applications. almost any business.

Data mining requires a separate, Because of advances in database technology, dedicated database. a dedicated database is not required, even

though it may be desirable.

Only those with advanced degrees can Newer Web-based tools enable managers of all do data mining. educational levels to do data mining.

Data mining is only for large firms that have If the data accurately reflect the business or its lots of customer data. customers, a company can use data mining.

D a t a m in in g v is io n a r ie s h a v e g a in e d e n o r m o u s c o m p e titiv e a d v a n ta g e b y u n d e r s ta n d in g

th a t t h e s e m y th s a r e ju s t th a t: m y th s . T h e f o llo w in g 1 0 d a ta m in in g m is ta k e s a r e o f t e n m a d e in p r a c tic e ( S k a la k , 2 0 0 1 ;

S h u ltz , 2 0 0 4 ) , a n d y o u s h o u ld try t o a v o id th e m :

1 . S e le c t in g t h e w r o n g p r o b l e m f o r d a ta m in in g . 2. Ig n o r in g w h a t y o u r s p o n s o r th in k s d a ta m in in g is a n d w h a t it re a lly c a n a n d c a n n o t d o .

Chapter 5 • D ata Mining 2 6 7

3. L e a v in g in s u f f ic ie n t t i m e f o r d a ta p r e p a r a tio n . It ta k e s m o r e e f f o r t t h a n is g e n e r a lly

4 . L o o k in g o n l y a t a g g r e g a te d re s u lts a n d n o t a t in d tv id u a l r e c o r d s . IB M ’s D B 2 IM S c a n

h ig h lig h t in d iv id u a l r e c o r d s o l in te re s t. 5 . B e i n g s l o p p y a b o u t k e e p i n g t r a c k o f t h e d a ta m in in g p r o c e d u r e a n d re s u lts.

6 I g n o r in g s u s p ic io u s fin d in g s a n d q u ic k ly m o v in g o n . 7 . R u n n in g m in in g a lg o r ith m s r e p e a t e d ly a n d b lin d ly . It is im p o r ta n t t o th in k h a r d

a b o u t t h e n e x t s ta g e o f d a ta a n a ly s is . D a ta m in in g rs a v e r y h a n d s - o n a c tiv ity .

8 B e l ie v i n g e v e r y th in g y o u a r e to ld a b o u t t h e d ata. 9 B e l ie v i n g e v e r y th in g y o u a r e t o ld a b o u t y o u r o w n d a ta m in in g a n a ly s is .

10. M e a s u r in g y o u r re s u lts d iffe r e n tly fr o m t h e w a y y o u r s p o n s o r m e a s u r e s th e m .

S E C T I O N 5 . 7 R E V I E W Q U E S T I O N S

X. W h a t a r e t h e p r iv a c y is s u e s in d a ta m in in g ? 2. H o w d o y o u th in k t h e d is c u s s io n b e t w e e n p r iv a c y a n d d a ta m in in g w ill p r o g r e s s . y. 3 . W h a t a r e t h e m o s t c o m m o n m y th s a b o u t d a ta m in in g ? 4 W h a t d o y o u th in k a r e t h e r e a s o n s f o r t h e s e m y th s a b o u t d a ta m in in g ? 5 . W h a t a r e t h e m o s t c o m m o n d a ta m in in g m is ta k e s/ b lu n d e rs ? H o w c a n t h e y b e m in i­

m iz e d a n d / o r e lim in a te d ?

Chapter Highlights • D a ta m in in g is t h e p r o c e s s o f d is c o v e r in g n e w

k n o w le d g e f r o m d a ta b a s e s . • D a ta m in in g c a n u s e s im p le fla t file s a s d a ta

s o u r c e s o r it c a n b e p e r f o r m e d o n d a ta in d a ta

w a r e h o u s e s . • T h e r e a r e m a n y a lte r n a tiv e n a m e s a n d d e fin itio n s

f o r d a ta m in in g . • D a ta m in in g is a t t h e in t e r s e c t io n o f m a n y d is ­

c ip lin e s , in c lu d in g s ta tis tic s , a rtific ia l i n t e llig e n c e ,

a n d m a th e m a tic a l m o d e lin g . • C o m p a n i e s u s e d a ta m in in g t o b e t t e r u n d e r s ta n d

t h e ir c u s t o m e r s a n d o p tim iz e t h e ir o p e r a tio n s . • D a ta m in in g a p p lic a tio n s c a n b e fo u n d in v irtu a lly

e v e r y a r e a o f b u s in e s s a n d g o v e r n m e n t, in c lu d ­ i n g h e a l t h c a r e , fin a n c e , m a r k e tin g , a n d h o m e l a n d

s e c u r ity . • T h r e e b r o a d c a t e g o r i e s o f d a ta m in in g ta s k s a re

p r e d ic t io n (c la s s i f i c a t i o n o r r e g r e s s io n ) , c lu s te r ­

in g , a n d a s s o c ia tio n . • S im ila r t o o t h e r in fo r m a tio n s y s te m s in itia tiv e s ,

a d a ta m in in g p r o je c t m u s t f o ll o w a s y s te m a tic p r o je c t m a n a g e m e n t p r o c e s s t o b e s u c c e s s fu l.

• S e v e r a l d a ta m in in g p r o c e s s e s h a v e b e e n p i o - p o s e d : C R IS P -D M , SEM M A , K D D , a n d s o fo rth .

• C R IS P -D M p r o v id e s a s y s te m a tic a n d o r d e r ly w a y t o c o n d u c t d a ta m in in g p r o je c ts .

• T h e e a r lie r s te p s in d a ta m in in g p r o je c t s ( i .e ., u n d e r s ta n d in g t h e d o m a in a n d t h e r e le v a n t d a ta ) c o n s u m e m o s t o f t h e to ta l p r o je c t tim e ( o f t e n m o r e t h a n 8 0 % o f t h e to ta l tim e ).

• D a ta p r e p r o c e s s i n g is e s s e n tia l to a n y s u c c e s s fu l d a ta m in in g s tu d y . G o o d d a ta le a d s to g o o d in fo r­ m a tio n ; g o o d in fo r m a tio n le a d s to g o o d d e c is io n s .

• D a ta p r e p r o c e s s i n g in c lu d e s f o u r m a in s te p s : d ata c o n s o li d a t i o n , d a ta c le a n in g , d a ta tr a n s fo r m a tio n ,

a n d d a ta r e d u c tio n . • C la s s ific a tio n m e th o d s le a r n fr o m p r e v io u s e x a m ­

p l e s c o n t a in in g in p u ts a n d t h e r e s u ltin g c la s s la b e ls , a n d o n c e p r o p e r ly t r a in e d th e y a r e a b le to

c la s s ify fu tu r e c a s e s . • C lu s te rin g p a r titio n s p a tte r n r e c o r d s in to n a tu ra l

s e g m e n ts o r c lu s te r s . E a c h s e g m e n t ’s m e m b e r s s h a r e s im ila r c h a r a c te r is tic s .

• A n u m b e r o f d iffe r e n t a lg o r ith m s a r e c o m m o n ly u s e d f o r c la s s ific a tio n . C o m m e r c ia l im p le m e n ta ­ tio n s i n c lu d e I D 3 , C 4 .5 , C 5 , C A R T , a n d S P R IN T .

• D e c is io n t r e e s p a r titio n d a ta b y b r a n c h i n g a lo n g d iffe r e n t a ttr ib u te s s o th a t e a c h l e a f n o d e h a s a ll t h e p a tte r n s o f o n e c la s s .

• T h e G in i in d e x a n d in fo r m a tio n g a in ( e n t r o p y ) a r e tw o p o p u la r w a y s t o d e te r m in e b r a n c h in g

c h o ic e s in a d e c i s io n tre e .

2 6 8 Part III • Predictive Analytics

• T h e G in i i n d e x m e a s u r e s t h e p u r ity o f a s a m p le . I f e v e r y th in g in a s a m p le b e lo n g s to o n e c la s s , t h e G in i i n d e x v a lu e is z e r o .

• S e v e r a l a s s e s s m e n t t e c h n iq u e s c a n m e a s u r e t h e p r e d i c t i o n a c c u r a c y o f c la s s ific a tio n m o d e ls , in c lu d in g s im p le sp lit, M o l d c r o s s -v a lid a tio n , b o o ts tr a p p in g , a n d a r e a u n d e r t h e R O C c u rv e .

• C lu ster a lg o rith m s a r e u s e d w h e n th e d a ta re c o rd s d o n o t h a v e p r e d e fin e d c la s s id en tifiers (i.e ., it is n o t k n o w n t o w h a t c la s s a p a rtic u la r r e c o r d b e lo n g s ).

• C lu s te r a lg o r ith m s c o m p u t e m e a s u r e s o f s im ila rity i n o r d e r t o g r o u p s im ila r c a s e s in to c lu s te rs .

• T h e m o s t c o m m o n ly u s e d s im ila rity m e a s u r e in c lu s te r a n a ly s is is a d is t a n c e m e a s u r e .

• T h e m o s t c o m m o n l y u s e d c lu s te r in g a lg o rith m s a r e fe -m e a n s a n d s e lf-o r g a n iz in g m a p s.

• A s s o c ia tio n r u le m in in g is u s e d t o d is c o v e r tw o o r m o r e ite m s ( o r e v e n t s o r c o n c e p t s ) th a t g o to g e th e r .

• A s s o c ia tio n r u le m in in g is c o m m o n l y r e f e r r e d to a s m a r k e t - b a s k e t a n a ly s is .

• T h e m o s t c o m m o n l y u s e d a s s o c ia t io n a lg o r ith m is A p rio ri, w h e r e b y f r e q u e n t ite m s e ts a r e id e n ti­ f i e d th r o u g h a b o t t o m - u p a p p r o a c h .

• A s s o c ia tio n r u le s a r e a s s e s s e d b a s e d o n t h e ir s u p ­ p o r t a n d c o n f i d e n c e m e a s u r e s .

• M a n y c o m m e r c i a l a n d f r e e d a ta m in in g t o o ls a re a v a ila b le .

• T h e m o s t p o p u la r c o m m e r c ia l d a ta m in in g to o ls a r e S P S S P A S W a n d SA S E n te r p r is e M in e r.

• T h e m o s t p o p u la r f r e e d a ta m in in g t o o ls a r e W e k a a n d R a p id M in e r .

Key Terms

A p rio ri a lg o r ith m a r e a u n d e r t h e R O C

c u r v e a s s o c ia tio n b o o ts tr a p p in g c a t e g o r ic a l d ata c la s s ific a tio n c lu s te r in g c o n f id e n c e C R IS P -D M

d a ta m in in g d e c is i o n tre e d is ta n c e m e a s u r e e n tr o p y G in i in d e x in fo r m a tio n g a in in te rv a l d ata & -fold c r o s s -v a lid a tio n k n o w le d g e d is c o v e r y in

d a t a b a s e s (K D D )

lift lin k a n a ly s is M ic r o s o ft E n te r p r is e

C o n s o r tiu m M ic r o s o ft S Q L S e r v e r n o m in a l d a ta n u m e r ic d a ta o r d in a l d ata p r e d ic tio n R a p id M in e r

r a tio d a ta r e g r e s s io n SEM M A s e q u e n c e m in in g s im p le s p lit s u p p o r t W e k a

Questions fo r Discussion 1 . D efine d a t a m in in g . W hy are there m any nam es and

definitions fo r data mining? 2 . W hat are th e m ain reasons for the recent popularity o f

data mining? 3 . D iscuss w h at an organization should con sid er before

m aking a d ecisio n to pu rch ase data m ining software. 4 . Distinguish data m ining from other analytical tools and

techniques. 5 . D iscuss th e main data m ining m ethods. W hat are the

fundam ental d ifferences am ong them? 6. W hat a re th e m ain data m ining application areas? Discuss

th e com m onalities o f th ese areas that m ake th em a pros­ p ect for data m ining studies.

7 . "Why do w e need a standardized data mining process? What are the most com m only used data mining processes?

8 . Explain h o w th e validity o f available data affects m odel d evelop m en t and assessm ent.

9 . In w hat step o f data processing should m issing data b e distinguished? H ow will it affect data clearing and transformation?

1 0 . W hy d o w e n e e d data preprocessing? W hat are th e main tasks and relevan t tech niques u sed in data preprocessing?

1 1 . Distinguish betw een an inconsistent value and missing value. W hen c a n inconsistent values b e rem oved from the data set and missing data b e ignored?

1 2 . W hat is the m ain d ifference betw een classification and clustering? Explain using con crete exam ples.

1 3 . D iscuss the com m o n w ays to deal with th e m issing data? H ow is m issing data d ealt with in a survey w here som e respond ents h av e n o t replied to a few questions?

1 4 . W hat are the privacy issues w ith data mining? D o you think they are substantiated?

1 5 . W hat are th e m ost com m o n myths and m istakes about data mining?

Chapter 5 * D ata Mining 2 6 9

Exercises

Teradata University Network (TUN) and Other Hands-on Exercises

1 . Visit teradatauniversitynetwork.com. Identify case studies and w h ite papers about data mining. D escribe recen t d evelopm ents in th e field.

2 . G o to teradatauniversitynetwork.com o r a URL pro­ vid ed by y o u r instructor. Locate W eb sem inars related to data m ining. In particular, locate a sem inar given by C. Im h o ff and T . Zouqes. W atch th e W eb sem inar. T h en answ er the follow ing questions:

a. W hat are som e o f the interesting applications o f data mining?

b. W hat types o f payoffs and costs ca n organizations e x p e c t fro m data m ining initiatives?

3 . F o r this ex ercise, your goal is to build a m odel to iden­ tify inputs o r predictors that differentiate risky custom ers from others (b a sed o n patterns pertaining to previous custom ers) and th en u se those inputs to predict new risky custom ers. This sam ple c a se is typical for this domain.

T h e sam p le data to b e used in this exercise are in O nline File W 5.1 in th e file CreditRisk.xlsx. T h e data set has 4 25 cases and 15 variables pertaining to past and current custom ers w h o have borrow ed from a bank for various reasons. T h e data set con tain s custom er-related information su ch as financial standing, reason for the loan, em ploym ent, dem ographic inform ation, and the o u tcom e o r d ep en den t variable for credit standing, clas­ sifying e a c h c a se as g ood o r bad, based o n the institu­ tion ’s past exp erien ce.

T a k e 4 0 0 o f th e c a se s as training c a se s and set asid e th e o th e r 2 5 for testing. B u ild a d ecisio n tree m od el to le a rn th e ch aracteristics o f th e problem . T e st its p erfo rm an ce o n th e o th e r 25 c a se s. R ep ort o n your m o d el’s learn in g and testing p erform an ce. P rep are a rep ort that identifies th e d ecisio n tree m o d el and train­ ing p aram eters, a s w ell a s th e resultin g p erfo rm an ce o n th e te st set. U se an y d ec isio n tree softw are. (T h is e x e r ­ c is e is cou rtesy o f StatSoft, In c., b a s e d o n a G erm an data set fro m ftp.ics.uci.edu/pub/m achine-learning- d atab ases/statlog/germ an ren am ed C reditRisk and altered .)

4 . F o r this e x e rc ise , you w ill rep licate (o n a sm aller s c a le ) the b o x -o ffic e p red ictio n m od elin g ex p la in ed in A pplication C ase 5.6. D ow n load th e training data set from O n lin e File W 5.2, MovieTrain.xlsx, w h ich is in M icrosoft E x c e l form at. U se th e data d escription given in A p p lication C ase 5 .6 to un derstand th e d om ain and th e p ro b lem you are trying to solve. P ick a n d c h o o se y o u r in d ep en d en t variables. D e v e lo p at least three clas­ sification m od els (e .g ., d ecisio n tree, logistic reg res­ sion , n eu ral netw ork s). C om pare th e a ccu racy results u sin g 10-fo ld cross-validation a n d p ercen tag e split

tech n iq u es, u s e c on fu sio n m atrices, and com m en t on th e o u tco m e. T e s t th e m od els y o u h av e d ev elo p ed on th e test set (s e e O n lin e File W 5.3, MovieTest.xlsx). A nalyze th e results w ith d ifferent m od els and c o m e up w ith th e b e st c lassificatio n m od el, supporting it w ith y o u r results.

5 . T his exercise is aim ed at introducing you to association rule mining. T h e E x cel data set b ask etsln tran s.xlsx has around 2 8 0 0 observations/records o f superm arket transaction data. E ach record con tain s the custom er’s ID and Products that they have purchased. U se this data set to understand th e relationships am on g prod­ ucts (i.e ., w h ich products are purchased together). Look for interesting relationships and add screenshots o f any subtle association patterns that you might find. More specifically, an sw er th e follow ing questions.

W hich association rules d o you think are most important?

• B ased o n som e o f the association rules you found, m ake at least three business recom m endations that might b e beneficial to the com pany. These recom m endations may include ideas about shelf organization, upselling, or cross-selling products. (B on us points will b e given to new/innovative ideas.) W hat are th e Support, C onfid ence, and Lift values for the follow ing rule?

• W ine, C anned V eg => Frozen Meal

Team Assignments and Role-Playing Projects 1 . Exam ine h o w new data-capture d evices su ch as

radio-frequency identification (RFID ) tags help organi­ zations accu rately identify and segm ent their custom ers for activities su ch as targeted marketing. Many o f these applications involve data mining. Scan the literature and th e W e b and th e n prop ose five potential n ew data m ining applications that c a n u se th e data created with RFID tech n olog )7. W hat issues cou ld arise if a country’s laws requ ired su ch d evices to b e em bedded in every­ o n e ’s b o d y fo r a national identification system?

2 . Interview administrators in your colleg e o r executives in you r organization to determ ine h o w data w arehous­ ing, data m ining, OLAP, and visualization tools could assist them in their w ork. W rite a proposal describing your findings. Include c o s t estim ates and benefits in your

report. 3. A very g o o d repository o f data that has b e e n used to

test th e p erform ance o f m any data mining algorithms is available at ics.uci.edu/~mlearn/MLRepository. html. Som e o f th e data sets are m eant to test th e limits o f current m achine-learning algorithm s and to com pare their p erform ance w ith new ap p roach es to learning. H ow ever, so m e o f th e sm aller data sets ca n b e useful for exp loring th e functionality o f any data m ining softw are

2 7 0 Part III • Predictive Analytics

o r th e softw are that is available as com p an ion softw are w ith this b o o k , su ch as Statistica Data Miner. D ow nload at least o n e data s e t from this repository (e .g ., Credit Screening D atabases, H ousing D atabase) and apply d ecision tree o r clustering m ethods, as appropriate. Prepare a report b ased o n you r results. (Som e o f these exercises m ay b e u sed a s sem ester-long term projects, for exam p le.)

4 . T h e re a re larg e a n d featu re rich data sets m ad e avail­ a b le by th e U .S. g o v ern m en t o r its su bsid iaries o n the In tern et. F o r in s ta n ce C en ters fo r D ise a s e C on trol and P reven tio n d ata sets (cdc.gov/D ataStatistics); the N ational C a n ce r In stitu te’s S u rv eillan ce E p id em iolog y and End R esu lts data sets (seer.can cer.g ov /d ata); and th e D ep a rtm en t o f T ran sp o rtatio n ’s Fatality A nalysis R ep o rtin g System crash data sets (nh tsa.go v / FARS). T h e s e data sets are not p re p r o c e s se d fo r data m ining, w h ic h m ak es th em a great re s o u rc e to e x p e r i­ e n c e th e c o m p le te data m ining p ro c e ss . A n oth er rich s o u rc e fo r a c o lle c tio n o f analytics data sets is listed o n KDNuggets.com (k d n u g gets.co m /d atasets/ index.htm l).

5. Consider the follow ing data set, w hich includes three attributes and a classification for adm ission d ecisions into an MBA program:

a. U sing th e data show n, d evelop you r o w n manual exp ert ru les fo r d ecision making.

b. U se the G in i ind ex to build a d ecision tree. Y ou can u se m anual calculation s o r a spreadsheet to perform th e b asic calculations.

c. U se an autom ated d ecision tree softw are program to build a tre e for the sam e data.

G M A T G PA Q u an titative G M A T Score (percentile) Decision

650 2.75 35 No 580 3.50 70 No 600 3.50 75 Yes 450 2.95 80 No 700 3.25 90 Yes 590 3.50 80 Yes 400 3.85 45 No 640 3.50 75 Yes 540 3.00 60 ?

690 2.85 80 ?

490 4.00 65 7

Internet Exercises 1. Visit the AI Exploratorium at cs.ualberta.ca/~aixplore.

C lick th e D ec isio n T re e link. Read th e narrative on bas­ ketball gam e statistics. Exam ine th e data and th en build a d ecisio n tree. Report you r im pressions o f the accuracy o f this d ecision tree. Also, exp lo re th e effects o f different algorithms.

2. Survey som e data m ining tools and vendors. Start with fairisaac.com and egain.com. Consult dmreview.com and identify so m e data m ining products and service pro­ viders that are n o t m en tioned in this chapter.

3 . Find recen t ca se s o f successful data m ining applications. Visit th e W eb sites o f som e data m ining vendors and lo o k for cases o r su ccess stories. Prepare a report sum­ marizing five n ew case studies.

4 . G o to vend or W e b sites (esp ecially th o se o f SAS, SPSS, Cognos, Teradata, StatSoft, and Fair Isa a c ) and lo o k at success stories for B I (OLAP and data m ining) tools. W hat d o th e various success stories have in common? H ow d o they differ?

5. G o to statsoft.com. D ow nload at least three white papers o n applications. W hich o f th ese applications may have used the data/text/Web m ining tech n iqu es dis­ cu ssed in this chapter?

6. G o to sas.com. D ow nload at least three w hite papers on applications. W h ich o f th ese applications may have used the data/text/Web m ining tech n iqu es discussed in this chapter?

7 . G o to spss.com. D ow nload at least th ree white papers o n applications. W hich o f th ese applications m ay have used the data/text/Web m ining tech n iqu es discussed in this chapter?

8 . G o to teradata.com . D ow nload at least three white papers o n applications. W hich o f th ese applications may have used th e data/text/Web m ining tech niques dis­ cussed in this chapter?

$>. G o to fairisaac.com. D ow nload at least three white papers o n applications. W hich o f th ese applications m ay have used th e data/text/Web m ining techniques d iscussed in this chapter?

10. G o to salfordsystems.com. D ow nload at least three w hite papers o n applications. W hich o f th ese applica­ tions may have used th e data/text/Web mining tech­ niques discussed in this chapter?

11. G o to rulequest.com. D ow nload at least three white papers o n applications. W hich o f th e se applications m ay have used the data/text/Web m ining techniques discussed in this chapter?

12. G o to kdnuggets.com. E xplore th e section s o n applica­ tions as well as softw are. Find nam es o f at least three additional p a ck ag es for data mining and text mining.

Chapter 5 • Data Mining 2 71

End-of-Chapter Application Case M a c y s .c o m Enhances Its Custom ers' Shopping Experience w ith A n alytics

After m ore than 8 0 y ears in b u sin ess, M acy’s In c. is o n e o f A m erica’s m o st ico n ic retailers. W ith ann u al rev en u es e x c e e d in g $ 2 0 b illion , M acy’s en jo y s a loyal b a s e o f cu stom ers w h o c o m e to its stores and sh o p o n lin e e a c h day. T o c o n tin u e its leg acy o f providing stellar cu stom er serv ice and th e right s ele c tio n o f prod ucts, th e retailer s e-co m m e rc e division— M acy s.com — is using analytics to better u n d erstan d and e n h a n c e its cu stom ers’ o n lin e s h o p p in g e x p e r ie n c e , w h ile h elp in g to in crea se th e retail­ e r’s overall profitability.

T o m o re effectively m easure and understand the im pact o f its o n lin e m arketing initiatives o n Macy’s store sales, M acys.com increased its analytical capabilities w ith SAS Enterprise M iner (o n e o f th e prem ier data m ining tools in the m arket), resulting in an e-m ail subscription churn reduction o f 2 0 p ercen t. It also u ses SAS to autom ate report generation, saving more th an $500 ,0 0 0 a year in com p analyst time.

E n d in g “O n e S ize F i t s A ll” E -M ail M a r k e tin g “W e w ant to understand custom er lifetim e value, explains K erem T om ak , vice p resid ent o f analytics for M acys.com . “W e w ant to understand how long o u r custom ers have b een with u s, h o w often an e-m ail from u s triggers a visit to our site. This help s us better understand w h o our b est custom ers are and h o w eng aged they are with us. [With that know ledge} w e c a n give o u r valuable custom ers the right prom otions in order to serve them the b est way possible.

“Custom ers share a lo t o f inform ation w ith us— their likes and dislikes— and our task is to support them in return for their loyalty by providing them with w hat they want, instantly,” adds Tom ak . M acys.com uses H adoop as a data platform for SAS Enterprise Miner.

Initially, Tom ak w as w orried that segm enting cu stom ­ ers and sen ding few er, bu t m ore sp ecific, e=m ails w ould reduce traffic to the W e b site. “T h e gen eral b e lie f w as that w e had to b la st ev ery on e,” T o m ak said. Today, e-m ails are sen t less frequently, bu t w ith m ore thought, and the retailer has red u ced subscription churn rate b y approxim ately

2 0 percent.

T im e Savin gs, Low er Costs T om ak’s group is responsible for creating a variety o f m ission critical reports— so m e daily, som e w eekly, others monthly that g o to em p lo y ees in marketing and finance. T h ese data-rich reports w ere taking analysts 4 to 12 hours to pro­ duce— m uch o f it busy w ork that involved cutting and past­ ing from E xcel spreadsheets. M acys.com is n ow using SAS to autom ate the reports. “This cuts th e time dramatically. It saves us m ore than $500 ,0 0 0 a y ear in terms o f com p FTE hours saved— a really b ig im pact," Tom ak says, noting that th e sav­ ings b eg an w ithin about 3 m onths o f installing SAS.

Now his staff c a n m axim ize tim e spent o n provid­ ing valu e-add ed analyses and insights to provide content, products, and o ffers that gu arantee a personalized shopping e x p erien c e for M acys.com custom ers.

“Macy’s is a very information-hungry organization, and requests for ad h o c reports com e from all over th e com pany. T h ese stream lined systems elim inate error, guarantee accuracy, and increase th e sp eed w ith w hich w e can address requests,” T om ak says. “E a ch time w e use the software, w e find new w ays o f doing things, and w e are m ore and m ore impressed b y the speed a t w hich it chum s out data and m odels.”

M o v in g F o r w a r d “W ith th e extra tim e, th e team has m oved from bein g rea c­ tionary to proactive, m eaning they c a n exam in e m ore data, sp en d quality tim e analyzing, and b e co m e internal con su l­ tants w h o provide m ore insight behind the data, h e says. “This will b e im portant to supporting the strategy and driving th e n ex t gen eration o f M a cy s.c o m .”

As com petition increases in th e onlin e retailing world, T o m ak says th ere is a push toward generating m ore accurate, real-tim e d ecision s about custom er p references. T h e ability to gain custom er insight across channels is a critical part o f improving custom er satisfaction and revenues, and M acys.com uses SAS Enterprise Miner to validate and guide th e site’s cross- and u p -sell offer algorithms.

Source: w w w . s a s . c o m / s u c c e s s / m a c y . h t m l .

References Bhandari, I ., E. Colet, J . Parker, Z. P ines, R. Pratap, and

K. Ram anujam . (1 9 9 7 ). "Advanced Scout: Data Mining and K n ow ledg e D iscovery in NBA D ata.” D a t a M in in g a n d K n o w le d g e D iscovery, Vol. 1, No. 1, pp. 1 2 1 -1 2 5 .

B u ck, N. (D e ce m b e r 2000/January 2 001). “Eureka! Know ledge D iscovery.” S o ftw a r e M a g a z in e .

Chan, P. K , W . Phan, A. Prodromidis, and S. Stolfo. (1999). “D istributed Data Mining in Credit Card Fraud D etection .” IEEE In te llig e n t System s, Vol. 14, No. 6, pp. 6 7 -7 4 .

CRISP-DM. (2 0 1 3 ). “Cross-Industry Standard P rocess for Data Mining (CRISP-DM ).” www.the-modeling-agency.com/ crisp-dm.pdf (a c cessed February 2, 2013)-

D avenport, T . H. (2006, Janu ary). “Com peting o n Analytics.” H a r v a r d B u s in e s s R eview .

D elen , D ., R. Sharda, and P. Kumar. (2007). “Movie Forecast G u m : A W eb-based DSS for H ollyw ood M anagers.” D e c is io n S u p p o r t Systems, Vol. 43 , No. 4, pp. 11 5 1 -1 1 7 0 .

D elen, D , D. Cogdell, and N. Kasap. (2012). “A Comparative Analysis o f Data Mining Methods in Predicting NCAA Bow l O utcom es,” In t e r n a t io n a l J o u r n a l o f F o re ca s tin g , Vol. 28, pp. 543 -5 5 2 .

D elen , D . (2 0 0 9 ). “Analysis o f C ancer Data: A Data Mining A pp roach.” E x p e r t System s, Vol. 26, No. 1, pp. 1 0 0 -1 1 2 .

Delen, D., G. W alker, and A. Kadam. (2005). “Predicting Breast C ancer Survivability: A Com parison o f T h ree Data Mining Methods.” A r tific ia l In te llig en c e in M ed icin e, Vol 34, No. 2, pp. 113 -1 2 7 .

D unham , M. (2 0 0 3 ). D a t a M in in g : In tr o d u c to r y a n d A d v a n c e d T opics. U pper Saddle River, NJ: P rentice Hall.

EPIC. (2 0 1 3 ). Electronic Privacy Inform ation Center. “Case Against Je tB lu e Airways Corporation and Acxiom Corporation. ” h t t p : / / e p i c . o r g / p r i v a c y / a i r t r a v e l / j e t b l u e / f t c c o m p l a i n t . h t m l (accessed Janu ary 14, 2013)-

Fayyad, U., G. Piatetsky-Shapiro, and P. Smyth. (1 9 9 6 ). “From K now led g e D iscovery in D atabases.” A I M a g a z in e . Vol. 17, N o. 3 , pp- 3 7 -5 4 .

H offm an, T . (1 9 9 8 , D ecem b er 7 ). “B an k s T urn to IT to R eclaim Most Profitable C ustom ers.” C o m p u terw o r ld .

H offman, T. (19 9 9 , April 19). “Insurers Mine for Age- A ppropriate O ffering.” C o m p u terw o r ld .

Kohonen, T. (1982). “Self-Organized Formation o f Topologically Correct Feature Maps.” B io lo g ic a l C ybern etics, Vol. 43, No. 1,

pp. 59-69- Nemati, H. R., and C. D. B arko. (2001). “Issues in Organizational

D ata Mining: A Survey o f Current Practices.” J o u r n a l o f D a t a W a r eh o u sin g , Vol. 6, No. 1, pp. 2 5 -3 6 .

2 7 2 Part III • Predictive Analytics

North, M. (2 0 1 2 ). Data mining for th e masses. A Global T e x t P ro ject B o ok , h t t p s : / / s i t e s . g o o g l e . c o m / s i t e / d a t a m i n i n g f o r t h e m a s s e s (a c cessed Ju n e 2013)-

Quinlan, J . R. (1 9 8 6 ). “Induction o f D ecisio n T re e s.” M a c h in e L e a r n in g , V ol. 1, pp. 8 1 -1 0 6 .

SEMMA. (2 0 0 9 ). “SAS’s D ata Mining P rocess: Sam ple, Explore. Modify, M odel, Assess.” s a s . c o m / o f f i c e s / e u r o p e / u k / t e c h n o l o g i e s / a n a l y t i c s / d a t a m i n i n g / m i n e r / s e m m a .

h t m l (a c ce s se d August 2009). Sharda, R., a n d D. D elen . (2 0 0 6 ). “Predicting B ox-office

Success o f M otion Pictures w ith Neural N etw orks.’' E x p ert S ystem s w ith A p p lic a tio n s, Vol. 30, pp. 2 4 3 -2 5 4 .

Shultz, R. (2 0 0 4 , D ec em b e r 7 ). “Live from NCDM: T ales o f D atabase B u ffoon ery .” d i r e c t m a g . c o m / n e w s / n c d m - 1 2 - 0 7 - 0 4 / i n d e x . h t m l (a c cessed April 2009).

Skalak, D. (2 0 0 1 ). “Data M ining Blunders Exposed!” D B 2 M a g a z in e , V ol. 6, No. 2, pp. 1 0 -1 3 .

StatSoft. (2 0 0 6 ). “D ata Mining T ech n iq u es.” statsoft.com/ t e x t b o o k / s t d a t m i n . h t m l (a c ce s se d August 2006).

W ald, M. L. (2 0 0 4 ). “U.S. Calls R elease o f Je tB lu e Data Im proper.” T h e N ew Y ork Tim es, February 21, 2004.

W ilson, R., a n d R. Sharda. (1 9 9 4 ). “B ankruptcy Prediction Using N eural N etw orks.” D e c is io n S u p p o r t System s, Vol. 11, pp. 5 4 5 -5 5 7 .

Wright, C. (2 0 1 2 ). “Statistical Predictors o f March Madness: An Exam ination o f th e NCAA M en’s B asketball Cham pionship.” h t t p : / / e c o n o m i c s - f i l e s . p o m o n a . e d u / G a r y S m i t h /

Econl90/Wright%20March%20Madness%20Final%20 P a p e r . p d f (a c cessed February 2, 2013).

Zaima, A. (2 0 0 3 ). “T h e Five Myths o f Data M ining.” W h a t W orks: B e s t P r a c t ic e s i n B u s in e s s In t e llig e n c e a n d D a t a W a r eh o u sin g , Vol. 15, th e Data W arehou sing Institute, Chatsworth, CA, pp. 4 2-43-

Techniques for Predictive Modeling

l e a r n i n g o b j e c t i v e s L e a m t h e a d v a n ta g e s a n d d is a d v a n ta g e s

. U n d e r s ta n d t h e c o n c e p t “ SV M c o m p a r e d to ANN o f artificial neural n etw ork CANN) _ ^ ^ ^ ^ fo r m u la tio n

■ L e a r n t h e d iffe r e n t ty p e s o f ANN ^ ^ -n e a r e s t n e i g h b o r a lg o r ith m (&NN)

a r c h ite c tu r e s . L e a m t h e a d v a n ta g e s a n d d is a d v a n ta g e s ■ K n o w h o w le a r n in g h a p p e n s in A - ^ feNN c o r n p a re d to A NN a n d SVM ■ U n d e r s ta n d t h e c o n c e p t a n d s tru c tu re o f

s u p p o r t v e c t o r m a c h in e s (SVM)

a llo w s d e c i s i o n m a k e r s to e s tim a te s tr u c tu r e s c a p a b ilitie s / lim ita tio n s , r t h e p a s t. I n th is c h a p te r , a s a rtific ia l a n d a p p lic a tio n s o f t h e m o s t p o p u la p fe. n e a r e s t n e ig h b o r . T h e s e te c h n iq u e s n e u ra l n e t w o r k s , s u p p o r t v e c t o r ' r e „ re s s io n -ty p e p r e d i c t i o n p r o b le m s , a re c a p a b le o f a d d r e s s in g b o t h t e c h n iq u e s a r e n o t

O fte n , t h e y a r e a p p lie d to c o ™ P le ^ ? a d d itio n to t h e s e t h r e e ( t h a t a r e c o v e r e d m c a p a b le o f p r o d u c i n g s a tis fa c to r y r e s u ^ t e c h n i q u e s in c lu d e r e g r e s s io n (lin e a r o r th is c h a p t e r ) , o t h e r n o t a b l e P ^ t i o ^ » ^ p r e d ic t io n p r o b le m s ) , n a iv e B a y e s

^ X ^ S S X ^ r ^ e l i n ^ d iffe r e n t ty p e s o f d e c i s i o n tr e e s

( c o v e r e d in C h a p te r } ) .

6 a o p e n i n g V i g n e t t e : P r e d i c t i v e M o d e l i n g H e lp s B e t t e r U n d e r s t a n d a n d M a n a g e

C o m p l e x M e d i c a l P r o c e d u r e s 2 7 4 6 .2 Basic C o n cep ts of N eural N etw orks 2 7 7

6 ^ D e v e l o p i n g N e u r a l N e t w o r k - B a s e d S y s t e m s >■ 7Q 2 6 * 4 I l l u m i n a t i n g t h e B l a c k B o x o f A N N w i t h S e n s i t i v e A n a ly s is - 9 .

2 7 4 Part III • Predictive Analytics

6 .5 S u p p o rt 'V ector M a ch in e s 2 S 6 6.6 A P r o c e s s - B a s e d A p p r o a c h to t h e U se o f S V M 303 6.7 N e a re st N e ig h b o r M eth o d fo r P re d ic tio n 3 0 5

6.1 OPENING VIGNETTE: Predictive Modeling Helps Better Understand and Manage Complex Medical Procedures

H e a lth ca re has b e c o m e o n e o f th e m o st im p ortan t issues to h av e a d ire ct im p act o n q uality o f life in the U n ited States an d a ro u n d the w o rld . W h ile th e d e m a n d for h e a lth ca re serv ices is in creasin g b e c a u s e o f th e ag in g p o p u latio n , th e su p p ly sid e is h avin g p ro b lem s k eep in g u p w ith th e lev el a n d quality o f se rv ice . In o rd e r to c lo s e th e g a p , h e a lth ca re system s o u g h t to significantly im p ro v e th eir o p eratio n al effectiven ess a n d efficien cy. E ffectiven ess (d o in g th e right th in g, su ch a s d iag n o sin g a n d treatin g a c c u ra te ly ) an d efficien cy (d o in g it th e rig h t w ay , su ch as u sin g th e le a st am ou n t o f re so u rce s a n d tim e) a re th e tw o fu n d am en tal pillars u p o n w h ich th e h ealth care system c a n b e revived. A p rom isin g w a y to im p ro v e h e a lth ca re is to tak e a d v a n ­ ta g e o f p red ictiv e m o d elin g tech n iq u es alo n g w ith large and featu re-rich d ata so u rce s (tru e reflection s o f m ed ical a n d h e a lth ca re e x p e r ie n c e s ) to su p p o rt a c c u ra te and tim ely d ecisio n m aking.

A ccording to th e A m erican H eart A ssociation, card iovascu lar d isease (CVD ) is the u nderlying cau se for o v e r 2 0 p e rce n t o f deaths in th e U nited States. Since 1 9 0 0 , CVD has b e e n th e n u m b er-on e killer ev ery y e a r e x c e p t 1 9 1 8 , w h ich w a s th e y e a r o f th e great flu pand em ic. CVD kills m ore p eo p le than th e n e x t fou r leading ca u se s o f death s com b in ed : ca n ce r, ch ro n ic lo w er respiratory disease, accid en ts, an d d iab etes mellitus. O u t o f all CVD d eaths, m o re th an half a re attributed to co ro n a ry d iseases. N ot on ly d o e s CVD take a huge toll o n th e p erson al health and w ell-b ein g o f the popu lation , b u t it is also a great drain o n th e h ealth care reso u rces in the Unites States and elsew h ere in th e w orld. The d irect and indirect co sts asso ciated w ith CVD fo r a y ear are estim ated to b e in e x c e s s o f $ 5 0 0 billion. A co m m o n surgical p ro ced u re to c u re a large variant o f CVD is called c o ro ­ n ary artery b ypass grafting (CABG). E ven th ou gh th e co st o f a CABG su rgery d ep en d s on th e patient an d service p ro v id er-related factors, the averag e rate is b e tw e e n $ 5 0 ,0 0 0 and $ 1 0 0 ,0 0 0 in the U nited States. As an illustrative e x a m p le, D elen e t al. ( 2 0 1 2 ) carried out an analytics study w h ere th ey u sed various p red ictive m odeling m eth od s to p red ict the o u tco m e o f a CABG and applied an inform ation fusion-based sensitivity analysis o n the trained m odels to b etter und erstan d the im p ortan ce o f th e p rogn ostic factors. T h e main goal w a s to illustrate that predictive an d e xp lan ato ry analysis o f large an d feature-rich d ata sets p rovid es invaluable inform ation to m ak e m o re efficient an d effective decisions in h ealthcare.

RESEARCH METHOD

Figure 6 .1 sh ow s th e m od el d ev elo p m en t an d testing p ro cess u sed b y D elen e t al. T hey em p lo y ed fou r different typ es o f p red iction m o d els (artificial neural netw orks, su pp ort v e cto r m ach in es, a n d tw o typ es o f d ecision trees, C5 and CART), an d w e n t th rou gh a large n u m b er o f exp erim ental runs to calibrate the m od eling param eters for e a ch m od el type. O n ce th e m odels w e re d ev elo p ed , th e y w en t o n th e te x t data set. Finally, the trained m od els w e re e x p o s e d to a sensitivity analysis p ro ced u re w h e re th e contribution o f the variables w a s m easu red . T able 6.1 sh ow s th e test results for th e fou r different types o f p red iction m odels.

Chapter 6 • T e ch n iq u e s fo r Predictive M odeling 2 7 5

. |np U t ---------------------- > -<■-------------------------- P ro c e s s in g --------------------------- > —* ------------------ Output

FIG U R E 6.1 A Process Map for Training and Testing o f the Four Predictive Models.

2 7 6 Part III • Predictive Analytics

T A B L E 6.1 Prediction Accuracy Results fo r All Four M odel Types Based on the Test Data Set

M odel T yp e1

C onfusion M atrices2

Pos (1) Neg (0) Accuracy3 Sensitivity3 Specificity3

ANN Pos (1) 7 4 9 2 3 0 7 4 . 7 2 % 7 6 .5 1 % 7 2 .9 3 % Neg (0) 2 6 5 7 1 4

SV M Pos (1) Neg (0)

8 7 6

1 37

1 03

8 4 2 8 7 . 7 4 % 8 9 .4 8 % 8 6 .0 1 %

C5 Pos (1) 876. . . .103 7 9 . 6 2 % 8 0 .2 9 % 7 8 .9 6 % Neg (0) 137 8 4 2

CART Pos (1) 6 6 0 3 1 9 7 1 . 1 5 % 6 7 .4 2 % 7 4 .8 7 % Neg (0) 2 4 6 7 3 3

'Acronyms for model types: ANN: Artificial Neural Networks; SVM: Support Vector Machines; C5: A popular decision tree algorithm; CART: Classification and Regression Trees.

Prediction results for the test data samples are shown in a confusion matrix, where the rows represent the actuals and columns represent the predicted cases.

Accuracy, Sensitivity, and Specificity are the three performance measures that were used in comparing the

four prediction models.

RESULTS In this stu d y, th ey s h o w e d th e p o w e r o f d ata m ining in p red icting th e o u tc o m e an d in an alyzin g th e p ro g n o stic facto rs o f c o m p le x m ed ical p ro ce d u re s su ch a s CABG su rgery. T h e y s h o w e d th at u sin g a n u m b er o f p red ictio n m eth o d s (a s o p p o s e d to on ly o n e ) in a com p etitiv e exp erim en tal setting h as th e p o ten tial to p ro d u ce b e tte r p red ictive as w ell as e x p la n a to ry results. A m ong th e fou r m eth o d s th at th ey u se d , SVMs p ro d u ce d th e b est results w ith p red ictio n a c c u r a c y o f 8 8 p e rc e n t o n th e te st d ata sam p le. T h e inform ation fu sio n -b ased sensitivity an alysis re v e a le d th e ran k ed im p o rtan ce o f th e in d ep en d en t variables. S o m e o f th e to p v ariab les identified in this analysis h avin g to ov erlap w ith th e m o st im p ortan t variab les identified in p rev iou sly c o n d u cte d clinical an d b iolo gical stu d ies con firm s th e validity an d effectiv en ess o f th e p ro p o se d d ata m in­

ing m eth o d o lo g y . From th e m anagerial standpoint, clinical d ecision su p p ort system s th at u se th e

o u tco m e o f data mining studies (su ch as th e o n e s p resen ted in this ca se stu d y) are not m ean t to rep lace h ealth care m an agers a n d /o r m ed ical professionals. Rather, they intend to su p p ort th em in m aking accu rate an d timely d ecision s to optim ally allocate resou rces in o rd er to in crease th e quantity and quality o f m ed ical services. T h ere still is a lon g w ay to g o b efore w e can see th ese d ecision aids b ein g u sed exten sively in h ealth care p rac­ tices. A m ong others, th ere a re behavioral, ethical, an d political reaso n s for this resistance to ad option . M aybe th e n e e d and the go v ern m en t incentives for b etter h ealth care system s

will ex p e d ite the adoption.

QUESTIONS FO R TH E OPENING VIGNETTE

1 . W h y is it im portant to study m ed ical p ro ced u res? W h at is th e valu e in predicting

ou tcom es? 2. W h at factors d o y ou think a re th e m ost im portan t in b etter understanding and

m anaging healthcare? C onsider b oth m an agerial an d clinical a sp ects o f h ealth care.

Chapter 6 • T ech n iq u es for Predictive M odeling 2 7 7

3 . W h a t w o u ld b e t h e im p a c t o f p r e d ic tiv e m o d e lin g o n h e a lt h c a r e a n d m e d ic in e ? C an predictive m odeling re p la ce m edical o r m an agerial personnel?

4. W h a t w e r e t h e o u t c o m e s o f t h e stu d y ? W h o c a n u s e t h e s e resu lts? H o w c a n th e r e s u lts b e im p le m e n te d ?

5 . S e a r c h t h e I n t e r n e t t o l o c a t e tw o a d d itio n a l c a s e s w h e r e p r e d ic tiv e m o d e li n g is u s e d t o u n d e r s ta n d a n d m a n a g e c o m p l e x m e d ic a l p r o c e d u r e s .

WHAT WE CAN LEARN FROM THIS VIGNETTE

A s y o u w ill s e e in th is c h a p t e r , p r e d ic t iv e m o d e li n g t e c h n i q u e s c a n b e a p p l i e d t o a w i d e r a n g e o f p r o b l e m a r e a s , f r o m s ta n d a r d b u s i n e s s p r o b l e m s o f a s s e s s i n g c u s t o m e r n e e d s to u n d e r s t a n d i n g a n d e n h a n c i n g e f f i c i e n c y o f p r o d u c t i o n p r o c e s s e s t o im p r o v in g h e a l t h c a r e a n d m e d ic in e . T h i s v ig n e t t e illu s tr a te s a n i n n o v a t iv e a p p l i c a t i o n o f p r e d ic t iv e m o d e li n g t o b e t t e r p r e d ic t, u n d e r s ta n d , a n d m a n a g e c o r o n a r y b y p a s s g r a ftin g p r o ­ c e d u r e s . A s t h e r e s u lts in d ic a t e , t h e s e s o p h i s t i c a t e d p r e d i c t iv e m o d e li n g t e c h n i q u e s a r e c a p a b le o f p r e d i c t i n g a n d e x p l a i n i n g s u c h c o m p l e x p h e n o m e n a . E v i d e n c e - b a s e d m e d i c i n e is a r e la tiv e ly n e w te r m c o i n e d in t h e h e a l t h c a r e a r e n a , w h e r e t h e m a in id e a is t o d ig d e e p in to p a s t e x p e r i e n c e s t o d is c o v e r n e w a n d u s e fu l k n o w le d g e t o im p r o v e m e d ic a l a n d m a n a g e r i a l p r o c e d u r e s i n h e a lt h c a r e . A s w e a ll k n o w , h e a l t h c a r e n e e d s a ll t h e h e l p t h a t it c a n g e t . C o m p a r e d t o tr a d itio n a l r e s e a r c h , w h i c h is c li n i c a l a n d b i o lo g i c a l in n a tu r e , d a ta -d r iv e n s tu d ie s p r o v i d e a n o u t - o f - t h e - b o x v ie w t o m e d i c i n e a n d m a n a g e m e n t o f m e d ic a l s y s te m s .

Sources: D. Delen, A. Oztekin, and L. Tomak, “An Analytic Approach to Better Understanding and Management o f Coronary Surgeries," D ecision Support Systems, Vol. 52, No. 3, 2012, pp. 698-705; and American Heart Association, "Heart Disease and Stroke Statistics— 2012 Update." h eart.o rg (accessed February 2013).

6.2 B A S IC CO NCEPTS OF N E U R A L N E T W O R K S N e u ra l n e tw o r k s r e p r e s e n t a b r a in m e t a p h o r f o r in fo r m a tio n p r o c e s s in g . T h e s e m o d e ls a r e b io lo g ic a lly in s p ir e d r a t h e r th a n a n e x a c t r e p l ic a o f h o w t h e b r a in a c tu a lly fu n c tio n s . N e u ra l n e tw o r k s h a v e b e e n s h o w n to b e v e r y p r o m is in g s y s te m s in m a n y fo r e c a s tin g a n d b u s in e s s c la s s ific a tio n a p p lic a tio n s d u e to th e ir a b ility to “l e a r n ” fr o m t h e d a ta , th e ir n o n p a r a m e t r ic n a tu re ( i .e ., n o rig id a s s u m p tio n s ), a n d th e ir a b ility to g e n e r a liz e . N eural com puting r e fe r s to a p a tte r n - r e c o g n itio n m e t h o d o l o g y f o r m a c h in e le a r n in g . T h e r e s u ltin g m o d e l fr o m n e u r a l c o m p u t in g is o f te n c a ll e d a n artificial neural netw ork (ANN) o r a neural netw ork. N e u ra l n e t w o r k s h a v e b e e n u s e d in m a n y b u s in e s s a p p lic a tio n s f o r p attern recognition, f o r e c a s tin g , p r e d ic tio n , a n d c la s s ific a tio n . N e u ra l n e t w o r k c o m p u t in g is a k e y c o m p o n e n t o f a n y d a ta m in in g to o lk it. A p p lic a tio n s o f n e u r a l n e tw o r k s a b o u n d in fin a n c e , m a r k e tin g , m a n u fa c tu r in g , o p e r a t io n s , in fo r m a tio n s y s te m s , a n d s o o n . T h e r e f o r e , w e d e v o t e th is c h a p t e r t o d e v e l o p in g a b e t t e r u n d e r s ta n d in g o f n e u r a l n e t w o r k m o d e ls , m e th o d s , a n d a p p lic a tio n s .

T h e h u m a n b r a in p o s s e s s e s b e w ild e r in g c a p a b il it ie s f o r in fo r m a tio n p r o c e s s in g a n d p r o b l e m s o lv in g th a t m o d e r n c o m p u te r s c a n n o t c o m p e t e w ith in m a n y a s p e c ts . It h a s b e e n p o s tu la te d th a t a m o d e l o r a s y s te m th a t is e n lig h t e n e d a n d s u p p o r te d b y t h e re s u lts f r o m b r a in r e s e a r c h , w ith a s tr u c tu r e s im ila r t o th a t o f b i o lo g i c a l n e u r a l n e tw o r k s , c o u ld e x h ib i t s im ila r in te llig e n t fu n c tio n a lity . B a s e d o n th is b o t t o m - u p a p p r o a c h , ANN ( a ls o k n o w n a s c o n n e c tio n is t models, p a r a lle l d istrib u ted p ro cessin g m odels, n eu ro m o rp h ic systems, o r s im p ly n e u r a l networks) h a v e b e e n d e v e l o p e d a s b io lo g ic a ll y in s p ir e d a n d p la u s ib le m o d e ls f o r v a r io u s ta s k s .

2 7 8 Part III • Predictive Analytics

B iological neural netw orks a re co m p o se d o f m an y m assively in terconn ected neurons. E ach n eu ro n p o ssesses a xo n s an d d endrites, fingerlike projection s th at enable the n eu ro n to com m u n icate w ith its neighboring n eu ron s b y transm itting and receiving electrical and ch em ical signals. M ore o r less resem bling th e structure o f their biological cou nterp arts, ANN are co m p o se d o f in tercon n ected , sim ple p ro cessin g elem ents called artificial neurons. W h en p ro cessin g information, th e p ro cessin g elem ents in an ANN op erate con curren tly and collectively, similar to b iological neurons. ANN p o ssess som e desirable traits similar to th o se o f biological neural n etw ork s, su ch as the abilities to learn, to self-organize, an d to su p p o rt fault toleran ce.

C om in g alo n g a w ind in g jo u rn ey, ANN h a v e b e e n in vestigated b y re se a rch e rs fo r m o re th an h alf a cen tu ry . T h e form al stu d y o f ANN b e g a n w ith th e p io n eerin g w o rk o f M cC ulloch an d Pitts in 19 4 3 - In sp ired b y th e results o f b io lo g ical exp erim en ts an d ob serv atio n s, M cC ulloch an d Pitts ( 1 9 4 3 ) in tro d u ced a sim ple m o d el o f a b in ary artificial n e u ro n th at ca p tu re d so m e o f th e fu n ction s o f b iological n eu ro n s. Using in fo rm atio n -p ro cessin g m ach in es to m o d el th e b rain , M cC ulloch an d Pitts built their n eural n etw o rk m o d el using a larg e n u m b er o f in te rco n n e cte d artificial b in ary n eu ron s. F ro m th ese b eg in n in gs, n eu ral n etw o rk re s e a rc h b e c a m e q uite p o p u la r in th e late 1 9 5 0 s an d e arly 1 9 6 0 s. After a th o ro u g h an alysis o f an early n eural n etw ork m o d el (ca lle d th e p erce p tro n , w h ich u se d n o h id d en la y e r) as w ell a s a p essim istic ev alu a­ tion o f the re s e a rc h p oten tial b y Minsky a n d P a p e rt in 1 9 6 9 , in terest in n eural netw orks

dim inished. During th e past tw o d ecad es, th ere has b een a n excitin g resu rgen ce in ANN studies

d ue to the introduction o f n e w n etw ork to p o logies, n e w activation functions, and n ew learning algorithm s, as w ell as p rogress in n eu ro scien ce and cognitive scien ce. A dvances in th eo ry an d m eth o d ology h av e o v e rco m e m an y o f th e o b stacles that hin dered neural n etw ork research a few d ecad es ag o . E v id en ced b y th e ap p ealin g results o f n um erou s studies, neural n etw orks are gaining in a cce p ta n ce a n d popularity. In addition, th e desir­ ab le features in neural inform ation p rocessin g m ak e n eural netw orks attractive for solving co m p le x p roblem s. ANN h ave b e e n ap plied to n u m erou s co m p le x p rob lem s in a variety o f ap plication settings. T h e su ccessfu l u se o f neural n etw ork applications has inspired ren ew ed interest from industry and business.

B io lo g ic a l a n d A r t if ic ia l N e u ra l N e tw o rk s

T h e h um an brain is co m p o se d o f sp ecial cells called n eu ro n s. T h e se cells d o n ot die an d rep len ish w h e n a p erso n is injured (all o th e r cells rep ro d u ce to re p la ce th em ­ selves an d th en d ie). This p h e n o m e n o n m ay e xp la in w h y hum ans retain inform ation fo r a n e x te n d e d p erio d o f tim e an d start to lo se it w h e n th e y g e t old— as th e brain cells gradually start to die. Inform ation sto rag e sp an s sets o f n eu ron s. T h e b rain has an yw h ere from 5 0 billion to 1 5 0 billion n eu ron s, o f w h ich th e re a re m o re th an 1 0 0 different kinds. N eu ron s are p artition ed into g ro u p s called netw orks. E a ch n etw ork con tain s several th ou san d highly in terco n n ected n eu ron s. Thus, th e brain can b e v iew ed a s a co llectio n

o f n eural netw orks. T h e ability to learn an d to re a ct to ch a n g e s in o u r en viron m en t req u ires intelligence.

T h e b rain an d the cen tral n erv o u s sy stem co n tro l thinking a n d intelligent behavior. P eo p le w h o suffer b rain d am ag e have difficulty learning an d reactin g to ch anging en viron m en ts. E ven so , u n d am ag ed p arts o f th e b rain c a n often co m p e n sa te w ith n ew

learning. A p ortion o f a n etw ork co m p o se d o f tw o cells is sh o w n in Figure 6 .2 . T h e cell itself

includes a nucleus (th e cen tral p rocessin g portion o f th e n eu ro n ). T o th e left o f cell 1, th e dendrites p rovid e input signals to th e cell. T o th e right, th e a x o n sen ds ou tput signals

Chapter 6 • T ech n iq u es for Predictive M odeling 2 7 9

FIGURE 6.2 Po rtio n o f a B io lo g ic a l N eural N etw o rk: T w o Inte rco nn ecte d Cells/N eu ro ns.

to cell 2 via th e a x o n terminals. T h ese a x o n term inals m erge w ith th e dendrites o f cell 2. Signals c a n b e transm itted u n ch an ged , o r they can b e altered by synapses. A syn ap se is ab le to in crease o r d e cre a se th e strength o f the co n n ectio n b e tw e e n n eu ron s a n d cau se excitation o r inhibition o f a su bsequ en t n euron. This is h o w inform ation is stored in the neural netw orks.

A n ANN e m u la te s a b io logical n eu ral n etw o rk . N eural co m p u tin g actu ally u se s a v e ry lim ited se t o f c o n c e p ts from b io lo g ical n eu ral sy stem s (s e e T e c h n o lo g y Insights 6 .1 ) . It is m o re o f a n a n a lo g y to th e h u m an brain th an a n a c c u ra te m o d el o f it. Neural c o n c e p ts usually a re im p lem en ted a s so ftw are sim ulation s o f th e m assively parallel p ro c e s s e s in v o lv ed in p ro ce ssin g in te rco n n e cte d elem en ts (a lso called artificial n e u ­ ro n s, o r n e u r o d e s ) in a n etw o rk a rch ite ctu re . T h e artificial n e u ro n re c e iv e s inputs a n a lo g o u s to th e e le c tro c h e m ic a l im pu lses th at d en d rites o f b io lo g ical n e u ro n s re ce iv e from o th e r n e u ro n s. T h e o u tp u t o f th e artificial n e u ro n co rre s p o n d s to sign als sen t from a b io lo g ical n e u ro n o v e r its a x o n . T h e se artificial signals c a n b e c h a n g e d b y w eigh ts in a m a n n e r sim ilar to th e p h ysical ch a n g e s th a t o c c u r in the sy n a p se s (s e e Fig u re 6.3 )-

Several ANN paradigm s h av e b een p ro p o sed for ap plications in a variety o f p ro b ­ lem d om ains. P erh ap s the easiest w ay to differentiate am on g the various n eural m odels is o n th e basis o f h o w they structurally em ulate th e h um an brain, th e w a y th e y p ro cess inform ation, an d h o w th ey learn to p erform their design ated tasks.

FIG U R E 6.3 Processing Information in an Artificial Neuron.

2 8 0 Part III • Predictive Analytics

TECHNOLOGY IN SIGH TS 6 .1 T h e R elatio n sh ip B etw e e n B io lo g ic a l and A rtificial N eural N etw orks

T h e follow ing list show s som e o f th e relationships b e tw e e n biological and artificial networks.

Biological Artificial

Soma Node

Dendrites Input

Axon Output

Synapse Weight

Slow Fast

Many neurons (109) Few neurons (a dozen to hundreds of thousands)

Sources: L. Medsker and J. Liebowitz, Design and Development o f Expert Systems and Neural Networks, Macmillan, New York, 1994, p. 163; and F. Zahedi, Intelligent Systems fo r Business: Expert Systems with Neural Networks, Wadsworth, Belmont, CA, 1993-

B e ca u se th ey are biologically inspired, th e m ain p ro cessin g elem ents o f a neural n etw ork are individual neurons, an alog ou s to th e b rain ’s neurons. T h ese artificial n eurons receive the inform ation from o th er n eurons o r extern al input stimuli, p erform a transfor­ m ation o n th e inputs, an d th en p ass o n th e transform ed inform ation to o th er n eurons or extern al outputs. This is similar to h o w it is currently th ou ght th at th e h um an brain works. Passing inform ation from n eu ron to n eu ro n c a n b e th ou ght o f a s a w ay to activate, o r trigger, a resp o n se from certain n eu ro n s b ased o n th e inform ation o r stimulus received.

H o w inform ation is p ro ce sse d b y a neural n etw ork is inherently a function o f its structure. Neural n etw orks c a n h av e o n e o r m ore layers o f neurons. T h ese n eurons can b e highly o r fully in tercon n ected , o r only certain layers can b e co n n ected . C onnections b e tw e e n n eurons have an asso ciated w eight. In e sse n ce , the “k n o w led g e” p o ssessed by the n etw ork is en cap su lated in th ese in terconn ection w eights. E a ch n eu ron calcu lates a w eigh ted su m o f the in com ing n eu ron values, transform s this input, an d p asses o n its neural valu e as the input to su bsequ en t neurons. Typically, although n ot alw ays, this in p ut/ou tp u t transform ation p ro cess at th e individual n eu ro n level is p erform ed in a n o n ­ linear fashion.

A p p lic a tio n C a s e 6 .1 p r o v id e s a n in te r e s tin g e x a m p l e o f t h e u s e o f n e u r a l n e tw o r k s

a s a p r e d ic t io n t o o l in t h e m in in g in d u stry .

Application Case 6.1 Neural N etw o rk s A re Helping to S a v e Lives in the In the m ining industry, m ost o f th e underground injuries and fatalities are d u e to ro ck falls (i.e ., fall o f hanging w all/ro o f). T h e m eth od that has b e e n u se d for m an y years in th e m ines w h e n determ in­ ing the integrity o f th e hanging wall is to tap the hanging wall w7ith a sou nd in g b a r an d listen to the so u n d em itted . An exp e rie n ce d m iner c a n differenti­ ate a n in tact/so lid hanging wall from a d e ta ch e d /

M ining Industry lo o se hanging w all b y th e so u n d that is em itted. This m eth od is subjective. T h e C ouncil for Scientific and Industrial R esearch (CSIR) in South Africa has d evel­ o p e d a d ev ice th at assists an y m iner in m aking an objective d ecision w h e n determ ining th e integrity o f the hanging wall. A trained neural n etw ork m od el is em b ed d ed into th e device. T h e d evice th en reco rd s the so u n d em itted w h e n a hanging wall is tapped.

Chapter 6 • T ech n iq u es for Predictive M odeling 281

T h e sou nd is th en p re p ro ce sse d b efo re b ein g input into a train ed neural n etw ork m odel, an d th e trained m o d el classifies th e hanging wall a s either intact o r d etach ed .

Mr. T e b o h o Nyareli, w orking as a research e n gin eer at CSIR, w h o holds a m aster’s d eg ree in electro n ic en gin eerin g fro m th e University o f Cape T o w n in South Africa, u sed NeuroSolutions, a p op u lar artificial neural n etw ork m odeling softw are d ev elo p ed b y N euroD im ensions, In c., to d evelop th e classification type p red iction m odels. T h e mul­ tilayer p ercep tro n -ty p e ANN arch itectu re th at he built ach iev ed b etter th an 7 0 p e rce n t prediction

a c c u ra c y o n th e h old -o u t sam ple. Currently, the p rototyp e system is und ergoing a final set o f tests b efore deployin g it as a d ecision aid, follow ed by the com m ercialization p hase. T h e follow ing figure sh ow s a sn apshot o f NeuroSolution’s m o d el building platform.

Q u e s t i o n s f o r D i s c u s s i o n

1. H o w did neural n etw orks help sav e lives in the mining industry?

2. W h at w e re th e ch allen ges, the p ro p o se d solu­ tion, and th e ob tain ed results?

Source: NeuroSolutions customer success story, n e u r o s o l u t i o n s . c o m / r e s o u r c e s / n y a r e l i . h t m l (accessed February 2013).

Elements of A N N

A neural n etw ork is co m p o se d o f p rocessin g elem ents that a re o rg an ized in different w ays to form th e n etw ork ’s structure. T h e b asic p rocessin g unit is th e n eu ro n . A n um ber o f n eurons are th en organ ized into a netw ork. N eurons c a n b e organ ized in a n um ber o f different w ay s; th ese various n etw ork patterns are referred to as topologies. O n e p opu lar ap p ro ach , k n o w n as the feed forw ard -b ack p rop agation p arad igm (o r sim ply backpropa- gation), allow s all n eurons to link th e ou tput in o n e layer to th e input o f the n e x t layer, but it d o e s n o t allow an y feed b ack linkage (H aykin, 2 0 0 9 ). B ack p ro p ag atio n is the m ost co m m on ly u se d n etw ork paradigm .

P R O C E S S IN G E L E M E N T S T h e p ro cessin g elem ents (P E ) o f a n ANN a re artificial n eu ­ rons. E a ch n e u ro n receiv es inputs, p ro cesses them , an d delivers a single ou tp ut, as sh ow n in Figure 6 .3 . T h e input c a n b e raw input data o r th e ou tput o f o th er p rocessin g elem ents. The o u tp u t c a n b e th e final result (e .g ., 1 m ean s y es, 0 m ean s n o ), o r it c a n b e input to o th er neurons.

2 8 2 Part III • P redictive Analytics

FIGURE 6.4 Neural Network with One Hidden Layer.

N FTW O R K STRU CTU R E E a c h A N N is c o m p o s e d o f a c o l l e c t i o n o f n e u r o n s th a t a re

to th e ou tput layer. T h e m o st co m m o n interpretation of the hidden 1 y e x t r a c t i o n m e c h a n is m ; th a t is, t h e h id d e n la y e r c o n v e r t s t h e o r ig m a l i n p u t , m t h e p r o b -

iem * —ai r r * t io n s a t t h e s a m e tim e . T h is p a ra lle l p ro ce ssm g r e s e m b le s t h e w a y t h e b r a in w o r k , it differs from the serial p rocessin g o f con ven tio n al com puting.

N e tw o rk In fo r m a t io n P ro ce ssin g O n c e t h e s tru c tu re o f a n e u r a l n e t w o r k is d e te r m in e d , in fo r m a tio n c a n b e p r o c e s s e d . W e n o w p r e s e n t t h e m a jo r c o n c e p t s r e l a t e d to n e t w o r k in fo r m a tio n p r o c e s s in g .

,N PU T E a c h in p u t c o r r e s p o n d s t o a s in g le attrib u te^ F o r e x a m p l e , i f t h e ^ to

wr- ?-=** - “ sag m eaningful inputs from sym bolic d ata o r to scale th e data.

Chapter 6 • T ech n iq u es for Predictive M odeling 2 8 3

values to th e ou tp ut, su ch as 1 for “y e s ” an d 0 for “n o .” T h e p u rp ose o f th e n etw ork is to com p u te the ou tp ut valu es. Often, p ostp ro cessin g o f the ou tp u t is required b e ca u s e som e netw orks use tw o ou tp uts: o n e fo r “y e s” an d an o th er for “n o .” It is co m m o n to ro u n d the

outputs to the n earest 0 o r 1.

CONNECTION WEIGHTS Connection weights a re the k ey elem ents o f a n ANN. T hey exp ress the relative strength (o r m athem atical v alu e) o f th e input data o r th e m an y c o n ­ n ection s th at transfer d ata from layer to layer. In o th er w ord s, w eights e xp ress th e relative im portan ce o f e a c h input to a p rocessin g elem en t an d , ultimately, the output. W eights are cru cial in that th ey sto re learn ed p atterns o f information. It is th rou gh rep eated adjust­ m ents o f w eights th at a n etw ork learns.

SUMMATION FUNCTION T h e summation function co m p u tes the w eigh ted su m s o f all the input elem ents en tering e a c h p rocessin g elem ent. A sum m ation function multiplies each input valu e b y its w eight an d totals the values for a w eig h ted sum Y. T h e formula for n inputs in o n e p rocessin g elem en t (s e e Figure 6 .5 a ) is:

n

> i = l

For th e yth n e u ro n o f several p ro cessin g n eu ro n s in a lay er (s e e Figure 6 .5 b ), the

form ula is: n

Yj = v a . - w ; , 2=1

TR A N S F O R M A T IO N (T R A N S F E R ) FU N C TIO N T h e s u m m a tio n f u n c tio n c o m p u t e s t h e in te r­ n a l s tim u la tio n , o r a c tiv a tio n le v e l, o f t h e n e u r o n . B a s e d o n th is le v e l, t h e n e u r o n m a y o r m a y n o t p r o d u c e a n o u tp u t. T h e r e la tio n s h ip b e t w e e n t h e in te r n a l a c tiv a tio n le v e l a n d t h e o u tp u t c a n b e l in e a r o r n o n lin e a r . T h e r e la tio n s h ip is e x p r e s s e d b y o n e o f s e v e r a l ty p e s o f tra n sfo rm a tio n (tra n sfe r) fu n ctio n s. T h e tr a n s fo r m a tio n f u n c t i o n c o m b in e s ( i .e ., a d d s u p ) t h e in p u ts c o m i n g in to a n e u r o n fr o m o t h e r n e u r o n s / s o u r c e s a n d t h e n p r o ­ d u c e s a n o u tp u t b a s e d o n t h e tr a n s fo r m a tio n fu n c tio n . S e le c t io n o f t h e s p e c if ic fu n c tio n a ffe c ts th e n e t w o r k ’s o p e r a tio n . T h e sig m o id (lo g ic a l activ atio n ) fu n c tio n ( o r s i g m o i d t r a n s f e r f u n c t i o n ) is a n 5 - s h a p e d tr a n s fe r f u n c tio n in t h e r a n g e o f 0 to 1, a n d it is a p o p u ­

la r a s w e ll a s u s e fu l n o n l i n e a r tr a n s fe r fu n c tio n :

2 8 4 Part III ♦ P redictive Analytics

( 1 + e - r )

w here YT is th e transformed (i.e., normalized) value o f Y ( se e Figure 6.6). The transform ation modifies th e ou tput levels to reason ab le v alu es t o p e t o t

b e tw e e n 0 an d 1). This transform ation is p erform ed b e fo re th e ou tput re a ch e s the n e x t level. W ithou t su ch a transform ation, the valu e o f th e ou tp u t b e co m e s very large especially w h en th ere are several layers o f neurons. S om etim es a th reshold valu e is used instead o f a transform ation ftm ction. A th r e s h o ld valu e is a hurdle valu e o f a n eu ro n to trigger th e n e x t level o f neurons. If an ou tp u t valu e is sm aller th an the th resh old value, it will n o t b e passed to the n e x t lev el o f neurons. F o r exam p le, any valu e o f 0 .5 o r less b e co m e s B, and an y valu e a b o v e 0 .5 b e co m e s 1. A transform ation can o c c u r at the output o f e a ch p ro cessin g elem ent, o r it can b e p erfo rm ed on ly at th e final

o u tp u t n o d e s .

H ID D E N L A Y E R S C o m p le x p r a c tic a l a p p lic a tio n s r e q u ir e o n e o r m o r e h id d e n la Ye i -s b e t w e e n t h e in p u t a n d o u tp u t n e u r o n s a n d a c o r r e s p o n d i n g ly la r g e n u m b e r o f w e ig h ts . M a n v c o m m e r c ia l A NN in c lu d e th r e e a n d s o m e t im e s u p to fiv e la y e r s , w i t h e a c h c o n ta in i n s 1 0 to 1 ,0 0 0 p r o c e s s i n g e le m e n ts . S o m e e x p e r im e n t a l ANN u s e m illio n s o f p r o c e s s ­ in g e le m e n ts . B e c a u s e e a c h la y e r i n c r e a s e s t h e tr a in in g e f f o r t e x p o n e n t ia l l y a n d a ls o i n c r e a s e s t h e c o m p u t a t io n r e q u ir e d , t h e u s e o f m o r e t h a n t h r e e h id d e n la y e r s

m o s t c o m m e r c i a l s y s te m s .

Neural Network Architectures T h ere a re several neural n etw ork arch itectures (fo r specifics o f m odels a n d /o r algo­ rithms, see H aykin, 2 0 0 9 ). T h e m ost c o m m o n o n e s in clude feed forw ard (m ultilayer

Sum m ation function: Y = 3 ( 0 .2 ) + 1 ( 0 .4 ] + 2 ( 0 .1 ) 1 . 2

T ra n s fe r function: YT = 1/ (1 + e 1'a ) = 0 . 7 7

F IG U R E 6.6 Example o f ANN Transfer Function.

Chapter 6 • T ech n iq u es fo r Predictive M odeling 2 8 5

Input 2

Input

Input 3

Input 4

Output 1

Output 2

H indicates a "hidden” neuron without a ta rge t output

FIGURE 6.7 A Recurrent Neural Network Architecture.

p ercep tro n w ith b ack p ro p ag atio n ), associative m em o ry , recu rren t netw orks, K oh onen's self-organizing feature m ap s, an d Hopfield netw orks. T h e g en eric arch itectu re o f a feed forw ard n etw o rk arch itectu re is sh ow n in Figure 6.4, w h ere th e inform ation flows unidirectionally from input layer to hid den layers to ou tput layer. In con trast, Figure 6 .7 sh ow s a pictorial represen tation o f a recu rren t neural n etw ork arch itectu re, w h ere the co n n ection s b e tw e e n th e layers are n ot unidirectional; rather, th ere are m an y con n ectio n s in every d irection b etw een th e layers and n eu ron s, creatin g a co m p le x co n n e ctio n struc­ ture. M any e x p e rts b elieve this b etter m im ics the w a y biological n eurons a re structured in th e h um an brain.

K O H O N E N 'S S E L F -O R G A N IZ IN G FE A T U R E M A P S First in trodu ced b y th e Finnish p rofes­ sor T e u v o K o h o n en , K ohonen’s self-organizing feature m aps (K o h o n e n n etw orks o r SOM, in sh o rt) p rovid e a w ay to rep resen t m ultidimensional data in m u ch low er d im ensional sp a c e s , usually o n e o r tw o dim ensions. O n e o f th e m o st interesting asp ects o f SOM is that th ey learn to classify d ata w ithout supervision (i.e ., th ere is n o ou tp u t v e c ­ tor). R em em b er, in su p eiv ised learning tech n iq u es, su ch as b ack p rop agatio n , the training data consists o f v e c to r pairs— an input v e c to r and a target v ecto r. B e c a u s e o f its self­ organizing capability, SOM a re com m o n ly u sed for clustering tasks w h e re a g ro u p of cases a re assign ed an arbitrary n um ber o f naturals grou p s. Figure 6 .8 a illustrates a very small K o h o n en n etw ork o f 4 x 4 n o d es co n n ected to th e input layer (w ith th ree inputs), rep resen ting a tw o-d im en sion al v ector.

H O P F IE LD N E T W O R K S T h e H opfield n etw ork is an o th er interesting n eural n etw ork architecture, first in trodu ced b y Jo h n Hopfield (1 9 8 2 ). Hopfield d em on strated in a series o f research articles in the early 1 9 8 0 s h o w highly in terco n n ected n etw o rk s o f nonlinear n eurons c a n b e extrem ely effective in solving co m p le x com p utational p rob lem s. T h ese netw orks w e re sh o w n to p rovid e n ovel an d quick solutions to a family o f p ro b lem s stated in term s o f a d esired objective subject to a n um ber o f constraints (i.e ., con straint optim i­ zation p rob lem s). O n e o f th e m ajor ad vantages o f Hopfield. neural netw orks is the fact that their structure c a n b e realized o n an electro n ic circuit b oard , possibly o n a VLSI (very large-scale integration) circuit, to b e u sed a s a n online so lver w ith a parallel-distributed

2 8 6 P a r t lll • P r e d ic tiv e A n a ly tic s

[a] Kohonen N etw ork (S O M ] [b] Hopfield Netw ork

FIGURE 6.8 Graphical Depiction of Kohonen and Hopfield ANN Structures.

p ro cess. Architecturally, a g en eral H opfield n etw o rk is rep resen ted as a single large layer o f n eu ro n s with total interconnectivity; th at is, e a c h n eu ron is co n n e cte d to ev ery oth er n eu ron within the n etw ork (s e e Figure 6 .8 b ).

Ultimately, th e arch itectu re o f a neural n etw ork m o d el is driven b y th e task it is in tended to carry out. F o r instance, neural n etw ork m odels h av e b e e n u sed as classifiers, as forecastin g tools, a s cu sto m er segm en tation m ech an ism s, an d as gen eral optim izers. As sh o w n later in this ch ap ter, neural n etw ork classifiers are typically multilayer m od ­ els in w h ich inform ation is p assed from o n e layer to th e n e x t, with th e ultim ate goal of m apping an input to th e n etw ork to a specific categ o ry , as identified b y an ou tp ut o f the netw ork. A neural m od el u sed as an optim izer, in contrast, c a n b e a single layer o f n eu ­ ron s, highly in terco n n ected , an d c a n co m p u te n e u ro n values iteratively until th e m od el co n v erg es to a stable state. This stable state rep resen ts an optim al solution to th e prob lem

u n d e r analysis. Application C ase 6 .2 sum m arizes the u se o f predictive m odeling (e .g ., neural n et­

w ork s) in addressing several ch an gin g prob lem s in th e electric p o w e r industry.

Application Case 6.2 P re d ictiv e M o d elin g Is Po w erin g the P o w e r G enerators T h e electrical p o w e r industry p rod u ces and delivers electric e n erg y (electricity o r p o w er) to b oth residen­ tial and business custom ers, w h erever and w h en ­ ev er they n e e d it. Electricity can b e gen erated from a m ultitude o f sources. Most often, electricity is p ro­ duced a t a p o w e r station using electrom echanical gen erators that are driven b y h eat engines fueled b y ch em ical com bustion (b y burning coal, petroleum , o r natural g a s) o r nuclear fusion (b y a nuclear reactor). G eneration o f electricity can also b e accom plished by o th er m ean s, such as kinetic en ergy (through fall­ ing/flow ing w ater o r w ind that activates turbines),

solar en erg y (through the en ergy emitted b y sun, either light o r h eat), o r geotherm al en ergy (through th e steam o r h ot w ater com ing from d e e p layers of th e earth). O n ce gen erated , the electric en ergy is dis­ tributed th rou gh a p o w er grid infrastructure.

E ven th ou gh so m e en ergy-gen eration m ethods are favored o v e r others, all forms o f electricity g en ­ eration h ave positive and negative asp ects. Som e are environm entally favored but are econ om ically unjus­ tifiable; others are econ om ically su perior b u t envi­ ronm entally prohibitive. In a m arket e co n o m y , th e op tions w ith few er overall co sts a re generally ch o sen

Chapter 6 • T ech n iq u es for Predictive M odeling 2 8 7

ab o v e all o th e r so u rces. It is n ot clear y et w hich form can best m e e t th e n ecessary d em an d for electricity' w ithout p erm anently dam aging the environm ent. Current tren d s indicate th at increasing the shares of ren ew ab le e n erg y and distributed gen eration from m ixed so u rce s h as the p rom ise o f red u cin g /b alan c­ ing en viron m en tal and e co n o m ic risks.

T h e electrical p ow er industry is a highly regulated, com p lex business endeavor. There are four distinct, roles that com p an ies ch o o se to participate in: pow er producers, transmitters, distributers, and retailers. Connecting all o f the producers to all o f the customers is accom plished through a com p lex structure, called the p o w er grid. Although all aspects o f the electricity industry are witnessing stiff competition, p ow er gen­ erators are perhaps the on es getting the lion’s share o f it. T o b e competitive, producers o f p o w er n eed to maximize th e u se o f their variety o f resources by mak­ ing the right decisions at the right rime.

StatSoft, o n e o f th e fastest g row in g p rovid­ ers o f cu stom ized analytics solutions, d ev elo p ed integrated d ecision su p p o rt tools for p o w e r g en ­ erators. Leveraging the data that co m e s from the prod uction p ro ce ss, th ese d ata m ining-driven soft­ w are tools h elp tech n ician s an d m an agers rapidly optim ize th e p ro cess param eters m axim ize the p o w e r ou tp u t w hile minimizing th e risk o f ad verse effects. Follow in g are a few e x am p les o f w h at these ad v an ced analytics tools, w h ich in clude ANN and SVM, can acco m p lish fo r p o w e r gen erators.

• O ptim ize O p era tio n P a ra m eters P ro b lem : A coal-bu rn in g 3 0 0 MW multi­ c y clo n e unit required optim ization for con sis­ tent high flam e tem peratu res to avoid forming slag a n d burning e x c e s s fuel oil. S o lu tio n : Using StatSoft’s predictive m o d el­ ing tools (alo n g with 12 m onths o f 3-m inute historical d ata), optim ized con trol p aram eter settings for stoichiom etric ratios, co al flows, p rim ary air, tertiary air, and split seco n d ary air d am p er flows w e re identified an d im plem ented. R esu lts: After optim izing th e con trol p aram ­ eters, flam e tem peratu res sh o w ed strong resp o n ses, resulting in cle a n e r com b u stion for h igh er an d m ore stable flame tem peratures.

• P re d ic t P ro b lem s B e fo re T hey H a p p en P ro b lem : A 4 0 0 M W coal-fired DRB-4Z b urner required optim ization fo r con sisten t and ro b u st lo w N O x op eration s to avoid excu rsio n s

and exp en siv e dow ntim e. Identify ro o t cau ses o f am m onia slip in a selective noncatalytic red u ction p ro ce ss fo r N O x reduction. S o lu tio n : A p p ly predictive analytics m eth od ­ olog ies (a lo n g w ith historical p ro ce ss d ata) to p red ict and co n tro l variability; then target p ro ­ cesses for b etter p erfo rm an ce, th ereb y red u c­ ing b oth a v e ra g e N O x an d variability. R esu lts: O ptim ized settings fo r com b in ations o f co n tro l p aram eters resulted in consistently lo w er N O x em ission s w ith less variability (an d n o e x cu rsio n s) o v e r con tin u ed op eratio n s at lo w load, including predicting failures o r u n e x ­ p e c te d m ain ten an ce issues.

• R e d u c e E m issio n (NO x, CO) P ro b lem : W hile N O x em issions for higher loads w e re w ithin a ccep tab le ran ges, a 4 0 0 MW coal-fired D R B-4Z b urner w as n o t optim ized fo r lo w -N O x op eratio n s u n d e r low load ( 5 0 - 1 7 5 MW). S o lu tio n : Using data-driven predictive m od ­ eling tech n o lo g ies w ith historical data, op ti­ m ized p aram eter settings for ch a n g e s to airflow w e re identified, resulting in a set o f specific, ach ievab le input p aram eter ran ges that w ere easily im plem en ted into th e existing DCS (digi­ tal con tro l system ). R esu lts: After optim ization, N O x em issions u nd er low -lo ad op eration s w e re co m p arab le to N O x em issions u n d er high er loads.

As these sp ecific exam p les illustrate, th ere are n um erou s opportunities for ad v a n ce d analytics to m ak e a significant contribution to th e p o w e r indus­ try. Using data an d predictive m odels cou ld help d ecision m akers g e t th e b est efficiency from their p rod u ction system w hile minimizing th e im pact on the environm ent.

Q u e s t i o n s f o r D i s c u s s i o n

1. W h at a re th e k ey environm ental co n ce rn s in the electric p o w e r industry?

2. W h at are th e m ain application areas for pred ic­ tive m odeling in th e electric p o w e r industry?

3. H o w w a s p red ictive m od eling u sed to address a variety o f p ro b lem s in the electric p o w er industry?

Source: StatSoft, Success Stories, power.statsoft.com/files/ statsoft-pow ersolu tions.p df (accessed February 2013).

2 8 8 Part III • Predictive Analytics

SECTION 6 .2 REVIEW QUESTIONS

1 . W h at is an ANN? 2 . Explain the follow ing term s: n eu ro n , a x o n , an d synapse. 3 . H o w d o w eights function in an ANN? 4 . W hat is the role o f th e su m m ation an d transform ation function? 5 . W h at a re the m o st co m m o n ANN architectures? H ow d o th e y differ from e a c h other?

6.3 D E V EL O P IN G N E U R A L N E T W O R K - B A S E D S Y S T E M S A lthough th e d evelop m en t p ro cess o f ANN is sim ilar to the structured design m e th o d o lo ­ gies o f traditional co m p u ter-b ased inform ation system s, so m e p h ases are unique o r have so m e uniqu e asp ects. In th e p ro cess d escrib ed h e re , w e assu m e that th e prelim inary steps o f system d evelop m en t, su ch as determ ining inform ation requirem ents, con du ctin g a fea­ sibility analysis, an d gaining a ch am p io n in top m an ag em en t for th e project, h ave b e e n co m p leted successfully. Such steps a re g en eric to an y inform ation system .

As sh o w n in Figure 6 .9 , the d evelop m en t p ro ce ss for an ANN application includes nine steps. In step 1, the d ata to b e u sed fo r training and testing th e n etw ork are co l­ lected. Im portant consid erations are that th e particular problem is am en ab le to a neural n etw ork solution an d th at ad eq u ate data exist a n d c a n b e obtained. In step 2, training d ata m ust b e identified, and a p lan m ust b e m ad e fo r testing the p erform an ce o f the

n etw ork. , In steps 3 and 4 , a n etw ork architecture an d a learning m ethod are selected, m e

availability o f a particular d evelop m en t to ol o r th e capabilities o f the d evelop m en t person ­ nel m ay determ ine the type o f neural netw ork to b e constructed. Also, certain problem types h ave dem on strated high su ccess rates w ith certain configurations (e.g ., multilayer feedforw ard neural n etw orks for b ank ru p tcy p red iction [Altman (1 9 6 8 ), W ilson and Sharda (1 9 9 4 ), and O lson e t al. (2 0 1 2 )]). Im portant considerations are the e x a ct n um ber o f n eu­ rons and th e n um ber o f layers. Som e p ack ag es u se gen etic algorithms to select the n et­

w ork design. T h ere are several param eters for tuning th e n etw ork to th e desired learning

p erfo rm an ce level. P ait o f th e p ro ce ss in step 5 is th e initialization o f th e n etw ork w eights an d p aram eters, follow ed b y the m odification o f th e param eters as traim n g-p erform an ce feed b ack is received. Often, the initial valu es a re im portant in determ ining th e efficiency and length o f training. Som e m eth od s ch an g e th e p aram eters during training to en h an ce

p erform an ce. . , _ , Step 6 transform s th e ap plication data into th e typ e an d format req u ired b y the

neural netw ork. This m ay require writing softw are to p rep ro cess th e d ata o r perform ing th ese op eratio n s directly in an ANN p ack ag e. D ata storage an d m anipulation techniques an d p ro cesses m ust b e designed for con ven ien tly an d efficiently retraining th e neural n etw ork, w h e n n eed ed . T h e ap plication data rep resen tatio n an d ord ering often influence th e efficiency and possibly the a ccu ra cy o f th e results.

In steps 7 an d 8 , training and testing are co n d u cted iteratively b y p resenting input a n d desired o r k now n ou tput d ata to th e n etw ork. T h e n etw ork co m p u tes th e outputs an d adjusts the w eights until the co m p u ted ou tp uts a re within a n a ccep tab le toleran ce o f the k now n outputs for th e input cases. T h e desired outputs and their relationships to input d ata are derived from historical d ata (i.e ., a portion o f th e d ata co llected in step 1).

In step 9 , a stable se t o f w eights is ob tain ed . N ow th e n etw ork c a n rep ro d u ce th e desired outputs, given inputs su ch as th o se in th e training set. T h e n etw ork is read y for u se as a stan d -alon e system o r as part o f an o th er softw are system w h ere n ew input data will b e p resen ted to it an d its ou tp ut will be a re co m m en d ed decision.

In th e follow ing sections, w e exam in e th e se step s in m o re detail.

Chapter 6 • T e ch n iq u e s for Predictive M odeling 2 8 9

FIGURE 6.9 Development Process of an ANN Model.

The G eneral A N N L e a rn in g Process

In s u p e rv is e d le a r n in g , the learning p ro cess is inductive; are derived from existin g cases. T h e usual p ro cess o f learning involves th ree (s e e

Figure 6 .1 0 ):

1 . C o m p u te te m p o r a r y o u tp u ts . 2 . C o m p a r e o u tp u ts w ith d e s ir e d ta rg e ts . 3 . A d ju st t h e w e i g h t s a n d r e p e a t t h e p r o c e s s .

2 9 0 Part III * Predictive Analytics

FIGURE 6.10 Supervised Learning Process of an ANN.

W h en existing outputs are available for co m p ariso n , th e learning p ro cess starts by setting th e co n n ectio n w eights. T h ese are set via rules o r at random . T h e difference b etw een th e actual ou tp u t ( F o r YT) an d the d esired ou tput (Z ) for a given set o f inputs is an error called d elta (in calculus, th e G reek sym bol delta, A, m ean s “d ifference”).

T h e objective is to minimize delta (i.e ., re d u ce it to 0 if p ossib le), w h ich is d o n e by adjusting th e n etw ork ’s w eights. T h e k ey is to c h a n g e th e w eights in th e right direction, making ch an g es that re d u ce delta (i.e ., error). W e will sh o w h o w this is d o n e later.

Inform ation p rocessin g w ith an ANN con sists o f attem pting to reco g n ize patterns o f activities (i.e ., pattern recogn ition ). During th e learning stages, th e interconn ection w eights ch an g e in resp o n se to training d ata p resen ted to the system .

Different ANN co m p u te delta in different w ay s, d ep en d in g o n th e learning alg o ­ rithm bein g used. H undreds o f learning algorithm s are available for various situations an d configurations o f ANN. Perh aps th e o n e that is m ost co m m on ly u sed an d is easiest to und erstan d is backp rop agation.

Backpropagation B ack p rop agatio n (sh o rt for b a ck -erro rp ro p a g a tio n ) is the m ost w idely u sed supervised learning algorithm in neural com p u tin g (P rin cip e e t al., 2 0 0 0 ). It is very ea sy to imple­ m en t. A b ack p rop agation n etw ork includes o n e o r m o re hidden layers. This typ e o f n etw ork is con sid ered feedforw ard b e ca u s e th ere are n o in terconn ections b etw een th e ou tput o f a p rocessin g elem en t and th e input o f a n o d e in th e sam e layer o r in a p re ce d ­ ing layer. E xternally provid ed co rre ct p atterns a re co m p ared w ith th e n eural n etw ork ’s ou tp u t during (su p ervised ) training, an d feed b ack is u sed to adjust th e w eights until the n etw ork h as catego rized all the training patterns as co rrectly as possible (th e e rro r toler­ a n c e is set in ad v an ce).

Starting w ith th e o u tp u t layer, erro rs b e tw e e n th e actu al an d d esired o u tp u ts are u se d to c o rr e c t th e w eigh ts fo r th e co n n e c tio n s to th e p rev io u s lay er (s e e Figu re 6 .1 1 ).

Chapter 6 • T e ch n iq u e s for Predictive M odeling 291

FIGURE 6.11 Backpropagation of Error for a Single Neuron.

F o r an y o u tp u t n e u ro n j , th e e rro r (d e lta ) — (Z j Yj) (.df/dx), w h e re Z a n d Y a ie th e d esired a n d a ctu a l ou tp u ts, resp ectiv ely . U sing th e sigm oid fu n ctio n ,/ = [1 + e x p ( - r ) ] , w h e re ^ is p ro p o rtio n a l to th e su m o f the w e ig h te d inputs to th e n eu ro n , is an effe c­

tive w ay to c o m p u te th e o u tp u t o f a n eu ro n in p ra ctice . W ith this fu n ctio n , th e d eriva­ tive o f th e sig m o id fu n ction d f/ d x = / ( I - / ) a n d th e e rro r is a sim ple fu n ctio n o f th e d esired a n d a ctu a l ou tp u ts. T h e f a c t o r / ( I - / ) is the logistic fu n ctio n , w h ich se rv e s to k e e p the e rro r c o rr e c tio n w ell b o u n d ed . T h e w eig h ts o f e a c h input to th e ,/th n eu ro n a re th en c h a n g e d in p ro p o rtio n to this ca lcu la te d erro r. A m o re co m p lica te d e x p re ssio n can b e d eriv ed to w o rk b ack w ard in a sim ilar w a y fro m th e o u tp u t n e u ro n s th rou gh th e h id d en layers to ca lcu la te th e co rre ctio n s to th e a sso cia te d w eigh ts o f th e in n er n eu ron s. This co m p lica te d m e th o d is an iterative a p p r o a c h to solv in g a n o n lin ear op ti­ m ization p ro b le m th at is v e ry sim ilar in m ean in g to th e o n e ch a ra cte riz in g m ultiple- linear reg ressio n .

T h e learning algorithm includes th e following p roced ures:

1 . Initialize w eigh ts w ith ran d o m values and set o th er param eters. 2 . R ead in th e input v e c to r an d th e desired output. 3 . C om pute th e actu al ou tp ut via th e calculations, w orking forw ard th ro u g h the layers. 4 . C om pute th e error. 5 . C han ge th e w eights b y w orking b ack w ard from th e ou tput layer th rou gh th e hidden

layers.

This p ro c e d u r e is re p e a te d fo r th e en tire se t o f input v e c to rs until th e d esired o u tp u t and th e actu al o u tp u t a g re e w ithin so m e p red eterm in ed to le ra n ce . G iven th e calcu latio n req u irem en ts fo r o n e iteration , a large n etw o rk can tak e a v e ry lo n g tim e to train; th erefo re, in o n e variation , a se t o f ca s e s is run fo iw a rd a n d a n a g g re g a te d error is fed b a ck w a rd to sp e e d u p learning. Som etim es, d ep en d in g o n th e initial ran d o m w eights an d n e tw o rk p aram eters, th e n etw o rk d o e s n o t c o n v e rg e to a satisfactory p erfo rm an ce level. W h en this is th e c a s e , n e w ran d om w eights m u st b e g en erated , an d the n etw o rk p a ram eters, o r e v e n its structure, m ay h a v e to b e m od ified b efo re a n o th e r attem p t is m ad e. C urren t re s e a rc h is aim ed at d ev elo p in g algorith m s an d using parallel co m p u te rs to im p ro v e this p ro ce ss. F o r e x a m p le , g e n e tic algorith m s c a n be u sed to gu id e th e sele ctio n o f th e n etw o rk p aram eters in o rd e r to m axim ize th e desired ou tp ut. In fact, m o st co m m e rcia l ANN so ftw are tools are n o w u sin g GA to h elp u sers "optim ize” th e n etw o rk p aram eters. T e ch n o lo g y Insights 6 .2 d iscu sses s o m e o f th e m ost p o p u lar n eu ral n etw o rk softw are a n d offers so m e W e b links to m o re co m p re h e n siv e A N N -related so ftw are sites.

2 9 2 Part III • Predictive Analytics

T E C H N O L O G Y IN SIG H T S 6 . 2 ANN S o f tw a r e

Many tools are available for d eveloping neural n etw orks (s e e this b o o k ’s W eb site and the resource lists at PC AI, p c a i.c o m ). Som e o f th ese tools function like softw are shells. T h e y pro­ vide a s e t o f standard architectures, learning algorithm s, and param eters, along with th e ability to m anipulate th e data. Som e d evelopm ent tools can support up to several d ozen netw ork para­ digms and learning algorithms.

Neural netw ork im plem entations a re also available in m ost o f th e com prehen sive data mining tools, su ch as the SAS Enterprise Miner, IBM SPSS M odeler (form erly C lem entine), and Statistica Data Miner. W eka, RapidMiner, and KNIME are o p en sou rce free data m ining softw are tools that include neural n etw ork capabilities. T h ese free tools can b e dow nload ed from their respective W eb sites; sim ple Internet search es o n th e nam es o f th ese tools should lead you to th e dow nload pag es. Also, m ost o f th e com m ercial softw are tools are available for dow n­ load and u se for evaluation purposes (usually, they are limited o n tim e o f availability and/or functionality).

M any specialized neural netw ork tools en ab le the building and deploym ent o f a neural netw ork m od el in practice. Any listing o f such tools w ou ld b e incom plete. O nline resou rces such as W ikipedia (e n .w ik ip e d ia .o r g /w ik i/A r tif ic ia l_ n e u r a l_ n e tw o r k ), G o o g le’s o r Y ah o o l’s softw are directory, and th e vend or listings o n p c a i .c o m are g o o d places to locate the latest inform ation o n neural netw ork softw are vendors. So m e o f th e vend ors that have b e e n around fo r a w hile and have reported industrial applications o f their neural netw ork softw are include California Scientific (BrainM aker), Neural W are; N euroD im ension Inc., W ard Systems Group (N euroshell), and M egaputer. Again, the list can n ev er b e com plete.

Som e ANN d evelopm ent tools are sp readsheet add-ins. Most can read spreadsheet, data­ b ase, and te x t files. Som e are freew are o r sharew are. Som e ANN system s have b e e n developed in Jav a to run directly o n th e W eb and are accessib le through a W eb brow ser interface. O ther ANN products are designed to interface with ex p ert system s as hybrid d evelopm ent products.

D evelop ers m ay instead prefer to use m ore gen eral program m ing languages, su c h as C++, or a spreadsheet to program the m odel and perform the calculations. A variation o n this is to u se a library o f ANN routines. For exam p le, hav.Softw are (h a v .c o m ) provides a library o f C++ classes for im plem enting stand-alone o r em bedded feedforw ard, sim ple recurrent, and random- ord er recurrent neural netw orks. Com putational softw are such as MATLAB also includes neural n etw o rk -sp ecific libraries.

S E C T I O N 6 . 3 R E V I E W Q U E S T I O N S

1 . List th e nine step s in con du ctin g a neural n etw ork project.

2 . W h at are so m e o f th e design param eters for d evelop in g a neural network?

3 . H o w d o e s b ack p rop agation learning work? 4 . D escrib e different typ es o f neural n etw ork softw are available today.

5 . H ow are neural n etw orks im plem ented in p ractice w h e n the training/testing is com plete?

6.4 ILLU M IN A T IN G THE B L A C K B O X O F A N N W ITH S E N S IT IV IT Y A N A L Y S IS

Neural n etw orks h av e b e e n u sed as an effective to ol for solving highly co m p le x real- world prob lem s in a w id e ran ge o f application areas. E ven thou gh ANN h ave b een p roven in m an y prob lem scen arios to b e su p erior p red ictors a n d /o r clu ster identifiers (co m p a re d to their traditional co u n terp arts), in s o m e applications th ere exists an addi­ tional n e e d to k n o w “h o w it d o e s w h at it d o e s .” ANN are typically th ou ght o f as b lack

Chapter 6 • T ech n iq u es for Predictive M odeling 2 9 3

b o x e s, cap ab le o f solving co m p le x problem s but lacking the exp lan atio n o f their capabili­ ties. This p h e n o m e n o n is com m on ly referred to as th e “b lack -b o x” syn drom e.

It is im portant to b e able to explain a m odel’s ‘In n er being”; such an explanation offers assu ran ce that the netw ork has b een properly trained and will behave as desired on ce deployed in a business intelligence environment. Such a n eed to “look under the h oo d ” might b e attributable to a relatively small training set (as a result o f th e high co st of data acquisition) o r a v ery high liability in ca se o f a system error. O ne exam p le o f such an application is th e deploym ent o f airbags in automobiles. H ere, both the co st o f data acqui­ sition (crashing cars) and the liability con cern s (d an ger to hum an lives) are rather signifi­ cant. A nother representative exam p le for the im portance o f explanation is loan-application processing. If an applicant is refused for a loan, h e o r she has the right to k now why. Having a prediction system that d oes a g o o d job o n differentiating g o o d and b ad applications may not b e sufficient if it d oes n ot also provide the justification o f its predictions.

A variety o f tech n iq u es has b e e n p ro p o sed fo r analysis an d evalu ation o f trained neural netw orks. T h ese tech n iq u es p rovid e a cle a r interpretation o f h o w a n eural n etw ork d oes w h at it d o es; that is, specifically h o w (a n d to w h at e x te n t) th e individual inputs factor into th e gen eration o f specific n etw ork output. Sensitivity analysis has b e e n the front ru n ner o f the tech n iq u es p ro p o se d for shedding light into the “b la ck -b o x ” ch aracter­ ization o f train ed neural netw orks.

Sensitivity analysis is a m eth od for extractin g the cau se-an d -effect relationships am o n g th e inputs an d th e outputs o f a trained neural n etw ork m odel. In th e p ro cess of perform ing sensitivity analysis, th e trained neural n etw ork ’s learning capability is disabled so that th e n etw o rk w eights a re n ot affected. T h e basic p ro ced u re behind sensitivity analysis is th at th e inputs to the n etw ork are system atically p ertu rb ed w ithin the allow ­ able v alu e ran g es and the corresp on d in g ch an g e in th e ou tp ut is re co rd e d for e a c h and every input variable (Principe e t al., 2 0 0 0 ). Figure 6 .1 2 sh ow s a grap h ical illustration of this p ro cess. T h e first input is varied b etw een its m ean plus-and-m inus a user-defined n um ber o f standard deviations (o r for categorical variables, all o f its possible values are u sed ) w h ile all o th er input variables are fixed a t their resp ective m ean s (o r m o d es). T h e n etw ork ou tp u t is co m p u ted for a u ser-defined n um ber o f steps ab o v e an d b elow th e m ean . This p ro ce ss is rep eated for e a c h input. As a result, a rep ort is g en erated to sum m arize th e variation o f e a ch ou tput w ith re sp e ct to th e variation in e a ch input. The gen erated re p o rt often con tain s a colu m n plot (alo n g w ith num eric valu es p resen ted on th e x-a x is), rep o rtin g th e relative sensitivity values fo r e a ch input variable. A rep resen ta­ tive e x a m p le o f sensitivity analysis o n ANN m od els is provid ed in A p p lication Case 6.3-

Trained A N N the black box”

Observed Change in

Outputs

Systematically P erturbed

Inputs

FIG U R E 6 .12 A Figurative Illustration o f Sensitivity Analysis on an ANN Model.

2 9 4 Part III • Predictive Analytics

Application Case 6.3 Se n sitivity A n alysis Reveals Injury S e v e rity Factors A ccord in g to th e National H ighw ay Traffic Safety Administration, o v e r 6 million traffic accid en ts claim m o re th an 4 1 ,0 0 0 lives e a c h y e a r in th e United States. C auses o f accid en ts and related injury severity are o f sp ecial in terest to traffic-safety research ers. Such research is aim ed n o t only a t red u cin g the n um ber o f accid en ts b u t also th e severity o f injury. O n e w a y to acco m p lish th e latter is to identify th e m ost profou n d factors that affect injury severity. U nderstanding the circu m stan ces u n d er w h ich drivers and p assen gers a re m o re likely to b e severely injured (o r killed) in an au to m ob ile accid en t c a n help im prove the overall driving safety situation. Factors that p o ten ­ tially elevate th e risk o f injury severity o f veh icle occu p an ts in th e even t o f an au tom otive accid en t in clude d em o g rap h ic a n d /o r behavioral ch aracteris­ tics o f th e p e rso n (e .g ., age, g en d er, seatb elt u sage, u se o f drugs o r alcoh ol while driving), en vironm en­ tal factors a n d /o r ro ad w ay conditions at th e time o f the a ccid e n t (e .g ., su rface conditions, w eath er o r light con dition s, th e direction o f im pact, veh icle orientation in th e crash , o ccu rre n ce o f a rollover), as well as tech n ical ch aracteristics o f th e veh icle itself (e .g ., v eh icle’s a g e , b o d y typ e).

In an e xp lo rato ry d ata mining study, D elen e t al. (2 0 0 6 ) u se d a large sam p le o f data— 3 0 ,3 5 8 p olice-rep o rted accid en t reco rd s ob tain ed from the G eneral Estim ates System o f the National H ighw ay Traffic Safety Administration— to identify w h ich factors b e c o m e increasingly m o re im portant in escalatin g th e probability o f injuiy severity during a traffic crash . A ccidents exam in ed in this study included a geog rap h ically rep resentative sam p le o f m ultiple-vehicle collision accid en ts, single-vehicle fixed-object collisions, an d single-vehicle noncolli­ sion (ro llover) crash es.

C ontrary to m an y o f the p reviou s studies co n d u cted in this dom ain , w h ich h ave prim ar­ ily u sed regression -typ e gen eralized linear m odels w h e re th e functional relationships b etw een injury severity an d crash -related factors a re assum ed to b e linear (w h ich is an oversim plification o f the reality in m o st real-w orld situations), D elen an d his colleag u es d ecid ed to g o in a different direction. B e ca u se ANN a re know n to b e su p erior in cap tu r­ ing highly n o n lin ear co m p le x relationships b etw een

in Traffic Accidents

the p red icto r variables (cra sh factors) and th e target variable (severity level o f the injuries), th ey d ecid ed to u se a series o f ANN m od els to estim ate the sig­ nificance o f th e cra sh factors o n the level o f injury severity sustained b y th e driver.

F ro m a m e th o d o lo g ic a l stan d p oin t, th ey fo llo w e d a tw o -s te p p ro c e s s . In th e first ste p , th e y d e v e lo p e d a s e rie s o f p re d ictio n m o d els (o n e for e a c h injury s e v e rity le v e l) to ca p tu re th e in -d ep th relatio n sh ip s b e tw e e n th e cra sh -re la te d facto rs an d a sp e cific le v e l o f injury sev erity . In th e s e c ­ o n d ste p , th e y c o n d u c te d sen sitivity an alysis o n th e tra in e d n e u ra l n e tw o rk m o d e ls to identify th e p rio ritized im p o r ta n c e o f cra s h -re la te d fa cto rs as th e y re la te to d ifferen t injury se v e rity levels. In th e fo rm u latio n o f th e stu d y, th e fiv e-class p re ­ d ictio n p ro b le m w a s d e c o m p o s e d in to a n u m b er o f b in ary classificatio n m o d els in o rd e r to ob tain th e granularity' o f in fo rm ation n e e d e d to identify th e “tr u e ” c a u s e -a n d -e ffe c t relatio n sh ip s b e tw e e n th e c ra s h -re la te d fa cto rs a n d d ifferen t lev els o f injury severity .

T h e resu lts re v e a le d co n sid e ra b le d ifferen ces a m o n g th e m o d els b uilt fo r different injuiy severity levels. This im plies th at th e m o st influential facto rs in p red ictio n m o d e ls highly d e p e n d o n th e level o f injury severity . F o r e x a m p le , th e stu d y re v ealed th at th e v ariab le se a tb e lt u se w a s th e m o st im p o r­ tan t d eterm in an t fo r p red ictin g h ig h er levels o f injury sev erity (s u c h as in cap acitatin g injuiy o r fatality), b ut it w a s o n e o f th e le a st significant p re d icto rs fo r lo w e r levels o f injury sev erity (s u c h a s n o n -in ca p a cita tin g injury a n d m in o r injury). A n o th er in terestin g finding in v o lv ed g e n d e r: The d rivers’ g e n d e r w a s a m o n g th e sign ifican t p re d ic­ to rs fo r lo w e r lev els o f injury sev erity , b u t it w as n o t a m o n g th e sign ifican t fa cto rs fo r h ig h er le v ­ els o f injury sev erity , in d icating th at m o re serio u s injuries d o n o t d e p e n d o n the d river b ein g a m ale o r a fem ale. Y e t a n o th e r in terestin g an d so m e w h a t intuitive finding o f th e stu d y in d icated th at ag e b e c o m e s an in creasin g ly m o re sign ifican t fa cto r as th e level o f injury se v e rity in cre a se s, im plying that o ld e r p e o p le a re m o r e likely to in cu r s e v e re inju­ ries (a n d fatalities) in serio u s a u to m o b ile c ra sh e s th an y o u n g e r p e o p le .

Chapter 6 • T ech n iq u es for Predictive M odeling 2 9 5

1. H o w d o e s sensitivity analysis sh ed light o n the black b o x (i.e ., neural netw orks)?

2. W h y w o u ld so m e o n e c h o o s e to u se a black- b o x to o l like neural netw orks o v e r theoretically sou nd , m ostly transparent statistical tools like logistic regression?

Q u e s t i o n s f o r D i s c u s s i o n 3. In this ca se , h o w did neural n etw orks an d sensi­ tivity analysis h elp identify injury-severity factors in traffic accidents?

Source: D. Delen, R. Sharda, and M. Bessonov, “Identifying Significant Predictors o f Injury Severity in Traffic Accidents Using a Series o f Artificial Neural Networks,"’ A cciden t A nalysis a n d P revention, Vol. 38, No. 3, 2006, pp. 434-444.

REVIEW QUESTIONS FO R SECTION 6 .4

1 . W h at is th e so-called “b lack -b o x” syndrom e? 2 . W h y is it im portan t to b e ab le to exp lain a n ANN’s m od el structure?

3 . H o w d o e s sensitivity analysis w ork? 4 . S earch th e Internet to find o th er ANN exp lan atio n m ethods.

6.5 SU PPO R T VEC TO R M A C H IN E S S upport v e c to r m ach in es (SVMs) a re o n e o f the p opu lar m ach in e-learn in g techniques, m ostly b e ca u s e o f their su p erior predictive p o w e r and their th eoretical foundation. SVMs are a m o n g th e su pervised learning m eth od s that p ro d u ce input-output functions from a set o f labeled training data. T h e function b e tw e e n th e input an d ou tp ut v ecto rs c a n be either a classification function (u sed to assign ca se s into pred efin ed cla sse s) o r a reg res­ sion function (u se d to estim ate th e con tin u ous num erical valu e o f th e desired output). F o r classification, nonlin ear kernel functions are often u sed to transform th e input data (naturally rep resen ting highly co m p le x nonlin ear relationships) to a high dim ensional feature sp a ce in w h ich th e input data b e co m e s linearly sep arab le. T hen , th e m axim um - margin h y perplanes are con stru cted to optim ally sep arate th e ou tp u t classes from each

o th er in th e training data. G iven a classification-type p red iction prob lem , generally speaking, m an y linear clas­

sifiers (h y p erp lan es) c a n sep arate th e data into multiple subsections, e a c h rep resen ting on e o f th e classes (s e e Figure 6 .1 3 a , w h e re th e tw o classes are rep resen ted with circles [“# ”] and sq uares [“■ " ]). H ow ev er, only o n e hyperplane ach ieves th e m axim u m sep ara­ tion b e tw e e n th e classes (s e e Figure 6 .1 3 b , w h e re th e hyperplane an d the tw o m axim u m margin h yp erp lan es a re sep arating th e tw o classes).

D ata u sed in SVMs m ay h ave m ore th an tw o dim ensions (i.e ., tw o distinct classes). In that ca se , w e w o u ld b e interested in sep arating d ata using th e n - 1 dim ensional h yp er­ plane, w h e re n is the n um ber o f dim ensions (i.e ., class labels). This m ay b e s e e n as a typical form o f linear classifier, w h ere w e are interested in finding th e n - 1 h yperplane so that th e d istan ce from th e h y perplanes to the n earest data points are m axim ized . T h e assum ption is that the larger the m argin o r d istan ce b etw een th ese parallel hyperplanes, th e b etter th e generalization p o w e r o f th e classifier (i.e ., p red iction p o w e r o f the SVM m od el). If s u c h h yperplanes exist, th ey c a n b e m athem atically rep resen ted using qua­ dratic optim ization m odeling. T h ese h y perplanes are k n ow n as th e m axim u m -m argm hyperplane, a n d su ch a linear classifier is k n ow n as a m axim u m m argin classifier.

In addition to their solid m athem atical foundation in statistical learning th eory, SVMs h ave also d em on strated highly com p etitive p erform an ce in n u m erou s real-w orld p red ic­ tion p rob lem s, su ch as m ed ical diagnosis, bioinform atics, fa c e /v o ic e recog n ition , d em and forecasting, im age p rocessin g , a n d t e x t m ining, w h ich has established SVMs a s o n e o f the

2 9 6 Part III • Predictive Analytics

F IG U R E 6.13 Separation of the Two Classes Using Hyperplanes.

m o st p opu lar analytics tools for k n ow led ge d iscov ery an d d ata mining. Similar to artificial n eural netw orks, SVMs p o ssess the w ell-k n ow n ability o f b ein g universal ap p roxim ators o f an y m ultivariate function to an y desired d e g re e o f a ccu racy . T h erefore, they are o f p ar­ ticular interest to m odeling highly nonlinear, c o m p le x p roblem s, system s, an d p rocesses. In the research study su m m arized in A pplication C ase 6 .4 , SVM are used to successfully predict freshm an student attrition.

Application Case 6.4 M an ag in g Stu d en t R eten tio n w ith Pred ictive M odeling G en erally, stu d en t attrition a t a u niversity is d efin ed b y th e n u m b er o f stu d en ts w h o d o n ot c o m p le te a d e g re e in th at institution. It h as b e c o m e o n e o f th e m o st ch allen g in g p rob lem s fo r d ecision m ak ers in a ca d e m ic institutions. In sp ite o f all o f th e p ro g ra m s a n d serv ices to h elp retain students, a cco rd in g to th e U.S. D ep artm en t o f E d u cation , C en ter fo r E d u cation al Statistics (n ces.ed .g o v ), on ly a b o u t h alf o f th o se w h o e n te r h ig h er e d u c a ­ tion actu ally g rad u ate w ith a b a ch e lo r’s d eg ree. E n ro llm en t m a n a g e m e n t a n d th e reten tio n o f stu­ d en ts h as b e c o m e a to p priority fo r adm inistrators o f c o lle g e s an d universities in th e U n ited States a n d o th e r d e v e lo p e d cou n tries aro u n d the w orld. High rates o f stu d en t attrition usually result in loss o f financial re so u rce s, lo w e r g rad u atio n rates, and inferior p e rc e p tio n o f the s ch o o l in th e e y e s o f all stak eh o ld ers. T h e legislators an d p olicym ak ers

w h o o v e rs e e h ig h er e d u ca tio n an d allo ca te funds, th e p aren ts w h o p ay fo r th eir ch ild ren ’s ed u catio n in o rd e r to p re p a re th em fo r a b e tte r fu tu re, and th e stu d en ts w h o m ak e co lle g e ch o ic e s lo o k for ev id e n ce o f institutional quality (s u c h a s low attrition ra te ) a n d rep u tatio n to g u id e th eir co lleg e selectio n d ecision s.

T h e statistics sh o w th at th e v ast m ajority o f stu d en ts w ith d raw fro m th e university during th eir first y e a r ( i .e ., fresh m an y e a r ) at th e c o l­ le g e . S in ce m o s t o f th e stu d en t d ro p o u ts o c c u r a t th e e n d o f th e first y e a r, m an y o f th e stu d en t re te n tio n /a ttritio n r e s e a r c h stu d ies (in clu d in g the o n e su m m arized h e re ) h a v e fo c u s e d o n first-year d ro p o u ts ( o r th e n u m b er o f stu d en ts th at d o n o t retu rn fo r th e s e c o n d y e a r ). T rad itionally, stu d en t r e te n tio n -re la te d re s e a rc h h as b e e n su rv e y d riven ( e .g ., su rv ey in g a stu d en t c o h o rt an d follow in g

Chapter 6 • T ech n iq u es for Predictive Modeling 2 9 7

th e m fo r a s p e cifie d p e rio d o f tim e to d eterm in e w h e th e r th e y co n tin u e th eir e d u c a tio n ). U sing su ch a re s e a rc h d esig n , re s e a rc h e rs w o rk e d o n d e v e l­ o p in g a n d valid atin g th e o re tica l m o d els in clu d in g th e fam o u s s tu d e n t in teg ratio n m o d el d e v e lo p e d b y T in to. A n altern ativ e ( o r a c o m p le m e n ta ry ) a p p r o a c h to th e tradition al su rv e y -b a se d reten tion re s e a rc h is a n an alytic a p p r o a c h w h e re th e d ata co m m o n ly fo u n d in in stitu tion al d a ta b a se s is used. E d u ca tio n a l institutions rou tinely c o lle c t a b ro a d ra n g e o f in fo rm atio n a b o u t th eir stu d en ts, in clu d ­ ing d e m o g ra p h ics, ed u ca tio n a l b a ck g ro u n d , social in v o lv em en t, s o c io e c o n o m ic status, a n d a c a d e m ic

p ro g re ss.

R e s e a r c h M e th o d

In o rd e r to im p ro ve stu d en t retention , o n e should try to u n d erstan d th e non-trivial reaso n s behind the attrition. T o b e su ccessfu l, o n e sh ould also b e able to a ccu rately identify th o se students th at a re at risk o f d rop p in g o u t. This is w h e re analytics c o m e in handy. U sing institutional data, p red iction m odels c a n b e d e v e lo p e d to a ccu rately identify th e students at risk o f d ro p o u t, s o th at lim ited reso u rces (p e o p le , m o n ey , tim e, e tc ., at an institution’s stu d en t su c­ c e s s ce n te r) c a n b e optim ally u sed to retain m o st

o f them . In this study, using 5 y ears o f freshm an student

d ata (o b tain ed from th e university’s existing data­ b a se s) alon g w ith several d ata mining techniques, fou r typ es o f p red iction m od els are d ev elo p ed and tested to identify th e best p red ictor o f freshm an attri­ tion. In o rd e r to exp lain the p h en o m en o n (identify th e relative im portan ce o f variables), a sensitivity analysis o f th e d ev elo p ed m od els is also con d u cted . T h e m ain g o als o f this and o th er similar analytic studies are to ( 1 ) d evelo p m odels to correctly iden­ tify th e freshm an students w h o are m o st likely to d rop o u t after their freshm an y ear, an d ( 2 ) identify th e m o st im portan t variables b y applying sensitiv­ ity an alyses o n d ev elo p ed m odels. T h e m od els that w e d e v e lo p e d are form ulated in su ch a w ay that th e p red iction o ccu rs at th e en d o f the first sem ester (usually at the e n d o f fall sem ester) in o rd e r for the d ecision m ak ers to p rop erly craft intervention p ro­ gram s during the n ext sem ester (th e spring sem es­ ter) in o rd e r to retain them.

Figure 6 .1 4 sh o w s th e graphical illustration o f the research m ythology. First, d ata from multiple so u rces ab ou t th e students a re co llected an d co n ­ solidated (s e e T ab le 6 .2 fo r th e variables u se d in this study). N ext, th e data is p rep ro cessed to handle m issing values a n d o th er an om alies. T h e p re p ro ­ ce sse d d ata is th en p u sh ed th rou gh a 10-fold cro ss- validation ex p e rim e n t w h e re fo r e a c h m o d el type, 10 different m o d els are d ev elo p ed an d tested for co m p arison p u rp oses.

R e s u lts

T h e results (s e e T ab le 6 .3 ) sh o w e d that, g iven suf­ ficient data w ith th e p ro p e r variables, data mining tech n iq u es are ca p a b le o f predicting freshm an stu­ d ent attrition w ith ap p roxim ately 8 0 p e rce n t a c c u ­ racy. A m ong th e fo u r individual p red iction m odels used in this study, su p p ort v e cto r m ach in es p er­ form ed the b est, follow ed b y d ecision trees, neural netw orks, an d logistic regression.

T h e sen sitiv ity an alysis o n th e tra in e d p re ­ d ictio n m o d e ls in d ica te d th a t th e m o s t im p o rtan t p re d icto rs fo r stu d e n t attrition a re th o s e re la te d to p ast a n d p re s e n t e d u c a tio n a l s u c c e s s (s u c h a s th e ratio o f c o m p le te d cre d it h o u rs in to to tal n u m ­ b e r o f h o u rs e n ro lle d ) o f th e stu d en t a n d w h e th e r th e y a re g ettin g fin an cial h elp .

Q u e s t i o n s f o r D i s c u s s i o n

1. W h y is attrition o n e o f the m o st im portant issues in higher education?

2. H o w c a n predictive analytics (ANN, SVM, and so forth) b e u sed to b etter m an age student retention?

3. W h at are th e m ain challen ges an d potential solutions to th e u se o f analytics in retention m anagem ent?

Sourcess Compiled from D. Delen, “A Comparative Analysis o f Machine Learning Techniques for Student Retention Management,” D ecision Support Systems, Vol. 49, No. 4, 2010, pp. 4 9 8 -5 0 6 ; V. Tinto, Leaving C ollege: R ethinking the Causes a n d C ures o f Student A ttrition, University o f Chicago Press, 1987; and D. Delen, “ Predicting Student Attrition with Data Mining Methods,” Jo u rn a l o f C ollege Student R etention, Vol. 13, No. 1, 2011, pp. 17-35-

0C o n tin u e d )

Application Case 6.4 (Continued)

2 9 8 Part III • Predictive Analytics

Raw Data

Data Preprocessing

Preprocessed Data

Design of Experiments

Experimental Design (10-fold C ro ss

Validation)

Model Building

Prediction Models

Model Testing

Decision T re e s Neural Networks

Sup p o rt V ec to r Machine

Logistic Regression

Experiment Results (Confusion Matrixes)

Y ES NO

Y E S # of correctly predicted YES |I||i

Model ^ NO predicted NO # of G orrectly predicted NO

Deployment

FIG U R E 6 .1 4 The Process o f D eveloping and Testing Prediction Models.

Chapter 6 • T ech n iq u es for Predictive Modeling

T A B L E 6 . 2 List o f V a r ia b le s U s e d in t h e S t u d e n t R e te n tio n P ro ject

No. Variables Data Type

1 College Multi Nominal

2 Degree Multi Nominal

3 Major Multi Nominal

4 Concentration Multi Nominal

5 Fall Hours Registered Number

6 Fall Earned Hours Number

7 Fall GPA Number

8 Fall Cumulative GPA Number

9 Spring Hours Registered Number

10 Spring Earned Hours Number

11 Spring GPA Number

12 Spring Cumulative GPA Number

13 Second Fall Registered (Y/N) Nominal

14 Ethnicity Nominal

15 Sex Binary Nominal

16 Residential Code Binary Nominal

17 Marital Status Binary Nominal

18 SAT High Score Comprehensive Number

19 SAT High Score English Number

20 SAT High Score Reading Number

21 SAT High Score Math Number

22 SAT High Score Science Number

23 Age Number

24 High School GPA Number

25 High School Graduation Year and Month Date

26 Starting Term as New Freshmen Multi Nominal

27 TOEFL Score Number

28 Transfer Hours Number

29 CLEP Earned Hours Number

30 Admission Type Multi Nominal

31 Permanent Address State Multi Nominal

32 Received Fall Financial Aid Binary Nominal

33 Received Spring Financial Aid Binary Nominal

34 Fall Student Loan Binary Nominal

35 Fall Granl/Tuition Waiver/Scholarship Binary Nominal

36 Fall Federal Work Study Binary Nominal

37 Spring Student Loan Binary Nominal

38 Spring Grant/Tuition Waiver/Scholarship Binary Nomina!

39 Spring Federal Work Study Binary Nominal

299

( C o n tin u e d )

3 0 0 Part III • Predictive Analytics

Application Case 6.4 (Continued) TABLE 6 .3 Prediction Results fo r the Four Data Mining Methods (A 10-fold cross-validation

w ith balanced data set is used to obtain these test results.) _______

ANN(MLP) DT(C5) SVM LR

No Yes No Yes No Yes No Yes

Confusion f No 2 3 0 9 4 6 4 2 3 1 1 4 1 7 2 3 1 3 3 8 6 2 1 2 5 6 2 6

Matrix 1 yes 781 2 6 2 6 7 7 9 2 6 7 3 7 7 7 2 7 0 4 9 6 5 2 4 6 4

SUM 3 0 9 0 3 0 9 0 3 0 9 0 3 0 9 0 3 0 9 0 3 0 9 0 3 0 9 0 3 0 9 0

Per-class Accuracy 7 4 .7 2 % 8 4 .9 8 % 7 4 .7 9 % 8 6 .5 0 % 7 4 . 8 5 % 8 7 . 5 1 % 6 8 .7 7 % 7 9 . 7 4 %

Overall Accuracy 7 9 .8 5 % 8 0 .6 5 % 81 .1 8 % 7 4 .2 6 %

M a t h e m a t ic a l F o rm u la t io n o f S V M s

C onsider data points in th e training d ata set o f th e form:

(X2,C2), . . . , (.XfijCyj)}

w h e re th e c is the class label taking a valu e o f eith er 1 (i.e ., “y e s ”) o r 0 (i.e ., “n o ”) w hile x is th e input variable v ecto r. T h at is, e a c h data p oin t is an m -dim ensional real v ecto r, u su ­ ally o f s ca le d [0, 11 o r [ - 1 , 1] values. T h e norm alization a n d /o r scaling are im portant steps to guard against variables/attributes w ith larger v arian ce that m ight oth erw ise dom inate th e classification form ulae. W e c a n v iew this as training data, w h ich d en o tes th e co rre ct classification (som eth in g that w e w o u ld like th e SVM to eventually a ch ie v e ) b y m ean s o f a dividing hyperplane, w h ich takes th e m athem atical form

W 'X — b = 0.

T h e v e c to r w points p erp en d icu lar to the sep aratin g hyperplane. Adding th e offset p aram eter b allow s u s to in crease th e margin. In its a b sen ce, the hyperplane is forced to p ass th rou gh th e origin, restricting th e solution. As w e are interested in th e m axim u m m argin, w e are interested in the su p p ort v ecto rs an d the parallel hyperplanes (to the optim al h y p erp lan e) clo sest to th ese su p p o rt v e c to rs in eith er class. It c a n b e sh o w n that th ese parallel h yperplanes c a n b e d escrib ed b y eq uations

W 'X - b = 1,

W 'X — b — —1.

If th e training data are linearly sep arab le, w e c a n select th ese h yperplanes so that th ere a re n o points b etw een th em an d th en try to m axim ize their d istan ce (se e Figure 6 .1 3 b ). B y using geom etry, w e find th e d istan ce b etw een th e h yperplanes is 2 / 1 w \, s o w e w an t to minimize \ w \ . T o e x clu d e data points, w e n e e d to en su re that for

all i either w X i — b S 1 o r

w ' X t — b < —1.

Chapter 6 • T ech n iq u es for Predictive M odeling 3 01

T h i s c a n b e r e w r itte n as:

Cj ( w • - b ) ^ 1 , 1 — i — n -

P rim a l Fo rm T l ie p r o b le m n o w is to m in im iz e | w | s u b je c t to th e c o n stra in t C j( w • Xj - b ) ^ 1, 1 — i — n . T h is is a q u a d ra tic p ro g ra m m in g ( Q P ) o p tim iz a tio n p ro b le m . M o re clearly ,

M in im iz e (1 / 2 ) || iv ||2

S u b je c t t o c t ( t v Xi - b ) s 1, 1 < i < n .

T h e f a c to r o f 1/2 is u s e d f o r m a th e m a tic a l c o n v e n ie n c e .

D u a l Form W ritin g t h e c la s s if ic a t io n r u le in its d u a l f o r m r e v e a ls th a t c la s s ific a tio n is o n ly a fu n c tio n o f t h e s u p p o r t v e c to r s , th a t is, t h e tra in in g d a ta th a t lie o n t h e m a r g in . T h e d u a l o f th e

SV M c a n b e s h o w n t o b e :

m a x ^ o c j — ' Ĵ ocl a j C !c jx ] x j i= 1 i-j

w h e r e t h e a te r m s c o n s titu te a d u a l r e p r e s e n t a t io n f o r t h e w e ig h t v e c t o r m te r m s o f th e

tr a in in g s et:

u) = 2 “ * * *

S o ft M a r g in I n 1 9 9 5 , C o r te s a n d V a p n ik s u g g e s te d a m o d ifie d m a x im u m m a r g in id e a th a t a llo w s f o r m i s la b e le d e x a m p l e s . I f th e r e e x is ts n o h y p e r p la n e t h a t c a n s p lit t h e “y e s ” a n d “n o ” e x a m ­ p le s , t h e s o f t m a r g in m e t h o d w ill c h o o s e a h y p e r p la n e th a t s p lits t h e e x a m p l e s a s c le a n ly a s p o s s i b le , w h i le s till m a x im iz in g t h e d is t a n c e t o t h e n e a r e s t c le a n l y s p lit e x a m p l e s T h is w o r k p o p u la r iz e d t h e e x p r e s s io n s u p p o r t v e c t o r m a c h in e o r SV M . T h e m e t h o d in tr o d u c e s s la c k v a r ia b le s , w h i c h m e a s u r e t h e d e g r e e o f m is c la s s ific a tio n o f t h e d a tu m .

d ( i v • X i - b ) S : 1 - 1 < z < w

T h e o b je c t i v e f u n c tio n is t h e n i n c r e a s e d b y a f u n c tio n th a t p e n a li z e s n o n - z e r o a n d t h e o p tim iz a tio n b e c o m e s a trade-off b e t w e e n a la r g e m a r g in a n d a s m a ll e r r o r p e n ­ alty. I f t h e p e n a lt y f u n c t io n is l in e a r , t h e e q u a t i o n n o w tr a n s fo r m s to

m i n M l2 + such that C j { w X f — b ) ^ 1 & 1 — ̂ — n i

T h i s c o n s tr a in t a l o n g w ith t h e o b je c t i v e o f m in im iz in g \w\ c a n b e s o l v e d u s in g L a g ra n g e m u ltip lie r s . T h e k e y a d v a n ta g e o f a l in e a r p e n a lty f u n c t io n is th a t t h e s la c k v a r ia b le s v a n i s h fr o m t h e d u a l p r o b le m , w ith t h e c o n s t a n t € a p p e a r i n g o n l y a s a n v a d d itio n a l c o n s tr a in t o n t h e L a g ra n g e m u ltip lie r s . N o n lin e a r p e n a lty fu n c tio n s h a v e b e e n u s e d , p a r tic u la r ly t o r e d u c e t h e e f f e c t o f o u tlie r s o n t h e c la s s ifie r , b u t u n le s s c a r e is ta k e n , th e p r o b l e m b e c o m e s n o n - c o n v e x , a n d th u s it is c o n s i d e r a b ly m o r e d iffic u lt t o fin d a

g l o b a l s o lu tio n .

3 0 2 Part III • Predictive Analytics

Nonlinear Classification T h e original optim al h yperplane algorithm p ro p o sed by Vladimir Vapnik in 1963, while h e w a s a d o cto ral stu d en t a t th e Institute o f C ontrol S cien ce in M oscow , w as a linear classifier. H o w ev er, in 1 9 9 2 , B oser, Guyon, an d V apnik su gg ested a w ay to create nonlin­ ea r classifiers b y applying the kernel trick (originally p ro p o sed by Aizerm an e t al., 1964) to m axim um -m argin h yperplanes. T h e resulting algorithm is formally similar, e x c e p t that every d o t p ro d u ct is rep laced b y a nonlin ear k ernel function. This allow s th e algorithm to fit th e m axim um -m argin hyperplane in th e transform ed feature sp a ce . T h e transformation m ay b e nonlin ear an d th e transform ed sp a ce high dim ensional; thus, th ou gh the classifier is a hyperplane in th e high-dim ensional feature s p a c e it m ay b e nonlin ear in the original

input sp ace. If the k ern el u sed is a G aussian radial basis function, th e corresp o n d in g feature

sp a ce is a Hilbert sp a ce o f infinite dim ension. M axim um m argin classifiers are well leg- ularized, so th e infinite d im en sion d oes n o t spoil the results. Som e co m m o n kernels

include,

Polynomial (hom ogeneous): k ( x , x ’ ) — { x ' x ' )

Polynomial (inhom ogeneous): k.(x, x ' ) = ( x ' x + l )

Radial basis function: k ( x , x ' ) — e x p (-y | | ^ x'\\ ) , for y >

, ||x — x G aussian radial basis function: k ( x , x ' ) = e x p (

1112

Sigmoid: k ( x , x ' ) = tan h ( k x ’ x ' + c ) for so m e k > 0 an d c < 0

Kernel Trick In m ach in e learning, th e kernel trick is a m eth od for con vertin g a linear classifier algorithm into a nonlinear o n e b y using a nonlinear function to m ap the original ob servations into a higher-dim ensional sp a ce ; this m ak es a linear classification in th e n e w sp a ce equivalent to nonlinear classification in th e original sp ace.

This is d o n e using M ercer’s th eorem , w h ich states that an y con tinuous, sym m etric, positive sem i-definite kernel function K (x , y ) c a n b e e x p re sse d as a d ot p ro d u ct in a high-dim ensional sp a ce . M ore specifically, if th e argu m en ts to th e kernel a re in a m easu r­ ab le sp a ce X , an d if th e kernel is positive sem i-definite — i.e.,

^ K ( x h xj) c i Cj > 0

fo r an y finite subset [xr, x n} o f X an d su bset {ch . . . , cn} o f objects (typically real num bers o r e v e n m olecu les)— th en th ere exists a function <p(x) w h o se ran ge is in an inner p ro d u ct sp ace o f p ossibly high dim ension, s u c h that

K (x , y ) = ip(x) • <p(y)

T h e k ern el trick transform s an y algorithm that solely d ep en d s o n th e d ot p rod u ct b e tw e e n tw o vecto rs. W h e re v e r a d ot p ro d u ct is u se d , it is re p la ce d w ith th e kernel func­ tion. Thus, a linear algorithm c a n easily b e transform ed into a nonlinear algorithm . This n onlin ear algorithm is equivalent to th e linear algorithm op erating in the ran ge sp a ce o f tp. H ow ever, b e ca u s e kernels are used, th e <p function is n ev er explicitly co m p u ted . This is

n, n

Chapter 6 • T ech n iq u es for Predictive M odeling 3 0 3

d esirable, b e ca u s e th e high-dim ensional sp a ce m ay b e infinite-dimensional (a s is th e case

w h e n the k ern el is a Gaussian). A lthough th e origin o f th e term k e r n e l trick is n o t know n, th e kernel trick w a s first

published b y A izerm an et al. (1 9 6 4 ). It has b e e n applied to several kinds o f algorithm in m ach in e learning an d statistics, including:

• P ercep tio n s • Support v e c to r m ach in es • Principal co m p o n en ts analysis • Fisher’s linear discrim inant analysis • Clustering

SECTION 6 .5 REVIEW QUESTIONS

1 . H o w d o SVM work? 2 . W h at are th e ad vantages and disadvantages o f SVM? 3. W h at is th e m eaning o f “m axim u m m argin h yp erp lan es”? W h y a re th e y im portant in

SVM? 4. W h at is “kernel trick”? H o w is it u sed in SVM?

6.6 A PR O C E S S- B A S ED A P PR O A C H TO THE U S E O F S V M D ue largely to th e b etter classification results, recen tly su p p ort v e cto r m ach in es (SVMs) h av e b e co m e a p op u lar tech n iq u e fo r classification-type p roblem s. E ven th ou gh p eop le con sid er th em as bein g easier to u se th an artificial neural netw orks, u sers w h o are not familiar w ith th e intricacies o f SVMs often g e t unsatisfactory results. In this section w e provide a p ro cess-b ased ap p ro a ch to the u se o f SVM, w h ich is m ore likely to p rod u ce b etter results. A pictorial rep resen tation o f th e th ree-step p ro cess is given in Figure 6 .1 5 .

N U M ER IC IZ IN G TH E D A T A SVMs require that e a ch data instance is rep resen ted as a v e cto r o f real n um bers. H e n ce , if th ere are categ orical attributes, w e first h av e to con vert th em into n u m eric data. A co m m o n reco m m en d atio n is to u se m pseudo-binary-variables to rep resen t a n m -class attribute (w h e re m ^ 3 ). In p ractice, on ly o n e o f th e m variables assu m es th e valu e o f “1 ” an d oth ers assu m e th e valu e o f “0 ” b ased o n th e actual class o f th e case (this is also called 1-of-m rep resen tation ). F o r exam p le, a th ree-categ o ry attribute su ch as {red, g reen , blue} c a n b e rep resen ted as ( 0 ,0 ,1 ) , ( 0 ,1 ,0 ) , an d ( 1 ,0 ,0 ) .

N O R M A LIZ IN G T H E D A T A As w as th e ca s e for artificial n eural netw orks, SVMs also require norm alization a n d /o r scaling o f num erical valu es. T h e m ain ad van tag e o f n or­ m alization is to avo id attributes in g reater n u m eric ran ges dom inating th o se o f in sm aller num eric ran ges. A n oth er ad van tag e is that it h elp s perform ing n um erical calculations during th e iterative p ro ce ss o f m od el building. B e ca u se kernel values usually d ep en d on th e inner p rod u cts o f feature v ecto rs (e .g ., th e linear kernel an d th e polyn om ial kernel), large attribute values m ight slow th e training p ro cess. Use recom m en d atio n s to norm alize each attribute to th e range [ - 1 , +1] o r [0, 1]. O f co u rse, w e h av e to u se th e sam e norm al­ ization m e th o d to scale testing d ata b efore testing.

S E L E C T T H E K E R N E L T Y P E A N D K E R N E L P A R A M E T E R S E ven th ou gh th e re are on ly four co m m on kern els m en tion ed in th e p revious sectio n , o n e m ust d ecid e w h ich o n e to use (o r w h eth er to tty th em all, o n e a t a tim e, using a sim ple exp erim en tal d esign a p p ro ach ). O n ce th e k ern el typ e is selected , th en o n e n eed s to s e le ct the v alu e o f p enalty p aram eter C and kernel p aram eters. G enerally speaking, R BF is a reaso n ab le first c h o ic e for th e kernel ivpe T h e RBF k ernel aim s to nonlinearly m ap d ata into a h igh er dim ensional s p a ce ; by doing so (unlike with a linear k ern el) it h and les th e ca se s w h e re th e relation b etw een

3 0 4 Part III • Predictive Analytics

FIGURE 6.15 A Simple Process Description for Developing SVM Models.

input an d ou tput v ecto rs is highly nonlinear. B esid es, o n e sh ould n o te that th e linear k ern el is just a sp ecial c a s e o f RBF kernel. T h ere are tw o p aram eters to ch o o s e for RBF kernels: C an d y . It is n o t k n o w n b eforeh an d w h ich C an d y are th e b est fo r a given p red iction prob lem ; therefore,, so m e kind o f p a ram eter search m eth od n eed s to b e used. T h e goal for the search is to identify optim al valu es for C and y so that the classifier can accu rately p red ict unk n ow n d ata (i.e ., testing d ata). T h e tw o m o st com m on ly used search m eth od s are cross-validation an d grid search.

D E P L O Y TH E M O D E L O n ce an “optim al” SVM p red iction m o d el h as b een d evelo p ed , th e n e x t step is to integrate it into th e d ecision su p p o rt system . F o r that, th ere are tw o options: ( 1 ) con verting th e m o d el into a com p u tational o b ject (e .g ., a W e b service, Java B ean , o r COM o b ject) that takes th e input p aram eter values and p rovides ou tput p red ic­ tion, (2 ) extractin g th e m od el coefficients and integrating th em directly into th e decision su pp ort system . T h e SVM m od els are useful (i.e ., accu rate, actio n ab le) on ly if th e b eh av ­ ior o f th e underlying dom ain stays th e sam e. F o r so m e reaso n , if it ch an g es, s o d oes the a c c u ra c y o f th e m odel. T h erefo re, o n e sh ould con tin u ously assess th e p erform an ce o f the m od els, d ecid e w h e n th ey n o lo n g er are a ccu ra te , and, h e n ce , n e e d to b e retrained.

Support Vector Machines Versus Artificial Neural Networks E ven th o u g h so m e p e o p le ch aracterize SVMs a s a special ca s e o f ANNs, m ost reco g n ize th em as tw o co m p etin g m ach in e-learn in g tech n iq u es w ith different qualities. H ere are a few points th at help SVMs stan d o u t against ANNs. Historically, the d evelop m en t o f ANNs

Chapter 6 • T e ch n iq u e s for Predictive Modeling

follow ed a heuristic path, w ith applications an d exten sive exp erim en tatio n preced in g th eory In con trast, th e d ev elo p m en t o f SVMs involved so u n d statistical learning theory first, th en im plem entation and exp erim ents. A significant ad v an tage o f SVMs is that while ANNs m ay suffer from multiple local minim a, th e solutions to SVMs a re global an d unique. T w o m o re ad van tages o f SVMs are th at th e y h ave a sim ple g eom etric in terpretation and give a sp arse solution. T h e reaso n that SVMs often ou tp erform ANNs in p ractice is that th ey successfully d eal w ith th e “o v e r fitting” prob lem , w h ich is a big issue w ith ANNs.

Besides th e se ad vantages o f SVMs (fro m a practical p oin t o f v iew ), th e y also have som e limitations. A n im portant issue th at is n ot entirely solv ed is th e selectio n of th e ker­ n el type and k ern el function param eters. A s e co n d and p erh ap s m ore im portant limitation o f SVMs are th e sp e e d an d size, b oth in the training an d testing cycles. M odel building in SVMs involves co m p le x an d tim e-d em an d in g calculations. F ro m the practical point o f view , p erh ap s th e m ost serious p rob lem with SVMs is the high algorithm ic com p lexity and exten sive m em o ry requirem ents o f the req u ired quadratic p rogram m in g m laige- scale tasks. D esp ite th ese limitations, b e ca u s e SVMs are b ased o n a so u n d theoretical foundation and th e solutions th ey p ro d u ce a re global and uniqu e in n ature (a s o p p o se d to netting stuck in a suboptim al alternative su ch as a local m inim a), n o w ad ay s they are arguably o n e o f th e m o st p op u lar p red iction m odeling tech n iq u es in th e data mining arena. T heir u se an d popularity will only in crease as th e p opu lar com m ercial data mining tools start to in co rp o rate th em into their m odeling arsenal.

S E C T I O N 6.6 R E V I E W Q U E S T I O N S 1„ W h at are th e m ain steps an d d ecision points in d evelop in g a SVM model?

2 . H o w d o y o u d eterm ine th e optim al kernel type an d kernel param eters?

3 . C om p ared to ANN, w h at a re th e ad vantages o f SVM? 4. W h at are th e co m m o n ap plication areas for SVM? C onduct a search o n th e Internet

to identify p o p u lar ap plication areas an d specific SVM softw are tools u se d in th ose

applications.

6.7 N E A R E S T N E IG H B O R M ETH O D FO R PRED IC TIO N Data mining algorithm s ten d to b e highly m athem atical an d com p u tationally intensive The tw o p o p u lar on es that a re c o v e re d in th e previous section (i.e ., ANNs an d SVMs) involve tim e-dem anding, com putationally intensive iterative m athem atical derivations. In con trast the ^-nearest neigh b or algorithm (o r kNN, in sh ort) seem s overly simplistic for a com p etitive p red iction m eth o d . It is s o ea sy to und erstan d (a n d exp lain to o th ­ ers) w h at it d o e s and h o w it d o e s it. fc-NN is a prediction m ethod for classification- as well as regression -typ e p red iction p roblem s. &-NN is a typ e o f in stan ce-b ased learning (o r lazy learning) w h e re th e function is on ly ap p ro xim ated locally an d all com p u tation s are deferred until th e actu al prediction. ,

T h e ^-n earest neigh bor algorithm is am on g the sim plest o f all m ach ine-learning algo­ rithms: F o r in stan ce, in the classification-type prediction, a ca se is classified b y a majority v o te o f its neighbors, w ith th e ob ject bein g assigned to the class m ost co m m o n am on g its k n earest neighbors (w h ere k is a positive integer). If k = 1, th en th e c a s e is simply assigned to the class o f its nearest neighbor. T o illustrate the co n ce p t w ith a n exam p le, let us look a t Figure 6 .1 6 , w h ere a sim ple tw o-dim ensional sp ace represents th e values tor the tw o variables (x, y); th e star represents a n ew case (o r ob ject); an d circles an d squares represent k n ow n cases (o r exam p les). T h e task is to assign the n ew ca se to either circles o r squares b ased o n its closen ess (similarity) to o n e o r th e other. If y o u se t th e value o f k to 1 ( k = 1 ). th e assignm ent sh ould b e m ad e to square, b e ca u s e th e clo sest exam p le to star is a sq uare.’ If y o u se t the valu e o f k to 3 ( k = 3 ), then the assignm ent sh ould be m ad e to

3 0 6 Part III • Predictive Analytics

FIGURE 6.16 The Importance of the Value of k in ZrNN Algorithm.

circle, b ecau se th ere tw o circles and o n e square, a n d h en ce from the sim ple majority vote rule, circle gets th e assignm ent o f th e n ew case. Similarly, if y o u set th e valu e o f k. to 5 ( k = 5 ), th en the assignm ent should b e m ad e to square-class. This overly simplified exam ­ p le is m eant to illustrate the im portan ce o f th e valu e that o n e assigns to k.

T h e sam e m eth od can also b e u sed fo r regression -typ e p red iction tasks, b y simply averaging th e values o f its k n earest neighbors and assigning this result to th e ca s e being p red icted . It c a n b e useful to w eight th e contributions o f the neighbors, s o th at th e n earer neighbors contribute m ore to th e average than th e m o re distant o n es. A co m m o n w eight­ ing sch em e is to give e a ch neigh bor a w eight o f 1 / d, w h e re d is th e distan ce to th e neigh­ bor. This sch e m e is essentially a generalization o f linear interpolation.

T h e n eigh bors are tak en from a set o f ca se s for w h ich th e co rre ct classification (or, in th e case o f regression, th e num erical valu e o f th e ou tput v alu e) is k now n. This can be thought o f as the training set for the algorithm , e v e n thou gh n o exp licit training step is required. T h e ^-nearest n eigh b or algorithm is sensitive to the local structure o f th e data.

Sim ilarity Measure: The Distance Metric O n e o f th e tw o critical d ecisions that an analyst has to m ak e w hile using &NN is to d eterm ine th e similarity m easu re (th e oth er is to determ in e th e valu e o f k, w h ich is exp lain ed n e x t). In th e &NN algorithm , the similarity m easu re is a m athem atically calcu ­ lable d istan ce m etric. Given a n e w c a s e , &NN m ak es predictions b ased o n th e o u tco m e o f th e k n eigh bors closest in d istan ce to th at poin t. T h erefore, to m ak e p red ictions with &NN, w e n e e d to define a m etric for m easuring th e d istan ce b e tw e e n th e n ew c a s e and th e cases from the exam p les. O n e o f the m o st p o p u lar ch o ices to m easu re this distance is k now n as Euclid ean (E q uation 3 ), w h ich is sim ply the linear d istan ce b etw een tw o points in a dim ensional sp a ce ; the o th er p op u lar o n e is th e rectilinear (a.k .a. City-block o r M anhattan d istan ce) (E q uation 2). B o th o f th ese d istan ce m easu res a re sp ecial ca se s o f Minkowski distance (E q uation 1).

Minkowski distance

Chapter 6 • T ech n iq u es for Predictive M odeling 3 0 7

w here i = ( * * , x a , % 3 and j = (xj h xj2, are tw o ^ d im e n s io n a l daw objects (e .g ., a n e w c a s e an d an exam p le in the d ata set), and q is a positive integer.

If q = 1 ? th en d is called M anhattan distance

d ( i , / ) = V | x n - x ; i | + |x/2 - x p_\ + ... + \xip - xjp \

I f q = 2, th en d is called Euclid ean distance

d { i j ) = V (|^/1 - xn \2 + | X [2 - xJ 2 12 + ••• + 1 % _ % | 2)

O bviously, th ese m easu res ap ply only to num erically rep resen ted data. H o w about nom inal data? T h ere are w ays to m easu re d istan ce for n on-nu m erical d ata as well. In the simplest c a s e , fo r a multi-value nom inal variable, if th e valu e o f that variable for th e n ew ca se and th at for the exam p le ca se a re th e sam e, th e distance w ou ld b e zero , otherw ise on e. In ca se s su ch as te x t classification, m o re sophisticated m etrics exist, su ch as the overlap m etric (o r H am m ing d istan ce). Often, the classification a c c u ra c y o f &NN can b e im proved significantly if th e distance m etric is determ ined through an exp erim ental design w h ere different m etrics are tried an d tested to identify th e b est o n e for th e given problem .

P a ra m e te r S e le c tio n

T h e b est ch o ic e o f k d epen d s u p o n the data; generally, larger v alu es o f k re d u ce the effect o f n oise o n the classification (o r regression ) but also m ak e b oun d aries b etw een classes less distinct. An “optim al” valu e o f k c a n be found b y so m e heuristic techniques, for instance, cross-validation. T h e special ca s e w h e re the class is p red icted to b e the class o f the clo se st training sam p le (i.e ., w h en k = 1) is called th e n earest n eig h b o r algorithm.

C R O S S -V A L ID A T IO N C ross-validation is a w ell-established exp erim en tation technique that c a n b e u se d to determ ine o p t i m a l valu es for a set o f unk n ow n m o d el p aram eters. It applies to m ost, if n o t all, o f the m ach in e-learn in g tech n iq u es, w h ere th ere are a num ber o f m od el p aram eters to b e determ ined. T h e gen eral idea o f this exp erim en tation m ethod is to divide th e d ata sam p le into a n um ber o f random ly draw n, disjointed sub-sam ples (i.e ., v n u m b er o f folds). F o r e a c h potential valu e o f k, th e kN N m od el is u sed to m ake predictions o n th e vth fold w hile using th e v- 1 folds as the exam p les, an d evalu ate the error. T h e c o m m o n ch o ice for this error is the ro o t-m ean -sq u ared -erro r (RMSE) fo r regres­ sion-type pred ictions an d p ercen tag e o f co rrectly classified in stan ces (i.e ., hit rate) for the classification-type predictions. This p ro cess o f testing e a ch told against th e rem aining o f exam p les rep eats v times. At th e en d o f th e v n um ber o f cy cles, th e co m p u te d errors are accu m u lated to yield a g o o d n ess m easu re o f the m od el (i.e ., h o w well th e m od el predicts w ith th e cu rren t valu e o f th e M). A t the end, the k v alu e that p ro d u ces th e smallest overall error is ch o se n as the optim al valu e fo r that problem . Figure 6 .1 7 sh ow s a sim ple p rocess w h ere th e training data is u sed to d eterm ine o p t i m a l values fo r k an d d i s t a n c e m e t r i c , w h ich are th en u sed to p red ict n ew in com ing cases.

As w e ob served in the sim ple exam p le given earlier, th e a ccu ra cy o f the &NN algo­ rithm c a n b e significantly different w ith different values o f k. Fu rth erm ore, th e predictive p o w e r o f th e &NN algorithm d eg rad es w ith th e p re se n ce o f noisy, in accu rate, o r irrelevant features. M uch research effort h as b e e n p u t into feature selection a n d n orm alization / scaling to en su re reliable p red iction results. A particularly p o p u lar a p p ro a ch is the u se o f evolutionary algorithm s (e .g ., g en etic algorithm s) to optim ize th e set o f features included in the &NN p red ictio n system . In binary (tw o class) classification p ro b lem s, it is helpful to ch o o s e k to b e an o d d n um ber a s this w o u ld avoid tied votes.

A d raw b ack to th e basic majority voting classification in &NN is th at th e classes with the m o re freq u en t exam p les ten d to d om in ate th e p red iction o f th e n e w v ecto r, as they

3 0 8 Part III • Predictive Analytics

FIGURE 6.17 The Process of Determining the Optimal Values for Distance Metric and k.

ten d to c o m e up in th e k n earest n eigh bors w h e n th e n eigh bors are co m p u ted due to their large num ber. O n e w ay to o v e rco m e this p rob lem is to w eig h th e classification taking into a c c o u n t th e d istan ce from the test point to e a ch o f its k n earest n eighbors. Another w a y to o v e rco m e this d raw b ack is b y o n e level o f ab straction in d ata representation.

T h e naive version o f th e algorithm is easy to im plem ent b y com p u tin g th e distances from th e test sam ple to all stored v e cto rs, but it is com p u tationally intensive, especially w h en th e size o f th e training set grow s. Many n earest n eigh b o r search algorithm s have b e e n p ro p o se d o v e r th e years; th ese gen erally seek to re d u ce th e n um ber o f distance evaluations actually p erform ed. Using an ap p rop riate n earest n eigh b o r se a rch algorithm m akes &NN com putationally tractab le ev en for large data sets. Application C ase 6 .5 talks a b ou t th e su p erior capabilities o f &NN in im age recogn ition and categorization.

Application Case 6.5 E fficie n t Im age Recognition and C ategorization w ith kNN Im age reco g n itio n is an em ergin g d ata mining appli­ cation field involved in p ro cessin g, analyzing, and categorizin g visual objects su ch as p ictures. In the p ro cess o f recogn ition (o r categ orization ), im ages are first transform ed into a m ultidimensional fea­ tu re s p a c e an d then, using m achine-learning tech n iq u es, are categ o rized in to a finite n u m b er o f classes. Application areas o f im age recogn ition and categ orization ran ge from agriculture to h om elan d secu rity, p erson alized m arketing to environm ental p rotectio n . Im age recogn ition is an integral p art o f an artificial intelligence field called co m p u te r vision.

A s a tech n olo gical discipline, co m p u ter vision seek s to d ev elo p co m p u ter system s that a re c a p a ­ ble o f “se e in g ” an d reactin g to their environm ent. E xam p les o f applications o f co m p u ter vision include system s fo r p ro ce ss au tom ation (industrial rob ots), navigation (au to n o m o u s v eh icles), m o n ito rin g/ d etectin g (visual su rveillan ce), search in g and sorting visuals (in d exin g d atab ases o f im ages and im age s e q u e n ce s), en gagin g (co m p u te r-h u m a n interac­ tion), an d in sp ection (m anufacturing p ro cesses).

While the field o f visual recognition and category recognition has been progressing rapidly, m u ch remains

Chapter 6 • T ech n iq u es for Predictive M odeling 309

to b e d one to reach human-level performance. Current approaches are capable o f dealing with only a limited num ber o f categories (1 0 0 o r so categories) and are

A n oth er g ro u p o f research ers (B o im an e t al., 2 0 0 8 ) argued th at tw o practices co m m o n ly u sed in im age classification m eth o d s (n am ely SVM- and ANN-type m od el-driven a p p ro a ch e s and kN N typecomputationally expensive. Many machine-learning

techniques (including ANN, SVM, and &NN) are used to develop com p u ter systems for visual recognition and categorization. Though com m endable results have b een obtained, generally speaking, none o f these tools in their current form is capable o f developing systems that can com p ete with humans.

In a research project, several research ers from the C o m p u ter S cien ce Division o f the Electrical Engineering a n d C om p uter S cience D ep artm en t at th e University o f California, B erk eley, u sed an innovative en sem b le a p p ro a ch to im age categorization (Z h an g e t al., 2 0 0 6 ). T h ey co n sid ered visual categ o ry re c ­ ognition in th e fram ew ork o f m easuring similari­ ties, o r p ercep tu al d istances, to d ev elo p exam p les o f categ o ries. T heir recogn ition an d categorization a p p ro ach w a s quite flexible, perm itting reco g n i­ tion b ased o n co lo r, textu re, and particularly shape. W hile n earest n eigh b or classifiers (i.e ., &NN) are nat­ ural in this setting, th ey suffered from th e problem o f high varian ce (in b ias-varian ce d ecom p ositio n ) in the ca s e o f limited sam pling. Alternatively, o n e co u ld c h o o s e to u se su p p ort v e c to r m ach in es but th ey also involve tim e-con su m in g optim ization and co m p u tations. T h ey p ro p o se d a hybrid o f th ese tw o m ethods, w h ich d eals naturally w ith th e multiclass setting, has reason ab le com p u tational co m p lexity b oth in training an d at run tim e, an d yields e x c e l­ lent results in p ractice. T h e b asic idea w a s to find c lo se n eigh bors to a q uery sam p le and train a local su p p ort v e c to r m ach in e that p reserves th e distance function o n th e collection o f neighbors.

T heir m eth od c a n b e ap plied to large, multi­ class d ata sets w h e re it ou tp erform s nearest neigh ­ b o r an d su p p o rt v e cto r m ach in es and rem ains effi­ cien t w h e n th e p rob lem b e co m e s intractable. A w ide variety o f d istan ce functions w e re used, an d their e xp erim en ts sh ow ed state-of-the-art p erform an ce on a n um ber o f b ench m ark data sets for sh ap e an d te x ­ ture classification (MNIST, USPS, CUReT) and ob ject recogn ition (C altech -101).

n on -p aram etric a p p ro a c h e s ) h av e led to less-than- d esired p erfo rm an ce o u tco m es. T h ey also claim that a hybrid m ethod c a n im prove th e p erform an ce of im age recogn ition and categorization. T h ey p ro p o se a trivial Naive B a y e s &NN-based classifier, w hich em p loys kN N d istan ces in th e sp a ce o f the local im age descrip tors (a n d n ot in th e sp a ce o f im ages). T h e y claim that, althou gh the m odified &NN m ethod is extrem ely sim ple, efficient, and requires n o learn­ ing/training p h ase, its p erfo rm an ce ranks a m o n g the top leading learning-based p aram etric im age classi­ fiers. Em pirical co m p ariso n s o f their m eth o d w ere sh ow n o n several challenging im age categorization d atab ases (C a lte ch -1 0 1 , C altech -256, an d G raz-01).

In addition to im age recogn ition and ca te g o ­ rization, kN N is successfully applied to co m p le x classification p rob lem s, su ch as co n ten t retrieval (handw riting d etectio n , vid eo co n ten t analysis, b ody a n d sign lan gu age, w h e re co m m u n ication is d on e using b o d y o r h an d gestu res), g e n e exp ressio n (this is an o th er are a w h e re kN N tends to p erform b et­ ter than o th er state-of-the-art tech n iq u es; in fact, a com b in ation o f &NN-SVM is o n e o f th e m o st p opu lar tech n iq u es u sed h e re ), an d protein-to-p rotein inter­ action and 3D structure prediction (g rap h -b ased &NN is often u sed for in teraction structure prediction).

Q u e s t i o n s f o r D i s c u s s i o n 1. W h y is im age recogn ition /classification a w orth y

but difficult problem ? 2. H o w c a n kN N b e effectively u sed for im age rec­

ogn ition/classification applications?

Sources: H. Zhang, A. C. Berg, M. Maire, and J. Malik, “SVM- KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition,” P roceedin gs o f the 2 0 0 6 IEEE C om puter Society C on feren ce on Com puter Vision a n d P attern R ecogn ition (CVPR’0 6 ), Vol. 2, 2006, pp. 2126-2136; O. Boiman, E. Shechtman, and M. Irani, ;'In Defense o f Nearest-Neighbor Based Image Classification," IEEE C on feren ce on Com puter Vision a n d P attern R ecogn ition, 20 0 8 (CVPR), 2008, pp. 1-8.

SECTION 6 . 7 REVIEW QUESTIONS

1 . W h at is sp ecial a b o u t th e kN N algorithm? 2 . W h at are th e ad vantages an d disadvantages o f kN N as co m p ared to ANN an d SVM?

3. W h at are the critical su ccess factors for a feNN implementation? 4 . W h at is a similarity (o r d istan ce m easu re)? H o w can it b e applied to b o th num erical

and nom inal valu ed variables? 5 . W h at are the co m m o n ap plications o f &NN?

3 1 0 Part III • Predictive Analytics

Chapter Highlights

• N eural co m p u tin g involves a set o f m ethod s that em ulate the w ay th e h um an brain w ork s. The b asic p ro cessin g unit is a n euron. Multiple n eu ­ ron s a re g ro u p ed into layers an d linked together.

• In a neural n etw ork, the k n o w led ge is stored in th e w eig h t asso ciated w ith e a ch con n ectio n b e tw e e n tw o neurons.

• B ackpropagation is the m ost popular paradigm in business applications o f neural networks. Most busi­ ness applications are handled using this algorithm.

• A b ack p ro p ag atio n -b ased neural n etw ork c o n ­ sists o f an input layer, an ou tput layer, an d a certain n u m b er o f hidden layers (usually o n e). T h e n o d e s in o n e layer a rc fully co n n e cte d to the n o d es in th e n e x t layer. Learning is d o n e through a trial-and-error p ro cess o f adjusting th e c o n n e c ­ tion w eights.

• E ach n o d e at the input layer typically rep resen ts a single attribute th at m ay affect the prediction.

• N eural n etw ork learning c a n o c c u r in supervised o r unsu p ervised m o d e.

• In su pervised learning m o d e, the training patterns in clude a co rre ct an sw er/classification /forecast.

• In unsu p ervised learning m o d e, th ere are n o k now n answ ers. Thus, unsupervised learning is u se d fo r clustering o r exp lo rato ry d ata analysis.

• T h e usual p ro ce ss o f learning in a n eural n etw ork involves th ree step s: ( 1 ) co m p u te tem porary ou tp uts b ased o n inputs and ran d o m w eights, ( 2 ) co m p u te outputs w ith desired targets, and ( 3 ) adjust th e w eights an d rep eat th e p rocess.

T h e delta rule is com m on ly u sed to adjust the w eights. It includes a learning rate and a m om en ­ tu m p aram eter. D eveloping neural n e tw o rk -b a se d system s requires a step -b y-step p ro cess. It includes data p rep aration an d p rep ro cessin g, training and testing, a n d con version o f the train ed m od el into a p ro d u ctio n system . N eural n etw o rk softw are is available to allow ea sy exp erim en tatio n w ith m any m od els. Neural n etw ork m o d u les a re included in all m ajor data mining softw are tools. Specific neural n etw ork p ack ag es a re also available. Som e neural netw ork tools are available as sp read sh eet add-ins.

> After a trained network has b een created, it is usually implemented in end-user systems through program­ ming languages such as C++, Java, and Visual Basic. Most neural network tools can generate co d e for the trained network in these languages.

• M any n eural n etw ork m odels b ey o n d b ack p ro p a­ gation exist, including radial basis functions, sup­ p ort v e c to r m ach in es, H opfield netw orks, and K o h on en ’s self-organizing m aps.

• Neural n etw ork applications ab ou n d in alm ost all b usiness disciplines as w ell as in virtually all o th er functional areas.

• Business applications o f neural n etw orks include finance, b a n k m p tcy p rediction, tim e-series fore­ casting, a n d s o on.

• N ew ap plications o f neural n etw orks are em erg­ ing in h ealth care, security, an d so on.

Key Terms

artificial n eural n etw ork (ANN)

axo n b ack p ro p ag atio n co n n ectio n w eigh t dendrite hidden layer

^-nearest neighbor K ohonen’s self-organizing

feature map neural computing neural network neuron nucleus

parallel p rocessin g p attern recogn ition p ercep tro n p rocessin g elem en t (P E ) sigm oid (logical

activation) function sum m ation function

supervised learning syn apse th reshold value transform ation (transfer)

function

Chapter 6 • T ech n iq u es for Predictive Modeling 3 1 1

Questions for Discussion 1 . D iscuss th e ev olu tion o f ANN. H ow have biological

netw orks contributed to th e d evelopm ent o f artificial networks? H ow are the tw o netw orks similar?

2 . W hat are the m ajor con cep ts related to netw ork informa­ tion processing in ANN? Explain the summ ation and trans­ form ation functions and their com bined effects o n ANN perform ance.

3 . Discuss the com m on ANN architectures. W hat are the main differences betw een K ohonen’s self-organizing feature m aps and Hopfield networks?

4 . Explain th e step s in neural n e tw o rk -b a se d system s devel­ opment? W hat p rocedu res a re involved in b a c k propaga­ tion learning algorithm s and h o w d o they work?

5 . A building so c iety uses neural netw ork to predict the creditworthiness o f m ortgage applicants. Th ere are two output nodes-, o n e for y es (1 — yes, 0 = n o ) and o n e lor no (1 = no, 0 = yes). I f a n applicant scores 0 .8 0 for the “y es” output nod e and 0 .3 9 for the "n o ” output nod e, w hat will be th e ou tcom e o f th e application? W ould the applicant b e a g o o d credit risk?

6 . Stock m arkets c a n b e un pred ictable. Factors that m ay c a u se ch an g es in sto c k p rices are only im per­ fectly kn ow n, a n d understanding them has not b een entirely su ccessfu l. W ould ANN b e a viable solution? Com pare and con trast ANN w ith o th er d ecisio n support tech n olog ies.

1.

Exercises Teradata University Network (TUN) and Other Hands-On Exercises

E xplore W eb sites o f neural netw ork vendors— Alyuda R esearch (alyu d a.com ), A pplied Analytic Systems (aasdt. co m ), B ioC om p Systems, Inc. (biocom p sy stem s.com ), NeuralWare (neu ralw are.com ), and W ard Systems Group, Inc. (w ard system s.com ). Review their products and com ­ pare th em by dow nloading and installing at least two dem o versions. a. W hich real-tim e application at Continental Airlines

may have used a neural network? b. W hat inputs and outputs ca n b e used in building a

neural netw ork application? c. G iven that Continental’s data m ining applications are

in real time, h o w m ight Continental im plem ent a neu­ ral netw ork in practice?

d. W hat o th e r neural netw ork applications w ould you p rop ose fo r the airline industry?

2 . G o to the Teradata University Netw ork W e b site (t e r a d a t a u n iv e r s i t y n e tw o r k .c o m ) o r th e URL given by your instructor. Locate the Harrah’s case. Read th e case and answ er th e follow ing questions: a. W hich o f th e Harrah’s data applications are m ost

likely im plem ented using neural networks? b. W hat o th e r applications cou ld Harrah’s d evelop using

th e data it is collecting from its customers? c. W hat are som e con cern s you might have as a cu s­

tom er at this casino? 3 . T h e bankruptcy-prediction problem c a n b e view ed as a

problem o f classification. T h e data s e t y o u will b e using for this p ro b lem includes five ratios that have b e e n com ­ puted from th e financial statem ents o f real-w orld firms. T h ese five ratios have b e e n used in studies involving bankruptcy prediction. T h e first sam ple includes data on firms that w en t bankrupt and firms that didn’t. This will b e your training sam ple for th e neural netw ork. I h e seco n d sam ple o f 10 firms also consists o f som e bankrup t firms

and som e nonban kru p t firms. Y o u r goal is to u se neural netw orks, support v ecto r m achines, and nearest n eigh bor algorithm s to build a m odel, using the first 20 data points, and th en test its p erform ance o n th e o th er 10 data points. (Tty to analyze th e new cases you rself manually before you run th e n eu ral netw ork and s e e h o w w ell you d o.) The follow ing tables show th e training sam ple and test data you should u s e for this exercise.

T r a in in g Sam ple

Firm W C/TA RE/TA EBIT/TA M VE/TD S/T A BR NB

1 0.1650 0.1192 0.2035 0.8130 1.6702

2 0.1415 0.3868 0.0681 0.5755 1.0579

3 0.5804 0.3331 0.0810 1.1964 1.3572

4 0.2304 0.2960 0.1225 0.4102 3.0809

5 0.3684 0.3913 0.0524 0.1658 1.1533

6 0.1527 0.3344 0.0783 0.7736 1.5046

7 0.1126 0.3071 0.0839 1.3429 1.5736

8 0.0141 0.2366 0.0905 0.5863 1.4651

9 0.2220 0.1797 0.1526 0.3459 1.7237

10 0.2776 0.2567 0.1642 0.2968 1.8904

11 0.2689 0.1729 0.0287 0.1224 0.9277 0

12 0.2039 -0.0476 0.1263 0.8965 1.0457 0

13 0.5056 -0.1951 0.2026 0.5380 1.9514 0

14 0.1759 0.1343 0.0946 0.1955 1.9218 0

15 0.3579 0.1515 0.0812 0.1991 1.4582 0

16 0.2845 0.2038 0.0171 0.3357 1.3258 0

17 0.1209 0.2823 -0.0113 0.3157 2.3219 0

18 0.1254 0.1956 0.0079 0.2073 1.4890 0

19 0.1777 0.0891 0.0695 0.1924 1.6871 0

20 0.2409 0.1660 0.0746 0.2516 1.8524 0

3 1 2 Part III * Predictive Analytics

Firm WC/TA RE/TA

Test Data EBIT/TA MVE/TD S/TA BR/NB

A 0.1759 0.1343 0.0946 0.1955 1.9218 ?

B 0.3732 0.3483 -0.0013 0.3483 1.8223 ?

C 0.1725 0.3238 0.1040 0.8847 0.5576 ?

D 0.1630 0.3555 0.0110 0.37,30 2.8307 ?

E 0.1904 0.2011 0.1329 0.5580 1.6623 ?

F 0.1123 0.2288 0.0100 0.1884 2.7186 ?

G 0.0732 0.3526 0.0587 0.2349 1.7432 7

H 0.2653 0.2683 0.0235 0.5118 1.8350 ?

1 0.1070 0.0787 0.0433 0.1083 1.2051 ?

J 0.2921 0.2390 0.0673 0.3402 0.9277 ?

necessary decisions to p reprocess th e data and build the b e st p ossible predictor. U se your favorite to o l to build the m odels fo r neural netw orks, support vector machines, and nearest n eigh bor algorithms, and d ocum ent the details o f v our results and exp erien ces in a written report. U se screenshots w ithin your report to illustrate important and interesting findings. Y o u are exp ected to discuss and justify any d ecision that you m ake along th e way.

T h e reu se o f this data set is unlim ited w ith retention o f copyright n otice for J o c k A. B lackard and Colorado

State University.

T e a m A s s ig n m e n ts a n d R o le -P la y in g P r o je c t s

1 Consider the follow ing set o f data that relates daily elec- ' tricity usage as a function o f outside high temperature

(for th e day):

D escrib e the results o f th e neural netw ork, support v ec­ to r m ach in es, and nearest n eigh bor m odel predictions, including softw are, architecture, and training information.

4 . T h e p u rpose o f this exercise is to d evelop m odels to predict forest cover type using a nu m ber o f cartographic m easures. T h e given data set (O nlin e File W 6.1) ^ elu d e s four w ilderness areas found in the Roosevelt National Forest o f northern Colorado. A total o f 12 cartographic m easures w ere utilized as ind ep endent variables; seven m ajor forest cover types w ere used as d ep endent variables. T h e follow ing table provides a short descrip­ tion o f th ese ind ep endent and d ep en d en t variables:

T h is is a n ex cellen t exam ple for a multiclass classifi­ cation problem . T h e data set is rather large (w ith }8 1 ,0 1 2 un iqu e instan ces) and feature rich. As you will s e e the data is also raw and skew ed (unbalanced for differ­ en t cov er types). As a model builder, y o u are to m ake

Temperature, X Kilowatts, Y

46.8 12,530

52.1 10,800

55.1 10,180

59.2 9,730

61.9 9,750

66.2 10,230

69.9 11,160

76.8 13,910

79.7 15,110

79.3 15,690

80.2 17,020

83.3 17,880

Number Name Description

1 Elevation

2 Aspect

3 Slope

4 Horizontal_Distance_To_Hydrology

5 Vertical_Distance_To_Hydrology

6 Horizontal_Distance_To_Roadways

7 Hillshade_9am

8 Hillshade_Noon

9 Hillshade_3pm

10 Horizontal_Distance_To_Fire_Points

11 Wilderness_Area (4 binary variables)

12 SoiLType (40 binary variables)

Number 1 Cover Type (7 unique types)

Independent Variables

Elevation in meters

Aspect in degrees azimuth

Slope in degrees Horizontal distance to nearest surface-water features

Vertical distance to nearest surface-water features Horizontal distance to nearest roadway Hill shade index at 9 a .m ., summer solstice Hill shade index at noon, summer solstice Hill shade index at 3 p.m., summer solstice Horizontal distance to nearest wildfire ignition points

Wilderness area designation Soil type designation

Dependent Variable Forest cover type designation

Note: More a b o u t t h . s e . * . o n t o . f f c .

Chapter 6 • T ech n iq u es for Predictive Modeling 3 1 3

a . P lot the raw data. W hat pattern d o you see? W hat do you think is really affecting electricity usage?

b . Solve this problem with lin ear regression Y = a + b X (in a spreadsheet). H ow w ell d oes this work? Plot your results. W hat is wrong? Calculate the sum -of- the-squ ares error and R2.

c . Solve this p roblem by using nonlinear regression. W e recom m en d a quadratic function, Y — a + b\X + &2-X2- H ow w ell d o es this w ork? Plot you r results. Is anything wrong? Calculate th e sum -of-the-squares error and R 2.

d . B reak up th e problem into three section s (lo o k at the p lot). Solve it using th ree lin ear regression m odels— o n e for e a c h section . H ow w ell d o es this w ork? Plot you r results. Calculate th e sum -of-the-squares error and R 1. Is this m odeling ap p roach appropriate? Why o r w h y not?

e. Build a neural netw ork to solve th e original p rob­ lem. (Y o u m ay have to scale th e X and Y values to b e b e tw e e n 0 and 1.) Train it (o n the entire set o f data) and solve th e problem (i.e., m ake predictions fo r ea ch o f th e original data item s). How w ell does this work? Plot your results. Calculate th e sum -of-the- squares error and R2.

f. W hich m eth od w orks b est and why? 2 . Build a real-w orld neural netw ork. Using dem o soft­

ware d ow nload ed from th e W eb (e .g ., NeuroSolutions at n e u r o d im e n s i o n .c o m o r an oth er site), identify real- world data (e .g ., start searching o n the W eb at ic s .u c i . e d u / -m le a r n / M L R e p o s ito r y .h tm l o r use data from an organization w ith w hich som eo n e in your group has a con tact) and build a neural netw ork to m ake predictions. T o p ics m ight inclu de sales forecasts, predicting success in an acad em ic program (e .g ., predict GPA from high sch o o l rating and SAT sco res, b ein g careful to lo o k out for “b ad ” data, su ch as GPAs o f 0 .0 ), o r hou sing prices; o r survey th e class for w eight, gender, and heigh t and try to predict height b ased o n th e o th er tw o factors. Y ou could also u se U.S. Census data o n this b o o k ’s W eb site o r at c e n s u s .g o v , b y state, to identify a relationship betw een edu cation level and incom e. How g o o d are your pred ic­ tions? Com pare th e results to predictions gen erated using standard statistical m ethods (regression ). W hich m ethod is better? H ow cou ld your system b e em bed ded in a DSS for real d ecisio n making?

3 . For e a c h o f th e follow ing applications, w ould it b e b e t­ ter to u se a n eu ral netw ork o r an exp ert system? Explain you r answ ers, including p ossible excep tio n s o r special conditions. a . Sea floor data acquisition b . Snow/rainfall prediction c . Automated voice-inquiry processing system d . Training o f new em ployees e . Handwriting recognition

4 . Consider the follow ing data set, w hich includes three attributes and a classification for adm ission decisions into an MBA program:

GMAT GPA Quantitative

GMAT Decision

650 2.75 35 NO

580 3.50 70 NO

600 3.50 75 YES

450 2.95 80 NO

700 3.25 90 YES

590 3.50 80 YES

400 3.85 45 NO

640 3.50 75 YES

540 3.00 60 ?

690 2.85 80 ?

490 4.00 65 ?

a. Using the data given h ere as exam ples, d evelop your o w n m anual exp ert ru les for d ecision making.

b . Build and te st a neural netw ork m odel using your favorite data m ining tool. E xperim ent w ith different m od el param eters to “optim ize” the predictive pow er o f your m odel.

c. B uild and test a support vector m achine m od el using you r favorite data m ining tool. Experim ent w ith dif­ ferent m od el param eters to “optim ize” th e predictive p o w er o f you r m odel. Com pare the results o f ANN and SVM.

d . Report th e pred ictions o n the last three observa­ tions from e a c h o f the th ree classification approaches (ANN, SVM, and &NN). Com m ent o n the results.

e. Com m ent o n th e similarity and d ifferences o f these three p red iction approaches. W hat did you learn from this exercise?

5- Y o u h av e w o rk ed o n neural netw ork s and o th e r data m ining tech n iq u es. G ive exam p les o f w h ere e a ch o f th ese h as b e e n used. B a sed o n you r kn ow led ge, h o w w ou ld you differentiate am on g th ese techniques? Assume that a fe w years from n ow you com e acro ss a situation in w h ic h n eu ral n etw o rk o r o th e r data min­ ing tech n iq u es co u ld b e u sed to build a n interesting ap p lication for y o u r organization. Y o u have an intern w ork in g w ith you to d o the grunt w ork. H ow w ill you d ecide w h eth er th e ap p lication is approp riate for a n eu ­ ral n etw o rk o r fo r an o th er data m ining m odel? B ased o n you r h o m ew o rk assignm ents, w h at sp ecific softw are gu idan ce c a n y o u provide to get you r intern to b e pro­ d uctive fo r y o u quickly? Y o u r answ er fo r this q u estion m ight m en tion th e sp ecific softw are, d escrib e h o w to g o a b o u t setting up th e model/neural netw ork , and validate th e application.

3 1 4 Part III • Predictive Analytics

Internet Exercises 1 . E xplore W eb sites o f neural netw ork vendors— Alyuda

Research (alyu d a.com ), Applied Analytic Systems (aasdt.com ), BioComp Systems, Inc. (biocom psystem s. com ), NeuralW are (neu ralw are.com ), and Ward Systems G roup, Inc. (w ardsystem s.com ). Review their products and com p are th em by dow nloading and installing at least two d em o versions.

2. A very g o o d rep ository o f data that has b e e n u sed to test th e p erform an ce o f neural netw ork and o th er m ach in e-learn in g algorithm s can b e a c ce sse d at ics.uci. edu/~mlearn/MLRepository.html. Som e o f the data sets a re really m ean t to test the lim its o f cu rren t m achine- learning algorithm s and com p are their p erform ance against n ew ap p roach es to learning. H ow ever, som e o f th e sm aller data sets ca n b e useful for exploring th e fun ctionality o f th e softw are you m ight d ow nload in In tern et E x ercise 1 o r th e softw are that is available at StatSoft.com (i.e ., Statistica Data M iner w ith ex ten ­ sive n eu ral n etw ork cap abilities). D ow n load at least on e data s e t fro m th e UCI rep ository (e .g ., Credit Screen in g D atab ases, H ousing D atabase). T h e n apply neural n et­ w ork s as w ell a s d ecisio n tre e m ethods, as appropriate. P repare a report o n y o u r results. (So m e o f th ese e x e r­ c ises co u ld a ls o b e com p leted in a group o r m ay even b e p ro p o sed a s sem ester-lon g projects for term papers and so o n .)

End-of-Chapter Application Case

Coors Im proves Beer Flavors w ith Neural N etw o rks

C oors B rew ers Ltd., based in B urton-upon-Trent, Britain’s brew ing capital, is proud o f having the United Kingdom 's to p b e e r brands, a 20 p ercent share o f the m arket, years o f exp erien ce, and som e o f the b est p e o p le in the business. Popular b ran d s include Carling (th e country’s bestselling lager), G rolsch , Coors Fine Light B eer, Sol, and Korenw olf.

P r o b le m T oday’s custom er has a w ide variety o f options regarding w hat h e o r sh e drinks. A drinker's c h o ic e dep en ds o n various factors, including m ood , ven u e, and occasion . Coors goal is to ensure that the custom er c h o o se s a Coors bran d n o matter w h at th e circu m stan ces are.

A ccording to Coors, creativity is the key to long-term su ccess. T o b e th e custom er’s c h o ic e brand, Coors need s to b e creative and anticipate the custom er’s ever so rapidly changing m ood s. An im portant issue w ith b eers is the flavor; each b e e r h as a distinctive flavor. T h ese flavors are m ostly determ ined throu gh panel tests. H ow ever, su ch tests take time. I f Coors cou ld understand the b e e r flavor based solely o n its chem ical com position, it w ould o p e n up n ew avenues to create b e e r that w ould suit custom er expectations.

T h e relationship b e tw e e n chem ical analysis and beer flavor is not clearly understood yet. Substantial data exist

3 . G o to calsci.com and read about th e com p an y’s various business applications. Prepare a report that summarizes th e applications.

4 . G o to nd.com. Read about the com pany’s applications in investm ent and trading. Prepare a report ab o u t them.

5 . G o to nd.com. D ow nload the trial version of N euroSolutions for E x cel and exp erim en t w ith it, using o n e o f the data sets from the exercises in this chapter. Prepare a rep ort about you r e x p erien c e w ith th e tool.

6 . G o to neoxi.com. Identify at least tw o softw are tools that have n o t b e e n m en tioned in this chapter. Visit W eb sites o f th o se tools and prepare a b rief report on the tools’ capabilities.

7 . G o to neuroshell.com. Look at G e e W hiz exam ples. Com m ent o n th e feasibility o f achieving th e results claim ed b y th e developers o f this neural netw ork model.

8 . G o to easynn.com. Dow nload th e trial version o f the software. After th e installation o f the software, find the sam ple file called Houseprices.tvq. Retrain th e neural netw ork a n d test the m odel by supplying som e data. Prepare a report about your exp erien ce with this software.

9. Visit statsoft.com. D ow nload at least three w hite papers o f applications. W hich o f th ese applications m ay have used neural networks?

10. G o to neuralware.com. Prepare a report about the products th e com pany offers.

o n the chem ical com position o f a b e e r and sensory analy­ sis. Coors n e e d ed a m echanism to link th o se tw o together. Neural netw orks w ere applied to create the link betw een chem ical com p osition and sensory analysis.

S o lu tio n O ver th e years, Coors Brew ers Ltd. has accum ulated a sig­ nificant am ount o f data related to th e final product analy­ sis, w h ich has b e e n supplem ented by sen sory data provided b y th e trained in-hou se testing panel. Som e o f th e analytical inputs and sen sory outputs are show n in the follow ing table:

Analytical Data: Inputs Sensory Data: Outputs

Alcohol Alcohol

Color Estery

Calculated bitterness Malty

Ethyl acetate Grainy

Isobutyl acetate Burnt

Ethyl butyrate Hoppy

Isoamyl acetate Toffee

Ethyl hexanoate Sweet

Chapter 6 • T ech n iq u es for Predictive Modeling 3 1 5

A single n eu ral netw ork, restricted to a single quality and flavor, w as first used to m od el the relationship betw een the analytical and sen so ry data. T h e neural netw ork w as based o n a p ack ag e solution supplied by N euroD im ension, Inc. ( n d .c o m ). T h e neural netw ork con sisted o f an MLP archi­ tecture w ith tw o hidden layers. Data w ere norm alized within the netw ork, th ereby enablin g com parison betw een the results for the various sensory outputs. T h e neural netw ork w as trained (to learn th e relationship betw een th e inputs and outputs) throu gh th e presentation o f m any com bina­ tions o f relevan t input/output com binations. W hen there was no observed im provem ent in the netw ork error in the last 100 ep o ch s, training w as autom atically terminated. Training w as carried o u t 50 times to ensure that a con sid erable m ean netw ork error co u ld b e calculated for com parison purposes. Prior to ea ch training run, a different training and cross- validation data set was p resented b y random izing the source data records, th ereb y rem oving any bias.

T h is tech n iq u e prod uced p o o r results, due to two m ajor factors. First, concentrating o n a single p rod uct’s quality m eant that th e variation in th e data was pretty low. T h e neural n etw o rk cou ld not extract useful relationships from th e data. Second , it w as probable that only o n e subset o f the provided inputs w ould have an im pact o n the selected b e e r flavor. P erform an ce o f th e neural netw ork w'as affected b y “n o ise" created by inputs that had n o impact o n flavor.

A more diverse product range was included in the training range to address the first factor. It was more challenging to identify the m ost important analytical inputs. This challenge was addressed by using a software switch that enabled the neural network to b e trained o n all possible com binations o f inputs. T h e switch w as not used to disable a significant input; if the significant input w ere disabled, w e could exp ect the net­ w ork eiTor to increase. If the disabled input w as insignificant, then the network en-or would either remain unchanged o r be reduced due to the removal o f noise. This approach is called an ex h a u s tiv e s e a r c h becau se all possible com binations are evalu­ ated. T h e technique, although conceptually simple, w as com ­ putationally impractical with the numerous inputs; the number o f possible com binations w as 16.7 million per flavor.

A m ore efficient m ethod o f searching for the relevant inputs w as required. A g en etic algorithm was the solution to the problem . A gen etic algorithm w as ab le to m anipulate the different input sw itches in response to the error term from the neural netw ork. T h e o b jectiv e o f the gen etic algorithm was to m inim ize the netw ork error term. W hen this minimum was reached, th e sw itch settings w ould identify the analytical inputs that w e re m ost likely to predict the flavor.

References Ainscough, T . L., and J . E. A ronson. (1 9 9 9 ). “A Neural

Networks A pproach for th e Analysis o f Scan ner D ata.” J o u r n a l o f R e ta ilin g a n d C o n s u m e r S erv ices. Vol. 6.

Aizerman, M ., E. Braverm an, and L. Rozonoer. (1964). “Theoretical Foundations o f the Potential Function Method

R e s u lts After determ ining w hat inputs w ere relevant, it w as p ossible to identify w h ich flavors cou ld b e predicted m ore skillfully. T h e netw ork w as trained using the relevant inputs previously identified m ultiple times. B efo re e a c h training run, the net­ w ork data w ere random ized to ensure that a different train­ ing and cross-validation data set w as used. Network error w as record ed after each training run. T h e testing set used for assessing th e perform an ce o f th e trained netw ork contained approxim ately 8 0 records out o f th e sam ple data. T h e neu­ ral netw ork accu rately pred icted a few flavors by using the chem ical inputs. F o r exam p le, “burnt" flavor w as predicted w ith a correlation coefficien t o f 0.87.

Today, a lim ited nu m ber o f flavors are b ein g predicted b y using the analytical data. Sensory resp on se is extrem ely c om p lex, with m an y potential interactions and hugely vari­ a b le sensitivity thresholds. Standard instrum ental analysis tend s to b e o f gross param eters, and for practical and e c o ­ n om ical reasons, m any flavor-active com poun ds are simply not m easured. T h e relationship o f flavor and analysis can b e effectively m od eled only if a large nu m ber o f flavor- contributory analytes are considered. W hat is m ore, in addi­ tion to th e obviou s flavor-active m aterials, m outh-feel and physical contributors should also b e con sid ered in th e over­ all sen sory profile. W ith further d evelopm ent o f th e input param eters, the accu racy o f the neural netw ork m odels will improve.

Q u e s t i o n s f o r t h e E n d - o f - C h a p t e r A p p l i c a t i o n C a s e

1 . W hy is b e e r flavor im portant to Coors' profitability? 2. W hat is th e o b jectiv e o f the neural netw ork used at

Coors? 3 . W hy w ere th e results o f Coors’ neural netw ork initially

poor, and w h at w as d one to im prove th e results? 4 . W hat b en efits m ight Coors derive if this p roject is

successful? 5. W hat m odifications w ould you m ake to im prove the

results o f b e e r flavor prediction?

Sources: Compiled from C. I. Wilson and L. Threapleton, "Application o f Artificial Intelligence for Predicting Beer Flavours from Chemical Analysis,” P roceedin gs o f th e 29th E uropean B rew ery Congress. Dublin, Ireland, May 1 7 -2 2 , 2003- neurosolutions.com /resources/ apps/beer.htm l (accessed February 2013); and R. Nischwitz, M. Goldsmith, M. Lees, P. Rogers, and L. MacLeod, “Developing Functional Malt Specifications for Improved Brewing Performance,” The Regional Institute Ltd., re g io n al.o rg .au /au /ab ts/1999/ nischwitz.htm (accessed February 2013).

in Pattern R ecognition Learning.” A u t o m a tio n a n d R e m o te C on trol, Vol. 25, pp. 8 2 1 -8 3 7 .

Altman, E. 1. (1 9 6 8 ). “Financial Ratios, D iscrim inant Analysis and the P rediction o f C orporate Bankruptcy.” J o u r n a l o f F i n a n c e , Vol. 23.

3 1 6 Part III • Predictive Analytics

California Scientific. “M axim ize Returns o n D irect Mail with BrainM aker Neural Networks Softw are.” calsci.com / DirectMail.html (a c cessed August 2009).

Collard, J . E. (1 9 9 0 ). “Com m odity Trading w ith a Neural N et.” N e u r a l N e tw o rk News, Vol. 2, No. 10.

Collins, E., S. G h osh , and C. L. Scofield. (1988). “An Application o f a Multiple Neural Network Learning System to Emulation o f Mortgage Underwriting Jud gm ents.” IEEE I n t e r n a t io n a l C o n fe r e n c e o n N e u r a l N etw orks, Vol. 2, pp. 4 5 9 -4 6 6 .

D as, R., I. Turkoglu, and A. Sengur. (2009). “Effective Diagnosis o f Heart D isease Through Neural Networks Ensem bles.” E x p ert S ystem s w ith A p p lica tio n s, Vol. 36, pp. 7 6 7 5 -7 6 8 0 .

Davis, J . T ., A. Episcop os, and S. W ettim uny. (2001). “Predicting D irection Shifts o n C anadian-U .S. E xchange Rates w ith Artificial Neural N etw orks.” I n t e r n a t io n a l J o u r n a l o f In te llig e n t S ystem s in A c c o u n tin g , F i n a n c e a n d M a n a g e m e n t , Vol. 10, No. 2.

D elen , D ., and E. Sirakaya. (2 0 0 6 ). “D eterm ining th e Efficacy o f Data-M ining M ethods in Predicting Gam ing B allot O u tcom es.” J o u r n a l o f H o s p ita lity & T o u rism R e s e a r c h , Vol. 30, No. 3, pp. 3 1 3 -3 3 2 .

D elen , D ., R. Sharda, and M. B esson ov. (2 0 0 6 ). “Identifying Significant Predictors o f Injury Severity in Traffic Accidents U sing a Series o f Artificial Neural N etw orks.” A c c id e n t A n a ly s is a n d P rev e n tio n , Vol. 38, No. 3, pp. 4 3 4 -4 4 4 .

Dutta, S., and S. Shakhar. (1 9 8 8 , Ju ly 2 4 -2 7 ). “Bond-Rating: A N on-Conservative A pplication o f Neural Networks/' P r o c e e d in g s o f t h e IEEE I n t e r n a t io n a l C o n f e r e n c e o n N e u r a l N etw orks, San D iego, CA.

Estevez, P. A ., M. H. Claudio, and C. A. Perez. “Prevention in Telecom m u nications Using Fuzzy Rules and Neural N etw orks.” cec.uchile.cl/~pestevez/RIO.pdf (accessed May 2009).

Fadlalla, A., and C. Lin. (2001). “An Analysis o f the Applications o f Neural Networks in F in an ce.” In te r fa c e s , Vol. 31, No. 4.

Fishm an, M., D. Barr, and W . Loick. (1991, April). “Using Neural N etw orks in Market Analysis.” T e c h n ic a l A n a ly sis o f S to ck s a n d C o m m o d ities .

Fozzard, R., G. Bradshaw, and L. Ceci. (1989)- ‘A Connectionist Expert System for Solar Flare Forecasting.” In D. S. Touretsky (ed .), A d v a n c e s in N e u r a l In fo r m a t io n P ro c ess in g Systems, Vol. 1. San M ateo, CA: Kaufman.

Francett, B . (1 9 8 9 , Janu ary). “Neural N ets Arrive.” C o m p u te r D e c is io n s .

Gallant, S. (1 9 8 8 , February). “Connectionist E x p eit System s.” C o m m u n ic a t io n s o f t h e ACM, Vol. 31, No. 2.

Giiler, I., Z. Gok^il, and E. Gulbandilar. (2 0 0 9 ). “Evaluating Traum atic B ra in Injuries Using Artificial Neural Networks." E x p e r t S ystem s w ith A p p lic a tio n s, Vol. 36, pp. 1 0 4 2 4 -1 0 4 2 7 .

Haykin, S. S. (2 0 0 9 ). N e u r a l N etw o rk s a n d l e a r n i n g M a ch in e s, 3rd ed. U pper Saddle River, NJ: P rentice Hall.

Hill, T ., T . M arquez, M. O ’C onnor, and M. Remus. (1994). “Neural N etw ork M odels for Forecasting and D ecision M aking.” I n t e r n a t i o n a l J o u r n a l o f F o r e c a s tin g , Vol. 10.

H opfield, J . (1 9 8 2 , April). “Neural Networks and Physical System s with Em ergent Collective Computational

A bilities.” P r o c e e d in g s o f N a t io n a l A c a d e m y o f S c ie n ce, Vol. 79, No. 8.

H opfield, J . J ., and D. W . Tank. (1 9 8 5 ). “Neural Com putation o f D ecision s in Optim ization P roblem s.” B io lo g i c a l C y b ern etics, Vol. 52.

Iyer, S. R., and R. Sharda. (2 0 0 9 ). “Prediction o f A thletes’ P erform ance U sing Neural N etworks: An A pplication in Cricket T eam S e le c tio n .” E x p ert S ystem s w ith A p p lica tio n s, Vol. 36, No. 3, pp. 5 5 1 0 -5 5 2 2 .

Kam ijo, K., and T . Tanigaw a. (1990, Ju n e 7 -1 1 ) . “Stock P rice Pattern Recognition: A Recurrent Neural Network A pproach.” I n t e r n a t i o n a l J o i n t C o n fe r e n c e o n N e u r a l N etw orks, San D iego.

Lee, P. Y ., S. C. Hui, and A. C. M. Fong. (2 0 0 2 , September/ O cto b er). “Neural Networks for W eb C ontent Filtering.” fF.F.F. I n te llig e n t System s.

Liang, T. P. (1 9 9 2 ). “A Com posite A pproach to Automated Know ledge A cquisition.” M a n a g e m e n t S c ie n c e , Vol. 38, No. 1.

Loeffelholz, B ., E. B ednar, and K. W . B auer. (2009). “Predicting N BA G am es Using Neural N etw orks.” J o u r n a l o f Q u a n t it a t iv e A n a ly s is in Sports, Vol. 5, No. 1.

M cCulloch, W. S., and W . H. Pitts. (1 9 4 3 ). “A Logical Calculus o f the Ideas Im m inent in Nervous Activity.” B u lle tin o f M a t h e m a t ic a l B io p h y s ic s , Vol. 5.

M edsker, L., and J . Liebowitz. (1 9 9 4 ). D esig n a n d D e v e lo p m e n t o f E x p ert System s a n d N e u r a l N etw orks. New Y ork : M acm illan, p. 163.

Mighell, D. (1 9 8 9 ). “Back-Propagation and Its Application to H andwritten Signature V erification.” In D. S. Touretsky (ed .), A d v a n c e s in N e u r a l I n f o r m a t i o n P r o c e s s in g System s. San M ateo, CA: Kaufman.

Minsky, M., and S. Papert. (1 9 6 9 ). P er c e p tr o n s . Cambridge, MA: MIT Press.

Neural T ech n o lo g ies. “Com bating Fraud: How a Leading T e le c o m C om pany Solved a Grow ing Problem ." neuralt. com /iqs/dlsfa.list/dlcpti.7/downloads.html ( accessed M arch 2009)-

Nischwitz, R., M. Goldsm ith, M. Lees, P. Rogers, and L. MacLeod. “D evelop ing Functional Malt Specifications for Im proved B rew in g P erform an ce.” T h e Regional Institute Ltd., regional.org.au/au/ a b ts /1 9 9 9 / nischwitz.htm (a c cessed May 2009).

O lson, D. L., D. D elen , and Y . Meng. (2 0 1 2 ). “Comparative Analysis o f D ata M ining Models for Bankruptcy Prediction.” D e c is io n S u p p o r t System s, Vol. 52, No. 2, pp. 4 6 4 -4 7 3 .

P iatesky-Shapiro, G. “ISR: M icrosoft Success Using Neural Network fo r D irect M arketing.” kdnuggets.com/ n e w s/9 4 /n 9 -tx t (accessed May 2009).

Principe, J . C., N. R. Euliano, and W . C. Lefebvre. (2000). N e u r a l a n d A d a p tiv e System s: F u n d a m e n t a l s T h ro u g h S im u la tio n s . N ew Y ork: Wiley.

R ochester, J . (e d .). (1 9 9 0 , February). “New B u sin ess U ses for N eurocom puting.” I/S A n a ly z e r .

Sirakaya, E., D. D elen , and H-S. Choi. (2005). “Forecasting G am ing R eferend a.” A n n a ls o f T o u rism R e s e a r c h , Vol. 32, No. 1, pp. 127 -1 4 9 -

Chapter 6 • T ech n iq u es fo r Predictive M odeling 3 1 7

Sordo, M_, H. B u xton , and D. W atson. (2 0 0 1 ). “A Hybrid A pproach to B reast C an cer D iagnosis.” In L. Ja in and P. D eW ilde (e d s .), P r a c t ic a l A p p lic a tio n s o f C o m p u t a t io n a l In t e llig e n c e T e c h n iq u e s , Vol. 16. Norwell, MA: Kluwer.

Surkan, A., and J . Singleton. (1 9 9 0 ). “Neural Networks for B ond Rating Im proved by Multiple Hidden Layers.” P r o c e e d in g s o f th e IE EE I n t e r n a t i o n a l C o n fe r e n c e o n N e u r a l N etw orks, Vol. 2.

Tang, Z., C. d e Almieda, and P. Fishwick. (1 9 9 1 ). “Tim e- Series Forecastin g Using Neural Networks vs. B ox-Jen kin s M ethodology.” S im u la tio n , Vol. 57, No. 5.

Thaler, S. L. (2 0 0 2 , January/February). “Al fo r Network P rotection: LITMUS:— Live Intrusion Tracking via Multiple U nsupervised STANNOs.” P C Ad.

W alczak, S., W . E. Pofahi, and R. J . Scorpio. (2002). “A D ecisio n Support T o o l for Allocating Hospital B ed R esources a n d D eterm ining Required Acuity o f C are.” D e c is io n S u p p o r t Systems, Vol. 34, No. 4.

W allace, M. P. (2 0 0 8 , Ju ly ). “Neural Networks and Their A pplications in F in an ce.” B u s in e s s I n t e llig e n c e J o u r n a l , pp. 6 7 -7 6 .

W en, U -P., K-M. Lan, and H-S. Shih. (2 0 0 9 ). “A Review o f H opfield Neural Networks for Solving Mathematical Program m ing P roblem s.” E u r o p e a n J o u r n a l o f O p e r a t io n a l R e s e a r c h , Vol. 198, pp. 6 7 5 -6 8 7 .

W ilson, C. I., a n d L. Threapleton. (20 0 3 , May 1 7 -2 2 ). "Application o f Artificial In telligence for Predicting B e e r Flavours fro m Chem ical Analysis.” P r o c e e d in g s o f t h e 2 9 th E u r o p e a n B r e w e r y C on gress, D ublin, Ireland. n e u r o s o l u t i o n s . c o m / r e s o u r c e s / a p p s / b e e r . h t m l

(a c cessed May 2009). W ilson, R., and R. Sharda. (1 9 9 4 ). “Bankruptcy Prediction

U sing Neural N etw orks.” D e c is io n S u p p o r t System s, Vol. 11.

Zahedi, F. (1 9 9 3 ). In te llig e n t System s f o r B u sin ess : E x p ert S ystem s w ith N e u r a l N etw orks. Belm on t, CA: Wadsworth.

Text Analytics, Text Mining, and Sentiment Analysis

L E A R N I N G O B J E C T I V E S

■ Describe text mining and understand the need for text mining

■ Differentiate among text analytics, text mining, and data mining

■ Understand the different application areas for text mining

■ Know the process for carrying out a text mining project

■ Appreciate the different methods to introduce structure to text-based data

* D escrib e sentim ent analysis

■ D ev elop familiarity w ith p opular applications o f sentim ent analysis

■ Learn the co m m o n m ethod s for sen tim ent analysis

* B e c o m e familiar w ith sp e e c h analytics as it relates to sentim ent analysis

"1 his ch a p te r p rovides a rather co m p reh en siv e overview o f te x t mining and o n e of its m ost p op u lar applications, sentim ent analysis, as they b oth relate to business analytics an d d ecision su p p o rt system s. G enerally speaking, sentim ent analysis

is a derivative o f te x t m ining, an d te x t mining is essentially a derivative o f data mining. B e ca u se textual d ata is increasing in volu m e m o re than the data in structured d atabases, it is im portant to k n o w so m e o f the tech n iq u es u se d to extract actionab le inform ation from this large quantity o f unstructured data.

7 .1 O p e n in g V ig n ette: M ach in e V ersus M en o n Jeopardy!-. T h e S tory o f W a tso n 3 1 9 7 .2 T e x t A n alytics an d T e x t M ining C o n c e p ts an d D efin ition s 3 2 1 7 . 3 N atural L a n g u a g e P ro c e s s in g 3 2 6 7 . 4 T e x t M ining A p p lica tio n s 3 3 0

7 .5 T e x t M ining P ro c e s s 3 3 7 7 . 6 T e x t M ining T o o ls 3 4 7 7 .7 Sentiment Analysis Overview 3 4 9 7 . 8 S en tim en t A n alysis A p p licatio n s 3 5 3 7 . 9 S en tim en t A nalysis P ro c e s s 3 5 5

7 . 1 0 S en tim en t A nalysis a n d S p e e ch A n alytics 3 5 9

3 1 8

7.1 OPENING VIGNETTE: Machine Versus Men on Jeopardy!: The Story of Watson

Chapter 7 • T e x t Analytics, T ex t Mining, and Sentim ent Analysis

Can m ach in e b eat the b est o f m an in w h at m an is su p p osed to b e th e b est at? Evidently, y es, an d th e m ach in e’s n am e is W atson . W atson is an extraord in ary co m p u te r system (a novel com b in ation o f a d v an ced hard w are an d softw are) d esign ed to a n sw e r questions p o sed in n atu ral h um an lan gu age. It w as d ev elo p ed in 2 0 1 0 b y an IBM R esearch team as part o f a D eep Q A p roject and w a s n am ed after IBM’s first president, T h o m as J. W atson.

BACKGROUND

Roughly 3 y ears ag o , IBM R esearch w as look in g fo r a m ajor research ch allen ge to rival th e scientific an d p op u lar interest o f D eep Blue, th e co m p u te r ch ess-p layin g ch am p ion , w h ich w o u ld also h ave clear relevan ce to IBM business interests. T h e g o a l w as to ad v an ce com p u ter s cie n ce b y exp loring n e w w ay s fo r co m p u ter tech n o lo g y to affect scien ce, busi­ n ess, and society. A ccordingly, IBM R esearch u n d ertoo k a ch allen ge to build a com p u ter system that co u ld co m p e te at th e hum an ch am p io n level in real time o n th e A m erican TV quiz sh o w , J eo p ard y ! T h e exten t o f the ch allen ge included fielding a real-tim e au tom atic con testan t o n th e sh o w , cap ab le o f listening, understanding, an d resp on ding— n o t m erely a lab oratory exercise.

COMPETING AGAINST THE BEST

In 2 0 1 1 , as a test o f its abilities, W atson co m p eted o n the quiz sh o w Jeopardy/, w hich was the first e v e r hum an-versus-m achine m atchup for th e show . In a tw o-gam e, com bined-point m atch (b road cast in th ree Jeopardy! episodes during February 1 4 -1 6 ) , W atso n b eat Brad Rutter, the b iggest all-time m on ey w inner o n Jeopardy!, and Ken Jennings, the reco rd holder for the longest cham pionship streak (7 5 days). In th ese episodes, W atson consistently out­ perform ed its hum an op p on en ts o n the gam e’s signaling device, but had trouble respond­ ing to a fe w categories, notably th ose having short clues containing on ly a few words. W atson h ad access to 2 0 0 million p ages o f structured and unstructured con ten t consum ing four terabytes o f disk storage. During the g am e W atson w as not co n n ected to the Internet.

M eeting th e Jeopardy! Challenge required advancing and in corporating a variety o f QA tech n ologies (te x t mining and natural language p rocessin g ) including parsing, question classification, question d ecom p osition, au tom atic so u rce acquisition and evaluation, entity and relation d etection, logical form gen eration, and know led ge rep resentation and reason ­ ing. W inning a t Jeopardy! required accu rately com puting co n fid en ce in y o u r answ ers. The questions an d co n ten t are am biguous and noisy an d n o n e o f the individual algorithms are

3 2 0 Part III • Predictive Analytics

p e r f e c t T h e r e f o r e , e a c h c o m p o n e n t m u s t p r o d u c e a c o n fid e n c e S to its

s h o u ld risk c h o o s in g t o a n s w e r a t all. I n J e o p a r d y ! p a r la n c e , t is c o n 1 o n f id e n c e

in . T h is is ro u g h ly b e t w e e n 1 a n d 6 s e c o n d s w it h a n a v e r a g e a r o u n d 3 s e c o n d .

HOW DOES WATSON DO IT? T h e s y s te m b e h in d W a ts o n , w h ic h is c a ll e d D e e p Q A , is a m a s s iv e ly p a r a lle l, t e x t m i m n g -

fo c u s e d , p r o b a b ilis t ic e v id e n c e - b a s e d c o m p u ta tio n a l , p n o p W a t s o n u s e d m o r e t h a n 1 0 0 d iffe r e n t t e c h n i q u e s f o r a n a ly z in g n a tu ra l ia n g u a g ,

^ D e ' e p Q A i s ^ n l r c t t e c t u r e w ith a n a c c o m p a n y i n g m e th o d o lo g y , w h i c h is n o t s p e c ific

— p^ s i = i s ^ s ^ x s s s s z in t e x t a n a ly tic s.

. M a s s i v e p a r a l l e l i s m : E x p lo it m a s s iv e p a r a lle lis m in t h e c o n s i d e r a t io n o f m u l-

. a p p lic a tio n , a n d c o n t e x t u a l e v a lu a tio n o f

a w id e r a n g e o f l o o s e ly c o u p l e d p r o b a b ilis t ic q u e s t io n a n d c o n t e n t a n a ly tic s . . P e n a ^ e c o n f i d e n c e e s t i m a t i o n : N o c o m p o n e n t c o m m its to a n a n s w e r ; all

r e d u c e f e a tu r e s a n d a s s o c ia t e d c o n f id e n c e s , s c o r in g d iffe re n t: q u e s ­ t i o n ^ a n d f o m e n t in te r p r e ta tio n s . A n u n d e r ly in g c o n f id e n c e - p r o c e s s i n g s u b s tr a te

1 p i r n s h o w to s ta c k a n d c o m b i n e t h e s c o r e s . • I n t e g r a t e s h a l l o w a n d d e e p k n o w l e d g e : B a l a n c e t h e u s e o f s tric t s e m a n tic s

a n d s h a llo w s e m a n tic s , le v e r a g in g m a n y l o o s e l y fo r m e d o n to lo g ie s .

Ficm re 7 1 illu s tra te s t h e D e e p Q A a r c h ite c tu r e a t a v e r y h ig h le v e l. M o re t e c h n ic a l d e t a i l s ^ b o u t t h e v a r io u s a r c h ite c tu r a l c o m p o n e n t s a n d t h e ir s p e c i f ic r o l e s a n d c a p a b i l i t y

c a n b e fo u n d in F e r r u c c i e t a l. ( 2 0 1 0 ) .

Prim ary search ; retrieval

Hypothesis and evidence scoring

Hypothesis and evidence scoring Answ er and confidence

Hypothesis generation

Answ er Evidence sources sources

Question Query Hypothesis analysis decomposition generation

- > Soft

filtering

- Soft

filtering

Trained models

-

j Synthesis •>- Final m erging and ranking

.

FIGURE 7 .1 A H igh-Level Depiction o f DeepQ A Architecture.

Chapter 7 • T e x t Analytics, T ex t Mining, and Sentim ent Analysis 321

CONCLUSION

T h e Je o p a r d y ! c h a lle n g e h e lp e d IB M ad d ress req u irem en ts th at led to th e d esig n o f the D e e p Q A arch ite ctu re a n d th e im p lem e n ta tio n o f W atso n . A fter 3 y e ars o f in te n se re se a rch a n d d e v e lo p m e n t b y a c o r e team o f a b o u t 2 0 re se a rch e rs, W atso n is p e rfo rm in g a t hu m an e x p e rt lev els in term s o f p re cisio n , co n fid e n c e , an d s p e e d a t th e Jeo p a rd y ! q u iz show .

IB M claim s to h av e d e v e lo p e d m an y com p u tatio n al an d lingu istic algorithm s to ad dress d iffe ren t kin d s o f issu es an d requ irem en ts in QA. E ven th o u g h th e internals o f th e s e alg orithm s are n o t k n o w n , it is im perative that th ey m ad e th e m o st o u t o f te x t analyt­ ics an d te x t m ining. N ow IB M is w o rk in g o n a v e rsio n o f W atson to ta k e o n su rm ou ntable p ro b lem s in h e alth care a n d m e d icin e (F e ld m an et al., 2012).

QUESTIONS FO R TH E OPENING VIGNETTE

1 . W h a t is W atson? W h a t is s p e c ia l a b o u t it? 2 . W h a t te c h n o lo g ie s w e re u s e d in b u ild in g W atson (b o th hard w are an d softw are)? 3 . W h a t a re th e innov ativ e ch aracteristics o f D ee p Q A arch ite ctu re th at m a d e W atson

superior? 4 . W h y d id IB M sp e n d all that tim e an d m o n e y to b u ild W atson? W h e re is th e ROI? 5 . C o n d u ct a n In te rn e t s e a rch to id entify o th e r p reviou sly d ev elo p e d “sm art m ach in es'

(b y IBM o r o th e rs) th at c o m p e te against th e b e s t o f m an. W h at te c h n o lo g ie s did

th e y use?

WHAT W E CAN LEARN FROM THIS VIGNETTE

It is safe to say th at com p u ter tech n o lo g y , o n b o th the hardw are and softw are fronts, is advancing faster th an anything e lse in th e last 50-plus years. Things that w e re to o b ig, to o co m p le x , im p ossible to solve are n o w w ell within th e reach o f inform ation tech n ology . O n e o f th o se e n ab lin g tech n o lo g ies is p erhap s te x t analytics/text m ining. W e cre a te d databases to structure th e data so that it c a n b e p ro cesse d b y com puters. T e x t, o n th e o th er hand, has alw ays b e e n m ean t for hum ans to process. Can m ach in es d o th e things th a t require hum an creativity and intelligence, and w h ich w e re n o t originally designed fo r m achines? Evidently, yes! W atson is a g reat e x am p le o f th e distance that w e have traveled in addressing the im pos­ sible. C om puters are n o w intelligent en o u g h to take o n m en at w h at w e think m e n are the b e s t at. U nderstanding th e q u estio n that w as p o se d in sp o k e n hu m an language, processing and digesting it, searching fo r an answ er, and replying w ithin a fe w se co n d s w as som ething that w e co u ld n o t h ave im agined p o ssible b e fo re W atson actually did it. In this chapter, you w ill learn th e to o ls and tech n iq u es em b ed d ed in W atson and m an y o th er sm art m ach in es to create m iracles in tackling p roblem s that w e re o n c e b elie v e d im possible to solve.

Sources: D. Ferrucci, E. Brown, J . Chu-Carroll, J . Fan, D. Gondek, A. A. Kalyanpur, A. Lally, J. W. Murdock, E. Nyberg, J . Prager, N. Schlaefer, and C. Welty, “Building Watson: An Overview o f the DeepQA Project,” AI Magazine, Vol. 31, No. 3, 2010; DeepQA, DeepQA Project: FAQ: IBM Corporation, 2011, research.ibm. com/deepqa/faq.shtml (accessed January 2013); and S. Feldman, J . Hanover, C. Burghard, and D. Schubmehl, “Unlocking the Power o f Unstructured Data,” IBM white paper, 2012, www-0 1 .ibm.com/software/ebusiness/ jstart/downloads/unlockingUnstructuredData.pdf (accessed February 2013).

7.2 TEXT A N A LY T IC S A N D TEXT M IN IN G CONCEPTS A N D D EFINITIO N S T h e in fo rm atio n age th at w e are living in is ch aracterize d b y th e rap id gro w th in the a m o u n t o f d ata an d in form ation c o lle cte d , stored , a n d m a d e av ailab le in e le ctro n ic form at. T h e vast m ajo rity o f b u sin e ss data is sto red in te x t d o cu m en ts th at are virtually u n stru c­ tured. A cco rd in g to a study b y M errill L ynch and G artner, 8 5 p e rce n t o f all co rp o ra te data

322 PartHI • Predictive Analytics

is cap tu red a n d sto red in so m e sort o f u n stru ctu red form (M cK night, 2 0 0 5 ). T h e sam e study also stated th at this unstructured data is d o u b lin g in siz e e v e iy 1 8 m onths. B e c a u s e k n o w le d g e is p o w e r in to d ay’s b u sin e ss w orld, an d k n o w le d g e is d eriv ed fro m d ata an d in form ation, b u s in e s s e s that effectiv ely an d e fficie n tly tap into th eir te x t d ata s o u rce s will h av e the n e cessa ry k n o w le d g e to m a k e b e tte r d e cisio n s, lead in g to a com p etitiv e ad van­ ta g e o v e r th o se b u s in e s s e s th at lag b eh in d . T h is is w h e re th e n e e d fo r te x t an aly tics an d

te x t m ining fits in to th e b ig p ictu re o f to d ay’s b u sin esses. E v e n th o u g h th e o v erarch in g g o al fo r b o th te x t an aly tics a n d te x t m in in g is to turn

u n stru ctu red textu al d ata in to a ctio n a b le in fo rm atio n th ro u g h th e a p p lica tio n o f natural la n g u a g e p ro ce s s in g (N LP) a n d analy tics, th eir d efin itio n s are so m e w h a t different, at least to so m e e x p e rts in th e field. A cco rd in g to th em , te x t an alytics is a b ro a d e r c o n c e p t that in clu d es in form ation retrieval (e .g ., s e a rch in g and identifying relev an t d o cu m en ts fo r a g iv e n set o f k e y term s) a s w e ll as in form ation e x tra ctio n , d ata m ining, an d W e b m ining, w h e re a s te x t m in in g is prim arily fo c u s e d o n d isco v e rin g n e w a n d u sefu l k n o w le d g e fro m th e textu al d ata so u rce s. Figure 7 .2 illustrates th e re latio n sh ip s b e tw e e n te x t an alytics an d te x t m ining a lo n g w ith o th e r re la ted a p p lica tio n areas. T h e b o tto m o f Figure 7 .2 lists th e m ain d iscip lin es (th e fo u n d atio n o f th e h o u s e ) that p la y a critical ro le in th e d ev elo p m en t o f th e s e in creasin g ly m o re p o p u la r a p p lica tio n a re a s. B a s e d o n this d efin itio n o f te x t an alytics a n d te x t m ining, o n e co u ld sim p ly fo rm u late th e d iffe re n ce b e tw e e n th e tw o as

fo llow s:

T e x t A n a ly tic s = In fo r m a tio n R e tr ie v a l + I n fo r m a tio n E x tr a c tio n + D a ta M in in g

+ W e b M in in g ,

o r sim ply

T e x t A n a ly tic s = I n f o r m a tio n R e trie v a l + T e x t M in in g

FIG U R E 7 .2 Text Analytics, Related Application Areas, and Enabling Disciplines.

Chapter 7 * T ex t Analytics, T e x t Mining, and Sentim ent Analysis 3 2 3

C o m p are d to text m ining, te x t an aly tics is a relativ ely n e w term . W ith th e re c e n t em p h asis o n an alytics, a s h a s b e e n th e c a s e in m any o th e r related te ch n ica l a p p licatio n a re a s (e .g ., c o n s u m e r analy tics, co m p le tiv e analytics, visual analytics, s o cia l analytics, an d so fo rth ), th e te x t field h a s a lso w a n te d to g e t o n th e analytics b a n d w a g o n . W h ile th e term text an alytics is m o re co m m o n ly u se d in a b u sin e ss a p p lica tio n c o n te x t, te x t m ining is freq u en tly u s e d in a ca d e m ic re s e a r c h circle s. E ven th o u g h th e y m ay b e d e fin e d s o m e ­ w h a t d ifferen tly at tim es, te x t an alytics an d te x t m ining are u su ally u se d sy n on ym ou sly, an d w e (th e au th o rs o f this b o o k ) co n c u r w ith this.

T e x t m ining (a lso k n o w n a s text d ata m ining o r kn ow ledge discovery in textual databases) is th e sem i-au to m ate d p ro ce s s o f extractin g pattern s (u sefu l in fo rm atio n and k n o w le d g e) fro m larg e am o u n ts o f u n stru ctu red data so u rce s. R e m em b e r th a t data m ining is th e p ro ce s s o f id entifying v alid, n o v e l, p o ten tially u sefu l, an d ultim ately u n d e rstan d ab le patterns in d ata s to red in stru ctured d a ta b a se s, w h e re th e data are o rg a n iz e d in record s stru ctured b y c a te g o rica l, ord inal, o r co n tin u o u s v ariab les. T e x t m in in g is th e sam e as d ata m in in g in th at it h a s th e sa m e p u rp o s e an d u s e s th e sam e p ro c e s s e s, b u t w ith text m ining th e in p u t to th e p ro ce s s is a c o lle c tio n o f u n stru ctu red (o r less stru ctu red ) data files s u ch as W o rd d o cu m en ts, P D F files, te x t e x ce rp ts , XML files, and s o o n . In e ss e n c e , te x t m ining c a n b e th o u g h t o f as a p ro c e s s (w ith tw o m ain ste p s) that starts w ith im p osin g structure o n th e te x t-b a s e d data s o u rc e s, fo llo w e d b y e x tra ctin g relev an t in fo rm atio n an d k n o w le d g e fro m this stru ctured te x t-b a s e d data u sin g d ata m ining te c h n iq u e s a n d to ols.

T h e b e n e fits o f te x t m ining are o b v io u s in th e areas w h e re very larg e am o u nts o f textu al d ata are b e in g g en erated , s u ch as law (co u rt o rd ers), a ca d em ic re se a rch (re s e a ic h articles) fin a n c e (q u arterly rep orts), m e d icin e (d isch arg e su m m aries), b io lo g y (m o le cu lar in te ractio n s), te c h n o lo g y (p a te n t files), and m arketing (cu sto m e r co m m e n ts). F o r exam p le, th e fre e -fo rm te x t-b a s e d in teractio n s w ith cu sto m ers in th e fo rm o f co m p lain ts (o r praises) an d w arranty claim s ca n b e u s e d to o b je ctiv e ly identify p ro d u ct an d serv ice characteristics th at are d e e m e d to b e less th an p e rfe ct an d c a n b e u se d as input to b e tte r p ro d u ct d ev el­ o p m en t a n d serv ice allocatio n s. Likew ise, m ark e t o u tre a ch p rogram s a n d fo cu s groups g en erate la rg e am o u n ts o f data. B y n o t restrictin g p ro d u ct o r serv ice fe e d b a c k to a co d i­ fied form , cu sto m ers c a n p resen t, in th e ir o w n w o rd s, w h at th ey think a b o u t a co m p a n y ’s products an d serv ices. A n oth er area w h e re th e au tom ated p ro cessin g o f un stru ctu red text has h a d a lo t o f im p act is in e le ctro n ic com m u n icatio n s an d e-m ail. T e x t m in in g n o t only can b e u s e d to classify a n d filter ju n k e-m ail, b u t it c a n a lso b e u se d to au tom atically prior­ itize e-m ail b a s e d o n im p o rtan ce lev el as w e ll as g e n era te au tom atic re s p o n s e s (W e n g an d Liu, 2 0 0 4 ). T h e fo llow in g are am o n g th e m o st p o p u lar ap p licatio n areas o f te x t m ining:

• I n f o r m a t i o n e x t r a c t i o n . Id e n tificatio n o f k e y p h rase s an d re latio n sh ip s w ithin te x t b y lo o k in g fo r p re d efin e d o b je c ts an d s e q u e n c e s in te x t b y w a y o f pattern m atch in g. P erh a p s th e m o st co m m o n ly u se d form o f in fo rm atio n e x tra ctio n is n am ed entity extraction. N am ed en tity e x tra ctio n in clu d es n am ed entity recognition (re c o g n itio n o f k n o w n en tity n am es— fo r p e o p le and organ izatio n s, p la ce n am es, te m p o ra l e x p re s s io n s , and ce rta in typ es o f n u m erical e x p re s s io n s , u sin g existin g k n o w le d g e o f th e d o m a in ), co-referen ce resolution (d e te c tio n o f c o -r e fe r e n c e an d a n a p h o ric links b e tw e e n te x t e n titie s), an d relation ship extraction (id e n tifica tio n o f re latio n s b e tw e e n en tities).

• T o p ic t r a c k in g . B a s e d o n a u s e r p ro file a n d d o cu m en ts th at a u s e r v ie w s, text m in in g c a n p re d ict o th e r d o cu m en ts o f in te rest to th e user.

• S u m m a r i z a t i o n . Sum m arizing a d o cu m e n t t o save tim e o n th e p a rt o f th e read er. • C a t e g o r i z a t i o n . Id entifying the m ain th e m e s o f a d o cu m en t a n d th e n p la cin g th e

d o cu m e n t into a p re d efin e d s e t o f ca te g o rie s b a s e d o n th o se th e m e s. • C lu s te r in g . G ro u p in g sim ilar d o cu m en ts w ith o u t h av in g a p re d e fin e d s e t o f

c ate g o rie s.

• C o n c e p t U n k in g . C o n n e cts related d o cu m e n ts b y identifying th e ir sh are d c o n ­ ce p ts and , b y d o in g s o , h e lp s u sers find in fo rm atio n that th e y p e rh a p s w o u ld n o t h a v e fo u n d u sin g trad itional s e a rch m e th o d s.

• Q u e s tio n a n s w e r in g . F ind ing th e b e s t a n s w e r to a g iv e n q u e stio n throu g k n o w le d g e-d riv en p attern m atch ing.

S e e T e c h n o lo g y Insigh ts 7 .1 fo r e x p la n a tio n s o f s o m e o f th e term s a n d c o n c e p ts u s e d in te x t m ining. A p p lication C a se 7 .1 d e s c rib e s th e u s e o f te x t m in in g in p aten t analysis.

3 2 4 P artH I • Predictive Analytics

T E C H N O L O G Y IN SIG H T S 7 . 1 T e x t M in in g L in g o

T h e follow ing list d escribes som e com m only used tex t m ining terms:

• U n s tr u c tu r e d d a ta ( versu s s tr u c tu r e d d a t a ). Structured data has a predeterm ined form at It is usually organized into records w ith sim ple data valu es (categorical, ordinal, and con tin uous variables) and stored in d atabases. In contrast, u n s t r u c t u r e d d a t a d oes not have a predeterm ined form at and is stored in th e form o f textual docum ents. In essen ce, the structured data is fo r th e com p u ters to p rocess w hile th e unstructured data is for humans to p rocess and understand.

• C o r p u s . In linguistics, a c o r p u s (plural c o r p o r a ) is a large and structured set o t texts (n o w usually stored and p rocessed electronically) prepared for the p u rpose o f conducting

kn ow led ge discovery. • Term s. A ter m is a single w ord or multiword phrase extracted directly from the corpus

o f a specific dom ain by m eans o f natural language p rocessing (NLP) m ethods. • C on cep ts. C o n c e p ts are features g en erated from a collection o f docum ents by m eans o f

m anual, statistical, ru le-based, o r hybrid categorization m ethodology. Com pared to terms, c on cep ts are the result o f higher level abstraction.

• Stem m ing. S t e m m i n g is the p rocess o f red ucing inflected w ords to their stem (o r base o r root) form . For instance, ste m m er , s t e m m in g , and s t e m m e d are all b ased o n the

t o o t stem . • S top w o rd s . S t o p w o r d s (o r n o is e w o r d s ) are words that are filtered ou t prioi to o r

after processing o f natural language data (i.e ., text). Even though there is n o universally accep ted list o f stop words, m ost natural language processing tools use a list that includes articles (.a, a m , the, of, etc.), auxiliary v erbs (is, a r e , w a s, w ere, e tc.), and con text-sp ecific w ords that are d eem ed n o t to have differentiating value.

• S y n o n y m s a n d p o ly se m e s. Synonym s are syntactically different words (i.e., spelled dif­ ferently) w ith identical or at least similar m eanings (e .g ., m ov ie, f i l m , and m o t io n p ic t u r e ) . In contrast, p o l y s e m e s , w hich are also called h o m o n y m s , are syntactically identical words (i.e., spelled exactly the sam e) with different m eanings (e.g ., h o w can m ean “to bend for­ w ard,” “the front o f the ship,” “th e w e a p o n that shoots arrows,” o r “a kind o f tied ribbon ).

• T o k e n iz in g . A t o k e n is a categorized b lo c k o f tex t in a sen ten ce. T h e b lo ck o f text corresp on ding to th e to k en is categ orized accord ing to th e function it perform s. This assignm ent o f m eaning to b lo ck s o f text is kn o w n as t o k e n i z i n g . A token c a n lo o k like anything; it just n eed s to b e a useful part o f th e structured text.

• T e rm d ic tio n a r y . A collection o f term s sp ecific to a narrow field that c a n b e used to restrict th e extracted term s w ithin a corpus.

• W o r d f r e q u e n c y . T h e nu m ber o f tim es a word is found in a sp ecific docum ent. • P a r t-o f -s p e e c h ta g g in g . T h e p rocess o f marking up th e words in a text a s corresp on d ­

ing to a particular part o f sp e ec h (su ch as n ou n s, verbs, ad jectives, adverbs, e tc.) based on a w ord’s definition and th e co n tex t in w h ich it is used.

• M o r p h o lo g y . A bran ch o f th e field o f linguistics and a part o f natural language pro­ cessin g that studies the internal -structure o f w ords (patterns o f w ord-form ation w ithin a

language o r across languages). • T e r m -b y -d o c u m e n t m a t r i x (o c c u r r e n c e m a t r i x ). A com m o n representation schem a

o f th e frequ en cy -based relationship b e tw e e n th e term s and d ocum ents in tabular format

Chapter 7 • T e x t Analytics, T e x t Mining, and Sentim ent Analysis 325

w h ere term s are listed in rows, d ocum ents are listed in colum ns, and th e frequ ency b e tw e e n th e terms and d ocum ents is listed in cells as integer values,

• S in g u la r -v a lu e d e c o m p o s itio n (la t e n t s e m a n tic i n d e x in g ). A dim ensionality reduction m ethod used to transform th e term -by-docum ent matrix to a m an ageable size by gen eratin g an interm ediate representation o f the frequ en cies using a m atrix m anipula­ tion m eth od similar to principal com p o n en t analysis.

Application Case 7.1 Text M ining fo r P a te n t Analysis A p a te n t is a s e t o f e xclu siv e rights granted b y a co u n try to a n in v e n to r fo r a lim ited p e rio d o f tim e in e x c h a n g e fo r a d isclosu re o f an in v e n tio n (n o te th at th e p ro c e d u re fo r g ranting p aten ts, th e re q u ire ­ m en ts p la c e d o n th e p a te n te e , an d th e e x te n t o f th e e xclu siv e rights vary w id ely fro m co u n try to co u n ­ try). T h e d isclo su re o f th e s e in v en tion s is critical to future a d v a n ce m e n ts in s c ie n c e a n d te ch n o lo g y . If carefu lly an aly zed , p a te n t d o cu m en ts ca n h e lp id e n ­ tify e m e rg in g te c h n o lo g ie s, in sp ire n o v e l solu tions, fo ste r sy m b io tic partnerships, a n d e n h a n c e overall a w a ren e ss o f b u s in e s s ’ cap ab ilitie s and lim itations.

P aten t analysis is th e u s e o f analytical te ch ­ n iq u es to e x tra ct v alu ab le k n o w le d g e fro m p atent d atabases. C o u ntries o r groups o f cou ntries that m aintain p aten t d atab ases (e .g ., th e U n ited States, th e E u ro p ean U n ion , Ja p a n ) add te n s o f m illions o f n e w p atents e a c h year. It is n early im p ossible to efficiently p ro ce s s s u c h en o rm o u s am o u nts o f sem istructured data (p a te n t d ocu m en ts usually co n tain partially stru ctured a n d partially textual data). P aten t analy­ sis w ith sem iau tom ated softw are to o ls is o n e w ay to e a s e th e p ro cessin g o f th e s e very large d atabases.

A R e p r e s e n t a t i v e E x a m p l e o f P a t e n t A n a ly s is

E astm an K o d a k em p lo y s m o re th an 5 ,0 0 0 scientists, e n g in e e rs, a n d te ch n icia n s aro u n d th e w o rld . D uring th e tw en tie th century, th e s e k n o w le d g e w o rk ers an d th eir p re d e c e s s o rs cla im e d n early 2 0 ,0 0 0 p at­ en ts, pu tting th e co m p a n y am o n g the to p 1 0 p aten t h old ers in th e w o rld . B e in g in th e b u sin ess o f c o n ­ stant c h a n g e , th e co m p a n y k n o w s that s u c c e s s (o r m e re survival) d e p e n d s o n its ability to apply m o re than a ce n tu ry 's w o rth o f k n o w le d g e a b o u t im aging s c ie n c e an d te ch n o lo g y to n e w u s e s a n d to s ecu re th o se n e w u s e s w ith patents.

A p p re ciatin g th e v a lu e o f p ate n ts, K o d a k n o t o n ly g e n e ra te s n e w p aten ts b u t a lso an aly zes th o se c re a te d b y o th e rs. U sing d ed icate d analysts an d state-o f-th e-art so ftw a re to o ls (in clu d in g sp e cia liz e d te x t m in in g to o ls fro m C learF orest C o rp .), K o d ak co n tin u o u sly d ig s d e e p in to vario u s data so u rces (p a te n t d a ta b a se s, new7 re le a s e arch iv es, an d p ro d ­ u ct a n n o u n c e m e n ts ) in o rd e r to d ev elo p a h o listic v ie w o f th e co m p etitiv e lan d scap e . P ro p e r analysis o f p aten ts c a n b rin g c o m p a n ie s lik e K o d a k a w id e ran g e o f b en e fits:

• It e n a b le s co m p etitiv e in te llig e n ce . K no w ing w h at c o m p etito rs a re d o in g c a n h e lp a co m ­ p a n y to d e v e lo p co u n te rm e asu re s.

• It c a n h e lp th e co m p a n y m a k e critical b u sin ess d ecisio n s, s u c h as w h a t n e w p ro d u cts, p ro d u ct lin es, and/or te c h n o lo g ie s to g e t in to o r w h at m ergers a n d a cq u isitio n s to pu rsue.

• It c a n aid in identifying an d recru iting th e b e s t and b rig h test n e w talen t, th o se w h o s e n am es a p p e a r o n th e p aten ts th at are critical to the co m p a n y ’s s u cce s s .

• It ca n h e lp th e co m p a n y to identify th e u n a u ­ th o rized u s e o f its p aten ts, e n a b lin g it to tak e a c tio n to p ro te c t its assets.

• It c a n id en tify co m p le m e n ta ry in v e n tio n s to b u ild sy m b io tic p artn ersh ip s o r to facilitate m e rg ers and/or acq u isition s.

• It p rev en ts co m p etito rs fro m creatin g sim ilar p ro d u cts a n d it c a n h e lp p ro te c t th e co m p a n y fro m p a te n t in frin g em en t law suits.

U sing p a te n t analysis as a rich s o u rc e o f k n o w le d g e a n d a strateg ic w e a p o n (b o th d efen siv e a s w e ll a s o ffe n siv e ), K o d a k n o t o n ly survives but e x c e ls in its m a rk e t s e g m e n t d efin e d b y in n ov atio n and co n sta n t ch a n g e .

0Continued)

3 2 6 Part III • Predictive Analytics

Application Case 7.1 (Continued)

Q u e s t i o n s f o r D i s c u s s i o n

1. W h y is it im portant fo r co m p a n ie s to k e e p up w ith p a te n t filings?

2. H o w d id K o d ak u se te x t analytics "to b e tte r a n a ­ ly ze patents?

3. W h a t w e r e th e c h a lle n g e s, th e p ro p o s e d so lu ­ tio n , an d th e o b ta in e d results?

SECTION 7 .2 QUESTIONS

1 . W h a t is te x t analytics? H o w d o e s it differ from te x t mining?

2 . W h at is te x t mining? H o w d o e s it differ fro m d ata mining? 3 . W h y is th e p o pularity o f te x t m in in g as a n an aly tics to o l increasing?

4 . W h at are s o m e o f th e m o st p o p u la r ap p lica tio n a re a s o f text mining?

7.3 N A T U R A L L A N G U A G E P R O C E S S IN G S o m e o f th e early te x t m in in g ap p lica tio n s u s e d a sim p lified re p re sen tatio n c a lle d bag - of-w ord s w h e n introd u cing stru cture to a c o lle c tio n o f te x t-b a s e d d o cu m en ts in ord er to classify th e m in to tw o o r m o re p re d eterm in e d cla ss e s o r to clu ste r th e m in to natural g ro u p in gs. In the b a g -o f-w o rd s m o d e l, text, s u ch a s a s e n te n c e , p arag rap h , o r co m p le te d o cu m en t, is re p re sen ted as a c o lle c tio n o f w ord s, d isregard in g th e gram m ar o r th e ord er in w h ic h th e w o rd s ap p ear. T h e b a g -o f-w o rd s m o d e l is still u s e d in s o m e sim p le d o cu ­ m e n t classificatio n to o ls . F o r in sta n ce, in sp am filtering a n e-m ail m e ssa g e can b e m o d ­ e le d as an u n o rd e red c o lle ctio n o f w o rd s (a b a g -o f-w o rd s) that is co m p a re d ag ain st tw o d ifferen t p re d eterm in e d b ag s. O n e b a g is filled w ith w o rd s fo u n d in sp am m e ssa g es an d th e o th e r is filled w ith w o rd s fo u n d in leg itim ate e -m ails. A lthough s o m e o f th e w o rd s are lik e ly to b e fo u n d in b o th b a g s, th e “sp a m ” b a g w ill co n ta in sp am -related w o rd s s u ch as stock, V iagra, an d bu y m u c h m o re freq u en tly th an th e legitim ate b a g , w h ich will co n ta in m o re w o rd s related to th e u s e r’s friend s o r w o rk p la c e . T h e lev el o f m atch b e tw e e n a s p e c ific e -m a il’s b ag -o f-w o rd s an d th e tw o b a g s co n ta in in g th e d escrip to rs d eterm in e s th e m e m b ersh ip o f th e e-m ail as e ith e r sp a m o r legitim ate.

Naturally, w e (h u m a n s) d o n o t u s e w o rd s w ith o u t s o m e ord e r o r stru cture. W e u se w o rd s in s e n te n c e s , w h ic h h av e s em an tic as w e ll a s sy n tactic structure. T h u s, au tom ated te ch n iq u e s (s u c h as te x t m ining ) n e e d to lo o k fo r w ays to g o b e y o n d th e b ag -of-w ord s in terp retation an d in co rp o rate m o re a n d m o re se m a n tic stru cture into th e ir op eration s. T h e cu rre n t trend in te x t m in in g is to w ard in clu d in g m any o f th e a d v a n ce d fe atu res th at c a n b e o b ta in e d u sin g natural lan g u ag e p ro cessin g .

It has b e e n sh o w n that th e b ag -of-w ord s m e th o d m ay n o t p ro d u ce g o o d en o u g h inform ation co n te n t fo r text m ining tasks (e .g ., classification, clustering, associatio n ). A g o o d e x a m p le o f this c a n b e found in e v id e n c e -b a se d m ed icin e. A critical c o m p o n e n t o f e v id e n ce -b a se d m e d icin e is in corp o ratin g th e b e s t available research findings into the clin ical d ecisio n -m ak in g p ro cess, w h ich involves appraisal o f th e inform ation c o lle cte d fro m th e printed m ed ia fo r validity an d relev an ce. Several research ers from th e University o f M aryland d ev elo p e d e v id e n ce a ssessm en t m o d e ls using a b ag -of-w ord s m ethod (Lin and D em n er, 2 0 0 5 ). T h e y e m p lo y ed p o p u lar m ach in e-learn in g m eth o d s alo n g w ith

Sources: P. X. Chiem, “Kodak Turns Knowledge Gained About Patents into Competitive Intelligence," K now ledge M anagem ent, 2001, pp. 1 1 -1 2 ; Y-H. Tsenga, C-J. Linb, and Y-I. Line, “Text Mining Techniques for Patent Analysis,” In form ation P rocessing & M anagem ent, Vol. 43, No. 5, 2007, pp. 1216-1247.

Chapter 7 • T ex t Analytics, T e x t iMining, and Sentim ent Analysis 327

m o re th a n h a lf a m illion research articles c o lle cte d fro m MEDLINE (M ed ical Literature Analysis and R etrieval System O n lin e ). In th eir m o d els, th ey re p re sen ted e a c h ab stract as a b a g -of-w ord s, w h e re e a c h stem m ed term re p re sen ted a featu re. D esp ite using p o p u la r c las­ sification m eth o d s w ith p ro v en exp erim en tal d esign m eth o d o lo g ies, their p red ictio n results w e re n o t m u ch b e tte r than sim p le g u essing, w h ich m ay indicate th at th e b ag -of-w ord s is n ot g en eratin g a g o o d e n o u g h rep resen tatio n o f th e re sea rch articles in this d om ain; h e n ce , m o re ad vanced te ch n iq u e s su ch as natural language p ro cessin g are n eed ed .

N atural language p ro cessin g (NLP) is a n im portant c o m p o n e n t o f te x t m in ­ ing an d is a su b fie ld o f artificial in te llig e n ce a n d co m p u tatio n al linguistics. It stu d ies th e p ro b le m o f “u n d erstan d in g ” th e natural h u m an lan g u ag e , w ith th e v ie w o f co n v ertin g d ep ictio n s o f h u m a n lan g u ag e (s u ch as textu al d o cu m e n ts ) into m o re fo rm al re p re sen ta ­ tions (in th e fo rm o f n u m e ric a n d s y m b o lic d ata) that are e a s ie r fo r co m p u te r p ro g ram s to m an ip u late. T h e g o al o f NLP is to m o v e b e y o n d syntax-d riv en te x t m an ip u latio n (w h ich is o fte n ca lle d “w o rd co u n tin g ”) to a tru e u n d erstan d in g and p ro ce s s in g o f natural lan­ g u ag e that co n sid e rs g ram m atical an d s em an tic con strain ts a s w e ll as th e co n te x t.

T h e d efin itio n an d s c o p e o f th e w o rd “u n d erstan d in g ” is o n e o f th e m a jo r d iscu s­ s io n to p ics in NLP. C o n sid erin g that th e natu ral h u m an lan g u ag e is v ag u e an d th at a true u n d erstan d in g o f m e a n in g req u ires e x te n siv e k n o w le d g e o f a to p ic (b e y o n d w h at is in th e w ord s, s e n te n c e s , and p arag rap h s), w ill co m p u te rs e v e r b e a b le to u n d e rstan d natural lan gu ag e th e s a m e w a y an d w ith th e sa m e a c c u ra c y th at h u m an s do? P ro b a b ly not! NLP has c o m e a lo n g w a y fro m th e days o f sim p le w o rd cou n tin g, b u t it has a n e v e n lo n g e r w a y to g o to really u n d erstan d in g natu ral h u m an lan g u ag e. T h e fo llo w in g a re ju st a few o f t h e ch a lle n g e s co m m o n ly a s s o cia te d w ith th e im p lem e n tatio n o f NLP:

• P a r t - o f- s p e e c h ta g g in g . It is d ifficult to m ark up term s in a te x t as c o rre s p o n d ­ ing to a p articu lar p art o f s p e e c h (s u c h as n o u n s, v erb s, a d je ctiv e s, a d v e rb s, e tc .) b e c a u s e th e part o f s p e e c h d e p e n d s n o t o n ly o n th e d efin ition o f th e te rm b u t also o n th e c o n te x t w ithin w h ich it is used .

• T ex t s e g m e n t a t io n . So m e w ritten lan g u ag es, s u ch as C h in e se, Ja p a n e s e , and T h ai, d o n o t h av e sin gle-w o rd b o u n d arie s. In th e s e in stan ces, th e text-p arsin g task req u ires th e id en tificatio n o f w o rd b o u n d a rie s, w h ich is o ften a d ifficu lt task. Sim ilar ch a lle n g e s in s p e e c h s e g m en ta tio n e m e rg e w h e n analyzing s p o k e n langu age, b e c a u s e so u n d s re p resen tin g s u cce s s iv e letters a n d w o rd s b le n d into e a c h other.

• W ord s e n s e d is a m b ig u a t io n . M any w o rd s h a v e m o re th an o n e m ean in g . Se le ctin g th e m e a n in g th at m a k e s th e m o st s e n s e c a n o n ly b e a c c o m p lis h e d b y tak­ ing in to a c c o u n t th e c o n te x t w ith in wrh ich th e w o rd is used .

• S y n ta c tic a m b ig u it y . T h e gram m ar fo r natu ral la n g u a g e s is a m b ig u o u s; that is, m u ltiple p o ss ib le s e n te n c e stru ctures o fte n n e e d to b e co n sid ere d . C h o o sin g th e m o st a p p ro p ria te stru cture u su ally req u ires a fu sio n o f se m a n tic an d co n tex tu a l

in form ation. • I m p e r fe c t o r i r r e g u la r in p u t. F o re ig n o r re g io n al a cce n ts an d v o c a l im p ed i­

m e n ts in s p e e c h an d ty p o g rap h ical o r g ram m atical errors in te x ts m a k e th e p ro c e s s ­ ing o f th e lan g u ag e a n e v e n m o re d ifficu lt task.

• S p e e c h a c t s . A s e n te n c e c a n o fte n b e c o n s id e re d a n a ctio n b y th e s p e a k e r. T h e s e n te n c e stru ctu re a lo n e m ay n o t co n ta in e n o u g h in form ation to d e fin e this action. F o r e x a m p le , “C an y o u p ass th e class?” re q u e sts a sim p le yes/no an sw er, w h ere a s “C an y o u p a s s th e salt?” is a re q u e st fo r a p h y sical a ctio n to b e p erfo rm ed .

It is a lo n g stan d in g d re a m o f th e artificial in te llig e n ce com m u n ity to h a v e algorithm s that are c a p a b le o f au to m atically re ad in g a n d o b ta in in g k n o w le d g e fro m te x t. B y apply ­ ing a learn in g alg o rith m to p a rse d te x t, re sea rch ers fro m Stanford U n iv ersity’s NLP lab h av e d e v e lo p e d m e th o d s th a t c a n au tom atically id entify th e c o n c e p ts a n d relatio n sh ip s b e tw e e n th o s e c o n c e p ts in th e text. B y ap p ly in g a u n iq u e p ro ced u re to la rg e am o u nts

o f text, th e ir algorithm s au tom atically a cq u ire h u n d red s o f th o u san d s o f item s o f w orld k n o w le d g e and u s e th e m to p ro d u ce significantly e n h a n c e d re p o sito rie s fo r W o rd N e t W ordN et is a lab o rio u sly h a n d -c o d e d d atab ase o f E n glish w ord s, th eir d efin ition s, sets o f sy n on y m s, an d vario u s se m a n tic re latio n s b e tw e e n sy n o n y m sets. It is a m a jo r re so u rce fo r NLP a p p licatio n s, b u t it h a s p ro v e n to b e v e r y e x p e n s iv e to b u ild an d m aintain m anu­ ally. B y au tom atically in d u cin g k n o w le d g e in to W ordN et, th e p o ten tial e x is ts to m a k e W ordN et an e v e n g re a te r an d m o re c o m p re h e n siv e re s o u rce fo r NLP at a fractio n o f th e co st. O n e p ro m in en t area w h e re th e b e n e fits o f NLP an d W ordN et a re alread y b e in g h arv este d is in cu sto m e r relatio n sh ip m a n a g e m e n t (CRM). B ro a d ly sp e ak in g , th e g o a l o f CRM is to m ax im ize cu sto m e r v alu e b y b e tte r u n d erstan d in g an d e ffe ctiv e ly resp o n d in g to th eir actu al a n d p e rc e iv e d n e ed s. An im p o rtan t area o f CRM, w h e re NLP is m aking a sig n ifican t im pact, is s en tim en t analysis. Sentim ent analysis is a te ch n iq u e u se d to d e te c t fav o rab le a n d u n fav o rab le o p in io n s to w ard s p e cific p ro d u cts an d s erv ices u sin g larg e n u m b e rs o f textu al data so u rce s (c u s to m e r fe e d b a c k in th e fo rm o f W e b postings). A d eta iled co v e ra g e o f s en tim en t an alysis an d W ordN et is g iv en in S e ctio n 7.7.

T e x t m ining is a lso u s e d in a ssessin g p u b lic com p lain ts. A p p lication C ase 7 .2 p rovides a n e x a m p le w h e re te x t m in in g is u s e d to an ticip a te an d ad dress p u b lic com p lain ts in

H o n g K ong.

B Part III • Predictive Analytics

Application Case 7.2 Text M ining Im proves Hong Kong G o vern m en t s A b ility to A n ticip ate and A ddress Pu blic Com plaints T h e 1 8 2 3 Call C entre o f th e H o n g K o n g g o v ern m e n t’s E fficie n cy U n it a c ts as a sin gle p o in t o f c o n ta c t for h an d lin g p u b lic in qu iries an d co m p lain ts o n b e h a lf o f m a n y g o v ern m e n t d ep artm ents. 182 3 o p erates r o u n d -th e -clo ck , in clu d in g during Sun days an d p u b ­ lic h olid ay s. E a ch y ear, it an sw e rs a b o u t 2 .6 5 m illion calls a n d 9 8 ,0 0 0 e -m ails, in clu d in g inqu iries, su g g es­ tion s, a n d co m p lain ts. “H av ing re c e iv e d s o m any c a lls a n d e-m ails, w e g ath e r su b stantial v o lu m e s o f data. T h e n e x t s tep is t o m a k e s e n se o f th e d a ta ,” says th e E fficie n cy U nit’s assistant d irecto r, W . F. Y u k. “N ow , w ith SAS te x t m in in g te ch n o lo g ie s, w e can o b ta in d e e p insights th ro u g h u n co v e rin g th e hidd en re latio n sh ip b e tw e e n w o rd s a n d s e n te n c e s o f c o m ­ plaints in form ation , s p o t e m e rg in g trend s an d p u b ­ lic c o n c e r n s , and p ro d u ce h igh -q u ality com p lain ts in te llig e n ce fo r th e d ep artm en ts w e s e rv e .”

B u ild in g a “C o m p la in ts I n te llig e n c e S y s te m ”

T h e E ffic ie n c y U nit aim s to b e th e p re fe rre d c o n ­ sulting p artn er fo r all g o v e rn m e n t b u re a u s and

d ep artm e n ts an d to ad v an ce th e d elivery o f w o rld - cla ss p u b lic s e rv ice s to th e p e o p le o f H o n g K o ng. T h e U nit la u n c h e d th e 1 8 2 3 Call C entre in 2 0 0 1 . O n e o f 1 8 2 3 ’s m ain fu n ctio n s is h an d lin g c o m ­ plaints— 10 p e r c e n t o f th e calls re c e iv e d la st y e a r w e re co m p la in ts. T h e E fficie n cy U nit re co g n iz e d th at th e re a r e s o cia l m e ssa g es h id d en in th e c o m ­ p lain ts data, w h ich p ro v id es im p ortan t fe e d b a c k o n p u b lic s e rv ic e an d hig hlights o p p o rtu n ities fo r s e r­ v ic e im p ro v em en t. R ath er th a n sim p ly h an d lin g calls a n d e-m ails, th e U n it s e e k s to u s e th e com p lain ts in form ation c o lle c te d to gain a b e tte r u n d erstan d in g o f d aily issu es fo r the p u blic.

“W e p r e v io u s ly c o m p i le d s o m e r e p o r ts o n c o m p la in t s ta tis tic s f o r r e f e r e n c e b y g o v e r n m e n t d e p a r t m e n t s ,” s a y s Y u k . “H o w e v e r , th r o u g h ‘e y e ­ b a l l’ o b s e r v a t i o n s , it w a s a b s o lu te ly im p o s s ib le t o e f f e c t i v e l y r e v e a l n e w o r m o r e c o m p l e x p o te n tia l p u b li c is s u e s a n d id e n tify t h e ir r o o t c a u s e s , a s m o s t o f t h e c o m p la in t s w e r e r e c o r d e d in u n s tr u c tu r e d t e x t u a l f o r m a t,” s a y s Y u k . A im in g to b u ild a p la t­ fo r m , c a lle d th e C o m p la in ts I n t e ll i g e n c e S y s te m , th e U n it r e q u ir e d a r o b u s t a n d p o w e r fu l s u ite o f te x t

C hapter 7 • T e x t Analytics, T ex t Mining, and Sentim ent Analysis 3 2 9

p ro c e s s in g a n d m in in g so lu tio n s that c o u ld u n co v e r th e tre n d s, p attern s, a n d relatio n sh ip s in h e re n t in th e com p lain ts.

U n c o v e r in g R o o t C a u s e s o f I s s u e s f r o m U n s t r u c t u r e d D a ta

T h e E fficie n cy U nit c h o s e to d ep lo y SAS T e x t M iner, w h ich c a n a c c e s s a n d an alyze v arious te x t fo r­ m ats, in clu d in g e-m ails re c e iv e d b y th e 182 3 Call C entre. “T h e so lu tio n co n so lid a te s all in form ation and u n co v e rs h id d en relatio n sh ip s th ro u g h sta­ tistical m o d e lin g a n a ly se s,” says Y u k. “It h e lp s us u n d erstan d h id d e n s o cia l issu es s o th at g o v ern m e n t d ep artm en ts c a n d isco v e r th e m b e fo r e th ey b e c o m e serio u s, and thus s e iz e th e o p p o rtu n ities fo r service im p ro v e m en t.”

E q u ip p e d w ith te x t analytics, th e d ep artm ents ca n b e tte r u n d erstan d u n d erlyin g issu es and q u ick ly re sp o n d e v e n as situ ations e v o lv e . S e n io r m a n a g e ­ m e n t c a n a c c e s s accu ra te , u p -to -d ate in form ation from th e C o m p lain ts In te llig e n ce System .

P e r f o r m a n c e R e p o r t s a t F in g e r tip s

W ith th e p latfo rm fo r SAS B u sin ess A nalytics in p la ce, the E fficie n cy U n it g e ts a b o o s t fro m th e sy stem ’s ability to instantly g e n e ra te rep orts. F or in sta n ce, it p reviou sly to o k a w e e k to co m p ile rep o rts o n k e y p e rfo rm a n ce in d icato rs s u ch as a b a n d o n e d call rate, cu sto m e r satisfactio n rate, an d first-tim e resolu tion rate N ow , th e s e rep o rts ca n b e c re a te d at th e click o f a m o u s e throu gh p e rfo rm a n ce d ash b o ard s, as all co m p la in ts in fo rm atio n is co n so lid a ted in to th e C o m p lain ts In te llig e n ce System . T h is e n a b le s e ffe c ­ tive m o n ito rin g o f th e 182 3 Call C en tre’s o p eratio n s and s e rv ic e quality.

S tr o n g L a n g u a g e C a p a b ilitie s , C u s to m iz e d S e r v ic e s

O f p articu lar im p o rta n ce in H o n g K o ng, SAS T e x t M in er h a s stro n g la n g u a g e cap ab ilitie s— su p p ortin g E nglish an d trad itio nal and sim p lified C h in ese— an d c a n p erfo rm au to m ated sp e llin g co rrectio n . T h e s o lu tio n is a lso a id e d b y th e SAS cap ab ility o f d e v e lo p in g cu sto m ized lists o f syn on ym s s u c h as

th e full a n d s h o rt fo rm s o f d iffe ren t go v ern m e n t d ep artm en ts an d to p a rse C h in e se te x t fo r sim ilar o r id e n tical term s w h o s e m e a n in g s a n d c o n n o ta tio n s c h a n g e , o fte n dram atically, d e p e n d in g o n th e c o n ­ te x t in w h ic h th e y are used . “A lso, th ro u g h o u t this 4 -m o n th p ro je ct, SAS h a s p ro v e d to b e o u r trusted p artn er,” said Y u k . “W e are satisfied w ith th e co m ­ p re h e n siv e su p p o rt pro v id ed b y th e SAS H o n g K o n g te a m .”

I n f o r m e d D e c is io n s D e v e lo p S m a r t S tr a te g ie s

“U sing SAS T e x t M iner, 1 8 2 3 ca n q u ick ly d isco v e r th e c o rrela tio n s am o n g s o m e k e y w o rd s in the co m p la in ts,” s a y s Y u k . “F o r in stan ce, w e c a n sp o t districts w ith fre q u e n t co m p lain ts re c e iv e d c o n c e rn ­ in g p u b lic h e a lth issu es s u ch as d ead b ird s found in resid en tial areas. W e c a n th e n inform relevan t g o v ern m e n t d ep artm en ts a n d p ro p e rty m a n a g e ­ m e n t c o m p a n ie s , s o th a t th e y ca n allo ca te ad eq u ate re s o u rc e s to ste p up clea n in g w o rk to av o id sp read o f p o ten tial p an d em ics.

“T h e p u b lic ’s v ie w s a re o f c o u rs e e x tr e m e ly im p o rta n t to th e g o v e rn m e n t. B y d e c o d in g th e ‘m e s s a g e s ’ th ro u g h statistical an d ro o t-c a u s e a n a ly ­ s e s o f c o m p la in ts d ata, th e g o v e r n m e n t c a n b e tte r u n d e rs ta n d th e v o ic e o f th e p e o p le , an d h e lp g o v ­ e rn m e n t d ep a rtm e n ts im p ro v e s e rv ic e d elivery, m a k e in fo r m e d d e c is io n s , a n d d e v e lo p sm a rt strat­ e g ie s . T h is in tu rn h e lp s b o o s t p u b lic s a tisfa ctio n w ith th e g o v e rn m e n t, a n d b u ild a q u a lity city ,” said W . F. Y u k , A ssistan t D ire c to r, H o n g K o n g E ffic ie n c y U nit.

Q u e s t i o n s f o r D i s c u s s i o n

1. H o w d id th e H o n g K o n g g o v e rn m e n t u se text m ining to b e tte r serv e its constituents?

2. W hat w e r e th e ch a lle n g e s, th e p ro p o s e d so lu ­ tio n , a n d th e o b ta in e d results?

Sources: SAS Institute, Customer Success Story, sas.com / success/pdf/hongkongeu.pdf (accessed February 2013); a n d e n t e r p r i s e i n n o v a t i o n . n e t / w h i t e p a p e r / t e x t - m i n i n g - im proves-hong-kong-governm ents-ability-anticipate-and- address-public.

NLP h a s s u cce ssfu lly b e e n ap p lie d to a v arie ty o f d o m ain s fo r a variety o f tasks via co m p u te r program s to au tom atically p ro ce s s natural h u m an lan g u ag e that p rev iou sly co u ld o n ly b e d o n e b y h u m ans. F o llo w in g are a m o n g th e m o st p o p u lar o f th e s e tasks:

. Question answering. T h e task o f a u to m atically an sw erin g a q u e stio n p o se d in natu ral la n g u ag e ; th a t is, p ro d u cin g a h u m an -lan g u ag e an sw e r w h e n g iv en a h u m an -la n g u ag e q u e stio n . T o fin d th e a n s w e r to a q u e stio n , th e co m p u te r p rogram m ay u se e ith e r a prestru ctured d a ta b a se o r a c o lle c tio n o f natu ral lan g u ag e d o cu ­ m e n ts (a te x t co rp u s s u ch a s th e W o rld W id e W e b ).

• A u t o m a t i c s u m m a r i z a t i o n . T h e c re a tio n o f a s h o rte n e d v e rs io n o a textu a d o cu m en t b y a co m p u te r p ro g ram th at co n ta in s th e m o st im p o itan t p o in ts o f th e

origin al d o cu m en t. . N a tu ra l language generation. Sy stem s co n v e rt in form ation fro m co m p u ter

d a tab ase s in to re a d a b le h u m an lang u age. • N a tu ra l language understanding. S ystem s co n v e rt sam p les o f h u m an lan­

g u a g e in to m o re form al re p re sen ta tio n s th a t are e a s ie r fo r co m p u te r p ro gram s to

m an ip u late. , , • M achine translation. T h e au to m atic tran slation o f o n e h u m an lan g u ag e to

a n o th er. . • Foreign language reading. A co m p u te r p ro gram th at assists a n o n n ativ e lan­

g u ag e s p e a k e r to re a d a fo reig n la n g u a g e w ith co r re c t p ro n u n ciatio n a n d a cce n ts

o n d ifferen t parts o f th e w ords. • F oreign language w riting. A co m p u te r p ro g ram th at assists a n o n n a tiv e lan­

g u ag e u s e r in w riting in a fo re ig n lan g u ag e. • S p e e c h r e c o g n i t i o n . C onverts s p o k e n w o rd s to m a ch in e -re a d a b le input. G iv e n a

s o u n d clip o f a p e rs o n sp e a k in g , th e sy stem p ro d u ce s a te x t dictation. • Text-to-speech. A lso ca lle d speech synthesis, a co m p u te r p ro gram au tom atically

co n v e rts n o rm al lan g u ag e te x t into h u m a n s p e e c h . • Text p ro o fin g. A co m p u te r p ro g ram read s a p r o o f c o p y o f a te x t in ord e r to

d e te c t an d co r re c t an y errors. • O p t i c a l c h a r a c t e r r e c o g n i t i o n . T h e a u to m atic tran slation o f im ag es o f h an d ­

w ritten, typ ew ritten, o r p rin ted te x t (u su ally cap tu red b y a s c a n n e r) in to m ach in e -

e d ita b le te xtu al d o cu m en ts. T h e s u c c e s s an d pop u larity o f te x t m in in g d e p e n d greatly o n ad v an ce m e n ts in NLP

in b o th g e n era tio n as w e ll as un derstand ing o f h u m a n langu ages. NLP e n a b le s the e x tr a c ­ tio n o f featu res fro m u n stru ctu red te x t s o th at a w id e variety o f data m in in g te ch n iq u es c a n b e u s e d to e x tra ct k n o w le d g e (n o v e l a n d u se fu l p attern s an d re latio n sh ip s) from it. In th a t se n se , sim p ly pu t, te x t m in in g is a c o m b in a tio n o f NLP a n d d ata m ining.

SECTION 7 . 3 REVIEW QUESTIONS

1 . W h a t is natu ral lan g u ag e p rocessing?

2 . H o w d o e s NLP relate to te x t mining? 3 . W h a t a re so m e o f th e b e n e fits an d ch a lle n g e s o f NLP? 4 . W h a t are th e m o st co m m o n ta sk s ad d resse d b y NLP?

7.4 TEXT M IN IN G A P P LIC A T IO N S As th e am o u n t o f u n stru ctu red data c o lle c te d b y o rg an ization s in cre a se s, s o d o e s the v a lu e p ro p o sitio n and pop u larity o f te x t m in in g to o ls. M any o rg an ization s a re n o w rea iz- ing th e im p o rtan ce o f e x tra ctin g k n o w le d g e fro m th eir d o cu m e n t-b a se d d ata re p o sito rie s th ro u g h th e u s e o f te x t m in in g to o ls. F o llo w in g are o n ly a sm all su b s e t o f th e exe m p lary

a p p lica tio n ca te g o rie s o f te x t m ining.

3 3 0 Part III * Predictive Analytics

Chapter 7 • T ex t Analytics, T e x t Mining, and Sentim ent Analysis 331

M arketing Applications T e x t m ining c a n b e u s e d to in cre ase cro ss-sellin g and u p -sellin g b y analyzing th e u n stru c­ tu red data g e n e ra te d b y call ce n te rs. T e x t g e n era ted b y call c e n te r n o te s as w e ll as tran ­ scrip tions o f v o ic e co n v e rsa tio n s w ith cu sto m ers ca n b e an aly zed b y te x t m in in g algorithm s to e x tract n o v e l, a ctio n a b le in form ation a b o u t cu sto m ers' p e rce p tio n s tow ard a co m p a n y ’s p ro d u cts a n d serv ices. A dditionally, b lo g s , u s e r review s o f p ro d u cts a t in d e p e n d e n t W e b sites, and d iscu ssio n b o a rd p o stin g s are a g o ld m in e o f cu sto m e r sen tim en ts. T h is rich c o l­ le ctio n o f in fo rm atio n , o n c e p ro p erly an alyzed , c a n b e u se d to in cre a se satisfactio n and th e ov erall lifetim e v alu e o f th e cu sto m e r (C o u sse m e n t a n d V an d e n P o e l, 2 0 0 8 ).

T e x t m in in g has b e c o m e in v alu ab le fo r cu sto m e r relatio n sh ip m an ag em en t. C o m p an ies c a n u s e te x t m in in g to an alyze rich sets o f un stru ctu red te x t d ata, c o m b in e d w ith th e re le v a n t stru ctured data e x tra cte d fro m org an izatio n al d a ta b a se s, to p re d ict cu s­ to m e r p e rce p tio n s an d s u b s e q u e n t p u rch asin g b eh av io r. C o u sse m en t a n d V a n d e n P o e l (2 0 0 9 ) su cce ssfu lly ap p lie d te x t m ining to sign ifican tly im p ro ve th e ab ility o f a m o d el to p re d ict cu sto m e r c h u m (i.e ., cu sto m e r attrition) s o that th o se cu sto m ers id e n tified as m o st likely to le a v e a co m p a n y are a ccu rate ly id en tified fo r re te n tio n tactics.

G h an i et al. (2 0 0 6 ) u se d te x t m ining to d ev elo p a sy stem ca p a b le o f inferring im plicit an d exp licit attributes o f p roducts to e n h a n c e retailers’ ability to analyze p ro d u ct d atabases. T reating p ro d u cts as sets o f attrib u te-v alu e pairs rath er th an as ato m ic en tities c a n p o ten ­ tially b o o s t th e e ffe ctiv e n ess o f m any b u sin ess ap p licatio n s, inclu d ing d em an d fo recasting , assortm ent op tim ization, p ro d u ct re com m en d ation s, assortm ent co m p ariso n acro ss retail­ ers an d m anu factu rers, and p rodu ct su p p lier s ele ctio n . T h e p ro p o se d sy stem allow s a b u sin ess to re p re se n t its p rodu cts in term s o f attributes and attribute v alu es w ith o u t m u ch m anual effort. T h e system learn s th e se attributes b y applying superv ised a n d sem i-su per- v ised learn ing te ch n iq u e s to p rodu ct d escrip tions fo u n d o n retailers’ W e b sites.

Security Applications O n e o f th e la rg est an d m o st p ro m in en t te x t m ining a p p lica tio n s in th e secu rity d om ain is p ro b a b ly th e h ig h ly classified ECHELON su rv eillan ce system . As ru m o r h a s it, ECHELON is assu m ed to b e c a p a b le o f identifying th e c o n te n t o f te le p h o n e calls, fa x e s, e-m ails, and o th e r ty p e s o f d ata a n d in te rcep tin g in form ation s e n t via satellites, p u b lic sw itch e d te le ­ p h o n e n e tw o rk s, a n d m icro w av e links.

In 2 0 0 7 , EU R O PO L d e v e lo p e d an in teg rated sy stem c a p a b le o f a c c e ss in g , storing, an d analyzing vast am o u n ts o f stru ctured an d u n stru ctu red data so u rce s in o rd e r to track transnational o rg a n iz e d crim e. C alled th e O verall A nalysis System fo r In te llig e n c e Su p p ort (O A SIS), this sy stem aim s to integrate th e m o st ad v an ce d data and te x t m in in g te c h ­ n o lo g ie s a v ailab le in to d ay ’s m arket. T h e system h a s e n a b le d EU R O PO L to m a k e sig­ nificant p ro g re ss in su p p o rtin g its law e n fo r c e m e n t o b je c tiv e s at th e in te rn atio n al lev el (EU R O PO L, 2 0 0 7 ).

T h e U .S. F e d e ra l B u re a u o f Inv estig atio n (F B I) an d th e C entral In te llig e n c e A g en cy (CIA), u n d e r th e d irectio n o f th e D ep artm en t fo r H o m elan d Security, are jo in tly d e v e lo p ­ ing a su p e rco m p u te r d ata a n d te x t m in in g system . T h e system is e x p e c te d to c re a te a gigantic data w a re h o u s e alo n g w ith a variety o f data an d te x t m in in g m o d u le s to m e e t th e k n o w le d g e -d isco v e ry n e e d s o f fe d eral, state, an d lo c a l la w e n fo r c e m e n t ag e n cie s. P rior to this p ro je c t, th e F B I a n d CIA e a c h h ad its o w n sep arate d atab ases, w ith little or n o in te rco n n e ctio n .

A n oth er secu rity -re lated a p p lica tio n o f te x t m in in g is in th e a re a o f d eception detection. A p plying te x t m ining to a la rg e s e t o f real-w o rld crim inal (p e rs o n -o f-in te re s t) statem en ts, F u ller e t al. (2 0 0 8 ) d e v e lo p e d p re d ictio n m o d e ls to d iffe ren tiate d ece p tiv e statem en ts fro m truthful o n e s . U sing a rich s e t o f c u e s e x tra cte d fro m th e te xtu al state­ m ents, th e m o d e l p re d icted the h o ld o u t sam p les w ith 7 0 p e rce n t a ccu ra cy , w h ich is

3 3 2 Part III • Predictive Analytics

b e lie v e d to b e a sig n ifican t s u c c e s s co n sid e rin g th at th e c u e s a re e x tra c te d o n ly from textu al state m e n ts (n o v e rb a l o r v isual c u e s a re p re sen t). F u rth erm ore, c o m p a re d to other d e c e p tio n -d e te c tio n te ch n iq u e s, s u ch a s p o ly g rap h , this m e th o d is n on in tru siv e and w id ely a p p lica b le to n o t o n ly te xtu al d ata, b u t a ls o (p o te n tially ) to tran scrip tion s o f v o ice record in g s. A m o re d eta iled d escrip tio n o f te x t-b a s e d d e c e p tio n d ete ctio n is p ro v id e d in

A p p lication C ase 7-3-

Application Case 7.3 M ining fo r Lies D riv en b y ad v an ce m e n ts in W e b -b a s e d inform a­ tio n te c h n o lo g ie s a n d in cre asin g globalizatio n , co m ­ p u ter-m e d iate d co m m u n ica tio n co n tin u e s to filter in to e v ery d ay life, b rin g in g w ith it n e w v e n u e s for d e c e p tio n . T h e vo lu m e o f te x t-b a s e d ch at, instant m essag in g , te x t m essag in g , an d te x t g e n e ra te d b y o n lin e co m m u n ities o f p ra ctice is in cre asin g rap ­ idly. E v e n e-m a il c o n tin u e s to gro w in use. W ith the m assiv e gro w th o f te x t-b a s e d co m m u n icatio n , the p o ten tia l fo r p e o p le to d e c e iv e o th ers th ro u g h c o m ­ p u ter-m e d ia te d co m m u n ica tio n has a lso g ro w n , and s u ch d e c e p tio n c a n h av e disastrous results.

U n fortu n ately, in g e n era l, h u m an s te n d to p e rfo rm p o o rly at d e c e p tio n -d e te c tio n tasks. This p h e n o m e n o n is e x a c e rb a te d in te x t-b a s e d c o m m u ­ n ica tio n s. A la rg e p art o f th e re se a rch o n d e c e p tio n d e te c tio n (a ls o k n o w n as cred ib ility assessm en t) h a s in v o lv e d fa c e -to -fa c e m eetin g s and interview s. Y et, w ith th e gro w th o f te x t-b a s e d co m m u n ica tio n , te x t- b a s e d d e c e p tio n -d e te c tio n te c h n iq u e s are essen tial.

T e ch n iq u es fo r successfully detecting d ece p tio n — that is, lies— hav e w id e applicability. Law' e n fo rcem en t ca n u se d ecisio n support to o ls a n d te ch ­ niqu es to investigate crim es, c on d u ct security screen in g in airports, and m onitor com m unications o f suspected terrorists. H um an resou rces professionals might use d ece p tio n d etectio n tools to scre en applicants. T h e se to o ls an d tech n iq u es also hav e the p otential to scre en e-m ails to u n co v er fraud o r oth er w rongd oings co m ­ m itted b y corp orate officers. Although som e p e o p le b e lie v e that they c a n readily identify th o se w h o are n o t b e in g truthful, a sum m ary o f d ecep tion research sh o w e d that, o n average, p e o p le are only 5 4 p ercen t accu rate in m aking v eracity determ inations (B o n d and D eP au lo , 2006). T h is figure m ay actually b e w o rse w h en h u m an s try to d etect d ecep tion in text.

U sin g a c o m b in a tio n o f te x t m in in g a n d data m in in g te ch n iq u e s , F u ller e t al. ( 2 0 0 8 ) an aly zed p e rs o n -o f-in te re s t state m e n ts c o m p le te d b y p e o p le

involved in crim e s o n m ilitary b a se s . In th e s e state­ m ents, su s p e cts a n d w itn e sses are req u ired to w rite th e ir r e c o lle c tio n o f th e e v e n t in th eir o w n w ord s. M ilitary la w e n fo rc e m e n t p e rs o n n e l s e a rch e d arch i­ v al data fo r sta te m e n ts th at th ey co u ld co n clu siv ely identify as b e in g truthful o r d e ce p tiv e . T h e s e d e c i- j sio n s w e re m ad e o n th e b asis o f co rro b o ratin g e v id e n c e a n d c a s e reso lu tio n . O n c e la b e le d as truthful o r d e c e p tiv e , th e law e n fo rc e m e n t p e rs o n ­ n e l re m o v e d identifying in form ation a n d g av e th e sta te m e n ts t o th e re se a rch team . In to tal, 371 u sab le statem en ts w e re re c e iv e d fo r analysis. T h e text- b a s e d d e c e p tio n -d e te c tio n m e th o d u s e d b y F u ller et al. ( 2 0 0 8 ) w as b a s e d o n a p ro ce s s k n o w n as m essag e fe a t u r e m in in g , w h ich re lie s o n e le m e n ts o f data a n d te x t m in in g te ch n iq u e s. A sim plified d ep ictio n o f th e p ro ce s s is p ro v id e d in Fig u re 7.3 .

First, th e research ers p rep ared the data fo r p ro ­ cessing. T h e original handw ritten statem ents had to b e tran scrib ed into a w ord p ro cessin g file. S econ d , featu res (i.e ., c u e s ) w e re identified. T h e research ers identified 31 features representing cate g o rie s o r types o f language that are relatively in d ep en d en t o f th e text co n ten t an d that ca n b e read ily analyzed b y auto­ m ated m e a n s. F o r exam p le, first-person pronou ns su ch as I o r m e c a n b e identified w ithout analysis o f th e su n o u n d in g text. T a b le 7 .1 lists th e categories and a n e x a m p le list o f featu res u se d in this study.

T h e fe a tu res w e re e x tra cte d fro m th e textual state m e n ts a n d input in to a flat file fo r fu rth er p ro ­ ce ssin g . U sing sev eral fe a tu re -s e le ctio n m eth o d s a lo n g w ith 10 -fo ld cro ss-v alid atio n , th e re search ers co m p a re d th e p re d ictio n a ccu ra cy o f th ree p o p u ­ lar data m in in g m eth o d s. T h e ir results ind icated that n eu ral n e tw o rk m o d e ls p e rfo rm e d th e b est, w ith 7 3 .4 6 p e rc e n t p re d ictio n a c c u ra c y o n test data sam p les; d e c is io n tre e s p e rfo rm e d s e c o n d b est, w ith 7 1 .6 0 p e r c e n t a ccu ra cy ; a n d lo g istic re g re ssio n w as last, w ith 6 7 .2 8 p e rce n t accu racy .

Chapter 7 • T ex t Analytics, T e x t Mining, and Sentim ent Analysis 3 3 3

FIGURE 7.3 Text-Based Deception-Detection Process. Source: C. M . Fuller, D. Biros, and D. Delen, "Exploration of Feature Selection and Advanced Classification Models for High-Stakes Deception Detection," in Proceedings o f the 41st A nnual Hawaii International Conference on System Sciences (HICSS), January 2008, Big Island, HI, IEEE Press, pp. 80-99.

T A B L E 7.1 Categories and Examples o f Linguistic Features Used in D eception Detection

Number Construct (Category) Example Cues

1 Quantity Verb count, noun-phrase count, etc.

2 Complexity Average number of clauses, average sentence length, etc.

3 Uncertainty Modifiers, modal verbs, etc.

4 Nonimmediacy Passive voice, objectification, etc.

5 Expressivity Emotiveness

6 Diversity Lexical diversity, redundancy, etc.

7 Informality Typographical error ratio

8 Specificity Spatiotemporal information, perceptual information, etc.

9 Affect Positive affect, negative affect, etc.

T h e resu lts in d icate that au to m ated te x t-b a s e d o f th e s e te c h n iq u e s e x c e e d e d th e a ccu ra cy o f m ost d e c e p tio n d e te c tio n h a s th e p otential to aid th o s e o th e r d e c e p tio n -d e te c tio n te ch n iq u e s e v e n th o u g h it w h o m u st try to d etect lies in t e x t and c a n b e su e- w as lim ited to te x tu a l cu es, cessfu lly a p p lie d to re al-w o rld data. T h e a ccu ra cy

( Continued)

3 3 4 Part III • Predictive Analytics

Application Case 7.3 (Continued) Q u e s t i o n s f o r D i s c u s s i o n

1. W h y is it d ifficult to d e te c t d ecep tion ? 2 . H o w c a n text/data m in in g be, u s e d to d e te ct

d e c e p tio n in text? 3. W h at d o y o u th in k a re th e m ain ch a lle n g e s fo r

s u ch a n au to m ated system?

S o u r c e s C. M. Fuller, D. Biros, and D. Delen, “Exploration of Feature Selection and Advanced Classification Models for High- Stakes D eception D etection,” in P roceedin gs o f the 41st A n nu al H aw aii In tern a tion al C on feren ce o n System S cien ces (HICSS), 2008, Big Island, HI, IEEE Press, pp. 8 0 -9 9 ; C. F. Bond and B. M DePaulo, “Accuracy o f D eception Judgm ents,” P ersonality a n d S o cial P sychology Reports, Vol. 10, No. 3, 2006, pp. 214-234.

Biomedical Applications T e x t m in in g h o ld s g reat p o ten tial fo r th e m ed ical field in g e n e ra l and b io m e d icin e in par­ ticu lar fo r s ev eral re aso n s. First, th e p u b lish e d literatu re and p u b lica tio n o u tlets (e sp e cially w ith th e ad v en t o f th e o p e n s o u rc e jo u rn a ls) in th e field are e x p a n d in g a t an e x p o n e n tia l rate. S e c o n d , c o m p a re d to m o st o th e r field s, th e m e d ica l literatu re is m o re standardized a n d orderly,1 m ak in g it a m o re “m in a b le ” in fo rm atio n s o u rc e . Finally, th e term in ology u s e d in this literatu re is relativ ely co n stan t, h av in g a fairly stand ard ized o n to lo g y . W hat fo llo w s are a fe w e x e m p la ry stu d ies w h e re te x t m in in g te c h n iq u e s w e re s u cce ssfu lly u se d in e x tra ctin g n o v e l pattern s fro m b io m e d ica l literature.

E xp e rim e n tal te c h n iq u e s s u ch as DNA m icroarray analysis, serial an alysis o f g e n e e x p r e s s io n (SA G E ), an d m ass sp e ctro m etry p ro te o m ics , a m o n g o th ers, are g e n era t­ ing large am o u n ts o f data related to g e n e s a n d p ro te in s. As in an y o th e r e xp erim en tal a p p ro a c h , it is n e ce s s a ry to an aly ze this vast am o u n t o f d ata in th e c o n te x t o f previ­ o u sly k n o w n in form ation a b o u t th e b io lo g ica l e n titie s u n d e r study. T h e literature is a particu larly v alu ab le s o u rc e o f in form ation fo r e x p e rim e n t v alid ation a n d in terp ietation . T h e re fo re , th e d ev e lo p m e n t o f au to m ate d te x t m in in g to o ls to assist in s u c h interpretation is o n e o f th e m ain ch a lle n g e s in cu rre n t b io in fo rm atics re search .

K n o w in g th e lo c a tio n o f a p ro te in w ith in a c e ll c a n h e lp to elu cid ate its ro le in b io lo g ica l p ro c e s s e s and to d eterm in e its p o ten tia l as a drug target. N um erous locatio n - p re d ictio n system s are d e s crib e d in th e literatu re; s o m e fo cu s o n s p e cific organism s, w h e re a s o th ers attem p t to an aly ze a w id e ra n g e o f organ ism s. Sh atkay e t al. (2 0 0 7 ) p ro p o s e d a co m p re h e n siv e sy stem th at u se s sev eral typ es o f s e q u e n c e - an d te x t-b a se d featu res to p re d ict th e lo c a tio n o f p ro teins. T h e m a in n ov elty o f th e ir sy stem lies in the w a y in w h ic h it s e le cts its te x t so u rce s an d fe a tu res a n d integrates th e m w ith s e q u e n c e - b a s e d featu res. T h e y te sted th e sy stem o n p re v io u sly u s e d data sets a n d o n n e w data sets d ev ised sp e cifica lly to te st its p red ictiv e p o w e r. T h e results s h o w e d that th eir system co n siste n tly b e a t p rev iou sly re p o rted results.

C h un e t al. ( 2 0 0 6 ) d e s crib e d a sy stem th at e x tra cts d is e a s e -g e n e relatio n sh ip s fro m literatu re a c c e s s e d via M edLine. T h e y co n stru cte d a d iction ary fo r d is e a s e an d g e n e n am es fro m s ix p u b lic d a tab ase s an d e x tra cte d re latio n can d id ates b y d ictio n ary m atch ing. B e c a u s e d iction ary m atch in g p ro d u ce s a larg e n u m b e r o f fa lse p o sitives, th ey d e v e lo p e d a m e th o d o f m a ch in e le a rn in g -b a s e d n a m e d en tity re co g n itio n (N ER ) to filter ou t false re co g n itio n s o f d isease/ gen e n am es. T h e y fo u n d th at th e s u cce s s o f relatio n e x tra ctio n is h eavily d e p e n d e n t o n th e p e rfo rm a n ce o f NER filtering an d th at th e filtering im p ro v ed th e p re cisio n o f re latio n e x tra ctio n b y 2 6 .7 p e rce n t, a t th e c o s t o f a sm all red u ctio n in recall.

Figure 7 .4 sh o w s a sim p lified d ep ictio n o f a m u ltilev el te x t an alysis p ro ce s s fo r d is­ co v erin g g e n e - p r o te in relatio n sh ip s (o r p ro te in -p ro te in in teractio n s) in th e b io m ed ica l literatu re (N akov e t al., 2 0 0 5 ). As c a n b e s e e n in this sim p lified e x a m p le that u s e s a sim ­ p le s e n te n c e fro m b io m e d ica l te x t, first (a t th e b o tto m th re e le v e ls ) th e te x t is to k en iz ed

Chapter 7 • T ex t Analytics, T e x t Mining, and Sentim ent Analysis 3 3 5

CD £ CD cl

a

. 2 in ® CO £ CL

5 9 6 . 1 2 0 4 3 2 4 2 2 4 2 8 1 0 2 0 ' " ' - --W -" - ' '

42722 397276

D019254

D007962

D 016923

D 001773

□044465 D001769 D002477 D003643 D016158

185 8 51112 9 23017 27 5874 2791 895 2 1623 5632 17 8252

NN IN NN IN VBZ JJ JJ NN NN CC NN

2523

NP PP NP NP PP NP NP PP NP

FIGURE 7.4 Multilevel Analysis of Text for Gene/Protein Interaction Identification. Source: P. Nakov, A . Schwartz, B. W olf, and M . A . Hearst, "Supporting A n n o ta tio n Layers fo r Natural Language Processing," Proceedings o f the Association for Computational Linguistics (ACL), interactive poster and dem onstration sessions, 2005, A n n A rb o r, M l, Association fo r Com putational Linguistics, pp. 65-68.

u sin g p art-of-speech ta g g in g a n d shallow -p arsing . T h e to k e n iz e d te rm s (w o rd s) are th e n m a tch e d (an d in terp reted ) a g ain st th e h iera rch ica l re p re se n ta tio n o f th e d om ain o n to lo g y t o d eriv e th e g e n e - p r o te in re latio n sh ip . A p p licatio n o f this m e th o d (and/or so m e variatio n o f it) to th e b io m e d ic a l literatu re o ffers g re a t p o ten tial to d e c o d e th e co m p le x itie s in th e H u m an G e n o m e P ro ject.

Academic Applications T h e issu e o f text m ining is o f g reat im p ortance to p u blish ers w h o h o ld large d atabases o f inform ation requ iring in d ex in g fo r b etter retrieval. T h is is particularly true in scientific d isciplines, in w h ich highly s p e c ific inform ation is o ften con tain ed w ith in w ritten text. Initiatives h a v e b e e n lau n ch ed , su ch as N ature's p ro p osal fo r a n O p e n T e x t M ining Interface (O TM I) a n d th e N ational Institutes o f H ealth’s co m m o n Jo u rn a l P u b lishing D o cu m e n t T ype D efinition (D T D ), w h ich w o u ld p rovide sem an tic cu e s to m a ch in e s to a n s w e r s p e cific q u e ­ ries co n ta in e d within te x t w ithout rem ov in g p u b lish e r barriers to p u b lic acce ss.

A cad em ic institutions hav e also lau n ch ed te x t m ining initiatives. F o r exam p le, the N ational C entre fo r T e x t M ining, a collab o rativ e effo rt betw een- th e U niversities o f M an ch ester a n d L iverpool, provides cu stom ized to ols, re sea rch facilities, an d ad vice o n te x t m ining to th e a ca d em ic com m unity. W ith a n initial fo cu s o n te x t m in in g in th e b io lo g i­ cal an d b io m ed ica l sc ie n c e s , re sea rch has sin c e e x p a n d e d into th e so cia l scie n ce s. In the U nited States, th e Sch o o l o f In fo rm ation a t the University o f California, B e r k e le y , is d ev el­ o p in g a p ro g ram called B io T e x t t o assist b io s c ie n c e research ers in te x t m in in g a nd analysis.

As d e s c rib e d in this s e ctio n , te x t m in in g h a s a w id e variety o f ap p lica tio n s in a n u m ­ b e r o f d iffe ren t d iscip lin es. S e e A p p lication C ase 7 .4 fo r a n e x a m p le o f h o w a fin an cial s erv ices firm is u sin g te x t m in in g to im p ro ve its cu sto m e r serv ice p e rfo rm a n ce .

3 3 6 Part III • Predictive Analytics

Application Case 7.4 Text M ining and Sen tim en t A nalysis Help Im p rove Custom er S e rv ice Perform ance

T h e c o m p a n y is a fin an cial s erv ices firm that p ro ­ v id es a b ro a d ra n g e o f so lu tio n s an d serv ices to a g lo b a l cu sto m e r b a s e . T h e co m p a n y has a c o m ­ p re h e n siv e n e tw o rk o f facilities aro u nd th e w orld, w ith o v e r 5 0 0 0 a sso cia tes assisting th e ir cu stom ers. C u stom ers lo d g e serv ice re q u e sts b y te le p h o n e , e m ail, o r th ro u g h a n o n lin e c h a t in terface.

As a B 2 C service provider, th e co m p an y strives to m aintain h ig h standards fo r effective com m unication b e tw e e n th eir associates and custom ers, and tries to m o n ito r cu stom er interactions at every op p or­ tunity. T h e b ro a d o b jectiv e o f this service perfor­ m a n ce m onitoring is to m aintain satisfactory quality o f service o v e r tim e and acro ss th e organization. T o this en d , th e com p an y has d evised a s e t o f standards for serv ice e x c e lle n c e , to w h ich all cu stom er interac­ tions are e x p e c te d to ad here. T h e se standards co m ­ prise d ifferent qualitative m easu res o f service levels (e .g ., associates shou ld use clear an d understand­ a b le lang u age, associates should always m aintain a p ro fession al an d friendly d em ean o r, e tc.) A ssociates p erfo rm an ces are m easured b a se d o n com p lian ce w ith th ese quality standards. O rganizational units at differ­ e n t lev els, like team s, departm ents, and th e com pany as a w h o le , also receiv e sco re s b a se d o n associate p erfo rm an ces. T h e evaluations an d rem unerations o f n ot only th e associates b u t also o f m an agem en t are in flu en ced b y th e se service p erfo rm ance scores.

C h a lle n g e

C o n tin u ally m o n ito rin g serv ice lev els is essen tial fo r s e rv ic e qu ality co n tro l. C u stom er surveys are a n e x c e lle n t w ay o f g ath erin g fe e d b a c k a b o u t s erv ice lev els. An e v e n rich e r so u rce o f inform ation is th e co rp u s o f a ss o cia te -cu s to m e r in teractio ns. H istorically th e c o m p a n y m anu ally e v alu ate d a sam p le o f a ss o cia te -cu s to m e r in teractio n s an d survey r e s p o n s e s fo r c o m p lia n ce w ith e x c e lle n c e standards. T h is a p p ro a c h , in ad d ition to b e in g s u b je ctiv e and e rro r-p ro n e , w a s tim e- an d lab o r-in ten siv e. A d vances in m a ch in e learn in g an d com p u tatio n al linguistics o ffe r a n op p o rtu n ity to o b je ctiv e ly e v alu ate all c u sto m e r in teractio n s in a tim ely m anner.

T h e co m p a n y n e e d s a system fo r ( 1 ) au tom ati­ c ally e v alu atin g a ss o cia te -cu s to m e r in teractio n s for

survey r e s p o n s e s to e x tra ct positive and negative fe e d b a c k . T h e a n alysis m ust b e a b le to a c c o u n t fo r the w id e diversity o f e x p r e s s io n in n atural lan gu age (e .g ., p lea sa n t and reassu rin g to n e , a c c e p ta b le lan g u ag e, a p p rop riate a b b re v ia tio n s, ad d ressin g all o f th e c u s ­ to m e rs ’ issu es, e tc .).

S o lu tio n

P olyA nalyst 6 .5 ™ b y M eg ap u ter In te llig e n ce is a d ata m ining an d an aly sis platform th at p ro v id es a c o m p re h e n siv e s e t o f to o ls fo r analyzing stru ctured an d u n stru ctu red data. P oly A nalyst’s te x t analysis to o ls are u s e d fo r e x tra ctin g c o m p le x w o rd pat­ tern s, gram m atical an d s em an tic re latio n sh ip s, and e x p re ssio n s o f sen tim en t. T h e results o f th e s e te x t a n aly se s a re th e n classified in to c o n te x t-s p e c ific th e m e s to id en tify a c tio n a b le issu es, w h ic h can b e assig n e d to relev an t individuals re s p o n s ib le for th eir re so lu tio n . T h e sy stem c a n b e p ro g ram m ed to pro v id e fe e d b a c k in c a s e o f in su fficien t classifica­ tio n s o th at an a ly se s c a n b e m o d ified o r am en d ed . T h e re latio n sh ip s b e tw e e n stru ctured field s an d text analysis results are a lso e sta b lis h e d in o rd e r to id e n ­ tify pattern s a n d in te ractio n s. T h e system p u blish es th e results o f an a ly se s th ro u g h g rap h ical, interactive, w e b -b a s e d re p o rts. U sers cre a te an alysis s ce n a r­ io s u sin g a d rag-an d -d ro p g rap h ical u s e r in terface (G U I). T h e s e s c e n a rio s are re u sa b le so lu tio n s that c a n b e p ro g ram m ed to au to m ate th e an alysis and rep o rt g e n e r a tio n p ro cess.

A s e t o f s p e c ific criteria w e re d esig n e d to c a p ­ ture and au tom atically d e te ct co m p lia n ce w ith th e c o m p a n y ’s Q u ality Standards. T h e fig u re b e lo w d isplays a n e x a m p le o f a n a s s o c ia te ’s re sp o n se , as w e ll as th e qu ality criteria that it s u c c e e d s o r fails to m atch .

As illu strated a b o v e , this co m m e n t m atch es se v e ra l criteria w h ile failin g to m a tch o n e , and co n trib u tes a cco rd in g ly to th e a s s o c ia te ’s p e rfo r­ m a n c e s co re . T h e s e s c o re s a re th e n au tom atically c a lc u la te d a n d ag g re g ate d a cro ss v a rio u s o rg an iza­ tio n al u n its. It is relativ ely e a sy to m o d ify th e sy s­ te m in c a s e o f c h a n g e s in qu ality stand ard s, an d th e c h a n g e s c a n b e q u ick ly ap p lie d to h istorical data. T h e sy stem a ls o has a n integrated c a s e m a n a g e m e n t

w h ic h g e n e r a te s em ail alerts in c a s e o fco m p lia n c e w ith q u ality stand ard s an d ( 2 ) an aly zin g sy stem ,

Chapter 7 • T e x t Analytics, T e x t Mining, and Sentim ent Analysis 3 3 7

Mentioned customer’s name Request sent to incorrect department

G re g , 1 am fo rw a rd in g this item to th e c o r r e c t d e p a rtm e n t for

Mentioned next department

A c c o u n tin g , upon processing, please advise Greg of the correct path for similar requests. Thank you and h a ve a nice

Pleasantry to next department

d ro p s in serv ice q u ality and allo w s u sers to track th e p ro g re ss o f issu e reso lu tio n .

T a n g ib le R e s u lts

1. C o m p letely au tom ated analysis; sav e s tim e. 2. A nalysis o f e n tire d ataset (> 1 m illion re co rd s

p e r y e a r); n o n e e d fo r sam pling. 3. 4 5 % c o s t savings o v e r trad itional analysis. 4 . W e e k ly p ro cessin g . In th e c a s e o f traditional

analysis, data co u ld o n ly b e p ro ce s s e d m o n th ly d u e t o tim e an d re s o u r c e constraints.

5. A nalysis n o t s u b je ctiv e to th e analyst. a. In c re a s e d accu racy . b. In cre a sed uniform ity.

6 . G re a te r acco u n tab ility . A sso ciate s c a n review th e analysis a n d raise co n c e rn s in c a s e o f d iscre p a n cies.

F u t u r e D ir e c tio n s

C urrently th e co rp u s o f a ss o cia te -cu s to m e r inter­ a ctio n s d o e s n o t in clu d e tran scrip ts o f p h o n e co n v e rsatio n s. B y in co rp o ratin g s p e e c h re co g n itio n cap ab ility, th e system c a n b e c o m e a o n e -s to p desti­ n a tio n fo r an aly zin g all cu sto m e r in te ractio n s. T h e sy stem co u ld a ls o p o ten tially b e u se d in real-tim e, in stead o f p e rio d ic analyses.

Q u e s t i o n s f o r D i s c u s s i o n

1. H o w d id th e fin an cial s erv ices firm u s e te x t m in in g a n d te x t an alytics to im p ro v e its cu sto m e r serv ice p erfo rm an ce?

2. W h a t w e re th e c h a lle n g e s, th e p ro p o s e d so lu ­ tion , a n d th e o b ta in e d results?

Source: Megaputer, Customer Success Story, megaputer.com (accessed Septem ber 2013)-

SECTION 7 . 4 REVIEW QUESTIONS

1 . List a n d b riefly d iscu ss s o m e o f th e te x t m in in g ap p lica tio n s in m arketing.

2 . H o w c a n te x t m in in g b e u s e d in secu rity an d cou nterterrorism ? 3 . W h a t a r e so m e p ro m isin g te x t m in in g ap p lica tio n s in b iom ed icin e?

7.5 TEX T M IN IN G PR O C E S S In o rd e r to b e su ccessfu l, text m ining stud ies sh o u ld fo llo w a so u n d m e th o d o lo g y b ase d o n b e s t p ractices. A standardized p ro cess m o d el is n e e d e d sim ilar to CRISP-DM, w h ich is the industry standard fo r data m ining p ro jects (s e e C h ap ter 5). E ven th o u g h m ost parts o f CRISP-DM a re also ap p licab le to te x t m ining p ro jects, a sp e cific p ro cess m o d el fo r te x t m in­ ing w o u ld in clu d e m u ch m o re e la b o ra te data p re p ro cessin g activities. Figu re 7 .5 d ep icts a hig h-lev el c o n te x t diagram o f a typical text m ining p ro cess (D e le n an d C rossland, 2 008). This c o n te x t diagram p resen ts th e s c o p e o f th e p ro cess, em phasizing its in terfaces w ith the larger en viron m en t. In e ss e n c e , it draw s b o u n d aries aro u nd th e sp e cific p ro ce s s to e xp lic­ itly identify w h at is in clu d ed in (a n d e x clu d e d fro m ) the text m ining p ro cess.

3 3 8 Part III • Predictive Analytics

So ftw a re /h a rd w a re limitations

Privacy issues

Linguistic limitations

U nstructured data (text)

Structured data [databases]

Extract knowledge from available data sources

AO

Context-specific knowledge

Dom ain expertise

Tools and techniques

FIGURE 7.5 Context Diagram for the Text Mining Process.

As th e co n te x t diagram indicates, th e input (inw ard co n n e ctio n to th e left ed g e o f the b o x ) into th e te xt-b ase d know led g e-d iscov ery p ro ce s s is the un structured as w ell as struc­ tured data co llected , stored, an d m ad e available to th e pro cess. T h e ou tpu t (outw ard e x te n ­ sio n from th e right ed g e o f th e b o x ) o f the p ro ce s s is th e co n tex t-sp e cific kn o w le d g e that c a n b e u s e d for d e cisio n m aking. T h e con trols, a lso calle d th e con strain ts (inw ard co n n e c ­ tion to th e top ed g e o f the b o x ), o f th e p ro cess in clu d e softw are and hard w are limitations, privacy issu es, and th e difficulties related to p ro cessin g th e te x t that is p re sen ted in the form o f natural language. T h e m ech an ism s (inw ard c o n n e c tio n to th e b o tto m ed g e o f th e b o x ) o f th e p ro cess in clu d e p ro p e r te ch n iq u es, softw are to o ls , an d d om ain exp ertise. T h e primary p u rp ose o f te x t m ining (w ithin th e co n te x t o f k n o w le d g e d iscov ery) is to p ro cess unstruc­ tu red (tex tu al) data (alo n g w ith structured data, i f relevan t to th e p ro b lem b e in g addressed an d available) to e xtract m eaningful an d a ctio n a b le patterns fo r b etter d ecisio n m aking.

At a v e ry h ig h lev el, th e te x t m ining p ro c e s s c a n b e b r o k e n d o w n into th ree c o n s e c ­ utive task s, e a c h o f w h ic h h a s s p e c ific inputs to g e n e ra te certain ou tp u ts (s e e Figu re 7 .6 ). If, for s o m e re aso n , th e ou tp u t o f a ta sk is n o t w h a t is e x p e c te d , a b ack w ard red irection to th e p re v io u s ta sk e x e c u tio n is n ecessary .

Task 1: Establish the Corpus T h e m ain p u rp o s e o f th e first ta sk activity is t o c o lle c t all o f th e d o cu m en ts related to th e c o n te x t (d o m ain o f in te rest) b e in g studied. T h is c o lle c tio n m ay in clu d e textu al d o cu - m en ts, XML files, e-m ails, W e b p ag e s, an d sh o rt n o tes. In ad d ition to th e read ily available textu al data, v o ic e reco rd in g s m ay also b e tra n scrib ed u sin g s p e e c h -re c o g m tio n a lg o ­

rithm s a n d m ad e a part o f th e te x t co lle ctio n . O n c e c o lle cte d , th e te x t d o cu m en ts a re tra n sfo rm e d a n d o rg an ize d in a m an n er

s u c h th at th e y are all in th e sam e re p re sen ta tio n a l fo rm (e .g ., ASCII te x t file s) fo r c o m ­ p u ter p ro cessin g . T h e org an izatio n o f th e d o cu m e n ts ca n b e as sim p le as a c o lle c tio n a t digitized te x t e x c e rp ts s to red in a file fo ld e r o r it c a n b e a list o f lin k s to a c o lle c tio n o f W e b p a g e s in a s p e c ific d om ain. M any co m m e rcia lly a v ailab le te x t m in in g softw are to o ls

Chapter 7 • T ex t Analytics, T e x t Mining, and Sentim ent Analysis 3 3 9

FIGURE 7.6 The Three-Step Text Mining Process.

c o u ld a c c e p t th e s e as input a n d co n v e rt th e m into a flat file fo r p ro cessin g . Alternatively, th e flat file c a n b e p re p are d ou tsid e th e te x t m in in g so ftw are an d th e n p re se n te d a s th e in p u t to th e te x t m in in g ap p licatio n .

Task 2: Create the Term-Document Matrix In this task , th e digitized an d organ ized d o cu m en ts (th e co rp u s) a re u sed to cre a te the term -d o cu m en t m atrix (TDM). In th e TD M , ro w s re p re sen t th e d o cu m en ts and colu m ns re p re se n t th e term s. T h e relatio n sh ip s b e tw e e n th e term s and d o cu m en ts are ch aracterized b y in d ice s (i.e ., a relatio n al m e asu re th at ca n b e as sim p le as th e n u m b e r o f o ccu rre n ce s o f th e term in resp ectiv e d o cu m en ts). Figure 7 .7 is a typical e x a m p le o f a TDM .

FIG U R E 7 .7 A Sim ple Term -D ocum ent Matrix.

3 4 0 Part III • P redictive Analytics

T h e g o al is to co n v e rt th e list o f organized d ocu m en ts (th e co rp u s) into a TD M w h ere th e ce lls are filled w ith th e m o st appropriate indices. T h e assu m p tion is that th e e s s e n c e o f a d o cu m en t c a n b e re p resen ted w ith a list an d fre q u e n cy o f th e term s u se d in that d ocu m ent. H ow ever, are all term s im portant w h e n characterizing d ocu m ents? O bviously, th e answ er is “n o .” So m e term s, su ch as articles, auxiliary v e rb s, and term s u s e d in alm ost all o f the d ocu m en ts in th e corp u s, have n o differentiating p o w e r an d th e re fo re shou ld b e e x clu d ed fro m th e ind exing p ro cess. T h is list o f term s, co m m o n ly calle d stop term s o r stop w ords, is sp e cific to th e d om ain o f study an d shou ld b e id entified b y th e d om ain exp erts. O n the oth er hand , o n e m ight c h o o s e a s e t o f p red eterm in ed term s u n d er w h ich th e d ocu m en ts are to b e in d e x ed (this list o f term s is co n v e n ie n tly calle d in clu d e term s o r diction ary ). Additionally, synonym s (pairs o f term s that are to b e treated th e sa m e ) an d sp e cific phrases (e .g ., “Eiffel T o w e r”) c a n also b e provid ed so that th e in d e x entries are m o re accu rate.

A nother filtration that shou ld tak e p la ce to accu rately create th e indices is stem m ing, w h ich refers t o th e red uction o f w o rd s to their ro ots s o that, for exam p le, different grammati­ ca l form s o r d eclinations o f a v e rb are identified an d in d e x ed as th e sam e word. F o r e x a m ­ ple, stem m ing w ill en su re that m odelin g an d m od eled will b e recog n ized as th e w o rd m odel.

T h e first g e n e ra tio n o f th e TD M in clu d es all o f th e u n iq u e term s id en tified in the co rp u s (a s its co lu m n s), e x clu d in g th e o n e s in th e sto p te rm list; all o f th e d ocu m en ts (a s its ro w s); an d th e o c c u rre n c e c o u n t o f e a c h te rm fo r e a c h d o cu m e n t (a s its c e ll valu es). If, as is co m m o n ly th e c a s e , th e co rp u s in clu d es a rath er large n u m b e r o f d o cu m en ts, th e n th e re is a very g o o d c h a n c e th a t th e TD M w ill h av e a v e iy large n u m b e r o f term s. P ro ce ssin g s u c h a larg e m atrix m ight b e tim e -co n su m in g and , m o re im portantly, m ight lea d to e x tra ctio n o f in a ccu ra te pattern s. At this p o in t, o n e h a s to d e cid e th e fo llow ing : ( 1 ) W h at is th e b e s t re p re se n ta tio n o f th e indices? a n d ( 2 ) H o w c a n w e re d u ce th e d im en ­ sio n ality o f this m atrix to a m a n a g e a b le size?

R E P R E S E N T IN G TH E IN D IC E S O n c e th e input d o cu m e n ts are in d e x e d a n d th e initial w o rd fre q u e n c ie s (b y d o cu m en t) com p u te d , a n u m b e r o f ad d itional tran sform atio ns c a n b e p e rfo rm e d to su m m arize a n d ag g reg ate th e e x tra cte d in form ation. T h e raw te rm fre­ q u e n c ie s g e n erally re fle c t o n h o w salien t o r im p o rtan t a w o rd is in e a c h d o cu m en t. S p ecifically , w o rd s th at o c c u r w ith g re a te r fre q u e n c y in a d o cu m e n t are b e tte r d escrip to rs o f th e co n te n ts o f th at d o cu m en t. H o w ev er, it is n o t re a s o n a b le to assu m e that th e w o rd co u n ts th e m se lv es are p ro p o rtio n a l to th eir im p o rta n ce as d escrip to rs o f th e d o cu m en ts. F o r e x a m p le , i f a w o rd o ccu rs o n e tim e in d o cu m e n t A, b u t th ree tim es in d o cu m e n t B, th e n it is n o t n e ce ssa rily re a s o n a b le to c o n c lu d e th a t this w o rd is th re e tim es a s im portant a d escrip to r o f d o cu m e n t B as c o m p a re d to d o cu m e n t A. In o rd e r to h a v e a m o re c o n ­ siste n t TD M fo r fu rth er analysis, th e se raw in d ice s n e e d to b e norm alized . As o p p o s e d to sh o w in g th e actu al fre q u e n c y cou n ts, th e n u m e rical re p re sen ta tio n b e tw e e n term s and d o cu m en ts c a n b e no rm alized u sin g a n u m b e r o f altern ativ e m e th o d s. T h e fo llo w in g are a fe w o f th e m o st co m m o n ly u se d n o rm alization m e th o d s (StatSoft, 2 0 0 9 ):

• L o g fr e q u e n c i e s . T h e raw fre q u e n cie s c a n b e tran sform ed u sin g th e lo g fu n ction . T h is tran sform atio n w o u ld “d am p en " th e raw fre q u e n c ie s a n d h o w th ey a ffe c t th e results o f s u b s e q u e n t analysis.

f ( w f ) = 1 + lo g (w f) fo r w f> 0

In th e form u la, w f is th e raw w o rd (o r term ) fre q u e n c y a n d f ( w f ) is th e resu lt o f th e lo g tran sform atio n. T h is tran sform atio n is a p p lie d to all o f th e raw fre q u e n c ie s in th e TD M w h e re th e fre q u e n c y is g re a te r th an zero .

• B i n a r y f r e q u e n c i e s . L ikew ise, a n e v e n sim p le r tran sform atio n c a n b e u se d to e n u m erate w h e th e r a te rm is u se d in a d o cu m en t.

f ( w f ) = 1 f o r w f> 0

T h e resu ltin g TD M m atrix w ill co n ta in o n ly I s a n d Os to in d icate th e p re s e n c e o r a b s e n c e o f th e re sp ectiv e w o rd s. Again, this tran sform atio n w ill d a m p e n th e e ffe c t o f th e raw fre q u e n cy co u n ts o n su b s e q u e n t co m p u tatio n s an d an aly ses.

• I n v e r s e d o c u m e n t fr e q u e n c i e s . A n o th er issu e that o n e m ay w a n t to co n s id e r m o re care fu lly an d re fle c t in th e in d ices u se d in fu rth er an aly ses is th e relativ e d o cu m en t fre q u e n cie s (d f) o f d ifferen t term s. F o r e x a m p le , a te rm s u ch a s gu ess m ay o c c u r freq u en tly in all d o cu m en ts, w h e re a s an o th e r term , s u c h as softw are, m ay a p p e a r o n ly a fe w tim es. T h e re a s o n is that o n e m ight m a k e g u esses in various co n te x ts , re g ard le ss o f th e s p e c ific to p ic, w h e re a s softw are is a m o re sem an tically fo cu se d te rm th at is o n ly likely to o c c u r in d o cu m en ts th at d ea l w ith co m p u te r so ft­ w are. A c o m m o n a n d very u sefu l tran sform atio n that re flects b o th th e sp e cificity o f w o rd s (d o cu m e n t fre q u e n c ie s ) a s w e ll a s th e ov erall fre q u e n cie s o f th e ir o c c u r­ r e n c e s (te rm fre q u e n c ie s ) is th e so -ca lle d inverse docum ent frequency (M anning an d S ch u tze, 2 0 0 9 ). T h is tran sform atio n fo r th e z'th w o rd a n d / th d o cu m e n t c a n b e

w ritten as:

( 0 ifu fij ~ 0 i d f i h J ) ~ ) N

U l + l o g ( t t $ , ) ) l o g ^ i f wfij > 1

In this fo rm u la, N is th e to tal n u m b e r o f d o cu m en ts, an d d f is th e d o cu m e n t fre ­ q u e n c y fo r th e zth w o rd (th e n u m b e r o f d o cu m en ts that in clu d e th is w o rd ). H e n ce , it c a n b e s e e n that this fo rm u la in clu d es b o th th e d am p e n in g o f th e s im p le -w o rd fre ­ q u e n c ie s via th e lo g fu n ctio n (d e s c r ib e d h e re ) a n d a w e ig h tin g fa cto r th at ev alu ates to 0 if th e w o rd o ccu rs in all d o cu m en ts [i.e., lo g ( N/ N= 1 ) = 0], an d to th e m axim u m v a lu e w h e n a w o rd o n ly o ccu rs in a sin gle d o cu m en t [i.e., logGV/1) = logOV)]- It c a n e asily b e s e e n h o w this tran sform atio n w ill cre a te in d ices that re fle c t b o th th e re la ­ tive fre q u e n c ie s o f o c c u rre n c e s o f w o rd s a s w e ll as th e ir s em an tic sp e cificitie s o v e r th e d o cu m e n ts in clu d e d in th e analysis. T h is is th e m o st c o m m o n ly u se d tran sfor­

m ation in th e field.

R ED U C IN G TH E D IM E N S IO N A L IT Y O F TH E M A T R IX B e c a u s e th e TD M is o fte n v e ry large an d rath er sp a rse (m o s t o f th e ce lls filled w ith z e ro s ), an o th e r im p ortan t q u e s tio n is “H o w d o w e re d u ce th e d im en sion ality o f this m atrix to a m a n a g e a b le size?” S e v e ra l o p tio n s are

available fo r m a n a g in g th e m atrix size:

• A d o m a in e x p e rt g o e s th ro u g h th e list o f term s and elim in ates th o s e th a t d o n o t m a k e m u ch s e n s e fo r th e co n te x t o f th e study (th is is a m an u al, lab o r-in ten siv e

p ro c e s s ). • E lim in ate term s w ith very fe w o c c u rre n c e s in v e ry fe w d o cu m en ts. • T ra n sfo rm th e m atrix u sin g sin gu lar v a lu e d eco m p o sitio n .

Singular value decom position (SVD), w h ich is clo s e ly re la ted to p rin cip al c o m ­ p o n en ts analy sis, re d u ce s th e ov erall d im en sion ality o f th e inp u t m atrix (n u m b e r o f inp u t d ocu m en ts b y n u m b e r o f e x tra c te d term s) to a lo w e r d im en sio n al s p a c e , w h e re e a c h co n se cu tiv e d im en sio n re p re sen ts th e larg est d eg re e o f variab ility (b e tw e e n w o rd s and d o cu m en ts) p o ss ib le (M an n in g a n d S ch u tze, 1 9 9 9 ). Id eally, th e analyst m ig h t id entify th e tw o o r th re e m o st s a lie n t d im en sio n s th a t a c c o u n t fo r m o st o f th e variab ility (d iffe re n ce s ) b e tw e e n th e w o rd s an d d o cu m en ts, thu s identifying th e laten t s em an tic s p a c e that orga­ n izes th e w o rd s an d d o cu m en ts in th e analysis. O n c e s u ch d im en sio n s a re id en tified , th e un derlying “m e a n in g ” o f w h a t is co n ta in e d (d iscu ssed o r d e s crib e d ) in t h e d o cu m en ts has b e e n e x tra cted . S p ecifically , a ssu m e that m atrix A re p re sen ts a n m X n te rm o c c u rre n c e m atrix w h e re m is th e n u m b e r o f in p u t d o cu m en ts a n d n is th e n u m b e r o f term s s e le cte d

Chapter 7 • T ex t Analytics, T e x t Mining, and Sentim ent Analysis 341

fo r analysis. T h e SVD co m p u te s th e m X r o rth o g o n al m atrix U , n X r o rth o g o n a l m atrix V, an d r X r m atrix D, s o that A = UDV' a n d r is th e n u m b e r o f e ig e n v a lu e s o f A A.

Task 3: Extract the Knowledge U sin g th e w ell-stru ctu red TD M , an d p o ten tially a u g m en ted w ith o th e r stru ctu red data e le m e n ts, n o v e l pattern s are e x tra cte d in th e c o n te x t o f th e s p e c ific p ro b le m b e in g ad d ressed . T h e m ain ca te g o rie s o f k n o w le d g e e x tr a c tio n m e th o d s are classificatio n , clu s­ terin g , a sso cia tio n , a n d tren d analysis. A sh o rt d e scrip tio n o f th e s e m e th o d s fo llow s.

C L A S S IF IC A T IO N A rguably th e m o st co m m o n k n o w le d g e -d isco v e ry to p ic in analyzing c o m p le x data so u rce s is th e classification (o r ca te g o riz a tio n ) o f c e rta in o b je c ts . T h e task is to classify a g iv en data in sta n ce in to a p re d eterm in e d s e t o f c a te g o rie s (o r c la ss e s ). As it a p p lie s to th e d o m ain o f te x t m ining, th e task is k n o w n as text ca teg o riz a tio n , w h ere fo r a g iv en set o f c a te g o rie s (su b je cts, to p ics , o r c o n c e p ts ) an d a c o lle c tio n o f te x t d o cu m en ts th e g o a l is to find th e co rre ct to p ic (s u b je c t o r c o n c e p t ) fo r e a c h d o cu m e n t u sin g m o d els d e v e lo p e d w ith a training d ata s e t th at in clu d es b o th th e d o cu m en ts an d actu al d o cu m en t cate g o rie s. T o d a y , au to m ate d te x t classificatio n is a p p lie d in a variety o f co n te x ts, inclu d ­ ing au to m atic o r sem iau to m atic (in te ra ctiv e ) in d e x in g o f text, sp a m filtering, W e b p ag e cate g o rizatio n u n d e r h ierarch ical ca ta lo g s, a u to m a tic g e n e ra tio n o f m etad ata, d e te c tio n of

g e n re , an d m an y oth ers. . , T h e tw o m ain a p p ro a ch e s to te x t cla ssifica tio n are k n o w le d g e e n g in e e rin g and

m a ch in e learn in g (F e ld m a n and San g er, 2 0 0 7 ). W ith th e k n o w le d g e -e n g in e e rin g ap p ro a ch , a n e x p e rt’s k n o w le d g e a b o u t th e c a te g o rie s is e n c o d e d in to th e sy stem e ith er d eclaratively o r in th e fo rm o f p ro ced u ra l classificatio n ru les. W ith th e m ach in e-learn in g ap p ro a ch , a g e n eral indu ctive p ro ce s s b u ild s a cla ssifier b y learn in g fro m a s e t o f re clas­ sified e x am p le s. As th e n u m b e r o f d o cu m en ts in cre a se s at a n e x p o n e n tia l rate an d as k n o w le d g e e x p e rts b e c o m e h ard e r to c o m e b y , th e pop u larity trend b e tw e e n th e tw o is sh iftin g to w ard th e m a ch in e -lea rn in g ap p ro ach .

C LU S T E R IN G C l u s t e r i n g is a n u n su p erv ised p r o c e s s w h e re b y o b je c ts a re classified into “natu ral” g ro u p s calle d clusters. C o m p ared to ca te g o riz a tio n , w h e re a c o lle c tio n o f p re ­ classified training e x a m p le s is u s e d to d ev elo p a m o d e l b a se d o n th e d escrip tiv e featu res o f th e cla ss e s in o rd e r to classify a n e w u n la b e le d e x a m p le , in clu sterin g th e p ro b le m is to gro u p a n u n la b e lle d c o lle c tio n o f o b je c ts (e .g ., d o cu m en ts, cu sto m e r co m m e n ts, W e b p a g e s ) into m ean in gfu l clu sters w ith o u t an y p rior k n o w le d g e.

C lustering is u sefu l in a w id e range o f ap p licatio n s, from d o cu m en t retrieval to e n ab lin g b e tte r W e b co n te n t search es. In fact, o n e o f th e pro m in en t ap p licatio n s o f Muster­ ing is th e analysis an d navigation o f very large te x t co lle ctio n s, s u ch as W e b p ag es. T h e b a sic underlying assu m p tio n is that relevan t d o cu m en ts te n d to b e m o re sim ilar to e a c h o th e r than to irrelevant o n e s. I f this assu m p tion h o ld s, th e clusterin g o f d o cu m en ts b a se d o n th e sim ilarity o f th eir c o n te n t im proves s ea rch e ffe ctiv e n ess (F eld m an an d S anger, 2 0 0 7 ):

• Im p r o v e d s e a r c h r e c a ll. Clustering, b e c a u s e it is b a s e d o n ov erall sim ilarity as o p p o s e d to th e p re s e n c e o f a sin gle term , c a n im p ro ve th e re ca ll o f a q u e ry -b ased s e a rch in su ch a w ay that w h e n a q u ery m a tc h e s a d o cu m e n t its w h o le clu ste r is

retu rned . . . • I m p r o v e d s e a r c h p r e c is io n . C lustering c a n a lso im prove s e a rch p re cisio n . As

th e n u m b e r o f d o cu m en ts in a c o lle c tio n gro w s, it b e c o m e s d ifficult to b ro w se th ro u g h th e list o f m a tch e d d o cu m en ts. C lustering c a n h e lp b y gro u p in g th e d o cu ­ m e n ts into a n u m b e r o f m u ch sm alle r g ro u p s o f re la ted d o cu m en ts, o rd erin g th e m b y re le v a n ce , an d retu rning o n ly the d o cu m e n ts fro m th e m o st re le v a n t g ro u p (o r

gro u p s).

3 4 2 Part III • Predictive Analytics

C hapter 7 • T ext Analytics, T ex t Mining, and Sentim ent Analysis 3 4 3

T h e tw o m o st p o p u la r clu sterin g m e th o d s are scatter/gather clu ste rin g an d q u ery- s p e cific clu sterin g:

• Scatter/gather. T h is d o cu m en t b ro w sin g m e th o d u se s clu ste rin g to e n h a n c e th e e ffic ie n c y o f h u m an b ro w sin g o f d o cu m en ts w h e n a s p e c ific s e a r c h q u e ry c a n n o t b e fo rm u lated . In a s e n se , th e m e th o d d ynam ically g e n e r a te s a ta b le o f c o n te n ts fo r th e c o lle c tio n and ad ap ts a n d m o d ifies it in re s p o n s e to th e u s e r se le ctio n .

• Q uery-specific clusterin g. T h is m e th o d e m p lo y s a h ie r a r c h ic a l c lu s te rin g a p p r o a c h w h e r e th e m o st re le v a n t d o c u m e n ts to th e p o s e d q u e r y a p p e a r in sm a ll tig h t clu s te rs th a t a r e n e s te d in la rg e r clu s te rs c o n ta in in g le s s sim ila r d o c ­ u m e n ts , c re a tin g a s p e c tru m o f re le v a n c e le v e ls a m o n g th e d o c u m e n ts . T h is m e th o d p e rfo rm s c o n s is te n tly w e ll fo r d o c u m e n t c o lle c tio n s o f re a lis tic a lly la rg e siz e s.

A S S O C IA T IO N A fo rm al d efin itio n an d d eta iled d e scrip tio n o f asso ciatio n w a s p ro ­ vid ed in th e ch a p te r o n d ata m in in g (C h a p te r 5). A sso ciatio n s, o r a sso c ia tio n ru le lea rn in g in d a ta m ining, is a p o p u la r an d w e ll-re s e a rc h e d te c h n iq u e fo r d isco v e rin g in terestin g re la tio n sh ip s am o n g v a ria b le s in larg e d a ta b a se s. T h e m ain id e a in g e n eratin g a s s o cia tio n ru le s (o r so lv in g m a rk e t-b a s k e t p ro b le m s ) is to id en tify th e fre q u e n t sets that g o to g eth er.

In te x t m ining, a sso cia tio n s s p e cifica lly re fe r to th e d irect re la tio n sh ip s b e tw e e n c o n c e p ts (te rm s) o r sets o f c o n c e p ts. T h e c o n c e p t s e t a s s o cia tio n ru le A => B, relat­ ing tw o fre q u e n t c o n c e p t sets A a n d C, c a n b e q u an tified b y th e tw o b a s ic m e a su re s o f su p p ort a n d co n fid e n c e . In this ca s e , c o n fid e n c e is th e p e rce n ta g e o f d o cu m en ts that in clu d e all th e c o n c e p ts in C w ithin th e sa m e su b se t o f th o s e d o cu m en ts th at in clu d e all th e c o n c e p ts in A. Su p p ort is th e p e rce n ta g e (o r n u m b e r) o f d o cu m en ts th at in clu d e all th e c o n c e p ts in A an d C. F o r in sta n ce, in a d o cu m en t c o lle c tio n th e c o n c e p t “Softw are Im p le m en tatio n F ailu re” m ay a p p e a r m o st o fte n in a s s o cia tio n w ith “E n terp rise R eso u rce P lan n in g ” an d “C u stom er R elationship M an ag em en t” w ith sig n ifican t su p p o rt (4 % ) and c o n fid e n ce (5 5 % ), m e a n in g that 4 p e rce n t o f th e d o cu m en ts h a d all th re e c o n c e p ts re p ­ re se n te d to g e th e r in th e sa m e d o cu m en t an d o f th e d o cu m en ts th at in clu d e d “Softw are Im p le m en tatio n F ailu re,” 55 p e rce n t o f th e m a lso in clu d e d “E n terp rise R e so u rc e P lan n in g ” and “C u sto m er R elation sh ip M an ag em en t.”

T e x t m in in g w ith a s s o c ia tio n ru le s w a s u s e d to a n a ly z e p u b lis h e d lite ratu re m e w s a n d a c a d e m ic a rtic le s p o s te d o n th e W e b ) to c h a rt th e o u tb r e a k an d p ro g re ss o f b ird flu (M a h g o u b e t a l., 2 0 0 8 ). T h e id e a w a s to a u to m a tica lly id e n tify th e a s s o ­ c ia tio n a m o n g th e g e o g r a p h ic a re a s , s p re a d in g a c r o s s s p e c ie s , a n d c o u n te rm e a s u re s (tre a tm e n ts ).

TREN D A N A L Y S I S R e ce n t m eth o d s o f trend an alysis in te x t m in in g h a v e b e e n b a s e d on th e n o tio n th a t th e vario u s ty p e s o f c o n c e p t distributions are fu n ctio n s o f d o cu m en t c o l­ lection s; th a t is, d ifferen t co lle c tio n s lea d to d ifferen t c o n c e p t d istrib u tions fo r th e sam e set o f c o n c e p ts. It is th e re fo re p o ss ib le to co m p a re tw o distributions th a t a re oth erw ise id entical e x c e p t that th ey a re fro m d ifferen t su b co lle ctio n s . O n e n o ta b le d irectio n o f this type o f an a ly se s is h av in g tw o co lle c tio n s fro m th e sam e s o u rc e (s u c h a s fro m th e sam e set o f a c a d e m ic jo u rn a ls) b u t from d ifferen t p o in ts in tim e. D e le n and C rosslan d (2 0 0 8 ) ap p lied tre n d analysis to a larg e n u m b e r o f a ca d e m ic articles (p u b lish e d in th e th ree h igh est-rated a ca d e m ic jo u rn a ls) to identify th e e v o lu tio n o f k e y c o n c e p ts in th e field o f inform ation system s.

As d e s c rib e d in this s e ctio n , a n u m b e r o f m e th o d s a re a v ailab le fo r te x t m ining. A p plication C a se 7 .5 d e s crib e s th e u s e o f a n u m b e r o f d iffe ren t te c h n iq u e s in an aly zin g a large s e t o f literature.

3 4 4 Part III • Predictive Analytics

Application Case 7.5 Research Literatu re S u rvey w ith Text M ining R e se a rc h e rs co n d u ctin g s e a r c h e s a n d rev iew s o f re l­ e v a n t literatu re fa c e a n in creasin g ly c o m p le x an d v o lu m in o u s task. In e x te n d in g th e b o d y o f relevan t k n o w le d g e , it h a s alw ays b e e n im portant to w o rk h ard to g a th e r, o rg an ize , an aly ze, a n d assim ilate e x is tin g in fo rm a tio n fro m th e literature, particularly fro m o n e ’s h o m e d iscip lin e. W ith th e in creasin g a b u n d a n c e o f p o ten tially sig n ifican t re s e a r c h b ein g re p o rte d in re la ted field s, and e v e n in w h a t are tra­ d ition ally d e e m e d to b e n o n rela te d field s o f study, th e re s e a r c h e r’s ta sk is e v e r m o re d aunting, i f a th o r­ o u g h jo b is d esired.

In n e w stream s o f re sea rch , th e re se a rch e r’s ta sk m ay b e e v e n m o re te d io u s a n d co m p le x . Trying to ferret o u t relev an t w o rk th at o th ers h av e rep o rted m ay b e difficu lt, at b e s t, an d p erh ap s e v e n n e a r im p o ssib le if trad itional, larg ely m anu al review s o f p u b lis h e d literatu re a re requ ired . E v e n w ith a leg io n o f d ed icate d grad u ate stu d ents o r h elp fu l c o l­ lea g u es, trying to c o v e r all p o ten tially relev an t p u b ­ lish ed w o rk is p ro b lem atic.

M any sch o larly c o n fe r e n c e s ta k e p la c e e v ery y e a r. In ad d itio n to e x te n d in g th e b o d y o f k n o w l­ e d g e o f th e cu rre n t fo cu s o f a c o n fe r e n c e , o rg an iz­ ers o fte n d esire to o ffe r ad d itional m in i-tracks and w o rk sh o p s. In m an y c a s e s , th e s e ad d itional e v en ts a re in te n d e d to in tro d u ce th e a tte n d e es to signifi­ c a n t stream s o f re se a rch in re la ted field s o f study and to try to identify th e “n e x t b ig th in g” in term s o f re se a rch in te rests a n d fo cu s. Id entify ing re a so n a b le ca n d id a te to p ic s fo r s u ch m in i-track s a n d w o rk sh o p s is o fte n s u b je ctiv e ra th er th an d eriv ed o b je ctiv e ly fro m th e e x istin g a n d e m e rg in g re search .

I n a r e c e n t stud y, D e le n a n d C rosslan d ( 2 0 0 8 ) p ro p o s e d a m e th o d to greatly assist a n d e n h a n c e th e e ffo rts o f th e re s e a r c h e rs b y e n a b lin g a sem i­ a u to m a te d an alysis o f la rg e v o lu m e s o f p u b lish e d literatu re th ro u g h th e ap p lica tio n o f te x t m ining. U sin g stan d ard d igital lib raries an d o n lin e p u b lica ­ tio n s e a r c h e n g in e s , th e a u th o rs d o w n lo a d e d an d c o lle c te d all o f th e a v ailab le articles fo r th e th ree m ajo r jo u rn a ls in th e field o f m an a g e m e n t in fo rm a ­ tio n sy stem s: MIS Q u arterly (M IS Q ), In fo rm a tio n System s R esearch (IS R ), a n d th e J o u r n a l o f M an ag em en t In fo rm a tio n System s (JM IS ). In ord er to m ain tain th e sa m e tim e interval fo r all th ree

jo u rn a ls (f o r p o te n tia l co m p a ra tiv e longitu d inal stu d ies), th e jo u rn a l w ith th e m o st re c e n t starting d ate fo r its d ig ital p u b lica tio n av ailability w a s u se d a s th e start tim e fo r this stud y ( i.e ., JM IS articles h av e b e e n d igitally a v a ilab le sin c e 1 9 9 4 ). F o r e a c h a rticle, th e y e x tr a c te d th e title, ab stra ct, a u th o r list, p u b lis h e d k e y w o rd s, v o lu m e , issu e n u m b e r, and y e a r o f p u b lica tio n . T h e y th e n lo a d e d all o f th e arti­ c le d ata in to a sim p le d a ta b a se file. A lso in clu d e d in th e c o m b in e d d ata s e t w as a field th at d esig n ate d th e jo u rn a l ty p e o f e a c h article fo r lik e ly d iscrim i­ n a to ry analy sis. Editorial n o te s , re s e a rch n o te s , an d e x e c u tiv e o v erv iew s w e re om itted fro m th e c o lle c ­ tion. T a b le 7 .2 sh o w s h o w th e d ata w as p re s e n te d in a ta b u la r form at.

In th e a n a ly s is p h a s e , th e y c h o s e to u s e o n ly th e a b stra c t o f a n a r tic le as th e s o u r c e o f in fo r­ m a tio n e x tr a c tio n . T h e y c h o s e n o t to in c lu d e th e k e y w o r d s lis te d w ith th e p u b lic a tio n s fo r tw o m a in re a s o n s : ( 1 ) u n d e r n o rm a l c irc u m s ta n c e s , th e a b stra c t w o u ld a lre a d y in c lu d e th e lis te d k e y ­ w o rd s , a n d th e r e fo r e in c lu s io n o f th e lis te d k e y ­ w o rd s fo r th e a n a ly sis w o u ld m e a n re p e a tin g th e s a m e in fo r m a tio n a n d p o te n tia lly g iv in g th e m u n m e rite d w e ig h t; a n d ( 2 ) th e liste d k e y w o rd s m ay b e te rm s th a t a u th o rs w o u ld lik e th e ii article to b e a s s o c ia te d w ith (a s o p p o s e d to w h a t is re a lly c o n ta in e d in th e a r tic le ), th e r e fo r e p o te n tia lly in tro d u c in g u n q u a n tifia b le b ia s to th e a n a ly sis o f th e c o n te n t.

T h e first e x p lo ra to ry stud y w as to lo o k at th e lo n g itu d in al p e rsp e ctiv e o f th e th ree jo u rn als ( i.e ., e v o lu tio n o f re s e a r c h to p ic s o v e r tim e ). In o rd e r to c o n d u c t a lon g itu d in al study, th e y d ivid ed th e 12-y e ar p e rio d (fro m 199 4 to 2 0 0 5 ) in to four 3 -y e a r p e rio d s fo r e a c h o f th e th re e jo u rn als. T h is fram ew o rk le d to 12 te x t m in in g e x p e rim e n ts w ith 12 m utually e x c lu s iv e d ata sets. At this p o in t, fo r e a c h o f th e 1 2 d ata sets th e y u s e d te x t m in in g to ex tra ct th e m o st d escrip tiv e te rm s fro m th e s e c o l­ le ctio n s o f article s re p re s e n te d b y th e ir abstracts. T h e results w e r e tab u lated an d e x a m in e d fo r tim e- v arying c h a n g e s in th e term s p u b lis h e d in th e s e th r e e jo u rn als.

As a s e c o n d e x p lo r a tio n , u s in g th e c o m p le te d ata s e t (in c lu d in g all th re e jo u rn a ls a n d all fo u r

Chapter 7 • T e x t Analytics, T e x t Mining, and Sentim ent Analysis 345

TABLE 7.2 Tabular R epresentation o f th e Fields Included in th e Com bined Data Set

Jo u rn al Y e a r A u thor(s) Title Vol/No Pages K eyw o rd s Abstract

A. Malhotra, Absorptive 29/1 145-187 knowledge The need for 5. Gossain, capacity management continual value and 0. A. configurations supply chain innovation is El Sawy in supply chains: absorptive capacity driving supply

Gearing for interorganizational chains to evolve partner-enabled information from a pure market systems transactional focus knowledge configuration to leveraging creation approaches interorganization

partnerships for sharing

D. Robey Accounting 165-185 organizational Although much and M . C. for the transformation contemporary Boudtreau contradictory impacts of thought considers

organizational technology advanced consequences organization information of information theory research technologies technology: methodology as either Theoretical intraorganizational determinants directions and power electronic or enablers methodological communication of radical implications misimplementation

culture systems organizational change, empirical studies have revealed inconsis­ tent findings to support the deterministic logic implicit in such arguments. This paper reviews the contradictory...

R. Aron and Achieving 65-88 information W hen producers E. K. Clemons the optimal products of goods

balance Internet (or services) are between advertising confronted by a investment product situation in in quality positioning which their and invest­ signaling offerings no ment in self­ signaling games longer perfectly promotion for match consumer information preferences, they products must determine

the extent to which the advertised features of...

( C ontinued)

3 4 6 Part III • Predictive Analytics

Application Case 7.5 (Continued)

p e rio d s), th e y co n d u c te d a clu ste rin g analysis. C lu sterin g is a rg u a b ly th e m o st co m m o n ly u s e d te x t m in in g te c h n iq u e . C lu sterin g w a s u s e d in this stud y to id en tify th e n atu ral g ro u p in g s o f th e arti­ c le s (b y p u ttin g th e m in to s e p a ra te c lu s te rs ) an d th e n to lis t th e m o st d e scrip tiv e te rm s th a t ch a r­ a c te riz e d th o s e clu ste rs. T h e y u s e d sin g u lar v a lu e d e c o m p o s itio n to re d u c e th e d im en sio n a lity o f th e te rm -b y -d o c u m e n t m atrix a n d th e n a n e x p e c ta ­ tio n -m a x im iz a tio n a lg o rith m to c re a te th e clu sters. T h e y c o n d u c te d s e v e r a l e x p e rim e n ts to id en tify th e o p tim a l n u m b e r o f clu ste rs, w h ic h tu rn e d o u t to b e n in e . A fter th e co n s tr u c tio n o f th e n in e c lu s ­ te rs. th e y a n a ly z ed th e c o n te n t o f th o se clu sters fro m tw o p e rs p e c tiv e s : ( 1 ) re p re s e n ta tio n o f th e jo u rn a l ty p e (s e e F ig u re 7 .8 ) a n d ( 2 ) re p re s e n ta ­ tio n o f tim e . T h e id e a w a s to e x p lo r e th e p o ten tia l d iffe r e n c e s and/or c o m m o n a litie s a m o n g th e th ree

jo u rn a ls a n d p o te n tia l ch a n g e s in th e e m p h a s is o n th o s e clu ste rs; th a t is, to a n s w e r q u e s tio n s s u c h as “A re th e re clu s te rs that r e p r e s e n t d iffe re n t re s e a r c h th e m e s s p e c ific to a sin g le jou rn al? an d Is th e re a tim e -v a ry in g c h a ra cte riz a tio n o f th o s e clusters? T h e y d is c o v e re d a n d d is cu ss e d se v e ra l in terestin g p attern s u sin g ta b u la r a n d g ra p h ica l re p re s e n ta tio n o f th e ir fin d in g s (fo r fu rth e r in fo rm a tio n s e e D e le n a n d C ro sslan d , 2 0 0 8 ).

Q u e s t i o n s f o r D i s c u s s i o n

1. H o w c a n te x t m in in g b e u s e d to e a s e th e task o f literature review ?

2. W h at are th e co m m o n o u tc o m e s o f a te x t m ining p ro je ct o n a s p e cific c o lle c tio n o f jo u rn al articles? C an y o u th in k o f o th e r p o ten tia l o u tc o m e s n o t m e n tio n e d in this case?

1

F i r U R E 7 8 D istrib u tio n o f th e N u m b er o f A rtic le s f o r th e T h re e Jo u rn a ls o v e r th e N ine Clu sters. Source: D. Delen and “ and, “ th e s u te y and Anaiysis of S e a r c h Literature w ith T e x t M in in g ,” Expert Systems « , t h A p r o n s ,

Vol. 34, No. 3, 2008, pp. 1707-1720.

C hapter 7 • T ex t Analytics, T e x t Mining, and Sentim ent Analysis 3 4 7

SECTION 7.5 REVIEW QUESTIONS

1 . W h at are th e m ain step s in th e te x t m in in g process?

2 . W h a t is th e re a s o n for n orm alizin g w o rd fre q u e n cies? W hat are th e c o m m o n m eth o d s fo r n orm alizin g w o rd freq u en cies?

3 . W h a t is sin gu lar v a lu e d eco m p o sitio n ? H ow is it u se d in te x t mining? 4 . W h at are th e m ain k n o w le d g e e x tra ctio n m e th o d s fro m corpus?

7.6 TEXT M IN IN G TOOLS As th e v a lu e o f te x t m in in g is b e in g realized b y m o re an d m o re organ izatio n s, th e n u m ­ b e r o f softw are to o ls o ffe re d b y softw are c o m p a n ie s an d n o n p ro fits is a ls o increasing. Fo llo w in g are s o m e o f th e p o p u la r te x t m ining to o ls, w h ic h w e classify a s co m m ercial softw are to o ls a n d fre e (and/or o p e n s o u rc e ) so ftw are to ols.

Commercial So ftw are Tools T h e fo llo w in g a re so m e o f th e m o st p o p u lar so ftw are to o ls u se d fo r te x t m ining. N ote that m an y c o m p a n ie s o ffe r d em o n stratio n v e rsio n s o f th eir p ro d u cts o n th e ir W e b sites.

1 . C learF o rest o ffers te x t an aly sis an d visu alization to ols. 2 . IB M o ffe rs SPSS M o d e ler an d d ata and te x t analytics toolkits. 3 . M eg ap u ter T e x t Analyst offers sem an tic analysis o f fre e -fo m i te xt, sum m arization,

clustering, n av igation, a n d n a airal lan g u ag e retrieval w ith sea rch d y n am ic refocu sing. 4 . SAS T e x t M in e r p ro v id e s a rich suite o f t e x t p ro ce ssin g an d analysis to o ls . 5. KXEN T e x t C o d er (K T C ) o ffers a te x t an alytics so lu tio n fo r au to m atically p rep aring

and tran sform in g unstructured te x t attributes in to a stru ctured re p re se n ta tio n fo r use in KXEN A n alytic Fram ew ork.

6. T h e Statistica T e x t M ining e n g in e p ro v id es e a sy -to -u se te x t m in in g fu n ction ality w ith e x c e p tio n a l visu alization cap ab ilities.

7. V a n ta g eP o in t p ro v id es a variety o f in teractiv e g rap hical v iew s an d an alysis to o ls w ith p o w e rfu l cap ab ilitie s to d isco v e r k n o w le d g e fro m te x t d atabases.

8 . T h e W ordStat analysis m o d u le fro m P rovalis R e se a rch an alyzes te x tu a l in form ation su ch as r e s p o n s e s to o p e n -e n d e d q u e stio n s, interview s, etc.

9. C larab rid ge te x t m in in g so ftw are p ro v id e s e n d -to -e n d so lu tio n s fo r cu sto m e r e x p e ri­ e n c e p ro fe ssio n a ls w ish in g to tran sform cu sto m e r fe e d b a c k fo r m ark etin g , serv ice, an d p ro d u ct im provem en ts.

Free So ftw are Tools F ree so ftw are to o ls , s o m e o f w h ic h are o p e n so u rce , are av ailab le fro m a n u m b e r o f n o n ­ profit o rgan ization s:

1 . RapidM iner, o n e o f th e m o st p o p u la r fre e , o p e n s o u rc e so ftw are to o ls fo r d ata m in­ ing an d te x t m ining, is tailo red w ith a g rap h ically a p p e alin g , d rag-an d -d ro p u ser in terface.

2 . O p e n C alais is an o p e n so u rce to o lk it fo r in clu d in g s em an tic fu n ctio n ality w ithin y o u r b lo g , c o n te n t m a n a g e m e n t system , W e b site, o r ap p licatio n .

3 . G A TE is a lead in g o p e n s o u rc e to o lk it fo r text m ining. It h a s a fre e o p e n so u rce fram ew o rk (o r SD K ) an d grap h ical d e v e lo p m e n t en viron m en t.

4 . LingP ipe is a su ite o f Ja v a lib raries fo r th e lingu istic analysis o f h u m an lan g u ag e. 5. S-EM (Sp y-E M ) is a te x t classificatio n sy stem that learn s fro m positive a n d u n lab ele d

e x am p le s. 6. V ivisim o/Clusty is a W e b s e a rch an d te x t-clu ste rin g en gin e.

O fte n , in n ov ativ e a p p lica tio n o f te x t m in in g c o m e s fro m th e co lle ctiv e u s e o f sev eral so ftw are to o ls. A p p lication C ase 7 .6 illustrates a fe w cu sto m e r c a s e study s y n o p se s w h e re te x t m in in g an d a d v a n ce d an alytics are u s e d to a d d ress a variety o f b u sin e ss ch a lle n g e s.

3 4 8 Part III • Predictive Analytics

Application Case 7.6 A Po tpo urri o f Text M ining Case Synopses

1 . A l b e r t a ’s P a r k s D iv is io n g a in s in s ig h t f r o m u n s t r u c t u r e d d a ta

Business Issue: A lb erta’s P ark s D iv isio n w as relyin g o n m an u al p ro ­ c e s s e s t o re s p o n d to s ta k e h o ld e rs , w h ic h w a s tim e - co n s u m in g a n d m a d e it d ifficu lt to g le a n insight fro m u n stru ctu red d ata s o u rc e s.

Solution : Using SAS T e x t M iner, the Parks D ivision is a b le to re d u ce a th ree -w ee k p ro cess d ow n to a co u p le o f days, and d iscov er n e w insights in a m atter o f m inutes.

B e n e f i t s :

T h e solu tion has n o t o n ly autom ated m anual tasks, but also provides insight into b o th structured and unstruc­ tured d ata sou rces that w a s previously n o t possible.

“W e n o w have op p ortu nities to ch a n n e l cu s­ to m e r co m m u n icatio n s into p ro d u cts and services th at m e e t th e ir n e ed s. H aving th e analytics w ill e n a b le us to b e tte r su p p ort ch a n g e s in p ro g ram delivery, said R o y Finzel, M anager o f B u sin ess In tegration and A nalysis, A lberta T ou rism , P arks and R ecreation .

F o r m ore d etails, p le a s e g o to http://w w w .sas. com /su ccess/albe? ia-p arks2 0 1 2 .h tm l

2 . A m erican H onda Saves M illions b y U sing T e x t an d D ata M ining

Business Issue: O n e o f th e m o st a d m ired a n d r e c o g n iz e d a u to m o ­ b ile b ra n d s in th e U n ite d S ta te s , A m e rica n H on d a w a n te d to d e te c t a n d c o n ta in w a rra n ty a n d ca ll

: c e n te r is s u e s b e fo r e th e y b e c o m e w id e s p re a d .

S olution : SAS T e x t M iner help s Am erican H onda spot patterns in a w id e range o f data and text to pinpoint problem s early, ensuring safety, quality, and custom er satisfaction.

Benefits: “SAS is h e lp in g us m ak e d isco v eries s o that w e can ad d ress th e co re issu es b e fo re th e y e v e r b e c o m e

p ro b lem s— an d w e c a n m ak e sure that w e are ad dressing th e right cau ses. W e ’re talk in g a b o u t hun­ dred s o f m illion s o f d ollars in savings,” said T racy C erm ack, P ro je ct M anager in th e Serv ice E n gin eerin g In fo rm ation D ep artm en t, A m erican H on da M otor Co. F o r m ore d etails, p le a s e g o to h ttp://w w w .sas.com / su ccess/h o n d a .htm l

3 . M a s p e x W a d o w ic e G ro u p A n a ly z e s O n lin e B r a n d I m a g e w ith T e x t M in in g

Business Issue: M asp ex W ad o w ice G roup, a d om inant p lay er am o n g fo o d an d b e v e ra g e m anufacturers in Central and E astern E u ro p e , w an ted t o analyze so cial m ed ia ch a n ­ n e ls to m o n ito r a p rodu ct’s brand im age an d s e e h ow it co m p ares w ith its g e n eral p e rce p tio n in th e m arket.

Solution: M a sp e x W ad o w ice G ro u p c h o o s e to u s e SAS T e x t M iner, w h ic h is a part o f t h e SAS B u s in e s s A nalytics cap ab ilitie s, t o tap into s o cia l m ed ia d ata so rce s.

Benefits: M a sp e x g a in ed a com p etitiv e ad van tage throu gh b e tte r c o n s u m e r insights, resultin g in m o re e ffe ctiv e an d e fficie n t m arketin g efforts.

“T h is w ill a llo w u s to p lan an d im p lem en t o u r m a rk e tin g and co m m u n ica tio n s activities m o re effe ctiv e ly , in particu lar th o se u sin g a W e b -b a s e d ch a n n e l,” sa id M arcin L esn iak, R e se arch M anager, M asp e x W ad o w ice G roup.

F o r m ore d etails, p le a s e g o to http://w w w .sas. com /su ccess/m asp ex -w ad ow ice.h tm l

4 . V iseca C ard Services R ed uces Fraud Loss w ith T e x t A nalytics

Business Issue: Sw itzerland ’s larg est cre d it ca rd co m p a n y aim ed to p re v e n t lo s s e s b y d etectin g a n d p re v e n tin g fraud o n V ise ca Card Se rv ices’ 1 m illion cre d it card s an d m o re th an 1 0 0 ,0 0 0 d aily tran saction s.

Chapter 7 • T e x t Analytics, T ex t Mining, and Sentim ent Analysis 3 4 9

Solution:

T h e y c h o o s e t o u s e a su ite o f an alytics to o ls fro m SAS in clu d in g SA S® E n terp rise M iner™ , SA S® E nterp rise G u id e ® , SAS T e x t M iner, an d SAS B I Server.

Benefits:

E ig h ty -o n e p e rc e n t o f all fraud c a s e s are fo u n d w ithin a d ay, and to tal fraud lo ss h a s b e e n re d u ce d b y 15 p e rc e n t. E v e n as th e n u m b e r o f frau d ca s e s a cro ss th e industry has d o u b led , V ise ca Card Se rv ices h a s re d u ce d loss p e r frau d c a s e b y 4 0 p ercen t.

“T h an k s to SAS Analytics o u r total fraud loss has b e e n re d u ce d b y 15 p ercen t. W e have o n e o f th e b e s t fraud p re v e n tio n ratings in Sw itzerland a n d ou r busi­ n e ss c a s e fo r fraud p rev en tio n is straightforward: O ur returns a re sim ply m o re than o u r investm ent,” said M arcel B ie le r, B u sin ess Analyst, V iseca Card Services.

F o r m ore d etails, p le a s e g o to http://w w w .sas. com /su ccess/V isecacard sv cs.h tm l

5 . I m p r o v i n g Q u a lity w ith T e x t M in in g a n d A d v a n c e d A n a ly tic s

Business Issue:

W h irlp o o l C orp., th e w o rld ’s lead in g m an u factu rer and m a rk e te r o f m a jo r h o m e a p p lia n ces, w a n te d to red u ce s e rv ic e calls b y find ing d e fe cts th ro u g h w ar­ ranty an aly sis an d co rre ctin g th e m quickly.

Solution:

SAS W arran ty A nalysis and early-w arn in g to o ls o n th e SAS E n terp rise B I Se rv e r distill and analyze

w arranty cla im s data to q u ick ly d e te c t p ro d u ct issu es. T h e to o ls u se d in this p ro je c t in clu d e d SAS E n terp rise B I Server, SAS W arran ty Analysis, SAS E n terp rise G u id e , an d SAS T e x t Miner.

Benefits: W h irlp o o l C orp. aim s to cu t ov erall c o s t o f quality, an d SAS is p lay in g a significant part in th at o b je c ­ tive. E x p e cta tio n s o f th e SAS W arran ty A nalysis so lu ­ tio n in clu d e a sig n ifican t red u ctio n in W h irlp o o l’s issu e d e te c tio n -to -c o rre c tio n c y c le , a th ree -m o n th d e c r e a s e in initial issu e d e te ctio n , an d a p o ten tial to cu t ov erall w arran ty e x p e n d itu re s w ith sign ifican t quality, pro d u ctiv ity an d e ffic ie n c y gains.

“SAS b rin g s a le v e l o f an aly tics to b u sin ess in te llig e n ce that n o o n e e ls e m a tc h e s ,” said J o h n K err, G e n e ra l M an ag er o f Q u ality a n d O p e ratio n al E x c e lle n c e , W h irlp o o l Corp.

F or m ore d etails, p le a s e g o to http://w w w .sas. com /su ccess/w h irlp ool.h tm l

Q u e s t i o n s f o r D i s c u s s i o n

1. W h a t d o y o u th in k are th e co m m o n ch aracteris­ tics o f th e k in d o f ch a lle n g e s th e s e fiv e co m p a ­ n ie s w e re facing?

2. W h a t a re th e ty p e s o f so lu tio n m e th o d s a n d to o ls p ro p o s e d in th e s e c a s e synopses?

3. W h a t d o y o u th in k are th e k e y b e n e fits o f using te x t m in in g an d ad v an ce d an aly tics (co m p a re d to th e trad itional w a y to d o th e sam e)?

Sources: SAS, w w w .s a s .c o m / s u c c e s s / (accessed September 2013).

SECTION 7 - 6 REVIEW QUESTIONS

1 . W h at a re s o m e o f th e m o st p o p u la r te x t m in in g so ftw are tools?

2 . W h y d o y o u th in k m o st o f th e te x t m in in g to o ls are o ffe re d b y statistics com p an ies? 3 . W h at d o y o u th in k are th e p ro s an d c o n s o f ch o o s in g a fre e te x t m in in g to o l o v e r a

co m m e rcia l tool?

7.7 SENTIM ENT A N A L Y S IS OVERVIEW W e, h u m a n s , a r e s o c ia l b e in g s . W e a re a d e p t a t u tiliz in g a v a r ie ty o f m e a n s to :o m m u n ic a te . W e o f te n c o n s u lt fin a n c ia l d is c u s s io n fo ru m s b e f o r e m a k in g a n in v e s tm e n t d e c is io n ; a s k o u r frie n d s fo r th e ir o p in io n s o n a n e w ly o p e n e d re sta u ra n t d t a n e w ly r e le a s e d m o v ie ; a n d c o n d u c t I n te r n e t s e a r c h e s an d re a d c o n s u m e r re v ie w s m d e x p e r t re p o r ts b e f o r e m a k in g a b ig p u r c h a s e lik e a h o u s e , a c a r , o r a n a p p li­ a n ce . W e r e ly o n o t h e r s ’ o p in io n s to m a k e b e tte r d e c is io n s , e s p e c ia lly in a n a re a

3 5 0 Part III • Predictive Analytics

w h e r e w e d o n ’t h a v e a lo t o f k n o w le d g e o r e x p e r ie n c e . T h a n k s to th e g ro w in g a v a ila b ility a n d p o p u la rity o f o p in io n - r ic h I n te r n e t r e s o u r c e s s u c h a s s o c ia l m ed ia o u tle ts ( e .g ., T w itte r, F a c e b o o k , e t c .) , o n lin e re v ie w s ite s , a n d p e r s o n a l b lo g s , it is n o w e a s ie r th a n e v e r to fin d o p in io n s o f o t h e r s (th o u s a n d s o f th e m , a s a m atter o f fa c t) o n e v e ry th in g fro m th e la te s t g a d g e ts to p o litic a l a n d p u b lic fig u r e s . E ven th o u g h n o t e v e ry b o d y e x p r e s s e s o p in io n s o v e r th e I n te rn e t, d u e m o stly to th e fast- g ro w in g n u m b e rs a n d c a p a b ilitie s o f s o c ia l c o m m u n ic a tio n c h a n n e ls , th e n u m b e rs are in c r e a s in g e x p o n e n tia lly .

Sentim ent is a d ifficu lt w o rd to d efin e . It is o fte n lin k e d to o r co n fu s e d w ith o th er term s lik e b elief, view, op in ion , a n d con v iction . S e n tim en t su gg ests a s ettle d op inion reflectiv e o f o n e ’s fe e lin g s (M ejova, 2 0 0 9 ). S e n tim en t h a s so m e u n iq u e p ro p erties that s e t it ap art fro m o th e r c o n c e p ts th at w e m ay w a n t to id entify in te xt. O fte n w e w an t to ca te g o riz e te x t b y to p ic, w h ic h m ay in v o lv e d ea lin g w ith w h o le tax o n o m ie s o f topics. S en tim en t classificatio n , o n th e o th e r h an d , u su ally d eals w ith tw o cla sse s (p o sitiv e versus n e g a tiv e ), a rang e o f polarity (e .g ., star ratings fo r m o v ie s ), o r e v e n a ran g e in strength o f o p in io n (P an g a n d L ee, 2 0 0 8 ). T h e s e cla ss e s s p a n m a n y to p ics, u se rs, an d d o cu m en ts. A lthough d ealin g w ith o n ly a fe w cla ss e s m ay s e e m lik e a n e a s ie r ta sk th a n standard text analysis, it is far fro m th e truth.

As a field o f re sea rch , sen tim en t an alysis is clo se ly related to com p u tatio n al linguis­ tics, natu ral lan g u ag e p ro cessin g , an d te x t m ining . S en tim en t an alysis h a s m an y nam es. It’s o fte n re fe rre d to as o p in io n m ining, su bjectiv ity an aly sis, and a p p ra isa l ex traction . w ith s o m e c o n n e c tio n s to affectiv e co m p u tin g (c o m p u te r re co g n itio n an d e x p re s s io n o f e m o tio n ). T h e su d d en u p su rg e o f in terest an d activity in th e a re a o f s en tim en t analysis (i.e ., o p in io n m in in g), w h ich d eals w ith th e a u to m a tic e x tra ctio n o f o p in io n s, feelings, a n d su b jectiv ity in text, is creatin g o p p o rtu n itie s a n d th reats fo r b u sin e sse s and individu­ als alik e. T h e o n e s w h o e m b r a c e a n d ta k e a d v an tag e o f it will greatly b e n e fit fro m it. Every o p in io n p u t o n th e In te rn e t b y a n individ ual o r a co m p a n y w ill b e a ccre d ite d to the orig in ato r (g o o d o r b a d ) an d w ill b e retriev ed an d m ined b y o th ers (o fte n autom atically

b y co m p u te r p ro gram s). S e n tim e n t a n aly sis is trying to a n s w e r th e q u e s tio n “W h a t d o p e o p le fe e l a b o u t

a c e rta in to p ic?” b y d ig g in g in to o p in io n s o f m a n y u sin g a v arie ty o f a u to m a te d to ols. B rin g in g to g e th e r r e s e a r c h e rs a n d p ra c titio n e rs in b u s in e s s , co m p u te r s c ie n c e , c o m ­ p u ta tio n a l lin g u istics, d ata m in in g , te x t m in in g , p s y c h o lo g y , an d e v e n s o c io lo g y , s e n ­ tim e n t a n aly sis aim s to e x p a n d trad itio n al fa c t-b a s e d te x t a n a ly sis to n e w fro n tiers, to re a liz e o p in io n -o rie n te d in fo rm a tio n sy stem s. I n a b u s in e s s s e ttin g , e s p e c ia lly in m ark e tin g and cu sto m e r re la tio n sh ip m a n a g e m e n t, s e n tim e n t a n aly sis s e e k s to d e te c t fa v o ra b le an d u n fa v o ra b le o p in io n s to w a rd s p e c ific p ro d u cts and / or s e r v ic e s using la rg e n u m b e rs o f te x tu a l d ata s o u r c e s (c u s to m e r f e e d b a c k in th e fo rm o f W e b p o stin g s, tw e e ts , b lo g s , e tc .).

Sen tim en t th at a p p e a rs in te x t co m e s in tw o flavors: e xp licit, w h e re th e su b jectiv e s e n te n c e directly e x p re s s e s a n o p in io n ( “It’s a w o nd erfu l d a y ’), an d im plicit, w h e re the te x t im p lies a n o p in io n ( “T h e h an d le b re a k s to o e asily ”). M ost o f th e earlie i w o rk d on e in s en tim en t analysis fo cu se d o n th e first kin d o f s en tim en t, sin c e it w as e a s ie r to analyze. Current trend s are to im p lem en t analytical m e th o d s to c o n s id e r b o th im plicit a n d e x p licit sen tim en ts. Sentim en t polarity is a particu lar fe a tu re o f te x t th at sen tim en t analysis prim ar­ ily fo cu se s on . It is u su ally d ich o to m iz e d in to tw o— p o sitiv e an d negative— b u t polarity c a n a lso b e th o u g h t o f as a range. A d o cu m en t co n tain in g sev eral o p in io n a te d statem ents w o u ld h av e a m ix e d polarity ov erall, w h ic h is d ifferen t fro m n o t h av in g a polarity at all

(b e in g o b je c tiv e ) (M ejova, 2 0 0 9 ). T im e ly c o lle c tio n a n d analysis o f te xtu al d ata, w h ic h m ay b e c o m in g fro m a variety

o f so u rce s— ran gin g fro m cu sto m e r call c e n te r tran scrip ts to so cia l m ed ia p o stin g s is a cru cial p art o f th e cap ab ilitie s o f p ro a ctiv e a n d cu sto m e r-fo cu se d co m p a n ie s , n ow ad ays.

Chapter 7 • T ex t Analytics, T e x t Mining, and Sentim ent Analysis 351

REAL l\m SOCIAL SIGNAL RTTEflflTV

jbud. m m ..........

°mm vmmm 0m J M W i i

FIGURE 7.9 A Sample Social Media Dashboard for Continuous Brand Analysis Source: Attensity.

T h e s e real-tim e a n aly se s o f textu al d ata a re o fte n visu alized in easy -to -u n d erstan d d ash bo ard s. A ttensity is o n e o f th o se co m p a n ie s that pro v id e s u ch e n d -to -e n d so lu tion s to c o m p a n ie s ’ te x t an alytics n e e d s (Figure 7 .9 sh o w s a n e x a m p le so cia l m e d ia analytics d ash b o ard c re a te d b y A ttensity). A p p lication C ase 7 .7 p ro v id es a n A ttensity’s cu sto m e r s u cce s s story, w h e re a larg e c o n s u m e r p ro d u ct m an u factu rer u s e d te x t an alytics and s en tim en t a n aly sis to b e tte r c o n n e c t w ith th e ir cu stom ers.

Application Case 7.7 W h irlp o o l A ch ieves Custom er Lo ya lty and Product Success w ith Text A n alytics

B a c k g r o u n d

Every d ay, a su b stantial am o u n t o f n e w cu sto m e r fe e d b a c k d ata— rich in sen tim en t, cu sto m e r issu es, a n d p ro d u ct insights— b e c o m e s av ailab le to orga­ n izatio n s th ro u g h e-m ails, rep air n o te s , CRM n otes, an d o n lin e in s o cia l m ed ia. W ithin that data exists a w e a lth o f in sig h t in to h o w cu sto m ers fe e l ab ou t p ro d u cts, serv ice s, b ran d s, an d m u ch m o re . That data a lso h o ld s in form ation a b o u t p o ten tial issu es that c o u ld e a sily im p act a p ro d u ct’s lon g -term

su c c e s s an d a c o m p a n y ’s b o tto m line. T h is data is in v a lu ab le to m ark etin g , p ro d u ct, and serv ice m a n ­ a g ers a c ro ss e v ery industry.

Attensity, a p re m ier te x t an alytics so lu tio n p ro v id er, c o m b in e s th e co m p a n y ’s rich te x t analyt­ ics ap p licatio n s w ith in cu sto m e r-sp e cific B I plat­ form s. T h e result is a n intuitive so lu tio n th at e n a b le s cu sto m e rs to fully lev erag e critical data a ssets to d isco v e r in v alu ab le b u sin e ss insight an d to fo ster b e tte r an d faster d e c is io n m aking.

( Continued)

3 5 2 Part III • Predictive Analytics

Application Case 7.7 (Continued) W h irlp o o l is th e w o rld ’s lead ing m anu factu rer

and m ark eter o f m ajor h o m e ap p lian ces, w ith annual sales o f ap p roxim ately $ 1 9 b illion, 6 7 ,0 0 0 em p loy ees, an d nearly 7 0 m anufacturing and te ch n o lo g y research ce n te rs aro u n d th e w orld. W hirlp ool recog n izes that co n su m e rs lead busy, active lives, an d co n tin u es to cre a te so lu tion s that h e lp consu m ers optim ize p ro­ ductivity an d e fficie n cy in the h o m e. In ad dition to designing ap p lian ce solutions b a se d o n con su m er insight, W h irlp o o l’s brand is d ed icated to creating EN ERG Y ST A R -qu alified a p p lian ces like th e R eso u rce Saver sid e-b y -sid e refrigerator, w h ich recen tly w as rated th e #1 b ra n d fo r sid e-by-sid e refrigerators.

B u s i n e s s C h a lle n g e

C u stom er satisfactio n an d fe e d b a c k are a t th e c e n ­ te r o f h o w W h irlp o o l drives its o v erarch in g b u sin ess strategy. A s s u c h , gain in g insight into cu sto m e r satis­ fa ctio n a n d p ro d u ct fe e d b a c k is p aram oun t. O n e o f W h irlp o o l’s g o als is to m o re e ffe ctiv e ly u n d erstan d an d re a c t to cu sto m e r an d p ro d u ct fe e d b a c k data, o rigin atin g fro m b lo g s , e -m ails, review s, forum s, rep air n o te s , a n d o th e r data so u rce s. W h irlp o o l also strives to e n a b le its m an ag ers to re p o rt o n lon g itu ­ d inal d ata, a n d b e a b le to co m p a re issu es b y b ran d o v e r tim e. W h irlp o o l h a s e n tru sted A ttensity’s text analytics solu tion s; a n d w ith th at, W h irlp o o l listens an d a cts o n cu sto m e r d ata in th e ir serv ice d ep art­ m en t, th e ir in n o v atio n and p ro d u ct d ev elo p m en ts g ro u p s, a n d in m ark et e v ery day.

M e th o d s a n d t h e B e n e f its

T o f a c e its b u sin e ss re q u irem en ts h e a d -o n , W h irlp ool u s e s A ttensity p ro d u cts fo r d e e p te x t analytics o f th e ir m u lti-ch a n n el cu sto m e r d ata, w h ic h in clu d es e-m ails, CRM n o te s , rep air n o te s , w arranty data, a n d s o c ia l m ed ia. M ore th a n 3 0 0 b u sin e ss u sers at W h irlp o o l u s e te x t an aly tics so lu tio n s e v ery d ay to g e t to th e ro o t ca u se o f p ro d u ct issu es an d re ce iv e alerts o n e m e rg in g issu es. U sers o f A ttensity s an aly tics p rodu cts at W h irlp o o l in clu d e product/ serv ice m an ag ers, corp orate/ p ro d u ct sa fe ty staff, c o n s u m e r a d v o ca tes, s e iv ic e qu ality staff, in n ov a­ tio n m a n a g e rs, th e C atego ry In sig h ts team , and all o f W h irlp o o l’s m anu factu ring d iv isions (a cro ss five co u n trie s).

A ttensity’s T e x t A nalytics a p p lica tio n has p lay ed a p articu larly critical ro le fo r W hirlp ool. W h irlp o o l re lie s o n th e a p p lica tio n to con d u ct d e e p analysis o f th e v o ic e o f th e cu sto m e r, w ith th e g o a l o f id entifying p ro d u ct quality issu es and in n o v a tio n o p p o rtu n itie s, an d drive th o se insights m o re b ro ad ly a c ro ss th e o rg an ization . U sers c o n ­ d u ct in -d ep th a n aly sis o f cu sto m e r d ata an d th e n e x te n d a c c e s s to th at an aly sis to b u sin e ss u sers all

o v e r th e w orld. W h irlp o o l h a s b e e n a b le to m o re p ro a c­

tively id en tify an d m itigate qu ality issu es b e fo re issu es e sca la te a n d claim s are filed . W h irlp o o l has a lso b e e n a b le to av o id recalls, w h ich h a s th e dual b e n e fit o f in cre a se d cu sto m e r loyalty and red u ced co s ts (realizin g 8 0 % savings o n th eir co s ts o f recalls d u e to early d e te c tio n ). H aving insight in to cu s­ to m e r fe e d b a c k an d p ro d u ct issu es h a s a lso resulted in m o re e ffic ie n t cu sto m e r su p p o rt and ultim ately in b e tte r p ro d u cts. W h irlp o o l’s cu sto m e r su p p ort ag e n ts n o w r e c e iv e fe w e r p ro d u ct serv ice su p p ort calls, and w h e n ag e n ts d o re ce iv e a call, it’s eas­ ier for th em to lev erag e th e in teractio n to im prove

pro d u cts a n d serv ices. T h e p ro c e s s o f lau n ch in g n e w p ro d u cts

has a lso b e e n e n h a n c e d b y having th e ability to an alyze its cu sto m e rs’ n e e d s an d fit n e w p rodu cts an d s erv ices t o th o se n e e d s ap p rop riately . W h e n a p ro d u ct is la u n ch e d , W h irlp o o l c a n u se e xte rn al cu sto m e r fe e d b a c k d ata to stay o n top o f p otential p ro d u ct issu es an d ad dress th e m in a tim ely fashion.

M ich ael P ag e, d ev e lo p m e n t a n d testin g m an­ ag e r fo r Q u ality A nalytics a t W h irp o o l C o rp o ratio n affirm s th e s e ty p e s o f b en e fits: “A ttensity’s p ro d ­ u cts h av e p ro v id e d im m e n se v alu e to o u r b u sin ess. W e ’v e b e e n a b le to p ro activ ely ad dress cu sto m e r fe e d b a c k and w o rk to w ard h ig h lev els o f cu sto m e r serv ice a n d p ro d u ct s u c c e s s .’

Q u e s t i o n s f o r D i s c u s s i o n

1. H o w did W h irlp o o l u s e cap a b ilitie s o f te x t an a ­ lytics to b e tte r u n d erstan d th eir cu sto m ers a n d im p ro ve p ro d u ct offerings?

2. W h at w e r e th e c h a lle n g e s, th e p ro p o se d so lu ­ tion , an d th e o b ta in e d results?

Source: Source: Attensity, Customer Success Stoiy, w w w . a t t e n s r t y . c o m / 2 0 1 0 / 0 8 / 2 l / w h i r l p o o l - 2 / (accessed August 2013)-

C hapter 7 • T e x t Analytics, T e x t Mining, and Sentim ent Analysis 3 5 3

SECTION 7 . 7 REVIEW QUESTIONS

1 . W h at is se n tim e n t analysis? H o w d o e s it relate to te x t mining?

2 . W hat are th e so u rce s o f data fo r se n tim e n t analysis? 3 . W h at are th e co m m o n c h a lle n g e s that s en tim en t an alysis h a s to d eal with?

7.8 SENTIMENT A N A LYSIS APPLICATIONS C o m p a re d to trad itio n al s e n tim e n t a n aly sis m e th o d s , w h ic h w e r e su rv e y b a s e d o r fo cu s g ro u p c e n te r e d , co s tly , a n d tim e -c o n s u m in g (a n d th e re fo r e d riv e n fro m s m a ll s a m p le s o f p articip a n ts), th e n e w fa c e o f te x t a n a ly tic s - b a s e d s e n tim e n t a n a ly sis is a lim it b re a k e r. C u rrent s o lu tio n s a u to m a te v e ry la rg e -s c a le d ata c o lle c tio n , filterin g , cla ss ific a tio n , and clu ste rin g m e th o d s via n a tu ra l la n g u a g e p r o c e s s in g a n d d ata m in in g te c h n o lo g ie s that h a n d le b o th fa c tu a l an d s u b je c tiv e in fo rm a tio n . S e n tim e n t a n a ly sis is p e rh a p s th e m o st p o p u la r a p p lic a tio n o f te x t an a ly tics, ta p p in g in to data s o u r c e s lik e tw e e ts , F a c e b o o k p o sts, o n lin e c o m m u n itie s , d is c u s s io n b o a rd s , W e b lo g s, p ro d u c t re v ie w s , ca ll c e n te r lo g s an d re c o rd in g , p ro d u c t ratin g s ite s , c h a t ro o m s , p ric e c o m p a ris o n p o rta ls , s e a rch e n g in e lo g s, a n d n e w sg ro u p s. T h e fo llo w in g a p p lic a tio n s o f s e n tim e n t a n aly sis are m e a n t to illu strate th e p o w e r and th e w id e s p r e a d c o v e r a g e o f this te c h n o lo g y .

V O IC E O F TH E C U ST O M E R (V O C ) Voice o f th e cu sto m er (VOC) is a n in teg ral part o f a n an alytic CRM a n d cu sto m e r e x p e r ie n c e m an a g e m e n t system s. As th e e n a b le r o f VOC, sen tim en t a n aly sis c a n a c c e s s a co m p a n y ’s p ro d u ct a n d serv ice review s (e ith e r co n tin u ­ o u sly o r p e rio d ica lly ) to b e tte r u n d erstan d an d b e tte r m a n a g e th e cu sto m e r com p lain ts an d p raises. F o r in sta n ce, a m o tio n p ictu re ad vertising/m arketing c o m p a n y m ay d etect th e n e g ativ e sen tim en ts tow ard a m o v ie th at is a b o u t to o p e n in th eatres (b a s e d o n its trailers), an d q u ick ly c h a n g e th e co m p o s itio n o f trailers an d ad vertising strategy (o n all m ed ia o u tlets) to m itigate th e n eg ativ e im pact. Sim ilarly, a softw are c o m p a n y m ay d etect th e n e g ativ e bu zz reg ard in g th e b u g s fo u n d in th e ir n e w ly re le a se d p ro d u ct early e n o u g h to re le a se p a tc h e s a n d q u ick fix e s to alleviate th e situation.

O fte n , th e fo cu s o f V O C is individual cu sto m ers, th e ir serv ice- and su p p o rt-related n e ed s, w an ts, a n d issu es. V O C d raw data fro m th e full s e t o f cu sto m e r to u ch points, inclu d ing e -m ails, surveys, call c e n te r n o tes/ record ing s, and so cia l m ed ia p o stin g s, and m atch cu sto m e r v o ic e s to tran sactio n s (in q u iries, p u rch a ses, retu rn s) an d individual cu s­ to m er p ro files ca p tu re d in e n te rp rise o p era tio n a l system s. V O C , m o stly d riven b y sen ti­ m en t analysis, is a k e y e le m e n t o f cu sto m er exp erien ce m anagem ent initiatives, w h e re th e g o al is to c re a te an intim ate relatio n sh ip w ith th e cu stom er.

V O IC E O F TH E M A R K E T (VO M ) Voice o f th e m ark et is a b o u t u n d erstan d in g ag g reg ate o p in io n s an d tren d s. It’s a b o u t k n o w in g w h at sta k e h o ld e rs— cu sto m e rs, p o te n tia l cu sto m ­ ers, in flu e n cers, w h o e v e r— are say in g a b o u t y o u r (a n d y o u r co m p e tito rs ’) p ro d u cts an d services. A w e ll-d o n e V O M analysis h e lp s c o m p a n ie s w ith co m p etitiv e in te llig e n ce and p rodu ct d e v e lo p m e n t and p o sition in g .

V O IC E O F T H E E M P L O Y E E (V O E ) T rad itio n ally V O E has b e e n lim ited to e m p lo y e e satis­ factio n surveys. T e x t an aly tics in g e n era l (an d s en tim en t an alysis in p articu lar) is a h u ge e n a b le r o f a ss e s s in g th e V O E . U sing rich, o p in io n a te d textu al d ata is a n e ffe ctiv e an d e fficie n t w a y to liste n to w h at e m p lo y e e s a re saying. As w e all k n o w , h ap p y e m p lo y e e s e m p o w er cu sto m e r e x p e r ie n c e e ffo rts and im p ro ve cu sto m e r satisfactio n .

B R A N D M A N A G E M E N T B ra n d m a n a g e m e n t fo c u s e s o n listen in g to s o cia l m ed ia w h e re a n y o n e (past/cu rrent/ prospective cu sto m ers, industry e x p e rts, o th e r a u th o rities) c a n p o st o p in io n s th at c a n d am ag e o r b o o s t y o u r rep u tatio n. T h e re are a n u m b e r o f relatively

3 5 4 Part III • Predictive Analytics

n ew ly la u n ch e d start-up c o m p a n ie s that o ffe r an aly tics-d riv en b ra n d m a n a g e m e n t ser­ v ic e s fo r oth ers. B ra n d m an a g e m e n t is p ro d u ct a n d co m p a n y (ra th er th an cu sto m er) fo cu se d . It attem pts to s h a p e p e rce p tio n s rath er th a n to m a n a g e e x p e r ie n c e s u sin g sen ti­

m e n t analysis te ch n iq u es.

F IN A N C IA L M A R K E T S P red ictin g th e future v a lu e s o f individ ual (o r a g ro u p of) s to c k s h a s b e e n a n in te restin g a n d s e e m in g ly u n so lv a b le p ro b le m . W h a t m a k e s a s to c k (o r a g ro u p o f s to c k s ) m o v e u p o r d o w n is an y th in g b u t a n e x a c t s c ie n c e . Many b e lie v e th a t th e s to c k m a rk e t is m o stly s e n tim e n t d riv en , m a k in g it a n y th in g b u t rational (e s p e c ia lly fo r sh o rt-term s to c k m o v e m e n ts ). T h e r e fo r e , u s e o f se n tim e n t an alysis in fin an cial m ark e ts h a s g a in e d sig n ifica n t p o p u larity . A u tom ated a n aly sis o f m a rk e t sen ti­ m e n ts u sin g s o c ia l m ed ia, n e w s , b lo g s , an d d is c u s s io n g ro u p s s e e m s to b e a p ro p e r w ay to c o m p u te th e m a rk e t m o v e m e n ts . I f d o n e c o r re c tly , s e n tim e n t a n aly sis c a n identify sh o rt-te rm s to c k m o v e m e n ts b a s e d o n th e b u z z in th e m ark e t, p o ten tia lly im p actin g

liquid ity a n d trading.

P O LIT IC S As w e all k n o w , o p in io n s m atter a g re a t d ea l in p o litics. B e c a u s e political d iscu ssio n s are d o m in ated b y q u o te s, sarcasm , a n d c o m p le x r e fe re n c e s to p e rso n s, o ig a - n izatio n s, a n d id eas, p o litics is o n e o f th e m o st difficult, a n d p o ten tially fruitful, areas fo r s en tim en t analysis. B y an aly zin g th e s en tim en t o n e le c tio n fo ru m s, o n e m ay pred ict w h o is m o re likely to w in o r lose. S e n tim en t a n aly sis c a n h e lp u n d erstan d w h at vo ters a re th in kin g and c a n clarify a ca n d id a te ’s p o sitio n o n issu es. S en tim en t analysis c a n h elp p o litical o rgan ization s, cam p aig n s, a n d n e w s an alysts to b e tte r u n d erstan d w h ich issu es a n d p o sitio n s m atter th e m o st to vo ters. T h e te c h n o lo g y w as s u cce ssfu lly ap p lied b y b o th parties to th e 2 0 0 8 an d 2 0 1 2 A m erican p resid en tial e le c tio n cam p aign s.

G O V E R N M E N T IN T E L L IG E N C E G o v ern m e n t in te llig e n ce is an o th e r a p p licatio n that has b e e n u s e d b y in te llig e n ce ag e n cie s. F o r e x a m p le , it has b e e n su g g e ste d that o n e c o u ld m o n ito r so u rce s fo r in cre a se s in h o stile o r n e g ativ e co m m u n icatio n s. S e n tim en t analysis c a n allo w th e au tom atic an alysis o f th e o p in io n s th a t p e o p le su b m it a b o u t p e n d in g p o licy o r g o v ern m e n t-re g u latio n p ro p o sals. F u rth erm ore, m o n ito rin g co m m u n icatio n s fo r sp ik es in n eg ativ e sen tim en t m ay b e o f u se to a g e n c ie s lik e H o m elan d Security.

O TH ER IN T E R E S T IN G A R E A S Sen tim en ts o f cu sto m e rs c a n b e u s e d to b e tte r d esign e -c o m m e r c e sites (p ro d u ct su g g e stio n s, up sell/ cross-sell ad vertising ), b e tte r p la c e ad ver­ tisem en ts (e .g ., p la cin g d y n am ic ad v ertisem en t o f p ro d u cts a n d s erv ices th a t c o n s id e r the sen tim en t o n th e p a g e th e u s e r is b ro w sin g ), an d m a n a g e o p in io n - o r re v iew -o rie n te d s e a rch e n g in e s (i.e ., a n o p in io n -a g g reg a tio n W e b site, a n altern ativ e to sites like E p in ions, sum m arizing u s e r re v iew s). S e n tim en t analysis c a n h e lp w ith e-m ail filtration b y c a te g o ­ rizing an d prioritizing in co m in g e-m ails (e .g ., it c a n d e te c t strongly n e g ativ e o r flam ing e-m ails a n d forw ard th e m to th e p ro p e r fo ld e r), as w e ll as citatio n analysis, w h e re it c a n d eterm in e w h e th e r a n a u th o r is citin g a p ie c e o f w o rk as su p p o rtin g e v id e n c e o i as re se a rch th a t h e o r s h e dism isses.

SECTION 7 . 8 REVIEW QUESTIONS

1 . W h a t are th e m o st p o p u la r a p p lica tio n are a s fo r s en tim en t analysis? W hy?

2 . H ow c a n s en tim en t an alysis b e u se d fo r b ra n d m anagem ent? 3 . W h a t w o u ld b e th e e x p e c te d b e n e fits a n d b e n e fic ia r ie s o f sen tim en t analysis in

politics? 4 . Howr c a n sen tim en t analysis b e u s e d in p re d ictin g fin an cial m aikets?

Chapter 7 • T ex t Analytics, T e x t Mining, and Sentim ent Analysis 3 5 5

7.9 SENTIMENT A N A LY SIS PROCESS B e ca u s e o f th e com p lex ity o f th e p ro b lem (underlying co n c ep ts, exp ressio n s in text, c o n ­ text in w h ich th e te x t is exp ressed , e tc .), th ere is n o readily available standardized p ro cess to co n d u ct sen tim en t analysis. H ow ever, b a se d o n th e p u blished w o rk in th e field o f sen ­ sitivity analysis s o far (b o th o n research m ethods and rang e o f ap p licatio ns), a multi-step, sim ple logical p ro cess, as g iven in Figure 7 .1 0 , se e m s to b e a n appropriate m e th o d o lo g y fo r sentim ent analysis. T h e s e logical steps are iterative (i.e., fe e d b ack , correctio n s, and iterations are part o f th e d iscovery p ro cess) an d exp erim ental in nature, an d o n c e c o m p le te d and com bin ed , ca p a b le o f p ro d u cing d esired insight a b o u t th e op in ion s in th e te x t collection .

STEP 1: S EN TIM EN T D ETECTIO N After th e retrieval and preparation o f the te x t docum ents, the first m ain task in sensitivity analysis is th e d etection o f objectivity. H ere th e g oal is to differentiate b e tw e e n a fact and an opinion, w h ich m ay b e view ed as classification o f text as ob jective o r subjective. T h is m ay also b e characterized as calculation o f O -S Polarity (Objectivity-Subjectivity Polarity, w h ich m ay b e represented w ith a num erical value ranging from 0 to 1). If th e objectivity value is clo se to 1, th e n th ere is n o o p in ion to m ine (i.e., it is a fact); therefore, th e p ro cess g o e s b a c k an d grabs th e n ext text data to analyze. Usually opinion

A statem ent

Step 1 Calculate the 0 -S Polarity

Lexicon L.

Step 2 Calculate th e N P

polarity of the sentim ent

N -P Polarity

□-S polarity measure

Step 3 Identify the target for the sentim ent

R ecord th e Polarity, Stre ngth, and the

T a rg e t of the sentiment

Target

Step 4 Tabulate and aggregate

th e sentiment analysis results

: GURE 7 .10 A Multi-Step Process to Sentim ent Analysis.

d etectio n is b ase d o n the exam ination o f adjectives in text. F o r exam p le, th e polarity o f “what a w ond erful w o rk ” c a n b e d eterm ined relatively easily b y looking at the adjective.

ST EP 2: N -P P O L A R IT Y C L A S S IF IC A T IO N T h e s e c o n d m ain ta sk is that o f polarity classificatio n . G iv e n a n o p in io n a te d p ie c e o f te x t, th e g o a l is to classify th e o p in io n as fallin g u n d er o n e o f tw o o p p o sin g sen tim en t p o larities, o r lo c a te its p o sitio n o n th e c o n ­ tinuu m b e tw e e n th e s e tw o p o larities (P a n g a n d L ee, 2 0 0 8 ). W h e n v ie w e d as a binary featu re p o larity classificatio n is th e b in ary cla ssifica tio n ta sk o f la b e lin g a n op in io n ate d d o cu m e n t as e x p re ssin g e ith e r a n ov erall p o sitiv e o r a n o v erall n eg ativ e o p in io n (e .g th u m b s u p o r th u m b s d o w n ). In ad d ition to th e id en tificatio n o f N-P polarity, o n e shou ld a ls o b e in te reste d in identifying th e stren g th o f th e se n tim e n t (a s o p p o s e d to ju st positive it m ay b e e x p r e s s e d a s m ildly, m o d e rate ly , stro n gly, o r v e ry stro n g ly p o sitiv e). M ost of this re se a rch w as d o n e o n p ro d u ct o r m o v ie re v iew s w h e re th e d efin itio n s o f positive an d “n e g a tiv e ” are q u ite clear. O th e r task s, s u c h as classifyin g n e w s as “g o o d ” o r bad, p re s e n t s o m e difficulty. F o r in stan ce a n article m ay co n ta in n e g ativ e n e w s w ith o u t e x p lic ­ itly u sin g a n y s u b je ctiv e w o rd s o r term s. F u rth erm ore, th e s e cla ss e s usually a p p e a r inter­ m ix e d w h e n a d o cu m en t e x p r e s s e s b o th p o sitiv e a n d n e g ativ e sen tim en ts. T h e n th e task c a n b e to id entify th e m ain (o r d o m in atin g ) se n tim e n t o f th e d o cu m en t. Still, fo r lengthy te x ts th e tasks o f classificatio n m ay n e e d to b e d o n e a t sev eral lev els: term , phrase, s e n te n c e a n d p e rh a p s d o cu m en t lev el. F o r th o s e , it is co m m o n to u s e th e ou tp u ts o f o n e lev el as th e inputs fo r th e n e x t h ig h e r layer. S ev eral m e th o d s u s e d to id entify th e polarity a n d stren g th s o f th e p o larity are e x p la in e d in th e n e x t sectio n .

ST EP 3: T A R G E T ID E N T IF IC A T IO N T h e g o al o f this Step is to a ccu rate ly identify th e target o f th e e x p re s s e d sen tim en t (e .g ., a p erso n , a p ro d u ct, a n e v e n t, e tc .). T h e difficulty o f this ta sk d e p e n d s largely o n th e d om ain o f th e analysis. E ven th o u g h it is usu ally e a sy to a c c u ­ rately identify th e targ e t fo r p ro d u ct o r m o vie rev iew s, b e c a u s e th e re v iew is directly co n ­ n e cte d to th e target, it m ay b e q u ite ch a lle n g in g in o th er d om ains. F o r in stan ce, lengthy, g e n era l-p u rp o se te x t su ch as W e b p a g e s, n e w s articles, an d b lo g s d o n o t alw ays h av e a p re d efin e d to p ic th at th ey are assig n e d to , and o fte n m e n tio n m an y o b je cts, an y o f w h ich m ay b e d e d u ce d a s th e target. So m etim es th e re is m o re th an o n e target in a sen tim ent s e n te n c e , w h ic h is th e c a s e in com p arative texts. A su b je ctiv e com p arative s e n te n c e orders o b je c ts in ord er o f p re fe re n ce s— fo r e x a m p le , “T h is lap to p co m p u te r is b e tte r th an my d esk to p P C .” T h e s e s e n te n c e s c a n b e id entified u sin g com p arative ad jectiv es an d ad verbs (m o re , less, b etter, lo n g e r), superlative ad je ctiv e s (m o st, least, b e s t), an d o th er w o rd s (s u ch as sam e , differ, w in , p refer, e tc.). O n c e th e s e n te n c e s have b e e n retrieved , th e o b je c ts can b e put in a n ord e r that is m o st rep resen tativ e o f th e ir m erits, as d e s crib e d m text.

ST EP 4: C O L L E C T IO N A N D A G G R E G A T IO N O n c e th e sen tim en ts o f all te x t data p o in ts in th e d o cu m en t are id en tified a n d calcu lated , in this ste p th ey are ag g re g ate d an d c o n ­ v e rte d to a sin gle s en tim en t m e a su re fo r th e w h o le d o cu m en t. T h is a g g reg atio n m ay b e a s sim p le as sum m ing up th e p o larities a n d stren g th s o f all texts, o r as c o m p le x as using sem an tic ag g re g a tio n te c h n iq u e s fro m natu ral la n g u a g e p ro ce s s in g to c o m e u p w ith th e

ultim ate sen tim ent.

3 5 6 Part III • Predictive Analytics

Methods for Polarity Identification As m e n tio n e d in th e p rev iou s s e c tio n , p o la rity id e n tific a tio n — identifying th e polarity o f a te x t— c a n b e m a d e a t th e w o rd , term , s e n te n c e , o r d o cu m e n t lev el. T h e m o st g ranu ­ lar lev el fo r p o larity id en tificatio n is a t th e w o rd lev el. O n c e th e p o larity id e n tificatio n is m ad e at th e w o rd lev el, th e n it c a n b e a g g re g a te d to th e n e x t h ig h er lev el, a n d th e n th e n e x t until th e lev el o f a g g re g a tio n d esire d fro m th e sen tim en t analysis is re a ch e d . T h e re

C hapter 7 • T e x t A nalytics, T ex t Mining, and Sentim ent Analysis 3 5 7

s ee m to b e tw o d o m in an t te ch n iq u e s u s e d fo r id e n tificatio n o f p o larity at th e word/term lev el, e a c h having its ad v an tag es and d isad vantages:

1 . U s in g a le x ic o n a s a r e fe r e n c e lib rary ( e it h e r d e v e lo p e d m a n u a lly o r au to m a tica lly , b y a n ind iv id u al fo r a s p e c ific ta s k o r d e v e lo p e d b y a n in s titu tio n fo r g e n e ra l u s e )

2 . U sin g a c o lle c tio n o f training d o cu m en ts as th e s o u rc e o f k n o w le d g e a b o u t the po larity o f term s w ith in a s p e cific d o m ain (i.e ., in d u cin g p re d ictiv e m o d e ls from o p in io n a te d textu al d o cu m en ts)

Using a Lexicon A le x ic o n is e sse n tia lly th e c a ta lo g o f w o rd s , th e ir sy n o n y m s, ancl th e ir m e a n in g s for a g iv en la n g u a g e . In ad d ition to le x ic o n s fo r m a n y o th e r la n g u a g e s, th e re a re sev eral g e n e ra l-p u rp o s e le x ic o n s c re a te d fo r E n g lish . O fte n g e n e ra l-p u rp o s e le x ic o n s are u se d to c r e a te a variety o f s p e c ia l-p u rp o s e le x ic o n s fo r u s e in se n tim e n t an aly sis p ro je cts. P erh ap s th e m o st p o p u la r g e n e ra l-p u rp o s e le x ic o n is W ord N et, c r e a te d at P rin ce to n U n iversity, w h ic h h a s b e e n e x te n d e d a n d u sed b y m an y re s e a r c h e rs an d p ractitio n e rs fo r s e n tim e n t an aly sis p u rp o se s. As d e s c rib e d o n th e W ordN et W e b site (wordnet. princeton.edu), it is a large le x ic a l d a ta b a se o f E nglish, in clu d in g n o u n s , v e rb s , a d je c ­ tives, a n d ad v e rb s g ro u p e d in to sets o f co g n itiv e sy n o n y m s (i.e ., s y n s e ts), e a c h e x p r e s s ­ ing a d istin ct c o n c e p t. Sy n sets a re in te rlin k e d b y m e a n s o f c o n c e p tu a l-s e m a n tic and lex ica l re latio n s.

An in te restin g e x te n s io n o f W ord N et w a s cre a te d b y E su li a n d S e b a stia n i (2 0 0 6 ) w h e re th e y ad d ed p o larity (P o sitiv e-N eg ativ e ) and o b je ctiv ity (S u b je c tiv e -O b je c tiv e ) la b e ls fo r e a c h te rm in th e le x ic o n . T o la b e l e a c h term , th e y classify th e s y n s e t (a gro u p o f sy n o n y m s) to w h ich this te rm b e lo n g s using a s e t o f ternary7 cla ssifiers (a m e asu re that a tta c h e s to e a c h o b je c t e x a c tly o n e o u t o f th re e la b e ls ), e a c h o f th e m c a p a b le o f d ecid in g w h e th e r a s y n s e t is P o sitiv e, o r N eg ativ e, o r O b je c tiv e . T h e resu ltin g s co re s ran g e fro m 0 .0 to 1.0, g ivin g a g rad ed e v a lu a tio n o f o p in io n -re la te d p ro p e rtie s o f th e term s. T h e s e c a n b e su m m ed up visu ally a s in F ig u re 7 .1 1 . T h e e d g e s o f th e triangle re p re se n t o n e o f th e th re e cla ssifica tio n s (p o sitiv e , n eg ativ e, a n d o b je c tiv e ). A te rm can b e lo c a te d in this s p a c e a s a p o in t, re p re se n tin g th e e x te n t to w h ic h it b e lo n g s to e a c h o f th e cla ssifica tio n s.

A sim ilar e x te n s io n m e th o d o lo g y is u se d to c re a te SentiW ord N et, a p u b licly avail­ a b le le x ic o n s p e cifica lly d e v e lo p e d fo r o p in io n m ining (se n tim e n t a n aly sis) p u rp oses.

Objective (□ ]

Subjective [S]

Positive (P) (+)

Negative [N ] H

FIG U R E 7.11 A Graphical Representation o f the P-N Polarity and S-O Polarity Relationship.

3 5 8 Part III • Predictive Analytics

SentiW ordNet assig n s to e a c h s y n set o f W ord N et th ree sen tim en t sco re s: positivity, n e g ­ ativity, ob jectiv ity . M o re a b o u t SentiW ord N et c a n b e fo u n d at sentiw ordnet.isti.cnr.it.

A n o th e r e x te n s io n to W ordN et is W ordN et-A ffect, d e v e lo p e d b y Strap parava and Valitutti (Strap p arav a an d Valitutti, 2 0 0 4 ). T h e y la b e l W ordN et sy n sets u sin g affectiv e labels re p re sen tin g d ifferen t affectiv e ca te g o rie s lik e e m o tio n , co g n itiv e state, attitude, feeling, an d so o n . W ordN et h a s a lso b e e n d irectly u s e d in sen tim en t analysis. F o r e x a m p le , Kim an d H ov y (K im a n d H ovy, 2 0 0 4 ) an d H u an d Liu (H u an d Liu, 2 0 0 5 ) g e n e ra te le x ic o n s o f p o sitiv e an d n eg ativ e term s b y starting w ith a sm all list o f “s e e d ” term s o f k n o w n polari­ ties (e .g ., lo've, lik e , n ice , e tc .) a n d th e n u sin g th e an to n ym y an d sy n o n y m y p ro p e rtie s o f term s to g ro u p th e m in to e ith e r o f th e p o larity cate g o rie s.

Using a Collection o f Training Documents It is p o ss ib le to p e rfo rm s en tim en t cla ssifica tio n u sin g statistical analysis a n d m ach in e - learn in g to o ls that tak e ad v an tag e o f th e vast re s o u rc e s o f la b e le d (m an u ally b y an n o ta­ tors o r u sin g a star/point sy stem ) d o cu m en ts av ailab le. P rod u ct re v iew W e b sites like A m azon, C-NET, e b a y , R o tten T o m ato es, an d th e In tern et M ovie D a ta b a se (IM D B ) have all b e e n e x te n siv e ly u s e d as s o u rc e s o f an n o ta te d data. T h e star (o r to m ato, as it w e re ) sy stem p ro v id e s a n e x p licit la b e l o f th e ov erall p o larity o f th e review , an d it is o fte n taken as a g o ld standard in algorithm evaluation.

A v arie ty o f m an u ally la b e le d textu al d ata is a v ailab le th ro u g h e v a lu a tio n efforts s u ch as th e T e x t REtrieval C o n fe re n c e (T R E C ), N il T e s t C o lle ctio n fo r IR System s (N TC IR ), a n d C ross L an gu age E v alu ation F o ru m (C LEF). T h e d ata s e ts th e s e e ffo rts p ro ­ d u c e o fte n serv e as a stan d ard in th e te x t m in in g com m u n ity , in clu d in g fo r sen tim en t an alysis re se a rch e rs. Individual re s e a rch e rs an d re se a rch g ro u p s h a v e a lso p ro d u ce d m an y in te restin g data sets. T e c h n o lo g y Insigh ts 7 .2 lists s o m e o f th e m o st p o p u la r o n e s. O n c e a n alread y la b e le d textu al data s e t is o b ta in e d , a v ariety o f p re d ictiv e m o d eling a n d o th e r m a ch in e -lea rn in g algorith m s c a n b e u se d to train se n tim e n t classifiers. S o m e o f th e m o st p o p u la r algorith m s u s e d fo r this ta s k in clu d e artificial n eu ral n e tw o rk s, su p ­ p o rt v e c to r m a ch in e s , ^ -n e are st n e ig h b o r, N aive B a y e s , d e c is io n tre e s, and e x p e c ta tio n m a x im iz a tio n -b a se d clusterin g.

Identifying Sem antic Orientation of Sentences and Phrases O n c e th e s e m a n tic o r ie n ta tio n o f in d iv id u al w o rd s h a s b e e n d e te rm in e d , it is o fte n d e s ir a b le to e x te n d th is to th e p h r a s e o r s e n t e n c e th e w o rd a p p e a rs in. T h e s im p le s t w a y to a c c o m p lis h s u c h a g g re g a tio n is to u s e s o m e ty p e o f a v e ra g in g fo r th e p o la ri­ tie s o f w o rd s in th e p h r a s e s o r s e n te n c e s . T h o u g h ra re ly a p p lie d , s u c h a g g re g a tio n c a n b e a s c o m p le x a s u s in g o n e o r m o re m a c h in e -le a r n in g t e c h n iq u e s to c r e a te a p r e d ic tiv e re la tio n s h ip b e t w e e n th e w o rd s (a n d th e ir p o la rity v a lu e s ) a n d p h r a s e s o r s e n te n c e s .

Identifying Sem antic Orientation of Document E v e n th o u g h th e v a st m a jo rity o f th e w o rk in th is a re a is d o n e in d e te rm in in g s e m a n ­ tic o r ie n ta tio n o f w o rd s a n d p h r a s e s / s e n te n c e s , s o m e ta s k s lik e s u m m a riz a tio n a n d in fo r m a tio n re triev a l m a y re q u ire s e m a n tic la b e lin g o f th e w h o le d o c u m e n t (R E F ). S im ilar to th e c a s e in a g g re g a tin g s e n tim e n t p o la rity fro m w o rd le v e l to p h r a s e o r s e n t e n c e le v e l, a g g r e g a tio n to d o c u m e n t lev el is a ls o a c c o m p lis h e d b y s o m e ty p e o f a v e ra g in g . S e n tim e n t o r ie n ta tio n o f th e d o c u m e n t m ay n o t m a k e s e n s e fo r v e ry la rg e d o c u m e n ts ; th e r e fo r e , it is o fte n u s e d o n sm all t o m e d iu m -s iz e d d o c u m e n ts p o s te d o n th e In te rn e t.

Chapter 7 • T e x t Analytics, T e x t Mining, and Sentim ent Analysis 3 5 9

T E C H N O L O G Y IN S IG H T S 7 . 2 L a r g e T e x tu a l D a ta S e ts f o r P r e d ic tiv e T e x t M in in g a n d S e n tim e n t A n a ly s is

Congressional Floor-Debate Transcripts: Published by Thomas et al. (Thomas and B. Pang, 2006); contains political speeches that are labeled to indicate whether the speaker supported or opposed the legislation discussed.

E con om ining: Published by Stem School at New York University; consists o f feed­ back postings for merchants at Amazon.com.

C ornell Movie-Review D ata Sets: Introduced by Pang and Lee (Pang and Lee, 2008); contains 1,000 positive and 1,000 negative automatically derived document-level labels, and 5,331 positive and 5,331 negative sentences/snippets.

Sta n ford —Large M ovie Review D ata Set: A set o f 25,000 highly polar movie reviews for training, and 25,000 for testing. There is additional unlabeled data for use as well. Raw text and already processed bag-of-words formats are provided. (See: h t t p : / / a i . s t a n f o r d . e d u / ~ a m a a s / d a t a / s e n t i m e n t . )

M PQ A Corpus: Corpus and Opinion Recognition System corpus; contains 535 manu­ ally annotated news articles from a variety o f news sources containing labels for opinions and private states (beliefs, emotions, speculations, etc.).

M ultiple-Aspect R estaurant Reviews: Introduced by Snyder and Barzilay (Snyder and Barzilay, 2007); contains 4,488 reviews with an explicit l-to-5 rating for five different aspects: food, ambiance, service, value, and overall experience.

S E C T I O N 7 .9 R E V I E W Q U E S T I O N S

1 . W h at a re th e m ain s te p s in carrying o u t sen tim en t analysis projects?

2 . W h at are th e tw o co m m o n m e th o d s fo r polarity identification? W h at is th e m ain dif­ fe re n c e b e tw e e n th e two?

3 . D e s c r ib e h o w sp e cia l le x ic o n s are u se d in id en tificatio n o f s e n tim e n t polarity.

7.10 SE N T IM E N T A N A L Y S IS A N D SPE E C H A N A LY T IC S S p e e c h a n a l y t i c s is a g ro w in g field o f s c ie n c e that allo w s u sers to an aly ze an d e xtract in form ation fro m b o th live a n d re co rd e d co n v ersatio n s. It is b e in g u se d e ffe ctiv e ly to g a th e r in te llig e n ce fo r secu rity p u rp o se s, to e n h a n c e th e p re sen tatio n and utility o f rich m edia a p p lica tio n s, a n d p e rh a p s m o st significantly, to d eliver m ean in gfu l an d quantitative b u sin ess in te llig e n ce th ro u g h th e analysis o f th e m illion s o f re co rd e d calls th at o c c u r in cu sto m e r c o n ta c t ce n te rs aro u n d th e w orld.

S en tim en t analysis, as it ap p lie s to s p e e c h analytics, fo c u s e s s p e cifica lly o n a s s e s s ­ ing th e e m o tio n a l states e x p r e s s e d in a co n v e rsa tio n and o n m easu rin g th e p re s e n c e an d stren g th o f p o sitiv e a n d n eg ativ e fe e lin g s that are e x h ib ite d b y th e participants. O n e co m m o n u s e o f sen tim en t analysis w ithin co n ta ct ce n te rs is to p ro v id e insight into a cu sto m e r’s fe e lin g s a b o u t an o rgan ization , its p ro d u cts, s e iv ic e s , a n d cu sto m e r serv ice p ro ce s s e s, as w e ll as an individual a g e n t’s b eh av io r. S en tim en t analysis data c a n b e used a cro ss a n o rg a n iz a tio n to aid in cu sto m e r relatio n sh ip m an ag e m e n t, ag e n t training, and in identifying a n d re so lv in g tro u b lin g issu es as th e y em e rg e.

How Is It Done? T h e c o r e o f a u to m a te d sen tim en t an alysis ce n te rs aro u n d cre a tin g a m o d e l to d e scrib e h o w ce rta in fe a tu res an d c o n te n t in th e au d io relate to th e sen tim en ts b e in g felt an d e x p re sse d b y th e p articip an ts in th e co n v e rsatio n . T w o prim ary m e th o d s h av e b e e n d ep loy ed to p re d ict s en tim en t w ith in au d io: aco u stic/ p h o n etic a n d lingu istic m od eling .

THE A C O U ST IC A PP R O A CH T h e a co u s tic a p p r o a c h to s en tim en t analysis relies o n e xtract­ ing and m easu rin g a s p e c ific s e t o f featu res (e .g ., to n e o f v o ic e , p itch o r v o lu m e , intensity an d rate o f s p e e c h ) o f th e au d io. T h e s e featu res c a n in s o m e circu m sta n ces p ro v id e b asic in d icato rs o f sen tim en t. F o r e x a m p le , th e s p e e c h o f a su rp rised s p e a k e r tend s to b e c o m e so m e w h a t faster, lou d er, a n d h ig h e r in p itch . Sa d n ess and d e p re s s io n a re p re se n te d as s lo w e r, so fter, a n d lo w e r in p itch ( s e e M o o re e t al„ 2 0 0 8 ). A n angry c a lle r m ay sp e a k m u c h faster, m u ch lou d er, a n d w ill in c re a se th e p itch o f stre sse d v o w e ls. T h e r e is a w ide v ariety o f au d io featu res th a t c a n b e m e asu re d . T h e m o st co m m o n o n e s are a s fo llow s:

• Intensity: en erg y , so u n d p re ssu re lev el • P itch: v ariation o f fu n d am en tal fre q u e n cy • Jitter: v ariation in am p litu d e o f v o c a l fo ld m o v e m e n ts • Shim m er: variatio n in fre q u e n cy o f v o cal fo ld m o v e m e n ts • G lo ttal p u lse: g lo ttal-so u rce s p e ctra l ch aracteristics • HNR: h a rm o n ics-to -n o ise ratio , • S p e ak in g rate: n u m b e r o f p h o n e m e s , v o w e ls , sy llables, o r w o rd s p e r u n it o f tim e

W h e n d ev elo p in g a n a co u stic analysis to o l, th e sy stem m u st b e b u ilt o n a m o d el th at d efin e s th e sen tim en ts b e in g m easu red . T h e m o d el is b a s e d o n a d a ta b a se o f the a u d io fe a tu res (s o m e o f w h ic h are liste d h e re ) a n d h o w th e ir p re s e n c e m ay in d icate e a c h o f th e sen tim en ts (a s sim p le a s p o sitive, n e g a tiv e , neutral, o r re fin ed , su ch as fear, anger, sa d n ess, hurt, su rp rise, relief, e tc .) that are b e in g m easu red . T o cre a te this d atabase, e a c h sin g le -e m o tio n e x a m p le is p re s e le c te d fro m a n o rigin al s e t o f r e c o r d i n g s manually- rev iew ed , and an n o ta te d to id entify w h ic h se n tim e n t it re p re sen ts. T h e fin al aco u stic an aly sis to o ls are th e n train ed (u sin g data m in in g te c h n iq u e s ) an d a p red ictiv e m o d e l is te sted a n d valid ated u sin g a d ifferen t s e t o f th e s a m e an n o ta te d record in g s.

As s o p h istica ted as it so u n d s, th e a co u s tic a p p ro a ch h a s its d eficien cie s. First, b e c a u s e a co u s tic analysis relies o n id entify ing th e au d io ch aracteristics o f a call, t e qu ality o f th e a u d io c a n significantly im p a ct th e ab ility to id entify th e s e featu res. S e c o n d sp e a k e rs o fte n e x p r e s s b le n d e d e m o tio n s, s u ch as b o th em p ath y a n d a n n o y a n c e (a s in d o u n d erstand , m ad am , b u t I h av e n o m iracle s o lu tio n ”), w h ic h a re e x trem ely d ifficult to classify b a s e d so le ly o n th e ir a co u s tic featu res. T h ird , a co u s tic analysis is o ften in ca p a b e o f re co g n iz in g an d ad ju sting fo r th e variety o f w ay s th at d ifferen t ca lle rs m ay e x p r e s s th e sa m e sen tim en t. Finally, its tim e -d em an d in g a n d la b o rio u s p ro c e s s m a k e it im p ractical lor

u s e w ith liv e au d io stream s.

THE LINGUISTIC A PP R O A CH C o n v ersely , th e ling u istic a p p ro a ch fo c u s e s o n th e exp licit in d icatio n s o f sen tim en t a n d c o n te x t o f th e s p o k e n c o n te n t w ith in th e au d io; linguistic m o d e ls a ck n o w le d g e that, w h e n in a ch a rg e d state , th e s p e a k e r h a s a h ig h er pro bab ility o f u sin g s p e c ific w o rd s, ex cla m a tio n s, o r p h ra se s in a particu lar order. T h e featu res that are m o st o fte n an aly zed in a lingu istic m o d e l in clu d e:

• L exical: w o rd s, p h rases, a n d o th e r lin gu istic patterns • D isflu en cie s: filled p au ses, h e sitatio n , restarts, an d n o n v e rb als s u ch as lau gh ter o r

b rea th in g . • H ig h er sem an tics: taxon o m y/ on to lo g y, d ia lo g u e history, an d p iag m atics

T h e sim p lest m e th o d , in th e lingu istic a p p ro a c h , is to c a tc h w ith in th e au d io a lim­ ited n u m b e r o f s p e cific k e y w o rd s ( a s p e cific le x ic o n ) that has d o m a m -sp ec ific sen tim en sig n ifica n ce . T h is a p p ro a ch is p e rh a p s th e le a s t p o p u la r d u e to its l i m i t e d ap p licab ility and less-th a n -d esired p re d ictio n accu racy . A lternatively, as w ith th e a co u s tic ap p ro a c a m o d e l is b u ilt b a s e d o n u n d erstan d in g w h ic h lingu istic e le m e n ts a r e p re d icto rs o l particu lar sen tim en ts, a n d this m o d e l is th e n ru n ag ain st a serie s o f re co rd m g s to d eter­ m in e th e sen tim en ts th at are co n ta in e d th e re in . T h e c h a lle n g e w ith this a p p ro a ch is in

3 6 0 Part III • Predictive Analytics

Chapter 7 • T ex t Analytics, T e x t Mining, and Sentim ent Analysis 3 61

c o lle ctin g th e ling u istic in fo rm atio n co n ta in e d in an y co rp u s o f aud io. T h is h a s trad ition­ ally b e e n d o n e u sin g a larg e v o ca b u la ry c o n tin u o u s s p e e c h re c o g n itio n (LVCSR) system , o ften re fe rre d to as sp e e c h -to -te x t. H o w ev er, LVCSR sy stem s are p ro n e to cre atin g signifi­ ca n t e rro r in th e textu al in d e x e s th e y cre a te . Tn ad dition, th e lev el o f c o m p u ta tio n a l effort th ey requ ire— th at is, th e am o u n t o f co m p u te r p ro ce ssin g p o w e r n e e d e d t o an aly ze large am o u n ts o f a u d io co n te n t— has m a d e th e m v e ry e x p e n s iv e to d e p lo y fo r m ass au d io analysis.

Y e t, a n o th e r a p p ro a ch to lingu istic analysis is th at o f p h o n e tic in d e x in g a n d search . A m on g th e sig n ifican t ad v an tag es a s s o cia te d w ith this a p p ro a ch to lin gu istic m o d elin g is th e m e th o d ’s ab ility to m ain tain a high d e g re e o f a ccu ra cy n o m atter w h a t th e quality o f th e au d io so u rce , an d its in co rp o ratio n o f co n v e rsa tio n a l c o n te x t th ro u g h th e u s e o f stru ctured q u e rie s during analysis (N exid ia, 2 0 0 9 ).

A p p lica tio n C ase 7 .8 is a g reat e x a m p le to h o w analytically savvy c o m p a n ie s find w ays to b e tte r “lis te n ” an d im p ro ve th eir cu sto m e rs’ e x p e rie n c e .

Application Case 7.8 Cutting Through the Confusion: Blu e Cross Blu e Shield o f North C arolina Uses N exidia's Speech A n alytics to Ease M em b er Experience in H ealthcare In tro d u ctio n

W ith th e p a s s a g e o f th e h e a lth ca re law , m an y health p lan m e m b e r s w e re p e rp le x e d b y n e w ru les and re g u latio n s an d c o n c e rn e d a b o u t the e ffe c ts m an ­ d a te s w o u ld h a v e o n th e ir b en e fits, co p a y s, an d p ro v id ers. In a n attem p t to e a s e c o n c e rn s , h e alth p lan s s u c h as B lu e C ross B lu e Sh ield o f North C arolin a (B C B S N C ) p u b lish e d literature, u p d ated W e b sites, a n d s e n t vario u s fo rm s o f co m m u n ica tio n to m e m b e r s to fu rth e r e d u c a te th e m o n th e ch an g e s. H o w ev er, m e m b ers c o n tin u e d to re a c h o u t via th e c o n ta c t c e n te r, s e e k in g a n sw e rs regard ing cu rren t claim s an d b e n e fits a n d h o w th e ir h e a lth in su ran ce c o v e ra g e m ig h t b e a ffe c te d in th e fu ture. As th e law m o v e s fo rw ard , m e m b ers w ill b e m o re e n g a g e d in m ak in g th e ir o w n d e c is io n s a b o u t h e a lth ca re p lan s and a b o u t w h e re to s e e k ca re , thu s b e c o m in g b e tte r co n s u m e rs . T h e tran sfo rm atio n to h e a lth ca re c o n ­ su m erism h a s m a d e it cru cial fo r h e alth p la n co n ta ct ce n te rs to d ilig ently w o rk to o p tim ize th e cu sto m e r e x p e rie n c e .

B C B SN C b e c a m e c o n c e rn e d th a t d esp ite its b e s t e ffo rts to co m m u n ica te ch a n g e s, c o n fu sio n re m ain ed am o n g its n e arly 4 m illion m e m b ers, w h ich w a s d riving u n n e ce s s a ry ca lls in to its co n ta ct ce n te r, w h ic h co u ld lea d to a d e c r e a s e in m e m b e r satisfactio n . A lso, lik e all p lan s, B C BSN C w a s lo o k ­ in g to trim co s ts a sso cia ted w ith its c o n ta c t ce n te r.

as th e h e a lth re fo rm law m an d ates h e alth plans sp e n d a m in im u m o f 8 0 p e rc e n t o f all p rem iu m pay­ m en ts o n h e a lth ca re . T h is rule lea v es less m o n e y fo r ad m inistrative e x p e n s e s , lik e th e c o n ta c t ce n te r.

H o w ev er, B C B SN C saw a n o p p o rtu n ity to lev erag e its p a rtn ersh ip w ith N exidia, a lead ing p ro v id e r o f cu sto m e r in te ractio n an aly tics, a n d u se s p e e c h an aly tics to b e tte r u n d erstan d th e c a u se and d ep th o f m e m b e r co n fu sio n . T h e u s e o f s p e e c h an a ­ lytics w a s a m o re attractive o p tio n fo r B C B SN C th an a sk in g th e ir cu sto m e r serv ice p ro fe ssio n a ls to m o re th o ro u g h ly d o c u m e n t th e n atu re o f th e calls w ithin th e c o n ta c t c e n te r d e s k to p a p p lica tio n , w h ic h w o u ld h av e d e c r e a s e d e ffic ie n c y a n d in cre a se d c o n ta c t c e n te r ad m inistrative e x p e n s e s . B y id e n ­ tifying th e s p e c ific ro o t c a u se o f th e in teractio n s w h e n m e m b e rs ca lle d th e c o n ta c t c e n te r, B C BSN C w o u ld b e a b le t o ta k e co rrectiv e actio n s to re d u ce call v o lu m e s a n d co s ts an d im p ro v e th e m e m b e r s ’ e x p e rie n c e . -

A lle v ia tin g t h e C o n fu s io n

B C B SN C h a s b e e n a h e a d o f th e cu rv e o n e n g ag in g an d e d u catin g its m e m b ers an d p ro vid ing e x e m ­ plary cu sto m e r s erv ice. T h e h e a lth p lan k n e w it n e e d e d to w o rk vig o ro u sly to m ain tain its cu sto m e r

0Continued)

3 6 2 Part III • Predictive Analytics

Application Case 7.8 (Continued) serv ice tra ck re co rd as th e h e a lth ca re m an d ates b e g a n . T h e first ste p w a s to b e tte r u n d erstan d h o w m e m b ers p e rc e iv e d th e v alu e th e y re ce iv e d fro m B C B SN C a n d th eir o v erall o p in io n o f th e com p an y . T o a c c o m p lis h this, B C B SN C e le c te d to co n d u c t s e n ­ tim e n t a n aly sis to g e t rich e r insights in to m e m b ers’ o p in io n s a n d in teractio ns.

W h e n co n d u ctin g sen tim en t analysis, tw o strateg ies c a n b e u se d to g a rn e r results. T h e a co u stic m o d el re lie s o n m easu rin g s p e c ific ch aracteristics o f th e au d io, su ch a s so u n d , to n e o f v o ic e , p itch , vol­ u m e, in ten sity , and rate o f s p e e c h . T h e o th er strat­ e g y , u sed b y N exidia, is lingu istic m o d elin g , w h ich fo c u s e s d irectly o n s p o k e n sen tim en t. A co u stic m o d e lin g results in in a ccu ra te data b e c a u s e o f p o o r re co rd in g quality, b a ck g ro u n d n o ise, an d a p e rs o n ’s inability to c h a n g e to n e o r c a d e n c e to re flect his o r h e r e m o tio n . T h e lingu istic a p p ro a c h , w h ic h fo cu se s d irectly o n w o rd s o r p h rase s u se d to c o n v e y a fe e l­ ing, h a s p ro v e n to b e m o st effe ctiv e .

S in c e B C B SN C s u s p e cte d its m e m b ers m ay p e rc e iv e th e ir h e alth co v e ra g e as co n fu sin g , B C BSN C u tilized N exid ia to p u t to g e th e r stru ctured s e a r c h e s fo r w o rd s o r p h rases u se d b y ca llers to e x p r e s s c o n fu s io n : “I’m a little c o n fu s e d ," “I d o n t u n d e rsta n d ,” “I d o n ’t g e t it,” an d “D o e s n ’t m a k e s e n s e .” T h e results w e re th e e x a c t p e rc e n ta g e o f ca lls co n ta in in g this s en tim en t and h e lp e d BCBSN C s p e cifica lly iso late th o s e circu m sta n ces and co v e r­ a g e in sta n ce s w h e re callers w e r e m o re lik e ly to b e c o n fu s e d w ith a b e n e fit o r claim . B C BSN C filtered th e ir “c o n fu s io n ca lls ” fro m th e ir o v erall ca ll v o lu m e s o th e s e ca lls w e re a v ailab le fo r fu rth er analysis.

T h e n e x t ste p w a s to u s e s p e e c h analytics to g e t to th e ro o t c a u se o f w h at w a s driving the d is c o n n e c tio n and d e v e lo p strate g ies to alleviate th e c o n fu s io n . B C B SN C u s e d N exid ia’s d iction ary in d e p e n d e n t p h o n e tic in d e x in g an d s e a rc h solu tio n , allo w in g fo r all p ro c e s s e d au d io to b e s e a r c h e d for a n y w o rd o r p h rase, to cre a te ad d itional structured s e a r c h e s . T h e s e s e a r c h e s fu rther classified th e call drivers, a n d w h e n c o m b in e d w ith targ e te d listenin g, B C B SN C p in p o in te d th e p ro blem s.

T h e find ings re v e a le d th at literatu re c re a te d b y B C B SN C u se d industry' te rm s that m e m b ers w e re u n fam iliar w ith an d d id n ’t cle a rly e x p la in th eir b e n e ­ fits, cla im s p ro ce s s e s, an d d ed u ctib les. Additionally,

in form ation o n th e W e b site w a s n e ith e r easily lo c a te d n o r u n d e rsto o d , a n d m e m b ers w e re u n ab le to “se lf-se rv e ,” resultin g in u n n e ce s s a ry co n ta ct c e n te r in te ractio n . F u rther, ad d in g to B C B SN C ’s tro u b le s, w'hen N exid ia’s s p e e c h an alytics c o m b in e d th e u n stru ctu red ca ll data w ith th e stru ctu red data a sso cia ted w ith th e call, it sh o w e d “c o n fu s io n calls ’ had a sig n ifican tly h ig h e r av erag e talk tim e (A TT), resultin g in a h ig h e r c o s t to serv e fo r BCBSN C.

T h e R e s u lts

B y listen in g to , and m o re sp ecifically u n d erstan d ­ ing, th e c o n fu s io n o f its m e m b ers regard ing b en efits, BCBSN C b e g a n im p lem en tin g strategies to im prove m e m b e r co m m u n ica tio n a n d cu sto m e r e x p e rie n c e . T h e h e alth p la n h a s d ev elo p e d m o re reader-friendly literature an d sim plified th e layou t to highlight p e r­ tin e n t in form ation . B C BSN C a lso h a s im plem en ted W e b site re d esig n s to su p p ort e a sie r navigation and ed u catio n . As a result o f th e m o d ifications, BCBSN C p ro je cts a 10 to 25 p e rce n t d rop in “c o n fu sio n ca lls ,” resulting in a b e tte r cu sto m e r .service e x p e rie n c e and a lo w e r c o s t to serve. Utilizing N exid ia’s analytic so lu tio n to con tin u o u sly m o n ito r a n d track ch a n g e s will b e p aram ou n t to B C BSN C ’s co n tin u ed s u cce s s as a lead ing h e a lth plan.

“B e c a u s e th e re is s o m u ch to d o in h ealth care to d ay a n d b e c a u s e o f th e c h a n g e s u n d e r w a y in th e industry, y o u really w a n t to invest in th e co n su m e r e x p e rie n c e s o th at cu sto m ers c a n g e t th e m o st o u t o f th eir h e a lth c a re c o v e r a g e ,” says G re tch e n Gray, d irecto r o f C u sto m er an d C o n su m e r E x p e rie n c e at B C B SN C . “I b e lie v e th a t u n le ss y o u u se [N exidia’sj ap p ro a ch , I d o n ’t k n o w h o w y o u p ick y o u r priori­ ties an d fo cu s . S p e e c h an aly tics is o n e o f th e m ain to o ls w e hav e w h e re w e c a n say, ‘h e re is w h e re w e c a n h a v e th e m o st im p act an d h e r e ’s w h at I n e e d to d o b e tte r o r d ifferen tly to assist m y cu sto m e rs.’”

Q u e s t i o n s f o r D i s c u s s i o n

1. F o r a large com p an y like BCBSNC w ith a lot o f cus­ tom ers, w h at d o es “listening to cu stom er” mean?

2. W h at w e re th e c h a lle n g e s, th e p ro p o se d so lu ­ tion , a n d th e o b ta in e d results fo r BCBSNC?

Source: Used with permission from Nexidia.com.

Chapter 7 • T ex t Analytics, T e x t Mining, and Sentim ent Analysis 3 6 3

SECTION 7 1 0 REVIEW QUESTIONS

1 . W h a t is s p e e c h analytics? H o w d o e s it relate to sen tim en t analysis?

2 . D e s c r ib e th e a co u s tic a p p r o a c h to s p e e c h analytics. 3 . D e s c r ib e th e ling u istic a p p ro a ch to s p e e c h analytics.

Chapter Highlights • T e x t m in in g is th e d iscov ery o f k n o w le d g e from

u n stru ctu red (m o stly te x t-b a s e d ) data so u rce s. G iv e n th at a g reat d ea l o f in fo rm a tio n is m text form , te x t m in in g is o n e o f th e fastest grow ing b ra n c h e s o f th e b u sin e ss in te llig e n ce field.

• C o m p an ies u se te x t m ining a n d W e b m ining to b e tte r un derstand th eir cu stom ers b y analyzing th eir fe e d b a c k left o n W e b form s, b logs, and wikis.

• T e x t m in in g a p p lica tio n s are in virtually every a re a o f b u sin e ss and g o v ern m e n t, inclu d ing m ar­ k e tin g, fin a n c e , h e a lth ca re , m e d icin e , a n d h o m e ­

lan d secu rity. • T e x t m in in g u s e s natural la n g u a g e p ro ce ssin g to

in d u ce stru ctu re into th e te x t c o lle c tio n an d th e n u s e s data m in in g algorithm s s u ch as classificatio n , clu sterin g , a sso cia tio n , an d s e q u e n c e d is c o v e r y to e x tra ct k n o w le d g e fro m it.

• S u cc essfu l a p p lica tio n o f te x t m in in g req u ires a s tru ctu red m e th o d o lo g y sim ilar to th e CRISP-DM m e th o d o lo g y in d ata m ining.

. T e x t m in in g is clo s e ly related to in form ation e x tra ctio n , natu ral lan g u ag e p ro cessin g , an d d o c­ u m en t sum m arization.

• T e x t m in in g en tails creatin g n u m e ric in d ice s fro m u n stru ctu red te x t an d th e n ap p lyin g data m ining alg orith m s to th e s e indices.

• S e n tim en t c a n b e d efin e d as a s ettle d o p in io n re flectiv e o f o n e ’s feelin gs.

• S e n tim en t classificatio n u su ally d eals w ith dif­ feren tiatin g b e tw e e n tw o cla sse s, p o sitiv e and

n eg ativ e. . . • As a field o f re sea rch , s en tim en t analysis is clo se ly

re lated to com p u tatio n al lingu istics, natural

la n g u a g e p ro ce s s in g , a n d te x t m ining. It may b e u s e d to e n h a n c e s e a rch results p ro d u ce d b y

s e a rch e n g in e s. . S en tim en t an alysis is trying to an sw e r th e q u e s ­

tio n o f “W h a t d o p e o p le fe e l a b o u t a certain topic?" b y d igging in to o p in io n s o f m any u sin g a variety o f au to m ate d to ols.

• V o ic e o f th e cu sto m e r is an in tegral part o f an an alytic CRM a n d cu sto m e r e x p e r ie n c e m a n a g e ­ m e n t sy stem s, a n d is o fte n p o w e re d b y sen tim en t

analysis. • V o ic e o f th e m arket is a b o u t un derstand ing

ag g re g ate o p in io n s and tren d s at the m ark et

lev el. . . , • B ra n d m a n a g e m e n t fo c u s e s o n listen in g to so cial

m ed ia w h e r e a n y o n e c a n p o st o p in io n s th a t can d am ag e o r b o o s t y o u r rep u tatio n.

• P olarity id en tificatio n in sen tim en t an alysis is a cco m p lis h e d e ith er b y u sin g a le x ic o n as a refer­ e n c e lib rary o r b y u sin g a c o lle c tio n o f training

d o cu m en ts. • W ord N et is a p o p u la r g e n e ra l-p u rp o se le x ic o n

cre a te d at P rin c e to n U niversity. _ • SentiW ord N et is a n e x te n s io n o f W ord N et to b e

u s e d fo r s en tim en t id entification. • S p e e c h an aly tics is a g ro w in g field o f s c ie n c e that

allow s u s e rs to an alyze and e x tra ct in form ation fro m b o th liv e a n d re co rd e d co n v ersatio n s.

• T h e a c o u s tic a p p r o a c h to s en tim en t an a y- sis re lie s o n e x tra ctin g and m easu rin g a s p e ­ cific s e t o f featu res (e .g ., to n e o f v o ic e , p itch o r v o lu m e , intensity a n d rate o f s p e e c h ) o f th e

aud io.

Key Terms

a sso cia tio n classification clusterin g corp u s

cu sto m e r e x p e rie n c e m a n a g e m e n t (CEM )

d e c e p tio n d ete ctio n

inv erse d o cu m e n t fre q u e n cy

natural lan g u a g e p ro ce s s in g (NLP)

p a rt-o f-s p e e ch tagging polarity id entificatio n p o ly se m e sen tim en t

3 6 4 Part III • Predictive Analytics

sen tim en t an alysis SentiW ord N et s e q u e n c e d isco v e ry sin gu lar valu e

d e c o m p o sitio n (SV D )

s p e e c h analytics stem m ing sto p w o rd s te rm -d o c u m e n t m atrix

(TD M )

te x t m ining to k en izin g tre n d analysis un stru ctu red d ata v o ic e o f cu sto m e r (V O C )

v o ic e o f th e m ark et W ordN et

Questions fo r Discussion 1 . Explain th e relationships am on g data mining, text m in­

ing, and sen tim ent analysis. 2 . W hat should an organization con sid er b efo re m aking a

d ecision to purchase text mining software?' 3 . D iscuss th e d ifferences and com m onalities b e tw e e n text

m ining a n d sentim ent analysis. 4 . In you r o w n words, d efine tex t m in in g and discuss its

m ost popu lar applications. 5 . Discuss th e similarities and d ifferences b e tw e e n the data

m ining p ro c e ss (e .g ., CRISP-DM) and th e three-step, high-level text mining p rocess explained in this c h a p te i.

6. W hat d o es it m ean to introduce stm ctu re into the text- b a sed data? D iscuss the alternative ways o f introducing structure into text-based data.

7 . W hat is th e role o f natural language processing in text mining? D iscu ss the capabilities and limitations o f NLP in th e co n tex t o f text mining.

8 . List and discuss three prom inent application areas foi text m ining. W hat is the com m on them e am ong the th ree ap p lication areas y o u chose?

9 . W hat is sentim ent analysis? H ow d oes it relate to text mining?

1 0 . W hat are th e sou rces o f data for sentim ent analysis? 1 1 . W hat are th e com m on challeng es that sentim ent analysis

has to deal with? 1 2 . W hat are th e m ost popu lar application areas for senti­

m en t analysis? Why? 1 3 . How c a n sen tim ent analysis b e used for brand

management? 1 4 . W hat w ould b e th e e x p ec ted b en efits and beneiiciaries

o f sentim ent analysis in politics? 1 5 . How can sentim ent analysis b e used in predicting finan­

cial markets? 1 6 . W hat are th e m ain steps in carrying out sentim ent analy­

sis projects? 1 7 . W hat a re th e tw o com m o n m ethods for polarity iden­

tification? W h at is th e m ain d ifferen ce b e tw e e n the

two? 1 8 . D escrib e h ow special lexico n s are used in identification

o f sen tim ent polarity. 1 9 . W hat is s p e e c h analytics? H ow d oes it relate to sentim ent

analysis? 2 0 . D escrib e th e acou stic ap p roach to sp e ec h analytics. 2 1 . D escribe th e linguistic ap p roach to s p e e c h analytics.

Exercises Teradata University Network (TUN) and Other Hands-On Exercises 1 . Visit t e r a d a t a u n i v e r s i t y n e t w o r k . c o m . Identify cases

ab o u t tex t mining. D escrib e recen t d evelopm ents in the field. I f you can n o t find en o u g h cases at th e Teradata University netw ork W eb site, broad en your search to o th er W e b -b a se d resources.

2 . G o to t e r a d a t a u n i v e r s i t y n e t w o r k . c o m or locate white papers, W e b seminars, and other materials related to text mining. Synthesize your findings into a short written report.

3 . B row se th e W e b and you r library’s digital d atabases to identify' articles that m ake the natural linkage betw een text/W eb m ining and contem porary business intelligence

systems. 4 . G o to t e r a d a t a u n i v e r s i t y n e t w o r k . c o m and find a case

study n am ed “e B a y Analytics." R ead the c a se carefully, exten d you r understanding o f th e case by search in g the Internet for additional inform ation, and answ er the case questions.

5 . G o to t e r a d a t a u n i v e r s i t y n e t w o r k . c o m and find a sen­ tim ent analysis case nam ed “H ow D o W e F ix and App

Like That!” R ead th e description and follow the direc­ tions to d ow nload th e data and th e to o l to carry ou t the exercise.

Team Assignments and Role-Playing Projects 1 . Exam ine h o w textual data c a n b e captured autom atically

using W eb -b ased tech n olog ies. O n c e captured, w hat are th e potential patterns that you can extract from these unstructured data sources?

2 . Interview administrators in you r c o lle g e o r execu tives in you r organization to determ ine h o w te x t m ining and W eb m ining co u ld assist them in their w ork. W rite a proposal d escribing your findings. Include a preliminary c o s t - b en efits analysis in you r report.

3 . G o to y o u r library’s onlin e resources. Learn h ow to d ow nload attributes o f a c ollectio n o f literature (journal articles) in a specific topic. D ow nload and p rocess the data using a m ethod ology similar to th e o n e explained in A pplication Case 7.5.

4 . Find a readily available sentiment text data set (see Tech nology Insights 7.2 for a list o f popular data sets) and

Chapter 7 • T ex t Analytics, T e x t Mining, and Sentim ent Analysis 3 6 5

download it into your computer. If you have an analytics tool that is capable o f text mining, use that; if not, download RapidMiner (rapid-i.com) and install it. Also install the text analytics add-on for RapidMiner. Process the downloaded data using your text mining tool (i.e., convert the data into a structured form). Build models and assess the sentiment detection accuracy of several classification models (e.g., support vector machines, decision trees, neural networks, logistic regression, etc.). Write a detailed report where you explain your finings and your experiences.

Internet Exercises 1 . Survey some text mining tools and vendors. Start with

clearforest.com and megaputer.com. Also consult with dmreview.com and identify some text mining products and service providers that are not mentioned in this chapter.

2. Find recent cases o f successful text mining and Web mining applications. Try text and Web mining software vendors and consultancy firms and look for cases or suc­ cess stories. Prepare a report summarizing five new case studies.

3. Go to statsoft.com. Select Downloads and download at least three white papers on applications. Which of these applications may have used the data/text/Web mining techniques discussed in this chapter?

4 . G o to sas.com . Download at least three white papers on applications. Which o f these applications may have used the data/text/Web mining techniques discussed in this chapter?

5. G o to ibm.com. Download at least three white papers on applications. Which o f these applications may have used the data/text/Web mining techniques discussed in this chapter?

6. Go to teradata.com . Download at least three white papers on applications. Which of these applications may have used the data/text/Web mining techniques dis­ cussed in this chapter?

7 . G o to fairisaac.com. Download at least three white papers on applications. Which o f these applications may have used the data/text/Web mining techniques dis­ cussed in this chapter?

8. G o to salfordsystems.com. Download at least three white papers on applications. Which o f these applica­ tions may have used the data/text/Web mining tech­ niques discussed in this chapter?

9 . G o to clarabridge.com. Download at least three white papers on applications. Which o f these applications may have used text mining in a creative way?

10. G o to kdnuggets.com. Explore the sections on applica­ tions as well as software. Find names o f at least three additional packages for data mining and text mining.

End-of-Chapter Application Case

B B V A Sea m lessly M onito rs and Im proves its Online Reputation

BBVA is a global group that offers individual and corpo­ rate customers a comprehensive range o f financial and non- financial products and services. It enjoys a solid leadership position in the Spanish market, where it first began its activi­ ties over 150 years ago. It also has a leading franchise in South America; it is the largest financial institution in Mexico; one of the 15 largest U.S. commercial banks and one o f the few large international groups operating in China and Turkey. BBVA employs approximately 104,000 people in over 30 countries around the world, and has more than 47 million customers and 900,000 shareholders.

L o o k in g f o r t o o l s t o r e d u c e r e p u ta tio n a l r is k s BBVA is interested in knowing what existing clients— and possible new ones— think about it through social media. Therefore, the bank has implemented an automated con­ sumer insight solution to monitor and measure the impact o f brand perception online— whether this b e customer com­ ments on social media sites (Twitter, Facebook, forums, blogs, etc.), the voices o f experts in online articles about BBVA and its competitors, or references to BBVA on news sites— to detect possible risks to its reputation or to possible business opportunities.

Insights derived from this analytical tool give BBVA the opportunity to address reputational challenges and continue to build on positive opinions. For example, the bank can now respond to negative (or positive) brand perception by focus­ ing its communication strategies on particular Internet sites, countering— or backing up— the most outspoken authors on Twitter, boards and blogs.

F in d in g a w a y f o r w a r d In 2009, BBVA began monitoring the w eb with an IBM social media research asset called Corporate Brand Reputation Analysis (COBRA), as a pilot betw een IBM and the bank’s Innovation department. This pilot proved highly successful for different areas o f the bank, including the Communications, Brand & Reputation, Corporate Social Responsibility, Consumer Insight, and Online Banking departments.

The BBVA Communication department then decided to tackle a new project, deploying a single tool that would enable the entire group to analyze online mentions o f BBVA and monitor the bank’s brand perception in various online communities.

The bank decided to implement IBM Cognos Consumer Insight to unify all its branches worldwide and allow them to

3 6 6 Part III • Predictive Analytics

use the same samples, models, and taxonomies. IBM Global Business Services is currently helping the bank to imple­ ment the solution, as well as design the focus of the analysis adapted to each country’s requirements.

IBM Cognos Consumer Insight will allow BBVA to monitor the voices o f current and potential clients on social media websites such as Twitter, Facebook and message boards, identify expert opinions about BBVA and its compet­ itors on blogs, and control the presence o f the bank in news channels to gain insights and detect possible reputational risks. All this new information will be distributed among the business departments o f BBVA, enabling the bank to take a holistic view across all areas o f its business.

S e a m le s s f o c u s o n o n lin e r e p u t a t i o n The solution has now been rolled out in Spain, and BBVA s Online Communications team is already seeing its benefits.

“Huge amounts o f data are being posted on Twitter every day, which makes it a great source o f information for us,” states the Online Communications Department o f this bank. "To make effective use o f this resource, wre needed to find a way to capture, store and analyze the data in a better, faster and more detailed fashion. W e believe that IBM Cognos Consumer Insight will help us to differentiate and categorize all the data w e collect according to pre- established criteria, such as author, date, country and sub­ ject. This enables us to focus only on comments and news items that are actually relevant, whether in a positive, nega­ tive or neutral sen se.”

The content o f the comments is subsequently analyzed using custom Spanish and English dictionaries, in order to identify whether the sentiments expressed are positive or negative. “What is great about this solution is that it helps us to focus our actions on the most important topics o f online discussions and immediately plan the correct and most suit­ able reaction,” adds the Department, “By building on what w e accomplished in the initial COBRA project, the new- solu­ tion enables BBVA to seamlessly monitor comments and postings, improve its decision-making processes, and thereby strengthen its online reputation.”

“W hen BBVA detects a negative comment, a reputa­ tional risk arises,” explains Miguel Iza Moreno, Business Analytics and Optimization Consultant at IBM Global Business Seivices. “Cognos Consumer Insight provides a reporting system which identifies the origin o f a negative statement and BBVA sets up an internal protocol to decide how to react. This can happen through press releases, direct

communication with users or, in some cases, no action is deemed to b e required; the solution also highlights those cases in w'hich the negative comment is considered ‘irrele­ vant’ or ‘harmless’. The same procedure applies to positive comments— the solution allows the bank to follow a standard and structured process, which, based on positive insights, en­ ables it to strengthen its reputation.

“Following the successful deployment in Spain, BBVA will b e able to easily replicate the Cognos Consumer Insight solution in other countries, providing a single solution that will help to consolidate and reaffirm the bank’s reputation management strategy,” says the Department.

T a n g ib le R e s u lts Starting with the COBRA pilot project, the solution delivered visible benefits during the first half o f 2011. Positive feed­ back about the company increased by more than one percent while negative feedback was reduced by 1.5 p e r c e n t - suggesting that hundreds o f customers and stakeholders across Spain are already enjoying a more satisfying experi­ ence from BBVA. Moreover, global monitoring improved, providing greater reliability when comparing results between branches and countries. Similar benefits are expected from the Cognos Consumer Insight project, and the initial results are expected shortly.

“BBVA is already seeing a remarkable improvement in the way that information is gathered and analyzed, which w e are sure will translate into the same kind o f tan­ gible benefits we saw from the COBRA pilot project, states the bank, “For the time being, we have already achieved what w e need ed the most: a single tool which unifies the online measuring o f our business strategies, enabling more detailed, structured and controlled online data analysis.”

Q u e s t i o n s f o r t h e E n d - o f - C h a p t e r

A p p l i c a t i o n C a s e 1 . How did BBVA use text mining? 2 . What were BBVA’s challenges? I low did BBVA over­

com e them with text mining and social media analysis? 3 . In what other areas, in your opinion, can BBVA use

text mining?

Source: IBM Customer Success Story, “BBVA seamlessly monitors and improves its online reputation” at http://w w w -01.ibm .com / so ftw a re /success/cssdb.nsf/CS/STRD-8NUD29?OpenDocumen t&Site=corp&cty=en_us (accessed August 2013).

References Chun, H. W., Y. Tsuruoka, J. D. Kim, R. Shiba, N. Nagata, and

T. Hishiki. (2006). “Extraction o f Gene-Disease Relations from Medline Using Domain Dictionaries and Machine Learning.” Proceedings o f the 11th P acific Symposium on Biocomputing, pp. 4-15-

Cohen, K. B., and L. Hunter. (2008). “Getting Started in Text Mining.” PLoS Compututional Biology, Vol. 4, No. I, pp. 1-10.

Coussement, K., and D. Van Den Poel. (2008). “Improving Customer Complaint Management by Automatic Email

Chapter 7 * T e x t Analytics, T e x t Mining, and Sentim ent Analysis 367

Classification Using Linguistic Style Features as P redictors.” D e c is io n S u p p o r t System s, Vol. 44, No. 4, pp. 8 7 0 -8 8 2 .

Coussem ent, K ., and D. V an D en P oel. (2009)- “Im proving C ustom er Attrition Prediction b y Integrating Em otions from Client/Company In teraction Emails and Evaluating Multiple Classifiers.” E x p ert S ystem s w ith A p p lic a tio n s, Vol. 36, No. 3, pp. 6 1 2 7 -6 1 3 4 .

D elen, D., and M. Crossland. (2008). “Seed ing the Survey and Analysis o f R esearch Literature with T ex t Mining.” E xpert S ystem s w ith A p p lic a tio n s, Vol. 34, No. 3, pp. 1 7 0 7 -1 7 2 0 .

Etzioni,0 .( 1 9 9 6 ) . “T h e World W ide W e b : Q uagm ireor GoldMine?” C o m m u n ic a t io n s o f t h e ACM, Vol. 39, No. 11, pp. 6 5 -6 8 .

EUROPOL. (2 0 0 7 ). “EUROPOL W ork Program 2007." s t a t e w a t c h . o r g / n e w s / 2006/ a p r / e u r o p o l - w o r k - p r o - g ram m e-2007.pdf (a c ce s se d O cto b e r 2008).

Feldm an, R., a n d j. Sanger. (2 0 0 7 ). T he T ext M in in g H a n d b o o k : A d v a n c e d A p p r o a c h e s in A n a ly z in g U n stru ctu red D ata . B oston : ABS Ventures.

Fuller, C. M., D . Biros, and D. D elen . (2 0 0 8 ). “Exploration o f Featu re S electio n and Advanced Classification Models for H igh-Stakes D ecep tion D etection .” P r o c e e d in g s o f th e 4 1 st A n n u a l H a w a i i I n t e r n a t io n a l C o n fe r e n c e o n System S c ie n c e s (1IICSS), B ig Island, HI: IEEE Press, pp. 8 0 -9 9 -

Ghani, R., K. P robst, Y . Liu, M. Krema, and A. Fano. (2006). “T e x t M ining for Product Attribute E xtraction.” SIGKDD E x p lo r a tio n s , Vol. 8, No. 1, pp. 4 1 -4 8 .

Grim es, S. (2 0 1 1 , February 17). “Seven Breakthrough Sentim ent Analysis Scen arios.” In fo r m a t io n W e e k .

Han, J ., and M. Kam ber. (2 0 0 6 ). D a t a M in in g : C o n c e p ts a n d T ec h n iq u es , 2nd ed. San Francisco: M organ Kaufmann.

Kanayama, H ., and T. Nasukawa. (2 0 0 6 ). “Fully Automatic L exicon Expanding for D om ain-oriented Sentim ent Analysis, EMNLP: Empirical M ethods in Natural Language P rocessing .” t r l .i b m .c o m /p r o j e c t s / t e x t m i n i n g /t a k m i / s e n tim e n t_ a n a ly s is _ e .h tm .

Kleinberg, J . (1 9 9 9 ). “Authoritative Sources in a Hyperlinked Environm ent.” f o u m a l o f t h e ACM, Vol. 46 , No. 5, pp. 6 0 4 -6 3 2 .

Lin, J . , and D. D em ner-Fushm an. (2 0 0 5 ). “‘B ag o f W ords’ Is Not Enough for Strength o f E vid en ce Classification.” AM1A A n n u a l S y m p o s iu m P r o c e e d in g s , pp. 10 3 1 -1 0 3 2 . p u b m ed - c e n tr a l.n ih .g o v /a r tic le r e n d e r .f c g i? a r tid = 1 5 6 0 8 9 7 .

Mahgoub, H., D. Rosner, N. Ismail, and F. T orkey. (2008). “A T ex t M ining T ech n iq u e U sing A ssociation Rules Extraction.” I n t e r n a t i o n a l f o u m a l o f C o m p u t a t io n a l In te llig e n c e , Vol. 4 , No. 1, pp. 2 1 -2 8 .

Manning, C. D ., and H. Schutze. (1 9 9 9 ). F o u n d a t io n s o f S ta tis tic a l N a t u r a l L a n g u a g e P r o c e s s in g . Cambridge, MA: MIT Press.

Masand, B . M., M. Spiliopoulou, J . Srivastava, and O. R. Za'iane. (2 0 0 2 ). “W e b Mining for Usage Patterns and Profiles.” SIGKDD E x p lo ra tio n s , Vol. 4, No. 2, pp. 1 2 5 -1 3 2 .

McKnight, W . ( 2 0 0 5 J a n u a r y 1). “T e x t Data Mining in B usiness Intelligence.” Information Management Magazine. in f o r m a tio n -m a n a g e m e n t.c o m /is s u e s /2 0 0 5 0 1 0 1 / 1 0 1 6 4 8 7 -l.h tm l (accessed May 22, 2009).

M ejova, Y . (2 0 0 9 ). “Sentim ent Analysis: An O verview .” C om prehensive ex a m paper. www.cs.uiowa. ed u /~ym ejova/publications/C om psYelenaM ejova. p d f (accessed February 2013).

Miller, T. W . (2 0 0 5 ). D a t a a n d Text M in in g : A B u s in e s s A p p lic a tio n s A p p r o a c h . U pper Saddle River, NJ: Prentice Hall.

Nakov, P., A. Schw artz, B . W olf, and M. A. Hearst. (2005). “Supporting Annotation Layers for Natural Language P rocessing .” P r o c e e d in g s o f t h e ACL, interactive p oster and dem onstration sessions, Ann Arbor, ML Association for Com putational Linguistics, pp. 6 5 -6 8 .

Nasraoui, O ., M. Spiliopoulou, J . Srivastava, B . M obasher, and B. Masand. (2 0 0 6 ). “W ebK D D 2006: W eb Mining and W eb U sag e Analysis Post-W orkshop Report.” A C M SIGKDD E x p lo r a tio n s N ew sletter, Vol. 8, No. 2, pp. 8 4 -8 9 .

Nexidia (2009). "State o f the art: Sentiment analysis” Nexidia White Paper, http://nexidia.com/files/resource_files/nexidia_ sentim ent_analysis_wp_8269.pdf (accessed February 2013).

Pang, B ., and L. Lee. (2 0 0 8 ). “O pinion M ining and Sentim ent Analysis.” N ow Pub. http://books.google.com .

P eterson, F.. T . (2 0 0 8 ). “T h e V oice o f Custom er: Qualitative Data as a Critical Input to W eb Site O ptim ization.” fore- s e e r e s u lts .c o m /F o r m _ E p e te r s o n _ W e b A n a ly tic s .h tm l (a c cessed May 22, 2009).

Shatkay, H., A. Ilo g lu n d , S. Brady, T. Blum , P. D on n es, and O. Kohlbacher. (2 0 0 7 ). “S h erL o c High-Accuracy Prediction o f Protein Su bcellular Localization by Integrating T ex t and Protein S e q u e n c e D ata.” B io in fo r m a t ic s , Vol. 23, No. 11, pp. 1 4 1 0 -1 4 1 7 .

SPSS. “M erck Sharp & D o h m e.” spss.com /success/tem - plate_view.cfm?Story_ID=185 (a c ce s se d May 15, 2009).

StatSoft. (2009)- S ta tis tic a D a t a a n d T ext M in e r U ser M a n u a l. Tulsa, OK: StatSoft, Inc.

Turetken, O ., and R. Sharda. (2 0 0 4 ). “D evelopm ent o f a Fish eye-B ased Inform ation Search Processing Aid (FISPA) for M anaging Inform ation Overload in th e W eb Environm ent.” D e c is io n S u p p o r t System s, Vol. 37 , No. 3, pp. 4 1 5 ^ 3 4 .

W eng, S, S., and C. K. Liu. (2 0 0 4 ) “U sing T ex t Classification and Multiple C on cepts to Answer E-Mails.” E x p e r t System s w ith A p p lic a tio n s , Vol. 26, No. 4, pp. 5 2 9 -5 4 3 .

Zhou, Y ., E. Reid, J . Q in, H. Chen, and G. Lai. (2005). “U.S. D o m estic Extrem ist G roups o n the W eb : Link and C ontent Analysis.” IEEE In te llig e n t System s, Vol. 20, No. 5, pp. 4 4 -5 1 .

Web Analytics, Web Mining, and Social Analytics

LEARNING OBJECTIVES

m D e fin e W eb m in in g a n d u n d erstan d its tax o n o m y a n d its a p p lica tio n areas

■ D ifferen tiate b e tw e e n W e b co n ten t m in in g and W e b stru cture m ining

8 U n d erstand th e internals o f W e b sea rch e n g in e s

* Learn th e d etails a b o u t s e a rch e n g in e o p tim ization

* D e fin e W eb u sag e m in in g an d le a rn its b u sin e ss ap p licatio n

* D e s c r ib e th e W e b an alytics m aturity m o d el an d its u se cases

■ U n d erstand so cia l n e tw o rk s and social analytics an d th e ir p ractical ap p licatio n s

■ D e fin e s o c ia l n etw ork a n a ly sis and b e c o m e fam iliar w ith its ap p licatio n areas

■ U n d erstan d so cia l m ed ia an alytics and its u s e fo r b e tte r cu sto m e r en g ag em en t

- h is ch a p te r is all a b o u t W e b m in in g a n d its a p p lica tio n areas. As you will see. | W e b m in in g is o n e o f th e fastest g ro w in g te c h n o lo g ie s in b u sin e ss intellig ence

JtL and b u s in e s s analytics. U n d e r th e u m b re lla o f W e b m ining, in this ch ap ter, w e w ill c o v e r W e b analytics, sea rch e n g in e s , s o c ia l an alytics a n d th e ir e n a b lin g m e th o d s algorithm s, an d te ch n o lo g ie s.

8 .1 O P E N IN G V IG N E T T E : S e c u rity F irst I n s u r a n c e D e e p e n s C o n n e c tio n w ith P o lic y h o ld e r s 3 6 9

8 .2 W e b M in in g O v e rv ie w 3 7 1 8 . 3 W e b C o n te n t a n d W e b S tru c tu re M in in g 3 7 4 8 . 4 S e a r c h E n g in e s 3 7 7 8 .5 S e a r c h E n g in e O p tim iz a tio n 3 8 4 8 . 6 W e b U sa g e M in in g (W e b A n a ly tic s ) 3 8 8 8 . 7 W e b A n a ly tic s M atu rity M o d e l a n d W e b A n a ly tic s T o o ls 3 9 6

Chapter 8 • W e b Analytics, W e b Mining, and Social Analytics 369

8 . 8 S o c ia l A n a ly tic s a n d S o c ia l N e tw o rk A n a ly sis 4 0 3

8 . 9 S o c ia l M e d ia D e f in itio n s a n d C o n c e p ts 4 0 7 8 . 1 0 S o c ia l M e d ia A n a ly tics 4 1 0

8.1 OPENING VIGNETTE: Security First Insurance Deepens Connection with Policyholders

S ecu rity First In s u ra n ce is o n e o f th e larg est h o m e o w n e rs ’ in su ra n ce c o m p a n ie s in Florida. H ead q u artered in O rm o n d B e a c h , it em p lo y s m o re th a n 8 0 in su ra n ce p ro fe ssio n a ls to serve its n e arly 1 9 0 ,0 0 0 cu sto m ers.

CHALLENGE

B eing T here fo r C ustom ers Storm A fter Storm , Y e a r A fter Y e a r

Florida h a s m o re p ro p e rty and p e o p le e x p o s e d t o h u rrican es th an an y state in th e cou ntry. E a c h y ear, th e A tlantic O c e a n av e rag e s 12 n a m e d storm s and n in e n a m e d h u rrican es. Secu rity First is o n e o f a fe w Florida h o m e o w n e rs ’ in su ran ce co m p a n ie s th a t h a s the fin an cial stren g th t o w ithstand m u ltip le natural d isasters. “O n e o f o u r p ro m ise s is to b e th e re fo r o u r cu sto m e rs, storm a fte r storm , y e ar a fte r y e a r,” says W e rn e r K ru ck , c h ie f o p eratin g o ffic e r fo r S ecu rity First.

D uring a ty p ical m o n th , S ecu rity First p ro c e s s e s 7 0 0 claim s. H o w ev er, in th e after- m ath o f a h u rrica n e, th a t n u m b e r c a n sw e ll to te n s o f th o u san d s w ithin days. It c a n b e a ch a lle n g e fo r th e c o m p a n y to q u ick ly s c a le u p to h an d le th e influx o f cu sto m e rs trying to file p o st-sto rm in su ra n ce claim s fo r d am ag ed p ro p e rty an d p o ss e s sio n s . In th e past, cu stom ers su b m itte d claim s prim arily b y p h o n e a n d so m e tim e s e m ail. T o d ay , p o licy h o ld ­ ers u s e a n y m e a n s av a ila b le to c o n n e c t w ith a n a g e n t o r claim s rep resen tativ e, in clu d in g p o stin g a q u e stio n o r co m m e n t o n th e co m p a n y ’s F a c e b o o k p a g e o r T w itte r a cco u n t.

A lthough Secu rity First p ro v id es o n g o in g m o n ito rin g o f its F a c e b o o k an d Tw itter acco u n ts, as w e ll as its m ultiple em ail ad d resses an d call ce n te rs, th e co m p a n y k n e w that the co m m u n icatio n v o lu m e after a m ajo r storm req u ired a m o re aggressive ap p ro ach . “W e w e re co n c e rn e d th a t if a m assive n u m b e r o f cu stom ers co n ta cte d u s th ro u g h em ail o r social m edia after a hu rrican e, w e w o u ld b e u n a b le to re sp o n d q u ick ly a n d appropriately, K ruck says. “W e n e e d t o b e available to o u r cu stom ers in w h atev er w ay th ey w an t to co n ta ct u s.” In addition, Secu rity First re co g n ize d th e n e e d to integrate its so cial m ed ia re s p o n s e s into the claim s p ro ce s s an d d o cu m en t th o se re sp o n se s to co m p ly w ith industry regulations.

SOLUTION

Providing R esponsive Service No M atter How C ustom ers Get in Touch

Security First c o n ta c te d IB M B u s in e s s P artner In tegritie fo r h e lp w ith h a rn e ssin g so cial m ed ia to im p ro ve th e cu sto m e r e x p e rie n c e . In tegritie co n fig u re d a solu tion built o n k e y IB M E n te rp rise C o n ten t M an ag em en t so ftw are co m p o n e n ts , featu ring IB M C ontent A nalytics w ith E n te rp rise S e arch , IB M C o n te n t C o lle cto r fo r E m ail and IB M ® F ileN et® C o n ten t M an ag er so ftw are . C alled S o cia l M ed ia C ap ture (SM C 4), th e In te g ritie so lu tio n o ffers fo u r critical ca p a b ilitie s fo r m an agin g so cia l m ed ia platform s: cap tu re , c o n tro l, c o m ­ p lian ce and co m m u n ica tio n . F o r e x a m p le , th e SM C4 so lu tio n lo g s all s o c ia l n e tw o rk in g in teractio n fo r S ecu rity First, ca p tu re s co n ten t, m o n ito rs in co m in g an d o u tg o in g m e ssag es and arch iv e s all co m m u n ica tio n fo r co m p lia n c e review .

B e c a u s e th e s o lu tio n u se s o p e n IB M E n terp rise C o n te n t M an ag em en t softw are, Security First c a n e a sily lin k it to critical co m p a n y ap p licatio n s, d a ta b a se s a n d p ro ce s s e s.

370 Part III • Predictive Analytics

F o r e x a m p le , C o n te n t C o lle cto r fo r E m ail s o ftw a re au tom atically cap tu res em ail co n ten t a n d attach m en ts a n d sen d s a n e m a il b a c k to th e p o licy h o ld e r ack n o w le d g in g rece ip t. In ad d ition, C o n te n t A nalytics w ith E n terp rise S e a rc h so ftw are sifts throu gh an d an aly zes th e c o n te n t o f cu sto m e rs’ p o sts an d em ails. T h e so ftw a re th e n ca p tu re s in form ation g lean ed fro m this an alysis d irectly into claim s d o cu m e n ts to b e g in th e claim s p ro ce ss. Virtually all in co m in g co m m u n ica tio n fro m th e c o m p a n y ’s w e b , th e In te rn e t an d em ails is p u led in to a ce n tra l FileN et C o n ten t M an ager so ftw a re re p o sito ry to m aintain, co n tro l an d link to th e ap p rop riate w o rk flo w . “W e c a n b rin g th e cu sto m e r co n v e rs a tio n and any pictures an d atta ch m en ts in to o u r p o licy a n d claim s m a n a g e m e n t system and u se it to trigger our claim s p ro ce s s a n d ad d to o u r d o cu m e n ta tio n ,” says K ruck.

Prioritizing Com m unications w ith A ccess to Sm arter Content

People w hose hom es have b ee n damaged or destroyed by a hurricane are often displaced qu ickly, w ith little m o re th an th e clo th e s o n th eir b a ck s. G ra b b in g a n in su ran ce policy o n the w av ou t th e d o o r is o fte n a n afterthough t. T h e y ’re relying o n th eir in su ran ce co m ­ pan ies to h av e th e inform ation th e y n e e d to h e lp th e m g e t th eir liv es b a c k in ord er as q u ickly as p o ssib le. W h e n te n s o f th o u san d s o f p o licy h o ld ers req u ire a ssistan ce w ithin a sh o rt p e rio d o f tim e, Secu rity First m ust triage re q u e sts q u ickly. T h e C o ntent A nalytics w ith E n terp rise Se arch softw are that a n ch o rs th e SM C4 so lu tio n p ro v id es th e inform ation n e c ­ essary to h e lp th e co m p a n y identify and ad d ress th e m o st u rg en t c a s e s first. T h e softw are au tom atically sifts throu gh d ata in em ail an d so cia l m ed ia p o sts, tw eets a n d co m m en ts using te x t m ining, te x t analytics, natu ral lan g u ag e p ro ce s s in g and sen tim en t analytics to d etect w o rd s an d to n e s th at identify sig n ifican t p ro p erty d am ag e o r th at co n v e y distress. Secu rity First c a n th e n prioritize th e m e ssa g es and ro u te th e m to th e p ro p e r p e rso n n e l to pro v id e re assu ran ce , h an d le co m p lain ts o r p ro c e s s a claim . “W ith a c c e s s to sm arter c o n ­ tent, w e ca n re sp o n d to o u r cu sto m ers in a m o re rapid, e fficie n t an d p e rso n a liz e d way, says K ruck. “W h e n cu sto m ers are having a bad e x p e rie n c e , it’s really im portant to g e t to th e m q u ick ly w ith th e lev el o f a ssista n ce ap p ro p riate to th eir p articu lar situ ation s.”

RESULTS

Successfully A ddressing Potential Com pliance Issu es

C o m p an ie s in all industries m u st stay co m p lia n t w ith n e w a n d e m e rg in g regulatory re q u irem en ts regard ing so cial m ed ia. T h e te x t analysis cap ab ilitie s provid ed m th e IBM so ftw a re h e lp Secu rity First filter in a p p ro p ria te in co m in g co m m u n ica tio n s and aud it ou t­ b o u n d co m m u n ica tio n s, avoid ing p o ten tial issu es w ith m e ssag e co n ten t. T h e co m p a n y ca n b e co n fid e n t that th e re s p o n s e s its e m p lo y e e s pro v id e are co m p lian t an d con trolled b a s e d o n b o th Secu rity First p o licie s and indu stry regulations.

Secu rity First c a n d esig n ate p e o p le o r ro le s in th e o rg an izatio n that are authorized to cre a te and su b m it re sp o n se s. T h e sy stem au to m atically v e rifies th e s e d esig n atio n s and an aly zes o u tg o in g m e ssa g e co n ten t, sto p p in g an y in effectiv e o r q u e s tio n a b le co m m u n i­ ca tio n s fo r fu rther review . “E verything is re co rd e d fo r co m p lia n ce , s o w e c a n effectiv ely tra ck and m aintain th e p ro ce ss. W e h a v e th e ab ility to co n tro l w h ich e m p lo y e e s resp o n d , th e ir lev el o f au thority a n d th e c o n te n t o f th e ir r e s p o n s e s ,” says K ruck.

T h e s e capabilities give Security First th e co n fid e n ce to exp an d its use o f social media. B e c a u s e com p lian ce is cov ered , th e co m p an y ca n fo cu s o n additional opportunities fo r direct dialog w ith custom ers. B e fo re this solution, Security First filtered cu stom er com m unications through agents. N ow it c a n reach ou t to cu stom ers directly an d proactively as a c o m p a n y

“W e ’re o n e o f the first in su ra n ce c o m p a n ie s in Florida to m a k e o u rselv e s available to cu sto m ers w h e n e v e r, w h e re v e r an d h o w e v e r th ey c h o o s e to co m m u n ica te . W e ’re also m an agin g internal p ro c e s s e s m o re e ffe ctiv e ly a n d p ro activ ely , re a ch in g o u t to cu stom ers in a co n tro lle d a n d co m p lian t m an n e r,” says K ruck.

Chapter 8 • W e b Analytics, W e b Mining, and Social Analytics 371

S o m e o f th e p rev ailin g b u sin e ss b e n e fits o f cre ativ e u se o f W e b a n d s o c ia l analytics

inclu de:

• T u rn s so cia l m ed ia in to an a c tio n a b le co m m u n ica tio n s c h a n n e l d u rin g a m ajo r

d isaster . . • S p e e d s cla im s p ro c e s s e s b y initiating claim s w ith in fom ration fro m e m a il an d so cial

m ed ia p o sts • Facilitates prioritizing u rg en t c a s e s b y analy zing s o cia l m ed ia co n te n t fo r sen tim en ts • H elp s e n s u re c o m p lia n ce b y au tom atically d o cu m en tin g s o cial m ed ia co m m u n ica tio n s

QUESTIONS FO R TH E OPENING VIGNETTE

1 . W h a t d o e s S ecu rity First do? 2 . W h at w e re th e m ain ch a lle n g e s Secu rity First w a s facing? 3 . W h at w as th e p ro p o se d so lu tio n ap p roach ? W h at ty p e s o f an alytics w e re in teg rated

in th e solution? 4 . B a s e d o n w h a t y o u le a rn fro m th e vig n ette, w h at d o y o u th in k are th e re latio n sh ip s

b e tw e e n W e b a nalytics, te x t m ining, and sen tim en t analysis? 5 . W h a t w e re th e results Security First ob tain ed ? W ere an y surprising b e n e fits realized?

WHAT W E CAN LEARN FROM THIS VIGNETTE

W e b analytics is b e c o m in g a w a y o f life fo r m an y b u sin e sse s, e sp ecia lly th e o n e s that are directly fa cin g th e co n su m e rs. C o m p a n ie s are e x p e c te d to find n e w an d in n o v ativ e w ay s :o c o n n e c t w ith th e ir cu sto m e rs, u n d erstan d th e ir n e e d s , w an ts, a n d o p in io n s, a n d p ro a c­ tively d ev elo p p ro d u cts an d s erv ices th at fit w e ll w ith th em . Tn this d ay an d a g e , ask in g cu sto m ers to te ll y o u e x a c tly w h at th e y lik e an d d islike is n o t a v iab le o p tio n . Instead , b u sin e sse s are e x p e c te d to d ed u ce th at inform ation b y ap p lyin g ad v an ce d an aly tics to o ls -o in valu ab le d ata g e n erated o n th e In tern et an d so cia l m ed ia sites (a lo n g w ith co rp o ra te d atab ases) S ecu rity First rea lized th e n e e d to re v o lu tio n ize th e ir b u s in e s s p r o c e s s e s to b e m ore e ffe ctiv e an d e fficie n t in th e w a y that th e y d eal w ith th e ir cu sto m ers an d cu sto m e r claim s. T h e y n o t o n ly u sed w h at th e In tern et an d s o cia l m ed ia have to o ffe r, b u t also :a p p e d into th e cu sto m e r ca ll record s/ record ings a n d o th er relev an t tra n sa ctio n d ata­ b ases. T h is v ig n e tte illustrates th e fact that an alytics te c h n o lo g ie s a re a d v a n ce d e n o u g h ro bring to g e th e r m an y d iffe ren t d ata so u rce s to cre a te a h o listic v iew o f th e cu sto m er. \nd that is p e rh a p s th e g re ate st s u c c e s s crite rio n fo r to d ay ’s b u s in e s s e s . In th e fo llo w in g ■sections, y o u w ill le a rn a b o u t m an y o f th e W e b -b a s e d analy tical te c h n iq u e s th at m a k e it

i l l h ap p en .

I J u rce: IBM Customer Success Story, “Security First Insurance deepens connection with policyholders" accessed h t t p :/ / w w w - 0 1 .i b m . c o m / s o f t w a r e / s u c c e s s / c s s d b . n s f / C S / S A K G - 9 7 5 H 4 N ? O p e n D o c u m e n t & S i t e = d e f

i a i l t & c t y = e n _ u s (accessed August 2 0 1 3 )-

8.2 W E B M IN IN G O V E R V IE W The Internet has forever ch an g e d th e landscape o f b u sin ess as w e kn o w it. B e c a u s e o f the highly co n n ecte d , flattened w orld an d b ro ad en ed com petitive field, today’s co m p an ies are xicreasingly facing greater opportunities (b ein g ab le to reach custom ers and m arkets that - e v m ay have n e v e r thought p o ssib le) and b igger challenge (a globalized a n d ever-changing : :nmpetitive m arketplace). O n e s w ith th e vision and capabilities to d eal w ith su ch a volatile

X S n t e m e t T h e y are n o t only buying products and

h a s m a d e data c r e a u o ,

data co lle ctio n , and data/inform ation/opinion & S w * M a y s m ufacturing/ sh ip p in g, d elivery, a n d te c h n o k ,

mmmm : ~ s = s =£==== e d g e d isco v ery (H a n a n d K am b e r, 2 0 0 6 ).

• The Web is too big f o r effective data m in in g. T h e W e b is .s o lar^e an g

p r a t e a ^ o ^ t h e d ata o n th e W e b , m ak in g data c o lle c tio n and in tegratio n a ch a lle n g e

. M e Web is too complex. T h e c o m p le x ity o f a W e b p a g e in a trad itional te x t d o cu m e n t c o lle c tio n . W e b p a g e s a c T h e y c o n ta in far m o re au th orin g style a n d c o n te n t v ariation th an an y s e t b o o k s ,

articles o r o th er traditional te x t-b a s e d d o cu m en t. . The Web is too dynamic. T h e W e b is a highly dynam ic infonn ation soul^ Qt ^

d o e s th e W e b grow rapidly, b u t its co n ten t is constantly b ein g updated. B 8 . stories, stock m arket results, w eath er reporte, sports scores, p ace s, m ents and num erous other types o f inform ation are updated regularly o n the W e a

■—SSSSEsSsSSS

3 7 2 P artH I • Predictive Analytics

Chapter 8 • W eb Analytics, W eb Mining, and Social Analytics

it is tm e that a particular p erson is generally interested in only a tiny portion o f th e W eb, w h ereas th e rest o f th e W e b contains inform ation that is uninteresting to th e user and m ay sw am p desired results. Finding th e portion o f d ie W e b that is truly relevant to a person an d th e task b ein g perform ed is a prom inent issue in W eb-related research.

T h e s e c h a lle n g e s h av e pro m p ted m an y re se a rch efforts to e n h a n c e th e e ffectiv en ess an d e fficie n cy o f d iscov erin g and using d ata assets o n th e W e b . A n u m b e r o f in d e x -b ase d W e b s e a rch e n g in e s con stan tly sea rch th e W e b an d in d e x W e b p a g e s u n d e r certain k e y ­ w ords. U sing th e se sea rch e n g in e s, a n e x p e rie n c e d u ser m ay b e a b le to lo ca te d o cu ­ m ents b y p ro vid ing a s e t o f tightly co n strain ed k ey w ord s o r p h rases. H o w ev er, a sim ple k ey w o rd -b ased s e a rc h e n g in e suffers from sev eral d eficien cie s. First, a to p ic o f an y breadth can e asily c o n ta in hu nd red s o r th o u san d s o f d o cu m en ts. T h is c a n lea d to a large n u m ber o f d o cu m en t e n tries retu rned b y th e s e a rch e n g in e , m any o f w h ich are m arginally relevan t to th e to p ic. S e c o n d , m an y d o cu m en ts that are highly relevan t to a to p ic m ay n o t co n tain th e e x a c t key w ord s d efin in g them . As w e will c o v e r in m o re d etail later in this chapter, co m p ared to k e y w o rd -b a se d W e b se a rch , W e b m in in g is a pro m in en t (a n d m o re ch a lle n g ­ ing) a p p ro a ch that c a n b e u s e d to substantially e n h a n c e th e p o w e r o f W e b s ea rch e n g in e s b e c a u s e W e b m ining ca n identify authoritative W e b p ag es, classify W e b d o cu m en ts, and resolve m any am b ig u ities a n d su b tleties raised in k e y w o rd -b ase d W e b s e a r c h engines.

W e b m i n i n g (o r W e b data m ining) is th e p ro cess o f d iscovering intrinsic relationships -i.e., interesting and useful inform ation) fro m W e b data, w h ich are e x p resse d in th e form o f textual, linkage, o r u sag e inform ation. T h e term Web m in in g w as first u se d b y Etzioni (1996); today, m any co n feren ce s, journals, an d b o o k s fo cu s o n W e b data m ining. It is a continu­ ally evolving a re a o f tech n ology and b u sin ess practice. W e b m ining is essentially the sam e as data m ining that uses data gen erated o v er the W e b . T h e goal is to turn vast repositories o f b u sin ess transactions, cu stom er interactions, and W e b site usage data into actionable inform ation (i.e ., kn o w led g e) to pro m ote b etter d ecisio n m aking throughout th e enteiprise. B e ca u s e o f th e in creased popularity o f the term an alytics, now adays m any hav e started to call W e b m ining W eb an alytics. H ow ever, th e se tw o term s are n o t the sam e. Although W eb analytics is prim arily W e b site usage data fo cu sed , W e b m ining is inclusive o f all data g en er­ ated via th e Internet, including transaction, social, an d u sag e data. W hile W e b analytics aims to d escribe w h at has h a p p e n e d o n th e W e b site (em p loy in g a pred efined , m etrics-driven descriptive analytics m ethodology), W e b m ining aims to d iscov er previously u n k n ow n pat­ terns an d relationships (em ploying a n ov el predictive o r p rescriptive analytics m ethodology). From a big-p ictu re perspective, W e b analytics ca n b e con sid ered a part o f W e b mining. Figure 8.1 presen ts a sim ple taxon o m y o f W e b mining, w h ere it is divided into th ree m ain areas: W e b co n te n t m ining, W e b structure m ining, an d W e b usage mining. I n th e figure, the data s o u rces u se d in th ese th ree m ain areas are also specified. Although th e s e three areas are show n sep arately, as y o u will s e e in the follow ing sectio n , they are o ften u s e d collectively and synergistically to address b u sin ess p roblem s and opportunities.

As Figure 8.1 indicates, W e b m ining relies heavily o n data m ining an d te x t m ining and their en ablin g to o ls and techniqu es, w h ich w e have cov ered in detail in th e previous tw o chapters (C hapters 6 and 7 ). T h e figure also indicates that th ese th ree generic areas are further extend ed into several very w ell-k n ow n application areas. So m e o f these areas w ere explained in th e previous chapters, and som e o f the others will b e cov ered in detail in this chapter.

S E C T I O N 8 . 2 R E V I E W Q U E S T I O N S

1 . W h a t a re s o m e o f th e m a in ch a lle n g e s th e W e b p o s e s fo r k n o w le d g e discovery.-'

2 . W h at is W e b m ining? H o w d o e s it d iffer fro m regu lar d ata m ining o r te x t mining?

3 . W h at a r e th e th re e m ain are a s o f W e b mining? 4 . Id entify th re e a p p lica tio n a re a s fo r W e b m ining (a t th e b o tto m o f F ig u re 8 .1 ). B a s e d

o n y o u r o w n e x p e rie n c e s , co m m e n t o n th e ir u se ca s e s in b u sin e ss settin gs.

3 7 4 Part III • Predictive Analytics

8.3 W E B CONTENT A N D W E B STRUCTURE M IN IN G W eb c o n te n t m in in g re fe rs to th e e x tra ctio n o f useful in form ation fro m W e b p a g e s. T h e d o cu m en ts m ay b e e x tra cte d in s o m e m a ch in e -re a d a b le fo rm at s o th a t au to m ate d te c h ­ n iq u e s c a n e x tra ct s o m e in form ation from th e s e W e b p a g e s. W eb c r a w le rs (a ls o called sp id e rs) are u s e d to read th ro u g h th e c o n te n t o f a W e b site au tom atically. T h e inform a­ tio n g ath e re d m ay in clu d e d o cu m en t ch a ra cteristics sim ilar to w h a t are u se d in te x t m in­ ing, b u t it m ay also in clu d e ad ditional c o n c e p ts , s u ch a s th e d o cu m e n t h ierarch y . Such a n au to m ate d (o r sem iau to m ated ) p ro ce s s o f c o lle ctin g an d m in in g W e b c o n te n t c a n b e u s e d fo r com p etitiv e in te llig e n ce (c o lle c tin g in te llig e n ce a b o u t co m p etito rs p ro d u cts, ser­ v ice s, and cu sto m e rs). It ca n a lso b e u se d fo r inform ation/new s/opinion c o lle c tio n and sum m arization, s en tim en t analysis, a u to m a te d data co lle c tio n , an d stru cturing fo r p re d ic­ tive m o d elin g. As a n illustrative e x a m p le to u sin g W e b c o n te n t m in in g as a n au tom ated data c o lle c tio n to o l, co n s id e r th e fo llow in g . F o r m o re th an 10 years, tw o o f th e th ree au th ors o f this b o o k (D rs. Sharda an d D e le n ) h av e b e e n d ev elo p in g m o d e ls to pred ict th e fin an cial s u c c e s s o f H o lly w o o d m o v ies b e fo re th e ir th eatrical re le a se . T h e data that th e y u s e fo r training th e m o d e ls c o m e fro m sev eral W e b sites, e a c h o f w h ic h h a s a differ­ e n t h iera rch ica l p a g e stru cture. C o llectin g a large s e t o f v ariab les on th o u san d s o f m o v ­ ies (fro m th e p ast sev eral y e a rs ) from th e s e W e b sites is a tim e-d em an d in g , erro r-p ro n e p ro ce ss. T h e re fo re , th ey u s e W e b co n te n t m in in g an d sp id e rs as a n e n a b lin g te ch n o lo g y to au to m atically c o lle c t, verify, valid ate ( i f th e s p e c ific d ata item is av ailab le o n m o re th a n o n e W e b site, th e n th e v a lu e s a re v alid ate d against e a c h o th e r and an o m a lie s are cap tu red an d re co rd e d ), an d sto re th e s e v a lu e s in a re latio n al d a ta b a se . T h a t w ay, th ey e n su re th e q u ality o f th e d ata w h ile saving v a lu a b le tim e (d ay s o r w e e k s ) in th e p ro cess.

In ad dition to te xt, W e b p a g e s also c o n ta in hyperlink s p o inting o n e p a g e to another. H yperlinks co n ta in a significant am o u n t o f h id d en h u m an a n n o tatio n that c a n potentially h elp to autom atically in fer th e n o tio n o f cen trality o r au thority. W h e n a W e b p a g e d ev el­ o p e r in clu d es a lin k p o inting to an o th e r W e b p a g e , this m ay b e regard ed as th e d ev elo p e r’s

C hapter 8 • W eb Analytics, W eb Mining, and Social Analytics

en d o rse m en t o f th e o th er p ag e. T h e co lle ctiv e e n d o rse m en t o f a given p a g e b y d ifferent d ev elo p ers o n th e W e b m ay in d icate th e im p o rtan ce o f the p a g e and m ay natu rally lea d to th e d isco v ery o f authoritative W e b p a g e s (M iller, 2 0 0 5 ). T h e re fo re , th e vast am o u n t o f W e b linkage in form ation provides a rich c o lle c tio n o f inform ation a b o u t th e re le v a n ce , quality, an d structure o f th e W e b ’s co n ten ts, an d thus is a rich so u rce fo r W e b m ining.

W e b c o n te n t m in in g c a n a lso b e u se d to e n h a n c e th e resu lts p ro d u c e d b y s e a rch en g in e s. In fa c t, s e a r c h is p e rh a p s th e m o st p re v ailin g a p p lica tio n o f W e b c o n te n t m in ­ ing an d W e b stru ctu re m ining. A s e a r c h o n th e W e b to o b ta in in fo rm a tio n o n a s p e ­ cific to p ic (p r e s e n te d as a c o lle c tio n o f k e y w o rd s o r a s e n te n c e ) u su ally retu rns a fe w relevan t, h ig h -q u a lity W e b p a g e s an d a larg er n u m b e r o f u n u s a b le W e b p ag e s. U se o f a re le v a n c e in d e x b a s e d o n k e y w o rd s an d au th oritative p a g e s (o r s o m e m e a s u re o f it) will im p ro v e th e s e a r c h results a n d ran k in g o f re le v a n t p a g e s. T h e id e a o f authority (o r a u th o r ita tiv e p a g e s ) ste m s fro m e a rlie r in fo rm a tio n retriev al w o rk u sin g citatio n s am o n g jo u rn a l articles to e v a lu a te th e im p a ct o f re s e a r c h p a p e rs (M iller, 2 0 0 5 ). T h o u g h th at w a s th e Origin o f th e id ea, th e r e are sig n ifican t d iffe re n c e s b e tw e e n th e citatio n s in re s e a r c h article s an d h y p e rlin k s o n W e b p a g e s. First, n o t e v e ry h y p e rlin k re p re sen ts a n e n d o rs e m e n t (s o m e lin k s are c re a te d fo r n a v ig a tio n p u rp o s e s a n d s o m e are fo r paid a d v e rtisem e n t). W h ile th is is tru e, i f th e m ajority o f th e h y p erlin k s are o f th e e n d o rs e ­ m e n t ty p e, th e n th e c o lle c tiv e o p in io n will still prevail. S e c o n d , fo r c o m m e rc ia l and co m p etitiv e in te re sts, o n e au thority w ill rarely h av e its W e b p a g e p o in t t o rival a u th o ri­ ties in th e s a m e d o m ain . F o r e x a m p le , M icroso ft m ay p re fe r n o t to in c lu d e links o n its W e b p a g e s to A p p le ’s W e b sites, b e c a u s e this m ay b e re g a rd ed as e n d o rs e m e n t o f its co m p e tito r’s au th ority . T h ird , au th oritativ e p a g e s a re seld o m p articu larly d escrip tiv e . F or e x a m p le , th e m ain W e b p a g e o f Y a h o o ! m ay n o t c o n ta in th e e x p lic it s e lf-d e s c rip tio n th at it is in fa c t a W e b s e a rch e n g in e .

T h e stru ctu re o f W e b h yp erlin k s h a s le d to an o th e r im portant c a te g o ry o f W e b pag es ca lle d a hu b . A h u b is o n e o r m o re W e b p a g e s that p ro v id e a c o lle c tio n o f lin ks to authoritative p a g e s. H ub p a g e s m ay n o t b e p ro m in en t and o n ly a fe w lin k s m ay p o in t to them ; h o w e v e r, th ey pro v id e lin ks to a c o lle c tio n o f p ro m in en t sites o n a s p e c ific to p ic o f interest. A h u b c o u ld b e a list o f re co m m e n d e d links o n a n individ ual’s h o m e p a g e , re c ­ o m m en d e d re fe re n c e sites o n a c o u rs e W e b p a g e , o r a p ro fe ssio n a lly a s s e m b le d re s o u rce list o n a s p e c ific to p ic. H u b p a g e s p lay th e ro le o f im plicitly co n ferrin g th e au th orities o n a n arro w field . In e s s e n c e , a c lo s e sy m b io tic relatio n sh ip ex ists b e tw e e n g o o d h u b s and authoritative p a g e s ; a g o o d h u b is g o o d b e c a u s e it p o in ts to m any g o o d au th orities, a n d a g o o d au th ority is g o o d b e c a u s e it is b e in g p o in te d to b y m an y g o o d hu b s. Su ch relatio n ­ ships b e tw e e n h u b s and authorities m ak e it p o ss ib le to au tom atically retriev e high-quality co n te n t fro m th e W e b .

T h e m o st p o p u lar p u b licly k n o w n a n d re fe re n c e d algorithm u s e d to ca lcu la te hu bs an d au th o rities is h y p e rlin k -in d u c e d to p ic s e a r c h (H IT S ). It w a s o rigin ally d ev elo p e d b y K lein b erg (1 9 9 9 ) and h a s s in c e b e e n im p ro v ed o n b y m an y re search ers. H ITS is a link- analysis algorith m th at rate s W e b p a g e s using th e h y p erlin k in form ation co n ta in e d w ithin them . In th e c o n te x t o f W e b se a rch , the H ITS algorithm co lle c ts a b a s e d o cu m en t set fo r a s p e cific q u e ry . It th e n recu rsiv ely ca lcu la te s th e h u b a n d authority v a lu e s fo r e a c h d o cu m en t. T o g a th e r th e b a s e d o cu m en t set, a ro o t s e t that m a tc h e s th e q u e ry is fe tch e d from a s e a rc h e n g in e . F o r e a c h d o cu m e n t retriev ed , a s e t o f d o cu m en ts th a t p o in ts to th e original d o c u m e n t an d a n o th e r s e t o f d o cu m en ts th at is p o in te d to b y th e original d o cu ­ m en t a re a d d e d to th e s e t as th e o rigin al d o cu m e n t’s n e ig h b o rh o o d . A re cu rsiv e p ro ce s s o f d o cu m e n t id e n tificatio n a n d lin k analysis co n tin u e s un til th e h u b a n d au th o rity valu es co n v e rg e. T h e s e v a lu e s a re th e n u s e d to in d e x and prioritize th e d o c u m e n t c o lle c tio n g e n erated fo r a s p e cific query.

W e b s t r u c t u r e m in in g is th e p r o c e s s o f e x tr a c tin g u se fu l in fo rm a tio n fro m th e lin k s e m b e d d e d in W e b d o c u m e n ts . It is u s e d to id e n tify a u th o rita tiv e p a g e s a n d h u b s,

w h ic h a r e th e c o r n e r s to n e s o f th e c o n te m p o ra ry p a g e -ra n k alg o rith m s th a t a r e ce n tra l to p o p u la r s e a r c h e n g in e s s u c h as G o o g le a n d Y a h o o !. Ju s t a s lin k s g o in g to a W e b p a g e m ay in d ic a te a s it e ’s p o p u la rity (o r a u th o rity ), lin k s w ith in th e W e b p a g e (o r th e c o m p e t e W e b s ite ) m a y in d ic a te th e d e p th o f c o v e r a g e o f a s p e c ific to p ic . A n aly sis o lin k s is v e ry im p o rta n t in u n d e rs ta n d in g th e in te rre la tio n s h ip s a m o n g la rg e n u m b e rs o f W e b p a g e s , le a d in g to a b e tte r u n d e rs ta n d in g o f a s p e c ific W e b co m m u n ity , cla n , o r c liq u e . A p p lica tio n C a se 8 .1 d e s c r ib e s a p r o je c t th a t u s e d b o th W e b c o n t e n t m in ­ in g a n d W e b stru ctu re m in in g to b e tt e r u n d e rs ta n d h o w U .S. e x tr e m is t g ro u p s are

c o n n e c te d .

3 7 6 Part III • Predictive Analytics

S E C T IO N 8 . 3 R E V IE W Q U E S T IO N S

1 . W h a t is W e b c o n te n t m ining? H o w c a n it b e u se d fo r co m p etitiv e advantage?

2 . W h a t is a n “authoritative p a g e ”? W h at is a “h u b ”? W h a t is th e d iffe re n ce b e tw e e n

th e tw o? 3 . W h at is W e b structure m ining? H o w d o e s it d iffer fro m W e b c o n te n t mining?

Application Case 8.1 Id en tifyin g Extrem ist Groups w ith W e b Link and C ontent A nalysis

W e n o rm a lly s e a r c h fo r a n s w e rs to o u r p ro b le m s o u ts id e o f o u r im m e d ia te e n v iro n m e n t. O fte n , h o w e v e r, th e tro u b le ste m s fro m w ith in . In tak in g a c tio n a g a in st g lo b a l terro rism , d o m e s tic e x tr e m ­ ist g ro u p s o fte n g o u n n o tic e d . H o w e v e r, d o m e stic e x tr e m is ts p o s e a s ig n ifica n t th re a t to U .S. secu rity b e c a u s e o f th e in fo rm a tio n th e y p o s s e s s , a s w e ll a s th e ir in c re a sin g ab ility , th ro u g h th e u s e o f th e In te rn e t, to re a c h o u t to e x tre m is t g ro u p s a ro u n d th e w o rld .

K e e p in g ta b s o n th e c o n te n t av ailab le o n the In te rn e t is difficult. R e se a rch e rs and au th orities n e ed s u p e rio r to o ls to an aly ze a n d m o n ito r th e activities o f e x trem ist groups. R e se arch e rs at th e University o f A rizo n a, w ith su p p o rt from th e D ep artm e n t o f H o m e la n d Secu rity a n d o th e r a g e n c ie s , h av e d e v e l­ o p e d a W e b m in in g m e th o d o lo g y to find an d an a ­ lyze W e b sites o p e ra te d b y d o m e stic extrem ists in o rd e r t o le a rn a b o u t th e se g ro u p s th ro u g h th e ir u se o f th e In te rn e t. E xtrem ist g ro u p s u se th e In te rn e t to c o m m u n ic a te , to a c c e s s private m e ssa g e s, a n d to raise m o n e y on lin e.

T h e re sea rch m e th o d o lo g y b e g in s b y gathering a su p erior-qu ality c o lle c tio n o f relev an t e xtrem ist and terrorist W e b sites. H yperlink analysis is p erfo rm ed , w h ich lead s to o th e r extrem ist a n d terrorist W e b sites. T h e in te rco n n e cte d n e ss w ith o th e r W e b sites is cru cial in estim ating th e sim ilarity o f th e o b je ctiv e s

o f various g ro u p s. T h e n e x t step is c o n te n t analysis, w h ic h fu rther co d ifies th e s e W e b sites b a s e d o n vari­ o u s attributes, s u ch as com m u n icatio n s, fu n d raising, a n d id e o lo g y sharing, to n a m e a few .

B a s e d o n lin k a n a ly sis a n d c o n te n t analy sis, re s e a r c h e rs h a v e id e n tified 9 7 W e b site s o f U .S. e x tre m ist ancl h a te g ro u p s. O fte n , th e lin k s b e tw e e n th e s e c o m m u n itie s d o n o t n e c e s s a rily re p re s e n t a n y c o o p e r a tio n b e tw e e n th em . H o w e v e r, fin d in g n u m e ro u s lin k s b e tw e e n c o m m o n in te re s t g ro u p s h e lp s in c lu s te rin g th e co m m u n itie s u n d e r a c o m ­ m o n b a n n e r . F u rth e r re s e a r c h u s in g d ata m in in g to a u to m a te th e p r o c e s s h a s a g lo b a l aim , w ith th e g o a l o f id e n tify in g lin k s b e tw e e n in te rn a tio n a l h a te a n d e x tr e m is t g ro u p s a n d th e ir U .S. co u n te rp a rts.

Q u e s t i o n s f o r D i s c u s s i o n

1 H o w c a n W e b link/content analysis b e u se d to identify e x tre m ist groups?

2. W h a t d o y o u th in k are th e c h a lle n g e s an d th e p o ten tial so lu tio n to s u c h in te llig e n ce gath erin g activities?

Source: Y. Zhou, E. Reid, J. Qin, H. Chen, and G. Lai, “U.S. Domestic Extremist Groups on the Web: Link and Content Analysis,” IEEE Intelligent Systems, Vol. 20, No. 5, September/ October 2005, pp. 44-51.

Chapter 8 • W e b Analytics, W e b M ining, and Social Analytics 377

8.4 SE A R C H E N G IN E S In this d ay an d age, th ere is n o d enying the im p ortance o f Internet search en gin es. As th e size and com p lex ity o f the W orld W id e W e b increases, finding w h at y o u w an t is b eco m in g a co m p le x an d laborio u s process. P e o p le use search en g in es for a variety o f reason s. W e u se th em to learn a b o u t a product o r a service b e fo re com m itting to b u y (includ ing w h o else is selling it, w h at th e p rices are at different locations/sellers, th e co m m o n issu es p e o p le are discussing a b o u t it, h o w satisfied previous buyers are, w h at oth er products o r services m ight b e better, e tc.) an d search fo r p laces to go, p e o p le to m eet, an d things to d o . In a sen se, search en g in es h a v e b e c o m e th e ce n te rp iece o f m ost Internet-b ased transactions and other activities. T h e incredible s u cce ss and popularity o f G o o g le, the m o st popular sea rch engine com pany, is a g o o d testam ent to this claim . W h at is som ew h at a mystery to m an y is h o w a search en g in e actu ally d o e s w h at it is m ean t to d o. In sim plest te n n s, a s e a r c h en gin e is a softw are p rogram th at s earch es for d ocu m en ts (Internet sites o r files) b ase d o n th e keyw ords (individual w o rd s, m ulti-w ord term s, o r a com p lete s e n te n ce ) that users hav e provided that hav e to d o w ith th e su b je ct o f th eir inquiry. Search en g in es are th e w o rk h o rses o f the Internet, resp on d in g to billions o f q u eries in hundreds o f d ifferent languages e v ery day.

T e ch n ica lly sp eak in g , sea rch en g in e is th e p o p u lar term fo r inform ation retrieval sys­ tem . A lthough W e b sea rch e n g in e s are the m o st popular, search e n g in e s are o ften u se d in a co n te x t o th e r th an th e W e b , su ch as d esk to p sea rch e n g in e s o r d o cu m en t s e a rch en g in es. As y o u w ill s e e in this sectio n , m any o f th e co n c e p ts a n d te ch n iq u es th at w e co v e re d in th e te x t analytics a n d te x t m ining ch a p ter (C h ap ter 7 ) also ap p ly h e re . T h e ov erall g o al o f a sea rch e n g in e is to return o n e o r m o re docu m ents/pages ( if m o re th an o n e docum ents/ pag es ap p lies, a ran k-ord er list is o ften p ro vid ed ) that b e s t m atch th e u se r’s q u ery. T h e tw o m etrics th at are o fte n u s e d to evalu ate search e n g in e s are effectiv en ess (o r quality— find­ ing th e right d ocu m en ts/ p ages) and efficien cy (o r sp e ed — retu rning a re sp o n se quickly). T h e s e tw o m etrics te n d to w o rk in rev erse d irection; im proving o n e ten d s to w o rse n th e other. O ften , b a s e d o n u se r e x p e cta tio n , s e a rc h e n g in e s fo cu s o n o n e at th e e x p e n s e o f th e other. B e tte r s ea rch e n g in e s are th e o n e s that e x c e l in b o th at the sam e tim e. B e c a u s e search e n g in e s n o t o n ly s e a rc h b u t, in fact, find an d return th e d ocu m en ts/ pages, p erhap s a m o re ap p rop riate n a m e fo r th e m w o u ld b e “find ing e n g in e s .”

Anatom y of a Search Engine X o w le t u s d is s e c t a s e a rch e n g in e an d lo o k in sid e it. At th e h ig h est lev el, a s e a rc h e n g in e system is c o m p o s e d o f tw o m ain cy cle s: a d e v e lo p m e n t cy cle a n d a re s p o n d in g c y c le (s e e th e stru cture o f a ty p ical In te rn e t s e a rch e n g in e in Figure 8 .2 ). W h ile o n e is in terfacing

FIGURE 8 .2 Structure o f a Typical Internet Search Engine.

w ith th e W orld W id e W e b , th e o th e r is in te rfacin g w ith th e u ser. O n e c a n th in k c y c le a s a p ro d u ctio n p ro ce s s (m an u factu rin g an d i n v e n t o r y d o c u m e n s/

n a e e s ) a n d th < Jr e s p o n d in g c y c le as a retailing p ro ce s s (p ro v id in g cu stom ers/users w tfh X t t ) foU ow ing s e c tio n th e s e tw o c y cle s are e x p la in e d rn m o re detail.

1. D e v e lo p m e n t C ycle T h e tw o m ain co m p o n e n ts o f th e d ev elo p m en t cy cle are d ie W e b craw ler an d in d ex er T h e p u rp ose o f this cy cle is to cre ate a hu g e d atabase o f docum ents/pa e s or^ n lzed and^indexed b a se d o n their co n ten t and inform ation value T h e o p in g su ch a rep ository o f d ocum ents/pages is qurte obviou s: D u e to com p lex ity, searching th e W e b to find p ag es in re s p o n s e to a user feasib le w ithin a re a so n a b le tim e fram e); th erefo re, search en g in es > « « rheir d atab ase an d u se s th e ca sh e d v ersio n o f th e W e b fo r search in g and findm 0 . O n ce l t e " “ aba^e allow s sea rch e n g in e s to rapidly and accu rately re sp o n d to u se r q u e n e s,

W eb C ra w le r A W e b craw le r (a ls o ca lle d a sp id e r o r a W e b sp id er) is a ° f softw are aticallv b ro w se s (craw ls th ro u g h ) th e W o rld W id e W e b fo r the p u rp o se o f find ing and fe tch in g W e b p ag e s. O fte n W e b craw le rs c o p y all th e p a g e s th ey visit to r la te r p ro ce s s in g

b y o th er fu n ctio n s o f a s e a rch e n g in e . .. , . , crhpHnler a n d A W e b craw le r starts w ith a list o f URLs t o v,sit, w h ic h are hsted

o fte n are ca lle d th e s ee d s. T h e s e URLs m ay c o m e fro m subi o r m o re o ften th e y c o m e fro m the in te rn al h y p e rlin k s o f p rev io u sly cra w le d d o c m in ts/ p ag e s As the craw le r visits th e s e URLs, it id en tifies all th e h y p erlin ks m th e pa and ad ds th e m to th e list o f URLs to visit (i.e ., th e s ch e d u le r). URLs in th e sch e d u le r ar recu rsiv ely visited a cco rd in g to a s e t o f p o lic ie s d eterm in ed b y the B e c a u s e th e re are large volum es: o f W e b p a g e s , t h e cra w le r ca n o n ly d o w n lo a d a lim ited n u m b e r o f th e m w ith in a g iv e n tim e; th e re fo re , it m ay n e e d to prioritize its d ow n lo ad s.

D o cu m e n t In d e xe r As th e d o cu m en ts a re fo u n d a n d fe tch e d b y th e craw ler, th e y are stored in a tem p o rary staging area fo r th e d o cu m e n t in d e x e r to g rab a n d p ro ce ss. T h e 3 * ^ re s p o n s ib le fo r p ro ce s s in g th e d o cu m en ts (W e b p a g e s o r d o cu m en t file s) an d p lacm g th e m ^ n to th e d o c u m e n t d atab ase. In o rd e r to co n v e rt th e d o c u m e n t s / p a ^ into t t e d esired , easily s e a rch a b le form at, th e d o cu m en t in d e x e r p erfo rm s th e fo llo w in g tasks.

ST EP r P R E P R O C E S S IN G T H E D O C U M EN TS B e e a u s f th e d o cu m en ts fe tc h e d b y the craw le r m ay all b e in d ifferen t fo rm ats, fo r th e e a se o f further, step th ey all are co n v e rte d to s o m e ty p e o f stan d ard re p re sen tatio n . F o r in sta n ce d ifferent c c m te n tly p e s (te x t, h yp erlin k, im age, e tc .) m ay b e sep arate d fro m e a c h oth er, form atted

(if n e ce s s a ry ), an d s to red in a p la c e fo r fu rth er p ro cessin g .

ST EP 2- P A R S IN G TH E D O CU M EN TS T h is step is essentially th e application o f text m ining (i e com p u tatio nal linguistic, natural lan g u age p ro cessin g ) to o ls an d te ch n iq u es to a to to n T r u m e n ts / p 8ages. In this step, first th e standardized d ocu m en ts are; pansed « co m o o n e n ts to identify ind ex-w o rthy words/terms. T h e n , using a s e t o f r a le s the words/

term s are in d e x ed M ore specifically, using to ken izatio n rules, th e words/terms/entities ar e X t e d ff o m t h e se n te n ce s in th e se d ocu m en ts. U sing p ro p e r lex ico n s, th e sp elling errors

and o th e r a n om alies in th e se words/terms are corrected . Not T h e non discrim inating w ords/terms (a lso k n o w n as stop w o rd s) are elim inated fro m m e us of^!ndex-worthy w ords/term s. B e c a u s e th e sa m e word/term can b e in m any d ifferent forms,

3 7 8 Part III • Predictive Analytics

C hapter 8 • W eb Analytics, W e b Mining, and Social Analytics 379

stem m ing is ap p lie d to red u ce th e words/terms to th eir ro o t form s. Again, u sin g lex ico n s and o th er lan g u ag e-sp ecific re so u rces (e .g ., W ordN et), synonym s a n d h o m o n y m s are id en­ tified and th e w ord/term co lle ctio n is p ro cesse d b e fo re m oving into th e in d ex in g phase.

STEP 3: C R E A T IN G TH E TER M -B Y-D O CU M EN T M A T R IX In this step, th e relationships b etw ee n th e words/terms an d d ocum ents/pages are identified. T h e w eight can b e as sim p le as assign­ ing 1 fo r p re sen ce o r 0 fo r a b se n ce o f th e word/term in th e docum ent/page. Usually m ore sophisticated w e ig h t sch em as are used. F or instance, as o p p o sed to binary, o n e m ay ch o o s e to assign freq u en cy o f occu rre n ce (n u m b er o f tim es the sam e word/term is fo u n d in a d o cu ­ m ent) as a w eight. As w e hav e s e e n in Chapter 7 , text m ining research and p ractice have clearly indicated that th e b e s t w eighting m ay co m e from th e u se o f ten n -Jrequ en cy divided b y in verse-docu m en t-frequ en cy (TF/IDF). T h is algorithm m easures the fre q u e n cy o f o ccu r­ re n ce o f e a c h w ord/term within a d ocu m en t, and th e n com p ares that freq u ency against th e freq u ency o f o ccu rre n ce in th e d ocu m en t collection. As w e all kn ow , n ot all high-freq uency words/term are g o o d d ocu m en t discriminators; and a g o o d d ocu m en t discrim inator m a d om ain m ay n o t b e o n e in an oth er dom ain. O n ce th e w eighing sch em a is d eterm ined , th e w eights are calculated and th e term -by-d ocu m ent in d ex file is created.

2. R e sp o n se C ycle T h e tw o m ain c o m p o n e n ts o f th e re sp o n d in g c y c le a re th e q u ery an alyzer an d th e d o c u ­

m e n t m atcher/ranker.

Q u ery A n a ly z e r T h e q u ery a n a ly z er is re sp o n sib le fo r re ce iv in g a s e a r c h re q u e st fro m th e u s e r (v ia th e search e n g in e 's W e b serv er in te rfa ce ) an d co n v e rtin g it in to a stand ard ized d ata stru cture, s o th at it c a n b e e asily q u eried / m atch ed ag ain st th e e n tries in th e d o cu m e n t d atabase. H o w th e q u ery a n aly zer d o e s w h at it is su p p o s e d to d o is q u ite sim ilar to w h a t th e d o cu m en t in d e x e r d o e s (a s w e h a v e ju st e x p la in e d ). T h e q u e iy a n a ly z er p arses the s ea rch string in to individual words/term s u sin g a serie s o f task s that in clu d e to k em z a tio n , rem oval o f s to p w o rd s, stem m in g, a n d w ord/term d isam b ig u ation (id e n tifica tio n o f s p e ll­ ing errors, sy n o n y m s, an d h o m o n y m s). T h e c lo s e sim ilarity b e tw e e n th e q u e iy an alyzer and d o cu m e n t in d e x e r is n o t co in cid e n tal. In fact, it is q u ite lo g ica l, b e c a u s e b o th are w o rkin g o f f o f th e d o cu m en t d a ta b a se ; o n e is putting in d o cu m en ts/ p ag es using a s p e ­ cific in d e x stru ctu res, and th e o th er is co n v e rtin g a q u ery string into th e s a m e stru cture s o that it c a n b e u se d to q u ick ly lo c a te m o st re le v a n t d ocu m en ts/ pages.

D o cu m en t M atcher/Ranker T his is w h e re th e stru ctured q u e ry d ata is m a tch e d against th e d o cu m e n t d atab ase to find th e m o st re le v a n t d o cu m en ts/ p ages a n d a lso ra n k th em in th e o rd e r o f relevan ce/ im p o rtan ce. T h e p ro ficie n cy o f this s te p is p e rh a p s th e m o st im portant c o m p o n e n t w h e n d ifferent s e a r c h e n g in e s are c o m p a re d to o n e an o th er. E very s e a r c h e n g in e h a s its o w n ( o ften p ro p rietary ) algorith m that it u se s to carry o u t th is im p ortan t step.

T h e early s e a r c h e n g in e s u s e d a sim p le k e y w o rd m atch a g ain st th e d o cu m e n t data­ b a se an d re tu rn ed a list o f o rd e re d d ocu m en ts/ p ag es, w h e re th e d eterm in an t o f th e ord e r w as a fu n ctio n th at u se d th e n u m b e r o f w ords/term s m atch ed b e tw e e n th e q u e ry and th e d o cu m en t a lo n g w ith th e w e ig h ts o f th o se words/term s. T h e qu ality a n d th e u sefu ln ess o f th e s e a rc h resu lts w e re n o t all th at g o o d . T h e n , in 1997, th e cre a to rs o f G o o g le cam e up w ith a n e w algorithm , ca lle d P ag eR an k . As th e n am e im p lies, P ag eR an k is a n algorith ­ m ic w ay to ran k -o rd e r d ocu m en ts/ p ag es b a se d o n th e ir re le v a n ce an d value/im portance. T e c h n o lo g y In sig h ts 8 .1 p ro v id e s a high -lev el d e scrip tio n o f this p a te n te d algorithm . E v e n

3 8 0 Part III • Predictive Analytics

T E C H N O L O G Y IN SIG H T S 8 . 1 P a g e R a n k A lg o r ith m

P a g e R a n k 13 a lin k an a ly sis a l g o r i t h m - n a m e d a fte r Larry P a g e , o n e o f tHe tw o m e n t o r * o f Google, which started as a research project at Stanford University in 1996— used b> the Google Web search engine. PageRank assigns a numerical weight to each element o f a hypeihnked se of documents, such as the ones found on the World Wide Web, with the purpose o f measuring its re la tiv e im p o rta n c e w ith in a g iv e n c o lle c tio n .

It Is believed that PageRank has been influenced by citation analysis, where citations in scholarly works are examined to discover relationships among researchers and their reseaich topics The applications o f citation analysis ranges from identification o f prominent experts in a given field o f study to providing invaluable information for a transparent review o f aca­ demic achievements, which can b e used for merit review, tenure, and promotion decisions. The PageRank algorithm aims to do the same thing: identifying reputable/important/valuable documents/pages that are highly regarded by other documents/pages. A graphical illustration o PageRank is shown in Figure 8.3.

How Does PageRank Work? C o m p u ta tio n a lly sp e a k in g , P a g e R a n k e x te n d s t h e c ita tio n an a ly sis id e a b y n o t from all pages equally and by normalizing by the number o f links on a page. PageRank

defined as follows: , , , . , ■ ■ -i _ Assume page A has pages P1 through P„ pointing to it (with hyperlinks, which is similar to

citations in citation analysis). The parameter d is a damping/smoothing factor that can assume values between 0 and 1. Also C(A) is defined as the number o f links going out o f page A. th e simple formula for the PageRank for page A can b e written as follows:

PageR anH J’i) PageRankiA ) — (1 - d) + d C(Pj)

F IG U R E 8 .3 A Graphical Example fo r the PageRank Algorithm ,

C hapter 8 • W e b Analytics, W eb M ining, and Social Analytics 381

N ote that th e PageRanks form a probability distribution over W eb pages, so th e sum o f all W eb p ag es’ PageR anks will b e 1. P a g e R a n k ( A ) can b e calculated using a sim ple iterative algo­ rithm and corresp on d s to th e principal eigen vector o f the norm alized link m atrix o f th e W eb. T h e algorithm is so com putationally efficien t that a P ageRank for 26 million W eb p ag es c a n b e com puted in a few hours o n a m edium -size w orkstation (B rin and P age, 2 012). O f cou rse, there are m ore details to th e actual calculation o f PageRank in G oog le. M ost o f th o se details are either not publicly available o r are b ey o n d the s co p e o f this sim ple explanation.

Justification o f the Formulation PageRank c a n b e thought o f as a model o f u ser behavior. It assum es there is a r a n d o m s u r fe r w ho is given a W eb p ag e at random and keep s clicking on hyperlinks, never hitting b a c k bu t eventually getting bored and starting on another random page. T h e probability that the random surfer visits a page is its PageRank. And, the d damping factor is the probability at each page the r a n d o m s u r fe r will get bored and request another random page. O n e important variation is to only add the damp­ ing factor d to a single page, o r a group o f pages. This allows for personalization and can m ake it nearly im possible to deliberaiely mislead th e system in ord er to get a higher ranking.

Another intuitive justification is that a page can have a high PageRank if there are many pages that point to it, or if there are som e pages that point to it and have a high PageRank. Intuitively, pages that are w ell cited from many places around the W eb are worth looking at. Also, pages that have perhaps only o n e citation from som ething like the Yahoo! hom epage are also generally worth looking at. If a page was not high quality, or was a broken link, it is quite likely that Y ahool’s hom epage w ould not link to it. T h e formulation o f PageRank handles both o f these cases and everything in betw een by recursively propagating weights through the link structure o f the Web.

th o u g h P a g e R a n k is a n in n ov ativ e w a y to ra n k d ocu m en ts/ p ages, it is a n a u g m en ta tio n to th e p ro ce s s o f retriev in g relev an t d o cu m en ts fro m th e d atab ase an d ra n k in g th em b a se d o n th e w e ig h ts o f the words/term s. G o o g le d o e s all o f th e se c o lle c tiv e ly an d m o re to co m e u p w ith th e m o st relev an t list o f d o cu m en ts/ p ag es fo r a g iv en s e a r c h req u est. O n c e an o rd e re d list o f d o cu m en ts/ p ages is cre a te d , it is p u sh e d b a c k to th e u s e r in a n easily d ig estib le fo rm at. At this p o in t, u sers m ay c h o o s e to c lic k o n a n y o f th e d o cu m e n ts in th e list, an d it m a y n o t b e th e o n e at th e to p . I f th e y c lic k o n a d o cu m en t/ p ag e link th a t is n o t at th e to p o f th e list, th e n c a n w e assu m e th at th e s e a rch e n g in e did n o t d o a g o o d jo b ranking them ? P erh ap s, y e s. Lead ing s e a rch e n g in e s lik e G o o g le m o n ito r th e p e rfo rm an ce o f th eir s e a r c h results b y captu ring, record in g , a n d analy zing p o std e liv e ry u se r a ctio n s an d e x p e r ie n c e s . T h e s e a n aly se s o fte n lea d to m o re and m o re ru les to fu rth e r re fin e th e ranking o f th e d o cu m en ts/ p ages s o that th e links a t th e to p are m o re p re fe ra b le to the

e n d u sers.

How D o es G o o g le D o It? E ven th o u g h c o m p le x lo w -le v e l co m p u ta tio n a l d e ta ils are trad e s e c r e t s a n d a re n o t k n o w n to th e p u b lic , th e h ig h -le v e l s tru ctu re o f th e G o o g le s e a r c h s y s te m is w e ll- k n o w n a n d q u ite sim p le . F ro m th e in frastru ctu re sta n d p o in t, th e G o o g le s e a r c h s y stem runs o n a d istrib u te d n e tw o rk o f te n s o f th o u sa n d s o f co m p u ters/ serv ers a n d c a n , th e r e ­ fo re, ca rry o u t its h e av y w o rk lo a d e ffe c tiv e ly and e ffic ie n tly u sin g s o p h is tic a te d p a ra llel p ro ce s s in g a lg o rith m s (a m e th o d o f co m p u ta tio n in w h ic h m a n y c a lc u la tio n s c a n b e distributed t o m a n y s erv e rs an d p e rfo rm e d s im u lta n e o u sly , sig n ifica n tly s p e e d in g up data p r o c e s s in g ). A t th e h ig h e s t le v e l, th e G o o g le s e a r c h s y s te m h a s th r e e d istin ct parts googleguide .com):

1 . G o o g le b o t, a W e b cra w le r th a t ro am s th e In te rn e t to fin d an d fe tc h W e b p a g e s 2 . T h e in d e x e r, w h ic h sorts e v ery w o rd o n e v ery p a g e a n d sto res th e resu ltin g in d e x

o f w o rd s in a h u g e d atab ase

3 8 2 Part III • Predictive Analytics

3 . T h e q u e ry p ro ce s s o r, w h ich co m p a re s y o u r s e a r c h q u e ry to th e in d e x a n d re co m ­ m en d s th e d o cu m en ts th at it co n sid ers m o st relevan t

1 . G o o g le b o t G o o g le b o t is G o o g le ’s W e b craw ling ro b o t, w h ich finds and retrieves p a g e s o n th e W eb a n d hand s th e m o f f to th e G o o g le indexer. It’s e a sy to im agine G o o g le b o t as a little sp id er scu rrying a cro ss th e strands o f cy b ersp ace , b u t in reality G o o g le b o t d o e s n ’t trav erse th e W eb at all. It fu n ction s, m u ch like yo u r W eb brow ser, b y sen d in g a re q u e st to a W eb serv e r fo r a W eb p ag e , d ow n­ load in g th e en tire p a g e , a n d th e n h an d in g it o f f to G o o g le ’s indexer. G o o g le b o t co n sists o f m an y com p u ters req u estin g and fetch in g p a g e s m u ch m o re quickly th a n y o u c a n w ith y ou r W eb brow ser. I n fa ct, G o o g le b o t ca n re q u e st thousands o f d ifferent p a g e s sim ultaneously. T o avoid o v erw h elm in g W eb servers, o r crow d­ ing o u t req u ests fro m h u m an users, G o o g le b o t d elib erately m ak e s req u ests o f e a c h individual W eb serv er m o re slow ly th an it’s ca p a b le o f doing.

W h e n G o o g le b o t fe tch e s a p ag e, it rem oves all th e links ap p earin g o n the p ag e a n d ad ds th em to a q u e u e fo r s u b s e q u e n t craw ling. G o o g le b o t tend s to e n co u n te r little sp am b e c a u se m o st W eb authors link only to w h at th ey b elie v e are high-quality p ages. B y harvesting links fro m every p ag e it en co u n ters, G o o g le b o t can q u ickly b u ild a list o f links that c a n co v er b road re a ch e s o f th e W eb. This te ch n iq u e, k n o w n as d eep craw ling, a lso allow s G o o g le b o t to p ro b e d e e p within individual sites. B e c a u s e o f their m assive scale , d e e p craw ls ca n re a ch alm ost every p a g e in th e W eb. T o k e e p th e in d e x current, G o o g le co n tin u ou sly recraw ls po p u lar freq uently ch an g in g W eb p a g e s a t a rate roughly proportional to h ow o ften th e p ag es ch an g e. S u ch craw ls k e e p an index cu rrent an d are k n o w n as fr e s h craw ls. N ew spaper p ag es are d o w n lo ad ed daily; p ag es w ith s to ck q u o tes are d ow n lo ad ed m u ch m o re frequently. O f cou rse, fresh craw ls return few er pages th an th e d e e p craw l. T h e co m b in ation o f d ie tw o types o f craw ls allow s G oo g le to b o th m ake efficien t u se o f its re so u rces and k e e p its in d ex re aso n ab ly current.

2 . G o o g le I n d e x e r G o o g le b o t gives th e in d exer th e full text o f the pages it finds. T h e s e p ag es are stored in G o o g le ’s in d e x d atabase. T h is in d ex is sorted alphabeti­ cally b y s earch term , w ith e a c h in d ex e n try storing a list o f d ocu m en ts in w h ich the term appears and th e location within th e text w h ere it occu rs. T h is data stm eture allow s rapid a cce ss to d ocu m en ts that co n tain u ser query term s. T o im prove search p erform ance, G o o g le ignores co m m o n w ords, called stop w ords (su ch as the, is, o n , or, o f a , a n , as w ell as certain single digits and single letters). Stop w ord s are so co m m o n that they d o little to narrow a search, an d therefore they c a n safely b e discarded. T h e in d ex er a lso ignores s o m e punctuation a nd multiple sp ace s, as well as converting all letters to low ercase, to im prove G o o g le ’s perform ance.

3 . G o o g le Q u ery P r o c e s s o r T h e q u e ry p ro c e s s o r has sev eral parts, inclu ding th e u ser in te rfa ce (s e a rc h b o x ), th e “e n g in e " that e v alu ate s q u e ries a n d m atch es th e m to relev an t d o cu m en ts, and th e results form atter.

G o o g le u s e s a p ro p rietary algorithm , ca lle d P ag eR an k , to ca lcu la te th e relativ e rank ord e r o f a g iv e n c o lle c tio n o f W e b p a g e s. P a g eR a n k is G o o g le ’s sy stem fo r ran kin g W e b p a g e s. A p a g e w ith a h ig h e r P ag eR a n k is d e e m e d m o re im p ortan t an d is m o re likely to b e listed a b o v e a p a g e w ith a lo w e r P ag eR an k . G o o g le co n sid e rs o v e r a hu nd red fa cto rs in co m p u tin g a P ag eR an k a n d d eterm in in g w h ic h d o cu m en ts are m o st relev an t to a query, in clu d in g th e p o p u larity o f th e p a g e , th e p o sitio n a n d siz e o f th e s e a r c h term s w ith in th e p a g e an d the p roxim ity o f th e s ea rch term s to o n e a n o th e r o n th e p age.

’G o o g le also applies m achine-learn ing te ch n iq u es to im prove its p erfo rm ance automati­ cally b y learning relationships an d associations w ithin th e stored data. F o r exam p le, the sp e - in g-correctin g system u se s su ch tech n iq u es to figure ou t likely alternative spellings. G o o g e

Chapter 8 • W eb Analytics, W e b Mining, and Social Analytics 383

closely guards th e form ulas it uses to calcu late relevan ce; th ey ’re tw eak ed t o im prove quality and p erfo rm an ce, and to outw it the latest d evious tech n iqu es u sed b y spam m ers.

In d ex in g th e full text o f th e W e b a llo w s G o o g le to g o b ey o n d sim p ly m atch in g sin gle s e a r c h term s. G o o g le gives m o re priority to p a g e s that h a v e s e a rc h term s n e a r e a c h o th er an d in th e sa m e ord e r as th e query. G o o g le c a n a lso m atch m u lti-w ord p h rase s and s e n te n ce s . B e c a u s e G o o g le in d e x e s HTML c o d e in ad d ition to th e te x t o n th e p a g e , users c a n restrict s e a r c h e s o n th e b a sis o f w h e re q u e ry w o rd s a p p e a r (e .g ., in th e title, in th e URL, in th e b o d y , a n d in lin k s to th e p ag e , o p tio n s o ffe re d b y G o o g le ’s A d v an ced S e a rch Form an d U sin g S e a rch O p erato rs).

U n d erstan d in g the in tern als o f p o p u la r sea rch e n g in e s h e lp s co m p a n ie s , w h o rely o n s e a rch e n g in e traffic, b etter d esig n th e ir e -c o m m e r c e sites to im p ro v e th e ir c h a n ce s o f g etting in d e x e d an d highly ra n k e d b y s e a r c h providers. A p p lication C a se 8 .2 gives an illustrative e x a m p le o f s u c h a p h e n o m e n o n , w h e re a n en te rtain m e n t co m p a n y in cre a se d its sea rch -o rig in a te d cu sto m e r traffic b y 15 0 0 p e rce n t.

Application Case 8.2 IGN Increases Search Traffic by 1500 Percen t IG N E n te r ta in m e n t o p e r a t e s th e In te r n e t’s la rg e s t n e tw o r k o f d e s tin a tio n s fo r v id e o g a m in g , e n te r ­ ta in m e n t, a n d c o m m u n ity g e a r e d to w a rd te e n s a n d 1 8 - t o 3 4 -y e a r -o ld m a le s . T h e c o m p a n y ’s p r o p e r tie s in c lu d e IG N .c o m , G a m e S p y , A sk M en . c o m , R o tt e n T o m a to e s , F ile P la n e t, T e a m X b o x , 3 D G a m e rs , V E 3 D , a n d D ir e c t2 D r iv e — m o re th a n 7 0 c o m m u n ity s ite s a n d a v a s t array o f o n lin e fo ru m s. IG N E n te r ta in m e n t is a ls o a le a d in g p r o ­ v id e r o f te c h n o l o g y fo r o n lin e g a m e p la y in v id e o g a m e s .

T h e C h a lle n g e

W h e n this c o m p a n y co n ta c te d S E O In c. in su m m er 2 0 0 3 , th e s ite w as a n e sta b lish e d an d w e ll-k n o w n site in th e g am in g com m u nity. T h e site a ls o h ad s o m e g o o d s e a r c h e n g in e ran kin gs an d w as getting a p p ro x im a te ly 2 .5 m illion u n iq u e visitors p e r m onth. At th e tim e IG N u se d p ro p rietary in -h o u se co n te n t m a n a g e m e n t a n d a te a m o f c o n te n t w riters. T h e p a g e s th at w e r e g e n era ted w h e n n e w g a m e review s and in fo rm a tio n w e re ad d ed to th e site w e re not v ery w e ll op tim ized . In ad d ition, th e re w e re serio u s arch itectu ral issu es w ith th e site, w h ich p re v e n te d s e a rch e n g in e sp id ers fro m th o ro u g h ly an d c o n s is ­ ten tly craw lin g th e site.

IG N ’s g o a ls w e re to “d o m in a te th e s ea rch ran kin gs fo r k e y w o rd s re la ted to an y v id e o gam es a n d g am in g sy stem s re v ie w e d o n th e s ite .” IG N

w a n te d to ra n k h ig h in th e s e a r c h e n g in e s , and m o st s p e c ific a lly , G o o g le , fo r a n y a n d a ll g am e titles an d va rian ts o n th o se g am e titles’ p h rases. IG N ’s re v e n u e is g e n e r a te d fro m ad v ertisin g s a le s, s o m o re traffic le a d s to m o re in v e n to ry fo r ad sales, m o re ad s b e in g so ld , and th e re fo r e m o re re v e n u e . In o rd e r to g e n e r a te m o re traffic, IG N k n e w th a t it n e e d e d to b e m u c h m o re v isib le w h e n p e o p le u se d th e s e a r c h e n g in e s .

T h e S tr a te g y

A fter sev eral c o n v e rsa tio n s w ith th e IGN team , SO E In c. cre a te d a cu sto m iz ed op tim izatio n p a c k a g e that w as d esig n e d t o a ch ie v e th e ir ra n k in g g o a ls and also fit th e c lie n t’s b u d g et. B e c a u s e IG N .co m had arch itectu ral p ro b le m s an d a p ro p rietary CMS (c o n ­ te n t m a n a g e m e n t sy stem ), it w as d e cid e d th at SEO In c. w o u ld w o rk w ith th e ir IT a n d W e b d ev elo p ­ m e n t te a m a t th e ir lo ca tio n . T h is a llo w e d SE O to se n d th e ir team to th e IG N lo ca tio n fo r sev eral days to learn h o w th e ir sy stem w o rk e d an d p artn er w ith th eir in -h o u s e p ro gram m ers to im p ro ve th e system and , h e n c e , im p ro v e s e a r c h e n g in e op tim ization. In ad d ition, SE O c re a te d cu sto m ized SE O b e s t p rac­ tic e s an d a rch ite cte d th e s e into th e ir p ro p rietaiy CMS. SE O a lso train ed th e ir c o n te n t w riters and p a g e d e v e lo p e rs o n SE O b e s t p ra ctice s. W h e n n e w g a m e s a n d p a g e s are ad d ed to th e site, th ey a re typi­ cally g etting ra n k e d w ith in w e e k s , i f n o t days.

( C ontinued)

3 8 4 Part III • Predictive Analytics

Application Case 8.2 (Continued)

T h e R e s u lts

T h is w a s a tru e a n d q u ick s u c c e s s story. O rg an ic s e a rch e n g in e ran kin gs sk y ro ck e te d an d th o u san d s o f p re v io u sly n o t-in d e x e d p a g e s w e re n o w b e in g craw le d reg u larly b y s e a rch e n g in e sp id ers. S o m e o f th e s p e c ific results w e re as fo llow s:

• U n iq u e visitors to th e site d ou bled within the first 2 m onths after th e optim ization w as com pleted.

• T h e r e w a s a 15 0 0 p e rc e n t in c re a se in org an ic s e a r c h e n g in e traffic.

• M a ss iv e g r o w th in traffic a n d r e v e n u e s e n a b l e d a c q u i s it io n o f a d d itio n a l W e b p r o p e r tie s in c lu d ­ i n g R o tte n to m a to e s .c o m a n d A s k m e n .c o m

IG N w a s a cq u ire d b y N ew s C orp in S e p te m b e r 200 5

fo r $ 6 5 0 m illion.

Q u e s t i o n s f o r D i s c u s s i o n 1. H o w d id IG N d ram atically in c re a se sea rch traffic

to its W e b portals? 2. W h at w e re th e c h a lle n g e s, th e p ro p o s e d so lu ­

tion , a n d th e o b ta in e d results?

Source: SOE Inc., Customer Case Study, seoinc.com/seo/case- studies/ign (accessed March 2013)-

SECTION 8 . 4 REVIEW QUESTIONS

1 . W h at is a s e a rc h engine? W h y are th e y im p ortan t fo r to d ay 's b u s in e s s e s '

2 . W hat is the re latio n sh ip b e tw e e n s e a rch e n g in e s an d te x t mining? 3 . W h a t ffice th e w o m ain c y cle s in s e a rch en g in e s? D e s c r ib e th e step s in e a c h e y e e ,

4 W h a t is a W e b craw ler? W h at is it u s e d fo r ? H o w doe® it w ork? 5. H ow d o e s a q u ery a n a ly zer w ork? W h a t is P a g e R a n k alg orithm a n d h o w d o e s it w ork?

8.5 SEA R C H E N G IN E O PT IM IZA TIO N S e a rc h e n g in e op tim izatio n CSE58 is t h e in te n tio n a l activity o f affectin g th e visibility o f a * B-co m m e rce s ite o r a W e b -site in a s e a r c h e n g in e 's natu ral (u n p aid o r o rg a n ic) s ea rch i results In g e n eral, th e h ig h e r ra n k e d o n th e s e a r c h results p a g e , an d m o re freq u en tly a site a p p e a rs in th e s e a rc h results list, th e m o re visitors it will re ce iv e fro m th e searc • M i n e 's u se rs. As a n In te rn e t m ark etin g strategy, SE O co n sid ers h o w s e a r f h en gm W ork w h a t p e o p le .search for, th e actu al s e a r c h term s o r k e y w o rd s ty p e d m t a s ea rch e n g in e s and w h ich s e a rch e n g in e s a re p re fe rre d b y th eir targ eted audience^ O ptim izing a W e b site m ay involve editing its co n ten t, HTML, an d a sso cia ted c o d in g to b o th in cre ase its re le v a n ce to s p e c ific k e y w o rd s a n d to re rn o v e b arriers to th e in d e x in g activ itie s o f s e a rch e n g in e s. P rom o tin g a site to in cre a se th e n u m b e r o f b a ck lm k s, o r in b o u n d links,

iS an<t N e a r l y days, in ord er to b e in d ex ed , all W eb m asters n e e d e d to d o w a s to subm it th e ad d ress o f a p ag e , o r URL, to th e v arious e n g in e s , w h ich w o u ld th e n se n d a spi er to “craw l" that p ag e , e xtract links to o th er p a g e s fro m it, an d returni inform ation fo u n d on Ih e p a g e to th e s tr v e r fo r indexing. T h e p ro ce ss, as e x p la in e d b efo re , involves a search e n g i n e sp id er d ow n lo ad in g a p a g e and storing it o n t h e search e n g in e ’s o w n server, w h ere a s e c o n d program , k n o w n as a n in d e x er, e xtracts v arious inform ation ab ou t th e p age, such as " h e w o rd s it co n tain s a n d w h e re th e se are lo cate d , as w e ll as a n y w e ig h t to r s p e ific w ords, an d all links the p ag e co n tain s, w h ich a r e th e n p la ce d m to at a later date. N ow adays s e a rch e n g in e s are n o lo n g e r relying o n W ebm asters URLs (e v e n th o u g h th e y still ca n ); instead, th e y a r e p ro activ ely an d con tin u o u sly era ing th e W e b , a n d find ing, fetch in g, an d in d e x in g everything a b o u t it.

Chapter 8 • W e b Analytics, W e b Mining, and Social Analytics 3 8 5

B e in g in d e x e d b y s e a r c h e n g in e s lik e G o o g le , B in g , a n d Y a h o o ! is n o t g o o d e n o u g h fo r b u s in e s s e s . G e ttin g r a n k e d o n th e m o s t w id e ly u s e d s e a r c h e n g in e s ( s e e T e c h n o lo g y In s ig h ts 8 .2 fo r a list o f m o s t w id e ly u s e d s e a r c h e n g in e s ) a n d g ettin g ra n k e d h ig h e r th a n y o u r c o m p e tito rs a re w h a t m a k e th e d iffe r e n c e . A v arie ty o f m e t l - o d s c a n in c r e a s e th e ra n k in g o f a W e b p a g e w ith in th e s e a r c h resu lts. C ro s s -lin k in g b e tw e e n p a g e s o f th e s a m e W e b s ite t o p ro v id e m o re lin k s to th e m o st im p o rta n t p a g e s m ay im p ro v e its v isib ility . W ritin g c o n te n t th a t in c lu d e s fre q u e n tly s e a r c h e d k e y w o r d p h ra se s , s o a s t o b e re le v a n t to a w id e v arie ty o f s e a r c h q u e rie s , w ill te n d to in c re a s e traffic. U p d a tin g c o n te n t s o as to k e e p s e a r c h e n g in e s c ra w lin g b a c k fre q u e n tly c a n oiv e ad d itio n a l w e ig h t t o a site. A d d ing re le v a n t k e y w o rd s to a W e b p a g e s m e ta d a ta , in clu d in g th e title tag an d m e ta d e s c rip tio n , w ill te n d to im p ro v e th e r e le v a n c y o f a s ite ’s s e a r c h listin g s, th u s in c re a s in g traffic. URL n o rm a liz a tio n o f W e b p a g e s s o th at th e y a re a c c e s s ib le via m u ltip le URLs a n d u s in g c a n o n ic a l lin k e le m e n t a n d re d ire cts c a n h e lp m a k e s u re lin k s t o d iffe re n t v e rs io n s o f th e URL all c o u n t to w a rd th e p a g e s

lin k p o p u la rity s c o re .

M ethods fo r Search E n g in e O p tim iza tio n In g e n era l, SE O te c h n iq u e s c a n b e classified in to tw o b ro a d ca te g o rie s: te c h n iq u e s that s e a rc h e n g in e s re co m m e n d as part o f g o o d site d esig n , and th o se te c h n iq u e s o f w h ich s e a rc h e n g in e s d o n o t ap p ro v e . T h e s e a r c h e n g in e s attem p t to m inim ize th e e ffe c t o f th e latter, w h ic h is o ften ca lle d sp am d ex in g (a ls o k n o w n a s sea rch spam , s ea rc h en g in e spam , o r’ s e a r c h en g in e p oiso n in g ). Ind ustry co m m e n ta to rs h av e classified th e s e m e th ­ od s, and th e p ractitio n e rs w h o e m p lo y th em , a s e ith e r w h ite -h a t SE O o r b la c k -h a t SEO

T E C H N O L O G Y IN S IG H T S 8 . 2 T o p 1 5 M o s t P o p u l a r S e a r c h E n g in e s

(M a r c h 2 0 1 3 )

Here are the 15 most popular search engines as derived from eBizMBA Rank (e b iz m b a .c o m / a r tic le s / s e a r c h - e n g in e s ) , which is a constantly updated average o f each W eb sites Alexa Global Traffic Rank, and U.S. Traffic Rank from both Compete and Quantcast.

Ran k N am e E stim ate d U n iq u e M o n th ly V isito rs

1 Google 900,000,000

2 Bing 165,000,000

3 Yahoo! Search 160,000,000

4 Ask 125,000,000

5 A O L Search 33,000,000

6 MyWebSearch 19,000,000

7 blekko 9,000,000

8 Lycos 4,300,000

9 Dogpile 2,900,000

10 WebCrawler 2,700,000

11 Info 2,600,000

12 InfoSpace 2,000,000

13 Search 1,450,000

14 Excite 1,150,000

15 GoodSearch 1,000,000

3 8 6 Part III • Predictive Analytics

(G o o d m a n , 2 0 0 5 ). W h ite h ats te n d to p ro d u ce resu lts that last a lo n g tim e, w h e re a s b la ck h ats a n ticip ate th at th e ir sites m ay ev en tu ally b e b a n n e d e ith er tem p o rarily o r p erm a­ n e n tly o n c e the s e a r c h e n g in e s d isco v e r w h at th e y are d oing.

A n S E O te c h n iq u e is c o n s id e r e d w h ite h a t i f it c o n fo r m s to th e s e a r c h e n g in e s ’ g u id e lin e s a n d in v o lv e s n o d e c e p tio n . B e c a u s e s e a r c h e n g in e g u id e lin e s a re n o t w rit­ te n a s a s e r ie s o f ru le s o r co m m a n d m e n ts , th is is a n im p o rta n t d is tin c tio n to n o te. W h ite -h a t S E O is n o t ju s t a b o u t fo llo w in g g u id e lin e s , b u t a b o u t e n s u rin g th a t th e c o n ­ te n t a s e a r c h e n g in e in d e x e s a n d s u b s e q u e n tly ra n k s is th e s a m e c o n t e n t a u s e r will s e e . W h ite -h a t a d v ic e is g e n e r a lly s u m m e d u p a s c re a tin g c o n te n t fo r u s e rs , n o t fo r s e a r c h e n g in e s , a n d th e n m a k in g th a t c o n t e n t e a s ily a c c e s s ib le to th e s p id e rs , rath er th a n a tte m p tin g to trick th e alg o rith m fro m its in te n d e d p u rp o s e . W h ite -h a t SE O is in m a n y w a y s sim ilar to W e b d e v e lo p m e n t th at p r o m o te s a c c e s s ib ility , a lth o u g h th e tw o a re n o t id e n tica l.

B la c k -h a t S E O atte m p ts to im p ro v e ra n k in g s in w ay s th a t a re d is a p p ro v e d b y the s e a r c h e n g in e s , o r in v o lv e d e c e p tio n . O n e b la c k -h a t te c h n iq u e u s e s te x t th a t is hid ­ d en , e ith e r a s te x t c o lo r e d sim ilar to th e b a c k g ro u n d , in a n in v isib le div, o r p o sitio n e d o ff-s c re e n . A n o th e r m e th o d g iv e s a d iffe re n t p a g e d e p e n d in g o n w h e th e r th e p a g e is b e in g re q u e s te d b y a h u m an v isito r o r a s e a r c h e n g in e , a te c h n iq u e k n o w n a s c lo a k ­ ing. S e a rc h e n g in e s m ay p e n a liz e site s th e y d is c o v e r u sin g b la c k -h a t m e th o d s , e ith e r b y re d u c in g th e ir ra n k in g s o r e lim in atin g th e ir listin gs fro m th e ir d a ta b a s e s a lto g e th e r. S u c h p e n a ltie s c a n b e a p p lie d e ith e r a u to m a tica lly b y th e s e a r c h e n g in e s ’ algorith m s, o r b y a m an u al site review'. O n e e x a m p le w a s th e F e b ru a ry 2 0 0 6 G o o g le re m o v a l o f b o th B M W G e rm a n y an d R ic o h G e rm a n y fo r u s e o f u n a p p ro v e d p ra c tic e s (C u tts, 2 0 0 6 ). B o th co m p a n ie s , h o w e v e r, q u ick ly a p o lo g iz e d , fix e d th e ir p ra c tic e s , an d w e r e resto red to G o o g le ’s list.

F o r s o m e b u s in e s s e s SEO m ay g e n e ra te sig n ifican t retu rn o n investm ent. H ow ever, o n e sh o u ld k e e p in m in d th at s e a rch e n g in e s a re n o t p aid fo r o rg a n ic s e a rch traffic, their algorithm s ch a n g e con stan tly , an d th e re are n o g u aran te es o f c o n tin u e d referrals. D u e to this la ck o f certain ty and stability, a b u sin e ss that relies heavily o n s e a rch e n g in e traffic c a n su ffe r m a jo r lo s s e s if th e s e a rc h e n g in e d e c id e s to c h a n g e its algorithm s an d stop se n d in g visitors. A cco rd in g to G o o g le ’s C E O , E ric Schm id t, in 2 0 1 0 , G o o g le m ad e o v er 5 0 0 algorithm c h a n g e s — alm o st 1 .5 p e r day. B e c a u s e o f th e d ifficulty in k e e p in g up w ith ch a n g in g s e a rch e n g in e ru les, c o m p a n ie s th at re ly o n s e a r c h traffic p ra c tic e o n e o r m ore o f th e fo llow in g : ( 1 ) Hire a co m p a n y that s p e cia liz e s in s e a r c h e n g in e o p tim ization (th ere s e e m to b e a n ab u n d an t n u m b e r o f th o se n o w a d a y s) to co n tin u o u sly im p ro v e y o u r site's a p p e a l to ch a n g in g p ractice s o f th e s e a rch e n g in e s ; (2 ) p a y th e s e a rc h e n g in e provid­ ers to b e listed o n th e p aid s p o n so rs se ctio n s ; an d (3 ) co n s id e r lib eratin g y o u rs e lf from d e p e n d e n c e o n s e a rch e n g in e traffic.

E ith e r o r ig in a tin g fro m a s e a r c h e n g in e (o r g a n ic a lly o r o th e r w is e ) o r c o m in g fro m o t h e r s ite s a n d p la c e s , w h a t is m o s t im p o rta n t fo r a n e -c o m m e r c e s ite is to m a x im iz e th e lik e lih o o d o f c u s to m e r tra n s a c tio n s . H av in g a lo t o f v is ito rs w ith o u t s a le s is n o t w h a t a ty p ic a l e -c o m m e r c e s ite is b u ilt fo r. A p p lic a tio n C a s e 8 .3 is a b o u t a la rg e In te r n e t-b a s e d s h o p p in g m all wrh e r e d e ta ile d a n a ly sis o f c u s to m e r b e h a v ­ io r (u s in g c lic k s tr e a m s a n d o t h e r d ata s o u r c e s ) is u s e d to s ig n ific a n tly im p ro v e th e c o n v e r s io n rate.

SECTION 8 . 5 REVIEW QUESTIONS

1 . W h at is “s e a r c h e n g in e op tim izatio n ”? W h o b e n e fits fro m it? 2 . D e s crib e th e o ld an d n e w w ays o f in d e x in g p e rfo rm e d b y s e a r c h e n g in e s. 3 . W h at a re th e th in gs th at h e lp W e b p a g e s ra n k h ig h er in th e sea rch e n g in e results? 4 . W h at are th e m o st co m m o n ly u s e d m e th o d s fo r s e a rch e n g in e optim ization?

Chapter 8 • W eb Analytics, W eb Mining, and Social Analytics 387

Application Case 8.3 U nderstanding W h y Custom ers A bandon Shopping

L o tte.com , th e lead in g In te rn e t sh o p p in g mall in K o re a w ith 13 m illion cu sto m e rs, h a s d e v e lo p e d an in teg rated W e b traffic analysis sy stem u sin g SAS fo r C u stom er E x p e rie n c e A nalytics. As a result, Lotte .c o m h a s b e e n a b le to im prove th e o n lin e e x p e ri­ e n c e fo r its cu sto m e rs, as w ell as g e n e r a te b etter retu rns fro m its m ark etin g cam p aign s. N ow , Lotte .co m e x e c u tiv e s ca n co n firm results an y w h ere, any­ tim e, a s w e ll a s m ak e im m ed iate ch an g es.

With alm ost 1 m illion W e b site visitors e a ch day, L otte.com n eed ed to k n o w h o w m any visitors w ere m aking pu rchases and w h ich channels w ere bringing the m ost valuable traffic. After review ing m any diverse solutions an d approaches. Lotte.com introduced its integrated W e b traffic analysis system using the SAS for Custom er E xp erien ce Analytics solution. This is the first online behavioral analysis system applied in Korea.

W ith this sy stem , L o tte .co m c a n accu rately m e asu re a n d an aly ze W e b site v isitor n u m b ers (U V ), p a g e v ie w (P V ) status o f site visitors an d p u rch a s­ ers, th e p o p u larity o f e a c h p ro d u ct c a te g o ry an d produ ct, c lick in g p re fe re n c e s fo r e a c h p a g e , th e effe ctiv e n e s s o f cam p aig n s, a n d m u ch m o re. T h is in form ation e n a b le s L o tte .co m to b e tte r u n d erstan d cu sto m ers a n d th e ir b e h a v io r o n lin e, an d co n d u ct so p h isticated , co st-e ffe c tiv e targ eted m arketing.

C o m m en tin g o n th e sy stem , A ssistant G e n e ral M anager J u n g H y o -h o o n o f th e M arketing P lanning T e a m fo r L o tte .co m said , “As a result o f in trod u cin g th e SAS sy stem o f analysis, m an y ’n e w tru ths’ w e re u n c o v e re d aro u n d cu sto m e r b eh av io r, a n d s o m e o f th e m w e re ‘in co n v e n ie n t tru ths.’” H e ad d ed , “S o m e site-p lan n in g activities that h a d b e e n u n d ertak en w ith th e e x p e c ta tio n o f certain results actu ally h a d a lo w re a c tio n fro m cu sto m ers, an d th e site p lan n ers had a d ifficu lt tim e re co g n izin g th e s e resu lts.”

B e n e f its

Introducing th e SAS fo r Custom er E xp erien ce Analytics solution fully transform ed the L otte.com W e b site. As a result, L otte.com h a s b e e n a b le to im prove the online e x p e rie n c e fo r its cu stom ers as w’e ll as gen erate better returns fro m its m arketing cam paigns. N ow , L otte.com e xecu tiv es c a n confirm results anyw here, anytim e, as well as m a k e im m ediate changes.

Carts Results in $10 M illion Sales Increase Since im plem enting SAS fo r Custom er E xp erien ce

Analytics, L otte.com has s e e n m any benefits:

A Ju m p in C u stom er L oy alty A larg e a m o u n t o f so p h istica ted activity in form a­ tio n ca n b e c o lle c te d u n d e r a v isitor en v iron m en t, in clu d in g q u ality o f traffic. D ep u ty A ssistant G e n e ral M anager Ju n g sa id that “b y analyzing actu al valid traffic an d lo o k in g o n ly a t o n e to tw o p a g e s, w e ca n carry o u t ca m p a ig n s to h e ig h te n th e lev el o f loyalty, a n d d eterm in e a certain ran g e o f e ffe ct, a cco rd in g ly .” H e ad d ed , “In ad d ition, it is p o ss ib le to classify an d co n firm th e o r d e r rate fo r e a c h c h a n n e l a n d s e e w h ic h ch a n n e ls h av e th e m o st v isito rs.”

O p tim ized M a rk etin g E ffic ien c y A n aly sis R ath er th an ju st analyzing v isitor n u m b ers o n ly , the sy stem is c a p a b le o f analyzing th e c o n v e rs io n rate (s h o p p in g cart, im m ed iate p u rch a se , w ish list, pur­ c h a s e c o m p le tio n ) c o m p a re d to actu al visitors for e a c h ca m p a ig n ty p e (affiliatio n o r e-m ail, b an n er, key w ord s, an d o th e rs), so d etailed an aly sis o f c h a n ­ n e l e ffe c tiv e n e s s is p o ssib le . A dditionally, it c a n c o n ­ firm th e m o st p o p u la r s e a rch w o rd s u s e d b y visitors fo r e a c h ca m p a ig n ty p e, lo ca tio n , an d p u rch ased pro d u cts. T h e p a g e ov erlay fu n ctio n c a n m e asu re th e n u m b e r o f c lick s and n u m b e r o f visitors fo r e a ch item in a p a g e to m e asu re th e v alu e fo r e a c h lo c a ­ tion in a p ag e . T h is cap ab ility e n a b le s L o tte .co m to p ro m p tly re p la c e o r r e n e w lo w traffic item s.

E n h a n c e d C u stom er S a tisfa c tio n a n d C u stom er E x p erien c e L e a d to H ig h er S ales L otte.com b u ilt a cu sto m e r b e h a v io r an alysis d ata­ b a s e that m e a su re s e a c h visitor, w h at p a g e s are v isited , h o w v isitors n av igate th e site, a n d w h at activities are u n d e rta k e n to e n a b le d iv erse analysis an d im p ro ve s ite e fficie n cy . In ad d ition, th e d ata­ b a se ca p tu re s cu sto m e r d em o g ra p h ic in form ation, sh o p p in g ca rt siz e a n d c o n v e rs io n rate, n u m b e r o f ord ers, a n d n u m b e r o f attem pts.

B y an aly zin g w h ic h stag e o f th e o rd erin g p ro ­ c e s s d eters th e m o st cu sto m ers a n d fix in g th o se stag es, c o n v e rs io n rates c a n b e in cre a se d . P reviously, analysis w as d o n e o n ly o n p la ce d orders. B y an aly z­ in g th e m o v e m e n t p attern o f visitors b e fo re ord ering an d at th e p o in t w h e re b rea k a w a y o ccu rs, cu sto m e r

(Continued)

3 8 8 Part III • Predictive Analytics

Application Case 8.3 (Continued) b e h a v io r c a n b e fo reca st, a n d so p h istica te d m arket­ ing activ ities c a n b e u n d e rta k e n . T h ro u g h a pattern analysis o f visitors, p u rch a se s ca n b e m o re e ffe c ­ tively in flu e n c e d a n d cu sto m e r d em an d c a n b e r e fle cte d in real tim e to e n su re q u ic k e r resp o n se s. C u sto m er satisfactio n h a s a lso im p ro ved as Lotte .c o m h a s b e tte r insight in to e a c h cu sto m e r’s b e h a v ­ io rs, n e e d s , a n d interests.

E valu ating th e system , Ju n g c o m m e n te d , “B y fin d in g o u t h o w e a c h cu sto m e r g ro u p m o v e s o n the b a sis o f th e d ata, it is p o ss ib le to d eterm in e c u s ­ to m e r s e rv ic e im p ro v em en ts an d targ e t m arketing s u b je c ts , an d this h a s aid ed th e s u c c e s s o f a n u m ­ b e r o f c a m p a ig n s .” H o w ev er, th e m o st significant

b e n e fit o f th e sy stem is gain in g in sig h t in to indi­ v id ual cu sto m e rs a n d vario u s cu sto m e r g ro u p s. B y u n d e rstan d in g w h e n cu sto m e rs w ill m a k e p u rch a ses a n d th e m a n n e r in w h ic h th e y nav igate th ro u g h o u t th e W e b p a g e , targeted c h a n n e l m ark etin g a n d b e t­ te r cu sto m e r e x p e r ie n c e c a n n o w b e a ch iev ed .

P lu s, w h e n SAS fo r C u stom er E x p e rie n ce A nalytics w a s im p lem e n te d b y L o tte .c o m ’s largest o v e rse a s d istributor, it resu lted in a first-year sale s in c re a se o f 8 m illio n e u ro s (U S $ 1 0 m illio n ) b y id en­ tifying th e c a u s e s o f s h o p p in g -ca rt ab an d o n m en t.

Source: SAS, Customer Success Stories, sas.com/success/lotte .htm l (accessed March 2013).

8.6 W E B U S A G E M IN IN G (W E B A N A LY T IC S ) Web u sage m ining (a ls o ca lle d W eb analytics) is th e e x tra ctio n o f u sefu l in form ation fro m data g e n e ra te d th ro u g h W e b p a g e visits a n d tran saction s. M asand e t al. ( 2 0 0 2 ) state th a t at le a s t th ree ty p e s o f d ata are g e n e ra te d th ro u g h W e b p a g e visits:

1 . A u tom atically g e n e ra te d d ata s to red in s e r v e r a c c e s s log s, re ferrer log s, a g e n t logs, an d clie n t-sid e c o o k ie s

2 . U ser p rofiles 3 . M etad ata, su ch as p a g e attributes, c o n te n t attrib u tes, a n d u s a g e data.

A nalysis o f th e in fo rm atio n c o lle c te d b y W e b servers c a n h e lp u s b e tte r u n d erstan d u s e r b eh av io r. Analysis o f this data is o fte n ca lle d clickstream analysis. B y u sin g th e data a n d te x t m ining te ch n iq u e s , a co m p a n y m ig h t b e a b le to d isce rn in terestin g patterns fro m th e click stream s. F or e x a m p le , it m ight le a rn th at 6 0 p e rce n t o f visitors w h o s ea rch ed fo r “h o te ls in M aui” h a d s e a r c h e d e a rlie r fo r “airfares to M aui.” S u c h in form ation co u ld b e u sefu l in d eterm in in g w h e re to p la c e o n lin e ad vertisem en ts. C lick stream analysis m ight also b e u sefu l fo r k n o w in g w hen visitors a c c e s s a site. F o r e x a m p le , if a co m p a n y k n e w th at 7 0 p e rce n t o f s oftw are d o w n lo ad s fro m its W e b site o ccu rre d b e tw e e n 7 a n d 11 p .m ., it co u ld p lan fo r b e tte r cu sto m e r su p p o rt a n d n e tw o rk b an d w id th during th o se hours. Figure 8 .4 s h o w s th e p ro c e s s o f e x tra ctin g k n o w le d g e fro m click stre am data a n d h o w th e g e n e r a te d k n o w le d g e is u se d to im p ro ve th e p ro c e s s , im p ro v e th e W e b site, and , m ost im portant, in c re a se th e cu sto m e r value.

W e b m in in g h a s w id e a ran g e o f b u s in e s s ap p licatio n s. F o r in stan ce, Nasraoui ( 2 0 0 6 ) listed th e fo llo w in g six m o st co m m o n ap p licatio n s:

1 . D eterm in e th e lifetim e v alu e o f clients. 2 . D e sig n cro ss-m ark e tin g strateg ies a cro ss produ cts. 3 . E valuate p ro m o tio n a l cam paigns. 4 . T a rg e t e le c tro n ic ads an d co u p o n s at u s e r g ro u p s b a s e d o n u se r a c c e s s patterns. 5 . P red ict u s e r b e h a v io r b a s e d o n p rev iou sly le a rn e d ru les and u sers’ profiles. 6. P rese n t d y n am ic in form ation to u sers b a s e d o n th e ir in terests a n d profiles.

Chapter 8 • W e b Analytics, W eb Mining, and Social Analytics 3 8 9

U s e r / C u s to m e r

Web site

Weblogs

P r e p r o c e s s D a ta Collecting Merging Cleaning Structuring • Identify users • Identify sessions • Identify page

views • Identify visits

E x t r a c t K n o w led ge Usage patterns U ser profiles Page profiles Visit profiles Customer value

f How to better the data

How to improve the Web site

How to increase the customer value

FIGURE 8.4 Extraction o f K n ow led ge from W eb Usage Data.

A m a z o n .co m p ro v id es a n e x c e lle n t e x a m p le o f h o w W e b u sa g e history c a n b e lev erag ed d ynam ically . A reg iste re d u s e r w h o revisits A m a z o n .co m is g re e te d b y nam e. This is a s im p le task that involves re co g n iz in g th e u s e r b y re ad in g a c o o k ie (i.e ., a sm all te x t file w ritten b y a W e b site o n th e visitor’s co m p u te r). A m a z o n .co m a lso p re se n ts th e u s e r w ith a c h o ic e o f p ro d u cts in a p e rso n a liz e d sto re, b a s e d o n p re v io u s p u rch a ses and an a s s o cia tio n an alysis o f sim ilar users. It a lso m a k e s s p e c ia l “G o ld B o x ” o ffers that are g o o d fo r a s h o rt am o u n t o f tim e. All th e se re co m m en d a tio n s involve a d eta iled analysis o f th e v isito r as w e ll as th e u s e r’s p e e r g ro u p d e v e lo p e d th ro u g h th e u s e o f clusterin g, s e q u e n c e p a ttern d iscov ery, a sso cia tio n , a n d o th er d ata an d text m in in g te ch n iq u e s.

Web A n a ly tic s Te ch n o lo g ie s T h ere are n u m e ro u s to o ls a n d te c h n o lo g ie s fo r W e b analytics in th e m a rk e tp la ce . B e c a u s e o f th e ir p o w e r to m e asu re , c o lle c t, an d an alyze In te rn e t d ata to b e tte r u n d erstan d and o p tim ize W e b u sage, th e p o p u larity o f W e b an alytics to o ls is in cre a sin g . W e b analytics h old s th e p ro m ise to re v o lu tio n ize h o w b u sin ess is d o n e o n th e W e b . W e b an alytics is n o t ju st a to o l fo r m easu rin g W e b traffic; it c a n a lso b e u se d as a to o l fo r e -b u s in e s s and m arket re s e a rch , a n d to assess a n d im p ro ve th e e ffe ctiv e n e s s o f a n e -c o m m e r c e W e b site. W e b an alytics a p p lica tio n s ca n a lso h e lp c o m p a n ie s m e asu re th e resu lts o f traditional print o r b ro a d c a s t ad vertising cam p aig n s. It c a n h e lp e stim ate h o w traffic to a W e b site ch an ges a fte r th e lau n ch o f a n e w ad vertising cam p aig n . W e b analytics p ro v id e s inform a­ tion a b o u t th e n u m b e r o f visitors to a W e b site a n d th e n u m b e r o f p a g e view s. It h elp s gau ge traffic a n d pop u larity trend s, w h ic h c a n b e u s e d fo r m ark e t re sea rch .

T h e re a r e tw o m ain ca te g o rie s o f w e b analytics; o ff-site and o n -site . O ff-site W e b analytics re fe rs to W e b m e a su re m e n t and analysis a b o u t y o u an d y o u r p ro d u cts that tak es p lace o u tsid e y o u r W e b site. It in clu d es th e m e a su re m e n t o f a W e b s ite ’s p o ten tial audi­ e n c e (p r o s p e c t o r o p p ortu n ity ), s h a re o f v o ic e (visibility7 o r w o rd -o f-m o u th ), an d buzz (co m m e n ts o r o p in io n s ) th at is h a p p e n in g o n the Internet.

W h at is m o re m ain stream is o n -site W e b analytics. H istorically, W e b analytics has referred to o n -s ite visitor m easu rem ent. H ow ever, in re c e n t y ears this h a s blurred, m ainly b e ca u se v e n d o rs are p ro d u cin g to o ls that sp a n b o th categ o ries. O n -site W e b analytics m easure a v isitors’ b e h a v io r o n c e th e y are o n y o u r W e b site. T h is in clu d es its drivers and conversion s— fo r e x a m p le , th e d e g re e to w h ic h d ifferent landing p a g e s a re a sso ciated w ith

3 9 0 Part III • Predictive Analytics

o n lin e p u rch ases. O n -site W e b analytics m e asu re th e p e rfo rm an ce o f y o u r W e b site in a co m m e rcial co n tex t. T h is data c o lle c te d o n th e W e b site is th e n co m p a re d against k e y p e rfo rm an ce ind icators fo r p e rfo rm an ce , an d u s e d to im p ro ve a W e b site’s o r m arketing cam p aig n ’s a u d ien ce re sp o n se . E ven th o u g h G o o g le A nalytics is th e m o st w id ely -u sed o n ­ site W e b analytics serv ice, th e re are o th ers pro v id ed b y Y a h o o ! a n d M icrosoft, a n d n e w e r a n d b e tte r to o ls are em ergin g con stan tiy that pro v id e ad ditional layers o f inform ation.

F o r o n -site W e b analytics, th ere are tw o te ch n ica l w ays o f co lle ctin g th e data. T h e first an d m o re traditional m eth o d is th e server log file analysis, w h e re the W e b serv er records file requ ests m ad e b y brow sers. T h e seco n d m e th o d is p ag e tagging, w h ich uses Jav aScrip t e m b e d d ed in the site p a g e c o d e to m a k e im age req u ests to a third-party analytics-ded icated server w h en e v e r a p a g e is ren d ered b y a W e b b ro w se r (o r w h e n a m o u se click occu rs). B o th c o lle c t data th at ca n b e p ro ce s s e d to p ro d u ce W e b traffic reports. In ad dition to th ese tw o m ain stream s, o th er data so u rce s m ay also b e ad d ed to au gm en t W e b site b eh av io r data. T h e s e oth er so u rces m ay in clu d e e-m ail, d irect-m ail cam p aign data, sales an d lea d his­ tory, o r s o cial m e d ia-o rig in ate d data. A p plication C ase 8 .4 sh o w s h o w A llegro im proved W e b site p erfo rm an ce b y 5 0 0 p e rce n t w ith analysis o f W e b traffic data.

Application Case 8.4 A lleg ro Boo sts Online Click-Through Rates by 500 P e rcen t w ith W e b Analysis

T h e A llegro G ro u p is head qu artered in P osnan, Poland, an d is con sid ered the largest n o n -e B a y on lin e m ark etp lace in th e w orld. Allegro, w h ich currently offers o v e r 7 5 proprietary W e b sites in 11 E u ro p ean cou ntries aro u n d th e w orld, hosts o v e r 15 m illion p rodu cts an d gen erates o v er 5 0 0 m illion p a g e view s p e r day. T h e ch allen g e it fa ce d w a s h o w to m atch the right o ffe r to th e right cu stom er w h ile still b e in g ab le to support th e extraord inary am o u n t o f data it held.

P r o b l e m

In to d ay’s m a rk e tp la ce , b u y ers h av e a w id e variety o f retail, c a ta lo g , and o n lin e o p tio n s fo r buying th eir g o o d s a n d serv ices. A llegro is a n e -m a rk e tp la ce w ith o v e r 2 0 m illio n cu sto m ers w h o th e m se lv es b u y fro m a n e tw o rk o f o v e r 3 0 th o u san d p ro fe ssio n a l retail s ellers u s in g th e A llegro n e tw o rk o f e -c o m m e r c e and au c tio n sites. A lleg ro h a d b e e n su p p o rtin g its internal re co m m e n d a tio n e n g in e so lely b y applying rules p ro v id e d b y its re-sellers.

T h e c h a lle n g e w a s fo r A llegro to in cre a se its in c o m e a n d g ro ss m e rch a n d ise v o lu m e from its cu r­ re n t n e tw o rk , as m easu red b y tw o k e y p e rfo rm a n ce ind icato rs.

• C lic k -T h r u R a t e s (C T R ): T h e n u m b e r o f c lic k s o n a p ro d u ct ad d ivid ed b y th e n u m b e r o f tim e s th e p ro d u ct is d isplayed .

• C o n v e r s io n R a t e s : T h e n u m b e r o f c o m ­ p leted s a le s tran sactio n s o f a p ro d u ct d ivid ed b y th e n u m b e r o f cu sto m ers re ce iv in g the p ro d u ct ad.

S o lu tio n

T h e o n lin e retail indu stry h a s e v o lv e d in to th e p re m ie r c h a n n e l fo r p e rs o n a liz e d p ro d u ct re c o m ­ m e n d a tio n s. T o s u c c e e d in this in cre asin g ly c o m ­ p etitiv e e -c o m m e r c e e n v iro n m en t, A llegro realized th at it n e e d e d t o cre a te a n e w , h ig h ly p e rso n a liz e d so lu tio n in te g ratin g p re d ictiv e a n a ly tics a n d ca m ­ p a ig n m a n a g e m e n t in to a real-tim e re co m m e n d a ­ tio n system .

A llegro d e c id e d to apply S o cia l N etw ork Analysis (SNA) as th e an alytic m e th o d o lo g y u n der­ lying its p ro d u ct re co m m e n d a tio n system . SNA fo c u s e s o n th e relatio n sh ip s o r lin ks b e tw e e n n o d e s (ind ivid uals o r p ro d u cts) in a n e tw o rk , rath er than th e n o d e s ’ attrib u tes as in trad itional statistical m e th ­ od s. SNA w as u se d to g ro u p sim ilar p ro d u cts into co m m u n ities b a s e d o n th e ir co m m o n a litie s; th en , co m m u n ities w e re w e ig h te d b a s e d o n v isitor click p ath s, item s p la c e d in sh o p p in g carts, and p u rch a ses to c re a te p red ictiv e attributes. T h e g rap h in Figure 8.5 displays a fe w o f th e p ro d u ct co m m u n ities g e n ­ erated b y A lleg ro u sin g th e KX EN 's In fin iteln sig h t S o cia l p ro d u ct fo r so cia l n e tw o rk an alysis (SN A).

C hapter 8 • W eb Analytics, W eb Mining, and Social Analytics 3 9 1

F IG U R E 8 .5 The Product Communities Generated b y Allegro Using KXEN's In fin iteln sigh t. Source: KXEN.

Statistical classificatio n m o d els w e re th e n built using KXEN In fin iteln sigh t M o d eler to p re d ict c o n ­ v ersion p ro p e n sity fo r e a c h p ro d u ct b a s e d o n th ese SNA p ro d u ct co m m u n ities and individual cu stom er attributes. T h e s e co n v e rsio n p ro p en sity s co re s are th e n u se d b y A llegro to d efin e p e rso n alize d offers p resen ted to m illions o f W e b site visitors in real tim e.

S o m e o f th e c h a lle n g e s A llegro fa c e d ap p lyin g so cial n e tw o rk an alysis in clu d ed :

• N e e d to b u ild m ultiple n etw o rk s, d ep en d in g o n th e p ro d u ct g ro u p cate g o rie s - Very large d iffe ren ce s in th e fre q u e n cy d is­

trib u tio n o f p articu lar p ro d u cts a n d th eir p o p u larity (c lic k s , tran sactio n s)

• A u tom atic settin g o f op tim al p aram eters, su ch as th e m inim um n u m b e r o f o c c u rre n c e s o f item s (su p p o rt)

• A u tom atio n throu g h scripting • O v e rc o n n e c te d p ro d u cts (b e s t-s e lle rs , m e g a­

h u b co m m u n ities).

Im p le m en tin g this so lu tio n a lso p re se n te d its o w n ch a lle n g e s in clu d in g:

• D ifferen t ru le sets are p ro d u ce d p e r W e b p ag e p la cem en t

• B u s in e s s o w n e rs d e cid e ap p ro p riate w e ig h t­ ings o f ru le sets fo r e a c h ty p e o f p la c e m e n t / b u sin e ss strategy

• B u ild in g 1 6 0 k ru les e v ery w e e k • A u tom atic co n v e rs io n o f so cia l n e tw o rk analy­

s e s in to m le s a n d ta b le -izatio n o f rules

R e s u lts

As a result o f im p lem e n tin g s o c ia l n e tw o rk a n aly ­ sis in its a u to m a te d re a l-tim e re co m m e n d a tio n p ro ­ ce s s , A llegro h a s s e e n a m a rk e d im p ro v e m e n t in all areas.

T o d a y A lleg ro o ffers 8 0 m illio n p e rso n a liz e d p ro d u ct re co m m e n d a tio n s daily, an d its p a g e view s h a v e in cre a se d b y o v e r 3 0 p e rce n t. B u t it’s in th e

0Continued)

3 9 2 Part III • Predictive Analytics

Application Case 4.4 (Continued)

Rule ID A n tece d e n t product ID Consequent product ID

Rule support

Rule confidence Rule Kl

Belong to th e same product com m unity?

1 D IG IT A L C A M E R A LENS 2 12 13 2 0 % 0.7 6 Y E S

2 D IG IT A L C A M E R A M E M O R Y C A R D 3145 1 8 % 0 .6 4 N O

3 PIN K S H O E S PIN K D R E S S 4 34 3 3 8 % 0.5 5 N O

n u m b ers d eliv e red b y A lleg ro’s tw o m o st critical K PIs th at th e results a re m o s t ob viou s:

• C lick -th ro u gh rate (C T R ) h a s in cre a se d b y m o re th a n 5 0 0 p e rc e n t as c o m p a re d to 'b est s e lle r1 ru les.

• C onversion rates are up b y a facto r o f o v e r 40X.

Q u e s t i o n s f o r D i s c u s s i o n

1. ITow did A lleg ro sig nificantly im p ro ve click ­ throu gh rate s w ith W e b analytics?

2, W h a t w e re th e c h a lle n g e s, th e p ro p o s e d s o lu ­ tio n , an d th e o b ta in e d results?

Source: kxen.com/customers/allegro (accessed July 2013).

W eb A n a ly tic s M etrics U sing a variety o f d ata so u rce s, W e b an aly tics p ro g ram s pro v id e a c c e s s to a lo t o f valu­ a b le m arketin g d ata, w h ic h c a n b e lev era g e d fo r b e tte r insights to g ro w y ou r b u sin ess an d b etter d o cu m e n t y o u r ROT. T h e insight an d in te llig e n ce g a in ed fro m W e b analytics c a n b e u s e d to e ffe ctiv e ly m a n a g e th e m arketin g e ffo rts o f a n o rg an izatio n an d its various p ro d u cts o r serv ices. W e b an aly tics p rogram s p ro v id e n e arly real-tim e data, w h ich ca n d o cu m en t y o u r m ark etin g ca m p a ig n s u c c e s s e s o r e m p o w e r y o u to m a k e tim ely ad just­ m en ts to y o u r cu rren t m arketin g strategies.

W h ile W e b analytics p ro v id es a b ro a d ran g e o f m etrics, th e re a re fo u r cate g o rie s o f m etrics th at a re g e n era lly a ctio n a b le and c a n d irectly im p a ct y o u r b u sin e ss o b je ctiv e s (T W G , 2 0 1 3 ). T h e s e ca te g o rie s inclu d e:

• W e b site usability: H o w w e re they using m y W e b site? • T ra ffic so u rce s: W h e re did th ey c o m e from? • V isitor p ro files: W h at d o m y visitors lo o k like? • C o n v ersio n statistics: W h at d o e s all this m ean fo r th e busin ess?

W eb Site U sa b ility B e g in n in g w ith y o u r W e b site, le t’s tak e a lo o k a t h o w w e ll it w o rk s fo r y o u r visitors. T h is is w h e re y o u c a n learn h o w “u ser-frien d ly ” it really is o r w h e th e r o r n o t y o u are providing th e right co n ten t.

1 . P a g e v iew s. T h e m o st b a s ic o f m e asu re m e n ts, this m etric is u su ally p re se n te d as th e “av erag e p a g e v iew s p e r visito r.” I f p e o p le c o m e to y o u r W e b site an d d o n ’t view m an y p a g e s, th e n y o u r W e b site m ay h a v e issu es w ith its d esig n o r structure. A n oth er e x p la n a tio n fo r lo w p a g e v ie w s is a d is c o n n e c t in th e m ark etin g m e ssa g es that b ro u g h t th e m to th e site an d th e c o n te n t th a t is actu ally a v ailab le.

2 . T im e o n s it e . Similar to p a g e view s, it’s a fu nd am ental m easu rem en t o f a visi­ to r’s interaction w ith y ou r W e b site. G enerally, th e lo n g e r a p e rso n sp en d s o n y o u r W e b site, th e b etter it is. T h at co u ld m e a n they’re carefully review in g y o u r co n ten t, utilizing inter­ active co m p o n e n ts y o u h av e available, a n d b u ild in g tow ard a n in form ed d ecisio n to buy,

Chapter 8 • W eb Analytics, W eb Mining, and Social Analytics

respond , o r ta k e th e n e x t step y o u ’v e provided. O n th e contrary, the tim e o n site also n e ed s to b e e x a m in e d against th e n u m b e r o f p a g e s v ie w e d to m ak e sure the visitor isn t spending his o r h e r tim e trying to lo ca te co n ten t that shou ld b e m o re readily acce ssib le .

3 . D o w n lo a d s. T h is in clu d es P D F s, v id eo s, a n d o th e r re so u rce s y o u m ak e avail­ a b le to y o u r v isitors. C o n sid er h o w a c c e s s ib le th e s e item s are as w ell as h o w w e ll th e y re p ro m oted . I f y o u r W e b statistics, fo r e x a m p le , rev eal that 6 0 p e rc e n t o f th e individuals w h o w a tch a d e m o v id e o a ls o m a k e a p u rch a se , th e n y o u ’ll w a n t to strateg ize to in cre a se

view ersh ip o f th a t v id eo . 4. C lick m a p . M o st an aly tics p rogram s c a n s h o w y o u th e p e rc e n ta g e o f click s

e a c h item o n y o u r W e b p a g e re ce iv e d . T h is in clu d es c lic k a b le p h o to s , te x t links in you r co p y , d o w n lo a d s, an d , o f co u rse , an y nav igation y o u m ay h a v e o n th e p ag e . A re th ey

click in g th e m o st im p ortan t items? 5 . C lick p a th s. Although an assessm ent o f click paths is m ore involved, it can

quickly reveal w h e re y o u m ight b e losing visitors in a sp ecific process. A w ell-d esig ned W eb site uses a com bination o f graphics and inform ation architecture to en co u rag e visitors to fol­ low “p red efined ” paths through you r W e b site. T h e s e are n ot rigid pathw ays but rather intui­ tive steps that align w ith th e various p ro cesses you’v e built into the W e b site. O n e process might b e that o f “educating” a visitor w h o has m inim um understanding o f y o u r product or service. A nother m ight b e a p ro cess o f “motivating” a returning visitor to co n sid er an upgrade o r repu rchase. A third p ro cess m ight b e structured around item s y ou m arket online. Y o u ’ll have as m any p ro cess pathw ays in y o u r W e b site as you have target au d ien ces, products, and services. E a ch c a n b e m easured through W e b analytics to d eterm ine h o w effective th ey are.

Tra ffic Sources Y o u r W e b an aly tics p ro g ram is a n in cre d ib le to o l fo r id entifying w h e re y o u r W e b traf­ fic orig in ates. B a s ic ca te g o rie s s u ch as s e a r c h e n g in e s, referral W e b sites, a n d visits from b o o k m a rk e d p a g e s (i.e ., d irect) a re co m p ile d w ith little in v o lv e m e n t b y th e m arketer. W ith a little effo rt, h o w e v e r, y o u c a n a lso identify W e b traffic th a t w a s g e n e ra te d b y y ou i various o fflin e o r o n lin e ad vertising cam p aign s.

1 . R e fe r r a l Web sites. O th e r W e b sites that co n tain links that s en d visitors directly :o your*W eb site are co n sid ered referral W e b sites. Y o u r analytics program w ill identify e a c h referral site y o u r traffic co m e s from , and a d ee p e r analysis will h elp you d eterm in e w h ich referrals p ro d u ce th e greatest volu m e, th e h ighest con v ersion s, the m ost n e w visitors, etc.

2 . S e a rc h e n g in e s . D ata in the s e a r c h e n g in e ca te g o ry is d iv id ed b e tw e e n paid « a r c h and o rg a n ic (o r natu ral) se a rch . Y o u c a n re v iew th e to p k e y w o rd s th at g e n era ted W e b traffic t o y o u r site a n d s e e i f th e y are rep resen tativ e o f y ou r p ro d u cts a n d services. D ep en d in g u p o n y o u r b u sin ess, y o u m ight w an t to h av e hu nd red s (o r th o u sa n d s) o f k e y ­ w ord s th at d raw p o ten tial cu sto m ers. E v e n th e sim p lest p ro d u ct s e a rc h c a n h av e m ultiple variations b a s e d o n h o w th e individual p h rase s th e s e a rch query.

3 . D irect. D irect s e a r c h e s are attributed to tw o so u rce s. An individ ual w h o b o o k - rrarks o n e o f y o u r W e b p a g e s in th e ir fav orites a n d click s that lin k w ill b e re co rd e d as x d irect s e a rc h . A n o th er s o u rc e o ccu rs w h e n s o m e o n e ty p e s y o u r URL d irectly in to theii brow ser. T h is h a p p e n s w h e n s o m e o n e retriev es y o u r URL fro m a b u s in e s s card , b io - d m re , print ad , rad io co m m e rcial, e tc. T h a t’s w h y it’s g o o d strategy to u s e co d e d URLs.

4 . O fflin e ca m p a ig n s. I f you utilize ad vertising o p tio n s o th e r th a n W e b -b a s e d cam p aign s, y o u r W e b an aly tics p ro g ram c a n cap tu re p e rfo rm a n ce d ata if y ou 'll in clu d e i m ech an ism fo r sen d in g th e m to y o u r W e b site. T y p ically, this is a d ed ica te d URL that j o o inclu de in y o u r ad v e rtisem e n t ( i.e ., “w w w . m y c o m p a n y . c o m / o f f e r 5 0 ”) th at delivers b a se visitors to a s p e c ific lan d in g p a g e . Y o u n o w h av e data o n h o w m a n y re s p o n d e d to

■fa*? ad b y visitin g y o u r W e b site.

5 . O n lin e c a m p a ig n s . I f y o u are ru n n in g a b a n n e r ad cam p aig n , s e a rch en g in e ad vertising cam p aig n , o r e v e n e-m ail cam p aig n s, y o u c a n m e asu re individual cam p aign e ffe ctiv e n e s s b y sim p ly u sin g a d ed icate d URL sim ilar to th e o fflin e ca m p a ig n strategy.

V is ito r P ro file s O n e o f th e w ay s y o u c a n lev erag e y o u r W e b an aly tics in to a really p o w erfu l m arketing to o l is th ro u g h seg m en tatio n . B y b le n d in g d ata fro m d ifferen t an aly tics rep orts, y o u 11

b e g in to s e e a variety o f u ser p ro file s em erg e. 1 K ey w o rd s. W ithin y o u r analytics report, you ca n s e e w h at keyw ord s visitors

u se d in search e n g in e s to lo ca te y o u r W e b site. I f y o u ag gregate y o u r keyw ords b y similar attributes, y o u ’ll b eg in to s e e distinct visitor gro u p s that are using y o u r W e b site. F o r e x a m ­ p le, th e particular s ea rch phrase that w as u se d c a n in d icate h o w w e ll they understand you r prod u ct o r its b en efits. I f they u se w o rd s that m irror y o u r o w n p rodu ct o r service d escrip­ tions th e n th ey p ro bab ly are already aw are o f y o u r offering s fro m effective advertisem ents, b ro ch u re s, e tc. I f th e term s are m o re general in natu re, th e n y o u r visitor is se e k in g a solu­ tio n fo r a p ro b lem an d h a s h a p p e n e d u p o n y ou r W e b site. I f this s e c o n d gro u p o f searchers is sizab le, then y o u ’ll w an t to e n su re th at y ou r site has a strong ed u catio n co m p o n e n t to co n v in ce th e m they’v e fo u n d their an sw er an d th e n m o v e th e m into y ou r sales channel.

2 . C o n te n t g r o u p in g s . D e p e n d in g u p o n h o w y o u gro u p y o u r c o n te n t, you may b e a b le to an aly ze se c tio n s o f y o u r W e b site th at c o r re s p o n d w ith s p e c ific p ro d u cts, ser­ v ic e s, cam p aig n s, an d o th er m ark etin g tactics. I f y o u co n d u c t a lo t o f trad e sh o w s and drive traffic to y o u r W e b site fo r s p e c ific p ro d u ct literatu re, th e n y o u r W e b an alytics will

highlight th e activity in that sectio n . 3 G e o g r a p h y . A nalytics perm its y o u to s e e w h e re y o u r traffic g eo g rap h ically

o rigin ates, in clu d in g cou ntry , state, a n d city lo ca tio n s . T h is c a n b e e sp e cia lly u sefu l if y ou use g e o -ta rg ete d cam p aig n s o r w an t to m e asu re y o u r visibility a cro ss a region.

4 T im e o f d a y . W e b traffic g e n erally h a s p e a k s at th e b eg in n in g o f th e w o rk ­ d ay during lu n ch , an d to w ard th e e n d o f th e w o rk d ay . It’s n o t unusual, h o w e v e r to find stro n g W e b traffic e n te rin g y o u r W e b site u p until th e late e v e n in g . Y o u ca n an aly ze this data to d eterm in e w h e n p e o p le b ro w s e v e rsu s b u y a n d a lso m a k e d e cis io n s o n w h at h o u rs y o u sh o u ld o ffe r cu sto m e r service.

5 . L a n d in g p a g e p r o file s . I f you structure y ou r various advertising cam paigns prop­ erly, you c a n drive e a c h o f you r targeted groups to a different landing page, w h ich y o u r W e analytics will capture and m easure. B y com bining th e se num bers w ith th e dem ographics o f you r cam paign media, you can kn o w w h at percen tage o f you r visitors fit e a c h dem ographic.

C o n ve rsio n S ta tistics E a ch org an ization w ill d efin e a “co n v e rs io n " a c c o r d in g to its s p e cific m arketin g o b je c ­ tives. So m e W e b an alytics p rogram s u se th e te rm “g o a l” to b e n c h m a rk certain W e b site o b je ctiv e s , w h e th e r th at b e a certain n u m b e r o f v isitors to a p a g e , a c o m p le te d registra­

tio n form , o r a n o n lin e p u rch ase. 1 . N ew v is it o r s . I f y o u ’re w o rk in g to in c re a se visibility, y o u ’ll w an t to study th e

tren d s in y o u r n e w visitors data. A nalytics id en tifies all visitors as e ith e r n e w o r returning.

2 . R e t u r n in g v is it o r s . I f y o u ’re in v o lv e d in loyalty p rogram s o r o ffe r a p ro d u ct th at has a lo n g p u rch a se c y c le , th e n y o u r retu rnin g visitors data w ill h e lp y o u m easu re

p ro g re ss in this area. 3 . L e a d s . O n c e a form is subm itted an d a thank-you p ag e is generated, you hav e

created a lead. W e b analytics will perm it y ou to calcu late a co m p letio n rate (o r ab and o nm ent rate) b y dividing th e n u m b e r o f com pleted form s b y th e n u m ber o f W e b visitors that cam e to y o u r page. A lo w co m p letio n percen tage w o u ld indicate a p ag e that n e ed s attention.

Chapter 8 • W e b Analytics, W e b Mining, and Social Analytics 395

4 . S a le s /c o n v e r s io n s . D ep en d in g u p o n th e in ten t o f y o u r W e b s ite, y o u c a n d e fin e a “s a le ” b y a n o n lin e p u rch a se , a c o m p le te d registration, a n o n lin e su b m issio n , o r an y n u m b e r o f o th er W e b activities. M onitoring th e s e figu res w ill a lert y o u to any c h a n g e s (o r s u c c e s s e s !) that o c c u r fu rth e r up stream .

5 . A b a n d o n m e n t /e x it ra tes. Ju st as im portant as th o se m oving through your W eb site are th o se w h o b eg a n a p ro cess and quit o r cam e to your W e b site an d left after a page o r two. In th e first case, y o u ’ll w an t to analyze w h ere th e visitor term inated th e process and w h eth er th ere are a n u m ber o f visitors quitting at the sam e place. T h e n investigate the situa­ tion fo r resolution. In th e latter case, a high e xit rate o n a W e b site o r a sp ecific p ag e generally indicates an issu e with expectations. Visitors click to y o u r W e b site b ase d o n so m e m essage contained in a n advertisem ent, a presentation, etc., an d e x p e ct so m e continuity in that m e s­ sage. M ake sure y o u ’re advertising a m essag e that y ou r W e b site can reinforce an d deliver.

W ith in e a c h o f th e s e item s are m etrics that c a n b e e sta b lish e d fo r y o u r s p e cific organization. Y o u c a n cre a te a w e e k ly d ash b o ard th at in clu d e s s p e c ific n u m b e rs o r per­ ce n ta g e s th at w ill in d icate w h e re y o u ’r e s u cce e d in g — o r highlig ht a m arketin g ch a lle n g e that sh o u ld b e ad d ressed . W h e n th e s e m etrics are e v alu ated co n siste n tly an d u se d in co n ju n ctio n w ith o th e r av ailab le m arketin g data, th e y can lea d y o u to a h ig h ly q u an ti­ fied m ark etin g program . Fig u re 8 .6 sh o w s a W e b analytics d ash b o ard c re a te d w ith fre e ly

av ailab le G o o g le A nalytics tools.

Top Metrics

Revenue

R evenue €35,525.85

Total: 4 0 0 ,0 0 1 4 ( £ 3 5 ,5 2 5 .8 5 )

- D E L E T E D A SH BO A RD

# Timeline

20,008

Feb 2 0 , 2 0 1 1 - Mar 22 , 2 0 1 1

# B row ser Breakdown

2Q,«Dfl

V isits 10,000 10,000

V i s i t s 389,427 Feb 20 Fab 27 Marfi Mar 13

r~ ‘ ..........; fa Oi lOlaU 1 Uir < w, to f Top Countries *

B o u n ce Rate *1*

B o u n c e R a t e 8 B B E 1 § § g g 2 2 1

66.03% U nited Sta te s 107,521 5 5 .9 6 % Site Avg: 66.03% (3.00%) U nited K in g d o m 26 ,14 1 7 4 .3 1 %

B ra zil 2 4 ,7 3 5 7 5 .8 3 %

A vg . Time on Site * C a n a d a 16 ,41 7 5 7 .3 2 %

A v g . T i m e o n S i t e S p a in 1 5 ,01 2 7 8 .8 6 %

53 sec. M exico 1 2 ,3 6 9 5 2 .7 6 % B i t e Avg: 53 see, (8.00%) In dia 11 ,81 6 6 8 .3 2 %

F ra n c e 1 0 ,2 0 4 6 0 .8 7 %

Italy 9 ,7 7 8 6 6 .1 9 %

49.72% Internet Explorer 193,614 Visits

! 21.92% Chrom e 85,350 Visits

1 18.55% Firefox 72,247 Visits

6.32% Safari 24,599 Visits

1.19 % Opera 4,629 Visits

i 2.31% Other 8,968 Visits

FIG U RE 8 .6 A Sam ple W eb Analytics Dashboard.

3 9 6 Part III * Predictive Analytics

SECTION 8 . 6 REVIEW QUESTIONS

1 . W h a t a re th e th re e ty p e s o f d ata g e n e ra te d th ro u g h W e b p a g e visits?

2. W h at is click stre am analysis? W h at is it u s e d for? 3 . W h a t are th e m ain ap p lica tio n s o f W e b m ining? 4 . W h at are co m m o n ly u s e d W e b an aly tics m etrics? W h a t is th e im p o rta n ce o f metrics?

8.7 WEB A N A L Y T IC S M ATURITY M O DEL AN D WEB A N A L Y T IC S TOOLS

T h e te rm “m aturity” relates to th e d e g re e o f p ro ficie n cy , form ality, a n d op tim izatio n o f b u sin e ss m o d e ls, m o vin g “a d h o c ” p ra ctice s to fo rm ally d efin e d s te p s a n d op tim al b u si­ n e s s p ro ce s s e s. A m aturity m o d e l is a form al d e p ic tio n o f critical d im en sio n s a n d th eir c o m p e te n c y lev els o f a b u sin e ss p ractice. C o llectiv ely, th e s e d im en sio n s an d lev els d efine th e m aturity lev el o f a n org an izatio n in that a re a o f p ra ctice . It o fte n d e s crib e s a n ev olu ­ tio n ary im p ro v em en t p ath fro m ad h o c , im m atu re p ra ctice s to d iscip lin ed , m atu re p ro­

c e s s e s w ith im p ro ved quality an d e fficie n cy . A a o o d e x a m p le o f maturity m o d els is th e B I Maturity M odel d ev elo p e d b y T h e Data

W are h o u se Institute (T D W I). In th e T D W I B I Maturity M odel th e m ain p u rp ose w as to g au g e w h ere organization data w areh o u sin g initiatives are a t a p o in t in tim e and w h ere it sh ou ld g o next. It w as re p resen ted in a six-stage fram ew ork (M anagem ent Reporting

Spreadm arts D ata Marts D ata W a re h o u se Enterprise D ata W are h o u se B I S ervices). A nother related ex am p le is th e sim ple b u sin ess analytics maturity m o d el m oving from sim ple descriptive m easu res to pred icting future ou tco m es, to o b tain in g sophisticated d ecisio n system s (i.e ., D escriptive Analytics Pred ictive Analytics P rescriptive A nalytics).

F o r W e b an aly tics p e rh a p s th e m o st co m p re h e n siv e m o d e l w as p ro p o s e d by S te p h a n e H am el (2 0 0 9 ). In this m o d e l, H am el u sed s ix d im en sio n s— ( 1 ) M anagem ent, G o v e rn a n ce an d A d option, ( 2 ) O b je ctiv e s D efin itio n , ( 3 ) S co p in g , ( 4 ) T h e A nalytics T e a m a n d E xp ertise, ( 5 ) T h e C o ntinu o u s Im p ro v e m e n t P ro c e s s a n d A nalysis M eth o d o lo g y , (6 ) T o o ls , T e c h n o lo g y an d D a ta In teg ratio n — a n d fo r e a c h d im en sio n h e u s e d s ix lev els o f p ro ficie n cy / co m p ete n ce . Figure 8 .7 s h o w s H am e l’s s ix d im en sio n s a n d th e resp ectiv e

p ro ficie n cy lev els. . . . T h e proficiency/ com p etence levels have d ifferent terms/labels for e a c h o f th e six dim en­

sions, d escribing specifically w hat e a c h level m eans. Essentially, th e six levels are indications o f analytical maturity ranging from “O-Analytically Im paired” to ^ -A n a ly tica l C om petitor.' A short description o f e ach o f th e six levels o f c o m p eten cie s is given h e re (H am el, 2009):

1. Im p a ire d : C h aracterized b y the u se o f o u t-o f-th e-b o x to o ls an d reports; limited re so u rces lackin g form al training (h a n d s-o n skills) an d ed u catio n (k n o w led g e ). W e b analyt­ ics is u se d o n an ad h o c b asis and is o f limited v alu e a n d s c o p e . So m e tactical o b jectiv es are defined , but results are n o t w ell com m u n icated an d th ere are m ultiple v ersion s o f the truth.

2. In itia te d : W o rk s w ith m etrics to o p tim ize s p e c ific are a s o f th e b u sin e ss (s u ch as m arketin g o r th e e -c o m m e rc e ca ta lo g u e ). R e so u rce s are still lim ited b u t th e p ro cess is g etting stream lined . R esults are co m m u n ica te d to vario u s b u sin e ss sta k e h o ld e rs (o lte n d irecto r lev el). H ow ever, W e b an aly tics m ight b e su p p o rtin g o b s o le te b u sin e ss p ro c e s s e s an d , thu s, b e lim ited in th e ab ility to p u sh fo r op tim izatio n b e y o n d th e o n lin e ch an n e l.

S u cc e s s is m o stly a n e cd o ta l. 3. O p era tio n a l: K e y p e rfo rm a n ce in d ica to rs an d d ash b o ard s a re d efin e d an d

a lig n e d w ith strateg ic b u sin e ss o b je ctiv e s . A m u ltid iscip linary te a m is in p la c e an d u ses vario u s so u rce s o f in form ation s u ch as co m p e titiv e d ata, v o ic e o f cu sto m e r, and s o cial m e d ia o r m o b ile analysis. M etrics are e x p lo ite d a n d e x p lo r e d throu gh s e g m en ta tio n an d m u ltivariate testin g . T h e In te rn e t c h a n n e l is b e in g o p tim ized ; p e rso n a s are b e in g d efined .

Chapter 8 • W eb Analytics, W eb Mining, and Social Analytics 3 9 7

Results start to a p p e a r a n d b e c o n s id e re d a t the e x e cu tiv e lev el. R esu lts are centrally driven, b u t b ro a d ly distributed.

4 . I n t e g r a t e d : A nalysts c a n n o w co rre la te o n lin e an d o fflin e data fro m vari­ o u s so u rce s t o pro v id e a n e a r 3 6 0 -d e g re e v ie w o f th e w h o le v alu e ch ain . O ptim ization e n c o m p a ss e s co m p le te p ro c e s s e s, in clu d in g b a c k -e n d and fro n t-en d . O n lin e activities are d efin e d fro m th e u s e r p e rsp e ctiv e an d p e rsu a sio n s ce n a rio s are d e fin e d . A co n tin u o u s im p ro v e m en t p ro ce s s a n d p ro b lem -so lv in g m e th o d o lo g ie s are prevalent. Insight an d re c ­ o m m en d atio n s re a c h th e C X O lev el.

5 . C o m p e tito r : T h is lev el is ch aracterize d b y sev eral attributes o f co m p a n ie s w ith a stro n g a n aly tical cu ltu re (D a v e n p o rt and Harris, 2 0 0 7 ):

a . O n e o r m o re s e n io r e x e cu tiv e s stro n g ly a d v o ca te fa c t-b a s e d d e c is io n m ak in g an d analytics

b . W id e sp re a d u s e o f n o t ju st d escrip tive statistics, b u t p red ictiv e m o d e lin g and c o m p le x op tim izatio n te ch n iq u e s

c . Su b stan tial u s e o f an aly tics acro ss m u ltiple b u sin e ss fu n ctio n s o r p ro c e s s e s d . M o v e m e n t to w ard a n e n te rp rise-le v e l a p p ro a ch to m an agin g an aly tical to ols,

d ata, an d organ ization al skills an d cap ab ilities.

6. A d d ic t e d : T h is level m atch es D av e n p o rt’s “Analytical C o m p e tito r” ch a ra c­ teristics: d e e p strategic insight, co n tin u o u s im p ro v em en t, integ rated , s k ille d re so u rces, top m a n a g e m e n t com m itm en t, fact-b ase d cu ltu re, c o n tin u o u s testin g, lea rn in g , an d m o st im portant: fa r b e y o n d th e b o u n d a rie s o f th e o n lin e ch an n e l.

In F ig u re 8 .7 , o n e c a n m ark th e lev el o f p ro ficie n cy in e a c h o f th e s ix d im en sio n s to c re a te th e ir o rg a n iz a tio n ’s m aturity m o d e l (w h ic h w o u ld lo o k lik e a s p id e r diagram ).

Tools S tra te g ic

C R M eM arketing

Behavioral Optimization W e b m e trics

N o W eb

M anagement [5 ] Com petitive analytics [5] (4 ] Culture (3 ] S e n io r m anagem ent

D ire cto r A project

champion

on analytics optimization

[3 ] e B u s in e s s optimization (2 ) eM arketing optimization (1 ) R e q u e st list (0 ) Undefined

Scop e (5) Com peting on analytics [4] Online eco system (3 ) S in g le w ebsite (2) Sp e c ific online activity (1) H IPPO (O) Im provisation

Agile approach Agile methodology (online] [4)

Continuous im provem ent p ro c e ss (3) D e p a rtm e n t/te a m m ethod [2)

A nalyst's own (1) N o methodology (0)

R esources Experienced/m ultidisciplinary (5]

M ultidisciplinary (4) D istributed te am (3)

Single an alyst (2) P ro je ct ap pro ach (1)

N o dedicated re so u rce (O)

FIG U RE 8 .7 A Framework fo r W eb Analytics Maturity Model.

S u c h a n a sse ssm e n t c a n h e lp o rg an ization s b etter u n d e rstan d a t w h at d im en sio n s th e y are lagging b eh in d , a n d ta k e co rrectiv e a ctio n s to m itigate it.

Web A n a ly tics Tools T h e r e are p le n ty o f W e b an aly tics ap p lica tio n s (d o w n lo a d a b le so ftw a re to o ls a n d W e b - b a se d / o n -d e m a n d serv ice p latfo rm s) in th e m a rk e t. C o m p a n ie s (la rg e , m ed iu m , o r sm all) are cre a tin g p ro d u cts an d s e rv ice s to g rab th e ir fa ir s h a re fro m th e e m e rg in g W e b a n a ­ ly tics m ark e tp la ce . W h at is th e m o st in te restin g is th a t m an y o f th e m o st p o p u la r W e b an aly tics to o ls a re fre e — y e s, fre e to d o w n lo a d a n d u s e fo r w h a te v e r re a s o n s , co m m e r­ cia l o r n o n p ro fit. T h e fo llo w in g a re am o n g th e m o st p o p u la r fre e (o r a lm o st fr e e ) W e b

an aly tics to ols:

G O O G LE W EB A N A L Y T IC S (G O O G L E .CO M /A N A LYT ICS) T h is is a serv ice o ffe re d b y G o o g le th at g e n e ra te s d etailed statistics a b o u t a W e b site’s traffic an d traffic so u rce s and. m e a su re s c o n v e rs io n s an d sales. T h e p ro d u ct is aim ed at m ark eters as o p p o s e d to the W e b m a ste rs and te ch n o lo g ists fro m w h ic h th e indu stry o f W e b analytics originally grew . It is th e m o st w id ely u s e d W e b an aly tics s erv ice. E v e n th o u g h th e b a sic serv ice is fre e o f ch a rg e , th e p rem iu m v e rs io n is av ailab le fo r a fe e .

Y A H O O ! W EB A N A L Y T IC S (W E B .A N A L Y T IC S .Y A H O O .C O M ) Y a h o o ! W e b an alytics is Y a h o o l’s altern ativ e to th e d o m in an t G o o g le A nalytics. It’s an e n te rp rise-le v e l, ro bu st W e b -b a s e d third-party so lu tio n th at m ak e s a c c e s s in g data e asy , e sp ecia lly fo r m u ltip le- u se r groups. It’s g o t all th e th in gs y o u ’d e x p e c t from a co m p re h e n siv e W e b analytics to o l, s u ch as pretty g rap h s, cu sto m -d e sig n e d (a n d p rin tab le ) rep orts, and real-tim e data

tracking.

OPEN W EB A N A L Y T IC S (O P E N W E B A N A L YT IC S.C O M ) O p e n W e b A nalytics (O W A ) is a p o p u la r o p e n s o u rc e W e b an alytics so ftw are that a n y o n e c a n u s e to tra ck an d an alyze h o w p e o p le use W e b sites an d ap p licatio n s. O W A is lic e n s e d u n d e r GPL a n d provides W e b site o w n e rs an d d ev elo p e rs w ith e a sy w ay s to a d d W e b an alytics to th eir sites usin g sim p le Jav ascrip t, PH P, o r R E ST -b ased APIs. O W A a lso c o m e s w ith b u ilt-in su p ­ p o rt fo r track in g W e b sites m ad e w ith p o p u lar c o n te n t m an ag e m e n t fram ew o rk s such as

W o rd P ress an d M ediaW iki.

PIWIK (P1W IK.ORG) P iw ik is th e o n e o f th e lead in g self-h o ste d , d ecen tralized , o p e n so u rce W e b an alytics platfo rm s, u se d b y 4 6 0 ,0 0 0 W e b sites in 1 5 0 co u n tries. P iw ik w as fo u n d e d b y M atthieu A ubry in 2 0 0 7 . O v e r th e la st 6 years, m o re talen ted and p a ssio n a te m e m b ers o f th e com m u n ity h av e jo in e d th e te a m . As is th e c a s e in m an y o p e n so u rce initiatives, th e y are activ ely lo o k in g fo r n e w d e v e lo p e rs , d esig n ers, d atavis a rch ite cts, an d

s p o n so rs to jo in them .

FIRESTAT (FIRESTATS.CC) FireStats is a sim p le an d straightforw ard W e b analytics ap p li­ ca tio n w ritten in PHP/MySQL. It su p p orts n u m e ro u s p latfo rm s an d set-u p s inclu d ing C# sites, D ja n g o sites, D ru pal, Jo o m la !, W ord P ress, a n d sev eral oth ers. FireStats h a s an intui­ tive API th at assists d e v e lo p e rs in creatin g th e ir o w n cu sto m a p p s o r p u b lish in g platform

c o m p o n e n ts.

SITE METER (SITEM ETER.COM ) Site M eter is a s e rv ic e that p ro v id es c o u n te r a n d track in g in form ation fo r W e b sites. B y lo g g in g IP a d d re sse s a n d u sin g Ja v a S crip t o r HTML to track visitor in form ation , Site M eter p ro v id es W e b site o w n e rs w ith in form ation a b o u t th eir v isitors, in clu d in g h o w th ey re a ch e d th e site, th e d a te and tim e o f th e ir visit, a n d m ore.

3 9 8 Part III • Predictive Analytics

Chapter 8 * W eb Analytics, W e b Mining, and Social Analytics 3 9 9

w o o p r a (W O O P R A .C O M ) W o o p ra is a re a l-tim e cu sto m e r a n a ly tics s e r v ic e th at p ro v id e s s o lu tio n s fo r s a le s , s e r v ic e , m a rk e tin g , and p ro d u ct te a m s. T h e p la tfo rm is d e s ig n e d to h e lp o rg a n iz a tio n s o p tim iz e th e c u s to m e r life c y c le b y d e liv e rin g live, g ran u lar b e h a v io ra l d ata fo r ind ivid u al W e b site v isito rs an d c u sto m e rs . It tie s this in d iv id u a l-lev el d ata to ag g re g a te a n a ly tics re p o rts fo r a fu ll life -c y c le v ie w th a t b rid g e s

d e p a rtm e n ta l gap s.

A W S T A T S (A W ST A T S .O R G ) AW Stats is a n o p e n s o u rc e W e b an aly tics re p o rtin g to o l, suit­ a b le fo r a n aly zin g data from In te rn e t serv ices s u ch as W e b , stream in g m e d ia , m ail, and FTP servers. AW Stats p a rse s an d an aly zes serv e r lo g files, p ro d u cin g HTML reports. D ata is visually p re s e n te d w ith in rep o rts b y ta b le s and b a r grap h s. Static re p o rts c a n b e c ie a te d th ro u g h a c o m m a n d lin e in te rface , an d o n -d e m a n d rep ortin g is su p p o rte d th ro u g h a W e b b ro w se r C G I program .

SN O O P (REINVIGORATE.NET) Sn o o p is a d esk to p -b ased ap p licatio n that runs o n th e Mac O S X and W ind ow s XP/Vista platforms. It sits nicely o n y o u r system status bar/system tray, notifying y o u w ith aud ible sou nd s w h en e v e r som ething hap p en s. A nother outstanding Sn o o p featu re is th e N am e Tags o p tio n , w h ich allow s you to “tag” visitors fo r easier identifi­ cation. So w h e n J o e o v er at the accou n tin g departm ent visits y o u r site, y o u ’ll instantly know .

M 0 C H IB 0 T (M O CH IB O T.CO M ) M o ch iB o t is a fre e W e b analytics/tracking to o l e sp ecially d esig n e d fo r F lash assets. W ith M o ch iB o t, y o u c a n s e e w h o ’s sh arin g y o u r F lash co n ten t, h o w m an y tim e s p e o p le v ie w y o u r c o n te n t, a s w e ll as h e lp y o u track w h e re y o u r Flash co n te n t is t o p re v e n t p ira cy and co n te n t theft. Installing M o ch iB o t is a b re e z e ; y o u sim ply c o p y a fe w lin e s o f A ctio nScrip t c o d e in th e .FLA file s y o u w an t to m o n ito r.

In ad d ition to th e s e free W e b analytics to o ls, T a b le 8 .1 p ro v id es a list o f co m m e r­ cially a v ailab le W e b an alytics to o ls.

T A B L E 8.1 Commercial W e b A n alytics So ftw a re Tools

Product Nam e Description URL

A n g o ss K n o w le d g e C o m b in e s A N G O S S K n o w le d g e angoss.com W e b M in e r S T U D IO a nd clickstream analysis

ClickTracks V isito r pattern s can be s h o w n on W e b site

clicktracks.com, n o w a t Lyris.com

LiveStats fro m Real-tim e log analysis, live d em o deepm etrix.com D e ep M etrix o n site

M e g a p u te r W e b A n a ly s t D ata an d tex t m ining capabilities m egaputer.com /site/textanalyst.php

M ic ro S tra te g y W e b T raffic highlights, c o n te n t analysis, m icrostrategy.com /Solutions/Applications/W TAM

Traffic A n alysis M o d u le an d W e b visitor analysis reports

S A S W e b A n alytics A n alyze s W e b site traffic sas.com /solutions/webanalytics

S P S S W e b M in in g for Extraction o f W e b events w w w - 01.ibm.com/software/analytics/spss/

C le m e n tin e

W e b T re n d s D ata m inin g o f W e b traffic info rm atio n .

w ebtrends.com

X M L M in e r A system and class library fo r m inin g d ata and tex t expressed in X M L , using fuzzy log ic expert system rules

scientio.com

4 0 0 Part III • Predictive Analytics

P u ttin g It A ll T o ge th e r—A Web S ite O p tim iza tio n Ecosystem It s e e m s th a t ju st a b o u t ev e ry th in g o n th e W e b c a n b e m e a su re d — e v ery c lic k c a n b e re co rd e d , e v e ry v ie w c a n b e cap tu re d , an d e v e ry visit c a n b e an aly zed all in an effort to co n tin u a lly an d a u to m atically o p tim iz e th e o n lin e e x p e r ie n c e . U n fortu n ately , th e n o tio n s o f “in fin ite m e asu rab ility ” a n d “a u to m a tic o p tim iz a tio n ” in th e o n lin e c h a n n e l a ie fa r m o re c o m p le x th a n m o st re alize . T h e a ssu m p tio n th at a n y sin gle a p p lica tio n o f W e b m in in g te c h n iq u e s w ill p ro v id e th e n e c e s s a ry ra n g e o f in sig h ts re q u ire d to u n d erstan d W e b site v isitor b e h a v io r is d e c e p tiv e an d p o ten tia lly risky. Id eally , a h o listic v ie w to cu sto m e r e x p e r ie n c e is n e e d e d th a t c a n o n ly b e ca p tu re d u sin g b o th q u an titativ e an d q u alitativ e data. Fo rw ard -th in k in g c o m p a n ie s h a v e a lread y ta k e n s te p s to w ard cap tu rin g an d an aly zin g a h o listic v ie w o f th e cu sto m e r e x p e r ie n c e , w h ich has le d to sig n ifican t g a in s, b o th in te rm s o f in cre m en ta l fin an cial g ro w th a n d in cre a sin g cu sto m e r loy alty an d satisfactio n .

A cco rd in g to P e te rs o n (2 0 0 8 ), th e inputs fo r W e b site op tim izatio n e ffo rts ca n b e classified alo n g tw o a x e s d escrib in g th e n atu re o f th e data and h o w that d ata c a n b e used. O n o n e a x is are data an d inform ation— data b e in g prim arily q u an titative and in form ation b e in g prim arily qualitative. O n th e o th e r a x is are m e a su re s a n d actio n s— m easu res b e in g re p o rts, analysis, an d re co m m en d a tio n s all d e sig n e d to drive actio n s, th e actu al ch a n g e s b e in g m ad e in th e o n g o in g p ro c e s s o f site an d m ark e tin g op tim ization . E a ch quadrant cre a te d b y th e s e d im en sio n s lev erag e s d ifferen t te c h n o lo g ie s an d cre a te s d ifferen t o u t­ pu ts, b u t m u ch like a b io lo g ica l e co sy stem , e a c h te ch n o lo g ica l n ic h e in teracts w ith th e o th ers to su p p o rt th e en tire o n lin e en v iro n m en t (s e e Fig u re 8 .8 ).

M o st b e lie v e th at th e W e b site o p tim ization e co s y s te m is d efin e d b y th e ability to log, p arse, a n d rep o rt o n th e click stre am b e h a v io r o f site visitors. T h e u n d erly ing te c h ­ n o lo g y o f this ability is g e n erally re fe rre d to as W eb an aly tics. A lthough W e b analytics

Actions (actual changes]

Personalization Testing and and Content

Targeting Management

Quantitative [data)

The nature of the data

IE

Voice of the Customer and

Customer Experience

Management

Measures [reports/analyses]

Qualitative (information)

F IG U R E 8 .8 T w o -D im e n sio n a l V ie w o f th e In p u ts f o r W e b Site O p tim iza tio n .

Chapter 8 • W eb Analytics, W eb Mining, and Social Analytics 401

to o ls p ro v id e in v alu ab le insights, u n d erstan d in g visitor b e h a v io r is as m u ch a fu n ctio n o f q u alitativ ely d eterm in in g in terests and intent as it is qu antifying c lic k s fro m p a g e to p age. F o rtu n ately th e re a re tw o o th e r cla ss e s o f a p p lica tio n s d esig n e d t o p ro v id e a m o re qualitative v ie w o f o n lin e visitor b e h a v io r d esig n e d to re p o rt o n th e o v erall u se r e x p e ri­ e n c e a n d re p o rt d irect fe e d b a c k g iv en b y visitors an d cu stom ers: cu sto m er experience m anagem ent (CEM) a n d voice o f cu sto m er (VOC):

• W e b an aly tics ap p lica tio n s fo cu s o n “w h e re and w h e n ” q u e stio n s b y aggregating, m in in g , an d visualizing large v o lu m e s o f d ata, b y rep orting o n o n lin e m arketin g and v isitor acq u isitio n effo rts, b y sum m arizing p ag e -le v e l visitor in te ra ctio n data, a n d b y su m m arizin g v isitor flow th ro u g h d efin e d m u ltistep p ro ce sse s.

• V o ic e o f cu sto m e r a p p lica tio n s fo c u s o n “w h o an d h o w '’ q u e stio n s b y gath erin g and re p o rtin g d irect fe e d b a c k fro m site visitors, b y b e n ch m a rk in g a g ain st o th e r sites and o fflin e ch a n n e ls, a n d b y su p p o rtin g p red ictive m o d e lin g o f future v isitor b eh av io r.

• C u sto m er e x p e rie n c e m a n a g e m e n t a p p lica tio n s fo cu s o n “w h at a n d w h y ” q u e s­ tio n s b y d etectin g W e b a p p lica tio n issu es an d p ro b lem s, b y tra ck in g an d re so lv ­ ing b u s in e s s p ro ce s s an d usability o b s ta cle s , b y re p o rtin g o n site p e rfo rm a n ce an d availability, b y e n a b lin g real-tim e alertin g an d m o n ito rin g , a n d b y su p p o rtin g d e e p d ia g n o sis o f o b s e rv e d visitor b eh av io r.

All th ree applications are n e ed e d to have a com p lete v iew o f th e visitor behav ior w h ere e a c h application plays a distinct and valu ab le role. W e b analytics, CEM, and VOC applications fo rm th e found ation o f the W e b site op tim ization ecosy stem that supports the online b u sin ess’s ability to positively in flu ence desired ou tcom es (a pictorial representation o f this p ro cess v ie w o f th e W e b site optim ization ecosy stem is given in Figure 8 .9 ). T h e se sim ilar-yet-distinct applications e a c h contribute to a site op erator's ability to recog n ize, react, and resp on d to th e on go in g ch allen g es faced b y every W e b site ow ner. Fundam ental to the optim ization p ro cess is m easurem ent, gathering data an d inform ation that ca n then b e trans­ form ed into tangible analysis, an d recom m end ations for im provem ent u sin g W e b m ining tools an d tech n iq u es. W h e n u se d properly, th e se applications allow fo r co n v erg en t valida­ tion— com b in in g different sets o f data co lle cte d fo r th e sam e au d ien ce to provide a richer and d ee p e r understanding o f au d ien ce behavior. T h e con v ergen t validation m odel— o n e

Customer Interaction Knowledge about the Holistic on the Web Analysis of Interactions View of the Customer

R G U R E 8 .9 A Process V iew o f the W eb Site O ptim ization Ecosystem.

4 0 2 Part III • Predictive Analytics

w h e re multiple so u rces o f data d escribing the sa m e population are integrated to increase th e d ep th and rich n ess o f th e resulting analysis— fo rm s th e fram ew ork o f th e W e b site opti­ m ization ecosystem . O n o n e side o f th e sp ectru m are th e primarily qualitative inputs from V O C applications; o n th e o th er sid e are th e prim arily quantitative inputs from CEM bridg­ ing the gap b y supporting k e y elem en ts o f data discovery. W h e n properly im plem ented, all th ree system s sam ple data from th e sam e au d ien ce. T h e com b in ation o f th e se data either through data integration projects o r sim ply via th e p ro cess o f cond u ctin g g o o d analysis— supports far m o re action ab le insights th an an y o f th e e co sy stem m em b ers individually.

A Fram ew o rk fo r V o ice o f th e C u sto m e r S tra te g y V o ice o f th e cu stom er (V O C ) is a term usually u se d to d escrib e th e analytic p ro cess o f capturing a cu stom er’s exp ectation s, p re fe re n ce s, an d aversions. It essentially is a m arket research tech n iqu e that p rodu ces a detailed set o f cu stom er w an ts and n eed s, organized into a hierarchical structure, an d then prioritized in term s o f relative im portance and satisfaction w ith current alternatives. Attensity, o n e o f th e innovative service providers in th e analytics m arketplace, d ev elop ed a n intuitive fram ew ork fo r V O C strategy that they called LARA, w h ich stands for Listen, Analyze, Relate, and Act. I t is a m eth o d o lo g y that outlines a process b y w h ich organizations c a n take user-generated co n te n t (U G C ), w h eth er generated b y c o n ­ sum ers talking in W e b forum s, o n m icro-blogging sites like Tw itter an d social netw orks like F a ce b o o k , o r in fe e d b a ck surveys, e-m ails, d ocu m en ts, research, etc., and using it as a busi­ ness asset in a b u sin ess process. Figure 8 .1 0 sh o w s a pictorial d ep iction o f this fram ew ork.

LISTEN T o “liste n ” is actu ally a p ro ce s s in its e lf th a t e n c o m p a ss e s b o th th e cap ab ility to listen to th e o p e n W e b (foru m s, b lo g s , tw ee ts, y o u n a m e it) an d th e cap ab ility to se a m ­ lessly a c c e s s e n te rp rise in form ation (CRM n o te s , d o cu m en ts, e-m ails, e tc .). It ta k e s a listen in g p o st, d e e p fe d era ted sea rch ca p a b ilitie s, scrap in g a n d e n te rp rise class data in te­ gration, an d a strategy to d eterm in e w h o a n d w h a t y o u w a n t to listen to.

A N A L Y Z E Th is is th e hard part. H o w c a n y o u ta k e all o f this m ass o f u n sta ic tu re d data a n d m a k e s e n s e o f it? T h is is w h e re the “s e c r e t s a u c e ” o f te x t an alytics c o m e s in to play. Look fo r so lu tio n s that in clu d e k ey w o rd , statistical, an d natural lan g u ag e ap p ro a ch e s

M A R K E T IN G

PRODUCT

H OPERATIONS

F IG U R E 8 .1 0 V o ic e o f th e C u s to m e r S tra te g y F ram e w o rk. Source: A tte n s ity .co m . Used with permission.

C h a p te r s • Web Analytics, W eb Mining, and Social Analytics 4 0 3

that w ill a llo w y o u to e sse n tia lly ta g o r b a rc o d e e v e ry w o rd an d th e re la tio n sh ip s am o n g w ord s, m ak in g it data th at ca n b e a cce ss e d , s e a rch e d , routed , co u n te d , a n a i y z ^ . charted , rep orted o n , an d e v e n reu sed . K e e p in m in d that, in ad dition to te c h n ic a l it has to b e e a sy to u s e , s o th a t y ou r b u sin e ss users c a n fo c u s o n th e insights, n o t th e te ch n o lo g y . It sh o u ld h av e an e n g in e th at d o e s n 't req u ire th e u s e r to d e fin e k e y w o rd s or term s th at th e y w an t th e ir sy stem to lo o k fo r o r in clu d e in a ru le b a s e . R ather it sh ou ld a u to m a tica lly identify term s ( “fa cts ,” p e o p le , p la c e s , things, e tc .) a n d th e,r r e l a t i o n s ^ w ith o th er term s o r co m b m atio n s o f term s— m ak in g it e a sy to u se, m aintain, an d a lso b m o re accu ra te , s o y o u c a n rely o n th e insig hts as a ctio n ab le .

RELATE N o w that y o u h av e fo u n d th e insights a n d c a n a n a ly z e th e unstructured data, the real v a lu e co m e s w h e n y o u c a n c o n n e c t th o se insights to y o u r stru ctured data, y ou r cu sto m e rs (w h ich cu sto m e r s e g m e n t is co m p lain in g a b o u t y o u r p ro d u ct m o s t ) ; y o u r p ro d u cts (w h ic h p ro d u ct is h av in g th e issue?); y o u r parts (is th e re a p ro b le m w ith a s p e c ific p art m an u factu red b y a s p e cific partner?); y o u r lo ca tio n s (is th e cu sto m e r w h o is tw eeting a b o u t w an tin g a s an d w ich n e a r y ou r n e are st restaurant?); an d s o o n . N o w you ca n ask q u e stio n s o f y o u r data a n d g e t d e e p , a c tio n a b le insights.

ACT H e re is w h e re it g ets e x citin g , a n d y ou r b u s in e s s strategy an d ra le s are critical. W h at d o y o u d o w ith th e n e w cu sto m e r insig ht y o u 'v e ob tained ? H o w d o y o u lev erag e th e p ro b le m re so lu tio n co n te n t c re a te d b y a cu sto m e r th at y o u ju st identified . H o w do you c o n n e c t w ith a cu sto m e r w h o is u n co v e rin g issu es th a t are im p ortan t to y o u r b u si­ n e ss o r w h o is ask in g fo r help? H o w d o y o u ro u te th e insights to th e right p e o p le . And h o w d o y o u e n g a g e w ith cu stom ers, p artn ers, an d i n f l u e n c e s o n c e y o u u n d e rstan d w h at th ey a re saying? Y o u u n d erstan d it; n o w y o u ’v e g o t to act.

SECTION 8 . 7 REVIEW QUESTIONS

1 . W h a t is a m aturity m odel? 2. List a n d co m m e n t o n th e s ix stag es o f T D W I’s B I m aturity fram ew ork . 3 . W h a t are th e six d im en sio n s u s e d in H am el's W e b an alytics m aturity m odel? 4 . D e s c r ib e A ttensity's fram ew o rk fo r V O C strategy. List an d d e s c rib e th e fo u r .stages.

8.8 S O C IA L A N A L Y T IC S A N D S O C IA L N ET W O R K A N A L Y S IS Social an aly tics m ay m e a n d ifferent things to d ifferen t p e o p l e b a s e d o n th eir and field o f study. F o r in sta n ce, th e d iction ary d efin itio n o f s o c ia l analytics refers a p h ilo so p h ica l p e rsp e ctiv e d e v e lo p e d b y th e D an ish h istorian a n d p h ilo s o p h e r Lars- H enrik Schm id t in th e 19 8 0 s. T h e th e o re tica l o b je c t o f th e p e rsp e ctiv e is socims^ a km o f “co m m o n n e s s " th at is n e ith er a u n iv ersal a c c o u n t n o r a com m u n ality s h a r e d b y every m e m b er o f a b o d y (Sch m id t, 1 9 9 6 ). T h u s, so cial analytics differs fro m traditional p h ilo so phy as w e ll as s o cio lo g y . It m ight b e v ie w e d as a p e rsp e ctiv e th at attem pts to articulate th e c o n te n tio n s b e tw e e n p h ilo so p h y a n d so cio lo g y .

O u r d efin itio n o f so cia l analytics is so m e w h a t d ifferen t; a s o p p o s e d to fo cu sin g o n th e “s o cia l" p art (a s is th e c s a e in its p h ilo so p h ica l d efin itio n ), w e a re m o re in terested m the “an aly tics” part o f th e term . G artn er d efin e d so cia l an aly tics as ‘ m o n ito rin g , analyzing, m e asu rin g a n d interpretin g digital in teractio n s a n d relatio n sh ip s o f p e o p le , to p ics, ideas an d c o n te n t." S o cia l analytics in clu d e m ining th e te x tu a l co n te n t c re a te d m so cia l m ed ia ( e g . , s en tim en t analysis, natural lan g u ag e p ro ce s s in g ) an d an aly zin g so cia lly esm b lish e d n etw o rks (e .g ., in flu e n cer id en tificatio n , p rofiling, p re d ictio n ) fo r th e p u rp o se o f gam ing insight a b o u t existin g an d p o te n tia l cu sto m ers' c u n e n t an d fu tu re b e h a v io rs , a n d a b o u t th e lik e s an d d islik es to w ard a firm ’s p ro d u cts an d serv ices. B a s e d o n this d efin ition and

4 0 4 Part III • Predictive Analytics

th e cu rre n t p ra ctice s, s o c ia l an aly tics c a n b e cla ssifie d in to tw o d ifferen t, b u t n o t n e c e s ­ sarily m utually e x clu siv e , b ra n ch e s: so cia l n e tw o rk an alysis a n d so cia l m ed ia analytics.

So cial N etw ork A n a ly sis A s o c i a l n e t w o r k is a s o c ia l stru ctu re c o m p o s e d o f in d iv id u als/ p eo p le (o r g ro u p s o f ind ivid u als o r o r g a n iz a tio n s ) lin k e d to o n e a n o t h e r w ith s o m e ty p e o f co n n e ctio n s/ re la tio n sh ip s . T h e s o c ia l n e tw o rk p e rs p e c tiv e p ro v id e s a h o listic a p p r o a c h to analyz­ in g stru ctu re an d d y n a m ics o f s o c ia l e n titie s. T h e stu d y o f th e s e stru ctu res u s e s s o cial n e tw o rk a n aly sis to id e n tify lo c a l an d g lo b a l p a tte rn s, lo c a te in flu e n tial e n titie s, and e x a m in e n e tw o rk d y n am ics. S o cia l n e tw o rk s a n d th e a n aly sis o f th e m is e ss e n tia lly an in te rd iscip lin a ry fie ld th a t e m e rg e d fro m s o c ia l p s y c h o lo g y , s o c io lo g y , statistics, an d g ra p h th e o ry . D e v e lo p m e n t a n d fo rm a liz a tio n o f th e m a th em a tica l e x te n t o f s o c ia l n e t­ w o rk a n a ly sis d a te s b a c k to th e 1 9 5 0 s ; th e d e v e lo p m e n t o f fo u n d a tio n a l th e o r ie s and m e th o d s o f s o c ia l n e tw o rk s d a te s b a c k to th e 1 9 8 0 s (S c o tt an d D avis, 2 0 0 3 )- S o cia l n e tw o rk a n aly sis is n o w o n e o f th e m a jo r p arad ig m s in b u s in e s s a n aly tics, c o n s u m e r in te llig e n c e , an d c o n te m p o ra ry s o c io lo g y , a n d is a ls o e m p lo y e d in a n u m b e r o f o th e r s o c ia l a n d fo rm al s c ie n c e s .

A s o cia l n e tw o rk is a th e o re tica l co n stru ct u sefu l in th e so cia l s c ie n c e s to study relatio n sh ip s b e tw e e n individuals, g ro u p s, organ izatio n s, o r e v e n e n tire s o c ie tie s (so cia l u n its). T h e term is u s e d to d e s c rib e a so cia l stru cture d eterm in e d b y s u ch interactions. T h e ties throu gh w h ic h an y g iv en so cia l u n it c o n n e c ts re p re se n t th e c o n v e rg e n c e o f the vario u s s o cia l co n ta cts o f th at unit. In g e n era l, s o c ia l n etw o rk s a re self-organ izin g , em e r­ g e n t, a n d co m p le x , s u ch th at a g lo b a lly c o h e r e n t p attern a p p e a rs fro m the lo ca l in terac­ tio n o f th e e le m e n ts (individuals an d gro u p s o f ind ivid u als) th at m a k e up th e system .

F ollow ing are a fe w typical social netw ork ty p es that are relevant to b u sin ess activities.

COMMUNICATION NETWORKS C o m m u n icatio n stu d ies are o fte n co n sid e re d a part o f b o th th e so cial s c ie n c e s an d th e hu m an ities, d raw in g h eavily o n field s s u ch as so cio lo g y , p sy ch o lo g y , an th ro p o lo g y , in form ation s c ie n c e , b io lo g y , p o litical s c ie n c e , a n d e c o n o m ­ ics. M any co m m u n icatio n s c o n c e p ts d e s c rib e th e tran sfer o f in form ation fro m o n e so u rce to an o th er, an d thu s c a n b e re p re s e n te d as a s o c ia l n etw o rk . T e le co m m u n ica tio n co m p a ­ n ies are tap p in g in to this rich in form ation s o u rc e to o p tim ize th e ir b u sin e ss p ra ctice s and to im p ro v e cu sto m e r relatio nsh ip s.

COMMUNITY NETWORKS T rad itionally, co m m u n ity re fe rre d to a s p e c ific g e o g ra p h ic lo ca tio n , a n d stud ies o f com m u n ity ties h ad to d o w ith w h o talk ed , a sso cia ted , traded, an d atten d ed s o cia l activities w ith w h o m . T o d a y , h o w e v e r, th e re are e x te n d e d “o n lin e ” co m m u n ities d e v e lo p e d th ro u g h s o cia l n e tw o rk in g to o ls and te le co m m u n ica tio n s d ev ices. S u ch to o ls an d d e v ice s co n tin u o u sly g e n e ra te la rg e am o u n ts o f d ata, w h ich c a n b e u sed b y c o m p a n ie s to d isco v e r in v alu ab le, a ctio n a b le in form ation.

CRIMINAL NETWORKS In crim inology and u rb an sociolog y, m u ch attention has b e e n paid to the social netw orks am o n g crim inal actors. F or exam p le, studying gan g m urders and o ther illegal activities as a series o f e x ch a n g e s b e tw e e n gang s ca n lead to b etter understanding and p revention o f su ch crim inal activities. N ow that w e live in a highly co n n ecte d w orld (thanks to th e Internet), m any o f th e crim inal netw orks’ form ations an d their activities are b ein g watched/pursued b y security ag e n cie s using state-of-the-art Internet tools and tactics. Even th o u g h th e Internet h a s changed th e lan d scap e fo r criminal netw orks an d law en forcem en t agencies, th e traditional social and p h ilo sop h ical th eories still apply to a large extent.

INNOVATION NETWORKS B u sin ess studies o n diffusion o f ideas and innovations in a n et­ w o rk environm ent fo cu s o n th e spread and u se o f ideas am o n g the m em b ers o f the social

Chapter 8 • W eb Analytics, W e b Mining, and Social Analytics 4 0 5

netw ork T h e id ea is t o understand w h y som e netw orks are m o re innovative, an d w h y so m e com m unities a re early ad opters o f ideas and innovations (i.e., exam ining th e im p act o f social n etw ork structure o n influencing th e spread o f a n innovation an d innovative b eh av io i).

So cial N e tw o rk A n a ly sis M etrics Social n e tw o rk analysis (SNA) is th e system atic exam ination o f social netw orks. Social net­ w ork analysis view s social relationships in term s o f netw ork theory, consisting o f nod es (representing individuals o r organizations w ithin the netw o rk) an d ties/connections (w h ich represent relationships b etw ee n th e individuals o r organizations, su ch as friendship, kinship, organizational position, e tc.). T h e se netw orks are o ften rep resented using social n etw o rk dia­ grams, w h ere n od es are rep resented as points and ties are rep resented as lines. Application Case 8 .5 gets in to th e details o f h o w SNA c a n b e used to h elp telecom m u n ication com panies.

O v e r th e years, vario u s m etrics (o r m e a su re m e n ts) h av e b e e n d e v e lo p e d to an alyze so cia l n e tw o rk stru ctu res fro m d iffe ren t p e rsp e ctiv es. T h e s e m etrics are o ften gro u p e d into th re e c a te g o rie s: c o n n e c tio n s , d istributions, a n d seg m en tatio n .

Application Case 8.5 Social N e tw o rk A nalysis Helps Telecom m unication B e c a u s e o f th e w id esp rea d u s e o f fre e In te rn e t to o ls an d te c h n iq u e s (V oIP , v id e o c o n fe re n c in g tools su ch as S k y p e , fre e p h o n e calls w ith in th e U nited States b y G o o g le V o ice , e tc .), th e te le co m m u n ica ­ tio n indu stry is g o in g th ro u g h a to u g h tim e. In ord er to stay v ia b le a n d co m p etitiv e , th ey n e e d to m a k e th e right d e cis io n s a n d u tilize th e ir lim ited re so u rce s optim ally. O n e o f th e k e y s u c c e s s facto rs fo r te le ­ c o m c o m p a n ie s is to m axim ize th e ir profitability b y listen in g a n d u n d erstan d in g th e n e e d s and w an ts o f th e cu sto m e rs, o ffe rin g co m m u n ica tio n p lan s, p rice s, a n d featu res that th ey w a n t a t th e p rice s that th ey are w illin g to pay.

T h e s e m ark e t p ressu res fo rc e te le co m m u ­ n icatio n c o m p a n ie s to b e m o re innovative. As w e all k n o w , “n e c e s s ity is th e m o th e r o f in v e n tio n .” T h e re fo re , m an y o f th e m o st p ro m isin g u se ca s e s fo r s o cia l n e tw o rk analysis (SNA) a re co m in g fro m the te le co m m u n ica tio n co m p a n ie s . U sing d etailed call re co rd s th a t are alread y in th e ir d a ta b a se s, th e y are trying to id en tify s o cia l n etw o rk s an d in flu en cers. In o rd e r to id entify th e so cia l n etw o rk s, th e y are ask in g q u e s tio n s lik e “W h o co n ta cts w h om ?” “H ow o ften ?” “H o w long?” “B o th directions?” “O n Net, o f f Net?” T h e y are a lso trying to an sw e r q u e stio n s that le a d to id en tificatio n o f in flu e n cers, su ch as “W h o in flu e n ce d w h o m h o w m u ch o n p u rchases?’ “W h o in flu e n c e s w h o m h o w m u ch o n churn?” and

Firms “W h o w ill a cq u ire others?” SNA m etrics lik e d eg ree (h o w m an y p e o p le a re d irectly in a p e rs o n ’s so cial n e tw o rk ), d en sity (h o w d e n s e is th e callin g pattern w ith in th e ca llin g c irc le ), b e tw e e n n e s s (h o w e ss e n ­ tial y o u are to facilitate co m m u n ica tio n w ith in y ou r callin g c irc le ), an d cen trality (h o w “im portant” you a re in th e s o c ia l n e tw o rk ) a re o fte n u s e d an sw e r th e s e q u e stio n s.

H e re are s o m e o f th e b e n e fits that c a n b e o b ta in e d fro m SNA:

• M anage cu sto m e r ch u m • R eactiv e (re d u ce co llateral ch u rn )— Id entify

su b scrib e rs w h o s e loyalty is th re a te n e d b y ch u rn aro u n d them .

• P rev en tiv e (re d u c e in flu en tial ch u rn ) Id en tify su b scrib e rs w h o , sh ou ld th ey ch u rn , w o u ld tak e a fe w frien d s w ith them .

• Im p ro v e cro ss-se ll a n d te c h n o lo g y transfer - R eactive (lev erage collateral ad option)—

Id entify subscribers w h o se affinity fo r prod­ ucts is increased due to ad op tion around th em an d stim ulate them .

- P ro a ctiv e (id en tify in flu e n cers fo r this a d o p ­ tio n )— Id entify su b scrib e rs w h o , sh ou ld th e y a d o p t, w o u ld p u sh a fe w friend s to d o th e sam e .

{Continued)

4 0 6 Part III • Predictive Analytics

Application Case 8.5 (Continued) • M an ag e viral cam p aig n s— U n derstand w h at

le a d s to h ig h -sca le sp re a d o f m e ssa g es a b o u t p ro d u cts an d serv ices, an d u s e this in form a­ tio n to y o u r b en e fit.

• Im p ro v e acq u isitio n — Id en tify w h o a re m ost lik e ly to re c o m m e n d a (o ff-N e t) frien d to b e c o m e a n e w s u b s c rib e r o f th e o p era to r. T h e re co m m e n d a tio n itself, as w e ll as th e su b scrip ­ tio n , is in cen tiv ized fo r b o th th e s u b scrib e r a n d th e re co m m en d in g p erson.

• Id en tify h o u seh o ld s, co m m u n ities, and c lo s e - g ro u p s to b e tte r m a n a g e y o u r relatio n sh ip s w ith them .

• Id entify cu sto m e r life-stages— Id entifying so cial n e tw o rk ch a n g e s an d fro m th e re identifying life -stag e ch a n g e s su ch as m oving, ch an g in g a jo b , g o in g to a university, starting a relatio n ­ s h ip s, g etting m arried, etc.

• Id en tify in g p re-ch u rn ers— D etectin g p o ten tial c h u rn e rs d uring th e p ro ce s s o f leav in g and m o tiv atin g th e m to stay w ith y ou .

• G a in co m p e tito r insights— T ra c k d ynam ic c h a n g e s in s o cia l n e tw o rk s b a s e d o n co m p e ti­ to r ’s m arketin g activities

• O th e rs in d u cin g id e n tify in g ro ta tio n a l ch u r­ n ers (s w itch in g b e tw e e n o p e ra to rs )— F a cilitatin g re - to p o stm ig ratio n , a n d track in g cu sto m e r’s n e tw o rk s d yn am ics o v e r his/her life c y c le .

A ctual c a s e s in d icate th at p ro p e r im p lem e n tatio n o f SNA c a n sig n ifican tly lo w e r c h u m , im p ro v e cro ss- sell, b o o s t n e w cu sto m e r a cq u isitio n , o p tim ize p ric­ ing and , h e n c e , m ax im ize profit, and im p ro ve o v er­ all co m p etitiv e n e ss.

Q u e s t i o n s f o r D i s c u s s i o n

1. H o w c a n s o cial n e tw o rk an alysis b e u s e d in th e te le co m m u n ica tio n s industry?

2. W h at d o you th in k are th e k e y ch a lle n g e s, p o ten tial solu tio n , an d p ro b a b le results in apply­ ing SNA in te le co m m u n ica tio n s firms?

Source: Compiled from “More Things W e Love About SNA: Return o f the Magnificent 10," February 2013, presentation by Judy Bayer and Fawad Qureshi, Teradata.

Connections H om o p h ily : T h e e x te n t to w h ic h a cto rs fo rm ties w ith sim ilar v ersu s d issim i­ lar o th ers. Sim ilarity ca n b e d efin ed b y g e n d er, ra ce , a g e , o ccu p a tio n , ed u catio n al a ch iev em en t, status, v alu es, o r an y o th e r salien t ch aracteristic.

M u ltip lexity : T h e n u m b e r o f co n ten t-fo rm s co n ta in e d in a tie. F o r e x a m p le , tw o p e o p le w h o are friend s a n d a lso w o rk to g e th e r w o u ld h av e a m u ltip lexity o f 2. M u ltiplexity has b e e n a sso cia ted w ith re latio n sh ip strength.

M u tu a lity /recip ro city : T h e e x te n t to w h ich tw o acto rs re c ip ro c a te e a c h o th e r’s frien d sh ip o r o th e r in teractio n. N etw ork c lo s u re : A m e asu re o f th e c o m p le te n e ss o f relatio n al triads. An indi­ v id u al’s assu m p tio n o f n e tw o rk clo su re (i.e ., th a t th e ir friend s are a lso frie n d s) is ca lle d transitivity. Transitivity is a n o u tc o m e o f th e individual o r situ ation al trait o f n e e d fo r co g n itiv e closu re. P ro p in q u ity : T h e te n d e n c y fo r a cto rs to h av e m o re tie s w ith g e o g rap h ically c lo s e oth ers.

D istrib u tio n s B r i d g e : A n individual w h o s e w e a k tie s fill a structural h o le , p ro vid ing th e o n ly lin k b e tw e e n tw o individuals o r clu sters. It a lso in clu d e s th e sh o rtest ro u te w h e n a lo n g e r o n e is u n fe a sib le d u e to a h ig h risk o f m e ssa g e d istortion o r d eliv ery failure.

Chapter 8 • W eb Analytics, W eb Mining, and Social Analytics 4 0 7

C en tra lity : R efers to a group o f m etrics that aim to quantify th e im p o ita n ce o r in flu e n ce (in a variety o f s e n se s ) o f a particular n o d e (o r g ro u p ) w ithin a netw ork. E x a m p le s o f co m m o n m ethods o f m easuring centrality include b e tw e e n n e ss central­ ity, clo s e n e s s centrality, e ig e n v e cto r centrality, alpha centrality, and d eg ree centrality.

D en sity : T h e p ro p o rtio n o f d irect ties in a n e tw o rk relativ e to th e to tal n u m b e r p o ssib le .

D is ta n c e : T h e m inim um n u m b e r o f ties req u ired to c o n n e c t tw o particu lar acto rs. S t r u c t u r a l holes: T h e a b s e n c e o f ties b e tw e e n tw o parts o f a n e tw o rk . Finding an d e x p lo itin g a structural h o le c a n give a n e n tre p re n e u r a co m p etitiv e ad vantage. T h is c o n c e p t w a s d e v e lo p e d b y s o c io lo g ist R o n ald B u rt a n d is so m e tim e s referred to a s a n altern ate c o n c e p tio n o f s o c ia l capital.

Tie s tre n g th : D e fin e d b y th e lin e ar co m b in a tio n o f tim e, e m o tio n a l intensity, intim acy , and re cip ro city (i.e ., m utuality). Stro ng ties a re a s s o c ia te d w ith hom op h ily , p ro p in q u ity , a n d transitivity, w h ile w e a k ties are a sso cia ted w ith brid ges.

Se g m e n ta tio n

C liq u es a n d so cia l circles: G ro u p s are id en tified as cliq u es i f every individual is d irectly tied to e v e ry o th e r individual o r s o c ia l circ les if th e re is less s trin g e n cy o f d ire ct c o n ta ct, w h ic h is im p re cise , o r as stru cturally c o h e s iv e b lo c k s if p re cisio n is w an ted .

C lu s te rin g c o e ffic ien t: A m e a su re o f th e lik e lih o o d that tw o m e m b ers o f a n o d e are a sso cia te s. A h ig h e r clu sterin g c o e ffic ie n t in d icate s a g re a te r cliqu ishn ess.

C o h esio n : T h e d e g re e to w h ic h a cto rs a re c o n n e c te d d irectly to e a c h o th e r b y co h e s iv e b o n d s. Structural c o h e s io n re fe rs to th e m inim um n u m b e r o f m e m b ers w h o , i f re m o v ed fro m a g ro u p , w o u ld d is c o n n e c t th e group.

SECTION 8 . 8 REVIEW QUESTIONS

1 . W hat is m e an t b y so cia l analytics? W h y is it a n im p ortan t b u s in e s s topic?

2. W h at is a so cia l netw ork? W h at is s o c ia l n e tw o rk analysis? 3 . List a n d b riefly d e s c rib e th e m o st co m m o n s o c ia l n e tw o rk types.

4 . List an d b riefly d e s c rib e th e so cia l n e tw o rk analysis m etrics.

8.9 SO CIA L MEDIA DEFINITIONS AND CONCEPTS Social m ed ia refers to the en ablin g te ch n o lo g ie s o f social interactions am o n g p e o p le in w h ich th ey create, share, and e x ch a n g e inform ation, ideas, an d o p inions in virtual com m uni­ ties and netw orks. It is a group o f Internet-based softw are applications, that build o n th e ide­ ological and tech n o lo g ical found ations o f W e b 2.0, and that allow th e creation and e xch an g e o f user-generated co n ten t (K ap lan and H aenlein, 2 0 1 0 ). Social m edia d ep en d s o n m o bile and o th er W e b -b a sed tech n olog ies to cre ate highly interactive platform s fo r individuals and com m unities to share, co -cre a te, discuss, and m odify user-generated, con ten t. It introduces substantial ch an g es to com m unication b etw ee n organizations, com m unities, an d individuals.

S in c e th e ir e m e rg e n c e in th e e a rly 19 9 0 s, W e b -b a s e d s o cia l m e d ia te ch n o lo g ie s have s e e n a sig n ifican t im p ro v e m en t in b o th quality an d quantity. T h e s e te c h n o lo ­ gies tak e o n m a n y d ifferen t fo rm s, in clu d in g o n lin e m ag azin es, In te rn e t fo ru m s, W e b log s, s o c ia l b lo g s , m icro b lo g g in g , w ikis, so cia l n etw o rk s, p o d casts, p ictu res, v id eo , and

4 0 8 Part III • Predictive Analytics

produ ct/service evaluations/ratings. B y ap p ly in g a s e t o f th e o rie s in th e field o f m ed ia re se a rch (s o c ia l p re s e n c e , m e d ia rich n e ss ) a n d s o cia l p ro c e s s e s (s e lf-p re sen ta tio n , self­ d isclo su re), K ap lan a n d H a e n le in ( 2 0 1 0 ) c re a te d a classificatio n s c h e m e w ith s ix d ifferent ty p e s o f s o cia l m ed ia: co lla b o ra tiv e p ro je cts ( e .g ., W ik ip e d ia ), b lo g s an d m icro b lo g s (e .g ., T w itter), c o n te n t co m m u n ities (e .g ., Y o u T u b e ), s o cia l n e tw o rk in g sites (e .g ., F a c e b o o k ), virtual g a m e w o rld s (e .g ., W o rld o f W arcraft), a n d virtual s o cia l w o rld s (e .g ., S e c o n d Life).

W e b -b a s e d so cial m ed ia are d ifferent fro m traditional/industrial m ed ia, s u ch as n e w sp ap e rs, telev isio n , a n d film , a s th e y are co m p arativ ely in ex p e n siv e an d a cce ss ib le to e n a b le a n y o n e (e v e n private individuals) to p u b lish o r access/ co n su m e inform ation. Industrial m ed ia gen erally requ ire sig n ifican t re so u rce s to pu blish inform ation, as in m ost ca s e s th e articles (o r b o o k s ) g o throu gh m any revisio n s b e fo re b e in g p u b lish e d (a s w as th e c a s e in th e p u b lica tio n o f this very b o o k ). H e re are s o m e o f th e m o st prevailing ch ar­ acteristics that h e lp differentiate b e tw e e n s o cial an d industrial m ed ia (M organ e t al., 2 0 1 0 ):

Q uality: In industrial p u blish in g— m e d ia te d b y a p u b lish e r th e ty p ical range o f qu ality is su b stantially n arro w e r th a n in n ich e, u n m ed iated m arkets. T h e m ain c h a lle n g e p o se d b y c o n te n t in so cia l m e d ia sites is th e fa ct th a t th e d istrib u tion o f quality h a s h ig h v arian ce: fro m very h ig h -q u ality item s to lo w -qu ality , so m e tim e s

ab u siv e, co n ten t. R e a c h : B o th industrial a n d so cia l m e d ia te c h n o lo g ie s pro v id e s c a le a n d are c a p a ­ b le o f re a ch in g a g lo b a l au d ie n ce . Ind u strial m ed ia, h o w e v e r, typically u s e a c e n ­ tralized fram ew o rk fo r o rgan ization , p ro d u ctio n , and d issem in atio n , w h e re a s s o cial m ed ia are b y th e ir v e ry n atu re m o re d e ce n tralized , less h ierarch ical, an d distin­ g u ish e d b y m u ltip le p o in ts o f p ro d u ctio n a n d utility.

F re q u e n c y : C o m p ared to industrial m ed ia, u p d atin g and re p o stin g o n s o cia l m e d ia platform s is e a sie r, faster, an d ch e a p e r, a n d th e re fo re p ra ctice d m o re fre­

q u en tly , resultin g in fre s h e r co n ten t. A ccessibility: T h e m e a n s o f p ro d u ctio n fo r industrial m ed ia a re typ ically g o v ­ e rn m e n t and/or co rp o ra te (p riv ately o w n e d ), an d a re costly, w h e re a s so cia l m ed ia to o ls are g e n erally a v ailab le to th e p u b lic at little o r n o cost.

Usability: Industrial m ed ia p ro d u ctio n typically req u ires s p e cia liz e d skills an d training. C o n versely , m o st s o c ia l m ed ia p ro d u ctio n re q u ires o n ly m o d e st rein ter­ pretatio n o f e x istin g skills; in th e o ry , a n y o n e w ith a c c e s s c a n o p e ra te th e m e a n s o f

so cia l m ed ia p ro d u ction . Im m ed ia cy : T h e tim e la g b e tw e e n c o m m u n ica tio n s p ro d u ce d b y industrial m ed ia c a n b e lo n g (w e e k s , m o n th s, o r e v e n y e a rs ) c o m p a re d to s o cia l m e d ia (w h ic h c a n b e c a p a b le o f virtually in stan tan e o u s re s p o n s e s ).

Updatability: Ind ustrial m ed ia, o n c e cre ate d , ca n n o t b e a ltered (o n c e a m aga­ z in e article is p rin ted an d d istributed , c h a n g e s c a n n o t b e m ad e to that sa m e article), w h e re a s s o cia l m e d ia c a n b e altered a lm o st in stan tan eo u sly b y co m m e n ts o r editing.

How Do People Use So cial M edia? N ot o n ly are th e n u m bers o n s o cia l n e tw o rk in g sites g ro w ing, b u t s o is the d e g re e to w h ich th ey are e n g a g e d w ith th e ch an n e l. B ro g a n an d B a s to n e (2 0 1 1 ) p re sen ted re sea rch results that stratify u sers acco rd in g to h o w a ctiv ely th e y u se so cial m ed ia an d tra ck e d ev o lu tio n o f th e se u se r seg m en ts o v e r tim e. T h e y listed s ix d ifferen t e n g a g em en t lev els (Figure 8 .1 1 ).

A cco rd in g to th e re se a rch results, th e o n lin e u s e r com m u n ity h a s b e e n steadily m igrating u p w ard s o n this e n g a g e m e n t h ierarch y . T h e m o st n o ta b le c h a n g e is am o n g In activ e s. Forty -fo u r p e rce n t o f th e o n lin e p o p u la tio n fe ll in to this cate g o ry . T w o y ears

Chapter 8 • W eb Analytics, W e b Mining, and Social Analytics 4 0 9

C r e a to r s

C ritic s

Jo in e rs

C o lle c to rs

S p e c ta to rs

In a c tiv e s

T i m e

F IG U R E 8 .1 1 E v o lu tio n o f S o cial M edia U ser E n g a g e m e n t.

later, m o re th a n h a lf o f th o s e In activ e s h a d ju m p e d into s o c ia l m e d ia in s o m e fo rm o r another. “N o w ro u gh ly 8 2 p e rc e n t o f th e ad ult p o p u la tio n o n lin e is in o n e o f th e u p p er c a te g o rie s,” said B a s to n e . “S o cia l m e d ia has truly re a ch e d a state o f m ass a d o p tio n .”

A p p licatio n C ase 8 .6 sh o w s th e p o sitiv e im p act o f so cial m e d ia a t L ollap aloo za.

Application Case 8.6 M easuring th e Im pact o f Social M ed ia a t Lollapalooza

w a n te d to k n o w o n e s im p le th in g : “D id it w o rk ?”C3 P resents creates, b o o k s , m arkets, and p ro d u ces live e xp e rien ces, con certs, events, an d just ab ou t anything that m akes p e o p le stand up an d ch eer. A m ong oth­ ers, th ey p ro d u ce th e Austin City Limits M usic Festival, L ollapalooza, as w ell as m o re th an 8 0 0 sh ow s nation­ w ide. T h e y h o p e to s e e you up in front som etim e.

A n e a rly a d o p te r o f so cia l m ed ia as a w ay to drive e v e n t a tte n d a n ce , L ollap aloo za o rg an ize r C3 P resen ts n e e d e d to k n o w th e im p act o f its s o cial m ed ia effo rts. T h e y ca m e to Cardinal P ath fo r a so cia l m ed ia m e a su re m e n t strategy an d e n d e d up w ith s o m e startling insights.

T h e C h a lle n g e

W hen th e Lollapalooza m usic festival d ecid ed to incor­ porate social m edia into their online marketing strat­ egy, they d id it w ith a bang. Using Facebo ok , MySpace, Twitter, and m ore, the Lollapalooza W e b site w as a first m over in allow ing its users to engage a nd share through social channels that w ere integrated into the site itself.

A fter in v e stin g th e tim e a n d r e s o u r c e s in b u ild ­ ing o u t t h e s e in te g ra tio n s an d th e ir fu n ctio n ality , C3

T o a n s w e r this, C 3 P re se n ts n e e d e d a m e a s u re m e n t strate g y th a t w o u ld p ro v id e a w e a lth o f in fo rm atio n a b o u t th e ir s o c ia l m e d ia im p le m e n ta tio n , s u c h as:

• W h ich fan s are u sin g s o cia l m e d ia an d sharing content?

• W h at s o c ia l m ed ia is b e in g u s e d th e m o st, and how ?

• A re v isitors th at in teract w ith so cial m ed ia m o re lik e ly to b u y a ticket?

• Is s o cial m e d ia driving m o re traffic to th e site? Is that traffic b u y in g tickets?

T h e S o lu tio n

Cardinal Path w a s a sk ed to architect an d im p lem en t a solution b a s e d o n a n existin g G o o g le Analytics im ple­ m entation that w o u ld to an sw er th e se questions.

A com bination o f custom ized event tracking, cam ­ paign tagging, cu stom variables, and a com p lex imple­ m entation and configuration w as dep loyed to include th e tracking o f e a c h social m edia outlet o n the site.

( Continued)

4 1 0 Part III • Predictive Analytics

Application Case 8.6 (Continued)

T h e R e s u lts

A s a re su lt o f th is m e a su re m e n t so lu tio n , it w a s e a sy to su rfa ce s o m e im p ressiv e insights that h e lp e d C3 q u an tify th e retu rn o n th e ir so cia l m e d ia investm ent:

• U sers o f th e s o c ia l m e d ia ap p licatio n s o n L o lla p a lo o z a .co m sp e n t tw ice as m u ch as n o n -u se rs.

• O v e r 6 6 p e rce n t o f the traffic referred from F a c e b o o k , M ySpace, and Tw itter w as a result o f sharing ap p licatio ns and L ollapalooza’s m essag­ ing to its fans o n th o se platforms.

• F an e n g a g em en t m etrics s u ch as tim e o n site, b o u n c e rate, p a g e view s p e r visit, an d inter­ actio n g o a ls im proved significantly acro ss the b o ard as a result o f so cial m ed ia applications.

Q u e s t i o n s f o r D i s c u s s i o n

1. H o w did C3 P rese n ts u se so cia l m ed ia analytics to im p ro ve its business?

2. W h at w e r e th e ch a lle n g e s, th e p ro p o s e d s o lu ­ tion , an d th e o b ta in e d results?

Source: www.cardinalpath.com/case-study/social-media- measurement (accessed March 2013).

SECTION 8 . 9 REVIEW QUESTIONS

1 . W h a t is so cia l m edia? H o w d o e s it relate to W e b 2.0? 2 . W h at are th e d iffe ren ce s and co m m o n alitie s b e tw e e n W e b -b a s e d so cia l m e d ia an d

traditional/industrial m edia? 3 . H o w d o p e o p le u s e s o cia l m edia? W h a t are th e ev o lu tio n ary lev els o f e n gagem en t?

8.10 SO C IA L M E D IA A N A LY T IC S S o cia l m e d ia an aly tics refers to th e sy stem atic a n d scie n tific w ays to c o n s u m e th e vast am o u n t o f c o n te n t cre a te d b y W e b -b a s e d s o cial m e d ia ou tlets, to o ls, a n d te c h n iq u e s fo r th e b ette rm e n t o f a n o rg an izatio n ’s co m p etitiv e n e ss. S o cia l m ed ia an alytics is rapidly b e c o m in g a n e w fo rce in org an izatio n s a ro u n d th e w o rld , allo w in g th em to re a c h o u t to an d u n d erstan d co n su m e rs as n e v e r b e fo re . In m a n y co m p a n ie s , it is b e c o m in g th e to o l fo r integ rated m arketin g an d co m m u n ica tio n s strateg ies.

T h e exp o n en tial grow th o f social m ed ia outlets, from b logs, F a ce b o o k , and Tw itter to Linkedln and Y o u T u b e, and analytics tools that tap into th ese rich data sou rces o ffe r organi­ zations th e ch a n ce to jo in a conversation w ith m illions o f cu stom ers around the g lo b e every day. This aptitude is w h y nearly tw o-thirds o f th e 2 ,1 0 0 com p an ies w h o participated in a recen t survey b y Harvard Business R eview (H B R ) Analytic Services said they are eith er cur­ rently using social m edia ch an n els o r h av e social m ed ia plans in th e w orks (H B R , 2010). But m any still say social m ed ia is a n experim ent, as th ey try to understand h o w to b e s t u se the different channels, g au ge their effectiveness, and integrate so cial m edia into their strategy.

D e s p ite th e vast p o ten tial s o cia l m e d ia an aly tics b rings, m any co m p a n ie s s e e m fo c u s e d o n so cia l m e d ia activity prim arily a s a o n e -w a y p ro m o tio n a l c h a n n e l a n d have y e t to cap italize o n th e ab ility to n o t o n ly liste n to , b u t a lso an aly ze, c o n s u m e r c o n v e rsa ­ tio n s an d turn th e in form ation in to insig hts th at im p a ct th e b o tto m line. H ere are s o m e o f th e results fro m th e H B R A nalytic Se rv ices survey (H B R , 2 0 1 0 ):

• T h re e -q u a rte rs (7 5 % ) o f th e co m p a n ie s in th e survey said th ey d id n o t k n o w w h e re th e ir m o st v alu ab le cu sto m ers w e re talk in g a b o u t them .

• N early o n e-th ird (3 1 % ) d o n o t m e a su re e ffe c tiv e n e s s o f so cia l m ed ia. • Less th an o n e -q u a rte r (2 3 % ) are u sin g s o c ia l m ed ia an alytic to ols. • A fractio n (7 % ) o f p articip ating c o m p a n ie s are a b le to in teg rate so cial m ed ia into

th e ir m ark etin g activities.

Chapter 8 * W eb Analytics, W e b Mining, and Social Analytics 4 1 1

W h ile still s e a rch in g fo r b e s t p ra ctice an d m e asu re m e n ts, tw o-th ird s o f the c o m ­ p an ie s su rv ey ed are co n v in ce d th e ir u se o f so cia l m ed ia will gro w , and m any an ticip ate investing m o re in it n e x t y ear, e v e n as sp e n d in g in traditional m ed ia d e clin e s. S o w h at is it s p e cifica lly th at th e c o m p a n ie s are in te reste d in m easu rin g in so cia l m edia?

M easuring the So cial M edia Im pact F o r o r g a n iz a tio n s , sm all o r la rg e , th e re is v a lu a b le in sig h t h id d e n in all th e u s e r-g e n - e ra te d c o n t e n t o n s o c ia l m e d ia s ite s . B u t h o w d o y o u d ig it o u t o f d o z e n s o f re v ie w sites, th o u s a n d s o f b lo g s , m illio n s o f F a c e b o o k p o sts, a n d b illio n s o f tw eets? O n c e y o u d o th a t, h o w d o y o u m e a s u re th e im p a c t o f y o u r effo rts? T h e s e q u e s tio n s c a n b e a d d r e s s e d b y th e a n a ly tics e x te n s io n o f th e s o c ia l m e d ia te c h n o lo g ie s . O n c e y o u d e c id e o n y o u r g o a l fo r s o c ia l m e d ia (w h a t it is th a t y o u w a n t to a c c o m p lis h ), th e re is a m u ltitu d e o f to o ls to h e lp y o u g e t th e re . T h e s e a n a ly sis to o ls u s u a lly fall in to th re e b ro a d c a te g o r ie s :

• D e s c r ip tiv e a n a ly t ic s : U se s sim p le statistics to id entify activity ch aracteristics an d tren d s, s u ch as h o w m an y fo llo w ers y o u h a v e, h o w m an y re v iew s w e re g e n e r­ ate d o n F a c e b o o k , an d w h ich ch a n n e ls are b e in g u s e d m o st often .

• S o c ia l n e tw o r k a n a ly s is : F o llo w s th e lin k s b e tw e e n frien d s, fan s, an d fo llo w ers to id entify c o n n e c tio n s o f in flu e n ce as w ell as th e b ig g e st so u rce s o f in flu e n ce.

• A d v a n c e d a n a ly t ic s : In clu d es p red ictiv e an alytics a n d te x t an aly tics that e x a m ­ in e th e content in o n lin e co n v e rsatio n s to identify th e m e s, sen tim en ts, a n d c o n n e c ­ tio n s th a t w o u ld n o t b e re v e a le d b y casu al su rveillance.

S o p h isticate d to o ls a n d so lu tio n s to so cia l m e d ia analytics u s e all th ree ca te g o rie s o f analytics ( i.e ., d escrip tive, p red ictiv e, an d p re scrip tiv e ) in a s o m e w h a t p ro g ressiv e fashion.

Best P ractice s in S o cia l M edia A n a ly tics

As a n e m e rg in g to ol, s o cia l m ed ia analytics is p ra ctice d b y co m p a n ie s in a so m e w h at hap h azard fa s h io n . B e c a u s e th e re are n o t w e ll e sta b lish e d m e th o d o lo g ie s , ev ery b o d y is trying to c re a te th e ir o w n b y trial a n d error. W hat fo llo w s are s o m e o f th e field -te ste d best p ractices fo r s o cia l m ed ia an aly tics p ro p o s e d b y P a in e an d C h aves (2 0 1 2 ).

THINKOFMEASUREMENTASAGUIDANCE SYSTEM, NOT A RATING SYSTEM M easu rem en ts are o ften u s e d fo r p u n ish m en t o r rew ard s; they sh o u ld n ot b e . T h e y sh o u ld b e a b o u t fig­ uring o u t w h a t th e m o st e ffe ctiv e to o ls an d p ra ctice s are, w h at n e e d s to b e d isco n tin u ed b e c a u s e it d o e s n ’t w o rk , an d w h at n e e d s to b e d o n e m o re b e c a u s e it d o e s w o rk very w ell. A g o o d an alytics system sh o u ld tell y o u w h e re y o u n e e d to fo cu s . M aybe all that em phasis o n F a c e b o o k d o e s n ’t really m atter, b e c a u s e that is n o t w h e r e y o u r a u d ien ce is. M ayb e th e y a re all o n T w itter, o r v ice versa. A cco rd in g to P ain e a n d C h av es, ch a n n e l p re fe re n ce w o n ’t n e c e s s a rily b e intuitive, “W e ju st w o rk e d w ith a h o te l th a t h a d virtually no activity o n T w itte r fo r o n e b ra n d but lots o f T w itte r activity fo r o n e o f th e ir high er brand s.” W ith o u t a n a ccu ra te m e a su re m e n t to o l, y o u w o u ld n o t k n o w .

TRACK THE ELUSIVE SENTIMENT C u stom ers w a n t to ta k e w h a t th e y a r e h e a rin g and learn ing fro m o n lin e co n v e rsa tio n s an d a c t o n it. T h e k e y is to b e p re c is e in extractin g m d tag g in g th e ir in te n tio n s b y m easu rin g th e ir sen tim en ts. As w e h av e s e e n in C h ap ter 7, :e x t analytic to o ls c a n ca te g o riz e o n lin e c o n te n t, u n c o v e r lin k e d c o n c e p ts , an d re v e a l th e -cn tim en t in a co n v e rs a tio n as “p o sitiv e ,” “n e g a tiv e ,” o r “n eu tral,” b a s e d o n the w ord s p e o p le u se. Id eally , y o u w o u ld like to b e a b le to attribute s en tim en t to a s p e c ific p ro d u ct, >ervice, a n d b u sin e ss unit. T h e m o re p re c ise y o u c a n g e t in u n d erstan d in g th e to n e and

p e rc e p tio n that p e o p le e x p re s s , th e m o re a c tio n a b le th e in form ation b e c o m e s , b ecau se y o u are m itigating c o n c e rn s a b o u t m ix e d polarity. A m ixed -p o larity p h rase , s u ch as hotel in g reat lo ca tio n b u t b a th ro o m w a s sm e lly ” sh o u ld n o t b e tagg ed as “n e u tral” b e c a u se y o u h av e p o sitiv es a n d n eg ativ es offsettin g e a c h oth er. T o b e a ctio n a b le , th e s e typ es o f p h rase s a re to b e treated sep arately ; “b a th ro o m w as sm e lly ” is so m e th in g s o m e o n e can o w n and im p ro ve u p o n . O n e c a n classify an d ca te g o riz e th e s e sen tim en ts, lo o k at trends o v e r tim e, an d s e e sig n ifican t d iffe re n ce s in th e w a y p e o p le sp e a k e ith er p o sitiv ely or neg ativ ely a b o u t y ou . F u rth erm ore, y o u c a n c o m p a re s en tim en t a b o u t y o u r b ra n d to your

com p etito rs.

CONTINUOUSLY IMPROVE THE ACCURACY OF TEXT ANALYSIS An ind u stry -sp ecific text an aly tics p a c k a g e will alread y k n o w th e v o ca b u la ry o f y o u r b u sin ess. T h e sy stem will h a v e lingu istic rules b u ilt into it, b u t it learn s o v e r tim e an d g e ts b e tte r and b etter. Much as y o u w o u ld tu n e a statistical m o d e l a s y o u g e t m o re d ata, b e tte r p aram eters, o r nev, te c h n iq u e s to d eliver b e tte r results, y o u w o u ld d o th e sa m e thing w ith th e natu ral lan ­ g u a g e p ro ce s s in g that g o e s in to s en tim en t analysis. Y o u s e t u p ru les, ta x o n o m ie s , ca te g o ­ rization, a n d m e an in g o f w o rd s; w a tch w h at th e results lo o k lik e ; a n d th e n g o b a c k and

d o it again.

LOOK AT THE RIPPLE EFFECT It is o n e th in g to g e t a g re a t hit o n a hig h -p ro file site, but th a t’s o n ly th e start. T h e r e ’s a d iffe ren ce b e tw e e n a g reat hit th a t ju st sits th e re an d g o e s aw ay v e rsu s a great hit th at is tw e e te d , re tw e e te d , a n d p ic k e d up b y influential b log g ers. Analysis sh ou ld s h o w y o u w h ic h s o cia l m ed ia activ ities g o “v iral” a n d w h ich q u ick ly go

dorm ant— an d w hy.

LOOK BEYOND THE BRAND O n e o f th e b ig g e s t m istak es p e o p le m a k e is to b e c o n ­ c e rn e d o n ly a b o u t th e ir b ran d . T o su cce ssfu lly an aly ze a n d a ct o n so cial m ed ia, y o u n e e d to u n d erstan d n o t ju st w h a t is b e in g said a b o u t y o u r b ran d , b u t th e b ro a d e r c o n ­ v e rsa tio n a b o u t th e sp e ctru m o f issu es su rrou n d in g y o u r p ro d u ct o r serv ice, as w ell. C u stom ers d o n ’t u su ally c a re a b o u t a firm ’s m e s s a g e o r its b rand ; th ey c a re a b o u t th e m ­ selv e s. T h e re fo re , y o u sh o u ld p a y atten tio n to w h a t th e y are talk in g a b o u t, w h e re they

are talking, an d w h e re th e ir in terests are.

IDENTIFY YOUR MOST POWERFUL INFLUENCERS O rg an izatio n s stru ggle to identify w h o h a s the m o st p o w e r in sh ap in g p u b lic o p in io n . It tu rns ou t, y o u r m o st im portant influ­ e n c e s are n o t n e c e s s a rily th e o n e s w h o a d v o c a te sp e cifica lly fo r y o u r b rand ; th e y are th e o n e s w h o in flu e n ce th e w h o le re a lm o f c o n v e rs a tio n a b o u t y o u r to p ic. Y o u n e e d to u n d erstan d w h e th e r th ey a re say in g n ic e things, e x p re ssin g su p p ort, o r sim p ly m aking o b serv a tio n s o r critiquing. W h at is th e natu re o f th e ir co nv ersatio ns? H o w is m y b ran d b e in g p o sitio n e d relativ e to th e co m p e titio n in th a t space?

LOOK CLOSELY AT THE ACCURACY OF YOUR ANALYTIC TOOL Until re ce n tly , co m p u te r- b a s e d a u to m a te d to o ls w e r e n o t as a ccu ra te a s h u m an s fo r sifting th ro u g h o n lin e c o n ­ ten t. E v e n n o w , a c c u ra c y v arie s d e p e n d in g o n th e m ed ia. F o r p ro d u ct re v iew sites, h o te re v ie w sites, and T w itter, it c a n re a c h a n y w h e re b e tw e e n 8 0 to 9 0 p e rc e n t a ccu ra cy , b e c a u s e th e c o n te x t is m o re b o x e d in. W h e n y o u start lo o k in g at b lo g s a n d d iscu s­ s io n fo ru m s, w h e re th e c o n v e rs a tio n is m o re w id e-ran g in g , th e so ftw are c a n d eliver 6 0 to 7 0 p e rc e n t a c c u ra c y (P a in e an d C h av es, 2 0 1 2 ). T h e s e fig u res w ill in c re a se o v e r tim e, b e c a u s e th e an aly tics to o ls a re co n tin u a lly u p g ra d ed w ith n e w ru le s a n d im p ro v ed alg orith m s to re fle c t field e x p e r ie n c e , n e w p ro d u cts, ch a n g in g m a rk e t co n d itio n s , an d

e m e rg in g p attern s o l s p e e c h .

4 1 2 Part III • Predictive Analytics

Chapter 8 • W eb Analytics, W e b Mining, and Social Analytics 4 1 3

INCORPORATE SOCIAL MEDIA INTELLIGENCE INTO PLANNING O n c e y o u h av e b ig -p ic- ture p e rs p e c tiv e an d d eta iled insight, y o u ca n b e g in to in co rp o ra te th is in form ation into y o u r p la n n in g cy cle . B u t th at is e a sie r said th a n d o n e . A q u ick a u d ie n c e p o ll re v e ale d that v e ry fe w p e o p le cu rren tly in co rp o rate learn in g fro m o n lin e c o n v e rsa tio n s into th eir p lan n in g cy c le s (P a in e an d C haves, 2 0 1 2 ). O n e w ay to a ch ie v e this is to fin d tim e-lin k ed a sso cia tio n s b e tw e e n so cia l m ed ia m etrics an d o th e r b u sin e ss activ ities o r m ark et events. S o cia l m e d ia is typ ically e ith er o rg an ically in v o k e d o r in v o k e d b y so m e th in g y o u r o rg a ­ nizatio n d o e s ; th e re fo re , i f y o u s e e a s p ik e in activity at s o m e p o in t in tim e, you w an t to k n o w w h a t w as b e h in d that.

A p p licatio n C a se 8 .7 sh o w s a n in terestin g c a s e w h e re eH arm o n y , o n e o f th e m o st p o p u lar o n lin e relatio n sh ip serv ice p ro vid ers, u s e s s o c ia l m e d ia an aly tics to b e tte r listen, u n d erstan d , a n d s e rv ic e its cu stom ers.

Application Case 8.7 eH arm on y Uses Social M ed ia to Help Take th e M ystery Out o f Online D ating e H a n n o n y lau n ch ed in the U nited States in 2 0 0 0 and is n o w th e n u m b er-o n e tm sted relationship services provider in th e United States. Millions o f p e o p le have u se d eH arm ony’s Com patibility M atching System to find com p atib le long-term relationships; a n average o f 5 4 2 eH arm ony m em b ers marry every day in the U nited States, as a result o f b ein g m atch ed o n th e site.

T h e C h allen ge

O n lin e d atin g h a s co n tin u e d to in c re a se in p o p u ­ larity, a n d w ith th e a d o p tio n o f s o c ia l m ed ia the s o cia l m e d ia te a m at e H arm o n y sa w a n e v e n g reater o p p o rtu n ity to c o n n e c t w ith b o th cu rren t a n d future m e m b ers. T h e te a m a t eH arm o n y sa w so cia l m ed ia a s a c h a n c e to d isp e l a n y m yths a n d p re co n ce iv e d n o tio n s a b o u t o n lin e dating and , m o re im portantly, hav e s o m e fu n w ith th e ir so cia l m e d ia p re s e n ce . “F o r u s it’s a b o u t b e in g h u m an, a n d sh arin g great c o n te n t th at w ill h e lp o u r m e m b ers an d o u r so cial m ed ia fo llo w e rs ,” says G rant L angston, d irecto r o f s o cia l m e d ia a t eH a n n o n y . “W e b e lie v e th at i f th ere are c o n v e rsa tio n s h a p p e n in g aro u nd o u r b ran d , w e n e e d t o b e th e re an d b e a p art o f that d ia lo g u e .”

T h e A p p ro ach

e H arm o n y started u sin g S a le s fo rc e M arketing C loud to liste n to c o n v e rsa tio n s aro u n d th e b ra n d and a ro u n d k e y w o rd s lik e “b a d d a te ” o r “first d a te .” T h e y a lso to o k to F a c e b o o k an d T w itter to c o n n e c t w ith m e m b e rs, sh are s u c c e s s sto rie s—-including e n g a g e ­ m e n t a n d w ed d in g v id eo s— an d a n s w e r q u e stio n s fro m th o s e lo o k in g fo r d atin g ad vice.

“W e w a n te d to e n su re o u r te a m fe lt co m fo rt­ a b le u sin g s o cia l m e d ia to c o n n e c t w ith o u r co m ­ m unity s o w e s e t u p g u id elin es fo r h o w to re sp o n d and p r o c e e d ,” e x p la in s G ran t L angston. “W e try to u se h u m o r an d h av e s o m e fu n w h e n w e re a c h ou t to p e o p le th ro u g h T w itte r o r F a c e b o o k . W e think it m a k e s a h u g e d iffe re n ce an d h e lp s m a k e p e o p le fe e l m o re c o m fo rta b le .”

T h e R esu lts

B y u sin g s o c ia l m e d ia to h e lp e d u c a te a n d cre a te a w a ren e ss aro u n d th e b en e fits o f o n lin e dating, eH arm o n y h a s built a stro n g a n d loy al com m u nity . T h e s o cia l m e d ia te a m n o w has e ig h t sta ff m e m b ers w o rk in g to re s p o n d to s o c ia l in teractio n s an d p o sts, h e lp in g th e m re a c h o u t to clie n ts an d re s p o n d to hu n d red s o f p o sts a w e e k . T h e y p la n to start c re ­ atin g F a c e b o o k a p p s that c e le b r a te th e ir m e m b e rs ’ s u cce s s , a n d th e y a re lo o k in g to c re a te s o m e n e w v id e o s a ro u n d s o m e co m m o n d atin g m istakes. T h e so cia l te a m at eH arm o n y is m ak in g all th e right m o v e s a n d th e ir hard w o rk is paying o f f fo r th eir m illions o f h a p p y m em b ers.

Q u e s t i o n s f o r D i s c u s s i o n

1. H o w d id e H arm o n y u se so cia l m ed ia to e n h a n c e o n lin e dating?

2. W h at w e re th e c h a lle n g e s, th e p ro p o s e d s o lu ­ tion , a n d th e o b ta in e d results?

Source: SalesForce Marketing Cloud, Case Study, salesforcem arketingcloud .com ; eharm ony.com .

4 1 4 Part III • Predictive Analytics

Monitoring social media, identifying interesting conversations among potential customers, an d in ferrin g w h a t th e y a re say in g a b o u t y o u r co m p a n y , y o u r p ro d u cts, an d s e rv ice s ts a n e sse n tia l y e t a ch a lle n g in g ta sk fo r m an y o rg an izatio n s. G e n e ra lly sp e a k in g , ther a re tw o m a in p aths th a t a n o rg a n iz a tio n c a n ta k e to a tta in socialt medra. ca p a b ilitie s: in -h o u s e d e v e lo p m e n t o r o u tso u rcin g . B e c a u s e th e SM A -related field evolving/m atu ring an d b e c a u s e b u ild in g a n e ffe c tiv e SMA sy stem re q u ires e x te n s iv e k n o w le d g e in se v e ra l re la te d field s (e .g ., W e b , te x t m ining, p re d ictiv e an aly tics, report ing, visu alization , p e rfo rm a n ce m a n a g e m e n t, e tc .) , w ith th e e x c e p tio n o f v e ry large e n te rp rise s, m o st o rg a n iz a tio n s c h o o s e th e e a s ie r path: ou tso u rcin g .

D u e to the astounding em phasis given to SMA, in th e last fe w years w e have w itnessed a n incredible em erg en ce o f start-up com p anies claim ing to provide practical cost-effective SMA solutions to organizations o f all sizes and types. B e c a u s e w h at they offered w as no m u ch m ore than just m onitoring a few keyw ords about brands/products/services in social m edia m an y o f th em did n o t su cceed . W h ile there still is a lot o f uncertainty and ch u m in h e m arketplace, a significant n u m ber o f th em hav e survived a nd evolved to provide services tha c o b ey o n d b asic m onitoring o f a fe w brand n am es and keyw ords; th ey provide an integrated ap p roach that help s m any parts o f th e business, including product d evelopm ent, custom er support, public outreach, lead generation, m ark et research, and cam paign m anagem ent.

In th e fo llow in g sectio n , w e list and briefly d efin e 10 SMA tools/vendors. T h is list is no t m e an t to b e “th e ab so lu te to p 1 0 ” o r th e co m p le te top-tier lead ers m th e m arket. It is to provide only 10 o f th e m any su ccessfu l SMA ven d ors and their re sp ectiv e tools/services

w ith w h ich w e h av e so m e familiarity.

ATTENSITY360 A tten sity360 operates on four k e y principles: liste n , analyze relate and act. A tten sity 360 helps monitor trending topics, influences, and the reach o f your brand while recommending ways to join the conversation. Attensity Analyze applies text ana­ lytics to unstructured text to extract meaning and uncover trends. Attensity Respond helps automate the routing o f incoming social media mentions into user-defined queues. Clients include Whirlpool, Vodofone, Versatel, TMobile, Oracle, and Wiley.

RADIAN6/SALESFORCE CLOUD R ad ian 6 , p u rc h a s e d b y S a le s fo rc e in 2 0 1 1 , w o rk s w ith b ran d s to h e lp th e m liste n m o re in te llig e n tly t o y o u r co n s u m e rs , co m p e tito rs, an d intlu - e n c e r s w ith th e g o a l o f g ro w in g y o u r b u s in e s s via d etailed , re al-tim e in sigh ts. B e y o n th e ir m o n ito rin g d a sh b o a rd , w h ic h tracks m e n tio n s o n m o re th an 1 0 0 m illio n so cia m ed ia sites, th ey o ffe r a n e n g a g e m e n t c o n s o le th a t allo w s y o u to c o o rd in a te y o u r in tern a re s p o n s e s to e x te rn a l activity b y im m ed iately u p d atin g y o u r b lo g , T w itter, an d F aceb cw a c c o u n ts all in o n e sp o t. T h e ir clie n ts in clu d e R e d C ross, A d o b e , AAA, C irq u e du S o l e i , H & R B lo c k , M arch o f D im es, M icroso ft, P e p s i, a n d S o u th w e st A irlines.

SYSOMOS M an ag in g c o n v e rs a tio n s in re a l tim e , S y s o m o s 's H e a rtb e a t is a re a l­ tim e m o n ito rin g a n d m e a s u re m e n t to o l th a t p ro v id e s c o n s ta n tly u p d a te d s n a p s h o s o f s o c ia l m e d ia c o n v e rs a tio n s d e liv e re d u s in g a v a rie ty o f u se r-frie n d ly graphics^ H e a rtb e a t o r g a n iz e s co n v e rs a tio n s , m a n a g e s w o rk flo w , fa c ilita te s co^la ^ ° ^ ĉ p ro v id e s w a y s to e n g a g e w ith k e y i n f l u e n c e s T h e ir c lie n ts in c lu d e IB M , H SB C . R o c h e K e tch u m , S o n y E ric s s o n , P h ilip s, C o nA gra, E d e lm a n , S h e ll O il, N o k ia , S a p ie n t, Cm ,

a n d In te rb ra n d . O w n e r: M ark etw ire.

COLLECTIVE INTELLECT B o u ld e r, C o lo ra d o -b a s e d C o llectiv e In te lle ct, w h ic h started out b y providing m o n ito rin g to fin an cial firms, h a s e v o lv e d into a to p -tier p lay er m th e m ar­ k e tp la ce o f "social m ed ia in te llig e n ce gath erin g . U sin g a co m b in a tio n o f self-se rv e clien t

Social M edia Analytics Tools and V endors

Chapter 8 • W eb Analytics, W eb Mining, and Social Analytics 4 1 5

d ash b o ard s a n d h u m an analy sis, C o llectiv e In te lle ct o ffe rs a ro b u st m o n ito rin g an d m e a ­ su rem e n t to o l su ite d to m id -size to large c o m p a n ie s w ith its S o cia l CRM In sigh ts platform . T h e ir clie n ts in clu d e G e n e ra l Mills, N BC U niversal, P ep si, W alm art, U n ilev er, M illerCoors, P aram ou nt, a n d Siem en s.

WEBTRENDS W e b tren d s o ffe rs s erv ices g e a re d tow ard m o nito ring, m e asu rin g , analyz­ ing, p rofiling, a n d targ eting a u d ie n c e s fo r a brand. T h e p artn er-b ase d p latfo rm allow s fo r cro w d -so u rce d im p ro v e m en ts a n d p ro b le m solving, cre a tin g tra n sp a ren cy fo r th eir p ro d u cts an d s e rv ic e s . T h e ir clie n ts in clu d e C B S, N BC U niversal, 2 0 th C en tu ry F o x , AOL, E le ctro n ic Arts, Lifetim e, an d N estle.

CRIMSON HEXAGON C am brid ge, M a s s a c h u se tts -b a s e d C rim son H e x a g o n tap s into b illion s o f c o n v e rsa tio n s taking p la c e in o n lin e m ed ia a n d turns th e m in to a ctio n a b le data fo r b e tte r b ra n d u n d erstan d in g and im p ro v em en t. B a s e d o n a te c h n o lo g y lice n se d from H arvard, its V o x T ro t O p in io n is a b le to an alyze v ast am o u n ts o f q u alitativ e infor­ m ation a n d d eterm in e quantitative p ro p o rtio n o f o p in io n . T h e ir clie n ts in clu d e CNN, H an es, AT&T, HP, J o h n s o n & J o h n s o n , M ash ab le, M icroso ft, M onster, T h o m s o n R euters, R ubberm aid , S y b a s e , and The W all Street Jo u rn a l.

CONVERSEON N ew Y o r k - b a s e d so cial m e d ia co n su ltin g firm C o n v e rse o n , n a m e d a lea d e r in th e s o c ia l m e d ia m o n ito rin g s e c to r b y F o rrester R e se a rch , b u ild s tailo red d ash ­ b oard s fo r its e n te rp rise installations and o ffers p ro fe ssio n a l s e iv ic e s a ro u n d e v ery step o f th e so cia l b u s in e s s in te llig e n ce p ro ce ss. C o n v e rse o n starts w ith th e te c h n o lo g y a n d adds h u m an analy sis, resultin g in high-quality d ata and im p ressiv e fu n ction ality . T h e ir clients inclu de D o w , A m w ay, G ra c o , and o th er m a jo r brand s.

SPIRAL16 Spiral 16 takes an in-depth lo o k at w h o is saying w h at a b o u t a b ran d an d co m ­ pares results w ith th o se o f top com petitors. T h e g o al is to h elp y o u m o n ito r th e effective­ ness o f y o u r so cia l m ed ia strategy, understand the sen tim ent b eh in d con v ersatio n s online, .md m ine larg e am ounts o f data. It u se s im pressive 3D displays and a standard d ashboard. Their clients in clu d e T oy o ta, Lee, a n d Cadbury.

5UZZLOGIC B uzzL ogic u ses its tech n o lo g y platform to identify and org an ize th e conv ersa­ tion universe, com b in in g b o th con v ersatio n to p ic and au d ien ce to h e lp b ran d s reach audi­ ences w h o a re p assio nate o n everything fro m th e latest te c h craze an d clo u d com p u ting to parenthood an d politics. T h e ir clien ts include Starbucks, A m erican E xp ress, H B O , and HP.

SPROUTSOCIAL F o u n d ed in 2 0 1 0 in C h icag o , Illinois, S p ro u tSo cial is a n in n o v ativ e s o cial m edia an alytics co m p a n y that p ro v id e s s o cia l an aly tics s erv ices to m a n y w e ll-k n o w n iinns a n d organ izatio n s. T h e ir clie n ts in clu d e Y a h o o !, N okia, P ep si, St. J u d e C h ild re n s Research C en ter, H yatt R e g e n cy , M cD o nald s, an d AMD. A sa m p le s c r e e n sh o t o f their social m ed ia so lu tio n d ash b o ard is sh o w n in Figure 8 .1 2 .

SECTION 8 . 1 0 REVIEW QUESTIONS

1 . W h at is s o c ia l m ed ia analytics? W h at typ e o f d ata is an aly zed w ith it? 2. W h at a r e th e reasons/ m otiv ations b e h in d th e e x p o n e n tia l g ro w th o f s o cia l m ed ia

analytics? 3 . H ow c a n y o u m e a su re th e im p a ct o f s o c ia l m e d ia analytics? 4 . List an d b rie fly d e s c rib e th e b e s t p ra ctice s in s o cia l m e d ia analytics. 5. W hy d o y o u th in k s o cia l m e d ia an alytics to o ls are usu ally o ffe re d as a serv ice and n o t

a tool?

4 1 6 Part III • P redictive Analytics

F IG U R E 8 .1 2 A S o cial M edia A n a ly tic s Scre en sh o t. Source: Courtesy of sp ro u tso cial.co m .

Chapter Highlights

• W e b m in in g c a n b e d e fin e d as th e d iscov ery an d an aly sis o f in terestin g an d u sefu l in form ation fro m th e W e b , a b o u t th e W e b , an d u su ally using W e b -b a s e d to ols.

• W e b m in in g c a n b e v ie w e d a s co n sistin g o f th ree areas: W e b c o n te n t m ining, W e b stru cture m in­ ing, a n d W e b u sa g e m ining.

• W e b c o n te n t m in in g re fe rs to th e au tom atic e x tra ctio n o f useful in form ation fro m W e b p ages. It m ay b e u s e d to e n h a n c e s e a rch results p ro ­ d u ce d b y s e a rc h e n g in e s.

• W e b s tru ctu re m in in g re fe rs to g e n e r a tin g in te r­ e stin g in fo rm a tio n fro m th e lin k s in clu d e d in W e b p a g e s . T h is is u s e d in G o o g le ’s p a g e ra n k a lg o rith m t o o rd e r th e d isp lay o f p a g e s , for e x a m p le .

• W e b structure m ining c a n also b e used to identify th e m e m b ers o f a sp e cific com m unity and p erhap s e v e n th e ro les o f th e m e m b ers in th e com m unity.

• W e b u sa g e m ining re fe rs to d ev elo p in g useful in form ation th ro u g h analysis o f W e b serv e r logs, u s e r p ro files, a n d tra n sa ctio n in form ation.

Chapter 8 • W eb Analytics, W e b Mining, and Social Analytics 4 1 7

• W e b u sa g e m in in g c a n assist in b e tte r CRM, p e r­ so n a liz a tio n , site n av ig atio n m o d ification s, and im p ro v ed b u sin e ss m od els.

• T e x t an d W e b m ining are em erging as critical c o m p o n e n ts o f th e n e x t g e n eratio n o f business in te llig e n ce to ols, e n ab lin g organizations to co m ­ p e te successfully.

• A s e a rch e n g in e is a softw are program that s earch es fo r d ocu m en ts (In tern e t sites o r files), b a se d o n k eyw ord s (individual w ord s, m ulti-w ord terms, o r a co m p le te s e n te n ce ) users h av e provided that h av e to d o w ith th e su b je ct o f their inquiry.

• P a g eR a n k is a lin k analysis algorithm n am ed after Larry P a g e , w h o is o n e o f th e tw o in v en to rs o f G o o g le , w h ich b e g a n as a re se a rch p r o je c t at Stanford U niversity in 1996. P ag eR an k is u se d b y th e G o o g le W e b s ea rch e n g in e .

• S e a rch e n g in e op tim izatio n (S E O ) is th e in te n ­ tio n al activity o f affectin g th e visibility o f an e -c o m m e r c e site o r a W e b site in a s ea rch e n g in e ’s natu ral (u n p a id o r o rg a n ic) s e a rch results.

• A m atu rity m o d e l is a fo rm al d e p ictio n o f criti­ ca l d im en sio n s a n d th e ir c o m p e te n c y lev els o f a b u s in e s s p ractice.

• V o ic e o f th e cu sto m e r (V O C ) is a te rm usually used to d e s c rib e th e an alytic p ro ce s s o f captu r­ in g a cu sto m e r’s e x p e cta tio n s , p re fe re n c e s, and aversion s.

• Social an aly tics is th e m o n ito rin g , an alyzing, m e a­ suring, an d in terp retin g o f digital in te ractio n s an d re latio n sh ip s am o n g p e o p le , to p ics, id eas, an d co n ten t.

• A s o cia l n e tw o rk is a so cia l stru cture c o m p o s e d o f individ uals/p eople (o r g ro u p s o f individuals or o rg an izatio n s) lin k e d to o n e a n o th e r w ith so m e ty p e o f con n ectio n s/ relatio n sh ip s.

• S o cia l m e d ia refers to th e e n a b lin g te c h n o lo g ie s o f s o cia l in te ractio n s am o n g p e o p le in w h ich th e y c re a te , sh are, a n d e x c h a n g e in form ation, id eas, an d o p in io n s in virtual co m m u n ities and n etw o rks.

• S o cia l m e d ia an alytics re fe rs to th e sy stem atic and scie n tific w ay s to c o n s u m e th e vast am o u n t o f c o n te n t c re a te d b y W e b -b a s e d so cia l m ed ia o u t­ lets, to o ls , a n d te c h n iq u e s fo r th e b ette rm e n t o f a n o rg a n iz a tio n ’s co m p etitiv en ess.

Key Terms

authoritative p a g e s clickstream an alysis cu stom er e x p e r ie n c e m an ag e m e n t

(CEM ) hubs h y p erlin k -in d u ced to p ic sea rch

(H ITS)

s e a rch e n g in e so cia l n e tw o rk spiders v o ic e o f cu sto m e r (V O C ) W e b analytics W e b c o n te n t m ining W e b craw ler

W e b m ining W e b stru cture m ining W e b u sa g e m ining

Questions for Discussion 1 . Explain h o w th e size and com plexity o f the W eb m akes

kn ow led ge discovery challenging. 2 . W hat are the limitations o f a sim ple keyword-based search

engine? H ow does W eb mining address these deficiencies? 3- Define W eb mining? H ow is it different from W eb analytics? 4 . How d o es W e b con ten t mining in crease a com pany’s

efficiency and com petitive advantage? 5. How can w e collectively use W eb content mining, W eb

structure mining, and W eb usage mining for business gains? 6 . W hat are authoritative pages, hu bs, and hyperlink-

induced to p ic search (HITS)? D iscuss the differences betw een citations in research articles and hyperlinks on W eb pages.

7. D efine a sea rch en g in e and discuss its anatomy. 8 . W hat is P ageR ank algorithm? W hat is th e relation

betw een P ageR ank and citation analysis? H ow d oes G o o g le u se PageRank?

9 . D efine SEO? W hat d o es it involve? D iscuss th e most com m only u sed m ethods for SEO.

1 0 . H ow w ould optim izing search eng in es help businesses? H ow d oes over-reliance o n search eng in e traffic harm busin esses a n d h o w c a n it b e avoided?

1 1 . W hat is W e b Analytics? D iscuss the application o f W eb analytics for businesses.

1 2 . D iscuss the im pact o f W eb analytics m etrics o n market insight.

1 3 . D iscuss W e b site optim ization ecosystem ? 1 4 . D iscuss so cia l netw ork analysis. W hich types o f social

netw orks are relevant to businesses?

4 1 8 Part III • Predictive Analytics

1 5 . H ow c a n w e m easure the im pact o f social m edia analyt ics? H ow c a n social m edia intelligence b e incorporated into planning?

Exercises Teradata University Network (TUN) and Other Hands-on Exercises 1 . Visit teradatauniversitynetwork.com. Identify cases

ab o u t W eb mining. D escrib e recen t d evelopm ents in the field. I f y o u cannot find en o u g h cases at th e Teradata University netw ork W e b site, b ro ad en you r search to o th er W eb -b ased resources.

2 . G o to te r a d a ta u n iv e r s ity n e tw o r k .c o m or locate white papers, W e b seminars, and other materials related to W eb mining. Synthesize your findings into a short written report.

3 . B row se th e W e b and your library’s digital databases to identify articles that m ake th e linkage betw een text/Web m ining a n d contem porary business intelligence systems.

Team Assignments and Role-Playing Projects 1 . Exam ine h o w W eb -b ased data c a n b e captured autom ati­

cally using th e latest tech n olog ies. O n c e captured, w hat are th e potential patterns that you c a n extract from these con tent-rich , m ostly unstructured data sources?

2 . Interview administrators in you r colleg e o r executives in your organization to determ ine h ow W e b mining is assisting (o r cou ld assist) them in their w ork. W rite a proposal d escribing your findings. Include a preliminary c o s t-b e n e fit analysis in you r report.

3 . G o o n lin e, search for publicly available W eb usage or social m ed ia data files, and dow nload o n e o f you r ch oice. Th en , d ow nload and u se o n e o f th e free tools to analyze th e data. W rite your findings and ex p erien ces in a p rofes­ sionally org anized report.

Internet Exercises 1 . W hat types o f o n lin e inform ation c a n b e co llected using

o f W e b crawlers? D iscuss h o w organizations cou ld use this inform ation for d ecision making.

2 . W rite a b rief report o n th e effectiven ess o f your edu­ cational institution’s current SEO illustrating w ith your findings and recom m endations. How c a n effective SEO strategies in flu en ce th e num ber o f potential students and their characteristics?

3 . How would W eb usage mining ensure content generation and user m anagem ent o n a site o f your choice? Referring to the chapter, carry out a basic off-site analysis for the site and suggest steps to improve its user retention strategy.

4 . U se social m ed ia W ebsites to gau ge op in ion s about a clothing b ran d am ong a sam ple o f your immediate friends. Identify exam p les o f h ow social netw orks influ­ e n c e product selection.

5 . Visit op en w eban aly tics.com and com p are the func­ tionality o f this o p e n sou rce tool with its com petitors. Distinguish b e tw e e n th e tw o. W hy w ou ld com panies pu rch ase w e b analytics solutions?

6 . G o to w ww.google.com/adwords and d ow nload at least th ree su ccess stories. D oes th e existen ce o f services su ch as G o o g le AdWords affect th e n eed for SEO in any way? Justify you r answer.

7 . G o to kdnuggets.com. E xplore the section s o n applica­ tions as w ell as softw are. Find nam es o f at least three additional p ackag es for W e b m ining and social media analytics.

End-of-Chapter Application Case

Keeping Stu d en ts on Track w ith W e b and Pred ictive

W hat m akes a colleg e student stay in school? O ver the years, educators h av e held a lot o f theories from student d em o­ graphics to attend ance in colleg e p rep cou rses but they ve lacked th e hard data to prove conclusively w hat really drives retention. As a result, colleg es and universities have struggled to understand h o w to low er dropout rates and k e e p students o n track all th e w ay to graduation.

T hat situation is changing at A m erican P ublic University System (APUS), an onlin e university serving 7 0 ,0 0 0 distance learners from th e United States and m ore than 1 00 countries.

nalytics APUS is break in g new ground by using analytics to zero in o n th o se factors that m ost influence a student’s d ecision to stay in sch o o l o r drop out.

“B y leveragin g th e p o w e r o f predictive analytics, w e c a n pred ict th e probability that any given stud en t will drop ou t,” says Phil Ic e , APUS’ director o f cou rse design, research , and d evelop m ent. “This translates into actio n ab le busi­ n ess intelligen ce that w e c a n d ep loy acro ss th e enterprise to create and m aintain th e con ditions for m axim um student reten tion .”

Chapter 8 • W e b Analytics, W eb Mining, and Social Analytics 4 1 9

Rich Data Sources As an online university, APUS has a rich store of student information available for analysis. “All o f the activities are technology mediated here,” says Ice, “so w e have a very nice record o f what goes on with the student. W e can pull demo­ graphic data, registration data, course level data, and more.” Ice and his colleagues then develop metrics that help APUS analyze and build predictive models o f student retention.

One o f the measures, for example, looks at the last time a student logged into the online system after starting a class. If too many days have passed, it may be a sign the student is about to drop out. Educators at APUS combine such online activity data with a sophisticated end-of-course survey to build a complete model o f student satisfaction and retention.

The course survey yields particularly valuable data for APUS. Designed around a theoretical framework known as the Community o f Inquiry, the survey seeks to understand the stu­ dent’s learning experience by analyzing three interdependent elements: social, cognitive, and teaching presence. Through the survey, Ice says, “We hone in on things such as a student’s perception o f being able to build effective community."

I n c r e a s i n g A c c u r a c y It turns out that the student’s sense o f being part o f a larger community— his or her “social presence”— is one of the key variables affecting the student’s likelihood of staying in school. Another one is the student’s perception of the effectiveness of online learning. In fact, when fed into IBM SPSS Modeler and measured against disenrolment rates, these two factors together accounted for nearly 25 percent o f the overall sta­ tistical variance, meaning they are strong predictors o f stu­ dent attrition. With the adoption o f advanced analytics in its retention efforts, APUS predicts with approximately 80 percent certainty whether a given student is going to drop out.

Some o f the findings generated by its predictive mod­ els actually cam e as a surprise to APUS. For example, it had long been assumed that gender and ethnicity were good predictors o f attrition, but the models proved otherwise. Educators also assumed that a preparatory course called "■College 100: Foundations o f Online Learning” was a major driver o f retention, but an in-depth analysis using Modeler came to a different conclusion. “When we ran the numbers, we found that students who took it were not retained the way we thought they were,” says Dr. Frank McCluskey, pro­ vost and executive vice president at APUS. “IBM SPSS predic­ tive analytics told us that our guess had been wrong.”

S tr a te g ic C o u r s e A d ju s tm e n ts The next step for APUS is to put its new-found predictive intelligence to work. Already the university is building online "dashboards” that are putting predictive analytics into the hands o f deans and other administrators who can design and implement strategies for boosting retention. Specific action plans could include targeting individual at-risk students with special communications and counseling. Analysis o f course

surveys can also help APUS adjust course content to better engage students and provide feedback to instructors to help improve their teaching methods.

Survey and modeling results are reinforcing the univer­ sity’s commitment to enriching the student’s sense o f commu­ nity— a key retention factor. Online courses, for example, are being refined to promote more interactions among students, and social media and online collaboration tools are being deployed to boost school spirit. “W e have an online student lounge, online student clubs, online student advisors,” says McCluskey. “W e want to duplicate a campus fully and com­ pletely, where students can grow in all sorts o f ways, learn things, exchange ideas— maybe even books— and get to know each other.”

S m a r t D e c is io n s While predictive modeling gives APUS an accurate picture o f the forces driving student attrition, tackling the problem means deciding among an array o f possible inteivention strategies. To help administrators sort out the options, APUS plans to implement a Decision Management System, a solu­ tion that turns IBM SPSS Modeler’s predictive power into intelligent, data-driven decisions. The solution will draw from Modeler’s analysis o f at-risk students and suggest the best intervention strategies for any given budget.

APUS also plans to delve deeper into the surveys by mining the open-ended text responses that are part o f each questionnaire (IBM SPSS Text Analytics for Surveys will help with that initiative). All of these data-driven initiatives aim to increase student learning, enhance the student’s experience, and build an environment that encourages retention at APUS. It’s no coincidence that achieving that goal also helps grow the university’s bottom line. “Attracting and enrolling new students is expensive, so losing them is costly for us and for students as well,” McCluskey says. “That is why we are excited about using predictive analytics to keep retention rates as high as possible.”

Q u e s t i o n s f o r t h e E n d - o f - C h a p t e r A p p l i c a t i o n C a s e

1 . Describe challenges that APUS was facing. Discuss the ramifications o f such challenges.

2. What types o f data did APUS tap into? What do you think are the main obstacles one would have to over­ com e when using data that comes from different domains and sources?

3 . What solutions did they pursue? What tools and tech­ nologies did they use?

4 . What w ere the results? D o you think these results are also applicable to other educational institutions? Why? Why not?

5 . What additional analysis are they planning on conduct­ ing? Can you think o f other data analyses that they could apply and benefit from?

Source: IBM, Customer Success Stories, w w w - 0 1 .i b m .c o m / s o f t w a r e / s u c c e s s / c s s d b .n s f / C S / G R E E - 8 F 8 M 7 6 (accessed March 2013).

4 2 0 Part III • Predictive Analytics

References Brin, S., and L. P age. (2012). ‘'Reprint o f the A natomy o f a

Large-Scale H ypertextual W e b Search E ngine.” C o m p u ter N etw orks, V ol. 56 , No. 18, pp. 3 8 2 5 -3 8 3 3 ,

Brogan, C„ and B aston e, J . (2 0 1 1 ). “Acting o n Custom er Intelligen ce from Social Media: T h e N ew Edge for Building C ustom er Loyalty and Y o u r B rand .” SAS w hite paper, s a s . c o m / r e s o u r c e s / w h ite p a p e r / w p _ 2 1 1 2 2 . p d f (accessed

March 2013). Coussement, K , and D. V an D en P oel. (2009). “Improving

Custom er Attrition Prediction by Integrating Emotions from Client/Company Interaction Emails and Evaluating Multiple Classifiers.” E x p ert S ystem s w ith A p p lica tio n s, Vol. 36, No. 3, pp. 6 1 2 7 -6 1 3 4 .

Cutts, M. (2 0 0 6 , February 4 ). “Ram ping t :p o n International W ebsp am .” m a ttc u tts .c o m / b lo g . m a ttc u tts .c o m / b lo g / r a m p i n g - u p - o n - i n t e r n a t i o n a l - w e b s p a m ( a c c e s s e d

M arch 2013). Etzioni, O . (1 9 9 6 ). “T h e W orld W ide W eb: Q uagm ire or G old

Mine?” C o m m u n ic a tio n s o f th e ACM, Vol. 39, No. 11, pp. 6 5 -6 8 .

G oodm an, A. (2005). “Search Engine Show dow n: B lack Hats Versus W hite Hats at SES.” SearchEngineW atch. s e a r c h e n g in e w a tch .co m /a rticle /2 0 6 6 0 9 0 /S e a rch -E n g in e - S h o w d o w n - B l a c k - H a t s - v s .- W h i t e - H a t s - a t - S E S ( a c ­

cessed February 2013). HBR. (2010). “T h e New Conversation: Taking Social Media from

Talk to Action,” A SAS-sponsored research report by Harvard Business Review Analytic Services, sas.co m / reso u rces/ w h ite p a p e r / w p _ 2 3 3 4 8 .p d f (accessed March 2013).

Kaplan, A. M., and M. H aenlein. (.2010). "Users o f th e World. Unite! T h e Challenges and Opportunities o f Social M edia.” B u sin ess H o r iz o n s, Vol. 53, No. 1, pp. 5 9 -6 8 .

K leinberg, J . (1999)- “Authoritative Sources in a Hvperlinked E nvironm ent.” J o u r n a l o f th e ACM, Vol. 4 6 , No. 5, pp. 6 0 4 -6 3 2 .

Lars-Henrik, S. (1 9 9 6 ). “C om m onness Across Cultures,” in Anindita N iyogi Balslev. C ro ss-C u ltu ra l C o n v ersa tio n : In itia tio n . O xford University Press.

Liddy, E. (2 0 1 2 ). “H ow a Search Engine W ork s.” w w w . in fo to d a y .c o m / s e a r c h e r / mayO 1 / lid d y .h tm (accessed N ovem ber 2012).

Masand, B . M., M. Spiliopoulou, J . Srivastava, and O. R. Za'fane. (2 0 0 2 ). “W e b Mining for U sage Patterns and P rofiles.” S IG K D D E x p lo ra tion s, Vol. 4 , No. 2, pp. 1 2 5 -1 3 2 .

Morgan, N., G. Jo n e s , and A. Hodges. (2010). “T h e Com plete G uide to Social Media from th e Social Media G uys.” t h e s o c i a l m e d i a g u y s . c o . u k / w p - c o n t e n t / u p l o a d s / d o w n l o a d s / 2 0 1 1 /0 3 / C o m p le t e G u id e t o S o c ia lM e d ia . p d f (a c ce s se d February 2013).

Nasraoui, O ., M. Spiliopoulou, J . Srivastava, B. M obasher, and B . Masand. (2 0 0 6 ). “W ebK D D 2006: W e b Mining and W eb Usage Analysis Post-W orkshop R eport.” ACM SIG KD D E x p lo r a tio n s N ew sletter, Vol. 8, No. 2, pp. 84-8 9 -

Pain e, K. D., a n d M. Chaves. (2 0 1 2 ). “Social Media M etrics.” SAS w hite paper, s a s .c o m / r e s o u r c e s / w h ite p a p e r / w p _ 1 9 8 6 l . p d f (a c cessed February 2013).

Peterson, E. T . (2 0 0 8 ). “T h e V oice o f Custom er: Qualitative D ata as a Critical Input to W eb Site O ptim ization.” f o r e s e e r e s u l t s . c o m / F o r m _ E p e t e r s o n _ W e b A n a l y t i c s . h t m l (a c ce s se d May 2009)-

Scott, W. Richard, and Gerald F. Davis. (2003). “Networks In and Around Organizations.” O rg a n iz a tio n s a n d O rg a n izin g . U pper Saddle River, NJ: Pearson Prentice Hall.

T h e W estover G roup. (2 0 1 3 ). “20 K ey W eb Analytics Metrics and H ow to U se T h em .” w w w .th e w e s to v e r g r o u p .c o m (accessed February 2013).

Turetken, O ., and R. Sharda. (2 0 0 4 ). “D evelop m ent o f a Fish eye-B ased Inform ation Search Processing Aid (FISPA) for Managing Inform ation O verload in the W eb Environm ent.” D e c is io n S u p p ort S ystem s, Vol. 37, No. 3, pp- 4 1 5 -4 3 4 .

Zhou, Y ., E. Reid, J . Qin, H. C hen, and G . Lai. (2005). “U.S. D om estic Extremist G roups o n the W eb: Link and Content Analysis.” IEEE In te llig e n t S ystem s, Vol. 20, No. 5,

pp. 4 4 -5 1 .

Prescriptive Analytics

LEARNING OBJECTIVES FOR PART IV

■ U n d e r s ta n d t h e a p p lic a tio n s o f p re s c r ip tiv e a n a ly tic s t e c h n i q u e s in c o m b i n a t io n w ith r e p o r tin g a n d p r e d ic tiv e a n a ly tic s

■ U n d e r s ta n d t h e c o n c e p t s o f a n a ly tic a l m o d e ls fo r s e l e c t e d d e c i s io n p r o b le m s in c lu d in g l in e a r p r o g r a m m in g a n d a n a ly tic h ie r a r c h y p r o c e s s

■ R e c o g n iz e th e c o n c e p t s o f h e u ris tic s e a r c h m e th ­ o d s a n d s im u la tio n m o d e ls fo r d e c is io n s u p p o rt

® U n d e r s ta n d th e c o n c e p t s a n d a p p lic a tio n s o f a u to m a te d r u le s y s te m s a n d e x p e r t s y s te m t e c h n o lo g ie s

B G a i n fa m ilia r ity w it h k n o w le d g e m a n a g e m e n t a n d c o l la b o r a t io n s u p p o r t s y s te m s

This part extends the decision support applications beyond reporting and data mining methods. It includes coverage of selected techniques that can be employed in combination with predictive models to help support decision making. We focus on techniques that can be implemented relatively easily using either spreadsheet tools or by using stand-alone software tools. Of course, there is much additional detail to be learned about management science models, but the objective of this part is to simply illustrate what is possible and how it has been implemented in some real settings. We also include coverage of automated decision systems that implement the results from various models described in this book, and expert system technologies. Finally, we conclude this part with a discussion of knowledge management issues and group support systems. Technologies and issues in knowledge management and group support systems directly impact the delivery and practice of prescriptive analytics.

Model-Based Decision Making: Optimization and

Multi-Criteria Systems

LEARNING OBJECTIVES

■ U n d erstand th e b a sic c o n c e p ts o f analytical d e cisio n m o d elin g

■ D e s crib e h o w prescrip tiv e m o d els in te ract w ith d ata and th e u s e r

■ U n d erstand s o m e d ifferen t, w e ll-k n o w n m o d e l classe s

■ U n d erstand h o w to stru cture d e cisio n m ak in g w ith a fe w alternatives

■ D e s c r ib e h o w s p re a d sh ee ts c a n b e u se d fo r analy tical m o d e lin g an d so lu tio n

■ E x p la in th e b a sic c o n c e p ts o f op tim izatio n and w h e n to u s e th em

■ D e s c r ib e h o w to structure a lin ear p ro gram m in g m o d el

* D e s crib e h o w to h a n d le m u ltiple g o als

■ E x p la in w h a t is m e a n t b y sensitivity analysis, w h a t-if a nalysis, a n d goal se e k in g

■ D e s c r ib e th e k e y issu es o f m ulti-criteria d e c is io n m aking

I n this ch a p te r w e d e s c rib e s e le c te d te c h n iq u e s e m p lo y e d in prescrip tiv e analytics. W e p re s e n t this m aterial w ith a n o te o f ca u tio n : M o d elin g c a n b e a very difficult to p ic an d is as m u ch a n art as a s c ie n c e . T h e p u rp o se o f this ch a p te r is n o t n e c e s ­ sarily fo r y o u to m aster th e topics o f m o d e lin g a n d analysis. Rather, th e m aterial is g e a re d to w ard g a in in g fa m ilia r ity w ith th e im p ortan t co n c e p ts as th ey relate to D SS an d th eir u se in d e c is io n m akin g. It is im portant to re c o g n iz e that th e m o d e lin g w e d iscu ss h e re is o n ly cu rsorily related to th e co n c e p ts o f data m o d elin g . Y o u sh o u ld n o t co n fu s e th e tw o. W e w a lk th ro u g h s o m e b a sic c o n c e p ts and d efin itio n s o f m o d e lin g b e fo r e in trod u cin g th e in flu e n ce diagram s, w h ich c a n aid a d e cisio n m a k e r in s k e tch in g a m o d e l o f a situ ation a n d e v e n solv in g it. W e n e x t in tro d u ce th e id ea o f m o d e lin g d irectly in sp re ad sh ee ts. W e th e n d iscu ss th e stru cture an d a p p lica tio n o f s o m e su cce ssfu l tim e -p ro v e n m o d e ls and m e th o d o lo g ie s: o p tim ization , d e c is io n analysis, d e c is io n trees, and an aly tic h ierarch y p ro ­ ce ss . T h is ch a p te r in clu d es th e fo llo w in g s ectio n s:

9 . 1 O p e n in g V ig n e tte : M id w e st IS O S a v e s B illio n s b y B e t t e r P la n n in g o f P o w e r P la n t O p e r a tio n s a n d C a p a c ity P la n n in g 4 2 3

9 . 2 D e c is io n S u p p o r t S y s te m s M o d e lin g 4 2 4

Chapter 9 • M odel-Based D ecisio n Making: O ptim ization and Multi-Criteria Systems 423

9 .3 S tru c tu re o f M a th e m a tic a l M o d e ls fo r D e c is io n S u p p o rt 4 2 9 9 . 4 C e rta in ty , U n c e rta in ty , a n d R is k 4 3 1 9 .5 D e c i s io n M o d e lin g w ith S p r e a d s h e e ts 4 3 4 9 .6 M a th e m a tic a l P ro g ra m m in g O p tim iz a tio n 4 3 7 9 .7 M u ltip le G o a ls , S e n s itiv ity A n a ly s is, W h a t- If A n a ly sis, an d G o a l S e e k i n g 4 4 6 9 .8 D e c is io n A n a ly sis w ith D e c is io n T a b le s a n d D e c is io n T r e e s 4 5 0 9 .9 M u lti-C riteria D e c is io n M a k in g w ith P a irw is e C o m p a ris o n s 4 5 3

9.1 OPENING VIGNETTE: Midwest ISO Saves Billions by Better Planning of Power Plant Operations and Capacity Planning

INTRODUCTION

M idw est ISO (M IS O ) o p e ra te s in 13 U.S. states as w e ll as th e p ro v in ce o f M an ito b a in Canada. It m a n a g e s 3 5 tran sm issio n o w n e rs a n d 100 non -tran sm issio n o w n e rs, ensuring that all m e m b e r s o f th e org a n izatio n h av e e q u a l a c c e s s to h ig h -v o ltag e p o w e r lines. T o g e th e r th e U n ited States a n d th e p ro v in ce o f M anito ba co n stitu te o n e o f th e largest energy m ark e ts in th e w o rld , w ith yearly e n erg y tran sactio n s am o u n tin g to a b o u t $23 b illion . B e f o r e M idw est ISO e x iste d , e a c h tran sm issio n co m p a n y o p e ra te d in d e p e n ­ dently. N ow , a fte r a co m p a n y jo in s M ISO , it still m ain tain s c o n tro l o f its p o w e r p lan ts and tran sm ission lin e s, and sh are s in th e resp on sibility o f su p p lyin g an d b u y in g e n erg y in a w h o le sale e le ctricity m ark e t to m e e t d em and. M ISO , h o w e v e r, h a s th e re sp o n sib ility o f deciding w h e n a n d h o w m u ch e n erg y to p ro d u ce an d ad m inister to th e m a rk e t in s u ch a

w ay as to in c re a s e b e n e fit to society.

PRESENTATION OF PROBLEM

individually, th e co m p a n ie s h a d to m a k e e x tra inv estm en ts to m a n a g e risk. T h e ir m o d e o f :o e r a tio n r e su lte d in in efficien t u se o f tran sm issio n lin e s. D ereg u latio n p o lic ie s w e re intro­ duced b y C o n g re ss an d w e re im p lem e n te d b y th e F e d e ra l E n erg y R eg u lato ry C om m ission FERC) fo r th e w h o le s a le ele ctricity industry. W h e n M ISO w a s fo rm e d , it first started

an e n e rg y -o n ly m ark et in 2 0 0 5 th at e n su re d u n b ia s e d a c c e s s to tran sm issio n lin es. In 2009, it a d d e d an cillary s erv ices (re g u la tio n an d co n tin g e n cy re se rv e s) t o its o p eratio n s. R egulation w a s su p p o s e d to e n su re th a t th e fre q u e n cy d id n o t d ev iate fro m 6 0 hertz, b o o tin g en cy re serv e s w e re su p p o s e d to h e lp en su re that in th e e v e n t o f u n e x p e c te d r o w e r loss, d em an d w as m e t w ithin 10 m inu tes o f th e p o w e r lo ss. O p e ra tio n s re sea rch s e i h o d s w e re c o n s id e re d as m e a n s to p ro v id e th e lev el o f p e rfo rm a n c e d e m a n d e d b y

dae ancillary serv ices.

METHODOLOGY/SOLUTION

Sequentially, tw o op tim izatio n algorithm s w e re u sed . T h e s e w e re th e com m itm en t i^ o r ith m a n d th e d isp atch algorithm . T h e co m m itm e n t algorithm co m m itted p o w e r ruints to b e e ith e r o n o r o ff. T h e d isp atch alg orith m d eterm in e d th e le v e l o f a p o w e r k m t s o u tp u t a n d p rice. W ith th e s e tw o algorithm s, facilities w e re g iv e n con strain ts o n b-w m u ch ele ctricity to carry w ith in th e ir p h y sical lim its in ord e r to a v o id o v e rlo a d an d m a g e to e x p e n s iv e e q u ip m en t. T h e co m m itm e n t p ro b le m fo r th e e n e rg y -o n ly m ar­ k s : m ad e u s e o f th e L agrangian re la x a tio n m eth o d . As m e n tio n e d e arlier, it d eteim in ed s - e n e a c h p la n t sh o u ld tu rn o n o r off. T h e d isp atch p ro b le m w as so lv e d w ith a lin ear

p rogram m ing m o d el. It h e lp e d d e cid e h o w m u c h ou tp u t sh o u ld b e p ro d u ce d b y e a ch p o w e r plant. It a lso h e lp e d d eterm in e th e p rice o f e n erg y b a s e d o n th e lo c a tio n o f th e p o w e r p lant. E v e n th o u g h th e s e m e th o d s w e re ju s t fin e , th e y w e re n o t a p p ro p riate to r th e an cillary serv ice m ark et co m m itm e n t p ro b lem . R ather, a m ix e d in te g er p rog ram m ing m o d e l w a s u s e d as a resu lt o f its s u p e rio r m o d e lin g capacity.

RESULTS/BENEFITS

B a s e d o n th e im p ro vem en ts m ad e, reliability o f th e tran sm issio n grid im p r o v e d A lso a d y n am ic tran sp aren t p ricing stru cture w as cre ate d . V alu e p ro p o sitio n stu d ies s h o w that M idw est ISO a ch ie v e d a b o u t $2.1 b illio n an d $3 b illio n d ollars in n e t cu m u lative savings b e tw e e n 2 0 0 7 and 2 0 1 0 . Future savin g s are e x p e c te d to a ccru e to a b o u t $6.1 b illion.

QUESTIONS FO R TH E OPENING VIGNETTE

1 . In w h at w ay s w e re th e individual c o m p a n ie s in M idw est ISO b e tte r o f f b e in g part o f M ISO as o p p o s e d to o p eratin g ind ep end en tly ?

2 . T h e d isp atch p ro b le m w as so lv e d w ith a lin e a r p rogram m ing m eth o d . E x p la in the n e e d o f s u c h m e th o d in light o f th e p ro b le m d iscu sse d in th e ca se .

3 . W h at w e re th e m o m ain op tim ization algorith m s used? B rie fly e x p la in th e u s e o f

e a c h algorithm .

LESSONS WE CAN LEARN FROM THIS VIGNETTE

O p e ra tio n s re s e a r c h (O R ) m e th o d s w e re u sed b y M idw est IS O to p rovide e fficie n t an d c h e a p e r so u rce s o f e n erg y fo r states in th e m id w estern re g io n o f th e U nited States. A co m b in a tio n o f lin e a r p ro gram m in g an d th e L agrangian re lax atio n m e th o d s w a s u se d to d eterm in e an o p tim ized a p p r o a c h to g e n e ra te a n d su p p ly p o w e r. B y e x te n s io n , this m e th o d o lo g y cou ld b e u s e d b y b o th g o v e rn m e n t a g e n cie s and th e private s e c to r to op tim ize th e c o s t a n d p ro v isio n o f s e rv ice s s u ch a s h e alth care and ed u cation .

Source: Brian Carlson. Yonghong Chen, Mingguo Hong, Roy Jon es, Kevin Larson, Xingwang Ma, Peter Nieuwesteeg, et al., “MISO Unlocks Billions in Savings Through the Application o f Operations Research tor Energy and Ancillary Services Markets,” In terfaces, Vol. 42, No. 1, 2012, pp. 58-73.

9.2 D EC ISIO N SU PPO R T S Y S T E M S M O D E L IN G M any read ily a c c e s s ib le a p p lica tio n s d e s c rib e h o w th e m o d els in co rp o ra te d in DSS co n trib u te to org an ization al su cce ss. T h e s e in clu d e P illo w te x (s e e P roM od el, 2 0 1 3 X Fiat (s e e P roM od el, 2 0 0 6 ), P ro cte r & G a m b le (s e e C am m e t al., 1 9 9 7 ), a n d o th ers. INFORMS p u b licatio n s s u ch as In terfaces, ORMS Today, a n d A n alytics m ag azin e all in clu d e s to n e s th at illustrate su cce ssfu l a p p lica tio n s o f d e c is io n m o d els in real settin gs. T h is ch ap ter in clu d es m an y e x a m p le s o f s u c h a p p licatio n s, a s d o e s th e n e x t ch ap ter.

Sim ulation m o d e ls c a n e n h a n c e a n o rg a n iz a tio n ’s d ecisio n -m ak in g p ro ce s s an d e n a b le it to s e e th e im p act o f its future c h o ic e s . F ia t (s e e ProM od el, 2 0 0 6 ) s a v e s $1 m illion annu ally in m an u factu rin g co s ts th ro u g h sim u latio n . IB M h a s p re d icted th e b e h a v io r o f th e ? 3 0 -m ile-lo n g G u ad alu p e R iv er a n d its m a n y tribu taries. T h e p red ictio n c a n b e m ad e sev eral days b e fo re th e im m in en t flo o d o f th e river. T h is is im portant as it w o u ld a llo w fo r e n o u g h tim e fo r d isaster m a n a g e m e n t an d p re p aratio n . IBM u s e d a c o m b in a tio n o f w e a th e r a n d s e n so r data to b u ild a river sy stem sim u latio n a p p lica tio n th at co u ld sim ulate th o u san d s o f river b ra n c h e s a t a tim e. B e s id e s flo o d p re d ictio n , th e a p p licatio n co u ld a lso b e u s e d fo r irrigation p lan n in g in s u c h a w a y as to av o id th e im p act o f d rou gh ts and surplu s w ater. E v e n co m p a n ie s u n d e r financial stress n e e d to in v est in s u ch so lu tio n s to

4 2 4 Part IV • Prescriptive Analytics

Chapter 9 • M odel-Based D ecision Making: Optim ization and Multi-Criteria Systems 4 2 5

s q u e e z e m o re e ffic ie n c y o u t o f th e ir lim ited re so u rce s— m a y b e e v e n m o re so. P illo w tex, a $2 b illio n co m p a n y th at m an u factu res pillow s, m attress p ad s, an d co m fo rters, h a d filed fo r b a n k ru p tcy an d n e e d e d to re o rg an ize its p lan ts to m ax im ize n e t profits fro m th e c o m p a n y ’s o p era tio n s. It e m p lo y e d a sim ulation m o d el to d ev elo p a n e w le a n m an u fac­ tu rin g e n v iro n m e n t th at w o u ld re d u ce th e co s ts an d in c re a se throu ghp u t. T h e co m p a n y e stim a ted th at th e u se o f this m o d e l resu lted in o v e r $ 1 2 m illion savin gs im m ediately. ( S e e promodel.com.) W e w ill study sim u latio n in th e n e x t ch ap ter.

M o d e lin g is a k e y e le m e n t in m ost D SS a n d a n e c e s s ity in a m o d e l-b a s e d D SS. T h e re are m a n y cla ss e s o f m o d e ls, and th e re are o fte n m an y s p e cia liz e d te c h n iq u e s fo r solving e a c h o n e . S im u lation is a co m m o n m o d e lin g ap p ro a ch , b u t th e re are sev eral others.

A p plying m o d els to real-w orld situations c a n sav e m illion s o f d ollars o r g e n era te m illion s o f d ollars in rev e n u e . C h ristian sen et al. (2 0 0 9 ) d e s c rib e th e a p p lica tio n s o f su ch m o d e ls in sh ip p in g co m p a n y o p era tio n s. T h e y d e s c rib e ap p lica tio n s o f T u rb o R o u ter, a D SS fo r sh ip routing a n d sch e d u lin g . T h e y claim that o v e r th e c o u rs e o f ju st a 3 -w e e k p erio d , a co m p a n y u s e d this m o d e l to b e tte r utilize its fle e t, g e n era tin g ad d itional profit o f $ 1 - 2 m illio n in ju st a sh o rt tim e. W e pro v id e an o th e r e x a m p le o f a m o d e l ap p licatio n in A p p lica tio n C ase 9 .1 .

Application Case 9.1 O ptim al Transport fo r ExxonM obil D ow nstream Through a DSS E x x o n M o b il, a p e tro le u m a n d natural g as com p an y , o p e ra te s in sev eral co u n trie s w o rld w id e. It p ro v id es s ev eral ran g es o f p e tro le u m p ro d u cts inclu d ing c le a n fu els, lu b rican ts, an d hig h -v alu e p ro d u cts and fe e d s to c k to sev eral cu stom ers. T h is is c o m p le te d th ro u g h a c o m p le x su p p ly ch a in b e tw e e n its refin­ e rie s a n d cu sto m ers. O n e o f th e m ain p rodu cts E x x o n M o b il tran sp orts is v acu u m g as oil (V G O ). E x x o n M o b il tran sp orts sev eral sh ip lo ad s o f v acu u m g as o il fro m E u ro p e to th e U n ited States. In a year, it is e stim ated that E x x o n M o b il tran sp orts a b o u t 6 0 - 7 0 sh ip s o f V G O a cro ss th e A tlantic O c e a n . H itherto, b o th E x x o n M o b il-m a n a g e d v e ssels and third-party v e s s e ls w e re sch e d u le d to tran sp ort V G O acro ss th e A tlan tic th ro u g h a cu m b e rso m e m anu al p ro cess. T h e w h o le p ro c e s s re q u ired th e co lla b o ra tio n o f s e v e ra l individuals acro ss th e su p p ly ch a in o rg an iza­ tion . Several cu sto m ized s p re a d sh ee ts w ith sp e cial co n strain ts, req u irem en ts, a n d e c o n o m ic trad e-offs w e re u s e d to d eterm in e th e tran sp ortation sch e d u le o f th e v e ssels. S o m e o f t h e con strain ts inclu ded :

1. C o n stan tly varying p ro d u ctio n and d em an d p ro je ctio n s

2 . M axim u m a n d m inim um in v e n to ry constraints 3 . A p o o l o f h e te ro g e n e o u s v essels (e .g ., sh ip s w ith

v arying sp e e d , c a rg o siz e )

4 . V e s s e ls th a t lo a d a n d d is c h a rg e a t m u ltip le p o rts

5- B o th E x x o n M o b il-m a n a g e d an d third-party su p ­ p lie s a n d ports

6 . C o m p le x tran sp ortation c o s t that in clu d es vari­ a b le o v e ra g e an d d em u rrag e co sts

7 . V essel siz e a n d draft limits fo r d ifferen t ports

T h e m an u al p ro c e s s co u ld n o t d eterm in e th e actu al ro u te s o f v e sse ls, th e tim ing o f e a c h v e ssel, and th e q u an tity o f V G O lo a d e d an d d ischarged . A dditionally, co n sid era tio n o f th e p ro d u ctio n an d co n s u m p tio n data at sev eral lo ca tio n s re n d e re d th e m an u al p ro c e s s b u rd e n s o m e a n d in efficien t.

M e t h o d o lo g y /S o lu tio n

A d e c is io n su p p o rt to o l that su p p o rte d s ch e d u l­ e rs in p la n n in g a n op tim al s ch e d u le fo r sh ip s to lo ad , tran sp ort, an d d isch arg e V G O to an d fro m m u ltiple lo c a tio n s w a s d ev elo p e d . T h e p ro b le m w as fo rm u late d as a m ix e d -in te g er lin e a r p rogram ­ m ing p ro b lem . T h e so lu tio n h a d to satisfy req u ire­ m e n ts fo r routing, tran sp ortation , sch e d u lin g , an d inventory m an a g e m e n t vis-a-vis v arying p ro d u ctio n a n d d e m a n d p ro files. A m ath em atical p ro gram ­ m ing lan g u a g e , GAMS, w as u se d fo r th e p ro b le m fo rm u latio n a n d M icroso ft E x c e l w as u s e d a s th e

( Continued.)

4 2 6 Part IV • Prescriptive Analytics

Application Case 9.1 (Continued) u s e r in te rfa ce . W h e n th e so lv e r (IL O G CPLEX) is run, a n o p tim al so lu tio n is re a ch e d a t a p o in t w h e n th e o b je c tiv e v a lu e o f th e in cu m b e n t so lu tio n stop s im proving. T h is sto p p in g crite rio n is d eterm in ed b y th e u s e r d u rin g e a c h p ro gram run.

R e s u l t s /B e n e f i t s

It w a s e x p e c te d that u sin g th e o p tim ization m o d el w ill le a d t o re d u ce d sh ip p in g c o s t an d less d em u r­ rag e e x p e n s e s . T h e s e w o u ld b e a ch ie v e d b e c a u s e th e to o l w o u ld b e a b le t o su p p o rt h ig h e r utilization o f sh ip s a n d h e lp m a k e s h ip s e le c tio n (e .g ., P an am a x versu s A fram ax) and d esig n m o re op tim al routing sch e d u le s. T h e re sea rch ers e x p e c te d to e x te n d th e re s e a r c h b y e x p lo rin g o th e r altern ate m ath em atical m e th o d s to s o lv e th e sch e d u lin g p ro b lem . T h e y a lso

in te n d e d to give th e D SS to o l th e cap ab ility to c o n ­ sid e r m u ltiple p ro d u cts fo r a p o o l o f v essels.

D iscussion Q uestions 1. List th ree w a y s in w h ic h m an u al s ch e d u lin g o f

sh ip s c o u ld re su lt in m o re o p era tio n a l c o s t as co m p a re d to th e to o l d ev elo p e d .

2 . In w h a t o th e r w ays c a n E x x o n M o b il le v e ra g e the d e c is io n s u p p o rt to o l d e v e lo p e d to e x p a n d and op tim ize th e ir o th e r b u sin e ss operation s?

3. W h a t are s o m e strateg ic d ecisio n s that co u ld b e m ad e b y d e c is io n m ak ers u sin g th e to o l d ev elo p e d .

Source: K. C. Furman, J. H. Song, G. R. Kocis, M. K. McDonald, and P. H. Warrick, '‘Feedstock Routing in the ExxonMobil Downstream Sector,” In terfaces, Vol. 41, No. 2, 2011, pp. 149-163.

C u rre n t M od e lin g Issues W e n e x t d iscu ss so m e m a jo r m o d elin g issues, s u c h as p ro b le m id entificatio n a n d envi­ ro nm ental analysis, variable id entification, fo recastin g , th e u s e o f m ultiple m od els, m o d el ca te g o rie s (o r ap p rop riate s e le c tio n ), m o d e l m an ag em en t, and k n o w le d g e-b a se d m od eling .

IDENTIFICATION OF THE PROBLEM AND ENVIRONMENTAL ANALYSIS O n e very im por­ tan t a s p e c t o f it is environmental scanning and analysis, w h ich is th e m onitoring, scan n in g , a n d in terp retation o f c o lle c te d in fo rm atio n . N o d e cisio n is m ad e m a vacu um . It is im portant to an aly ze th e s c o p e o f th e d o m a in an d th e fo rc e s a n d d y n am ics o f th e en v iron m en t. A d e c is io n m a k e r n e e d s to id en tify th e o rgan ization al cu ltu re and th e c o rp o ra te d ecisio n -m a k in g p ro c e s s e s (e .g ., w h o m a k e s d ecisio n s, d e g re e o f cen traliza­ tio n ). It is en tirely p o ss ib le th at en v iro n m en tal fa cto rs h av e c re a te d th e cu rre n t p ro blem . B i/ b u sin ess an aly tics (B A ) to o ls c a n h e lp id entify p ro b lem s b y sca n n in g fo r them . T h e p ro b le m m u st b e u n d e rsto o d an d e v e ry o n e in v o lv e d sh o u ld sh a re th e sam e fram e o f u n d erstand ing, b e c a u s e th e p ro b le m w ill u ltim ately b e re p re s e n te d b y th e m o d e l in o n e fo rm o r an o th er. O th e rw ise, th e m o d e l w ill n o t h e lp th e d e c is io n m aker.

VARIABLE IDENTIFICATION Identification o f a m o d e l’s variables (e .g ., d ecisio n , result, u n co n tro llab le ) is critical, as are th e relationsh ips am o n g th e variables. In flu e n ce diagram s, w h ich a re graphical m o d els o f m ath em atical m o d els, c a n facilitate the identification pro cess. A m o re g en eral fo rm o f an in flu en ce diagram , a co g n itiv e m ap , c a n h e lp a d ecisio n m aker d ev elo p a b etter u nderstand ing o f a p ro blem , e sp ecially o f v ariab les a n d th eir interactions.

FORECASTING (PREDICTIVE ANALYTICS) Forecasting is p red ictin g th e future. T h is form o f p red ictive an aly tics is e sse n tia l fo r co n stru ctio n a n d m an ip u latin g m o d e ls, b e c a u s e w h e n a d e c is io n is im p lem e n te d th e results u su ally o c c u r in th e future. W h e re a s D SS are typ ically d esig n e d to d eterm in e w h a t w ill b e , trad itional MIS rep o rt w h at is o r w h a t w as. T h e r e is n o p o in t in ru n n in g a w h a t-if (sen sitiv ity ) analysis o n th e past, b e c a u s e d ecisio n s m a d e th e n h av e n o im p act o n th e fu ture. F o re ca stin g is g etting e a s ie r a s softw are v e n d o rs au to m ate m a n y o f t h e co m p lica tio n s o f d e v e lo p in g s u ch m od els.

Chapter 9 • M odel-Based D ecisio n Making: O ptim ization and Multi-Criteria Systems 4 2 7

E -c o m m e rc e h a s cre a te d a n im m e n se n e e d fo r fo re ca stin g a n d a n a b u n d a n ce o f a v ailab le in form ation fo r p erfo rm in g it. E -co m m e rce activities o c c u r q u ick ly , y e t in form ation a b o u t p u rch a se s is g ath e re d an d sh o u ld b e an aly zed to p ro d u ce fo recasts. P art o f th e a n aly sis in v olv es sim ply p red ictin g d em an d ; h o w e v e r, fo re ca s tin g m o d e ls c a n u s e p ro d u ct life -cy cle n e e d s a n d in form ation a b o u t th e m a rk e tp la ce a n d co n su m e rs to an aly ze th e e n tire situ ation , id eally lead in g to ad d itional sa le s o f p ro d u cts a n d serv ices.

M any organizations have accurately pred icted d em and fo r products and services, using a variety o f qualitative an d quantitative m ethods. B u t until recently, m ost co m p an ies view ed their cu stom ers and potential cu stom ers b y categorizing th em into o n ly a few , tim e-tested groupings. T od ay, it is critical n o t only to co n sid er cu stom er characteristics, b u t also to co n sid er h o w to g e t th e right produ ct(s) to th e right cu stom ers at th e right p rice at th e right tim e in th e right format/packaging. T h e m o re accu rately a firm d o e s this, th e m o re profit­ a b le th e firm is. In addition, a firm n eed s to reco g n ize w h en n o t to sell a particular product o r b u n d le o f products to a particular set o f custom ers. Part o f this effort involves identify­ ing lifelong cu stom er profitability. T h e se cu stom er relationship m an ag em en t (CRM) system and rev en u e m anagem ent system (RMS) ap p roach es rely heavily o n forecasting techniqu es, w h ich are typically d escribed as p red ictiv e an alytics. T h e se system s attem pt to pred ict w h o th eir b e s t (i.e ., m ost profitable) cu stom ers (an d w orst o n e s as w ell) are an d fo cu s o n identify­ ing products an d services at appropriate prices to ap p eal to them . W e d escrib e a n effective e x am p le o f su ch forecasting at Harrah’s C h ero k ee C asino and H otel in A p plication C ase 9-2.

Application Case 9.2 Forecasting/Predictive Analytics Proves to Be a G ood G am ble fo r H arrah's Cherokee Casino and Hotel

H arrah’s C h ero k ee Casino an d H otel u ses a revenue m an agem en t (RM) system to optim ize its profits. T h e system h elp s Harrah’s attain a n average 9 8 .6 p ercen t o ccu p a n cy rate 7 days a w e e k all year, w ith the e x c e p ­ tion o f D e ce m b e r, a n d a 6 0 p erce n t gross revenu e profit margin. O n e asp ect o f the RM system is providing its cu stom ers w ith T otal Rew ards cards, w h ich track h ow m u ch m o n e y e a c h cu stom er gam bles. T h e system also tracks reservations a n d overbookings, w ith the e x c e p ­ tion o f th o se m ad e through third parties su ch as travel a gencies. T h e RM system calculates th e opportunity co st o f saving room s fo r p o ssible custom ers w h o gam ­ b le m o re th an others, b e ca u se gam bling is Harrah’s m ain so u rce o f revenue. U nlike the traditional m ethod o f co m p a n y em p lo y e es only tracking th e “b ig sp end ­ ers,” th e RM system also tracks the “mid-tier” spend­ ers. T h is has h elp ed increase th e com p an y’s profits. O nly cu stom ers w h o g am b le o v er a certain dollar am ount are recom m en d ed b y the RM system to b e given ro o m s at the hotel; tho se w h o sp en d less may b e given com plim entary room s at n earby hotels in ord er to k e ep the b igger spenders c lo s e by. T h e RM system a lso tracks w h ich gam ing m achines are m ost p o p u lar s o that m anagem ent c a n p lace th em strate­ gically throughout th e casin o in ord er to en cou rag e cu stom ers to g am b le m o re m oney. Additionally, the

system help s track the su ccess o f different m arketing projects an d incentives.

T h e ca sin o collects dem and data, w h ich are then u se d b y a forecasting algorithm w ith several co m p o ­ nents: sm oo th ed values fo r b a se dem and, dem and trends, annu al and d ay-of-the-w eek seasonality, and sp ecial event factors. T h e forecasts are u sed b y over­ b o o k in g and optim ization m odels fo r inventory-control recom m endations. T h e b o o k in g recom m end ation sys­ tem includes a linear program (to b e introduced later in th e chapter). T h e m o d el up dates th e recom m end a­ tions fo r b o o k in g a ro om periodically o r w h e n certain events d em and it. T h e bid-price m odel is updated or optim ized after 2 4 hours hav e p assed sin ce the last optim ization, w h e n five room s hav e b e e n b o o k e d sin ce the last optim ization, o r w h en th e RM analyst m anually starts a n e w optim ization. T h e m odel is a g o o d ex a m p le o f th e p ro cess o f forecasting dem and and then using this inform ation to em p lo y a m odel- b a se d DSS fo r m aking optimal decisions.

Source: Based on R. Metters, C. Queenan, M. Ferguson, L. Harrison, J. Higbie, S. Ward, B. Barfield, T. Farley, H. A. Kuyumcu, and A. Duggasani, “The ‘Killer Application’ o f Revenue Management: Harrah’s Cherokee Casino & Hotel,” In terfaces, Vol. 38, No. 3, May/june 2008, pp. 161-175-

4 2 8 Part IV • Prescriptive Analytics

MODEL CATEGORIES T a b le 9.1 classifies D SS m o d e ls into se v e n g ro u p s an d lists several rep resen tativ e te ch n iq u es fo r e a c h categ ory. E a c h te ch n iq u e c a n b e ap p lied to eith er a static o r a dynamic m odel, w h ich c a n b e co n stru cte d u n d e r assu m ed en viron m en ts o f certainty, uncertainty, o r risk. T o e x p e d ite m o d e l con stru ction , w e ca n u s e sp e cia l d ecisio n analysis system s that h av e m o d elin g lan gu ages an d cap ab ilities e m b e d d ed in them . T h e s e in clu d e sp read sh eets, data m in in g sy stem s, OLAP system s, and m o d elin g lan g u ages that h e lp a n analyst build a m o d el. W e will introd u ce o n e o f th e s e system s later in th e chapter.

MODEL MANAGEMENT M odels, lik e data, m ust b e m an ag ed to m aintain th eir integrity, and thus th eir applicability. S u ch m an ag e m e n t is d o n e w ith th e aid o f m o d el b a se m an ag e­ m e n t system s (M BM S), w h ich a re a n a lo g o u s to d a ta b a se m an ag e m e n t sy stem s (D B M S).

KNOWLEDGE-BASED MODELING D SS u s e s m o stly quantitative m o d e ls, w h e re a s e x p e rt system s u s e qualitative, k n o w le d g e -b a s e d m o d e ls in th eir ap p licatio n s. S o m e k n o w le d g e is n e c e s s a ry to co n stru ct s o lv a b le (a n d th e re fo re u s a b le ) m o d e ls. M any o f th e p red ictive an aly tics te ch n iq u e s s u ch as classificatio n , clu sterin g, and s o o n c a n b e u se d in build ing k n o w le d g e -b a s e d m o d els. As d escrib e d , s u ch m o d e ls ca n a lso b e b u ilt fro m an alysis o f e x p e rtis e a n d in co rp o ra tio n o f s u ch e x p e rtis e in m o d els.

CURRENT TRENDS IN MODELING O n e r e c e n t tre n d in m o d e lin g involves th e d ev elo p ­ m e n t o f m o d e l lib raries an d so lu tio n te ch n iq u e lib raries. S o m e o f th e se c o d e s c a n b e run d irectly o n th e o w n e r’s W e b serv e r fo r fre e , a n d o th ers c a n b e d o w n lo ad e d a n d ru n on a lo c a l com p u ter. T h e availability o f th e s e c o d e s m e a n s that p o w erfu l op tim izatio n an d sim u latio n p a ck a g e s are av ailab le to d e cisio n m a k e rs w h o m ay h av e o n ly e x p e rie n c e d th e s e to o ls fro m th e p e rsp e ctiv e o f cla ssro o m p ro b lem s. F o r e x a m p le , th e M athem atics an d C o m p u ter S c ie n c e D ivision at A rg on n e N ational L aboratory (A rg o n n e, Illin o is) m ain­ tains th e N EOS Se rv e r fo r O p tim izatio n at n eo s.m cs.an l.g o v /n eo s/in d ex.h tm l. Y o u c a n find lin k s to o th e r sites b y click in g th e R e so u rce s lin k at inform s.org, th e W e b site o f th e Institute fo r O p e ratio n s R e se arch and th e M an ag e m e n t S c ie n c e s (IN FO R M S). A w ealth

TABLE 9.1 C a t e g o r i e s o f M o d e ls

Category Process and O bjective Representative Techniques

O p tim izatio n o f problem s w ith f e w alternatives

O ptim ization via algorithm

O p tim izatio n via an analytic form u la

Sim ulation

Heuristics

Predictive m odels

O th e r m odels

Find th e best solution fro m a small n u m b e r o f alternatives

Find th e best solution fro m a large nu m b er o f alternatives, using a step-by-step im p ro ve m e n t process

Find th e best solution in o n e step, using a fo rm u la

Find a g o o d e n o u g h so lu tion or th e best a m o n g th e alternatives che ck e d , using ex p erim en tatio n

Find a g o o d e n o u g h solution, using rules

Pred ict th e fu tu re fo r a given scenario

So lv e a w h at-if case, using a fo rm u la

Decision tables, decision trees, analytic hierarchy process

Linear a n d o th e r m athem atical program m ing m odels, n e tw o rk m odels

S o m e in ven tory m odels

Several type s o f sim ulation

Heuristic p rogram m ing, expert system s

Forecasting m odels, M a rk o v analysis

Financial m o d eling , w a itin g lines

C hapter 9 • M odel-Based D ecision Making: Optim ization and Multi-Criteria Systems 4 2 9

o f m o d e lin g a n d so lu tio n inform ation is a v ailab le fro m INFORMS. T h e W e b site fo r o n e o f IN FO R M S’ p u b licatio n s, OR/MS Today, a t lionhrtpub.com /O R M S.shtm l in clu d es links to m any c a te g o rie s o f m o d e lin g so ftw are. W e w ill le a rn a b o u t s o m e o f th e s e shortly.

T h e r e is a c le a r trend tow ard d ev elo p in g a n d u sin g W e b to o ls an d softw are to a cce ss an d e v e n ru n softw are to p erfo rm m o d elin g , op tim ization, sim ulation, a n d s o on . This h as, in m an y w ays, sim plified th e a p p licatio n o f m any m o d els to real-w o rld p ro blem s. H o w ev er, to u se m o d els an d so lu tio n te ch n iq u e s effectiv ely , it is n e ce s s a ry to truly gain e x p e rie n c e th ro u g h d ev elo p in g and solving sim p le o n e s. T h is a s p e c t is o ften o v erlo o k ed . A n o th er tren d , unfortunately, involves th e la ck o f understand ing o f w h a t m o d els an d their so lu tion s c a n d o in th e real w orld. O rg anizatio ns that h av e k e y analysts w h o understand h o w t o ap p ly m o d els in d e e d a p p ly th e m very effectively . 1’his is m o st n o ta b ly o ccu rrin g in th e re v e n u e m a n a g e m e n t area, w h ich h a s m o v e d fro m th e p ro v in ce o f airlines, h o tels, and au to m o b ile rental to retail, in su ran ce, en tertainm ent, a n d m an y o th e r areas. CRM a lso uses m o d e ls, b u t th ey are o fte n tran sp arent to th e user. W ith m an ag e m e n t m o d e ls, th e am o u nt o f data a n d m o d el sizes are q u ite large, n ecessitatin g th e u s e o f data w a re h o u se s to supp ly th e data an d parallel com p u tin g hard w are to o b ta in so lu tio n s in a re a s o n a b le tim e fram e.

T h e r e is a co n tin u in g trend tow ard m a k in g analytics m o d e ls c o m p le te ly tran sp arent to th e d e c is io n m ak er. F o r e x a m p le , m ultidim ensional analysis (m odeling) involves d ata an alysis in sev eral d im en sio n s. In m u ltid im en sion al analysis (m o d e lin g ) an d so m e o th e r c a s e s , d ata are g e n era lly sh o w n in a sp re a d sh e e t form at, w ith w h ic h m o st d e cisio n m ak ers a re fam iliar. M any d e cisio n m ak ers accu sto m e d to slicin g an d d icin g data c u b e s are n o w u sin g OLAP system s th at a c c e s s d ata w a reh o u se s. A lthough th e s e m eth o d s m ay m ak e m o d e lin g p alatab le, th e y a lso elim in ate m a n y im portant an d a p p lica b le m o d el cla ss e s fro m co n sid era tio n , an d th e y elim in ate s o m e im p ortan t a n d su b tle so lu tio n in ter­ p re tatio n a sp e cts. M o d elin g involves m u ch m o re th an ju st data a n aly sis w ith tre n d lin es an d e stab lish in g relatio n sh ip s w ith statistical m ethod s.

T h e re is also a trend to build a m o d e l o f a m o d el to h elp in its analysis. An influence diagram is a graphical representation o f a m odel; that is, it is a m odel o f a m odel. S o m e influ­ e n c e diagram softw are p ackages are cap ab le o f generating and solving th e resultant m odel.

SECTION 9 2 REVIEW QUESTIONS

1 . List th re e le s s o n s le a rn e d fro m m od eling . 2 . List an d d e s c rib e th e m a jo r issu es in m o d elin g. 3 . W h a t are th e m a jo r typ es o f m o d els u s e d in DSS? 4 . W h y are m o d e ls n o t u s e d in industry as fre q u e n tly a s th e y s h o u ld o r co u ld be? 5 . W h a t a re th e cu rren t trend s in m odeling?

9.3 STRUCTURE OF M A T H E M A T IC A L M O D E L S FO R D EC ISIO N SU PPO R T

In th e follow ing sectio n s, w e presen t th e topics o f analytical m athem atical m o d els (e.g., math­ em atical, financial, engineering). T h e se include th e com p on en ts and th e structure o f m odels.

The C o m p o n e n ts o f D e cisio n S u p p o rt M ath em atical M odels All quantitative m odels a re typically m a d e up o f fo u r b a sic c o m p o n e n ts ( s e e Figu re 9 .1 ): result ( o r o u tc o m e ) v ariab les, d e c is io n v ariab le s, u n co n tro lla b le v ariab les (and/or p aram ­ e te rs), a n d interm ed iate resu lt v ariab les. M athem atical re la tio n sh ip s link th e s e c o m p o ­ n e n ts to g eth e r. In n on -qu an titativ e m o d e ls, th e re latio n sh ip s are s y m b o lic o r qualitative. T h e resu lts o f d ecisio n s are d eterm in ed b a s e d o n th e d e c is io n m a d e (i.e ., th e v a lu e s o f th e d e c is io n v ariab le s), th e fa cto rs th at ca n n o t b e co n tro lle d b y th e d e c is io n m a k e r ( in th e

4 3 0 Part IV • Prescriptive Analytics

FIGURE 9.1 T h e G e n eral Stru ctu re o f a Q u a n tita tiv e M odel.

en v iro n m en t), a n d th e re latio n sh ip s am o n g th e v ariab les. T h e m o d e lin g p ro ce s s involves id entifying th e v a ria b le s a n d re latio n sh ip s am o n g them . Solving a m o d el d eterm in es the v a lu e s o f th e se a n d th e result v aria b le (s).

RESULT (OUTCOME) VARIABLES Result (outcom e) variables reflect th e level o f effectiveness o f a system ; that is, they indicate h o w w ell th e system perform s o r attains its goal(s). T h ese variables are outputs. E xam ples o f result variables are show n in T a b le 9-2. Result variables are consid ered d ep en d en t variables. Interm ediate result variables are som etim es u sed in m odeling to identify interm ediate ou tcom es. In the ca se o f a d ep en d en t variable, another event must o ccu r first b efo re th e event described b y th e variable can occu r. Result variables d ep en d o n th e o ccu rren ce o f th e d ecisio n variables an d th e un controllable variables.

DECISION VARIABLES D ecision variables d e s c rib e alternativ e co u rs e s o f a ctio n . T h e d e c is io n m a k e r co n tro ls th e d e cisio n v ariab les. F o r e x a m p le , fo r a n in v estm en t p ro b lem , th e a m o u n t to in v est in b o n d s is a d e c is io n v aria b le . In a s ch e d u lin g p ro b lem , the d e ci­ s io n v a ria b le s are p e o p le , tim e s, a n d sch e d u le s . O th e r e x a m p le s are listed in T a b le 9.2.

T A B L E 9.2 Examples o f th e Com ponents o f M odels

Area Decision Variables Result Variables Uncontrollable V ariables

and Param eters

Financial investm ent In vestm ent alternatives and To tal profit, risk Inflation rate

am o u n ts Rate o f return on inv e stm e n t (RO I)

Earnings per share Liquidity level

Prim e rate C o m p e titio n

M a rk e tin g A dvertisin g b u d g e t M a rk e t share C u sto m e r's incom e

W h e r e to advertise C u sto m e r satisfaction C o m p e tito r's actions

M a n u fa c tu rin g W h a t a n d h o w m u ch to Total cost M a c h in e cap acity

produce Q u ality level T e ch n olo g y

In ve n to ry levels C o m p en satio n program s

Em p lo yee satisfaction M a te rials prices

A c c o u n tin g Use o f com p u ters D ata processing co s t C o m p u te r te c h n o lo g y

A u d it sch edule Error rate Tax rates Legal requirem en ts

T ran sp o rtation Sh ip m e n ts schedule Total tra n s p o rt cost D elivery distan ce

U se o f sm art cards P a y m e n t flo a t tim e Regulations

Services S taffin g levels C u sto m e r satisfaction D e m an d fo r services

UNCONTROLLABLE VARIABLES, OR PARAMETERS In an y d ecisio n-m ak ing situation, there are factors that affect the result v ariables b u t a re n ot u n d er th e co n tro l o f th e d ecisio n m aker. E ith er th e se facto rs ca n b e fixed, in w h ich c a s e they are called uncontrollable variables, o r parameters, o r they c a n vary, in w h ich ca se th ey are called variables. E xam p les o f fac­ tors are th e p rim e interest rate, a city’s building co d e , tax regulations, an d utilities costs. Most o f th ese factors are un controllable b e ca u se th ey are in and d eterm ined b y elem en ts o t the system en v iron m en t in w h ich the d ecisio n m ak er w orks. S o m e o f th ese v ariab les limit the d ecisio n m ak er a n d therefore form w h at are called th e con strain ts o f th e problem .

INTERMEDIATE RESULT VARIABLES Interm ediate result variables re flect interm ed iate o u tco m e s in m ath em atical m o d els. F o r e x a m p le , in d eterm in in g m a ch in e sch e d u lin g sp o il­ a g e is a n in te rm e d iate resu lt v ariab le , an d to tal p ro fit is th e result v ariab le (i.e ., s p o ila g e is o n e d eterm in an t o f to tal p ro fit). A n o th er e x a m p le is e m p lo y e e salaries. T h is co n stitu te s a d e c is io n v a ria b le fo r m an ag e m e n t: It d eterm in e s e m p lo y e e satisfactio n (i.e . interm ed iate o u tc o m e ), w h ic h , in a i m , d eterm in e s th e p rodu ctivity le v e l (i.e ., final result).

The Stru ctu re o f M athem atical M odels T h e c o m p o n e n ts o f a quantitative m o d e l are lin k e d to g e th e r b y m a th em a tica l (a lg e b ra ic)

e x p re s s io n s — e q u a tio n s o r ineq u alities. A v e ry sim p le fin an cial m o d e l is

P - R - C

w h e re P = p rofit, R = re v e n u e , a n d C = co st. T h is e q u a tio n d escrib e s th e relationsh ip am o n g th e v ariab les. A n o th er w e ll-k n o w n fin an cial m o d el is th e s im p le presen t-v alu e c a s h flo w m o d e l, w h e re P = p re s e n t v alu e , F = a future sin gle p a y m e n t in d ollars * - in te rest rate (p e rc e n ta g e ), and n = n u m b e r o f y e a rs. W ith this m o d e l, it is p o ss ib le to d eterm in e th e p re s e n t v alu e o f a p a y m en t o f $ 1 0 0 ,0 0 0 to b e m a d e 5 y e a rs fro m today, at

a 10 p e r c e n t ( 0 .1 ) in te rest rate, as fo llow s:

P = 100,000/ (1 + 0 .1 ) 5 = 6 2 ,0 9 2

W e p re s e n t m o re in terestin g an d c o m p le x m ath em atical m o d e ls in th e fo llo w in g

s ectio n s.

SECTION 9 - 3 REVIEW QUESTIONS

1 . W h a t is a d e c is io n variable? 2 . List a n d b riefly d iscu ss the th ree m ajo r c o m p o n e n ts o f lin e a r p rogram m ing. 3 . E x p la in th e ro le o f in term ed iate result v ariab les.

9.4 C ER T A IN T Y, U N C ER TA IN TY, A N D R IS K 1 P art o f S im o n ’s d ecisio n -m a k in g p ro ce s s d e s crib e d in C h ap ter 2 in v o lv e s evalu atin g and co m p a rin g alternativ es; during this p ro c e s s , it is n e c e s s a ry to p re d ict th e future o u tco m e o f e a c h p ro p o s e d alternativ e. D e c is io n situations a re o fte n classified o n th e b a sis o f w h at th e d e c is io n m a k e r k n o w s (o r b e lie v e s ) a b o u t th e fo re c a s te d results. W e cu stom arily cla s­ sify this k n o w le d g e in to th re e ca te g o rie s (s e e F igu re 9 -2 ), ran gin g fro m c o m p le te k n o w l­

e d g e to c o m p le te ig n o ran ce :

• C ertainty • Risk • U n certain ty

Chapter 9 • M odel-Based D ecisio n Making: O ptim ization and Multi-Criteria Systems 4 3 1

'Some parts o f the original versions of these sections were adapted from Turban and Meredith (1994).

4 3 2 Part IV • Prescriptive Analytics

Increasing knowledge

Decreasing knowledge

F IG U R E 9 .2 T h e Z o n e s o f D ecisio n M akin g .

W h e n w e d ev elo p m o d e ls, an y o f th e s e c o n d itio n s ca n o ccu r, an d d ifferent kinds o f m o d e ls are ap p rop riate fo r e a c h ca s e . N ext, w e d iscu ss b o th th e b a sic d efin itio n s o f th e se term s a n d s o m e im portant m o d e lin g issu es fo r e a c h co n d itio n .

D e cisio n M ak ing Under C e rta in ty In d ecisio n m aking under certainty, it is assu m ed that co m p le te k n ow led g e is available s o that the d ecisio n m aker k n o w s exactly w h at th e ou tco m e o f ea c h cou rse o f a ctio n w ill b e (a s in a determ inistic environm ent). It m ay n o t b e true that th e ou tcom es are 100 p ercen t k now n, n o r is it n ecessary to really evaluate a ll th e o u tcom es, b u t o ften this assum ption sim­ plifies the m o d el an d m akes it tractable. T h e d ecisio n m aker is view ed a s a p erfect predictor o f the future b e c a u se it is assum ed th at there is o n ly o n e ou tco m e fo r e a ch alternative, t o r exam p le, th e alternative o f investing in U.S. Treasury bills is o n e fo r w h ich there is com p lete availability o f inform ation ab ou t th e future return o n th e investm ent if it is h eld to matunty. \ situation involving d ecisio n m aking u n d er certainty o ccu rs m o st o ften w ith structured prob­ lem s w ith short tim e horizons (u p to 1 year). Certainty m o d els are relatively e a sy to d evelop and solve, and they ca n y ield optim al solutions. Many financial m o d els are constructed un der assum ed certainty, e v e n th o u gh the m arket is anything b u t 100 p ercen t certain.

D ecision M aking Under U ncertainty In d ecisio n m aking under uncertainty, th e d ecisio n m ak er consid ers situations in w h ich several ou tcom es are possible for e a c h course o f action. In contrast to th e risk situation, in this case, th e d ecisio n m aker d o es n o t kn ow , o r ca n n o t estim ate, th e probability o f o ccu rren ce o f the possible ou tcom es. D ecisio n m aking u n der uncertainty is m ore difficult than decision m aking u n der certainty b ecau se there is insufficient information. M odeling o f s u ch situations involves assessm ent o f the d ecision m ak er’s (o r th e organization’s) attitude tow ard risk.

M anagers attem pt to avoid uncertainty a s m u ch as p o ssible, ev en to the p o in t o l assum ­ ing it aw ay. Instead o f d ealing w ith uncertainty, they attem pt to ob tain m o re information^so that th e p ro b lem c a n b e treated u n d er certainty (b e c a u s e it c a n b e “almost-’ certain) o r under calculated (i.e., assum ed ) risk. I f m o re inform ation is n ot available, th e p ro b lem m ust b e treated u n d er a cond ition o f uncertainty, w h ich is less definitive th an th e oth er categories.

D ecision M akin g U nder R isk (R isk A n a ly sis) A d e c is io n m a d e u n d e r r is k 2 (a ls o k n o w n a s a p r o b a b ilis tic o r s to c h a s tic d e c is io n ­ m a k in g s itu a tio n ) is o n e in w h ic h th e d e c is io n m a k e r m u s t c o n s id e r se v e r a l p o s ­ s ib le o u tc o m e s fo r e a c h a lte r n a tiv e , e a c h w ith a g iv e n p r o b a b ility o f o c c u r r e n c e .

2Our definitions o f the terms risk and u n certain ty w ere formulated by F. H. Knight o f the University o f Chicago in 1933. Other, comparable definitions also are in use.

Chapter 9 • M od el-Based D ecision Making: O ptim ization and Multi-Criteria System s 4 3 3

T h e lo n g -r u n p r o b a b ilitie s th a t th e g iv e n o u tc o m e s w ill o c c u r a re a s s u m e d to b e k n o w n o r c a n b e e s tim a te d . U n d e r th e s e a s s u m p tio n s , th e d e c is io n m a k e r c a n a s s e s s th e d e g r e e o f ris k a s s o c ia te d w ith e a c h a lte r n a tiv e (c a lle d c a lc u la te d ris k ). M ost m a jo r b u s in e s s d e c is io n s a r e m a d e u n d e r a s s u m e d risk. R isk an aly sis ( i .e ., c a lc u ­ la te d ris k ) is a d e c is io n -m a k in g m e th o d th a t a n a ly z e s th e ris k ( b a s e d o n a s s u m e d k n o w n p r o b a b ilit ie s ) a s s o c ia te d w ith d iffe r e n t a lte r n a tiv e s . R isk a n a ly s is c a n b e p e r ­ fo rm e d b y c a lc u la tin g th e e x p e c t e d v a lu e o f e a c h a lte r n a tiv e an d s e le c tin g th e o n e w ith th e b e s t e x p e c t e d v a lu e . A p p lic a tio n C a s e 9 -3 illu s tra te s o n e a p p lic a t io n to r e d u c e u n c e rta in ty .

Application Case 9.3 Am erican A irlin es Uses Should-Cost M od elin g to Assess the U n certain ty o f Bids fo r Shipm ent Routes

I n t r o d u c t i o n

A m erican A irlines, In c. (AA) is o n e o f th e w o rld ’s largest airlin es. Its c o r e b u s in e s s is p a s s e n g e r tran s­ p o rtatio n b u t it h a s o th e r vital an cillary fu n ctio n s that in clu d e fu ll-tm ck lo a d (F T L ) freigh t sh ip m en t o f m a in te n a n c e e q u ip m e n t a n d in -flig ht sh ip m e n t o f p a s s e n g e r serv ice item s th at co u ld add up to o v e r $1 b illio n in in v e n to ry a t an y g iv en tim e. AA re c e iv e s n u m e ro u s b id s fro m s u p p liers in re sp o n se to re q u e st fo r q u o te s (R F Q s) fo r in v e n to rie s. AA’s R FQ s co u ld to ta l o v e r 5 0 0 in an y g iv e n year. B id q u o te s vary sig n ifican tly as a resu lt o f th e large n u m b e r o f b id s and resu ltan t c o m p le x b id d in g p ro ­ c e s s . S o m etim e s, a sin g le co n tra ct b id c o u ld dev iate b y a b o u t 2 0 0 p e rc e n t. As a resu lt o f th e c o m p le x p ro c e s s , it is co m m o n to e ith e r o v erp a y o r u n d er­ p a y s u p p liers fo r th e ir serv ice s. T o this end , AA w a n te d a s h o u ld -c o s t m o d e l th a t w o u ld stream lin e a n d a s s e s s b id q u o te s fro m su p p liers in o rd e r to c h o o s e b id q u o te s th at w e re fair to b o th th e m and th e ir su p p liers.

M e th o d o lo g y /S o lu tio n

In o rd e r to d eterm in e fair c o s t fo r su p p lier p rodu cts an d serv ice s, th ree ste p s w e re taken :

1 . Prim ary (e .g ., in terview s) a n d s e co n d a ry (e .g .. In te rn e t) s o u rc e s w e r e sco u te d fo r b a s e -c a s e and ran g e d ata that w o u ld inform c o s t v ariab les th at a ffect a n FTL bid.

2 . C o st v a ria b le s w e re c h o s e n s o th a t th e y w e re m utually e xclu siv e a n d co lle ctiv e ly exh au stiv e.

3 . T h e D PL d e c is io n an aly sis so ftw are w a s u s e d to m o d e l th e uncertainty.

F u rth erm ore, E x ten d e d Sw an so n-M egill (ESM ) a p p ro x im a tio n w a s u s e d to m o d e l th e p ro bab ility distribution o f th e m o st sen sitiv e c o s t v a ria b le s used . T h is w a s d o n e in o rd e r to a c c o u n t fo r th e high vari­ ab ility in th e b id s in th e initial m o d el.

R e s u l t s /B e n e f i t s

A p ilo t te s t w a s d o n e o n a n R FQ th a t attracted b id s from s ix FTL carriers. O u t o f th e s ix b id s pre­ se n te d , five w e re w ithin th re e stand ard d eviation s fro m th e m e a n w h ile o n e w a s co n sid e re d a n o u t­ lier. S u b seq u en tly , AA u se d th e sh o u ld -co st FTL m o d e l o n m o re th a n 2 0 R FQ s to d eterm in e w h a t a fair a n d a ccu ra te c o s t o f g o o d s and s erv ices sh ou ld b e . It is e x p e c te d th at this m o d e l w ill h e lp in re d u c­ ing th e risk o f e ith e r o v erp ay in g o r u n d erp ay in g its supp liers.

Q uestions f o r D iscussion 1. B e s id e s re d u cin g th e risk o f ov erp ay in g o r u n d er­

p ay in g su p p liers, w h a t are s o m e o th er b en e fits AA w o u ld d eriv e fro m its “sh ou ld b e ” m odel?

2. C an y o u th in k o f o th e r d om ain s b e s id e s air trans­ p o rtatio n w h e r e s u c h a m o d e l c o u ld b e used?

3. D iscu ss Other p o ss ib le m e th o d s w ith w h ich AA c o u ld h a v e so lv e d its b id ov erp ay m e n t and u n d e rp ay m en t p ro b lem .

Source: M. J . Bailey, J . Snapp, S. Yetur, J. S. Stonebraker, S. A. Edwards, A. Davis, and R. Cox, “Practice Summaries: American Airlines Uses Should-Cost Modeling to Assess the Uncertainty of Bids for Its Full-Truckload Shipment Routes,” In terfaces, Vol. 41, No. 2, 2011, pp. 194-196.

4 3 4 Part IV • Prescriptive Analytics

SECTION 9 . 4 REVIEW QUESTIONS

1 . D e fin e w h at it m e a n s to p e rfo rm d e c is io n m a k in g u n d e r assu m ed certain ty, risk, and

uncertainty. 2 . H o w ca n d ecisio n -m a k in g p ro b le m s u n d er a ssu m e d certain ty b e handled? 3 . H ow c a n d ecisio n -m a k in g p ro b lem s u n d er a ssu m e d u n certain ty b e handled? 4 . H o w c a n d ecisio n -m a k in g p ro b lem s u n d e r assu m ed risk b e handled?

9.5 D EC ISIO N M O D E L IN G W ITH S P R E A D S H E E T S M o d els c a n b e d e v e lo p e d an d im p le m e n te d in a variety o f p ro g ram m in g lan g u ag es a n d sy stem s. T h e s e ran g e fro m third -, fo u rth -, a n d fifth -g e n e ra tio n p ro g ram m in g lan ­ g u a g e s to co m p u te r-a id e d so ftw are e n g in e e rin g (C A SE) sy stem s a n d o th e r sy stem s that a u to m atically g e n e r a te u s a b le so ftw are . W e fo c u s p rim arily o n sp rea d sh eets (w ith th eir a d d -in s), m o d e lin g la n g u a g e s, a n d tra n sp a ren t d ata a n aly sis to o ls . W ith th e ir strength a n d flex ib ility , s p re a d sh e e t p a c k a g e s w e re q u ick ly r e c o g n iz e d as e a sy -to -u se im p e- m e n ta tio n so ftw a re fo r th e d e v e lo p m e n t o f a w id e ra n g e o f ap p lica tio n s in b u sin ess, en g in e e rin g , m a th em a tics, an d s c ie n c e . S p re a d s h e e ts in clu d e e x te n s iv e statistical fo re ­ castin g . an d o th e r m o d e lin g an d d a ta b a se m a n a g e m e n t ca p a b ilitie s, fu n ctio n s, an d ro u ­ tin e s. As s p re a d sh e e t p a c k a g e s e v o lv e d , ad d -in s w e re d e v e lo p e d fo r stru ctu ring an d so lv in g s p e c ific m o d e l cla ss e s . A m o n g th e a d d -in p a ck a g e s , m a n y w e re d e v e lo p e d fo r D SS d e v e lo p m e n t. T h e s e D S S -re la ted ad d -in s in clu d e S o lv e r (F ro n tlin e Sy stem s In c solver.com ) an d W h a t'sBest! (a v e rs io n o f U n d o , fro m L ind o System s, In c ., lindo.com) fo r p erfo rm in g lin e a r a n d n o n lin e a r o p tim iz a tio n ; B ra in c e l (J u n k R e se a rc h Softw are In c ., jurikres.com ) a n d N e u ra lT o o ls (P a lisa d e C o rp ., palisade.com ) fo r artificial n eu ral n e tw o rk s; E v o lv e r (P a lisa d e C o rp .) fo r g e n e tic alg o rith m s; an d @ R ISK (P a lisa d e C o rp .) fo r p e rfo rm in g sim u latio n stud ies. C o m p a ra b le ad d -ins a re av a ila b le fo r fre e o r a t a very lo w co st. (C o n d u ct a W e b s e a rch to find th e m ; n e w o n e s are a d d e d to th e m a rk e tp la ce

o n a regu lar b a sis .) T h e s p re a d sh ee t is cle a rly th e m o st p o p u la r en d -u ser m od elin g to o l b e c a u s e it

in co rp o rate s m an y p o w erfu l fin an cial, statistical, m ath em atical, a n d o th e r fu nctions. Sp re ad sh ee ts c a n p erfo rm m o d e l so lu tio n task s s u c h a s lin e ar p ro gram m in g an d reg res sio n analysis. T h e sp re a d sh e e t h a s e v o lv e d in to a n im p ortan t to o l fo r analysis, p lanning, an d m o d e lin g (s e e Farasyn e t al., 2 0 0 8 ; H u rley a n d B a le z , 2 0 0 8 ; an d O v ch in n ik o v and M ilner, 2 0 0 8 ). A p p lication C a se 9-4 d e s crib e s a n in terestin g a p p lica tio n o f a sp re ad sh ee t-

b a s e d op tim izatio n m o d e l in a sm all b u sin ess.

Application Case 9.4 Sh o w case Scheduling a t Fred A sta ire East Sid e Dance Studio

T h e F re d A staire E a s t S id e D a n c e S tu d io in N ew Y o r k C ity p r e s e n ts tw o b a llr o o m s h o w c a s e s a y e a r. T h e s tu d io w a n te d a c h e a p , u s e r-frie n d ly , a n d q u ic k c o m p u te r p ro g ra m to c r e a te s c h e d u le s fo r its s h o w c a s e s th a t in v o lv e d h e a ts la stin g a r o u n d 7 5 s e c o n d s an d s o lo s la stin g a ro u n d 3 m in u te s . T h e p ro g ra m w a s c r e a te d u s in g a n in te ­ g e r p ro g ra m m in g o p tim iz a tio n m o d e l in V isu al

B a s ic a n d E x c e l . T h e e m p lo y e e s ju s t h a v e to e n te r th e s tu d e n ts ’ n a m e s , th e ty p e s o f d a n c e s th e stu ­ d e n ts w a n t to p a rtic ip a te in , th e te a c h e r s th e stu ­ d e n ts w a n t to d a n c e w ith , h o w m a n y tim e s th e s tu d e n ts w a n t to d o e a c h ty p e o f d a n c e , w h a t tim e s th e s tu d e n ts a re u n a v a ila b le , a n d w h a t tim e s th e t e a c h e r s a re u n a v a ila b le . T h is is e n te r e d in to an E x c e l s p r e a d s h e e t. T h e p ro g ra m th e n u s e s

Chapter 9 • M odel-Based D ecisio n M aking: Optim ization and Multi-Criteria Systems 4 3 5

g u id e lin e s p ro v id e d b y th e b u s in e s s to d e s ig n th e s c h e d u le . T h e g u id e lin e s in c lu d e a d a n c e ty p e n o t b e in g p e r fo r m e d tw ic e in a r o w i f p o s s ib le , a stu ­ d e n t p a rtic ip a tin g in e a c h q u a rte r o f th e s h o w ­ c a s e in o r d e r to k e e p h im / h er a c tiv e th r o u g h o u t, all p a rtic ip a n ts in e a c h h e a t p e rfo rm in g th e s a m e ty p e o f d a n c e (w ith a m a x im u m o f s e v e n c o u p le s p e r h e a t) , e lim in a tin g a s m a n y o n e - c o u p l e h e a ts a s p o s s ib le , e a c h s tu d e n t a n d te a c h e r o n ly b e in g s c h e d u le d o n c e p e r h e a t, a n d a llo w in g stu d e n ts a n d t e a c h e r s to d a n c e m u ltip le tim e s p e r d a n c e ty p e i f d e s ire d . A tw o -s te p h e u ris tic m e th o d w a s u s e d to h e lp m in im iz e th e n u m b e r o f o n e - c o u p le h e a ts . In th e e n d , th e p ro g ra m c u t d o w n th e tim e

th e e m p lo y e e s s p e n t c re a tin g th e s c h e d u le a n d a llo w e d f o r c h a n g e s to b e c a lc u la te d a n d m a d e q u ic k ly a s c o m p a r e d t o w h e n m a d e m a n u a lly . F o r th e s u m m e r 2 0 0 7 s h o w c a s e , th e s y s te m s c h e d u le d 5 8 3 h e a t e n tr ie s , 1 9 d a n c e ty p e s , 18 s o l o e n trie s , 2 8 s tu d e n ts , a n d 8 te a c h e r s . T h is c o m b in a tio n o f M ic ro s o ft E x c e l a n d V isu a l B a s ic e n a b le d th e s tu ­ d io to u s e a m o d e l-b a s e d d e c is io n s u p p o rt s y s te m fo r a p r o b le m th a t c o u ld b e tim e -c o n s u m in g to s o lv e .

Source: Based o n M. A. Lejeune and N. Yafcova, “Showcase Scheduling at Fred Astaire East Side Dance Studio,” In terfaces, Vol. 38, No. 3, May/June 2008, pp. 176-186.

O th e r im p o rta n t s p re a d s h e e t fe a tu re s in clu d e w h a t-if an aly sis, g o a l s e e k in g , data m a n a g e m e n t, an d p ro g ram m ab ility ( i.e ., m a c ro s ). W ith a s p re a d sh e e t, it is e a sy to c h a n g e a c e ll’s v a lu e an d im m e d iate ly s e e th e result. G o a l s e e k in g is p e rfo rm e d b y in d ica tin g a targ e t c e ll, its d esire d v a lu e , a n d a c h a n g in g ce ll. E x te n s iv e d a ta b a s e m an ­ a g e m e n t c a n b e p e rfo rm e d w ith sm all d ata s e ts, o r parts o f a d a ta b a s e c a n b e im p o rted fo r a n a ly sis (w h ic h is e sse n tia lly h o w OLAP w o rk s w ith m u ltid im en sio n al d ata c u b e s; in fa ct, m o st OLAP sy stem s h av e th e lo o k a n d fe e l o f a d v a n ce d s p re a d s h e e t so ftw are a fte r th e d ata are lo a d e d ). T e m p la te s, m a cro s, a n d o th e r to o ls e n h a n c e th e pro d u ctiv ity o f b u ild in g D SS.

M o st s p re a d sh e e t p a ck a g e s p rovide fairly sea m le ss in teg ratio n b e c a u s e th e y read and w rite co m m o n file stru ctures an d e asily in te rfa ce w ith d a ta b a se s an d o th er tools. M icroso ft E x c e l is th e m o st p o p u la r s p re a d sh e e t p a ck a g e . In Figure 9 -3 , w e s h o w a sim p le lo a n ca lcu la tio n m o d e l in w h ic h th e b o x e s o n th e s p re a d sh ee t d e s c rib e th e co n te n ts o f th e ce lls, w h ich c o n ta in form u las. A c h a n g e in th e in te rest rate in c e ll E 7 is im m ed iately re fle cte d in th e m o n th ly p ay m en t in c e ll E 13. T h e results ca n b e o b s e r v e d an d an aly zed im m ed iately. I f w e re q u ire a s p e cific m o n th ly p ay m en t, w e ca n u s e g o a l s e e k in g to d eter­ m in e a n a p p ro p ria te in terest rate o r lo a n am ount.

S ta tic o r d y n a m ic m o d e ls c a n b e b u ilt in a s p re a d s h e e t. F o r e x a m p le , th e m o n th ly lo a n c a lc u la tio n s p r e a d s h e e t s h o w n in F ig u re 9 -3 is s ta tic. A lth o u g h th e p r o b le m a ffe c ts th e b o r r o w e r o v e r tim e , th e m o d e l in d ica te s a s in g le m o n th ’s p e r fo r m a n c e , w h ic h is re p lic a te d . A d y n a m ic m o d e l, in c o n tra s t, re p re s e n ts b e h a v io r o v e r tim e. T h e lo a n c a lc u la tio n s in th e s p r e a d s h e e t s h o w n in F ig u re 9 -4 in d ic a te th e e ffe c t o f p re p a y ­ m e n t o n th e p rin c ip a l o v e r tim e . R isk a n a ly sis c a n b e in c o r p o r a te d in to s p re a d sh e e ts b y u s in g b u ilt-in ra n d o m -n u m b e r g e n e r a to rs to d e v e lo p s im u la tio n m o d e ls ( s e e th e n e x t c h a p te r).

S p re a d s h e e t ap p licatio n s fo r m o d e ls a re re p o rted regularly. W e w ill le a rn h o w to u se a s p re a d sh e e t-b a s e d op tim izatio n m o d e l in th e n e x t sectio n .

SECTION 9 5 REVIEW QUESTIONS

1 . W h a t is a sp read sh eet? 2 . W h a t is a sp re a d sh e e t add-in? H o w c a n ad d -in s h e lp in D SS c re a tio n a n d use? 3 . E x p la in w h y a s p re a d sh e e t is s o co n d u civ e to th e d e v e lo p m e n t o f D SS.

4 3 6 Part IV • Prescriptive Analytics

FIGURE

General Calibri

Conditional Format Cell Fo rm attin g ' as T ab le'’ Styles

y es , _.

u * * In s e r t* 51

So rtSt Find a Format - <2* Filter - Select -

Sim ple Loan Calculation Mode! in Excel

6 Lo an A m o u n t $ 1 5 0 ,0 0 0

7 In te re st Rate 8 .0 0 %

8 N u m be r of Y e a rs 30

9 10 N u m be r o f M onths 3 6 0

11 In te re st Rate/M onth 0 .6 7 %

12 $ 1 ,1 0 0 .6 5 ^

13 M onthly Lo an Paym ent

= E8 * 1 2

Excel S p re ad sh e e t Sta tic M odel Exam ple o f a Sim ple Loan

.3 E x c e l Sp re a d s h e e t S ta tic M odel Ex a m p le o f a S im p le Lo an C a lc u la tio n o f M o n th ly Paym en ts.

Review ViewFormulasHome 1 Insert

ieneral Calibn

Conditional Format Cell Formatting ’ as Table ’ Styles

NumberAlignment pboard

Dynamic Loan C alc ulatio n M odel w ith P repay m ent

I I U lO G .O O : y Normal \ Prep ay Totai

M on th \ Payment \ Am ount Paym ent!

Excel S p r e a d s h e e t Dynam ic M od el E xam ple o f a S im p le l e a n

A $ 1 0 0 Prepaym ent every M on th -L o an i s p a id o ffir * M on th 2 7 0

P rin cip le

1 $ 1 ,1 0 0 .6 5 $ 1 0 0 .0 0 $ 1 ,2 0 0 .6 5

2 $ 1 ,1 0 0 .6 5 $ 1 0 0 .0 0 $ 1 ,2 0 0 .6 5 3 $ 1 ,1 0 0 .6 5 $ 1 0 0 .0 0 $ 1 ,2 0 0 .6 5

4 $ 1 ,1 0 0 .6 5 $ 1 0 0 . 0 0 $ 1 ,2 0 0 .6 5 5 $ 1 ,1 0 0 .6 5 $ 1 0 0 .0 0 $ 1 ,2 0 0 .6 5

$ 1 4 9 ,7 9 9 = E 23*(1+ $E$111"D 24

$ 1 4 9 ,5 9 7 $ 1 4 9 ,3 9 4 $ 1 4 9 ,1 8 9 . ------------ 1------- - $ 1 4 3 ,9 8 3 Copy t h e C ells in How

25th ro u g h

Row 3 8 3 t o g e l 3 6 0 M o n t h s o t R ssults

'■ ' ■ ̂ »—-aa———............ ... ........... p F IG U R E 9 .4 E x ce l Sp re a d sh e e t D yn am ic M odel Ex a m p le o f a S im p le Lo a n C a lc u la tio n o f M o n th ly P aym en ts an d th e Effects

of Prepaym ent.

Loan Amount

In te r e s t Rate Number o f Y ears

Number o f M o n th s In te r e s t Rate/Month 0 - 6 7 * ^ =£7/12 j

M on th ly lo a n P ay m ent $ 1 ,1 0 0 .6 5 < -------\ =PMT ( E U .E 1 0 .E 6 .C

Chapter 9 • M odel-Based D ecisio n Making: Optim ization and Multi-Criteria Systems 4 3 7

9.6 M A T H E M A T IC A L P R O G R A M M IN G O PTIM IZA TIO N T h e b a s ic id e a o f op tim izatio n w a s in tro d u ced in C h ap ter 2. L i n e a r p r o g r a m m i n g ( L P ) is th e b e s t-k n o w n te c h n iq u e in a fam ily o f op tim izatio n to o ls c a lle d m a th em a tica l program m ing-, in LP, all relatio n sh ip s am o n g th e v ariab les are linear. It is u s e d e x te n siv e ly in D SS (s e e A p p lication C ase 9 .5 ). LP m o d e ls h av e m an y im p ortan t ap p licatio n s in p ractice. T h e s e in clu d e su p p ly ch a in m an ag e m e n t, p ro d u ct m ix d e c is io n s , routing, and so o n . S p e cia l fo rm s o f the m o d e ls c a n b e u se d fo r s p e cific a p p lica tio n s. F o r e x a m p le , A p p licatio n C ase 9 .5 d e s crib e s a sp re a d sh e e t m o d e l th at w as u s e d to cre a te a s ch e d u le fo r m e d ica l interns.

Application Case 9.5 Sp rea d sh e et M od el Helps Assign M ed ical Residents F le tc h e r A llen H e a lth C are (FA H C ) is a te a c h in g h o sp ita l th at w o rk s w ith t h e U n iv ersity o f V e rm o n t’s C o lle g e o f M e d icin e . In this p a rticu la r c a s e , FAHC e m p lo y s 15 re s id e n ts w ith h o p e s o f a d d in g 5 m o re in th e d ia g n o s tic ra d io lo g y p ro g ram . E a c h y e a r th e c h i e f ra d io lo g y re s id e n t is re q u ire d to m a k e a y e a r­ lo n g s c h e d u le fo r all o f th e re sid e n ts in rad io lo g y . T h is is a tim e -c o n s u m in g p r o c e s s to d o m an u ally b e c a u s e th e r e a re m a n y lim ita tio n s o n w h e n e a c h re s id e n t is a n d is n o t a llo w e d to w o rk . D u ring th e w e e k d a y w o rk in g h o u rs, th e re sid e n ts w o rk w ith c e rtifie d ra d io lo g ists, b u t n ig h ts, w e e k e n d s , a n d h o lid a y s a re all sta ffed b y re s id e n ts o n ly . T h e r e s id e n ts a re a ls o re q u ire d to ta k e th e “e m e rg e n c y r o ta tio n s ,” w h ich in v o lv e ta k in g c a r e o f th e ra d io l­ o g y n e e d s o f th e e m e r g e n c y ro o m , w h ic h is o fte n th e b u s ie s t o n w e e k e n d s . T h e ra d io lo g y p ro g ram is a 4 -y e a r p ro g ram , and th e re a re d iffe re n t ru les fo r th e w o r k s c h e d u le s o f th e re s id e n ts fo r e a c h y e a r th e y a re th e re . F o r e x a m p le , first- an d fo u rth - y e a r re s id e n ts c a n n o t b e o n ca ll o n h o lid a y s, s e c ­ o n d -y e a r re s id e n ts c a n n o t b e o n ca ll o r a s s ig n e d ER s h ifts d u rin g 1 3 -w e e k b lo c k s w h e n th e y are a s s ig n e d to w o rk in B o s to n , an d th ird -y e a r re sid e n ts m u st w o r k o n e E R ro ta tio n d u rin g o n ly o n e o f th e m a jo r w in te r h o lid a y s (T h a n k sg iv in g o r Christm as/ N ew Y e a r ’s ). A lso, firs t-y e a r re sid e n ts c a n n o t b e o n c a ll until a fte r Ja n u a r y 1, and fo u rth -y e a r re s i­ d en ts c a n n o t b e o n ca ll a fte r D e c e m b e r 3 1 , a n d s o o n . T h e g o a l that th e v a rio u s c h i e f re s id e n ts h a v e e a c h y e a r is to g iv e e a c h p e r s o n th e m a x im u m n u m b e r o f d ay s b e tw e e n o n -c a ll d ay s a s is p o s ­ s ib le . M anu ally, o n ly 3 d ay s b e tw e e n o n - c a ll days

w a s th e m o s t a c h i e f re s id e n t h a d b e e n a b le to a c c o m p lis h .

In o r d e r to c r e a te a m o r e e ffic ie n t m e th o d o f c re a tin g a s c h e d u le , th e c h i e f r e s id e n t w o rk e d w ith a n MS c la s s o f M B A s tu d e n ts to d e v e lo p a s p r e a d s h e e t m o d e l to c r e a te th e s c h e d u le . T o s o lv e th is m u ltip le -o b je c tiv e d e c is io n -m a k in g p r o b le m , th e c la s s u s e d a c o n s tr a in t m e th o d m a d e up o f tw o s ta g e s . T h e first s ta g e w a s to u s e th e s p r e a d s h e e t c r e a te d in E x c e l a s a c a lc u la to r a n d to n o t u s e it fo r o p tim iz in g . T h is a llo w e d th e c re a to r s “t o m e a s u re th e k e y m e tric s o f th e r e s i­ d e n ts ’ a s s ig n m e n ts , s u c h a s th e n u m b e r o f d ays w o r k e d in e a c h c a te g o r y .” T h e s e c o n d s ta g e w as a n o p tim iz a tio n m o d e l, w h ic h w a s la y e r e d o n th e c a lc u la to r s p r e a d s h e e t. A ssig n m e n t c o n s tr a in ts a n d th e o b je c t i v e w e r e a d d e d . T h e S o lv e r e n g in e in E x c e l w a s th e n in v o k e d t o fin d a f e a s ib le s o lu ­ tio n . T h e d e v e lo p e r s u s e d P re m iu m S o lv e r b y F r o n tlin e a n d th e X p r e s s MP S o lv e r e n g in e b y D a s h O p tim iz a tio n t o s o lv e th e y e a r lo n g m o d e l. F in a lly , u s in g E x c e l fu n c tio n s , th e d e v e lo p e r s c o n ­ v e rte d th e s o lu tio n fo r a y e a r lo n g s c h e d u le fro m z e r o s a n d o n e s t o a n e a s y -to -r e a d fo rm a t fo r th e re s id e n ts . In th e e n d , th e p ro g ra m c o u ld s o lv e th e p r o b le m o f a s c h e d u le w ith 3 to 4 d ay s in b e t w e e n o n c a lls in s ta n tly a n d w ith 5 d ay s in b e t w e e n o n ca lls ( w h ic h w a s n e v e r a c c o m p lis h e d m a n u a lly ).

Source: Based on A. Ovchinnikov and J. Milner, “Spreadsheet Model Helps to Assign Medical Residents at the University of Vermont’s College o f Medicine,” In terfaces, Vol. 38, N'o. 4, July/ August 2008, pp. 311-323-

4 3 8 Part IV • Prescriptive Analytics

M ath em atical P ro g ra m m in g Mathematical programming is a fam ily o f to o ls d e sig n e d to h e lp s o lv e m anagerial p ro b lem s in w h ic h th e d e c is io n m a k e r m u st a llo c a te sca rce re so u rce s am o n g com p etin g activities to o p tim ize a m e a su ra b le goal. F o r e x a m p le , th e distribution o f m a ch in e time (th e re s o u r c e ) am o n g vario u s p ro d u cts (th e activ ities) is a ty p ical a llo ca tio n p ro b lem . LP a llo ca tio n p ro b le m s usu ally display th e fo llo w in g ch aracteristics:

• A lim ited qu antity o f e c o n o m ic re so u rce s is a v ailab le fo r allo catio n . • T h e re so u rce s are u s e d in th e p ro d u ctio n o f p ro d u cts o r serv ices. • T h e r e are tw o o r m o re w ay s in w h ich th e re so u rce s c a n b e u sed . E a c h is c a lle d a

solu tion o r a p rog ram . . • E a c h activity (p ro d u ct o r se rv ice ) in w h ic h th e re so u rce s are u s e d y ield s a retu rn in

term s o f t h e stated goal. . „ , • T h e a llo ca tio n is u su ally restricted b y se v e ra l lim itations a n d req u irem en ts, calle d

con strain ts. T h e LP allo ca tio n m o d e l is b a s e d o n th e fo llo w in g rational e c o n o m ic assum p tions:

• R eturns fro m d ifferen t a llo ca tio n s c a n b e co m p a re d ; th at is, th e y ca n b e m easu red

b y a co m m o n u n it (e .g ., d ollars, utility). • T h e retu rn fro m an y a llo ca tio n is in d e p e n d e n t o f o th e r allo catio n s. • T h e to tal return is th e su m o f th e retu rns y ie ld e d b y th e d iffe ren t activities.

• All d ata are k n o w n w ith certainty. • T h e re so u rce s are to b e u s e d in th e m o st e c o n o m ic a l m ann er.

A llo catio n p ro b lem s ty p ically h a v e a la rg e n u m b e r o f p o s s i b l e solu tion s. D ep en d in g o n th e u n d erlyin g assu m p tio ns, th e n u m b e r o f so lu tio n s c a n b e e ith er infinite o r mite. U su ally d ifferen t so lu tio n s y ield d iffe ren t rew ard s. O f th e a v ailab le so lu tio n s, a t le a st o n e is th e b est, in th e s e n s e th at th e d e g re e o f g o a l attain m en t a s s o cia te d w ith it is th e h ig h est (i.e ., th e to tal rew ard is m ax im ized ). T h is is ca lle d a n o p t i m a l s o l u t i o n , a n d it c a n b e

fo u n d b y u sin g a sp e cia l algorithm .

Lin e a r P ro g ra m m in g E v ery LP p ro b le m is c o m p o s e d o f d ecisio n v a ria b les (w h o s e v a lu e s are u n k n o w n and are s e a r c h e d fo r), a n ob jectiv e fu n c tio n ( a lin e ar m ath em atical fu n ctio n th a t relates th e d e c is io n v a ria b le s to th e g o a l, m e a su re s g o a l attainm ent, an d is to b e o p tim ize ), ob jectiv e fu n c tio n c o effic ien ts (u n it p ro fit o r c o s t c o e ffic ie n ts ind icating th e con trib u tio n to th e o b je c tiv e o f o n e u n it o f a d e c is io n v a ria b le ), con strain ts (e x p re s s e d in th e fo rm o f lin e a r in eq u alities o r e q u alities that lim it re so u rce s and/or req u irem en ts; th e s e relate th e v a riab les th ro u g h lin e a r re latio n sh ip s), c a p a c ities (w h ic h d e scrib e th e u p p e r and so m e tim e s lo w e r lim its o n th e con strain ts a n d v aria b le s), an d in pu t/ou tpu t (techn ology) co effic ien ts (w h ic h in d icate re s o u rce u tilizatio n fo r a d e cisio n v ariab le ).

Let u s lo o k a t a n e x a m p le . M B I C o rp o ra tio n , w h ic h m an u factu re s sp e cia l-p u rp o se co m p u te rs, n e e d s to m a k e a d e cis io n : H o w m an y co m p u te rs sh o u ld « P ro d u ce n e x t m o n th a t th e B o s to n plant? M B I is c o n s id e rin g tw o ty p e s o f co m p u te rs: th e CC-7, w h ich re q u ires 3 0 0 d ays o f la b o r a n d $ 1 0 ,0 0 0 in m aterials, an d th e C C -8, w h ic h ^ e q m r e s 5 0 0 days o f la b o r a n d $ 1 5 ,0 0 0 in m aterials. T h e p ro fit co n trib u tio n o f e a c h CC-7 is $ 8 ,0 0 0 , w h e re a s that o f e a c h C C -8 is $ 1 2 ,0 0 0 . T h e p la n t h a s a ca p a city o f 2 0 0 ,0 0 0 w o rk in g days p e r m o n th , a n d th e m aterial b u d g e t is $8 m illio n p e r m o n th . M arketing re q u ire s th at at le a s t 1 0 0 units o f th e CC -7 a n d a t le a st 2 0 0 u n its o f th e CC -8 b e p ro d u ce d e a c h m onth. T h e p ro b le m is to m ax im ize th e c o m p a n y ’s p ro fits b y d eterm in in g h o w m an y units o f th e CC -7 a n d h o w m a n y u n its o f th e CC -8 s h o u ld b e p ro d u ce d e a c h m o n th . N ote th a t m a real-w o rld e n v iro n m en t, it c o u ld p o ssib ly ta k e m o n th s to o b ta in th e d ata in th e p ro b le m

Chapter 9 • M odel-Based D ecision Making: O ptim ization and Multi-Criteria Systems 4 3 9

state m e n t, a n d w h ile gath erin g th e d ata th e d e c is io n m a k e r w o u ld n o d o u b t u n c o v e r fa cts a b o u t h o w to stru cture th e m o d e l to b e so lv ed . W e b -b a s e d to o ls fo r g ath erin g

data ca n h e lp .

M od e lin g in LP: An E xa m p le A standard LP m odel can b e developed fo r the M B I Corporation problem just described. As discussed in T echnology Insights 9 .1 , the LP m odel has three com ponents: decrsron variables, result variables, and uncontrollable variables (constraints).

T h e d e c is io n v a ria b le s are a s fo llow s:

X1 = u n it o f CC -7 to b e p ro d u ce d X2 = u n it o f CC -8 to b e p ro d u ce d

T h e resu lt v ariab le is as fo llow s:

T o ta l p ro fit = Z

T h e o b je c tiv e is to m ax im ize to tal profit:

Z = 8 ,0 0 Q *i + 12,000^2

T h e u n c o n tro lla b le v a ria b le s (co n stra in ts) are as fo llow s:

L abo r constraint: 3 0 0 ^ + 5 0 0 X 2 < 2 0 0 ,0 0 0 (in d ays)

B u d g e t con strain t: 1 0 ,0 0 0 X a + 1 5 ,0 0 0 X 2 < 8 ,0 0 0 ,0 0 0 (in d o llars)

M arketing req u irm en t fo r CC-7: X j > 100 (in un its) M arketing requ irm en t fo r CC-8: X2 >: 2 0 0 (in un its)

T h is in fo rm atio n is su m m arized in Figu re 9-5- T h e m o d e l a ls o h a s a fo u rth , h id d e n c o m p o n e n t. E v e ry LP m o d e l h a s s o m e

in te rn a l in te rm e d ia te v a r ia b le s th a t a re n o t e x p lic itly state d . T h e la b o r a n d b u d g e t c o n ­ strain ts m a y e a c h h a v e s o m e s la c k in th e m w h e n th e le ft-h a n d s id e is s trictly le s s th a n th e rig h t-h a n d sid e . T h is s la c k is r e p re s e n te d in te rn a lly b y s la c k v a r ia b le s th a t in d i­ c a te e x c e s s r e s o u r c e s a v a ila b le . T h e m a rk e tin g r e q u ir e m e n t co n s tr a in ts m ay e a c h h a v e s o m e s u rp lu s in th e m w h e n th e le ft-h a n d s id e is strictly g re a te r th a n th e rig h t-h an d s id e T h is s u rp lu s is r e p r e s e n te d in te rn a lly b y su rp lu s v a r ia b le s in d ic a tin g th a t th e r e is s o m e ro o m to a d ju st th e rig h t-h a n d s id e s o f th e s e c o n s tra in ts. T h e s e s la c k a n d su rp lu s v a r ia b le s a r e in te rm e d ia te . T h e y c a n b e o f g re a t v a lu e to a d e c is io n m a k e r b e c a u s e LP s o lu tio n m e th o d s u s e th e m in e s ta b lis h in g s e n sitiv ity p a ra m e te rs fo r e c o n o m ic w h a t-if

a n a ly se s.

T E C H N O L O G Y IN S IG H T S 9 .1 L in ear P ro g ra m m in g

LP is perhaps the best-known optimization model. It deals with the optimal allocation o f resources among competing activities. The allocation problem is represented by the model described here

The problem is to find the values o f the decision variables Xh X2, and so on, such that the value o f the result variable Z is maximized, subject to a set o f linear constraints that express the technology, market conditions, and other uncontrollable variables. The mathematical rela­ tionships are all linear equations and inequalities. Theoretically, any allocation problem o f this type has an infinite number o f possible solutions. Using special mathematical procedures, the LP approach applies a unique computerized search procedure that finds a best solution(s) in a matter o f seconds. Furthermore, the solution approach provides automatic sensitivity analysis.

4 4 0 Part IV • Prescriptive Analytics

FIGURE 9.5 M ath em atical M o del o f a P ro d u ct-M ix Ex am p le .

T h e p ro d u c t-m ix m o d e l h a s a n in fin ite n u m b e r o f p o s s ib le s o lu tio n s . A ssu m in g th a t a p ro d u c tio n p la n is n o t re s tric te d to w h o le n u m b e rs — w h ic h is a r e a s o n a b le a s s u m p tio n in a m o n th ly p ro d u c tio n p l a n - w e w a n t a s o lu tio n th a t m a x im iz e s to ta l profit- a n o p tim a l s o lu tio n . F o rtu n a te ly , E x c e l c o m e s w ith th e a d d -in S o lv e r w h ic h c a n re a d ily o b ta in a n o p tim a l ( b e s t ) s o lu tio n to this p ro b le m . A lth o u g h th e lo c a tio n o f S o lv e r A d d -in h a s m o v e d fro m o n e v e r s io n o f E x c e l to a n o th e r , it is still a v a ila b le as a fr e e A d d -in. L o o k fo r it u n d e r D a ta ta b a n d o n th e A n alysis rib b o n . I f it is n o t th e r e , y o u sh o u ld b e a b le t o e n a b le it b y g o in g to E x c e l ’s O p tio n s M en u a n d s e le c tin g

W e e n te r th e s e d ata d irectly in to a n E x c e l s p re a d sh e e t, a ctiv a te So lv er, a n d id e n ­ tify th e g o a l (b y settin g T a rg e t C ell e q u a l t o M ax ), d e c is io n v a ria b le s (b y settin g y C h an g in g C ells), an d co n strain ts (b y e n su rin g th at T o ta l C o n su m e d e le m e n ts is le s s th an o r e q u a l to Lim it fo r th e first tw o ro w s a n d is g re a te r th a n o r e q u a l to Lim it fo r th e third an d fo u rth ro w s). C ells C 7 a n d D 7 co n stitu te th e d e c is io n v a ria b le ce lls. R esu lts m th e se c e lls w o u ld b e filled a fte r ru n n in g th e S o lv e r A dd-in. T a rg e t C ell is C ell E7, w h ic h is also th e re su lt v aria b le , re p re sen tin g a p ro d u ct o f d e c is io n v a ria b le c e lls an d th e ir p e r unit p ro fit c o e ffic ie n ts (in C ells C 8 an d D 8 ). N ote th a t all th e n u m b e rs h av e b e e n d iv id ed b y 1 0 0 0 to m a k e it e a s ie r to ty p e ( e x c e p t th e d e c is io n v a ria b le s ). R o w s 9 - 1 2 d e s c rib e th e co n strain ts o f th e p ro b le m : th e co n stra in ts o n la b o r ca p a city , b u d g e t, a n d th e d e sire d m inim um p ro d u ctio n o f th e tw o p ro d u cts X1 an d X2. C o lu m n s C a n d D d e fin e th e c o e - ficie n ts o f th e s e co n strain ts. C o lu m n E in clu d e s th e fo rm u la e that m u ltiply th e d e c is io n v a ria b le s (C e lls C 7 a n d D 7 ) w ith th e ir r e s p e c tiv e c o e ffic ie n ts in e a c h ro w . C o lu m n t d e fin e s th e rig h t-h an d sid e v a lu e o f th e s e co n stra in ts. E x c e l’s m atrix m u ltip lica tio n c a p a ­ b ilitie s (e .g ., SU M PRO D U C T fu n c tio n ) c a n b e u s e d to d e v e lo p s u ch ro w a n d co lu m n

m u ltip licatio n s easily . . . , A fter th e m o d e l’s ca lc u la tio n s h a v e b e e n s e t u p in E x c e l, it is tim e to in v o k e the

S o lv e r A d d -in. C lick in g o n th e S o lv e r A d d -in (a g a in u n d e r th e A n alysis g ro u p u n d e r D ata T a b ) o p e n s a d ia lo g b o x (w in d o w ) th a t lets y o u s p e c ify th e c e lls o r ra n g e s th a t d e fin e th e o b je c tiv e fu n c tio n c e ll, d ecis io n / ch a n g in g v a r ia b le s (c e lls ), a n d th e c o n ­ straints. A lso, in O p tio n s , w e s e le c t th e s o lu tio n m e th o d (u su a lly S im p le x L P), an d th e n w e s o lv e th e p ro b le m . N ext, w e s e le c t all th re e re p o rts -A n s w e r, S en siu v ity an d L i m i t s - t o o b ta in a n o p tim a l s o lu tio n o f X j = 3 3 3 -3 3 , * 2 = 2 0 0 , a n d P ro fit = $ 5 ,0 6 6 ,6 6 7 a s s h o w n in F ig u re 9-6. S o lv e r p ro d u c e s th r e e u se fu l re p o rts a b o u t th e s o lu tio n . T ry l . S o lv e r n o w a ls o in c lu d e s th e ab ility to s o lv e n o n lin e a r p ro g ram m in g p ro b le m s a n d in te ­ g e r p ro g ra m m in g p ro b le m s b y u s in g o th e r s o lu tio n m e th o d s a v a ila b le w ith in it.

Chapter 9 • M odel-Based D ecisio n Making: O ptim ization and Multi-Criteria Systems 441

T h e fo llo w in g e x a m p le w a s c r e a te d b y P ro f. R ic k W ils o n o f O k la h o m a S ta te U n iv e rsity to fu rth e r illu stra te th e p o w e r o f s p r e a d s h e e t m o d e lin g fo r d e c

SUPP°The t a b l e in F ig u r e 9 .7 d e s c r i b e s s o m e e s t im a t e d d a ta a n d a ttr ib u te s o f n in e “s w in g s ta te s fo r t i e 2 0 1 2 e le c t i o n . A ttrib u te s o f th e n in e s ta te s in c lu d e th e ir n u m b e r o f e l e c t o r a l v o t e s , tw o r e g io n a l d e s c r ip to r s ( n o t e th a t t h r e e s t a t e s a r e c a s ^ d a s L i t h e r N o r th % S o u th ) a n d a n e s t im a t e d “i n f l u e n c e f u n c t i o n ,’ w h ic h r e la t e s t o i n c r e a s e d c a n d id a te s u p p o r t p e r u n it o f c a m p a ig n fin a n c i a l m v e s tm e

“ th F o r T n s ta n c e , in flu e n ce fu n ctio n F I sh o w s th a t fo r e v ery fin a n c ia l_ unit m vested in that state th e re w ill b e a total o f a 10-u n it in cre a se m v o te r su p p ort (u n its w ill stay g e n eral h e r e ), m ad e up o f a n in c re a se in y o u n g m e n su p p o rt b y 3 units, o ld m e n su p p o h v 1 u n it an d v o u n g an d o ld w o m e n e a c h b y 3 units.

T h e c a m p a ig n h a s 1 ,0 5 0 fin an cial u n its to invest in th e 9 states. It m u st in v est at 5 p e rc e n t “ state o th e to tal ov erall in vested , b u t n o m o re th a n 2 5 p e rc e n t o f he o v e r S t o t ll in v e sted c a n b e in an y o n e state. All 1 ,0 5 0 units d o n o t h a v e to b e inv ested

as w ell. F r o m a ^ i —

stand p oint th e W e st states (in to tal) m u st h av e cam p aig n investm ent a t lev els th a t are at le a st 60 p e rc e n t o f th e total inv ested in E ast states. In term s o f p e o p le in flu en ced , th e d e cisio n to a llo ca te fin an cial investm ents to states m ust le a d to at le a s t 9 ,2 0 0 total p eo p l in flu e n ced O v erall th e total n u m b e r o f fe m a le s in flu e n ced m u st b e g re a te r th an o r equ a to th e total n u m b e r o f m ales influ en ced . A lso, at le a s t 4 6 p e rc e n t o f all p e o p le in flu en ced

m ust b e “o ld .”

4 4 2 Part IV * Prescriptive Analytics

Electoral Influence

---- — — NV 6 W e st F1

CO 9 W e st F2

IA 6 W e st North F3

WI 10 W e st North F1

□H 18 East North F2

VA 13 East South F2

NC 15 East South F1

FL 29 East South F3

NH 4 E ast F3

Old

Men 2 .5 2 .5 5

Women 1 2 3

3 .5 4 .5 8

Men 3 1 4

Women 3 3 6 | 6 4 1 0 Total