discussion-0903
twinkle
Chapter1and2.pdf
1
P A R T
Introduction to Analytics and AI
I
2
1
LEARNING OBJECTIVES
Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence: Systems for Decision Support
■■ Understand the need for computerized support of managerial decision making
■■ Understand the development of systems for providing decision-making support
■■ Recognize the evolution of such computerized support to the current state of analytics/data science and artificial intelligence
■■ Describe the business intelligence (BI) methodology and concepts
■■ Understand the different types of analytics and review selected applications
■■ Understand the basic concepts of artificial intelligence (AI) and see selected applications
■■ Understand the analytics ecosystem to identify various key players and career opportunities
T he business environment (climate) is constantly changing, and it is becoming more and more complex. Organizations, both private and public, are under pres-sures that force them to respond quickly to changing conditions and to be in- novative in the way they operate. Such activities require organizations to be agile and to make frequent and quick strategic, tactical, and operational decisions, some of which are very complex. Making such decisions may require considerable amounts of relevant data, information, and knowledge. Processing these in the framework of the needed decisions must be done quickly, frequently in real time, and usually requires some computerized support. As technologies are evolving, many decisions are being automated, leading to a major impact on knowledge work and workers in many ways.
This book is about using business analytics and artificial intelligence (AI) as a computerized support portfolio for managerial decision making. It concentrates on the
C H A P T E R
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 3
theoretical and conceptual foundations of decision support as well as on the commercial tools and techniques that are available. The book presents the fundamentals of the tech- niques and the manner in which these systems are constructed and used. We follow an EEE (exposure, experience, and exploration) approach to introducing these topics. The book primarily provides exposure to various analytics/AI techniques and their applica- tions. The idea is that students will be inspired to learn from how various organizations have employed these technologies to make decisions or to gain a competitive edge. We believe that such exposure to what is being accomplished with analytics and that how it can be achieved is the key component of learning about analytics. In describing the techniques, we also give examples of specific software tools that can be used for devel- oping such applications. However, the book is not limited to any one software tool, so students can experience these techniques using any number of available software tools. We hope that this exposure and experience enable and motivate readers to explore the potential of these techniques in their own domain. To facilitate such exploration, we include exercises that direct the reader to Teradata University Network (TUN) and other sites that include team-oriented exercises where appropriate. In our own teaching experi- ence, projects undertaken in the class facilitate such exploration after students have been exposed to the myriad of applications and concepts in the book and they have experi- enced specific software introduced by the professor.
This introductory chapter provides an introduction to analytics and artificial intel- ligence as well as an overview of the book. The chapter has the following sections:
1.1 Opening Vignette: How Intelligent Systems Work for KONE Elevators and Escalators Company 3
1.2 Changing Business Environments and Evolving Needs for Decision Support and Analytics 5
1.3 Decision-Making Processes and Computer Decision Support Framework 9 1.4 Evolution of Computerized Decision Support to Business Intelligence/
Analytics/Data Science 22 1.5 Analytics Overview 30 1.6 Analytics Examples in Selected Domains 38 1.7 Artificial Intelligence Overview 52 1.8 Convergence of Analytics and AI 59 1.9 Overview of the Analytics Ecosystem 63
1.10 Plan of the Book 65 1.11 Resources, Links, and the Teradata University Network Connection 66
1.1 OPENING VIGNETTE: How Intelligent Systems Work for KONE Elevators and Escalators Company
KONE is a global industrial company (based in Finland) that manufactures mostly eleva- tors and escalators and also services over 1.1 million elevators, escalators, and related equipment in several countries. The company employs over 50,000 people.
THE PROBLEM
Over 1 billion people use the elevators and escalators manufactured and serviced by KONE every day. If equipment does not work properly, people may be late to work, can- not get home in time, and may miss important meetings and events. So, KONE’s objective is to minimize the downtime and users’ suffering.
4 Part I • Introduction to Analytics and AI
The company has over 20,000 technicians who are dispatched to deal with the elevators anytime a problem occurs. As buildings are getting higher (the trend in many places), more people are using elevators, and there is more pressure on elevators to handle the growing amount of traffic. KONE faced the responsibility to serve users smoothly and safely.
THE SOLUTION
KONE decided to use IBM Watson IoT Cloud platform. As we will see in Chapter 6, IBM installed cognitive abilities in buildings that make it possible to recognize situations and behavior of both people and equipment. The Internet of Things (IoT), as we will see in Chapter 13, is a platform that can connect millions of “things” together and to a central command that can manipulate the connected things. Also, the IoT connects sensors that are attached to KONE’s elevators and escalators. The sensors collect information and data about the elevators (such as noise level) and other equipment in real time. Then, the IoT transfers to information centers via the collected data “cloud.” There, analytic systems (IBM Advanced Analytic Engine) and AI process the collected data and predict things such as potential failures. The systems also identify the likely causes of problems and suggest poten- tial remedies. Note the predictive power of IBM Watson Analytics (using machine learning, an AI technology described in Chapters 4–6) for finding problems before they occur.
The KONE system collects a significant amount of data that are analyzed for other purposes so that future design of equipment can be improved. This is because Watson Analytics offers a convenient environment for communication of and collaboration around the data. In addition, the analysis suggests how to optimize buildings and equip- ment operations. Finally, KONE and its customers can get insights regarding the financial aspects of managing the elevators.
KONE also integrates the Watson capabilities with Salesforce’s service tools (Service Cloud Lightning and Field Service Lightning). This combination helps KONE to immedi- ately respond to emergencies or soon-to-occur failures as quickly as possible, dispatch- ing some of its 20,000 technicians to the problems’ sites. Salesforce also provides superb customer relationship management (CRM). The people–machine communication, query, and collaboration in the system are in a natural language (an AI capability of Watson Analytics; see Chapter 6). Note that IBM Watson analytics includes two types of analytics: predictive, which predicts when failures may occur, and prescriptive, which recommends actions (e.g., preventive maintenance).
THE RESULTS
KONE has minimized downtime and shortened the repair time. Obviously, elevators/ escalators users are much happier if they do not have problems because of equipment downtime, so they enjoy trouble-free rides. The prediction of “soon-to-happen” can save many problems for the equipment owners. The owners can also optimize the schedule of their own employees (e.g., cleaners and maintenance workers). All in all, the decision mak- ers at both KONE and the buildings can make informed and better decisions. Some day in the future, robots may perform maintenance and repairs of elevators and escalators.
Note: This case is a sample of IBM Watson’s success using its cognitive buildings capability. To learn more, we suggest you view the following YouTube videos: (1) youtube.com/watch?v=6UPJHyiJft0 (1:31 min.) (2017); (2) youtube.com/watch?v=EVbd3ejEXus (2:49 min.) (2017).
Sources: Compiled from J. Fernandez. (2017, April). “A Billion People a Day. Millions of Elevators. No Room for Downtime.” IBM developer Works Blog. developer.ibm.com/dwblog/2017/kone-watson-video/ (accessed September 2018); H. Srikanthan. “KONE Improves ‘People Flow’ in 1.1 Million Elevators with IBM Watson IoT.” Generis. https://generisgp.com/2018/01/08/ibm-case-study-kone-corp/ (accessed September 2018); L. Slowey. (2017, February 16). “Look Who’s Talking: KONE Makes Elevator Services Truly Intelligent with Watson IoT.” IBM Internet of Things Blog. ibm.com/blogs/internet-of-things/kone/ (accessed September 2018).
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 5
u QUESTIONS FOR THE OPENING VIGNETTE
1. It is said that KONE is embedding intelligence across its supply chain and enables smarter buildings. Explain.
2. Describe the role of IoT in this case. 3. What makes IBM Watson a necessity in this case? 4. Check IBM Advanced Analytics. What tools were included that relate to this case? 5. Check IBM cognitive buildings. How do they relate to this case?
WHAT CAN WE LEARN FROM THIS VIGNETTE?
Today, intelligent technologies can embark on large-scale complex projects when they include AI combined with IoT. The capabilities of integrated intelligent platforms, such as IBM Watson, make it possible to solve problems that were economically and techno- logically unsolvable just a few years ago. The case introduces the reader to several of the technologies, including advanced analytics, sensors, IoT, and AI that are covered in this book. The case also points to the use of “cloud.” The cloud is used to centrally process large amounts of information using analytics and AI algorithms, involving “things” in dif- ferent locations. This vignette also introduces us to two major types of analytics: predic- tive analytics (Chapters 4–6) and prescriptive analytics (Chapter 8).
Several AI technologies are discussed: machine learning, natural language process- ing, computer vision, and prescriptive analysis.
The case is an example of augmented intelligence in which people and machines work together. The case illustrates the benefits to the vendor, the implementing compa- nies, and their employees and to the users of the elevators and escalators.
1.2 CHANGING BUSINESS ENVIRONMENTS AND EVOLVING NEEDS FOR DECISION SUPPORT AND ANALYTICS
Decision making is one of the most important activities in organizations of all kind— probably the most important one. Decision making leads to the success or failure of orga- nizations and how well they perform. Making decisions is getting difficult due to internal and external factors. The rewards of making appropriate decisions can be very high and so can the loss of inappropriate ones.
Unfortunately, it is not simple to make decisions. To begin with, there are several types of decisions, each of which requires a different decision-making approach. For ex- ample, De Smet et al. (2017) of McKinsey & Company management consultants classify organizational decision into the following four groups:
• Big-bet, high-risk decisions. • Cross-cutting decisions, which are repetitive but high risk that require group work
(Chapter 11). • Ad hoc decisions that arise episodically. • Delegated decisions to individuals or small groups.
Therefore, it is necessary first to understand the nature of decision making. For a comprehensive discussion, see (De Smet et al. 2017).
Modern business is full of uncertainties and rapid changes. To deal with these, or- ganizational decision makers need to deal with ever-increasing and changing data. This book is about the technologies that can assist decision makers in their jobs.
6 Part I • Introduction to Analytics and AI
Decision-Making Process
For years, managers considered decision making purely an art—a talent acquired over a long period through experience (i.e., learning by trial and error) and by using intuition. Management was considered an art because a variety of individual styles could be used in approaching and successfully solving the same types of manage- rial problems. These styles were often based on creativity, judgment, intuition, and experience rather than on systematic quantitative methods grounded in a scientific ap- proach. However, recent research suggests that companies with top managers who are more focused on persistent work tend to outperform those with leaders whose main strengths are interpersonal communication skills. It is more important to emphasize methodical, thoughtful, analytical decision making rather than flashiness and interper- sonal communication skills.
Managers usually make decisions by following a four-step process (we learn more about these in the next section):
1. Define the problem (i.e., a decision situation that may deal with some difficulty or with an opportunity).
2. Construct a model that describes the real-world problem. 3. Identify possible solutions to the modeled problem and evaluate the solutions. 4. Compare, choose, and recommend a potential solution to the problem.
A more detailed process is offered by Quain (2018), who suggests the following steps:
1. Understand the decision you have to make. 2. Collect all the information. 3. Identify the alternatives. 4. Evaluate the pros and cons. 5. Select the best alternative. 6. Make the decision. 7. Evaluate the impact of your decision.
We will return to this process in Section 1.3.
The Influence of the External and Internal Environments on the Process
To follow these decision-making processes, one must make sure that sufficient alterna- tive solutions, including good ones, are being considered, that the consequences of using these alternatives can be reasonably predicted, and that comparisons are done properly. However, rapid changes in internal and external environments make such an evaluation process difficult for the following reasons:
• Technology, information systems, advanced search engines, and globalization re- sult in more and more alternatives from which to choose.
• Government regulations and the need for compliance, political instability and ter- rorism, competition, and changing consumer demands produce more uncertainty, making it more difficult to predict consequences and the future. • Political factors. Major decisions may be influenced by both external and
internal politics. An example is the 2018 trade war on tariffs. • Economic factors. These range from competition to the genera and state
of the economy. These factors, both in the short and long run, need to be considered.
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 7
• Sociological and psychological factors regarding employees and customers. These need to be considered when changes are being made.
• Environment factors. The impact on the physical environment must be assessed in many decision-making situations.
Other factors include the need to make rapid decisions, the frequent and unpredict- able changes that make trial-and-error learning difficult, and the potential costs of making mistakes that may be large.
These environments are growing more complex every day. Therefore, making deci- sions today is indeed a complex task. For further discussion, see Charles (2018). For how to make effective decisions under uncertainty and pressure, see Zane (2016).
Because of these trends and changes, it is nearly impossible to rely on a trial- and-error approach to management. Managers must be more sophisticated; they must use the new tools and techniques of their fields. Most of those tools and techniques are discussed in this book. Using them to support decision making can be extremely rewarding in making effective decisions. Further, many tools that are evolving impact even the very existence of several decision-making tasks that are being automated. This impacts future demand for knowledge workers and begs many legal and societal impact questions.
Data and Its Analysis in Decision Making
We will see several times in this book how an entire industry can employ analytics to develop reports on what is happening, predict what is likely to happen, and then make decisions to make the best use of the situation at hand. These steps require an organiza- tion to collect and analyze vast stores of data. In general, the amount of data doubles every two years. From traditional uses in payroll and bookkeeping functions, computer- ized systems are now used for complex managerial areas ranging from the design and management of automated factories to the application of analytical methods for the eval- uation of proposed mergers and acquisitions. Nearly all executives know that information technology is vital to their business and extensively use these technologies.
Computer applications have moved from transaction-processing and monitoring ac- tivities to problem analysis and solution applications, and much of the activity is done with cloud-based technologies, in many cases accessed through mobile devices. Analytics and BI tools such as data warehousing, data mining, online analytical processing (OLAP), dashboards, and the use of cloud-based systems for decision support are the cornerstones of today’s modern management. Managers must have high-speed, networked information systems (wired or wireless) to assist them with their most important task: making deci- sions. In many cases, such decisions are routinely being fully automated (see Chapter 2), eliminating the need for any managerial intervention.
Technologies for Data Analysis and Decision Support
Besides the obvious growth in hardware, software, and network capacities, some devel- opments have clearly contributed to facilitating the growth of decision support and ana- lytics technologies in a number of ways:
• Group communication and collaboration. Many decisions are made today by groups whose members may be in different locations. Groups can collaborate and communicate readily by using collaboration tools as well as the ubiquitous smartphones. Collaboration is especially important along the supply chain, where partners—all the way from vendors to customers—must share information. Assembling a group of decision makers, especially experts, in one place can be
8 Part I • Introduction to Analytics and AI
costly. Information systems can improve the collaboration process of a group and enable its members to be at different locations (saving travel costs). More critically, such supply chain collaboration permits manufacturers to know about the changing patterns of demand in near real time and thus react to marketplace changes faster. For a comprehensive coverage and the impact of AI, see Chapters 2, 10, and 14.
• Improved data management. Many decisions involve complex computations. Data for these can be stored in different databases anywhere in the organization and even possibly outside the organization. The data may include text, sound, graphics, and video, and these can be in different languages. Many times it is neces- sary to transmit data quickly from distant locations. Systems today can search, store, and transmit needed data quickly, economically, securely, and transparently. See Chapters 3 and 9 and the online chapter for details.
• Managing giant data warehouses and Big Data. Large data warehouses (DWs), like the ones operated by Walmart, contain huge amounts of data. Special methods, including parallel computing and Hadoop/Spark, are available to orga- nize, search, and mine the data. The costs related to data storage and mining are declining rapidly. Technologies that fall under the broad category of Big Data have enabled massive data coming from a variety of sources and in many different forms, which allows a very different view of organizational performance that was not pos- sible in the past. See Chapter 9 for details.
• Analytical support. With more data and analysis technologies, more alternatives can be evaluated, forecasts can be improved, risk analysis can be performed quickly, and the views of experts (some of whom may be in remote locations) can be collected quickly and at a reduced cost. Expertise can even be derived directly from analytical systems. With such tools, decision makers can perform complex simulations, check many possible scenarios, and assess diverse impacts quickly and economically.This, of course, is the focus of several chapters in the book. See Chapters 4–7.
• Overcoming cognitive limits in processing and storing information. The human mind has only a limited ability to process and store information. People sometimes find it difficult to recall and use information in an error-free fashion due to their cognitive limits. The term cognitive limits indicates that an individual’s problem-solving capability is limited when a wide range of diverse information and knowledge is required. Computerized systems enable people to overcome their cognitive limits by quickly accessing and processing vast amounts of stored infor- mation. One way to overcome humans’ cognitive limitations is to use AI support. For coverage of cognitive aspects, see Chapter 6.
• Knowledge management. Organizations have gathered vast stores of informa- tion about their own operations, customers, internal procedures, employee interac- tions, and so forth through the unstructured and structured communications taking place among various stakeholders. Knowledge management systems (KMS) have become sources of formal and informal support for decision making to manag- ers, although sometimes they may not even be called KMS. Technologies such as text analytics and IBM Watson are making it possible to generate value from such knowledge stores. (See Chapters 6 and 12 for details.
• Anywhere, anytime support. Using wireless technology, managers can access information anytime and from any place, analyze and interpret it, and communicate with those using it. This perhaps is the biggest change that has occurred in the last few years. The speed at which information needs to be processed and converted into decisions has truly changed expectations for both consumers and businesses. These and other capabilities have been driving the use of computerized decision support since the late 1960s, especially since the mid-1990s. The growth of mobile technologies, social media platforms, and analytical tools has enabled a different level of information systems (IS) to support managers. This growth in providing
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 9
data-driven support for any decision extends not just to managers but also to con- sumers. We will first study an overview of technologies that have been broadly referred to as BI. From there we will broaden our horizons to introduce various types of analytics.
• Innovation and artificial intelligence. Because of the complexities in the decision-making process discussed earlier and the environment surrounding the process, a more innovative approach is frequently need. A major facilitation of innovation is provided by AI. Almost every step in the decision-making process can be influenced by AI. AI is also integrated with analytics, creating synergy in making decisions (Section 1.8).
u SECTION 1.2 REVIEW QUESTIONS
1. Why is it difficult to make organizational decisions? 2. Describe the major steps in the decision-making process. 3. Describe the major external environments that can impact decision making. 4. What are some of the key system-oriented trends that have fostered IS-supported
decision making to a new level?
5. List some capabilities of information technologies that can facilitate managerial deci- sion making.
1.3 DECISION-MAKING PROCESSES AND COMPUTERIZED DECISION SUPPORT FRAMEWORK
In this section, we focus on some classical decision-making fundamentals and in more detail on the decision-making process. These two concepts will help us ground much of what we will learn in terms of analytics, data science, and artificial intelligence.
Decision making is a process of choosing among two or more alternative courses of action for the purpose of attaining one or more goals. According to Simon (1977), mana- gerial decision making is synonymous with the entire management process. Consider the important managerial function of planning. Planning involves a series of decisions: What should be done? When? Where? Why? How? By whom? Managers set goals, or plan; hence, planning implies decision making. Other managerial functions, such as organizing and controlling, also involve decision making.
Simon’s Process: Intelligence, Design, and Choice
It is advisable to follow a systematic decision-making process. Simon (1977) said that this involves three major phases: intelligence, design, and choice. He later added a fourth phase: implementation. Monitoring can be considered a fifth phase—a form of feedback. However, we view monitoring as the intelligence phase applied to the imple- mentation phase. Simon’s model is the most concise and yet complete characterization of rational decision making. A conceptual picture of the decision-making process is shown in Figure 1.1. It is also illustrated as a decision support approach using modeling.
There is a continuous flow of activity from intelligence to design to choice (see the solid lines in Figure 1.1), but at any phase, there may be a return to a previous phase (feedback). Modeling is an essential part of this process. The seemingly chaotic nature of following a haphazard path from problem discovery to solution via decision making can be explained by these feedback loops.
The decision-making process starts with the intelligence phase; in this phase, the decision maker examines reality and identifies and defines the problem. Problem owner- ship is established as well. In the design phase, a model that represents the system is constructed. This is done by making assumptions that simplify reality and by writing down
10 Part I • Introduction to Analytics and AI
the relationships among all the variables. The model is then validated, and criteria are de- termined in a principle of choice for evaluation of the alternative courses of action that are identified. Often, the process of model development identifies alternative solutions and vice versa.
The choice phase includes the selection of a proposed solution to the model (not necessarily to the problem it represents). This solution is tested to determine its viability. When the proposed solution seems reasonable, we are ready for the last phase: imple- mentation of the decision (not necessarily of a system). Successful implementation results in solving the real problem. Failure leads to a return to an earlier phase of the process. In fact, we can return to an earlier phase during any of the latter three phases. The decision- making situations described in the opening vignette follow Simon’s four-phase model, as do almost all other decision-making situations.
The Intelligence Phase: Problem (or Opportunity) Identification
The intelligence phase begins with the identification of organizational goals and objectives related to an issue of concern (e.g., inventory management, job selection, lack of or incorrect Web presence) and determination of whether they are being met. Problems occur because of dissatisfaction with the status quo. Dissatisfaction is the result of a difference between what people desire (or expect) and what is occurring. In this first phase, a decision maker attempts to determine whether a problem exists, identify its symptoms, determine its magnitude, and
Success
Organization objectives Search and scanning procedures Data collection Problem identification Problem ownership Problem classification Problem statement
Solution to the model Sensitivity analysis Selection to the best (good) alternative(s)
Plan for implementation
Formulate a model Set criteria for choice Search for alternatives Predict and measure outcomes
Assumptions
Simplification
Problem Statement
Alternatives
Validation of the Model
Verification, Testing of the Proposed Solution
Implementation of the solution
Failure
Intelligence
Design
Choice
Reality
FIGURE 1.1 The Decision-Making/Modeling Process.
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 11
explicitly define it. Often, what is described as a problem (e.g., excessive costs) may be only a symptom (i.e., measure) of a problem (e.g., improper inventory levels). Because real-world problems are usually complicated by many interrelated factors, it is sometimes difficult to distinguish between the symptoms and the real problem. New opportunities and problems certainly may be uncovered while investigating the causes of symptoms.
The existence of a problem can be determined by monitoring and analyzing the organization’s productivity level. The measurement of productivity and the construction of a model are based on real data. The collection of data and the estimation of future data are among the most difficult steps in the analysis.
ISSUES IN DATA COLLECTION The following are some issues that may arise during data collection and estimation and thus plague decision makers:
• Data are not available. As a result, the model is made with and relies on potentially inaccurate estimates.
• Obtaining data may be expensive. • Data may not be accurate or precise enough. • Data estimation is often subjective. • Data may be insecure. • Important data that influence the results may be qualitative (soft). • There may be too many data (i.e., information overload). • Outcomes (or results) may occur over an extended period. As a result, revenues,
expenses, and profits will be recorded at different points in time. To overcome this difficulty, a present-value approach can be used if the results are quantifiable.
• It is assumed that future data will be similar to historical data. If this is not the case, the nature of the change has to be predicted and included in the analysis.
When the preliminary investigation is completed, it is possible to determine whether a problem really exists, where it is located, and how significant it is. A key issue is whether an information system is reporting a problem or only the symptoms of a problem. For example, if reports indicate that sales are down, there is a problem, but the situation, no doubt, is symptomatic of the problem. It is critical to know the real problem. Sometimes it may be a problem of perception, incentive mismatch, or organizational processes rather than a poor decision model.
To illustrate why it is important to identify the problem correctly, we provide a clas- sical example in Application Case 1.1.
This story has been reported in numerous places and has almost become a classic example to explain the need for problem identification. Ackoff (as cited in Larson, 1987) described the problem of manag- ing complaints about slow elevators in a tall hotel tower. After trying many solutions for reducing the complaint—staggering elevators to go to different floors, adding operators, and so on—the manage- ment determined that the real problem was not
about the actual waiting time but rather the per- ceived waiting time. So the solution was to install full-length mirrors on elevator doors on each floor. As Hesse and Woolsey (1975) put it, “The women would look at themselves in the mirrors and make adjustments, while the men would look at the women, and before they knew it, the elevator was there.” By reducing the perceived waiting time, the problem went away. Baker and Cameron (1996)
Application Case 1.1 Making Elevators Go Faster!
(Continued )
12 Part I • Introduction to Analytics and AI
PROBLEM CLASSIFICATION Problem classification is the conceptualization of a problem in an attempt to place it in a definable category, possibly leading to a standard solution approach. An important approach classifies problems according to the degree of struc- turedness evident in them. This ranges from totally structured (i.e., programmed) to to- tally unstructured (i.e., unprogrammed).
PROBLEM DECOMPOSITION Many complex problems can be divided into subproblems. Solving the simpler subproblems may help in solving a complex problem. Also, seemingly poorly structured problems sometimes have highly structured subproblems. Just as a sem- istructured problem results when some phases of decision making are structured whereas other phases are unstructured, and when some subproblems of a decision- making prob- lem are structured with others unstructured, the problem itself is semistructured. As a de- cision support system is developed and the decision maker and development staff learn more about the problem, it gains structure.
PROBLEM OWNERSHIP In the intelligence phase, it is important to establish problem ownership. A problem exists in an organization only if someone or some group takes the responsibility for attacking it and if the organization has the ability to solve it. The assign- ment of authority to solve the problem is called problem ownership. For example, a man- ager may feel that he or she has a problem because interest rates are too high. Because interest rate levels are determined at the national and international levels and most manag- ers can do nothing about them, high interest rates are the problem of the government, not a problem for a specific company to solve. The problem that companies actually face is how to operate in a high interest-rate environment. For an individual company, the interest rate level should be handled as an uncontrollable (environmental) factor to be predicted.
When problem ownership is not established, either someone is not doing his or her job or the problem at hand has yet to be identified as belonging to anyone. It is then important for someone to either volunteer to own it or assign it to someone.
The intelligence phase ends with a formal problem statement.
The Design Phase
The design phase involves finding or developing and analyzing possible courses of action. These include understanding the problem and testing solutions for feasibility. A model of the decision-making problem is constructed, tested, and validated. Let us first define a model.
give several other examples of distractions, includ- ing lighting and displays, that organizations use to reduce perceived waiting time. If the real problem is identified as perceived waiting time, it can make a big difference in the proposed solutions and their costs. For example, full-length mirrors probably cost a whole lot less than adding an elevator!
Sources: Based on J. Baker and M. Cameron. (1996, September). “The Effects of the Service Environment on Affect and Consumer Perception of Waiting Time: An Integrative Review and Research Propositions,” Journal of the Academy of Marketing
Science, 24, pp. 338–349; R. Hesse and G. Woolsey (1975). Applied Management Science: A Quick and Dirty Approach. Chicago, IL: SRA Inc; R. C. Larson. (1987, November/December). “Perspectives on Queues: Social Justice and the Psychology of Queuing.” Operations Research, 35(6), pp. 895–905.
Questions for Case 1.1
1. Why this is an example relevant to decision making?
2. Relate this situation to the intelligence phase of decision making.
Application Case 1.1 (Continued)
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 13
MODELS A major characteristic of computerized decision support and many BI tools (notably those of business analytics) is the inclusion of at least one model. The basic idea is to perform the analysis on a model of reality rather than on the real system. A model is a simplified representation or abstraction of reality. It is usually simplified because reality is too complex to describe exactly and because much of the complexity is actually irrel- evant in solving a specific problem.
Modeling involves conceptualizing a problem and abstracting it to quantitative and/ or qualitative form. For a mathematical model, the variables are identified and their mu- tual relationships are established. Simplifications are made, whenever necessary, through assumptions. For example, a relationship between two variables may be assumed to be linear even though in reality there may be some nonlinear effects. A proper balance be- tween the level of model simplification and the representation of reality must be obtained because of the cost–benefit trade-off. A simpler model leads to lower development costs, easier manipulation, and a faster solution but is less representative of the real problem and can produce inaccurate results. However, a simpler model generally requires fewer data, or the data are aggregated and easier to obtain.
The Choice Phase
Choice is the critical act of decision making. The choice phase is the one in which the actual decision and the commitment to follow a certain course of action are made. The boundary between the design and choice phases is often unclear because certain activi- ties can be performed during both of them and because the decision maker can return frequently from choice activities to design activities (e.g., generate new alternatives while performing an evaluation of existing ones). The choice phase includes the search for, evaluation of, and recommendation of an appropriate solution to a model. A solution to a model is a specific set of values for the decision variables in a selected alternative. Choices can be evaluated as to their viability and profitability.
Each alternative must be evaluated. If an alternative has multiple goals, they must all be examined and balanced against each other. Sensitivity analysis is used to determine the robustness of any given alternative; slight changes in the parameters should ideally lead to slight or no changes in the alternative chosen. What-if analysis is used to explore major changes in the parameters. Goal seeking helps a manager determine values of the decision variables to meet a specific objective. These topics are addressed in Chapter 8.
The Implementation Phase
In The Prince, Machiavelli astutely noted some 500 years ago that there was “nothing more difficult to carry out, nor more doubtful of success, nor more dangerous to handle, than to initiate a new order of things.” The implementation of a proposed solution to a problem is, in effect, the initiation of a new order of things or the introduction of change. And change must be managed. User expectations must be managed as part of change management.
The definition of implementation is somewhat complicated because implementation is a long, involved process with vague boundaries. Simplistically, the implementation phase involves putting a recommended solution to work, not necessarily implementing a computer system. Many generic implementation issues, such as resistance to change, degree of support of top management, and user training, are important in dealing with information system–supported decision making. Indeed, many previous technology- related waves (e.g., business process reengineering [BPR] and knowledge management) have faced mixed results mainly because of change management challenges and issues. Management of change is almost an entire discipline in itself, so we recognize its impor- tance and encourage readers to focus on it independently. Implementation also includes
14 Part I • Introduction to Analytics and AI
a thorough understanding of project management. The importance of project manage- ment goes far beyond analytics, so the last few years have witnessed a major growth in certification programs for project managers. A very popular certification now is the Project Management Professional (PMP). See pmi.org for more details.
Implementation must also involve collecting and analyzing data to learn from the previous decisions and improve the next decision. Although analysis of data is usually conducted to identify the problem and/or the solution, analytics should also be employed in the feedback process. This is especially true for any public policy decisions. We need to be sure that the data being used for problem identification is valid. Sometimes people find this out only after the implementation phase.
The decision-making process, though conducted by people, can be improved with computer support, which is introduced next.
The Classical Decision Support System Framework
The early definitions of decision support system (DSS) identified it as a system intended to support managerial decision makers in semistructured and unstructured decision situ- ations. DSS was meant to be an adjunct to decision makers, extending their capabilities but not replacing their judgment. DSS was aimed at decisions that required judgment or at decisions that could not be completely supported by algorithms. Not specifically stated but implied in the early definitions was the notion that the system would be computer based, would operate interactively online, and preferably would have graphical output capabilities, now simplified via browsers and mobile devices.
An early framework for computerized decision support includes several major con- cepts that are used in forthcoming sections and chapters of this book. Gorry and Scott- Morton created and used this framework in the early 1970s, and the framework then evolved into a new technology called DSS.
Gorry and Scott-Morton (1971) proposed a framework that is a 3-by-3 matrix, as shown in Figure 1.2. The two dimensions are the degree of structuredness and the types of control.
DEGREE OF STRUCTUREDNESS The left side of Figure 1.2 is based on Simon’s (1977) idea that decision-making processes fall along a continuum that ranges from highly struc- tured (sometimes called programmed) to highly unstructured (i.e., non-programmed) decisions. Structured processes are routine and typically repetitive problems for which standard solution methods exist. Unstructured processes are fuzzy, complex problems for which there are no cut-and-dried solution methods.
An unstructured problem is one where the articulation of the problem or the solu- tion approach may be unstructured in itself. In a structured problem, the procedures for obtaining the best (or at least a good enough) solution are known. Whether the problem involves finding an appropriate inventory level or choosing an optimal investment strat- egy, the objectives are clearly defined. Common objectives are cost minimization and profit maximization.
Semistructured problems fall between structured and unstructured problems, hav- ing some structured elements and some unstructured elements. Keen and Scott-Morton (1978) mentioned trading bonds, setting marketing budgets for consumer products, and performing capital acquisition analysis as semistructured problems.
TYPES OF CONTROL The second half of the Gorry and Scott-Morton (1971) framework (refer to Figure 1.2) is based on Anthony’s (1965) taxonomy, which defines three broad categories that encompass all managerial activities: strategic planning, which involves defining long-range goals and policies for resource allocation; management control, the
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 15
acquisition and efficient use of resources in the accomplishment of organizational goals; and operational control, the efficient and effective execution of specific tasks.
THE DECISION SUPPORT MATRIX Anthony’s (1965) and Simon’s (1977) taxonomies are combined in the nine-cell decision support matrix shown in Figure 1.2. The initial pur- pose of this matrix was to suggest different types of computerized support to differ- ent cells in the matrix. Gorry and Scott-Morton (1971) suggested, for example, that for making semistructured decisions and unstructured decisions, conventional management information systems (MIS) and management science (MS) tools are insufficient. Human intellect and a different approach to computer technologies are necessary. They proposed the use of a supportive information system, which they called a DSS.
Note that the more structured and operational control-oriented tasks (such as those in cells 1, 2, and 4 of Figure 1.2) are usually performed by lower-level managers, whereas the tasks in cells 6, 8, and 9 are the responsibility of top executives or highly trained specialists.
COMPUTER SUPPORT FOR STRUCTURED DECISIONS Since the 1960s, computers have his- torically supported structured and some semistructured decisions, especially those that involve operational and managerial control. Operational and managerial control decisions are made in all functional areas, especially in finance and production (i.e., operations) management.
Monitoring accounts receivable Monitoring accounts payable Placing order entries
Operational Control
Structured
Managerial Control
Strategic Planning
Semistructured
Unstructured
Analyzing budget Forecasting short-term Reporting on personnel Making or buying
Scheduling production Controlling inventory
Evaluating credit Preparing budget Laying out plant Scheduling project Designing reward system Categorizing inventory
Building a new plant Planning mergers and acquisitions Planning new products Planning compensation Providing quality assurance Establishing human resources policies Planning inventory
Buying software Approving loans Operating a help desk Selecting a cover for a magazine
Negotiating Recruiting an executive Buying hardware Lobbying
Planning research and development Developing new technologies Planning social responsibility
Type of Decision
Type of Control
1 2 3
4 5 6
7 8 9
Managing finances Monitoring investment portfolio Locating warehouse Monitoring distribution systems
FIGURE 1.2 Decision Support Frameworks.
16 Part I • Introduction to Analytics and AI
Structured problems, which are encountered repeatedly, have a high level of struc- ture, as their name suggests. It is therefore possible to abstract, analyze, and classify them into specific categories. For example, a make-or-buy decision is one category. Other examples of categories are capital budgeting, allocation of resources, distribution, pro- curement, planning, and inventory control decisions. For each category of decision, an easy-to-apply prescribed model and solution approach have been developed, generally as quantitative formulas. Therefore, it is possible to use a scientific approach for automat- ing portions of managerial decision making. Solutions to many structured problems can be fully automated (see Chapters 2 and 12).
COMPUTER SUPPORT FOR UNSTRUCTURED DECISIONS Unstructured problems can be only partially supported by standard computerized quantitative methods. It is usually necessary to develop customized solutions. However, such solutions may benefit from data and information generated from corporate or external data sources. Intuition and judgment may play a large role in these types of decisions, as may computerized com- munication and collaboration technologies, as well as cognitive computing (Chapter 6) and deep learning (Chapter 5).
COMPUTER SUPPORT FOR SEMISTRUCTURED PROBLEMS Solving semistructured prob- lems may involve a combination of standard solution procedures and human judgment. Management science can provide models for the portion of a decision-making problem that is structured. For the unstructured portion, a DSS can improve the quality of the information on which the decision is based by providing, for example, not only a single solution, but also a range of alternative solutions along with their potential impacts. These capabilities help managers to better understand the nature of problems and, thus, to make better decisions.
DECISION SUPPORT SYSTEM: CAPABILITIES The early definitions of DSS identified it as a system intended to support managerial decision makers in semistructured and unstructured decision situations. DSS was meant to be an adjunct to decision makers, extending their capabilities but not replacing their judgment. It was aimed at decisions that required judgment or at decisions that could not be completely supported by al- gorithms. Not specifically stated but implied in the early definitions was the notion that the system would be computer based, would operate interactively online, and prefer- ably would have graphical output capabilities, now simplified via browsers and mobile devices.
A DSS Application
A DSS is typically built to support the solution of a certain problem or to evaluate an op- portunity. This is a key difference between DSS and BI applications. In a very strict sense, business intelligence (BI) systems monitor situations and identify problems and/or opportunities using analytic methods. Reporting plays a major role in BI; the user gener- ally must identify whether a particular situation warrants attention and then can apply analytical methods. Again, although models and data access (generally through a data warehouse) are included in BI, a DSS may have its own databases and is developed to solve a specific problem or set of problems and are therefore called DSS applications.
Formally, a DSS is an approach (or methodology) for supporting decision mak- ing. It uses an interactive, flexible, adaptable computer-based information system (CBIS) especially developed for supporting the solution to a specific unstructured management problem. It uses data, provides an easy user interface, and can incorporate the decision maker’s own insights. In addition, a DSS includes models and is developed (possibly by
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 17
end users) through an interactive and iterative process. It can support all phases of deci- sion making and may include a knowledge component. Finally, a DSS can be used by a single user or can be Web based for use by many people at several locations.
THE CHARACTERISTICS AND CAPABILITIES OF DSS Because there is no consensus on exactly what a DSS is, there is obviously no agreement on the standard characteristics and capabilities of DSS. The capabilities in Figure 1.3 constitute an ideal set, some members of which are described in the definitions of DSS and illustrated in the application cases.
The key characteristics and capabilities of DSS (as shown in Figure 1.3) are as follows:
1. Supports decision makers, mainly in semistructured and unstructured situations, by bringing together human judgment and computerized information. Such problems cannot be solved (or cannot be solved conveniently) by other computerized systems or through use of standard quantitative methods or tools. Generally, these problems gain structure as the DSS is developed. Even some structured problems have been solved by DSS.
2. Supports all managerial levels, ranging from top executives to line managers. 3. Supports individuals as well as groups. Less-structured problems often require the
involvement of individuals from different departments and organizational levels or even from different organizations. DSS supports virtual teams through collaborative Web tools. DSS has been developed to support individual and group work as well
Is adaptable and flexible
7
Provides interactivity, ease of use
8
Support variety of decision
processes and styles
6
Improves effectiveness and efficiency
9
Supports intelligence,
design, choice, and implementation
5
Provides complete human control of
the process
10
Supports managers at
all levels
2 Provides support for semistructured or unstructured
problems
1 Can be stand-
alone, integrated, and Web- based tool
14
Provides data access
13
Supports individuals and groups
3
Provides models and analysis
12
Supports interdependent or sequential
decisions
4
Provides ease of development by end users
11 Decision Support System (DSS)
FIGURE 1.3 Key Characteristics and Capabilities of DSS.
18 Part I • Introduction to Analytics and AI
as to support individual decision making and groups of decision makers working somewhat independently.
4. Supports interdependent and/or sequential decisions. The decisions may be made once, several times, or repeatedly.
5. Supports all phases of the decision-making process: intelligence, design, choice, and implementation.
6. Supports a variety of decision-making processes and styles. 7. Is flexible, so users can add, delete, combine, change, or rearrange basic elements.
The decision maker should be reactive, able to confront changing conditions quickly, and able to adapt the DSS to meet these changes. It is also flexible in that it can be readily modified to solve other, similar problems.
8. Is user-friendly, has strong graphical capabilities, and a natural language interactive human-machine interface can greatly increase the effectiveness of DSS. Most new DSS applications use Web-based interfaces or mobile platform interfaces.
9. Improves the effectiveness of decision making (e.g., accuracy, timeliness, quality) rather than its efficiency (e.g., the cost of making decisions). When DSS is deployed, decision making often takes longer, but the decisions are better.
10. Provides complete control by the decision maker over all steps of the decision- making process in solving a problem. A DSS specifically aims to support, not to replace, the decision maker.
11. Enables end users to develop and modify simple systems by themselves. Larger systems can be built with assistance from IS specialists. Spreadsheet packages have been utilized in developing simpler systems. OLAP and data mining soft- ware in conjunction with data warehouses enable users to build fairly large, complex DSS.
12. Provides models that are generally utilized to analyze decision-making situations. The modeling capability enables experimentation with different strategies under dif- ferent configurations.
13. Provides access to a variety of data sources, formats, and types, including GIS, mul- timedia, and object-oriented data.
14. Can be employed as a stand-alone tool used by an individual decision maker in one location or distributed throughout an organization and in several organiza- tions along the supply chain. It can be integrated with other DSS and/or applica- tions, and it can be distributed internally and externally, using networking and Web technologies.
These key DSS characteristics and capabilities allow decision makers to make bet- ter, more consistent decisions in a timely manner, and they are provided by major DSS components,
Components of a Decision Support System
A DSS application can be composed of a data management subsystem, a model manage- ment subsystem, a user interface subsystem, and a knowledge-based management sub- system. We show these in Figure 1.4.
The Data Management Subsystem
The data management subsystem includes a database that contains relevant data for the situation and is managed by software called the database management system (DBMS). DBMS is used as both singular and plural (system and systems) terms, as are many other acronyms in this text. The data management subsystem can be interconnected with the corporate data warehouse, a repository for corporate relevant decision-making data.
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 19
Usually, the data are stored or accessed via a database Web server. The data management subsystem is composed of the following elements:
• DSS database • Database management system • Data directory • Query facility
Many of the BI or descriptive analytics applications derive their strength from the data management side of the subsystems.
The Model Management Subsystem
The model management subsystem is the component that includes financial, statistical, management science, or other quantitative models that provide the system’s analytical capabilities and appropriate software management. Modeling languages for building cus- tom models are also included. This software is often called a model base management system (MBMS). This component can be connected to corporate or external storage of models. Model solution methods and management systems are implemented in Web de- velopment systems (such as Java) to run on application servers. The model management subsystem of a DSS is composed of the following elements:
• Model base • MBMS • Modeling language • Model directory • Model execution, integration, and command processor
Because DSS deals with semistructured or unstructured problems, it is often neces- sary to customize models, using programming tools and languages. Some examples of these are .NET Framework languages, C++, and Java. OLAP software may also be used to work with models in data analysis. Even languages for simulations such as Arena and
Data: internal and/or external
ERP/POS
Legacy
Web, etc.
Other Computer-Based
Systems
Internet, Intranet, Extranet
Data Management
User Interface
Knowledge-Based Subsystems
Manager (user)Organizational Knowledgebase
External Models
Model Management
FIGURE 1.4 Schematic View of DSS.
20 Part I • Introduction to Analytics and AI
statistical packages such as those of SPSS offer modeling tools developed through the use of a proprietary programming language. For small- and medium-sized DSS or for less complex ones, a spreadsheet (e.g., Excel) is usually used. We use Excel for several ex- amples in this book. Application Case 1.2 describes a spreadsheet-based DSS.
The User Interface Subsystem
The user communicates with and commands the DSS through the user interface subsys- tem. The user is considered part of the system. Researchers assert that some of the unique contributions of DSS are derived from the intensive interaction between the computer and the decision maker. A difficult user interface is one of the major reasons that man- agers do not use computers and quantitative analyses as much as they could, given the availability of these technologies. The Web browser provided a familiar, consistent GUI structure for many DSS in the 2000s. For locally used DSS, a spreadsheet also provides a familiar user interface. The Web browser has been recognized as an effective DSS GUI because it is flexible, user-friendly, and a gateway to almost all sources of necessary infor- mation and data. Essentially, Web browsers have led to the development of portals and dashboards, which front end many DSS.
Explosive growth in portable devices, including smartphones and tablets, has changed the DSS user interfaces as well. These devices allow either handwritten input or
Telecommunications network services to educational institutions and government entities are typically pro- vided by a mix of private and public organizations. Many states in the United States have one or more state agencies that are responsible for providing net- work services to schools, colleges, and other state agencies. One example of such an agency is OneNet in Oklahoma. OneNet is a division of the Oklahoma State Regents for Higher Education and operated in cooperation with the Office of State Finance.
Usually agencies such as OneNet operate as an enterprise-type fund. They must recover their costs through billing their clients and/or by justifying appropriations directly from the state legislatures. This cost recovery should occur through a pricing mechanism that is efficient, simple to implement, and equitable. This pricing model typically needs to recognize many factors: convergence of voice, data, and video traffic on the same infrastructure; diversity of user base in terms of educational institu- tions and state agencies; diversity of applications in use by state clients from e-mail to videoconferences, IP telephoning, and distance learning; recovery of current costs as well as planning for upgrades and
future developments; and leverage of the shared infrastructure to enable further economic develop- ment and collaborative work across the state that leads to innovative uses of OneNet.
These considerations led to the development of a spreadsheet-based model. The system, SNAP-DSS, or Service Network Application and Pricing (SNAP)- based DSS, was developed in Microsoft Excel 2007 and used the VBA programming language.
The SNAP-DSS offers OneNet the ability to select the rate card options that best fit the preferred pricing strategies by providing a real-time, user- friendly, graphical user interface (GUI). In addition, the SNAP-DSS not only illustrates the influence of the changes in the pricing factors on each rate card option but also allows the user to analyze various rate card options in different scenarios using dif- ferent parameters. This model has been used by OneNet financial planners to gain insights into their customers and analyze many what-if scenarios of different rate plan options.
Source: Based on J. Chongwatpol and R. Sharda. (2010, December). “SNAP: A DSS to Analyze Network Service Pricing for State Networks.” Decision Support Systems, 50(1), pp. 347–359.
Application Case 1.2 SNAP DSS Helps OneNet Make Telecommunications Rate Decisions
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 21
typed input from internal or external keyboards. Some DSS user interfaces utilize natural language input (i.e., text in a human language) so that the users can easily express them- selves in a meaningful way. Cell phone inputs through short message service (SMS) or chatbots are becoming more common for at least some consumer DSS-type applications. For example, one can send an SMS request for search on any topic to GOOGL (46645). Such capabilities are most useful in locating nearby businesses, addresses, or phone numbers, but it can also be used for many other decision support tasks. For example, users can find definitions of words by entering the word “define” followed by a word, such as “define extenuate.” Some of the other capabilities include
• Price lookups: “Price 64GB iPhone X.” • Currency conversions: “10 US dollars in euros.” • Sports scores and game times: Just enter the name of a team (“NYC Giants”), and Google
SMS will send the most recent game’s score and the date and time of the next match.
This type of SMS-based search capability is also available for other search engines such as Microsoft’s search engine Bing.
With the emergence of smartphones such as Apple’s iPhone and Android smartphones from many vendors, many companies are developing apps to provide purchasing-decision support. For example, Amazon’s app allows a user to take a picture of any item in a store (or wherever) and send it to Amazon.com. Amazon.com’s graphics-understanding al- gorithm tries to match the image to a real product in its databases and sends the user a page similar to Amazon.com’s product info pages, allowing users to perform price com- parisons in real time. Millions of other apps have been developed that provide consumers support for decision making on finding and selecting stores/restaurants/service providers on the basis of location, recommendations from others, and especially from your own so- cial circles. Search activities noted in the previous paragraph are also largely accomplished now through apps provided by each search provider.
Voice input for these devices and the new smart speakers such as Amazon Echo (Alexa) and Google Home is common and fairly accurate (but not perfect). When voice input with accompanying speech-recognition software (and readily available text-to- speech software) is used, verbal instructions with accompanied actions and outputs can be invoked. These are readily available for DSS and are incorporated into the portable devices described earlier. An example of voice inputs that can be used for a general- purpose DSS is Apple’s Siri application and Google’s Google Now service. For example, a user can give her or his zip code and say “pizza delivery.” These devices provide the search results and can even place a call to a business.
The Knowledge-Based Management Subsystem
Many of the user interface developments are closely tied to the major new advances in their knowledge-based systems. The knowledge-based management subsystem can support any of the other subsystems or act as an independent component. It provides intelligence to aug- ment the decision maker’s own or to help understand a user’s query so as to provide a consis- tent answer. It can be interconnected with the organization’s knowledge repository (part of a KMS), which is sometimes called the organizational knowledge base, or connect to thousands of external knowledge sources. Many artificial intelligence methods have been implemented in the current generation of learning systems and are easy to integrate into the other DSS com- ponents. One of the most widely publicized knowledge-based DSS is IBM’s Watson, which was introduced in the opening vignette and will be described in more detail later.
This section has covered the history and progression of Decision Support Systems in brief. In the next section we discuss evolution of this support to business intelligence, analytics, and data science.
22 Part I • Introduction to Analytics and AI
u SECTION 1.3 REVIEW QUESTIONS
1. List and briefly describe Simon’s four phases of decision making. 2. What is the difference between a problem and its symptoms? 3. Why is it important to classify a problem? 4. Define implementation. 5. What are structured, unstructured, and semistructured decisions? Provide two exam-
ples of each.
6. Define operational control, managerial control, and strategic planning. Provide two examples of each.
7. What are the nine cells of the decision framework? Explain what each is for. 8. How can computers provide support for making structured decisions? 9. How can computers provide support for making semistructured and unstructured
decisions?
1.4 EVOLUTION OF COMPUTERIZED DECISION SUPPORT TO BUSINESS INTELLIGENCE/ANALYTICS/DATA SCIENCE
The timeline in Figure 1.5 shows the terminology used to describe analytics since the 1970s. During the 1970s, the primary focus of information systems support for decision making focused on providing structured, periodic reports that a manager could use for decision making (or ignore them). Businesses began to create routine reports to inform decision makers (managers) about what had happened in the previous period (e.g., day, week, month, quarter). Although it was useful to know what had happened in the past, managers needed more than this: They needed a variety of reports at different levels of granularity to better understand and address changing needs and challenges of the busi- ness. These were usually called management information systems (MIS). In the early 1970s, Scott-Morton first articulated the major concepts of DSS. He defined DSS as “inter- active computer-based systems, which help decision makers utilize data and models to solve unstructured problems” (Gorry and Scott-Morton, 1971). The following is another classic DSS definition provided by Keen and Scott-Morton (1978):
Decision support systems couple the intellectual resources of individuals with the capabilities of the computer to improve the quality of decisions. It is a computer-based support system for management decision makers who deal with semistructured problems.
Big Data AutomationAnalyticsBusiness IntelligenceEnterprise/Executive IS
Routine Reporting
AI/Expert System s
D ecision Support System
s
Relational D BM
S
O n-D
em and Static Reporting
Enterprise Resource Planning
D ata W
arehousing
D ashboards, Scorecards
Executive Inform ation System
s
Softw are as a Service
D ata/Text/W
eb M ining
Business Intelligence, BPM
Cloud, Big D ata Analytics
In-M em
ory/In-D atabase/M
PP
Social N etw
ork/M edia Analytics
Autom ated Analytics
Al/D eep Learning, loT/Sensors
Robotics, Sm art Robo-Assistants
Decision Support Systems
1970s 1980s 1990s 2000s 2010s 2020s
FIGURE 1.5 Evolution of Decision Support, Business Intelligence, Analytics, and AI.
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 23
Note that the term decision support system, like management information system and several other terms in the field of IT, is a content-free expression (i.e., it means dif- ferent things to different people). Therefore, there is no universally accepted definition of DSS.
During the early days of analytics, data were often obtained from the domain ex- perts using manual processes (i.e., interviews and surveys) to build mathematical or knowledge-based models to solve constrained optimization problems. The idea was to do the best with limited resources. Such decision support models were typically called operations research (OR). The problems that were too complex to solve optimally (using linear or nonlinear mathematical programming techniques) were tackled using heuristic methods such as simulation models. (We will introduce these as prescriptive analytics later in this chapter).
In the late 1970s and early 1980s, in addition to the mature OR models that were being used in many industries and government systems, a new and exciting line of mod- els had emerged: rule-based expert systems (ESs). These systems promised to capture ex- perts’ knowledge in a format that computers could process (via a collection of if-then-else rules or heuristics) so that these could be used for consultation much the same way that one would use domain experts to identify a structured problem and to prescribe the most probable solution. ESs allowed scarce expertise to be made available where and when needed, using an “intelligent” DSS.
The 1980s saw a significant change in the way organizations captured business- related data. The old practice had been to have multiple disjointed information systems tailored to capture transactional data of different organizational units or functions (e.g., accounting, marketing and sales, finance, manufacturing). In the 1980s, these systems were integrated as enterprise-level information systems that we now commonly call en- terprise resource planning (ERP) systems. The old mostly sequential and nonstandardized data representation schemas were replaced by relational database management (RDBM) systems. These systems made it possible to improve the capture and storage of data as well as the relationships between organizational data fields while significantly reducing the replication of information. The need for RDBM and ERP systems emerged when data integrity and consistency became an issue, significantly hindering the effectiveness of business practices. With ERP, all the data from every corner of the enterprise is collected and integrated into a consistent schema so that every part of the organization has access to the single version of the truth when and where needed. In addition to the emergence of ERP systems, or perhaps because of these systems, business reporting became an on- demand, as-needed business practice. Decision makers could decide when they needed to or wanted to create specialized reports to investigate organizational problems and opportunities.
In the 1990s, the need for more versatile reporting led to the development of execu- tive information systems (EISs; DSS designed and developed specifically for executives and their decision-making needs). These systems were designed as graphical dashboards and scorecards so that they could serve as visually appealing displays while focusing on the most important factors for decision makers to keep track of the key performance in- dicators. To make this highly versatile reporting possible while keeping the transactional integrity of the business information systems intact, it was necessary to create a middle data tier known as a DW as a repository to specifically support business reporting and decision making. In a very short time, most large- to medium-sized businesses adopted data warehousing as their platform for enterprise-wide decision making. The dashboards and scorecards got their data from a DW, and by doing so, they were not hindering the efficiency of the business transaction systems mostly referred to as ERP systems.
In the 2000s, the DW-driven DSS began to be called BI systems. As the amount of longitudinal data accumulated in the DWs increased, so did the capabilities of hardware
24 Part I • Introduction to Analytics and AI
and software to keep up with the rapidly changing and evolving needs of the decision makers. Because of the globalized competitive marketplace, decision makers needed current information in a very digestible format to address business problems and to take advantage of market opportunities in a timely manner. Because the data in a DW are up- dated periodically, they do not reflect the latest information. To elevate this information latency problem, DW vendors developed a system to update the data more frequently, which led to the terms real-time data warehousing and, more realistically, right-time data warehousing, which differs from the former by adopting a data-refreshing policy based on the needed freshness of the data items (i.e., not all data items need to be refreshed in real time). DWs are very large and feature rich, and it became necessary to “mine” the corporate data to “discover” new and useful knowledge nuggets to improve business pro- cesses and practices, hence, the terms data mining and text mining. With the increasing volumes and varieties of data, the needs for more storage and more processing power emerged. Although large corporations had the means to tackle this problem, small- to medium-sized companies needed more financially manageable business models. This need led to service-oriented architecture and software and infrastructure-as-a-service ana- lytics business models. Smaller companies, therefore, gained access to analytics capabili- ties on an as-needed basis and paid only for what they used, as opposed to investing in financially prohibitive hardware and software resources.
In the 2010s, we are seeing yet another paradigm shift in the way that data are captured and used. Largely because of the widespread use of the Internet, new data gen- eration mediums have emerged. Of all the new data sources (e.g., radio-frequency iden- tification [RFID] tags, digital energy meters, clickstream Web logs, smart home devices, wearable health monitoring equipment), perhaps the most interesting and challenging is social networking/social media. These unstructured data are rich in information content, but analysis of such data sources poses significant challenges to computational systems from both software and hardware perspectives. Recently, the term Big Data has been coined to highlight the challenges that these new data streams have brought on us. Many advancements in both hardware (e.g., massively parallel processing with very large com- putational memory and highly parallel multiprocessor computing systems) and software/ algorithms (e.g., Hadoop with MapReduce and NoSQL, Spark) have been developed to address the challenges of Big Data.
The last few years and the upcoming decade are bringing massive growth in many exciting dimensions. For example, streaming analytics and the sensor technologies have enabled the IoT. Artificial Intelligence is changing the shape of BI by enabling new ways of analyzing images through deep learning, not just traditional visualization of data. Deep learning and AI are also helping grow voice recognition and speech synthesis, leading to new interfaces in interacting with technologies. Almost half of U.S. households already have a smart speaker such as Amazon Echo or Google Home and have begun to interact with data and systems using voice interfaces. Growth in video interfaces will eventually enable gesture-based interaction with systems. All of these are being enabled due to massive cloud- based data storage and amazingly fast processing capabilities. And more is yet to come.
It is hard to predict what the next decade will bring and what the new analytics-related terms will be. The time between new paradigm shifts in information systems and particularly in analytics has been shrinking, and this trend will continue for the foreseeable future. Even though analytics is not new, the explosion in its popularity is very new. Thanks to the recent explosion in Big Data, ways to collect and store these data and intuitive software tools, data- driven insights are more accessible to business professionals than ever before. Therefore, in the midst of global competition, there is a huge opportunity to make better managerial decisions by using data and analytics to increase revenue while decreasing costs by building better products, improving customer experience, and catching fraud before it happens, im- proving customer engagement through targeting and customization, and developing entirely
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 25
new lines of business, all with the power of analytics and data. More and more companies are now preparing their employees with the know-how of business analytics to drive effec- tiveness and efficiency in their day-to-day decision-making processes.
The next section focuses on a framework for BI. Although most people would agree that BI has evolved into analytics and data science, many vendors and researchers still use that term. So the next few paragraphs pay homage to that history by specifically focusing on what has been called BI. Following the next section, we introduce analytics and use that as the label for classifying all related concepts.
A Framework for Business Intelligence
The decision support concepts presented in Sections 1.2 and 1.3 have been implemented incrementally, under different names, by many vendors that have created tools and meth- odologies for decision support. As noted in Section 1.2, as the enterprise-wide systems grew, managers were able to access user-friendly reports that enabled them to make deci- sions quickly. These systems, which were generally called EISs, then began to offer addi- tional visualization, alerts, and performance measurement capabilities. By 2006, the major commercial products and services appeared under the term business intelligence (BI).
DEFINITIONS OF BI Business intelligence (BI) is an umbrella term that combines architec- tures, tools, databases, analytical tools, applications, and methodologies. It is, like DSS, a content-free expression, so it means different things to different people. Part of the confu- sion about BI lies in the flurry of acronyms and buzzwords that are associated with it (e.g., business performance management [BPM]). BI’s major objective is to enable interactive access (sometimes in real time) to data, to enable manipulation of data, and to give busi- ness managers and analysts the ability to conduct appropriate analyses. By analyzing his- torical and current data, situations, and performances, decision makers get valuable insights that enable them to make more informed and better decisions. The process of BI is based on the transformation of data to information, then to decisions, and finally to actions.
A BRIEF HISTORY OF BI The term BI was coined by the Gartner Group in the mid-1990s. However, as the history in the previous section points out, the concept is much older; it has its roots in the MIS reporting systems of the 1970s. During that period, reporting sys- tems were static, were two dimensional, and had no analytical capabilities. In the early 1980s, the concept of EISs emerged. This concept expanded the computerized support to top-level managers and executives. Some of the capabilities introduced were dynamic mul- tidimensional (ad hoc or on-demand) reporting, forecasting and prediction, trend analysis, drill-down to details, status access, and critical success factors. These features appeared in dozens of commercial products until the mid-1990s. Then the same capabilities and some new ones appeared under the name BI. Today, a good BI-based enterprise information system contains all the information that executives need. So, the original concept of EIS was transformed into BI. By 2005, BI systems started to include artificial intelligence ca- pabilities as well as powerful analytical capabilities. Figure 1.6 illustrates the various tools and techniques that may be included in a BI system. It illustrates the evolution of BI as well. The tools shown in Figure 1.6 provide the capabilities of BI. The most sophisticated BI products include most of these capabilities; others specialize in only some of them.
The Architecture of BI
A BI system has four major components: a DW, with its source data; business analytics, a collection of tools for manipulating, mining, and analyzing the data in the DW; BPM for monitoring and analyzing performance; and a user interface (e.g., a dashboard). The re- lationship among these components is illustrated in Figure 1.7.
26 Part I • Introduction to Analytics and AI
The Origins and Drivers of BI
Where did modern approaches to DW and BI come from? What are their roots, and how do those roots affect the way organizations are managing these initiatives today? Today’s investments in information technology are under increased scrutiny in terms of their bottom-line impact and potential. The same is true of DW and the BI applications that make these initiatives possible.
Business Intelligence
Spreadsheets (MS Excel)
DSS
ETL
Data Warehouse
Data Marts
Metadata
Querying and Reporting
EIS/ESS
Broadcasting Tools
Portals
OLAP
Scorecards and Dashboards
Alerts and Notifications
Data and Text Mining Predictive
Analytics
Digital Cockpits and Dashboards
Workflow
Financial Reporting
FIGURE 1.6 Evolution of Business Intelligence (BI).
Technical staff
Build the data warehouse
- Organizing - Summarizing - Standardizing
Data warehouse
Business users
Access
Manipulation, results
Managers/executives
BPM strategies
Future component: Intelligent systems
User interface
- Browser - Portal - Dashboard
Data Warehouse Environment
Business Analytics Environment
Performance and Strategy
Data Sources
FIGURE 1.7 A High-Level Architecture of BI. Source: Based on W. Eckerson. (2003). Smart Companies in the 21st Century: The Secrets of Creating Successful Business Intelligent Solutions.
Seattle, WA: The Data Warehousing Institute, p. 32, Illustration 5.
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 27
Organizations are being compelled to capture, understand, and harness their data to support decision making to improve business operations. Legislation and regulation (e.g., the Sarbanes-Oxley Act of 2002) now require business leaders to document their business processes and to sign off on the legitimacy of the information they rely on and report to stakeholders. Moreover, business cycle times are now extremely compressed; faster, more informed, and better decision making is, therefore, a competitive impera- tive. Managers need the right information at the right time and in the right place. This is the mantra for modern approaches to BI.
Organizations have to work smart. Paying careful attention to the management of BI initiatives is a necessary aspect of doing business. It is no surprise, then, that organiza- tions are increasingly championing BI and under its new incarnation as analytics.
Data Warehouse as a Foundation for Business Intelligence
BI systems rely on a DW as the information source for creating insight and supporting managerial decisions. A multitude of organizational and external data is captured, trans- formed, and stored in a DW to support timely and accurate decisions through enriched business insight. In simple terms, a DW is a pool of data produced to support decision making; it is also a repository of current and historical data of potential interest to man- agers throughout the organization. Data are usually structured to be available in a form ready for analytical processing activities (i.e., OLAP, data mining, querying, reporting, and other decision support applications). A DW is a subject-oriented, integrated, time-variant, nonvolatile collection of data in support of management’s decision-making process.
Whereas a DW is a repository of data, data warehousing is literally the entire process. Data warehousing is a discipline that results in applications that provide decision support capability, allows ready access to business information, and creates business insight. The three main types of data warehouses are data marts (DMs), operational data stores (ODS), and enterprise data warehouses (EDW). Whereas a DW combines databases across an en- tire enterprise, a DM is usually smaller and focuses on a particular subject or department. A DM is a subset of a data warehouse, typically consisting of a single subject area (e.g., marketing, operations). An operational data store (ODS) provides a fairly recent form of customer information file. This type of database is often used as an interim staging area for a DW. Unlike the static contents of a DW, the contents of an ODS are updated throughout the course of business operations. An EDW is a large-scale data warehouse that is used across the enterprise for decision support. The large-scale nature of an EDW provides in- tegration of data from many sources into a standard format for effective BI and decision support applications. EDWs are used to provide data for many types of DSS, including CRM, supply chain management (SCM), BPM, business activity monitoring, product life- cycle management, revenue management, and sometimes even KMS.
In Figure 1.8, we show the DW concept. Data from many different sources can be extracted, transformed, and loaded into a DW for further access and analytics for decision support. Further details of DW are available in an online chapter on the book’s Web site.
Transaction Processing versus Analytic Processing
To illustrate the major characteristics of BI, first we will show what BI is not—namely, transaction processing. We are all familiar with the information systems that support our transactions, like ATM withdrawals, bank deposits, and cash register scans at the grocery store. These transaction processing systems are constantly involved in handling updates to what we might call operational databases. For example, in an ATM withdrawal transac- tion, we need to reduce our bank balance accordingly; a bank deposit adds to an account; and a grocery store purchase is likely reflected in the store’s calculation of total sales for the day, and it should reflect an appropriate reduction in the store’s inventory for the items we bought, and so on. These online transaction processing (OLTP) systems handle a
28 Part I • Introduction to Analytics and AI
company’s routine ongoing business. In contrast, a DW is typically a distinct system that provides storage for data that will be used for analysis. The intent of that analysis is to give management the ability to scour data for information about the business, and it can be used to provide tactical or operational decision support whereby, for example, line per- sonnel can make quicker and/or more informed decisions. DWs are intended to work with informational data used for online analytical processing (OLAP) systems.
Most operational data in ERP systems—and in their complementary siblings like SCM or CRM—are stored in an OLTP system, which is a type of computer processing where the computer responds immediately to user requests. Each request is considered to be a transaction, which is a computerized record of a discrete event, such as the receipt of inventory or a customer order. In other words, a transaction requires a set of two or more database updates that must be completed in an all-or-nothing fashion.
The very design that makes an OLTP system efficient for transaction processing makes it inefficient for end-user ad hoc reports, queries, and analysis. In the 1980s, many business users referred to their mainframes as “black holes” because all the information went into them, but none ever came back. All requests for reports had to be programmed by the IT staff, whereas only “precanned” reports could be generated on a scheduled basis, and ad hoc real-time querying was virtually impossible. Although the client/server-based ERP sys- tems of the 1990s were somewhat more report friendly, they have still been a far cry from a desired usability by regular, nontechnical end users for things such as operational reporting and interactive analysis. To resolve these issues, the notions of DW and BI were created.
DWs contain a wide variety of data that present a coherent picture of business con- ditions at a single point in time. The idea was to create a database infrastructure that was always online and contained all the information from the OLTP systems, including histori- cal data, but reorganized and structured in such a way that it was fast and efficient for querying, analysis, and decision support. Separating the OLTP from analysis and decision support enables the benefits of BI that were described earlier.
A Multimedia Exercise in Business Intelligence
TUN includes videos (similar to the television show CSI) to illustrate concepts of analytics in different industries. These are called “BSI Videos (Business Scenario Investigations).” Not only are these entertaining, but they also provide the class with some questions for discussion. For starters, please go to https://www.teradatauniversitynetwork.com/ Library/Items/BSI-The-Case-of-the-Misconnecting-Passengers/ or www.youtube.
Data Marts
Applications (Visualization)
Data/Text Mining
OLAP, Dashboard, Web
Routine Business Reporting
Custom-Built Applications
Data Sources
POS
Other OLTP/Web
External Data
ETL Process
Select
Extract
Transform
Integrate
Load
Metadata
Enterprise Data
Warehouse
Replication
Data mart (Marketing)
Data mart (Operations)
Data mart (Finance)
Data mart (...)
No data mart options
ERP
Legacy
A P
I/ M
id d le
w a re
FIGURE 1.8 Data Warehouse Framework and Views.
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 29
com/watch?v=NXEL5F4_aKA. Watch the video that appears on YouTube. Essentially, you have to assume the role of a customer service center professional. An incoming flight is run- ning late, and several passengers are likely to miss their connecting flights. There are seats on one outgoing flight that can accommodate two of the four passengers. Which two passengers should be given priority? You are given information about customers’ profiles and relationships with the airline. Your decisions might change as you learn more about those customers’ profiles.
Watch the video, pause it as appropriate, and answer the questions on which pas- sengers should be given priority. Then resume the video to get more information. After the video is complete, you can see the slides related to this video and how the analy- sis was prepared on a slide set at www.slideshare.net/teradata/bsi-how-we-did-it- the-case-of-the-misconnecting-passengers.
This multimedia excursion provides an example of how additional available information through an enterprise DW can assist in decision making.
Although some people equate DSS with BI, these systems are not, at present, the same. It is interesting to note that some people believe that DSS is a part of BI—one of its analytical tools. Others think that BI is a special case of DSS that deals mostly with re- porting, communication, and collaboration (a form of data-oriented DSS). Another expla- nation (Watson, 2005) is that BI is a result of a continuous revolution, and as such, DSS is one of BI’s original elements. Further, as noted in the next section onward, in many circles, BI has been subsumed by the new terms analytics or data science.
APPROPRIATE PLANNING AND ALIGNMENT WITH THE BUSINESS STRATEGY First and foremost, the fundamental reasons for investing in BI must be aligned with the company’s business strategy. BI cannot simply be a technical exercise for the information systems department. It has to serve as a way to change the manner in which the company con- ducts business by improving its business processes and transforming decision- making processes to be more data driven. Many BI consultants and practitioners involved in suc- cessful BI initiatives advise that a framework for planning is a necessary precondition. One framework, proposed by Gartner, Inc. (2004), decomposed planning and execution into business, organization, functionality, and infrastructure components. At the busi- ness and organizational levels, strategic and operational objectives must be defined while considering the available organizational skills to achieve those objectives. Issues of orga- nizational culture surrounding BI initiatives and building enthusiasm for those initiatives and procedures for the intra-organizational sharing of BI best practices must be consid- ered by upper management—with plans in place to prepare the organization for change. One of the first steps in that process is to assess the IS organization, the skill sets of the potential classes of users, and whether the culture is amenable to change. From this as- sessment, and assuming there are justification and the need to move ahead, a company can prepare a detailed action plan. Another critical issue for BI implementation success is the integration of several BI projects (most enterprises use several BI projects) among themselves and with the other IT systems in the organization and its business partners.
Gartner and many other analytics consulting organizations promoted the concept of a BI competence center that would serve the following functions:
• A center can demonstrate how BI is clearly linked to strategy and execution of strategy. • A center can serve to encourage interaction between the potential business user
communities and the IS organization. • A center can serve as a repository and disseminator of best BI practices between
and among the different lines of business. • Standards of excellence in BI practices can be advocated and encouraged through-
out the company. • The IS organization can learn a great deal through interaction with the user communi-
ties, such as knowledge about the variety of types of analytical tools that are needed.
30 Part I • Introduction to Analytics and AI
• The business user community and IS organization can better understand why the DW platform must be flexible enough to provide for changing business requirements.
• The center can help important stakeholders like high-level executives see how BI can play an important role.
Over the last 10 years, the idea of a BI competence center has been abandoned because many advanced technologies covered in this book have reduced the need for a central group to organize many of these functions. Basic BI has now evolved to a point where much of it can be done in “self-service” mode by the end users. For example, many data visualizations are easily accomplished by end users using the latest visualization pack- ages (Chapter 3 will introduce some of these). As noted by Duncan (2016), the BI team would now be more focused on producing curated data sets to enable self- service BI. Because analytics is now permeating across the whole organization, the BI competency center could evolve into an analytics community of excellence to promote best practices and ensure overall alignment of analytics initiatives with organizational strategy.
BI tools sometimes needed to be integrated among themselves, creating synergy. The need for integration pushed software vendors to continuously add capabilities to their products. Customers who buy an all-in-one software package deal with only one vendor and do not have to deal with system connectivity. But they may lose the advan- tage of creating systems composed from the “best-of-breed” components. This led to major chaos in the BI market space. Many of the software tools that rode the BI wave (e.g., Savvion, Vitria, Tibco, MicroStrategy, Hyperion) have either been acquired by other companies or have expanded their offerings to take advantage of six key trends that have emerged since the initial wave of surge in business intelligence:
• Big Data. • Focus on customer experience as opposed to just operational efficiency. • Mobile and even newer user interfaces—visual, voice, mobile. • Predictive and prescriptive analytics, machine learning, artificial intelligence. • Migration to cloud. • Much greater focus on security and privacy protection.
This book covers many of these topics in significant detail by giving examples of how the technologies are evolving and being applied, and the managerial implications.
u SECTION 1.4 REVIEW QUESTIONS
1. List three of the terms that have been predecessors of analytics. 2. What was the primary difference between the systems called MIS, DSS, and Executive
Information Systems?
3. Did DSS evolve into BI or vice versa? 4. Define BI. 5. List and describe the major components of BI. 6. Define OLTP. 7. Define OLAP. 8. List some of the implementation topics addressed by Gartner’s report. 9. List some other success factors of BI.
1.5 ANALYTICS OVERVIEW
The word analytics has largely replaced the previous individual components of comput- erized decision support technologies that have been available under various labels in the past. Indeed, many practitioners and academics now use the word analytics in place of BI. Although many authors and consultants have defined it slightly differently, one can
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 31
view analytics as the process of developing actionable decisions or recommendations for actions based on insights generated from historical data. According to the Institute for Operations Research and Management Science (INFORMS), analytics represents the com- bination of computer technology, management science techniques, and statistics to solve real problems. Of course, many other organizations have proposed their own interpreta- tions and motivations for analytics. For example, SAS Institute Inc. proposed eight levels of analytics that begin with standardized reports from a computer system. These reports essentially provide a sense of what is happening with an organization. Additional technolo- gies have enabled us to create more customized reports that can be generated on an ad hoc basis. The next extension of reporting takes us to OLAP-type queries that allow a user to dig deeper and determine specific sources of concern or opportunities. Technologies available today can also automatically issue alerts for a decision maker when performance warrants such alerts. At a consumer level, we see such alerts for weather or other issues. But similar alerts can also be generated in specific settings when sales fall above or below a certain level within a certain time period or when the inventory for a specific product is running low. All of these applications are made possible through analysis and queries of data being collected by an organization. The next level of analysis might entail statistical analysis to better understand patterns. These can then be taken a step further to develop forecasts or models for predicting how customers might respond to a specific marketing campaign or ongoing service/product offerings. When an organization has a good view of what is happening and what is likely to happen, it can also employ other techniques to make the best decisions under the circumstances.
This idea of looking at all the data to understand what is happening, what will happen, and how to make the best of it has also been encapsulated by INFORMS in pro- posing three levels of analytics. These three levels are identified as descriptive, predictive, and prescriptive. Figure 1.9 presents a graphical view of these three levels of analytics. It suggests that these three are somewhat independent steps and one type of analytics
Descriptive
What happened? What is happening?
Q u e s ti o n s
E n a b le
rs
Well-defined business problems and opportunities
O u tc
o m
e s
Business reporting Dashboards Scorecards Data warehousing
Business Analytics
Predictive
What will happen? Why will it happen?
Prescriptive
What should I do? Why should I do it?
Accurate projections of future events and
outcomes
Best possible business decisions
and actions
Data mining Text mining Web/media mining Forecasting
Optimization Simulation Decision modeling Expert systems
FIGURE 1.9 Three Types of Analytics.
32 Part I • Introduction to Analytics and AI
applications leads to another. It also suggests that there is actually some overlap across these three types of analytics. In either case, the interconnected nature of different types of analytics applications is evident. We next introduce these three levels of analytics.
Descriptive Analytics
Descriptive (or reporting) analytics refers to knowing what is happening in the or- ganization and understanding some underlying trends and causes of such occurrences. First, this involves the consolidation of data sources and availability of all relevant data in a form that enables appropriate reporting and analysis. Usually, the development of this data infrastructure is part of DWs. From this data infrastructure, we can develop ap- propriate reports, queries, alerts, and trends using various reporting tools and techniques.
A significant technology that has become a key player in this area is visualization. Using the latest visualization tools in the marketplace, we can now develop powerful in- sights in the operations of our organization. Application Cases 1.3 and 1.4 highlight some such applications.
Silvaris Corporation was founded in 2000 by a team of forest industry professionals to pro- vide technological advancement in the lumber and building material sector. Silvaris is the first e- commerce platform in the United States spe- cifically for forest products and is headquartered in Seattle, Washington. It is a leading wholesale provider of industrial wood products and surplus building materials.
Silvaris sells its products and provides interna- tional logistics services to more than 3,500 custom- ers. To manage various processes that are involved in a transaction, the company created a proprietary online trading platform to track information flow related to transactions between traders, accounting, credit, and logistics. This allowed Silvaris to share its real-time information with its customers and partners. But due to the rapidly changing prices of materials, it became necessary for Silvaris to get a real-time view of data without moving them into a separate reporting format.
Silvaris started using Tableau because of its abil- ity to connect with and visualize live data. With dash- boards created by Tableau that are easy to understand and explain, Silvaris started using it for reporting pur- poses. This helped Silvaris in pulling out informa- tion quickly from the data and identifying issues that impact its business. Silvaris succeeded in managing
online versus offline orders with the help of reports generated by Tableau. Now, Silvaris keeps track of online orders placed by customers and knows when to send renew pushes to which customers to keep them purchasing online. Also, analysts of Silvaris can save time by generating dashboards instead of writ- ing hundreds of pages of reports by using Tableau.
Sources: Tableau.com. “Silvaris Augments Proprietary Technology Platform with Tableau’s Real-Time Reporting Capabilities.” http:// www.tableau.com/sites/default/files/case-studies/silvaris- business-dashboards_0.pdf (accessed September 2018); Silvaris. com. http://www.silvaris.com (accessed September 2018).
Questions for Case 1.3
1. What was the challenge faced by Silvaris?
2. How did Silvaris solve its problem using data visualization with Tableau?
What We Can Learn from This Application Case
Many industries need to analyze data in real time. Real-time analysis enables the analysts to identify issues that impact their business. Visualization is sometimes the best way to begin analyzing the live data streams. Tableau is one such data visualization tool that has the capability to analyze live data with- out bringing live data into a separate reporting format.
Application Case 1.3 Silvaris Increases Business with Visual Analysis and Real-Time Reporting Capabilities
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 33
Predictive Analytics
Predictive analytics aims to determine what is likely to happen in the future. This analysis is based on statistical techniques as well as other more recently developed techniques that fall under the general category of data mining. The goal of these techniques is to be able to predict whether the customer is likely to switch to a com- petitor (“churn”), what and how much the customer would likely buy next, what promotions the customer would respond to, whether the customer is a creditworthy risk, and so forth. A number of techniques are used in developing predictive analytical applications, including various classification algorithms. For example, as described in Chapters 4 and 5, we can use classification techniques such as logistic regression, de- cision tree models, and neural networks to predict how well a motion picture will do at the box office. We can also use clustering algorithms for segmenting customers into different clusters to be able to target specific promotions to them. Finally, we can use association mining techniques (Chapters 4 and 5) to estimate relationships between different purchasing behaviors. That is, if a customer buys one product, what else is the customer likely to purchase? Such analysis can assist a retailer in recommending or promoting related products. For example, any product search on Amazon.com results in the retailer also suggesting similar products that a customer may be interested in. We will study these techniques and their applications in Chapters 3 through 6. Application Case 1.5 illustrates one such application in sports.
Siemens is a German company headquartered in Berlin, Germany. It is one of the world’s largest companies focusing on the areas of electrification, automation, and digitalization. It has an annual rev- enue of 76 billion euros.
The visual analytics group of Siemens is tasked with end-to-end reporting solutions and consulting for all of Siemens internal BI needs. This group was fac- ing the challenge of providing reporting solutions to the entire Siemens organization across different depart- ments while maintaining a balance between gover- nance and self-service capabilities. Siemens needed a platform that could analyze its multiple cases of cus- tomer satisfaction surveys, logistic processes, and finan- cial reporting. This platform should be easy to use for their employees so that they could use these data for analysis and decision making. In addition, the platform should be easily integrated with existing Siemens sys- tems and give employees a seamless user experience.
Siemens started using Dundas BI, a leading global provider of BI and data visualization solutions. It allowed Siemens to create highly interactive dash- boards that enabled it to detect issues early and thus save a significant amount of money. The dashboards developed by Dundas BI helped Siemens global
logistics organization answer questions like how dif- ferent supply rates at different locations affect the operation, thus helping the company reduce cycle time by 12 percent and scrap cost by 25 percent.
Questions for Case 1.4
1. What challenges were faced by Siemens visual analytics group?
2. How did the data visualization tool Dundas BI help Siemens in reducing cost?
What We Can Learn from This Application Case
Many organizations want tools that can be used to analyze data from multiple divisions. These tools can help them improve performance and make data discovery transparent to their users so that they can identify issues within the business easily.
Sources: Dundas.com. “How Siemens Drastically Reduced Cost with Managed BI Applications.” https://www.dundas.com/Content/ pdf/siemens-case-study.pdf (accessed September 2018); Wikipedia. org. “SIEMENS.” https://en.wikipedia.org/wiki/Siemens (ac- cessed September 2018); Siemens.com. “About Siemens.” http:// www.siemens.com/about/en/ (accessed September 2018).
Application Case 1.4 Siemens Reduces Cost with the Use of Data Visualization
34 Part I • Introduction to Analytics and AI
Any athletic activity is prone to injuries. If the inju- ries are not handled properly, then the team suffers. Using analytics to understand injuries can help in deriving valuable insights that would enable coaches and team doctors to manage the team composition, understand player profiles, and ultimately aid in bet- ter decision making concerning which players might be available to play at any given time.
In an exploratory study, Oklahoma State University analyzed U.S. football-related sports inju- ries by using reporting and predictive analytics. The project followed the Cross-Industry Standard Process for Data Mining (CRISP-DM) methodology (to be described in Chapter 4) to understand the problem of making recommendations on managing injuries, understanding the various data elements collected about injuries, cleaning the data, developing visual- izations to draw various inferences, building predic- tive models to analyze the injury healing time period, and drawing sequence rules to predict the relation- ships among the injuries and the various body part parts afflicted with injuries.
The injury data set consisted of more than 560 football injury records, which were categorized into injury-specific variables—body part/site/laterality, action taken, severity, injury type, injury start and healing dates—and player/sport-specific variables— player ID, position played, activity, onset, and game location. Healing time was calculated for each record, which was classified into different sets of time periods: 0–1 month, 1–2 months, 2–4 months, 4–6 months, and 6–24 months.
Various visualizations were built to draw infer- ences from injury–data set information depicting the healing time period associated with players’ positions, severity of injuries and the healing time period, treat- ment offered and the associated healing time period, major injuries afflicting body parts, and so forth.
Neural network models were built to pre- dict each of the healing categories using IBM SPSS
Modeler. Some of the predictor variables were current status of injury, severity, body part, body site, type of injury, activity, event location, action taken, and position played. The success of classifying the healing category was quite good: Accuracy was 79.6 percent. Based on the analysis, many recommendations were suggested, including employing more specialists’ input from injury onset instead of letting the training room staff screen the injured players; training players at defensive positions to avoid being injured; and hold- ing practice to thoroughly safety-check mechanisms.
Sources: “Sharda, R., Asamoah, D., & Ponna, N. (2013). “Research and Pedagogy in Business Analytics: Opportunities and Illustrative Examples.” Journal of Computing and Information Technology, 21(3), pp. 171–182.
Questions for Case 1.5
1. What types of analytics are applied in the injury analysis?
2. How do visualizations aid in understanding the data and delivering insights into the data?
3. What is a classification problem?
4. What can be derived by performing sequence analysis?
What We Can Learn from This Application Case
For any analytics project, it is always important to understand the business domain and the cur- rent state of the business problem through exten- sive analysis of the only resource—historical data. Visualizations often provide a great tool for gaining the initial insights into data, which can be further refined based on expert opinions to identify the rela- tive importance of the data elements related to the problem. Visualizations also aid in generating ideas for obscure problems, which can be pursued in building PMs that could help organizations in deci- sion making.
Application Case 1.5 Analyzing Athletic Injuries
Prescriptive Analytics
The third category of analytics is termed prescriptive analytics. The goal of prescriptive analytics is to recognize what is going on as well as the likely forecast and make deci- sions to achieve the best performance possible. This group of techniques has historically been studied under the umbrella of OR or management sciences and is generally aimed at
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 35
optimizing the performance of a system. The goal here is to provide a decision or a recom- mendation for a specific action. These recommendations can be in the form of a specific yes/no decision for a problem, a specific amount (say, price for a specific item or airfare to charge), or a complete set of production plans. The decisions may be presented to a deci- sion maker in a report or may be used directly in an automated decision rules system (e.g., in airline pricing systems). Thus, these types of analytics can also be termed decision or normative analytics. Application Case 1.6 gives an example of such prescriptive analytic applications. We will learn about some aspects of prescriptive analytics in Chapter 8.
ANALYTICS APPLIED TO DIFFERENT DOMAINS Applications of analytics in various in- dustry sectors have spawned many related areas or at least buzzwords. It is almost fashionable to attach the word analytics to any specific industry or type of data. Besides the general category of text analytics—aimed at getting value out of text (to be studied in Chapter 7)—or Web analytics—analyzing Web data streams (also in
This application case is based on a project that involved one of the coauthors A company that does not wish to disclose its name (or even its precise industry) was facing a major problem of making decisions on which inventory of raw materials to use to satisfy which customers. This company sup- plies custom configured steel bars to its customers. These bars may be cut into specific shapes or sizes and may have unique material and finishing require- ments. The company procures raw materials from around the world and stores them in its warehouse. When a prospective customer calls the company to request a quote for the specialty bars meeting spe- cific material requirements (composition, origin of the metal, quality, shapes, sizes, etc.), the salesper- son usually has just a little bit of time to submit such a quote including the date when the product can be delivered and, of course, prices, and so on. It must make available-to-promise (ATP) decisions, which determine in real time the dates when the salesper- son can promise delivery of products that customers requested during the quotation stage. Previously, a salesperson had to make such decisions by analyz- ing reports on available inventory of raw materials. Some of the available raw material may have already been committed to another customer’s order. Thus, the inventory in stock might not really be inven- tory available. On the other hand, there may be raw material that is expected to be delivered in the near future that could also be used for satisfying the order
from this prospective customer. Finally, there might even be an opportunity to charge a premium for a new order by repurposing previously committed inventory to satisfy this new order while delaying an already committed order. Of course, such deci- sions should be based on the cost–benefit analyses of delaying a previous order. The system should thus be able to pull real-time data about inventory, committed orders, incoming raw material, produc- tion constraints, and so on.
To support these ATP decisions, a real-time DSS was developed to find an optimal assignment of the available inventory and to support additional what-if analysis. The DSS uses a suite of mixed- integer pro- gramming models that are solved using commercial software. The company has incorporated the DSS into its enterprise resource planning system to seam- lessly facilitate its use of business analytics.
Questions for Case 1.6
1. Why would reallocation of inventory from one customer to another be a major issue for discussion?
2. How could a DSS help make these decisions?
Source: M. Pajouh Foad, D. Xing, S. Hariharan, Y. Zhou, B. Balasundaram, T. Liu, & R. Sharda, R. (2013). “Available-to-Promise in Practice: An Application of Analytics in the Specialty Steel Bar Products Industry.” Interfaces, 43(6), pp. 503–517. http://dx.doi. org/10.1287/inte.2013.0693 (accessed September 2018).
Application Case 1.6 A Specialty Steel Bar Company Uses Analytics to Determine Available-to-Promise Dates
36 Part I • Introduction to Analytics and AI
Chapter 7)—many industry- or problem-specific analytics professions/streams have been developed. Examples of such areas are marketing analytics, retail analytics, fraud analytics, transportation analytics, health analytics, sports analytics, talent ana- lytics, behavioral analytics, and so forth. For example, we will soon see several appli- cations in sports analytics. Application Case 1.5 could also be termed a case study in health analytics. The next section will introduce health analytics and market analytics broadly. Literally, any systematic analysis of data in a specific sector is being labeled as “(fill-in-blanks)” analytics. Although this may result in overselling the concept of analytics, the benefit is that more people in specific industries are aware of the power and potential of analytics. It also provides a focus to professionals developing and ap- plying the concepts of analytics in a vertical sector. Although many of the techniques to develop analytics applications may be common, there are unique issues within each vertical segment that influence how the data may be collected, processed, ana- lyzed, and the applications implemented. Thus, the differentiation of analytics based on a vertical focus is good for the overall growth of the discipline.
ANALYTICS OR DATA SCIENCE? Even as the concept of analytics is receiving more at- tention in industry and academic circles, another term has already been introduced and is becoming popular. The new term is data science. Thus, the practitioners of data sci- ence are data scientists. D. J. Patil of LinkedIn is sometimes credited with creating the term data science. There have been some attempts to describe the differences between data analysts and data scientists (e.g., see “Data Science Revealed,” 2018) (emc.com/ collateral/about/news/emc-data-science-study-wp.pdf). One view is that data analyst is just another term for professionals who were doing BI in the form of data compila- tion, cleaning, reporting, and perhaps some visualization. Their skill sets included Excel use, some SQL knowledge, and reporting. You would recognize those capabilities as descriptive or reporting analytics. In contrast, data scientists are responsible for predic- tive analysis, statistical analysis, and use of more advanced analytical tools and algo- rithms. They may have a deeper knowledge of algorithms and may recognize them under various labels—data mining, knowledge discovery, or machine learning. Some of these professionals may also need deeper programming knowledge to be able to write code for data cleaning/analysis in current Web-oriented languages such as Java or Python and statistical languages such as R. Many analytics professionals also need to build signifi- cant expertise in statistical modeling, experimentation, and analysis. Again, our readers should recognize that these fall under the predictive and prescriptive analytics umbrella. However, prescriptive analytics also includes more significant expertise in OR including optimization, simulation, and decision analysis. Those who cover these fields are more likely to be called data scientists than analytics professionals.
Our view is that the distinction between analytics professional and data scientist is more of a degree of technical knowledge and skill sets than functions. It may also be more of a distinction across disciplines. Computer science, statistics, and applied mathematics programs appear to prefer the data science label, reserving the analytics label for more business-oriented professionals. As another example of this, applied physics professionals have proposed using network science as the term for describing analytics that relate to groups of people—social networks, supply chain networks, and so forth. See http://barabasi.com/networksciencebook/ for an evolving textbook on this topic.
Aside from a clear difference in the skill sets of professionals who only have to do descriptive/reporting analytics versus those who engage in all three types of analyt- ics, the distinction between the two labels is fuzzy at best. We observe that graduates of our analytics programs tend to be responsible for tasks that are more in line with data
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 37
science professionals (as defined by some circles) than just reporting analytics. This book is clearly aimed at introducing the capabilities and functionality of all analytics (which include data science), not just reporting analytics. From now on, we will use these terms interchangeably.
WHAT IS BIG DATA? Any book on analytics and data science has to include significant coverage of what is called Big Data analytics. We cover it in Chapter 9 but here is a very brief introduction. Our brains work extremely quickly and efficiently and are ver- satile in processing large amounts of all kinds of data: images, text, sounds, smells, and video. We process all different forms of data relatively easily. Computers, on the other hand, are still finding it hard to keep up with the pace at which data are generated, let alone analyze them quickly. This is why we have the problem of Big Data. So, what is Big Data? Simply put, Big Data refers to data that cannot be stored in a single storage unit. Big Data typically refers to data that come in many different forms: structured, un- structured, in a stream, and so forth. Major sources of such data are clickstreams from Web sites, postings on social media sites such as Facebook, and data from traffic, sensors, or weather. A Web search engine such as Google needs to search and index billions of Web pages to give you relevant search results in a fraction of a second. Although this is not done in real time, generating an index of all the Web pages on the Internet is not an easy task. Luckily for Google, it was able to solve this problem. Among other tools, it has employed Big Data analytical techniques.
There are two aspects to managing data on this scale: storing and processing. If we could purchase an extremely expensive storage solution to store all this at one place on one unit, making this unit fault tolerant would involve a major expense. An ingenious solution was proposed that involved storing these data in chunks on different machines connected by a network—putting a copy or two of this chunk in different locations on the network, both logically and physically. It was originally used at Google (then called the Google File System) and later developed and released by an Apache project as the Hadoop Distributed File System (HDFS).
However, storing these data is only half of the problem. Data are worthless if they do not provide business value, and for them to provide business value, they must be analyzed. How can such vast amounts of data be analyzed? Passing all computation to one powerful computer does not work; this scale would create a huge overhead on such a powerful computer. Another ingenious solution was proposed: Push computa- tion to the data instead of pushing data to a computing node. This was a new paradigm and gave rise to a whole new way of processing data. This is what we know today as the MapReduce programming paradigm, which made processing Big Data a reality. MapReduce was originally developed at Google, and a subsequent version was released by the Apache project called Hadoop MapReduce.
Today, when we talk about storing, processing, or analyzing Big Data, HDFS and MapReduce are involved at some level. Other relevant standards and software solutions have been proposed. Although the major toolkit is available as an open source, several companies have been launched to provide training or specialized analytical hardware or software services in this space. Some examples are HortonWorks, Cloudera, and Teradata Aster.
Over the past few years, what was called Big Data changed more and more as Big Data applications appeared. The need to process data coming in at a rapid rate added ve- locity to the equation. An example of fast data processing is algorithmic trading. This uses electronic platforms based on algorithms for trading shares on the financial market, which operates in microseconds. The need to process different kinds of data added variety to the equation. Another example of a wide variety of data is sentiment analysis, which
38 Part I • Introduction to Analytics and AI
uses various forms of data from social media platforms and customer responses to gauge sentiments. Today, Big Data is associated with almost any kind of large data that have the characteristics of volume, velocity, and variety. As noted before, these are evolving quickly to encompass stream analytics, IoT, cloud computing, and deep learning– enabled AI. We will study these in various chapters in the book.
u SECTION 1.5 REVIEW QUESTIONS
1. Define analytics. 2. What is descriptive analytics? What are the various tools that are employed in descrip-
tive analytics?
3. How is descriptive analytics different from traditional reporting? 4. What is a DW? How can DW technology help enable analytics? 5. What is predictive analytics? How can organizations employ predictive analytics? 6. What is prescriptive analytics? What kinds of problems can be solved by prescriptive
analytics?
7. Define modeling from the analytics perspective. 8. Is it a good idea to follow a hierarchy of descriptive and predictive analytics before
applying prescriptive analytics?
9. How can analytics aid in objective decision making? 10. What is Big Data analytics? 11. What are the sources of Big Data? 12. What are the characteristics of Big Data? 13. What processing technique is applied to process Big Data?
1.6 ANALYTICS EXAMPLES IN SELECTED DOMAINS
You will see examples of analytics applications throughout various chapters. That is one of the primary approaches (exposure) of this book. In this section, we highlight three ap- plication areas—sports, healthcare, and retail—where there have been the most reported applications and successes.
Sports Analytics—An Exciting Frontier for Learning and Understanding Applications of Analytics
The application of analytics to business problems is a key skill, one that you will learn in this book. Many of these techniques are now being applied to improve decision making in all aspects of sports, a very hot area called sports analytics. It is the art and science of gathering data about athletes and teams to create insights that improve sports decisions, such as deciding which players to recruit, how much to pay them, who to play, how to train them, how to keep them healthy, and when they should be traded or retired. For teams, it involves business decisions such as ticket pricing as well as roster decisions, analysis of each competitor’s strengths and weaknesses, and many game-day decisions.
Indeed, sports analytics is becoming a specialty within analytics. It is an important area because sport is a big business, generating about $145 billion in revenues each year plus an additional $100 billion in legal and $300 billion in illegal gambling, accord- ing to Price Waterhouse (“Changing the Game: Outlook for the Global Sports Market to 2015” (2015)). In 2014, only $125 million was spent on analytics (less than 0.1 percent
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 39
of revenues). This is expected to grow at a healthy rate to $4.7 billion by 2021 (“Sports Analytics Market Worth $4.7B by 2021” (2015)).
The use of analytics for sports was popularized by the Moneyball book by Michael Lewis in 2003 and the movie starring Brad Pitt in 2011. It showcased Oakland A’s general manager Billy Beane and his use of data and analytics to turn a losing team into a winner. In particular, he hired an analyst who used analytics to draft players who were able to get on base as opposed to players who excelled at traditional measures like runs batted in or stolen bases. These insights allowed the team to draft prospects overlooked by other teams at reasonable starting salaries. It worked—the team made it to the playoffs in 2002 and 2003.
Now analytics are being used in all parts of sports. The analytics can be divided between the front office and back office. A good description with 30 examples appears in Tom Davenport’s survey article (). Front-office business analytics include analyzing fan behavior ranging from predictive models for season ticket renewals and regular ticket sales to scoring tweets by fans regarding the team, athletes, coaches, and owners. This is very similar to traditional CRM. Financial analysis is also a key area such as when salary cap (for pros) or scholarship (for colleges) limits are part of the equation.
Back-office uses include analysis of both individual athletes and team play. For in- dividual players, there is a focus on recruitment models and scouting analytics, analytics for strength and fitness as well as development, and PMs for avoiding overtraining and injuries. Concussion research is a hot field. Team analytics include strategies and tactics, competitive assessments, and optimal roster choices under various on-field or on-court situations.
The following representative examples illustrate how two sports organizations use data and analytics to improve sports operations in the same way that analytics have im- proved traditional industry decision making.
Example 1: The Business Office
Dave Ward works as a business analyst for a major pro baseball team, focusing on rev- enue. He analyzes ticket sales, both from season ticket holders and single-ticket buyers. Sample questions in his area of responsibility include why season ticket holders renew (or do not renew) their tickets as well as what factors drive last-minute individual seat ticket purchases. Another question is how to price the tickets.
Some of the analytical techniques Dave uses include simple statistics on fan be- havior such as overall attendance and answers to survey questions about likelihood to purchase again. However, what fans say versus what they do can be different. Dave runs a survey of fans by ticket seat location (“tier”) and asks about their likelihood of renew- ing their season tickets. But when he compares what they say versus what they do, he discovers big differences. (See Figure 1.10.) He found that 69 percent of fans in Tier 1 seats who said on the survey that they would “probably not renew” actually did. This
Tier Highly Likely Likely Maybe Probably Not Certainly Not
1 92 88 75 69 45
2 88 81 70 65 38
3 80 76 68 55 36
4 77 72 65 45 25
5 75 70 60 35 25
FIGURE 1.10 Season Ticket Renewals—Survey Scores.
40 Part I • Introduction to Analytics and AI
is useful insight that leads to action—customers in the green cells are the most likely to renew tickets and so require fewer marketing touches and dollars to convert compared to customers in the blue cells.
However, many factors influence fan ticket purchase behavior, especially price, which drives more sophisticated statistics and data analysis. For both areas, but especially single-game tickets, Dave is driving the use of dynamic pricing—moving the business from simple static pricing by seat location tier to day-by-day up-and-down pricing of indi- vidual seats. This is a rich research area for many sports teams and has huge upside po- tential for revenue enhancement. For example, his pricing takes into account the team’s record, who they are playing, game dates and times, which star athletes play for each team, each fan’s history of renewing season tickets or buying single tickets, and factors such as seat location, number of seats, and real-time information like traffic congestion historically at game time and even the weather. See Figure 1.11.
Which of these factors are important and by how much? Given his extensive sta- tistics background, Dave builds regression models to pick out key factors driving these historic behaviors and create PMs to identify how to spend marketing resources to drive revenues. He builds churn models for season ticket holders to create segments of custom- ers who will renew, will not renew, or are fence-sitters, which then drives more refined marketing campaigns.
In addition, Dave does sentiment scoring on fan comments such as tweets that help him segment fans into different loyalty segments. Other studies about single-game atten- dance drivers help the marketing department understand the impact of giveaways like bobble-heads or T-shirts or suggestions on where to make spot TV ad buys.
Beyond revenues, there are many other analytical areas that Dave’s team works on, including merchandising, TV and radio broadcast revenues, inputs to the general manager on salary negotiations, draft analytics especially given salary caps, promotion effectiveness including advertising channels, and brand awareness, as well as partner analytics. He’s a very busy guy!
Seat Location
Team Performance
Time-Related Variables
Game Start Time
Part of the Season
Days before the Game
Home Team Performance in Past 10 Games
Opponent Made Playoffs Previous Year
Individual Player Reputations
Which Pitcher? What’s His Earned Run Average?
Number of All Stars on Opponent’s Roster
Opponent from Same Division
FIGURE 1.11 Dynamic Pricing Previous Work—Major League Baseball. Source: Based on C. Kemper and C. Breuer, “How Efficient is Dynamic Pricing for Sports Events? Designing a Dynamic Pricing Model
for Bayern Munich”, Intl. Journal of Sports Finance, 11, pp. 4–25, 2016.
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 41
Example 2: The Coach
Bob Breedlove is the football coach for a major college team. For him, everything is about winning games. His areas of focus include recruiting the best high school play- ers, developing them to fit his offense and defense systems, and getting maximum effort from them on game days. Sample questions in his area of responsibility include: Whom do we recruit? What drills help develop their skills? How hard do I push our athletes? Where are opponents strong or weak, and how do we figure out their play tendencies?
Fortunately, his team has hired a new team operations expert, Dar Beranek, who specializes in helping the coaches make tactical decisions. She is working with a team of student interns who are creating opponent analytics. They used the coach’s annotated game film to build a cascaded decision tree model (Figure 1.12) to predict whether the next play will be a running play or passing play. For the defensive coordinator, they have built heat maps (Figure 1.13) of each opponent’s passing offense, illustrating their tenden- cies to throw left or right and into which defensive coverage zones. Finally, they built some time-series analytics (Figure 1.14) on explosive plays (defined as a gain of more than 16 yards for a passing play or more than 12 yards for a run play). For each play, they compare the outcome with their own defensive formations and the other team’s offensive formations, which help Coach Breedlove react more quickly to formation shifts during a game. We explain the analytical techniques that generated these figures in much more depth in Chapters 3–6 and Chapter 9.
New work that Dar is fostering involves building better high school athlete recruit- ing models. For example, each year the team gives scholarships to three students who are wide receiver recruits. For Dar, picking out the best players goes beyond simple
Total # of Plays: 540 Percentage of Run: 46.48% Percentage of Pass: 53.52%
Total # of Plays: 155 Percentage of Run: 79.35% Percentage of Pass: 20.65%
Total # of Plays: 385 Percentage of Run: 33.25% Percentage of Pass: 66.75%
Total # of Plays: 294 Percentage of Run: 38.78% Percentage of Pass: 61.22%
Total # of Plays: 91 Percentage of Run: 15.38% Percentage of Pass: 84.62%
Total # of Plays: 162 Percentage of Run: 50.62% Percentage of Pass: 49.38%
Total # of Plays: 132 Percentage of Run: 24.24% Percentage of Pass: 75.67%
Total # of Plays: 25 Percentage of Run: 44.00% Percentage of Pass: 56.00%
Total # of Plays: 66 Percentage of Run: 4.55%
Percentage of Pass: 95.45%
12, 21, 30, 31, 32 10, 11, 20, 22, or Missing
If it is...
If... If the distance to achieve the next down is
More than 5 yardsLess than 5 yardsWe are behind We are leading
or it is a tie
1st or 2nd Down 3rd or 4th Down
If Off_Pers is
FIGURE 1.12 Cascaded Decision Tree for Run or Pass Plays. Source: Contributed by Dr. Dave Schrader, who retired after 24 years in advanced development and marketing at Teradata. He has remained on the Board of Advisors of the Teradata University
Network, where he spends his retirement helping students and faculty learn more about sports analytics. Graphics by Peter Liang
and Jacob Pearson, graduate students at Oklahoma State University, as part of a student project in the spring of 2016 in Prof.
Ramesh Sharda’s class under Dr. Dave Schrader’s coaching.
42 Part I • Introduction to Analytics and AI
measures like how fast athletes run, how high they jump, or how long their arms are to newer criteria like how quickly they can rotate their heads to catch a pass, what kinds of reaction times they exhibit to multiple stimuli, and how accurately they run pass routes. Some of her ideas illustrating these concepts appear on the TUN Web site; look for the Business Scenario Investigation (2015) “The Case of Precision Football.”
WHAT CAN WE LEARN FROM THESE EXAMPLES? Beyond the front-office business ana- lysts, the coaches, trainers, and performance experts, there are many other people in sports who use data, ranging from golf groundskeepers who measure soil and turf condi- tions for PGA tournaments to baseball and basketball referees who are rated on the cor- rect and incorrect calls they make. In fact, it is hard to find an area of sports that is not being impacted by the availability of more data, especially from sensors.
Skills you will learn in this book for business analytics will apply to sports. If you want to dig deeper into this area, we encourage you to look at the Sports Analytics sec- tion of the TUN, a free resource for students and faculty. On its Web site, you will find descriptions of what to read to find out more about sports analytics, compilations of places where you can find publically available data sets for analysis, as well as examples
A Complete: 35
Total: 46 76.08%
Explosive: 4
1 Complete: 25
Total: 35 71.4%
Explosive: 1
B Complete: 6
Total: 8 75.00%
Explosive: 5
2 Complete: 12
Total: 24 50%
Explosive: 0
3 Complete: 14
Total: 28 50%
Explosive: 0
4 Complete: 8
Total: 14 57.14%
Explosive: 0
6 Complete: 7
Total: 10 70%
Explosive: 2
7 Complete: 13
Total: 21 61.9%
Explosive: 9
8 Complete: 7
Total: 10 70%
Explosive: 6
9 Complete: 15
Total: 27 55.55%
Explosive: 8
5 Complete: 25
Total: 44 56.81%
Explosive: 1
C Complete: 22
Total: 27 81.48%
Explosive: 2
X Complete: 1
Total: 13 7.69%
Explosive: 1
Y Complete: 7
Total: 18 38.88%
Explosive: 7
Z Complete: 5
Total: 15 33.33%
Explosive: 6
Line of Scrimmage
Defense
Offense
FIGURE 1.13 Heat Map Zone Analysis for Passing Plays. Source: Contributed by Dr. Dave Schrader, who retired after 24 years in advanced development and marketing at Teradata. He has remained on
the Board of Advisors of the Teradata University Network, where he spends his retirement helping
students and faculty learn more about sports analytics. Graphics by Peter Liang and Jacob Pearson,
graduate students at Oklahoma State University, as part of a student project in the spring of 2016 in
Prof. Ramesh Sharda’s class under Dr. Dave Schrader’s coaching.
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 43
of student projects in sports analytics and interviews of sports professionals who use data and analytics to do their jobs. Good luck learning analytics!
Analytics Applications in Healthcare—Humana Examples
Although healthcare analytics span a wide variety of applications from prevention to diagnosis to efficient operations and fraud prevention, we focus on some applications that have been developed at a major health insurance company in the United States, Humana. According to its Web site, “The company’s strategy integrates care delivery, the member experience, and clinical and consumer insights to encourage en- gagement, behavior change, proactive clinical outreach and wellness. . . .” Achieving these strategic goals includes significant investments in information technology in general and analytics in particular. Brian LeClaire is senior vice president and CIO of Humana. He has a PhD in MIS from Oklahoma State University. He has championed analytics as a competitive differentiator at Humana— including cosponsoring the creation of a center for excellence in analytics. He described the following projects as examples of Humana’s analytics initiatives, led by Humana’s chief clinical analytics officer, Vipin Gopal.
Humana Example 1: Preventing Falls in a Senior Population— An Analytic Approach
Accidental falls are a major health risk for adults age 65 years and older with one-third experiencing a fall every year.1 The costs of falls pose a significant strain on the U.S. healthcare system; the direct costs of falls were estimated at $34 billion in 2013 alone.1
ud_d_off_pers
ud_d_cov
Expl_
Expl_Y
ud_d_cov0
ud_d_cov2
ud_d_cov3
ud_d_cov4
ud_d_cov4 JAM ud_d_cov6
ud_d_covBUZZ ud_d_covFALL
ud_d_covFIRES
ud_d_covFLAME
ud_d_covHANDS
ud_d_covHARD ud_d_covHERO
ud_d_covHOT
ud_d_covLEVELS
ud_d_covMIX
ud_d_covROBBER
ud_d_covROLL
ud_d_covSKY
ud_d_covSMOKE
ud_d_covSPARK
ud_d_covSQUAT
ud_d_covSTATE
ud_d_covWALL
ud_d_off_pers10
ud_d_off_pers11
ud_d_off_pers12
ud_d_off_pers20
ud_d_off_pers21
ud_d_off_pers22
ud_d_off_pers23
ud_d_off_pers30
ud_d_off_pers31
ud_d_off_pers32
FIGURE 1.14 Time-Series Analysis of Explosive Plays.
44 Part I • Introduction to Analytics and AI
With the percent of seniors in the U.S. population on the rise, falls and associated costs are anticipated to increase. According to the Centers for Disease Control and Prevention (CDC), “Falls are a public health problem that is largely preventable” (www.cdc.gov/ homeandrecreationalsafety/falls/adultfalls.html).1 Falls are also the leading factor for both fatal and nonfatal injuries in older adults with injurious falls increasing the risk of disability by up to 50 percent (Gill et al., 2013).2 Humana is the nation’s second-largest provider of Medicare Advantage benefits with approximately 3.2 million members, most of whom are seniors. Keeping its senior members well and helping them live safely at their homes is a key business objective of which prevention of falls is an important com- ponent. However, no rigorous methodology was available to identify individuals most likely to fall, for whom falls prevention efforts would be beneficial. Unlike chronic medi- cal conditions such as diabetes and cancer, a fall is not a well-defined medical condition. In addition, falls are usually underreported in claims data as physicians typically tend to code the consequence of a fall such as fractures and dislocations. Although many clini- cally administered assessments to identify fallers exist, they have limited reach and lack sufficient predictive power (Gates et al., 2008).3 As such, there is a need for a prospec- tive and accurate method to identify individuals at greatest risk of falling so that they can be proactively managed for fall prevention. The Humana analytics team undertook the development of a Falls Predictive Model in this context. It is the first comprehensive PM reported that utilizes administrative medical and pharmacy claims, clinical data, temporal clinical patterns, consumer information, and other data to identify individuals at high risk of falling over a time horizon.
Today, the Falls PM is central to Humana’s ability to identify seniors who could benefit from fall mitigation interventions. An initial proof-of-concept with Humana con- sumers, representing the top 2 percent of those at the highest risk of falling, demonstrated that the consumers had increased utilization of physical therapy services, indicating con- sumers are taking active steps to reduce their risk for falls. A second initiative utilizes the Falls PM to identify high-risk individuals for remote monitoring programs. Using the PM, Humana was able to identify 20,000 consumers at a high risk of falling who benefited from this program. Identified consumers wear a device that detects falls and alerts a 24/7 service for immediate assistance.
This work was recognized by the Analytics Leadership Award by Indiana University Kelly School of Business in 2015, for innovative adoption of analytics in a business environment.
Contributors: Harpreet Singh, PhD; Vipin Gopal, PhD; Philip Painter, MD.
Humana Example 2: Humana’s Bold Goal—Application of Analytics to Define the Right Metrics
In 2014, Humana, Inc. announced its organization’s Bold Goal to improve the health of the communities it serves by 20 percent by 2020 by making it easy for people to achieve their best health. The communities that Humana serves can be defined in many ways, including geographically (state, city, neighborhood), by product (Medicare Advantage, employer-based plans, individually purchased), or by clinical profile (priority conditions including diabetes, hypertension, congestive heart failure [CHF], coronary artery disease [CAD], chronic obstructive pulmonary disease [COPD], or depression). Understanding the health of these communities and how they track over time is critical not only for the evaluation of the goal, but also in crafting strategies to improve the health of the whole membership in its entirety.
A challenge before the analytics organization was to identify a metric that cap- tures the essence of the Bold Goal. Objectively measured traditional health insurance
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 45
metrics such as hospital admissions or emergency room visits per 1,000 persons would not capture the spirit of this new mission. The goal was to identify a metric that captures health and its improvement in a community and was relevant to Humana as a business. Through rigorous analytic evaluations, Humana eventually selected “Healthy Days,” a four- question, quality-of-life questionnaire originally developed by the CDC to track and measure Humana’s overall progress toward the Bold Goal.
It was critical to make sure that the selected metric was highly correlated to health and business metrics so that any improvement in Healthy Days resulted in improved health and better business results. Some examples of how “Healthy Days” is correlated to metrics of interest include the following:
• Individuals with more unhealthy days (UHDs) exhibit higher utilization and cost patterns. For a five-day increase in UHDs, there are (1) an $82 increase in average monthly medical and pharmacy costs, (2) an increase of 52 inpatient admits per 1,000 patients, and (3) a 0.28-day increase in average length of stay (Havens, Peña, Slabaugh, Cordier, Renda, & Gopal, 2015).1
• Individuals who exhibit healthy behaviors and have their chronic conditions well managed have fewer UHDs. For example, when we look at individuals with diabe- tes, UHDs are lower if they obtained an LDL screening ( -4.3 UHDs) or a diabetic eye exam ( -2.3 UHDs). Likewise, if they have controlled blood sugar levels mea- sured by HbA1C (-1.8 UHDs) or LDL levels (-1.3 UHDs) (Havens, Slabaugh, Peña, Haugh, & Gopal 2015).2
• Individuals with chronic conditions have more UHDs than those who do not have (1) CHF (16.9 UHDs), (2) CAD (14.4 UHDs), (3) hypertension (13.3 UHDs), (4) dia- betes (14.7 UHDs), (5) COPD (17.4 UHDs), or (6) depression (22.4 UHDs) (Havens, Peña, Slabaugh et al., 2015; Chiguluri, Guthikonda, Slabaugh, Havens, Peña, & Cordier, 2015; Cordier et al., 2015).1,3,4
Humana has since adopted Healthy Days as their metric for the measurement of progress toward Bold Goal (Humana, http://populationhealth.humana.com/wp- content/uploads/2016/05/BoldGoal2016ProgressReport_1.pdf).5
Contributors: Tristan Cordier, MPH; Gil Haugh, MS; Jonathan Peña, MS; Eriv Havens, MS; Vipin Gopal, PhD.
Humana Example 3: Predictive Models to Identify the Highest Risk Membership in a Health Insurer
The 80/20 rule generally applies in healthcare; that is, roughly 20 percent of consum- ers account for 80 percent of healthcare resources due to their deteriorating health and chronic conditions. Health insurers like Humana have typically enrolled the highest-risk enrollees in clinical and disease management programs to help manage the chronic con- ditions the members have.
Identification of the correct members is critical for this exercise, and in the recent years, PMs have been developed to identify enrollees with high future risk. Many of these PMs were developed with heavy reliance on medical claims data, which results from the medical services that the enrollees use. Because of the lag that exists in submitting and processing claims data, there is a corresponding lag in identification of high-risk members for clinical program enrollment. This issue is especially relevant when new members join a health insurer as they would not have a claims history with an insurer. A claims-based PM could take on average of 9–12 months after enrollment of new members to identify them for referral to clinical programs.
In the early part of this decade, Humana attracted large numbers of new members in its Medicare Advantage products and needed a better way to clinically manage this
46 Part I • Introduction to Analytics and AI
membership. As such, it became extremely important that a different analytic approach be developed to rapidly and accurately identify high-risk new members for clinical man- agement, to keep this group healthy and costs down.
Humana’s Clinical Analytics team developed the New Member Predictive Model (NMPM) that would quickly identify at-risk individuals soon after their new plan enrollments with Humana rather than waiting for sufficient claim history to become available for compil- ing clinical profiles and predicting future health risk. Designed to address the unique chal- lenges associated with new members, NMPM developed a novel approach that leveraged and integrated broader data sets beyond medical claims data such as self-reported health risk assessment data and early indicators from pharmacy data, employed advanced data min- ing techniques for pattern discovery, and scored every Medicare Advantage (MA, a specific insurance plan) consumer daily based on the most recent data Humana has to date. The model was deployed with a cross-functional team of analytics, IT, and operations to ensure seamless operational and business integration.
Since NMPM was implemented in January 2013, it has been rapidly identifying high- risk new members for enrollment in Humana’s clinical programs. The positive outcomes achieved through this model have been highlighted in multiple senior leader commu- nications from Humana. In the first quarter 2013 earnings release presentation to inves- tors, Bruce Broussard, CEO of Humana, stated the significance of “improvement in new member PMs and clinical assessment processes,” which resulted in 31,000 new members enrolled in clinical programs, compared to 4,000 in the same period a year earlier, a 675 percent increase. In addition to the increased volume of clinical program enrollments, outcome studies showed that the newly enrolled consumers identified by NMPM were also referred to clinical programs sooner with over 50 percent of the referrals identified within the first three months after new MA plan enrollments. The consumers identified also participated at a higher rate and had longer tenure in the programs.
Contributors: Sandy Chiu, MS; Vipin Gopal, PhD.
These examples illustrate how an organization explores and implements analytics applications to meet its strategic goals. You will see several other examples of healthcare applications throughout various chapters in the book.
ANALYTICS IN THE RETAIL VALUE CHAIN The retail sector is where you would perhaps see the most applications of analytics. This is the domain where the volumes are large but the margins are usually thin. Customers’ tastes and preferences change frequently. Physical and online stores face many challenges to succeed. And market dominance at one time does not guarantee continued success. So investing in learning about your suppliers, cus- tomers, employees, and all the stakeholders that enable a retail value chain to succeed and using that information to make better decisions has been a goal of the analytics industry for a long time. Even casual readers of analytics probably know about Amazon’s enormous investments in analytics to power their value chain. Similarly, Walmart, Target, and other major retailers have invested millions of dollars in analytics for their supply chains. Most of the analytics technology and service providers have a major presence in retail analytics. Coverage of even a small portion of those applications to achieve our exposure goal could fill a whole book. So this section highlights just a few potential applications. Most of these have been fielded by many retailers and are available through many technology providers, so in this section, we will take a more general view rather than point to specific cases. This general view has been proposed by Abhishek Rathi, CEO of vCreaTek.com. vCreaTek, LLC is a boutique analytics software and service company that has offices in India, the United States, the United Arab Emirates (UAE), and Belgium. The company develops ap- plications in multiple domains, but retail analytics is one of its key focus areas.
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 47
Figure 1.15 highlights selected components of a retail value chain. It starts with suppliers and concludes with customers but illustrates many intermediate strategic and operational planning decision points where analytics—descriptive, predictive, or prescriptive—can play a role in making better data-driven decisions. Table 1.1 also il- lustrates some of the important areas of analytics applications, examples of key questions that can be answered through analytics, and of course, the potential business value de- rived from fielding such analytics. Some examples are discussed next.
An online retail site usually knows its customer as soon as the customer signs in, and thus they can offer customized pages/offerings to enhance the experience. For any retail store, knowing its customer at the store entrance is still a huge challenge. By com- bining the video analytics and information/badge issued through its loyalty program, the store may be able to identify the customer at the entrance itself and thus enable an extra opportunity for a cross-selling or up-selling. Moreover, a personalized shopping experi- ence can be provided with more customized engagement during the customer’s time in the store.
Store retailers invest lots of money in attractive window displays, promotional events, customized graphics, store decorations, printed ads, and banners. To discern the effectiveness of these marketing methods, the team can use shopper analytics by observ- ing closed-circuit television (CCTV) images to figure out the demographic details of the in-store foot traffic. The CCTV images can be analyzed using advanced algorithms to de- rive demographic details such as age, gender, and mood of the person browsing through the store.
Further, the customer’s in-store movement data when combined with shelf layout and planogram can give more insight to the store manager to identify the hot-selling/ profitable areas within the store. Moreover, the store manager also can use this informa- tion to plan the workforce allocation for those areas for peak periods.
Retail Value Chain Critical needs at every touch point of the Retail Value Chain
• Shelf-space optimization • Location analysis • Shelf and floor planning • Promotions and markdown optimization
• Supply chain management • Inventory cost optimization • Inventory shortage and excess management • Less unwanted costs
• Targeted promotions • Customized inventory • Promotions and price optimization • Customized shopping experience
• On-time product availability at low costs • Order fulfillment and clubbing • Reduced transportation costs
• Trend analysis • Category management • Predicting trigger events for sales • Better forecasts of demand
• Deliver seamless customer experience • Understand relative performance of channels • Optimize marketing strategies
Vendors Customers Planning Merchandizing Buying
Warehouse & Logistics
Multichannel Operations
• Building retention and satisfaction • Understanding the needs of the customer better • Serving high LTV customers better
FIGURE 1.15 Example of Analytics Applications in a Retail Value Chain. Source: Contributed by Abhishek Rathi, CEO, vCreaTek.com.
48 Part I • Introduction to Analytics and AI
TABLE 1.1 Examples of Analytics Applications in the Retail Value Chain
Analytic Application Business Question Business Value
Inventory Optimization
1. Which products have high demand? 2. Which products are slow moving or becoming
obsolete?
1. Forecast the consumption of fast-moving products and order them with sufficient inventory to avoid a stock out scenario.
2. Perform fast inventory turnover of slow-moving products by combining them with one in high demand.
Price Elasticity 1. How much net margin do I have on the product?
2. How much discount can I give on this product?
1. Markdown prices for each product can be optimized to reduce the margin dollar loss.
2. Optimized price for the bundle of products is identified to save the margin dollar.
Market-Basket Analysis
1. What products should I combine to create a bundle offer?
2. Should I combine products based on slow- moving and fast-moving characteristics?
3. Should I create a bundle from the same category or a different category line?
1. The affinity analysis identifies the hidden correlations between the products, which can help in following values: a. Strategize the product bundle offering based on
focus on inventory or margin. b. Increase cross-selling or up-selling by creating
bundle from different categories or the same categories, respectively.
Shopper Insight
1. Which customer is buying what product at what location?
1. By customer segmentation, the business owner can create personalized offers resulting in better customer experience and retention of the customer.
Customer Churn Analysis
1. Who are the customers who will not return? 2. How much business will I lose? 3. How can I retain the customers? 4. What demography of customer is my loyal
customer?
1. Businesses can identify the customer and product relationships that are not working and show high churn. Thus, they can have better focus on product quality and the reason for that churn.
2. Based on the customer lifetime value (LTV), the business can do targeted marketing resulting in retention of the customer.
Channel Analysis
1. Which channel has lower customer acquisition cost?
2. Which channel has better customer retention? 3. Which channel is more profitable?
1. Marketing budget can be optimized based on insight for better return on investment.
New Store Analysis
1. What location should I open? 2. What and how much opening
inventory should I keep?
1. Best practices of other locations and channels can be used to get a jump-start.
2. Comparison with competitor data can help to create a differentiator to attract the new customers.
Store Layout 1. How should I do store layout for better topline?
2. How can I increase my in-store customer experience?
1. Understand the association of products to decide store layout and better alignment with customer needs.
2. Workforce deployment can be planned for better customer interactivity and thus satisfying customer experience.
Video Analytics
1. What demography is entering the store during the peak period of sales?
2. How can I identify a customer with high LTV at the store entrance so that a better personalized experience can be provided to this customer?
1. In-store promotions and events can be planned based on the demography of incoming traffic.
2. Targeted customer engagement and instant discount enhances the customer experience resulting in higher retention.
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 49
Market-basket analysis has commonly been used by the category managers to push the sale of slowly moving stock keeping units (SKUs). By using advanced analytics of data available, the product affinity can be identified at the lowest level of SKU to drive better returns on investments (ROIs) on the bundle offers. Moreover, by using price elasticity techniques, the markdown or optimum price of the bundle offer can also be deduced, thus reducing any loss in the profit margin.
Thus, by using data analytics, a retailer can not only get information on its current operations but can also get further insight to increase the revenue and decrease the operational cost for higher profit. A fairly comprehensive list of cur- rent and potential retail analytics applications that a major retailer such as Amazon could use is proposed by a blogger at Data Science Central. That list is available at www. datasciencecentral.com/profiles/blogs/20-data-science-systems-used-by- amazon-to-operate-its-business. As noted earlier, there are too many examples of these opportunities to list here, but you will see many examples of such applications throughout the book.
IMAGE ANALYTICS As seen in this section, analytics techniques are being applied to many diverse industries and data. An area of particular growth has been analysis of visual images. Advances in image capturing through high-resolution cameras, storage capabili- ties, and deep learning algorithms have enabled very interesting analyses. Satellite data have often proven their utility in many different fields. The benefits of satellite data at high resolution and in different forms of imagery including multi-spectral are significant to scientists who need to regularly monitor global change, land usage, and weather. In fact, by combining the satellite imagery and other data including information on social media, government filings, and so on, one can surmise business planning activities, traffic pat- terns, changes in parking lots or open spaces. Companies, government agencies, and non- governmental organizations (NGOs) have invested in satellites to try to image the whole globe every day so that daily changes can be tracked at any location and the information can be used for forecasting. In the last few months, many interesting examples of more reliable and advanced forecasts have been reported. Indeed, this activity is being led by different industries across the globe, and has added a term to Big Data called Alternative Data. Here are a few examples from Tartar et al. (2018). We will see more in Chapter 9 when we study Big Data.
• World Bank researchers used satellite data to propose strategic recommendations for urban planners and officials from developing nations. This analysis arose due to the recent natural disaster where at least 400 people died in Freetown, Sierra Leone. Researchers clearly demonstrated that Freetown and some other developing cities lacked systematic planning of their infrastructure that resulted in the loss of life. The bank researchers are using satellite imagery now to make critical decisions regard- ing risk-prone urban areas.
• EarthCast provides accurate weather updates for a large commercial U.S. airline based on the data it pulls from a constellation of 60 government-operated satellites combined with ground and aircraft-based sensors, tracking almost anything from lightning to turbulence. It has even developed the capability to map out conditions along a flight path and provides customized forecasts for everything from hot air balloons to drones.
• Imazon started using satellite data to develop a picture of close real-time informa- tion on Amazon deforestation. It uses advanced optical and infrared imagery that has led to identifying illegal sawmills. Imazon is now focused more on getting data to local governments through its “green municipalities” program that trains officials to identify and curb deforestation.
50 Part I • Introduction to Analytics and AI
• The Indonesian government teamed up with international nonprofit Global Fishing Watch, which processes satellite extracted information on ship movement to spot where and when vessels are fishing illegally (Emmert, 2018). This initiative delivered instant results: Government revenue from fishing went up by 129 percent in 2017 compared to 2014. It is expected that by next decade, the organization would track vessels that are responsible for 75 percent of the world’s catch.
These examples illustrate just a sample of ways that satellite data can be combined with analytics to generate new insights. In anticipation of the coming era of abundant earth observations from satellites, scientists and communities must put some thought into recognizing key applications and key scientific issues for the betterment of society. Although such concerns will eventually be resolved by policymakers, what is clear is that new and interesting ways of combining satellite data and many other data sources is spawning a new crop of analytics companies.
Such image analysis is not limited to satellite images. Cameras mounted on drones and traffic lights on every conveyable pole in buildings and streets provide the ability to capture images from just a few feet high. Analysis of these images coupled with facial recognition technologies is enabling all kinds of new applications from customer recognition to govern- ments’ ability to track all subjects of interest. See Yue (2017) as an example. Applications of this type are leading to much discussion on privacy issues. In Application Case 1.7, we learn about a more benevolent application of image analytics where the images are captured by a phone and a mobile application provides immediate value to the user of the app.
Estimating how much ground is covered by green vegetation is important in analysis of a forest or even a farm. In case of a forest, such analysis helps users understand how the forest is evolving, its impact on surrounding areas, and even climate. For a farm, similar analysis can help understand likely plant growth and help estimate future crop yields. It is obviously impossible to measure all forest cover manually and is challenging for a farm. The com- mon method is to record images of a forest/farm and then analyze these images to estimate the ground cover. Such analysis is expensive to perform visually and is also error prone. Different experts looking at the ground cover might estimate the percentage of ground covering differently. Thus, automated meth- ods to analyze these images and estimate the per- centage of ground covered by vegetation are being developed. One such method and an app to make it practical through a mobile phone has been devel- oped at Oklahoma State University by researchers in the Department of Plant and Soil Sciences in part- nership with the university’s App Center and the Information Technology group within the Division of Agricultural Sciences and Natural Resources.
Canopeo is a free desktop or mobile app that estimates green canopy cover in near real-time from images taken with a smartphone or digital camera. In experiments in corn, wheat, canola, and other crops, Canopeo calculated the percentage of canopy covering dozens to thousands of times faster than existing software without sacrificing accuracy. And unlike other programs, the app can acquire and ana- lyze video images, says Oklahoma State University (OSU) soil physicist, Tyson Ochsner—a feature that should reduce the sampling error associated with canopy cover estimates. “We know that plant cover, plant canopies, can be quite variable in space,” says Ochsner, who led the app’s development with former doctoral student Andres Patrignani, now a faculty member at Kansas State University. “With Canopeo, you can just turn on your [video] device, start walk- ing across a portion of a field, and get results for every frame of video that you’re recording.” By using a smartphone or tablet’s digital camera, Canopeo users in the field can take photos or videos of green plants, including crops, forages, and turf, and import them to the app, which analyzes each image pixel, classifying them based on its red-green-blue (RGB)
Application Case 1.7 Image Analysis Helps Estimate Plant Cover
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 51
Analytics/data science initiatives are quickly embracing and even merging with new developments in artificial intelligence. The next section provides an overview of artificial intelligence followed by a brief discussion of convergence of the two.
u SECTION 1.6 REVIEW QUESTIONS
1. What are three factors that might be part of a PM for season ticket renewals? 2. What are two techniques that football teams can use to do opponent analysis? 3. What other analytics uses can you envision in sports? 4. Why would a health insurance company invest in analytics beyond fraud detection?
Why is it in its best interest to predict the likelihood of falls by patients?
5. What other applications similar to prediction of falls can you envision? 6. How would you convince a new health insurance customer to adopt healthier life-
styles (Humana Example 3)?
7. Identify at least three other opportunities for applying analytics in the retail value chain beyond those covered in this section.
8. Which retail stores that you know of employ some of the analytics applications iden- tified in this section?
9. What is a common thread in the examples discussed in image analytics? 10. Can you think of other applications using satellite data along the lines presented in
this section?
color values. Canopeo analyzes pixels based on a ratio of red to green and blue to green pixels as well as an excess green index. The result is an image where color pixels are converted into black and white with white pixels corresponding to green can- opy and black pixels representing the background. Comparison tests showed that Canopeo analyzes images more quickly and just as accurately as two other available software packages.
Developers of Canopeo expect the app to help producers judge when to remove grazing cattle from winter wheat in “dual-purpose” systems where wheat is also harvested for grain. Research by oth- ers at OSU found that taking cattle off fields when at least 60 percent green canopy cover remained ensured a good grain yield. “So, Canopeo would be useful for that decision,” Patrignani says. He and Ochsner also think the app could find use in turf- grass management; in assessments of crop damage from weather or herbicide drift; as a surrogate for the Normalized Difference Vegetation Index (NDVI) in fertilizer recommendations; and even in UAV- based photos of forests or aquatic systems.
Analysis of images is a growing applica- tion area for deep learning as well as many other AI techniques. Chapter 9 includes several exam- ples of image analysis that have spawned another
term—alternative data. Applications of alternative data are emerging in many fields. Chapter 6 also highlights some applications. Imagining innovative applications by being exposed to others’ ideas is one of the main goals of this book!
Questions for DisCussion
1. What is the purpose of knowing how much ground is covered by green foliage on a farm? In a forest?
2. Why would image analysis of foliage through an app be better than a visual check?
3. Explore research papers to understand the underlying algorithmic logic of image analysis. What did you learn?
4. What other applications of image analysis can you think of?
Source: Compiled from A. Patrignani and T. E. Ochsner. (2015). “Canopeo: A Powerful New Tool for Measuring Fractional Green Canopy Cover.” Agronomy Journal, 107(6), pp. 2312–2320; R. Lollato, A. Patrignani, T. E. Ochsner, A. Rocatelli, P. Tomlinson, & J. T. Edwards. (2015). Improving Grazing Management Using a Smartphone App. www.bookstore.ksre.ksu.edu/pubs/ MF3304.pdf (accessed October 2018); http://canopeoapp. com/ (accessed October 2018); Oklahoma State University press releases.
52 Part I • Introduction to Analytics and AI
1.7 ARTIFICIAL INTELLIGENCE OVERVIEW
On September 1, 2017, the first day of the school year in Russia, Vladimir Putin, the Russian President, lectured to over 1,000,000 school children in what is called in Russia the National Open Lesson Day. The televised speech was titled “Russia Focused on the Future.” In this presentation, the viewers saw what Russian scientists are achieving in sev- eral fields. But, what everyone remembers from this presentation is one sentence: “The country that takes the lead in the sphere of computer-based artificial intelligence will become the ruler of the world.”
Putin is not the only one who knows the value of AI. Governments and corpora- tions are spending billions of dollars in a race to become a leader in AI. For example, in July 2017, China unveiled a plan to create an AI industry worth $150 billion to the Chinese economy by 2030 (Metz, 2018). China’s Badu Company today employs over 5,000 AI en- gineers. The Chinese government facilitates research and applications as a national top priority. The accounting firm PricewaterhouseCoopers calculated that AI will add $15.7 trillion to the global economy by 2030 (about 14 percent; see Liberto, 2017). Thus, there is no wonder that AI is clearly the most talked about technology topic in 2018.
What Is Artificial Intelligence?
There are several definitions of what is AI (Chapter 2). The reason is that AI is based on theories from several scientific fields, and it encompasses a wide collection of tech- nologies and applications. So, it may be beneficial to look at some of the characteris- tics of AI in order to understand what it is. The major goal of AI is to create intelligent machines that can do tasks currently done by people. Ideally, these tasks include reasoning, thinking, learning, and problem solving. AI studies the human thought processes’ ability to understand what intelligence is so AI scientists can duplicate the human processes in machines. eMarketer (2017) provides a comprehensive report, describing AI as
• Technology that can learn to do things better over time. • Technology that can understand human language. • Technology that can answer questions.
The Major Benefits of AI
Since AI appears in many shapes, it has many benefits. They are listed in Chapter 2. The major benefits are as follows:
• Significant reduction in the cost of performing work. This reduction continues over time while the cost of doing the same work manually increases with time.
• Work can be performed much faster. • Work is consistent in general, more consistent than human work. • Increased productivity and profitability as well as a competitive advantage are the
major drivers of AI.
The Landscape of AI
There are many parts in the landscape (or ecosystem) of AI. We decided to organize them into five groups as illustrated in Figure 1.16. Four of the groups constitute the basis for the fifth one, which is the AI applications. The groups are as follows:
MAJOR TECHNOLOGIES Here we elected to include machine learning (Chapter 5), deep learning (Chapter 6), and intelligent agents (Chapter 2).
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 53
KNOWLEDGE-BASED TECHNOLOGIES (all covered in Chapter 12) Topics covered are expert systems, recommendation engines, chatbots, virtual personal assistants, and robo-advisors.
BIOMETRIC-RELATED TECHNOLOGIES This includes natural language processing (under- standing and generation, machine vision and scene and image recognition and voice and other biometric recognition (Chapter 6).
SUPPORT THEORIES, TOOLS, AND PLATFORMS Academic disciplines include computer science, cognitive science, control theory, linguistics, mathematics, neuroscience, philoso- phy, psychology, and statistics.
Devices and methods include sensors, augmented reality, context awareness, logic, gestural computing collaborative filtering, content recognition, neural networks, data mining, humanoid theories, case-based reasoning, predictive application programming interfaces (APIs), knowledge management, fuzzy logic, genetic algorithm, bin data, and much more.
TOOLS AND PLATFORMS These are available from IBM, Microsoft, Nvidia, and several hundred vendors specializing in the various aspects of AI.
AI APPLICATIONS There are several hundred or may be thousands of them. We provide here only a sample:
Smart cities, smart homes, autonomous vehicles (Chapter 13), automatic decisions (Chapter 2), language translation, robotics (Chapter 10), fraud detection, security protec- tion, content screening, prediction, personalized services, and more. Applications are in all business areas (Chapter 2), and in almost any other area ranging from medicine and healthcare to transportation and education.
Note: Lists of all these are available at Faggela (2018) and Jacquet (2017). Also see Wikipedia, “Outline of artificial intelligence,” and a list of “AI projects” (several hundred items.)
In Application Case 1.8, we describe how several of these technologies are com- bined in improving security and in expediting the processing of passengers in airports.
Major AI Technologies
Knowledge-Based Technologies
Biometric-Based Technologies
Support Theories, Tools, Platforms,
Mechanisms
AI Applications
FIGURE 1.16 The Landscape (Ecosystem) of AI. Source: Drawn by E. Turban.
54 Part I • Introduction to Analytics and AI
NARROW (WEAK) VERSUS GENERAL (STRONG) AI The AI field can be divided into two major categories of applications: narrow (or weak) and general (or strong).
A Narrow AI Field Focuses on One Narrow Field (Domain). Well-known examples of this are SIRI and Alexa (Chapter 12) that, at least in their early years of life, operated in limited, predefined areas. As time has passed, they have become more general, acquiring ad- ditional knowledge. Most expert systems (Chapter 12) were operating in fairly narrow domains. If you notice, when you converse with an automated call center, the computer
We may not like the security lines at airports or the idea that terrorists may board our plane or enter our country. Several AI devices are designed to mini- mize these possibilities.
1. Facial recognition at airports. Jet Blue is ex- perimenting with facial-recognition technology (a kind of machine vision to match travelers’ faces against prestored photos, such as pass- port, driver’s license). This will eliminate the need for boarding passes and increase security. The match is of high quality. The technology pioneered by British Airways is used by Delta, KLM, and other airlines using similar technol- ogies for self-checking of bags. Similar tech- nology is used by the U.S. Immigration and Customs Enforcement agency where people’s photos taken at arrivals are matched against the database of photos and other documents.
2. China’s system. The major airports in China are using a system similar to that of Jet Blue, us- ing facial recognition for verifying the identity of passengers. The idea is to eliminate board- ing passes and expedite the flow of boarding. The system is also used to recognize airport employees entering restricted areas.
3. Using bots. Several airports (e.g., New York, Beijing) offer conversational bots (Chapter 12) to provide travelers with airport guidance. Bots provide also information about customs and immigration services.
4. Spotting liars at airport. This application is emerging to help immigration services to vet passengers at airports and land entry borders. With increased security, both immigration and airline personal may need to query passengers. Here is the solution that can be economically used to query all passengers at high speed, so
there will be short waiting lines. This emerging system is called Automated Virtual Agent for Truth Assessments in Real Time (AVATAR). The essentials of the system are as follows:
a. AVATAR is a bot in which you first scan your passport.
b. AVATAR asks you a few questions. Several AI technologies are used in this project, such as AI, Big Data analytics, the “Cloud,” robotics, machine learning, machine vision, and bots.
c. You answer the questions. d. AVATAR’s sensors and other AI technolo-
gies collect data from your body, such as voice variability, facial expression (e.g., muscle engagement), eyes’ position and movements, mouth movements, and body posture. Researchers feel that it takes less effort to tell the truth than to die, so re- searchers compared the answers to routine questions.
The machine then will flag suspects for fur- ther investigation. The machine is already in use by immigration agents in several countries.
Sources: Condensed from Thibodeaux, W. (2017, June 29). “This Artificial Intelligence Kiosk Is Designed to Spot Liars at Airports.” Inc.com.; Silk, R. (2017, November). “Biometrics: Facial Recognition Tech Coming to an Airport Near You.” Travel Weekly, 21.
Questions for Case 1.8
1. List the benefits of AI devices to travelers.
2. List the benefits to governments and airline companies.
3. Relate this case to machine vision and other AI tools that deal with people’s biometrics.
Application Case 1.8 AI Increases Passengers’ Comfort and Security in Airports and Borders
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 55
(which is usually based on some AI technology) is not too intelligent. But, it is get- ting “smarter” with time. Speech recognition allows computers to convert sound to text with great accuracy. Similarly, computer vision is improving, recognizes objects, classifies them, and even understands their movements. In sum, there are millions of narrow AI ap- plications, and the technology is improving every day. However, AI is not strong enough yet because it does not exhibit the true capabilities of human intelligence (Chapter 2).
GENERAL (STRONG) AI To exhibit real intelligence, machines need to perform the full range of human cognitive capabilities. Computers can have some cognitive capabilities (e.g., some reasoning and problem solving) as will be shown in Chapter 6 on cognitive computing.
The difference between the two classes of AI is getting smaller as AI is getting smarter. Ideally, strong AI will be able to replicate humans. But true intelligence is happening only in narrow domains, such as game playing, medical diagnosis, and equipment failure diagnosis.
Some feel that we never will be able to build a truly strong AI machine. Others think differently; see the debate in Section 14.9. The following is an example of a strong AI bot in a narrow domain.
Example 3: AI Makes Coke Vending Machine Smarter
If you live in Australia or New Zealand and you are near a Coca-Cola vending machine, you can order a can or a bottle of the soft drink using your smartphone. The machines are cloud connected, which means you can order the Coke from any place in the world, not only for yourself but also for any friend who is near a vending machine in Australia or New Zealand. See Olshansky (2017).
In addition, the company can adjust pricing remotely, offer promotions, and collect inventory data so that restocking can be made. Converting existing machines to AI-enabled takes about 1 hour each.
Wait a minute, what if something goes wrong? No problem, you can chat with Coca- Cola’s bot via Facebook Messenger (Chapter 12).
The Three Flavors of AI Decisions
Staff (2017) divided the capabilities of AI systems into three levels: assisted, autonomous, and augmented.
ASSISTED INTELLIGENCE This is equivalent mostly to the weak AI, which works only in narrow domains. It requires clearly defined inputs and outputs. Examples are some monitoring systems and low-level virtual personal assistants (Chapter 12). Our cars are full of such monitoring systems that give us alerts. Similarly, there are many healthcare applications (monitoring, diagnosis).
Autonomous AI
These systems are in the realm of the strong AI but in very narrow domain. Eventually, the computer will take over. Machines will act as experts and have absolute decision-making power. Pure robo-advisors (Chapter 12) are examples of such machines. Autonomous vehicles and robots that can fix themselves are also good examples.
AUGMENTED INTELLIGENCE Most of the existing AI applications are between assisted and autonomous and/are referred to as augmented intelligence (or intelligence aug- mentation). The technology focuses on augmenting computer abilities to extend human cognitive abilities (see Chapter 6 on cognitive computing), resulting in high performance as described in Technology Insights 1.1.
56 Part I • Introduction to Analytics and AI
TECHNOLOGY INSIGHTS 1.1 Augmented Intelligence
The idea of combining the performance of people and machines is not new. Here we combine (augmenting) human capabilities with powerful machine intelligence. That is, not replacing people which is done by autonomous AI, but extending human cognitive abilities. The result is the ability to solve complex human problems as in the opening vignette to this chapter. The computers enabled people to solve problems that were unsolved before. Padmanabhan (2018) distinguishes the following differences between traditional and augmented AI:
1. Augmented machines extend rather than replace human decision making, and they facilitate creativity.
2. Augmentation excels in solving complex human and industry problems in specific domains in contrast with strong, general AI.
3. In contrast with a “black box” model of some AI and analytics, augmented intelligence provides insights and recommendations, including explanations.
4. In addition, augmentation technology can offer new solutions by combining existing and discovered information in contrast with assisted AI, which identifies problems or symp- toms and suggests predetermined solutions.
Padmanabhan (2018) and many others believe that at the moment, augmented AI is the best option to move toward the transformation of the AI world.
In contrast with autonomous AI, which describes machines with a wide range of cogni- tive abilities (e.g., driverless vehicles), augmented intelligence has only a few cognitive abilities.
Examples of Augmented Intelligence Staff (2017) provides the following examples:
• Cybercrime fighting. AI can identify forthcoming attacks and suggest solutions. • e-Commerce decisions. Marketing tools make testing 100 times faster and adapt the
layout and response functions of a Web site to users. The machines make recommenda- tions and the marketers can accept or reject them.
• High-frequency stock market trading. This is done either completely autonomously or in some cases with control and calibration by humans.
DISCUSSION QUESTIONS
1. What is the basic premise of augmented intelligence?
2. List the major differences between augmented intelligence and traditional AI.
3. What are some benefits of augmented intelligence?
4. How does technology relate to cognitive computing?
Societal Impacts
Much talk is on the topics of AI and productivity, speed, and cost reduction. In a national conference hosted by Gartner, the famous IT consulting company, nearly half of 3,000 participating U.S. CIOs reported plans to deploy AI-now (Weldon, 2018). Industry can- not ignore the potential benefits of AI, especially its increased productivity gains, cost reduction and quality, and speed. Conference participants there talked about strategy and implementation (Chapter 14). It seems that every company is at least involved in pilot- ing and experimentation AI. However, in all this excitement, we should not neglect the societal impacts. Many of these are positive, some are negative, and most are unknown. A comprehensive discussion is provided in Chapter 14. Here we provide three examples of AI impacts.
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 57
IMPACT ON AGRICULTURE A major impact of AI will be on agriculture. One major an- ticipated result is to provide more food, especially in third world countries. Here are a few examples:
• According to Lopez (2017), using AI and robots can help farmers produce 70 per- cent more food by 2050. This increase is a result of higher productivity of farm equipment boosted by IoT (see opening vignette to Chapter 13) and a reduced cost of producing food. (Today only 10 percent of a family’s budget is spent on food versus. 17.5 percent in 1960).
• Machine vision helps in improved planting and harvesting. Also, AI helps to pick good kernels of grain.
• AI will help to improve the nutrition of food. • AI will reduce the cost of food processing. • Driverless tractors are already being experimented with. • Robots know how to pick fruits and to plant vegetables can solve the shortage of
farm workers. • Crop yields are continuously increasing in India and other countries. • Pest control improves. For example, AI can predict pest attacks, facilitating planning. • Weather conditions are monitored by satellites. AI algorithms tell farmers when to
plant and/or harvest.
The list can go on and on. For countries such as India and Bangladesh, these activi- ties will critically improve the life of farmers. All in all, AI will help farmers make better decisions. For a Bangladesh case, see PE Report (2017). See alsonews.microsoft.com/ en-in/features/ai-agriculture-icrisat-upl-india/.
Note: AI can help hungry pets too. A food and water dispenser, called Catspad, is available in the United Kingdom for about US $470 You need to put an ID tag on your pet (only cats and small dogs). The dispenser knows which pet comes to the food and dispenses the type and amount of appropriate food. In addition, sensors (Chapter 13) can tell you how much food each pet ate. You will also be notified if water needs to be added. Interested? See Deahl (2018) for details.
INTELLIGENT SYSTEMS CONTRIBUTION TO HEALTH AND MEDICAL CARE Intelligent sys- tems provide a major contribution to our health and medical care. New innovations ar- rive almost any day in some place in the world (governments, research institutions, and corporation-sponsored active medical AI research). Here are some interesting innovations.
• AI excels in disease prediction (e.g., predicting the occurrence of infective diseases one week in advance).
• AI can detect brain bleeds. • AI can track medication intake, send medical alerts, order medicine refills, and
improve prescription compliance. • Mobile telepresence robots remotely connect physicians and patients. • NVIDIA’s medical imaging supercomputer helps diagnosticians and facilitates cures
of diseases. • Robotics and AI can redesign pharmaceutical supply chains. • AI predicts cardiovascular risks from retinal images. • Cancer predictions are made with deep learning, and machine learning performs
melanoma diagnosis. • A virtual personal assistant can assess a patient’s mood and feeling by cues pro-
vided (e.g., speech gesture or inflection). • Many portals provide medical information to patients and even surgeons. Adoptive
spine IT is an example.
58 Part I • Introduction to Analytics and AI
• Aging-based AI center for research on people who are elderly operates in the United States. Similar activities exist in Japan.
• The use of bionic hands and legs is improving with AI. • Healthcare IT News (2017) describes how AI is solving healthcare problems by
using virtual assistants (Chapter 12).
The list can go on and on. Norman (2018) describes the scenario of replacing doc- tors with intelligent systems.
Note: AI in medicine is recognized as a scientific field with national and interna- tional annual conferences. For a comprehensive book on the subject, see Agah (2017).
OTHER SOCIETAL APPLICATIONS There are many AI applications in transportation, utili- ties, education, social services, and other fields. Some are covered under the topic of smart cities (Chapter 13). AI is used by social media and others to control content in- cluding fake news. Finally, how about using technology to eradicate child slavery in the Middle East? See Application Case 1.9.
In several Middle Eastern countries, notably Jordan, Abu Dhabi, and other Gulf nations, racing camels has been a popular activity for generations. The owners of the winning camels can make a huge bonus (up to $1,000,000 for first place). Also, the events are considered cultural and social.
The Problem
For a long time, the racing camels were guided by human jockeys. The lighter the weight of the rider, the better is the chance to win. So the owners of the camels trained children (as young as seven) to be jockeys. Young male children were bought (or kidnapped) from poor families in Sudan, India, Bangladesh, and other poor countries and were trained as child jockeys. In fact, this practice was using child slave labor to race the camels. This prac- tice was used for generations until it was banned in all Middle Eastern countries during 2005–2010. A major factor that resulted in the banning was the uti- lization of robots.
The Robots’ Solution
Racing camels was a tradition for many generations and become a lucrative sport. So, no one wanted to discontinue it. According to Opfer (2016), there was a humanistic reason for using robots to race camels—to save the children. Today, all camel race tracks in the Middle East employ only robots. The
robots are tied to the hump of the camels, look- ing like small jockeys and are remote controlled from cars that drive parallel to the racing camels. The owners can command the camels by voice, and they can also operate a mechanical whip to beat the animals so they will run faster, much like human jockeys do. Note that camels would not run unless they hear the voice of a human or see something that looks like a human on their humps.
The Technology
There is a video camera that shows the people that are in cars driving alongside of the camels, what is going on in real time. The owner can provide voice commands to the camel from the car. A mechanical whip attached to the hump of the camel can be remotely operated to induce the animal.
The Results
The results are astonishing. Not only was the child slavery practice eliminated, but also the speed obtained by the camels increased. After all, the robots used weigh only 6 pounds and do not get tired. To see how this works watch the video at you- tube.com/watch?v=GVeVhWXB7sk (2:47 min.). To view a complete race, see youtube.com/ watch?v=xFCRhk4GYds (9:08 min .). You may have
Application Case 1.9 Robots Took the Job of Camel-Racing Jockeys for Societal Benefits
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 59
u SECTION 1.7 REVIEW QUESTIONS
1. What are the major characteristics of AI? 2. List the major benefits of AI. 3. What are the major groups in the ecosystem of AI? List the major contents of each. 4. Why is machine learning so important? 5. Differentiate between narrow and general AI. 6. Some say that no AI application is strong. Why? 7. Define assisted intelligence, augmented intelligence, and autonomous intelligence. 8. What is the difference between traditional AI and augmented intelligence? 9. Relate types of AI to cognitive computing.
10. List five major AI applications for increasing the food supply. 11. List five contributions of AI in medical care.
1.8 CONVERGENCE OF ANALYTICS AND AI
Until now we have presented analytics and AI as two independent entities. But, as illustrated in the opening vignette, these technologies can be combined in solv- ing complex problems. In this section, we discuss the convergence of these tech- niques and how they complement each other. We also describe the possible addition of other technologies, especially IoT, that enable the solutions to very complex problems.
Major Differences between Analytics and AI
As you recall from Section 1.4, analytics process historical data using statistical, man- agement science and other computational tools to describe situations (descriptive ana- lytics), to predict results including forecasting (predictive analytics), and to propose recommendations for solutions to problems (predictive analytics). The emphasis is on the statistical, management science, and other computational tools that help analyze historical data.
AI, on the other hand, also uses different tools, but its major objective is to mimic the manner in which people think, learn, reason, make decisions, and solve problems. The emphasis here is on knowledge and intelligence as major tools for solving problems rather than relying on computation, which we do in analysis. Furthermore, AI also is dealing with cognitive computing. In reality, this difference is not so clear because in advanced analytic applications, there are situations of using machine learning (an AI
a chance to see the royal family when you go to the track. Finally, you can see more details in youtube. com/watch?v=C1uYAXJIbYg (8:08 min .).
Sources: Compiled from C. Chung. (2016, April 4). “Dubai Camel Race Ride-Along.” YouTube.com. youtube.com/watch?v=xFCRhk4 GYds (accessed September 2018); P. Boddington. (2017, January 3). “Case Study: Robot Camel Jockeys. Yes, really.” Ethics for Artificial Intelligence; and L. Slade. (2017, December 21). “Meet the Jordanian Camel Races Using Robot Jockeys.” Sbs. com.au.
DisCussion Questions
1. It is said that the robots eradicated the child slav- ery. Explain.
2. Why do the owners need to drive by their cam- els while they are racing?
3. Why not duplicate the technology for horse racing?
4. Summarize ethical aspects of this case (Read Boddington, 2017). Do this exercise after you have read about ethics in Chapter 14.
60 Part I • Introduction to Analytics and AI
technology), supporting analytics in both prediction and prescription. In this section, we describe the convergence of intelligent technologies.
Why Combine Intelligent Systems?
Both analytics and AI and their different technologies are making useful contributions to many organizations when each is applied by itself. But each does have limitations According to a Gartner study, the chance that business analytics initiatives will not meet the enterprise objectives is 70–80 percent. Namely, at least 70 percent of corporate needs are not fulfilled. In other words, there is only a small chance that business intelligence initiatives will result in organizational excellence. There are several reasons for this situation including:
• Predictive models have unintended effects (see Chapter 14). • Models must be used ethically, responsibly, and mindfully (Chapter 14). They may
not be used this way. • The results of analytics may be very good for some applications but not for
others. • Models are as good as their input data and assumptions (garbage-in, garbage-out). • Data could be incomplete. Changing environments can make data obsolete very
quickly. Models may be unable to adapt. • Data that come from people may not be accurate. • Data collected from different sources can vary in format and quality.
Additional reasons for combining intelligent systems are generic to IT projects, and they are discussed in Section 14.2.
The failure rate of AI initiatives is also high. Some of the reasons are similar to the rate of analytics. However, a major reason is that some AI technologies need a large amount of data, sometimes Big Data. For example, many millions of data items are fed to Alexa every day to increase its knowledge. Without continuous flow of data, there would not be good learning in AI.
The question is whether AI and analytics (and other intelligent systems) can be combined in such a way that there will be synergy for better results.
How Convergence Can Help?
According to Nadav (2017), business intelligence and its analytics answer most of the why and what questions regarding the sufficiency of problem solving. Adding prescrip- tive analytics will add more cost but not necessarily better performance. Therefore, the next generation of business intelligence platforms will use AI to automatically locate, visualize, and narrate important things. This can also be used to create automatic alerts and notifications. In addition, machine learning and deep learning can support ana- lytics by conducting pattern recognition and more accurate predictions. AI will help to compare actual performance with the predicted one (see Section 14.6). Machine learning and other AI technologies also provide for constant improvement strategy. Nadav also suggested adding expert opinions via collective intelligence, as presented in Chapter 11.
In the remaining part of this section, we present detailed aspects of convergence of some intelligent systems.
Big Data Is Empowering AI Technologies
Big Data is characterized by its volume, variety, and velocity that exceed the reach of commonly used hardware environments and/or the capabilities of software tools to pro- cess data. However, today there are technologies and methods that enable capturing,
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 61
cleaning, and analyzing Big Data. These technologies and methods enable companies to make real-time decisions. The convergence with AI and machine learning is a major force in this direction. The availability of new Big Data analytics enables new capabilities in AI technologies that were not possible until recently. According to Bean (2017), Big Data can empower AI due to:
• The new capabilities of processing Big Data at a much reduced cost. • The availability of large data sets online. • The scale up of algorithms, including deep learning, is enabling powerful AI
capabilities.
MetLife Example: Convergence of AI and Big Data
MetLife is a Canadian-based global insurance company that is known for its use of IT to smooth its operation and increase customer satisfaction. To get the most from technology, the company uses AI that has been enabled by Big Data analysis as follows:
• Tracking incidents and their outcomes has been improved by speech recognition. • Machine learning indicates pending failures. In addition, handwritten reports made
by doctors about people injured or were sick and claims paid by the insurance com- pany are analyzed in seconds by the system.
• Expediting the execution of underwriting policies in property and casualty insur- ance is done by using both AI and analytics.
• The back-office side of claim processing includes many unstructured data that are incorporated in claims. Part of the analysis includes patients’ health data. Machine learning is used to recognize anomalies in reports very quickly.
For more about AI and the insurance business, see Chapter 2. For more on the con- vergence of Big Data and AI in general and at MetLife, see Bean (2017).
The Convergence of AI and the IoT
The opening vignette illustrated to us how AI technologies when combined with IoT can provide solutions to complex problems. IoT collects a large amount of data from sensors and other “things.” These data need to be processed for decision support. Later we will see how Microsoft’s Cortana does this. Butner (2018) describes how combining AI and IoT can lead to the “next-level solutions and experiences.” The emphasis in such combination is on learning more about customers and their needs. This integration also can facilitate competitive analysis and business operation (see the opening vignette). The combined pair of AI and IoT, especially when combined with Big Data, can help facilitate the discovery of new products, business processes, and opportunities. The full potential of IoT can be leveraged with AI technologies. In addition, the only way to make sense of the data streamed from the “things” via IoT and to obtain the insight from them is to subject them to AI analysis. Faggela (2017) provides the following three examples of combining AI and IoT:
1. The smart thermostat of Nest Labs (see smart homes in Chapter 13). 2. Automated vacuum cleaners, like iRobot Roomba (see Chapter 2, intelligent
vacuums). 3. Self-driving vehicles (see Chapter 13).
The IoT can become very intelligent when combined with IBM Watson Analytics that includes machine learning. Examples are presented in the opening vignette and the opening vignette to Chapter 13.
62 Part I • Introduction to Analytics and AI
The Convergence with Blockchain and Other Technologies
Several experts raise the possibility of the convergence of AI, analytics, and blockchain (e.g., Corea, 2017; Kranz, 2017). The idea is that such convergence may contribute to design or redesign of paradigms and technologies. The blockchain technology can add security to data shared by all parties in a distributed network, where transaction data can be recorded. Kranz believes that the conversion with blockchain will power new solu- tions to complex problems. Such a convergence should include the IoT. Kranz also see a role for fog computing (Chapter 9). Such a combination can be very useful in complex applications such as autonomous vehicles and in Amazon’s Go (Application Case 1.10).
In early 2018, Amazon.com opened its first fully automated convenience store in downtown Seattle. The company had had success with this type of store during 2017, experimenting with only the company’s employees.
Shoppers enter the store, pick up products, and go home. Their accounts are charged later on. Sounds great! No more waiting in line for the packing of your goods and paying for them – no cashiers, no hassle.
In some sense, shoppers are going through a process similar to what they do online—find desired products/services, buy them, and wait for the monthly electronic charge.
The Shopping Process
To participate, you need a special free app on your smartphone. You need to connect it to your regular Amazon.com account. Here is what you do next:
1. Open your app. 2. Wave your smartphone at a gate to the store. It
will work with a QR code there. 3. Enter the store. 4. Start shopping. All products are prepacked.
You put them in a shopping bag (yours or one borrowed at the store). The minute you pick an item from the shelf, it is recorded in a virtual shopping cart. This activity is done by sensors/cameras. Your account is debited. If you change your mind, and return an item, the system will credit your account instantly. The sensors also track your movements in the store. (This is an issue of digital privacy; see Chapter 14, Section 14.3). The sensors are of RFID type (Chapter 13).
5. Finished shopping? Just leave the store (make sure your app is open for the gate to let you leave). The system knows that you have left and
what products you took, and your shopping trip is finished. The system will total your cost, which you can check anytime on your smartphone.
6. Amazon.com records your shopping habits (again, a privacy issue), which will help your fu- ture shopping experience and will help Amazon to build recommendations for you (Chapter 2). The objective of Go is to guide you to healthy food! (Amazon sells its meal kits of healthy food there.)
Note: Today, only few people work in the store! Employees stock shelves and assist you otherwise. The company plans to open several additional stores in 2018.
The Technology Used
Amazon disclosed some of the technologies used. These are deep learning algorithms, computer vision, and sensor fusion. Other technologies were not disclosed. See the videoyoutube.com/ watch?v=NrmMk1Myrxc (1:50 min .).
Sources: Condensed forC. Jarrett. (2018). “Amazon Set to Open Doors on AI-Powered Grocery Store.” Venturebeat.com. venturebeat.com/2018/01/21/amazon-set-to-open-doors- on-ai-powered-grocery-store/ (accessed September 2018); D. Reisinger. (2018, February 22). “Here Are the Next Cities to Get Amazon Go Cashier-Less Stores.” Fortune.
Questions for Case 1.9
1. Watch the video. What did you like in it, and what did you dislike?
2. Compare the process described here to a self- check available today in many supermarkets and “big box” stores (Home Depot, etc.).
3. The store was opened in downtown Seattle. Why was the downtown location selected?
4. What are the benefits to customers? To Amazon?
5. Will customers be ready to trade privacy for con- venience? Discuss.
Application Case 1.10 Amazon Go Is Open for Business
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 63
For a comprehensive report regarding convergence of intelligent technologies, see reportbuyer.com/product/5023639/.
In addition to blockchain, one can include IoT and Big Data, as suggested earlier, as well as more intelligent technologies (e.g., machine vision, voice technologies). These may have enrichment effects. In general, the more technologies are used (presumably properly), the more complex problems may be solved, and the more efficient the perfor- mance of the convergence systems (e.g., speed, accuracy) will be. For a discussion, see i-scoop.eu/convergence-ai-iot-big-data-analytics/.
IBM and Microsoft Support for Intelligent Systems Convergence
Many companies provide tools or platforms for supporting intelligent systems conver- gence. Two examples follow.
IBM IBM is combining two of its platforms to support the convergence of AI and ana- lytics. Power AI is a distribution platform for AI and machine learning. This is a way to support the IBM analytics platform called Data Science Experience (cloud enabled). The combination of the two enables improvements in data analytics process. It also en- ables data scientists to facilitate the training of complex AI models and neural networks. Researchers can use the combined system for deep learning projects. All in all, this combi- nation provides better insight to problem solving. For details, see FinTech Futures (2017).
As you may recall from the opening vignette, IBM Watson is also combining analyt- ics, AI, and IoT in cognitive buildings projects.
MICROSOFT’S CORTANA INTELLIGENCE SUITE Microsoft offers from its AZURE cloud (Chapter 13) a combination of advanced analytics, traditional BI, and Big Data analytics. The suite enables users to transform data into intelligent actions.
Using Cortana, one can transform data from several sources, including from IoT sensors, and apply both advanced analytics (e.g., data mining) and AI (e.g., ma- chine learning) and extract insights and actionable recommendations, which are de- livered to decision makers, to apps, or to fully automated systems. For the details of the system and the architecture of Cortana, see mssqltips.com/sqlservertip/4360/ introduction-to-microsoft-cortana-intelligence-suite/.
u SECTION 1.8 REVIEW QUESTIONS
1. What are the major benefits of intelligent systems convergences? 2. Why did analytics initiatives fail at such a high rate in the past? 3. What synergy can be created by combining AI and analytics? 4. Why is Big Data preparation essential for AI initiatives? 5. What are the benefits of adding IoT to intelligent technology applications? 6. Why it is recommended to use blockchain in support of intelligent applications?
1.9 OVERVIEW OF THE ANALYTICS ECOSYSTEM
So you are excited about the potential of analytics, data science, and AI and want to join this growing industry. Who are the current players, and what to do they do? Where might you fit in? The objective of this section is to identify various sectors of the analytics in- dustry, provide a classification of different types of industry participants, and illustrate the types of opportunities that exist for analytics professionals. Eleven different types of play- ers are identified in an analytics ecosystem. An understanding of the ecosystem also gives the reader a broader view of how the various players come together. A secondary
64 Part I • Introduction to Analytics and AI
purpose of understanding the analytics ecosystem for a professional is also to be aware of organizations and new offerings and opportunities in sectors allied with analytics.
Although some researchers have distinguished business analytics professionals from data scientists (Davenport and Patil, 2012), as pointed out previously, for the purpose of understanding the overall analytics ecosystem, we treat them as one broad profession. Clearly, skill needs can vary for a strong mathematician to a programmer to a modeler to a communicator, and we believe this issue is resolved at a more micro/individual level rather than at a macro level of understanding the opportunity pool. We also take the wid- est definition of analytics to include all three types as defined by INFORMS—descriptive/ reporting/visualization, predictive, and prescriptive as described earlier. We also include AI within this same pool.
Figure 1.17 illustrates one view of the analytics ecosystem. The components of the ecosystem are represented by the petals of an analytics flower. Eleven key sectors or clus- ters in the analytics space are identified. The components of the analytics ecosystem are grouped into three categories represented by the inner petals, outer petals, and the seed (middle part) of the flower. The outer six petals can be broadly termed technology provid- ers. Their primary revenue comes from providing technology, solutions, and training to analytics user organizations so they can employ these technologies in the most effective and efficient manner. The inner petals can be generally defined as the analytics accelera- tors. The accelerators work with both technology providers and users. Finally, the core of the ecosystem comprises the analytics user organizations. This is the most important component as every analytics industry cluster is driven by the user organizations.
The metaphor of a flower is well suited for the analytics ecosystem as multiple com- ponents overlap each other. Similar to a living organism like a flower, all these petals grow and wither together. Many companies play in multiple sectors within the analytics industry and thus offer opportunities for movement within the field both horizontally and vertically.
Data Generation
Infrastructure Providers
Analytics- Focused Software
Developers
Data Service Providers
Middleware Providers
Data Warehouse Providers
Analytics User
Organization
Regulators and Policy Makers
Data Management Infrastructure
Providers
Academic Institutions and
Certification Agencies
Analytics Industry
Analysts and InfluencersApplication
Developers: Industry Specific
or General
FIGURE 1.17 Analytics Ecosystem.
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 65
More details for the analytics ecosystem are included in our shorter book (Sharda, Delen, and Turban, 2017) as well as in Sharda and Kalgotra (2018). Matt Turck, a venture capitalist with FirstMark, has also developed and updates an analytics ecosystem focused on Big Data. His goal is to keep track of new and established players in various segments of the Big Data industry. A very nice visual image of his interpretation of the ecosystem and a comprehensive listing of companies is available through his Web site: http:// mattturck.com/2016/02/01/big-data-landscape/ (accessed September 2018).
1.10 PLAN OF THE BOOK
The previous sections have given you an understanding of the need for information technology in decision making, the evolution of BI, analytics, data science, and artificial intelligence. In the last several sections, we have seen an overview of various types of analytics and their applications. Now we are ready for a more detailed managerial excur- sion into these topics along with some deep hands-on experience in some of the techni- cal topics. Figure 1.18 presents a plan on the rest of the book.
Analytics, Data Science, and Artificial Intelligence: Systems for Decision Support (11th Edition)
Chapter 1 An Overview of BA, DSS, BI,
DS and AI
Chapter 2 Artificial
Intelligence Concepts, Drivers, Major
Technologies, and Business Applications
Chapter 3 Nature of Data,
Statistical Modeling and Visualization
Chapter 8 Optimization
and Simulation
Chapter 9 Big Data, Data Centers, and
Cloud Computing
Chapter 4 Algorithms rather than Applications in the title
Chapter 5 Machine- Learning
Techniques for Predictive
Analytics
Chapter 7 Text Mining, Sentiment Analysis, and Social Analytics
Chapter 6 Deep Learning and Cognitive Computing
Chapter 10 Robotics;
Industrial and Consumer
Applications
Chapter 11 Group Decision
Making, Collaborative
Systems, and AI Support
Chapter 13 TThe Internet of
Things As A Platform For
Intelligent Applications
Chapter 12 Knowledge Systems:
Expert Systems, Recommenders, Chatbots, Virtual
Personal Assistants, and Robo Advisors
Chapter 14 Implementation
Issues: From Ethics and
Privacy to Organizational and Societal
Impacts
Introduction to Analytics and Al
Predictive Analytics/ Machine Learning
Prescriptive Analytics and Big Data
Robotics, Social Networks, Al and IoT
Caveats of Analytics and Al
FIGURE 1.18 Plan of the Book.
66 Part I • Introduction to Analytics and AI
In this chapter, we have provided an introduction, definitions, and overview of DSS, BI, and analytics, including Big Data analytics and data science. We also gave you an overview of the analytics ecosystem to have you appreciate the breadth and depth of the industry. Chapters 2 and 3 cover descriptive analytics and data issues. Data clearly form the foundation for any analytics application. Thus, we cover an introduction to data warehousing issues, applications, and technologies. This chapter also covers business reporting and visualization technologies and applications.
We follow the current chapter with a deeper introduction to artificial intelligence in Chapter 2. Because data are fundamental to any analysis, Chapter 3 introduces data issues as well as descriptive analytics, including statistical concepts and visualization. An online chapter covers data warehousing processes and fundamentals for those who like to dig more deeply into these issues. The next section of the book covers predictive analytics and machine learning. Chapter 4 provides an introduction to data mining applications and the data mining process. Chapter 5 introduces many of the common data mining techniques: classification, clustering, association mining, and so forth. Chapter 6 includes coverage of deep learning and cognitive computing. Chapter 7 focuses on text mining applications as well as Web analytics, including social media analytics, sentiment analysis, and other related topics. The following section brings the “data science” angle into further depth. Chapter 8 covers prescriptive analytics. Chapter 9 includes more details of Big Data analytics. It also includes an introduction to cloud-based analytics as well as location analytics. The next section covers robotics, social networks, AI, and IoT. Chapter 10 introduces robots in business and consumer applications and discusses the future impact of such devices on society. Chapter 11 focuses on collaboration systems, crowdsourcing, and social networks. Chapter 12 reviews personal assistants, chatbots, and the exciting developments in this space. Chapter 13 studies IoT and its potential in decision support and a smarter society. The ubiquity of wireless and GPS devices and other sensors is resulting in the creation of massive new databases and unique applications. A new breed of analytics companies is emerging to analyze these new databases and create a much better and deeper un- derstanding of customers’ behaviors and movements. It is leading to the automation of analytics and has spanned a new area called the “Internet of Things.” Finally, Chapter 14 concludes with a brief discussion of security, privacy, and societal dimensions of analytics/AI.
1.11 RESOURCES, LINKS, AND THE TERADATA UNIVERSITY NETWORK CONNECTION
The use of this chapter and most other chapters in this book can be enhanced by the tools described in the following sections.
Resources and Links
We recommend the following major organizational resources and links:
• The Data Warehousing Institute (tdwi.org). • Data Science Central (datasciencecentral.com). • DSS Resources (dssresources.com). • Microsoft Enterprise Consortium (enterprise.waltoncollege.uark.edu/mec.asp).
Vendors, Products, and Demos
Most vendors provide software demos of their products and applications. Information about products, architecture, and software is available at dssresources.com.
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 67
Periodicals
We recommend the following periodicals:
• Decision Support Systems (www.journals.elsevier.com/decision-support-systems). • CIO Insight (www.cioinsight.com).
The Teradata University Network Connection
This book is tightly connected with the free resources provided by TUN (see www. teradatauniversitynetwork.com). The TUN portal is divided into two major parts: one for students and one for faculty. This book is connected to the TUN portal via a special section at the end of each chapter. That section includes appropriate links for the specific chapter, pointing to relevant resources. In addition, we provide hands-on exercises using software and other materials (e.g., cases) available at TUN.
The Book’s Web Site
This book’s Web site, pearsonhighered.com/sharda, contains supplemental textual mate- rial organized as Web chapters that correspond to the printed book’s chapters. The topics of these chapters are listed in the online chapter table of contents.
As this book went to press, we verified that all cited Web sites were active and valid. However, URLs are dynamic. Web sites to which we refer in the text sometimes change or are discontinued because companies change names, are bought or sold, merge, or fail. Sometimes Web sites are down for maintenance, repair, or redesign. Many organizations have dropped the initial “www” designation for their sites, but some still use it. If you have a problem connecting to a Web site that we mention, please be patient and simply run a Web search to try to identify a possible new site. Most times, you can quickly find the new site through one of the popular search engines. We apologize in advance for this inconvenience.
Chapter Highlights
• The business environment is becoming more complex and is rapidly changing, making deci- sion making more difficult.
• Businesses must respond and adapt to the chang- ing environment rapidly by making faster and better decisions.
• A model is a simplified representation or abstrac- tion of reality.
• Decision making involves four major phases: in- telligence, design, choice, and implementation.
• In the intelligence phase, the problem (op- portunity) is identified, classified, and decom- posed (if needed), and problem ownership is established.
• In the design phase, a model of the system is built, criteria for selection are agreed on, alterna- tives are generated, results are predicted, and a decision methodology is created.
• In the choice phase, alternatives are compared, and a search for the best (or a good-enough)
solution is launched. Many search techniques are available.
• In implementing alternatives, a decision maker should consider multiple goals and sensitivity- analysis issues.
• The time frame for making decisions is shrinking, whereas the global nature of decision making is ex- panding, necessitating the development and use of computerized DSS.
• An early decision support framework divides decision situations into nine categories, de- pending on the degree of structuredness and managerial activities. Each category is sup- ported differently.
• Structured repetitive decisions are supported by standard quantitative analysis methods, such as MS, MIS, and rule-based automated decision support.
• DSS use data, models, and sometimes knowledge management to find solutions for semistructured and some unstructured problems.
68 Part I • Introduction to Analytics and AI
• The major components of a DSS are a database and its management, a model base and its man- agement, and a user-friendly interface. An intel- ligent (knowledge-based) component can also be included. The user is also considered to be a component of a DSS.
• BI methods utilize a central repository called a DW that enables efficient data mining, OLAP, BPM, and data visualization.
• BI architecture includes a DW, business analyt- ics tools used by end users, and a user interface (such as a dashboard).
• Many organizations employ descriptive analytics to replace their traditional flat reporting with in- teractive reporting that provides insights, trends, and patterns in the transactional data.
• Predictive analytics enables organizations to es- tablish predictive rules that drive the business outcomes through historical data analysis of the existing behavior of the customers.
• Prescriptive analytics helps in building models that involve forecasting and optimization tech- niques based on the principles of OR and man- agement science to help organizations to make better decisions.
• Big Data analytics focuses on unstructured, large data sets that may also include vastly different types of data for analysis.
• Analytics as a field is also known by industry- specific application names, such as sports analyt- ics. It is also known by other related names such as data science or network science.
• Healthcare and retail chains are two areas where analytics applications abound, with much more to come.
• Image analytics is a rapidly evolving field leading to many applications of deep learning.
• The analytics ecosystem can be first viewed as a collection of providers, users, and facilitators. It can be broken into 11 clusters.
Key Terms
analytics analytics ecosystem artificial intelligence augmented intelligence Big Data analytics business intelligence (BI) choice phase
dashboard data mining decision or normative analytics descriptive (or reporting) analytics design phase implementation phase intelligence phase
online analytical processing (OLAP)
online transaction processing (OLTP)
predictive analytics prescriptive analytics
Questions for Discussion
1. Survey the literature from the past six months to find one application each for DSS, BI, and analytics. Summarize the applications on one page, and submit it with the exact sources.
2. Your company is considering opening a branch in China. List typical activities in each phase of the deci- sion (intelligence, design, choice, and implementation) regarding whether to open a branch.
3. You are about to buy a car. Using Simon’s (1977) four- phase model, describe your activities at each step in making the decision.
4. Explain, through an example, the support given to deci- sion makers by computers in each phase of the decision process.
5. Comment on Simon’s (1977) philosophy that managerial decision making is synonymous with the whole process of management. Does this make sense? Explain. Use a real-world example in your explanation.
6. Review the major characteristics and capabilities of DSS. How does each of them relate to the major components of DSS?
7. List some internal data and external data that could be found in a DSS for a university’s admissions office.
8. Distinguish BI from DSS. 9. Compare and contrast predictive analytics with prescrip-
tive and descriptive analytics. Use examples. 10. Discuss the major issues in implementing BI.
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 69
Exercises
Teradata University Network and Other Hands-On Exercises
1. Go to the TUN site teradatauniversitynetwork.com. Using the site password your instructor provides, register for the site if you have not already previously registered. Log on and learn the content of the site. You will receive assignments related to this site. Prepare a list of 20 items on the site that you think could be beneficial to you.
2. Go to. Explore the Sports Analytics page, and summa- rize at least two applications of analytics in any sport of your choice.
3. Go to. The TUN site, and select “Cases, Projects, and Assignments.” Then select the case study “Harrah’s High Payoff from Customer Information.” Answer the follow- ing questions about this case: a. What information does the data mining generate? b. How is this information helpful to management in
decision making? (Be specific.) c. List the types of data that are mined. d. Is this a DSS or BI application? Why?
4. Go to teradatauniversitynetwork.com and find the paper titled “Data Warehousing Supports Corporate Strategy at First American Corporation” (by Watson, Wixom, and Goodhue). Read the paper, and answer the following questions: a. What were the drivers for the DW/BI project in the
company? b. What strategic advantages were realized? c. What operational and tactical advantages were achieved? d. What were the critical success factors for the
implementation? 5. Go to http://analytics-magazine.org/issues/digital-
editions and find the January/February 2012 edition titled “Special Issue: The Future of Healthcare.” Read the article “Predictive Analytics—Saving Lives and Lowering Medical Bills.” Answer the following questions: a. What problem is being addressed by applying pre-
dictive analytics? b. What is the FICO Medication Adherence Score? c. How is a prediction model trained to predict the FICO
Medication Adherence Score HoH? Did the prediction model classify the FICO Medication Adherence Score?
d. Zoom in on Figure 4, and explain what technique is applied to the generated results.
e. List some of the actionable decisions that were based on the prediction results.
6. Go to http://analytics-magazine.org/issues/digital- editions, and find the January/February 2013 edition titled “Work Social.” Read the article “Big Data, Analytics and Elections,” and answer the following questions: a. What kinds of Big Data were analyzed in the article’s
Coo? Comment on some of the sources of Big Data.
b. Explain the term integrated system. What is the other technical term that suits an integrated system?
c. What data analysis techniques are employed in the project? Comment on some initiatives that resulted from data analysis.
d. What are the different prediction problems answered by the models?
e. List some of the actionable decisions taken that were based on the prediction results.
f. Identify two applications of Big Data analytics that are not listed in the article.
7. Search the Internet for material regarding the work of managers and the role analytics plays in it. What kinds of references to consulting firms, academic departments, and programs do you find? What major areas are represented? Select five sites that cover one area, and report your findings.
8. Explore the public areas of dssresources.com. Prepare a list of its major available resources. You might want to refer to this site as you work through the book.
9. Go to microstrategy.com. Find information on the five styles of BI. Prepare a summary table for each style.
10. Go to oracle.com, and click the Hyperion link under Applications. Determine what the company’s major products are. Relate these to the support technologies cited in this chapter.
11. Go to the TUN questions site. Look for BSI videos. Review the video of “Case of Retail Tweeters.” Prepare a one-page summary of the problem, proposed solu- tion, and the reported results. You can also find associ- ated slides on slideshare.net.
12. Review the Analytics Ecosystem section. Identify at least two additional companies in at least five of the industry clusters noted in the discussion.
13. The discussion for the analytics ecosystem also included several typical job titles for graduates of analytics and data science programs. Research Web sites such as datasciencecentral.com and tdwi.org to locate at least three similar job titles that you may find interesting for your career.
14. Go to Brainspace at MIT lab brainspace.com. View the video about “Augmented Human Intelligence.” Find the activities that deal with the enabling of meaningful combination of people and machines. Write a report.
15. Find information about IBM Watson’s activities in the healthcare field. Write a report.
16. Examine Daniel Power’s DSS Resources site at dssresources.com. Take the Decision Support Systems Web Tour (dssresources.com/tour/index.html). Explore other areas of the Web site. List at least three recent resources related to analytics. What topics do these cover?
70 Part I • Introduction to Analytics and AI
References
http://canopeoapp.com/ (accessed October 2018). http://imazon.org.br/en/imprensa/mapping-change-
in-the-amazon-how-satellite-images-are-halting- deforestation/ (accessed October 2018).
http://www.earthcastdemo.com/2018/07/bloomberg- earthcast-customizing-weather/ (accessed October 2018)
h t t p s : / / w w w. w o r l d b a n k . o r g / e n / n e w s / p r e s s - release/2018/02/22/world-bank-supports-sierra- leones-efforts-in-landslide-recovery (accessed October 2018)
Siemens.com. About Siemens. siemens.com/about/en/ (accessed September 2018).
Silvaris.com. Silvaris overview. silvaris.com (accessed September 2018).
Agah, A. (2017). Medical Applications of Artificial Intelligence. Boca Raton, FL: CRC Press.
Anthony, R. N. (1965). Planning and Control Systems: A Framework for Analysis. Cambridge, MA: Harvard Univer- sity Graduate School of Business.
Baker, J., and M. Cameron. (1996, September). “The Effects of the Service Environment on Affect and Consumer Perception of Waiting Time: An Integrative Review and Research Propositions.” Journal of the Academy of Market- ing Science, 24, pp. 338–349.
Bean, R. (2017, May 8). “How Big Data Is Empowering AI and Machine Learning at Scale.” MIT Sloan Management Review.
Boddington, P. (2017, January 3). “Case Study: Robot Camel Jockeys. Yes, really.” Ethics for Artificial Intelligence.
Brainspace. (2016, June 13). “Augmenting Human Intelli- gence.” MIT Technology Review Insights.
Butner, K. (2018, January 8). “Combining Artificial Intelligence with the Internet of Things Could Make Your Business Smarter.” IBM Consulting Blog.
CDC.gov. (2015, September 21). “Important Facts about Falls.” cdc.gov/homeandrecreationalsafety/falls/adultfalls. html (accessed September 2018).
Charles, T. (2018, May 21). “Influence of the External Environment on Strategic Decision.” Azcentral. yourbusiness. azcentral. com/influence-external-environment- strategic- decisions-17628.html/ (accessed October 2018).
Chiguluri, V., Guthikonda, K., Slabaugh, S., Havens, E., Peña, J., & Cordier, T. (2015, June). Relationship Between Diabetes Complications and Health Related Quality of Life Among an Elderly Population in the Unit- ed States. Poster presentation at the American Diabe- tes Association Seventy- Fifth Annual Scientific Sessions. Boston, MA.
Chongwatpol, J., & R. Sharda. (2010, December). “SNAP: A DSS to Analyze Network Service Pricing for State Net- works.” Decision Support Systems, 50(1), pp. 347–359.
Chung, C. (2016). “Dubai Camel Race Ride-Along.” YouTube. com. youtube.com/watch?v=xFCRhk4GYds (accessed September 2018).
Cordier, T., Slabaugh, L., Haugh, G., Gopal, V., Cusano, D., Andrews, G., & Renda, A. (2015, September). Quality of Life Changes with Progressing Congestive Heart Failure. Poster presentation at the Nineteenth Annual Scientific Meeting of the Heart Failure Society of America, Wash- ington, DC.
Corea, F. (2017, December 1). “The Convergence of AI and Blockchain: What’s the Deal?” Medium.com. medium. com/@Francesco_AI/the-convergence-of-ai-and- blockchain-whats-the-deal-60c618e3accc (accessed September 2018).
Davenport, T., & SAS Institute Inc. (2014, February). “Analytics in Sports: The New Science of Winning.” sas.com/con- tent/dam/SAS/en_us/doc/whitepaper2/iia-analytics- in-sports-106993.pdf (accessed September 2018).
Davenport, T. H., & Patil, D. J. (2012). “Data Scientist.” Har- vard Business Review, 90, 70–76.
Deahl, D. (2018, January 7). “This Automatic Feeder Can Tell the Difference Between Your Pets.” The Verge.
De Smet, A., et al. (2017, June). “Untangling Your Organiza- tion’s Decision Making.” McKinsey Quarterly.
Duncan, A. (2016). “The BICC Is Dead.” https://blogs. gartner.com/alan-duncan/2016/03/11/the-bicc-is- dead/ (accessed October 2018).
Dundas.com. “How Siemens Drastically Reduced Cost with Managed BI Applications.” www.dundas.com/Con- tent/pdf/siemens-case-study.pdf (accessed September 2018).
Eckerson, W. (2003). Smart Companies in the 21st Century: The Secrets of Creating Successful Business Intelligent Solu- tions. Seattle, WA: The Data Warehousing Institute.
eMarketer. (2017, May). “Artificial Intelligence: What’s Now, What’s New and What’s Next?” EMarketer Inc.
Emc.com. (n.d.). “Data Science Revealed: A Data-Driven Glimpse into the Burgeoning New Field.” emc.com/col- lateral/about/news/emc-data-science-study-wp.pdf (accessed September 2018)
Emmert, Samantha. (2018, March 19). “Fighting Illegal Fishing.” Global Fishing Watch. globalfishingwatch.org/research/ fighting-illegal-fishing/ (accessed October 2018).
Faggela, D. (2017, August 24). “Artificial Intelligence Plus the Internet of Things (IoT): 3 Examples Worth Learning From.” TechEmergence.
Faggela, D. (2018, March 29). “Artificial Intelligence Industry: An Overview by Segment.” TechEmergence.
Fernandez, J. (2017, April). “A Billion People a Day. Millions of Elevators. No Room for Downtime.” IBM developer Works Blog. developer.ibm.com/dwblog/2017/kone- watson-video/ (accessed September 2018).
Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 71
FinTech Futures. (2017, October 11). “IBM Combining Data Sci- ence and AI for Analytics Advance.” BankingTech.com.
Gartner, Inc. (2004). Using Business Intelligence to Gain a Competitive Edge. A special report.
Gates, S., Smith, L. A., Fisher, J. D., et al. (2008). Systematic Re- view of Accuracy of Screening Instruments for Predicting Fall Risk Among Independently Living Older Adults. Jour- nal of Rehabilitation Research and Development, 45(8), pp. 1105–1116.
Gill, T. M., Murphy, T. E., Gahbauer, E. A., et al. (2013). “Associ- ation of Injurious Falls with Disability Outcomes and Nurs- ing Home Admissions in Community Living Older Persons.” American Journal of Epidemiology, 178(3), pp. 418–425.
Gorry, G. A., & Scott-Morton, M. S. (1971). “A Framework for Management Information Systems.” Sloan Management Review, 13(1), pp. 55–70.
Havens, E., Peña, J., Slabaugh, S., Cordier, T., Renda, A., & Gopal, V. (2015, October). Exploring the Relationship Be- tween Health-Related Quality of Life and Health Condi- tions, Costs, Resource Utilization, and Quality Measures. Podium presentation at the ISOQOL Twenty-Seventh Annual Conference, Vancouver, Canada.
Havens, E., Slabaugh, L., Peña, J., Haugh, G., & Gopal, V. (2015, February). Are There Differences in Healthy Days Based on Compliance to Preventive Health Screening Measures? Poster presentation at Preventive Medicine 2015, Atlanta, GA.
Healthcare IT News. (2017, November 9). “How AI Is Trans- forming Healthcare and Solving Problems in 2017.” Slideshow. healthcareitnews.com/slideshow/how- ai-transforming-healthcare-and-solving-problems- 2017?page=4/ (accessed September 2018).
Hesse, R., & G. Woolsey. (1975). Applied Management Sci- ence: A Quick and Dirty Approach. Chicago, IL: SRA Inc.
Humana. 2016 Progress Report. populationhealth. h u m a n a . c o m / w p - c o n t e n t / u p l o a d s / 2 0 1 6 / 0 5 / BoldGoal2016ProgressReport_1.pdf (accessed Sep- tember 2018).
INFORMS. Analytics Section Overview. informs.org/ Community/Analytics (accessed September 2018).
Jacquet, F. (2017, July 4). “Exploring the Artificial Intelligence Ecosystem: AI, Machine Learning, and Deep Learning.” DZone.
Jarrett, C. (2018, January 21). “Amazon Set to Open Doors on AI-Powered Grocery Store.” Venturebeat.com. venture- beat.com/2018/01/21/amazon-set-to-open-doors-on- ai-powered-grocery-store/ (accessed September 2018).
Keen, P. G. W., & M. S. Scott-Morton. (1978). Decision Sup- port Systems: An Organizational Perspective. Reading, MA: Addison-Wesley.
Kemper, C., and C. Breuer. (2016). “How Efficient Is Dynamic Pricing for Sports Events? Designing a Dynamic Pricing Model for Bayern Munich.” International Journal of Sports Finance, 11, pp. 4–25.
Kranz, M. (2017, December 27). “In 2018, Get Ready for the Convergence of IoT, AI, Fog, and Blockchain.” Insights.
Liberto, D. (2017, June 29). “Artificial Intelligence Will Add Trillion to the Global Economy: PwC.” Investopedia.
Lollato, R., Patrignani, A., Ochsner, T. E., Rocatelli, A., Tomlinson, P. & Edwards, J. T. (2015). “Improving Grazing Management Using a Smartphone App.” www. bookstore.ksre.ksu.edu/pubs/MF3304.pdf (accessed October 2018).
Lopez, J. (2017, August 11). “Smart Farm Equipment Helps Feed the World.” IQintel.com.
Metz, C. (2018, February 12). “As China Marches Forward on A.I., the White House Is Silent.” The New York Times.
Nadav, S. (2017, August 9). “Business Intelligence Is Failing; Here Is What Is Coming Next.” Huffington Post.
Norman, A. (2018, January 31). “Your Future Doctor May Not Be Human. This Is the Rise of AI in Medicine.” Futurism.com.
Olshansky, C. (2017, August 24). “Coca-Cola Is Bringing Artifi- cial Intelligence to Vending Machines.” Food & Wine.
Opfer, C. (2016, June 22). “There’s One Terrific Reason to Race Camels Using Robot Jockeys.” Howstuffworks.com.
Padmanabhan, G. (2018, January 4). “Industry-Specific Augment- ed Intelligence: A Catalyst for AI in the Enterprise.” Forbes.
Pajouh Foad, M., Xing, D., Hariharan, S., Zhou, Y., Balasunda- ram, B., Liu, T., & Sharda, R. (2013). Available-to-Promise in Practice: An Application of Analytics in the Specialty Steel Bar Products Industry. Interfaces, 43(6), pp. 503–517. dx. doi.org/10.1287/inte.2013.0693 (accessed September 2018).
Patrignani, A., & Ochsner, T. E., (2015). Canopeo: A Powerful New Tool for Measuring Fractional Green Canopy Cover. Agronomy Journal, 107(6), pp. 2312–2320;
PE Report. (2017, July 29). “Satellite-Based Advance Can Help Raise Farm Output by 20 Percent Experts.” Financial Express.
PricewaterhouseCoopers Report. (2011, December). “Chang- ing the Game: Outlook for the Global Sports Market to 2015.” pwc.com/gx/en/ hospitality-leisure/pdf/ changing-the-game-outlook-for-the-global-sports- market-to-2015.pdf (accessed September 2018).
Quain, S. (2018, June 29). “The Decision-Making Process in an Organization.” Small Business Chron.
Reisinger, D. (2018, February 22). “Here Are the Next Cities to Get Amazon Go Cashier-Less Stores.” Fortune.
Sharda, R., Asamoah, D., & Ponna, N. (2013). “Research and Pedagogy in Business Analytics: Opportunities and Illus- trative Examples.” Journal of Computing and Information Technology, 21(3), pp. 171–182.
Sharda, R., Delen, D., & Turban, E. (2016). Business Intelligence, Analytics, and Data Science: A Managerial Perspective on Analytics. 4th ed. NJ: Pearson.
Sharda, R., & P. Kalgotra. (2018). “The Blossoming Analytics Talent Pool: An Overview of the Analytics Ecosystem.” In James J. Cochran (ed.). INFORMS Analytics Body of Knowl- edge. John Wiley, Hoboken, NJ
72 Part I • Introduction to Analytics and AI
Silk, R. (2017, November). “Biometrics: Facial Recognition Tech Coming to an Airport Near You.” Travel Weekly, 21.
Simon, H. (1977). The New Science of Management Decision. Englewood Cliffs, NJ: Prentice Hall.
Slade, L. (2017, December 21). “Meet the Jordanian Camel Races Using Robot Jockeys.” Sbs.com.au.
Slowey, L. (2017, February 16). “Look Who’s Talking: KONE Makes Elevator Services Truly Intelligent with Watson IoT.” IBM Internet of Things Blog. ibm.com/blogs/internet- of-things/kone/ (accessed September 2018).
“Sports Analytics Market Worth by 2021.” (2015, June 25). Wintergreen Research Press Release. Covered by PR Newswire at http://www.prnewswire.com/news- releases/sports-analytics-market-worth-47-billion- by-2021-509869871.html.
Srikanthan, H. . (2018, January 8). “KONE Improves ‘People Flow’ in 1.1 Million Elevators with IBM Watson IoT.” Ge- neris. https://generisgp.com/2018/01/08/ibm-case- study-kone-corp/ (accessed September 2018).
Staff. “Assisted, Augmented and Autonomous: The 3Flavours of AI Decisions.” (2017, June 28). Software and Technology.
Tableau.com. Silvaris Augments Proprietary Technology Platform with Tableau’s Real-Time Reporting Capabilities. tableau.com/sites/default/files/case-studies/silvaris- business-dashboards_0.pdf (accessed September 2018).
Tartar, Andre, et al. (2018, 26 July). “All the Things Satellites Can Now See from Space.” Bloomberg.com. www. bloomberg.com/news/features/2018-07-26/all-the- things-satellites-can-now-see-from-space (accessed October 2018).
TeradataUniversityNetwork.com. (2015, Fall). “BSI: Sports Analytics—Precision Football” (video). teradatauniver- sity network.com/About-Us/Whats-New/BSI–Sports- Analytics-Precision-Football/ (accessed September 2018).
Thibodeaux, W. (2017, June 29). “This Artificial Intelligence Kiosk Is Designed to Spot Liars at Airports.” Inc.com.
Turck, Matt. “Is Big Data Still a Thing? (The 2016 Big Data Landscape).” http://mattturck.com/2016/02/01/big- data-landscape/ (accessed September 2018).
Watson, H. (2005, Winter). Sorting Out What’s New in Deci- sion Support. Business Intelligence Journal.
Weldon, D. (2018, March 6). “Nearly Half of CIOs Now Plan to Deploy Artificial Intelligence.” Information Management.
Wikipedia.org. On-base Percentage. wikipedia.org/wiki/ On_base_percentage. (accessed September 2018).
Wikipedia.org. Sabermetrics. wikipedia.org/wiki/ Sabermetrics (accessed September 2018).
Wikipedia.org. SIEMENS. wikipedia.org/wiki/Siemens (accessed September 2018).
YouTube.com. (2013, December 17). CenterPoint Energy Talks Real Time Big Data Analytics. youtube.com/ watch?v=s7CzeSlIEfI (accessed September 2018).
Yue, P. (2017, August 24). “Baidu, Beijing Airport Launch Facial Recognition for Passenger Check-In.” China Money Network. https://www.chinamoneynetwork.com/ 2017/08/24/baidu-capital-airport-launch-facial- recognition-system-airport (accessed October 2018).
Zane, E. B. (2016). Effective Decision-Making: How to Make Better Decisions Under Uncertainty And Pressure. Kindle ed. Seattle, WA: Amazon Digital Services.
73
LEARNING OBJECTIVES
Artificial Intelligence Concepts, Drivers, Major Technologies, and Business Applications
2 C H A P T E R
■■ Understand the concepts of artificial intelligence (AI) ■■ Become familiar with the drivers, capabilities, and benefits of AI
■■ Describe human and machine intelligence ■■ Describe the major AI technologies and some derivatives
■■ Discuss the manner in which AI supports decision making
■■ Describe AI applications in accounting ■■ Describe AI applications in banking and financial services
■■ Describe AI in human resource management ■■ Describe AI in marketing ■■ Describe AI in production-operation management
A rtificial intelligence (AI), which was a curiosity for generations, is rapidly de-veloping into a major applied technology with many applications in a variety of fields. OpenAI’s (an AI research institution described in Chapter 14) mission states that AI will be the most significant technology ever created by humans. AI appears in several shapes and has several definitions. In a crude way, it can be said that AI’s aim is to make machines exhibit intelligence as close as possible to what people exhibit, hopefully for the benefit of humans. The latest developments in computing technologies drive AI to new levels and achievements. For example, IDC Spending Guide (March 22, 2018) forecasted that worldwide spending on AI will reach $19.1 billion in 2018. It also predicted annual double-digit spending growth for the near future. According to Sharma (2017), China expects to be the world leader in AI, with a spending of $60 billion in 2025. For the business value of AI, see Greig (2018).
In this chapter, we provide the essentials of AI, its major technologies, its support for decision making, and a sample of its applications in the major business functional areas.
74 Part I • Introduction to Analytics and AI
The chapter has the following sections:
2.1 Opening Vignette: INRIX Solves Transportation Problems 74 2.2 Introduction to Artificial Intelligence 76 2.3 Human and Computer Intelligence 83 2.4 Major AI Technologies and Some Derivatives 87 2.5 AI Support for Decision Making 95 2.6 AI Applications in Accounting 99 2.7 AI Applications in Financial Services 101 2.8 AI in Human Resource Management (HRM) 105 2.9 AI in Marketing, Advertising, and CRM 107
2.10 AI Applications in Production-Operation Management (POM) 110
2.1 OPENING VIGNETTE: INRIX Solves Transportation Problems
THE PROBLEM
Traffic congestion is an ever-increasing problem in many large metropolitan areas. Drivers may spend several hours on the roads each day. In addition, air pollution is increasing, and more accidents are occurring.
THE SOLUTION
INRIX corporation (inrix.com) enables drivers to get real-time traffic information. They can download the INRIX-XD Traffic App for iOS and Android. The information provided is generated by a predictive analysis of massive data obtained from consumers and the environment (e.g., road construction, accidents). Information sources include:
• Traffic data collected by helicopters, drones, and so on, which include real-time traffic flow and accident information.
• Information provided by participating delivery companies and over 100 million anonymous volunteer drivers, who have GPS-enabled smartphones, all reporting in real time.
• Information provided by traffic congestion reports (e.g., delays due to road maintenance).
INRIX processes the collected information with proprietary analytical tools and for- mulas, some of which are AI-based. The processed information is used to generate traffic predictions. For example, it creates a picture of anticipated traffic flows and delays for the next 15 to 20 minutes, few hours, and few days for many locations. These predictions enable drivers to plan their optimal routes. As of 2018, INRIX had offered global coverage in 45 countries and in many major cities, and the company analyzed traffic information from over 100 sources. This service is combined with digital maps. In Seattle, for exam- ple, traffic information is disseminated via smartphones and color codes on billboards along the freeways. Smartphones also display estimated times for the roads to be either clear or jammed. As of 2018, the company had covered over 5,000,000 miles of highways worldwide, delivering upon request the best recommended routes to use, all in real time.
The INRIX system provides information (or recommendations) for decisions such as:
• Optional routes for delivery vehicles and other travelers to take • The best time to go to work or to other places from a given location
Chapter 2 • Artificial Intelligence 75
• Information for rerouting a trip to avoid encountering a traffic jam that just occurred
• Fees to be paid on highways, which are based on traffic conditions and time of the day
The technologies used to collect data are:
• Closed-circuit TV cameras and radar that monitor traffic conditions • Public safety reports and traffic information • Information about freeway access and departure flows • Technologies that measure toll collection queues • Magnetic sensing detectors embedded under the road surface (expensive) • Smartphones and other data collection devices that gather data for INRIX
The information is processed by several AI techniques such as expert systems; see Chapter 12 and different analytical models (such as simulation).
Several of the sources of information are connected to the company via the Internet of Things (IoT) (Chapter 13). According to its Web site, INRIX has partnered with Clear Channel Radio to broadcast real-time traffic data directly to vehicles via Ln Carr or via portable navigation systems, broadcast media, and wireless and Internet-based services. Clear Channel’s Total Traffic Network is available in more than 125 metropolitan areas in four countries (inrix.com/press-releases/2654/). In 2018, the system was installed in over 275 million cars and data collection devices. The system collects real-time traffic information from these devices.
THE RESULTS
In addition to being used by individual drivers, the processed information is shared by organizations and city planners for making planning decisions. Also, less traffic con- gestion has been recorded in participating cities, which results in less pollution, fewer road accidents, and increased productivity by happier employees who spend less time commuting.
The INRIX Traffic App (available for download at inrix.com/mobile-apps) is suit- able for all smartphones; it supports 10 languages, including English, French, and Spanish. For the free INRIX traffic features, see inrixtraffic.com/features. For interesting case studies, see inrix.com/case-studies.
As of 2016, INRIX had released an improved traffic app that uses both AI and crowdsourcing (Chapter 11) to support drivers’ decisions as to the best route to take (Korosec, 2016). The AI technology analyzes drivers’ historical activities to infer their future activities.
Note: Popular smartphone apps, such as Waze and Moovit, provide navigation and data collection similar to INRIX.
Sources: Based on inrix.com, Gitlin (2016), Korosec (2016), and inrix.com/mobile-apps (all accessed June 2018).
u QUESTIONS FOR THE OPENING VIGNETTE
1. Explain why traffic may be down while congestion is up (see the London case at inrix.com/uk-highways-agency/).
2. How does this case relate to decision support? 3. Identify the AI elements in this system. 4. Identify developments related to AI by viewing the company’s press releases from
the most recent four months at inrix.com/press-releases. Write a report.
76 Part I • Introduction to Analytics and AI
5. According to Gitlin (2016), INRIX’s new mobile traffic app is a threat to Waze. Explain why.
6. Go to sitezeus.com/data/inrix and describe the relationship between INRIX and Zeus. View the 2:07 min. video at sitezeus.com/data/inrix/. Why is the system in the video called a “decision helper”?
WHAT WE CAN LEARN FROM THE VIGNETTE
The INRIX case illustrates to us how the collection and analysis of a very large amount of information (Big Data) can improve vehicles’ mobility in large cities. Specifically, by collecting information from drivers and other sources instead of only from expensive sensors, INRIX has been able to optimize mobility. This has been achieved by sup- porting decisions made by drivers and by analyzing traffic flows. INRIX is also using applications from the IoT to connect vehicles and devices with its computing system. This application is one of the building blocks of smart cities (see Chapter 13). The analysis of the collected data is done by using powerful algorithms, some of which are applications of AI.
2.2 INTRODUCTION TO ARTIFICIAL INTELLIGENCE
We would all like to see computerized decision making being simpler, easier to use, more intuitive, and less threatening. And indeed, efforts have been made over time to simplify and automate several tasks in the decision-making process. Just think of the day that refrigerators will be able to measure and evaluate their contents and place orders for goods that need replenishment. Such a day is not too far in the future, and the task will be supported by AI.
CIO Insight projected that by 2035, intelligent computer technologies will result in $5–$8.3 trillion in economic value (see cioinsight.com/blogs/how-ai-will-impact- the-global-economy.html). Among the technologies listed as intelligent ones are the IoT, advanced robotics, and self-driven vehicles, all described in this book. Gartner, a leading technology consulting firm, listed the following in its 2016 and 2017 Hype Cycles for Emerging Technologies: expert advisors, natural language questions and answering, commercial drones, smart workspaces, IoT platforms, smart data discovery, general- purpose machine intelligence, and virtual personal assistants. Most are described or cited in this book (see also Greengard, 2016). For the history of AI, see Zarkadakis (2016) and en.wikipedia.org/wiki/History_of_artificial_intelligence.
Definitions
Artificial intelligence has several definitions (for an overview see Marr 2018); however, many experts agree that AI is concerned with two basic ideas: (1) the study of human thought processes (to understand what intelligence is) and (2) the representation and duplication of those thought processes in machines (e.g., computers, robots). That is, the machines are expected to have humanlike thought processes.
One well-publicized definition of AI is “the capabilities of a machine to imitate intel- ligent human behavior” (per Merriam-Webster Dictionary). The theoretical background of AI is based on logic, which is also used in several computer science innovations. Therefore, AI is considered a subfield of computer science. For the relationship between AI and logic, see plato.stanford.edu/entries/logic-ai.
A well-known early application of artificial intelligence was the chess program hosted at IBM’s supercomputer (Deep Blue). The system beat the famous world cham- pion, Grand Master Garry Kasparov.
Chapter 2 • Artificial Intelligence 77
AI is an umbrella term for many techniques that share similar capabilities and char- acteristics. For a list of 50 unique AI technologies, see Steffi (2017). For 33 types of AI, see simplicable.com/new/types-of-artificial-intelligence.
Major Characteristics of AI Machines
There is an increasing trend to make computers “smarter.” For example, Web 3.0 sup- poses to enable computerized systems that exhibit significantly more intelligence than Web 2.0. Several applications are already based on multiple AI techniques. For example, the area of machine translation of languages is helping people who speak different lan- guages to collaborate as well as to buy online products that are advertised in languages they do not speak. Similarly, machine translation can help people who know only their own language to converse with people speaking other languages and to make decisions jointly in real time.
Major Elements of AI
As described in Chapter 1, the landscape of AI is huge, including hundreds or more components. We illustrate the foundation and the major technologies in Figure 2.1. Notice that we divide them into two groups: Foundations, and Technologies and Applications. The major technologies will be defined later in this chapter and described throughout this book.
Intelligence; Tutoring
Autonomous Vehicles
Speech Understanding
Automatic Programming
Game Playing
Expert SystemsComputer Vision
Intelligent Agents
Natural Language Processing
Neutral Networks
Voice Recognition
Genetic Algorithms
Deep Learning
Engineering
Philosophy
Pattern Recognition
Neurology M2M
Logic Sociology
IoT
Human Cognition
Statistics
Information Systems
Linguistics
Augmented Reality
F ou
nd at
io ns
T ec
hn ol
og ie
s an
d A
pp lic
at io
ns
Machine Learning
Robo Advisors
Smart Cities
Smart Homes
Human Behavior
Psychology
Biology
Fuzzy Logic
Management Science
Robotics
Computer ScienceMathematics
Personal Assistant
The AI
Tree
Smart Factories
FIGURE 2.1 The Functionalities and Applications of Artificial Intelligence.
78 Part I • Introduction to Analytics and AI
AI Applications
The technologies of AI are used in the creation of a large number of applications. In Sections 2.6–2.10, we provide a sampler of applications in the major functional areas of business.
Example
Smart or intelligent applications include those that can help machines to answer cus- tomers’ questions asked in natural languages. Another area is that of knowledge-based systems which can provide advice, assist people to make decisions, and even make de- cisions on their own. For example, such systems can approve or reject buyers’ requests to purchase online (if the buyers are not preapproved or do not have an open line of credit). Other examples include the automatic generating of online purchasing orders and arranging fulfillment of orders placed online. Both Google and Facebook are experiment- ing with projects that attempt to teach machines how to learn and support or even make autonomous decisions. For smart applications in enterprises, see Dodge (2016), Finlay (2017), McPherson (2017), and Reinharz (2017). For how AI solutions are used to facili- tate government services, see BrandStudio (2017).
AI-based systems are also important for innovation and are related to the areas of analytics and Big Data processing. One of the most advanced projects in this area is IBM Watson Analytics (see Chapter 6). For comprehensive coverage of AI, including defini- tions and its history, frontiers, and future, see Kaplan (2016).
Note: In January 2016, Mark Zuckerberg, the CEO of Facebook, announced publicly that his goal for 2016 was to build an AI-based assistant to help with his personal and business activities and decisions. Zuckerberg was teaching a machine to understand his voice and follow his basic commands as well as to recognize the faces of his friends and business partners. Personal assistants are used today by millions of people (see Chapter 12).
Example: Pitney Bowes Is Getting Smarter with AI
Pitney Bowes Inc. is a U.S.-based global business solutions provider in areas such as product shipments, location intelligence, customer engagement, and customer infor- mation management. The company powers billions of physical and digital transactions annually across the connected and borderless world of commerce.
Today, at Pitney Bowes, shipping prices are determined automatically based on the dimensions, weight, and packaging of each package. The fee calculations create data that are fed into AI algorithms. The more data processed, the more accurate are the calcula- tions (a machine-learning characteristic). The company estimates a 25 percent improve- ment in calculations achieved from their algorithms. This gives Pitney Bowes an accurate base for pricing, better customer satisfaction, and improved competitive advantage.
Major Goals of AI
The overall goal of AI is to create intelligent machines that are capable of executing a variety of tasks currently done by people. Ideally, AI machines should be able to reason, think abstractly, plan, solve problems, and learn.
Some specific goals are to:
• Perceive and properly react to changes in the environment that influence specific business processes and operations.
• Introduce creativity in business processes and decision making.
Chapter 2 • Artificial Intelligence 79
Drivers of AI
The use of AI has been driven by the following forces:
• People’s interest in smart machines and artificial brains • The low cost of AI applications versus the high cost of manual labor (doing the
same work) • The desire of large tech companies to capture competitive advantage and market
share of the AI market and their willingness to invest billions of dollars in AI • The pressure on management to increase productivity and speed • The availability of quality data contributing to the progress of AI • The increasing functionalities and reduced cost of computers in general • The development of new technologies, particularly cloud computing
Benefits of AI
The major benefits of AI are as follows:
• AI has the ability to complete certain tasks much faster than humans. • The consistency of the completed AI work can be much better than that of humans.
AI machines do not make mistakes. • AI systems allow for continuous improvement projects. • AI can be used for predictive analysis via its capability of pattern recognition. • AI can manage delays and blockages in business processes. • AI machines do not stop to rest or sleep. • AI machines can work autonomously or be assistants to humans. • The functionalities of AI machines are ever increasing. • AI machines can learn and improve their performance. • AI machines can work in environments that are hazardous to people. • AI machines can facilitate innovations by human (i.e., support research and devel-
opment [R&D]). • No emotional barriers interfere with AI work. • AI excels in fraud detection and in security facilitations. • AI improves industrial operations. • AI optimizes knowledge work. • AI increases speed and enables scale. • AI helps with the integration and consolidating of business operations. • AI applications can reduce risk. • AI can free employees to work on more complex and productive jobs. • AI improves customer care. • AI can solve difficult problems that previously were unsolved (Kharpal, 2017). • AI increases collaboration and speeds up learning.
These benefits facilitate competitive advantage as reported by Agrawal (2018).
Note: Not all AI systems deliver all these benefits. Specific systems may deliver only some of them.
The capability of reducing costs and increasing productivity may result in large in- creases in profit (Violino, 2017). In addition to benefiting individual companies, AI can dramatically increase a country’s economic growth, as it is doing in Singapore.
80 Part I • Introduction to Analytics and AI
EXAMPLES OF AI BENEFITS The following are typical benefits of AI in various areas of applications:
1. The International Swabs and Derivatives Association (ISDA) uses AI to eliminate tedious activities in contract procedures. For example, by using optical character recognition (OCR) integrated with AI, ISDA digitizes contracts and then defines, ex- tracts, and archives the contracts.
2. AI is starting to revolutionize business recruitment by (1) conducting more efficient and fairer candidate screening, (2) making better matches of candidates to jobs, and (3) helping safeguard future talent pipelines for organizations. For details, see SMBWorld Asia Editors (2017) and Section 2.8.
3. AI is redefining management. According to Kolbjørnsrud et al. (2016), the following five practices result from the use of AI:
• It can perform routine administrative tasks. • Managers can focus on the judgment portions of work. • Intelligent machines are treated as colleagues (i.e., managers trust the advice gener-
ated by AI). In addition, there is people–machine collaboration (see Chapter 11). • Managers concentrate on creative abilities that can be supported by AI machines. • Managers are developing social skills, which are needed for better collaboration,
leadership, and coaching.
4. Accenture Inc. developed AI-powered solutions using natural language processing (NLP) and image recognition to help blind people in India improve the way that they can experience the world around them. This enables them to have a better life, and those who work can work better, faster, and do jobs that are more challenging.
5. Ford Motor Credit uses machine learning to spot overlooked borrowers. In addition, it uses machine learning to help its underwriters better understand loan applicants. The program helps the productivity of both underwriters and overlooked applicants. Finally, the system predicts potential borrowers’ creditworthiness, thus minimizing losses for Ford.
6. Alastair Cole uses data collected from several sources with IBM Watson to predict what customers are expecting from the company. The generated data are used for supporting more efficient business decisions.
7. Companies are building businesses around AI. There are many examples of start-ups or existing companies that are attempting to create new businesses.
2000 2010 2020 2030 Time
Hum ans
Wo rk
AI/Robotics
Cost $
FIGURE 2.2 Cost of Human Work versus the Cost of AI Work.
Chapter 2 • Artificial Intelligence 81
Two areas in which large benefits have already been reaped are customer experi- ence and enjoyment. According to a global survey reported by CMO Innovation Editors (2017), 91 percent of top-performing companies deployed AI solutions to support customer experience.
Some Limitations of AI Machines
The following are the major limitations of AI machines:
• Lack human touch and feel • Lack attention to non-task surroundings • Can lead people to rely on AI machines too much (e.g., people may stop to think
on their own) • Can be programmed to create destruction (see discussion in Chapter 14) • Can cause many people to lose their jobs (see Chapter 14) • Can start to think by themselves, causing significant damage (see Chapter 14)
Some of the limitations are diminishing with time. However, risks exist. Therefore, it is necessary to properly manage the development of AI and try to minimize risks.
WHAT AI CAN AND CANNOT DO The limitations just identified constrain the capabili- ties of commercial AI. For example, it could cost too much to be commercially used. Ng (2016) provides an assessment of what AI was able to do by 2016. This is important for two reasons: (1) executives need to know what AI can do economically and how compa- nies can use it to benefit their business and (2) executives need to know what AI cannot economically do.
AI is already transforming Web search, retailing and banking services, logistics, online commerce, entertainment, and more. Hundreds of millions of people use AI on their smartphones and in other ways. However, according to Ng (2016), applications in these areas are based on how simple input is converted to simple output as a response; for example, in automatic loan approval, the input is the profile of the applicant and the output will be an approval or rejection.
Applications in these areas are normally fully automated. Automated tasks are usu- ally repetitive and done by people with short periods of training. AI machines depend on data that may be difficult to get (e.g., belong to someone else) or inaccurate. A second barrier is the need for AI experts, who are difficult to find and/or expensive to hire. For other barriers, see Chapter 14.
Three Flavors of AI Decisions
Staff (2017) divided the capabilities of AI systems into three levels: assisted, autonomous, and augmented.
ASSISTED INTELLIGENCE This is equivalent mostly to weak AI, which works only in narrow domains. It requires clearly defined inputs and outputs. Examples are some moni- toring systems and low-level virtual personal assistants (Chapter 12). Such systems and assistants are used in our vehicles for giving us alerts. Similar systems can be used in many healthcare applications (e.g., monitoring, diagnosing).
AUTONOMOUS AI These systems are in the realm of the strong AI but in a very narrow domain. Eventually, a computer will take over many tasks, automating them completely. Machines act as experts and have absolute decision-making power. Pure robo-advisors (Chapter 12) are examples of such machines. Autonomous vehicles and robots that can fix themselves are also good examples.
82 Part I • Introduction to Analytics and AI
AUGMENTED INTELLIGENCE Most of the existing AI applications, which are between assisted and autonomous, are referred to as augmented intelligence (or intelligence augmentation). Their technology can augment computer tasks to extend human cogni- tive abilities (see Chapter 6 on cognitive computing), resulting in high performance, as described in Technology Insight 2.1.
Artificial Brain
The artificial brain is a people-made machine that is desired to be as intelligent, creative, and self-aware as humans. To date, no one has been able to create such a machine; see artificialbrains.com. A leader in this area is IBM. IBM and the U.S. Air Force have built a system equivalent to 64 million artificial neurons that aims to reach 10 billion neurons
TECHNOLOGY INSIGHT 2.1 Augmented Intelligence
The idea of combining the performance of people and machines is not new. In this section, we discuss combining (augmenting) human abilities with powerful machine intelligence—not re- placing people, which autonomous AI does, but extending human cognitive abilities. The result is the ability of humans to solve more complex problems, as in the opening vignette to Chapter 1. Computers have provided data to help people solve problems for which no solution had been available. Padmanabhan (2018) specifies the following differences between traditional and aug- mented AI:
1. Augmented machines extend human thinking capabilities rather than replace human decision making. These machines facilitate creativity.
2. Augmentation excels in solving complex human and industry problems in specific do- mains in contrast with strong, general AI machines, which are still in development.
3. In contrast with a “black box” model of some AI and analytics, the augmented intelli- gence provides insights and recommendations, including explanations.
4. In addition, augmented technology can offer new solutions by combining existing and discovered information in contrast to assisted AI that identifies problems or symptoms and suggests predetermined known solutions.
Padmanabhan (2018) and many others believe that at the moment, augmented AI is the best option to deal with practical problems and transform organizations to be “smarter.”
In contrast with autonomous AI, which describes machines with a wide range of cognitive abilities (e.g., driverless cars), augmented intelligence has only a few cognitive abilities.
Examples of Augmented Intelligence Staff (2017) provides the following areas for which AI is useful:
• Cybercrime fighting. For example, AI can identify forthcoming attacks and suggest solutions.
• E-commerce decisions. AI marketing tools can make testing results 100 times faster, and adapt the layout and response functions of a Web site to users. Machines also make recommendations, and marketers can accept or reject them.
• High-frequency stock market trading. This process can be done either completely autonomously or in some cases with human control and calibration.
Questions for DisCussion
1. What is the basic premise of augmented intelligence?
2. List the major differences between augmented intelligence and assisted AI applications.
3. What are some benefits of augmented intelligence?
4. How does the technology relate to cognitive computing?
Chapter 2 • Artificial Intelligence 83
by 2020. Note that a human brain contains about 100 billion neurons. The system tries to imitate a biological brain and be energy efficient. IBM’s project is called TrueNorth or BlueBrain, and it learns from humans’ brains. Many believe that it will be a long and slow process for AI machines to be as creative as people (e.g., Dormehl, 2017).
u SECTION 2.2 REVIEW QUESTIONS
1. Define AI. 2. What are the major aims and goals of AI? 3. List some characteristics of AI. 4. List some AI drivers. 5. List some benefits of AI applications. 6. List some AI limitations. 7. Describe the artificial brain. 8. List the three flavors of AI and describe augmentation.
2.3 HUMAN AND COMPUTER INTELLIGENCE
AI usage is growing rapidly due to its increased capabilities. To understand AI, we need to first explore the meaning of intelligence.
What Is Intelligence?
Intelligence can be considered to be an umbrella term and is usually measured by an IQ test. However, some claim that there are several types of intelligence. For example, Dr. Howard Gardner of Harvard University proposed the following types of intelligence:
• Linguistic and verbal • Logical • Spatial • Body/movement • Musical • Interpersonal • Intrapersonal • Naturalist
Thus, intelligence is not a simple concept.
CONTENT OF INTELLIGENCE Intelligence is composed of reasoning, learning, logic, problem-solving ability, perception, and linguistic ability.
Obviously, the concept of intelligence is not simple.
CAPABILITIES OF INTELLIGENCE To understand what artificial intelligence is, it is useful to first examine those abilities that are considered signs of human intelligence:
• Learning or understanding from experience • Making sense out of ambiguous, incomplete, or even contradictory messages and
information • Responding quickly and successfully to a new situation (i.e., using the most cor-
rect responses) • Understanding and inferring in a rational way, solving problems, and directing
conduct effectively
84 Part I • Introduction to Analytics and AI
• Applying knowledge to manipulate environments and situations • Recognizing and judging the relative importance of different elements in a situation
AI attempts to provide some, hopefully all, of these capabilities, but in general, it is still not capable of matching human intelligence.
How Intelligent Is AI?
AI machines have demonstrated superiority over humans in playing complex games such as chess (beating the world champion), Jeopardy! (beating the best players), and Go (a complex Chinese game) whose top players were beaten by a computer using the well- known program, Google’s DeepMind (see Hughes, 2016). Despite these remarkable dem- onstrations (whose cost is extremely high), many AI applications still show significantly less intelligence than humans.
COMPARING HUMAN INTELLIGENCE WITH AI Several attempts have been made to com- pare human intelligence with AI. There is difficulty in doing so because it is a multidi- mensional situation. A comparison is presented in Table 2.1.
TABLE 2.1 Artificial Intelligence versus Human Intelligence
Area AI Human
Execution Very fast Can be slow
Emotions Not yet Can be positive or negative
Computation speed Very fast Slow, may have trouble
Imagination Only what is programmed for Can expand existing knowledge
Answers to questions What is in the program Can be innovative
Flexibility Rigid Large, flexible
Foundation A binary code Five senses
Consistency High Variable, can be poor
Process As modeled Cognitive
Form Numbers Signals
Memory Built in, or accessed in the cloud
Use of content and scheme memory
Brain Independent Connected to a body
Creativity Uninspired Truly creative
Durability Permanent, but can get obsolete if not updated
Perishable, but can be updated
Duplication, documentation, and dissemination
Easy Difficult
Cost Usually low and declining Maybe high and increasing
Consistency Stable Erratic at times
Reasoning process Clear, visible Difficult to trace at times
Perception By rules and data By patterns
Figure missing data Usually cannot Frequently can
Chapter 2 • Artificial Intelligence 85
For additional comparisons and who had the advantage in which area, see www. dennisgorelik.com/ai/ComputerintelligenceVsHumanIntelligence.htm.
Measuring AI
The Turing Test is a well-known attempt to measure the intelligence level of AI machines.
TURING TEST: THE CLASSICAL MEASURE OF MACHINE INTELLIGENCE Alan Turing de- signed a test known as the Turing Test to determine whether a computer exhibits intel- ligent behavior. According to this test, a computer can be considered smart only when a human interviewer asking the same questions to both an unseen human and an unseen computer cannot determine which is which (see Figure 2.3). Note that this test is limited to a question-and-answer (Q&A) mode.
To pass the Turing Test, a computer needs to be able to understand a human language (NLP), to possess human intelligence (e.g., have a knowledge base), to rea- son using its stored knowledge, and to be able to learn from its experiences (machine learning).
Note: The $100,000 Leobner prize is waiting for the person or persons who develop software that is truly intelligent (i.e., passing the Turing Test).
OTHER TESTS Over the years, there have been several other proposals of how to measure machine intelligence. For example, improvements in the Turing Test appear in several variants. Major U.S. universities (e.g., University of Illinois, Massachusetts Institute of Technology [MIT], Stanford University) are engaged in studying the IQ of AI. In addition, there are several other measuring tests. Let’s examine one test in Application Case 2.1.
In conclusion, it is difficult to measure the level of intelligence of humans as well as that of machines. Doing so depends on the circumstances and the metrics used.
Human Interviewer
Questions Unseen Human
Screen
Unseen Computer
FIGURE 2.3 A Pictorial Representation of the Turing Test
86 Part I • Introduction to Analytics and AI
If you do not know it, vacuum cleaners can be smart. Some of you may use the Roomba from iRo- bot. This vacuum cleaner can be left alone to clean floors, and it exhibits some intelligence.
However, in smart homes (Chapter 13), we expect to see even smarter vacuum cleaners. One is Roboking Turbo Plus from LG in Korea. Researchers at South Korea’s Seoul National University Robotics and Intelligent System Lab studied the Roboking and verified that its deep-learning algorithm makes it as intelligent as a six- or seven-year-old child. If we have self-driving cars, why can’t we have a self- driving vacuum cleaner, which is much simpler than a car. The cleaner needs only to move around an entire room. To do so, the machine needs to “see” its location in a room and identify obstacles in front of it. Then the cleaner’s knowledge base needs to find what is the best thing to do (given worked in the past). This is basically what many AI machines’ sensors, knowledge bases, and rules do. In addition, the AI machine needs to learn from its past experi- ence (e.g., what it should not do because it did not work in the past).
Roboking is equipped with LG’s Deep Thin QTM AI program, which enables the vacuum cleaner to figure out the nature of an encountered
obstacle. The program tells it to go around furniture, wait for a dog to move, or stop. So, how intelligent is the machine? To answer this question, the Korean researchers developed 100 metrics and tested vac- uum cleaners that were boasted as autonomous. The performance of the tested cleaners was divided into three levels based on their performance regard- ing the 100 metrics. The levels were as intelligent as a dolphin, as intelligent as an ape, and as intelli- gent as a six-to-seven-year-old child. The study con- firmed that Roboking performed tasks at the upper level of machine intelligence.
Sources: Compiled from Fuller (2017) and webwire.com/ ViewPressRel.asp?aId=211017 news dated July 18, 2017.
Questions for Case 2.1
1. How did the Korean researchers determine the performance of the vacuum cleaners?
2. If you own (or have seen) the Roomba, how intelligent do you think it is?
3. What capability can be generated by the deep learning feature? (You need to do some research.)
4. Find recent information about LG’s Roboking. Specifically, what are the newest improvements to the product?
Application Case 2.1 How Smart Can a Vacuum Cleaner Be?
Regardless of the determination of how intelligent a machine is, AI exhibits a large num- ber of benefits as described earlier.
It is important to note that the capabilities of AI are increasing with time. For ex- ample, an experiment at Stanford University (Pham, 2018) found that AI programs at Microsoft and Alibaba Co. have scored higher than hundreds of individual people at read- ing comprehension tests. (Of course, these are very expensive AI programs.) For a discus- sion of AI versus human intelligence, see Carney (2018).
u SECTION 2.3 REVIEW QUESTIONS
1. What is intelligence? 2. What are the major capabilities of human intelligence? Which are superior to that of
AI machines?
3. How intelligent is AI? 4. How can we measure AI’s intelligence? 5. What is the Turing Test and what are its limitations? 6. How can one measure the intelligence level of a vacuum cleaner?
Chapter 2 • Artificial Intelligence 87
2.4 MAJOR AI TECHNOLOGIES AND SOME DERIVATIVES
The AI field is very broad; we can find AI technologies and applications in hundreds of disciplines ranging from medicine to sports. Press (2017) lists 10 top AI technolo- gies similar to what is covered in this book. Press also provides the status of the life cycle (ecosystem phase) of the technologies. In this section, we present some major AI technologies and their derivatives as related to business. The selected list is illustrated in Figure 2.4.
Intelligent Agents
An intelligent agent (IA) is an autonomous, relatively small computer software pro- gram that observes and acts upon changes in its environment by running specific tasks autonomously. An IA directs an agent’s activities to achieve specific goals related to the changes in the surrounding environment. Intelligent agents may have the ability to learn by using and expanding the knowledge embedded in them. Intelligent agents are effective tools for overcoming the most critical burden of the Internet information overload and making computers more viable decision support tools. Interest in using intelligent agents for business and e-commerce started in the academic world in the mid-1990s. However, only since 2014, when the capabilities of IA increased remark- ably, have we started to see powerful applications in many areas of business, econom- ics, government, and services.
Initially, intelligent agents were used mainly to support routine activities such as searching for products, getting recommendations, determining products’ pricing, plan- ning marketing, improving computer security, managing auctions, facilitating payments, and improving inventory management. However, these applications were very simple, using a low level of intelligence. Their major benefits were increasing speed, reducing costs, reducing errors, and improving customer service. Today’s applications, as we will see throughout this chapter, are much more sophisticated.
Artificial Intelligence
Machine Learning Neural Network
Deep LearningIntelligent Agents Natural Language
Process
Chatbots
Knowledge Systems
Robotics
Autonomous Vehicle
Machine Computer Visions
Video Analysis
Image Generations
Cognitive Computing
Machine Translation of Languages
Speech and Voice Understanding and Generation
Degenerated Reality
FIGURE 2.4 The Major AI Technologies
88 Part I • Introduction to Analytics and AI
Example 1: Virus Detection Program
A simple example of an intelligent software agent is a virus detection program. It resides in a computer, scans all incoming data, and removes found viruses automatically while learning to detect new virus types and detection methods.
Example 2
Allstate Business Insurance is using an intelligent agent to reduce call center traffic and provide help to human insurance agents during the rate-quoting process with business customers. In these cases, rate quotes can be fairly complicated. Using this system, agents can quickly answer questions posted by corporate customers, even if the agents are not fully familiar with the related issue.
Intelligent agents are also utilized in e-mail servers, news filtering and distribution, appointment handling, and automated information gathering.
Machine Learning
At this time, AI systems do not have the same learning capabilities that humans have; rather, they have simplistic (but improving) machine learning (modeled after human learning methods. The machine-learning scientists try to teach computers to identify pat- terns and make connections by showing the machines a large volume of examples and related data. Machine learning also allows computer systems to monitor and sense their environmental activities so the machines can adjust their behavior to deal with changes in the environment. The technology can also be used to predict performance, to reconfigure programs based on changing conditions, and much more. Technically speaking, machine learning is a scientific discipline concerned with the design and development of algorithms that allow computers to learn based on data coming from sensors, databases, and other sources. This learning is then used for making predictions, recognizing patterns, and sup- porting decision makers. For an overview, see Alpaydin (2016) and Theobald (2017).
Machine-learning algorithms (see Chapter 5 for description and discussion) are used today by many companies. For an executive guide to machine learning, see Pyle and San Jose (2015).
The process of machine learning involves computer programs that learn as they face new situations. Such programs collect data and analyze them and then “train” them- selves to arrive at conclusions. For example, by showing examples of situations to a machine-learning program, the program can find elements not easily visible without it. A well-known example is that of computers detecting credit card fraud.
Application Case 2.2 illustrates how machine learning can improve companies’ busi- ness processes.
According to Taylor (2016), the “increased computing power, coupled with other improvements including better algorithms and deep neural networks for image pro- cessing, and ultra-fast in-memory databases like SAP HANA, are the reasons why ma- chine learning is one of the hottest areas of development in enterprise software today.” Machine-learning applications are also expanding due to the availability of Big Data sources, especially those provided by the IoT (Chapter 13). Machine learning is basically learning from data.
There are several methods of machine learning. They range from neural networks to case-based reasoning. The major ones are presented in Chapter 5.
DEEP LEARNING One subset, or refinement, of machine learning is called deep learning. This technology, which is discussed in Chapter 6, tries to mimic how the human brain
Chapter 2 • Artificial Intelligence 89
The following examples of using machine learn- ing are provided by Wellers, et al. (2017), who stated that “today’s leading organizations are using machine learning-based tools to automate decision processes. . . .”
1. Improving customer loyalty and retention. Companies mine customers’ activities, transac- tions, and social interactions and sentiments to predict customer loyalty and retention. Com- panies can use machine learning, for example, to predict people’s desire to change jobs and then employers can make attractive offers to keep the existing employees or to lure poten- tial employees who work elsewhere to move to new employers.
2. Hiring the right people. Given an average of 250 applicants for a good job in certain com- panies, an AI-based program can analyze applicants’ resumes and find qualified can- didates who did not apply but placed their resume online.
3. Automating finance. Incomplete financial transactions that lack some data (e.g., order numbers) require special attention. Machine- learning systems can learn how to detect and correct such situations, very quickly and at minimal cost. The AI program can take the necessary corrective action automatically.
4. Detecting fraud. Machine-learning algorithms use pattern recognition to detect fraud in real time. The program is looking for anomalies, and then it makes inferences regarding the type of detected activities to look for fraud.
Financial institutions are the major users of this program.
5. Providing predictive maintenance. Machine learning can find anomalies in the operation of equipment before it fails. Thus, corrective actions are done immediately at a fraction of a cost to repair equipment after it fails. In ad- dition, optimal preventive maintenance can be done (see Opening Vignette Chapter 1).
6. Providing retail shelf analysis. Machine learn- ing combined with machine vision can analyze displays in physical stores to find whether items are in proper locations on the shelves, whether the shelves are properly stocked, and whether the product labels (including prices) are properly shown.
7. Making other predictions. Machine learn- ing has been used for making many types of predictions ranging in areas from medicine to investments. An example is Google Flights, which predicts delays that have not been flagged yet by the airlines.
Source: Compiled from Wellers, et al. (2017) and Theobald (2017).
Questions for Case 2.2
1. Discuss the benefits of combining machine learn- ing with other AI technologies.
2. How can machine learning improve marketing?
3. Discuss the opportunities of improving human resource management.
4. Discuss the benefits for customer service.
Application Case 2.2 How Machine Learning Is Improving Work in Business
works. Deep learning uses artificial neural technology and plays a major role in dealing with complex applications that regular machine learning and other AI technologies can- not handle. Deep learning (DL) delivers systems that not only think but also keep learn- ing, enabling self-direction based on fresh data that flow in. DL can tackle previously unsolvable problems using its powerful learning algorithms.
For example, DL is a key technology in autonomous vehicles by helping to interpret road signs and road obstacles. DL is also playing critical roles in smartphones, robotics, tablets, smart homes, and smart cities (Chapter 13). For a discussion of these and other applications, see Mittal (2017). DL is mostly useful in real-time interactive applications in the areas of machine vision, scene recognition, robotics, and speech and voice process- ing. The key is continuous learning. As long as new data arrive, learning occurs.
90 Part I • Introduction to Analytics and AI
Example
Cargill Corp. offers conventional analytics, and DL-based analytics help farmers to do more profitable work. For example, farmers can produce better shrimp at lower cost. DL is used extensively in stock market analysis and predictions. For details, see Smith (2017) and Chapter 6.
Machine and Computer Vision
The definitions of machine vision vary because several different computer vision sys- tems include different hardware and software as well as other components. Generally speaking, the classical definition is that the term machine vision includes “the technol- ogy and methods used to provide imaging-based automated inspection and analysis for applications such as robot guidance, process control, autonomous vehicles, and inspec- tion.” Machine vision is an important tool for the optimization of production and robotic processes. A major part of machine vision is the industrial camera, which captures, stores, and archives visual information. This information is then presented to users or computer programs for analysis and eventually for automatic decision making or for support of human decision making. Machine vision can be confused with computer vision because sometimes the two are used as synonyms, but some users and researchers treat them as different entities. Machine vision is treated more as an engineering subfield, while com- puter vision belongs to the computer science area.
COMPUTER VISION Computer vision, according to Wikipedia, “is an interdisciplinary field that deals with how computers can be made for gaining high-level understanding from digital images or videos. From the perspective of engineering, it seeks to automate tasks that the human visual system can do.” Computer vision acquires or processes, analyzes, and interprets digital images and produces meaningful information for making decisions. Image data can take several formats, such as photos or videos, and they can come from multidimensional sources (e.g., medical scanners). Scene and item recogni- tions are important elements in computer vision. The computer vision field plays a vital role in the domains of safety, security, health, and entertainment. Computer vision is con- sidered a technology of AI, which enables robots and autonomous vehicles to see (refer to the description in Chapter 6). Both computer vision and machine vision automate many human tasks (e.g., inspection). These tasks can deal with one image or a sequence of images. The major benefit of both technologies is lowering the costs of performing tasks, especially those that are repetitive and make the human eyes tired. The two tech- nologies are also combined with image processing that facilitates complex applications, such as in visual quality control. Another view shows them as being interrelated based on image processing and sharing a variety of contributing fields.
An applied area of machine vision is scene recognition, which is performed by computer vision. Scene recognition enables recognition and interpretation of objects, scenery, and photos.
Example of Application
Significant illegal logging exists in many countries. To comply with the laws in the United States, Europe, and other countries, it is necessary to examine wood in the field. This requires expertise. According to the U.S. Department of Agriculture, “the urgent need for such field expertise, training and deploying humans to identify processed wood in the field [i.e., at ports, border crossings, weigh-stations, airports, and other points of entry for commerce] is prohibitively expensive and difficult logistically. The machine vision wood
Chapter 2 • Artificial Intelligence 91
identification project (MV) has developed a prototype machine vision system for wood identification.” Similarly, AI computer vision combined with deep learning is used to identify illegal poachers of animals (see USC, 2018).
Another example of this application is facial recognition in several security appli- cations, such as those used by the Chinese police that employ smart glasses to identify (via facial recognition) potential suspects. In 2018, the Chinese police identified a suspect who attended a pop concert. There were 60,000 people in the crowd. The person was recognized at the entrance gate where a camera took his picture; see the video at you- tube.com/watch?v=Fq1SEqNT-7c. In 2018, US Citizenship and Immigration Services identified people that used false passports in the same manner.
VIDEO ANALYTICS Applying computer vision techniques to videos enables the recog- nition of patterns (e.g., for detecting fraud) and identifying events. This is a derivative application of computer vision. Another example is one in which, by letting computers view TV shows, it is possible to train the computers to make predictions regarding human interactions and the success of advertising.
Robotic Systems
Sensory systems, such as those for scene recognition and signal processing, when com- bined with other AI technologies, define a broad category of integrated, possibly com- plex, systems, generally called robotics (Chapter 10). There are several definitions of robots, and they are changing over time. A classical definition is this: “A robot is an elec- tromechanical device that is guided by a computer program to perform manual and/or mental tasks.” The Robotics Institute of America formally defines a robot as “a program- mable multifunctional manipulator designed to move materials, parts, tools, or special- ized devices through variable programmed motions for the performance of a variety of tasks.” This definition ignores the many mental tasks done by today’s robots.
An “intelligent” robot has some kind of sensory apparatus, such as a camera, that collects information about the robot’s surroundings and its operations. The collected data are interpreted by the robot’s “brain,” allowing it to respond to the changes in the environment.
Robots can be fully autonomous (programmed to do tasks completely on their own, even repair themselves), or can be remotely controlled by a human. Some robots known as androids resemble humans, but most industrial robots are not this type. Autonomous robots are equipped with AI intelligent agents. The more advanced smart robots are not only autonomous but also can learn from their environment, building their capa- bilities. Some robots today can learn complex tasks by watching what humans do. This leads to better human–robot collaboration. The Interactive Group at MIT is experimenting with this capability by teaching robots to make complex decisions. For details, see Shah (2016). For an overview of the robot revolution, see Waxer (2016).
Example: Walmart Is Using Robots to Properly Stock Shelves
The efficiency of Walmart stores depends on appropriately stocking their shelves. Using manual labor for checking what is going on is expensive and may be inaccurate. As of late 2017, robots were supporting the company’s stocking decisions.
At Walmart, the 2-foot-tall robots use a camera/sensor to scan the shelves to look for misplaced, missing, or mispriced items. The collected information and the interpre- tation of problems are done by these self-moving robots. The results are transmitted to humans who take corrective actions. The robots carry out their tasks faster and frequently more accurately than humans. The company experimented with this in 50 stores in 2018.
92 Part I • Introduction to Analytics and AI
Preliminary results are significantly positive and are also expected to increase customer satisfaction. The robots will not cause employees to lose their jobs.
Robots are used extensively in e-commerce warehouses (e.g., tens of thousands are used by Amazon.com). They also are used in make-to-order manufacturing as well as in mass production (e.g., cars), lately of self-driven vehicles. A new generation of robots is designed to work as advisors, as described in Chapter 12. These robots are already advis- ing on topics such as investments, travel, healthcare, and legal issues. Robots can serve as front desk receptionists and even can be used as teachers and trainers.
Robots can help with online shopping by collecting shopping information, match- ing buyers and products, and conducting price and capability comparisons. These are known as shopbots (e.g., see igi-global.com/dictionary/shopbot/26826). Robots can carry goods for shoppers in open air markets. Walmart is experimenting now with robotic shopping carts (Knight, 2016). For a video (4:41 min.), see businessinsider. com/personal-robots-for-shopping-and-e-commerce-2016-9?IR=T. The Japanese company SoftBank opened a cellphone store in Tokyo entirely staffed by robots, each named Pepper. Each robot is mobile (on wheels) and can approach customers. Initially, communication with customers was done by entering information into a tablet attached to each Pepper. A major issue with robots is their trend to take human jobs. For a discus- sion of this topic, see Section 14.6.
Natural Language Processing
Natural language processing (NLP) is a technology that gives users the ability to communicate with a computer in their native language. The communication can be in written text and/or in voice (speech). This technology allows for a conversational type of interface in contrast with using a programming language that consists of computer jargon, syntax, and commands. NLP includes two subfields:
• Natural language understanding that investigates methods of enabling computers to comprehend instructions or queries provided in ordinary English or other human languages.
• Natural language generation that strives to have computers produce ordinary spoken language so that people can understand the computers more easily. For details and the history of NLP, see en.wikipedia.org/wiki/Natural_language_processing and Chapter 6.
NLP is related to voice-generated data as well as text and other communication forms.
SPEECH (VOICE) UNDERSTANDING Speech (voice) understanding is the recognition and understanding of spoken languages by a computer. Applications of this technol- ogy have become more popular. For instance, many companies have adopted this technology in their automated call centers. For an interesting application, see cs.cmu. edu/~./listen.
Related to NLP is machine translation of languages, which is done by both written text (e.g., Web content) and voice conversation.
MACHINE TRANSLATION OF LANGUAGES Machine translation uses computer programs to translate words and sentences from one language to another. For example, Babel Fish Translation, available at babelfish.com, offers more than 25 different combinations of language translations. Similarly, Google’s Translate (translate.google.com) can translate dozens of different languages. Finally, users can post their status on Facebook in several languages.
Chapter 2 • Artificial Intelligence 93
Example: Sogou’s Travel Translator
This Chinese company introduced, in 2018, an AI-powered portable travel device. Chinese people are now traveling to other countries in increasing numbers (200 million expected in 2020 versus 122 million in 2016). The objective of the device is to enable Chinese tourists to plan trips (so they can read Web sites like Trip Advisor, available in English). The AI-powered portable travel device enables tourists to read menus, street signs, and communicate with native speakers. The device, which is using NLP and image recognition, is connected to Sogou search (a search engine). In contrast with the regular Chinese-English dictionaries, this device is structured specifically for travelers and their needs.
Knowledge and Expert Systems and Recommenders
These systems, which are presented in Chapter 12, are computer programs that store knowledge, which their applications use to generate expert advice and/or perform prob- lem solving. Knowledge-based expert systems also help people to verify information and make certain types of automated routine decisions.
Recommendation systems (Chapter 12) are knowledge-based systems that make shopping and other recommendations to people. Another knowledge system is chatbots (see Chapter 12).
KNOWLEDGE SOURCES AND ACQUISITION FOR INTELLIGENT SYSTEMS For many intel- ligent systems to work, it is necessary for them to have knowledge. The process of acquiring this knowledge is referred to as knowledge acquisition. This activity can be complex because it is necessary to make sure what knowledge is needed. It must fit the desired system. In addition, the sources of the knowledge need to be identified to ensure the feasibility of acquiring the knowledge. The specific methods of acquiring the knowledge need to be identified and if expert(s) are the source of knowledge, their cooperation must be ensured. In addition, the method of knowledge representation and reasoning from the collected knowledge must be taken into account, and knowledge must be validated and be consistent.
Given this information, it is easy to see that the process of knowledge acquisition (see Figure 2.5) can be very complex. It includes extracting and structuring knowledge. It has several methods (e.g., observing, interviewing, scenario building, and discussing), so specially trained knowledge engineers may be needed for knowledge acquisition and system building. In many cases, teams of experts with different skills are created for knowledge acquisition. Knowledge can be generated from data, and then experts may be used to verify it. The acquired knowledge needs to be organized in an activity referred to as knowledge representation.
KNOWLEDGE REPRESENTATION Acquired knowledge needs to be organized and stored. There are several methods of doing this, depending on what the knowledge will be used for, how the reasoning from this knowledge will be done, how users will interact with the knowledge, and more. A simple way to represent knowledge is in the form of questions and matching answers (Q&A).
REASONING FROM KNOWLEDGE Perhaps the most important component in an intelli- gent system is its reasoning feature. This feature processes users’ requests and provides answers (e.g., solutions, recommendations) to the user. The major difference among the various types of the intelligent technologies is the type of reasoning they use.
94 Part I • Introduction to Analytics and AI
Chatbots
Robots come in several shapes and types. One type that has become popular in recent years is the chatbot. A chatbot, which will be presented in Chapter 12, is a conversional robot that is used for chatting with people. (A “bot” is short for “robot.”) Depending on the purpose of the chat, which can be done in writing or by voice, bots can be in the form of intelligent agents that retrieve information or personal assistants that provide ad- vice. In either case, chatbots are usually equipped with NLP that enables conversations in natural human languages rather than in a programmed computer language. Note that Google has rolled out six different voices to its Google’s Assistant.
Emerging AI Technologies
Several new AI technologies are emerging. Here are a few examples:
• Effective computing. These technologies detect the emotional conditions of people and suggest how to deal with discovered problems
• Biometric analysis. These technologies can verify an identity based on unique bio- logical traits that are compared to stored ones (e.g., facial recognition).
COGNITIVE COMPUTING Cognitive computing is the application of knowledge derived from cognitive science (the study of the human brain) and computer science theories in order to simulate the human thought processes (an AI objective) so that computers can exhibit and/or support decision-making and problem-solving capabilities (see Chapter 6). To do so, computers must be able to use self-learning algorithms, pattern recognition, NLP, machine vision, and other AI technologies. IBM is a major proponent of the concept by developing technologies (e.g., Watson) that support people in making complex de- cisions. Cognitive computing systems learn to reason with purpose, and interact with people naturally. For details, see Chapter 6 and Marr (2016).
Sources of Knowledge
Knowledge Acquisition Validation Verification
Natural Language Understanding
Knowledge Organization and Representation
Knowledge Repository
Knowledge Refining
Response Generation
Explanation Justification
Natural Language Generation Q&A
Problem Analysis
IdentificationUser Interface
Knowledge Users
Documented Knowledge Data Information
System Brain, Search
Inferencing, Reasoning
FIGURE 2.5 Automated Decision-Making Process
Chapter 2 • Artificial Intelligence 95
AUGMENTED REALITY Augmented reality (AR) refers to the integration of digital infor- mation with the user environment in real time (mostly vision and sound). The technology provides people real-world interactive experience with the environment. Therefore, infor- mation may change the way people work, learn, play, buy, and connect. Sophisticated AI programs may include machine vision, scene recognition, and gesture recognition. AR is available on iPhones as ARKit. (Also see Metz, 2017.)
These AR systems use data captured by sensors (e.g., vision, sound, temperature) to augment and supplement real-world environments. For example, if you take a photo of a house with your cellphone, you can immediately get the publicly available information about its configuration, ownership, and tax liabilities on your cellphone.
u SECTION 2.4 REVIEW QUESTIONS
1. Define intelligent agents and list some of their capabilities. 2. Prepare a list of applications of intelligent agents. 3. What is machine learning? How can it be used in business? 4. Define deep learning. 5. Define robotics and explain its importance for manufacturing and transportation. 6. What is NLP? What are its two major formats? 7. Describe machine translation of languages. Why it is important in business? 8. What are knowledge systems? 9. What is cognitive computing?
10. What is augmented reality?
2.5 AI SUPPORT FOR DECISION MAKING
Almost since the inception of AI, researchers have recognized the opportunity of using it for supporting the decision-making process and for completely automating decision making. Jeff Bezos, the CEO of Amazon.com, said in May 2017 that AI is in a golden age, and it is solving problems that were once in the realm of science fiction (Kharpal, 2017). Bezos also said that Amazon.com is using AI in literally hundreds of applications, and AI is really of amazing assistance. Amazon.com has been using AI, for example, for product recommendations for over 20 years. The company also uses AI for product pricing, and as Bezos said, to solve many difficult problems. And indeed, since its inception, AI has been related to problem solving and decision making. AI technologies allow people to make better decisions. The fact is that AI can:
• Solve complex problems that people have not been able to solve. (Note that solving problems frequently involves making decisions.)
• Make much faster decisions. For example, Amazon makes millions of pricing and recommendation decisions, each in a split second.
• Find relevant information, even in large data sources, very fast. • Make complex calculations rapidly. • Conduct complex comparisons and evaluations in real time.
In a nutshell, AI can drive some types of decisions many times faster and more consistently than humans can. For details, watch the video at youtube.com/ watch?v=Dr9jeRy9whQ/. The nature of decision making, especially nonroutine ones, as noted in Chapter 1, is complex. We discussed in Chapter 1 the fact that there are sev- eral types of decisions and several managerial levels of making them, and we looked at the typical process of making decisions. Making decisions, many of which are used for problem solving, requires intelligence and expertise. AI’s aim is to provide both. As a
96 Part I • Introduction to Analytics and AI
result, it is clear that using AI to facilitate decision making involves many opportunities, benefits, and variations. For example, AI can successfully support certain types of deci- sion making and fully automate others.
In this section, we discuss some general issues of AI decision support. The section also distinguishes between support of decision making and fully automating decision making.
Some Issues and Factors in Using AI in Decision Making
Several issues determine the justification of using AI and its chance of success. These include:
• The nature of the decision. For example, routine decisions are more likely to be fully automated, especially if they are simple.
• The method of support, what technology(ies) is (are) used. Initially, automated decision supports were rule-based. Practically, expert systems were created to gen- erate solutions to specific decision situations in well-defined domains. Another popular technology mentioned earlier was “recommender,” which appeared with e-commerce in the 1990s. Today, there is an increased use of machine learning and deep learning. A related technology is that of pattern recognition. Today, attention is also given to biometric types of recognition.
For example, research continues to develop an AI machine that will interview peo- ple at airports, asking one or two questions, and then determining whether they are telling the truth. Similar algorithms can be used to vet refugees and other types of immigrants.
• Cost-benefit and risk analyses. These are necessary for making large-scale decisions, but computing these values may not be simple with AI models due to difficulties in measuring costs, risks, and benefits. For example, as we cited earlier, researchers used 100 metrics to measure the intelligence level of vacuum cleaners.
• Using business rules. Many AI systems are based on business or other types of rules. The quality of automated decisions depends on the quality of these rules. Advanced AI systems can learn and improve business rules.
• AI algorithms. There is an explosion in the number of AI algorithms that are the basis for automated decisions and decision support. The quality of the decisions depends on the input of the algorithms, which may be affected by changes in the business environment.
• Speed. Decision automation is also dependent on the speed within which decisions need to be made. Some decisions cannot be automated because it takes too much time to get all the relevant input data. On the other hand, manual decisions may be too slow for certain circumstances.
AI Support of the Decision-Making Process
Much AI support can be applied today to the various steps of the decision-making pro- cess. Fully automated decisions are common in routine situations and will be discussed in the next section. Here we follow the steps in the decision-making process described in Chapter 1.
PROBLEM IDENTIFICATION AI systems are used extensively in problem identification typically in diagnosing equipment malfunction and medical problems, finding security breaches, estimating financial health, and so on. Several technologies are used. For ex- ample, sensor-collected data are used by AI algorithms. Performance levels of machines are compared to standards, and trend analysis can point to opportunities or troubles.
Chapter 2 • Artificial Intelligence 97
GENERATING OR FINDING ALTERNATIVE SOLUTIONS Several AI technologies offer alter- native solutions by matching problem characteristics with best practices or proven solu- tions stored in databases. Both expert systems and chatbots employ this approach. They can generate recommended solutions or provide several options from which to choose. AI tools such as case-based reasoning and neural computing are used for this purpose.
SELECTING A SOLUTION AI models are used to evaluate proposed solutions, for ex- ample, by predicting their future impact (predictive analysis), assessing their chance of success, or predicting a company’s reply to action taken by a competitor.
IMPLEMENTING THE SOLUTIONS AI can be used to support the implementation of com- plex solutions. For example, it can be used to demonstrate the superiority of proposals and to assess resistance to changes.
Applying AI to one or more of the decision-making processes and steps enables companies to solve complex real-world problems, as shown in Application Case 2.3.
Automated Decision Making
As the power of AI technologies increases, so does its ability to fully automate more and more complex decision-making situations.
The following examples were extracted from Forrest (2017):
Google’s Cloud Machine Learning Engine and Tensor Flow allow unique access to machine learn- ing tools without the need for PhD-educated data scientists.
The following companies use Google’s tools to solve the listed problem.
1. Axa International. This global insurance com- pany uses machine learning to predict which drivers would be more likely to cause major accidents. The analysis provides prediction ac- curacy of 78 percent. This prediction is used to determine appropriate insurance premiums.
2. Airbus Defense & Space. Detecting clouds in satellite imagery was done manually for de- cades. Using machine learning, the process has been expedited by 40 percent, and the error rate has been reduced from 11 percent to 3 percent.
3. Preventing overfishing globally. A government agency previously monitored only small sam- ple regions globally to find fishing violators. Now, using satellite AIS positioning, the agen-
cy can watch the entire ocean. Using machine learning, the agency can track all fishing ves- sels to find violators.
4. Detecting credit card fraud in Japan. SMFG, a Japanese financial services company, uses Google’s machine learning (a deep learning application) to monitor fraud related to credit card use, with an 80–90 percent accuracy of detection. The detection generates an alarm for taking actions.
5. Kewpie Food of Japan. This company detected defective potato cubes manually using a slow and expensive process. Using Google AI tools enables it to automatically monitor video feeds and alert inspectors to remove defective potatoes.
Source: Condensed and compiled from Forrest (2017).
Questions for Case 2.3
1. Why use machine learning for predictions?
2. Why use machine learning for detections?
3. What specific decisions were supported in the five cases?
Application Case 2.3 How Companies Solve Real-World Problems Using Google’s Machine-Learning Tools
98 Part I • Introduction to Analytics and AI
INTELLIGENT AND AUTOMATED DECISION SUPPORT As early as 1970, there were at- tempts to automate decision making. These attempts were typically done with the use of rule-based expert systems that provided recommended solutions to repetitive managerial problems. Examples of decisions made automatically include the following:
• Small loan approvals • Initial screening of job applicants • Simple restocking • Prices of products and services (when and how to change them) • Product recommendation (e.g., at Amazon.com)
The process of automated decision making is illustrated in Figure 2.5. The pro- cess starts with knowledge acquisition and creation of a knowledge repository. Users submit questions to the system brain, which generates a response and submits it to the users. In addition, the solutions are evaluated so that the knowledge repository and the reasoning from it can be improved. Complex situations are forwarded to humans’ attention. This process is especially used in knowledge-based systems. Note that the process in Figure 2.5 for knowledge acquisition illustrates automatic decision making as well. Companies use automated decision making for both their external operations (e.g., sales) and internal operations (e.g., resource allocation, inventory management). An example follows.
Example: Supporting Nurses’ Diagnosis Decisions
A study conducted in a Taiwanese hospital (Liao, et al., 2015) investigated the use of AI to generate nursing diagnoses and compared them to diagnoses generated by humans. Diagnoses required comprehensive knowledge, clinical experience, and instinct. The re- searchers used several AI tools, including machine learning, to conduct data mining and analysis to predict the probable success of automated nursing diagnoses based on patient characteristics. The results indicated an 87 percent agreement between the AI and human diagnosis decisions.
Such technology can be used in places that have no human nursing staff as well as by nursing staff who want to verify the accuracy of their own diagnostic predictions. The system can facilitate the training of nursing staff as well.
Automated decisions can take several forms, as illustrated in Technology Insight 2.2.
Conclusion
There is little doubt that AI can change the decision-making process for businesses; for an example, see Sincavage (2017). The nature of the change varies based on the circum- stances. But, in general, we expect AI to have a major impact for making better, faster, and more efficient decisions. Note that, in some cases, an AI watchdog is needed to regu- late the process (see Sample, 2017, for details).
u SECTION 2.5 REVIEW QUESTIONS
1. Distinguish between fully automated and supported decision making. 2. List the benefits of AI for decision support. 3. What factors influence the use of AI for decision support? 4. Relate AI to the steps in the classical decision-making process. 5. What are the necessary conditions for AI to be able to automate decision making? 6. Describe Schrage’s four models.
Chapter 2 • Artificial Intelligence 99
TECHNOLOGY INSIGHT 2.2 Schrage’s Models for Using AI to Make Decisions
Schrage (2017) of MIT’s Sloan School has proposed the following four models for AI to make autonomous business decisions:
1. The Autonomous Advisor. This is a data-driven management model that uses AI algorithms to generate best strategies and instructions on what to do and makes specific recommendations. However, only humans can approve the recommendations (e.g., proposed solutions).
Schrage provided an example in which an American retailing company replaced an entire merchandising department with an AI machine, ordering employees to obey direc- tives from it. Obviously, resistance and resentment followed. To ensure compliance, the company had to install monitoring and auditing software.
2. The Autonomous Outsource. Here, the traditional business process outsourcing model is changed to a business process algorithm. To automate this activity, it is necessary to cre- ate crystal-clear rules and instructions. It is a complex scenario since it involves resource allocation. Correct predictability and reliability are essential.
3. People–Machine Collaboration. Assuming that algorithms can generate optimal decisions in this model, humans need to collaborate with the brilliant, but constrained, fully auto- mated machines. To ensure such collaboration, it is necessary to train people to work with the AI machines (see the discussion in Chapter 14). This model is used by tech giants such as Netflix, Alibaba, and Google.
4. Complete Machine Autonomy. In this model, organizations fully automate entire processes. Management needs to completely trust AI models, a process that may take years. Schrage provides an example of a hedge fund that trades very frequently based on a machine’s rec- ommendations. The company uses machine learning to train the trading algorithms.
Implementing these four models requires appropriate management leadership and col- laboration with data scientists. For suggestions of how to do so, consult Schrage (2017), who has written several related books. Kiron (2017) discusses why managers should consider AI for decision support.
An interesting note is that some competition among companies will actually occur among data-driven autonomous algorithms and related business models.
Questions for DisCussion
1. Differentiate between the autonomous advisor and the people–machine collabora- tion models.
2. In all four models, there are some degrees of people–machine interaction. Discuss.
3. Why it is easier to use model 4 for investment decisions than, for example, market- ing strategies?
4. Why is it important for data scientists to work with top management in autono- mous AI machines?
2.6 AI APPLICATIONS IN ACCOUNTING
Throughout this book, we provide many examples of AI applications in business, ser- vices, and government. In the following five sections, we provide additional applications in the traditional areas of business: accounting; finance; human resource management; marketing, advertising, and CRM; and production-operation management.
AI in Accounting: An Overview
The CEO of SlickPie Accounting Software for small businesses, Chandi (2017), noticed trends among professional accountants: their use of AI, including bots in professional
100 Part I • Introduction to Analytics and AI
routines, increased. Chandi observed that the major drivers for this are perceived sav- ings in time and money and increased accuracy and productivity. The adoption has been rapid and it has been followed by significant improvements. An example is the execution of compliance procedures, where, for instance, Ernst & Young (EY) is using machine learning for detecting anomalous data (e.g., fraudulent invoices).
AI in Big Accounting Companies
Major users of AI are the big tax and accounting companies as illustrated in Application Case 2.4.
Accounting Applications in Small Firms
Small accounting firms also use AI. For example, Crowe Horwath of Chicago is using AI to solve complex billing problems in the healthcare industry. This helps its clients to deal with claims processing and reimbursements. The firm can now solve difficult problems that had previously resisted solutions. Many other applications are used with the support of AI, ranging from analyzing real estate contracts to risk analysis. It is only a question of time before even smaller firms will be able to utilize AI as well.
The big accounting companies use AI to replace or support human activities in tasks such as tax prepa- ration, auditing, strategy consulting, and accoun- tancy services. They mostly use NLP, robotic pro- cess automation, text mining, and machine learning. However, they use different strategies as described by Zhou (2017):
• EY attempts to show quick, positive return on investment (ROI) on a small scale. The strategy concentrates on business value. EY uses AI, for example, to review legal documents related to leasing (e.g., to meet new government regulations).
• PricewaterhouseCoopers (PwC) favors small projects that can be completely functioning in four weeks. The objective is to demonstrate the value of AI to client companies. Once demon- strated to clients, the projects are refined. PwC demonstrates 70 to 80 such projects annually.
• Deloitte Touche Tohmatsu Limited, commonly referred to as Deloitte, builds cases that guide AI-based projects for both clients and internal use. The objective is to facilitate innovation. One successful area is the use of NLP for review of large contracts that may include hundreds of thousands of legal documents. The company reduced such review time from six months to
less than a month, and it reduced the number of employees who had performed the review by more than 70 percent. Deloitte, like its com- petitors, is using AI to evaluate potential pro- curement synergies for merger and acquisition decisions. Such evaluation is a time-consuming task since it is necessary to check huge quanti- ties of data (sometime millions of data lines). As a result, Deloitte can finish such evaluation in a week compared to the four to five months required earlier. Deloitte said that with AI, it is viewing data in ways never even contemplated before (Ovaska-Few, 2017).
All big accounting companies use AI to assist in generating reports and to conduct many other routine, high-volume tasks. AI has produced high- quality work, and its accuracy has become better and better with time.
Sources: Compiled from Chandi (2017), Zhou (2017), and Ovaska- Few (2017).
Questions for Case 2.4
1. What are the characteristics of the tasks for which AI is used?
2. Why do the big accounting firms use different implementation strategies?
Application Case 2.4 How EY, Deloitte, and PwC Are Using AI
Chapter 2 • Artificial Intelligence 101
COMPREHENSIVE STUDY OF AI USE IN ACCOUNTING The ICAEW information technology (IT) faculty provides a free comprehensive study, “AI and the Future of Accountancy.” This report (ICAEW, 2017) provides an assessment of AI use in accounting today and in the future. The report sees the advantage of AI by:
• Providing cheaper and better data to support decision making and solve accounting problems
• Generating insight from data analysis • Freeing time of accountants to concentrate on problem solving and decision making
The report points to the use of the following:
• Machine learning for detecting fraud and predicting fraudulent activities • Machine-learning and knowledge-based systems for verifying of accounting tasks • Deep learning to analyze unstructured data, such as in contracts and e-mails
Job of Accountants
AI and analytics will automate many routine tasks done today by accountants (see discus- sion in Chapter 14), many of whom may lose their jobs. On the other hand, accountants will need to manage AI-based accounting systems. Finally, accountants need to drive AI innovation in order to succeed or even survive (see Warawa, 2017).
u SECTION 2.6 REVIEW QUESTIONS
1. What are the major reasons for using AI in accounting? 2. List some applications big accounting firms use. 3. Why do big accounting firms lead the use of applied AI? 4. What are some of the advantages of using AI cited by the ICAEW report? 5. How may the job of the accountant be impacted by AI?
2.7 AI APPLICATIONS IN FINANCIAL SERVICES
Financial services are much diversified, and so is AI usage in the area. One way to orga- nize the AI activities is by major segments of services. In this section, we discuss only two segments: banking and insurance.
AI Activities in Financial Services
Singh (2017) observed the following activities that may be found across various types of financial services:
• Extreme personalization (e.g., using chatbots, personal assistants, and robo invest- ment advisors) (Chapter 12)
• Shifting customer behavior both online and in brick-and-mortar branches • Facilitating trust in digital identity • Revolutionizing payments • Sharing economic activities (e.g., person-to-person loans) • Offering financial services 24/7 and globally (connecting the world)
AI in Banking: An Overview
Consultancy.uk (2017) provides an overview of how AI is transforming the banking in- dustry. It found AI applications mostly in IT, finance and accounting, marketing and sales, human resource management (HRM), customer service, and operations. A comprehensive
102 Part I • Introduction to Analytics and AI
survey on AI in banking was conducted in 2017, and a report is available for purchase (see Tiwan, 2017).
The key findings of this report are as follows:
• AI technologies in banking include all those listed in Section 2.7 and several other analytical tools (Chapters 3 to 11 of this book).
• These technologies help banks improve both their front-office and back-office operations.
• Major activities are the use of chatbots to improve customer service and com- municating with customers (see Chapter 12), and robo advising is used by some financial institutions (see Chapter 12).
• Facial recognition is used for safer online banking. • Advanced analytics helps customers with investment decisions. For examples of
this help, see Nordrum (2017), E. V. Staff (2017), and Agrawal (2018). • AI algorithms help banks identify and block fraudulent activities including money
laundering. • AI algorithms can help in assessing the creditworthiness of loan applicants. (For
a case study of an application of AI in credit screening, see ai-toolkit.blogspot. com/2017/01/case-study-artificial-intelligence-in.html.)
Illustrative AI Applications in Banking
The following are banking institutions that use AI:
• Banks are using AI machines, such as IBM Watson, to step up employee sur- veillance. This is important in preventing illegal activities such as those that occurred at Wells Fargo, the financial services and banking company. For de- tails, see information-management.com/articles/banks-using-algorithms- to-step-up-employee-surveillance.
• Banks use applications for tax preparation. H&R Block is using IBM Watson to review tax returns. The program makes sure that individuals pay only what they owe. Using interactive conversations, the machine attempts to lower peo- ple’s tax bills.
• Answering many queries in real time. For example, Rainbird Co. (rainbird.ai/) is an AI vendor that trains machines to answer customers’ queries. Millions of cus- tomers’ questions keep bank employees busy. Bots assist staff members to quickly find the appropriate answers to queries. This is especially important in banks where turnovers of employees are high. Also, there is knowledge degrading over- time, due to frequent changes in policies and regulations.
Rainbird is integrated with IBM Watson, which is using AI capabilities and cognitive reasoning to understand the nature of the queries and provide solutions. The program–employee conversations are done via chatbots, which are deployed to all U.K. branches of the banks served by Rainbird.
• At Capital One and several other banks, customers can talk with Amazon’s Alexa to pay credit card bills and check their accounts.
• TD Bank and others (see Yurcan, 2017) experiment with Alexa, which provides machine learning and augmented reality capabilities for answering queries.
• Bank Danamon uses machine learning for fraud detection and anti–money- laundering activities. It also improves the customer experience.
• At HSBC, customers can converse with the virtual banking assistant, Olivia, to find information about their accounts and even learn about security. Olivia can learn from its experiences and become more useful.
Chapter 2 • Artificial Intelligence 103
• Santander Bank employs a virtual assistant (called Nina) that can transfer money, pay bills, and do more. Nina can also authenticate its customers via an AI-based voice recognition system. Luvo of RBS is a customer service and customer relation- ship management (CRM) bot that answers customers’ queries.
• At Accenture, Collette is a virtual mortgage advisor that provides personalized advice.
• A robot named NaO can analyze facial expression and behavior of customers that enter the branches of certain banks and determine their nationality. Then the machine selects a matching language ( Japanese, Chinese, or English) to interact with the customer.
IBM Watson can provide banks many other services ranging from crime fighting to regulatory compliance as illustrated next.
Example: How Watson Helps Banks Manage Compliance and Supports Decision Making
Government regulations place a burden on banks and other financial institutions. To comply with regulations, banks must spend a considerable amount of time examining huge amounts of data generated daily.
Developed by Promontory Financial Group (an IBM subsidiary), IBM Watson (Chapter 6) developed a set of tools to deal with the compliance problem. The set of tools was trained by using the knowledge of former regulators and examining data from over 200 different sources. All in all, the program is based on over 60,000 regulatory cita- tions. It includes three sets of cognitive tools that deal with regulatory compliance. One of the tools deals with financial crimes, flagging potential suspicious transactions and pos- sible fraud. The second tool monitors compliance, and the third one deals with the large volume of data. Watson is acting as a banking financial consultant for these and other banking issues.
IBM’s tools are designed to assist financial institutions to justify important decisions. The AI algorithms examine the data inputs and outputs in managerial decision making. For example, when the program spots suspicious activity, it will notify the appropriate manager, who then will take the necessary action. For details, see Clozel (2017).
Application Case 2.5 illustrates US Bank’s use of AI to improve customer service.
Insurance Services
Advancements in AI are improving several areas in the insurance industry, mostly in issu- ing policies and handling claims.
According to Hauari (2017), the major objectives of the AI support are to improve analysis results and enhance customer experience. Incoming claims are analyzed by AI, and, depending on their nature, are sent to appropriate available adjusters. The technolo- gies used are NLP and text recognition (Chapters 6 and 7). The AI software can help in data collection and analysis and in data mining old claims.
Agents previously spent considerable time asking routine questions from people submitting insurance claims. AI machines, according to Beauchamp (2016), provide speed, accuracy, and efficiency in performing this process. Then AI can facilitate the un- derwriting process.
Similarly, claims processing is streamlined with the help of AI. It reduces process- ing time (by up to 90 percent) and improves accuracy. Capabilities of machine-learning and other AI programs can be shared in seconds in multi-office configurations, including global settings.
104 Part I • Introduction to Analytics and AI
Insurers, like other adopters of AI, will have to go through a transformation and adapt to change. Companies and individual agents can learn from early adopters. For how this is done at MetLife, see Blog (2017).
Example: Metromile Uses AI in Claim Processing
Metromile is an innovator in vehicle insurance, using the pay-per-mile model. It operates in seven U.S. states. In mid-2017, it started using AI-based programs to automate accident data, process accident claims, and pay customer claims. The automated platform, accord- ing to Santana (2017), is powered by a smart claim bot called AVA. It processes images forwarded by customers, extracting the pertinent telematic data. The AI bot simulates the accidents’ major points and makes a verification based on decision rules; authoriza- tion for payments provides for successful verification. The process takes minutes. Only complex cases are sent to investigation by human processors. Customers are delighted since they can get fast resolutions. While at the moment AVA is limited to certain types of claims, its range of suitability is increasing with the learning capabilities of machine learn- ing and the advances in AI algorithms.
Note: A 2015 start-up, Lemonade (lemonade.com) provides an AI-based platform for insurance that includes bots and machine learning. For details, see Gagliordi (2017).
u SECTION 2.7 REVIEW QUESTIONS
1. What are the new ways that banks interact with customers by using AI? 2. It is said that financial services are more personalized with AI support. Explain. 3. What back-office activities in banks are facilitated by AI? 4. How can AI contribute to security and safety?
As of July 2017, US Bank has been able to automati- cally identify military service members and veterans when they call or enter one of its branches This is not a simple task. The service members are recog- nized by Einstein, an AI-based CRM service from Salesforce Inc. (see Section 2.9).
What US Bank is trying to do is to recognize customers and understand their needs. Einstein helps the bank gain a competitive advantage in doing so. Knowledge provided is important not only for marketing and providing targeted professional financial services but also for greeting customers on their birthdays or thanking them for using the bank’s services.
The bank now has considerable informa- tion about customers available to human agents in real time. Such information helps customers
when online and when at one of the bank’s actual locations.
The AI application tells the rep all about the customer so the rep can offer appropriate services. For example, if the customer needs insurance, the AI will detect this need and the rep will offer a good alternative. It also offers information to an online customer: “Hello, Mary; I see you are checking your mortgage payments. I have good news for you. . . .”
Source: Compiled from Crosman (2017) and Carey (2017).
Questions for Case 2.5
1. What are Einstein’s advantages to US Bank?
2. What are its advantages to customers?
3. What are the benefits of voice communication?
Application Case 2.5 US Bank Customer Recognition and Services
Chapter 2 • Artificial Intelligence 105
5. What is the role of chatbots and virtual assistants in financial services? 6. How can IBM Watson help banking services? 7. Relate Salesforce Einstein to CRM in financial services. 8. How can AI help in processing insurance claims?
2.8 AI IN HUMAN RESOURCE MANAGEMENT (HRM)
As in other business functional areas, the use of AI technologies is spreading rapidly in HRM. And as in other areas, the AI services reduce cost and increase productivity, consis- tency, and speed of execution.
AI in HRM: An Overview
Savar (2017) points to the following reasons for AI to transform HRM, especially in re- cruiting: (1) reducing human bias, (2) increasing efficiency, productivity, and insight in evaluating candidates, and (3) improving relationships with current employees.
Wislow (2017) sees the use of AI as a continuation of automation that supports HRM and keeps changing it. Wislow suggests that such automation changes how HRM employ- ees work and are engaged. This change also strengthens teamwork. Wislow divided the impact of AI into the following areas:
RECRUITMENT (TALENT ACQUISITION) One of the cumbersome tasks in HRM, especially in large organizations, is recruiting new employees. The fact is that many job positions are unfilled due to difficulties in finding the right employees. At the same time, many qualified people cannot find the right jobs.
AI improves the recruiting process as illustrated in Application Case 2.6. The use of chatbots to facilitate recruitment is also described by Meister (2017). Companies that help recruiters and job seekers, especially LinkedIn, are using AI
algorithms to suggest matches to both recruiters and job seekers. Haines (2017) describes the process, noting that a key benefit of this process is the removal of unconscious biases and prejudices of humans.
AI FACILITIES TRAINING The rapid technological developments make it necessary to train and retrain employees. AI methods can be used to facilitate learning. For example, chatbots can be used as a source of knowledge to answer learners’ queries. Online courses are popular with employees. AI can be used to test progress, for example. In addition, AI can be used to personalize online teaching for individuals and to design group lectures.
AI SUPPORTS PERFORMANCE ANALYSIS (EVALUATION) AI tools enable HR management to conduct performance analysis by breaking work into many small components and by measuring the performance of each employee and team on each component. The perfor- mance is compared to objectives, which are provided to employees and teams. AI also can track changes and progress by combining AI with analytical tools.
AI USE IN RETENTION AND ATTRITION DETECTION In order to keep employees from leaving, it is necessary for businesses to analyze and predict how to make workers happy. Machine learning can be used to detect reasons why employees leave companies by identifying influencing patterns.
AI in Onboarding
Once new employees are hired, the HR department needs help introducing them to the organizational culture and operating processes. Some new employees require much
106 Part I • Introduction to Analytics and AI
attention. AI helps HRM prepare customized onboarding paths that are best for the new- comers. Results showed that those employees supported by AI-based plans tend to stay longer in organizations (Wislow, 2017).
USING CHATBOTS FOR SUPPORTING HRM The use of chatbots in HRM is increasing rap- idly. Their ability to provide current information to employees anytime is a major reason. Dickson (2017) refers to the following chatbots: Mya, a recruiting assistant, and Job Bot, which supports the recruitment of hourly workers. This bot is also used as a plug-in to Craigslist. Another chatbot mentioned earlier is Olivia; see olivia.paradox.ai/.
Introducing AI to HRM Operations
Introducing AI to HRM operations is similar to introducing AI to other functional areas. Meister (2017) suggests the following activities:
1. Experiment with a variety of chatbots 2. Develop a team approach involving other functional areas 3. Properly plan a technology roadmap for both the short and long term, including
shared vision with other functional areas 4. Identify new job roles and modifications in existing job roles in the transformed
environment 5. Train and educate the HRM team to understand AI and gain expertise in it
Alexander Mann is a Chicago-based company that offers AI solutions to support the employee recruit- ment process. The major objective is to help com- panies solve HRM problems and challenges. The AI is used to:
1. Help companies evaluate applicants and their resumes by using machine learning. The result is the decision regarding which applicants to invite for an interview.
2. Help companies evaluate resumes that are posted on the Web. The AI software can use key words for the search related to the back- ground of employees (e.g., training, years of experience).
3. Evaluate the resumes of the best employees who currently work in a company and create, accordingly, desired profiles to be used when vacancies occur. These profiles are then com- pared to those of applying candidates, and the top ones are ranked by their fit to each job opening. In addition to the ranking, the AI program shows the fit with each desired criterion. At this stage, the human recruiter
can make a final selection decision. This way, the selection process is faster and much more accurate.
The accuracy of the process solves the candidate volume problem, ensuring that qualified people are not missed and poorly fit applicants are not selected.
Alexander Mann is also helping its clients to install chatbots that can provide candidates’ answers to questions related to the jobs and the working con- ditions at the employing company. (For the recruiting chatbot, see Dickson, 2017).
Sources: Compiled from Huang (2017), Dickson (2017), and alexandermannsolutions.com, accessed June 2018.
Questions for Case 2.6
1. What types of decisions are supported?
2. Comment on the human–machine collaboration.
3. What are the benefits to recruiters? To applicants?
4. Which tasks in the recruiting process are fully automated?
5. What are the benefits of such automation?
Application Case 2.6 How Alexander Mann Solutions (AMS) Is Using AI to Support the Recruiting Process
Chapter 2 • Artificial Intelligence 107
For additional information and discussion, see Essex (2017).
u SECTION 2.8 REVIEW QUESTIONS
1. List the activities in recruiting and explain the support provided by AI to each. 2. What are the benefits rewarded to recruiters by AI? 3. What are the benefits to job seekers? 4. How does AI facilitate training? 5. How is performance evaluation of employees improved by AI? 6. How can companies increase retention and reduce attrition with AI? 7. Describe the role of chatbots in supporting HRM.
2.9 AI IN MARKETING, ADVERTISING, AND CRM
Compared to other business areas, there are probably more applications of AI in market- ing and advertising. For example, AI-based product recommendations have been in use by Amazon.com and other e-commerce companies for more than 20 years. Due to the large number of applications, we provide only a few examples here.
Overview of Major Applications
Davis (2016) provides 15 examples of AI in marketing as listed with explanations by the authors of this book and from Martin (2017). Also see Pennington (2018).
1. Product and personal recommendations. Starting with Amazon.com’s book recom- mendations for Netflix’s movies, AI-based technologies are used extensively for per- sonalized recommendations (e.g., see Martin, 2017).
2. Smart search engines. Google is using RankBrain’s AI system to interpret users’ que- ries. Using NLP helps in understanding the products or services for which online users are searching. This includes the use of voice communication.
3. Fraud and data breaches detection. Application for this has covered credit/debit card use for many years, protecting Visa and other card issuers. Similar technologies protect retailers (such as Target and Neiman Marcus) from hackers’ attacks.
4. Social semantics. Using AI-based technologies, such as sentiment analysis and image and voice recognitions, retailers can learn about customers’ needs and provide tar- geted advertisements and product recommendations directly (e.g., via e-mail) and through social media.
5. Web site design. Using AI methods, marketers are able to design attractive Web sites. 6. Producer pricing. AI algorithms help retailers price products and services in a dynamic
fashion based on the competition, customers’ requirements, and more. For example, AI provides predictive analysis to forecast the impact of different price levels.
7. Predictive customer service. Similar to predicting the impact of pricing, AI can help in predicting the impact of different customer service options.
8. Ad targeting. Similar to product recommendations, which are based on user profiles, marketers can tailor ads to individual customers. The AI machines attempt to match different ads with individuals.
9. Speech recognition. As the trend to use voice in human–machine interaction is in- creasing, the use of bots by marketers to provide product information and prices accelerates. Customers prefer to talk to bots rather than to key in dialogue.
10. Language translation. AI enables conversations between people who speak differ- ent languages. Also, customers can buy from Web sites written in languages they do not speak by using GoogleTranslate, for example.
108 Part I • Introduction to Analytics and AI
11. Customer segmentation. Marketers are segmenting customers into groups and then tailoring ads to each group. While less effective than targeting individuals, this is more effective than mass advertising. AI can use data and text mining to help mar- keters identify the characteristics of specific segments (e.g., by mining historical files) as well as help tailor the best ads for each segment.
12. Sales forecasting. Marketers’ strategy and planning are based on sales forecasting. Such forecasting may be very difficult for certain products. Uncertainties may exist in many situations such as in customer need assessment. Predictive analytics and other AI tools can provide better forecasting than traditional statistical tools.
13. Image recognition. This can be useful in market research (e.g., for identifying con- sumer preferences of one company’s products versus those of its competition). It can also be used for detecting defects in producing and/or packaging products.
14. Content generation. Marketers continuously create ads and product information. AI can expedite this task and make sure that it is consistent and complies with regu- lations. Also, AI can help in generating targeted content to both individuals and segments of consumers.
15. Using bots, assistants, and robo advisors. In Chapter 12, we describe how bots, per- sonal assistants, and robo advisors help consumers of products and services. Also, these AI machines excel in facilitating customer experience and strengthen customer relationship management. Some experts call bots and virtual personal assistants the “face of marketing.”
Another list is provided at en.wikipedia.org/wiki/Marketing_and_artificial_ intelligence.
AI Marketing Assistants in Action
There are many ways that AI can be used in marketing. One way is illustrated in Application Case 2.7 about Kraft Foods.
Customer Experiences and CRM
As described earlier, a major impact of AI technologies is changing customer experiences. A notable example is the use of conversational bots. Bots (e.g., Alexa) can provide infor- mation about products and companies and can provide advice and guidance (e.g., robo advisors for investment; see Chapter 12). Gangwani (2016) lists the following ways to improve customers’ experiences:
1. Use NLP for generating user documentation. This capability also improves the customer–machine dialogue.
2. Use visual categorization to organize images (for example, see IBM’s Visual Recognition and Clarifai)
3. Provide personalized and segmented services by analyzing customer data. This in- cludes improving shopping experience and CRM.
A well-known example of AI in CRM is Salesforce’s Einstein.
Example: Salesforce’s AI Einstein
Salesforce Einstein is an AI set of technologies (e.g., Einstein Vision for image recogni- tion) that is used for enhancing customer interactions and supporting sales. For example, the system delivers dynamic sales dashboards to sales reps. It also tracks performance and manages teamwork by using sales analytics. The AI product also can provide predictions
Chapter 2 • Artificial Intelligence 109
The number of mobile users is growing rapidly as is the number of mobile shoppers. Kraft Foods took notice of that. The company is adapting its adver- tising and sales to this trend. Mobile customers are looking for brands and interacting with Kraft brands. Kraft Foods wanted to make it easy for customers to interact with the company whenever and wherever they want. To achieve this interaction goal, Kraft Foods created a “Food Assistant,” also known as Kraft Food Assistant.
The Kraft Food Assistant
Kraft’s Food Assistant is an app for smartphones that allows customers to access more than 700 recipes. Thus, the consumer can browse easily for ideas. Customers enter a virtual store and open the “recipe of the day.” The app tells the user all the ingredients needed for that recipe or for any desired recipe. The Food Assistant also posts all the relevant coupons available for the ingredients on users’ smartphone. Users need only to take the smartphone to a supermarket, scan the coupons, and save on the ingredients. The recipe of the day is also demonstrated on a video. Unique to this app is the inclusion of an AI algorithm that learns from users’ orders and can infer, for exam- ple, the users’ family size. The more the AI learns about users, the more suggestions it makes. For example, it tells users what to do with their left- over ingredients. In addition, the more the Food Assistant learns about users, the more useful sug- gestions for recipes and cooking it can offer. It is like the Netflix recommender. The more Kraft products that users buy (the ingredients), the more advice they get. The Food Assistant also directs users to the nearest store that has the recipes’ ingredients. Users can get assistance on how to prepare food in 20 minutes and on many cooking-related topics.
The AI is tracking consumers’ behavior. Information is stored on each user’s loyalty card. The system makes inferences about what consum- ers like and targets related promotions to them. This process is called behavioral pattern recognition, and is based on AI techniques such as “collaborative fil- tering.” (See Chapter 12.)
AI assistants also can tweak messages to users, and they know if users are interested in their top- ics. The assistant also knows whether customers are responding positively and whether they are or are not motivated to try a new product or purchase more of what they previously purchased. The Kraft AI Food Assistant actually is trying to influence and sometimes to modify consumer behavior. Like other vendors, Kraft is using the information collected by the AI assistant to forge and execute mobile and regular commerce strategies.
Using the information collected, Kraft and sim- ilar vendors can expand their mobile marketing pro- grams both online and in physical stores.
Note: Users can interact with the system with voice powered by Nuance Communication. The system is based on natural lan- guage processing.
Sources: Compiled from Celentano (2016), press releases at nuance.com, and kraftrecipes.com/media/iphoneassistant. aspx/, accessed March 2018.
Questions for Case 2.7
1. Identify all AI technologies used in the Food Assistant.
2. List the benefits to the customers.
3. List the benefits to Kraft Foods.
4. How is advertising done?
5. What role is “behavioral pattern recognition” playing?
6. Compare Kraft’s Food Assistant to Amazon.com and Netflix recommendation systems.
Application Case 2.7 Kraft Foods Uses AI for Marketing and CRM
and recommendations. It supports Salesforce Customer Successful Platform and other Salesforce products.
Einstein’s automatically prioritized sales leads make sales reps more productive when dealing with sales leads and potential opportunities. The sales reps also get insights about customers’ sentiments, competitors’ involvement, and other information.
110 Part I • Introduction to Analytics and AI
For information and a demo, see salesforce.com/products/einstein/overview/. For features and description of the product, see zdnet.com/article/salesforces- einstein- ai-platform-what-you-need-to-know/. For additional features, see salesforce.com/ products/einstein/features/.
Other Uses of AI in Marketing
The following show the diversity of AI technologies used in marketing:
• It is used to mimic the expertise of in-store salespeople. In many physical stores, humans are not readily available to help customers who do not want to wait very long. Thus, shopping is made easier when bots provide guidance. A Japanese store already provides all services in a physical store by speaking robots.
• It provides lead generation. As seen in the case of Einstein, AI can help generate sales leads by analyzing customers’ data. The program can generate predictions. Insights can be generated by intelligent analytics.
• It can increase customer loyalty using personalization. For example, some AI tech- niques can recognize regular customers (e.g., in banks). IBM Watson can learn about people from their tweets.
• Salesforce.com provides a free e-book, “Everything You Need to Know about AI for CRM” (salesforce.com/form/pdf/ai-for-crm.jsp).
• It can improve the sales pipeline. Narayan (2018) provides a process of how compa- nies can use AI and robots to do this. Specifically, robots convert unknown visitors into customers. Robots use three stages: (1) prepare a list of target customers in the database, (2) send information, ads, videos, and so on to prospects on the list cre- ated earlier, and (3) provide the company sales department with a list of leads that successfully convert potential customers to buyers.
u SECTION 2.9 REVIEW QUESTIONS
1. List five of the 15 applications of Davis (2016). Comment on each. 2. Which of the 15 applications relate to sales? 3. Which of the 15 applications relate to advertising? 4. Which of the 15 applications relate to customer service and CRM? 5. For what are the prediction capabilities of AI used? 6. What is the Salesforce’s Einstein? 7. How can AI be used to improve CRM?
2.10 AI APPLICATIONS IN PRODUCTION-OPERATION MANAGEMENT (POM)
The field of POM is much diversified, and its use of AI is evident today in many areas. To describe all of them, we would need more than a whole book. In the remaining chapters, we provide dozens of examples about AI applications in POM. Here, we pro- vide only a brief discussion regarding two related application areas: manufacturing and logistics.
AI in Manufacturing
To handle ever-increasing labor costs, changes in customers’ requirements, increased global competition, and government regulations (Chapter 1), manufacturing compa- nies are using elevated levels of automation and digitization. According to Bollard et al.
Chapter 2 • Artificial Intelligence 111
(2017), companies need to be more agile, and react quicker and more effectively. They also need to be more efficient and improve customers’ (organizations’ and individuals’) experiences. Companies are pressured to cut costs and increase quality and transparency. To achieve these goals, they need to automate processes and make use of AI and other cutting-edge technologies.
Implementation Model
Bollard, et al. (2017) proposed a five-component model for manufacturing companies to use intelligent technologies. This model includes:
• Streamlining processes, including minimizing waste, redesigning processes, and using business process management (BPM)
• Outsourcing certain business processes, including going offshore • Using intelligence in decision making by deploying AI and analytics • Replacing human tasks with intelligent automation • Digitizing customers’ experiences
Companies have used this model for a long time. Actually, robotics have been used since around 1960 (e.g., Unimate in General Motors). However, the robots were “dumb,” each usually doing one simple task. Today, companies use intelligent robots for complex tasks, enabling make-to-order products and mass customization. In other words, many mental and cognitive tasks are being automated. These developments, involving AI and sensors, allow supporting or even automating production decisions in real time.
Example
When a sensor detects a defective product or a malfunction, the data are processed by an AI algorithm. An action then takes place instantly and automatically. For example, a defective item can be removed or replaced. AI can even make predictions about equip- ment failures before they occur (see the opening vignette in Chapter 1). This real-time action saves a huge amount of money for manufacturers. (This process may involve the IoT; see Chapter 13.)
Intelligent Factories
Ultimately, companies will use smart or intelligent factories (see Chapter 13). These facto- ries use complex software and sensors. An example of a lead supplier is General Electric, which provides software such as OEE Performance Analyzer and Production Execution Supervisor. The software is maintained in the “cloud” and it is provided as a “software-as- a-service.” GE partners with Cisco and FTC to provide security, connectivity, and special analytics.
In addition to GE, well-known companies such as Siemens and Hitachi provide comprehensive solutions. For an example, see Hitachi AI Technology’s Report (social- innovation.hitachi/ph/solutions/ai/pdf/ai_en_170310.pdf).
Many small vendors are specializing in different aspects of AI for manufacturing. For example, BellHawk Systems Corporation, which provides services to small companies, specializes in real-time operations tracking (see Green, 2016).
Early successes were recorded by large companies such as Procter & Gamble and Toyota.
However, as time passes, medium-size and small companies can also afford AI ser- vices. For additional information, see bellhawk.com.
112 Part I • Introduction to Analytics and AI
Logistics and Transportation
AI and intelligent robots are used extensively in corporate logistics and internal and exter- nal transportation, as well as in supply chain management. For example, Amazon.com is using over 50,000 robots to move items in its distribution centers (other e-commerce com- panies are doing the same). Soon, we will see driverless trucks and other autonomous vehicles all over the world (see Chapter 13).
Example: DHL Supply Chain
DHL is a global delivery company (competing with FedEx and UPS). It has a supply chain division that works with many business partners. AI and IoT are changing the manner by which the company, its partners, and even its competitors operate. DHL is developing innovative logistics and transportation business models, mostly with AI, IoT, and machine learning. These models also help DHL’s customers gain a competitive advantage (and this is why the company cannot provide details in its reports).
Several of the IoT projects are linked to machine learning, specifically in the areas of sensors, communication, device management, security, and analysis. Machine learning in such cases assists in tailoring solutions to specific requirements.
Overall, DHL concentrates on the areas of supply chains (e.g., identifies inventories and controls them along the supply chain) and warehouse management. Machine learn- ing and other AI algorithms enable more accurate procurement, production planning, and work coordination. Tagging and tracking items using Radio Frequency Identification (RFID) and Quick Response (QR) code allow for item tracking along the supply chain. Finally, AI facilitates predictive analytics, scheduling, and resource planning. For details, see Coward (2017).
u SECTION 2.10 REVIEW QUESTIONS
1. Describe the role of robots in manufacturing. 2. Why use AI in manufacturing? 3. Describe the Bollard et al. implementation model. 4. What is an intelligent factory? 5. How are a company’s internal and external logistics supported by AI technologies?
Chapter Highlights
• The aim of artificial intelligence is to make ma- chines perform tasks intelligently, possibly like people do.
• A major reason for using AI is to cause work and decision making to be easier to perform. AI can be more capable (enable new applications and business models), more intuitive, and less threat- ening than other decision support applications.
• A major reason to use AI is to reduce cost and/or increase productivity.
• AI systems can work autonomously, saving time and money, and perform work consistently. They can also work in rural and remote areas where human expertise is rare or not available.
• AI can be used to improve all decision-making steps.
• Intelligent virtual systems can act as assistants to humans.
• AI systems are computer systems that exhibit low (but increasing) levels of intelligence.
• AI has several definitions and derivatives, and its importance is growing rapidly. The U.S. govern- ment postulated that AI will be a “critical driver of the U.S. economy” (Gaudin 2016).
• The major technologies of AI are intelligent agents, machine learning, robotic systems, NLP and speech recognition, computer vision, and knowledge systems.
Chapter 2 • Artificial Intelligence 113
• Expert systems, recommendation systems, chat- bots, and robo advisors are all based on knowl- edge transferred to machines.
• The major limitations of AI are the lack of human touch and feel, the fear that it will take jobs from people, and the possibility that it could be destructive.
• AI is not a match to humans in many cognitive tasks, but it can perform many manual tasks quicker and at a lower cost.
• There are several types of intelligence, so it is dif- ficult to measure AI’s capacity.
• In general, human intelligence is superior to that of machines. However, machines can beat peo- ple in complex games.
• Machine learning is currently the most useful AI technology. It attempts to learn from its experi- ence to improve operations.
• Deep learning enables AI technologies to learn from each other, creating synergy in learning.
• Intelligent agents excel in performing simple tasks considerably faster and more consistently than humans (e.g., detecting viruses in computers).
• The major power of machine learning is a result of the machine’s ability to learn from data and its manipulation.
• Deep learning can solve many difficult problems. • Computer vision can provide understandings
from images, including from videos. • Robots are electromechanical computerized sys-
tems that can perform physical and mental tasks. When provided with sensory devices, they can become intelligent.
• Computers can understand human languages and can generate text or voice in human languages.
• Cognitive computing simulates the human thought process for solving problems and making decisions.
• Computers can be fully automated in simple manual and mental tasks using AI.
• Several types of decision making are fully auto- mated using AI; other types can be supported.
• AI is used extensively in all functional business departments, reducing cost and increasing pro- ductivity, accuracy, and consistency. There is a tendency to increase the use of chatbots. They all support decision making well.
• AI is used extensively in accounting, automating simple transactions, helping deal with Big Data, finding fraudulent transactions, increasing secu- rity, and assisting in auditing and compliance.
• AI is used extensively in financial services to im- prove customer service, provide investment ad- vice, increase security, and facilitate payments among other tasks. Notable applications are in banking and insurance.
• HRM is using AI to facilitate recruitment, en- hance training, help onboarding, and streamline operations.
• There is considerable use of AI in marketing, sales, and advertising. AI is used to support prod- uct recommendation, help in search of products and services, facilitate Web site design, support pricing decisions, provide language translation in globe trade, assist in forecasting and predictions, and use chatbots for many marketing and cus- tomer service activities.
• AI has been used in manufacturing for decades. Now it is applied to support planning, supply chain coordination, logistics and transportation, and operation of intelligent factories.
Key Terms
artificial brain artificial intelligence (AI) augmented intelligence chatbots computer vision deep learning
intelligent agent machine learning machine vision natural language processing
(NLP) robot
scene recognition shopbot speech (voice) understanding Turing Test
Questions for Discussion
1. Discuss the difficulties in measuring the intelligence of machines.
2. Discuss the process that generates the power of AI.
3. Discuss the differences between machine learning and deep learning.
114 Part I • Introduction to Analytics and AI
4. Describe the difference between machine vision and computer vision.
5. How can a vacuum cleaner be as intelligent as a six- year-old child?
6. Why are NLP and machine vision so prevalent in industry?
7. Why are chatbots becoming very popular? 8. Discuss the advantages and disadvantages of the Turing
Test. 9. Why is augmented reality related to AI?
10. Discuss the support that AI can provide to decision makers.
11. Discuss the benefits of automatic and autonomous deci- sion making.
12. Why is general (strong) AI considered to be “the most significant technology ever created by humans”?
13. Why is the cost of labor increasing, whereas the cost of AI is declining?
14. If an artificial brain someday contains as many neurons as the human brain, will it be as smart as a human brain? (Students need to do extra research.)
15. Distinguish between dumb robots and intelligent ones. 16. Discuss why applications of natural language processing
and computer vision are popular and have many uses.
Exercises
1. Go to itunes.apple.com/us/app/public-transit-app- moovit/id498477945?mt=8. Compare Moovit opera- tions to the operation of INRIX.
2. Go to sitezeus.com and view the 2:07 min. video. Explain how the technology works as a decision helper.
3. Go to Investopedia and learn about investors’ tolerance. Then find out how AI can be used to contain this risk, and write a report.
4. In 2017, McKinsey & Company created a five-part video titled “Ask the AI Experts: What Advice Would You Give to Executives About AI?” View the video and summarize the advice given to the major issues discussed. (Note: This is a class project.)
5. Watch the McKinsey & Company video (3:06 min.) on today’s drivers of AI at youtube.com/ watch?v=yv0IG1D-OdU and identify the major AI drivers. Write a report.
6. Go to the Web site of the Association for the Advancement of Artificial Intelligence aaai.org/home. html and describe its content. Compare it to that of ai. sri.com and csail.mit.edu/.
7. Go to crosschx.com and find information about Olive. Explain how it works, what its limitations and advan- tages are, and which types of decisions it automates and which it only supports.
8. Go to waze.com and moovitapp.com and find their capabilities. Summarize the help they can provide users.
9. Go to sentient.ai. Find its products that facilitate e- commerce. Write a report.
10. Go to artificialbrain.org and report the latest progress there.
11. Find recent information on research that is aimed to measure artificial intelligence. Write a report.
12. Go to salesforce.com and find recent developments on AI Einstein. Why it is so popular?
13. Find the latest information on IBM Watson’s advising activities. Write a report.
14. Find information on the use of AI in iPhones. Explore the role of Edge AI. Write a report.
15. Explore the AI-related products and services of Nuance Inc. (nuance.com). Explore the Dragon voice recogni- tion product.
16. Go to the Netradyne report at cs_netradyne.com/ and read about the use of its product for road safety. Write a report.
17. Go to salesforce.com and investigate the capabilities of Gecko HRM. Relate it to Salesforce Einstein. Provide examples of two applications.
18. Enter McKinsey & Company and find in its Fifty Five “The Value AI Can Bring to Your Business” (mckinsey. com/featured-insights/artificial-intelligence/five- fifty-real-world-ai). Then look for “Real-World AI.” Find the banking section and dive more deeply into its content.
19. Find material on the impact of AI on advertising. Write a report.
20. Go to strategicsourceror.com/2018/03/giant-scale- supply-chains-can-make.html. Summarize the use of AI.
References
Agrawal, V. “How Successful Investors Are Using AI to Stay Ahead of the Competition.” ValueWalk, January 28, 2018.
Alpaydin, E. Machine Learning: The New AI (The MIT Press Essential Knowledge Series). Boston, MA: MIT Press, 2016.
Beauchamp, P. “Artificial Intelligence and the Insurance In- dustry: What You Need to Know.” The Huffington Post, October 27, 2016.
Blog. “Welcome to the Future: How AI Is Transforming Insur- ance.” Blog.metlife.com, October 1, 2017.
Bollard, A., et al. “The next-generation operating model for the digital world.” McKinsey & Company, March 2017.
BrandStudio. “Future-Proof: How Today’s Artificial Intelli- gence Solutions Are Taking Government Services to the Next Frontier.” Washington Post, August 22, 2017.
Chapter 2 • Artificial Intelligence 115
Carey, S. “US Bank Doubles Its Conversion Rate for Wealth Customers Using Salesforce Einstein.” Computerworld UK, November 10, 2017.
Carney, P. “Pat Carney: Artificial Intelligence versus Human Intelligence.” Vancouver Sun, April 7, 2018.
Celentano, D. “Kraft Foods iPhone Assistant Appeals to Time Starved Consumers.” The Balance, September 18, 2016.
Chandi, N. “How AI is Reshaping the Accounting Industry.” Forbes.com, July 20, 2017.
Clozel, L. “IBM Unveils New Watson tools to Help Banks Manage Compliance, AML.” American Banker, June 14, 2017.
Consultancy.uk. “How Artificial Intelligence Is Transforming the Banking Industry.” September 28, 2017. consultancy .uk/news/14017/how-ar tif icial-intelligence-is- transforming-the-banking-industry/ (accessed June 2018).
Coward, J. “Artificial Intelligence Is Unshackling DHL’s Supply Chain Potential.” IoT Institute, April 18, 2017. ioti.com/ industrial-iot/artificial-intelligence-unshackling- dhls-supply-chain-potential (accessed June 2018).
Crosman, P. “U.S. Bank Bets AI Can Finally Deliver 360- Degree View.” American Banker, July 20, 2017.
Davis, B. “15 Examples of Artificial Intelligence in Marketing.” Econsultancy, April 19, 2016.
Dickson, B. “How Artificial Intelligence Optimizes Recruit- ment.” The Next Web, June 3, 2017.
Dodge, J. “Artificial Intelligence in the Enterprise: It’s On.” Computerworld, February 10, 2016.
Dormehl, L. Thinking Machines: The Quest for Artificial Intelligence—and Where It’s Taking Us Next. New York, NY: Tarcher-Perigee, 2017.
Essex, D. “AI in HR: Artificial Intelligence to Bring Out the Best in People.” TechTargetEssential Guide, April 2017.
E. V. Staff. “Artificial Intelligence Used to Predict Short-Term Share Price Movements.” The Economic Voice, June 22, 2017.
Finlay, S. Artificial Intelligence and Machine Learning for Business: A No-Nonsense Guide to Data Driven Technolo- gies. 2nd ed. Seattle, WA: Relativistic, 2017.
Forrest, C. “7 Companies That Used Machine Learning to Solve Real Business Problems.” Tech Republic, March 8, 2017.
Fuller, D. “LG Claims Its Roboking Vacuum Is As Smart As a Child.” Androidheadlines.com, July 18, 2017.
Gagliordi, N. “Softbank Leads $120M Investment in AI-Based Insurance Startup Lemonade.” ZDNET, December 19, 2017.
Gangwani, T. “3 Ways to Improve Customer Experience Using A.I.” CIO Contributor Network, October 12, 2016.
Gaudin, S. “White House: A.I. Will Be Critical Driver of U.S. Economy.” Computerworld, October 12, 2016.
Gitlin, J. M. “Watch Out, Waze: INRIX’s New Traffic App Is Coming for You.” Ars Technica, March 30, 2016. arstechnica. com/cars/2016/watch-out-waze-inrixs-new-traffic- app-is-coming-for-you/ (accessed June 2018).
Greengard, S. “Delving into Gartner’s 2016 Hype Cycle.” Base- line, September 7, 2016.
Greig, J. “Gartner: AI Business Value Up 70% in 2018, and These Industries Will Benefit the Most.” Tech Republic, April 25, 2018.
Haines, D. “Is Artificial Intelligence Making It Easier and Quicker to Get a New Job?” Huffington Post UK, Decem- ber 4, 2017.
Hauari, G. “InsurersLeverage AI to Unlock Legacy Claims Data.” Information Management, July 3, 2017.
Huang, G. “Why AI Doesn’t Mean Taking the ‘Human’ Out of Human Resources.” Forbes.com, September 27, 2017.
Hughes, T. “Google DeepMind’s Program Beat Human at Go.” USA Today, January 27, 2016.
ICAEW. “Artificial Intelligence and the Future of Accountan- cy.” artificial-intelligence-report.ashx/, 2017.
Kaplan, J. Artificial Intelligence: What Everyone Needs to Know. London, UK: Oxford University Press, 2016.
Kharpal, A. “A.I. Is in a ‘Golden Age’ and Solving Problems That Were Once in the Realm of Sci-Fi, Jeff Bezos Says.” CNBC News, May 8, 2017.
Kiron, D. “What Managers Need to Know About Artificial Intel- ligence?” MITSloan Management Review, January 25, 2017.
Knight, W. “Walmart’s Robotic Shopping Carts Are the Latest Sign That Automation Is Eating Commerce.” Technology Review, June 15, 2016.
Kolbjørnsrud, V., R. Amico, and R. J. Thomas. “How Artificial Intelligence Will Redefine Management.” Harvard Busi- ness Review, November 2, 2016.
Korosec, K. “Inrix Updates Traffic App to Learn Your Daily Habits.” Fortune Tech, March 30, 2016.
Liao, P-H., et al. “Applying Artificial Intelligence Technology to Support Decision-Making in Nursing: A Case Study in Taiwan.” Health Informatics Journal, June 2015.
Marr, B., “The Key Definitions of Artificial Intelligence That Explain Its Importance.” Forbes, February 14, 2018.
Marr, B. “What Everyone Should Know About Cognitive Com- puting.” Forbes.com, March 23, 2016.
Martin, J. “10 Things Marketers Need to Know about AI.” CIO. com, February 13, 2017.
McPherson, S.S. Artificial Intelligence: Building Smarter Ma- chines. Breckenridge, CO: Twenty-First Century Books, 2017.
Meister, J. “The Future of Work: How Artificial Intelligence Will Transform the Employee Experience.” Forbes.com, November 9, 2017.
Metz, C. “Facebook’s Augmented Reality Engine Brings AI Right to Your Phone.” Wired, April 19, 2017.
Mittal, V. “Top 15 Deep Learning Applications That Will Rule the World in 2018 and Beyond.” Medium.com, October 3, 2017.
Narayan, K. “Leverage Artificial Intelligence to Build your Sales Pipeline.” LinkedIn, February 14, 2018.
Ng, A. “What Artificial Intelligence Can and Can’t Do Right Now.” Harvard Business Review, November 9, 2016.
Nordrum, A. “Hedge Funds Look to Machine Learning, Crowd- sourcing for Competitive Advantage.” IEEE Spectrum, June 28, 2017.
Ovaska-Few, S. “How Artificial Intelligence Is Changing Accounting.” Journal of Accountancy, October 9, 2017.
116 Part I • Introduction to Analytics and AI
Padmanabhan, G. “Industry-Specific Augmented Intelligence: A Catalysts for AI in the Enterprise.” Forbes, January 4, 2018.
Pennington, R. “Artificial Intelligence: The New Tool for Ac- complishing an Old Goal in Marketing.” Huffington Post, January 16, 2018.
Press, G. “Top 10 Hot Artificial Intelligence (AI) Technolo- gies.” Forbes, January 23, 2017.
Pyle, D., and C. San José. “An Executive’s Guide to Machine Learning.” McKinsey & Company, June 2015.
Reinharz, S. An Introduction to Artificial Intelligence: Profes- sional Edition: An Introductory Guide to the Evolution of Artificial Intelligence. Kindle Edition. Seattle, WA: Simulta- neous Device Usage (Amazon Digital Service), 2017.
Sample, I. “AI Watchdog Needed to Regulate Automated Decision-Making, Say Experts.” The Guardian, January 27, 2017.
Santana, D. “Metromile Launches AI Claims Platform.” Digital Insurance, July 25, 2017.
Savar, A. “3 Ways That A.I. Is Transforming HR and Recruit- ing.” INC.com, June 26, 2017.
Schrage, M. “4 Models for Using AI to Make Decisions.” Har- vard Business Review, January 27, 2017.
Shah, J. “Robots Are Learning Complex Tasks Just by Watch- ing Humans Do Them.” Harvard Business Review, June 21, 2016.
Sharma, G. “China Unveils Multi-Billion Dollar Artificial Intel- ligence Plan.” International Business Times, July 20, 2017. ibtimes.co.uk/china-unveils-multi-billion-dollar- artificial-intelligence-plan-1631171/ (accessed January 2018).
Sincavage, D. “How Artificial Intelligence Will Change Decision-Making for Businesses.” Business 2 Community, August 24, 2017.
Singh, H. “How Artificial Intelligence Will Transform Financial Services.” Information Management, June 6, 2017.
SMBWorld Asia Editors. “Hays: Artificial Intelligence Set to Revolutionize Recruitment.” Enterprise Innovation, August 30, 2017.
Smith, J. Machine Learning: Machine Learning for Beginners. Can Machines Really Learn Like Humans? All About Artificial Intelligence (AI), Deep Learning and Digital Neu-
ral Networks. Kindle Edition. Seattle, WA: Amazon Digital Service, 2017.
Staff. “Assisted, Augmented and Autonomous: The 3 Flavours of AI Decisions.” Software and Technology, June 28, 2017. tgdaily.com/technology/assisted-augmented-and- autonomous-the-3-flavours-of-ai-decisions
Steffi, S. “List of 50 Unique AI Technologies.” Hacker Noon. com, October 18, 2017.
Taylor, P. “Welcome to the Machine – Learning.” Forbes Brand- Voice, June 3, 2016. forbes.com/sites/sap/2016/06/03/ welcome-to-the-machine-learning/#3175d50940fe (accessed June 2017).
Theobald, O. Machine Learning for Absolute Beginners: A Plain English Introduction. Kindle Edition. Seattle, WA, 2017.
Tiwan, R. “Artificial Intelligence (AI) in Banking Case Study Report 2017.” iCrowd Newswire, July 7, 2017.
USC. “AI Computer Vision Breakthrough IDs Poachers in Less Than Half a Second.” Press Release, February 8, 2018.
Violino, B. “Most Firms Expect Rapid Returns on Artificial Intelligence Investments.” Information Management, November 1, 2017.
Warawa, J. “Here’s Why Accountants (Yes, YOU!) Should Be Driving AI Innovation.” CPA Practice Advisor, November 1, 2017.
Waxer, C. “Get Ready for the BOT Revolution.” Computer- world, October 17, 2016.
Wellers, D., et al. “8 Ways Machine Learning Is Improving Companies’ Work Processes.” Harvard Business Review, May 31, 2017.
Wislow, E. “5 Ways to Use Artificial Intelligence (AI) in Human Resources.” Big Data Made Simple, October 24, 2017. bigdata-madesimple.com/5-ways-to-use-artificial- intelligence-ai-in-human-resources/.
Yurcan, B. “TD’s Innovation Agenda: Experiments with Alexa, AI and Augmented Reality.” Information Management, December 27, 2017.
Zarkadakis, G. In Our Own Image: Savior or Destroyer? The History and Future of Artificial Intelligence. New York, NY: Pegasus Books, 2016.
Zhou, A. “EY, Deloitte and PwC Embrace Artificial Intelligence for Tax and Accounting.” Forbes.com, November 14, 2017.
117
LEARNING OBJECTIVES
Nature of Data, Statistical Modeling, and Visualization
■■ Understand the nature of data as they relate to business intelligence (BI) and analytics
■■ Learn the methods used to make real-world data analytics ready
■■ Describe statistical modeling and its relationship to business analytics
■■ Learn about descriptive and inferential statistics ■■ Define business reporting and understand its historical evolution
■■ Understand the importance of data/information visualization
■■ Learn different types of visualization techniques ■■ Appreciate the value that visual analytics brings to business analytics
■■ Know the capabilities and limitations of dashboards
I n the age of Big Data and business analytics in which we are living, the importance of data is undeniable. Newly coined phrases such as “data are the oil,” “data are the new bacon,” “data are the new currency,” and “data are the king” are further stress- ing the renewed importance of data. But the type of data we are talking about is obvi- ously not just any data. The “garbage in garbage out—GIGO” concept/principle applies to today’s Big Data phenomenon more so than any data definition that we have had in the past. To live up to their promise, value proposition, and ability to turn into insight, data have to be carefully created/identified, collected, integrated, cleaned, transformed, and properly contextualized for use in accurate and timely decision making.
Data are the main theme of this chapter. Accordingly, the chapter starts with a de- scription of the nature of data: what they are, what different types and forms they can come in, and how they can be preprocessed and made ready for analytics. The first few sections of the chapter are dedicated to a deep yet necessary understanding and process- ing of data. The next few sections describe the statistical methods used to prepare data as input to produce both descriptive and inferential measures. Following the statistics sections are sections on reporting and visualization. A report is a communication artifact
3 C H A P T E R
118 Part I • Introduction to Analytics and AI
prepared with the specific intention of converting data into information and knowledge and relaying that information in an easily understandable/digestible format. Today, these reports are visually oriented, often using colors and graphical icons that collectively look like a dashboard to enhance the information content. Therefore, the latter part of the chapter is dedicated to subsections that present the design, implementation, and best practices regarding information visualization, storytelling, and information dashboards.
This chapter has the following sections:
3.1 Opening Vignette: SiriusXM Attracts and Engages a New Generation of Radio Consumers with Data-Driven Marketing 118
3.2 Nature of Data 121 3.3 Simple Taxonomy of Data 125 3.4 Art and Science of Data Preprocessing 129 3.5 Statistical Modeling for Business Analytics 139 3.6 Regression Modeling for Inferential Statistics 151 3.7 Business Reporting 163 3.8 Data Visualization 166 3.9 Different Types of Charts and Graphs 171
3.10 Emergence of Visual Analytics 176 3.11 Information Dashboards 182
3.1 OPENING VIGNETTE: SiriusXM Attracts and Engages a New Generation of Radio Consumers with Data-Driven Marketing
SiriusXM Radio is a satellite radio powerhouse, the largest radio company in the world with $3.8 billion in annual revenues and a wide range of hugely popular music, sports, news, talk, and entertainment stations. The company, which began broadcasting in 2001 with 50,000 subscribers, had 18.8 million subscribers in 2009, and today has nearly 29 million.
Much of SiriusXM’s growth to date is rooted in creative arrangements with automo- bile manufacturers; today, nearly 70 percent of new cars are SiriusXM enabled. Yet the company’s reach extends far beyond car radios in the United States to a worldwide pres- ence on the Internet, on smartphones, and through other services and distribution chan- nels, including SONOS, JetBlue, and Dish.
BUSINESS CHALLENGE
Despite these remarkable successes, changes in customer demographics, technology, and a competitive landscape over the past few years have posed a new series of business challenges and opportunities for SiriusXM. Here are some notable ones:
• As its market penetration among new cars increased, the demographics of its buy- ers changed, skewing toward younger people with less discretionary income. How could SiriusXM reach this new demographic?
• As new cars become used cars and change hands, how could SiriusXM identify, engage, and convert second owners to paying customers?
• With its acquisition of the connected vehicle business from Agero—the leading pro- vider of telematics in the U.S. car market—SiriusXM gained the ability to deliver its service via satellite and wireless networks. How could it successfully use this acqui- sition to capture new revenue streams?
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 119
PROPOSED SOLUTION: SHIFTING THE VISION TOWARD DATA-DRIVEN MARKETING
SiriusXM recognized that to address these challenges, it would need to become a high- performance, data-driven marketing organization. The company began making that shift by establishing three fundamental tenets. First, personalized interactions—not mass marketing—would rule the day. The company quickly understood that to conduct more personalized marketing, it would have to draw on past history and interactions as well as on a keen understanding of the consumer’s place in the subscription life cycle.
Second, to gain that understanding, information technology (IT) and its external tech- nology partners would need the ability to deliver integrated data, advanced analytics, integrated marketing platforms, and multichannel delivery systems.
And third, the company could not achieve its business goals without an integrated and consistent point of view across the company. Most important, the technology and business sides of SiriusXM would have to become true partners to best address the chal- lenges involved in becoming a high-performance marketing organization that draws on data-driven insights to speak directly with consumers in strikingly relevant ways.
Those data-driven insights, for example, would enable the company to differentiate between consumers, owners, drivers, listeners, and account holders. The insights would help SiriusXM to understand what other vehicles and services are part of each household and cre- ate new opportunities for engagement. In addition, by constructing a coherent and reliable 360-degree view of all its consumers, SiriusXM could ensure that all messaging in all cam- paigns and interactions would be tailored, relevant, and consistent across all channels. The important bonus is that a more tailored and effective marketing is typically more cost-efficient.
IMPLEMENTATION: CREATING AND FOLLOWING THE PATH TO HIGH-PERFORMANCE MARKETING
At the time of its decision to become a high-performance marketing company, SiriusXM was working with a third-party marketing platform that did not have the capacity to support SiriusXM’s ambitions. The company then made an important, forward-thinking decision to bring its marketing capabilities in-house—and then carefully plotted what it would need to do to make the transition successfully.
1. Improve data cleanliness through improved master data management and governance. Although the company was understandably impatient to put ideas into action, data hygiene was a necessary first step to create a reliable window into consumer behavior.
2. Bring marketing analytics in-house and expand the data warehouse to enable scale and fully support integrated marketing analytics.
3. Develop new segmentation and scoring models to run in databases, eliminating la- tency and data duplication.
4. Extend the integrated data warehouse to include marketing data and scoring, lever- aging in-database analytics.
5. Adopt a marketing platform for campaign development. 6. Bring all of its capability together to deliver real-time offer management across all
marketing channels: call center, mobile, Web, and in-app.
Completing those steps meant finding the right technology partner. SiriusXM chose Teradata because its strengths were a powerful match for the project and company. Teradata offered the ability to:
• Consolidate data sources with an integrated data warehouse (IDW), advanced ana- lytics, and powerful marketing applications.
• Solve data-latency issues.
120 Part I • Introduction to Analytics and AI
• Significantly reduce data movement across multiple databases and applications. • Seamlessly interact with applications and modules for all marketing areas. • Scale and perform at very high levels for running campaigns and analytics in-database. • Conduct real-time communications with customers. • Provide operational support, either via the cloud or on premise.
This partnership has enabled SiriusXM to move smoothly and swiftly along its road map, and the company is now in the midst of a transformational, five-year process. After establishing its strong data governance process, SiriusXM began by implementing its IDW, which allowed the company to quickly and reliably operationalize insights through- out the organization.
Next, the company implemented Customer Interaction Manager—part of the Teradata Integrated Marketing Cloud, which enables real-time, dialog-based customer interaction across the full spectrum of digital and traditional communication channels. SiriusXM also will incorporate the Teradata Digital Messaging Center.
Together, the suite of capabilities allows SiriusXM to handle direct communications across multiple channels. This evolution will enable real-time offers, marketing messages, and recommendations based on previous behavior.
In addition to streamlining the way it executes and optimizes outbound marketing activities, SiriusXM is also taking control of its internal marketing operations with the implementation of Marketing Resource Management, also part of the Teradata Integrated Marketing Cloud. The solution will allow SiriusXM to streamline workflow, optimize mar- keting resources, and drive efficiency through every penny of its marketing budget.
RESULTS: REAPING THE BENEFITS
As SiriusXM continues its evolution into a high-performance marketing organization, it already is benefiting from its thoughtfully executed strategy. Household-level consumer insights and a complete view of marketing touch strategy with each consumer enable SiriusXM to create more targeted offers at the household, consumer, and device levels. By bringing the data and marketing analytics capabilities in-house, SiriusXM achieved the following:
• Campaign results in near real time rather than four days, resulting in massive reduc- tions in cycle times for campaigns and the analysts who support them.
• Closed-loop visibility, allowing the analysts to support multistage dialogs and in-campaign modifications to increase campaign effectiveness.
• Real-time modeling and scoring to increase marketing intelligence and sharpen cam- paign offers and responses at the speed of their business.
Finally, SiriusXM’s experience has reinforced the idea that high-performance market- ing is a constantly evolving concept. The company has implemented both processes and the technology that give it the capacity for continued and flexible growth.
u QUESTIONS FOR THE OPENING VIGNETTE
1. What does SiriusXM do? In what type of market does it conduct its business? 2. What were its challenges? Comment on both technology and data-related
challenges.
3. What were the proposed solutions? 4. How did the company implement the proposed solutions? Did it face any
implementation challenges?
5. What were the results and benefits? Were they worth the effort/investment? 6. Can you think of other companies facing similar challenges that can potentially
benefit from similar data-driven marketing solutions?
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 121
WHAT WE CAN LEARN FROM THIS VIGNETTE
Striving to thrive in a fast-changing competitive industry, SiriusXM realized the need for a new and improved marketing infrastructure (one that relies on data and analytics) to effectively communicate its value proposition to its existing and potential custom- ers. As is the case in any industry, success or mere survival in entertainment depends on intelligently sensing the changing trends (likes and dislikes) and putting together the right messages and policies to win new customers while retaining the existing ones. The key is to create and manage successful marketing campaigns that resonate with the target population of customers and have a close feedback loop to adjust and modify the message to optimize the outcome. At the end, it was all about the preci- sion in the way that SiriusXM conducted business: being proactive about the changing nature of the clientele and creating and transmitting the right products and services in a timely manner using a fact-based/data-driven holistic marketing strategy. Source identification, source creation, access and collection, integration, cleaning, transforma- tion, storage, and processing of relevant data played a critical role in SiriusXM’s suc- cess in designing and implementing a marketing analytics strategy as is the case in any analytically savvy successful company today, regardless of the industry in which they are participating.
Sources: C. Quinn, “Data-Driven Marketing at SiriusXM,” Teradata Articles & News, 2016. http://bigdata. teradata.com/US/Articles-News/Data-Driven-Marketing-At-SiriusXM/ (accessed August 2016); “SiriusXM Attracts and Engages a New Generation of Radio Consumers.” http://assets.teradata.com/resourceCenter/ downloads/CaseStudies/EB8597.pdf?processed=1.
3.2 NATURE OF DATA
Data are the main ingredient for any BI, data science, and business analytics initiative. In fact, they can be viewed as the raw material for what popular decision technolo- gies produce—information, insight, and knowledge. Without data, none of these tech- nologies could exist and be popularized—although traditionally we have built analytics models using expert knowledge and experience coupled with very little or no data at all; however, those were the old days, and now data are of the essence. Once perceived as a big challenge to collect, store, and manage, data today are widely considered among the most valuable assets of an organization with the potential to create invaluable insight to better understand customers, competitors, and the business processes.
Data can be small or very large. They can be structured (nicely organized for computers to process), or they can be unstructured (e.g., text that is created for humans and hence not readily understandable/consumable by computers). Data can come in small batches continuously or can pour in all at once as a large batch. These are some of the characteristics that define the inherent nature of today’s data, which we often call Big Data. Even though these characteristics of data make them more challenging to process and consume, they also make the data more valuable because the character- istics enrich them beyond their conventional limits, allowing for the discovery of new and novel knowledge. Traditional ways to manually collect data (via either surveys or human-entered business transactions) mostly left their places to modern-day data collection mechanisms that use Internet and/or sensor/radio frequency identification (RFID)–based computerized networks. These automated data collection systems are not only enabling us to collect more volumes of data but also enhancing the data quality and integrity. Figure 3.1 illustrates a typical analytics continuum—data to analytics to actionable information.
122 Part I • Introduction to Analytics and AI
Although their value proposition is undeniable, to live up their promise, data must comply with some basic usability and quality metrics. Not all data are useful for all tasks, obviously. That is, data must match with (have the coverage of the specifics for) the task for which they are intended to be used. Even for a specific task, the relevant data on hand need to comply with the quality and quantity requirements. Essentially, data have to be analytics ready. So what does it mean to make data analytics ready? In addition to its relevancy to the problem at hand and the quality/quantity requirements, it also has to have a certain structure in place with key fields/variables with properly normalized val- ues. Furthermore, there must be an organization-wide agreed-on definition for common variables and subject matters (sometimes also called master data management), such as how to define a customer (what characteristics of customers are used to produce a holis- tic enough representation to analytics) and where in the business process the customer- related information is captured, validated, stored, and updated.
Sometimes the representation of the data depends on the type of analytics being employed. Predictive algorithms generally require a flat file with a target variable, so mak- ing data analytics ready for prediction means that data sets must be transformed into a flat-file format and made ready for ingestion into those predictive algorithms. It is also imperative to match the data to the needs and wants of a specific predictive algorithm and/or a software tool. For instance, neural network algorithms require all input variables
UOB 1.0
X
UOB 2.2
UOB 2.1
UOB 3.0
ERP CRM SCM
Business Process
Google+
Linked In
YouTube
Tumblr Flicker
Instagram Pinterest
Snapchat
Reddit Foursquare
Internet/Social Media
Machines/Internet of Things
Data Storage Analytics
Data Protection
Cloud Storage and Computing
Pa tte
rn s
Trends
Knowledge
Applications
End Users
Validate
Built
Test
X
FIGURE 3.1 A Data to Knowledge Continuum.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 123
to be numerically represented (even the nominal variables need to be converted into pseudo binary numeric variables), whereas decision tree algorithms do not require such numerical transformation—they can easily and natively handle a mix of nominal and nu- meric variables.
Analytics projects that overlook data-related tasks (some of the most critical steps) often end up with the wrong answer for the right problem, and these unintentionally cre- ated, seemingly good answers could lead to inaccurate and untimely decisions. Following are some of the characteristics (metrics) that define the readiness level of data for an ana- lytics study (Delen, 2015; Kock, McQueen, & Corner, 1997).
• Data source reliability. This term refers to the originality and appropriateness of the storage medium where the data are obtained—answering the question of “Do we have the right confidence and belief in this data source?” If at all possible, one should always look for the original source/creator of the data to eliminate/mitigate the possibilities of data misrepresentation and data transformation caused by the mishandling of the data as they moved from the source to destination through one or more steps and stops along the way. Every move of the data creates a chance to unintentionally drop or reformat data items, which limits the integrity and perhaps true accuracy of the data set.
• Data content accuracy. This means that data are correct and are a good match for the analytics problem—answering the question of “Do we have the right data for the job?” The data should represent what was intended or defined by the original source of the data. For example, the customer’s contact information recorded within a database should be the same as what the customer said it was. Data accuracy will be covered in more detail in the following subsection.
• Data accessibility. This term means that the data are easily and readily obtainable— answering the question of “Can we easily get to the data when we need to?” Access to data can be tricky, especially if they are stored in more than one location and storage medium and need to be merged/transformed while accessing and obtaining them. As the traditional relational database management systems leave their place (or coexist with a new generation of data storage mediums such as data lakes and Hadoop infra- structure), the importance/criticality of data accessibility is also increasing.
• Data security and data privacy. Data security means that the data are secured to allow only those people who have the authority and the need to access them and to prevent anyone else from reaching them. Increasing popularity in educational degrees and certificate programs for Information Assurance is evidence of the criti- cality and the increasing urgency of this data quality metric. Any organization that maintains health records for individual patients must have systems in place that not only safeguard the data from unauthorized access (which is mandated by federal laws such as the Health Insurance Portability and Accountability Act [HIPAA]) but also accurately identify each patient to allow proper and timely access to records by authorized users (Annas, 2003).
• Data richness. This means that all required data elements are included in the data set. In essence, richness (or comprehensiveness) means that the available variables portray a rich enough dimensionality of the underlying subject matter for an accurate and worthy analytics study. It also means that the information content is complete (or near complete) to build a predictive and/or prescriptive analytics model.
• Data consistency. This means that the data are accurately collected and com- bined/merged. Consistent data represent the dimensional information (variables of interest) coming from potentially disparate sources but pertaining to the same sub- ject. If the data integration/merging is not done properly, some of the variables of different subjects could appear in the same record—having two different patient
124 Part I • Introduction to Analytics and AI
records mixed up; for instance, this could happen while merging the demographic and clinical test result data records.
• Data currency/data timeliness. This means that the data should be up-to-date (or as recent/new as they need to be) for a given analytics model. It also means that the data are recorded at or near the time of the event or observation so that the time delay–related misrepresentation (incorrectly remembering and encoding) of the data is prevented. Because accurate analytics relies on accurate and timely data, an essential characteristic of analytics-ready data is the timeliness of the creation and access to data elements.
• Data granularity. This requires that the variables and data values be defined at the lowest (or as low as required) level of detail for the intended use of the data. If the data are aggregated, they might not contain the level of detail needed for an analytics algorithm to learn how to discern different records/cases from one another. For example, in a medical setting, numerical values for laboratory results should be recorded to the appropriate decimal place as required for the meaning- ful interpretation of test results and proper use of those values within an analytics algorithm. Similarly, in the collection of demographic data, data elements should be defined at a granular level to determine the differences in outcomes of care among various subpopulations. One thing to remember is that the data that are aggregated cannot be disaggregated (without access to the original source), but they can easily be aggregated from its granular representation.
• Data validity. This is the term used to describe a match/mismatch between the actual and expected data values of a given variable. As part of data definition, the acceptable values or value ranges for each data element must be defined. For example, a valid data definition related to gender would include three values: male, female, and unknown.
• Data relevancy. This means that the variables in the data set are all relevant to the study being conducted. Relevancy is not a dichotomous measure (whether a variable is relevant or not); rather, it has a spectrum of relevancy from least relevant to most relevant. Based on the analytics algorithms being used, one can choose to include only the most relevant information (i.e., variables) or, if the algorithm is capable enough to sort them out, can choose to include all the relevant ones regard- less of their levels. One thing that analytics studies should avoid is including totally irrelevant data into the model building because this could contaminate the informa- tion for the algorithm, resulting in inaccurate and misleading results.
The above-listed characteristics are perhaps the most prevailing metrics to keep up with; the true data quality and excellent analytics readiness for a specific application do- main would require different levels of emphasis to be placed on these metric dimensions and perhaps add more specific ones to this collection. The following section will delve into the nature of data from a taxonomical perspective to list and define different data types as they relate to different analytics projects.
u SECTION 3.2 REVIEW QUESTIONS
1. How do you describe the importance of data in analytics? Can we think of analytics without data?
2. Considering the new and broad definition of business analytics, what are the main inputs and outputs to the analytics continuum?
3. Where do the data for business analytics come from? 4. In your opinion, what are the top three data-related challenges for better analytics? 5. What are the most common metrics that make for analytics-ready data?
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 125
3.3 SIMPLE TAXONOMY OF DATA
The term data (datum in singular form) refers to a collection of facts usually obtained as the result of experiments, observations, transactions, or experiences. Data can consist of numbers, letters, words, images, voice recordings, and so on, as measurements of a set of variables (characteristics of the subject or event that we are interested in studying). Data are often viewed as the lowest level of abstraction from which information and then knowledge is derived.
At the highest level of abstraction, one can classify data as structured and unstruc- tured (or semistructured). Unstructured data/semistructured data are composed of any combination of textual, imagery, voice, and Web content. Unstructured/semistruc- tured data will be covered in more detail in the text mining and Web mining chapter. Structured data are what data mining algorithms use and can be classified as categori- cal or numeric. The categorical data can be subdivided into nominal or ordinal data, whereas numeric data can be subdivided into intervals or ratios. Figure 3.2 shows a simple data taxonomy.
• Categorical data. These represent the labels of multiple classes used to divide a variable into specific groups. Examples of categorical variables include race, sex, age group, and educational level. Although the latter two variables can also be considered in a numerical manner by using exact values for age and highest grade completed, for example, it is often more informative to categorize such variables into a relatively small number of ordered classes. The categorical data can also be called discrete data, implying that they represent a finite number of values with no continuum between them. Even if the values used for the categorical (or discrete) variables are numeric, these numbers are nothing more than symbols and do not imply the possibility of calculating fractional values.
• Nominal data. These contain measurements of simple codes assigned to objects as labels, which are not measurements. For example, the variable marital status can be generally categorized as (1) single, (2) married, and (3) divorced. Nominal
Data in Analytics
Structured Data Unstructured or Semi-Structured Data
Nominal
Ordinal
Textual
Multimedia
XML/JSON
Categorical Numerical
Interval
Ratio
Image
Audio
Video
FIGURE 3.2 A Simple Taxonomy of Data.
126 Part I • Introduction to Analytics and AI
data can be represented with binomial values having two possible values (e.g., yes/no, true/false, good/bad) or multinomial values having three or more pos- sible values (e.g., brown/green/blue, white/black/Latino/Asian, single/married/ divorced).
• Ordinal data. These contain codes assigned to objects or events as labels that also represent the rank order among them. For example, the variable credit score can be generally categorized as (1) low, (2) medium, or (3) high. Similar ordered relationships can be seen in variables such as age group (i.e., child, young, middle-aged, elderly) and educational level (i.e., high school, college, graduate school). Some predictive analytic algorithms, such as ordinal multiple logistic regression, take into account this additional rank-order information to build a better classification model.
• Numeric data. These represent the numeric values of specific variables. Examples of numerically valued variables include age, number of children, total household income (in U.S. dollars), travel distance (in miles), and temperature (in Fahrenheit degrees). Numeric values representing a variable can be integers (only whole numbers) or real (also fractional numbers). The numeric data can also be called continuous data, implying that the variable contains continuous measures on a specific scale that allows insertion of interim values. Unlike a discrete vari- able, which represents finite, countable data, a continuous variable represents scal- able measurements, and it is possible for the data to contain an infinite number of fractional values.
• Interval data. These are variables that can be measured on interval scales. A common example of interval scale measurement is temperature on the Celsius scale. In this particular scale, the unit of measurement is 1/100 of the difference between the melting temperature and the boiling temperature of water in atmospheric pres- sure; that is, there is not an absolute zero value.
• Ratio data. These include measurement variables commonly found in the physical sciences and engineering. Mass, length, time, plane angle, energy, and electric charge are examples of physical measures that are ratio scales. The scale type takes its name from the fact that measurement is the estimation of the ratio between a magnitude of a continuous quantity and a unit magnitude of the same kind. Informally, the dis- tinguishing feature of a ratio scale is the possession of a nonarbitrary zero value. For example, the Kelvin temperature scale has a nonarbitrary zero point of absolute zero, which is equal to –273.15 degrees Celsius. This zero point is nonarbitrary because the particles that comprise matter at this temperature have zero kinetic energy.
Other data types, including textual, spatial, imagery, video, and voice, need to be converted into some form of categorical or numeric representation before they can be pro- cessed by analytics methods (data mining algorithms; Delen, 2015). Data can also be classi- fied as static or dynamic (i.e., temporal or time series).
Some predictive analytics (i.e., data mining) methods and machine-learning algorithms are very selective about the type of data that they can handle. Providing them with incompatible data types can lead to incorrect models or (more often) halt the model development process. For example, some data mining methods need all the variables (both input and output) represented as numerically valued variables (e.g., neural net- works, support vector machines, logistic regression). The nominal or ordinal variables are converted into numeric representations using some type of 1-of-N pseudo variables (e.g., a categorical variable with three unique values can be transformed into three pseudo variables with binary values—1 or 0). Because this process could increase the number of variables, one should be cautious about the effect of such representations, especially for the categorical variables that have large numbers of unique values.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 127
Similarly, some predictive analytics methods, such as ID3 (a classic decision tree algorithm) and rough sets (a relatively new rule induction algorithm), need all the vari- ables represented as categorically valued variables. Early versions of these methods re- quired the user to discretize numeric variables into categorical representations before they could be processed by the algorithm. The good news is that most implementa- tions of these algorithms in widely available software tools accept a mix of numeric and nominal variables and internally make the necessary conversions before process- ing the data.
Data come in many different variable types and representation schemas. Business analytics tools are continuously improving in their ability to help data scientists in the daunting task of data transformation and data representation so that the data require- ments of specific predictive models and algorithms can be properly executed. Application Case 3.1 illustrates a business scenario in which one of the largest telecommunication companies streamlined and used a wide variety of rich data sources to generate customers insight to prevent churn and to create new revenue sources.
The Problem
In the ultra-competitive telecommunications indus- try, staying relevant to consumers while finding new sources of revenue is critical, especially since cur- rent revenue sources are in decline.
For Fortune 13 powerhouse Verizon, the secret weapon that catapulted the company into the nation’s largest and most reliable network provider is also guiding the business toward future success (see the following figure for some numbers about Verizon). The secret weapon? Data and analytics. Because telecommunication companies are typically rich in data, having the right analytics solution and personnel in place can uncover critical insights that benefit every area of the business.
The Backbone of the Company
Since its inception in 2000, Verizon has partnered with Teradata to create a data and analytics archi- tecture that drives innovation and science-based decision making. The goal is to stay relevant to cus- tomers while also identifying new business oppor- tunities and making adjustments that result in more cost-effective operations.
“With business intelligence, we help the business identify new business opportunities or
make course corrections to operate the business in a more cost-effective way,” said Grace Hwang, executive director of Financial Performance & Analytics, BI, for Verizon. “We support decision makers with the most relevant information to improve the competitive advantage of Verizon.”
By leveraging data and analytics, Verizon is able to offer a reliable network, ensure customer satisfaction, and develop products and services that consumers want to buy.
“Our incubator of new products and services will help bring the future to our customers,” Hwang said. “We’re using our network to make breakthroughs in
Application Case 3.1 Verizon Answers the Call for Innovation: The Nation’s Largest Network Provider uses Advanced Analytics to Bring the Future to its Customers
Verizon by the Numbers
The top ranked wireless carrier in the U.S. has:
$131.6B
177K
1,700
112.1M
106.5M
13M
retail locations
retail connections
postpaid customers
TV and internet subscribers
in revenue
employees
$
(Continued )
128 Part I • Introduction to Analytics and AI
interactive entertainment, digital media, the Internet of Things, and broadband services.”
Data Insights across Three Business Units
Verizon relies on advanced analytics that are exe- cuted on the Teradata® Unified Data Architecture™ to support its business units. The analytics enable Verizon to deliver on its promise to help customers innovate their lifestyles and provide key insights to support these three areas:
• Identify new revenue sources. Research and development teams use data, analytics, and strategic partnerships to test and develop with the Internet of Things (IoT). The new frontier in data is IoT, which will lead to new revenues that in turn generate opportunities for top-line growth. Smart cars, smart agricul- ture, and smart IoT will all be part of this new growth.
• Predict churn in the core mobile business. Verizon has multiple use cases that demonstrate how its advanced analytics enable laser-accurate churn prediction—within a one to two percent margin—in the mobile space. For a $131 billion company, predicting churn with such precision is significant. By recognizing specific patterns in tablet data usage, Verizon can identify which customers most often access their tablets, then engage those who do not.
• Forecast mobile phone plans. Customer behav- ioral analytics allow finance to better predict earnings in fast-changing market conditions. The U.S. wireless industry is moving from monthly payments for both the phone and the service to paying for the phone independently. This opens up a new opportunity for Verizon to gain busi- ness. The analytic environment helps Verizon better predict churn with new plans and forecast the impact of changes to pricing plans.
The analytics deliver what Verizon refers to as “hon- est data” that inform various business units. “Our mission is to be the honest voice and the indepen- dent third-party opinion on the success or oppor- tunities for improvement to the business,” Hwang
explains. “So my unit is viewed as the golden source of information, and we come across with the honest voice, and a lot of the business decisions are through various rungs of course correction.”
Hwang adds that oftentimes, what forces a company to react is competitors affecting change in the marketplace, rather than the company making the wrong decisions. “So we try to guide the business through the best course of correc- tion, wherever applicable, timely, so that we can continue to deliver record-breaking results year after year,” she said. “I have no doubt that the business intelligence had led to such success in the past.”
Disrupt and Innovate
Verizon leverages advanced analytics to optimize marketing by sending the most relevant offers to customers. At the same time, the company relies on analytics to ensure they have the financial acumen to stay number one in the U.S. mobile market. By continuing to disrupt the industry with innovative products and solutions, Verizon is positioned to remain the wireless standard for the industry.
“We need the marketing vision and the sales rigor to produce the most relevant offer to our customers, and then at the same time we need to have the finance rigor to ensure that whatever we offer to the customer is also profitable to the business so that we’re responsible to our share- holders,” Hwang says.
In Summary—Executing the Seven Ps of Modern Marketing
Telecommunications giant Verizon uses seven Ps to drive its modern-day marketing efforts. The Ps, when used in unison, help Verizon penetrate the market in the way it predicted.
1. People: Understanding customers and their needs to create the product.
2. Place: Where customers shop. 3. Product: The item that’s been manufactured
and is for sale.
Application Case 3.1 (Continued)
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 129
u SECTION 3.3 REVIEW QUESTIONS
1. What are data? How do data differ from information and knowledge? 2. What are the main categories of data? What types of data can we use for BI and
analytics?
3. Can we use the same data representation for all analytics models? Why, or why not?
4. What is a 1-of-N data representation? Why and where is it used in analytics?
3.4 ART AND SCIENCE OF DATA PREPROCESSING
Data in their original form (i.e., the real-world data) are not usually ready to be used in analytics tasks. They are often dirty, misaligned, overly complex, and inaccurate. A te- dious and time-demanding process (so-called data preprocessing) is necessary to con- vert the raw real-world data into a well-refined form for analytics algorithms (Kotsiantis, Kanellopoulos, & Pintelas, 2006). Many analytics professionals would testify that the time spent on data preprocessing (which is perhaps the least enjoyable phase in the whole process) is significantly longer than the time spent on the rest of the analytics tasks (the fun of analytics model building and assessment). Figure 3.3 shows the main steps in the data preprocessing endeavor.
In the first step of data preprocessing, the relevant data are collected from the iden- tified sources, the necessary records and variables are selected (based on an intimate understanding of the data, the unnecessary information is filtered out), and the records coming from multiple data sources are integrated/merged (again, using the intimate un- derstanding of the data, the synonyms and homonyms are able to be handled properly).
In the second step of data preprocessing, the data are cleaned (this step is also known as data scrubbing). Data in their original/raw/real-world form are usually dirty (Hernández & Stolfo, 1998; Kim et al., 2003). In this phase, the values in the data set are identified and dealt with. In some cases, missing values are an anomaly in the data set, in which case they need to be imputed (filled with a most probable value) or ignored; in other cases, the missing values are a natural part of the data set
4. Process: How customers get to the shop or place to buy the product.
5. Pricing: Working with promotions to get cus- tomers’ attention.
6. Promo: Working with pricing to get customers’ attention.
7. Physical evidence: The business intelligence that gives insights.
“The Aster and Hadoop environment allows us to explore things we suspect could be the rea- sons for breakdown in the seven Ps,” says Grace Hwang, executive director of Financial Performance & Analytics, BI, for Verizon. “This goes back to
providing the business value to our decision- makers. With each step in the seven Ps, we ought to be able to tell them where there are opportunities for improvement.”
Questions for Case 3.1
1. What was the challenge Verizon was facing?
2. What was the data-driven solution proposed for Verizon’s business units?
3. What were the results?
Source: Teradata Case Study “Verizon Answers the Call for Innovation” https://www.teradata.com/Resources/Case-Studies/ Verizon-answers-the-call-for-innovation (accessed July 2018).
130 Part I • Introduction to Analytics and AI
(e.g., the household income field is often left unanswered by people who are in the top income tier). In this step, the analyst should also identify noisy values in the data (i.e., the outliers) and smooth them out. In addition, inconsistencies (unusual values within a variable) in the data should be handled using domain knowledge and/or expert opinion.
In the third step of data preprocessing, the data are transformed for better process- ing. For instance, in many cases, the data are normalized between a certain minimum and maximum for all variables to mitigate the potential bias of one variable having
DW
Well-Formed Data
Social Data
Legacy DBWeb Data
Data Consolidation Collect data Select data Integrate data
Data Cleaning Impute values Reduce noise Eliminate duplicates
Data Transformation Normalize data Discretize data Create attributes
Data Reduction Reduce dimension Reduce volume Balance data
OLTP
Raw Data Sources
F e e d b a c k
FIGURE 3.3 Data Preprocessing Steps.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 131
large numeric values (such as household income) dominating other variables (such as number of dependents or years in service, which could be more important) having smaller values. Another transformation that takes place is discretization and/or aggrega- tion. In some cases, the numeric variables are converted to categorical values (e.g., low, medium, high); in other cases, a nominal variable’s unique value range is reduced to a smaller set using concept hierarchies (e.g., as opposed to using the individual states with 50 different values, one could choose to use several regions for a variable that shows location) to have a data set that is more amenable to computer processing. Still, in other cases, one might choose to create new variables based on the existing ones to magnify the information found in a collection of variables in the data set. For instance, in an organ transplantation data set, one might choose to use a single variable show- ing the blood-type match (1: match, 0: no match) as opposed to separate multinominal values for the blood type of both the donor and the recipient. Such simplification could increase the information content while reducing the complexity of the relationships in the data.
The final phase of data preprocessing is data reduction. Even though data scientists (i.e., analytics professionals) like to have large data sets, too much data can also be a problem. In the simplest sense, one can visualize the data commonly used in predictive analytics projects as a flat file consisting of two dimensions: variables (the number of columns) and cases/records (the number of rows). In some cases (e.g., image process- ing and genome projects with complex microarray data), the number of variables can be rather large, and the analyst must reduce the number to a manageable size. Because the variables are treated as different dimensions that describe the phenomenon from differ- ent perspectives, in predictive analytics and data mining, this process is commonly called dimensional reduction (or variable selection). Even though there is not a single best way to accomplish this task, one can use the findings from previously published litera- ture; consult domain experts; run appropriate statistical tests (e.g., principal component analysis or independent component analysis); and, more preferably, use a combination of these techniques to successfully reduce the dimensions in the data into a more manage- able and most relevant subset.
With respect to the other dimension (i.e., the number of cases), some data sets can include millions or billions of records. Even though computing power is increasing ex- ponentially, processing such a large number of records cannot be practical or feasible. In such cases, one might need to sample a subset of the data for analysis. The underlying assumption of sampling is that the subset of the data will contain all relevant patterns of the complete data set. In a homogeneous data set, such an assumption could hold well, but real-world data are hardly ever homogeneous. The analyst should be extremely careful in selecting a subset of the data that reflects the essence of the complete data set and is not specific to a subgroup or subcategory. The data are usually sorted on some variable, and taking a section of the data from the top or bottom could lead to a biased data set on specific values of the indexed variable; therefore, always try to randomly select the records on the sample set. For skewed data, straightforward random sampling might not be sufficient, and stratified sampling (a proportional representation of different subgroups in the data is represented in the sample data set) might be required. Speaking of skewed data, it is a good practice to balance the highly skewed data by either oversampling the less represented or undersampling the more represented classes. Research has shown that balanced data sets tend to produce better prediction models than unbalanced ones (Thammasiri et al., 2014).
The essence of data preprocessing is summarized in Table 3.1, which maps the main phases (along with their problem descriptions) to a representative list of tasks and algorithms.
132 Part I • Introduction to Analytics and AI
TABLE 3.1 A Summary of Data Preprocessing Tasks and Potential Methods
Main Task Subtasks Popular Methods
Data consolidation Access and collect the data Select and filter the data Integrate and unify the data
SQL queries, software agents, Web services. Domain expertise, SQL queries, statistical tests. SQL queries, domain expertise, ontology-driven data mapping.
Data cleaning Handle missing values in the data
Fill in missing values (imputations) with most appropriate val- ues (mean, median, min/max, mode, etc.); recode the missing values with a constant such as “ML”; remove the record of the missing value; do nothing.
Identify and reduce noise in the data
Identify the outliers in data with simple statistical techniques (such as averages and standard deviations) or with cluster analysis; once identified, either remove the outliers or smooth them by using binning, regression, or simple averages.
Find and eliminate erroneous data
Identify the erroneous values in data (other than outliers), such as odd values, inconsistent class labels, odd distributions; once identified, use domain expertise to correct the values or remove the records holding the erroneous values.
Data transformation Normalize the data Reduce the range of values in each numerically valued variable to a standard range (e.g., 0 to 1 or -1 to +1) by using a vari- ety of normalization or scaling techniques.
Discretize or aggregate the data
If needed, convert the numeric variables into discrete represen- tations using range- or frequency-based binning techniques; for categorical variables, reduce the number of values by applying proper concept hierarchies.
Construct new attributes Derive new and more informative variables from the existing ones using a wide range of mathematical functions (as simple as addition and multiplication or as complex as a hybrid combi- nation of log transformations).
Data reduction Reduce number of attributes Use principal component analysis, independent component analysis, chi-square testing, correlation analysis, and decision tree induction.
Reduce number of records Perform random sampling, stratified sampling, expert- knowledge-driven purposeful sampling.
Balance skewed data Oversample the less represented or undersample the more represented classes.
It is almost impossible to underestimate the value proposition of data preprocess- ing. It is one of those time-demanding activities in which investment of time and effort pays off without a perceivable limit for diminishing returns. That is, the more resources you invest in it, the more you will gain at the end. Application Case 3.2 illustrates an interesting study that used raw, readily available academic data within an educational organization to develop predictive models to better understand attrition and improve freshman student retention in a large higher education institution. As the application case clearly states, each and every data preprocessing task described in Table 3.1 was criti- cal to a successful execution of the underlying analytics project, especially the task that related to the balancing of the data set.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 133
Student attrition has become one of the most chal- lenging problems for decision makers in academic institutions. Despite all the programs and services that are put in place to help retain students, accord- ing to the U.S. Department of Education’s Center for Educational Statistics (nces.ed.gov), only about half of those who enter higher education actually earn a bachelor’s degree. Enrollment management and the retention of students have become a top priority for administrators of colleges and universities in the United States and other countries around the world. High dropout of students usually results in overall financial loss, lower graduation rates, and an inferior school reputation in the eyes of all stakeholders. The legislators and policy makers who oversee higher education and allocate funds, the parents who pay for their children’s education to prepare them for a better future, and the students who make college choices look for evidence of institutional quality and reputation to guide their decision-making processes.
The Proposed Solution
To improve student retention, one should try to understand the nontrivial reasons behind the attrition. To be successful, one should also be able to accu- rately identify those students who are at risk of drop- ping out. So far, the vast majority of student attrition research has been devoted to understanding this com- plex, yet crucial, social phenomenon. Even though these qualitative, behavioral, and survey-based stud- ies revealed invaluable insight by developing and testing a wide range of theories, they do not pro- vide the much-needed instruments to accurately pre- dict (and potentially improve) student attrition. The project summarized in this case study proposed a quantitative research approach in which the histori- cal institutional data from student databases could be used to develop models that are capable of pre- dicting as well as explaining the institution-specific nature of the attrition problem. The proposed analyt- ics approach is shown in Figure 3.4.
Although the concept is relatively new to higher education, for more than a decade now, similar problems in the field of marketing man- agement have been studied using predictive data
analytics techniques under the name of “churn analysis” where the purpose has been to identify a sample among current customers to answer the question, “Who among our current customers are more likely to stop buying our products or services?” so that some kind of mediation or intervention pro- cess can be executed to retain them. Retaining exist- ing customers is crucial because, as we all know and as the related research has shown time and time again, acquiring a new customer costs on an order of magnitude more effort, time, and money than try- ing to keep the one that you already have.
Data Are of the Essence
The data for this research project came from a sin- gle institution (a comprehensive public university located in the Midwest region of the United States) with an average enrollment of 23,000 students, of which roughly 80 percent are the residents of the same state and roughly 19 percent of the students are listed under some minority classification. There is no significant difference between the two genders in the enrollment numbers. The average freshman student retention rate for the institution was about 80 percent, and the average six-year graduation rate was about 60 percent.
The study used five years of institutional data, which entailed 16,000+ students enrolled as fresh- men, consolidated from various and diverse univer- sity student databases. The data contained variables related to students’ academic, financial, and demo- graphic characteristics. After merging and convert- ing the multidimensional student data into a single flat file (a file with columns representing the vari- ables and rows representing the student records), the resultant file was assessed and preprocessed to identify and remedy anomalies and unusable val- ues. As an example, the study removed all inter- national student records from the data set because they did not contain information about some of the most reputed predictors (e.g., high school GPA, SAT scores). In the data transformation phase, some of the variables were aggregated (e.g., “Major” and “Concentration” variables aggregated to binary vari- ables MajorDeclared and ConcentrationSpecified)
Application Case 3.2 Improving Student Retention with Data-Driven Analytics
(Continued )
134 Part I • Introduction to Analytics and AI
FIGURE 3.4 An Analytics Approach to Predicting Student Attrition.
Application Case 3.2 (Continued)
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 135
for better interpretation for the predictive model- ing. In addition, some of the variables were used to derive new variables (e.g., Earned/Registered ratio and YearsAfterHighSchool).
Earned>Registered = EarnedHours> RegisteredHours
YearsAfterHigh = FreshmenEnrollmentYear - School HighSchoolGraduationYear
The Earned/Registered ratio was created to have a better representation of the students’ resiliency and determination in their first semester of the freshman year. Intuitively, one would expect greater values for this variable to have a positive impact on retention/ persistence. The YearsAfterHighSchool was created to measure the impact of the time taken between high school graduation and initial college enrollment. Intuitively, one would expect this variable to be a contributor to the prediction of attrition. These aggre- gations and derived variables are determined based on a number of experiments conducted for a number of logical hypotheses. The ones that made more com- mon sense and the ones that led to better prediction accuracy were kept in the final variable set. Reflecting the true nature of the subpopulation (i.e., the fresh- men students), the dependent variable (i.e., “Second Fall Registered”) contained many more yes records (~80%) than no records (~20%; see Figure 3.5).
Research shows that having such imbalanced data has a negative impact on model performance.
Therefore, the study experimented with the options of using and comparing the results of the same type of models built with the original imbalanced data (biased for the yes records) and the well-balanced data.
Modeling and Assessment
The study employed four popular classification meth- ods (i.e., artificial neural networks, decision trees, sup- port vector machines, and logistic regression) along with three model ensemble techniques (i.e., bagging, busting, and information fusion). The results obtained from all model types were then compared to each other using regular classification model assessment methods (e.g., overall predictive accuracy, sensitivity, specificity) on the holdout samples.
In machine-learning algorithms (some of which will be covered in Chapter 4), sensitivity analysis is a method for identifying the “cause-and-effect” relationship between the inputs and outputs of a given prediction model. The fundamental idea behind sensitivity analysis is that it measures the importance of predictor variables based on the change in modeling performance that occurs if a predictor variable is not included in the model. This modeling and experimentation practice is also called a leave-one-out assessment. Hence, the mea- sure of sensitivity of a specific predictor variable is the ratio of the error of the trained model without the predictor variable to the error of the model that includes this predictor variable. The more sensitive
50% No
50% No
B a la
n c e d D
a ta
Model Building, Testing, and Validating
Model Assessment
TP FP
FN TN
Yes No
Yes
No
(80%, 80%, 80%)
(90%, 100%, 50%)
Which one is better?
Input Data
80% No
20% Yes
*Yes: dropped out, No: persisted.
(accuracy, precision+, precision–)
(accuracy, precision+, precision–)
Validate
Built
Test
Im b a la
n c e d D
a ta
FIGURE 3.5 A Graphical Depiction of the Class Imbalance Problem.
(Continued )
136 Part I • Introduction to Analytics and AI
the network is to a particular variable, the greater the performance decrease would be in the absence of that variable and therefore the greater the ratio of importance. In addition to the predictive power of the models, the study also conducted sensitivity analyses to determine the relative importance of the input variables.
The Results
In the first set of experiments, the study used the original imbalanced data set. Based on the 10-fold cross-validation assessment results, the support vector machines produced the best accuracy with an overall prediction rate of 87.23 percent, and the decision tree was the runner-up with an overall prediction rate of 87.16 percent, followed by artificial neural networks and logistic regression with overall prediction rates of 86.45 percent and 86.12 percent, respectively (see Table 3.2). A careful examination of these results reveals that the prediction accuracy for the “Yes” class is significantly higher than the prediction accuracy of the “No” class. In fact, all four model types predicted the students who are likely to return for the second year with better than 90 percent accuracy, but the types did poorly on predicting the students who are likely to drop out after the freshman year with less than 50 percent accuracy. Because the prediction of the “No” class is the main purpose of this study, less than 50 percent accuracy for this class was deemed not acceptable. Such a difference in prediction accu- racy of the two classes can (and should) be attributed to the imbalanced nature of the training data set (i.e., ~80% “Yes” and ~20% “No” samples).
The next round of experiments used a well- balanced data set in which the two classes are represented nearly equally in counts. In realizing this approach, the study took all samples from the minority class (i.e., the “No” class herein), randomly selected an equal number of samples from the major- ity class (i.e., the “Yes” class herein), and repeated this process 10 times to reduce potential bias of random sampling. Each of these sampling processes resulted in a data set of 7,000+ records, of which both class labels (“Yes” and “No”) were equally represented. Again, using a 10-fold cross-validation methodology, the study developed and tested prediction models for all four model types. The results of these experi- ments are shown in Table 3.3. Based on the hold- out sample results, support vector machines once again generated the best overall prediction accuracy with 81.18 percent followed by decision trees, artifi- cial neural networks, and logistic regression with an overall prediction accuracy of 80.65 percent, 79.85 percent, and 74.26 percent, respectively. As can be seen in the per-class accuracy figures, the prediction models did significantly better on predicting the “No” class with the well-balanced data than they did with the unbalanced data. Overall, the three machine- learning techniques performed significantly better than their statistical counterpart, logistic regression.
Next, another set of experiments was con- ducted to assess the predictive ability of the three ensemble models. Based on the 10-fold cross- validation methodology, the information fusion– type ensemble model produced the best results with an overall prediction rate of 82.10 percent, followed by the bagging-type ensembles and boosting-type
TABLE 3.2 Prediction Results for the Original/Unbalanced Data Set
ANN(MLP) DT(C5) SVM LR
No Yes No Yes No Yes No Yes
No 1,494 384 1,518 304 1,478 255 1,438 376
Yes 1,596 11,142 1,572 11,222 1,612 11,271 1,652 11,150
SUM 3,090 11,526 3,090 11,526 3,090 11,526 3,090 11,526
Per-class accuracy 48.35% 96.67% 49.13% 97.36% 47.83% 97.79% 46.54% 96.74%
Overall accuracy 86.45% 87.16% 87.23% 86.12%
*ANN: Artificial Neural Network; MLP: Multi-Layer Perceptron; DT: Decision Tree; SVM: Support Vector Machine; LR: Logistic Regression
Application Case 3.2 (Continued)
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 137
ensembles with overall prediction rates of 81.80 per- cent and 80.21 percent, respectively (see Table 3.4). Even though the prediction results are slightly better than those of the individual models, ensembles are known to produce more robust prediction systems compared to a single-best prediction model (more on this can be found in Chapter 4).
In addition to assessing the prediction accu- racy for each model type, a sensitivity analysis was also conducted using the developed prediction models to identify the relative importance of the independent variables (i.e., the predictors). In real- izing the overall sensitivity analysis results, each of the four individual model types generated its own sensitivity measures, ranking all independent vari- ables in a prioritized list. As expected, each model type generated slightly different sensitivity rank- ings of the independent variables. After collecting all four sets of sensitivity numbers, the sensitivity numbers are normalized and aggregated and plot- ted in a horizontal bar chart (see Figure 3.6).
The Conclusions
The study showed that, given sufficient data with the proper variables, data mining methods are capable of predicting freshmen student attrition with approximately 80 percent accuracy. Results also showed that, regardless of the prediction model employed, the balanced data set (compared to unbalanced/original data set) produced better prediction models for identifying the students who are likely to drop out of the college prior to their sophomore year. Among the four individual pre- diction models used in this study, support vector machines performed the best, followed by deci- sion trees, neural networks, and logistic regres- sion. From the usability standpoint, despite the fact that support vector machines showed better prediction results, one might choose to use deci- sion trees, because compared to support vector machines and neural networks, they portray a more transparent model structure. Decision trees
TABLE 3.3 Prediction Results for the Balanced Data Set
Confusion Matrix
ANN(MLP) DT(C5) SVM LR
No Yes No Yes No Yes No Yes
No 2,309 464 2311 417 2,313 386 2,125 626
Yes 781 2,626 779 2,673 777 2,704 965 2,464
SUM 3,090 3,090 3,090 3,090 3,090 3,090 3,090 3,090
Per-class accuracy 74.72% 84.98% 74.79% 86.50% 74.85% 87.51% 68.77% 79.74%
Overall accuracy 79.85% 80.65% 81.18% 74.26%
TABLE 3.4 Prediction Results for the Three Ensemble Models
Boosting Bagging Information Fusion (boosted trees) (random forest) (weighted average)
No Yes No Yes No Yes
No 2,242 375 2,327 362 2,335 351
Yes 848 2,715 763 2,728 755 2,739
SUM 3,090 3,090 3,090 3,090 3,090 3,090
Per-class accuracy 72.56% 87.86% 75.31% 88.28% 75.57% 88.64%
Overall accuracy 80.21% 81.80% 82.10%
(Continued )
138 Part I • Introduction to Analytics and AI
explicitly show the reasoning process of different predictions, providing a justification for a specific outcome, whereas support vector machines and artificial neural networks are mathematical models that do not provide such a transparent view of “how they do what they do.”
Questions for Case 3.2
1. What is student attrition, and why is it an impor- tant problem in higher education?
2. What were the traditional methods to deal with the attrition problem?
3. List and discuss the data-related challenges within the context of this case study.
4. What was the proposed solution? What were the results?
Sources: D. Thammasiri, D. Delen, P. Meesad, & N. Kasap, “A Critical Assessment of Imbalanced Class Distribution Problem: The Case of Predicting Freshmen Student Attrition,” Expert Systems with Applications, 41(2), 2014, pp. 321–330; D. Delen, “A Comparative Analysis of Machine Learning Techniques for Student Retention Management,” Decision Support Systems, 49(4), 2010, pp. 498–506, and “Predicting Student Attrition with Data Mining Methods,” Journal of College Student Retention 13(1), 2011, pp. 17–35.
EarnedByRegistered
SpringStudentLoan
FallGPA
SpringGrantTuitionWaiverScholarship
FallRegisteredHours
FallStudentLoan
MaritalStatus
AdmissionType
Ethnicity
SATHighMath
SATHighEnglish
FallFederalWorkStudy
SpringFederalWorkStudy
FallGrantTuitionWaiverScholarship
PermanentAddressState
SATHighScience
SATHighComprehensive
SATHighReading
TransferredHours
ReceivedFallAid
MajorDeclared
ConcentrationSpecified
StartingTerm
HighSchoolGraduationMonth
HighSchoolGPA
YearsAfterHS
Age
0.00 0.20 0.40 0.60 1.000.80 1.20
Sex
CLEPHours
FIGURE 3.6 Sensitivity-Analysis-Based Variable Importance Results.
Application Case 3.2 (Continued)
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 139
u SECTION 3.4 REVIEW QUESTIONS
1. Why are the original/raw data not readily usable by analytics tasks? 2. What are the main data preprocessing steps? 3. What does it mean to clean/scrub the data? What activities are performed in this
phase?
4. Why do we need data transformation? What are the commonly used data transforma- tion tasks?
5. Data reduction can be applied to rows (sampling) and/or columns (variable selec- tion). Which is more challenging?
3.5 STATISTICAL MODELING FOR BUSINESS ANALYTICS
Because of the increasing popularity of business analytics, the traditional statistical meth- ods and underlying techniques are also regaining their attractiveness as enabling tools to support evidence-based managerial decision making. Not only are they regaining atten- tion and admiration, but this time, they are attracting business users in addition to statisti- cians and analytics professionals.
Statistics (statistical methods and underlying techniques) is usually considered as part of descriptive analytics (see Figure 3.7). Some of the statistical methods can also be considered as part of predictive analytics, such as discriminant analysis, multiple regres- sion, logistic regression, and k-means clustering. As shown in Figure 3.7, descriptive ana- lytics has two main branches: statistics and online analytics processing (OLAP). OLAP is the term used for analyzing, characterizing, and summarizing structured data stored in organizational databases (often stored in a data warehouse or in a data mart) using cubes (i.e., multidimensional data structures that are created to extract a subset of data values to answer a specific business question). The OLAP branch of descriptive analytics has also been called business intelligence. Statistics, on the other hand, helps to characterize the data, either one variable at a time or multivariable, all together using either descriptive or inferential methods.
Statistics—a collection of mathematical techniques to characterize and interpret data—has been around for a very long time. Many methods and techniques have been developed to address the needs of the end users and the unique characteristics of the data being analyzed. Generally speaking, at the highest level, statistical methods can be
Descriptive Inferential
OLAP Statistics
Business Analytics
Descriptive Predictive Prescriptive
FIGURE 3.7 Relationship between Statistics and Descriptive Analytics.
140 Part I • Introduction to Analytics and AI
classified as either descriptive or inferential. The main difference between descriptive and inferential statistics is the data used in these methods—whereas descriptive statistics is all about describing the sample data on hand, inferential statistics is about drawing inferences or conclusions about the characteristics of the population. In this section, we briefly describe descriptive statistics (because of the fact that it lays the foundation for, and is the integral part of, descriptive analytics), and in the following section we cover regression (both linear and logistic regression) as part of inferential statistics.
Descriptive Statistics for Descriptive Analytics
Descriptive statistics, as the name implies, describes the basic characteristics of the data at hand, often one variable at a time. Using formulas and numerical aggregations, descrip- tive statistics summarizes the data in such a way that often meaningful and easily under- standable patterns emerge from the study. Although it is very useful in data analytics and very popular among the statistical methods, descriptive statistics does not allow making conclusions (or inferences) beyond the sample of the data being analyzed. That is, it is simply a nice way to characterize and describe the data on hand without making conclu- sions (inferences or extrapolations) regarding the population of related hypotheses we might have in mind.
In business analytics, descriptive statistics plays a critical role—it allows us to un- derstand and explain/present our data in a meaningful manner using aggregated num- bers, data tables, or charts/graphs. In essence, descriptive statistics helps us convert our numbers and symbols into meaningful representations for anyone to understand and use. Such an understanding helps not only business users in their decision-making processes but also analytics professionals and data scientists to characterize and validate the data for other more sophisticated analytics tasks. Descriptive statistics allows analysts to identify data concertation, unusually large or small values (i.e., outliers), and unexpectedly dis- tributed data values for numeric variables. Therefore, the methods in descriptive statistics can be classified as either measures for central tendency or measures of dispersion. In the following section, we use a simple description and mathematical formulation/repre- sentation of these measures. In mathematical representation, we will use x1, x2, . . . , xn to represent individual values (observations) of the variable (measure) that we are interested in characterizing.
Measures of Centrality Tendency (Also Called Measures of Location or Centrality)
Measures of centrality are the mathematical methods by which we estimate or describe central positioning of a given variable of interest. A measure of central tendency is a single numerical value that aims to describe a set of data by simply identifying or estimat- ing the central position within the data. The mean (often called the arithmetic mean or the simple average) is the most commonly used measure of central tendency. In addition to mean, you could also see median or mode being used to describe the centrality of a given variable. Although, the mean, median, and mode are all valid measures of central tendency, under different circumstances, one of these measures of centrality becomes more appropriate than the others. What follows are short descriptions of these measures, including how to calculate them mathematically and pointers on the circumstances in which they are the most appropriate measure to use.
Arithmetic Mean
The arithmetic mean (or simply mean or average) is the sum of all the values/observa- tions divided by the number of observations in the data set. It is by far the most popular
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 141
and most commonly used measure of central tendency. It is used with continuous or discrete numeric data. For a given variable x, if we happen to have n values/observations 1x1, x2, . . ., xn2, we can write the arithmetic mean of the data sample (x, pronounced as x-bar) as follows:
x = x1 + x2 + g + xn
n
or
x = a i=1 n
xi n
The mean has several unique characteristics. For instance, the sum of the absolute devia- tions (differences between the mean and the observations) above the mean is the same as the sum of the deviations below the mean, balancing the values on either side of it. That said, it does not suggest, however, that half the observations are above and the other half are below the mean (a common misconception among those who do not know basic statistics). Also, the mean is unique for every data set and is meaningful and calculable for both interval- and ratio-type numeric data. One major downside is that the mean can be affected by outliers (observations that are considerably larger or smaller than the rest of the data points). Outliers can pull the mean toward their direction and, hence, bias the centrality representation. Therefore, if there are outliers or if the data are erratically dis- persed and skewed, one should either avoid using the mean as the measure of centrality or augment it with other central tendency measures, such as median and mode.
Median
The median is the measure of center value in a given data set. It is the number in the middle of a given set of data that has been arranged/sorted in order of magnitude (either ascending or descending). If the number of observations is an odd number, identifying the median is very easy—just sort the observations based on their values and pick the value right in the middle. If the number of observations is an even number, identify the two middle values, and then take the simple average of these two values. The median is meaningful and calculable for ratio, interval, and ordinal data types. Once determined, one-half of the data points in the data is above and the other half is below the median. In contrary to the mean, the median is not affected by outliers or skewed data.
Mode
The mode is the observation that occurs most frequently (the most frequent value in our data set). On a histogram, it represents the highest bar in a bar chart, and, hence, it can be considered as the most popular option/value. The mode is most useful for data sets that contain a relatively small number of unique values. That is, it could be useless if the data have too many unique values (as is the case in many engineering measurements that capture high precision with a large number of decimal places), rendering each value having either one or a very small number representing its frequency. Although it is a useful measure (especially for nominal data), mode is not a very good representation of centrality, and therefore, it should not be used as the only measure of central tendency for a given data set.
In summary, which central tendency measure is the best? Although there is not a clear answer to this question, here are a few hints—use the mean when the data are not prone to outliers and there is no significant level of skewness; use the median when the data have outliers and/or it is ordinal in nature; use the mode when the data are nominal.
142 Part I • Introduction to Analytics and AI
Perhaps the best practice is to use all three together so that the central tendency of the data set can be captured and represented from three perspectives. Mostly because “av- erage” is a very familiar and highly used concept to everyone in regular daily activities, managers (as well as some scientists and journalists) often use the centrality measures (especially mean) inappropriately when other statistical information should be consid- ered along with the centrality. It is a better practice to present descriptive statistics as a package—a combination of centrality and dispersion measures—as opposed to a single measure such as mean.
Measures of Dispersion (Also Called Measures of Spread or Decentrality)
Measures of dispersion are the mathematical methods used to estimate or describe the degree of variation in a given variable of interest. They represent the numerical spread (compactness or lack thereof) of a given data set. To describe this dispersion, a number of statistical measures are developed; the most notable ones are range, variance, and standard deviation (and also quartiles and absolute deviation). One of the main reasons why the measures of dispersion/spread of data values are important is the fact that they give us a framework within which we can judge the central tendency—give us the indica- tion of how well the mean (or other centrality measures) represents the sample data. If the dispersion of values in the data set is large, the mean is not deemed to be a very good representation of the data. This is because a large dispersion measure indicates large dif- ferences between individual scores. Also, in research, it is often perceived as a positive sign to see a small variation within each data sample, as it may indicate homogeneity, similarity, and robustness within the collected data.
Range
The range is perhaps the simplest measure of dispersion. It is the difference between the largest and the smallest values in a given data set (i.e., variables). So we calculate range by simply identifying the smallest value in the data set (minimum), identifying the largest value in the data set (maximum), and calculating the difference between them (range = maximum - minimum).
Variance
A more comprehensive and sophisticated measure of dispersion is the variance. It is a method used to calculate the deviation of all data points in a given data set from the mean. The larger the variance, the more the data are spread out from the mean and the more variability one can observe in the data sample. To prevent the offsetting of negative and positive differences, the variance takes into account the square of the distances from the mean. The formula for a data sample can be written as
s2 = a ni = 1( xi - x )2
n - 1
where n is the number of samples, x is the mean of the sample, and xi is the i th
value in the data set. The larger values of variance indicate more dispersion, whereas smaller values indicate compression in the overall data set. Because the differences are squared, larger deviations from the mean contribute significantly to the value of variance. Again, because the differences are squared, the numbers that represent de- viation/variance become somewhat meaningless (as opposed to a dollar difference, here you are given a squared dollar difference). Therefore, instead of variance, in
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 143
many business applications, we use a more meaningful dispersion measure, called standard deviation.
Standard Deviation
The standard deviation is also a measure of the spread of values within a set of data. The standard deviation is calculated by simply taking the square root of the variations. The following formula shows the calculation of standard deviation from a given sample of data points.
s = A a ni = 1(xi - x )2n - 1 Mean Absolute Deviation
In addition to variance and standard deviation, sometimes we also use mean absolute deviation to measure dispersion in a data set. It is a simpler way to calculate the overall deviation from the mean. Specifically, the mean absolute deviation is calculated by mea- suring the absolute values of the differences between each data point and the mean and then summing them. This process provides a measure of spread without being specific about the data point being lower or higher than the mean. The following formula shows the calculation of the mean absolute deviation:
MAD = a ni =1� xi - x �
n
Quartiles and Interquartile Range
Quartiles help us identify spread within a subset of the data. A quartile is a quarter of the number of data points given in a data set. Quartiles are determined by first sorting the data and then splitting the sorted data into four disjoint smaller data sets. Quartiles are a useful measure of dispersion because they are much less affected by outliers or a skewness in the data set than the equivalent measures in the whole data set. Quartiles are often reported along with the median as the best choice of measure of dispersion and central tendency, respectively, when dealing with skewed and/or data with outliers. A common way of expressing quartiles is as an interquartile range, which describes the difference between the third quartile (Q3) and the first quartile (Q1), telling us about the range of the middle half of the scores in the distribution. The quartile-driven descriptive measures (both centrality and dispersion) are best explained with a popular plot called a box-and-whiskers plot (or box plot).
Box-and-Whiskers Plot
The box-and-whiskers plot (or simply a box plot) is a graphical illustration of several descriptive statistics about a given data set. They can be either horizontal or vertical, but vertical is the most common representation, especially in modern-day analytics software products. It is known to be first created and presented by John W. Tukey in 1969. Box plot is often used to illustrate both centrality and dispersion of a given data set (i.e., the distribution of the sample data) in an easy-to-understand graphical notation. Figure 3.8 shows two box plots side by side, sharing the same y-axis. As shown therein, a single chart can have one or more box plots for visual comparison purposes. In such cases, the y-axis would be the common measure of magnitude (the numerical value of the
144 Part I • Introduction to Analytics and AI
variable), with the x-axis showing different classes/subsets such as different time dimen- sions (e.g., descriptive statistics for annual Medicare expenses in 2015 versus 2016) or different categories (e.g., descriptive statistics for marketing expenses versus total sales).
Although historically speaking, the box plot has not been used widely and often enough (especially in areas outside of statistics), with the emerging popularity of business analytics, it is gaining fame in less technical areas of the business world. Its information richness and ease of understanding are largely to credit for its recent popularity.
The box plot shows the centrality (median and sometimes also mean) as well as the dispersion (the density of the data within the middle half—drawn as a box between the first and third quartiles), the minimum and maximum ranges (shown as extended lines from the box, looking like whiskers, that are calculated as 1.5 times the upper or lower end of the quartile box), and the outliers that are larger than the limits of the whis- kers. A box plot also shows whether the data are symmetrically distributed with respect to the mean or sway one way or another. The relative position of the median versus mean and the lengths of the whiskers on both side of the box give a good indication of the potential skewness in the data.
x
Max
Upper Quartile
Median
Lower Quartile
Min
Outliers
Outliers Larger than 1.5 times the upper quartile
Largest value, excluding larger outliers
25% of data is larger than this value
25% of data is smaller than this value
Smallest value, excluding smaller outliers
Smaller than 1.5 times the lower quartile
Mean
50% of data is larger than this value—middle of dataset
Simple average of the dataset
x
Variable 1 Variable 2
FIGURE 3.8 Understanding the Specifics about Box-and-Whiskers Plots.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 145
Shape of a Distribution
Although not as common as the centrality and dispersion, the shape of the data distribu- tion is also a useful measure for the descriptive statistics. Before delving into the shape of the distribution, we first need to define the distribution itself. Simply put, distribution is the frequency of data points counted and plotted over a small number of class labels or numerical ranges (i.e., bins). In a graphical illustration of distribution, the y-axis shows the frequency (count or percentage), and the x-axis shows the individual classes or bins in a rank-ordered fashion. A very well-known distribution is called normal distribution, which is perfectly symmetric on both sides of the mean and has numerous well-founded mathematical properties that make it a very useful tool for research and practice. As the dispersion of a data set increases, so does the standard deviation, and the shape of the distribution looks wider. A graphic illustration of the relationship between dispersion and distribution shape (in the context of normal distribution) is shown in Figure 3.9.
There are two commonly used measures to calculate the shape characteristics of a distribution: skewness and kurtosis. A histogram (frequency plot) is often used to visually illustrate both skewness and kurtosis.
Skewness is a measure of asymmetry (sway) in a distribution of the data that por- trays a unimodal structure—only one peak exists in the distribution of the data. Because normal distribution is a perfectly symmetric unimodal distribution, it does not have
0 1 2 3212223
(a)
(c)
(d)
(b)
FIGURE 3.9 Relationship between Dispersion and Distribution Shape Properties.
146 Part I • Introduction to Analytics and AI
skewness; that is, its skewness measure (i.e., the value of the coefficient of skewness) is equal to zero. The skewness measure/value can be either positive or negative. If the dis- tribution sways left (i.e., the tail is on the right side and the mean is smaller than median), then it produces a positive skewness measure; if the distribution sways right (i.e., the tail is on the left side and the mean is larger than median), then it produces a negative skew- ness measure. In Figure 3.9, (c) represents a positively skewed distribution whereas (d) represents a negatively skewed distribution. In the same figure, both (a) and (b) represent perfect symmetry and hence zero measure for skewness.
Skewness = S = a ni = 1(xi - x)3
(n - 1)s3
where s is the standard deviation and n is the number of samples. Kurtosis is another measure to use in characterizing the shape of a unimodal dis-
tribution. As opposed to the sway in shape, kurtosis focuses more on characterizing the peak/tall/skinny nature of the distribution. Specifically, kurtosis measures the degree to which a distribution is more or less peaked than a normal distribution. Whereas a posi- tive kurtosis indicates a relatively peaked/tall distribution, a negative kurtosis indicates a relatively flat/short distribution. As a reference point, a normal distribution has a kurtosis of 3. The formula for kurtosis can be written as
Kurtosis = K = a ni = 1(xi - x)4
ns4 -3
Descriptive statistics (as well as inferential statistics) can easily be calculated using com- mercially viable statistical software packages (e.g., SAS, SPSS, Minitab, JMP, Statistica) or free/open source tools (e.g., R). Perhaps the most convenient way to calculate descriptive and some of the inferential statistics is to use Excel. Technology Insights 3.1 describes in detail how to use Microsoft Excel to calculate descriptive statistics.
TECHNOLOGY INSIGHTS 3.1 How to Calculate Descriptive Statistics in Microsoft Excel
Excel, arguably the most popular data analysis tool in the world, can easily be used for descriptive statistics. Although the base configuration of Excel does not seem to have the statistics function readily available for end users, those functions come with the Excel installation and can be acti- vated (turned on) with only a few mouse clicks. Figure 3.10 shows how these statistics functions (as part of the Analysis ToolPak) can be activated in Microsoft Excel 2016.
Once activated, the Analysis ToolPak will appear in the Data menu option under the name of Data Analysis. When you click on Data Analysis in the Analysis group under the Data tab in the Excel menu bar, you will see Descriptive Statistics as one of the options within the list of data analysis tools (see Figure 3.11, steps 1, 2); click on OK, and the Descriptive Statistics dialog box will appear (see the middle of Figure 3.11). In this dialog box, you need to enter the range of the data, which can be one or more numerical columns, along with the preference check boxes, and click OK (see Figure 3.11, steps 3, 4). If the selection includes more than one numeric column, the tool treats each column as a separate data set and provides descriptive statistics for each column separately.
As a simple example, we selected two columns (labeled as Expense and Demand) and executed the Descriptive Statistics option. The bottom section of Figure 3.11 shows the output created by Excel. As can be seen, Excel produced all descriptive statistics that are covered in the previous section and added a few more to the list. In Excel 2016, it is also very easy (a few
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 147
1
2
3
4
FIGURE 3.10 Activating Statistics Function in Excel 2016.
148 Part I • Introduction to Analytics and AI
3
1
4
2
FIGURE 3.11 Obtaining Descriptive Statistics in Excel.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 149
1
2 3
FIGURE 3.12 Creating a Box-and-Whiskers Plot in Excel 2016.
mouse clicks) to create a box-and-whiskers plot. Figure 3.12 shows the simple three-step pro- cess of creating a box-and-whiskers plot in Excel.
Although Analysis ToolPak is a very useful tool in Excel, one should be aware of an im- portant point related to the results that it generates, which have a different behavior than other ordinary Excel functions: Although Excel functions dynamically change as the underlying data in the spreadsheet are changed, the results generated by the Analysis ToolPak do not. For example, if you change the values in either or both of these columns, the Descriptive Statistics results produced by the Analysis ToolPak will stay the same. However, the same is not true for ordinary Excel func- tions. If you were to calculate the mean value of a given column (using “=AVERAGE(A1:A121)”) and then change the values within the data range, the mean value would automatically change. In summary, the results produced by Analysis ToolPak do not have a dynamic link to the underlying data, and if the data change, the analysis needs to be redone using the dialog box.
Successful applications of data analytics cover a wide range of business and organizational settings, addressing problems once thought unsolvable. Application Case 3.3 is an excellent il- lustration of those success stories in which a small municipality administration adopted a data analytics approach to intelligently detect and solve problems by continuously analyzing demand and consumption patterns.
u SECTION 3.5 REVIEW QUESTIONS
1. What is the relationship between statistics and business analytics? 2. What are the main differences between descriptive and inferential statistics? 3. List and briefly define the central tendency measures of descriptive statistics. 4. List and briefly define the dispersion measures of descriptive statistics. 5. What is a box-and-whiskers plot? What types of statistical information does it represent? 6. What are the two most commonly used shape characteristics to describe a data distribution?
150 Part I • Introduction to Analytics and AI
A leaky faucet. A malfunctioning dishwasher. A cracked sprinkler head. These are more than just a headache for a home owner or business to fix. They can be costly, unpredictable, and, unfortunately, hard to pin- point. Through a combination of wireless water meters and a data-analytics-driven, customer-accessible portal, the Town of Cary, North Carolina, is making it much easier to find and fix water loss issues. In the process, the town has gained a big-picture view of water usage critical to planning future water plant expansions and promoting targeted conservation efforts.
When the town of Cary installed wireless meters for 60,000 customers in 2010, it knew the new technology wouldn’t just save money by eliminating manual monthly readings; the town also realized it would get more accurate and timely information about water consumption. The Aquastar wireless system reads meters once an hour—that is 8,760 data points per customer each year instead of 12 monthly readings. The data had tremendous potential if they could be easily consumed.
“Monthly readings are like having a gallon of water’s worth of data. Hourly meter readings are more like an Olympic-size pool of data,” says Karen Mills, finance director for Cary. “SAS helps us man- age the volume of that data nicely.” In fact, the solu- tion enables the town to analyze half a billion data points on water usage and make them available to and easily consumable by all customers.
The ability to visually look at data by house- hold or commercial customer by the hour has led to some very practical applications:
• The town can notify customers of potential leaks within days.
• Customers can set alerts that notify them with- in hours if there is a spike in water usage.
• Customers can track their water usage online, helping them to be more proactive in conserv- ing water.
Through the online portal, one business in the town saw a spike in water consumption on weekends when employees are away. This seemed odd, and the unusual reading helped the company learn that a commercial dishwasher was malfunctioning, running continuously over weekends. Without the wireless
water-meter data and the customer-accessible portal, this problem could have gone unnoticed, continuing to waste water and money.
The town has a much more accurate picture of daily water usage per person, critical for planning future water plant expansions. Perhaps the most interesting perk is that the town was able to verify a hunch that has far-reaching cost ramifications: Cary residents are very economical in their use of water. “We calculate that with modern high-efficiency appli- ances, indoor water use could be as low as 35 gal- lons per person per day. Cary residents average 45 gallons, which is still phenomenally low,” explains town Water Resource Manager Leila Goodwin. Why is this important? The town was spending money to encourage water efficiency—rebates on low-flow toilets or discounts on rain barrels. Now it can take a more targeted approach, helping specific consum- ers understand and manage both their indoor and outdoor water use.
SAS was critical not just for enabling residents to understand their water use but also working behind the scenes to link two disparate databases. “We have a billing database and the meter-reading database. We needed to bring that together and make it presentable,” Mills says.
The town estimates that by just removing the need for manual readings, the Aquastar system will save more than $10 million above the cost of the project. But the analytics component could provide even bigger savings. Already, both the town and individual citizens have saved money by catch- ing water leaks early. As Cary continues to plan its future infrastructure needs, having accurate infor- mation on water usage will help it invest in the right amount of infrastructure at the right time. In addition, understanding water usage will help the town if it experiences something detrimental like a drought.
“We went through a drought in 2007,” says Goodwin. “If we go through another, we have a plan in place to use Aquastar data to see exactly how much water we are using on a day-by-day basis and communicate with customers. We can show ‘here’s what’s happening, and here is how much you can use because our supply is low.’ Hopefully, we’ll never have to use it, but we’re prepared.”
Application Case 3.3 Town of Cary Uses Analytics to Analyze Data from Sensors, Assess Demand, and Detect Problems
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 151
Questions for Case 3.3
1. What were the challenges the Town of Cary was facing?
2. What was the proposed solution? 3. What were the results? 4. What other problems and data analytics solutions
do you foresee for towns like Cary?
Source: “Municipality Puts Wireless Water Meter-Reading Data To Work (SAS' Analytics)—The Town of Cary, North Carolina Uses SAS Analytics to Analyze Data from Wireless Water Meters, Assess Demand, Detect Problems and Engage Customers.” Copyright © 2016 SAS Institute Inc., Cary, NC, USA. Reprinted with permis- sion. All rights reserved.
3.6 REGRESSION MODELING FOR INFERENTIAL STATISTICS
Regression, especially linear regression, is perhaps the most widely known and used analytics technique in statistics. Historically speaking, the roots of regression date back to the 1920s and 1930s, to the earlier work on inherited characteristics of sweet peas by Sir Francis Galton and subsequently by Karl Pearson. Since then, regression has become the statistical technique for characterization of relationships between explanatory (input) variable(s) and response (output) variable(s).
As popular as it is, regression essentially is a relatively simple statistical technique to model the dependence of a variable (response or output variable) on one (or more) explanatory (input) variables. Once identified, this relationship between the variables can be formally represented as a linear/additive function/equation. As is the case with many other modeling techniques, regression aims to capture the functional relationship be- tween and among the characteristics of the real world and describe this relationship with a mathematical model, which can then be used to discover and understand the complexi- ties of reality—explore and explain relationships or forecast future occurrences.
Regression can be used for one of two purposes: hypothesis testing—investigating potential relationships between different variables—and prediction/forecasting— estimating values of a response variable based on one or more explanatory variables. These two uses are not mutually exclusive. The explanatory power of regression is also the foundation of its predictive ability. In hypothesis testing (theory building), regression analysis can reveal the existence/strength and the directions of relationships between a number of explanatory variables (often represented with xi) and the response variable (often represented with y). In prediction, regression identifies additive mathematical relationships (in the form of an equation) between one or more explanatory variables and a response variable. Once deter- mined, this equation can be used to forecast the values of the response variable for a given set of values of the explanatory variables.
CORRELATION VERSUS REGRESSION Because regression analysis originated from cor- relation studies, and because both methods attempt to describe the association between two (or more) variables, these two terms are often confused by professionals and even by scientists. Correlation makes no a priori assumption of whether one variable is de- pendent on the other(s) and is not concerned with the relationship between variables; instead it gives an estimate on the degree of association between the variables. On the other hand, regression attempts to describe the dependence of a response variable on one (or more) explanatory variables where it implicitly assumes that there is a one- way causal effect from the explanatory variable(s) to the response variable, regardless of whether the path of effect is direct or indirect. Also, although correlation is interested in the low-level relationships between two variables, regression is concerned with the rela- tionships between all explanatory variables and the response variable.
152 Part I • Introduction to Analytics and AI
SIMPLE VERSUS MULTIPLE REGRESSION If the regression equation is built between one response variable and one explanatory variable, then it is called simple regression. For instance, the regression equation built to predict/explain the relationship between the height of a person (explanatory variable) and the weight of a person (response variable) is a good example of simple regression. Multiple regression is the extension of simple regression when the explanatory variables are more than one. For instance, in the pre- vious example, if we were to include not only the height of the person but also other personal characteristics (e.g., BMI, gender, ethnicity) to predict the person’s weight, then we would be performing multiple regression analysis. In both cases, the relationship between the response variable and the explanatory variable(s) is linear and additive in nature. If the relationships are not linear, then we might want to use one of many other nonlinear regression methods to better capture the relationships between the input and output variables.
How Do We Develop the Linear Regression Model?
To understand the relationship between two variables, the simplest thing that one can do is to draw a scatter plot where the y-axis represents the values of the response variable and the x-axis represents the values of the explanatory variable (see Figure 3.13). A scat- ter plot would show the changes in the response variable as a function of the changes in the explanatory variable. In the case shown in Figure 3.13, there seems to be a positive relationship between the two; as the explanatory variable values increase, so does the response variable.
Simple regression analysis aims to find a mathematical representation of this rela- tionship. In reality, it tries to find the signature of a straight line passing through right between the plotted dots (representing the observation/historical data) in such a way that it minimizes the distance between the dots and the line (the predicted values on the
R e s p o n s e V
a ri a b le
: y
Explanatory Variable: x
b0
b1
(xi, yi)
(xi, yi)
(xi, yi)
Regression Line
FIGURE 3.13 A Scatter Plot and a Linear Regression Line.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 153
theoretical regression line). Even though there are several methods/algorithms proposed to identify the regression line, the one that is most commonly used is called the ordinary least squares (OLS) method. The OLS method aims to minimize the sum of squared residuals (squared vertical distances between the observation and the regression point) and leads to a mathematical expression for the estimated value of the regression line (which are known as b parameters). For simple linear regression, the aforementioned relationship between the response variable 1y2 and the explanatory variable(s) 1x2 can be shown as a simple equation as follows:
y = b0 + b1x
In this equation, b0 is called the intercept, and b1 is called the slope. Once OLS deter- mines the values of these two coefficients, the simple equation can be used to forecast the values of y for given values of x. The sign and the value of b1 also reveal the direc- tion and the strengths of relationship between the two variables.
If the model is of a multiple linear regression type, then there would be more coef- ficients to be determined, one for each additional explanatory variable. As the following formula shows, the additional explanatory variable would be multiplied with the new bi coefficients and summed together to establish a linear additive representation of the response variable.
y = b0 + b1x1 + b2x2 + b3x3 + # + bnxn
How Do We Know If the Model Is Good Enough?
Because of a variety of reasons, sometimes models as representations of the reality do not prove to be good. Regardless of the number of explanatory variables included, there is always a possibility of not having a good model, and therefore the linear regression model needs to be assessed for its fit (the degree to which it represents the response variable). In the simplest sense, a well-fitting regression model results in predicted values close to the observed data values. For the numerical assessment, three statistical measures are often used in evaluating the fit of a regression model: R 2(R - squared), the overall F-test, and the root mean square error (RMSE). All three of these measures are based on the sums of the square errors (how far the data are from the mean and how far the data are from the model’s predicted values). Different combinations of these two values pro- vide different information about how the regression model compares to the mean model.
Of the three, R 2 has the most useful and understandable meaning because of its intuitive scale. The value of R 2 ranges from 0 to 1 (corresponding to the amount of vari- ability explained in percentage) with 0 indicating that the relationship and the prediction power of the proposed model is not good, and 1 indicating that the proposed model is a perfect fit that produces exact predictions (which is almost never the case). The good R 2 values would usually come close to one, and the closeness is a matter of the phe- nomenon being modeled—whereas an R 2 value of 0.3 for a linear regression model in social sciences can be considered good enough, an R 2 value of 0.7 in engineering might be considered as not a good enough fit. The improvement in the regression model can be achieved by adding more explanatory variables or using different data transforma- tion techniques, which would result in comparative increases in an R 2 value. Figure 3.14 shows the process flow of developing regression models. As can be seen in the process flow, the model development task is followed by the model assessment task in which not only is the fit of the model assessed, but because of restrictive assumptions with which the linear models have to comply, the validity of the model also needs to be put under the microscope.
154 Part I • Introduction to Analytics and AI
What Are the Most Important Assumptions in Linear Regression?
Even though they are still the choice of many for data analyses (both for explanatory and for predictive modeling purposes), linear regression models suffer from several highly restrictive assumptions. The validity of the linear model built depends on its ability to comply with these assumptions. Here are the most commonly pronounced assumptions:
1. Linearity. This assumption states that the relationship between the response variable and the explanatory variables is linear. That is, the expected value of the response variable is a straight-line function of each explanatory variable while holding all other explanatory variables fixed. Also, the slope of the line does not depend on the values of the other variables. It also implies that the effects of dif- ferent explanatory variables on the expected value of the response variable are additive in nature.
2. Independence (of errors). This assumption states that the errors of the response variable are uncorrelated with each other. This independence of the errors is weaker
Tabulated Data
Data Assessment
Scatter plot
Correlations
Model Fitting
Transform data
Estimate parameters
Model Assessment
Test assumptions
Assess model fit
Deployment
One-time use
Recurrent use
FIGURE 3.14 A Process Flow for Developing Regression Models.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 155
than actual statistical independence, which is a stronger condition and is often not needed for linear regression analysis.
3. Normality (of errors). This assumption states that the errors of the response vari- able are normally distributed. That is, they are supposed to be totally random and should not represent any nonrandom patterns.
4. Constant variance (of errors). This assumption, also called homoscedasticity, states that the response variables have the same variance in their error regardless of the values of the explanatory variables. In practice, this assumption is invalid if the response variable varies over a wide enough range/scale.
5. Multicollinearity. This assumption states that the explanatory variables are not correlated (i.e., do not replicate the same but provide a different perspective of the information needed for the model). Multicollinearity can be triggered by having two or more perfectly correlated explanatory variables presented to the model (e.g., if the same explanatory variable is mistakenly included in the model twice, one with a slight transformation of the same variable). A correlation-based data assessment usually catches this error.
There are statistical techniques developed to identify the violation of these assump- tions and techniques to mitigate them. The most important part for a modeler is to be aware of their existence and to put in place the means to assess the models to make sure that they are compliant with the assumptions they are built on.
Logistic Regression
Logistic regression is a very popular, statistically sound, probability-based classifica- tion algorithm that employs supervised learning. It was developed in the 1940s as a complement to linear regression and linear discriminant analysis methods. It has been used extensively in numerous disciplines, including the medical and social sciences fields. Logistic regression is similar to linear regression in that it also aims to regress to a mathematical function that explains the relationship between the response vari- able and the explanatory variables using a sample of past observations (training data). Logistic regression differs from linear regression with one major point: its output (re- sponse variable) is a class as opposed to a numerical variable. That is, whereas linear regression is used to estimate a continuous numerical variable, logistic regression is used to classify a categorical variable. Even though the original form of logistic regres- sion was developed for a binary output variable (e.g., 1/0, yes/no, pass/fail, accept/ reject), the present-day modified version is capable of predicting multiclass output variables (i.e., multinomial logistic regression). If there is only one predictor variable and one predicted variable, the method is called simple logistic regression (similar to calling linear regression models with only one independent variable simple linear regression).
In predictive analytics, logistic regression models are used to develop probabilis- tic models between one or more explanatory/predictor variables (which can be a mix of both continuous and categorical in nature) and a class/response variable (which can be binomial/binary or multinomial/multiclass). Unlike ordinary linear regression, logis- tic regression is used for predicting categorical (often binary) outcomes of the response variable—treating the response variable as the outcome of a Bernoulli trial. Therefore, logistic regression takes the natural logarithm of the odds of the response variable to create a continuous criterion as a transformed version of the response variable. Thus, the logit transformation is referred to as the link function in logistic regression—even though the response variable in logistic regression is categorical or binomial, the logit is the con- tinuous criterion on which linear regression is conducted. Figure 3.15 shows a logistic
156 Part I • Introduction to Analytics and AI
regression function where the odds are represented in the x-axis (a linear function of the independent variables), whereas the probabilistic outcome is shown in the y-axis (i.e., response variable values change between 0 and 1).
The logistic function, f1y2 in Figure 3.15 is the core of logistic regression, which can take values only between 0 and 1. The following equation is a simple mathematical representation of this function:
f1y2 = 1 1 + e-1b0 +b1x 2
The logistic regression coefficients (the bs) are usually estimated using the maximum likelihood estimation method. Unlike linear regression with normally distributed residu- als, it is not possible to find a closed-form expression for the coefficient values that maxi- mizes the likelihood function, so an iterative process must be used instead. This process begins with a tentative starting solution, then revises the parameters slightly to see if the solution can be improved, and repeats this iterative revision until no improvement can be achieved or is very minimal, at which point the process is said to have completed/ converged.
Sports analytics—use of data and statistical/analytics techniques to better manage sports teams/organizations—has been gaining tremendous popularity. Use of data-driven analytics techniques has become mainstream for not only professional teams but also col- lege and amateur sports. Application Case 3.4 is an example of how existing and readily available public data sources can be used to predict college football bowl game outcomes using both classification and regression-type prediction models.
Time-Series Forecasting
Sometimes the variable that we are interested in (i.e., the response variable) might not have distinctly identifiable explanatory variables, or there might be too many of them in a highly complex relationship. In such cases, if the data are available in a desired format, a prediction model, the so-called time series, can be developed. A time series is a sequence of data points of the variable of interest, measured and represented at successive points in time spaced at uniform time intervals. Examples of time series include monthly rain volumes in a geographic area, the daily closing value of the stock market indexes, and
f (y) 1
26 24 22 0 2 4 6
b0 1 b1x
0.5
FIGURE 3.15 The Logistic Function.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 157
Predicting the outcome of a college football game (or any sports game, for that matter) is an interesting and challenging problem. Therefore, challenge-seeking researchers from both academics and industry have spent a great deal of effort on forecasting the out- come of sporting events. Large amounts of historic data exist in different media outlets (often publicly available) regarding the structure and outcomes of sporting events in the form of a variety of numeri- cally or symbolically represented factors that are assumed to contribute to those outcomes.
The end-of-season bowl games are very impor- tant to colleges in terms of both finance (bring- ing in millions of dollars of additional revenue) and reputation—for recruiting quality students and highly regarded high school athletes for their athletic pro- grams (Freeman & Brewer, 2016). Teams that are selected to compete in a given bowl game split a purse, the size of which depends on the specific bowl (some bowls are more prestigious and have higher payouts for the two teams), and therefore securing an invitation to a bowl game is the main goal of any division I-A college football program. The decision makers of the bowl games are given the authority to select and invite bowl-eligible (a team that has six
wins against its Division I-A opponents in that season) successful teams (as per the ratings and rankings) that will play in an exciting and competitive game, attract fans of both schools, and keep the remaining fans tuned in via a variety of media outlets for advertising.
In a recent data mining study, Delen et al. (2012) used eight years of bowl game data along with three popular data mining techniques (decision trees, neural networks, and support vector machines) to predict both the classification-type outcome of a game (win versus loss) and the regression-type out- come (projected point difference between the scores of the two opponents). What follows is a shorthand description of their study.
The Methodology
In this research, Delen and his colleagues followed a popular data mining methodology, CRISP-DM (Cross-Industry Standard Process for Data Mining), which is a six-step process. This popular meth- odology, which is covered in detail in Chapter 4, provided them with a systematic and structured way to conduct the underlying data mining study and hence improved the likelihood of obtaining accurate
Application Case 3.4 Predicting NCAA Bowl Game Outcomes
(Continued )
158 Part I • Introduction to Analytics and AI
and reliable results. To objectively assess the pre- diction power of the different model types, they used a cross-validation methodology k-fold cross- validation. Details on k-fold cross-validation can be found in Chapter 4. Figure 3.16 graphically illustrates the methodology employed by the researchers.
Data Acquisition and Data Preprocessing
The sample data for this study are collected from a variety of sports databases available on the Web,
including jhowel.net, ESPN.com, Covers.com, ncaa.org, and rauzulusstreet.com. The data set included 244 bowl games representing a com- plete set of eight seasons of college football bowl games played between 2002 and 2009. Delen et al. also included an out-of-sample data set (2010– 2011 bowl games) for additional validation pur- poses. Exercising one of the popular data mining rules of thumb, they included as much relevant information in the model as possible. Therefore, after an in-depth variable identification and
Classification & Regression Trees
Neural Networks
X1
X2
Support Vector Machines
M ax
im um
-m ar
gin h yp
er pla
ne
M argin
Data Collection, Organization, Cleaning, and Transformation
Raw Data Sources
Built Classification
Models
Test Model
Tabulate the Results
Built Regression
Models
Transform and Tabulate Results
Compare the Prediction Results
Test Model
Classification Modeling
Regression Modeling
DBs
Output: Binary (win/loss) Output: Integer (point difference)
Win Loss
Win
Loss
...
......
...
10 %
10 %
10 %
10 % 10 %
10 %
10 %
10 %
10 % 10 % 10 %
10 %
10 %
10 % 10 %
10 %
10 %
10 %
10 % 10 %
FIGURE 3.16 The Graphical Illustration of the Methodology Employed in the Study.
Application Case 3.4 (Continued)
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 159
collection process, they ended up with a data set that included 36 variables, of which the first 6 were the identifying variables (i.e., name and the year of the bowl game, home and away team names, and their athletic conferences—see variables 1–6 in
Table 3.5), followed by 28 input variables (which included variables delineating a team’s seasonal sta- tistics on offense and defense, game outcomes, team composition characteristics, athletic conference char- acteristics, and how they fared against the odds—see
TABLE 3.5 Description of Variables Used in the Study
No Cat Variable Name Description
1 ID1 YEAR Year of the bowl game
2 ID BOWLGAME Name of the bowl game
3 ID HOMETEAM Home team (as listed by the bowl organizers)
4 ID AWAYTEAM Away team (as listed by the bowl organizers)
5 ID HOMECONFERENCE Conference of the home team
6 ID AWAYCONFERENCE Conference of the away team
7 I12 DEFPTPGM Defensive points per game
8 I1 DEFRYDPGM Defensive rush yards per game
9 I1 DEFYDPGM Defensive yards per game
10 I1 PPG Average number of points a given team scored per game
11 I1 PYDPGM Average total pass yards per game
12 I1 RYDPGM Team’s average total rush yards per game
13 I1 YRDPGM Average total offensive yards per game
14 I2 HMWIN% Home winning percentage
15 I2 LAST7 How many games the team won out of their last 7 games
16 I2 MARGOVIC Average margin of victory
17 I2 NCTW Nonconference team winning percentage
18 I2 PREVAPP Did the team appear in a bowl game previous year
19 I2 RDWIN% Road winning percentage
20 I2 SEASTW Winning percentage for the year
21 I2 TOP25 Winning percentage against AP top 25 teams for the year
22 I3 TSOS Strength of schedule for the year
23 I3 FR% Percentage of games played by freshmen class players for the year
24 I3 SO% Percentage of games played by sophomore class players for the year
25 I3 JR% Percentage of games played by junior class players for the year
26 I3 SR% Percentage of games played by senior class players for the year
27 I4 SEASOvUn% Percentage of times a team went over the O/U3 in the current season
28 I4 ATSCOV% Against the spread cover percentage of the team in previous bowl games
(Continued )
160 Part I • Introduction to Analytics and AI
TABLE 3.5 (Continued)
No Cat Variable Name Description
29 I4 UNDER% Percentage of times a team went under in previous bowl games
30 I4 OVER% Percentage of times a team went over in previous bowl games
31 I4 SEASATS% Percentage of covering against the spread for the current season
32 I5 CONCH Did the team win their respective conference championship game
33 I5 CONFSOS Conference strength of schedule
34 I5 CONFWIN% Conference winning percentage
35 O1 ScoreDiff4 Score difference (HomeTeamScore – AwayTeamScore)
36 O2 WinLoss4 Whether the home team wins or loses the game
1ID: Identifier variables; O1: output variable for regression models; O2: output variable for classification models. 2Offense/defense; I2: game outcome; I3: team configuration; I4: against the odds; I5: conference stats. 3Over/Under—Whether or not a team will go over or under the expected score difference. 4Output variables—ScoreDiff for regression models and WinLoss for binary classification models.
variables 7–34 in Table 3.5), and finally the last two were the output variables (i.e., ScoreDiff—the score difference between the home team and the away team represented with an integer number—and WinLoss—whether the home team won or lost the bowl game represented with a nominal label).
In the formulation of the data set, each row (a.k.a. tuple, case, sample, example, etc.) represented a bowl game, and each column stood for a variable (i.e., identifier/input or output type). To represent the game-related comparative characteristics of the two opponent teams in the input variables, Delen et al. calculated and used the differences between the measures of the home and away teams. All these variable values are calculated from the home team’s perspective. For instance, the variable PPG (average number of points a team scored per game) repre- sents the difference between the home team’s PPG and away team’s PPG. The output variables repre- sent whether the home team wins or loses the bowl game. That is, if the ScoreDiff variable takes a posi- tive integer number, then the home team is expected to win the game by that margin; otherwise (if the ScoreDiff variable takes a negative integer number), the home team is expected to lose the game by that margin. In the case of WinLoss, the value of the out- put variable is a binary label, “Win” or “Loss,” indi- cating the outcome of the game for the home team.
The Results and Evaluation
In this study, three popular prediction techniques are used to build models (and to compare them to each other): artificial neural networks, decision trees, and support vector machines. These prediction techniques are selected based on their capability of modeling both classification and regression-type prediction problems and their popularity in recently published data mining literature. More details about these popular data min- ing methods can be found in Chapter 4.
To compare predictive accuracy of all models to one another, the researchers used a stratified k-fold cross-validation methodology. In a stratified version of k-fold cross-validation, the folds are created in a way that they contain approximately the same proportion of predictor labels (i.e., classes) as the original data set. In this study, the value of k is set to 10 (i.e., the com- plete set of 244 samples are split into 10 subsets, each having about 25 samples), which is a common prac- tice in predictive data mining applications. A graphical depiction of the 10-fold cross-validations was shown earlier in this chapter. To compare the prediction mod- els that were developed using the aforementioned three data mining techniques, the researchers chose to use three common performance criteria: accuracy, sen- sitivity, and specificity. The simple formulas for these metrics were also explained earlier in this chapter.
Application Case 3.4 (Continued)
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 161
The prediction results of the three model- ing techniques are presented in Tables 3.6 and 3.7. Table 3.6 presents the 10-fold cross-validation results of the classification methodology in which the three data mining techniques are formulated to have a binary-nominal output variable (i.e., WinLoss). Table 3.7 presents the 10-fold cross- validation results of the regression-based classifica- tion methodology in which the three data mining techniques are formulated to have a numerical out- put variable (i.e., ScoreDiff). In the regression-based classification prediction, the numerical output of the models is converted to a classification type by label- ing the positive WinLoss numbers with a “Win” and
negative WinLoss numbers with a “Loss” and then tabulating them in the confusion matrixes. Using the confusion matrices, the overall prediction accuracy, sensitivity, and specificity of each model type are calculated and presented in Tables 3.6 and 3.7. As the results indicate, the classification-type prediction methods performed better than regression-based classification-type prediction methodology. Among the three data mining technologies, classification and regression trees produced better prediction accuracy in both prediction methodologies. Overall, classification and regression tree classification mod- els produced a 10-fold cross-validation accuracy of 86.48 percent followed by support vector machines
TABLE 3.6 Prediction Results for the Direct Classification Methodology
Prediction Method (classification1)
Confusion Matrix
Accuracy2 (in %)
Sensitivity (in %)
Specificity (in %)
Win Loss
ANN (MLP) Win 92 42 75.00 68.66 82.73
Loss 19 91
SVM (RBF) Win 105 29 79.51 78.36 80.91
Loss 21 89
DT (C&RT) Win 113 21 86.48 84.33 89.09
Loss 12 98
1The output variable is a binary categorical variable (Win or Loss). 2Differences were significant.
TABLE 3.7 Prediction Results for the Regression-Based Classification Methodology
Prediction Method (regression based1)
Confusion Matrix
Accuracy2
Sensitivity
Specificity
Win Loss
ANN (MLP) Win 94 40 72.54 70.15 75.45
Loss 27 83
SVM (RBF) Win 100 34 74.59 74.63 74.55
Loss 28 82
DT (C&RT) Win 106 28 77.87 76.36 79.10
Loss 26 84
1The output variable is a numerical/integer variable (point-diff). 2Differences were sig p 6 0.01.
(Continued )
162 Part I • Introduction to Analytics and AI
daily sales totals for a grocery store. Often, time series are visualized using a line chart. Figure 3.17 shows an example time series of sales volumes for the years 2008 through 2012 on a quarterly basis.
Time-series forecasting is the use of mathematical modeling to predict future values of the variable of interest based on previously observed values. The time-series plots/charts look and feel very similar to simple linear regression in that, as was the case in simple linear regression, in time series there are two variables: the response variable and the time variable presented in a scatter plot. Beyond this appearance similarity, there is hardly any other commonality between the two. Although regression analysis is often employed in testing theories to see if current values of one or more explanatory variables explain (and hence predict) the response variable, the time-series models are focused on extrapolating on their time-varying behavior to estimate the future values.
Time-series-forecasting assumes that all of the explanatory variables are aggregated into the response variable as a time-variant behavior. Therefore, capturing the time- variant behavior is the way to predict the future values of the response variable. To do that, the pattern is analyzed and decomposed into its main components: random variations, time trends, and seasonal cycles. The time-series example shown in Figure 3.17 illustrates all of these distinct patterns.
The techniques used to develop time-series forecasts range from very simple (the naïve forecast that suggests today’s forecast is the same as yesterday’s actual) to very complex like ARIMA (a method that combines autoregressive and moving average pat- terns in data). Most popular techniques are perhaps the averaging methods that include simple average, moving average, weighted moving average, and exponential smoothing. Many of these techniques also have advanced versions when seasonality and trend can also be taken into account for better and more accurate forecasting. The accuracy of a method is usually assessed by computing its error (calculated deviation between actuals and forecasts for the past observations) via mean absolute error (MAE), mean squared error (MSE), or mean absolute percent error (MAPE). Even though they all use the same
(with a 10-fold cross-validation accuracy of 79.51 percent) and neural networks (with a 10-fold cross- validation accuracy of 75.00 percent). Using a t-test, researchers found that these accuracy values were significantly different at 0.05 alpha level; that is, the decision tree is a significantly better predictor of this domain than the neural network and support vec- tor machine, and the support vector machine is a significantly better predictor than neural networks.
The results of the study showed that the classification-type models predict the game out- comes better than regression-based classification models. Even though these results are specific to the application domain and the data used in this study and therefore should not be generalized beyond the scope of the study, they are exciting because deci- sion trees are not only the best predictors but also the best in understanding and deployment, com- pared to the other two machine-learning techniques
employed in this study. More details about this study can be found in Delen et al. (2012).
Questions for Case 3.4
1. What are the foreseeable challenges in predicting sporting event outcomes (e.g., college bowl games)?
2. How did the researchers formulate/design the prediction problem (i.e., what were the inputs and output, and what was the representation of a single sample—row of data)?
3. How successful were the prediction results? What else can they do to improve the accuracy?
Sources: D. Delen, D. Cogdell, and N. Kasap, “A Comparative Analysis of Data Mining Methods in Predicting NCAA Bowl Outcomes,” International Journal of Forecasting, 28, 2012, pp. 543–552; K. M. Freeman, and R. M. Brewer, “The Politics of American College Football,” Journal of Applied Business and Economics, 18(2), 2016, pp. 97–101.
Application Case 3.4 (Continued)
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 163
core error measure, these three assessment methods emphasize different aspects of the error, some penalizing larger errors more so than the others.
u SECTION 3.6 REVIEW QUESTIONS
1. What is regression, and what statistical purpose does it serve? 2. What are the commonalities and differences between regression and correlation? 3. What is OLS? How does OLS determine the linear regression line? 4. List and describe the main steps to follow in developing a linear repression model. 5. What are the most commonly pronounced assumptions for linear regression? 6. What is logistics regression? How does it differ from linear regression? 7. What is time series? What are the main forecasting techniques for time-series data?
3.7 BUSINESS REPORTING
Decision makers need information to make accurate and timely decisions. Information is essentially the contextualization of data. In addition to statistical means that were ex- plained in the previous section, information (descriptive analytics) can also be obtained using OLTP systems (see the simple taxonomy of descriptive analytics in Figure 3.7). The information is usually provided to decision makers in the form of a written report (digital or on paper), although it can also be provided orally. Simply put, a report is any com- munication artifact prepared with the specific intention of conveying information in a di- gestible form to whoever needs it whenever and wherever. It is typically a document that contains information (usually driven from data) organized in a narrative, graphic, and/or tabular form, prepared periodically (recurring) or on an as-needed (ad hoc) basis, refer- ring to specific time periods, events, occurrences, or subjects. Business reports can fulfill many different (but often related) functions. Here are a few of the most prevailing ones:
• To ensure that all departments are functioning properly. • To provide information.
0
1
2
3
4
5
6
7
8
9
10
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
2008 2009 2010 2011 2012
Quarterly Product Sales (in millions)
FIGURE 3.17 A Sample Time Series of Data on Quarterly Sales Volumes.
164 Part I • Introduction to Analytics and AI
• To provide the results of an analysis. • To persuade others to act. • To create an organizational memory (as part of a knowledge management
system).
Business reporting (also called OLAP or BI) is an essential part of the larger drive to- ward improved, evidence-based, optimal managerial decision making. The foundation of these business reports is various sources of data coming from both inside and outside the organization (OLTP systems). Creation of these reports involves extract, transform, and load (ETL) procedures in coordination with a data warehouse and then using one or more reporting tools.
Due to the rapid expansion of IT coupled with the need for improved competitive- ness in business, there has been an increase in the use of computing power to produce unified reports that join different views of the enterprise in one place. Usually, this report- ing process involves querying structured data sources, most of which were created using different logical data models and data dictionaries, to produce a human-readable, easily digestible report. These types of business reports allow managers and coworkers to stay informed and involved, review options and alternatives, and make informed decisions. Figure 3.18 shows the continuous cycle of data acquisition S information generation S decision-making S business process management. Perhaps the most critical task in this cyclical process is the reporting (i.e., information generation)—converting data from dif- ferent sources into actionable information.
Key to any successful report are clarity, brevity, completeness, and correctness. The nature of the report and the level of importance of these success factors changes significantly based on for whom the report is created. Most of the research in effective reporting is dedicated to internal reports that inform stakeholders and decision makers within the organization. There are also external reports between businesses and the government (e.g., for tax purposes or for regular filings to the Securities and Exchange Commission). Even though there is a wide variety of business reports, the ones that are often used for managerial purposes can be grouped into three major categories (Hill, 2016).
Data Repositories
Business Functions
UOB 1.0 X
UOB 2.2
UOB 2.1 X UOB 3.0
Symbol Count Description
1 Machine Failure
Exception Event
Transactional Records
Information (reporting)
Decision Maker
Action (decision)
Data
1 2 3 4
5
FIGURE 3.18 The Role of Information Reporting in Managerial Decision Making.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 165
METRIC MANAGEMENT REPORTS In many organizations, business performance is man- aged through outcome-oriented metrics. For external groups, these are service-level agreements. For internal management, they are key performance indicators (KPIs). Typically, there are enterprise-wide agreed upon targets to be tracked against over a pe- riod of time. They can be used as part of other management strategies such as Six Sigma or total quality management.
DASHBOARD-TYPE REPORTS A popular idea in business reporting in recent years has been to present a range of different performance indicators on one page like a dashboard in a car. Typically, dashboard vendors would provide a set of predefined reports with static elements and fixed structure but also allow for customization of the dashboard wid- gets, views, and set targets for various metrics. It is common to have color-coded traffic lights defined for performance (red, orange, green) to draw management’s attention to particular areas. A more detailed description of dashboards can be found in a later section of this chapter.
BALANCED SCORECARD–TYPE REPORTS This is a method developed by Kaplan and Norton that attempts to present an integrated view of success in an organization. In addition to financial performance, balanced scorecard–type reports also include cus- tomer, business process, and learning and growth perspectives. More details on balanced scorecards are provided in a later section in this chapter.
Application Case 3.5 is an example to illustrate the power and the utility of auto- mated report generation for a large (and, at a time of natural crisis, somewhat chaotic) organization such as the Federal Emergency Management Agency.
Staff at the Federal Emergency Management Agency (FEMA), the U.S. federal agency that coordinates disaster response when the president declares a national disaster, always got two floods at once. First, water covered the land. Next, a flood of paper required to administer the National Flood Insurance Program (NFIP) covered their desks— pallets and pallets of green-striped reports poured off a mainframe printer and into their offices. Individual reports were sometimes 18 inches thick with a nugget of information about insurance claims, premiums, or payments buried in them somewhere.
Bill Barton and Mike Miles do not claim to be able to do anything about the weather, but the project manager and computer scientist, respec- tively, from Computer Sciences Corporation (CSC) have used WebFOCUS software from Information Builders to turn back the flood of paper generated
by the NFIP. The program allows the government to work with national insurance companies to col- lect flood insurance premiums and pay claims for flooding in communities that adopt flood control measures. As a result of CSC’s work, FEMA staffs no longer leaf through paper reports to find the data they need. Instead, they browse insurance data posted on NFIP’s BureauNet intranet site, select just the information they want to see, and get an on-screen report or download the data as a spread- sheet. And that is only the start of the savings that WebFOCUS has provided. The number of times that NFIP staff ask CSC for special reports has dropped in half because NFIP staff can generate many of the special reports they need without calling on a pro- grammer to develop them. Then there is the cost of creating BureauNet in the first place. Barton esti- mates that using conventional Web and database software to export data from FEMA’s mainframe,
Application Case 3.5 Flood of Paper Ends at FEMA
(Continued )
166 Part I • Introduction to Analytics and AI
u SECTION 3.7 REVIEW QUESTIONS
1. What is a report? What are reports used for? 2. What is a business report? What are the main characteristics of a good business report? 3. Describe the cyclic process of management, and comment on the role of business
reports.
4. List and describe the three major categories of business reports. 5. What are the main components of a business reporting system?
3.8 DATA VISUALIZATION
Data visualization (or more appropriately, information visualization) has been defined as “the use of visual representations to explore, make sense of, and communicate data” (Few, 2007). Although the name that is commonly used is data visualization, usually what this means is information visualization. Because information is the aggregation, summarization, and contextualization of data (raw facts), what is portrayed in visualiza- tions is the information, not the data. However, because the two terms data visualiza- tion and information visualization are used interchangeably and synonymously, in this chapter we will follow suit.
Data visualization is closely related to the fields of information graphics, information visualization, scientific visualization, and statistical graphics. Until recently, the major forms of data visualization available in both BI applications have included charts and graphs as well as the other types of visual elements used to create scorecards and dashboards.
To better understand the current and future trends in the field of data visualization, it helps to begin with some historical context.
store it in a new database, and link that to a Web server would have cost about 100 times as much— more than $500,000—and taken about two years to complete compared with the few months Miles spent on the WebFOCUS solution.
When Tropical Storm Allison, a huge slug of sodden, swirling cloud, moved out of the Gulf of Mexico onto the Texas and Louisiana coastline in June 2001, it killed 34 people, most from drowning; damaged or destroyed 16,000 homes and businesses; and displaced more than 10,000 families. President George W. Bush declared 28 Texas counties disaster areas, and FEMA moved in to help. This was the first serious test for BureauNet, and it delivered. This first comprehensive use of BureauNet resulted in FEMA field staff readily accessing what they needed when they needed it and asking for many new types of reports. Fortunately, Miles and WebFOCUS were up to the task. In some cases, Barton says, “FEMA would ask for a new type of report one day, and Miles would have it on BureauNet the next day,
thanks to the speed with which he could create new reports in WebFOCUS.”
The sudden demand on the system had little impact on its performance, noted Barton. “It han- dled the demand just fine,” he says. “We had no problems with it at all. And it made a huge differ- ence to FEMA and the job they had to do. They had never had that level of access before, never had been able to just click on their desktop and generate such detailed and specific reports.”
Questions for Case 3.5
1. What is FEMA, and what does it do?
2. What are the main challenges that FEMA faces?
3. How did FEMA improve its inefficient reporting practices?
Source: Used with permission from Information Builders. Useful information flows at disaster response agency. informationbuild- ers.com/applications/fema (accessed July 2018); and fema.gov.
Application Case 3.5 (Continued)
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 167
Brief History of Data Visualization
Despite the fact that predecessors to data visualization date back to the second century AD, most developments have occurred in the last two and a half centuries, predominantly during the last 30 years (Few, 2007). Although visualization has not been widely recognized as a discipline until fairly recently, today’s most popular visual forms date back a few centuries. Geographical exploration, mathematics, and popularized history spurred the creation of early maps, graphs, and timelines as far back as the 1600s, but William Playfair is widely credited as the inventor of the modern chart, having created the first widely distributed line and bar charts in his Commercial and Political Atlas of 1786 and what is generally considered to be the first time-series line chart in his Statistical Breviary published in 1801 (see Figure 3.19).
Perhaps the most notable innovator of information graphics during this period was Charles Joseph Minard, who graphically portrayed the losses suffered by Napoleon’s army in the Russian campaign of 1812 (see Figure 3.20). Beginning at the Polish–Russian border, the thick band shows the size of the army at each position. The path of Napoleon’s retreat from Moscow in the bitterly cold winter is depicted by the dark lower band, which is tied to tem- perature and time scales. Popular visualization expert, author, and critic Edward Tufte says that this “may well be the best statistical graphic ever drawn.” In this graphic, Minard man- aged to simultaneously represent several data dimensions (the size of the army, direction of movement, geographic locations, outside temperature, etc.) in an artistic and informative
FIGURE 3.19 The First Time-Series Line Chart Created by William PlayFair in 1801.
168 Part I • Introduction to Analytics and AI
manner. Many more excellent visualizations were created in the 1800s, and most of them are chronicled on Tufte’s Web site (edwardtufte.com) and his visualization books.
The 1900s saw the rise of a more formal, empirical attitude toward visualization, which tended to focus on aspects such as color, value scales, and labeling. In the mid- 1900s, cartographer and theorist Jacques Bertin published his Semiologie Graphique, which some say serves as the theoretical foundation of modern information visualization. Although most of his patterns are either outdated by more recent research or completely inapplicable to digital media, many are still very relevant.
In the 2000s, the Internet emerged as a new medium for visualization and brought with it many new tricks and capabilities. Not only has the worldwide, digital distribution of both data and visualization made them more accessible to a broader audience (raising visual literacy along the way), but also it has spurred the design of new forms that incorporate interaction, animation, and graphics-rendering technology unique to screen media and real- time data feeds to create immersive environments for communicating and consuming data.
Companies and individuals are, seemingly all of a sudden, interested in data; that in- terest has in turn sparked a need for visual tools that help them understand it. Cheap hard- ware sensors and do-it-yourself frameworks for building your own system are driving down the costs of collecting and processing data. Countless other applications, software tools, and low-level code libraries are springing up to help people collect, organize, manipulate, visualize, and understand data from practically any source. The Internet has also served as a fantastic distribution channel for visualizations; a diverse community of designers, program- mers, cartographers, tinkerers, and data wonks has assembled to disseminate all sorts of new ideas and tools for working with data in both visual and nonvisual forms.
Google Maps has also single-handedly democratized both the interface conven- tions (click to pan, double-click to zoom) and the technology (256-pixel square map tiles with predictable file names) for displaying interactive geography online to the extent that most people just know what to do when they are presented with a map online. Flash has served well as a cross-browser platform on which to design and develop rich, beautiful Internet applications incorporating interactive data visualization and maps; now, new
FIGURE 3.20 Decimation of Napoleon’s Army during the 1812 Russian Campaign.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 169
browser-native technologies such as canvas and SVG (sometimes collectively included under the umbrella of HTML5) are emerging to challenge Flash’s supremacy and extend the reach of dynamic visualization interfaces to mobile devices.
The future of data/information visualization is very hard to predict. We can only extrapolate from what has already been invented: more three-dimensional visualization, immersive experience with multidimensional data in a virtual reality environment, and holographic visualization of information. There is a pretty good chance that we will see something that we have never seen in the information visualization realm invented be- fore the end of this decade. Application Case 3.6 shows how visual analytics/reporting tools such as Tableau can help facilitate effective and efficient decision making through information/insight creation and sharing.
The Background
Macfarlan Smith has earned its place in medical history. The company held a royal appointment to provide medicine to Her Majesty Queen Victoria and supplied groundbreaking obstetrician Sir James Simpson with chloroform for his experiments in pain relief during labor and delivery. Today, Macfarlan Smith is a sub- sidiary of the Fine Chemical and Catalysts division of Johnson Matthey plc. The pharmaceutical manufac- turer is the world’s leading manufacturer of opiate narcotics such as codeine and morphine.
Every day, Macfarlan Smith is making decisions based on its data. The company collects and ana- lyzes manufacturing operational data, for example, to allow it to meet continuous improvement goals. Sales, marketing, and finance rely on data to identify new pharmaceutical business opportunities, grow revenues, and satisfy customer needs. Additionally, the company’s manufacturing facility in Edinburgh needs to monitor, trend, and report quality data to ensure the identity, quality, and purity of its phar- maceutical ingredients for customers and regulatory authorities such as the U.S. FDA and others as part of current good manufacturing practice (CGMP).
Challenges: Multiple Sources of Truth and Slow, Onerous Reporting Processes
The process of gathering that data, making decisions, and reporting was not easy, though. The data were
scattered across the business including in the compa- ny’s bespoke enterprise resource planning (ERP) plat- form, inside legacy departmental databases such as SQL, Access databases, and stand-alone spreadsheets. When those data were needed for decision mak- ing, excessive time and resources were devoted to extracting the data, integrating them, and presenting them in a spreadsheet or other presentation outlet.
Data quality was another concern. Because teams relied on their own individual sources of data, there were multiple versions of the truth and con- flicts between the data. And it was sometimes hard to tell which version of the data was correct and which was not.
It didn’t stop there. Even once the data had been gathered and presented, making changes “on the fly” was slow and difficult. In fact, whenever a member of the Macfarlan Smith team wanted to per- form trend or other analysis, the changes to the data needed to be approved. The end result was that the data were frequently out of date by the time they were used for decision making.
Liam Mills, Head of Continuous Improvement at Macfarlan Smith highlights a typical reporting scenario:
One of our main reporting processes is the “Corrective Action and Preventive Action,” or CAPA, which is an analysis of Macfarlan Smith’s manufacturing processes taken to eliminate causes of non-conformities or other unde- sirable situations. Hundreds of hours every month were devoted to pulling data together for CAPA—and it took days to produce each
Application Case 3.6 Macfarlan Smith Improves Operational Performance Insight with Tableau Online
(Continued )
170 Part I • Introduction to Analytics and AI
report. Trend analysis was tricky too, because the data was static. In other reporting scenar- ios, we often had to wait for spreadsheet pivot table analysis; which was then presented on a graph, printed out, and pinned to a wall for everyone to review.
Slow, labor-intensive reporting processes, dif- ferent versions of the truth, and static data were all catalysts for change. “Many people were frustrated because they believed they didn’t have a complete picture of the business,” says Mills. “We were having more and more discussions about issues we faced— when we should have been talking about business intelligence reporting.”
The Solution: Interactive Data Visualizations
One of the Macfarlan Smith team had previous expe- rience in using Tableau and recommended Mills explore the solution further. A free trial of Tableau Online quickly convinced Mills that the hosted inter- active data visualization solution could conquer the data battles the company was facing.
“I was won over almost immediately,” he says. “The ease of use, the functionality and the breadth of data visualizations are all very impressive. And of course being a software-as-a-service (SaaS)-based solution, there’s no technology infrastructure invest- ment, we can be live almost immediately, and we have the flexibility to add users whenever we need.”
One of the key questions that needed to be answered concerned the security of the online data. “Our parent company Johnson Matthey has a cloud- first strategy, but has to be certain that any hosted solution is completely secure. Tableau Online fea- tures like single sign-on and allowing only autho- rized users to interact with the data provide that watertight security and confidence.”
The other security question that Macfarlan Smith and Johnson Matthey wanted answered was this: Where are the data physically stored? Mills again: “We are satisfied Tableau Online meets our criteria for data security and privacy. The data and workbooks are all hosted in Tableau’s new Dublin data center, so it never leaves Europe.”
Following a six-week trial, the Tableau sales manager worked with Mills and his team to build a
business case for Tableau Online. The management team approved it almost straight away, and a pilot program involving 10 users began. The pilot involved a manufacturing quality improvement initiative: look- ing at deviations from the norm, such as when a heat- ing device used in the opiate narcotics manufacturing process exceeds a temperature threshold. From this, a “quality operations” dashboard was created to track and measure deviations and put in place measures to improve operational quality and performance.
“That dashboard immediately signaled where deviations might be. We weren’t ploughing through rows of data—we reached answers straight away,” says Mills.
Throughout this initial trial and pilot, the team used Tableau training aids, such as the free training videos, product walk-throughs, and live online train- ing. They also participated in a two-day “fundamen- tals training” event in London. According to Mills, “The training was expert, precise and pitched just at the right level. It demonstrated to everyone just how intuitive Tableau Online is. We can visualize 10 years’ worth of data in just a few clicks.” The com- pany now has five Tableau Desktop users and up to 200 Tableau Online licensed users.
Mills and his team particularly like the Tableau Union feature in Version 9.3, which allows them to piece together data that have been split into little files. “It’s sometimes hard to bring together the data we use for analysis. The Union feature lets us work with data spread across multiple tabs or files, reducing the time we spend on prepping the data,” he says.
The Results: Cloud Analytics Transform Decision Making and Reporting
By standardizing on Tableau Online, Macfarlan Smith has transformed the speed and accuracy of its deci- sion making and business reporting. This includes:
• New interactive dashboards can be produced within one hour. Previously, it used to take days to integrate and present data in a static spreadsheet.
• The CAPA manufacturing process report, which used to absorb hundreds of man-hours every month and days to produce, can now be produced in minutes—with insights shared in the cloud.
Application Case 3.6 (Continued)
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 171
u SECTION 3.8 REVIEW QUESTIONS
1. What is data visualization? Why is it needed? 2. What are the historical roots of data visualization? 3. Carefully analyze Charles Joseph Minard’s graphical portrayal of Napoleon’s march.
Identify and comment on all the information dimensions captured in this ancient diagram.
4. Who is Edward Tufte? Why do you think we should know about his work? 5. What do you think is the “next big thing” in data visualization?
3.9 DIFFERENT TYPES OF CHARTS AND GRAPHS
Often end users of business analytics systems are not sure what type of chart or graph to use for a specific purpose. Some charts or graphs are better at answering certain types of questions. Some look better than others. Some are simple; some are rather complex and crowded. What follows is a short description of the types of charts and/or graphs com- monly found in most business analytics tools and the types of questions they are better at answering/analyzing. This material is compiled from several published articles and other literature (Abela, 2008; Hardin et al., 2012; SAS, 2014).
Basic Charts and Graphs
What follows are the basic charts and graphs that are commonly used for information visualization.
LINE CHART The line chart is perhaps the most frequently used graphical visuals for time-series data. Line charts (or line graphs) show the relationship between two variables; they are most often used to track changes or trends over time (having one of the vari- ables set to time on the x-axis). Line charts sequentially connect individual data points to help infer changing trends over a period of time. Line charts are often used to show time-dependent changes in the values of some measure, such as changes in a specific
• Reports can be changed and interrogated “on the fly” quickly and easily, without technical intervention. Macfarlan Smith has the flexibility to publish dashboards with Tableau Desktop and share them with colleagues, partners, or customers.
• The company has one, single, trusted version of the truth.
• Macfarlan Smith is now having discussions about its data—not about the issues surround- ing data integration and data quality.
• New users can be brought online almost instantly—and there’s no technical infrastruc- ture to manage.
Following this initial success, Macfarlan Smith is now extending Tableau Online to financial report- ing, supply chain analytics, and sales forecasting. Mills
concludes, “Our business strategy is now based on data-driven decisions, not opinions. The interactive visualizations enable us to spot trends instantly, identify process improvements and take business intelligence to the next level. I’ll define my career by Tableau.”
Questions for Case 3.6
1. What were the data and reporting related chal- lenges that Macfarlan Smith faced?
2. What were the solution and the obtained results/ benefits?
Source: Tableau Customer Case Study, “Macfarlan Smith improves operational performance insight with Tableau Online,” http://www. tableau.com/stories/customer/macfarlan-smith- improves- operational-performance-insight-tableau-online (accessed June 2018). Used with permission from Tableau Software, Inc.
172 Part I • Introduction to Analytics and AI
stock price over a five-year period or changes in the number of daily customer service calls over a month.
BAR CHART The bar chart is among the most basic visuals used for data representation. They are effective when you have nominal data or numerical data that split nicely into different categories so you can quickly see comparative results and trends within your data. Bar charts are often used to compare data across multiple categories such as the percentage of advertising spending by departments or by product categories. Bar charts can be vertically or horizontally oriented. They can also be stacked on top of each other to show multiple dimensions in a single chart.
PIE CHART The pie chart is visually appealing, as the name implies, pie-looking charts. Because they are so visually attractive, they are often incorrectly used. Pie charts should be used only to illustrate relative proportions of a specific measure. For instance, they can be used to show the relative percentage of an advertising budget spent on differ- ent product lines, or they can show relative proportions of majors declared by college students in their sophomore year. If the number of categories to show is more than just a few (say more than four), one should seriously consider using a bar chart instead of a pie chart.
SCATTER PLOT The scatter plot is often used to explore the relationship between two or three variables (in 2D or 3D visuals). Because scatter plots are visual exploration tools, translating more than three variables into more than three dimensions is not easily achiev- able. Scatter plots are an effective way to explore the existence of trends, concentrations, and outliers. For instance, in a two-variable (two-axis) graph, a scatter plot can be used to illustrate the co-relationship between age and weight of heart disease patients, or it can illustrate the relationship between the number of customer care representatives and the number of open customer service claims. Often, a trend line is superimposed on a two- dimensional scatter plot to illustrate the nature of the relationship.
BUBBLE CHART The bubble chart is often an enhanced version of scatter plots. Bubble charts, though, are not a new visualization type; instead, they should be viewed as a tech- nique to enrich data illustrated in scatter plots (or even geographic maps). By varying the size and/or color of the circles, one can add additional data dimensions, offering more enriched meaning about the data. For instance, a bubble chart can be used to show a competitive view of college-level class attendance by major and by time of the day, and it can be used to show profit margin by product type and by geographic region.
Specialized Charts and Graphs
The graphs and charts that we review in this section are either derived from the basic charts as special cases or they are relatively new and are specific to a problem type and/ or an application area.
HISTOGRAM Graphically speaking, a histogram looks just like a bar chart. The dif- ference between histograms and generic bar charts is the information that is portrayed. Histograms are used to show the frequency distribution of one variable or several vari- ables. In a histogram, the x-axis is often used to show the categories or ranges, and the y-axis is used to show the measures/values/frequencies. Histograms show the distribu- tional shape of the data. That way, one can visually examine whether the data are nor- mally or exponentially distributed. For instance, one can use a histogram to illustrate the
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 173
exam performance of a class, to show distribution of the grades as well as comparative analysis of individual results, or to show the age distribution of the customer base.
GANTT CHART A Gantt chart is a special case of horizontal bar charts used to portray project timelines, project tasks/activity durations, and overlap among the tasks/activities. By showing start and end dates/times of tasks/activities and the overlapping relation- ships, Gantt charts provide an invaluable aid for management and control of projects. For instance, Gantt charts are often used to show project timelines, task overlaps, relative task completions (a partial bar illustrating the completion percentage inside a bar that shows the actual task duration), resources assigned to each task, milestones, and deliverables.
PERT CHART The PERT chart (also called a network diagram) is developed primarily to simplify the planning and scheduling of large and complex projects. A PERT chart shows precedence relationships among project activities/tasks. It is composed of nodes (rep- resented as circles or rectangles) and edges (represented with directed arrows). Based on the selected PERT chart convention, either nodes or the edges can be used to repre- sent the project activities/tasks (activity-on-node versus activity-on-arrow representation schema).
GEOGRAPHIC MAP When the data set includes any kind of location data (e.g., physical addresses, postal codes, state names or abbreviations, country names, latitude/longitude, or some type of custom geographic encoding), it is better and more informative to see the data on a map. Maps usually are used in conjunction with other charts and graphs rather than by themselves. For instance, one can use maps to show the distribution of customer service requests by product type (depicted in pie charts) by geographic locations. Often a large variety of information (e.g., age distribution, income distribution, education, eco- nomic growth, population changes) can be portrayed in a geographic map to help decide where to open a new restaurant or a new service station. These types of systems are often called geographic information systems (GIS).
BULLET A bullet graph is often used to show progress toward a goal. This graph is essentially a variation of a bar chart. Often bullet graphs are used in place of gauges, meters, and thermometers in a dashboard to more intuitively convey the meaning within a much smaller space. Bullet graphs compare a primary measure (e.g., year-to-date rev- enue) to one or more other measures (e.g., annual revenue target) and present this in the context of defined performance metrics (e.g., sales quotas). A bullet graph can intuitively illustrate how the primary measure is performing against overall goals (e.g., how close a sales representative is to achieving his or her annual quota).
HEAT MAP The heat map is a great visual to illustrate the comparison of continuous values across two categories using color. The goal is to help the user quickly see where the intersection of the categories is strongest and weakest in terms of numerical values of the measure being analyzed. For instance, one can use a heat map to show segmenta- tion analysis of target markets where the measure (color gradient would be the purchase amount) and the dimensions would be age and income distribution.
HIGHLIGHT TABLE The highlight table is intended to take heat maps one step further. In addition to showing how data intersect by using color, highlight tables add a number on top to provide additional detail. That is, they are two-dimensional tables with cells popu- lated with numerical values and gradients of colors. For instance, one can show sales representatives’ performance by product type and by sales volume.
174 Part I • Introduction to Analytics and AI
TREE MAP A tree map displays hierarchical (tree-structured) data as a set of nested rectangles. Each branch of the tree is given a rectangle, which is then tiled with smaller rectangles representing subbranches. A leaf node’s rectangle has an area proportional to a specified dimension on the data. Often the leaf nodes are colored to show a separate dimension of the data. When the color and size dimensions are correlated in some way with the tree structure, one can often easily see patterns that would be difficult to spot in other ways, such as a certain color that is particularly relevant. A second advantage of tree maps is that, by construction, they make efficient use of space. As a result, they can legibly display thousands of items on the screen simultaneously.
Which Chart or Graph Should You Use?
Which chart or graph that we explained in the previous section is the best? The answer is rather easy: There is not one best chart or graph because if there were, we would not have so many chart and graph types. They all have somewhat different data representa- tion “skills.” Therefore, the right question should be, “Which chart or graph is the best for a given task?” The capabilities of the charts given in the previous section can help in selecting and using the proper chart/graph for a specific task, but doing so still is not easy to sort out. Several different chart/graph types can be used for the same visualiza- tion task. One rule of thumb is to select and use the simplest one from the alternatives to make it easy for the intended audience to understand and digest.
Although there is not a widely accepted, all-encompassing chart selection algorithm or chart/graph taxonomy, Figure 3.21 presents a rather comprehensive and highly logical
Single Variable
What would you like to show in your chart or graph?
Composition
DistributionRelationship
Two Variables
Three Variables
Changing over Time
Static Few Periods Many Periods
Only Relative Difference Matters
Relative and Absolute Difference
Matter
Only Relative Difference Matters
Relative and Absolute Difference
Matter
Simple Share
of Total
Accumulation or Subtraction
to Total
Components of
Components
Two Variables
Three Variables
Among Items Over Time
Two Variables per Item
One Variable per Item
Many Categories Few Categories
Few ItemsMany Items
Many Periods Few Periods
Cyclic Data Non-Cyclic Data Single or Few Categories
Many Categories
Many Data Points
Few Data Points
Comparison
FIGURE 3.21 A Taxonomy of Charts and Graphs. Source: Adapted from Abela, A. (2008). Advanced Presentations by Design: Creating Communication That Drives Action. New York: Wiley.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 175
organization of chart/graph types in a taxonomy-like structure (the original version was published in Abela, 2008). The taxonomic structure is organized around the questions of “What would you like to show in your chart or graph?”—that is, what the purpose of the chart or graph will be. At that level, the taxonomy divides the purpose into four different types—relationship, comparison, distribution, and composition—and further divides the branches into subcategories based on the number of variables involved and time depen- dency of the visualization.
Even though these charts and graphs cover a major part of what is commonly used in information visualization, they by no means cover all. Today, one can find many other specialized graphs and charts that serve a specific purpose. Furthermore, the cur- rent trend is to combine/hybridize and animate these charts for better-looking and more intuitive visualization of today’s complex and volatile data sources. For instance, the interactive, animated, bubble charts available at the Gapminder Web site (gapminder. org) provide an intriguing way of exploring world health, wealth, and population data from a multidimensional perspective. Figure 3.22 depicts the types of displays available at that site. In this graph, population size, life expectancy, and per capita income at the continent level are shown; also given is a time-varying animation that shows how these variables change over time.
FIGURE 3.22 A Gapminder Chart That Shows the Wealth and Health of Nations. Source: gapminder.org.
176 Part I • Introduction to Analytics and AI
u SECTION 3.9 REVIEW QUESTIONS
1. Why do you think there are many different types of charts and graphs? 2. What are the main differences among line, bar, and pie charts? When should you use
one over the others?
3. Why would you use a geographic map? What other types of charts can be combined with it?
4. Find and explain the role of two types of charts that are not covered in this section.
3.10 EMERGENCE OF VISUAL ANALYTICS
As Seth Grimes (2009a, b) has noted, there is a “growing palate” of data visualization tech- niques and tools that enable the users of business analytics and BI systems to better “commu- nicate relationships, add historical context, uncover hidden correlations, and tell persuasive stories that clarify and call to action.” The latest Magic Quadrant for Business Intelligence and Analytics Platforms released by Gartner in February 2016 further emphasizes the impor- tance of data visualization in BI and analytics. As the chart in Figure 3.23 shows, all solution
FIGURE 3.23 Magic Quadrant for Business Intelligence and Analytics Platforms. Source: Used with permission from Gartner Inc.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 177
providers in the Leaders and Visionaries quadrants are either relatively recently founded information visualization companies (e.g., Tableau Software, QlikTech) or well-established large analytics companies (e.g., Microsoft, SAS, IBM, SAP, MicroStrategy, Alteryx) that are increasingly focusing their efforts on information visualization and visual analytics. More de- tails on Gartner’s latest Magic Quadrant are given in Technology Insights 3.2.
In BI and analytics, the key challenges for visualization have revolved around the intuitive representation of large, complex data sets with multiple dimensions and mea- sures. For the most part, the typical charts, graphs, and other visual elements used in these applications usually involve two dimensions, sometimes three, and fairly small sub- sets of data sets. In contrast, the data in these systems reside in a data warehouse. At a
TECHNOLOGY INSIGHTS 3.2 Gartner Magic Quadrant for Business Intelligence and Analytics Platforms
Gartner, Inc., the creator of Magic Quadrants, is the leading IT research and advisory company publically traded in the United States with over $2 billion annual revenues in 2015. Founded in 1979, Gartner has 7,600 associates, including 1,600 research analysts and consultants and numer- ous clients in 90 countries.
Magic Quadrant is a research method designed and implemented by Gartner to monitor and evaluate the progress and positions of companies in a specific, technology-based market. By applying a graphical treatment and a uniform set of evaluation criteria, Magic Quadrant helps users to understand how technology providers are positioned within a market.
Gartner changed the name of this Magic Quadrant from “Business Intelligence Platforms” to “Business Intelligence and Analytics Platforms” to emphasize the growing importance of ana- lytics capabilities to the information systems that organizations are now building. Gartner defines the BI and analytics platform market as a software platform that delivers 15 capabilities across three categories: integration, information delivery, and analysis. These capabilities enable orga- nizations to build precise systems of classification and measurement to support decision making and improve performance.
Figure 3.23 illustrates the latest Magic Quadrant for Business Intelligence and Analytics Platforms. Magic Quadrant places providers in four groups (niche players, challengers, visionar- ies, and leaders) along two dimensions: completeness of vision (x-axis) and ability to execute (y-axis). As the quadrant clearly shows, most of the well-known BI/BA (business analytics) providers are positioned in the “leaders” category while many of the less known, relatively new, emerging providers are positioned in the “niche players” category.
The BI and analytics platform market’s multiyear shift from IT-led enterprise reporting to business-led self-service analytics seems to have passed the tipping point. Most new buying is of modern, business-user-centric visual analytics platforms forcing a new market perspective, sig- nificantly reordering the vendor landscape. Most of the activity in the BI and analytics platform market is from organizations that are trying to mature their visualization capabilities and to move from descriptive to predictive and prescriptive analytics echelons. The vendors in the market have overwhelmingly concentrated on meeting this user demand. If there were a single market theme in 2015, it would be that data discovery/visualization became a mainstream architec- ture. While data discovery/visualization vendors such as Tableau, Qlik, and Microsoft are solidi- fying their position in the Leaders quadrant, others (both emerging and large, well-established tool/solution providers) are trying to move out of Visionaries into the Leaders quadrant.
This emphasis on data discovery/visualization from most of the leaders and visionar- ies in the market—which are now promoting tools with business-user-friendly data integration coupled with embedded storage and computing layers and unfettered drilling—continues to accelerate the trend toward decentralization and user empowerment of BI and analytics and greatly enables organizations’ ability to perform diagnostic analytics.
Source: Gartner Magic Quadrant, released on February 4, 2016, gartner.com (accessed August 2016). Used with permission from Gartner Inc.
178 Part I • Introduction to Analytics and AI
minimum, these warehouses involve a range of dimensions (e.g., product, location, orga- nizational structure, time), a range of measures, and millions of cells of data. In an effort to address these challenges, a number of researchers have developed a variety of new visualization techniques.
Visual Analytics
Visual analytics is a recently coined term that is often used loosely to mean nothing more than information visualization. What is meant by visual analytics is the combi- nation of visualization and predictive analytics. Whereas information visualization is aimed at answering “What happened?” and “What is happening?” and is closely associ- ated with BI (routine reports, scorecards, and dashboards), visual analytics is aimed at answering “Why is it happening?” and “What is more likely to happen?” and is usu- ally associated with business analytics (forecasting, segmentation, correlation analysis). Many of the information visualization vendors are adding the capabilities to call them- selves visual analytics solution providers. One of the top, long-time analytics solution providers, SAS Institute, is approaching it from another direction. It is embedding its analytics capabilities into a high-performance data visualization environment that it calls visual analytics.
Visual or not visual, automated or manual, online or paper based, business report- ing is not much different than telling a story. Technology Insights 3.3 provides a different, unorthodox viewpoint on better business reporting.
TECHNOLOGY INSIGHTS 3.3 Telling Great Stories with Data and Visualization
Everyone who has data to analyze has stories to tell, whether it’s diagnosing the reasons for manufacturing defects, selling a new idea in a way that captures the imagination of your target audience, or informing colleagues about a particular customer service improvement program. And when it’s telling the story behind a big strategic choice so that you and your senior management team can make a solid decision, providing a fact-based story can be es- pecially challenging. In all cases, it’s a big job. You want to be interesting and memorable; you know you need to keep it simple for your busy executives and colleagues. Yet you also know you have to be factual, detail oriented, and data driven, especially in today’s metric- centric world.
It’s tempting to present just the data and facts, but when colleagues and senior manage- ment are overwhelmed by data and facts without context, you lose. We have all experienced presentations with large slide decks only to find that the audience is so overwhelmed with data that they don’t know what to think, or they are so completely tuned out that they take away only a fraction of the key points.
Start engaging your executive team and explaining your strategies and results more powerfully by approaching your assignment as a story. You will need the “what” of your story (the facts and data), but you also need the “Who?” “How?” “Why?” and the often-missed “So what?” It’s these story elements that will make your data relevant and tangible for your audience. Creating a good story can aid you and senior management in focusing on what is important.
Why Story? Stories bring life to data and facts. They can help you make sense and order out of a disparate collection of facts. They make it easier to remember key points and can paint a vivid picture of what the future can look like. Stories also create interactivity—people put themselves into stories and can relate to the situation.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 179
Cultures have long used storytelling to pass on knowledge and content. In some cultures, storytelling is critical to their identity. For example, in New Zealand, some of the Maori people tattoo their faces with mokus. A moku is a facial tattoo containing a story about ancestors—the family tribe. A man may have a tattoo design on his face that shows features of a hammerhead to highlight unique qualities about his lineage. The design he chooses signifies what is part of his “true self” and his ancestral home.
Likewise, when we are trying to understand a story, the storyteller navigates to finding the “true north.” If senior management is looking to discuss how they will respond to a competitive change, a good story can make sense and order out of a lot of noise. For example, you may have facts and data from two studies, one including results from an advertising study and one from a product satisfaction study. Developing a story for what you measured across both studies can help people see the whole where there were disparate parts. For rallying your distributors around a new product, you can employ a story to give vision to what the future can look like. Most important, storytelling is interactive—typically, the presenter uses words and pictures that audience members can put themselves into. As a result, they become more engaged and better understand the information.
So What Is a Good Story? Most people can easily rattle off their favorite film or book. Or they remember a funny story that a colleague recently shared. Why do people remember these stories? Because they contain cer- tain characteristics. First, a good story has great characters. In some cases, the reader or viewer has a vicarious experience where they become involved with the character. The character then has to be faced with a challenge that is difficult but believable. There must be hurdles that the character overcomes. And finally, the outcome or prognosis is clear by the end of the story. The situation may not be resolved—but the story has a clear endpoint.
Think of Your Analysis as a Story—Use a Story Structure When crafting a data-rich story, the first objective is to find the story. Who are the characters? What is the drama or challenge? What hurdles have to be overcome? And at the end of your story, what do you want your audience to do as a result?
Once you know the core story, craft your other story elements: define your characters, understand the challenge, identify the hurdles, and crystallize the outcome or decision question. Make sure you are clear with what you want people to do as a result. This will shape how your audience will recall your story. With the story elements in place, write out the storyboard, which represents the structure and form of your story. Although it’s tempting to skip this step, it is bet- ter first to understand the story you are telling and then to focus on the presentation structure and form. Once the storyboard is in place, the other elements will fall into place. The storyboard will help you think about the best analogies or metaphors, clearly set up challenge or oppor- tunity, and finally see the flow and transitions needed. The storyboard also helps you focus on key visuals (graphs, charts, and graphics) that you need your executives to recall. Figure 3.24 shows a storyline for the impact of small loans in a worldwide view within the Tableau visual analytics environment.
In summary, do not be afraid to use data to tell great stories. Being factual, detail oriented, and data driven is critical in today’s metric-centric world, but it does not have to mean being boring and lengthy. In fact, by finding the real stories in your data and following the best prac- tices, you can get people to focus on your message—and thus on what’s important. Here are those best practices:
1. Think of your analysis as a story—use a story structure. 2. Be authentic—your story will flow. 3. Be visual—think of yourself as a film editor. 4. Make it easy for your audience and you. 5. Invite and direct discussion.
Source: Fink, E., & Moore, S. J. (2012). “Five Best Practices for Telling Great Stories with Data.” White paper by Tableau Software, Inc., www.tableau.com/whitepapers/telling-data-stories (accessed May 2016).
180 Part I • Introduction to Analytics and AI
High-Powered Visual Analytics Environments
Due to the increasing demand for visual analytics coupled with fast-growing data volumes, there is an exponential movement toward investing in highly efficient visualization sys- tems. With its latest move into visual analytics, the statistical software giant SAS Institute is now among those who are leading this wave. Its new product, SAS Visual Analytics, is a very high-performance computing, in-memory solution for exploring massive amounts of data in a very short time (almost instantaneously). It empowers users to spot patterns, identify opportunities for further analysis, and convey visual results via Web reports or mobile platforms such as tablets and smartphones. Figure 3.24 shows the high-level ar- chitecture of the SAS Visual Analytics platform. On one end of the architecture, there are universal data builder and administrator capabilities, leading into explorer, report designer, and mobile BI modules, collectively providing an end-to-end visual analytics solution.
Some of the key benefits proposed by the SAS analytics platform (see Figure 3.25) are the following:
• Empowers all users with data exploration techniques and approachable analytics to drive improved decision making. SAS Visual Analytics enables different types of users to conduct fast, thorough explorations on all available data. Sampling to re- duce the data is not required and not preferred.
• Has easy-to-use, interactive Web interfaces that broaden the audience for analyt- ics, enabling everyone to glean new insights. Users can look at additional options, make more precise decisions, and drive success even faster than before.
FIGURE 3.24 A Storyline Visualization in Tableau Software. Source: Used with permission from Tableau Software, Inc.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 181
• Answers complex questions faster, enhancing the contributions from your analytic talent. SAS Visual Analytics augments the data discovery and exploration process by providing extremely fast results to enable better, more focused analysis. Analytically savvy users can identify areas of opportunity or concern from vast amounts of data so further investigation can take place quickly.
• Improves information sharing and collaboration. Large numbers of users, including those with limited analytical skills, can quickly view and interact with reports and charts via the Web, Adobe PDF files, and iPad mobile devices while IT maintains control of the underlying data and security. SAS Visual Analytics provides the right information to the right person at the right time to improve productivity and orga- nizational knowledge.
• Liberates IT by giving users a new way to access the information they need. Frees IT from the constant barrage of demands from users who need access to different amounts of data, different data views, ad hoc reports, and one-off requests for infor- mation. SAS Visual Analytics enables IT to easily load and prepare data for multiple users. Once data are loaded and available, users can dynamically explore data, cre- ate reports, and share information on their own.
• Provides room to grow at a self-determined pace. SAS Visual Analytics provides the option of using commodity hardware or database appliances from EMC Greenplum and Teradata. It is designed from the ground up for performance optimization and scalability to meet the needs of any size organization.
Figure 3.26 shows a screenshot of a SAS Analytics platform where time-series fore- casting and confidence interval around the forecast are depicted.
u SECTION 3.10 REVIEW QUESTIONS
1. What are the main reasons for the recent emergence of visual analytics? 2. Look at Gartner’s Magic Quadrant for Business Intelligence and Analytics Platforms.
What do you see? Discuss and justify your observations.
3. What is the difference between information visualization and visual analytics?
FIGURE 3.25 An Overview of SAS Visual Analytics Architecture. Source: Copyright © SAS Institute, Inc. Used with permission.
182 Part I • Introduction to Analytics and AI
4. Why should storytelling be a part of your reporting and data visualization? 5. What is a high-powered visual analytics environment? Why do we need it?
3.11 INFORMATION DASHBOARDS
Information dashboards are common components of most, if not all, BI or business ana- lytics platforms, business performance management systems, and performance measure- ment software suites. Dashboards provide visual displays of important information that is consolidated and arranged on a single screen so that the information can be digested at a single glance and easily drilled in and further explored. A typical dashboard is shown in Figure 3.27. This particular executive dashboard displays a variety of key per- formance indicators (KPIs) for a hypothetical software company called Sonatica (selling audio tools). This executive dashboard shows a high-level view of the different functional groups surrounding the products, starting from a general overview to the marketing ef- forts, sales, finance, and support departments. All of this is intended to give executive
FIGURE 3.26 A Screenshot from SAS Visual Analytics. Source: Copyright © SAS Institute, Inc. Used with permission.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 183
decision makers a quick and accurate idea of what is going on within the organization. On the left side of the dashboard, we can see (in a time-series fashion) the quarterly changes in revenues, expenses, and margins as well as the comparison of those figures to previous years’ monthly numbers. On the upper-right side are two dials with color- coded regions showing the amount of monthly expenses for support services (dial on the left) and the amount of other expenses (dial on the right). As the color coding in- dicates, although the monthly support expenses are well within the normal ranges, the other expenses are in the red region, indicating excessive values. The geographic map on the bottom right shows the distribution of sales at the country level throughout the world. Behind these graphical icons there are various mathematical functions aggregating numerous data points to their highest level of meaningful figures. By clicking on these graphical icons, the consumer of this information can drill down to more granular levels of information and data.
Dashboards are used in a wide variety of businesses for a wide variety of reasons. For instance, in Application Case 3.7, you will find the summary of a successful imple- mentation of information dashboards by the Dallas Cowboys football team.
FIGURE 3.27 A Sample Executive Dashboard. Source: A Sample Executive Dashboard from Dundas Data Visualization, Inc., www.dundas.com, reprinted with permission.
184 Part I • Introduction to Analytics and AI
Founded in 1960, the Dallas Cowboys are a pro- fessional American football team headquartered in Irving, Texas. The team has a large national follow- ing, which is perhaps best represented by their NFL record for number of consecutive games at sold-out stadiums.
The Challenge
Bill Priakos, chief operating officer (COO) of the Dallas Cowboys Merchandising Division, and his team needed more visibility into their data so they could run it more profitably. Microsoft was selected as the baseline platform for this upgrade as well as a number of other sales, logistics, and e-commerce (per MW) applications. The Cowboys expected that this new information architecture would provide the needed analytics and reporting. Unfortunately, this was not the case, and the search began for a robust dashboarding, analytics, and reporting tool to fill this gap.
The Solution and Results
Tableau and Teknion together provided real-time reporting and dashboard capabilities that exceeded the Cowboys’ requirements. Systematically and methodically, the Teknion team worked side by side with data owners and data users within the Dallas Cowboys to deliver all required functionality on time and under budget. “Early in the process, we were able to get a clear understanding of what it would take to run a more profitable operation for the Cowboys,” said Teknion Vice President Bill Luisi. “This process step is a key step in Teknion’s approach with any client, and it always pays huge
dividends as the implementation plan progresses.” Added Luisi, “Of course, Tableau worked very closely with us and the Cowboys during the entire project. Together, we made sure that the Cowboys could achieve their reporting and analytical goals in record time.”
Now, for the first time, the Dallas Cowboys are able to monitor their complete merchandising activi- ties from manufacture to end customer and not only see what is happening across the life cycle but also drill down even further into why it is happening.
Today, this BI solution is used to report and analyze the business activities of the Merchandising Division, which is responsible for all of the Dallas Cowboys’ brand sales. Industry estimates say that the Cowboys generate 20 percent of all NFL mer- chandise sales, which reflects the fact that they are the most recognized sports franchise in the world.
According to Eric Lai, a ComputerWorld reporter, Tony Romo and the rest of the Dallas Cowboys may have been only average on the foot- ball field in the last few years, but off the field, especially in the merchandising arena, they remain America’s team.
Questions for Case 3.7
1. How did the Dallas Cowboys use information visualization?
2. What were the challenge, the proposed solution, and the obtained results?
Sources: Lai, E. (2009, October 8). “BI Visualization Tool Helps Dallas Cowboys Sell More Tony Romo Jerseys,” ComputerWorld. Tableau case study. tableau.com/blog/computerworld-dallas- cowboys-business-intelligence (accessed July 2018).
Application Case 3.7 Dallas Cowboys Score Big with Tableau and Teknion
Dashboard Design
Dashboards are not a new concept. Their roots can be traced at least to the executive information system of the 1980s. Today, dashboards are ubiquitous. For example, a few years back, Forrester Research estimated that over 40 percent of the largest 2,000 com- panies in the world used the technology (Ante & McGregor, 2006). Since then, one can safely assume that this number has gone up quite significantly. In fact, today it would be rather unusual to see a large company using a BI system that does not employ some sort of performance dashboards. The Dashboard Spy Web site (dashboardspy.com/about) provides further evidence of their ubiquity. The site contains descriptions and screenshots
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 185
of thousands of BI dashboards, scorecards, and BI interfaces used by businesses of all sizes and industries, nonprofits, and government agencies.
According to Eckerson (2006), a well-known expert on BI in general and dash- boards in particular, the most distinctive feature of a dashboard is its three layers of information:
1. Monitoring: Graphical, abstracted data to monitor key performance metrics. 2. Analysis: Summarized dimensional data to analyze the root cause of problems. 3. Management: Detailed operational data that identify what actions to take to re-
solve a problem.
Because of these layers, dashboards pack a large amount of information into a sin- gle screen. According to Few (2005), “The fundamental challenge of dashboard design is to display all the required information on a single screen, clearly and without distraction, in a manner that can be assimilated quickly.” To speed assimilation of the numbers, they need to be placed in context. This can be done by comparing the numbers of interest to other baseline or target numbers, by indicating whether the numbers are good or bad, by denoting whether a trend is better or worse, and by using specialized display widgets or components to set the comparative and evaluative context. Some of the common comparisons that are typically made in BI systems include comparisons against past val- ues, forecasted values, targeted values, benchmark or average values, multiple instances of the same measure, and the values of other measures (e.g., revenues versus costs).
Even with comparative measures, it is important to specifically point out whether a particular number is good or bad and whether it is trending in the right direction. Without these types of evaluative designations, it can be time consuming to determine the status of a particular number or result. Typically, either specialized visual objects (e.g., traffic lights, dials, and gauges) or visual attributes (e.g., color coding) are used to set the evalu- ative context. An interactive dashboard-driven reporting data exploration solution built by an energy company is featured in Application Case 3.8.
Energy markets all around the world are going through a significant change and transformation, creating ample opportunities along with significant challenges. As is the case in any industry, oppor- tunities are attracting more players in the market- place, increasing the competition, and reducing the tolerances for less-than-optimal business decision making. Success requires creating and disseminat- ing accurate and timely information to whomever whenever it is needed. For instance, if you need to easily track marketing budgets, balance employee workloads, and target customers with tailored mar- keting messages, you would need three different reporting solutions. Electrabel GDF SUEZ is doing all of that for its marketing and sales business unit with SAS'Analytics Visual Analytics platform.
The one-solution approach is a great time-saver for marketing professionals in an industry that is undergoing tremendous change. “It is a huge chal- lenge to stabilize our market position in the energy market. That includes volume, prices, and margins for both retail and business customers,” notes Danny Noppe, manager of Reporting Architecture and Development in the Electrabel Marketing and Sales business unit. The company is the largest supplier of electricity in Belgium and the largest producer of elec- tricity for Belgium and the Netherlands. Noppe says it is critical that Electrabel increase the efficiency of its customer communications as it explores new digital channels and develops new energy-related services.
“The better we know the customer, the bet- ter our likelihood of success,” he says. “That is why
Application Case 3.8 Visual Analytics Helps Energy Supplier Make Better Connections
(Continued )
186 Part I • Introduction to Analytics and AI
What to Look for in a Dashboard
Although performance dashboards and other information visualization frameworks differ, they all share some common design characteristics. First, they all fit within the larger BI and/or performance measurement system. This means that their underlying architecture is the BI or performance management architecture of the larger system. Second, all well- designed dashboards and other information visualizations possess the following charac- teristics (Novell, 2009):
we combine information from various sources— phone traffic with the customer, online questions, text messages, and mail campaigns. This enhanced knowledge of our customer and prospect base will be an additional advantage within our competitive market.”
One Version of the Truth
Electrabel was using various platforms and tools for reporting purposes. This sometimes led to ambigu- ity in the reported figures. The utility also had per- formance issues in processing large data volumes. SAS Visual Analytics with in-memory technology removes the ambiguity and the performance issues. “We have the autonomy and flexibility to respond to the need for customer insight and data visualization internally,” Noppe says. “After all, fast reporting is an essential requirement for action-oriented depart- ments such as sales and marketing.”
Working More Efficiently at a Lower Cost
SAS Visual Analytics automates the process of updating information in reports. Instead of building a report that is out of date by the time it is com- pleted, the data are refreshed for all the reports once a week and is available on dashboards. In deploying the solution, Electrabel chose a phased approach, starting with simple reports and moving on to more complex ones. The first report took a few weeks to build, and the rest came quickly. The successes include the following:
• Reduction of data preparation from two days to only two hours.
• Clear graphic insight into the invoicing and composition of invoices for business-to-busi- ness (B2B) customers.
• A workload management report by the op- erational teams. Managers can evaluate team
workloads on a weekly or long-term basis and can make adjustments accordingly.
“We have significantly improved our effi- ciency and can deliver quality data and reports more frequently, and at a significantly lower cost,” says Noppe. And if the company needs to combine data from multiple sources, the process is equally easy. “Building visual reports, based on these data marts, can be achieved in a few days, or even a few hours.”
Noppe says the company plans to continue broadening its insight into the digital behavior of its customers, combining data from Web analytics, e-mail, and social media with data from back-end systems. “Eventually, we want to replace all labor- intensive reporting with SAS Visual Analytics,” he says, adding that the flexibility of SAS Visual Analytics is critical for his department. “This will give us more time to tackle other challenges. We also want to make this tool available on our mobile devices. This will allow our account managers to use up-to-date, insightful, and adaptable reports when visiting cus- tomers. We’ve got a future-oriented reporting plat- form to do all we need.”
Questions for Case 3.8
1. Why do you think energy supply companies are among the prime users of information visualiza- tion tools?
2. How did Electrabel use information visualization for the single version of the truth?
3. What were their challenges, the proposed solu- tion, and the obtained results?
Source: SAS Customer Story, “Visual Analytics Helps Energy Supplier Make Better Connections.” http://www.sas.com/ en_us/customers/electrabel-be.html (accessed July 2018). Copyright © 2018 SAS Institute Inc., Cary, NC, United States. Reprinted with permission. All rights reserved.
Application Case 3.8 (Continued)
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 187
• They use visual components (e.g., charts, performance bars, sparklines, gauges, meters, stoplights) to highlight, at a glance, the data and exceptions that require action.
• They are transparent to the user, meaning that they require minimal training and are extremely easy to use.
• They combine data from a variety of systems into a single, summarized, unified view of the business.
• They enable drill-down or drill-through to underlying data sources or reports, pro- viding more detail about the underlying comparative and evaluative context.
• They present a dynamic, real-world view with timely data refreshes, enabling the end user to stay up-to-date with any recent changes in the business.
• They require little, if any, customized coding to implement, deploy, and maintain.
Best Practices in Dashboard Design
The real estate saying “location, location, location” makes it obvious that the most im- portant attribute for a piece of real estate property is where it is located. For dashboards, it is “data, data, data.” Often overlooked, data are considered one of the most important things to focus on in designing dashboards (Carotenuto, 2007). Even if a dashboard’s ap- pearance looks professional, is aesthetically pleasing, and includes graphs and tables cre- ated according to accepted visual design standards, it is also important to ask about the data: Are they reliable? Are they timely? Are any data missing? Are they consistent across all dashboards? Here are some of the experience-driven best practices in dashboard de- sign (Radha, 2008).
Benchmark Key Performance Indicators with Industry Standards
Many customers, at some point in time, want to know if the metrics they are measuring are the right metrics to monitor. Sometimes customers have found that the metrics they are tracking are not the right ones to track. Doing a gap assessment with industry bench- marks aligns you with industry best practices.
Wrap the Dashboard Metrics with Contextual Metadata
Often when a report or a visual dashboard/scorecard is presented to business users, questions remain unanswered. The following are some examples:
• Where did you source these data? • While loading the data warehouse, what percentage of the data was rejected/
encountered data quality problems? • Is the dashboard presenting “fresh” information or “stale” information? • When was the data warehouse last refreshed? • When is it going to be refreshed next? • Were any high-value transactions that would skew the overall trends rejected as a
part of the loading process?
Validate the Dashboard Design by a Usability Specialist
In most dashboard environments, the dashboard is designed by a tool specialist without giving consideration to usability principles. Even though it is a well-engineered data warehouse that can perform well, many business users do not use the dashboard because it is perceived as not being user friendly, leading to poor adoption of the infrastructure and change management issues. Up-front validation of the dashboard design by a usabil- ity specialist can mitigate this risk.
188 Part I • Introduction to Analytics and AI
Prioritize and Rank Alerts/Exceptions Streamed to the Dashboard
Because there are tons of raw data, having a mechanism by which important exceptions/ behaviors are proactively pushed to the information consumers is important. A business rule can be codified, which detects the alert pattern of interest. It can be coded into a pro- gram, using database-stored procedures, which can crawl through the fact tables and detect patterns that need immediate attention. This way, information finds the business user as opposed to the business user polling the fact tables for the occurrence of critical patterns.
Enrich the Dashboard with Business-User Comments
When the same dashboard information is presented to multiple business users, a small text box can be provided that can capture the comments from an end user’s perspective. This can often be tagged to the dashboard to put the information in context, adding per- spective to the structured KPIs being rendered.
Present Information in Three Different Levels
Information can be presented in three layers depending on the granularity of the infor- mation: the visual dashboard level, the static report level, and the self-service cube level. When a user navigates the dashboard, a simple set of 8 to 12 KPIs can be presented, which would give a sense of what is going well and what is not.
Pick the Right Visual Construct Using Dashboard Design Principles
In presenting information in a dashboard, some information is presented best with bar charts and some with time-series line graphs, and when presenting correlations, a scatter plot is useful. Sometimes merely rendering it as simple tables is effective. Once the dashboard design principles are explicitly documented, all the developers working on the front end can adhere to the same principles while rendering the reports and dashboard.
Provide for Guided Analytics
In a typical organization, business users can be at various levels of analytical maturity. The capability of the dashboard can be used to guide the “average” business user to access the same navigational path as that of an analytically savvy business user.
u SECTION 3.11 REVIEW QUESTIONS
1. What is an information dashboard? Why is it so popular? 2. What are the graphical widgets commonly used in dashboards? Why? 3. List and describe the three layers of information portrayed on dashboards. 4. What are the common characteristics of dashboards and other information visuals? 5. What are the best practices in dashboard design?
Chapter Highlights
• Data have become one of the most valuable assets of today’s organizations.
• Data are the main ingredient for any BI, data science, and business analytics initiative.
• Although its value proposition is undeniable, to live up its promise, the data must comply with some basic usability and quality metrics.
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 189
• The term data (datum in singular form) refers to a collection of facts usually obtained as the result of experiments, observations, transactions, or experiences.
• At the highest level of abstraction, data can be classified as structured and unstructured.
• Data in original/raw form are not usually ready to be useful in analytics tasks.
• Data preprocessing is a tedious, time-demanding, yet crucial task in business analytics.
• Statistics is a collection of mathematical tech- niques to characterize and interpret data.
• Statistical methods can be classified as either de- scriptive or inferential.
• Statistics in general, as well as descriptive statistics in particular, is a critical part of BI and business analytics.
• Descriptive statistics methods can be used to mea- sure central tendency, dispersion, or the shape of a given data set.
• Regression, especially linear regression, is per- haps the most widely known and used analytics technique in statistics.
• Linear regression and logistic regression are the two major regression types in statistics.
• Logistics regression is a probability-based classifi- cation algorithm.
• Time series is a sequence of data points of a vari- able, measured and recorded at successive points in time spaced at uniform time intervals.
• A report is any communication artifact prepared with the specific intention of conveying informa- tion in a presentable form.
• A business report is a written document that con- tains information regarding business matters.
• The key to any successful business report is clar- ity, brevity, completeness, and correctness.
• Data visualization is the use of visual representations to explore, make sense of, and communicate data.
• Perhaps the most notable information graphic of the past was developed by Charles J. Minard, who graphically portrayed the losses suffered by Napoleon’s army in the Russian campaign of 1812.
• Basic chart types include line, bar, and pie chart. • Specialized charts are often derived from the
basic charts as exceptional cases. • Data visualization techniques and tools make the
users of business analytics and BI systems better information consumers.
• Visual analytics is the combination of visualiza- tion and predictive analytics.
• Increasing demand for visual analytics coupled with fast-growing data volumes led to exponen- tial growth in highly efficient visualization sys- tems investment.
• Dashboards provide visual displays of important information that is consolidated and arranged on a single screen so that information can be di- gested at a single glance and easily drilled in and further explored.
Key terms
analytics ready arithmetic mean box-and-whiskers plot box plot bubble chart business report categorical data centrality correlation dashboards data preprocessing data quality data security data taxonomy data visualization datum descriptive statistics dimensional reduction
dispersion high-performance computing histogram inferential statistics key performance indicator
(KPI) knowledge kurtosis learning linear regression logistic regression mean absolute deviation median mode nominal data online analytics processing
(OLAP) ordinal data
ordinary least squares (OLS) pie chart quartile range ratio data regression report scatter plot skewness standard deviation statistics storytelling structured data time-series forecasting unstructured data variable selection variance visual analytics
190 Part I • Introduction to Analytics and AI
Questions for Discussion
1. How do you describe the importance of data in analyt- ics? Can we think of analytics without data? Explain.
2. Considering the new and broad definition of business analytics, what are the main inputs and outputs to the analytics continuum?
3. Where do the data for business analytics come from? What are the sources and the nature of those incoming data?
4. What are the most common metrics that make for analytics-ready data?
5. What are the main categories of data? What types of data can we use for BI and analytics?
6. Can we use the same data representation for all analytics models (i.e., do different analytics models require different data representation schema)? Why, or why not?
7. Why are the original/raw data not readily usable by ana- lytics tasks?
8. What are the main data preprocessing steps? List and explain their importance in analytics.
9. What does it mean to clean/scrub the data? What activi- ties are performed in this phase?
10. Data reduction can be applied to rows (sampling) and/ or columns (variable selection). Which is more chal- lenging? Explain.
11. What is the relationship between statistics and business analytics? (Consider the placement of statistics in a busi- ness analytics taxonomy.)
12. What are the main differences between descriptive and inferential statistics?
13. What is a box-and-whiskers plot? What types of statisti- cal information does it represent?
14. What are the two most commonly used shape character- istics to describe a data distribution?
15. List and briefly define the central tendency measures of descriptive statistics.
16. What are the commonalities and differences between regression and correlation?
17. List and describe the main steps to follow in developing a linear regression model.
18. What are the most commonly pronounced assumptions for linear regression? What is crucial to the regression models against these assumptions?
19. What are the commonalities and differences between linear regression and logistic regression?
20. What is time series? What are the main forecasting tech- niques for time-series data?
21. What is a business report? Why is it needed? 22. What are the best practices in business reporting? How
can we make our reports stand out? 23. Describe the cyclic process of management, and com-
ment on the role of business reports. 24. List and describe the three major categories of business
reports. 25. Why has information visualization become a center-
piece in BI and business analytics? Is there a difference between information visualization and visual analytics?
26. What are the main types of charts/graphs? Why are there so many of them?
27. How do you determine the right chart for a job? Explain and defend your reasoning.
28. What is the difference between information visualiza- tion and visual analytics?
29. Why should storytelling be a part of your reporting and data visualization?
30. What is an information dashboard? What does it present? 31. What are the best practices in designing highly informa-
tive dashboards? 32. Do you think information/performance dashboards are
here to stay? Or are they about to be outdated? What do you think will be the next big wave in BI and business analytics in terms of data/information visualization?
Exercises
Teradata University and Other Hands-on Exercises
1. Download the “Voting Behavior” data and the brief data description from the book’s Web site. This is a data set manually compiled from counties all around the United States. The data are partially processed, that is, some derived variables have been created. Your task is to thoroughly preprocess the data by identifying the error and anomalies and proposing remedies and solutions. At the end, you should have an analytics-ready version of these data. Once the pre- processing is completed, pull these data into Tableau (or into some other data visualization software tool)
to extract useful visual information from it. To do so, conceptualize relevant questions and hypotheses (come up with at least three of them) and create proper visualizations that address those questions of “tests” of those hypotheses.
2. Download Tableau (at tableau.com, following aca- demic free software download instructions on the site). Using the Visualization_MFG_Sample data set (available as an Excel file on this book’s Web site), answer the fol- lowing questions: a. What is the relationship between gross box office
revenue and other movie-related parameters given in the data set?
Chapter 3 • Nature of Data, Statistical Modeling, and Visualization 191
b. How does this relationship vary across different years? Prepare a professional-looking written report that is enhanced with screenshots of your graphic findings.
3. Go to teradatauniversitynetwork.com. Look for an article that deals with the nature of data, management of data, and/or governance of data as it relates to BI and analytics, and critically analyze the content of the article.
4. Go to UCI data repository (archive.ics.uci.edu/ml/ datasets.html) and identify a large data set that con- tains both numeric and nominal values. Using Microsoft Excel or any other statistical software: a. Calculate and interpret central tendency measures
for each and every variable. b. Calculate and interpret the dispersion/spread mea-
sures for each and every variable. 5. Go to UCI data repository (archive.ics.uci.edu/ml/
datasets.html) and identify two data sets, one for estimation/regression and one for classification. Using Microsoft Excel or any other statistical software: a. Develop and interpret a linear regression model. b. Develop and interpret a logistic regression model.
6. Go to KDnuggest.com and become familiar with the range of analytics resources available on this portal. Then identify an article, a white paper, or an interview script that deals with the nature of data, management of data, and/or governance of data as they relate to BI and business analytics, and critically analyze the content of the article.
7. Go to Stephen Few’s blog, “The Perceptual Edge” (perceptualedge.com). Go to the section of “Examples.” In this section, he provides critiques of various dashboard examples. Read a handful of these examples. Now go to dundas.com. Select the “Gallery” section of the site. Once there, click the “Digital Dashboard” selection. You will be shown a variety of different dashboard demos. Run a couple of them. a. What types of information and metrics are shown on
the demos? What types of actions can you take? b. Using some of the basic concepts from Few’s cri-
tiques, describe some of the good design points and bad design points of the demos.
8. Download an information visualization tool, such as Tableau, QlikView, or Spotfire. If your school does not have an educational agreement with these companies, a trial version would be sufficient for this exercise. Use your own data (if you have any) or use one of the data sets that comes with the tool (such tools usually have one or more data sets for demonstration purposes). Study the data, come up with several business prob- lems, and use data visualization to analyze, visualize, and potentially solve those problems.
9. Go to teradatauniversitynetwork.com. Find the “Tableau Software Project.” Read the description, exe- cute the tasks, and answer the questions.
10. Go to teradatauniversitynetwork.com. Find the assignments for SAS Visual Analytics. Using the infor- mation and step-by-step instructions provided in the
assignment, execute the analysis on the SAS Visual Analytics tool (which is a Web-enabled system that does not require any local installation). Answer the questions posed in the assignment.
11. Find at least two articles (one journal article and one white paper) that talk about storytelling, especially within the context of analytics (i.e., data-driven storytell- ing). Read and critically analyze the article and paper, and write a report to reflect your understanding and opinions about the importance of storytelling in BI and business analytics.
12. Go to data.gov—a U.S. government–sponsored data portal that has a very large number of data sets on a wide variety of topics ranging from healthcare to edu- cation, climate to public safety. Pick a topic that you are most passionate about. Go through the topic- specific information and explanation provided on the site. Explore the possibilities of downloading the data, and use your favorite data visualization tool to create your own meaningful information and visualizations.
Team Assignments and Role-Playing Projects
1. Analytics starts with data. Identifying, accessing, obtain- ing, and processing of relevant data is the most essential task in any analytics study. As a team, you are tasked to find a large enough real-world data (either from your own organization, which is the most preferred, or from the Internet that can start with a simple search, or from the data links posted on KDnuggets.com), one that has tens of thousands of rows and more than 20 vari- ables to go through, and document a thorough data preprocessing project. In your processing of the data, identify anomalies and discrepancies using descriptive statistics methods and measures, and make the data an- alytics ready. List and justify your preprocessing steps and decisions in a comprehensive report.
2. Go to a well-known information dashboard provider Web site (dundas.com, idashboards.com, enterprise- dashboard.com). These sites provide a number of exam- ples of executive dashboards. As a team, select a particular industry (e.g., healthcare, banking, airline). Locate a hand- ful of example dashboards for that industry. Describe the types of metrics found on the dashboards. What types of displays are used to provide the information? Using what you know about dashboard design, provide a paper pro- totype of a dashboard for this information.
3. Go to teradatauniversitynetwork.com. From there, go to University of Arkansas data sources. Choose one of the large data sets, and download a large number of records (this could require you to write an SQL state- ment that creates the variables that you want to include in the data set). Come up with at least 10 questions that can be addressed with information visualization. Using your favorite data visualization tool (e.g., Tableau), ana- lyze the data, and prepare a detailed report that includes screenshots and other visuals.
192 Part I • Introduction to Analytics and AI
References
Abela, A. (2008). Advanced Presentations by Design: Creating Communication That Drives Action. New York, NY: Wiley.
Annas, G. (2003). “HIPAA Regulations—A New Era of Medical-Record Privacy?” New England Journal of Medicine, 348(15), 1486–1490.
Ante, S., & J. McGregor. (2006). “Giving the Boss the Big Pic- ture: A Dashboard Pulls Up Everything the CEO Needs to Run the Show.” Business Week, 43–51.
Carotenuto, D. (2007). “Business Intelligence Best Practices for Dashboard Design.” WebFOCUS. www. datawarehouse. inf.br/papers/information_builders_dashboard_ best_practices.pdf (accessed August 2016).
Dell Customer Case Study. “Medical Device Company Ensures Product Quality While Saving Hundreds of Thousands of Dollars.” https://software.dell.com/ documents/instrumentation-laboratory-medical- device- companyensures-product-quality-while- saving-hundreds-of thousands-of-dollars-case- study-80048.pdf (accessed August 2016).
Delen, D. (2010). “A Comparative Analysis of Machine Learn- ing Techniques for Student Retention Management.” Deci- sion Support Systems, 49(4), 498–506.
Delen, D. (2011). “Predicting Student Attrition with Data Min- ing Methods.” Journal of College Student Retention 13(1), 17–35.
Delen, D. (2015). Real-World Data Mining: Applied Business Analytics and Decision Making. Upper Saddle River, NJ: Financial Times Press (A Pearson Company).
Delen, D., D. Cogdell, & N. Kasap. (2012). “A Comparative Analysis of Data Mining Methods in Predicting NCAA Bowl Outcomes.” International Journal of Forecasting, 28, 543–552.
Eckerson, W. (2006). Performance Dashboards. New York: Wiley.
Few, S. (2005, Winter). “Dashboard Design: Beyond Meters, Gauges, and Traffic Lights.” Business Intelligence Journal, 10(1).
Few, S. (2007). “Data Visualization: Past, Present and Future.” Perceptualedge.com/articles/Whitepapers/Data_ Visualization.pdf (accessed July 2016).
Fink, E., & S. J. Moore. (2012). “Five Best Practices for Telling Great Stories with Data.” Tableau Software, Inc. www. tableau.com/whitepapers/telling-data-stories (accessed May 2016).
Freeman, K., & R. M. Brewer. (2016). “The Politics of Ameri- can College Football.” Journal of Applied Business and Economics, 18(2), 97–101.
Gartner Magic Quadrant. (2016, February 4). gartner.com (accessed August 2016).
Grimes, S. (2009a, May 2). “Seeing Connections: Visualizations Makes Sense of Data. Intelligent Enterprise.” i.cmpnet. com/intelligententer pr ise/next-era-business- intelligence/Intelligent_Enterprise_Next_Era_BI_ Visualization.pdf (accessed January 2010).
Grimes, S. (2009b). Text “Analytics 2009: User Perspectives on Solutions and Providers.” Alta Plana. altaplana.com/ TextAnalyticsPerspectives2009.pdf (accessed July, 2016).
Hardin, M. Hom, R. Perez, & Williams L. (2012). “Which Chart or Graph Is Right for You?” Tableau Software. http:// www.tableau.com/sites/default/files/media/which_ chart_v6_final_0.pdf (accessed August 2016).
Hernández, M., & S. J. Stolfo. (1998, January). “Real-World Data Is Dirty: Data Cleansing and the Merge/Purge Problem.” Data Mining and Knowledge Discovery, 2(1), 9–37.
Hill, G. (2016). “A Guide to Enterprise Reporting.” Ghill. customer.netspace.net.au/reporting/definition.html (accessed July 2016).
Kim, W., B. J. Choi, E. K. Hong, S. K. Kim, & D. Lee. (2003). “A Taxonomy of Dirty Data.” Data Mining and Knowledge Discovery, 7(1), 81–99.
Kock, N. F., R. J. McQueen, & J. L. Corner. (1997). “The Na- ture of Data, Information and Knowledge Exchanges in Business Processes: Implications for Process Improvement and Organizational Learning.” The Learning Organization, 4(2), 70–80.
Kotsiantis, S., D. Kanellopoulos, & P. E. Pintelas. (2006). “Data Preprocessing for Supervised Leaning.” International Jour- nal of Computer Science, 1(2), 111–117.
Lai, E. (2009, October 8). “BI Visualization Tool Helps Dallas Cowboys Sell More Tony Romo Jerseys.” Com- puterWorld.
Quinn, C. (2016). “Data-Driven Marketing at SiriusXM,” Teradata Articles & News. http://bigdata.teradata. com/US/Articles-News/Data-Driven-Marketing-At- SiriusXM/ (accessed August 2016); “SiriusXM Attracts and Engages a New Generation of Radio Consumers.” http:// assets. teradata.com/resourceCenter/downloads/ CaseStudies/EB8597.pdf?processed=1 (accessed August 2018).
Novell. (2009, April). “Executive Dashboards Elements of Success.” Novell white paper. www.novell.com/ docrep/documents/3rkw3et fc3/Executive%20 Dashboards_Elements_of_Success_White_Paper_ en.pdf (accessed June 2016).
Radha, R. (2008). “Eight Best Practices in Dashboard Design.” Information Management. www.information- management.com/news/columns/-10001129-1.html (accessed July 2016).
SAS. (2014). “Data Visualization Techniques: From Basics to Big Data.” http://www.sas.com/content/dam/ SAS/en_us/doc/whitepaper1/data-visualization- techniques-106006.pdf (accessed July 2016).
Thammasiri, D., D. Delen, P. Meesad, & N. Kasap. (2014). “A Critical Assessment of Imbalanced Class Distribution Prob- lem: The Case of Predicting Freshmen Student Attrition.” Expert Systems with Applications, 41(2), 321–330.
193
P A R T
Predictive Analytics/ Machine Learning
II
194
LEARNING OBJECTIVES
Data Mining Process, Methods, and Algorithms
4 C H A P T E R
■■ Define data mining as an enabling technology for business analytics
■■ Understand the objectives and benefits of data mining
■■ Become familiar with the wide range of applications of data mining
■■ Learn the standardized data mining processes
■■ Learn different methods and algorithms of data mining
■■ Build awareness of existing data mining software tools
■■ Understand the privacy issues, pitfalls, and myths of data mining
G enerally speaking, data mining is a way to develop intelligence (i.e., actionable information or knowledge) from data that an organization collects, organizes, and stores. A wide range of data mining techniques is being used by organiza- tions to gain a better understanding of their customers and their operations and to solve complex organizational problems. In this chapter, we study data mining as an enabling technology for business analytics and predictive analytics; learn about the standard pro- cesses of conducting data mining projects; understand and build expertise in the use of major data mining techniques; develop awareness of the existing software tools; and explore privacy issues, common myths, and pitfalls that are often associated with data mining.
4.1 Opening Vignette: Miami-Dade Police Department Is Using Predictive Analytics to Foresee and Fight Crime 195
4.2 Data Mining Concepts 198 4.3 Data Mining Applications 208 4.4 Data Mining Process 211 4.5 Data Mining Methods 220 4.6 Data Mining Software Tools 236 4.7 Data Mining Privacy Issues, Myths, and Blunders 242
Chapter 4 • Data Mining Process, Methods, and Algorithms 195
4.1 OPENING VIGNETTE: Miami-Dade Police Department Is Using Predictive Analytics to Foresee and Fight Crime
Predictive analytics and data mining have become an integral part of many law enforce- ment agencies including the Miami-Dade Police Department whose mission is not only to protect the safety of Florida’s largest county, with 2.5 million citizens (making it the seventh largest in the United States), but also to provide a safe and inviting climate for the millions of tourists who come from around the world to enjoy the county’s natural beauty, warm climate, and stunning beaches. With tourists spending nearly US$20 billion every year and generating nearly one-third of Florida’s sales taxes, it is hard to overstate the importance of tourism to the region’s economy. So although few of the county’s police officers would likely list economic development in their job description, nearly all grasp the vital link between safe streets and the region’s tourist-driven prosperity.
That connection is paramount for Lieutenant Arnold Palmer, currently supervising the Robbery Investigations Section and a former supervisor of the department’s Robbery Intervention Detail. This specialized team of detectives is focused on intensely policing the county’s robbery hot spots and worst repeat offenders. He and the team occupy mod- est offices on the second floor of a modern-looking concrete building set back from a palm-lined street on the western edge of Miami. In his 10 years in the unit and 23 in total on the force, Palmer has seen many changes—not just in policing practices like the way his team used to mark street crime hot spots with colored pushpins on a map.
POLICING WITH LESS
Palmer and the team have also seen the impact of a growing population, shifting demo- graphics, and a changing economy on the streets they patrol. Like any good police force officers, they have continually adapted their methods and practices to meet a policing challenge that has grown in scope and complexity. But like nearly all branches of the county’s government, intensifying budget pressures have placed the department in a squeeze between rising demands and shrinking resources.
Palmer, who sees detectives as front-line fighters against a rising tide of street crime and the looming prospect of ever-tightening resources, put it this way: “Our basic chal- lenge was how to cut street crime even as tighter resources have reduced the number of cops on the street.” Over the years, the team has been open to trying new tools, the most notable of which is a program called “analysis-driven enforcement,” which used crime history data as the basis for positioning teams of detectives. “We’ve evolved a lot since then in our ability to predict where robberies are likely to occur, both through the use of analysis and our own collective experience.”
NEW THINKING ON COLD CASES
The more confounding challenge for Palmer and his team of investigators, one shared with the police of all major urban areas, is in closing the hardest cases whose leads, wit- nesses, video—any facts or evidence that can help solve a case—are lacking. This is not surprising, explains Palmer, because “the standard practices we used to generate leads, like talking to informants or to the community or to patrol officers, haven’t changed much, if at all. That kind of an approach works okay, but it relies a lot on the experience our detectives carry in their head. When the detectives retire or move on, that experience goes with them.”
Palmer’s conundrum was that turnover resulting from the retirement of many of his most experienced detectives was on an upward trend. True, he saw the infusion of young blood as an inherently good thing, especially given this group’s increased comfort with
196 Part II • Predictive Analytics/Machine Learning
the new types of information—from e-mails, social media, and traffic cameras, to name a few—to which his team had access. But as Palmer recounts, the problem came when the handful of new detectives coming into the unit turned to look for guidance from the senior officers “and it’s just not there. We knew at that point we needed a different way to fill the experience gap going forward.”
His ad hoc efforts to come up with a solution led to blue-sky speculation. What if new detectives on the squad could pose the same questions to a computer database as they would to a veteran detective? That speculation planted a seed in Palmer’s mind that wouldn’t go away.
THE BIG PICTURE STARTS SMALL
What was taking shape within the robbery unit demonstrated how big ideas can come from small places. But more importantly, it showed that for these ideas to reach frui- tion, the “right” conditions need to be in alignment at the right time. On a leadership level, this means a driving figure in the organization who knows what it takes to nurture top-down support as well as crucial bottom-up buy-in from the ranks while keeping the department’s information technology (IT) personnel on the same page. That person was Palmer. At the organizational level, the robbery unit served as a particularly good launching point for lead modeling because of the prevalence of repeat offenders among perpetrators. Ultimately, the department’s ability to unleash the broader transformative potential of lead modeling would hinge in large part on the team’s ability to deliver results on a small scale.
When early tests and demos proved encouraging—with the model yielding accu- rate results when the details of solved cases were fed into it—the team started gaining attention. The initiative received a critical boost when the robbery bureau’s unit major and captain voiced their support for the direction of the project, telling Palmer that “if you can make this work, run with it.” But more important than the encouragement, Palmer explains, was their willingness to advocate for the project among the department’s higher-ups. “I can’t get it off the ground if the brass doesn’t buy in,” says Palmer. “So their support was crucial.”
SUCCESS BRINGS CREDIBILITY
Having been appointed the official liaison between IT and the robbery unit, Palmer set out to strengthen the case for the lead modeling tool—now officially called Blue PALMS (for Predictive Analytics Lead Modeling Software)—by building a series of successes. His constituency was not only the department brass but also the detectives whose support would be critical to its successful adoption as a robbery-solving tool. In his attempts to introduce Blue PALMS, resistance was predictably stronger among veteran detectives who saw no reason to give up their long-standing practices. Palmer knew that dictates or coercion would not win their hearts and minds. He would need to build a beachhead of credibility.
Palmer found that opportunity in one of his best and most experienced detectives. Early in a robbery investigation, the detective indicated to Palmer that he had a strong hunch who the perpetrator was and wanted, in essence, to test the Blue PALMS system. At the detective’s request, the department analyst fed key details of the crime into the sys- tem, including the modus operandi (MO). The system’s statistical models compared these details to a database of historical data, looking for important correlations and similarities in the crime’s signature. The report that came out of the process included a list of 20 suspects ranked in order of match strength, or likelihood. When the analyst handed the detective the report, his “hunch” suspect was listed in the top five. Soon after his arrest, the suspect confessed, and Palmer had gained a solid convert.
Chapter 4 • Data Mining Process, Methods, and Algorithms 197
Although it was a useful exercise, Palmer realized that the true test was not in con- firming hunches but in breaking cases that had come to a dead end. Such was the situ- ation in a carjacking that had, in Palmer’s words, “no witnesses, no video, and no crime scene—nothing to go on.” When the senior detective on the stalled case went on leave after three months, the junior detective to whom it was assigned requested a Blue PALMS report. Shown photographs of the top people on the suspect list, the victim made a posi- tive identification of the suspect leading to the successful conclusion of the case. That suspect was number one on the list.
JUST THE FACTS
The success that Blue PALMS continues to build has been a major factor in Palmer’s get- ting his detectives on board. But if there is a part of his message that resonates even more with his detectives, it is the fact that Blue PALMS is designed not to change the basics of policing practices but to enhance them by giving them a second chance of cracking the case. “Police work is at the core about human relations—about talking to witnesses, to victims, to the community—and we’re not out to change that,” says Palmer. “Our aim is to give investigators factual insights from information we already have that might make a difference, so even if we’re successful 5 percent of the time, we’re going to take a lot of offenders off the street.”
The growing list of cold cases solved has helped Palmer in his efforts to reinforce the merits of Blue PALMS. But, in showing where his loyalty lies, he sees the detectives who have closed these cold cases—not the program—as most deserving of the spotlight, and that approach has gone over well. At his chief’s request, Palmer is beginning to use his liai- son role as a platform for reaching out to other areas in the Miami-Dade Police Department.
SAFER STREETS FOR A SMARTER CITY
When he speaks of the impact of tourism, a thread that runs through Miami-Dade’s Smarter Cities vision, Palmer sees Blue PALMS as an important tool to protect one of the county’s greatest assets. “The threat to tourism posed by rising street crime was a big rea- son the unit was established,” says Palmer. “The fact that we’re able to use analytics and intelligence to help us close more cases and keep more criminals off the street is good news for our citizens and our tourist industry.”
u QUESTIONS FOR THE OPENING VIGNETTE
1. Why do law enforcement agencies and departments like the Miami-Dade Police Department embrace advanced analytics and data mining?
2. What are the top challenges for law enforcement agencies and departments like the Miami-Dade Police Department? Can you think of other challenges (not mentioned in this case) that can benefit from data mining?
3. What are the sources of data that law enforcement agencies and departments like the Miami-Dade Police Department use for their predictive modeling and data mining projects?
4. What type of analytics do law enforcement agencies and departments like the Miami-Dade Police Department use to fight crime?
5. What does “the big picture starts small” mean in this case? Explain.
WHAT WE CAN LEARN FROM THIS VIGNETTE
Law enforcement agencies and departments are under tremendous pressure to carry out their mission of safeguarding people with limited resources. The environment within which they perform their duties is becoming increasingly more challenging so that they
198 Part II • Predictive Analytics/Machine Learning
have to constantly adopt and perhaps stay a few steps ahead to prevent the likelihood of catastrophes. Understanding the changing nature of crime and criminals is an ongo- ing challenge. In the midst of these challenges, what works in favor of these agencies is the availability of the data and analytics technologies to better analyze past occur- rences and to foresee future events. Data have become available more now than in the past. Applying advanced analytics and data mining tools (i.e., knowledge discovery techniques) to these large and rich data sources provides them with the insight that they need to better prepare and act on their duties. Therefore, law enforcement agencies are becoming one of the leading users of the new face of analytics. Data mining is a prime candidate for better understanding and management of these mission-critical tasks with a high level of accuracy and timeliness. The study described here clearly illustrates the power of analytics and data mining to create a holistic view of the world of crime and criminals for better and faster reaction and management. In this chapter, you will see a wide variety of data mining applications solving complex problems in a variety of indus- tries and organizational settings where the data are used to discover actionable insight to improve mission readiness, operational efficiency, and competitive advantage.
Sources: “Miami-Dade Police Department: Predictive modeling pinpoints likely suspects based on common crime signatures of previous crimes,” IBM Customer Case Studies. www-03.ibm.com/software/ businesscasestudies/om/en/corp?synkey=C894638H25952N07; “Law Enforcement Analytics: Intelligence- Led and Predictive Policing by Information Builder.” www.informationbuilders.com/solutions/gov-lea.
4.2 DATA MINING CONCEPTS
Data mining, a relatively new and exciting technology, has become a common practice for a vast majority of organizations. In an interview with Computerworld magazine in January 1999, Dr. Arno Penzias (Nobel laureate and former chief scientist of Bell Labs) identified data mining from organizational databases as a key application for corporations of the near future. In response to Computerworld’s age-old question of “What will be the killer applica- tions in the corporation?” Dr. Penzias replied, “Data mining.” He then added, “Data mining will become much more important and companies will throw away nothing about their customers because it will be so valuable. If you’re not doing this, you’re out of business.” Similarly, in an article in Harvard Business Review, Thomas Davenport (2006) argued that the latest strategic weapon for companies is analytical decision making, providing examples of companies such as Amazon.com, Capital One, Marriott International, and others that have used analytics to better understand their customers and optimize their extended sup- ply chains to maximize their returns on investment while providing the best customer ser- vice. This level of success is highly dependent on a company’s thorough understanding of its customers, vendors, business processes, and the extended supply chain.
A large portion of “understanding the customer” can come from analyzing the vast amount of data that a company collects. The cost of storing and processing data has decreased dramatically in the recent past, and, as a result, the amount of data stored in electronic form has grown at an explosive rate. With the creation of large databases, the possibility of ana- lyzing the data stored in them has emerged. The term data mining was originally used to describe the process through which previously unknown patterns in data were discovered. This definition has since been stretched beyond those limits by some software vendors to include most forms of data analysis in order to increase sales with the popularity of the data mining label. In this chapter, we accept the original definition of data mining.
Although the term data mining is relatively new, the ideas behind it are not. Many of the techniques used in data mining have their roots in traditional statistical analysis and artificial intelligence work done since the early part of the 1980s. Why, then, has it
Chapter 4 • Data Mining Process, Methods, and Algorithms 199
suddenly gained the attention of the business world? Following are some of the most important reasons:
• More intense competition at the global scale driven by customers’ ever-changing needs and wants in an increasingly saturated marketplace.
• General recognition of the untapped value hidden in large data sources. • Consolidation and integration of database records, which enables a single view of
customers, vendors, transactions, and so on. • Consolidation of databases and other data repositories into a single location in the
form of a data warehouse. • The exponential increase in data processing and storage technologies. • Significant reduction in the cost of hardware and software for data storage and
processing. • Movement toward the demassification (conversion of information resources into
nonphysical form) of business practices.
Data generated by the Internet are increasing rapidly in both volume and complex- ity. Large amounts of genomic data are being generated and accumulated all over the world. Disciplines such as astronomy and nuclear physics create huge quantities of data on a regular basis. Medical and pharmaceutical researchers constantly generate and store data that can then be used in data mining applications to identify better ways to accu- rately diagnose and treat illnesses and to discover new and improved drugs.
On the commercial side, perhaps the most common use of data mining has been in the finance, retail, and healthcare sectors. Data mining is used to detect and reduce fraudulent activities, especially in insurance claims and credit card use (Chan et al., 1999); to identify customer buying patterns (Hoffman, 1999); to reclaim profitable customers (Hoffman, 1998); to identify trading rules from historical data; and to aid in increased profitability using market-basket analysis. Data mining is already widely used to bet- ter target clients, and with the widespread development of e-commerce, this can only become more imperative with time. See Application Case 4.1 for information on how Infinity P&C has used predictive analytics and data mining to improve customer service, combat fraud, and increase profit.
When card issuers first started using automated busi- ness rules software to counter debit and credit card fraud, the limits on that technology were quickly evident: Customers reported frustrating payment rejections on dream vacations or critical business trips. Visa works with its clients to improve cus- tomer experience by providing cutting-edge fraud risk tools and consulting services that make its strategies more effective. Through this approach, Visa enhances customer experience and minimizes invalid transaction declines.
The company’s global network connects thousands of financial institutions with millions of merchants and cardholders every day. It has been
a pioneer in cashless payments for more than 50 years. By using SAS® Analytics, Visa is supporting financial institutions to reduce fraud without upset- ting customers with unnecessary payment rejections. Whenever it processes a transaction, Visa analyzes up to 500 unique variables in real time to assess the risk of that transaction. Using vast data sets, includ- ing global fraud hot spots and transactional patterns, the company can more accurately assess whether you’re buying escargot in Paris or someone who stole your credit card is.
“What that means is that if you are likely to travel we know it, and we tell your financial insti- tution so you’re not declined at the point of sale,”
Application Case 4.1 Visa Is Enhancing the Customer Experience while Reducing Fraud with Predictive Analytics and Data Mining
(Continued )
200 Part II • Predictive Analytics/Machine Learning
says Nathan Falkenborg, head of Visa Performance Solutions for North Asia. “We also will assist your bank in developing the right strategies for using the Visa tools and scoring systems,” he adds. Visa esti- mates that Big Data analytics works; state-of-the-art models and scoring systems have the potential to prevent an incremental $2 billion of fraudulent pay- ment volume annually.
A globally recognized name, Visa facilitates electronic funds transfer through branded products that are issued by its thousands of financial institu- tion partners. The company processed 64.9 billion transactions in 2014, and $4.7 trillion in purchases were made with Visa cards in the same year.
Visa has the computing capability to process 56,000 transaction messages per second, which is more than four times the actual peak transaction rate to date. Visa does not just process and compute—it is continually using analytics to share strategic and operational insights with its partner financial insti- tutions and assist them in improving performance. This business goal is supported by a robust data management system. Visa also assists its clients in improving performance by developing and deliver- ing deep analytical insight.
“We understand patterns of behavior by per- forming clustering and segmentation at a granular level, and we provide this insight to our financial institution partners,” says Falkenborg. “It’s an effec- tive way to help our clients communicate better and deepen their understanding of the customer.”
As an example of marketing support, Visa has assisted clients globally in identifying segments of customers that should be offered a different Visa product. “Understanding the customer lifecycle is incredibly important, and Visa provides information to clients that help them take action and offer the right product to the right customer before a value proposition becomes stale,” says Falkenborg.
How Can Using In-Memory Analytics Make a Difference?
In a recent proof of concept, Visa used a high- performance solution from SAS that relies on in-memory computing to power statistical and machine-learning algorithms and then present the information visually. In-memory analytics reduces
the need to move data and perform additional model iterations, making it much faster and accurate.
Falkenborg describes the solution as like having the information memorized versus having to get up and go to a filing cabinet to retrieve it. “In-memory analytics is just taking your brain and making it bigger. Everything is instantly accessible.”
Ultimately, solid analytics helps the company do more than just process payments. “We can deepen the client conversation and serve our clients even better with our incredible big data set and expertise in mining transaction data,” says Falkenborg. “We use our consulting and analytics capabilities to assist our clients in tackling business challenges and pro- tect the payment ecosystem. And that’s what we do with high-performance analytics.”
Falkenborg elaborates,
The challenge that we have, as with any com- pany managing and using massive data sets, is how we use all necessary information to solve a business challenge—whether that is improving our fraud models, or assisting a cli- ent to more effectively communicate with its customers. In-memory analytics enables us to be more nimble; with a 100* analytical system processing speed improvement, our data and decision scientists can iterate much faster.
Fast and accurate predictive analytics allows Visa to better serve clients with tailored consult- ing services, helping them succeed in today’s fast- changing payments industry.
Questions for Case 4.1
1. What challenges were Visa and the rest of the credit card industry facing?
2. How did Visa improve customer service while also improving concepts related to retention of fraud?
3. What is in-memory analytics, and why was it necessary?
Source: “Enhancing the Customer Experience While Reducing Fraud (SAS® Analytics) High-Performance Analytics Empowers Visa to Enhance Customer Experience While Reducing Debit and Credit Card Fraud.” Copyright © 2018 SAS Institute Inc., Cary, NC, USA. Reprinted with permission. All rights reserved.
Application Case 4.1 (Continued)
Chapter 4 • Data Mining Process, Methods, and Algorithms 201
Definitions, Characteristics, and Benefits
Simply defined, data mining is a term used to describe discovering or “mining” knowl- edge from large amounts of data. When considered by analogy, one can easily realize that the term data mining is a misnomer; that is, mining of gold from within rocks or dirt is referred to as “gold” mining rather than “rock” or “dirt” mining. Therefore, data min- ing perhaps should have been named “knowledge mining” or “knowledge discovery.” Despite the mismatch between the term and its meaning, data mining has become the choice of the community. Many other names that are associated with data mining include knowledge extraction, pattern analysis, data archaeology, information harvesting, pattern searching, and data dredging.
Technically speaking, data mining is a process that uses statistical, mathematical, and artificial intelligence techniques to extract and identify useful information and subse- quent knowledge (or patterns) from large sets of data. These patterns can be in the form of business rules, affinities, correlations, trends, or prediction models (see Nemati and Barko, 2001). Most literature defines data mining as “the nontrivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data stored in structured databases,” where the data are organized in records structured by categori- cal, ordinal, and continuous variables (Fayyad, Piatetsky-Shapiro, and Smyth, 1996, pp. 40–41). In this definition, the meanings of the key term are as follows:
• Process implies that data mining comprises many iterative steps. • Nontrivial means that some experimental type search or inference is involved; that
is, it is not as straightforward as a computation of predefined quantities. • Valid means that the discovered patterns should hold true on new data with a suf-
ficient degree of certainty. • Novel means that the patterns are not previously known to the user within the con-
text of the system being analyzed. • Potentially useful means that the discovered patterns should lead to some benefit to
the user or task. • Ultimately understandable means that the pattern should make business sense that
leads to the user saying, “Mmm! It makes sense; why didn’t I think of that,” if not immediately, at least after some postprocessing.
Data mining is not a new discipline but rather a new definition for the use of many disciplines. Data mining is tightly positioned at the intersection of many disci- plines, including statistics, artificial intelligence, machine learning, management sci- ence, information systems (IS), and databases (see Figure 4.1). Using advances in all of these disciplines, data mining strives to make progress in extracting useful information and knowledge from large databases. It is an emerging field that has attracted much attention in a very short time.
The following are the major characteristics and objectives of data mining:
• Data are often buried deep within very large databases, which sometimes contain data from several years. In many cases, the data are cleansed and consolidated into a data warehouse. Data can be presented in a variety of formats (see Chapter 3 for a brief taxonomy of data).
• The data mining environment is usually a client/server architecture or a Web-based IS architecture.
• Sophisticated new tools, including advanced visualization tools, help remove the information ore buried in corporate files or archival public records. Finding it involves massaging and synchronizing the data to get the right results. Cutting-edge data min- ers are also exploring the usefulness of soft data (i.e., unstructured text stored in such places as Lotus Notes databases, text files on the Internet, or enterprisewide intranets).
202 Part II • Predictive Analytics/Machine Learning
• The miner is often an end user empowered by data drills and other powerful query tools to ask ad hoc questions and obtain answers quickly with little or no programming skill.
• “Striking it rich” often involves finding an unexpected result and requires end users to think creatively throughout the process, including the interpretation of the findings.
• Data mining tools are readily combined with spreadsheets and other software devel- opment tools. Thus, the mined data can be analyzed and deployed quickly and easily.
• Because of the large amounts of data and massive search efforts, it is sometimes necessary to use parallel processing for data mining.
A company that effectively leverages data mining tools and technologies can ac- quire and maintain a strategic competitive advantage. Data mining offers organizations an indispensable decision-enhancing environment to exploit new opportunities by trans- forming data into a strategic weapon. See Nemati and Barko (2001) for a more detailed discussion on the strategic benefits of data mining.
How Data Mining Works
Using existing and relevant data obtained from within and outside the organization, data mining builds models to discover patterns among the attributes presented in the data set. Models are the mathematical representations (simple linear relationships/affinities and/or complex and highly nonlinear relationships) that identify the patterns among the attributes of the things (e.g., customers, events) described within the data set. Some of these patterns are explanatory (explaining the interrelationships and affinities among the attributes), whereas others are predictive (foretelling future values of certain attributes). In general, data mining seeks to identify four major types of patterns:
1. Associations find the commonly co-occurring groupings of things, such as beer and diapers going together in market-basket analysis.
Statistics
Artificial Intelligence
Information Visualization
Database Management
& Data Warehousing
Management Science & Information Systems
Machine Learning &
Pattern Recognition
DATA MINING (Knowledge Discovery)
FIGURE 4.1 Data Mining is a Blend of Multiple Disciplines.
Chapter 4 • Data Mining Process, Methods, and Algorithms 203
2. Predictions tell the nature of future occurrences of certain events based on what has happened in the past, such as predicting the winner of the Super Bowl or forecasting the absolute temperature of a particular day.
3. Clusters identify natural groupings of things based on their known characteristics, such as assigning customers in different segments based on their demographics and past purchase behaviors.
4. Sequential relationships discover time-ordered events, such as predicting that an existing banking customer who already has a checking account will open a savings account followed by an investment account within a year.
Application Case 4.2 shows how American Honda uses data mining (a critical component of advanced analytics tools) to enhance their understanding of the warranty claims, forecast feature part and resource needs, and better understand customer needs, wants, and opinions.
Background
When a car or truck owner brings a vehicle into an Acura or Honda dealership in the United States, there’s more to the visit than a repair or a service check. During each visit, the service technicians gen- erate data on the repairs, including any warranty claims to American Honda Motor Co., Inc., that feed directly into its database. This includes what type of work was performed, what the customer paid, ser- vice advisor comments, and many other data points.
Now, multiply this process by dozens of visits a day at over 1,200 dealerships nationwide, and it’s clear—American Honda has big data. It’s up to peo- ple like Kendrick Kau, assistant manager of American Honda’s Advanced Analytics group, to draw insights from this data and turn it into a useful asset.
Examining Warranty Data to Make Maintenance More Efficient
Like any other major automobile distributor, American Honda works with a network of dealer- ships that perform warrantied repair work on its vehicles. This can be a significant cost for the com- pany, so American Honda uses analytics to make sure that warranty claims are complete and accurate upon submission.
In the case of warranty claims, Kau’s team helps empower dealers to understand the appro- priate warranty processes by providing them with useful information via an online report. To support a goal of reducing inappropriate warranty costs,
Kau and his team must sift through information on repairs, parts, customers, and other details. They chose a visual approach to business intelligence and analytics, powered by SAS, to identify cost reduction opportunities.
To decrease warranty expense, the Advanced Analytics team used SAS Analytics to create a propri- etary process to surface suspicious warranty claims for scrutiny on a daily basis to make sure they are in compliance with existing guidelines. The effort to identify and scrutinize claims was once fairly man- ual, tedious, and time-intensive.
“Before SAS, it took one of our staff members one week out of each month to aggregate and report warranty data within Microsoft Excel spreadsheets,” Kau says. “Now, with SAS, we populate those same reports on an easily accessible online dashboard automatically, and we recovered a week of man- power that we could put on other projects.”
By applying SAS Analytics to warranty data, the Advanced Analytics group gave the Claims group and field personnel the ability to quickly and accu- rately identify claims that were incomplete, inaccu- rate, or noncompliant. The results were impressive.
“Initially, it took our examiners over three min- utes on average to identify a potentially noncompli- ant claim, and even then, they were only finding a truly noncompliant claim 35 percent of the time,” Kau says. “Now, with SAS, it takes less than a minute to identify a suspicious claim. And in that time, they are finding a noncompliant claim 76 percent of the time.”
Application Case 4.2 American Honda Uses Advanced Analytics to Improve Warranty Claims
(Continued )
204 Part II • Predictive Analytics/Machine Learning
The effort to increase warranty compliance has paid off for American Honda. Through more com- plete analysis of warranty claims—and more edu- cation at the dealerships—American Honda saw a reduction in labor costs for 52 percent of its available labor codes.
Using Service Data to Forecast Future Needs
The American Honda Advanced Analytics team also uses service and parts data to develop stronger bonds with customers by ensuring dealers have in-demand parts available for customer repairs. Having the right parts available—at the right time—is paramount, so vehicle repairs data feed directly into American Honda’s marketing and customer retention efforts.
“For the marketing team, we provide strate- gic insight to help shape their programs that are designed to drive customers to the dealers, and ulti- mately, keep them loyal to our brand,” Kau says. “The goal of Honda is lifetime owner loyalty. We want our customers to have a good experience, and one of the ways to do that is through exceptional service.”
American Honda uses SAS Forecast Server to assist with business planning to ensure adequate resources are available to meet future demands for services. Using historical information on repair orders and certifications, they developed a time series using years of previous repairs. By combining time series information with sales data, Kau’s team can project where the company’s greatest opportu- nities are in the years ahead.
“Our goal is to forecast the number of vehi- cles in operation in order to predict the volume of customers coming into the dealerships,” Kau says.
“And that translates to how many parts we should have on hand and helps us to plan staffing to meet customer demands. Looking backward on a year-by- year basis, we’ve been within 1 percent of where we forecast to be. That’s extremely good for a forecast, and I attribute much of that to the abilities of the SAS software.”
Customer Feedback that Drives the Business
Another way American Honda uses analytics is to quickly evaluate customer survey data. Using SAS, the Advanced Analytics team mines survey data to gain insight into how vehicles are being used and identify design changes that are most likely to improve customer satisfaction.
On a weekly basis, the analytics team exam- ines customer survey data. Kau’s team uses SAS to flag emerging trends that may require the attention of design, manufacturing, engineering, or other groups. With SAS technology, users can drill down from high-level issues to more specific responses to understand a potential root cause.
“We can look into the data and see what the customers are saying,” Kau says. “And that leads to a number of questions that we can tackle. Is a component designed in the most optimal way? Is it a customer education issue? Is it something that we should address at the manufacturing process? Because of SAS, these are critical questions that we can now identify using our data.”
Questions for Case 4.2
1. How does American Honda use analytics to improve warranty claims?
2. In addition to warranty claims, for what other purposes does American Honda use advanced analytics methods?
3. Can you think of other uses of advanced analyt- ics in the automotive industry? You can search the Web to find some answers to this question.
Source: SAS Case Study “American Honda Motor Co., Inc. uses SAS advanced analytics to improve warranty claims” https:// www.sas.com/en_us/customers/american-honda.html (accessed June 2018)
Application Case 4.2 (Continued)
Honda–Facts & Figures
Faster claims analysis
Reduced labor costs
3X $$ 1200
Dealerships nationwide
Chapter 4 • Data Mining Process, Methods, and Algorithms 205
These types of patterns have been manually extracted from data by humans for centuries, but the increasing volume of data in modern times has created the need for more automatic approaches. As data sets have grown in size and complexity, direct manual data analysis has increasingly been augmented with indirect, automatic data- processing tools that use sophisticated methodologies, methods, and algorithms. The manifestation of such evolution of automated and semiautomated means of processing large data sets is now commonly referred to as data mining.
Generally speaking, data mining tasks can be classified into three main categories: prediction, association, and clustering. Based on the way in which the patterns are ex- tracted from the historical data, the learning algorithms of data mining methods can be classified as either supervised or unsupervised. With supervised learning algorithms, the training data includes both the descriptive attribute (i.e., independent variables or deci- sion variables) and the class attribute (i.e., output variable or result variable). In contrast, with unsupervised learning, the training includes only descriptive attributes. Figure 4.2 shows a simple taxonomy for data mining tasks along with the learning methods and popular algorithms for each of the data mining tasks.
PREDICTION Prediction is commonly referred to as the act of telling about the future. It differs from simple guessing by taking into account the experiences, opinions, and other relevant information in conducting the task of foretelling. A term that is commonly associ- ated with prediction is forecasting. Even though many believe that these two terms are synonymous, there is a subtle but critical difference between the two. Whereas prediction is largely experience and opinion based, forecasting is data and model based. That is, in order of increasing reliability, one might list the relevant terms as guessing, predicting, and forecasting, respectively. In data mining terminology, prediction and forecasting are used synonymously, and the term prediction is used as the common representation of the act. Depending on the nature of what is being predicted, prediction can be named more specifically as classification (where the predicted thing, such as tomorrow’s forecast, is a class label such as “rainy” or “sunny”) or regression (where the predicted thing, such as tomorrow’s temperature, is a real number, such as “65°F”).
CLASSIFICATION Classification, or supervised induction, is perhaps the most common of all data mining tasks. The objective of classification is to analyze historical data stored in a database and automatically generate a model that can predict future behavior. This induced model consists of generalizations over the records of a training data set, which help distin- guish predefined classes. The hope is that the model can then be used to predict the classes of other unclassified records and, more importantly, to accurately predict actual future events.
Common classification tools include neural networks and decision trees (from machine learning), logistic regression and discriminant analysis (from traditional statis- tics), and emerging tools such as rough sets, support vector machines (SVMs), and ge- netic algorithms. Statistics-based classification techniques (e.g., logistic regression and discriminant analysis) have received their share of criticism—that they make unrealistic assumptions about the data, such as independence and normality—which limit their use in classification-type data mining projects.
Neural networks involve the development of mathematical structures (somewhat resembling the biological neural networks in the human brain) that have the capability to learn from past experiences presented in the form of well-structured data sets. They tend to be more effective when the number of variables involved is rather large and the relationships among them are complex and imprecise. Neural networks have disadvan- tages as well as advantages. For example, providing a good rationale for the predictions made by a neural network is usually very difficult. Also, training neural networks usually
206 Part II • Predictive Analytics/Machine Learning
Data Mining Algorithms
K-means, Expectation Maximization (EM)
Autoregressive Methods, Averaging Methods, Exponential Smoothing, ARIMA
Expectation Maximization, Apriori Algorithm, Graph-Based Matching
Apriori, OneR, ZeroR, Eclat, GA
Linear/Nonlinear Regression, ANN, Regression Trees, SVM, kNN, GA
Decision Trees, Neural Networks, Support Vector Machines, kNN, Naïve Bayes, GA
Data Mining Tasks and Methods
Prediction
Classification
Regression
Segmentation
Association
Link Analysis
Sequence Analysis
Clustering
Apriori Algorithm, FP-Growth, Graph-Based Matching
Time series
Market-Basket
Outlier Analysis
Learning Type
K-means, Expectation Maximization (EM)
Supervised
Unsupervised
Supervised
Supervised
Unsupervised
Unsupervised
Unsupervised
Unsupervised
FIGURE 4.2 Simple Taxonomy for Data Mining Tasks, Methods, and Algorithms.
takes a considerable amount of time. Unfortunately, the time needed for training tends to increase exponentially as the volume of data increases, and in general, neural networks cannot be trained on very large databases. These and other factors have limited the ap- plicability of neural networks in data-rich domains.
Decision trees classify data into a finite number of classes based on the values of the input variables. Decision trees are essentially a hierarchy of if-then statements and are thus significantly faster than neural networks. They are most appropriate for categorical data and interval data. Therefore, incorporating continuous variables into a decision
Chapter 4 • Data Mining Process, Methods, and Algorithms 207
tree framework requires discretization, that is, converting continuous valued numerical variables to ranges and categories.
A related category of classification tools is rule induction. Unlike with a decision tree, with rule induction the if-then statements are induced from the training data directly, and they need not be hierarchical in nature. Other, more recent techniques such as SVM, rough sets, and genetic algorithms are gradually finding their way into the arsenal of clas- sification algorithms.
CLUSTERING Clustering partitions a collection of things (e.g., objects, events, presented in a structured data set) into segments (or natural groupings) whose members share simi- lar characteristics. Unlike in classification, in clustering, the class labels are unknown. As the selected algorithm goes through the data set, identifying the commonalities of things based on their characteristics, the clusters are established. Because the clusters are de- termined using a heuristic-type algorithm and because different algorithms could end up with different sets of clusters for the same data set, before the results of clustering tech- niques are put to actual use, it could be necessary for an expert to interpret, and poten- tially modify, the suggested clusters. After reasonable clusters have been identified, they can be used to classify and interpret new data.
Not surprisingly, clustering techniques include optimization. The goal of clustering is to create groups so that the members within each group have maximum similarity and the members across groups have minimum similarity. The most commonly used cluster- ing techniques include k-means (from statistics) and self-organizing maps (from machine learning), which is a unique neural network architecture developed by Kohonen (1982).
Firms often effectively use their data mining systems to perform market segmenta- tion with cluster analysis. Cluster analysis is a means of identifying classes of items so that items in a cluster have more in common with each other than with items in other clusters. Cluster analysis can be used in segmenting customers and directing appropriate market- ing products to the segments at the right time in the right format at the right price. Cluster analysis is also used to identify natural groupings of events or objects so that a common set of characteristics of these groups can be identified to describe them.
ASSOCIATIONS Associations, or association rule learning in data mining, is a popular and well-researched technique for discovering interesting relationships among variables in large databases. Thanks to automated data-gathering technologies such as bar code scanners, the use of association rules for discovering regularities among products in large- scale transactions recorded by point-of-sale systems in supermarkets has become a com- mon knowledge discovery task in the retail industry. In the context of the retail industry, association rule mining is often called market-basket analysis.
Two commonly used derivatives of association rule mining are link analysis and sequence mining. With link analysis, the linkage among many objects of interest is dis- covered automatically, such as the link between Web pages and referential relationships among groups of academic publication authors. With sequence mining, relationships are examined in terms of their order of occurrence to identify associations over time. Algorithms used in association rule mining include the popular Apriori (where frequent itemsets are identified) and FP-Growth, OneR, ZeroR, and Eclat.
VISUALIZATION AND TIME-SERIES FORECASTING Two techniques often associated with data mining are visualization and time-series forecasting. Visualization can be used in conjunction with other data mining techniques to gain a clearer understanding of underly- ing relationships. As the importance of visualization has increased in recent years, a new term, visual analytics, has emerged. The idea is to combine analytics and visualization in a single environment for easier and faster knowledge creation. Visual analytics is covered in
208 Part II • Predictive Analytics/Machine Learning
detail in Chapter 3. In time-series forecasting, the data consist of values of the same vari- able that are captured and stored over time in regular intervals. These data are then used to develop forecasting models to extrapolate the future values of the same variable.
Data Mining Versus Statistics
Data mining and statistics have a lot in common. They both look for relationships within data. Most people call statistics the “foundation of data mining.” The main difference between the two is that statistics starts with a well-defined proposition and hypothesis whereas data mining starts with a loosely defined discovery statement. Statistics collects sample data (i.e., primary data) to test the hypothesis whereas data mining and analyt- ics use all the existing data (i.e., often observational, secondary data) to discover novel patterns and relationships. Another difference comes from the size of data that they use. Data mining looks for data sets that are as “big” as possible, whereas statistics looks for the right size of data (if the data are larger than what is needed/required for the statisti- cal analysis, a sample of them is used). The meaning of “large data” is rather different between statistics and data mining. A few hundred to a thousand data points are large enough to a statistician, but several million to a few billion data points are considered large for data mining studies.
uSECTION 4.2 REVIEW QUESTIONS
1. Define data mining. Why are there many different names and definitions for data mining?
2. What recent factors have increased the popularity of data mining? 3. Is data mining a new discipline? Explain. 4. What are some major data mining methods and algorithms? 5. What are the key differences between the major data mining tasks?
4.3 DATA MINING APPLICATIONS
Data mining has become a popular tool in addressing many complex business problems and opportunities. It has been proven to be very successful and helpful in many areas, some of which are shown by the following representative examples. The goal of many of these business data mining applications is to solve a pressing problem or to explore an emerging business opportunity to create a sustainable competitive advantage.
• Customer relationship management. CRM is the extension of traditional mar- keting. The goal of CRM is to create one-on-one relationships with customers by developing an intimate understanding of their needs and wants. As businesses build relationships with their customers over time through a variety of interactions (e.g., product inquiries, sales, service requests, warranty calls, product reviews, social media connections), they accumulate tremendous amounts of data. When combined with demographic and socioeconomic attributes, this information-rich data can be used to (1) identify most likely responders/buyers of new products/services (i.e., customer profiling), (2) understand the root causes of customer attrition to improve customer retention (i.e., churn analysis), (3) discover time-variant associations be- tween products and services to maximize sales and customer value, and (4) identify the most profitable customers and their preferential needs to strengthen relation- ships and to maximize sales.
• Banking. Data mining can help banks with the following: (1) automating the loan application process by accurately predicting the most probable defaulters,
Chapter 4 • Data Mining Process, Methods, and Algorithms 209
(2) detecting fraudulent credit card and online banking transactions, (3) identifying ways to maximize value for customers by selling them products and services that they are most likely to buy, and (4) optimizing the cash return by accurately fore- casting the cash flow on banking entities (e.g., ATM machines, banking branches).
• Retailing and logistics. In the retailing industry, data mining can be used to (1) predict accurate sales volumes at specific retail locations to determine correct inven- tory levels, (2) identify sales relationships between different products (with market- basket analysis) to improve the store layout and optimize sales promotions, (3) forecast consumption levels of different product types (based on seasonal and environmental conditions) to optimize logistics and, hence, maximize sales, and (4) discover interest- ing patterns in the movement of products (especially for products that have a limited shelf life because they are prone to expiration, perishability, and contamination) in a supply chain by analyzing sensory and radio-frequency identification (RFID) data.
• Manufacturing and production. Manufacturers can use data mining to (1) pre- dict machinery failures before they occur through the use of sensory data (enabling what is called condition-based maintenance), (2) identify anomalies and common- alities in production systems to optimize manufacturing capacity, and (3) discover novel patterns to identify and improve product quality.
• Brokerage and securities trading. Brokers and traders use data mining to (1) predict when and how much certain bond prices will change, (2) forecast the range and direction of stock fluctuations, (3) assess the effect of particular issues and events on overall market movements, and (4) identify and prevent fraudulent activities in securities trading.
• Insurance. The insurance industry uses data mining techniques to (1) forecast claim amounts for property and medical coverage costs for better business planning, (2) determine optimal rate plans based on the analysis of claims and customer data, (3) predict which customers are more likely to buy new policies with special fea- tures, and (4) identify and prevent incorrect claim payments and fraudulent activities.
• Computer hardware and software. Data mining can be used to (1) predict disk drive failures well before they actually occur, (2) identify and filter unwanted Web content and e-mail messages, (3) detect and prevent computer network security breaches, and (4) identify potentially unsecure software products.
• Government and defense. Data mining also has a number of military appli- cations. It can be used to (1) forecast the cost of moving military personnel and equipment, (2) predict an adversary’s moves and, hence, develop more successful strategies for military engagements, (3) predict resource consumption for better planning and budgeting, and (4) identify classes of unique experiences, strategies, and lessons learned from military operations for better knowledge sharing through- out the organization.
• Travel industry (airlines, hotels/resorts, rental car companies). Data mining has a variety of uses in the travel industry. It is successfully used to (1) predict sales of different services (seat types in airplanes, room types in hotels/resorts, car types in rental car companies) in order to optimally price services to maximize revenues as a function of time-varying transactions (commonly referred to as yield management), (2) forecast demand at different locations to better allocate limited organizational resources, (3) identify the most profitable customers and provide them with person- alized services to maintain their repeat business, and (4) retain valuable employees by identifying and acting on the root causes for attrition.
• Healthcare. Data mining has a number of healthcare applications. It can be used to (1) identify people without health insurance and the factors underlying this unde- sired phenomenon, (2) identify novel cost–benefit relationships between different
210 Part II • Predictive Analytics/Machine Learning
treatments to develop more effective strategies, (3) forecast the level and the time of demand at different service locations to optimally allocate organizational resources, and (4) understand the underlying reasons for customer and employee attrition.
• Medicine. Use of data mining in medicine should be viewed as an invaluable complement to traditional medical research, which is mainly clinical and biological in nature. Data mining analyses can (1) identify novel patterns to improve surviv- ability of patients with cancer, (2) predict success rates of organ transplantation patients to develop better organ donor matching policies, (3) identify the functions of different genes in the human chromosome (known as genomics), and (4) dis- cover the relationships between symptoms and illnesses (as well as illnesses and successful treatments) to help medical professionals make informed and correct decisions in a timely manner.
• Entertainment industry. Data mining is successfully used by the entertainment industry to (1) analyze viewer data to decide what programs to show during prime time and how to maximize returns by knowing where to insert advertisements, (2) predict the financial success of movies before they are produced to make invest- ment decisions and to optimize the returns, (3) forecast the demand at different loca- tions and different times to better schedule entertainment events and to optimally allocate resources, and (4) develop optimal pricing policies to maximize revenues.
• Homeland security and law enforcement. Data mining has a number of home- land security and law enforcement applications. It is often used to (1) identify patterns of terrorist behaviors (see Application Case 4.3 for an example of the use of data min- ing to track funding of terrorists’ activities), (2) discover crime patterns (e.g., locations, timings, criminal behaviors, and other related attributes) to help solve criminal cases in a timely manner, (3) predict and eliminate potential biological and chemical attacks to the nation’s critical infrastructure by analyzing special-purpose sensory data, and (4) identify and stop malicious attacks on critical information infrastructures (often called information warfare).
The terrorist attack on the World Trade Center on September 11, 2001, underlined the importance of open source intelligence. The USA PATRIOT Act and the creation of the U.S. Department of Homeland Security heralded the potential application of infor- mation technology and data mining techniques to detect money laundering and other forms of terror- ist financing. Law enforcement agencies had been focusing on money laundering activities via normal transactions through banks and other financial ser- vice organizations.
Law enforcement agencies are now focusing on international trade pricing as a terrorism fund- ing tool. Money launderers have used international trade to move money silently out of a country with- out attracting government attention. They achieve
this transfer by overvaluing imports and undervalu- ing exports. For example, a domestic importer and foreign exporter could form a partnership and over- value imports, thereby transferring money from the home country, resulting in crimes related to customs fraud, income tax evasion, and money laundering. The foreign exporter could be a member of a terror- ist organization.
Data mining techniques focus on analysis of data on import and export transactions from the U.S. Department of Commerce and commerce- related entities. Import prices that exceed the upper quartile of import prices and export prices that are lower than the lower quartile of export prices are tracked. The focus is on abnormal transfer prices between corporations that might result in shifting
Application Case 4.3 Predictive Analytic and Data Mining Help Stop Terrorist Funding
Chapter 4 • Data Mining Process, Methods, and Algorithms 211
taxable income and taxes out of the United States. An observed price deviation could be related to income tax avoidance/evasion, money laundering, or terrorist financing. The observed price devia- tion could also be due to an error in the U.S. trade database.
Data mining will result in efficient evalua- tion of data, which, in turn, will aid in the fight against terrorism. The application of information technology and data mining techniques to financial transactions can contribute to better intelligence information.
Questions for Case 4.3
1. How can data mining be used to fight terrorism? Comment on what else can be done beyond what is covered in this short application case.
2. Do you think data mining, although essential for fighting terrorist cells, also jeopardizes individu- als’ rights of privacy?
Sources: J. S. Zdanowic, “Detecting Money Laundering and Terrorist Financing via Data Mining,” Communications of the ACM, 47(5), May 2004, p. 53; R. J. Bolton, “Statistical Fraud Detection: A Review,” Statistical Science, 17(3), January 2002, p. 235.
• Sports. Data mining was used to improve the performance of National Basketball Association (NBA) teams in the United States. Major League Baseball teams are into predictive analytics and data mining to optimally utilize their limited resources for a winning season. In fact, most, if not all, professional sports today employ data crunchers and use data mining to increase their chances of winning. Data mining applications are not limited to professional sports. In an article, Delen at al. (2012) developed data mining models to predict National Collegiate Athletic Association (NCAA) Bowl Game outcomes using a wide range of variables about the two op- posing teams’ previous games statistics (more details about this case study are pro- vided in Chapter 3). Wright (2012) used a variety of predictors for examination of the NCAA men’s basketball championship (a.k.a. March Madness) bracket.
uSECTION 4.3 REVIEW QUESTIONS
1. What are the major application areas for data mining? 2. Identify at least five specific applications of data mining and list five common charac-
teristics of these applications.
3. What do you think is the most prominent application area for data mining? Why? 4. Can you think of other application areas for data mining not discussed in this section?
Explain.
4.4 DATA MINING PROCESS
To systematically carry out data mining projects, a general process is usually followed. Based on best practices, data mining researchers and practitioners have proposed sev- eral processes (workflows or simple step-by-step approaches) to maximize the chances of success in conducting data mining projects. These efforts have led to several standard- ized processes, some of which (a few of the most popular ones) are described in this section.
One such standardized process, arguably the most popular one, the Cross-Industry Standard Process for Data Mining—CRISP-DM—was proposed in the mid-1990s by a European consortium of companies to serve as a nonproprietary standard methodology for data mining (CRISP-DM, 2013). Figure 4.3 illustrates this proposed process, which is a sequence of six steps that starts with a good understanding of the business and the
212 Part II • Predictive Analytics/Machine Learning
need for the data mining project (i.e., the application domain) and ends with the deploy- ment of the solution that satisfies the specific business need. Even though these steps are sequential in nature, there is usually a great deal of backtracking. Because data mining is driven by experience and experimentation, depending on the problem situation and the knowledge/experience of the analyst, the whole process can be very iterative (i.e., one should expect to go back and forth through the steps quite a few times) and time consuming. Because later steps are built on the outcomes of the former ones, one should pay extra attention to the earlier steps in order to not put the whole study on an incorrect path from the onset.
Step 1: Business Understanding
The key element of any data mining study is to know what the study is for. Determining this begins with a thorough understanding of the managerial need for new knowledge and an explicit specification of the business objective regarding the study to be con- ducted. Specific goals answering questions such as “What are the common characteristics of the customers we have lost to our competitors recently?” or “What are typical profiles of our customers, and how much value does each of them provide to us?” are needed. Then a project plan for finding such knowledge is developed that specifies the people responsible for collecting the data, analyzing the data, and reporting the findings. At this early stage, a budget to support the study should also be established at least at a high level with rough numbers.
Step 2: Data Understanding
A data mining study is specific to addressing a well-defined business task, and differ- ent business tasks require different sets of data. Following the business understanding
Business Understanding
1
Data Understanding
2
Data Preparation
3
Model Building
4
Testing and Evaluation
5
Deployment
6
Data
FIGURE 4.3 Six-Step CRISP-DM Data Mining Process.
Chapter 4 • Data Mining Process, Methods, and Algorithms 213
step, the main activity of the data mining process is to identify the relevant data from many available databases. Some key points must be considered in the data iden- tification and selection phase. First and foremost, the analyst should be clear and concise about the description of the data mining task so that the most relevant data can be identified. For example, a retail data mining project could seek to identify spending behaviors of female shoppers who purchase seasonal clothes based on their demographics, credit card transactions, and socioeconomic attributes. Furthermore, the analyst should build an intimate understanding of the data sources (e.g., where the relevant data are stored and in what form; what the process of collecting the data is—automated versus manual; who the collectors of the data are and how often the data are updated) and the variables (e.g., What are the most relevant variables? Are there any synonymous and/or homonymous variables? Are the variables independent of each other—do they stand as a complete information source without overlapping or conflicting information?).
To better understand the data, the analyst often uses a variety of statistical and graphical techniques, such as simple statistical summaries of each variable (e.g., for nu- meric variables, the average, minimum/maximum, median, and standard deviation are among the calculated measures whereas for categorical variables, the mode and fre- quency tables are calculated), and correlation analysis, scatterplots, histograms, and box plots can be used. A careful identification and selection of data sources and the most relevant variables can make it easier for data mining algorithms to quickly discover useful knowledge patterns.
Data sources for data selection can vary. Traditionally, data sources for business applications include demographic data (such as income, education, number of house- holds, and age), sociographic data (such as hobby, club membership, and entertain- ment), transactional data (sales record, credit card spending, issued checks), and so on. Today, data sources also use external (open or commercial) data repositories, social media, and machine-generated data.
Data can be categorized as quantitative and qualitative. Quantitative data are mea- sured using numeric values, or numeric data. They can be discrete (such as integers) or continuous (such as real numbers). Qualitative data, also known as categorical data, contain both nominal and ordinal data. Nominal data have finite nonordered values (e.g., gender data, which have two values: male and female). Ordinal data have finite ordered values. For example, customer credit ratings are considered ordinal data because the ratings can be excellent, fair, and bad. A simple taxonomy of data (i.e., the nature of data) is provided in Chapter 3.
Quantitative data can be readily represented by some sort of probability distribu- tion. A probability distribution describes how the data are dispersed and shaped. For instance, normally distributed data are symmetric and are commonly referred to as being a bell-shaped curve. Qualitative data can be coded to numbers and then described by frequency distributions. Once the relevant data are selected according to the data mining business objective, data preprocessing should be pursued.
Step 3: Data Preparation
The purpose of data preparation (more commonly called data preprocessing) is to take the data identified in the previous step and prepare it for analysis by data mining meth- ods. Compared to the other steps in CRISP-DM, data preprocessing consumes the most time and effort; most people believe that this step accounts for roughly 80 percent of the total time spent on a data mining project. The reason for such an enormous effort spent on this step is the fact that real-world data are generally incomplete (lacking at- tribute values, lacking certain attributes of interest, or containing only aggregate data),
214 Part II • Predictive Analytics/Machine Learning
noisy (containing errors or outliers), and inconsistent (containing discrepancies in codes or names). The nature of the data and the issues related to the preprocessing of data for analytics are explained in detail in Chapter 3.
Step 4: Model Building
In this step, various modeling techniques are selected and applied to an already prepared data set to address the specific business need. The model-building step also encompasses the assessment and comparative analysis of the various models built. Because there is not a universally known best method or algorithm for a data mining task, one should use a variety of viable model types along with a well-defined experimentation and assessment strategy to identify the “best” method for a given purpose. Even for a single method or algorithm, a number of parameters need to be calibrated to obtain optimal results. Some methods could have specific requirements in the way that the data are to be formatted; thus, stepping back to the data preparation step is often necessary. Application Case 4.4 presents a research study in which a number of model types are developed and com- pared to each other.
According to the American Cancer Society, half of all men and one-third of all women in the United States will develop cancer during their lifetimes; approxi- mately 1.5 million new cancer cases were expected to be diagnosed in 2013. Cancer is the second most common cause of death in the United States and in the world, exceeded only by cardiovascular dis- ease. This year, more than 500,000 Americans are expected to die of cancer—more than 1,300 peo- ple a day—accounting for nearly one of every four deaths.
Cancer is a group of diseases generally char- acterized by uncontrolled growth and spread of abnormal cells. If the growth and/or spread are not controlled, cancer can result in death. Even though the exact reasons are not known, cancer is believed to be caused by both external factors (e.g., tobacco, infectious organisms, chemicals, and radiation) and internal factors (e.g., inherited mutations, hormones, immune conditions, and mutations that occur from metabolism). These causal factors can act together or in sequence to initiate or promote carcinogenesis. Cancer is treated with surgery, radiation, chemo- therapy, hormone therapy, biological therapy, and targeted therapy. Survival statistics vary greatly by cancer type and stage at diagnosis.
The five-year relative survival rate for all can- cers is improving, and the decline in cancer mortal- ity had reached 20 percent in 2013, translating into the avoidance of about 1.2 million deaths from can- cer since 1991. That’s more than 400 lives saved per day! The improvement in survival reflects progress in diagnosing certain cancers at an earlier stage and improvements in treatment. Further improvements are needed to prevent and treat cancer.
Even though cancer research has traditionally been clinical and biological in nature, in recent years, data-driven analytic studies have become a common complement. In medical domains where data- and analytics-driven research has been applied success- fully, novel research directions have been identified to further advance the clinical and biological stud- ies. Using various types of data, including molecular, clinical, literature-based, and clinical trial data, along with suitable data mining tools and techniques, researchers have been able to identify novel pat- terns, paving the road toward a cancer-free society.
In one study, Delen (2009) used three popular data mining techniques (decision trees, artificial neu- ral networks, and SVMs) in conjunction with logistic regression to develop prediction models for prostate cancer survivability. The data set contained around
Application Case 4.4 Data Mining Helps in Cancer Research
Chapter 4 • Data Mining Process, Methods, and Algorithms 215
120,000 records and 77 variables. A k-fold cross- validation methodology was used in model build- ing, evaluation, and comparison. The results showed that support vector models are the most accurate predictor (with a test set accuracy of 92.85%) for this domain followed by artificial neural networks and decision trees. Furthermore, using a sensitivity– analysis-based evaluation method, the study also revealed novel patterns related to prognostic factors of prostate cancer.
In a related study, Delen, Walker, and Kadam (2005) used two data mining algorithms (artificial neural networks and decision trees) and logistic regression to develop prediction models for breast cancer survival using a large data set (more than 200,000 cases). Using a 10-fold cross-validation method to measure the unbiased estimate of the prediction models for performance comparison pur- poses, the researchers determined that the results indicated that the decision tree (C5 algorithm) was the best predictor with 93.6 percent accuracy on the holdout sample (which was the best prediction accuracy reported in the literature) followed by arti- ficial neural networks with 91.2 percent accuracy, and logistic regression, with 89.2 percent accu- racy. Further analysis of prediction models revealed prioritized importance of the prognostic factors, which can then be used as a basis for further clinical and biological research studies.
In the most recent study, Zolbanin et al. (2015) studied the impact of comorbidity in cancer sur- vivability. Although prior research has shown that diagnostic and treatment recommendations might be altered based on the severity of comorbidities, chronic diseases are still being investigated in isola- tion from one another in most cases. To illustrate the significance of concurrent chronic diseases in the course of treatment, their study used the Surveillance, Epidemiology, and End Results (SEER) Program’s cancer data to create two comorbid data sets: one for breast and female genital cancers and another for prostate and urinal cancers. Several pop- ular machine-learning techniques are then applied to the resultant data sets to build predictive mod- els (see Figure 4.4). Comparison of the results has
shown that having more information about comor- bid conditions of patients can improve models’ pre- dictive power, which in turn can help practitioners make better diagnostic and treatment decisions. Therefore, the study suggested that proper identi- fication, recording, and use of patients’ comorbidity status can potentially lower treatment costs and ease the healthcare-related economic challenges.
These examples (among many others in the medical literature) show that advanced data min- ing techniques can be used to develop models that possess a high degree of predictive as well as explanatory power. Although data mining meth- ods are capable of extracting patterns and relation- ships hidden deep in large and complex medical databases, without the cooperation and feedback from medical experts, their results are not of much use. The patterns found via data mining methods should be evaluated by medical professionals who have years of experience in the problem domain to decide whether they are logical, actionable, and novel enough to warrant new research directions. In short, data mining is not meant to replace medi- cal professionals and researchers but to comple- ment their invaluable efforts to provide data-driven new research directions and to ultimately save more human lives.
Questions for Case 4.4
1. How can data mining be used for ultimately cur- ing illnesses like cancer?
2. What do you think are the promises and major challenges for data miners in contributing to medical and biological research endeavors?
Sources: H. M. Zolbanin, D. Delen, & A. H. Zadeh, “Predicting Overall Survivability in Comorbidity of Cancers: A Data Mining Approach,” Decision Support Systems, 74, 2015, pp. 150–161; D. Delen, “Analysis of Cancer Data: A Data Mining Approach,” Expert Systems, 26(1), 2009, pp. 100–112; J. Thongkam, G. Xu, Y. Zhang, & F. Huang, “Toward Breast Cancer Survivability Prediction Models Through Improving Training Space,” Expert Systems with Applications, 36(10), 2009, pp. 12200–12209; D. Delen, G. Walker, & A. Kadam, “Predicting Breast Cancer Survivability: A Comparison of Three Data Mining Methods,” Artificial Intelligence in Medicine, 34(2), 2005, pp. 113–127.
(Continued )
216 Part II • Predictive Analytics/Machine Learning
Training and calibrating the
model
Testing the model
Artificial Neural Networks (ANN)
Tabulated Model Testing Results
(Accuracy, Sensitivity, and
Specificity)
Partitioned data (training &
testing)
Partitioned data (training
& testing)
Training and calibrating the
model
Testing the model
Logistic Regression (LR)
Training and calibrating the
model
Testing the model
Random Forest (RF)
Assess variable
importance
Tabulated Relative Variable
Importance Results
Data Preprocessing Cleaning Selecting Transforming
Cancer DB 1 Cancer DB 2 Cancer DB n
Combined Cancer DB
Partitioned data (training
& testing)
FIGURE 4.4 Data Mining Methodology for Investigation of Comorbidity in Cancer Survivability.
Depending on the business need, the data mining task can be of a prediction (either classification or regression), an association, or a clustering type. Each of these data min- ing tasks can use a variety of data mining methods and algorithms. Some of these data mining methods were explained earlier in this chapter, and some of the most popular
Application Case 4.4 (Continued)
Chapter 4 • Data Mining Process, Methods, and Algorithms 217
algorithms, including decision trees for classification, k-means for clustering, and the Apriori algorithm for association rule mining, are described later in this chapter.
Step 5: Testing and Evaluation
In step 5, the developed models are assessed and evaluated for their accuracy and gen- erality. This step assesses the degree to which the selected model (or models) meets the business objectives and, if so, to what extent (i.e., Do more models need to be developed and assessed?). Another option is to test the developed model(s) in a real- world scenario if time and budget constraints permit. Even though the outcome of the developed models is expected to relate to the original business objectives, other find- ings that are not necessarily related to the original business objectives but that might also unveil additional information or hints for future directions often are discovered.
The testing and evaluation step is a critical and challenging task. No value is added by the data mining task until the business value obtained from discovered knowledge patterns is identified and recognized. Determining the business value from discovered knowledge patterns is somewhat similar to playing with puzzles. The extracted knowledge patterns are pieces of the puzzle that need to be put together in the context of the specific business purpose. The success of this identification operation depends on the interaction among data analysts, business analysts, and decision makers (such as business managers). Because data analysts might not have the full understanding of the data mining objectives and what they mean to the business and the business analysts, and decision makers might not have the technical knowledge to interpret the results of sophisticated mathematical solutions, interaction among them is necessary. To properly interpret knowledge patterns, it is often necessary to use a variety of tabulation and visualization techniques (e.g., pivot tables, cross-tabulation of findings, pie charts, histograms, box plots, scatterplots).
Step 6: Deployment
Development and assessment of the models is not the end of the data mining project. Even if the purpose of the model is to have a simple exploration of the data, the knowl- edge gained from such exploration will need to be organized and presented in a way that the end user can understand and benefit from. Depending on the requirements, the de- ployment phase can be as simple as generating a report or as complex as implementing a repeatable data mining process across the enterprise. In many cases, it is the customer, not the data analyst, who carries out the deployment steps. However, even if the analyst will not carry out the deployment effort, it is important for the customer to understand up front what actions need to be carried out to actually make use of the created models.
The deployment step can also include maintenance activities for the deployed models. Because everything about the business is constantly changing, the data that reflect the busi- ness activities also are changing. Over time, the models (and the patterns embedded within them) built on the old data can become obsolete, irrelevant, or misleading. Therefore, moni- toring and maintenance of the models are important if the data mining results are to become a part of the day-to-day business and its environment. A careful preparation of a maintenance strategy helps avoid unnecessarily long periods of incorrect usage of data mining results. To monitor the deployment of the data mining result(s), the project needs a detailed plan on the monitoring process, which might not be a trivial task for complex data mining models.
Other Data Mining Standardized Processes and Methodologies
To be applied successfully, a data mining study must be viewed as a process that follows a standardized methodology rather than as a set of automated software tools and tech- niques. In addition to CRISP-DM, there is another well-known methodology developed
218 Part II • Predictive Analytics/Machine Learning
by the SAS Institute, called SEMMA (2009). The acronym SEMMA stands for “sample, explore, modify, model, and assess.”
Beginning with a statistically representative sample of the data, SEMMA makes it easy to apply exploratory statistical and visualization techniques, select and transform the most significant predictive variables, model the variables to predict outcomes, and confirm a model’s accuracy. A pictorial representation of SEMMA is given in Figure 4.5.
By assessing the outcome of each stage in the SEMMA process, the model devel- oper can determine how to model new questions raised by the previous results and thus proceed back to the exploration phase for additional refinement of the data; that is, as with CRISP-DM, SEMMA is driven by a highly iterative experimentation cycle. The main difference between CRISP-DM and SEMMA is that CRISP-DM takes a more comprehensive approach—including understanding of the business and the relevant data—to data mining projects whereas SEMMA implicitly assumes that the data mining project’s goals and objectives along with the appropriate data sources have been identi- fied and understood.
Some practitioners commonly use the term knowledge discovery in databases (KDD) as a synonym for data mining. Fayyad et al. (1996) defined knowledge discov- ery in databases as a process of using data mining methods to find useful information and patterns in the data as opposed to data mining, which involves using algorithms to identify patterns in data derived through the KDD process (see Figure 4.6). KDD is a comprehensive process that encompasses data mining. The input to the KDD pro- cess consists of organizational data. The enterprise data warehouse enables KDD to be implemented efficiently because it provides a single source for data to be mined. Dunham (2003) summarized the KDD process as consisting of the following steps: data selection, data preprocessing, data transformation, data mining, and interpretation/ evaluation.
Figure 4.7 shows the polling results for the question, “What main methodology are you using for data mining?” (conducted by KDnuggets.com in August 2007).
SEMMA
Feedback
Sample (Generate a representative
sample of the data.)
Explore (Visualize and provide a
basic description of the data.)
Assess (Evaluate the accuracy and usefulness of the models.)
Modify (Select variables, transform variable representations.)
Model (Use a variety of statistical
and machine learning models.)
FIGURE 4.5 SEMMA Data Mining Process.
Chapter 4 • Data Mining Process, Methods, and Algorithms 219
Target Data
Preprocessed Data
1 2 3 4 5
Transformed Data
Extracted Patterns
Knowledge “Actionable
Insight”
Data Selection
Data Cleaning
Data Transformation
Data Mining
Internalization
Feedback
Sources for Raw Data
FIGURE 4.6 KDD (Knowledge Discovery in Databases) Process.
CRISP-DM
My own
SEMMA
KDD Process
Domain-specific methodology
None
Other methodology (not domain specific)
My organization’s
0 10 20 30 40 50 60 70
FIGURE 4.7 Ranking of Data Mining Methodologies/Processes. Source: Used with permission from KDnuggets.com.
220 Part II • Predictive Analytics/Machine Learning
uSECTION 4.4 REVIEW QUESTIONS
1. What are the major data mining processes? 2. Why do you think the early phases (understanding of the business and understand-
ing of the data) take the longest amount of time in data mining projects?
3. List and briefly define the phases in the CRISP-DM process. 4. What are the main data-preprocessing steps? Briefly describe each step and provide
relevant examples.
5. How does CRISP-DM differ from SEMMA?
4.5 DATA MINING METHODS
Various methods are available for performing data mining studies, including classification, regression, clustering, and association. Most data mining software tools employ more than one technique (or algorithm) for each of these methods. This section describes the most popular data mining methods and explains their representative techniques.
Classification
Classification is perhaps the most frequently used data mining method for real-world problems. As a popular member of the machine-learning family of techniques, classifica- tion learns patterns from past data (a set of information—traits, variables, features—on characteristics of the previously labeled items, objects, or events) to place new instances (with unknown labels) into their respective groups or classes. For example, one could use classification to predict whether the weather on a particular day will be “sunny,” “rainy,” or “cloudy.” Popular classification tasks include credit approval (i.e., good or bad credit risk), store location (e.g., good, moderate, bad), target marketing (e.g., likely customer, no hope), fraud detection (i.e., yes/no), and telecommunication (e.g., likely to turn to another phone company, yes/no). If what is being predicted is a class label (e.g., “sunny,” “rainy,” or “cloudy”), the prediction problem is called a classification; if it is a numeric value (e.g., temperature, such as 68°F), the prediction problem is called a regression.
Even though clustering (another popular data mining method) can also be used to determine groups (or class memberships) of things, there is a significant difference between the two. Classification learns the function between the characteristics of things (i.e., independent variables) and their membership (i.e., output variable) through a su- pervised learning process in which both types (input and output) of variables are pre- sented to the algorithm; in clustering, the membership of the objects is learned through an unsupervised learning process by which only the input variables are presented to the algorithm. Unlike classification, clustering does not have a supervising (or controlling) mechanism that enforces the learning process; instead, clustering algorithms use one or more heuristics (e.g., multidimensional distance measure) to discover natural groupings of objects.
The most common two-step methodology of classification-type prediction involves model development/training and model testing/deployment. In the model development phase, a collection of input data, including the actual class labels, is used. After a model has been trained, the model is tested against the holdout sample for accuracy assessment and eventually deployed for actual use where it is to predict classes of new data instances (where the class label is unknown). Several factors are considered in assessing the model, including the following.
• Predictive accuracy. The model’s ability to correctly predict the class label of new or previously unseen data. Prediction accuracy is the most commonly used assessment factor for classification models. To compute this measure, actual class
Chapter 4 • Data Mining Process, Methods, and Algorithms 221
labels of a test data set are matched against the class labels predicted by the model. The accuracy can then be computed as the accuracy rate, which is the percentage of test data set samples correctly classified by the model (more on this topic is pro- vided later in the chapter).
• Speed. The computational costs involved in generating and using the model where faster is deemed to be better.
• Robustness. The model’s ability to make reasonably accurate predictions given noisy data or data with missing and erroneous values.
• Scalability. The ability to construct a prediction model efficiently given a rather large amount of data.
• Interpretability. The level of understanding and insight provided by the model (e.g., how and/or what the model concludes on certain predictions).
Estimating the True Accuracy of Classification Models
In classification problems, the primary source for accuracy estimation is the confusion matrix (also called a classification matrix or a contingency table). Figure 4.8 shows a confusion matrix for a two-class classification problem. The numbers along the diagonal from the upper left to the lower right represent correct decisions, and the numbers out- side this diagonal represent the errors.
Table 4.1 provides equations for common accuracy metrics for classification models. When the classification problem is not binary, the confusion matrix gets bigger (a
square matrix with the size of the unique number of class labels), and accuracy metrics become limited to per class accuracy rates and the overall classifier accuracy.
(True Classification Rate)i = (True Classification)
a n
i = 1 1False Classification2
(Overall Classifier Accuracy)i = a n
i = 1 (Ture Classification)i
Total Number of Cases
Estimating the accuracy of a classification model (or classifier) induced by a super- vised learning algorithm is important for the following two reasons: First, it can be used
True Positive (TP)
Count
False Negative (FN)
Count
True Negative (TN)
Count
False Positive (FP)
Count
Positive
True/Observed Class
N e g a ti
ve P
o s it
iv e
Negative
P re
d ic
te d C
la s s
FIGURE 4.8 Simple Confusion Matrix for Tabulation of Two-Class Classification Results.
222 Part II • Predictive Analytics/Machine Learning
to estimate its future prediction accuracy, which could imply the level of confidence one should have in the classifier’s output in the prediction system. Second, it can be used for choosing a classifier from a given set (identifying the “best” classification model among the many trained). The following are among the most popular estimation methodologies used for classification-type data mining models.
SIMPLE SPLIT The simple split (or holdout or test sample estimation) partitions the data into two mutually exclusive subsets called a training set and a test set (or holdout set). It is common to designate two-thirds of the data as the training set and the remaining one- third as the test set. The training set is used by the inducer (model builder), and the built classifier is then tested on the test set. An exception to this rule occurs when the classifier is an artificial neural network. In this case, the data are partitioned into three mutually ex- clusive subsets: training, validation, and testing. The validation set is used during model building to prevent overfitting. Figure 4.9 shows the simple split methodology.
The main criticism of this method is that it makes the assumption that the data in the two subsets are of the same kind (i.e., have the exact same properties). Because this is a simple random partitioning, in most realistic data sets where the data are skewed on the classification variable, such an assumption might not hold true. To improve this
TABLE 4.1 Common Accuracy Metrics for Classification Models
Metric Description
Accuracy = TP + TN
TP + TN + FP + FN The ratio of correctly classified instances (positives and negatives) divided by the total numbers of instances
True Positive Rate = TP
TP + FN (a.k.a. Sensitivity) The ratio of correctly classified positives divided by the total positive count (i.e., hit rate or recall)
True Negative Rate = TN
TN + FP (a.k.a. Specificity) The ratio of correctly classified negatives divided by the total negative count (i.e., false alarm rate)
Precision = TP
TP + FP The ratio of correctly classified positives divided by the sum of correctly classified positives and incorrectly classified positives
Recall = TP
TP + FN Ratio of correctly classified positives divided by the sum of correctly classified positives and incorrectly classified negatives
Preprocessed Data
Training Data
Model Development
Model Assessment (scoring)
Testing Data
1/3
2/3
Trained Classifier
Prediction Accuracy
FN TN
TP FP
FIGURE 4.9 Simple Random Data Splitting.
Chapter 4 • Data Mining Process, Methods, and Algorithms 223
situation, stratified sampling is suggested, where the strata become the output variable. Even though this is an improvement over the simple split, it still has a bias associated from the single random partitioning.
K-FOLD CROSS-VALIDATION To minimize the bias associated with the random sampling of the training and holdout data samples in comparing the predictive accuracy of two or more methods, one can use a methodology called k-fold cross-validation. In k-fold cross-validation, also called rotation estimation, the complete data set is randomly split into k mutually exclusive subsets of approximately equal size. The classification model is trained and tested k times. Each time it is trained on all but one fold and then tested on the remaining single fold. The cross-validation estimate of the overall accuracy of a model is calculated by simply averaging the k individual accuracy measures, as shown in the following equation:
CVA = 1
k a k
i = 1 Ai
where CVA stands for cross-validation accuracy, k is the number of folds used, and A is the accuracy measure (e.g., hit rate, sensitivity, specificity) of each fold. Figure 4.10 shows a graphical illustration of k-fold cross-validation where k is set to 10.
ADDITIONAL CLASSIFICATION ASSESSMENT METHODOLOGIES Other popular assessment methodologies include the following:
• Leave one out. The leave-one-out method is similar to the k-fold cross-validation where the k takes the value of 1; that is, every data point is used for testing once as many models are developed as there are data points. This is a time-consuming methodology, but sometimes for small data sets, it is a viable option.
• Bootstrapping. With bootstrapping, a fixed number of instances from the origi- nal data are sampled (with replacement) for training, and the rest of the data set is used for testing. This process is repeated as many times as desired.
• Jackknifing. Though similar to the leave-one-out methodology, with jackknifing, the accuracy is calculated by leaving one sample out at each iteration of the estima- tion process.
• Area under the ROC curve. The area under the ROC curve is a graphical assessment technique that plots the true positive rate on the y-axis and the false positive rate on the x-axis. The area under the ROC curve determines the accuracy
...
10% 10%10% 10%
10%10%
10%10%
10%10% 10%
10% 10% 10%
10%10%
10%10%
10%10% 10%
10%
10%
10%10%
10%10%
10%10% 10% Repeated for
all 10 folds
FIGURE 4.10 Graphical Depiction of k-Fold Cross-Validation.
224 Part II • Predictive Analytics/Machine Learning
measure of a classifier: A value of 1 indicates a perfect classifier; 0.5 indicates no better than random chance; in reality, the values would range between the two extreme cases. For example, in Figure 4.11, A has a better classification performance than B, whereas C is not any better than the random chance of flipping a coin.
Estimating the Relative Importance of Predictor Variables
Data mining methods (i.e., machine-learning algorithms) are really good at capturing complex relationships between input and output variables (producing very accurate prediction models) but are not nearly as good at explaining how they do what they do (i.e., model transparency). To mitigate this deficiency (also called the black-box syn- drome), the machine-learning community proposed several methods, most of which are characterized as sensitivity analysis. In the context of predictive modeling, sensitivity analysis refers to an exclusive experimentation process aimed at discovering the cause- and-effect relationship between the input variables and output variable. Some of the variable importance methods are algorithm specific (i.e., applied to decision trees) and some are algorithm agnostic. Here are the most commonly used variable importance methods employed in machine learning and predictive modeling:
1. Developing and observing a well-trained decision tree model to see the relative dis- cernibility of the input variables—the closer to the root of the tree a variable is used to split, the greater is its importance/relative-contribution to the prediction model.
2. Developing and observing a rich and large random forest model and assessing the variable split statistics. If the ratio of a given variable’s selection into candidate counts (i.e., number of times a variable selected as the level-0 splitter is divided by number of times it was picked randomly as one of split candidates) is larger, its importance/ relative-contribution is also greater.
10.90.80.70.60.50.40.30.20.10 0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
1
0.9
0.8
False Positive Rate (1-Specificity)
T ru
e P
o s it
iv e R
a te
( S
e n s it
iv it
y) A
B
C
FIGURE 4.11 Sample ROC Curve.
Chapter 4 • Data Mining Process, Methods, and Algorithms 225
3. Sensitivity analysis based on input value perturbation by which the input variables are gradually changed/perturbed one at a time and the relative change in the output is observed—the larger the change in the output, the greater the importance of the perturbed variable. This method is often used in feed-forward neural network mod- eling when all of the variables are numeric and standardized/normalized. Because this method is covered in Chapter 6 within the context of deep learning and deep neural networks, it is not be explained here.
4. Sensitivity analysis based on leave-one-out methodology. This method can be used for any type of predictive analytics method and therefore is further explained as follows.
The sensitivity analysis (based on leave-one-out methodology) relies on the experi- mental process of systematically removing input variables, one at a time, from the input variable set, developing and testing a model, and observing the impact of the absence of this variable on the predictive performance of the machine-learning model. The model is trained and tested (often using a k-fold cross validation) for each input variable (i.e., its absence in the input variable collection) to measure its contribution/importance to the model. A graphical depiction of the process is shown in Figure 4.12.
This method is often used for support vector machines, decision trees, logistic re- gression, and artificial neural networks. In his sensitivity analysis book, Saltelli (2002) formalized the algebraic representation of this measurement process:
Si = Vi
V (Ft) =
V (E(Ft|Xi))
V (Ft)
In the denominator of the equation, V(Ft) refers to the variance in the output vari- able. In the numerator, V(E(Ft|Xi)), E is the expectation operator to call for an integral over parameter Xi; that is, inclusive of all input variables except Xi, the V, the variance operator, applies a further integral over Xi. The variable contribution (i.e., importance), represented as Si for the i
th variable, is calculated as the normalized sensitivity mea- sure. In a later study, Saltelli et al. (2004) proved that this equation is the most probable
Systematically Perturb, Add/ Remove Inputs
Trained ML Model – “the black-box” Observed Change in Outputs
w∙ x–
b= –1
w∙ x–
b= 0
w∙ x–
b= 1
2 w
–6 –4X1
X2
–2 0 2
1
0.5
0 4 6
Im p o rt
a n c e
Variable Names
100 90
80 70
60 50 50
40 30
40
1
2 3
7654
8 9 10 11 12 13 14 15
D1
FIGURE 4.12 Graphical Depiction of the Sensitivity Analysis Process.
226 Part II • Predictive Analytics/Machine Learning
measure of model sensitivity that is capable of ranking input variables (i.e., the predic- tors) in the order of importance for any combination of interactions including the non- orthogonal relationships among the input variables. To properly combine the sensitivity analysis results for several prediction methods, one can use an information fusion–based methodology, particularly by modifying the preceding equation in such a way that the sensitivity measure of an input variable n obtained based on the information combined (i.e., fused) from m number of prediction models. The following equation represents this weighted summation function.
Sn(fused ) = a m
i = 1 viSin = v1S1n + v2S2n + ... + vmSmn
In this equation, vi represents the normalized contribution/weight for each predic- tion model in which the level of contribution/weight of a model is calculated as a func- tion of its relative predictive power—the larger the prediction power (i.e., accuracy) is, the higher is the value of v.
CLASSIFICATION TECHNIQUES A number of techniques (or algorithms) are used for clas- sification modeling, including the following:
• Decision tree analysis. Decision tree analysis (a machine-learning technique) is arguably the most popular classification technique in the data mining arena. A detailed description of this technique is given in the following section.
• Statistical analysis. Statistical techniques were the primary classification algo- rithm for many years until the emergence of machine-learning techniques. Statistical classification techniques include logistic regression and discriminant analysis, both of which make the assumptions that the relationships between the input and output variables are linear in nature, the data are normally distributed, and the variables are not correlated and are independent of each other. The questionable nature of these assumptions has led to the shift toward machine-learning techniques.
• Neural networks. These are among the most popular machine-learning tech- niques that can be used for classification-type problems.
• Case-based reasoning. This approach uses historical cases to recognize com- monalities to assign a new case into the most probable category.
• Bayesian classifiers. This approach uses probability theory to build classifi- cation models based on the past occurrences that are capable of placing a new instance into a most probable class (or category).
• Genetic algorithms. This is the use of the analogy of natural evolution to build directed-search-based mechanisms to classify data samples.
• Rough sets. This method takes into account the partial membership of class labels to predefined categories in building models (collection of rules) for classifica- tion problems.
A complete description of all of these classification techniques is beyond the scope of this book; thus, only several of the most popular ones are presented here.
DECISION TREES Before describing the details of decision trees, we need to discuss some simple terminology. First, decision trees include many input variables that might have an impact on the classification of different patterns. These input variables are usu- ally called attributes. For example, if we were to build a model to classify loan risks on the basis of just two characteristics—income and a credit rating—these two characteristics would be the attributes, and the resulting output would be the class label (e.g., low, me- dium, or high risk). Second, a tree consists of branches and nodes. A branch represents
Chapter 4 • Data Mining Process, Methods, and Algorithms 227
the outcome of a test to classify a pattern using one of the attributes. A leaf node at the end represents the final class choice for a pattern (a chain of branches from the root node to the leaf node, which can be represented as a complex if-then statement).
The basic idea behind a decision tree is that it recursively divides a training set until each division consists entirely or primarily of examples from one class. Each nonleaf node of the tree contains a split point, which is a test of one or more attributes and deter- mines how the data are to be divided further. Decision tree algorithms, in general, build an initial tree from training data such that each leaf node is pure, and they then prune the tree to increase its generalization, and hence, the prediction accuracy of test data.
In the growth phase, the tree is built by recursively dividing the data until each divi- sion is either pure (i.e., contains members of the same class) or relatively small. The basic idea is to ask questions whose answers would provide the most information, similar to what we do when playing the game “Twenty Questions.”
The split used to partition the data depends on the type of the attribute used in the split. For a continuous attribute A, splits are of the form value(A) 6 x, where x is some “optimal” split value of A. For example, the split based on income could be “Income 6 50,000.” For the categorical attribute A, splits are of the form that value(A) belongs to x where x is a subset of A. As an example, the split could be on the basis of gender: “Male versus Female.”
A general algorithm for building a decision tree is as follows:
1. Create a root node and assign all of the training data to it. 2. Select the best splitting attribute. 3. Add a branch to the root node for each value of the split. Split the data into mutually
exclusive (nonoverlapping) subsets along the lines of the specific split and move to the branches.
4. Repeat steps 2 and 3 for each and every leaf node until a stopping criterion is reached (e.g., the node is dominated by a single class label).
Many different algorithms have been proposed for creating decision trees. These algorithms differ primarily in terms of the way in which they determine the splitting attri- bute (and its split values), the order of splitting the attributes (splitting the same attribute only once or many times), the number of splits at each node (binary versus ternary), the stopping criteria, and the pruning of the tree (pre- versus postpruning). Some of the most well-known algorithms are ID3 (followed by C4.5 and C5 as the improved versions of ID3) from machine learning, classification and regression trees (CART) from statistics, and the chi-squared automatic interaction detector (CHAID) from pattern recognition.
When building a decision tree, the goal at each node is to determine the attribute and the split point of that attribute that best divides the training records to purify the class representation at that node. To evaluate the goodness of the split, some splitting indices have been proposed. Two of the most common ones are the Gini index and information gain. The Gini index is used in CART and Scalable PaRallelizable INduction of Decision Trees (SPRINT) algorithms. Versions of information gain are used in ID3 (and its newer versions, C4.5 and C5).
The Gini index has been used in economics to measure the diversity of a popu- lation. The same concept can be used to determine the purity of a specific class as the result of a decision to branch along a particular attribute or variable. The best split is the one that increases the purity of the sets resulting from a proposed split. Let us briefly look into a simple calculation of the Gini index.
If a data set S contains examples from n classes, the Gini index is defined as
gini(S) = 1 - a n
j = 1 p2j
228 Part II • Predictive Analytics/Machine Learning
where pj is a relative frequency of class j in S. If a data set S is split into two subsets, S1 and S2 with sizes N1 and N2, respectively, the Gini index of the split data contains examples from n classes, and the Gini index is defined as
ginisplit(S) = N1 N
gini(S1) + N2 N
gini(S2)
The attribute/split combination that provides the smallest ginisplit(S) is chosen to split the node. In such a determination, one should enumerate all possible splitting points for each attribute.
Information gain is the splitting mechanism used in ID3, which is perhaps the most widely known decision tree algorithm. It was developed by Ross Quinlan in 1986, and since then, he has evolved this algorithm into the C4.5 and C5 algorithms. The basic idea behind ID3 (and its variants) is to use a concept called entropy in place of the Gini index. Entropy measures the extent of uncertainty or randomness in a data set. If all the data in a subset belong to just one class, there is no uncertainty or randomness in that data set, so the entropy is zero. The objective of this approach is to build subtrees so that the entropy of each final subset is zero (or close to zero). Let us also look at the calcula- tion of the information gain.
Assume that there are two classes: P (positive) and N (negative). Let the set of ex- amples S contain p counts of class P and n counts of class N. The amount of information needed to decide if an arbitrary example in S belongs to P or N is defined as
I(p, n) = - p
p + n log2
p
p + n -
n
p + n log2
n
p + n
Assume that using attribute A, the set S will be partitioned into sets {S1, S2, . . ., g}. If Si contains pi examples of P and ni examples of N, the entropy, or the expected information needed to classify objects in all subtrees, Si, is
E(A) = a n
i = 1
pi + ni p + n
I(pi,ni)
Then the information that would be gained by branching on attribute A would be
Gain(A) = I(p,n) - E(A)
These calculations are repeated for each and every attribute, and the one with the highest information gain is selected as the splitting attribute. The basic ideas behind these splitting indices are rather similar, but the specific algorithmic details vary. A detailed definition of the ID3 algorithm and its splitting mechanism can be found in Quinlan (1986).
Application Case 4.5 illustrates how significant the gains can be if the right data min- ing techniques are used for a well-defined business problem.
Cluster Analysis for Data Mining
Cluster analysis is an essential data mining method for classifying items, events, or con- cepts into common groupings called clusters. The method is commonly used in biology, medicine, genetics, social network analysis, anthropology, archaeology, astronomy, char- acter recognition, and even in management information systems (MIS) development. As data mining has increased in popularity, its underlying techniques have been applied to business, especially to marketing. Cluster analysis has been used extensively for fraud
Chapter 4 • Data Mining Process, Methods, and Algorithms 229
Influence Health, Inc. provides the healthcare industry’s only integrated digital consumer engage- ment and activation platform. It enables providers, employers, and payers to positively influence con- sumer decision making and health behaviors well beyond the physical care setting through personal- ized and interactive multichannel engagement. Since 1996, the Birmingham, Alabama–based company has helped more than 1,100 provider organizations influence consumers in a way that transforms finan- cial and quality outcomes.
Healthcare is a personal business. Each patient’s needs are different and require an indi- vidual response. On the other hand—as the cost of providing healthcare services continues to rise— hospitals and health systems increasingly need to harness economies of scale by catering to larger and larger populations. The challenge then becomes to provide a personalized approach while operat- ing on a large scale. Influence Health specializes in helping its healthcare sector clients solve this chal- lenge by getting to know their existing and poten- tial patients better and targeting each individual with the appropriate health services at the right time. Advanced predictive analytics technology from IBM allows Influence Health to help its clients discover the factors that have the most influence on patients’ healthcare decisions. By assessing the propensity of hundreds of millions of prospects to require spe- cific healthcare services, Influence Health is able to boost revenues and response rates for healthcare campaigns, improving outcomes for its clients and their patients alike.
Targeting the Savvy Consumer
Today’s healthcare industry is becoming more com- petitive than ever before. If the use of an organiza- tion’s services drops, so do its profits. Rather than simply seeking out the nearest hospital or clinic, consumers are now more likely to make positive choices among healthcare providers. Paralleling efforts that are common in other industries, health- care organizations must make increased efforts to market themselves effectively to both existing and potential patients, building long-term engagement and loyalty.
The keys to successful healthcare marketing are timeliness and relevance. If you can predict what kind of health services an individual prospect might need, you can engage and influence her or him much more effectively for wellness care.
Venky Ravirala, chief analytics officer at Influence Health, explains, “Healthcare organiza- tions risk losing people’s attention if they bombard them with irrelevant messaging. We help our clients avoid this risk by using analytics to segment their existing and potential prospects and market to them in a much more personal and relevant way.”
Faster and More Flexible Analytics
As its client base has expanded, the total volume of data in Influence Health’s analytics systems has grown to include over 195 million patient records with a detailed disease encounter history for several million patients. Ravirala comments, “With so much data to analyze, our existing method of scoring data was becoming too complex and time-consuming. We wanted to be able to extract insights at greater speed and accuracy.”
By leveraging predictive analytics software from IBM, Influence Health is now able to develop models that calculate how likely each patient is to require particular services and express this likelihood as a percentage score. Microsegmentation and numerous disease-specific models draw on demographic, socio- economic, geographical, behavioral, disease history, and census data and examine different aspects of each patient’s predicted healthcare needs.
“The IBM solution allows us to combine all these models using an ensemble technique, which helps to overcome the limitations of individual mod- els and provide more accurate results,” comments Venky Ravirala, chief analytics officer at Influence Health. “It gives us the flexibility to apply multiple techniques to solve a problem and arrive at the best solution. It also automates much of the analytics process, enabling us to respond to clients’ requests faster than before, and often give them a much deeper level of insight into their patient population.”
For example, Influence Health decided to find out how disease prevalence and risk vary between different cohorts within the general population. By
Application Case 4.5 Influence Health Uses Advanced Predictive Analytics to Focus on the Factors That Really Influence People’s Healthcare Decisions
(Continued )
230 Part II • Predictive Analytics/Machine Learning
detection (both credit card and e-commerce) and market segmentation of customers in contemporary CRM systems. More applications in business continue to be developed as the strength of cluster analysis is recognized and used.
Cluster analysis is an exploratory data analysis tool for solving classification prob- lems. The objective is to sort cases (e.g., people, things, events) into groups, or clusters, so that the degree of association is strong among members of the same cluster and weak among members of different clusters. Each cluster describes the class to which its mem- bers belong. An obvious one-dimensional example of cluster analysis is to establish score ranges into which to assign class grades for a college class. This is similar to the cluster analysis problem that the U.S. Treasury faced when establishing new tax brackets in the 1980s. A fictional example of clustering occurs in J. K. Rowling’s Harry Potter books. The Sorting Hat determines to which House (e.g., dormitory) to assign first-year students at the Hogwarts School. Another example involves determining how to seat guests at a wed- ding. As far as data mining goes, the importance of cluster analysis is that it can reveal
using very sophisticated cluster analysis techniques, the company was able to discover new comorbidity patterns that improve risk predictability for over 100 common diseases by up to 800 percent.
This helps to reliably differentiate between high-risk and very high-risk patients—making it eas- ier to target campaigns at the patients and prospects who need them most. With insights like these in hand, Influence Health is able to use its healthcare marketing expertise to advise its clients on how best to allocate marketing resources.
“Our clients make significant budgeting deci- sions based on the guidance we give them,” states Ravirala. “We help them maximize the impact of one-off campaigns—such as health insurance mar- ketplace campaigns when Obamacare began—as well as their long-term strategic plans and ongoing marketing communications.”
Reaching the Right Audience
By enabling its clients to target their marketing activi- ties more effectively, Influence Health is helping to drive increased revenue and enhance population health. “Working with us, clients have been able to achieve return on investment of up to 12 to 1 through better targeted marketing,” elaborates Ravirala. “And it’s not just about revenues: by ensuring that vital healthcare information gets sent to the people who need it, we are helping our clients improve general health levels in the communities they serve.”
Influence Health continues to refine its modeling techniques, gaining an ever-deeper understanding of the critical attributes that influence healthcare deci- sions. With a flexible analytics toolset at its fingertips, the company is well equipped to keep improving its service to clients. Ravirala explains, “In the future, we want to take our understanding of patient and prospect data to the next level, identifying patterns in behavior and incorporating analysis with machine- learning libraries. IBM SPSS has already given us the ability to apply and combine multiple models without writing a single line of code. We’re eager to further leverage this IBM solution as we expand our health- care analytics to support clinical outcomes and popu- lation health management services.”
“We are achieving analytics on an unprec- edented scale. Today, we can analyze 195 million records with 35 different models in less than two days—a task which was simply not possible for us in the past,” says Ravirala.
Questions for Case 4.5
1. What does Influence Health do?
2. What were the company’s challenges, proposed solutions, and obtained results?
3. How can data mining help companies in the healthcare industry (in ways other than the ones mentioned in this case)?
Source: Reprint Courtesy of International Business Machines Corporation, © (2018) International Business Machines Corporation.
Application Case 4.5 (Continued)
Chapter 4 • Data Mining Process, Methods, and Algorithms 231
associations and structures in data that were not previously apparent but are sensible and useful once found.
Cluster analysis results can be used to
• Identify a classification scheme (e.g., types of customers). • Suggest statistical models to describe populations. • Indicate rules for assigning new cases to classes for identification, targeting, and
diagnostic purposes. • Provide measures of definition, size, and change in what were previously broad
concepts. • Find typical cases to label and represent classes. • Decrease the size and complexity of the problem space for other data mining
methods. • Identify outliers in a specific domain (e.g., rare-event detection).
DETERMINING THE OPTIMAL NUMBER OF CLUSTERS Clustering algorithms usually re- quire one to specify the number of clusters to find. If this number is not known from prior knowledge, it should be chosen in some way. Unfortunately, there is not an op- timal way to calculate what this number is supposed to be. Therefore, several different heuristic methods have been proposed. The following are among the most commonly referenced ones:
• Look at the percentage of variance explained as a function of the number of clus- ters; that is, choose a number of clusters so that adding another cluster would not give much better modeling of the data. Specifically, if one graphs the percentage of variance explained by the clusters, there is a point at which the marginal gain will drop (giving an angle in the graph), indicating the number of clusters to be chosen.
• Set the number of clusters to (n>2)1>2, where n is the number of data points. • Use the Akaike information criterion (AIC), which is a measure of the goodness of
fit (based on the concept of entropy), to determine the number of clusters. • Use Bayesian information criterion, a model-selection criterion (based on maximum
likelihood estimation), to determine the number of clusters.
ANALYSIS METHODS Cluster analysis might be based on one or more of the following general methods:
• Statistical methods (including both hierarchical and nonhierarchical), such as k- means or k-modes.
• Neural networks (with the architecture called self-organizing map). • Fuzzy logic (e.g., fuzzy c-means algorithm). • Genetic algorithms.
Each of these methods generally works with one of two general method classes:
• Divisive. With divisive classes, all items start in one cluster and are broken apart. • Agglomerative. With agglomerative classes, all items start in individual clusters,
and the clusters are joined together.
Most cluster analysis methods involve the use of a distance measure to calculate the closeness between pairs of items. Popular distance measures include Euclidian dis- tance (the ordinary distance between two points that one would measure with a ruler) and Manhattan distance (also called the rectilinear distance or taxicab distance) between two points. Often, they are based on true distances that are measured, but this need not be so, as is typically the case in IS development. Weighted averages can be used to es- tablish these distances. For example, in an IS development project, individual modules of
232 Part II • Predictive Analytics/Machine Learning
the system can be related by the similarity between their inputs, outputs, processes, and the specific data used. These factors are then aggregated, pairwise by item, into a single distance measure.
K-MEANS CLUSTERING ALGORITHM The k-means algorithm (where k stands for the pre- determined number of clusters) is arguably the most referenced clustering algorithm. It has its roots in traditional statistical analysis. As the name implies, the algorithm assigns each data point (customer, event, object, etc.) to the cluster whose center (also called the cen- troid) is the nearest. The center is calculated as the average of all the points in the cluster; that is, its coordinates are the arithmetic mean for each dimension separately over all the points in the cluster. The algorithm steps follow and are shown graphically in Figure 4.13:
Initialization step: Choose the number of clusters (i.e., the value of K ).
Step 1: Randomly generate k random points as initial cluster centers. Step 2: Assign each point to the nearest cluster center. Step 3: Recompute the new cluster centers.
Repetition step: Repeat steps 2 and 3 until some convergence criterion is met (usually that the assignment of points to clusters becomes stable).
Association Rule Mining
Association rule mining (also known as affinity analysis or market-basket analysis) is a popular data mining method that is commonly used as an example to explain what data mining is and what it can do to a technologically less-savvy audience. Most of you might have heard the famous (or infamous, depending on how you look at it) relationship dis- covered between the sales of beer and diapers at grocery stores. As the story goes, a large supermarket chain (maybe Walmart, maybe not; there is no consensus on which super- market chain it was) did an analysis of customers’ buying habits and found a statistically significant correlation between purchases of beer and purchases of diapers. It was theo- rized that the reason for this was that fathers (presumably young men) were stopping off at the supermarket to buy diapers for their babies (especially on Thursdays), and because they could no longer go down to the sports bar as often, would buy beer as well. As a result of this finding, the supermarket chain is alleged to have placed the diapers next to the beer, resulting in increased sales of both.
In essence, association rule mining aims to find interesting relationships (affinities) between variables (items) in large databases. Because of its successful application to retail business problems, it is commonly called market-basket analysis. The main idea
Step 1 Step 2 Step 3
FIGURE 4.13 Graphical Illustration of the Steps in the k-Means Algorithm.
Chapter 4 • Data Mining Process, Methods, and Algorithms 233
in market-basket analysis is to identify strong relationships among different products (or services) that are usually purchased together (show up in the same basket together, ei- ther a physical basket at a grocery store or a virtual basket at an e-commerce Web site). For example, 65 percent of those who buy comprehensive automobile insurance also buy health insurance; 80 percent of those who buy books online also buy music online; 60 percent of those who have high blood pressure and are overweight have high cho- lesterol; 70 percent of the customers who buy a laptop computer and virus protection software also buy extended service plans.
The input to market-basket analysis is the simple point-of-sale transaction data when a number of products and/or services purchased together (just like the content of a purchase receipt) are tabulated under a single transaction instance. The outcome of the analysis is invaluable information that can be used to better understand customer- purchase behavior to maximize the profit from business transactions. A business can take advantage of such knowledge by (1) putting the items next to each other to make it more convenient for the customers to pick them up together and not forget to buy one when buying the others (increasing sales volume), (2) promoting the items as a package—do not put one on sale if the other(s) are on sale, and (3) placing them apart from each other so that the customer has to walk the aisles to search for it, and by doing so, potentially seeing and buying other items.
Applications of market-basket analysis include cross-marketing, cross-selling, store design, catalog design, e-commerce site design, optimization of online advertising, prod- uct pricing, and sales/promotion configuration. In essence, market-basket analysis helps businesses infer customer needs and preferences from their purchase patterns. Outside the business realm, association rules are successfully used to discover relationships between symptoms and illnesses, diagnosis and patient characteristics and treatments (which can be used in a medical decision support system), and genes and their functions (which can be used in genomics projects), among others. Here are a few common areas and uses for association rule mining:
• Sales transactions: Combinations of retail products purchased together can be used to improve product placement on the sales floor (placing products that go to- gether in close proximity) and promotional pricing of products (not having promo- tions on both products that are often purchased together).
• Credit card transactions: Items purchased with a credit card provide insight into other products the customer is likely to purchase or fraudulent use of credit card numbers.
• Banking services: The sequential patterns of services used by customers (check- ing account followed by savings account) can be used to identify other services they might be interested in (investment account).
• Insurance service products: Bundles of insurance products bought by custom- ers (car insurance followed by home insurance) can be used to propose additional insurance products (life insurance), or unusual combinations of insurance claims can be a sign of fraud.
• Telecommunication services: Commonly purchased groups of options (e.g., call waiting, caller ID, three-way calling) help better structure product bundles to maximize revenue; the same is also applicable to multichannel telecom providers with phone, television, and Internet service offerings.
• Medical records: Certain combinations of conditions can indicate increased risk of various complications; or, certain treatment procedures at certain medical facili- ties can be tied to certain types of infections.
A good question to ask with respect to the patterns/relationships that association rule mining can discover is “Are all association rules interesting and useful?” To answer
234 Part II • Predictive Analytics/Machine Learning
such a question, association rule mining uses two common metrics: support, and con- fidence and lift. Before defining these terms, let’s get a little technical by showing what an association rule looks like:
X 1 Y 3Supp(%), Conf (%)4 {Laptop Computer, Antivirus software} 1 {Extended Service Plan} [30%, 70%]
Here, X (products and/or service—called the left-hand side, LHS, or antecedent) is associated with Y (products and/or service—called the right-hand side, RHS, or con- sequent). S is the support, and C is the confidence for this particular rule. Here are the simple formulas for Supp, Conf, and Lift.
Support = Supp(X 1 Y ) = Number of Baskets that contains both X and Y
Total Number of Baskets
Confidence = Conf (X 1 Y) = Supp (X 1 Y )
Supp (X)
Lift(X 1 Y ) = Conf (X 1 Y )
Expected Conf (X 1 Y ) =
S(X 1 Y ) S(X)
S(X) * S(Y )
S(X)
= S(X 1 Y )
S(X) * S(Y )
The support (S) of a collection of products is the measure of how often these products and/or services (i.e., LHS + RHS = Laptop Computer, Antivirus Software, and Extended Service Plan) appear together in the same transaction; that is, the proportion of transactions in the data set that contain all of the products and/or services mentioned in a specific rule. In this example, 30 percent of all transactions in the hypothetical store database had all three products present in a single sales ticket. The confidence of a rule is the measure of how often the products and/or services on the RHS (consequent) go together with the products and/or services on the LHS (antecedent), that is, the propor- tion of transactions that include LHS while also including the RHS. In other words, it is the conditional probability of finding the RHS of the rule present in transactions where the LHS of the rule already exists. The lift value of an association rule is the ratio of the confidence of the rule and the expected confidence of the rule. The expected confidence of a rule is defined as the product of the support values of the LHS and the RHS divided by the support of the LHS.
Several algorithms are available for discovering association rules. Some well-known algorithms include Apriori, Eclat, and FP-Growth. These algorithms only do half the job, which is to identify the frequent itemsets in the database. Once the frequent itemsets are identified, they need to be converted into rules with antecedent and consequent parts. Determination of the rules from frequent itemsets is a straightforward matching process, but the process can be time consuming with large transaction databases. Even though there can be many items on each section of the rule, in practice the consequent part usu- ally contains a single item. In the following section, one of the most popular algorithms for identification of frequent itemsets is explained.
APRIORI ALGORITHM The Apriori algorithm is the most commonly used algorithm to discover association rules. Given a set of itemsets (e.g., sets of retail transactions, each listing individual items purchased), the algorithm attempts to find subsets that are common to at least a minimum number of the itemsets (i.e., complies with a minimum support). Apriori uses a bottom-up approach by which frequent subsets are extended
Chapter 4 • Data Mining Process, Methods, and Algorithms 235
one item at a time (a method known as candidate generation by which the size of frequent subsets increases from one-item subsets to two-item subsets, then three-item subsets, etc.), and groups of candidates at each level are tested against the data for minimum support. The algorithm terminates when no further successful extensions are found.
As an illustrative example, consider the following. A grocery store tracks sales trans- actions by SKU (stock-keeping unit) and thus knows which items are typically purchased together. The database of transactions along with the subsequent steps in identifying the frequent itemsets is shown in Figure 4.14. Each SKU in the transaction database cor- responds to a product, such as “1 = butter,” “2 = bread,” “3 = water,” and so on. The first step in Apriori is to count the frequencies (i.e., the supports) of each item (one-item itemsets). For this overly simplified example, let us set the minimum support to 3 (or 50, meaning an itemset is considered to be a frequent itemset if it shows up in at least 3 of 6 transactions in the database). Because all the one-item itemsets have at least 3 in the sup- port column, they are all considered frequent itemsets. However, had any of the one-item itemsets not been frequent, they would not have been included as a possible member of possible two-item pairs. In this way, Apriori prunes the tree of all possible itemsets. As Figure 4.14 shows, using one-item itemsets, all possible two-item itemsets are generated and the transaction database is used to calculate their support values. Because the two- item itemset {1, 3} has a support less than 3, it should not be included in the frequent itemsets that will be used to generate the next-level itemsets (three-item itemsets). The algorithm seems deceivingly simple, but only for small data sets. In much larger data sets, especially those with huge amounts of items present in low quantities and small amounts of items present in big quantities, the search and calculation become a computationally intensive process.
uSECTION 4.5 REVIEW QUESTIONS
1. Identify at least three of the main data mining methods. 2. Give examples of situations in which classification would be an appropriate data
mining technique. Give examples of situations in which regression would be an appropriate data mining technique.
3. List and briefly define at least two classification techniques. 4. What are some of the criteria for comparing and selecting the best classification
technique?
Itemset (SKUs)
Support Transaction
No SKUs
(item no.)
1001234
1001235
1001236
1001237
1001238
1001239
1, 2, 3, 4
2, 3, 4
2, 3
1, 2, 4
1, 2, 3, 4
2, 4
Raw Transaction Data
1
2
3
4
3
6
4
5
Itemset (SKUs)
Support
1, 2
1, 3
1, 4
2, 3
3
2
3
4
3, 4
5
3
2, 4
Itemset (SKUs)
Support
1, 2, 4
2, 3, 4
3
3
One-Item Itemsets Two-Item Itemsets Three-Item Itemsets
FIGURE 4.14 Identification of Frequent Itemsets in the Apriori Algorithm.
236 Part II • Predictive Analytics/Machine Learning
5. Briefly describe the general algorithm used in decision trees. 6. Define Gini index. What does it measure? 7. Give examples of situations in which cluster analysis would be an appropriate data
mining technique.
8. What is the major difference between cluster analysis and classification? 9. What are some of the methods for cluster analysis?
10. Give examples of situations in which association would be an appropriate data min- ing technique.
4.6 DATA MINING SOFTWARE TOOLS
Many software vendors provide powerful data mining tools. Examples of these ven- dors include IBM (IBM SPSS Modeler, formerly known as SPSS PASW Modeler and Clementine), SAS (Enterprise Miner), Dell (Statistica, formerly known as StatSoft Statistica Data Miner), SAP (Infinite Insight, formerly known as KXEN Infinite Insight), Salford Systems (CART, MARS, TreeNet, RandomForest), Angoss (KnowledgeSTUDIO, KnowledgeSEEKER), and Megaputer (PolyAnalyst). Noticeably but not surprisingly, the most popular data mining tools are developed by the well-established statistical soft- ware companies (SAS, SPSS, and StatSoft)—largely because statistics is the foundation of data mining, and these companies have the means to cost-effectively develop them into full-scale data mining systems. Most of the business intelligence tool vendors (e.g., IBM Cognos, Oracle Hyperion, SAP Business Objects, Tableau, Tibco, Qlik, MicroStrategy, Teradata, and Microsoft) also have some level of data mining capabilities integrated into their software offerings. These BI tools are still primarily focused on multidimensional modeling and data visualization and are not considered to be direct competitors of the data mining tool vendors.
In addition to these commercial tools, several open source and/or free data mining software tools are available online. Traditionally, especially in educational circles, the most popular free and open source data mining tool is Weka, which was developed by a number of researchers from the University of Waikato in New Zealand (the tool can be downloaded from cs.waikato.ac.nz/ml/weka). Weka includes a large number of algo- rithms for different data mining tasks and has an intuitive user interface. Recently, a num- ber of free open source, highly capable data mining tools emerged: leading the pack are KNIME (knime.org) and RapidMiner (rapidminer.com). Their graphically enhanced user interfaces, employment of a rather large number of algorithms, and incorporation of a variety of data visualization features set them apart from the rest of the free tools. These two free software tools are also platform agnostic (i.e., can natively run on both Windows and Mac operating systems). With a recent change in its offerings, RapidMiner has created a scaled-down version of its analytics tool for free (i.e., community edition) while making the full commercial product. Therefore, once listed under the free/open source tools cat- egory, RapidMiner today is often listed under commercial tools. The main difference be- tween commercial tools, such as SAS Enterprise Miner, IBM SPSS Modeler, and Statistica, and free tools, such as Weka, RapidMiner (community edition), and KNIME, is the com- putational efficiency. The same data mining task involving a rather large and feature-rich data set can take much longer to complete with the free software tools, and for some algorithms, the job might not even be completed (i.e., crashing due to the inefficient use of computer memory). Table 4.2 lists a few of the major products and their Web sites.
A suite of business intelligence and analytics capabilities that has become increas- ingly more popular for data mining studies is Microsoft’s SQL Server (it has included
Chapter 4 • Data Mining Process, Methods, and Algorithms 237
increasingly more analytics capabilities, such as BI and predictive modeling modules, starting with the SQL Server 2012 version) where data and the models are stored in the same relational database environment, making model management a considerably easier task. Microsoft Enterprise Consortium serves as the worldwide source for access to Microsoft’s SQL Server software suite for academic purposes—teaching and research. The consortium has been established to enable universities around the world to access en- terprise technology without having to maintain the necessary hardware and software on their own campus. The consortium provides a wide range of business intelligence devel- opment tools (e.g., data mining, cube building, business reporting) as well as a number of large, realistic data sets from Sam’s Club, Dillard’s, and Tyson Foods. The Microsoft Enterprise Consortium is free of charge and can be used only for academic purposes. The Sam M. Walton College of Business at the University of Arkansas hosts the enterprise sys- tem and allows consortium members and their students to access these resources using a simple remote desktop connection. The details about becoming a part of the consortium along with easy-to-follow tutorials and examples can be found at walton.uark.edu/ enterprise/.
In May 2016, KDnuggets.com conducted the 13th Annual Software Poll on the following question: “What software have you used for Analytics, Data Mining, Data Science, Machine Learning projects in the past 12 months?” The poll received remarkable participation from analytics and data science community and vendors, attracting 2,895
TABLE 4.2 Selected Data Mining Software
Product Name Web Site (URL)
IBM SPSS Modeler www-01.ibm.com/software/analytics/spss/products/ modeler/
IBM Watson Analytics ibm.com/analytics/watson-analytics/
SAS Enterprise Miner sas.com/en_id/software/analytics/enterprise-miner.html
Dell Statistica statsoft.com/products/statistica/product-index
PolyAnalyst megaputer.com/site/polyanalyst.php
CART, RandomForest salford-systems.com
Insightful Miner solutionmetrics.com.au/products/iminer/default.html
XLMiner solver.com/xlminer-data-mining
SAP InfiniteInsight (KXEN) help.sap.com/ii
GhostMiner qs.pl/ghostminer
SQL Server Data Mining msdn.microsoft.com/en-us/library/bb510516.aspx
Knowledge Miner knowledgeminer.com
Teradata Warehouse Miner teradata.com/products-and-services/teradata-warehouse- miner/
Oracle Data Mining (ODM) oracle.com/technetwork/database/options/odm/
FICO Decision Management fico.com/en/analytics/decision-management-suite/
Orange Data Mining Tool orange.biolab.si/
Zementis Predictive Analytics zementis.com
238 Part II • Predictive Analytics/Machine Learning
voters, who chose from a record number of 102 different tools. Here are some of the interesting findings that came from the poll:
• R remains the leading tool, with 49 percent shares (up from 46.9% in 2015), but Python usage grew faster and almost caught up to R with 45.8 percent shares (up from 30.3%).
• RapidMiner remains the most popular general platform for data mining/data science with about 33 percent shares. Notable tools with the most growth in popularity include g, Dataiku, MLlib, H2O, Amazon Machine Learning, scikit-learn, and IBM Watson.
• The increased choice of tools is reflected in wider usage. The average number of tools used was 6.0 (versus 4.8 in May 2015).
• The usage of Hadoop/Big Data tools increased to 39 percent up from 29 percent in 2015 (and 17% in 2014) driven by Apache Spark, MLlib (Spark Machine Learning Library), and H2O.
• The participation by region was United States/Canada (40%), Europe (39%), Asia (9.4%), Latin America (5.8%), Africa/MidEast (2.9%), and Australia/New Zealand (2.2%).
• This year, 86 percent of voters used commercial software, and 75 percent used free software. About 25 percent used only commercial software, and 13 percent used only open source/free software. A majority of 61 percent used both free and com- mercial software, similar to 64 percent in 2015.
• The use of Hadoop/Big Data tools increased to 39 percent, up from 29 percent in 2015 and 17 percent in 2014, driven mainly by big growth in Apache Spark, MLlib (Spark Machine Learning Library), and H2O, which we include among Big Data tools.
• For the second year, KDnuggets.com’s poll included Deep Learning tools. This year, 18 percent of voters used Deep Learning tools, doubling the 9 percent in 2015—Google Tensorflow jumped to first place, displacing last year’s leader, Theano/Pylearn2 ecosystem.
• In the programming languages category, Python, Java, Unix tools, and Scala grew in popularity, while C/C++, Perl, Julia, F#, Clojure, and Lisp declined.
To reduce bias through multiple voting, in this poll KDnuggets.com used e-mail verification and, by doing so, aimed to make results more representative of the reality in the analytics world. The results for the top 40 software tools (as per total number of votes received) are shown in Figure 4.15. The horizontal bar chart also makes a differentiation among free/open source, commercial, and Big Data/Hadoop tools using a color-coding schema.
Application Case 4.6 is about a research study in which a number of software tools and data mining techniques were used to build data mining models to predict financial success (box-office receipts) of Hollywood movies while they are nothing more than ideas.
uSECTION 4.6 REVIEW QUESTIONS
1. What are the most popular commercial data mining tools? 2. Why do you think the most popular tools are developed by statistics-based
companies?
3. What are the most popular free data mining tools? Why are they gaining overwhelm- ing popularity (especially R)?
4. What are the main differences between commercial and free data mining software tools?
5. What would be your top five selection criteria for a data mining tool? Explain.
Chapter 4 • Data Mining Process, Methods, and Algorithms 239
0
Orange 89 Gnu Octave 89
Salford SPM/CART/RF/MARS/TreeNet 100 Rattle 103
121 Apache Pig 132
IBM Watson
Other Hadoop/HDFS-based tools 141 Microsoft Azure Machine Learning 147
QlikView 153 Hbase 158
Microsoft Power BI 161
Scala 180
SAS Enterprise Miner 162
H2O 193 Other programming and data languages 197
Other free analytics/data mining tools 198 C/C++ 210
211 222 225 227 242 263
301 314 315 337 359
462 487 497 521 536
624
944 972
1,029 1,325
1,419
641
SQL on Hadoop tools IBM SPSS Modeler
SAS base Dataiku
IBM SPSS Statistics MATLAB
Unix shell/awk/gawk Microsoft SQL Server
Weka Mllib Hive
Anaconda Java
SciKit-Learn KNIME Tableau
Spark Hadoop
RapidMiner Excel SQL
Python R
300 600 900 1200 1500
Free/Open Source tools
Commercial tools Hadoop/Big Data tools
FIGURE 4.15 Popular Data Mining Software Tools (Poll Results). Source: Used with permission from KDnuggets.com.
Predicting box-office receipts (i.e., financial success) of a particular motion picture is an interesting and challenging problem. According to some domain
experts, the movie industry is the “land of hunches and wild guesses” due to the difficulty associated with forecasting product demand, making the movie
Application Case 4.6 Data Mining goes to Hollywood: Predicting Financial Success of Movies
(Continued )
240 Part II • Predictive Analytics/Machine Learning
business in Hollywood a risky endeavor. In sup- port of such observations, Jack Valenti (the longtime president and CEO of the Motion Picture Association of America) once mentioned that “no one can tell you how a movie is going to do in the market- place . . . not until the film opens in darkened the- atre and sparks fly up between the screen and the audience.” Entertainment industry trade journals and magazines have been full of examples, statements, and experiences that support such a claim.
Like many other researchers who have attempted to shed light on this challenging real-world problem, Ramesh Sharda and Dursun Delen have been explor- ing the use of data mining to predict the financial per- formance of a motion picture at the box office before it even enters production (while the movie is nothing more than a conceptual idea). In their highly publi- cized prediction models, they convert the forecasting (or regression) problem into a classification problem; that is, rather than forecasting the point estimate of box-office receipts, they classify a movie based on its box-office receipts in one of nine categories, rang- ing from “flop” to “blockbuster,” making the problem a multinomial classification problem. Table 4.3 illus- trates the definition of the nine classes in terms of the range of box-office receipts.
Data
Data were collected from a variety of movie-related databases (e.g., ShowBiz, IMDb, IMSDb, AllMovie, BoxofficeMojo) and consolidated into a single data set. The data set for the most recently developed models contained 2,632 movies released between 1998 and 2006. A summary of the independent vari- ables along with their specifications is provided in Table 4.4. For more descriptive details and justifi- cation for inclusion of these independent variables, the reader is referred to Sharda and Delen (2006).
The Methodology
Using a variety of data mining methods, including neural networks, decision trees, SVMs, and three types of ensembles, Sharda and Delen (2006) devel- oped the prediction models. The data from 1998 to 2005 were used as training data to build the pre- diction models, and the data from 2006 were used as the test data to assess and compare the models’ prediction accuracy. Figure 4.16 shows a screenshot of IBM SPSS Modeler (formerly Clementine data mining tool) depicting the process map employed for the prediction problem. The upper-left side of the process map shows the model development
Application Case 4.6 (Continued)
TABLE 4.3 Movie Classification based on Receipts
Class No. 1 2 3 4 5 6 7 8 9
Range (in millions of dollars)
71 71 710 720 740 765 7100 7150 7200 (Flop) 66 10 620 66 40 66 65 66 100 66 150 66 200 (Blockbuster)
TABLE 4.4 Summary of Independent Variables
Independent Variable Number of Values Possible Values
MPAA Rating 5 G, PG, PG-13, R, NR
Competition 3 High, medium, low
Star value 3 High, medium, low
Genre 10 Sci-Fi, Historic Epic Drama, Modern Drama, Politically Related, Thriller, Horror, Comedy, Cartoon, Action, Documentary
Special effects 3 High, medium, low
Sequel 2 Yes, no
Number of screens 1 A positive integer between 1 and 3,876
Chapter 4 • Data Mining Process, Methods, and Algorithms 241
process, and the lower-right corner of the process map shows the model assessment (i.e., testing or scoring) process (more details on the IBM SPSS Modeler tool and its usage can be found on the book’s Web site).
The Results
Table 4.5 provides the prediction results of all three data mining methods as well as the results of the three different ensembles. The first performance measure is the percentage of correct classification rate, which is called Bingo. Also reported in the table is the 1-Away correct classification rate (i.e., within one category). The results indicate that SVM performed the best among the individual prediction models followed by ANN; the worst of the three was the CART decision tree algorithm. In general, the ensemble models per- formed better than the individual prediction models of which the fusion algorithm performed the best. What is probably more important to decision makers and standing out in the results table is the significantly
low standard deviation obtained from the ensembles compared to the individual models.
The Conclusion
The researchers claim that these prediction results are better than any reported in the published literature for this problem domain. Beyond the attractive accuracy of their prediction results of the box-office receipts, these models could also be used to further analyze (and potentially optimize) the decision variables to maximize the financial return. Specifically, the param- eters used for modeling could be altered using the already trained prediction models to better understand the impact of different parameters on the end results. During this process, which is commonly referred to as sensitivity analysis, the decision maker of a given enter- tainment firm could find out, with a fairly high accuracy level, how much value a specific actor (or a specific release date, or the addition of more technical effects, etc.) brings to the financial success of a film, making the underlying system an invaluable decision aid.
Model Development Process
Model Assessment Process
Table
DataMining_Movie_All.
1996–2005 Data
SVM
2006 Data
Class
Class
Class
Class Analysis
Analysis
Analysis
CART Decision Tree
Neural Net
FIGURE 4.16 Process Flow Screenshot for the Box-Office Prediction System. Source: Reprint Courtesy of International Business Machines Corporation, © International Business Machines Corporation.
(Continued )
242 Part II • Predictive Analytics/Machine Learning
Questions for Case 4.6
1. Why is it important for many Hollywood pro- fessionals to predict the financial success of movies?
2. How can data mining be used for predicting financial success of movies before the start of their production process?
3. How do you think Hollywood performed, and perhaps still is performing, this task without the help of data mining tools and techniques?
Sources: R. Sharda & D. Delen, “Predicting Box-Office Success of Motion Pictures with Neural Networks,” Expert Systems with Applications, 30, 2006, pp. 243–254; D. Delen, R. Sharda, & P. Kumar, “Movie Forecast Guru: A Web-Based DSS for Hollywood Managers,” Decision Support Systems, 43(4), 2007, pp. 1151–1170.
Application Case 4.6 (Continued)
4.7 DATA MINING PRIVACY ISSUES, MYTHS, AND BLUNDERS
Data that are collected, stored, and analyzed in data mining often contain information about real people. Such information can include identification data (name, address, Social Security number, driver’s license number, employee number, etc.), demographic data (e.g., age, sex, ethnicity, marital status, number of children), financial data (e.g., salary, gross family income, checking or savings account balance, home ownership, mortgage or loan account specifics, credit card limits and balances, investment account specifics), purchase history (i.e., what is bought from where and when—either from vendor’s transaction records or from credit card transaction specifics), and other personal data (e.g., anniversary, preg- nancy, illness, loss in the family, bankruptcy filings). Most of these data can be accessed through some third-party data providers. The main question here is the privacy of the per- son to whom the data belong. To maintain the privacy and protection of individuals’ rights, data mining professionals have ethical (and often legal) obligations. One way to accomplish this is the process of de-identification of the customer records prior to applying data mining applications so that the records cannot be traced to an individual. Many publicly available data sources (e.g., CDC data, SEER data, UNOS data) are already de-identified. Prior to ac- cessing these data sources, users are often asked to consent that under no circumstances will they try to identify the individuals behind those figures.
There have been a number of instances in the recent past when companies shared their customer data with others without seeking the explicit consent of their customers. For instance, as most of you might recall, in 2003, JetBlue Airlines provided more than 1 million passenger records of customers to Torch Concepts, a U.S. government contractor. Torch
TABLE 4.5 Tabulated Prediction Results for Individual and Ensemble Models
Prediction Models
Individual Models Ensemble Models
Performance Measure SVM ANN CART Random Forest Boosted Tree Fusion (average)
Count (Bingo) 192 182 140 189 187 194
Count (1-Away) 104 120 126 121 104 120
Accuracy (% Bingo) 55.49% 52.60% 40.46% 54.62% 54.05% 56.07%
Accuracy (% 1-Away) 85.55% 87.28% 76.88% 89.60% 84.10% 90.75%
Standard deviation 0.93 0.87 1.05 0.76 0.84 0.63
Chapter 4 • Data Mining Process, Methods, and Algorithms 243
then subsequently augmented the passenger data with additional information such as fam- ily sizes and Social Security numbers—information purchased from the data broker Acxiom. The consolidated personal database was intended to be used for a data mining project to develop potential terrorist profiles. All of this was done without notification or consent of passengers. When news of the activities got out, however, dozens of privacy lawsuits were filed against JetBlue, Torch, and Acxiom, and several U.S. senators called for an investiga- tion into the incident (Wald, 2004). Similar, but not as dramatic, privacy-related news was reported in the recent past about popular social network companies that allegedly were selling customer-specific data to other companies for personalized target marketing.
Another peculiar story about privacy concerns made it to the headlines in 2012. In this instance, the company, Target, did not even use any private and/or personal data. Legally speaking, there was no violation of any laws. The story is summarized in Application Case 4.7.
In early 2012, an infamous story appeared concern- ing Target’s practice of predictive analytics. The story was about a teenage girl who was being sent adver- tising flyers and coupons by Target for the kinds of things that a mother-to-be would buy from a store like Target. The story goes like this: An angry man went into a Target outside of Minneapolis, demanding to talk to a manager: “My daughter got this in the mail!” he said. “She’s still in high school, and you’re sending her coupons for baby clothes and cribs? Are you trying to encourage her to get pregnant?” The manager had no idea what the man was talking about. He looked at the mailer. Sure enough, it was addressed to the man’s daughter and contained advertisements for maternity clothing, nursery furniture, and pictures of smiling infants. The manager apologized and then called a few days later to apologize again. On the phone, though, the father was somewhat abashed. “I had a talk with my daughter,” he said. “It turns out there’s been some activities in my house I haven’t been completely aware of. She’s due in August. I owe you an apology.”
As it turns out, Target figured out a teen girl was pregnant before her father did! Here is how the company did it. Target assigns every customer a Guest ID number (tied to his or her credit card, name, or e-mail address) that becomes a placeholder that keeps a history of everything the person has bought. Target augments these data with any demographic information that it had collected from the customer or had bought from other information sources. Using this information, Target looked at historical buying
data for all the females who had signed up for Target baby registries in the past. They analyzed the data from all directions, and soon enough, some useful patterns emerged. For example, lotions and special vitamins were among the products with interesting purchase patterns. Lots of people buy lotion, but what an analyst noticed was that women on the baby registry were buying larger quantities of unscented lotion around the beginning of their second trimester. Another analyst noted that sometime in the first 20 weeks, pregnant women loaded up on supplements like calcium, magnesium, and zinc. Many shoppers purchase soap and cotton balls, but when someone suddenly starts buying lots of scent-free soap and extra-large bags of cotton balls, in addition to hand sanitizers and washcloths, it signals that they could be getting close to their delivery date. In the end, the analysts were able to identify about 25 products that, when analyzed together, allowed them to assign each shopper a “pregnancy prediction” score. More impor- tant, they could also estimate a woman’s due date to within a small window, so Target could send cou- pons timed to very specific stages of her pregnancy.
If you look at this practice from a legal perspec- tive, you would conclude that Target did not use any information that violates customer privacy; rather, they used transactional data that almost every other retail chain is collecting and storing (and perhaps analyzing) about their customers. What was disturb- ing in this scenario was perhaps the targeted con- cept: pregnancy. Certain events or concepts should
Application Case 4.7 Predicting Customer Buying Patterns—The Target Story
(Continued )
244 Part II • Predictive Analytics/Machine Learning
Data Mining Myths and Blunders
Data mining is a powerful analytical tool that enables business executives to advance from describing the nature of the past (looking at a rearview mirror) to predicting the future (looking ahead) to better manage their business operations (making accurate and timely decisions). Data mining helps marketers find patterns that unlock the mysteries of customer behavior. The results of data mining can be used to increase revenue and reduce cost by identifying fraud and discovering business opportunities, offering a whole new realm of competitive advantage. As an evolving and maturing field, data mining is often associated with a number of myths, including those listed in Table 4.6 (Delen, 2014; Zaima, 2003).
Data mining visionaries have gained enormous competitive advantage by under- standing that these myths are just that: myths.
Although the value proposition and therefore its necessity are obvious to anyone, those who carry out data mining projects (from novice to seasoned data scientist) some- times make mistakes that result in projects with less-than-desirable outcomes. The follow- ing 16 data mining mistakes (also called blunders, pitfalls, or bloopers) are often made in practice (Nisbet et al., 2009; Shultz, 2004; Skalak, 2001), and data scientists should be aware of them and, to the extent that is possible, do their best to avoid them:
1. Selecting the wrong problem for data mining. Not every business problem can be solved with data mining (i.e., the magic bullet syndrome). When there are no represen- tative data (large and feature rich), there cannot be a practicable data mining project.
2. Ignoring what your sponsor thinks data mining is and what it really can and cannot do. Expectation management is the key for successful data mining projects.
TABLE 4.6 Data Mining Myths
Myth Reality
Data mining provides instant, crystal-ball- like predictions.
Data mining is a multistep process that requires deliberate, proactive design and use.
Data mining is not yet viable for mainstream business applications.
The current state of the art is ready for almost any business type and/or size.
Data mining requires a separate, dedicated database.
Because of the advances in database technology, a dedicated database is not required.
Only those with advanced degrees can do data mining.
Newer Web-based tools enable managers of all educational levels to do data mining.
Data mining is only for large firms that have lots of customer data.
If the data accurately reflect the business or its customers, any company can use data mining.
be off limits or treated extremely cautiously, such as terminal disease, divorce, and bankruptcy.
Questions for Case 4.7
1. What do you think about data mining and its implication for privacy? What is the threshold between discovery of knowledge and infringe- ment of privacy?
2. Did Target go too far? Did it do anything ille- gal? What do you think Target should have done? What do you think Target should do next (quit these types of practices)?
Sources: K. Hill, “How Target Figured Out a Teen Girl Was Pregnant Before Her Father Did,” Forbes, February 16, 2012; R. Nolan, “Behind the Cover Story: How Much Does Target Know?”, February 21, 2012. NYTimes.com.
Application Case 4.7 (Continued)
Chapter 4 • Data Mining Process, Methods, and Algorithms 245
3. Beginning without the end in mind. Although data mining is a process of knowledge discovery, one should have a goal/objective (a stated business problem) in mind to succeed. Because, as the saying goes, “If you don’t know where you are going, you will never get there.”
4. Defining the project around a foundation that your data cannot support. Data mining is all about data; that is, the biggest constraint that you have in a data mining project is the richness of the data. Knowing what the limitations of data are helps you craft feasible projects that deliver results and meet expectations.
5. Leaving insufficient time for data preparation. It takes more effort than is generally understood. The common knowledge suggests that up to one-third of the total proj- ect time is spent on data acquisition, understanding, and preparation tasks. To suc- ceed, avoid proceeding into modeling until after your data are properly processed (aggregated, cleaned, and transformed).
6. Looking only at aggregated results, not at individual records. Data mining is at its best when the data are at a granular representation. Try to avoid unnecessarily ag- gregating and overly simplifying data to help data mining algorithms—they don’t really need your help; they are more than capable of figuring it out themselves.
7. Being sloppy about keeping track of the data mining procedure and results. Because data mining is a discovery process that involves many iterations and experimenta- tions, its user is highly likely to lose track of the findings. Success requires a system- atic and orderly planning, execution, and tracking/recording of all data mining tasks.
8. Using data from the future to predict the future. Because of the lack of description and understanding of the data, oftentimes analysts include variables that are unknown at the time when the prediction is supposed to be made. By doing so, their prediction models produce unbelievably accurate results (a phenomenon that is often called fool’s gold). If your prediction results are too good to be true, they usually are; in that case, the first thing that you need to look for is the incorrect use of a variable from the future.
9. Ignoring suspicious findings and quickly moving on. The unexpected findings are often the indicators of real novelties in data mining projects. Proper investigation of such oddities can lead to surprisingly pleasing discoveries.
10. Starting with a high-profile complex project that will make you a superstar. Data mining projects often fail if they are not thought out carefully from start to end. Success often comes with a systematic and orderly progression of projects from smaller/simpler to larger/complex ones. The goal should be to show incremental and continuous value added as opposed to taking on a large project that will con- sume resources without producing any valuable outcomes.
11. Running data mining algorithms repeatedly and blindly. Although today’s data mining tools are capable of consuming data and setting algorithmic parameters to produce results, one should know how to transform the data and set the proper parameter values to obtain the best possible results. Each algorithm has its own unique way to process data, and knowing that is necessary to get the most out of each model type.
12. Ignore the subject matter experts. Understanding the problem domain and the related data requires a highly involved collaboration between the data mining and the domain experts. Working together helps the data mining expert to go beyond the syntactic representation and to obtain semantic nature (i.e., the true meaning of the variables) of the data.
13. Believing everything you are told about the data. Although it is necessary to talk to domain experts to better understand the data and the business problem, the data scientist should not take anything for granted. Validation and verification through a critical analysis is the key to intimate understanding and processing of the data.
14. Assuming that the keepers of the data will be fully on board with cooperation. Many data mining projects fail because the data mining expert did not know/understand the organizational politics. One of the biggest obstacles in data mining projects can
246 Part II • Predictive Analytics/Machine Learning
be the people who own and control the data. Understanding and managing the politics is a key to identify, access, and properly understand the data to produce a successful data mining project.
15. Measuring your results differently from the way your sponsor measures them. The results should talk/appeal to the end user (manager/decision maker) who will be using them. Therefore, producing the results in a measure and format that appeals to the end user tremendously increases the likelihood of true understanding and proper use of the data mining outcomes.
16. Follow the advice in a well-known quote: “If you build it, they will come”: don’t worry about how to serve it up. Usually, data mining experts think they have finished once they build models that meet and hopefully exceed the needs/wants/expectations of the end user (i.e., the customer). Without a proper deployment, the value deliverance of data mining outcomes is rather limited. Therefore, deployment is a necessary last step in the data mining process in which models are integrated into the organizational decision support infrastructure for enablement of better and faster decision making.
uSECTION 4.7 REVIEW QUESTIONS
1. What are the privacy issues in data mining? 2. How do you think the discussion between privacy and data mining will progress? Why? 3. What are the most common myths about data mining? 4. What do you think are the reasons for these myths about data mining? 5. What are the most common data mining mistakes/blunders? How can they be allevi-
ated or completely eliminated?
Chapter Highlights
• Data mining is the process of discovering new knowledge from databases.
• Data mining can use simple flat files as data sources, or it can be performed on data in data warehouses.
• There are many alternative names and definitions for data mining.
• Data mining is at the intersection of many dis- ciplines, including statistics, artificial intelligence, and mathematical modeling.
• Companies use data mining to better understand their customers and optimize their operations.
• Data mining applications can be found in virtually every area of business and government, including healthcare, finance, marketing, and homeland security.
• Three broad categories of data mining tasks are prediction (classification or regression), cluster- ing, and association.
• Similar to other IS initiatives, a data mining proj- ect must follow a systematic project management process to be successful.
• Several data mining processes have been pro- posed: CRISP-DM, SEMMA, KDD, for example.
• CRISP-DM provides a systematic and orderly way to conduct data mining projects.
• The earlier steps in data mining projects (i.e., un- derstanding the domain and the relevant data) consume most of the total project time (often more than 80% of the total time).
• Data preprocessing is essential to any successful data mining study. Good data lead to good infor- mation; good information leads to good decisions.
• Data preprocessing includes four main steps: data consolidation, data cleaning, data transformation, and data reduction.
• Classification methods learn from previous ex- amples containing inputs and the resulting class labels, and once properly trained, they are able to classify future cases.
• Clustering partitions pattern records into natural segments or clusters. Each segment’s members share similar characteristics.
• A number of different algorithms are commonly used for classification. Commercial implementa- tions include ID3, C4.5, C5, CART, CHAID, and SPRINT.
• Decision trees partition data by branching along different attributes so that each leaf node has all the patterns of one class.
Chapter 4 • Data Mining Process, Methods, and Algorithms 247
• The Gini index and information gain (entropy) are two popular ways to determine branching choices in a decision tree.
• The Gini index measures the purity of a sample. If everything in a sample belongs to one class, the Gini index value is zero.
• Several assessment techniques can measure the prediction accuracy of classification models, in- cluding simple split, k-fold cross-validation, boot- strapping, and the area under the ROC curve.
• There are a number of methods to assess the vari- able importance of data mining models. Some of these methods are model type specific, some are model type agnostic.
• Cluster algorithms are used when data records do not have predefined class identifiers (i.e., it is not known to what class a particular record belongs).
• Cluster algorithms compute measures of similarity in order to group similar cases into clusters.
• The most commonly used similarity measure in cluster analysis is a distance measure.
• The most commonly used clustering algorithms are k-means and self-organizing maps.
• Association rule mining is used to discover two or more items (or events or concepts) that go together.
• Association rule mining is commonly referred to as market-basket analysis.
• The most commonly used association algorithm is Apriori by which frequent itemsets are identi- fied through a bottom-up approach.
• Association rules are assessed based on their sup- port and confidence measures.
• Many commercial and free data mining tools are available.
• The most popular commercial data mining tools are IBM SPSS Modeler and SAS Enterprise Miner.
• The most popular free data mining tools are KNIME, RapidMiner, and Weka.
Key Terms
Apriori algorithm area under the ROC curve association bootstrapping categorical data classification clustering confidence CRISP-DM data mining
decision tree distance measure ensemble entropy Gini index information gain interval data k-fold cross-
validation KNIME
knowledge discovery in databases (KDD)
lift link analysis Microsoft Enterprise
Consortium Microsoft SQL Server nominal data numeric data ordinal data
prediction RapidMiner regression SEMMA sensitivity analysis sequence mining simple split support Weka
Questions for Discussion
1. Define data mining. Why are there many names and definitions for data mining?
2. What are the main reasons for the recent popularity of data mining?
3. Discuss what an organization should consider before making a decision to purchase data mining software.
4. Distinguish data mining from other analytical tools and techniques.
5. Discuss the main data mining methods. What are the fundamental differences among them?
6. What are the main data mining application areas? Discuss the commonalities of these areas that make them a prospect for data mining studies.
7. Why do we need a standardized data mining pro- cess? What are the most commonly used data mining processes?
8. Discuss the differences between the two most com- monly used data mining processes.
9. Are data mining processes a mere sequential set of activities? Explain.
10. Why do we need data preprocessing? What are the main tasks and relevant techniques used in data preprocessing?
11. Discuss the reasoning behind the assessment of clas- sification models.
12. What is the main difference between classification and clustering? Explain using concrete examples.
13. Moving beyond the chapter discussion, where else can association be used?
14. What are the privacy issues with data mining? Do you think they are substantiated?
15. What are the most common myths and mistakes about data mining?
248 Part II • Predictive Analytics/Machine Learning
Exercises
Teradata University Network (TUN) and Other Hands-On Exercises
1. Visit teradatauniversitynetwork.com. Identify case studies and white papers about data mining. Describe recent developments in the field of data mining and predictive modeling.
2. Go to teradatauniversitynetwork.com. Locate Web seminars related to data mining. In particular, locate and watch a seminar given by C. Imhoff and T. Zouqes. Then answer the following questions: a. What are some of the interesting applications of data
mining? b. What types of payoffs and costs can organizations
expect from data mining initiatives? 3. For this exercise, your goal is to build a model to iden-
tify inputs or predictors that differentiate risky custom- ers from others (based on patterns pertaining to previous customers) and then use those inputs to predict new risky customers. This sample case is typical for this domain.
The sample data to be used in this exercise are in Online File W4.1 in the file CreditRisk.xlsx. The data set has 425 cases and 15 variables pertaining to past and current customers who have borrowed from a bank for various reasons. The data set contains customer-related information such as financial standing, reason for the loan, employment, demographic information, and the outcome or dependent variable for credit standing, clas- sifying each case as good or bad based on the institu- tion’s past experience.
Take 400 of the cases as training cases and set aside the other 25 for testing. Build a decision tree mod- el to learn the characteristics of the problem. Test its per- formance on the other 25 cases. Report on your model’s learning and testing performance. Prepare a report that identifies the decision tree model and training param- eters as well as the resulting performance on the test set. Use any decision tree software. (This exercise is cour- tesy of StatSoft, Inc., based on a German data set from ftp.ics.uc,i.edu/pub/machine-learning-databases/ statlog/german renamed CreditRisk and altered.)
4. For this exercise, you will replicate (on a smaller scale) the box-office prediction modeling explained in Application Case 4.6. Download the training data set from Online File W4.2, MovieTrain.xlsx, which is in Microsoft Excel format. Use the data description given in Application Case 4.6 to understand the domain and the problem you are trying to solve. Pick and choose your independent variables. Develop at least three clas- sification models (e.g., decision tree, logistic regression, neural networks). Compare the accuracy results using 10-fold cross-validation and percentage split techniques, use confusion matrices, and comment on the outcome. Test the models you have developed on the test set (see Online File W4.3, MovieTest.xlsx). Analyze the results
with different models, and find the best classification model, supporting it with your results.
5. This exercise introduces you to association rule min- ing. The Excel data set baskets1ntrans.xlsx has around 2,800 observations/records of supermarket transaction products data. Each record contains the customer’s ID and products that they have purchased. Use this data set to understand the relationships among products (i.e., which products are purchased together). Look for inter- esting relationships and add screenshots of any subtle association patterns that you might find. More specifi- cally, answer the following questions. • Which association rules do you think are most important? • Based on some of the association rules you found,
make at least three business recommendations that might be beneficial to the company. These recom- mendations can include ideas about shelf organiza- tion, up-selling, or cross-selling products. (Bonus points will be given to new/innovative ideas.)
• What are the Support, Confidence, and Lift values for the following rule?
Wine, Canned Veg S Frozen Meal 6. In this assignment, you will use a free/open source
data mining tool, KNIME (knime.org), to build pre- dictive models for a relatively small Customer Churn Analysis data set. You are to analyze the given data set (about the customer retention/attrition behavior for 1,000 customers) to develop and compare at least three prediction (i.e., classification) models. For exam- ple, you can include decision trees, neural networks, SVM, k nearest neighbor, and/or logistic regression models in your comparison. Here are the specifics for this assignment: • Install and use the KNIME software tool from
(knime.org). • You can also use MS Excel to preprocess the data (if
you need to/want to). • Download CustomerChurnData.csv data file from the
book’s Web site. • The data are given in comma-separated value (CSV)
format. This format is the most common flat-file for- mat that many software tools can easily open/handle (including KNIME and MS Excel).
• Present your results in a well-organized professional document.
• Include a cover page (with proper information about you and the assignment).
• Make sure to nicely integrate figures (graphs, charts, tables, screenshots) within your textual description in a professional manner. The report should have six main sections (resembling CRISP-DM phases).
• Try not to exceed 15 pages in total, including the cover (use 12-point Times New Roman fonts, and 1.5- line spacing).
Chapter 4 • Data Mining Process, Methods, and Algorithms 249
Team Assignments and Role-Playing Projects
1. Examine how new data capture devices such as RFID tags help organizations accurately identify and segment their customers for activities such as targeted marketing. Many of these applications involve data mining. Scan the literature and the Web and then propose five potential new data mining applications that can use the data cre- ated with RFID technology. What issues could arise if a country’s laws required such devices to be embedded in everyone’s body for a national identification system?
2. Interview administrators in your college or executives in your organization to determine how data mining, data warehousing, Online Analytics Processing (OLAP), and visualization tools could assist them in their work. Write a proposal describing your findings. Include cost esti- mates and benefits in your report.
3. A very good repository of data that has been used to test the performance of many data mining algorithms is available at ics.uci.edu/~mlearn/MLRepository.html. Some of the data sets are meant to test the limits of cur- rent machine-learning algorithms and to compare their performance with new approaches to learning. However, some of the smaller data sets can be useful for exploring the functionality of any data mining software, such as RapidMiner or KNIME. Download at least one data set from this repository (e.g., Credit Screening Databases, Housing Database) and apply decision tree or clustering methods, as appropriate. Prepare a report based on your results. (Some of these exercises, especially the ones that involve large/challenging data/problem may be used as semester-long term projects.)
4. Large and feature-rich data sets are made available by the U.S. government or its subsidiaries on the Internet. For instance, see a large collection of government data sets (data.gov), the Centers for Disease Control and Prevention data sets (www.cdc.gov/DataStatistics), Surveillance, Cancer.org’s Epidemiology and End Results data sets (http://seer.cancer.gov/data), and the Department of Transportation’s Fatality Analysis Reporting System crash data sets (www.nhtsa.gov/FARS). These data sets are not preprocessed for data mining, which makes them a great resource to experience the complete data mining process. Another rich source for a collec- tion of analytics data sets is listed on KDnuggets.com (KDnuggets.com/datasets/index.html).
5. Consider the following data set, which includes three attributes and a classification for admission decisions into an MBA program:
GMAT GPA Quantitative GMAT Score (percentile) Decision
650 2.75 35 No
580 3.50 70 No
600 3.50 75 Yes
450 2.95 80 No
700 3.25 90 Yes
GMAT GPA Quantitative GMAT Score (percentile) Decision
590 3.50 80 Yes
400 3.85 45 No
640 3.50 75 Yes
540 3.00 60 ?
690 2.85 80 ?
490 4.00 65 ?
a. Using the data shown, develop your own manual expert rules for decision making.
b. Use the Gini index to build a decision tree. You can use manual calculations or a spreadsheet to perform the basic calculations.
c. Use an automated decision tree software program to build a tree for the same data.
Internet Exercises
1. Visit the AI Exploratorium at cs.ualberta.ca/~aixplore. Click the Decision Tree link. Read the narrative on bas- ketball game statistics. Examine the data, and then build a decision tree. Report your impressions of its accuracy. Also explore the effects of different algorithms.
2. Survey some data mining tools and vendors. Start with fico.com and egain.com. Consult dmreview.com, and identify some data mining products and service providers that are not mentioned in this chapter.
3. Find recent cases of successful data mining applications. Visit the Web sites of some data mining vendors, and look for cases or success stories. Prepare a report sum- marizing five new case studies.
4. Go to vendor Web sites (especially those of SAS, SPSS, Cognos, Teradata, StatSoft, and Fair Isaac) and look at success stories for BI (OLAP and data mining) tools. What do the various success stories have in common? How do they differ?
5. Go to statsoft.com (now a Dell company). Download at least three white papers on applications. Which of these applications might have used the data/text/Web mining techniques discussed in this chapter?
6. Go to sas.com. Download at least three white papers on applications. Which of these applications could have used the data/text/Web mining techniques discussed in this chapter?
7. Go to spss.com (an IBM company). Download at least three white papers on applications. Which of these applications could have used the data/text/Web mining techniques discussed in this chapter?
8. Go to teradata.com. Download at least three white papers on applications. Which of these applications could have used the data/text/Web mining techniques discussed in this chapter?
9. Go to fico.com. Download at least three white papers on applications. Which of these applications could have used the data/text/Web mining techniques discussed in this chapter?
250 Part II • Predictive Analytics/Machine Learning
10. Go to salfordsystems.com. Download at least three white papers on applications. Which of these applica- tions could have used the data/text/Web mining tech- niques discussed in this chapter?
11. Go to rulequest.com. Download at least three white papers on applications. Which of these applications
could have used the data/text/Web mining techniques discussed in this chapter?
12. Go to KDnuggets.com. Explore the sections on applications as well as software. Find names of at least three additional packages for data mining and text mining.
References
Chan, P., Phan, W., Prodromidis, A., & Stolfo, S. (1999). “Dis- tributed Data Mining in Credit Card Fraud Detection.” IEEE Intelligent Systems, 14(6), 67–74.
CRISP-DM. (2013). “Cross-Industry Standard Process for Data Min- ing (CRISP-DM).” http://crisp-dm.orgwww.the- modeling- agency.com/crisp-dm.pdf (accessed February 2, 2013).
Davenport, T. (2006, January). “Competing on Analytics.” Harvard Business Review, 99–107.
Delen, D. (2009). “Analysis of Cancer Data: A Data Mining Approach.” Expert Systems, 26(1), 100–112.
Delen, D. (2014). Real-World Data Mining: Applied Business Analytics and Decision Making. Upper Saddle River, NJ: Pearson.
Delen, D., Cogdell, D., & Kasap, N. (2012). “A Comparative Analysis of Data Mining Methods in Predicting NCAA Bowl Outcomes.” International Journal of Forecasting, 28, 543–552.
Delen, D., Sharda, R., & Kumar, P. (2007). “Movie Forecast Guru: A Web-Based DSS for Hollywood Managers.” Deci- sion Support Systems, 43(4), 1151–1170.
Delen, D., Walker, G., & Kadam, A. (2005). “Predicting Breast Cancer Survivability: A Comparison of Three Data Mining Methods.” Artificial Intelligence in Medicine, 34(2), 113–127.
Dunham, M. (2003). Data Mining: Introductory and Ad- vanced Topics. Upper Saddle River, NJ: Prentice Hall.
Fayyad, U., Piatetsky-Shapiro, G., & Smyth, P. (1996). “From Knowl- edge Discovery in Databases.” AI Magazine, 17(3), 37–54.
Hoffman, T. (1998, December 7). “Banks Turn to IT to Re- claim Most Profitable Customers.” Computerworld.
Hoffman, T. (1999, April 19). “Insurers Mine for Age- Appropriate Offering.” Computerworld.
Kohonen, T. (1982). “Self-Organized Formation of Topologically Correct Feature Maps.” Biological Cybernetics, 43(1), 59–69.
Nemati, H., & Barko, C. (2001). “Issues in Organizational Data Mining: A Survey of Current Practices.” Journal of Data Warehousing, 6(1), 25–36.
Nisbet, R., Miner, G., & Elder IV, J. (2009). “Top 10 Data Min- ing Mistakes.” Handbook of Statistical Analysis and Data Mining Applications. Academic Press, pp. 733–754.
Quinlan, J. (1986). “Induction of Decision Trees.” Machine Learning, 1, 81–106.
Saltelli, A. (2002). “Making Best Use of Model Evaluations to Compute Sensitivity Indices,” Computer Physics Communi- cations, 145, 280–297.
Saltelli, A., Tarantola, S., Campolongo, F., & Ratto, M. (2004). Sensitivity Analysis in Practice – A Guide to Assessing Sci- entific Models. Hoboken, NJ: John Wiley.
SEMMA. (2009). “SAS’s Data Mining Process: Sample, Explore, Modify, Model, Assess.” sas.com/offices/europe/uk/ technologies/analytics/datamining/miner/semma. html (accessed August 2009).
Sharda, R., & Delen, D. (2006). “Predicting Box-Office Success of Motion Pictures with Neural Networks.” Expert Systems with Applications, 30, 243–254.
Shultz, R. (2004, December 7). “Live from NCDM: Tales of Database Buffoonery.” directmag.com/news/ncdm-12- 07-04/index.html (accessed April 2009).
Skalak, D. (2001). “Data Mining Blunders Exposed!” DB2 Magazine, 6(2), 10–13.
Thongkam, J., Xu, G., Zhang, Y., & Huang, F. (2009). “Toward Breast Cancer Survivability Prediction Models Through Im- proving Training Space.” Expert Systems with Applications, 36(10), 12200–12209.
Wald, M. (2004, February 21). “U.S. Calls Release of JetBlue Data Improper.” The New York Times.
Wright, C. (2012). “Statistical Predictors of March Mad- ness: An Examination of the NCAA Men’s Basketball Championship.” http://economics-files.pomona. edu/GarySmith/Econ190/Wright%20March%20 Madness%20Final%20Paper.pdf (accessed February 2, 2013).
Zaima, A. (2003). “The Five Myths of Data Mining.” What Works: Best Practices in Business Intelligence and Data Warehousing, Vol. 15. Chatsworth, CA: The Data Ware- housing Institute, pp. 42–43.
Zolbanin, H., Delen, D., & Zadeh, A. (2015). “Predicting Over- all Survivability in Comorbidity of Cancers: A Data Mining Approach.” Decision Support Systems, 74, 150–161.
