Business Intelligence IV

364 Part II • Predictive Analytics/Machine Learning

notes or social media feedback. In today’s world, unstructured notes are part of core communications in virtually every industry, for example:

• Medical professionals record patient obser- vations.

• Auto technicians write down safety information. • Retailers track social media for consumer

comments. • Call centers monitor customer feedback and

take notes.

Bringing together notes, which are usually avail- able as free-form text, with other data for analysis has been difficult. That is because each industry has its own specific terms, slang, shorthand, and acronyms embed- ded in the data. Finding meaning and business insights first requires the text to be changed into a structured form. This manual process is expensive, time consum- ing, and prone to errors, especially as data scales to ever-increasing volumes. One way that companies can leverage notes without codifying the text is to use text clustering. This analytic technique quickly identifies common words or phrases for rapid insights.

Text and Notes Can Lead to New and Improved Products

Leveraging the insights and customer sentiment uncovered during a text and sentiment analysis can spark innovation. Companies such as vehicle manufacturers can use the intelligence to improve customer service and deliver an elevated customer experience. By learning what customers like and dislike about current products, companies can improve their design, such as adding new features to a vehicle to enhance the driving experience.

Forming word clusters also allows companies to identify safety issues. If an auto manufacturer sees that numerous customers are expressing negative sen- timents about black smoke coming from their vehicle, the company can respond. Likewise, manufacturers can address safety issues that are a concern to custom- ers. With comments grouped into buckets, companies have the ability to focus on specific customers who experienced a similar problem. This allows a com- pany to, for instance, offer a rebate or special promo- tion to those who experienced black smoke.

Understanding sentiments can better inform a vehicle manufacturer’s policies. For example,

customers have different lifetime values. A cus- tomer who complains just once but has a very large lifetime value can be a more urgent candidate for complaint resolution than a customer with a lower lifetime value with multiple issues. One may have spent $5,000 buying the vehicle from a used vehicle lot. Another may have a history of buying new cars from the manufacturer and spent $30,000 to buy the vehicle on the showroom floor.

Analyzing Notes Enables High-Value Business Outcomes

Managing the life cycle of products and services continues to be a struggle for most companies. The massive volumes of data now available have com- plicated life cycle management, creating new chal- lenges for innovation. At the same time, the rapid rise of consumer feedback through social media has left businesses without a strategy for digesting, mea- suring, or incorporating the information into their product innovation cycle—meaning they miss a cru- cial amount of intelligence that reflects a customer’s actual thoughts, feelings, and emotions.

Text and sentiment analysis is one solution to this problem. Deconstructing topics from masses of text allows companies to see what common issues, com- plaints, or positive or negative sentiments customers have about products. These insights can lead to high- value outcomes, such as improving products or cre- ating new ones that deliver a better user experience, responding timely to safety issues, and identifying which product lines are most popular with consumers.

Example: Visualizing Auto Issues with “The Safety Cloud”

The Teradata Art of Analytics uses data science, Teradata® Aster® Analytics, and visualization tech- niques to turn data into one-of-a-kind artwork. To demonstrate the unique insights offered by text clus- tering, data scientists used the Art of Analytics to create “The Safety Cloud.”

The scientists used advanced analytics algo- rithms on safety inspector and call center notes from an automobile manufacturer. The analytics identi- fied and systematically extracted common words and phrases embedded in the data volumes. The blue cluster represents power steering failure. The pink is engine stalls. Yellow is black smoke in the exhaust.

Application Case 6.7 (Continued)

Chapter 6 • Deep Learning and Cognitive Computing 365

Orange is brake failure. The manufacturer can use this information to gauge how big the problem is and whether it is safety related, and if so, then take actions to fix it.

For a visual summary, you can watch the video (http://www.teradata.com/Resources/Videos/ Art-of-Analytics-Safety-Cloud).

Questions for Case 6.7

1. Why do you think sentiment analysis is gaining overwhelming popularity?

2. How does sentiment analysis work? What does it produce?

3. In addition to the specific examples in this case, can you think of other businesses and industries that can benefit from sentiment analysis? What is common among the companies that can benefit greatly from sentiment analysis?

Source: Teradata Case Study. “Deliver Innovation by Understanding Customer Sentiments.” http://assets.teradata. com/resourceCenter/downloads/CaseStudies/EB9859.pdf (accessed August 2018). Used with permission.

LSTM Networks Applications

Since their emergence in the late 1990s (Hochreiter & Schmidhuber, 1997), LSTM networks have been widely used in many sequence modeling applications, includ- ing image captioning (i.e., automatically describing the content of images) (Vinyals, Toshev, Bengio, and Erhan, 2017, 2015; Xu et al., 2015), handwriting recognition and generation (Graves, 2013; Graves and Schmidhuber, 2009; Keysers et al. 2017), parsing (Liang et al. 2016; Vinyals, Kaiser, et al., 2015), speech recognition (Graves and Jaitly, 2014; Graves, Jaitly, and Mohamed, 2013; Graves, Mohamed, and Hinton, 2013), and machine translation (Bahdanau, Cho, and Bengio, 2014; Sutskever, Vinyals, and Le, 2014).

366 Part II • Predictive Analytics/Machine Learning

Currently, we are surrounded by multiple deep learning solutions working on the basis of speech recognition, such as Apple’s Siri, Google Now, Microsoft’s Cortana, and Amazon’s Alexa, several of which we deal with on a daily basis (e.g., checking on the weather, asking for a Web search, calling a friend, and asking for directions on the map). Note taking is not a difficult, frustrating task anymore since we can easily record a speech or lecture, upload the digital recording on one of the several cloud-based speech-to-text service providers’ platforms, and download the transcript in a few seconds. The Google cloud-based speech-to-text service, for example, supports 120 languages and their vari- ants and has the ability to convert speech to text either in real time or using recorded audios. The Google service automatically handles the noise in the audio; accurately punc- tuates the transcripts with commas, question marks, and periods; and can be customized by the user to a specific context by getting a set of terms and phrases that are very likely to be used in a speech and recognizing them appropriately.

Machine translation refers to a subfield of AI that employs computer programs to translate speech or text from one language to another. One of the most comprehensive machine translation systems is the Google’s Neural Machine Translation (GNMT) platform. GNMT is basically an LSTM network with eight encoder and eight decoder layers designed by a group of Google researchers in 2016 (Wu et al., 2016). GNMT is specialized for trans- lating whole sentences at a time as opposed to the former version of Google Translate platform, which was a phrase-based translator. This network is capable of naturally han- dling the translation of rare words (that previously was a challenge in machine translation) by dividing the words into a set of common subword units. GNMT currently supports au- tomatic sentence translations between more than 100 languages. Figure 6.34 shows how a sample sentence was translated from French to English by GNMT and a human translator. It also indicates how closely the GNMT translations between different language pairs were ranked by the human speakers compared with translations made by humans.

For the former secretary of state, this is to forget a month of bungling and convince the audience that Mr. Trump has not the makings of a president

Phrase Based†

Input Sentence

Neural Network† Human

English French

Chinese

Spanish

Spanish

French

Chinese

Translation Method Phrase Based† Neural Network† Human

543

Pour l’ancienne secrétaire d’Etat, il s’agit de faire oublier un mois de cafouillages et de convaincre l’auditoire que M. Trump n’a pas l’étoffe d’un président

For the former secretary of state, it is a question of forgetting a month of muddles and convincing the audience that Mr. Trump does not have the stuff of a president

The former secretary of state has to put behind her a month of setbacks and convince the audience that Mr. Trump does not have what it takes to be a president

Perfect Translation 5 6

English

English

English

FIGURE 6.34 Example Indicating the Close-to-Human Performance of the Google Neural Machine Translator (GNMT)

Chapter 6 • Deep Learning and Cognitive Computing 367

Although machine translation has been revolutionized by the virtue of LSTMs, it en- counters challenges that make it far from a fully automated high-quality translation. Like image-processing applications, there is a lack of sufficient training data (manually trans- lated by humans) for many language pairs on which the network can be trained. As a result, translations between rare languages are usually done through a bridging language (mostly English) that may result in higher chances of error.

In 2014, Microsoft launched its Skype Translator service, a free voice translation service involving both speech recognition and machine translation with the ability of translating real-time conversations in 10 languages. Using this service, people speaking different languages can talk to each other in their own languages via a Skype voice or video call, and the system recognizes their voices and translates their every sentence through a translator bot in near real time for the other party. To provide more accurate translations, the deep networks used in the backend of this system were trained using conversational language (i.e., using materials such as translated Web pages, movie sub- titles, and casual phrases taken from people’s conversations in social networking Web sites) rather than the formal language commonly used in documents. The output of the speech recognition module of the system then goes through TrueText, a Microsoft tech- nology for normalizing text that is capable of identifying mistakes and disfluencies (e.g., pauses during the speech or repeating some parts of speech, or adding fillers like “um” and “ah” when speaking) that people commonly conduct in their conversations and ac- count for them for making better translations. Figure 6.35 shows the four-step process involved in the Skype Translator by Microsoft, each of which relies on the LSTM type of deep neural networks.

u SECTION 6.8 REVIEW QUESTIONS

1. What is RNN? How does it differ from CNN? 2. What is the significance of “context,” “sequence,” and “memory” in RNN? 3. Draw and explain the functioning of a typical recurrent neural network unit. 4. What is the LSTM network, and how does it differ from RNNs? 5. List and briefly describe three different types of LSTM applications. 6. How do Google’s Neural Machine Translation and Microsoft Skype Translator work?

Can you hear me?

can can you hear me

Speech

Automatic Speech Recognition Machine

Translation

Text to Speech

?

me

hear

You

Can

Speech

TrueText

can can you here me

hear

A B C

FIGURE 6.35 Four-Step Process of Translating Speech Using Deep Networks in the Microsoft Skype Translator.

368 Part II • Predictive Analytics/Machine Learning

6.9 COMPUTER FRAMEWORKS FOR IMPLEMENTATION OF DEEP LEARNING

Advances in deep learning owe its recent popularity, to a great extent, to advances in the software and hardware infrastructure required for its implementation. In the past few de- cades, GPUs have been revolutionized to support the playing of high-resolution videos as well as advanced video games and virtual reality applications. However, GPUs’ huge pro- cessing potential had not been effectively utilized for purposes other than graphics pro- cessing up until a few years ago. Thanks to software libraries such as Theano (Bergstra et al., 2010), Torch (Collobert, Kavukcuoglu, and Farabet, 2011), Caffe (Jia et al., 2014), PyLearn2 (Goodfellow et al., 2013), Tensorflow (Abadi et al., 2016), and MXNet (Chen et al., 2015) developed with the purpose of programming GPUs for general-purpose processing (just as CPUs), and particularly for deep learning and analysis of Big Data, GPUs have become a critical enabler for the modern-day analytics. The operation of these libraries mostly relies on a parallel computing platform and application programming in- terface (API) developed by NVIDIA called Compute Unified Device Architecture (CUDA), which enables software developers to use GPUs made by NVIDIA for general-purpose processing. In fact, each deep learning framework consists of a high-level scripting lan- guage (e.g., Python, R, Lua) and a library of deep learning routines usually written in C (for using CPUs) or CUDA (for using GPUs).

We next introduce some of the most popular software libraries used for deep learn- ing by researchers and practitioners, including Torch, Caffe, Tensorflow, Theano, and Keras, and discuss some of their specific properties.

Torch

Torch (Collobert et al., 2011) is an open-source scientific computing framework (avail- able at www.torch.ch) for implementing machine-learning algorithms using GPUs. The Torch framework is a library based on LuaJIT, a compiled version of the popular Lua pro- gramming language (www.lua.org). In fact, Torch adds a number of valuable features to Lua that make deep learning analyses possible; it enables supporting n-dimensional arrays (i.e., tensors), whereas tables (i.e., two-dimensional arrays) normally are the only data-structuring method used by Lua. Additionally, Torch includes routine libraries for manipulating (i.e., indexing, slicing, transposing) tensors, linear algebra, neural network functions, and optimization. More importantly, while Lua by default uses CPU to run the programs, Torch enables use of GPUs for running programs written in the Lua language.

The easy and extremely fast scripting properties of LuaJIT along with its flexibility have made Torch a very popular framework for practical deep learning applications such that today its latest version, Torch7, is widely used by a number of big companies in the deep learning area, including Facebook, Google, and IBM, in their research labs, as well as for their commercial applications.

Caffe

Caffe is another open-source deep learning framework (available at http://caffe. berkeleyvision.org) created by Yangqing Jia (2013), a PhD student at the University of California–Berkeley, which the Berkeley AI Research (BAIR) then further developed. Caffe has multiple options to be used as a high-level scripting language, including the command line, Python, and MATLAB interfaces. The deep learning libraries in Caffe are written in the C++ programming language.

In Caffe, everything is done using text files instead of code. That is, to implement a network, generally we need to prepare two text files with the .prototxt extension that are communicated by the Caffe engine via JavaScript Object Notation (JSON) format.

Chapter 6 • Deep Learning and Cognitive Computing 369

The first text file, known as the architecture file, defines the architecture of the network layer by layer, where each layer is defined by a name, a type (e.g., data, convolution, output), the names of its previous (bottom) and next (top) layers in the architecture, and some required parameters (e.g., kernel size and stride for a convolutional layer). The sec- ond text file, known as the solver file, specifies the properties of the training algorithm, including the learning rate, maximum number of iterations, and processing unit (CPU or GPU) to be used for training the network.

While Caffe supports multiple types of deep network architectures like CNN and LSTM, it is particularly known to be an efficient framework for image processing due to its incredible speed in processing image files. According to its developers, it is able to pro- cess over 60 million images per day (i.e., 1 ms/image) using a single NVIDIA K40 GPU. In 2017, Facebook released an improved version of Caffe called Caffe2 (www.caffe2.ai) with the aim of improving the original framework to be effectively used for deep learning architectures other than CNN and with a special emphasis on portability for performing cloud and mobile computations while maintaining scalability and performance.

TensorFlow

Another popular open-source deep learning framework is TensorFlow. It was origi- nally developed and written in Python and C++ by the Google Brain Group in 2011 as DistBelief, but it was further developed into TensorFlow in 2015. TensorFlow at this time is the only deep learning framework that, in addition to CPUs and GPUs, supports Tensor Processing Units (TPUs), a type of processor developed by Google in 2016 for the specific purpose of neural network machine learning. In fact, TPUs were specifically designed by Google for the TensorFlow framework.

Although Google has not yet made TPUs available to the market, it is reported that it has used them in a number of its commercial services such as Google search, Street View, Google Photos, and Google Translate with significant improvements reported. A detailed study performed by Google shows that TPUs deliver 30 to 80 times higher perfor- mance per watt than contemporary CPUs and GPUs (Sato, Young, and Patterson, 2017). For example, it has been reported (Ung, 2016) that in Google Photos, an individual TPU can process over 100 million images per day (i.e., 0.86 ms/image). Such a unique feature will probably put TensorFlow way ahead of the other alternative frameworks in the near future as soon as Google makes TPUs commercially available.

Another interesting feature of TensorFlow is its visualization module, TensorBoard. Implementing a deep neural network is a complex and confusing task. TensorBoard re- fers to a Web application involving a handful of visualization tools to visualize network graphs and plot quantitative network metrics with the aim of helping users to better un- derstand what is going on during training procedures and to debug possible issues.

Theano

In 2007, the Deep Learning Group at the University of Montreal developed the initial version of a Python library, Theano (http://deeplearning.net/software/theano), to define, optimize, and evaluate mathematical expressions involving multi-dimensional ar- rays (i.e., tensors) on CPU or GPU platforms. Theano was one of the first deep learning frameworks but later became a source of inspiration for the developers of TensorFlow. Theano and TensorFlow both pursue a similar procedure in the sense that in both a typi- cal network implementation involves two sections: in the first section, a computational graph is built by defining the network variables and operations to be done on them; and the second section runs that graph (in Theano by compiling the graph into a function and in TensorFlow by creating a session). In fact, what happens in these libraries is that the user defines the structure of the network by providing some simple and symbolic

370 Part II • Predictive Analytics/Machine Learning

syntax understandable even for beginners in programming, and the library automatically generates appropriate codes in either C (for processing on CPU) or CUDA (for process- ing on GPU) to implement the defined network. Hence, users without any knowledge of programming in C or CUDA and with just a minimum knowledge of Python are able to efficiently design and implement deep learning networks on the GPU platforms.

Theano also includes some built-in functions to visualize computational graphs as well as to plot the network performance metrics even though its visualization features are not comparable to TensorBoard.

Keras: An Application Programming Interface

While all described deep learning frameworks require users to be familiar with their own syntax (through reading their documentations) to be able to successfully train a network, fortunately there are some easier, more user-friendly ways to do so. Keras (https:// keras.io/) is an open-source neural network library written in Python that functions as a high-level application programming interface (API) and is able to run on top of various deep learning frameworks including Theano and TensorFlow. In essence, Keras just by getting the key properties of network building blocks (i.e., type of layers, transfer func- tions, and optimizers) via an extremely simple syntax automatically generates syntax in one of the deep learning frameworks and runs that framework in the backend. While Keras is efficient enough to build and run general deep learning models in just a few minutes, it does not provide several advanced operations provided by TensorFlow or Theano. Therefore, in dealing with special deep network models that require advanced settings, one still needs to directly use those frameworks instead of Keras (or other APIs such as Lasagne) as a proxy.

u SECTION 6.9 REVIEW QUESTIONS

1. Despite the short tenure of deep learning implementation, why do you think there are several different computing frameworks for it?

2. Define CPU, NVIDIA, CUDA, and deep learning, and comment on the relationship between them.

3. List and briefly define the characteristics of different deep learning frameworks. 4. What is Keras, and how is it different from the other frameworks?

6.10 COGNITIVE COMPUTING

We are witnessing a significant increase in the way technology is evolving. Things that once took decades are now taking months, and the things that we see only in SciFi movies are becoming reality, one after another. Therefore, it is safe to say that in the next decade or two, technological advancements will transform how people live, learn, and work in a rather dramatic fashion. The interactions between humans and technology will become in- tuitive, seamless, and perhaps transparent. Cognitive computing will have a significant role to play in this transformation. Generally speaking, cognitive computing refers to the com- puting systems that use mathematical models to emulate (or partially simulate) the human cognition process to find solutions to complex problems and situations where the potential answers can be imprecise. While the term cognitive computing is often used interchange- ably with AI and smart search engines, the phrase itself is closely associated with IBM’s cognitive computer system Watson and its success on the television show Jeopardy! Details on Watson’s success on Jeopardy! can be found in Application Case 6.8.

According to Cognitive Computing Consortium (2018), cognitive computing makes a new class of problems computable. It addresses highly complex situations that are

Chapter 6 • Deep Learning and Cognitive Computing 371

characterized by ambiguity and uncertainty; in other words, it handles the kinds of prob- lems that are thought to be solvable by human ingenuity and creativity. In today’s dy- namic, information-rich, and unstable situations, data tend to change frequently, and they often conflict. The goals of users evolve as they learn more and redefine their objectives. To respond to the fluid nature of users’ understanding of their problems, the cognitive computing system offers a synthesis not just of information sources but also of influences, contexts, and insights. To achieve such a high-level of performance, cognitive systems often need to weigh conflicting evidence and suggest an answer that is “best” rather than “right.” Figure 6.36 illustrates a general framework for cognitive computing where data and AI technologies are used to solve complex real-world problems.

How Does Cognitive Computing Work?

As one would guess from the name, cognitive computing works much like a human thought process, reasoning mechanism, and cognitive system. These cutting-edge compu- tation systems can find and synthesize data from various information sources and weigh context and conflicting evidence inherent in the data to provide the best possible answers to a given question or problem. To achieve this, cognitive systems include self-learning technologies that use data mining, pattern recognition, deep learning, and NLP to mimic the way the human brain works.

Outcomes

Cognitive Computing

Saved lives

Improved economy

Better security

Engaged customers

Higher revenues Reduced risks

Improved living

Test

Built

Validate

Structured Data

(POS, transactions, OLAP, CRM, SCM,

external, etc.)

Unstructured Data

(social media, multimedia, loT, literature, etc.)

Complex Problems

(health, economic, humanitarian, social, etc.)

AI Algorithms Soft/Hardware

(machine learning, NLP, search, cloud,

GPU, etc.)

FIGURE 6.36 Conceptual Framework for Cognitive Computing and Its Promises.

372 Part II • Predictive Analytics/Machine Learning

Using computer systems to solve the types of problems that humans are typically tasked with requires vast amounts of structured and unstructured data fed to machine- learning algorithms. Over time, cognitive systems are able to refine the way in which they learn and recognize patterns and the way they process data to become capable of antici- pating new problems and modeling and proposing possible solutions.

To achieve those capabilities, cognitive computing systems must have the following key attributes as defined by the Cognitive Computing Consortium (2018):

• Adaptive: Cognitive systems must be flexible enough to learn as information changes and goals evolve. The systems must be able to digest dynamic data in real time and make adjustments as the data and environment change.

• Interactive: Human-computer interaction (HCI) is a critical component in cogni- tive systems. Users must be able to interact with cognitive machines and define their needs as those needs change. The technologies must also be able to interact with other processors, devices, and cloud platforms.

• Iterative and stateful: Cognitive computing technologies can also identify prob- lems by asking questions or pulling in additional data if a stated problem is vague or incomplete. The systems do this by maintaining information about similar situa- tions that have previously occurred.

• Contextual: Understanding context is critical in thought processes, so cogni- tive systems must understand, identify, and mine contextual data, such as syntax, time, location, domain, requirements, and a specific user’s profile, tasks, or goals. Cognitive systems may draw on multiple sources of information, including struc- tured and unstructured data and visual, auditory, or sensor data.

How Does Cognitive Computing Differ from AI?

Cognitive computing is often used interchangeably with AI, the umbrella term used for technologies that rely on data and scientific methods/computations to make (or help/sup- port in making) decisions. But there are differences between the two terms, which can largely be found within their purposes and applications. AI technologies include—but are not limited to—machine learning, neural computing, NLP, and, most recently, deep learn- ing. With AI systems, especially in machine-learning systems, data are fed into the algo- rithm for processing (an iterative and time-demanding process that is often called training) so that the systems “learn” variables and interrelationships among those variables so that it can produce predictions (or characterizations) about a given complex problem or situa- tion. Applications based on AI and cognitive computing include intelligent assistants, such as Amazon’s Alexa, Google Home, and Apple’s Siri. A simple comparison between cogni- tive computing and AI is given in Table 6.3 (Reynolds and Feldman, 2014; CCC, 2018).

As can be seen in Table 6.3, the differences between AI and cognitive computing are rather marginal. This is expected because cognitive computing is often character- ized as a subcomponent of AI or an application of AI technologies tailored for a specific purpose. AI and cognitive computing both utilize similar technologies and are applied to similar industry segments and verticals. The main difference between the two is the pur- pose: while cognitive computing is aimed at helping humans to solve complex problems, AI is aimed at automating processes that are performed by humans; at the extreme, AI is striving to replace humans with machines for tasks requiring “intelligence,” one at a time.

In recent years, cognitive computing typically has been used to describe AI systems that aim to simulate human thought process. Human cognition involves real-time analysis of environment, context, and intent among many other variables that inform a person’s ability to solve problems. A number of AI technologies are required for a computer sys- tem to build cognitive models that mimic human thought processes, including machine learning, deep learning, neural networks, NLP, text mining, and sentiment analysis.

Chapter 6 • Deep Learning and Cognitive Computing 373

In general, cognitive computing is used to assist humans in their decision-making process. Some examples of cognitive computing applications include supporting medical doctors in their treatment of disease. IBM Watson for Oncology, for example, has been used at Memorial Sloan Kettering Cancer Center to provide oncologists evidence-based treatment options for cancer patients. When medical staff input questions, Watson generates a list of hypotheses and offers treatment options for doctors to consider. Whereas AI relies on algo- rithms to solve a problem or to identify patterns hidden in data, cognitive computing systems have the loftier goal of creating algorithms that mimic the human brain’s reasoning process to help humans solve an array of problems as the data and the problems constantly change.

In dealing with complex situations, context is important, and cognitive computing systems make context computable. They identify and extract context features such as time, location, task, history, or profile to present a specific set of information that is ap- propriate for an individual or for a dependent application engaged in a specific process at a specific time and place. According to the Cognitive Computing Consortium, they provide machine-aided serendipity by wading through massive collections of diverse information to find patterns and then apply those patterns to respond to the needs of the user at a particular moment. In a sense, cognitive computing systems aim at redefining the nature of the relationship between people and their increasingly pervasive digital environment. They may play the role of assistant or coach for the user, and they may act virtually autonomously in many problem-solving situations. The boundaries of the pro- cesses and domains these systems can affect are still elastic and emergent. Their output may be prescriptive, suggestive, instructive, or simply entertaining.

In the short time of its existence, cognitive computing has proved to be useful in many domain and complex situations and is evolving into many more. The typical use cases for cognitive computing include the following:

• Development of smart and adaptive search engines • Effective use of natural language processing • Speech recognition • Language translation • Context-based sentiment analysis

TABLE 6.3 Cognitive Computing versus Artificial Intelligence (AI)

Characteristic Cognitive Computing Artificial Intelligence (AI)

Technologies used • Machine learning • Natural language processing • Neural networks • Deep learning • Text mining • Sentiment analysis

• Machine learning • Natural language processing • Neural networks • Deep learning

Capabilities offered Simulate human thought processes to assist humans in finding solutions to complex problems

Find hidden patterns in a variety of data sources to identify problems and provide potential solutions

Purpose Augment human capability Automate complex processes by acting like a human in certain situations

Industries Customer service, marketing, healthcare, entertainment, service sector

Manufacturing, finance, healthcare, banking, securities, retail, government

374 Part II • Predictive Analytics/Machine Learning

• Face recognition and facial emotion detection • Risk assessment and mitigation • Fraud detection and mitigation • Behavioral assessment and recommendations

Cognitive analytics is a term that refers to cognitive computing–branded technol- ogy platforms, such as IBM Watson, that specialize in processing and analyzing large, unstructured data sets. Typically, word processing documents, e-mails, videos, images, audio files, presentations, Web pages, social media, and many other data formats need to be manually tagged with metadata before they can be fed into a traditional analytics engine and Big Data tools for computational analyses and insight generation. The princi- pal benefit of utilizing cognitive analytics over those traditional Big Data analytics tools is that for cognitive analytics such data sets do not need to be pretagged. Cognitive analyt- ics systems can use machine learning to adapt to different contexts with minimal human supervision. These systems can be equipped with a chatbot or search assistant that un- derstands queries, explains data insights, and interacts with humans in human languages.

Cognitive Search

Cognitive search is the new generation search method that uses AI (advanced indexing, NLP, and machine learning) to return results that are much more relevant to users. Forrester de- fines cognitive search and knowledge discovery solutions as “a new generation of enterprise search solutions that employ AI technologies such as natural language processing and ma- chine learning to ingest, understand, organize, and query digital content from multiple data sources” (Gualtieri, 2017). Cognitive search creates searchable information out of nonsearch- able content by leveraging cognitive computing algorithms to create an indexing platform.

Searching for information is a tedious task. Although current search engines do a very good job in finding relevant information in a timely manner, their sources are limited to publically available data over the Internet. Cognitive search proposes the next genera- tion of search tailored for use in enterprises. It is different from traditional search because, according to Gualtieri (2017), it:

• Can handle a variety of data types. Search is no longer just about unstructured text contained in documents and in Web pages. Cognitive search solutions can also accommodate structured data contained in databases and even nontraditional enter- prise data such as images, video, audio, and machine-/sensor-generated logs from IoT devices.

• Can contextualize the search space. In information retrieval, the context is important. Context takes the traditional syntax-/symbol-driven search to a new level where it is defined by semantics and meaning.

• Employ advanced AI technologies. The distinguishing characteristic of cogni- tive search solutions is that they use NLP and machine learning to understand and organize data, predict the intent of the search query, improve the relevancy of results, and automatically tune the relevancy of results over time.

• Enable developers to build enterprise-specific search applications. Search is not just about a text box on an enterprise portal. Enterprises build search applica- tions that embed search in customer 360 applications, pharma research tools, and many other business process applications. Virtual digital assistants such as Amazon Alexa, Google Now, and Siri would be useless without powerful searches behind the scenes. Enterprises wishing to build similar applications for their customers will also benefit from cognitive search solutions. Cognitive search solutions provide soft- ware development kits (SDKs), APIs, and/or visual design tools that allow develop- ers to embed the power of the search engine in other applications.

Chapter 6 • Deep Learning and Cognitive Computing 375

Figure 6.37 shows the progressive evolution of search methods from good old key- word search to modern-day cognitive search on two dimensions—ease of use and value proposition.

IBM Watson: Analytics at Its Best

IBM Watson is perhaps the smartest computer system built to date. Since the emergence of computers and subsequently AI in the late 1940s, scientists have compared the per- formance of these “smart” machines with human minds. Accordingly, in the mid- to late-1990s, IBM researchers built a smart machine and used the game of chess (generally credited as the game of smart humans) to test its ability against the best of human players. On May 11, 1997, an IBM computer called Deep Blue beat the world chess grandmaster after a six-game match series: two wins for Deep Blue, one for the champion, and three draws. The match lasted several days and received massive media coverage around the world. It was the classic plot line of human versus machine. Beyond the chess contest, the intention of developing this kind of computer intelligence was to make computers able to handle the kinds of complex calculations needed to help discover new drugs and to do the broad financial modeling needed to identify trends and do risk analysis, handle large database searches, and perform massive calculations needed in advanced fields of science.

After a couple of decades, IBM researchers came up with another idea that was perhaps more challenging: a machine that could not only play the American TV quiz show Jeopardy! but also beat the best of the best. Compared to chess, Jeopardy! is much more challenging. While chess is well structured and has very simple rules and therefore is a very good match for computer processing, Jeopardy! is neither simple nor structured. Jeopardy! is a game designed to test human intelligence and creativity. Therefore, a com- puter designed to play the game needed to be a cognitive computing system that can work and think like a human. Making sense of imprecision inherent in human language was the key to success.

Value Proposition

Natural Human Interaction (NHI)

Cognitive Search

Contextual Search

Indexing NLP

Indexing NLP Machine Learning

Semantic Search

Keyword Search

Machine Learning

Natural Language Processing (NLP)

E as

e of

U se

Indexing

Indexing

FIGURE 6.37 Progressive Evolution of Search Methods.

376 Part II • Predictive Analytics/Machine Learning

In 2010, an IBM research team developed Watson, an extraordinary computer system—a novel combination of advanced hardware and software—designed to answer questions posed in natural human language. The team built Watson as part of the DeepQA project and named it after IBM’s first president, Thomas J. Watson. The team that built Watson was looking for a major research challenge: one that could rival the scientific and popular interest of Deep Blue and would have clear relevance to IBM’s business interests. The goal was to advance computational science by exploring new ways for computer technology to affect science, business, and society at large. Accordingly, IBM research undertook a challenge to build Watson as a computer system that could compete at the human champion level in real time on Jeopardy! The team wanted to create a real-time automatic contestant on the show capable of listening, understanding, and responding, not merely a laboratory exercise. Application Case 6.8 provides some of the details on IBM Watson’s participation in the game show.

In 2011, to test its cognitive abilities, Watson com- peted on the quiz show Jeopardy! in the first-ever human-versus-machine matchup for the show. In a two-game, combined-point match (broadcast in three Jeopardy! episodes during February 14–16), Watson beat Brad Rutter, the highest all-time money winner on Jeopardy! and Ken Jennings, the record holder for the longest championship streak (75 days). In these episodes, Watson consistently outperformed its human opponents on the game’s signaling device, but it had trouble responding to a few categories, notably those having short clues containing only a few words. Watson had access to 200 million pages of structured and unstructured content, consum- ing four terabytes of disk storage. During the game, Watson was not connected to the Internet.

Meeting the Jeopardy! challenge required advancing and incorporating a variety of text mining and NLP technologies, including parsing, question classification, question decomposition, automatic source acquisition and evaluation, entity and rela- tionship detection, logical form generation, and knowledge representation and reasoning. Winning at Jeopardy! required accurately computing confi- dence in answers. The questions and content are ambiguous and noisy, and none of the individual algorithms is perfect. Therefore, each component must produce a confidence in its output, and indi- vidual component confidences must be combined to compute the overall confidence of the final

answer. The final confidence is used to determine whether the computer system should risk choosing to answer at all. In Jeopardy! this confidence is used to determine whether the computer will “ring in” or “buzz in” for a question. The confidence must be computed during the time the question is read and before the opportunity to buzz in. This is roughly between one and six seconds with an average around three seconds.

Watson was an excellent example for the rapid advancement of the computing technology and what it is capable of doing. Although still not as creatively/natively smart as human beings, com- puter systems like Watson are evolving to change the world we are living in, hopefully for the better.

Questions for Case 6.8

1. In your opinion, what are the most unique fea- tures about Watson?

2. In what other challenging games would you like to see Watson compete against humans? Why?

3. What are the similarities and differences between Watson’s and humans’ intelligence?

Sources: Ferrucci, D., E. Brown, J. Chu-Carroll, J. Fan, D. Gondek, D. Kalyanpur, A. Lally, J. Murdock, E. Nyberg, J. Prager, N. Schlaefer, and C. Welty. (2010). “Building Watson: An Overview of the DeepQA Project.” AI Magazine, 31(3), pp. 59–79; IBM Corporation. (2011). “The DeepQA Project.” https://researcher.watson.ibm. com/researcher/view_group.php?id=2099 (accessed May 2018).

Application Case 6.8 IBM Watson Competes against the Best at Jeopardy!

Chapter 6 • Deep Learning and Cognitive Computing 377

How Does Watson Do It?

What is under the hood of Watson? How does it do what it does? The system behind Watson, which is called DeepQA, is a massively parallel, text mining–focused, probabilis- tic evidence–based computational architecture. For the Jeopardy! challenge, Watson used more than 100 different techniques for analyzing natural language, identifying sources, finding and generating hypotheses, finding and scoring evidence, and merging and ranking hypotheses. What is far more important than any particular technique the IBM team used was how it combined them in DeepQA such that overlapping approaches could bring their strengths to bear and contribute to improvements in accuracy, confidence, and speed.

DeepQA is architecture with an accompanying methodology that is not specific to the Jeopardy! challenge. These are the overarching principles in DeepQA:

• Massive parallelism. Watson needed to exploit massive parallelism in the con- sideration of multiple interpretations and hypotheses.

• Many experts. Watson needed to be able to integrate, apply, and contextually evaluate a wide range of loosely coupled probabilistic questions and content analytics.

• Pervasive confidence estimation. No component of Watson committed to an answer; all components produced features and associated confidences, scoring dif- ferent question and content interpretations. An underlying confidence-processing substrate learned how to stack and combine the scores.

• Integration of shallow and deep knowledge. Watson needed to balance the use of strict semantics and shallow semantics, leveraging many loosely formed ontologies.

Figure 6.38 illustrates the DeepQA architecture at a very high level. More technical details about the various architectural components and their specific roles and capabilities can be found in Ferrucci et al. (2010).

What Is the Future for Watson?

The Jeopardy! challenge helped IBM address requirements that led to the design of the DeepQA architecture and the implementation of Watson. After three years of intense re- search and development by a core team of about 20 researchers, as well as a significant

Hypothesis n Soft Filtering Evidence Scoring

Hypothesis 3 Soft Filtering Evidence Scoring

Hypothesis 2 Soft Filtering Evidence Scoring

Hypothesis 1

Question (in natural language)

Soft Filtering Evidence Scoring

Question (translation to digital)

Analysis (decomposition)

Primary Search

Answer Sources

Evidence Sources

Synthesis (combining)

Answer (and level of confidence)

Merging and Ranking

... ... ...

Candidate Generation

Support Evidence Retrieval

Deep Evidence Scoring

1 2 3

4 5

FIGURE 6.38 A High-Level Depiction of DeepQA Architecture

378 Part II • Predictive Analytics/Machine Learning

R&D budget, Watson managed to perform at human expert levels in terms of precision, confidence, and speed on the Jeopardy! quiz show.

After the show, the big question was “So what now?” Was developing Watson all for a quiz show? Absolutely not! Showing the rest of the world what Watson (and the cognitive system behind it) could do became an inspiration for the next generation of intelligent information systems. For IBM, it was a demonstration of what is possible with cutting-edge analytics and computational sciences. The message is clear: If a smart ma- chine can beat the best of the best in humans at what they are the best at, think about what it can do for your organizational problems.

The innovative and futuristic technologies that made Watson one of the most ac- claimed technological advances of this decade are being leveraged as computational foundation for several tools to analyze and characterize unstructured data for prediction- type problems. These experimental tools include Tone Analyzer and Personality Insights. Using textual content, these tools have shown the ability to predict outcomes of complex social events and globally popular competitions.

WATSON PREDICTS THE WINNER OF 2017 EUROVISION SONG CONTEST. A tool developed on the foundations of IBM Watson, Watson Tone Analyzer, uses computational linguistics to identify tone in written text. Its broader goal is to have business managers use the Tone Analyzer to understand posts, conversations, and communications of target customer pop- ulations and to respond to their needs and wants in a timely manner. One could, for ex- ample, use this tool to monitor social media and other Web-based content, including wall posts, tweets, product reviews, and discussion boards as well as longer documents such as articles and blog posts. Or one could use it to monitor customer service interactions and support related conversations. Although it sounds as if any other text-based detection system can build on sentiment analysis, Tone Analyzer differs from these systems in that it analyzes and characterizes textual content. Watson Tone Analyzer measures social tenden- cies and opinions, using a version of the Big-5, the five categories of personality traits (i.e., openness, agreeableness, conscientiousness, extroversion, and neuroticism), along with other emotional categories to detect the tone in a given textual content. As an example, Slowey (2017b) used IBM’s Watson Tone Analyzer to predict the winner of the 2017 Eurovision Songs Contest. Using nothing but the lyrics of the previous years’ competitions, Slowey discovered a pattern that suggested most winners had high levels of agreeableness and conscientiousness. The results (produced before the contest) indicated that Portugal would win the contest, and that is exactly what happened. Try it out yourself:

• Go to Watson Tone Analyzer (https://tone-analyzer-demo.ng.bluemix.net). • Copy and paste your own text in the provided text entry field. • Click “Analyze.” • Observe the summary results as well as the specific sentences where specific tones

are the strongest

Another tool built on the linguistic foundations of IBM Watson is Watson Personality Insight, which seems to work quite similar to Watson Tone Analyzer. In another fun applica- tion case, Slowey (2017a) used Watson Personality Insight to predict the winner of the best picture category at the 2017 Oscar Academy Awards. Using the scripts of the movies from the past years, Slowey developed a generalized profile for winners and then compared that profile to those of the newly nominated movies to identify the upcoming winner. Although in this case, Slowey incorrectly predicted Hidden Figures as the winner, the methodology she followed was unique and innovative and hence deserves credit. To try Watson Personality Insight tool yourself, just go to https://personality-insights-demo.ng.bluemix.net/, copy and paste your own textual content into the “Body of Text” section, and observe the outcome.

Chapter 6 • Deep Learning and Cognitive Computing 379

One of the worthiest endeavors for Watson (or Watson-like large-scale cognitive computing systems) is to help doctors and other medical professionals to diagnose dis- eases and identify the best treatment options that would work for an individual patient. Although Watson is new, this very novel and worthy task is not new to the world of com- puting. In the early 1970s, several researchers at Stanford University developed a com- puter system, MYCIN, to identify bacteria causing severe infections, such as bacteremia and meningitis, and to recommend antibiotics with the dosage adjusted for the specifics of an individual patient (Buchanan and Shortliffe, 1984). This six-year effort relied on a rule-based expert system, a type of AI system, where the diagnoses and treatment knowl- edge nuggets/rules were elicited from a large number of experts (i.e., doctors with ample experience in the specific medical domain). The resulting system was then tested on new patients, and its performance was compared to those of the experienced doctors used as the knowledge sources/experts. The results favored MYCIN, providing a clear indication that properly designed and implemented AI-based computer systems can meet and often exceed the effectiveness and efficiency of even the best medical experts. After more than four decades, Watson is now trying to pick up where MYCIN left the mission of using smart computer systems to improve the health and well-being of humans by helping doc- tors with the contextual information that they need to better and more quickly diagnose and treat their patients.

The first industry targeted to utilize Watson was healthcare, followed by security, finance, retail, education, public services, and research. The following sections pro- vide short descriptions of what Watson can do (and, in many cases, is doing) for these industries.

HEALTHCARE AND MEDICINE The challenges that healthcare is facing today are rather big and multifaceted. With the aging U.S. population, which may be partially attributed to better living conditions and advanced medical discoveries fueled by a variety of tech- nological innovations, demand for healthcare services is increasing faster than the supply of resources. As we all know, when there is an imbalance between demand and supply, prices go up and quality suffers. Therefore, we need cognitive systems like Watson to help decision makers optimize the use of their resources in both clinical and managerial settings.

According to healthcare experts, only 20 percent of the knowledge that physicians use to diagnose and treat patients is evidence based. Considering that the amount of medical information available is doubling every five years and that much of these data are unstructured, physicians simply do not have time to read every journal that can help them keep up-to-date with the latest advances. Given the growing demand for services and the complexity of medical decision making, how can healthcare providers address these problems? The answer could be to use Watson or similar cognitive systems that have the ability to help physicians in diagnosing and treating patients by analyzing large amounts of data—both structured data coming from electronic medical record databases and unstructured text coming from physician notes and published literature—to provide evidence for faster and better decision making. First, the physician and the patient can describe symptoms and other related factors to the system in natural language. Watson can then identify the key pieces of information and mine the patient’s data to find rel- evant facts about family history, current medications, and other existing conditions. It can then combine that information with current findings from tests and then can form and test hypotheses for potential diagnoses by examining a variety of data sources—treatment guidelines, electronic medical record data, doctors’ and nurses’ notes, and peer-reviewed research and clinical studies. Next, Watson can suggest potential diagnostics and treat- ment options with a confidence rating for each suggestion.

380 Part II • Predictive Analytics/Machine Learning

Watson also has the potential to transform healthcare by intelligently synthesizing fragmented research findings published in a variety of outlets. It can dramatically change the way medical students learn. It can help healthcare managers to be proactive about upcoming demand patterns, optimally allocate resources, and improve processing of pay- ments. Early examples of leading healthcare providers that use Watson-like cognitive systems include MD Anderson, The Cleveland Clinic, and Memorial Sloan Kettering.

SECURITY As the Internet expands into every facet of our lives—e-commerce, e-business, smart grids for energy, smart homes for remote control of residential gad- gets and appliances—to make things easier to manage, it also opens up the potential for ill-intended people to intrude in our lives. We need smart systems like Watson that are capable of constantly monitoring for abnormal behavior and, when it is identified, preventing people from accessing our lives and harming us. This could be at the corpo- rate or even national security system level; it could also be at the personal level. Such a smart system could learn who we are and become a digital guardian that could make inferences about activities related to our life and alert us whenever abnormal things happen.

FINANCE The financial services industry faces complex challenges. Regulatory measures as well as social and governmental pressures for financial institutions to be more inclusive have increased. And the customers the industry serves are more empowered, demand- ing, and sophisticated than ever before. With so much financial information generated each day, it is difficult to properly harness the appropriate information on which to act. Perhaps the solution is to create smarter client engagement by better understanding risk profiles and the operating environment. Major financial institutions are already working with Watson to infuse intelligence into their business processes. Watson is tackling data- intensive challenges across the financial services sector, including banking, financial plan- ning, and investing.

RETAIL The retail industry is rapidly changing according to customers’ needs and wants. Empowered by mobile devices and social networks that give them easier access to more information faster than ever before, customers have high expectations for products and services. While retailers are using analytics to keep up with those expectations, their big- ger challenge is efficiently and effectively analyzing the growing mountain of real-time insights that could give them a competitive advantage. Watson’s cognitive computing capabilities related to analyzing massive amounts of unstructured data can help retail- ers reinvent their decision-making processes around pricing, purchasing, distribution, and staffing. Because of Watson’s ability to understand and answer questions in natural language, Watson is an effective and scalable solution for analyzing and responding to social sentiment based on data obtained from social interactions, blogs, and customer reviews.

EDUCATION With the rapidly changing characteristics of students—who are more visu- ally oriented/stimulated, constantly connected to social media and social networks, and with increasingly shorter attention spans—what should the future of education and the classroom look like? The next generation of educational systems should be tailored to fit the needs of the new generation with customized learning plans, personalized textbooks (digital ones with integrated multimedia—audio, video, animated graphs/charts, etc.), dynamically adjusted curriculum, and perhaps smart digital tutors and 24/7 personal advi- sors. Watson seems to have what it takes to make all this happen. With its NLP capability, students can converse with it just as they do with their teachers, advisors, and friends.

Chapter 6 • Deep Learning and Cognitive Computing 381

This smart assistant can answer students’ questions, satisfy their curiosity, and help them keep up with the endeavors of the educational journey.

GOVERNMENT For local, regional, and national governments, the exponential rise of Big Data presents an enormous dilemma. Today’s citizens are more informed and em- powered than ever before, and that means they have high expectations for the value of the public sector serving them. And government organizations can now gather enormous volumes of unstructured, unverified data that could serve their citizens, but only if those data can be analyzed efficiently and effectively. IBM Watson’s cognitive computing may help make sense of this data deluge, speeding governments’ decision-making processes and helping public employees to focus on innovation and discovery.

RESEARCH Every year, hundreds of billions of dollars are spent on research and develop- ment, most of it documented in patents and publications, creating an enormous amount of unstructured data. To contribute to the extant body of knowledge, one needs to sift through these data sources to find the outer boundaries of research in a particular field. This is very difficult, if not impossible, work if it is done with traditional means, but Watson can act as a research assistant to help collect and synthesize information to keep people updated on recent findings and insights. For instance, the New York Genome Center is using the IBM Watson cognitive computing system to analyze the genomic data of patients diagnosed with a highly aggressive and malignant brain cancer and to more rapidly deliver personalized, life-saving treatment to patients with this disease (Royyuru, 2014).

u SECTION 6.10 REVIEW QUESTIONS

1. What is cognitive computing, and how does it differ from other computing paradigms? 2. Draw a diagram and explain the conceptual framework of cognitive computing.

Make sure to include inputs, enablers, and expected outcomes in your framework.

3. List and briefly define the key attributes of cognitive computing. 4. How does cognitive computing differ from ordinary AI techniques? 5. What are the typical use cases for cognitive analytics? 6. Explain what the terms cognitive analytics and cognitive search mean. 7. What is IBM Watson and what is its significance to the world of computing? 8. How does Watson work? 9. List and briefly explain five use cases for IBM Watson.

Chapter Highlights

• Deep learning is among the latest trends in AI that come with great expectations.

• The goal of deep learning is similar to those of the other machine-leaning methods, which is to use sophisticated mathematical algorithms to learn from data similar to the way that humans learn.

• What deep learning has added to the classic machine-learning methods is the ability to auto- matically acquire the features required to accom- plish highly complex and unstructured tasks.

• Deep learning belongs to the representation learning within the AI learning family of methods.

• The recent emergence and popularity of deep learning can largely be attributed to very large data sets and rapidly advancing commuting infrastructures.

• Artificial neural networks emulate the way the human brain works. The basic processing unit is a neuron. Multiple neurons are grouped into lay- ers and linked together.

382 Part II • Predictive Analytics/Machine Learning

• In a neural network, knowledge is stored in the weight associated with the connections between neurons.

• Backpropagation is the most popular learning paradigm of feedforward neural networks.

• An MLP-type neural network consists of an input layer, an output layer, and a number of hidden layers. The nodes in one layer are connected to the nodes in the next layer.

• Each node at the input layer typically represents a single attribute that may affect the prediction.

• The usual process of learning in a neural network involves three steps: (1) compute temporary out- puts based on inputs and random weights, (2) compute outputs with desired targets, and (3) ad- just the weights and repeat the process.

• Developing neural network–based systems re- quires a step-by-step process. It includes data preparation and preprocessing, training and test- ing, and conversion of the trained model into a production system.

• Neural network software allows for easy experi- mentation with many models. Although neural network modules are included in all major data mining software tools, specific neural network packages are also available.

• Neural network applications abound in almost all business disciplines as well as in virtually all other functional areas.

• Overfitting occurs when neural networks are trained for a large number of iterations with rela- tively small data sets. To prevent overfitting, the training process is controlled by an assessment process using a separate validation data set.

• Neural networks are known as black-box models. Sensitivity analysis is often used to shed light into the black box to assess the relative importance of input features.

• Deep neural networks broke the generally ac- cepted notion of “no more than two hidden lay- ers are needed to formulate complex prediction problems.” They promote increasing the hidden layer to arbitrarily large numbers to better repre- sent the complexity in the data set.

• MLP deep networks, also known as deep feedfor- ward networks, are the most general type of deep networks.

• The impact of random weights in the learning process of deep MLP is shown to be a signifi- cant issue. Nonrandom assignment of the initial weights seems to significantly improve the learn- ing process in deep MLP.

• Although there is no generally accepted theoreti- cal basis for this, it is believed and empirically shown that in deep MLP networks, multiple lay- ers perform better and converge faster than few layers with many neurons.

• CNNs are arguably the most popular and most successful deep learning methods.

• CNNs were initially designed for computer vision applications (e.g., image processing, video process- ing, text recognition) but also have been shown to be applicable to nonimage or non-text data sets.

• The main characteristic of the convolutional net- works is having at least one layer involving a convolution weight function instead of general matrix multiplication.

• The convolution function is a method to address the issue of having too many network weight pa- rameters by introducing the notion of parameter sharing.

• In CNN, a convolution layer is often followed by another layer known as the pooling (a.k.a. sub- sampling) layer. The purpose of a pooling layer is to consolidate elements in the input matrix in order to produce a smaller output matrix while maintaining the important features.

• ImageNet is an ongoing research project that provides researchers with a large database of images, each linked to a set of synonym words (known as synset) from WordNet (a word hierar- chy database).

• AlexNet is one of the first convolutional net- works designed for image classification using the ImageNet data set. Its success rapidly popularized the use and reputation of CNNs.

• GoogLeNet (a.k.a. Inception), a deep convolu- tional network architecture designed by Google researchers, was the winning architecture at ILSVRC 2014.

• Google Lens is an app that uses deep learning ar- tificial neural network algorithms to deliver infor- mation about the images captured by users from their nearby objects.

• Google’s word2vec project remarkably increased the use of CNN-type deep learning for text min- ing applications.

• RNN is another deep learning architecture de- signed to process sequential inputs.

• RNNs have memory to remember previous in- formation in determining context-specific, time- dependent outcomes.

• A variation of RNN, the LSTM network is today known as the most effective sequence modeling

Chapter 6 • Deep Learning and Cognitive Computing 383

technique and is the base of many practical applications.

• Two emerging LSTM applications are Google Neural Machine Translator and Microsoft Skype Translator.

• Deep learning implementation frameworks include Torch, Caffe, TensorFlow, Theano, and Keras.

• Cognitive computing makes a new class of prob- lems computable by addressing highly complex situations that are characterized by ambiguity and uncertainty; in other words, it handles the kinds of problems that are thought to be solvable by human ingenuity and creativity.

• Cognitive computing finds and synthesizes data from various information sources and weighs the context and conflicting evidence inherent in the data in order to provide the best possible answers to a given question or problem.

• The key attributes of cognitive computing include adaptability, interactivity, being iterative, stateful, and contextual.

• Cognitive analytics is a term that refers to cognitive computing–branded technology platforms, such as IBM Watson, that specialize in the processing and analysis of large unstructured data sets.

• Cognitive search is the new generation of search method that uses AI (advanced indexing, NLP, and machine learning) to return results that are much more relevant to the user than traditional search methods.

• IBM Watson is perhaps the smartest computer system built to date. It has coined and popular- ized the term cognitive computing.

• IBM Watson beat the best of men (the two most winning competitors) at the quiz game Jeopardy!, showcasing the ability of commut- ers to do tasks that are designed for human intelligence.

• Watson and systems like it are now in use in many application areas including healthcare, fi- nance, security, and retail.

Key Terms

activation function artificial intelligence (AI) artificial neural networks (ANN) backpropagation black-box syndrome Caffe cognitive analytics cognitive computing cognitive search connection weight constant error carousel (CEC) convolution function convolutional neural network

(CNN) deep belief network (DBN) deep learning deep neural network DeepQA

Google Lens GoogLeNet Google Neural Machine Translator

(GNMT) graphics processing unit (GPU) hidden layer IBM Watson ImageNet Keras long short-term memory (LSTM) machine learning Microsoft Skype Translator multilayer perceptron (MLP) MYCIN network structure neural network neuron overfitting

perceptron performance function pooling processing element (PE) recurrent neural network (RNN) representation learning sensitivity analysis stochastic gradient

descent (SGD) summation function supervised learning TensorFlow Theano threshold value Torch transfer function word embeddings word2vec

Questions for Discussion

1. What is deep learning? What can deep learning do that traditional machine-learning methods cannot?

2. List and briefly explain different learning paradigms/ methods in AI.

3. What is representation learning, and how does it relate to machine learning and deep learning?

4. List and briefly describe the most commonly used ANN activation functions.

5. What is MLP, and how does it work? Explain the function of summation and activation weights in MLP-type ANN.

6. List and briefly describe the nine-step process in con- ducting a neural network project.

384 Part II • Predictive Analytics/Machine Learning

7. Draw and briefly explain the three-step process of learning in ANN.

8. How does the backpropagation learning algorithm work? 9. What is overfitting in ANN learning? How does it hap-

pen, and how can it be prevented? 10. What is the so-called black-box syndrome? Why is

it important to be able to explain an ANN’s model structure?

11. How does sensitivity analysis work in ANN? Search the Internet to find other methods to explain ANN methods.

12. What is meant by “deep” in deep neural networks? Compare deep neural network to shallow neural network.

13. What is GPU? How does it relate to deep neural networks?

14. How does a feedforward multilayer perceptron–type deep network work?

15. Comment on the impact of random weights in develop- ing deep MLP.

16. Which strategy is better: more hidden layers versus more neurons?

17. What is CNN? 18. For what type of applications can CNN be used? 19. What is the convolution function in CNN, and how does

it work? 20. What is pooling in CNN? How does it work? 21. What is ImageNet, and how does it relate to deep

learning? 22. What is the significance of AlexNet? Draw and describe

its architecture. 23. What is GoogLeNet? How does it work? 24. How does CNN process text? What is word embeddings,

and how does it work? 25. What is word2vec, and what does it add to the tradi-

tional text mining?

26. What is RNN? How does it differ from CNN? 27. What is the significance of context, sequence, and mem-

ory in RNN? 28. Draw and explain the functioning of a typical recurrent

neural network unit. 29. What is LSTM network, and how does it differ from

RNNs? 30. List and briefly describe three different types of LSTM

applications. 31. How do Google’s Neural Machine Translation and

Microsoft Skype Translator work? 32. Despite its short tenure, why do you think deep learn-

ing implementation has several different computing frameworks?

33. Define and comment on the relationship between CPU, NVIDIA, CUDA, and deep learning.

34. List and briefly define the characteristics of different deep learning frameworks.

35. What is Keras, and how does it differ from other frameworks?

36. What is cognitive computing and how does it differ from other computing paradigms?

37. Draw a diagram and explain the conceptual frame- work of cognitive computing. Make sure to include inputs, enablers, and expected outcomes in your framework.

38. List and briefly define the key attributes of cognitive computing.

39. How does cognitive computing differ from ordinary AI techniques?

40. What are the typical use cases for cognitive analytics? 41. What is cognitive analytics? What is cognitive search? 42. What is IBM Watson, and what is its significance to the

world of computing? 43. How does IBM Watson work? 44. List and briefly explain five use cases for IBM Watson.

Exercises

Teradata University Network (TUN) and Other Hands-On and Internet Exercises

1. Go to the Teradata University Network Web site (teradatauniversitynetwork.com). Search for teach- ing and learning materials (e.g., articles, application cases, white papers, videos, exercises) on deep learn- ing, cognitive computing, and IBM Watson. Read the material you have found. If needed, also conduct a search on the Web to enhance your findings. Write a report on your findings.

2. Deep learning is relatively new to the world of analytics. Its application cases and success stories are just start- ing to emerge in the Web. Conduct a comprehensive search on your school’s digital library resources to iden- tify at least five journal articles where interesting deep

learning applications are described. Write a report on your findings.

3. Most of the applications of deep learning today are developed using R- and/or Python-based open-source computing resources. Identify those resources (frame- works such as Torch, Caffe, TensorFlow, Theano, Keras) available for building deep learning models and applications. Compare and contrast their capabilities and limitations. Based on your findings and understand- ing of these resources, if you were to develop a deep learning application, which one would you choose to employ? Explain and justify/defend your choice.

4. Cognitive computing has become a popular term to define and characterize the extent of the ability of machines/ computers to show “intelligent” behavior. Thanks to IBM

Chapter 6 • Deep Learning and Cognitive Computing 385

Watson and its success on Jeopardy!, cognitive comput- ing and cognitive analytics are now part of many real- world intelligent systems. In this exercise, identify at least three application cases where cognitive computing was used to solve complex real-world problems. Summarize your findings in a professionally organized report.

5. Download KNIME analytics platform, one of the most popular free/open-source software tools from knime. org. Identify the deep learning examples (where Keras is used to build some exemplary prediction/classifica- tion models) in its example folder. Study the models in detail. Understand what it does and how exactly it does it. Then, using a different but similar data set, build and test your own deep learning prediction model. Report your findings and experiences in a written document.

6. Search for articles related to “cognitive search.” Identify at least five pieces of written material (a combination of journal articles, white papers, blog posts, application cases, etc.). Read and summarize your findings. Explain your understanding of cognitive search and how it dif- fers from regular search methods.

7. Go to Teradata.com. Search and find application case studies and white papers on deep learning and/or cogni- tive computing. Write a report to summarize your find- ings, and comment on the capabilities and limitations (based on your understanding) of these technologies.

8. Go to SAS.com. Search and find application case stud- ies and white papers on deep learning and/or cognitive computing. Write a report to summarize your findings, and comment on the capabilities and limitations (based on your understanding) of these technologies.

9. Go to IBM.com. Search and find application case stud- ies and white papers on deep learning and/or cognitive computing. Write a report to summarize your findings, and comment on the capabilities and limitations (based on your understanding) of these technologies.

10. Go to TIBCO.com or some other advanced analytics company Web site. Search and find application case studies and white papers on deep learning and/or cog- nitive computing. Write a report to summarize your find- ings, and comment on the capabilities and limitations (based on your understanding) of these technologies.

References

Abad, M., P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, . . . M. Isard. (2016). “TensorFlow: A System for Large-Scale Machine Learning.” OSDI, 16, pp. 265–283.

Altman, E. I. (1968). “Financial Ratios, Discriminant Analysis and the Prediction of Corporate Bankruptcy.” The Journal of Finance, 23(4), pp. 589–609.

Bahdanau, D., K. Cho, & Y. Bengio. (2014). “Neural Machine Translation by Jointly Learning to Align and Translate.” ArXiv Preprint ArXiv:1409.0473.

Bengio, Y. (2009). “Learning Deep Architectures for AI.” Foundations and Trends® in Machine Learning, 2(1), pp. 1–127.

Bergstra, J., O. Breuleux, F. Bastien, P. Lamblin, R. Pascanu, G. Desjardins, . . . Y. Bengio. (2010). “Theano: A CPU and GPU Math Compiler in Python.” Proceedings of the Ninth Python in Science Conference, Vol. 1.

Bi, R. (2014). “When Watson Meets Machine Learning.” www. kdnuggets.com/2014/07/watson-meets-machine- learning.html (accessed June 2018).

Boureau, Y.-L., N. Le Roux, F. Bach, J. Ponce, & Y. LeCun (2011). “Ask the Locals: Multi-Way Local Pooling for Image Recognition.” Proceedings of the International Com- puter Vision (ICCV’11) IEEE International Conference, pp. 2651–2658.

Boureau, Y.-L., J. Ponce, & Y. LeCun. (2010). “A Theoretical Analysis of Feature Pooling in Visual Recognition.” Pro- ceedings of International Conference on Machine Learn- ing (ICML’10), pp. 111–118.

Buchanan, B. G., & E. H. Shortliffe. (1984). Rule Based Ex- pert Systems: The MYCIN Experiments of the Stanford

Heuristic Programming Project. Reading, MA: Addison- Wesley.

Cognitive Computing Consortium. (2018). https://cogni- tivecomputingconsortium.com/resources/cognitive- computing-defined/#1467829079735-c0934399- 599a (accessed July 2018).

Chen, T., M. Li, Y. Li, M. Lin, N. Wang, M. Wang, . . . Z. Zhang. (2015). “Mxnet: A Flexible and Efficient Machine Learn- ing Library for Heterogeneous Distributed Systems.” ArXiv Preprint ArXiv:1512.01274.

Collobert, R., K. Kavukcuoglu, & C. Farabet. (2011). “Torch7: A Matlab-like Environment for Machine Learning.” Big- Learn, NIPS workshop.

Cybenko, G. (1989). “Approximation by Superpositions of a Sigmoidal Function.” Mathematics of Control, Signals and Systems, 2(4), 303–314.

DeepQA. (2011). “DeepQA Project: FAQ, IBM Corporation.” https://researcher.watson.ibm.com/researcher/ view_group.php?id=2099 (accessed May 2018).

Delen, D., R. Sharda, & M. Bessonov, M. (2006). “Identifying Sig- nificant Predictors of Injury Severity in Traffic Accidents Us- ing a Series of Artificial Neural Networks.” Accident Analysis & Prevention, 38(3), 434–444.

Denyer, S. (2018, January). “Beijing Bets on Facial Recognition in a Big Drive for Total Surveillance.” The Washington Post. https://www.washingtonpost.com/news/world/ w p / 2 0 1 8 / 0 1 / 0 7 / f e a t u r e / i n - c h i n a - f a c i a l - recognition-is-sharp-end-of-a-drive-for-total- su r ve i l l ance / ?nored i r e c t=on&utm_ te r m= . e73091681b31.

386 Part II • Predictive Analytics/Machine Learning

Feldman, S., J. Hanover, C. Burghard, & D. Schubmehl. (2012). “Unlocking the Power of Unstructured Data.” IBM White Paper. http://.www-01.ibm.com/software/ebusiness/ jstart/downloads/unlockingUnstructuredData.pdf. (accessed May 2018).

Ferrucci, D., E. Brown, J. Chu-Carroll, J. Fan, D. Gondek, Kalyanpur, A. A. Lally, J. W. Murdock, E. Nyberg, J. Prag- er, N. Schlaefer, & C. Welty. (2010). “Building Watson: An Overview of the DeepQA Project.” AI Magazine, 31(3), pp. 59–79.

Goodfellow, I., Y. Bengio, & A. Courville. (2016). “Deep Learning.” Cambridge, MA: MIT Press.

Goodfellow, I. J., D. Warde-Farley, P. Lamblin, V. Dumoulin, M. Mirza, R. Pascanu, . . . Y. Bengio. (2013). “Pylearn2: A Machine Learning Research Library.” ArXiv Preprint ArX- iv:1308.4214.

Graves, A. (2013). “Generating Sequences with Recurrent Neural Networks.” ArXiv Preprint ArXiv:1308.0850.

Graves, A., & N. Jaitly. (2014). “Towards End-to-End Speech Recognition with Recurrent Neural Networks.” Proceed- ings on International Conference on Machine Learning, pp. 1764–1772.

Graves, A., N. Jaitly, & A. Mohamed. (2013). “Hybrid Speech Recognition with Deep Bidirectional LSTM.” IEEE Work- shop on Automatic Speech Recognition and Understand- ing, pp. 273–278.

Graves, A., A. Mohamed, & G. Hinton. (2013). “Speech Recog- nition with Deep Recurrent Neural Networks.” IEEE Acous- tics, Speech and Signal Processing (ICASSP) International Conference, pp. 6645–6649.

Graves, A., & J. Schmidhuber. (2009). “Offline Handwriting Recognition with Multidimensional Recurrent Neural Net- works.” Advances in Neural Information Processing Sys- tems. Cambridge, MA: MIT Press, pp. 545–552.

Gualtieri, M. (2017). “Cognitive Search Is the AI Version of Enterprise Search, Forrester.” go.forrester.com/blogs/ 17-06-12-cognitive_search_is_the_ai_version_of_ enterprise_search/ (accessed July 2018).

Haykin, S. S. (2009). Neural Networks and Learning Machines, 3rd ed. Upper Saddle River, NJ: Prentice Hall.

He, K., X. Zhang, S. Ren, & J. Sun. (2015). “Delving Deep into Rectifiers: Surpassing Human-Level Performance on Imagenet Classification.” Proceedings of the IEEE Interna- tional Conference on Computer Vision, pp. 1026–1034.

Hinton, G. E., S. Osindero, & Y.-W. Teh. (2006). “A Fast Learn- ing Algorithm for Deep Belief Nets.” Neural Computation, 18(7), 1527–1554.

Hochreiter, S., & J. Schmidhuber (1997). “Long Short-Term Memory.” Neural Computation, 9(8), 1735–1780.

Hornik, K. (1991). “Approximation Capabilities of Multilayer Feedforward Networks.” Neural Networks, 4(2), 251–257.

IBM. (2011). “IBM Watson.” www.ibm.com/watson/ (ac- cessed July 2017).

Jia, Y. (2013). “Caffe: An Open Source Convolutional Architec- ture for Fast Feature Embedding.” http://Goo.Gl/Fo9YO8 (accessed June 2018).

Jia, Y., E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, . . . T. Darrell, T. (2014). “Caffe: Convolutional Architecture for Fast Feature Embedding.” Proceedings of the ACM International Conference on Multimedia, pp. 675–678.

Keysers, D., T. Deselaers, H. A. Rowley, L.-L. Wang, & V.  Carbune. (2017). “Multi-Language Online Handwriting Recognition.” IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), pp. 1180–1194.

Krizhevsky, A., I. Sutskever, & G. Hinton. (2012). “Imagenet Classification with Deep Convolutional Neural Networks.” Advances in Neural Information Processing Systems, pp. 1097–1105S.

Kumar, S. (2017). “A Survey of Deep Learning Methods for Relation Extraction.” http://arxiv.org/abs/1705.03645. (accessed June 2018)

LeCun, Y., B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, & L. D. Jackel. (1989). “Backpropagation Ap- plied to Handwritten ZIP Code Recognition.” Neural Com- putation, 1(4), 541–551.

Liang, X., X. Shen, J. Feng, L. Lin, & S. Yan. (2016). “Seman- tic Object Parsing with Graph LSTM.” European Con- ference on Computer Vision. New York, NY: Springer, pp. 125–143.

Mahajan, D., R. Girshick, V. Ramanathan, M. Paluri, & L. van der Maaten. (2018). “Advancing State-of-the-Art Image Recognition with Deep Learning on Hashtags.” https:// code.facebook.com/posts/1700437286678763/ advancing-state-of-the-art-image-recognition-with- deep-learning-on-hashtags/. (accessed June 2018)

Mikolov, T., K. Chen, G. Corrado, & J. Dean. (2013). “Effi- cient Estimation of Word Representations in Vector Space.” ArXiv Preprint ArXiv:1301.3781.

Mikolov, T., I. Sutskever, K. Chen, G. S. Corrado, & J. Dean. (2013). “Distributed Representations of Words and Phrases and Their Compositionality” Advances in Neural Informa- tion Processing Systems, pp. 3111–3119.

Mintz, M., S. Bills, R. Snow, & D. Jurafsky. (2009). “Distant Supervision for Relation Extraction Without Labeled Data.” Proceedings of the Joint Conference of the Forty-Seventh Annual Meeting of the Association for Computational Lin- guistics and the Fourth International Joint Conference on Natural Language Processing of the AFNLP, Vol. 2, pp. 1003–1011.

Mozur, P. (2018, June 8). “Inside China’s Dystopian Dreams: A.I., Shame and Lots of Cameras.” The New York Times, is- sue June 8, 2018.

Nguyen, T. H., & R. Grishman. (2015). “Relation Extraction: Perspective from Convolutional Neural Networks.” Pro- ceedings of the First Workshop on Vector Space Modeling for Natural Language Processing, pp. 39–48.

Olson, D. L., D. Delen, and Y. Meng. (2012). “Comparative Analysis of Data Mining Models for Bankruptcy Predic- tion.” Decision Support Systems, 52(2), pp. 464–473.

Principe, J. C., N. R. Euliano, and W. C. Lefebvre. (2000). Neu- ral and Adaptive Systems: Fundamentals Through Simula- tions. New York: Wiley.

Chapter 6 • Deep Learning and Cognitive Computing 387

Reynolds, H., & S. Feldman. (2014, July/August). “Cognitive Computing: Beyond the Hype.” KM World, 23(7), p. 21.

Riedel, S., L. Yao, & A. McCallum. (2010). “Modeling Rela- tions and Their Mentions Without Labeled Text.” Joint European Conference on Machine Learning and Knowl- edge Discovery in Databases., New York, NY: Springer, pp. 148–163

Robinson, A., J. Levis, & G. Bennett. (2010, October). “Informs to Officially Join Analytics Movement.” ORMS Today.

Royyuru, A. (2014). “IBM’s Watson Takes on Brain Cancer: Analyzing Genomes to Accelerate and Help Clinicians Per- sonalize Treatments.” Thomas J. Watson Research Center, www.research.ibm.com/articles/genomics.shtml (ac- cessed September 2014).

Rumelhart, D. E., G. E. Hinton, & R. J. Williams. (1986). “Learn- ing Representations by Back-Propagating Errors.” Nature, 323(6088), pp. 533.

Russakovsky, O., J. Deng, H. Su, J. Krause, S. Satheesh, S.  Ma,  .  .  . M. Bernstein. (2015). “Imagenet Large Scale Visual Recognition Challenge.” International Journal of Computer Vision, 115(3), 211–252.

Sato, K., C. Young, & D. Patterson. (2017). “An In-Depth Look at Google’s First Tensor Processing Unit (TPU).” https:// cloud.google.com/blog/big-data/2017/05/an-in- depth-look-at-googles-first-tensor-processing-unit- tpu. (accessed June 2018)

Scherer, D., A. Müller, & S. Behnke. (2010). “Evaluation of Pooling Operations in Convolutional Architectures for Object Recognition.” International Conference on Artificial Neural Networks., New York, NY: Springer, 92–101.

Slowey, L. (2017a, January 25). “Winning the Best Picture Os- car: IBM Watson and Winning Predictions.” https://www. ibm.com/blogs/internet-of-things/best-picture-oscar- watson-predicts/(accessed August 2018).

Slowey, L. (2017b, May 10). “Watson Predicts the Winners: Euro- vision 2017.” https://www.ibm.com/blogs/internet-of- things/eurovision-watson-tone-predictions/(accessed August 2018).

Sutskever, I., O. Vinyals, & Q. V. Le. (2014). “Sequence to Sequence Learning with Neural Networks. Advances in Neural Information Processing Systems, pp. 3104–3112.

Ung, G. M. (2016, May). “Google’s Tensor Processing Unit Could Advance Moore’s Law 7 Years into the Future.” PC- World. https://www.pcworld.com/article/3072256/ google-io/googles-tensor-processing-unit-said-to- advance-moores-law-seven-years-into-the-future.html (accessed July 2018).

Vinyals, O., L. Kaiser, T. Koo, S. Petrov, I. Sutskever, & G. Hinton, G. (2015). “Grammar As a Foreign Language.” Advances in Neural Information Processing Systems, pp. 2773–2781.

Vinyals, O., A. Toshev, S. Bengio, & D. Erhan. (2015). “Show and Tell: A Neural Image Caption Generator.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3156–3164.

Vinyals, O., A. Toshev, S. Bengio, & D. Erhan. (2017). “Show and Tell: Lessons Learned from the 2015 MSCOCO Image Cap- tioning Challenge.” Proceedings of the IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4), 652–663.

Wilson, R. L., & R. Sharda. (1994). “Bankruptcy Prediction Using Neural Networks.” Decision Support Systems, 11(5), 545–557.

Wu, Y., M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Ma- cherey, & K. Macherey. (2016). “Google’s Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation.” ArXiv Preprint ArXiv:1609.08144.

Xu, K., J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, & Y. Bengio. (2015). “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention.” Proceedings of the Thirty-Second International Conference on Machine Learning, pp. 2048–2057.

Zeng, D., K. Liu, S. Lai, G. Zhou, & J. Zhao (2014). “Relation Classification via Convolutional Deep Neural Network.” http://doi.org/http://aclweb.org/anthology/C/C14/ C14-1220.pdf. (accessed June 2018).

Zhou, Y.-T., R. Chellappa, A. Vaid, & B. K. Jenkins. (1988). “Image Restoration Using a Neural Network.” IEEE Trans- actions on Acoustics, Speech, and Signal Processing, 36(7), pp. 1141–1151.

388

Text Mining, Sentiment Analysis, and Social Analytics

LEARNING OBJECTIVES

■■ Describe text analytics and understand the need for text mining

■■ Differentiate among text analytics, text mining, and data mining

■■ Understand the different application areas for text mining

■■ Know the process of carrying out a text mining project

■■ Appreciate the different methods to introduce structure- to text-based data

■■ Describe sentiment analysis ■■ Develop familiarity with popular applications of sentiment analysis

■■ Learn the common methods for sentiment analysis ■■ Become familiar with speech analytics as it relates to sentiment analysis

■■ Learn three facets of Web analytics—content, structure, and usage mining

■■ Know social analytics including social media and social network analyses

T his chapter provides a comprehensive overview of text analytics/mining and Web analytics/mining along with their popular application areas such as search engines, sentiment analysis, and social network/media analytics. As we have been witness- ing in recent years, the unstructured data generated over the Internet of Things (IoT) (Web, sensor networks, radio-frequency identification [RFID]–enabled supply chain sys- tems, surveillance networks, etc.) are increasing at an exponential pace, and there is no in- dication of its slowing down. This changing nature of data is forcing organizations to make text and Web analytics a critical part of their business intelligence/analytics infrastructure.

7.1 Opening Vignette: Amadori Group Converts Consumer Sentiments into Near-Real-Time Sales 389

7.2 Text Analytics and Text Mining Overview 392 7.3 Natural Language Processing (NLP) 397 7.4 Text Mining Applications 402 7.5 Text Mining Process 410 7.6 Sentiment Analysis 418

7 C H A P T E R

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 389

7.7 Web Mining Overview 429 7.8 Search Engines 433 7.9 Web Usage Mining 441

7.10 Social Analytics 446

7.1 OPENING VIGNETTE: Amadori Group Converts Consumer Sentiments into Near-Real-Time Sales

BACKGROUND

Amadori Group, or Gruppo Amadori in Italian, is a leading manufacturing company in Italy that produces and markets food products. Headquartered in San Vittore di Cesena, Italy, the company employs more than 7,000 people and operates 16 production plants.

Amadori wanted to evolve its marketing to dynamically align with the changing life- styles and dietary needs of young people aged 25-35. It sought to create fun ways to engage this target segment by exploiting the potential of online marketing and social media. The company wanted to boost brand visibility, encourage customer loyalty, and gauge consumers’ reactions to products and marketing campaigns.

ENGAGING YOUNG ADULTS WITH CREATIVE DIGITAL MARKETING PROMOTIONS

Together with Tecla (a digital business company), Amadori used IBM WebSphere® Portal and IBM Web Content Manager software to create and manage interactive content for four mini websites, or “minisites,” which promote ready-to-eat and quick-to-prepare products that fit young adults’ preferences and lifestyles. For example, to market its new Evviva sausage product, the company created the “Evviva Il Würstel Italiano” minisite and let consumers upload images and videos of themselves attending events organized by Amadori. To encourage participation, the company offered the winner a spot in its next national ad campaign.

With this and other campaigns, the Amadori marketing staff compiled a database of consumer profiles by asking minisite visitors to share data to enter competitions, down- load applications, receive regular newsletters, and sign up for events. Additionally, the company uses Facebook Insights technology to obtain metrics on its Facebook page, including the number of new fans and favorite content.

MONITORING MARKETPLACE PERCEPTIONS OF THE AMADORI BRAND

The company capitalizes on IBM SPSS® Data Collection software to help assess peo- ples’ opinions of its products and draw conclusions about any fluctuation in Amadori brands’ popularity among consumers. For example, as it launched its Evviva campaign

390 Part II • Predictive Analytics/Machine Learning

TV advertisements and Beach Party Tour, Amadori experienced a flood of consumer conversation. The company captured comments about the product from its Web site and social media networks using SPSS software’s sentiment analysis functionality and suc- cessfully adjusted its marketing efforts in near-real time. The software does not depend solely on keyword searches but also analyzes the syntax of languages, connotations, and even slang to reveal hidden speech patterns that help gauge whether comments about the company or Amadori products express a positive, negative, or neutral opinion. Figure 7.1 shows Amadori’s three facets of commerce analytics to improve consumer engagement.

MAINTAINING BRAND INTEGRITY AND CONSISTENCY ACROSS PRODUCT LINES

Building on the success of marketing minisites, Amadori launched a new corporate Web site built on the same IBM portal and content management technology. The company now concentrates on bringing visitors to the corporate Web site. Instead of individual minisites, Amadori offers sections within the Web site, some with different templates and graphics and a user interface specific to a particular marketing or ad campaign. “For example, we introduced a new product that is made from organic, free-range chickens,” says Fabbri. “As part of the marketing plan, we offered webcam viewing in a new sec- tion of our corporate site so visitors could see how the poultry live and grow. We created a new graphic, but the URL, the header and the footer are always the same so visitors understand that they are always in the Amadori site.”

Visitors can move from one section to another, remaining longer and learning about other offerings. With new content added weekly, the Amadori site has become bigger and is gaining greater prominence on Google and other search engines. “The first year after implementation, our Website traffic grew to approximately 240,000 unique visitors, with 30 percent becoming loyal users,” says Fabbri.

KEEPING CONTENT CURRENT AND ENGAGING TO DIVERSE AUDIENCES

With more content and a high volume of traffic on the Web, it is important that visitors continue to easily find what they are looking for no matter how they access the Amadori site. With this aim in mind, the Amadori project team created a content taxonomy organized

Instrumented The interactive digital platform supports rapid, accurate data collection from business partners and customers.

Interconnected The digital platform also provides an integrated view of the company’s end-to-end processes from production plan to marketing and sales.

Intelligent Content management, data collection, and predictive analytics applications monitor and analyze social media relevant to the Amadori brand, helping the company anticipate issues and better align products and market- ing promotions with customers’ needs and desires.

FIGURE 7.1 Smarter Commerce—Improving Consumer Engagement through Analytics.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 391

by role and area of interest. For example, when people visit the Amadori Web site, they see a banner inviting them to “Reorganize the contents.” They can identify themselves as consumers, buyers, or journalists/bloggers and slide selection bars to indicate interest level in corporate, cooking, and/or entertainment information. The content appearing on the site changes in real time based on these selections. “If the visitor identifies himself as a profes- sional buyer interested predominantly in corporate information, the icons he sees at the top of the screen invite him to either view a digital product catalog online or download a PDF,” says Fabbri. “In that same area of the screen, a consumer interested in cooking sees an icon that clicks through to pages with recipes for preparing dishes using Amadori products.”

Amadori’s advanced analytics projects have been producing significant business ben- efits, making a very strong case for the company to venture into more innovative use of social data. Following are a few of the most prevalent ones:

• Boost by 100 percent the company’s ability to dynamically monitor and learn about the health of its brand using sentiment analysis.

• Improve the company’s social media presence by 100 percent using near-real-time marketing insights, gaining 45,000 Facebook fans in less than one year.

• Establish direct communication with the target segment through Web integration with social media.

• Increase sales by facilitating timely promotions such as eCoupons.

As this case illustrates, in this age of Internet and social media, customer-focused companies are in a race to better communicate with their customers to obtain an inti- mate understanding of their needs, wants, likes, and dislikes. Social analytics that builds on social media—providing both content and the social network–related data—enables these companies to gain deeper insights than ever before.

u QUESTIONS FOR THE OPENING VIGNETTE

1. According to the vignette and based on your opinion, what are the challenges that the food industry is facing today?

2. How can analytics help businesses in the food industry to survive and thrive in this competitive marketplace?

3. What were and still are the main objectives for Amadori to embark into analytics? What were the results?

4. Can you think of other businesses in the food industry that utilize analytics to become more competitive and customer focused? If not, an Internet search could help find relevant information to answer this question.

WHAT WE CAN LEARN FROM THIS VIGNETTE

It is safe to say that computer technology, both on the hardware and software fronts, is advancing faster than anything else in the last 50-plus years. Things that were too big, too complex, and impossible to solve are now well within the reach of information technol- ogy. One of the enabling technologies is perhaps text analytics/text mining and its deriva- tive called sentiment analysis. Traditionally, we have created databases to structure the data so that they can be processed by computers. Textual content, on the other hand, has always been meant for humans to process. Can machines do the things that are meant for humans’ creativity and intelligence? Evidently, yes! This case illustrates the viability and value proposition of collecting and processing customer opinions to develop new and improved products and services, managing the company’s brand name, and engag- ing and energizing the customer base for mutually beneficial and closer relationships. Under the overarching name of “digital marketing,” Amadori showcases the use of text

392 Part II • Predictive Analytics/Machine Learning

mining, sentiment analysis, and social media analytics to significantly advance the bottom line through improved customer satisfaction, increased sales, and enhanced brand loyalty.

Sources: IBM Customer Case Study. “Amadori Group Converts Consumer Sentiments into Near-Real-Time Sales.” Used with permission of IBM.

7.2 TEXT ANALYTICS AND TEXT MINING OVERVIEW

The information age that we are living in is characterized by the rapid growth in the amount of data and information collected, stored, and made available in electronic format. A vast majority of business data are stored in text documents that are virtually un- structured. According to a study by Merrill Lynch and Gartner, 85 percent of all corporate data are captured and stored in some sort of unstructured form (McKnight, 2005). The same study also stated that these unstructured data are doubling in size every 18 months. Because knowledge is power in today’s business world and knowledge is derived from data and information, businesses that effectively and efficiently tap into their text data sources will have the necessary knowledge to make better decisions, leading to a com- petitive advantage over those businesses that lag behind. This is where the need for text analytics and text mining fits into the big picture of today’s businesses.

Even though the overarching goal for both text analytics and text mining is to turn unstructured textual data into actionable information through the application of natural language processing (NLP) and analytics, the definitions of these terms are somewhat different, at least to some experts in the field. According to them, “text analytics” is a broader concept that includes information retrieval (e.g., searching and identifying rel- evant documents for a given set of key terms), as well as information extraction, data mining, and Web mining, whereas “text mining” is primarily focused on discovering new and useful knowledge from the textual data sources. Figure 7.2 illustrates the relation- ships between text analytics and text mining along with other related application areas. The bottom of Figure 7.2 lists the main disciplines (the foundation of the house) that play a critical role in the development of these increasingly more popular application areas. Based on this definition of text analytics and text mining, one could simply formulate the difference between the two as follows:

Text Analytics = Information Retrieval + Information Extraction + Data Mining + Web Mining

or simply

Text Analytics = Information Retrieval + Text Mining

Compared to text mining, text analytics is a relatively new term. With the recent emphasis on analytics, as has been the case in many other related technical application areas (e.g., consumer analytics, completive analytics, visual analytics, social analytics), the field of text has also wanted to get on the analytics bandwagon. Although the term text analytics is more commonly used in a business application context, text mining is frequently used in academic research circles. Even though the two can be defined some- what differently at times, text analytics and text mining are usually used synonymously, and we (authors of this book) concur with this.

Text mining (also known as text data mining or knowledge discovery in textual databases) is the semiautomated process of extracting patterns (useful information and knowledge) from large amounts of unstructured data sources. Remember that data mining is the process of identifying valid, novel, potentially useful, and ultimately understandable

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 393

patterns in data stored in structured databases where the data are organized in records structured by categorical, ordinal, or continuous variables. Text mining is the same as data mining in that it has the same purpose and uses the same processes, but with text mining, the input to the process is a collection of unstructured (or less structured) data files such as Word documents, PDF files, text excerpts, and XML files. In essence, text mining can be thought of as a process (with two main steps) that starts with imposing structure on the text-based data sources followed by extracting relevant information and knowledge from these structured text-based data using data mining techniques and tools.

The benefits of text mining are obvious in the areas in which very large amounts of textual data are being generated, such as law (court orders), academic research (research articles), finance (quarterly reports), medicine (discharge summaries), biology (molecular interactions), technology (patent files), and marketing (customer comments). For example, the free-form text-based interactions with customers in the form of complaints (or com- pliments) and warranty claims can be used to objectively identify product and service characteristics that are deemed to be less than perfect and can be used as input to better product development and service allocations. Likewise, market outreach programs and focus groups generate large amounts of data. By not restricting product or service feed- back to a codified form, customers can present, in their own words, what they think about a company’s products and services. Another area where the automated processing of unstructured text has had much impact is in electronic communications and e-mail. Text mining can be used not only to classify and filter junk e-mail but also to automatically

Text Mining “Knowledge Discovery in

Textual Data”

Information Retrieval

Natural Language

Processing

Web Mining

Data Mining

Web Content Mining

Web Structure Mining

Web Usage Mining

Classification

Clustering

Association

Document Matching

Link Analysis

Search Engines

POS Tagging

Lemmatization

Word Disambiguation

TEXT ANALYTICS

Statistics

Artificial Intelligence

Machine Learning

Computer Science

Management Science

Other Disciplines

FIGURE 7.2 Text Analytics, Related Application Areas, and Enabling Disciplines.

394 Part II • Predictive Analytics/Machine Learning

prioritize e-mail based on importance level as well as generate automatic responses (Weng & Liu, 2004). Following are among the most popular application areas of text mining:

• Information extraction. Identifying key phrases and relationships within text by looking for predefined objects and sequences in text by way of pattern matching.

• Topic tracking. Based on a user profile and documents that a user views, pre- dicting other documents of interest to the user.

• Summarization. Summarizing a document to save the reader time. • Categorization. Identifying the main themes of a document and then placing the

document into a predefined set of categories based on those themes. • Clustering. Grouping similar documents without having a predefined set of

categories. • Concept linking. Connecting related documents by identifying their shared con-

cepts and, by doing so, helping users find information that they perhaps would not have found using traditional search methods.

• Question answering. Finding the best answer to a given question through knowledge-driven pattern matching.

See Technology Insights 7.1 for explanations of some of the terms and concepts used in text mining. Application Case 7.1 describes the use of text mining in the insurance industry.

Application Case 7.1 shows how text mining and a variety of user-generated data sources enable Netflix stay innovative in its business practices, generate deeper customer insight, and drive very successful content for its viewers.

TECHNOLOGY INSIGHTS 7.1 Text Mining Terminology

The following list describes some commonly used text mining terms:

• Unstructured data (versus structured data). Structured data have a predetermined format. They are usually organized into records with simple data values (categorical, or- dinal, and continuous variables) and stored in databases. In contrast, unstructured data do not have a predetermined format and are stored in the form of textual documents. In essence, structured data are for the computers to process, whereas unstructured data are for humans to process and understand.

• Corpus. In linguistics, a corpus (plural corpora) is a large and structured set of texts (now usually stored and processed electronically) prepared for the purpose of conducting knowledge discovery.

• Terms. A term is a single word or multiword phrase extracted directly from the corpus of a specific domain by means of NLP methods.

• Concepts. Concepts are features generated from a collection of documents by means of manual, statistical, rule-based, or hybrid categorization methodology. Compared to terms, concepts are the result of higher-level abstraction.

• Stemming. Stemming is the process of reducing inflected words to their stem (or base or root) form. For instance, stemmer, stemming, stemmed are all based on the root stem.

• Stop words. Stop words (or noise words) are words that are filtered out prior to or after processing natural language data (i.e., text). Even though there is no universally accepted list of stop words, most NLP tools use a list that includes articles (a, an, the), preposi- tions (of, on, for), auxiliary verbs (is, are, was, were), and context-specific words that are deemed not to have differentiating value.

• Synonyms and polysemes. Synonyms are syntactically different words (i.e., spelled differently) with identical or at least similar meanings (e.g., movie, film, and motion pic- ture). In contrast, polysemes, which are also called homonyms, are syntactically identi- cal words (i.e., spelled exactly the same) with different meanings (e.g., bow can mean “to bend forward,” “the front of the ship,” “the weapon that shoots arrows,” or “a kind of tied ribbon”).

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 395

• Tokenizing. A token is a categorized block of text in a sentence. The block of text corresponding to the token is categorized according to the function it performs. This as- signment of meaning to blocks of text is known as tokenizing. A token can look like anything; it just needs to be a useful part of the structured text.

• Term dictionary. This is a collection of terms specific to a narrow field that can be used to restrict the extracted terms within a corpus.

• Word frequency. This is the number of times a word is found in a specific document. • Part-of-speech tagging. This is the process of marking the words in a text as corre-

sponding to a particular part of speech (nouns, verbs, adjectives, adverbs, etc.) based on a word’s definition and the context in which it is used.

• Morphology. This is the branch of the field of linguistics and a part of NLP that studies the internal structure of words (patterns of word formation within a language or across languages).

• Term-by-document matrix (occurrence matrix). This term refers to the common representation schema of the frequency-based relationship between the terms and docu- ments in tabular format where terms are listed in columns, documents are listed in rows, and the frequency between the terms and documents is listed in cells as integer values.

• Singular value decomposition (latent semantic indexing). This dimensionality re- duction method is used to transform the term-by-document matrix to a manageable size by generating an intermediate representation of the frequencies using a matrix manipula- tion method similar to principal component analysis.

The Problem

In today’s hyper-connected world, businesses are under enormous pressure to build relationships with fully engaged consumers who keep coming back for more.

In theory, fostering more intimate consumer relationships becomes easier as new sources of data emerge, data volumes continue their unprecedented growth, and technology becomes more sophisticated. These developments should enable businesses to do a much better job of personalizing marketing campaigns and generating precise content recommendations that drive engagement, adoption, and value for subscribers.

Yet achieving an advanced understanding of one’s audience is a continuous process of testing and learning. It demands the ability to quickly gather and reliably analyze thousands, millions, even billions of events every day found in a variety of data sources, formats, and locations—otherwise known as Big Data. Technology platforms crafted to gather these data and conduct the analyses must be powerful enough to deliver timely insights today and flexible enough to change and grow in business and technol- ogy landscapes that morph with remarkable speed.

Netflix, an undisputed leader and innovator in the over-the-top (OTT ) content space, understands this context better than most. It has staked its busi- ness and its brand on delivering highly targeted, personalized experiences for every subscriber— and has even begun using its remarkably detailed insights to change the way it buys, licenses, and develops content, causing many throughout the Media and Entertainment industries to sit up and take notice.

To support these efforts, Netflix leverages Teradata as a critical component of its data and ana- lytics platform. More recently, the two companies partnered to transition Netflix to the Teradata Cloud, which has given Netflix the power and flexibility it needs—and, so, the ability to maintain its focus on those initiatives at the core of its business.

A Model for Data-Driven, Consumer-Focused Business

The Netflix story is a model for data-driven, direct- to-consumer, and subscriber-based companies— and, in fact, for any business that needs engaged audiences to thrive in a rapidly changing world.

Application Case 7.1 Netflix: Using Big Data to Drive Big Engagement: Unlocking the Power of Analytics to Drive Content and Consumer Insight

(Continued )

396 Part II • Predictive Analytics/Machine Learning

After beginning as a mail-order DVD business, Netflix became the first prominent OTT content pro- vider and turned the media world on its head; wit- ness recent decisions by other major media compa- nies to begin delivering OTT content.

One major element in Netflix’s success is the way it relentlessly tweaks its recommendation engines, constantly adapting to meet each consum- er’s preferred style. Most of the company’s streaming activity emerges from its recommendations, which generate enormous consumer engagement and loy- alty. Every interaction a Netflix subscriber has with the service is based on meticulously culled and ana- lyzed interactions—no two experiences are the same.

In addition, as noted above, Netflix has applied its understanding of subscribers and potential subscribers—as individuals and as groups—to make strategic purchasing, licensing, and content develop- ment decisions. It has created two highly successful dramatic series—House of Cards and Orange is the New Black—that are informed in part by the compa- ny’s extraordinary understanding of its subscribers.

While those efforts and the business minds that drive them make up the heart of the company’s busi- ness, the technology that supports these initiatives must be more powerful and reliable than that of its competi- tors. The data and analytics platform must be able to:

• Rapidly and reliably handle staggering work- loads; it must support insightful analysis of bil- lions of transactional events each day—every search, browse, stop, and start—in whatever data format that records the events.

• Work with a variety of analytics approaches, including neural networks, Python, Pig, as well as varied Business Intelligence tools, like Mi- croStrategy.

• Easily scale and contract as necessary with ex- ceptional elasticity.

• Provide a safe and redundant repository for all of the company’s data.

• Fit within the company’s cost structure and de- sired profit margins.

Bringing Teradata Analytics to the Cloud

With these considerations in mind, Netflix and Teradata teamed up to launch a successful ven- ture to bring Netflix’s Teradata Data Warehouse into the cloud.

Power and Maturity: Teradata’s well-earned reputa- tion for exceptional performance is especially im- portant to a company like Netflix, which pounds its analytics platform with hundreds of concurrent queries. Netflix also needed data warehousing and analytics tools that enable complex work- load management—essential for creating differ- ent queues for different users, and thus allowing for the constant and reliable filtering of what each user needs.

Hybrid Analytical Ecosystems and a Unified Data Architecture: Netflix’s reliance on a hybrid analytical ecosystem that leverages Ha- doop where appropriate but refuses to compro- mise on speed and agility was the perfect fit for Teradata. Netflix’s cloud environment relies on a Teradata-Hadoop connector that enables Net- flix to seamlessly move cloud-based data from another provider into the Teradata Cloud. The result is that Netflix can do much of its analytics off a world-class data warehouse in the Teradata Cloud that offers peace-of-mind redundancy, the ability to expand and contract in response to changing business conditions, and a significantly reduced need for data movement. And, Netflix’s no-holds-barred approach to allowing their ana- lysts to use whatever analytical tools fit the bill demanded a unique analytics platform that could accommodate them. Having a partner that works efficiently with the full complement of analyti- cal applications—both its own and other leading software providers—was critical.

Teradata’s Unified Data Architecture (UDA) helps provide this by recognizing that most companies need a safe, cost-effective collection of services, platforms, applications, and tools for smarter data management, processing, and analytics. In turn, organizations can get the most from all their data. The Teradata UDA includes:

• An integrated data warehouse, which enables or- ganizations to access a comprehensive and shared data environment to quickly and reliably opera- tionalize insights throughout an organization.

• A powerful discovery platform offers companies discovery analytics that rapidly unlock insights from all available data using a variety of techniques accessible to mainstream business analysts.

Application Case 7.1 (Continued)

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 397

u SECTION 7.2 REVIEW QUESTIONS

1. What is text analytics? How does it differ from text mining? 2. What is text mining? How does it differ from data mining? 3. Why is the popularity of text mining as an analytics tool increasing? 4. What are some of the most popular application areas of text mining?

7.3 NATURAL LANGUAGE PROCESSING (NLP)

Some of the early text mining applications used a simplified representation called bag-of-words when introducing structure to a collection of text-based documents to classify them into two or more predetermined classes or to cluster them into natu- ral groupings. In the bag-of-words model, text, such as a sentence, paragraph, or complete document, is represented as a collection of words, disregarding the gram- mar or the order in which the words appear. The bag-of-words model is still used in some simple document classification tools. For instance, in spam filtering, an e-mail message can be modeled as an unordered collection of words (a bag-of-words) that is compared against two different predetermined bags. One bag is filled with words found in spam messages and the other is filled with words found in legitimate e-mails. Although some of the words are likely to be found in both bags, the “spam” bag will contain spam-related words such as stock, Viagra, and buy much more frequently than the legitimate bag, which will contain more words related to the user’s friends or workplace. The level of match between a specific e-mail’s bag-of-words and the two bags containing the descriptors determines the membership of the e-mail as either spam or legitimate.

Naturally, we (humans) do not use words without some order or structure. We use words in sentences, which have semantic as well as syntactic structure. Thus, automated techniques (such as text mining) need to look for ways to go beyond the bag-of-words

• A data platform (e.g., Hadoop) provides the means to economically gather, store, and refine all a company’s data and facilitate the type of discovery never before believed possible.

The Proof Is in the Eyeballs

Netflix scrupulously adheres to a few simple and powerful metrics when evaluating the success of its personalization capabilities: eyeballs. Are subscrib- ers watching? Are they watching more? Are they watching more of what interests them?

With engagement always top of mind, it’s no surprise that Netflix is among the world’s leaders in personalizing content to successfully attract and retain profitable consumers. It has achieved this standing by drawing on its understanding that in a rapidly changing business and technology landscape, one key to success is constantly testing new ways of gathering and analyzing data to deliver the most effective and targeted recommendations. Working with technology partners that make such testing pos- sible frees Netflix to focus on its core business.

Moving ahead, Netflix believes that making increased use of cloud-based technology will fur- ther empower its customer engagement initiatives. By relying on technology partners that understand how to tailor solutions and provide peace of mind about the redundancy of Netflix’s data, the company expects to continue its organic growth and expand its capacity to respond nimbly to technological change and the inevitable ebbs and flows of business.

Questions for Case 7.1

1. What does Netflix do? How did they evolve into this current business model?

2. In the case of Netflix, what was it meant to be data-driven and customer-focused?

3. How did Netflix use Teradata technologies in its analytics endeavors?

Source: Teradata Case Study “Netflix: Using Big Data to Drive Big Engagement” https://www.teradata.com/Resources/Case- Studies/Netflix-Using-Big-Data-to-Drive-Big-Engageme (accessed July 2018).

398 Part II • Predictive Analytics/Machine Learning

interpretation and incorporate more and more semantic structure into their operations. The current trend in text mining is toward including many of the advanced features that can be obtained using NLP.

It has been shown that the bag-of-words method might not produce good enough information content for text mining tasks (e.g., classification, clustering, association). A good example of this can be found in evidence-based medicine. A critical component of evidence-based medicine is incorporating the best available research findings into the clinical decision-making process, which involves appraisal of the information collected from the printed media for validity and relevance. Several researchers from the University of Maryland developed evidence assessment models using a bag-of-words method (Lin and Demner-Fushman, 2005). They employed popular machine-learning methods along with more than half a million research articles collected from Medical Literature Analysis and Retrieval System Online (MEDLINE). In their models, the researchers represented each abstract as a bag-of-words, where each stemmed term represented a feature. Despite using popular classification methods with proven experimental design methodologies, their prediction results were not much better than simple guessing, which could indicate that the bag-of-words is not generating a good enough representation of the research ar- ticles in this domain; hence, more advanced techniques such as NLP were needed.

Natural language processing (NLP) is an important component of text mining and is a subfield of artificial intelligence and computational linguistics. It studies the problem of “understanding” the natural human language with the task of converting depictions of human language (such as textual documents) into more formal repre- sentations (in the form of numeric and symbolic data) that are easier for computer programs to manipulate. The goal of NLP is to move beyond syntax-driven text ma- nipulation (which is often called word counting) to a true understanding and process- ing of natural language that considers grammatical and semantic constraints as well as the context.

The definition and scope of the word understanding is one of the major discus- sion topics in NLP. Considering that the natural human language is vague and that a true understanding of meaning requires extensive knowledge of a topic (beyond what is in the words, sentences, and paragraphs), will computers ever be able to un- derstand natural language the same way and with the same accuracy that humans do? Probably not! NLP has come a long way from the days of simple word counting, but it has an even longer way to go to really understand natural human language. The following are just a few of the challenges commonly associated with the implementa- tion of NLP:

• Part-of-speech tagging. It is difficult to mark up terms in a text as corresponding to a particular part of speech (such as nouns, verbs, adjectives, or adverbs) because the part of speech depends not only on the definition of the term but also on the context within which it is used.

• Text segmentation. Some written languages, such as Chinese, Japanese, and Thai, do not have single-word boundaries. In these instances, the text-parsing task requires the identification of word boundaries, which is often difficult. Similar chal- lenges in speech segmentation emerge when analyzing spoken language because sounds representing successive letters and words blend into each other.

• Word sense disambiguation. Many words have more than one meaning. Selecting the meaning that makes the most sense can be accomplished only by tak- ing into account the context within which the word is used.

• Syntactic ambiguity. The grammar for natural languages is ambiguous; that is, multiple possible sentence structures often need to be considered. Choosing the most appropriate structure usually requires a fusion of semantic and contextual information.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 399

• Imperfect or irregular input. Foreign or regional accents and vocal impedi- ments in speech and typographical or grammatical errors in texts make the process- ing of the language an even more difficult task.

• Speech acts. A sentence can often be considered an action by the speaker. The sentence structure alone might not contain enough information to define this action. For example, “Can you pass the class?” requests a simple yes/no answer, whereas “Can you pass the salt?” is a request for a physical action to be performed.

A long-standing dream of the artificial intelligence community is to have algo- rithms that are capable of automatically reading and obtaining knowledge from text. By applying a learning algorithm to parsed text, researchers from Stanford University’s NLP lab have developed methods that can automatically identify the concepts and rela- tionships between those concepts in the text. By applying a unique procedure to large amounts of text, the lab’s algorithms automatically acquire hundreds of thousands of items of world knowledge and use them to produce significantly enhanced repositories for WordNet. WordNet is a laboriously hand-coded database of English words, their definitions, sets of synonyms, and various semantic relations between synonym sets. It is a major resource for NLP applications, but it has proven to be very expensive to build and maintain manually. By automatically inducing knowledge into WordNet, the potential exists to make it an even greater and more comprehensive resource for NLP at a fraction of the cost. One prominent area in which the benefits of NLP and WordNet are already being harvested is in customer relationship management (CRM). Broadly speaking, the goal of CRM is to maximize customer value by better understanding and effectively responding to customers’ actual and perceived needs. An important area of CRM in which NLP is making a significant impact is sentiment analysis. Sentiment analysis is a technique used to detect favorable and unfavorable opinions toward specific products and services using a large number of textual data sources (customer feedback in the form of Web postings). A detailed coverage of sentiment analysis and WordNet is given in Section 7.6.

Analytics in general and text analytics and text mining in particular can be used in the broadcasting industry. Application Case 7.2 provides an example that uses a wide range of analytics capabilities to capture new viewers, predict ratings, and add business value to a broadcasting company.

Over the past 10 years, the cable television sector in the United States has enjoyed a period of growth that has enabled unprecedented creativity in the creation of high-quality content. AMC Networks has been at the forefront of this new golden age of television, producing a string of successful, critically acclaimed shows such as Breaking Bad, Mad Men, and The Walking Dead.

Dedicated to producing quality program- ming and movie content for more than 30 years, AMC Networks owns and operates several of the most popular and award-winning brands in cable

television, producing and delivering distinctive, compelling, and culturally relevant content that engages audiences across multiple platforms.

Getting Ahead of the Game

Despite its success, AMC Networks has no plans to rest on its laurels. As Vitaly Tsivin, SVP Business Intelligence, explains:

We have no interest in standing still. Although a large percentage of our business is still linear

Application Case 7.2 AMC Networks Is Using Analytics to Capture New Viewers, Predict Ratings, and Add Value for Advertisers in a Multichannel World

(Continued )

400 Part II • Predictive Analytics/Machine Learning

cable TV, we need to appeal to a new gen- eration of millennials who consume content in very different ways.

TV has evolved into a multichannel, mul- tistream business, and cable networks need to get smarter about how they market to and connect with audiences across all of those streams. Relying on traditional ratings data and third-party analytics providers is going to be a losing strategy: you need to take ownership of your data, and use it to get a richer picture of who your viewers are, what they want, and how you can keep their attention in an increas- ingly crowded entertainment marketplace

Zoning in on the Viewer

The challenge is that there is just so much information available—hundreds of billions of rows of data from industry data-providers such as Nielsen and com- Score, from channels such as AMC’s TV Everywhere live Web streaming and video-on-demand service,

from retail partners such as iTunes and Amazon, and from third-party online video services such as Netflix and Hulu.

“We can’t rely on high-level summaries; we need to be able to analyze both structured and unstructured data, minute-by-minute and viewer- by-viewer,” says Tsivin. “We need to know who’s watching and why—and we need to know it quickly so that we can decide, for example, whether to run an ad or a promo in a particular slot during tomor- row night’s episode of Mad Men.”

AMC decided it needed to develop an industry- leading analytics capability in-house and focused on delivering this capability as quickly as possible. Instead of conducting a prolonged and expensive vendor and product selection process, AMC decided to leverage its existing relationship with IBM as its trusted strategic technology partner. The time and money traditionally spent on procurement were instead invested in realizing the solution, accelerat- ing AMC’s progress on its analytics roadmap by at least six months.

Application Case 7.2 (Continued)

Web-Based Dashboard Used by AMC Networks. Source: Used with permission of AMC Networks.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 401

Empowering the Research Department

In the past, AMC’s research team spent a large por- tion of its time processing data. Today, thanks to its new analytics tools, it is able to focus most of its energy on gaining actionable insights.

“By investing in big data analytics technology from IBM, we’ve been able to increase the pace and detail of our research an order of magnitude,” says Tsivin. “Analyses that used to take days and weeks are now possible in minutes, or even seconds.” He added,

Bringing analytics in-house will provide major ongoing cost-savings. Instead of paying hun- dreds of thousands of dollars to external ven- dors when we need some analysis done, we can do it ourselves—more quickly, more accu- rately, and much more cost-effectively. We’re expecting to see a rapid return on investment.

As more sources of potential insight become available and analytics becomes more strategic to the business, an in-house approach is really the only viable way forward for any network that truly wants to gain competitive advantage from its data.

Driving Decisions with Data

Many of the results delivered by this new analytics capability demonstrate a real transformation in the way AMC operates. For example, the company’s business intelligence department has been able to create sophis- ticated statistical models that help the company refine its marketing strategies and make smarter decisions about how intensively it should promote each show.

Instrumented AMC combines ratings data with viewer information from a wide range of digital channels: its own video- on-demand and live-streaming services, retailers, and online TV services.

Interconnected A powerful and comprehensive big data and analytics engine centralizes the data and makes them available to a range of descriptive and predictive analytics tools for accelerated modeling, reporting, and analysis.

Intelligent AMC can predict which shows will be successful, how it should schedule them, what promos it should create, and to whom it should market them—helping to win new audience share in an increasingly competitive market.

segmentation and look-alike modeling helped the company target new and existing viewers so effec- tively that AMC video-on-demand transactions were higher than would be expected otherwise.

This newfound ability to reach out to new viewers based on their individual needs and prefer- ences is not just valuable for AMC; it also has huge potential value for the company’s advertising part- ners. AMC is currently working on providing access to its rich data sets and analytics tools as a service for advertisers, helping them fine-tune their cam- paigns to appeal to ever-larger audiences across both linear and digital channels.

Tsivin concludes, “Now that we can really har- ness the value of big data, we can build a much more attractive proposition for both consumers and advertisers—creating even better content, market- ing it more effectively, and helping it reach a wider audience by taking full advantage of our multichan- nel capabilities.”

Questions for Case 7.2

1. What are the common challenges that broadcast- ing companies are facing today? How can analyt- ics help to alleviate these challenges?

2. How did AMC leverage analytics to enhance its business performance?

3. What were the types of text analytics and text minisolutions developed by AMC networks? Can you think of other potential uses of text mining applications in the broadcasting industry?

Sources: IBM Customer Case Study. “Using Analytics to Capture New Viewers, Predict Ratings and Add Value for Advertisers in a Multichannel World.” http://www-03.ibm.com/software/ businesscasestudies/us/en/corp?synkey=A023603A76220M60 (accessed July 2016); www.ibm.com; www.amcnetworks.com.

With deeper insight into viewership, AMC’s direct marketing campaigns are also much more suc- cessful than before. In one recent example, intelligent

402 Part II • Predictive Analytics/Machine Learning

NLP has successfully been applied to a variety of domains for a wide range of tasks via computer programs to automatically process natural human language that previously could be done only by humans. Following are among the most popular of these tasks:

• Question answering. The task of automatically answering a question posed in natural language; that is, producing a human language answer when given a human language question. To find the answer to a question, the computer program can use either a prestructured database or a collection of natural language documents (a text corpus such as the World Wide Web).

• Automatic summarization. The creation of a shortened version of a textual document by a computer program that contains the most important points of the original document.

• Natural language generation. The conversion of information from computer databases into readable human language.

• Natural language understanding. The conversion of samples of human lan- guage into more formal representations that are easier for computer programs to manipulate.

• Machine translation. The automatic translation of one human language to another.

• Foreign language reading. A computer program that assists a nonnative lan- guage speaker in reading a foreign language with correct pronunciation and accents on different parts of the words.

• Foreign language writing. A computer program that assists a nonnative lan- guage user in writing in a foreign language.

• Speech recognition. Conversion of spoken words to machine-readable input. Given a sound clip of a person speaking, the system produces a text dictation.

• Text to speech. Also called speech synthesis, a computer program that automati- cally converts normal language text into human speech.

• Text proofing. A computer program that reads a proof copy of a text to detect and correct any errors.

• Optical character recognition. The automatic translation of images of hand- written, typewritten, or printed text (usually captured by a scanner) into machine- editable textual documents.

The success and popularity of text mining depends greatly on advancements in NLP in both generating and understanding human languages. NLP enables the extraction of features from unstructured text so that a wide variety of data mining techniques can be used to extract knowledge (novel and useful patterns and relationships) from it. In that sense, simply put, text mining is a combination of NLP and data mining.

u SECTION 7.3 REVIEW QUESTIONS

1. What is NLP? 2. How does NLP relate to text mining? 3. What are some of the benefits and challenges of NLP? 4. What are the most common tasks addressed by NLP?

7.4 TEXT MINING APPLICATIONS

As the amount of unstructured data collected by organizations increases, so do the value proposition and popularity of text mining tools. Many organizations are now realizing the importance of extracting knowledge from their document-based data repositories through

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 403

the use of text mining tools. The following is only a small subset of the exemplary ap- plication categories of text mining.

Marketing Applications

Text mining can be used to increase cross-selling and up-selling by analyzing the un- structured data generated by call centers. Text generated notes from call center as well as transcriptions of voice conversations with customers can be analyzed by text min- ing algorithms to extract novel, actionable information about customers’ perceptions toward a company’s products and services. In addition, blogs, user reviews of products at independent Web sites, and discussion board postings are gold mines of customer sentiments. This rich collection of information, once properly analyzed, can be used to increase satisfaction and the overall lifetime value of the customer (Coussement & Van den Poel, 2008).

Text mining has become invaluable for CRM. Companies can use text mining to analyze rich sets of unstructured text data combined with the relevant structured data extracted from organizational databases to predict customer perceptions and subsequent purchasing behavior. Coussement and Van den Poel (2009) successfully applied text min- ing to significantly improve a model’s ability to predict customer churn (i.e., customer at- trition) so that those customers identified as most likely to leave a company are accurately identified for retention tactics.

Ghani et al. (2006) used text mining to develop a system capable of inferring im- plicit and explicit attributes of products to enhance retailers’ ability to analyze product databases. Treating products as sets of attribute–value pairs rather than as atomic entities can potentially boost the effectiveness of many business applications, including demand forecasting, assortment optimization, product recommendations, assortment comparison across retailers and manufacturers, and product supplier selection. The proposed system allows a business to represent its products in terms of attributes and attribute values with- out much manual effort. The system learns these attributes by applying supervised and semi-supervised learning techniques to product descriptions found on retailers’ Web sites.

Security Applications

One of the largest and most prominent text mining applications in the security domain is probably the highly classified ECHELON surveillance system. As rumor has it, ECHELON is assumed to be capable of identifying the content of telephone calls, faxes, e-mails, and other types of data, intercepting information sent via satellites, public-switched telephone networks, and microwave links.

In 2007, the European Union Agency for Law Enforcement Cooperation (EUROPOL) developed an integrated system capable of accessing, storing, and analyzing vast amounts of structured and unstructured data sources to track transnational organized crime. Called the Overall Analysis System for Intelligence Support (OASIS), it aims to integrate the most advanced data and text mining technologies available in today’s market. The system has enabled EUROPOL to make significant progress in supporting its law enforcement objec- tives at the international level (EUROPOL, 2007).

The U.S. Federal Bureau of Investigation (FBI) and the Central Intelligence Agency (CIA), under the direction of the Department for Homeland Security, are jointly develop- ing a supercomputer data and text mining system. The system is expected to create a gigantic data warehouse along with a variety of data and text mining modules to meet the knowledge-discovery needs of federal, state, and local law enforcement agencies. Prior to this project, the FBI and CIA each had its own separate database with little or no interconnection.

404 Part II • Predictive Analytics/Machine Learning

Another security-related application of text mining is in the area of deception detection. Applying text mining to a large set of real-world criminal (person-of-interest) statements, Fuller, Biros, and Delen (2008) developed prediction models to differentiate deceptive statements from truthful ones. Using a rich set of cues extracted from textual state- ments, the model predicted the holdout samples with 70 percent accuracy, which is believed to be a significant success considering that the cues are extracted only from textual state- ments (no verbal or visual cues are present). Furthermore, compared to other deception- detection techniques, such as polygraphs, this method is nonintrusive and widely applicable to not only textual data but also (potentially) transcriptions of voice recordings. A more detailed description of text-based deception detection is provided in Application Case 7.3.

Biomedical Applications

Text mining holds great potential for the medical field in general and biomedicine in particular for several reasons. First, published literature and publication outlets (especially with the advent of the open source journals) in the field are expanding at an exponential rate. Second, compared to most other fields, medical literature is more standardized and orderly, making it a more “minable” information source. Finally, the terminology used in

Driven by advancements in Web-based informa- tion technologies and increasing globalization, computer-mediated communication continues to fil- ter into everyday life, bringing with it new venues for deception. The volume of text-based chat, instant messaging, text messaging, and text generated by online communities of practice is increasing rapidly. Even the use of e-mail continues to increase. With the massive growth of text-based communication, the potential for people to deceive others through computer-mediated communication has also grown, and such deception can have disastrous results.

Unfortunately, in general, humans tend to perform poorly at deception-detection tasks. This phenomenon is exacerbated in text-based commu- nications. A large part of the research on deception detection (also known as credibility assessment) has involved face-to-face meetings and interviews. Yet with the growth of text-based communication, text- based deception-detection techniques are essential.

Techniques for successfully detecting deception—that is, lies—have wide applicability. Law enforcement can use decision support tools and tech- niques to investigate crimes, conduct security screening in airports, and monitor communications of suspected terrorists. Human resources professionals might use deception-detection tools to screen applicants. These tools and techniques also have the potential to screen

e-mails to uncover fraud or other wrongdoings com- mitted by corporate officers. Although some people believe that they can readily identify those who are not being truthful, a summary of deception research showed that, on average, people are only 54 percent accurate in making veracity determinations (Bond & DePaulo, 2006). This figure may actually be worse when humans try to detect deception in text.

Using a combination of text mining and data mining techniques, Fuller et al. (2008) analyzed person-of-interest statements completed by peo- ple involved in crimes on military bases. In these statements, suspects and witnesses are required to write their recollection of the event in their own words. Military law enforcement personnel searched archival data for statements that they could conclu- sively identify as being truthful or deceptive. These decisions were made on the basis of corroborating evidence and case resolution. Once labeled as truth- ful or deceptive, the law enforcement personnel removed identifying information and gave the state- ments to the research team. In total, 371 usable state- ments were received for analysis. The text-based deception-detection method used by Fuller et al. was based on a process known as message feature mining, which relies on elements of data and text mining techniques. A simplified depiction of the process is provided in Figure 7.3.

Application Case 7.3 Mining for Lies

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 405

First, the researchers prepared the data for pro- cessing. The original handwritten statements had to be transcribed into a word processing file. Second, features (i.e., cues) were identified. The research- ers identified 31 features representing categories or types of language that are relatively independent of

the text content and that can be readily analyzed by automated means. For example, first-person pronouns such as I or me can be identified with- out analysis of the surrounding text. Table 7.1 lists the categories and examples of features used in this study.

Statements Transcribed for

Processing

Text Processing Software Identified Cues in Statements

Statements Labeled as Truthful or Deceptive by Law Enforcement

Text Processing Software Generated

Quantified Cues

Classification Models Trained and Tested on Quantified Cues

Cues Extracted & Selected

FIGURE 7.3 Text-Based Deception-Detection Process. Source: Fuller, C. M., D. Biros, & D. Delen. (2008, January). Exploration of Feature Selection and Advanced Classification Models for High-Stakes Deception Detection. Proceedings of

the Forty-First Annual Hawaii International Conference on System Sciences (HICSS), Big Island, HI: IEEE Press, pp. 80–99.

TABLE 7.1 Categories and Examples of Linguistic Features Used in Deception Detection

Number Construct (Category) Example Cues

1 Quantity Verb count, noun phrase count, etc.

2 Complexity Average number of clauses, average sentence length, etc.

3 Uncertainty Modifiers, modal verbs, etc.

4 Nonimmediacy Passive voice, objectification, etc.

5 Expressivity Emotiveness

6 Diversity Lexical diversity, redundancy, etc.

7 Informality Typographical error ratio

8 Specificity Spatiotemporal information, perceptual information, etc.

9 Affect Positive affect, negative affect, etc.

(Continued )

406 Part II • Predictive Analytics/Machine Learning

this literature is relatively constant, having a fairly standardized ontology. What follows are a few exemplary studies that successfully used text mining techniques in extracting novel patterns from biomedical literature.

Experimental techniques such as DNA microarray analysis, serial analysis of gene ex- pression (SAGE), and mass spectrometry proteomics, among others, are generating large amounts of data related to genes and proteins. As in any other experimental approach, it is necessary to analyze this vast amount of data in the context of previously known information about the biological entities under study. The literature is a particularly valu- able source of information for experiment validation and interpretation. Therefore, the development of automated text mining tools to assist in such interpretation is one of the main challenges in current bioinformatics research.

Knowing the location of a protein within a cell can help to elucidate its role in biological processes and to determine its potential as a drug target. Numerous location- prediction systems are described in the literature; some focus on specific organisms, whereas others attempt to analyze a wide range of organisms. Shatkay et al. (2007) pro- posed a comprehensive system that uses several types of sequence- and text-based fea- tures to predict the location of proteins. The main novelty of their system lies in the way in which it selects its text sources and features and integrates them with sequence-based features. They tested the system on previously used and new data sets devised specifi- cally to test its predictive power. The results showed that their system consistently beat previously reported results.

Chun et al. (2006) described a system that extracts disease–gene relationships from literature accessed via MEDLINE. They constructed a dictionary for disease and gene names from six public databases and extracted relation candidates by dictionary matching. Because dictionary matching produces a large number of false positives, they developed a method of machine-learning–based system, named entity recognition (NER), to filter out false recognition of disease/gene names. They found that the success of relation extraction is heavily dependent on the performance of NER filtering and that the filtering improved the precision of relation extraction by 26.7 percent at the cost of a small reduction in recall.

Figure 7.4 shows a simplified depiction of a multilevel text analysis process for dis- covering gene–protein relationships (or protein–protein interactions) in the biomedical

The features were extracted from the textual statements and input into a flat file for further pro- cessing. Using several feature-selection methods along with 10-fold cross-validation, the researchers compared the prediction accuracy of three popu- lar data mining methods. Their results indicated that neural network models performed the best, with 73.46 percent prediction accuracy on test data samples; decision trees performed second best, with 71.60 percent accuracy; and logistic regression was last, with 65.28 percent accuracy.

The results indicate that automated text-based deception detection has the potential to aid those who must try to detect lies in text and can be suc- cessfully applied to real-world data. The accuracy of these techniques exceeded the accuracy of most

other deception-detection techniques, even though it was limited to textual cues.

Questions for Case 7.3

1. Why is it difficult to detect deception?

2. How can text/data mining be used to detect deception in text?

3. What do you think are the main challenges for such an automated system?

Sources: Fuller, C. M., D. Biros, & D. Delen. (2008, January). “Exploration of Feature Selection and Advanced Classification Models for High-Stakes Deception Detection.” Proceedings of the Forty-First Annual Hawaii International Conference on System Sciences (HICSS), Big Island, HI: IEEE Press, pp. 80–99; Bond, C. F., & B. M. DePaulo. (2006). “Accuracy of Deception Judgments.” Personality and Social Psychology Reports, 10(3), pp. 214–234.

Application Case 7.3 (Continued)

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 407

literature (Nakov, Schwartz, Wolf, and Hearst, 2005). As can be seen in this simplified ex- ample that uses a simple sentence from biomedical text, first (at the bottom three levels) the text is tokenized using part-of-speech tagging and shallow parsing. The tokenized terms (words) are then matched (and interpreted) against the hierarchical representa- tion of the domain ontology to derive the gene–protein relationship. Application of this method (and/or some variation of it) to the biomedical literature offers great potential to decode the complexities in the Human Genome Project.

Academic Applications

The issue of text mining is of great importance to publishers who hold large databases of information requiring indexing for better retrieval. This is particularly true in scien- tific disciplines in which highly specific information is often contained within written text. Initiatives have been launched, such as Nature’s proposal for an Open Text Mining Interface and the National Institutes of Health’s common Journal Publishing Document Type Definition, which would provide semantic cues to machines to answer specific que- ries contained within text without removing publisher barriers to public access.

Academic institutions have also launched text mining initiatives. For example, the National Centre for Text Mining, a collaborative effort between the Universities of Manchester and Liverpool, provides customized tools, research facilities, and advice on text mining to the academic community. With an initial focus on text mining in the biological and bio- medical sciences, research has since expanded into the social sciences. In the United States, the School of Information at the University of California–Berkeley is developing a program called BioText to assist bioscience researchers in text mining and analysis.

G en

e/ P

ro te

in 596 12043 24224 281020 42722 397276

D007962

D016923

D001773

D019254 D044465 D001769 D002477 D003643 D016158

185 8 51112 9 23017 27 5874 2791 8952 1623 5632 17 8252 8 2523

NN IN NN IN VBZ IN JJ JJ NN NN NN CC NN IN NN

NP PP NP NP PP NP NP PP NP

O nt

ol og

y W

or d

P O

S S

ha llo

w P

ar se

... expression of Bcl-2 is correlated with insufficient white blood cell death and activation of p53.

FIGURE 7.4 Multilevel Analysis of Text for Gene/Protein Interaction Identification. Source: Used with permission of Nakov, P., Schwartz, A., Wolf, B., & Hearst, M. A. (2005). Supporting annotation layers for natural language processing. Proceedings of the

Association for Computational Linguistics (ACL), Interactive Poster and Demonstration Sessions, Ann Arbor, MI. Association for

Computational Linguistics, 65–68.

408 Part II • Predictive Analytics/Machine Learning

As described in this section, text mining has a wide variety of applications in a number of different disciplines. See Application Case 7.4 for an example of how a leading computing product manufacturer uses text mining to better understand its current and potential customers’ needs and wants related to product quality and product design.

Advanced analytics techniques that use both structured and unstructured data have been successfully used in many application domains. Application Case 7.4 provides an interesting example where a wide range of analytics capabilities are used to successfully manage the Orlando Magic organization both on and off the courts of NBA.

From ticket sales to starting lineups, the Orlando Magic have come a long way since their inaugural season in 1989. There weren’t many wins in those early years, but the franchise has weathered the ups and downs to compete at the highest levels of the NBA.

Professional sports teams in smaller markets often struggle to build a big enough revenue base to compete against their larger market rivals. By using SAS® Analytics and SAS® Data Management, the Orlando Magic are among the top revenue earners in the NBA, despite being in the 20th-largest market.

The Magic accomplish this feat by studying the resale ticket market to price tickets better, to predict season ticket holders at risk of defection (and lure them back), and to analyze concession and product merchandise sales to make sure the organization has what the fans want every time they enter the arena. The club has even used SAS to help coaches put together the best lineup.

“Our biggest challenge is to customize the fan experience, and SAS helps us manage all that in a robust way,” says Alex Martins, CEO of the Orlando Magic. Having been with the Magic since the begin- ning (working his way up from PR Director to President to CEO), Martins has seen it all and knows the value that analytics adds. Under Martins’ leader- ship, the season-ticket base has grown as large as 14,200, and the corporate sales department has seen tremendous growth.

The Challenge: Filling Every Seat

But like all professional sports teams, the Magic are constantly looking for new strategies that will keep the seats filled at each of the 41 yearly home games. “Generating new revenue streams in this day of escalating player salaries and escalating expenses is important,” says Anthony Perez, vice president of Business Strategy. But with the advent of a robust

online secondary market for tickets, reaching the industry benchmark of 90 percent renewal of season tickets has become more difficult.

“In the first year, we saw ticket revenue increase around 50 percent. Over the last three years—for that period, we’ve seen it grow maybe 75 percent. It’s had a huge impact” said Anthony Perez, vice president of Business Strategy, Orlando Magic.

Perez’s group takes a holistic approach by combining data from all revenue streams (conces- sion, merchandise, and ticket sales) with outside data (secondary ticket market) to develop models that benefit the whole enterprise. “We’re like an in- house consulting group,” explains Perez.

In the case of season ticket holders, the team uses historical purchasing data and renewal patterns to build decision tree models that place subscrib- ers into three categories: most likely to renew, least likely to renew, and fence sitters. The fence sitters then get the customer service department’s attention come renewal time.

“SAS has helped us grow our business. It is probably one of the greatest investments that we’ve made as an organization over the last half-dozen years because we can point to top-line revenue growth that SAS has helped us create through the specific messaging that we’re able to direct to each one of our client groups.”

How Do They Predict Season Ticket Renewals?

When analytics showed the team that 80 percent of revenue was from season ticket holders, it decided to take a proactive approach to renewals and at- risk accounts. The Magic don’t have a crystal ball, but they do have SAS® Enterprise Miner™, which allowed them to better understand their data and

Application Case 7.4 The Magic Behind the Magic: Instant Access to Information Helps the Orlando Magic Up their Game and the Fan’s Experience

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 409

develop analytic models that combine three pillars for predicting season ticket holder renewals:

• Tenure (how long had the customer been a ticket holder?).

• Ticket use (did the customer actually attend the games?).

• Secondary market activity (were the unused tickets successfully sold on secondary sites?).

The data mining tools allowed the team to accomplish more accurate scoring that led to a difference—and marked improvement—in the way it approached customer retention and marketing.

Ease of Use Helps Spread Analytics Message

Perez likes how easy it is to use SAS—it was a factor in opting to do the work in-house rather than out- sourcing it. Perez’s team has set up recurring pro- cesses and automated them. Data manipulation is minimal, “allowing us more time to interpret rather than just manually crunching the numbers.” Business users throughout the organization, including execu- tives, have instant access to information through SAS® Visual Analytics. “It’s not just that we’re using the tools daily; we are using them throughout the day to make decisions,” Perez says.

Being Data-Driven

“We adopted an analytics approach years ago, and we're seeing it transform our entire organization,” says Martins. “Analytics helps us understand custom- ers better, helps in business planning (ticket pricing, etc.), and provides game-to-game and year-to-year data on demand by game and even by seat.”

“And analytics has helped transform the game. GMs and analytics teams look at every aspect of the game, including movements of players on the court, to transform data to predict defense against certain teams. We can now ask ourselves, ‘What are the most efficient lineups in a game? Which team can produce more points vs. another lineup? Which team is better defensively than another?’”

“We used to produce a series of reports man- ually, but now we can do it with five clicks of a mouse (instead of five hours overnight in anticipa- tion of tomorrow’s game). We can have dozens of reports available to staff in minutes. Analytics has made us smarter,” says Martins.

What’s Next?

“Getting real-time data is the next step for us in our analytical growth process,” says Martins. “On a game day, getting real-time data to track what tickets are available and how to maximize yield of those tickets is critical. Additionally, you're going to see major techno- logical changes and acceptance of the technology on the bench to see how the games are played moving forward. Maybe as soon as next season you’ll see our assistant coaches with iPad® tablets getting real-time data, learning what the opponent is doing and what plays are working. It’ll be necessary in the future.

“We’re setting ourselves up to be successful moving forward. And in the very near future, we’ll be in a position again to compete for a conference championship and an NBA championship,” says Martins. “All of the moves made this year and the ones to come in the future will be done in order to build success on [and off] the court.’’

Questions for Case 7.4

1. According to the application case, what were the main challenges the Orlando Magic was facing?

2. How did analytics help the Orlando Magic to overcome some of its most significant challenges on and off the court?

3. Can you think of other uses of analytics in sports and especially in the case of the Orlando Magic? You can search the Web to find some answers to this question.

Source: SAS Customer Story, “The magic behind the Magic: Instant access to information helps the Orlando Magic up their game and the fan’s experience” at https://www.sas.com/en_us/customers/ orlando-magic.html and https://www.nba.com/magic/news/ denton-25-years-magic-history (accessed November 2018).

u SECTION 7.4 REVIEW QUESTIONS

1. List and briefly discuss some of the text mining applications in marketing. 2. How can text mining be used in security and counterterrorism? 3. What are some promising text mining applications in biomedicine?

410 Part II • Predictive Analytics/Machine Learning

7.5 TEXT MINING PROCESS

To be successful, text mining studies should follow a sound methodology based on best practices. A standardized process model is needed similar to Cross-Industry Standard Process for Data Mining (CRISP-DM), which is the industry standard for data mining projects (see Chapter 4). Even though most parts of CRISP-DM are also applicable to text mining projects, a specific process model for text mining would include much more elab- orate data preprocessing activities. Figure 7.5 depicts a high-level context diagram of a typical text mining process (Delen & Crossland, 2008). This context diagram presents the scope of the process, emphasizing its interfaces with the larger environment. In essence, it draws boundaries around the specific process to explicitly identify what is included in (and excluded from) the text mining process.

As the context diagram indicates, the input (inward connection to the left edge of the box) into the text-based knowledge-discovery process is the unstructured as well as struc- tured data collected, stored, and made available to the process. The output (outward exten- sion from the right edge of the box) of the process is the context-specific knowledge that can be used for decision making. The controls, also called the constraints (inward connec- tion to the top edge of the box), of the process include software and hardware limitations, privacy issues, and difficulties related to processing the text that is presented in the form of natural language. The mechanisms (inward connection to the bottom edge of the box) of the process include proper techniques, software tools, and domain expertise. The primary purpose of text mining (within the context of knowledge discovery) is to process unstruc- tured (textual) data (along with structured data if relevant to the problem being addressed and available) to extract meaningful and actionable patterns for better decision making.

At a very high level, the text mining process can be broken down into three consec- utive tasks, each of which has specific inputs to generate certain outputs (see Figure 7.6). If, for some reason, the output of a task is not what is expected, a backward redirection to the previous task execution is necessary.

Task 1: Establish the Corpus

The main purpose of the first task activity is to collect all the documents related to the context (domain of interest) being studied. This collection may include textual docu- ments, XML files, e-mails, Web pages, and short notes. In addition to the readily available

Extract knowledge from available data sources

A0

Unstructured data (text)

Structured data (databases)

Context-specific knowledge

Software/hardware limitations

Privacy issues

Linguistic limitations

Tools and techniques Domain expertise

FIGURE 7.5 Context Diagram for the Text Mining Process.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 411

textual data, voice recordings may also be transcribed using speech-recognition algo- rithms and made a part of the text collection.

Once collected, the text documents are transformed and organized in a manner such that they are all in the same representational form (e.g., ASCII text files) for com- puter processing. The organization of the documents can be as simple as a collection of digitized text excerpts stored in a file folder or a list of links to a collection of Web pages in a specific domain. Many commercially available text mining software tools could ac- cept these as input and convert them into a flat file for processing. Alternatively, the flat file can be prepared outside the text mining software and then presented as the input to the text mining application.

Task 2: Create the Term–Document Matrix

In this task, the digitized and organized documents (the corpus) are used to create the term–document matrix (TDM). In the TDM, the rows represent the documents and the columns represent the terms. The relationships between the terms and documents are characterized by indices (i.e., a relational measure that can be as simple as the num- ber of occurrences of the term in respective documents). Figure 7.7 is a typical example of a TDM.

Establish the Corpus: Collect and organize the domain-specific unstructured data

Create the Term- Document Matrix: Introduce structure

to the corpus

Extract Knowledge: Discover novel

patterns from the T-D matrix

The inputs to the process include a variety of relevant unstructured (and semi- structured) data sources such as text, XML, HTML

The output of Task 1 is a collection of documents in some digitized format for computer processing

The output of Task 2 is a flat file called term-document matrix where the cells are populated with the term frequencies

The output of Task 3 is a number of problem-specific classification, association, clustering models, and visualizations

Task 1 Task 2 Task 3

FeedbackFeedback

Knowledge 1 2

3 4 5

DataText

FIGURE 7.6 The Three-Step/Task Text Mining Process.

Inv est

me nt

Ris k

Pro jec

t M ana

gem ent

So ftw

are En

gin eer

ing

De vel

opm ent

1

SA P

...

Document 1

Document 2

Document 3

Document 4

Document 5

Document 6

...

Documents

Terms

1

1

1

2

1

1

1

3

1

FIGURE 7.7 Simple Term–Document Matrix.

412 Part II • Predictive Analytics/Machine Learning

The goal is to convert the list of organized documents (the corpus) into a TDM where the cells are filled with the most appropriate indices. The assumption is that the essence of a document can be represented with a list and frequency of the terms used in that document. However, are all terms important when characterizing documents? Obviously, the answer is “no.” Some terms, such as articles, auxiliary verbs, and terms used in almost all the documents in the corpus, have no differentiating power and, there- fore, should be excluded from the indexing process. This list of terms, commonly called stop terms or stop words, is specific to the domain of study and should be identified by the domain experts. On the other hand, one might choose a set of predetermined terms under which the documents are to be indexed (this list of terms is conveniently called in- clude terms or dictionary). In addition, synonyms (pairs of terms that are to be treated the same) and specific phrases (e.g., “Eiffel Tower”) can also be provided so that the index entries are more accurate.

Another filtration that should take place to accurately create the indices is stem- ming, which refers to the reduction of words to their roots so that, for example, different grammatical forms or declinations of a verb are identified and indexed as the same word. For example, stemming will ensure that modeling and modeled will be recognized as the word model.

The first generation of the TDM includes all the unique terms identified in the corpus (as its columns), excluding the ones in the stop term list; all the documents (as its rows); and the occurrence count of each term for each document (as its cell values). If, as is commonly the case, the corpus includes a rather large number of documents, then there is a very good chance that the TDM will have a very large number of terms. Processing such a large matrix might be time consuming and, more important, might lead to extraction of inaccurate patterns. At this point, one has to decide the following: (1) What is the best representation of the indices? and (2) How can we reduce the dimen- sionality of this matrix to a manageable size?

REPRESENTING THE INDICES Once the input documents have been indexed and the ini- tial word frequencies (by document) computed, a number of additional transformations can be performed to summarize and aggregate the extracted information. The raw term frequencies generally reflect on how salient or important a word is in each document. Specifically, words that occur with greater frequency in a document are better descriptors of the contents of that document. However, it is not reasonable to assume that the word counts themselves are proportional to their importance as descriptors of the documents. For example, if a word occurs one time in document A but three times in document B, it is not necessarily reasonable to conclude that this word is three times as important a descriptor of document B as compared to document A. To have a more consistent TDM for further analysis, these raw indices need to be normalized. As opposed to showing the actual frequency counts, the numerical representation between terms and documents can be normalized using a number of alternative methods, such as log frequencies, binary frequencies, and inverse document frequencies.

REDUCING THE DIMENSIONALITY OF THE MATRIX Because the TDM is often very large and rather sparse (most of the cells filled with zeros), another important question is, “How do we reduce the dimensionality of this matrix to a manageable size?” Several op- tions are available for managing the matrix size:

• A domain expert goes through the list of terms and eliminates those that do not make much sense for the context of the study (this is a manual, labor-intensive process).

• Eliminate terms with very few occurrences in very few documents. • Transform the matrix using SVD.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 413

Singular value decomposition (SVD), which is closely related to principal com- ponents analysis, reduces the overall dimensionality of the input matrix (number of input documents by number of extracted terms) to a lower-dimensional space when each con- secutive dimension represents the largest degree of variability (between words and docu- ments) possible (Manning and Schutze, 1999). Ideally, the analyst might identify the two or three most salient dimensions that account for most of the variability (differences) between the words and documents, thus identifying the latent semantic space that orga- nizes the words and documents in the analysis. Once such dimensions are identified, the underlying “meaning” of what is contained (discussed or described) in the documents has been extracted.

Task 3: Extract the Knowledge

Using the well-structured TDM and potentially augmented with other structured data elements, novel patterns are extracted in the context of the specific problem being ad- dressed. The main categories of knowledge extraction methods are classification, cluster- ing, association, and trend analysis. A short description of these methods follows.

CLASSIFICATION Arguably the most common knowledge-discovery topic in analyzing complex data sources is the classification (or categorization) of certain objects. The task is to classify a given data instance into a predetermined set of categories (or classes). As it applies to the domain of text mining, the task is known as text categorization for a given set of categories (subjects, topics, or concepts) and a collection of text documents whose goal is to find the correct topic (subject or concept) for each document using models developed with a training data set that includes both the documents and actual document categories. Today, automated text classification is applied in a variety of contexts, including automatic or semi-automatic (interactive) indexing of text, spam filtering, Web page categorization under hierarchical catalogs, automatic generation of metadata, and detection of genre.

The two main approaches to text classification are knowledge engineering and machine learning (Feldman and Sanger, 2007). With the knowledge-engineering ap- proach, an expert’s knowledge about the categories is encoded into the system either declaratively or in the form of procedural classification rules. With the machine-learning approach, a general inductive process builds a classifier by learning from a set of reclas- sified examples. As the number of documents increases at an exponential rate and as knowledge experts become harder to come by, the popularity trend between the two is shifting toward the machine-learning approach.

CLUSTERING Clustering is an unsupervised process whereby objects are classified into “natural” groups called clusters. Compared to categorization that uses a collection of preclassified training examples to develop a model based on the descriptive features of the classes to classify a new unlabeled example, in clustering the problem is to group an unlabeled collection of objects (e.g., documents, customer comments, Web pages) into meaningful clusters without any prior knowledge.

Clustering is useful in a wide range of applications from document retrieval to en- abling better Web content searches. In fact, one of the prominent applications of clustering is the analysis and navigation of very large text collections, such as Web pages. The basic underlying assumption is that relevant documents tend to be more similar to each other than to irrelevant ones. If this assumption holds, the clustering of documents based on the similarity of their content improves search effectiveness (Feldman and Sanger, 2007):

• Improved search recall. Because it is based on overall similarity as opposed to the presence of a single term, clustering can improve the recall of a query-based search in such a way that when a query matches a document, its whole cluster is returned.

414 Part II • Predictive Analytics/Machine Learning

• Improved search precision. Clustering can also improve search precision. As the number of documents in a collection grows, it becomes difficult to browse through the list of matched documents. Clustering can help by grouping the documents into a number of much smaller groups of related documents, ordering them by relevance, and returning only the documents from the most relevant group (or groups).

The two most popular clustering methods are scatter/gather clustering and query-specific clustering:

• Scatter/gather. This document browsing method uses clustering to enhance the efficiency of human browsing of documents when a specific search query cannot be formulated. In a sense, the method dynamically generates a table of contents for the collection and adapts and modifies it in response to the user selection.

• Query-specific clustering. This method employs a hierarchical clustering approach where the most relevant documents to the posed query appear in small tight clusters that are nested in larger clusters containing less-similar documents, creating a spectrum of relevance levels among the documents. This method per- forms consistently well for document collections of realistically large sizes.

ASSOCIATION Associations, or association rule learning in data mining, is a popular and well-researched technique for discovering interesting relationships among variables in large databases. The main idea in generating association rules (or solving market- basket problems) is to identify the frequent sets that go together.

In text mining, associations specifically refer to the direct relationships between con- cepts (terms) or sets of concepts. The concept set association rule A + C relating two frequent concept sets A and C can be quantified by the two basic measures of support and confidence. In this case, confidence is the percentage of documents that include all concepts in C within the same subset of those documents that include all concepts in A. Support is the percentage (or number) of documents that include all the concepts in A and C. For instance, in a document collection the concept “Software Implementation Failure” could appear most often in association with “Enterprise Resource Planning” and “Customer Relationship Management” with significant support (4%) and confidence (55%), meaning that 4 percent of the documents had all three concepts represented in the same document, and of the documents that included “Software Implementation Failure,” 55 percent of them also included “Enterprise Resource Planning” and “Customer Relationship Management.”

Text mining with association rules was used to analyze published literature (news and academic articles posted on the Web) to chart the outbreak and progress of the bird flu (Mahgoub et al., 2008). The idea was to automatically identify the association among the geographic areas, spreading across species, and countermeasures (treatments).

TREND ANALYSIS Recent methods of trend analysis in text mining have been based on the notion that the various types of concept distributions are functions of document collections; that is, different collections lead to different concept distributions for the same set of concepts. It is, therefore, possible to compare two distributions that are otherwise identical except that they are from different subcollections. One notable direction of this type of analysis is hav- ing two collections from the same source (such as from the same set of academic journals) but from different points in time. Delen and Crossland (2008) applied trend analysis to a large number of academic articles (published in the three highest-rated academic journals) to identify the evolution of key concepts in the field of information systems.

As described in this section, a number of methods are available for text mining. Application Case 7.5 describes the use of a number of different techniques in analyzing a large set of literature.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 415

Researchers conducting searches and reviews of rel- evant literature face an increasingly complex and voluminous task. In extending the body of relevant knowledge, it has always been important to work hard to gather, organize, analyze, and assimilate existing information from the literature, particularly from one’s home discipline. With the increasing abundance of potentially significant research being reported in related fields, and even in what are tra- ditionally deemed to be nonrelated fields of study, the researcher’s task is ever more daunting if a thor- ough job is desired.

In new streams of research, the researcher’s task can be even more tedious and complex. Trying to ferret out relevant work that others have reported can be difficult, at best, and perhaps even nearly impossible if traditional, largely manual reviews of published literature are required. Even with a legion of dedicated graduate students or helpful col- leagues, trying to cover all potentially relevant pub- lished work is problematic.

Many scholarly conferences take place every year. In addition to extending the body of knowl- edge of the current focus of a conference, organiz- ers often desire to offer additional minitracks and workshops. In many cases, these additional events are intended to introduce attendees to significant streams of research in related fields of study and to try to identify the “next big thing” in terms of research interests and focus. Identifying reasonable candidate topics for such minitracks and workshops is often subjective rather than derived objectively from the existing and emerging research.

In a recent study, Delen and Crossland (2008) proposed a method to greatly assist and enhance the efforts of the researchers by enabling a semi- automated analysis of large volumes of published literature through the application of text mining. Using standard digital libraries and online publica- tion search engines, the authors downloaded and collected all the available articles for the three major journals in the field of management information sys- tems: MIS Quarterly (MISQ), Information Systems Research (ISR), and the Journal of Management Information Systems (JMIS). To maintain the same

time interval for all three journals (for potential comparative longitudinal studies), the journal with the most recent starting date for its digital publica- tion availability was used as the start time for this study (i.e., JMIS articles have been digitally available since 1994). For each article, Delen and Crossland extracted the title, abstract, author list, published keywords, volume, issue number, and year of pub- lication. They then loaded all the article data into a simple database file. Also included in the com- bined data set was a field that designated the journal type of each article for likely discriminatory analysis. Editorial notes, research notes, and executive over- views were omitted from the collection. Table  7.2 shows how the data were presented in a tabular format.

In the analysis phase, the researchers chose to use only the abstract of an article as the source of information extraction. They chose not to include the keywords listed with the publica- tions for two main reasons: (1) under normal cir- cumstances, the abstract would already include the listed keywords, and therefore inclusion of the listed keywords for the analysis would mean repeating the same information and potentially giving them unmerited weight and (2) the listed keywords could be terms that authors would like their article to be associated with (as opposed to what is really contained in the article), therefore, potentially introducing unquantifiable bias to the analysis of the content.

The first exploratory study was to look at the longitudinal perspective of the three journals (i.e., evolution of research topics over time). To conduct a longitudinal study, Delen and Crossland divided the 12-year period (from 1994 to 2005) into four 3-year periods for each of the three journals. This framework led to 12 text mining experiments with 12 mutually exclusive data sets. At this point, for each of the 12 data sets, the researchers used text mining to extract the most descriptive terms from these collections of articles represented by their abstracts. The results were tabulated and examined for time-varying changes in the terms published in these three journals.

Application Case 7.5 Research Literature Survey with Text Mining

(Continued )

416 Part II • Predictive Analytics/Machine Learning

As a second exploration, using the complete data set (including all three journals and all four periods), Delen and Crossland conducted a cluster- ing analysis. Clustering is arguably the most com- monly used text mining technique. Clustering was used in this study to identify the natural groupings of the articles (by putting them into separate clus- ters) and then to list the most descriptive terms that characterized those clusters. They used SVD to reduce the dimensionality of the term-by-document matrix and then an expectation-maximization algo- rithm to create the clusters. They conducted sev- eral experiments to identify the optimal number of clusters, which turned out to be nine. After the construction of the nine clusters, they analyzed the content of those clusters from two perspectives: (1) representation of the journal type (see Figure 7.8a) and (2) representation of time (Figure 7.8b). The idea was to explore the potential differences

and/or commonalities among the three journals and potential changes in the emphasis on those clusters; that is, to answer questions such as “Are there clus- ters that represent different research themes specific to a single journal?” and “Is there a time-varying characterization of those clusters?” The researchers discovered and discussed several interesting pat- terns using tabular and graphical representation of their findings (for further information, see Delen and Crossland, 2008).

Questions for Case 7.5

1. How can text mining be used to ease the insur- mountable task of literature review?

2. What are the common outcomes of a text mining project on a specific collection of journal articles? Can you think of other potential outcomes not mentioned in this case?

Application Case 7.5 (Continued)

TABLE 7.2 Tabular Representation of the Fields Included in the Combined Data Set

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 417

0 20

MISQ ISR

Histogram of JOURNAL; categorized by CLUSTER

JMIS

40 60

(a)

80 100 120 140

MISQ ISR JMIS MISQ ISR JMIS

0 20

MISQ ISR

CLUSTER: 3 CLUSTER: 8 CLUSTER: 6

JMIS

40 60 80

N o

of o

bs

100 120 140

MISQ ISR JMIS MISQ ISR JMIS

0 20

MISQ ISR

CLUSTER: 0 CLUSTER: 4 CLUSTER: 5

CLUSTER: 1 CLUSTER: 7

JOURNAL

CLUSTER: 2

JMIS

40 60 80

100 120 140

MISQ ISR JMIS MISQ ISR JMIS

FIGURE 7.8 (a) Distribution of the Number of Articles for the Three Journals over the Nine Clusters. (b) Development of the Nine Clusters over the Years.

Source: Used with permission of Delen, D., & M. Crossland. (2008). “Seeding the Survey and Analysis of Research Literature

with Text Mining.” Expert Systems with Applications, 34(3), pp. 1707–1720.

Histogram of YEAR; categorized by CLUSTER

N o

of o

bs

0 1994 1996 1998 2000 2002 2004

1995 1997 1999 2001 2003 2005

CLUSTER: 3

40 35 30 25 20 15 10

5

1994 1996 1998 2000 2002 2004 1995 1997 1999 2001 2003 2005

CLUSTER: 8

1994 1996 1998 2000 2002 2004 1995 1997 1999 2001 2003 2005

CLUSTER: 6

0 1994 1996 1998 2000 2002 2004

1995 1997 1999 2001 2003 2005

CLUSTER: 0

40 35 30 25 20 15 10

5

1994 1996 1998 2000 2002 2004 1995 1997 1999 2001 2003 2005

CLUSTER: 4

1994 1996 1998 2000 2002 2004 1995 1997 1999 2001 2003 2005

CLUSTER: 5

0 1994 1996 1998 2000 2002 2004

1995 1997 1999 2001 2003 2005

CLUSTER: 1

40 35 30 25 20 15 10

5

1994 1996 1998 2000 2002 2004 1995 1997 1999 2001 2003 2005

CLUSTER: 7 YEAR

1994 1996 1998 2000 2002 2004 1995 1997 1999 2001 2003 2005

CLUSTER: 2

(b)

418 Part II • Predictive Analytics/Machine Learning

u SECTION 7.5 REVIEW QUESTIONS

1. What are the main steps in the text mining process? 2. What is the reason for normalizing word frequencies? What are the common methods

for normalizing word frequencies?

3. What is SVD? How is it used in text mining? 4. What are the main knowledge extraction methods from corpus?

7.6 SENTIMENT ANALYSIS

We humans are social beings. We are adept at utilizing a variety of means to communicate. We often consult financial discussion forums before making an investment decision; ask our friends for their opinions on a newly opened restaurant or a newly released movie; and conduct Internet searches and read consumer reviews and expert reports before making a big purchase like a house, a car, or an appliance. We rely on others’ opinions to make better decisions, especially in an area where we do not have much knowledge or experience. Thanks to the growing availability and popularity of opinion-rich Internet resources such as social media outlets (e.g., Twitter, Facebook), online review sites, and personal blogs, it is now easier than ever to find opinions of others (thousands of them, as a matter of fact) on everything from the latest gadgets to political and public figures. Even though not everybody expresses opinions over the Internet—due mostly to the fast-growing number and capabilities of social communication channels—the numbers are increasing exponentially.

Sentiment is a difficult word to define. It is often linked to or confused with other terms like belief, view, opinion, and conviction. Sentiment suggests a settled opinion reflective of one’s feelings (Mejova, 2009). Sentiment has some unique properties that set it apart from other concepts that we might want to identify in text. Often we want to categorize text by topic, which could involve dealing with whole taxonomies of topics. Sentiment classification, on the other hand, usually deals with two classes (positive versus negative), a range of polar- ity (e.g., star ratings for movies), or even a range in strength of opinion (Pang and Lee, 2008). These classes span many topics, users, and documents. Although dealing with only a few classes might seem like an easier task than standard text analysis, this is far from the truth.

As a field of research, sentiment analysis is closely related to computational linguis- tics, NLP, and text mining. Sentiment analysis has many names. It is often referred to as opinion mining, subjectivity analysis, and appraisal extraction with some connections to affective computing (computer recognition and expression of emotion). The sudden upsurge of interest and activity in the area of sentiment analysis (i.e., opinion mining), which deals with the automatic extraction of opinions, feelings, and subjectivity in text, is creating opportunities and threats for businesses and individuals alike. The ones who embrace and take advantage of it will greatly benefit from it. Every opinion put on the Internet by an individual or a company will be accredited to the originator (good or bad) and will be retrieved and mined by others (often automatically by computer programs).

Sentiment analysis is trying to answer the question, “What do people feel about a certain topic?” by digging into opinions held by many using a variety of automated tools. Bringing together researchers and practitioners in business, computer science, computa- tional linguistics, data mining, text mining, psychology, and even sociology, sentiment analysis aims to expand the traditional fact-based text analysis to new frontiers, to real- ize opinion-oriented information systems. In a business setting, especially in marketing and CRM, sentiment analysis seeks to detect favorable and unfavorable opinions toward specific products and/or services using large numbers of textual data sources (customer feedback in the form of Web postings, tweets, blogs, etc.).

Sentiment that appears in text comes in two flavors: explicit in which the subjective sentence directly expresses an opinion (“It’s a wonderful day”), and implicit in which the text

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 419

implies an opinion (“The handle breaks too easily”). Most of the earlier work done in senti- ment analysis focused on the first kind of sentiment because it is easier to analyze. Current trends are to implement analytical methods to consider both implicit and explicit sentiments. Sentiment polarity is a particular feature of text on which sentiment analysis primarily focuses. It is usually dichotomized into two—positive and negative—but polarity can also be thought of as a range. A document containing several opinionated statements will have a mixed polarity overall, which is different from not having a polarity at all (being objective; Mejova, 2009). Timely collection and analysis of textual data, which may be coming from a variety of sources—ranging from customer call center transcripts to social media postings—is a crucial part of the capabilities of proactive and customer-focused companies today. These real-time analyses of textual data are often visualized in easy-to-understand dashboards. Application Case 7.6 provides a customer success story in which a collection of analytics solutions is col- lectively used to enhance viewers’ experience at the Wimbledon tennis tournament.

Known to millions of fans simply as “Wimbledon,” The Championships are the oldest of tennis’s four Grand Slams, and one of the world’s highest-profile sporting events. Organized by the All England Lawn Tennis Club (AELTC), it has been a global sporting and cultural institution since 1877.

The Champion of Championships

The organizers of The Championships, Wimbledon, and AELTC have a simple objective: every year, they want to host the best tennis championships in the world—in every way and by every metric.

Application Case 7.6 Creating a Unique Digital Experience to Capture Moments That Matter at Wimbledon

(Continued )

Live scores displayed on Wimbledon.com, the official website of the championships, wimbledon,

Copyright AELTC and IBM. Used with permission.

420 Part II • Predictive Analytics/Machine Learning

The motivation behind this commitment is not simply pride; it also has a commercial basis. Wimbledon’s brand is built on its premier status; this is what attracts both fans and partners. The world’s best media organizations and greatest corporations—IBM included—want to be associated with Wimbledon precisely because of its reputation for excellence.

For this reason, maintaining the prestige of The Championships is one of AELTC’s top priorities, but there are only two ways that the organization can directly control how the rest of the world per- ceives The Championships.

The first, and most important, is to provide an outstanding experience for the players, journalists, and spectators who are lucky enough to visit and watch the tennis courtside. AELTC has vast experi- ence in this area. Since 1877, it has delivered two weeks of memorable, exciting competition in an idyllic setting: tennis in an English country garden.

The second is The Championships’ online pres- ence, which is delivered via the wimbledon.com Web site, mobile apps, and social media channels. The con- stant evolution of these digital platforms is the result of a 26-year partnership between AELTC and IBM.

Mick Desmond, commercial and media director at AELTC, explains, “When you watch Wimbledon on TV, you are seeing it through the broadcaster’s lens. We do everything we can to help our media partners put on the best possible show, but at the end of the day, their broadcast is their presentation of The Championships.”

He adds, “Digital is different: it’s our platform, where we can speak directly to our fans—so it’s vital that we give them the best possible experience. No sporting event or media channel has the right to demand a viewer’s attention, so if we want to strengthen our brand, we need people to see our digital experience as the number-one place to fol- low The Championships online.”

To that end, AELTC set a target of attracting 70 million visits, 20 million unique devices, and 8 million social followers during the two weeks of The Championships in 2015. It was up to IBM and AELTC to find a way to deliver.

Delivering a Unique Digital Experience

IBM and AELTC embarked on a complete redesign of the digital platform, using their intimate knowledge

of The Championships’ audience to develop an experience tailor-made to attract and retain tennis fans from across the globe.

“We recognized that while mobile is increas- ingly important, 80% of our visitors are using desktop computers to access our Web site,” says Alexandra Willis, head of Digital and Content at AELTC. She continued,

Our challenge for 2015 was how to update our digital properties to adapt to a mobile-first world, while still offering the best possible desk- top experience. We wanted our new site to take maximum advantage of that large screen size and give desktop users the richest possible expe- rience in terms of high-definition visuals and video content—while also reacting and adapting seamlessly to smaller tablet or mobile formats.

Second, we placed a major emphasis on putting content in context—integrating articles with relevant photos, videos, stats and snip- pets of information, and simplifying the navi- gation so that users could move seamlessly to the content that interests them most.

On the mobile side, the team recognized that the wider availability of high bandwidth 4G connec- tions meant that the mobile Web site would become more popular than ever—and ensured that it would offer easy access to all rich media content. At the same time, The Championships’ mobile apps were enhanced with real-time notifications of match scores and events—and could even greet visitors as they passed through stations on the way to the grounds.

The team also built a special set of Web sites for the most important tennis fans of all: the play- ers themselves. Using IBM' Bluemix' technology, it built a secure Web application that provided players a personalized view of their court bookings, trans- port, and on-court times, as well as helping them review their performance with access to stats on every match they played.

Turning Data into Insight—and Insight into Narrative

To supply its digital platforms with the most com- pelling possible content, the team took advantage of a unique opportunity: its access to real-time, shot-by-shot data on every match played during

Application Case 7.6 (Continued)

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 421

The Championships. Over the course of the Wimbledon fortnight, 48 courtside experts capture approximately 3.4 million data points, tracking the type of shot, strategies, and outcome of each and every point.

These data are collected and analyzed in real time to produce statistics for TV commentators and journalists—and for the digital platform’s own edito- rial team.

Willis went on to explain:

This year IBM gave us an advantage that we had never had before—using data streaming technology to provide our editorial team with real-time insight into significant milestones and breaking news.

The system automatically watched the streams of data coming in from all 19 courts, and whenever something significant happened— such as Sam Groth hitting the second-fastest serve in Championships’ history—it let us know instantly. Within seconds, we were able to bring that news to our digital audience and share it on social media to drive even more traffic to our site.

The ability to capture the moments that matter and uncover the compelling narratives within the data, faster than anyone else, was key. If you wanted to experience the emotions of The Championships live, the next best thing to being there in person was to follow the action on wimbledon.com.

Harnessing the Power of Natural Language

Another new capability tried in 2015 was the use of IBM’s NLP technologies to help mine AELTC’s huge library of tennis history for interesting contextual infor- mation. The team trained IBM Watson™ Engagement Advisor to digest this rich unstructured data set and use it to answer queries from the press desk.

The same NLP front-end was also connected to a comprehensive structured database of match statistics, dating back to the first Championships in 1877—providing a one-stop shop for both basic questions and more complex inquiries.

“The Watson trial showed a huge amount of potential. Next year, as part of our annual innova- tion planning process, we will look at how we can

use it more widely—ultimately in pursuit of giving fans more access to this incredibly rich source of tennis knowledge,” says Desmond.

Taking to the Cloud

IBM hosted the whole digital environment in its Hybrid Cloud. IBM used sophisticated modeling techniques to predict peaks in demand based on the schedule, popularity of each player, time of day, and many other factors—enabling it to dynamically allocate cloud resources appropriately to each piece of digital content and ensure a seamless experience for millions of visitors around the world.

In addition to the powerful private cloud plat- form that has supported The Championships for several years, IBM also used a separate SoftLayer' cloud to host the Wimbledon Social Command Centre and provide additional incremental capacity to supplement the main cloud environment during times of peak demand.

The elasticity of the cloud environment is key because The Championships’ digital platforms need to be able to scale efficiently by a factor of more than 100 within a matter of days as the interest builds ahead of the first match on Centre Court.

Keeping Wimbledon Safe and Secure

Online security is a key concern today for all orga- nizations. For major sporting events in particu- lar, brand reputation is everything—and while the world is watching, it is particularly important to avoid becoming a high-profile victim of cyber crime. For these reasons, security has a vital role to play in IBM’s partnership with AELTC.

Over the first five months of 2015, IBM secu- rity systems detected a 94 percent increase in secu- rity events on the wimbledon.com infrastructure compared to the same period in 2014.

As security threats—in particular distributed denial of service (DDoS) attacks—become ever more prevalent, IBM continually increases its focus on providing industry-leading levels of security for AELTC’s entire digital platform.

A full suite of IBM security products, includ- ing IBM QRadar' SIEM and IBM Preventia Intrusion Prevention, enabled the 2015 Championships to run smoothly and securely and the digital platform to deliver a high-quality user experience at all times.

(Continued )

422 Part II • Predictive Analytics/Machine Learning

Sentiment Analysis Applications

Compared to traditional sentiment analysis methods, which were survey based or focus group centered, costly, and time consuming (and therefore driven from a small sample of participants), the new face of text analytics–based sentiment analysis is a limit breaker. Current solutions automate very large-scale data collection, filtering, classification, and clustering methods via NLP and data mining technologies that handle both factual and subjective information. Sentiment analysis is perhaps the most popular application of text analytics, tapping into data sources such as tweets, Facebook posts, online communities, discussion boards, Web logs, product reviews, call center logs and recordings, product rating sites, chat rooms, price comparison portals, search engine logs, and newsgroups. The following applications of sentiment analysis are meant to illustrate the power and the widespread coverage of this technology.

VOICE OF THE CUSTOMER Voice of the customer (VOC) is an integral part of ana- lytic CRM and customer experience management systems. As the enabler of VOC, sentiment analysis can access a company’s product and service reviews (either con- tinuously or periodically) to better understand and better manage customer com- plaints and compliments. For instance, a motion picture advertising/marketing company can detect negative sentiments about a movie that is soon to open in theatres (based on its trailers) and quickly change the composition of trailers and advertising strategy (on all media outlets) to mitigate the negative impact. Similarly, a software company can detect the negative buzz regarding the bugs found in their newly released product early enough to release patches and quick fixes to alleviate the situation.

Capturing Hearts and Minds

The success of the new digital platform for 2015— supported by IBM cloud, analytics, mobile, social, and security technologies—was immediate and complete. Targets for total visits and unique visitors were not only met but also exceeded. Achieving 71 million visits and 542 million page views from 21.1 million unique devices demonstrates the platform’s success in attracting a larger audience than ever before and keeping those viewers engaged through- out The Championships.

“Overall, we had 13% more visits from 23% more devices than in 2014, and the growth in the use of wimbledon.com on mobile was even more impressive,” says Willis. “We saw 125% growth in unique devices on mobile, 98% growth in total vis- its, and 79% growth in total page views.”

Desmond concludes, “The results show that in 2015, we won the battle for fans’ hearts and minds. People may have favorite newspapers and sports

website that they visit for 50 weeks of the year—but for two weeks, they came to us instead.”

He continued, “That’s a testament to the sheer quality of the experience we can provide— harnessing our unique advantages to bring them closer to the action than any other media channel. The ability to capture and communicate relevant content in real time helped our fans experience The Championships more vividly than ever before.”

Questions for Case 7.6

1. How did Wimbledon use analytics capabilities to enhance viewers’ experience?

2. What were the challenges, proposed solution, and obtained results?

Source: IBM Case Study. “Creating a Unique Digital Experience to Capture the Moments That Matter.” http:// www-03.ibm.com/software/businesscasestudies/us/en/ corp?synkey=D140192K15783Q68 (accessed May 2016).

Application Case 7.6 (Continued)

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 423

Often, the focus of VOC is individual customers, their service- and support-related needs, wants, and issues. VOC draws data from the full set of customer touch points, including e-mails, surveys, call center notes/recordings, and social media postings, and matches customer voices to transactions (inquiries, purchases, returns) and individual customer profiles captured in enterprise operational systems. VOC, mostly driven by sen- timent analysis, is a key element of customer experience management initiatives, where the goal is to create an intimate relationship with the customer.

VOICE OF THE MARKET (VOM) VOM is about understanding aggregate opinions and trends. It is about knowing what stakeholders—customers, potential customers, influenc- ers, whoever—are saying about your (and your competitors’) products and services. A well-done VOM analysis helps companies with competitive intelligence and product de- velopment and positioning.

VOICE OF THE EMPLOYEE (VOE) Traditionally, VOE has been limited to employee satis- faction surveys. Text analytics in general (and sentiment analysis in particular) is a huge enabler of assessing the VOE. Using rich, opinionated textual data provides an effective and efficient way to listen to what employees are saying. As we all know, happy employ- ees empower customer experience efforts and improve customer satisfaction.

BRAND MANAGEMENT Brand management focuses on listening to social media where anyone (past/current/prospective customers, industry experts, other authorities) can post opinions that can damage or boost a company’s reputation. A number of relatively newly launched start-up companies offer analytics-driven brand management services for oth- ers. Brand management is product and company (rather than customer) focused. It at- tempts to shape perceptions rather than to manage experiences using sentiment analysis techniques.

FINANCIAL MARKETS Predicting the future values of individual (or a group of) stocks has been an interesting and seemingly unsolvable problem. What makes a stock (or a group of stocks) move up or down is anything but an exact science. Many believe that the stock market is mostly sentiment driven, making it anything but rational (especially for short-term stock movements). Therefore, the use of sentiment analysis in financial markets has gained significant popularity. Automated analysis of market sentiment using social media, news, blogs, and discussion groups seems to be a proper way to compute the market movements. If done correctly, sentiment analysis can identify short-term stock movements based on the buzz in the market, potentially impacting liquidity and trading.

POLITICS As we all know, opinions matter a great deal in politics. Because political discussions are dominated by quotes, sarcasm, and complex references to persons, or- ganizations, and ideas, politics is one of the most difficult, and potentially fruitful, areas for sentiment analysis. By analyzing the sentiment on election forums, one might predict who is more likely to win or lose a race. Sentiment analysis can help understand what voters are thinking and can clarify a candidate’s position on issues. Sentiment analysis can help political organizations, campaigns, and news analysts to better understand which is- sues and positions matter the most to voters. The technology was successfully applied by both parties to the 2008 and 2012 U.S. presidential election campaigns.

GOVERNMENT INTELLIGENCE Government intelligence is another application that has been used by intelligence agencies. For example, it has been suggested that one could monitor sources for increases in hostile or negative communications. Sentiment analysis can allow the automatic analysis of the opinions that people submit about pending policy

424 Part II • Predictive Analytics/Machine Learning

or government regulation proposals. Furthermore, monitoring communications for spikes in negative sentiment could be of use to agencies such as Homeland Security.

OTHER INTERESTING AREAS Sentiments of customers can be used to better design e-commerce sites (product suggestions, up-sell/cross-sell advertising), better place adver- tisements (e.g., placing dynamic advertisements of products and services that consider the sentiment on the page the user is browsing), and manage opinion- or review-oriented search engines (i.e., an opinion-aggregation Web site, an alternative to sites similar to Epinions, summarizing user reviews). Sentiment analysis can help with e-mail filtration by categorizing and prioritizing incoming e-mails (e.g., it can detect strongly negative or flaming e-mails and forward them to a proper folder), and citation analysis can determine whether an author is citing a piece of work as supporting evidence or in research but dismisses.

Sentiment Analysis Process

Because of the complexity of the problem (underlying concepts, expressions in text, con- text in which text is expressed, etc.), there is no readily available standardized process to conduct sentiment analysis. However, based on the published work in the field of sensi- tivity analysis so far (both on research methods and range of applications), a multistep, simple logical process as given in Figure 7.9 seems to be an appropriate methodology for sentiment analysis. These logical steps are iterative (i.e., feedback, corrections, and iterations are part of the discovery process) and experimental in nature, and once com- pleted and combined, capable of producing desired insight about the opinions in the text collection.

STEP 1: SENTIMENT DETECTION After retrieval and preparation of the text documents, the first main task in sensitivity analysis is the detection of objectivity. Here the goal is to differentiate between a fact and an opinion, which can be viewed as classification of text as objective or subjective. This can also be characterized as calculation of Objectivity– Subjectivity (3O-S4 Polarity, which can be represented with a numerical value ranging from 0 to 1). If the objectivity value is close to 1, there is no opinion to mine (i.e., it is a fact); therefore, the process goes back and grabs the next text data to analyze. Usually opinion detection is based on the examination of adjectives in text. For example, the po- larity of “what a wonderful work” can be determined relatively easily by looking at the adjective.

STEP 2: N–P (NEGATIVE OR POSITIVE) POLARITY CLASSIFICATION The second main task is that of polarity classification. Given an opinionated piece of text, the goal is to classify the opinion as falling under one of two opposing sentiment polarities or to locate its posi- tion on the continuum between these two polarities (Pang & Lee, 2008). When viewed as a binary feature, polarity classification is the binary classification task of labeling an opin- ionated document as expressing either an overall positive or an overall negative opinion (e.g., thumbs up or thumbs down). In addition to the identification of N–P polarity, one should also be interested in identifying the strength of the sentiment (as opposed to just positive, it can be expressed as mildly, moderately, strongly, or very strongly positive). Most of this research was done on product or movie reviews where the definitions of “positive” and “negative” are quite clear. Other tasks, such as classifying news as “good” or “bad,” present some difficulty. For instance, an article could contain negative news without explicitly using any subjective words or terms. Furthermore, these classes usually appear intermixed when a document expresses both positive and negative sentiments. Then the task can be to identify the main (or dominating) sentiment of the document.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 425

Still, for lengthy texts, the tasks of classification might need to be done at several levels: term, phrase, sentence, and perhaps document level. For those, it is common to use the outputs of one level as the inputs for the next higher layer. Several methods used to iden- tify the polarity and strengths of the polarity are explained in the next section.

STEP 3: TARGET IDENTIFICATION The goal of this step is to accurately identify the target of the expressed sentiment (e.g., a person, a product, an event). The difficulty of this task depends largely on the domain of the analysis. Even though it is usually easy to accurately identify the target for product or movie reviews because the review is directly connected to the target, it can be quite challenging in other domains. For instance, lengthy, general-purpose text such as Web pages, news articles, and blogs do not always have a predefined topic assigned to them and often mention many ob- jects, any of which could be deduced as the target. Sometimes there is more than one target in a sentiment sentence, which is the case in comparative texts. A subjective comparative sentence orders objects in order of preferences—for example, “This lap- top computer is better than my desktop PC.” These sentences can be identified using

Identify the Target for the sentiment

Calculate the N–P Polarity of the

sentiment

Is there a sentiment?

Record the Polarity, Strength,

and the Target of the sentiment.

Textual Data

Calculate the O–S Polarity

YesNo

A statement

Yes

Lexicon

Lexicon

O–S Polarity measure

N–P Polarity

Target

Step 1

Step 2

Step 3

Tabulate & aggregate the sentiment

analysis results

Step 4

FIGURE 7.9 Multistep Process to Sentiment Analysis.

426 Part II • Predictive Analytics/Machine Learning

comparative adjectives and adverbs (more, less, better, longer), superlative adjectives (most, least, best), and other words (such as same, differ, win, prefer). Once the sen- tences have been retrieved, the objects can be put in an order that is most representa- tive of their merits as described in the text.

STEP 4: COLLECTION AND AGGREGATION Once the sentiments of all text data points in the document have been identified and calculated, in this step they are aggregated and converted to a single sentiment measure for the entire document. This aggregation could be as simple as summing up the polarities and strengths of all texts or as complex as using semantic aggregation techniques from NLP to identify the ultimate sentiment.

Methods for Polarity Identification

As mentioned in the previous section, polarity identification can be made at the word, term, sentence, or document level. The most granular level for polarity identification is at the word level. Once the polarity identification has been made at the word level, then it can be aggregated to the next higher level, and then the next until the level of aggrega- tion desired from the sentiment analysis is reached. Two dominant techniques have been used for identification of polarity at the word/term level, each having its advantages and disadvantages:

1. Using a lexicon as a reference library (developed either manually or automat- ically by an individual for a specific task or developed by an institution for general use).

2. Using a collection of training documents as the source of knowledge about the polarity of terms within a specific domain (i.e., inducing predictive models from opinionated textual documents).

Using a Lexicon

A lexicon is essentially the catalog of words, their synonyms, and their meanings for a given language. In addition to lexicons for many other languages, there are several general-purpose lexicons created for English. Often general-purpose lexicons are used to create a variety of special-purpose lexicons for use in sentiment analysis projects. Perhaps the most popular general-purpose lexicon is WordNet created at Princeton University; it has been extended and used by many researchers and practitioners for sen- timent analysis purposes. As described on the WordNet Web site (wordnet. princeton. edu), it is a large lexical database of English, including nouns, verbs, adjectives, and adverbs grouped into sets of cognitive synonyms (i.e., synsets), each expressing a dis- tinct concept. Synsets are interlinked by means of conceptual–semantic and lexical relations.

An interesting extension of WordNet was created by Esuli and Sebastiani (2006) where they added polarity (Positive–Negative; P–N) and objectivity (Subjective– Objective; S–O) labels for each term in the lexicon. To label each term, they classified the synset (a group of synonyms) to which a term belongs using a set of ternary classi- fiers (a measure that attaches to each object exactly one of three labels), each capable of deciding whether a synset is positive, or negative, or objective. The resulting scores range from 0.0 to 1.0, giving a graded evaluation of opinion-related properties of the terms. These can be summed up visually as in Figure 7.10. The edges of the triangle represent one of the three classifications (positive, negative, and objective). A term can be located in this space as a point representing the extent to which it belongs to each of the classifications.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 427

A similar extension methodology is used to create SentiWordNet, a publicly avail- able lexicon specifically developed for opinion mining (sentiment analysis) purposes. SentiWordNet assigns to each synset of WordNet three sentiment scores: positivity, negativity, and objectivity. More about SentiWordNet can be found at sentiwordnet. isti.cnr.it.

Another extension to WordNet is WordNet-Affect, developed by Strapparava and Valitutti (2004). They label WordNet synsets using affective labels representing different affective categories (emotion, cognitive state, attitude, and feeling). WordNet has also been directly used in sentiment analysis. For example, Kim and Hovy (2004) and Liu, Hu, and Cheng (2005) generate lexicons of positive and negative terms by starting with a small list of “seed” terms of known polarities (e.g., love, like, nice) and then using the antonymy and synonymy properties of terms to group them into either of the polarity categories.

Using a Collection of Training Documents

It is possible to perform sentiment classification using statistical analysis and machine- learning tools that take advantage of the vast resources of labeled (manually by annota- tors or using a star/point system) documents available. Product review Web sites such as Amazon, C-NET, eBay, RottenTomatoes, and the Internet Movie Database have all been extensively used as sources of annotated data. The star (or tomato, as it were) system provides an explicit label of the overall polarity of the review, and it is often taken as a gold standard in algorithm evaluation.

A variety of manually labeled textual data is available through evaluation ef- forts such as the Text REtrieval Conference, NII Test Collection for IR Systems, and Cross Language Evaluation Forum. The data sets these efforts produce often serve as a standard in the text mining community including sentiment analysis research- ers. Individual researchers and research groups have also produced many interesting data sets. Technology Insights 7.2 lists some of the most popular ones. Once an al- ready labeled textual data set has been obtained, a variety of predictive modeling and other machine-learning algorithms can be used to train sentiment classifiers. Some of the most popular algorithms used for this task include artificial neural networks, sup- port vector machines, k-nearest neighbor, Naïve Bayes, decision trees, and expectation maximization-based clustering.

Positive (P) (1)

Negative (N) (2)

Objective (O)

Subjective (S)

P–N Polarity

S –

O P

ol ar

it y

FIGURE 7.10 Graphical Representation of the P–N Polarity and S–O Polarity Relationship.

428 Part II • Predictive Analytics/Machine Learning

TECHNOLOGY INSIGHTS 7.2 Large Textual Data Sets for Predictive Text Mining and Sentiment Analysis

Following are a few of the most commonly used examples to large textual data sets:

Congressional Floor-Debate Transcripts: Published by Thomas, Pang, and Lee (2006); contains political speeches that are labeled to indicate whether the speaker supported or opposed the legislation discussed.

Economining: Published by the Stern School at New York University; consists of feedback postings for merchants at Amazon.com.

Cornell Movie-Review Data Sets: Introduced by Pang and Lee (2008); contains 1,000 positive and 1,000 negative automatically derived document-level labels and 5,331 positive and 5,331 negative sentences/snippets.

Stanford—Large Movie Review Data Set: A set of 25,000 highly polar movie reviews for training and 25,000 for testing. There is additional unlabeled data for use as well. Raw text and already processed bag-of-words formats are provided. (See http://ai.stanford. edu/~amaas/data/sentiment.)

MPQA Corpus: Corpus and Opinion Recognition System corpus; contains 535 manually annotated news articles from a variety of news sources containing labels for opinions and private states (beliefs, emotions, speculations, etc.).

Multiple-Aspect Restaurant Reviews: Introduced by Snyder and Barzilay (2007); con- tains 4,488 reviews with an explicit 1-to-5 rating for five different aspects: food, ambiance, service, value, and overall experience.

Identifying Semantic Orientation of Sentences and Phrases

Once the semantic orientation of individual words has been determined, it is often desir- able to extend this to the phrase or sentence in which the word appears. The simplest way to accomplish such aggregation is to use some type of averaging for the polarities of words in the phrases or sentences. Though rarely applied, such aggregation can be as complex as using one or more machine-learning techniques to create a predictive rela- tionship between the words (and their polarity values) and phrases or sentences.

Identifying Semantic Orientation of Documents

Even though the vast majority of the work in this area is done in determining semantic orientation of words and phrases/sentences, some tasks such as summarization and in- formation retrieval could require semantic labeling of the whole document (Ramage et al., 2009). Similar to the case in aggregating sentiment polarity from word level to phrase or sentence level, aggregation to document level is also accomplished by some type of averaging. Sentiment orientation of the document might not make sense for very large documents; therefore, it is often used on small to medium-size documents posted on the Internet.

u SECTION 7.6 REVIEW QUESTIONS

1. What is sentiment analysis? How does it relate to text mining? 2. What are the most popular application areas for sentiment analysis? Why? 3. What would be the expected benefits and beneficiaries of sentiment analysis in

politics?

4. What are the main steps in carrying out sentiment analysis projects? 5. What are the two common methods for polarity identification? Explain.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 429

7.7 WEB MINING OVERVIEW

The Internet has changed the landscape for conducting business forever. Because of the highly connected, flattened world and broadened competition field, today’s companies are increasingly facing more opportunities (being able to reach customers and markets that they might never have thought possible) and more challenges (a globalized and ever- changing competitive marketplace). Companies with the vision and capabilities to deal with such a volatile environment are greatly benefiting from it, whereas others who resist adapting are having difficulty surviving. Having an engaged presence on the Internet is not a choice anymore; it is a business requirement. Customers are expecting companies to offer their products and/or services over the Internet. Customers are not only buying products and services but also talking about companies and sharing their transactional and usage experiences with others over the Internet.

The growth of the Internet and its enabling technologies has made data creation, data collection, and data/information/opinion exchange easier. Delays in service, manufacturing, shipping, delivery, and customer inquiries are no longer private incidents and are accepted as necessary evils. Now, thanks to social media tools and technologies on the Internet, ev- erybody knows everything. Successful companies are the ones that embrace these Internet technologies and use them to improve their business processes to better communicate with their customers, understand their needs and wants, and serve them thoroughly and expedi- tiously. Being customer focused and keeping customers happy has never been as important a concept for businesses as they are now in this age of the Internet and social media.

The World Wide Web (or for short, Web) serves as an enormous repository of data and information on virtually everything one can conceive; business, personal, you name it—an abundant amount of it is there. The Web is perhaps the world’s largest data and text repository, and the amount of information on the Web is growing rapidly. Much interesting information can be found online: whose home page is linked to which other pages, how many people have links to a specific Web page, and how a particular site is organized. In addition, each visitor to a Web site, each search on a search engine, each click on a link, and each transaction on an e-commerce site create additional data. Although unstructured textual data in the form of Web pages coded in HTML or XML are the dominant content of the Web, the Web infrastructure also contains hyperlink informa- tion (connections to other Web pages) and usage information (logs of visitors’ interac- tions with Web sites), all of which provide rich data for knowledge discovery. Analysis of this information can help us make better use of Web sites and also aid us in enhancing relationships and value for the visitors to our own Web sites.

Because of its sheer size and complexity, mining the Web is not an easy undertak- ing by any means. The Web also poses great challenges for effective and efficient knowl- edge discovery (Han & Kamber, 2006):

• The Web is too big for effective data mining. The Web is so large and growing so rapidly that it is difficult to even quantify its size. Because of the sheer size of the Web, it is not feasible to set up a data warehouse to replicate, store, and integrate all of the data on the Web, making data collection and integration a challenge.

• The Web is too complex. The complexity of a Web page is far greater than that of a page in a traditional text document collection. Web pages lack a unified struc- ture. They contain far more authoring style and content variation than any set of books, articles, or other traditional text-based document.

• The Web is too dynamic. The Web is a highly dynamic information source. Not only does the Web grow rapidly but also its content is constantly being updated. Blogs, news stories, stock market results, weather reports, sports scores, prices, company advertisements, and numerous other types of information are updated regularly on the Web.

430 Part II • Predictive Analytics/Machine Learning

• The Web is not specific to a domain. The Web serves a broad diversity of com- munities and connects billions of workstations. Web users have very different back- grounds, interests, and usage purposes. Most users might not have good knowledge of the structure of the information network and might not be aware of the heavy cost of a particular search that they perform.

• The Web has everything. Only a small portion of the information on the Web is truly relevant or useful to someone (or some task). It is said that 99 percent of the information on the Web is useless to 99 percent of Web users. Although this might not seem obvious, it is true that a particular person is generally interested in only a tiny portion of the Web, whereas the rest of the Web contains information that is uninteresting to the user and could swamp desired results. Finding the portion of the Web that is truly relevant to a person and the task being performed is a promi- nent issue in Web-related research.

These challenges have prompted many research efforts to enhance the effectiveness and efficiency of discovering and using data assets on the Web. A number of index-based Web search engines constantly search the Web and index Web pages under certain keywords. Using these search engines, an experienced user might be able to locate docu- ments by providing a set of tightly constrained keywords or phrases. However, a sim- ple keyword-based search engine suffers from several deficiencies. First, a topic of any breadth can easily contain hundreds or thousands of documents. This can lead to a large number of document entries returned by the search engine, many of which are marginally relevant to the topic. Second, many documents that are highly relevant to a topic might not contain the exact keywords defining them. As we cover in more detail later in this chapter, compared to keyword-based Web search, Web mining is a prominent (and more challeng- ing) approach that can be used to substantially enhance the power of Web search engines because Web mining can identify authoritative Web pages, classify Web documents, and resolve many ambiguities and subtleties raised in keyword-based Web search engines.

Web mining (or Web data mining) is the process of discovering intrinsic relation- ships (i.e., interesting and useful information) from Web data, which are expressed in the form of textual, linkage, or usage information. The term Web mining was first used by Etzioni (1996); today, many conferences, journals, and books focus on Web data min- ing. It is a continually evolving area of technology and business practice. Web mining is essentially the same as data mining that uses data generated over the Web. The goal is to turn vast repositories of business transactions, customer interactions, and Web site usage data into actionable information (i.e., knowledge) to promote better decision mak- ing throughout the enterprise. Because of the increased popularity of the term analytics, today many have started to refer to Web mining as Web analytics. However, these two terms are not the same. Whereas Web analytics is primarily Web site usage focused data, Web mining is inclusive of all data generated via the Internet including transaction, social, and usage data. Where Web analytics aims to describe what has happened on the Web site (employing a predefined, metrics-driven descriptive analytics methodology), Web mining aims to discover previously unknown patterns and relationships (employing a novel predictive or prescriptive analytics methodology). From a big-picture perspective, Web analytics can be considered to be a part of Web mining. Figure 7.11 presents a sim- ple taxonomy of Web mining divided into three main areas: Web content mining, Web structure mining, and Web usage mining. In the figure, the data sources used in these three main areas are also specified. Although these three areas are shown separately, as you will see in the following section, they are often used collectively and synergistically to address business problems and opportunities.

As Figure 7.11 indicates, Web mining relies heavily on data mining and text mining and their enabling tools and techniques, which we covered in detail early in this chapter

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 431

and in the previous chapter (Chapter 4). The figure also indicates that these three generic areas are further extended into several very well-known application areas. Some of these areas were explained in previous chapters, and some of the others are covered in detail in this chapter.

Web Content and Web Structure Mining

Web content mining refers to the extraction of useful information from Web pages. The documents can be extracted in some machine-readable format so that automated tech- niques can extract some information from these Web pages. Web crawlers (also called spiders) are used to read through the content of a Web site automatically. The informa- tion gathered could include document characteristics similar to what is used in text min- ing, but it could also include additional concepts, such as the document hierarchy. Such an automated (or semiautomated) process of collecting and mining of Web content can be used for competitive intelligence (collecting intelligence about competitors’ products, services, and customers). It can also be used for information/news/opinion collection and summarization, sentiment analysis, and automated data collection and structuring for predictive modeling. As an illustrative example of using Web content mining as an automated data collection tool, consider the following. For more than 10 years now, two of the three authors of this book (Drs. Sharda and Delen) have been developing mod- els to predict the financial success of Hollywood movies before their theatrical release. The data that they use for training of the models come from several Web sites, each having a different hierarchical page structure. Collecting a large set of variables on thou- sands of movies (from the past several years) from these Web sites is a time-demanding,

Marketing Attribution Customer Analytics 360 Customer View Voice of the Customer

Search Engine Optimization Social Network Analysis Social Media Analytics Weblog Analysis

Page Rank Information Retrieval Graph Mining Social Analytics Clickstream Analysis

Data Mining

Text Mining

Web Mining

Web Structure Mining Source: the unified

resource locator (URL) links contained in the

Web pages

Web Content Mining Source: unstructured textual content of the Web pages (usually in

HTML format)

Web Usage Mining Source: the detailed description of a Web site’s visits (sequence of clicks by sessions)

Web AnalyticsSearch Engines Sentiment Analysis Semantic Webs

FIGURE 7.11 Simple Taxonomy of Web Mining.

432 Part II • Predictive Analytics/Machine Learning

error-prone process. Therefore, Sharda and Delen use Web content mining and spiders as an enabling technology to automatically collect, verify, validate (if the specific data item is available on more than one Web site, then the values are validated against each other and anomalies are captured and recorded), and store these values in a relational database. That way, they ensure the quality of the data while saving valuable time (days or weeks) in the process.

In addition to text, Web pages also contain hyperlinks pointing one page to an- other. Hyperlinks contain a significant amount of hidden human annotation that can potentially help to automatically infer the notion of centrality or authority. When a Web page developer includes a link pointing to another Web page, this could be regarded as the developer’s endorsement of the other page. The collective endorsement of a given page by different developers on the Web might indicate the importance of the page and might naturally lead to the discovery of authoritative Web pages (Miller, 2005). Therefore, the vast amount of Web linkage information provides a rich collection of information about the relevance, quality, and structure of the Web’s contents and thus is a rich source for Web mining.

Web content mining can also be used to enhance the results produced by search engines. In fact, search is perhaps the most prevailing application of Web content min- ing and Web structure mining. A search on the Web to obtain information on a specific topic (presented as a collection of keywords or a sentence) usually returns a few relevant, high-quality Web pages and a larger number of unusable Web pages. Use of a relevance index based on keywords and authoritative pages (or some measure of it) improves the search results and ranking of relevant pages. The idea of authority (or authoritative pages) stems from earlier information retrieval work using citations among journal ar- ticles to evaluate the impact of research papers (Miller, 2005). Although that was the origination of the idea, there are significant differences between the citations in research articles and hyperlinks on Web pages. First, not every hyperlink represents an endorse- ment (some links are created for navigation purposes and some are for paid advertise- ments). Although this is true, if the majority of the hyperlinks are of the endorsement type, then the collective opinion will still prevail. Second, for commercial and competi- tive interests, one authority rarely has its Web page point to rival authorities in the same domain. For example, Microsoft might prefer not to include links on its Web pages to Apple’s Web sites because this could be regarded as an endorsement of its competitor’s authority. Third, authoritative pages are seldom particularly descriptive. For example, the main Web page of Yahoo! might not contain the explicit self-description that it is in fact a Web search engine.

The structure of Web hyperlinks has led to another important category of Web pages called a hub, which is one or more Web pages that provide a collection of links to authoritative pages. Hub pages might not be prominent, and only a few links might point to them; however, hubs provide links to a collection of prominent sites on a specific topic of interest. A hub could be a list of recommended links on an individual’s home page, recommended reference sites on a course Web page, or a professionally assembled resource list on a specific topic. Hub pages play the role of implicitly conferring the authorities on a narrow field. In essence, a close symbiotic relationship exists between good hubs and authoritative pages; a good hub is good because it points to many good authorities; and a good authority is good because it is being pointed to by many good hubs. Such relationships between hubs and authorities make it possible to automatically retrieve high-quality content from the Web.

The most popular publicly known and referenced algorithm used to calculate hubs and authorities is hyperlink-induced topic search (HITS). It was originally developed by Kleinberg (1999) and has since been improved by many researchers. HITS is a link- analysis algorithm that rates Web pages using the hyperlink information contained within

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 433

them. In the context of Web search, the HITS algorithm collects a base document set for a specific query. It then recursively calculates the hub and authority values for each document. To gather the base document set, a root set that matches the query is fetched from a search engine. For each document retrieved, a set of documents that points to the original document and another set of documents that is pointed to by the original docu- ment are added to the set as the original document’s neighborhood. A recursive process of document identification and link analysis continues until the hub and authority values converge. These values are then used to index and prioritize the document collection generated for a specific query.

Web structure mining is the process of extracting useful information from the links embedded in Web documents. It is used to identify authoritative pages and hubs, which are the cornerstones of the contemporary page-rank algorithms that are central to popular search engines such as Google and Yahoo! Just as links going to a Web page can indicate a site’s popularity (or authority), links within the Web page (or the complete Web site) can indicate the depth of coverage of a specific topic. Analysis of links is very important in understanding the interrelationships among large numbers of Web pages, leading to a better understanding of a specific Web community, clan, or clique.

u SECTION 7.7 REVIEW QUESTIONS

1. What are some of the main challenges the Web poses for knowledge discovery? 2. What is Web mining? How does it differ from regular data mining or text mining? 3. What are the three main areas of Web mining? 4. What is Web content mining? How can it be used for competitive advantage? 5. What is Web structure mining? How does it differ from Web content mining?

7.8 SEARCH ENGINES

In this day and age, there is no denying the importance of Internet search engines. As the size and complexity of the World Wide Web increase, finding what you want is becoming a complex and laborious process. People use search engines for a variety of reasons. We use them to learn about a product or service before committing to buy it (including who else is selling it, what the prices are at different locations/sellers, common issues people are discussing about it, how satisfied previous buyers are, and what other products or services might be better) and to search for places to go, people to meet, things to do. In a sense, search engines have become the centerpiece of most Internet-based transactions and other activities. The incredible success and popularity of Google, the most popular search engine company, is a good testament to this claim. What is somewhat of a mystery to many is how a search engine actually does what it is meant to do. In simplest terms, a search engine is a software program that searches for documents (Internet sites or files) based on the keywords (individual words, multiword terms, or a complete sentence) users have provided that have to do with the subject of their inquiry. Search engines are the workhorses of the Internet, responding to billions of queries in hundreds of different languages every day.

Technically speaking, “search engine” is the popular term for information retrieval systems. Although Web search engines are the most popular, search engines are often used in contexts other than the Web, such as desktop search engines and document search engines. As you will see in this section, many of the concepts and techniques that we covered in text analytics and text mining early in this chapter also apply here. The overall goal of a search engine is to return one or more documents/pages (if more than one document/page applies, then a ranked-order list is often provided) that best match the user’s query. The two metrics that are often used to evaluate search engines are

434 Part II • Predictive Analytics/Machine Learning

effectiveness (or quality—finding the right documents/pages) and efficiency (or speed— returning a response quickly). These two metrics tend to work in reverse directions; improving one tends to worsen the other. Often, based on the user expectation, search engines focus on one at the expense of the other. Better search engines are the ones that excel in both at the same time. Because search engines not only search but also find and return documents/pages, perhaps a more appropriate name for them would have been finding engines.

Anatomy of a Search Engine

Now let us dissect a search engine and look inside it. At the highest level, a search engine system is composed of two main cycles: a development cycle and a responding cycle (see the structure of a typical Internet search engine in Figure 7.12). While one is interfacing with the World Wide Web, the other is interfacing with the user. One can think of the development cycle as a production process (manufacturing and inventorying documents/ pages) and the responding cycle as a retailing process (providing customers/users what they want). In the following section, these two cycles are explained in more detail.

1. Development Cycle

The two main components of the development cycle are the Web crawler and document indexer. The purpose of this cycle is to create a huge database of documents/pages orga- nized and indexed based on their content and information value. The reason for devel- oping such a repository of documents/pages is quite obvious: Due to its sheer size and complexity, searching the Web to find pages in response to a user query is not practical (or feasible within a reasonable time frame); therefore, search engines “cache the Web” into their database and use the cached version of the Web for searching and finding. Once created, this database allows search engines to rapidly and accurately respond to user queries.

WEB CRAWLER A Web crawler (also called a spider or Web spider) is a piece of software that systematically browses (crawls through) the World Wide Web for the purpose of finding and fetching Web pages. Often Web crawlers copy all the pages they visit for later processing by other functions of a search engine.

Query Analyzer

Document Matcher/Ranker

Web Crawler

Document Indexer

Scheduler

Cashed/Indexed Documents DB

User World Wide Web

Se arc

h

Qu ery

Processed Query

Lis t o

f U RL

s t o

Cra wl

Crawling the Web

Un pro

ces sed

We b P

age s

Processed Pages Lis

t o f M

atc hed

Pa ges

Ranked-

Ordered Pages

Responding Cycle Development Cycle

M et

ad at

a

In de

x

FIGURE 7.12 Structure of a Typical Internet Search Engine.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 435

A Web crawler starts with a list of URLs to visit, which are listed in the scheduler and often are called the seeds. These URLs can come from submissions made by Webmasters or, more often, from the internal hyperlinks of previously crawled documents/pages. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit (i.e., the scheduler). URLs in the scheduler are recursively visited according to a set of policies determined by the specific search engine. Because there are large volumes of Web pages, the crawler can download only a limited number of them within a given time; therefore, it might need to prioritize its downloads.

DOCUMENT INDEXER As the documents are found and fetched by the crawler, they are stored in a temporary staging area for the document indexer to grab and process. The document indexer is responsible for processing the documents (Web pages or document files) and placing them into the document database. To convert the documents/pages into the desired, easily searchable format, the document indexer performs the following tasks.

STEP 1: PREPROCESSING THE DOCUMENTS Because the documents fetched by the crawler might all be in different formats, for the ease of processing them further, in this step they all are converted to some type of standard representation. For instance, different content types (text, hyperlink, image, etc.) could be separated from each other, formatted (if necessary), and stored in a place for further processing.

STEP 2: PARSING THE DOCUMENTS This step is essentially the application of text mining (i.e., computational linguistic, NLP) tools and techniques to a collection of documents/ pages. In this step, first the standardized documents are parsed into components to iden- tify index-worthy words/terms. Then, using a set of rules, the words/terms are indexed. More specifically, using tokenization rules, the words/terms/entities are extracted from the sentences in these documents. Using proper lexicons, the spelling errors and other anomalies in these words/terms are corrected. Not all the terms are discriminators. The nondiscriminating words/terms (also known as stop words) are eliminated from the list of index-worthy words/terms. Because the same word/term can be in many different forms, stemming is applied to reduce the words/terms to their root forms. Again, using lexicons and other language-specific resources (e.g., WordNet), synonyms and homonyms are iden- tified, and the word/term collection is processed before moving into the indexing phase.

STEP 3: CREATING THE TERM-BY-DOCUMENT MATRIX In this step, the relationships be- tween the words/terms and documents/pages are identified. The weight can be as simple as assigning 1 for presence or 0 for absence of the word/term in the document/page. Usually more sophisticated weight schemas are used. For instance, as opposed to binary, one can choose to assign frequency of occurrence (number of times the same word/ term is found in a document) as a weight. As we saw earlier in this chapter, text mining research and practice have clearly indicated that the best weighting could come from the use of term frequency divided by inverse document frequency (TF/IDF). This algorithm measures the frequency of occurrence of each word/term within a document and then compares that frequency against the frequency of occurrence in the document collection. As we all know, not all high-frequency words/terms are good document discriminators, and a good document discriminator in a domain might not be one in another domain. Once the weighing schema is determined, the weights are calculated and the term-by- document index file is created.

2. Response Cycle

The two main components of the responding cycle are the query analyzer and the document matcher/ranker.

436 Part II • Predictive Analytics/Machine Learning

QUERY ANALYZER The query analyzer is responsible for receiving a search request from the user (via the search engine’s Web server interface) and converting it into a standardized data structure so that it can be easily queried/matched against the entries in the document database. How the query analyzer does what it is supposed to do is quite similar to what the document indexer does (as we have just explained). The query analyzer parses the search string into individual words/terms using a series of tasks that include tokenization, removal of stop words, stemming, and word/term dis- ambiguation (identification of spelling errors, synonyms, and homonyms). The close similarity between the query analyzer and the document indexer is not coincidental. In fact, it is quite logical because both are working off the document database; one is putting in documents/pages using a specific index structure, and the other is convert- ing a query string into the same structure so that it can be used to quickly locate most relevant documents/pages.

DOCUMENT MATCHER/RANKER This is where the structured query data are matched against the document database to find the most relevant documents/pages and rank them in the order of relevance/importance. The proficiency of this step is perhaps the most important component when different search engines are compared to one another. Every search engine has its own (often proprietary) algorithm that it uses to carry out this im- portant step.

The early search engines used a simple keyword match against the document data- base and returned a list of ordered documents/pages when the determinant of the order was a function that used the number of words/terms matched between the query and the document along with the weights of those words/terms. The quality and the usefulness of the search results were not all that good. Then, in 1997, the creators of Google came up with a new algorithm, called PageRank. As the name implies, PageRank is an algorith- mic way to rank order documents/pages based on their relevance and value/importance. Even though PageRank is an innovative way to rank documents/pages, it is an augmenta- tion to the process of retrieving relevant documents from the database and ranking them based on the weights of the words/terms. Google does all of these collectively and more to identify the most relevant list of documents/pages for a given search request. Once an ordered list of documents/pages is created, it is pushed back to the user in an easily digestible format. At this point, users might choose to click on any of the documents in the list, and it might not be the one at the top. If they click on a document/page link that is not at the top of the list, then can we assume that the search engine did not do a good job ranking them? Perhaps, yes. Leading search engines like Google monitor the performance of their search results by capturing, recording, and analyzing postdelivery user actions and experiences. These analyses often lead to more and more rules to further refine the ranking of the documents/pages so that the links at the top are more preferable to the end users.

Search Engine Optimization

Search engine optimization (SEO) is the intentional activity of affecting the visibility of an e-commerce site or a Web site in a search engine’s natural (unpaid or organic) search results. In general, the higher it is ranked on the search results page, and the more frequently a site appears in the search results list, the more visitors it will receive from the search engine’s users. As an Internet marketing strategy, SEO considers how search engines work, what people search for, the actual search terms or keywords typed into search engines, and which search engines are preferred by their targeted audience. Optimizing a Web site can involve editing its content, HTML, and associated coding to both increase its relevance to specific keywords and to remove barriers to the indexing

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 437

activities of search engines. Promoting a site to increase the number of backlinks, or in- bound links, is another SEO tactic.

In the early days, in order to be indexed, the only thing that Webmasters needed to do was to submit the address of a page, or URL, to the various engines, which would then send a “spider” to “crawl” that page, extract links to other pages from it, and return infor- mation found on the page to the server for indexing. The process, as explained before, involves a search engine spider downloading a page and storing it on the search engine’s own server, where a second program, known as an indexer, extracts various informa- tion about the page, such as the words it contains and where they are located as well as any weight for specific words, and all links the page contains, which are then placed into a scheduler for crawling at a later date. Today, search engines are no longer rely- ing on Webmasters submitting URLs (even though they still can); instead, search engines are proactively and continuously crawling the Web and finding, fetching, and indexing everything about it.

Being indexed by search engines such as Google, Bing, and Yahoo! is not good enough for businesses. Getting ranked on the most widely used search engines (see Technology Insights 7.3 for a list of most widely used search engines) and getting ranked higher than your competitors are what make a difference in the eye of the customers and other constituents. A variety of methods can increase the ranking of a Web page within the search results. Cross-linking between pages of the same Web site to provide more links to the most important pages could improve its visibility. Writing content that includes frequently searched keyword phrases to be relevant to a wide variety of search queries tends to increase traffic. Updating content to keep search engines crawling back frequently can give additional weight to a site. Adding relevant keywords to a Web page’s metadata, including the title tag and metadescription, will tend to improve the relevancy of a site’s search listings, thus increasing traffic. URL normalization of Web pages (so that they are accessible via multiple and simpler URLs) and using canonical link elements and redirects can help make sure that the links to different versions of the Web pages and their URLs all count toward the Web site’s link popularity score.

Methods for Search Engine Optimization

In general, SEO techniques can be classified into two broad categories: techniques that search engines recommend as part of good site design and those techniques of which search engines do not approve. The search engines attempt to minimize the effect of the latter, which is often called spamdexing (also known as search spam, search engine spam, or search engine poisoning). Industry commentators, and the practitioners who employ them, have classified these methods as either white-hat SEO or black-hat SEO (Goodman, 2005). White hats tend to produce results that last a long time, whereas black hats anticipate that their sites might eventually be banned either temporarily or perma- nently once the search engines discover what they are doing.

An SEO technique is considered white hat if it conforms to the search engine’s guidelines and involves no deception. Because search engine guidelines are not written as a series of rules or commandments, this is an important distinction to note. White- hat SEO is not just about following guidelines but also about ensuring that the content a search engine indexes and subsequently ranks is the same content a user will see. White-hat advice is generally summed up as creating content for users, not for search engines, and then making that content easily accessible to the spiders rather than at- tempting to trick the algorithm from its intended purpose. White-hat SEO is in many ways similar to Web development that promotes accessibility, although the two are not identical.

438 Part II • Predictive Analytics/Machine Learning

TECHNOLOGY INSIGHTS 7.3 Top 15 Most Popular Search Engines (August 2016)

These are the 15 most popular search engines as derived from eBizMBA Rank (ebizmba.com/ articles/search-engines), which is a constantly updated average of each Web site’s Alexa Global Traffic Rank and U.S. Traffic Rank from both Compete and Quantcast.

Rank Name Estimated Unique Monthly Visitors

1 Google 1,600,000,000

2 Bing 400,000,000

3 Yahoo! Search 300,000,000

4 Ask 245,000,000

5 AOL Search 125,000,000

6 Wow 100,000,000

7 WebCrawler 65,000,000

8 MyWebSearch 60,000,000

9 Infospace 24,000,000

10 Info 13,500,000

11 DuckDuckGo 11,000,000

12 Contenko 10,500,000

13 Dogpile 7,500,000

14 Alhea 4,000,000

15 ixQuick 1,000,000

Black-hat SEO attempts to improve rankings in ways that are not approved by the search engines or involve deception. One black-hat technique uses text that is hidden, either as text colored similar to the background, in an invisible div tag (that defines a division or a section in an HTML document), or positioned off-screen. Another method gives a different page depend- ing on whether the page is being requested by a human visitor or a search engine, a technique known as cloaking. Search engines can penalize sites they discover using black-hat methods, either by reducing their rankings or eliminating their listings from their databases altogether. Such penalties can be applied either automatically by the search engines’ algorithms or by a manual site review. One example was the February 2006 Google removal of both BMW Germany and Ricoh Germany for use of unapproved practices (Cutts, 2006). Both companies, however, quickly apologized, fixed their practices, and were restored to Google’s list.

For some businesses, SEO can generate a significant return on investment. However, one should keep in mind that search engines are not paid for organic search traffic, their algorithms change constantly, and there are no guarantees of continued referrals. Due to this lack of certainty and stability, a business that relies heavily on search engine traffic can suffer major losses if the search engine decides to change its algorithms and stop sending visitors. According to Google’s CEO, Eric Schmidt, in 2010, Google made over 500 algo- rithm changes—almost 1.5 per day. Because of the difficulty in keeping up with changing search engine rules, companies that rely on search traffic practice one or more of the fol- lowing: (1) hire a company that specializes in SEO (there seem to be an abundant number of those today) to continuously improve your site’s appeal to changing practices of the search engines, (2) pay the search engine providers to be listed on the paid sponsors’ sec- tions, and (3) consider liberating yourself from dependence on search engine traffic.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 439

Either originating from a search engine (organically or otherwise) or coming from other sites and places, what is most important for an e-commerce site is to maximize the likelihood of customer transactions. Having many visitors without sales is not what a typical e-commerce site is built for. Application Case 7.7 discusses a large Internet- based shopping mall where detailed analysis of customer behavior (using clickstreams and other data sources) is used to significantly improve the conversion rate.

Either originating from a search engine (organically or otherwise), responding to email- based marketing campaigns, or coming from social media sites, what is most important for an e-commerce site is to maximize its leads and subsequent customer sales transactions. Application Case 7.7 shows how a century-old fashionable cloth and accessory company used email-based campaigns to generate large number of new leads for its e-commerce business.

respected around the world, Barbour was aware of the importance in establishing direct relationships with its target audience—especially when encouraging users to engage with its new e-commerce platform. It also understood that it needed to take more control of shaping the customer journey. That way Barbour could create and maintain the same exceptional level of quality in the user experience as that applied to the manufacturing of its products. To do this, the com- pany needed to develop its understanding of its target market’s online behavior. With the goal of reaching its target audience in order to build meaningful cus- tomer relationships, Barbour approached Teradata. Barbour’s marketing department needed Teradata Interactive to offer a solution that would increase its knowledge of the unique characteristics and needs of its individual customers, as well as support the launch of its new UK e-commerce website.

The Solution: Implementing a Lead Nurture Program

The increasing shift to global e-commerce and the growth in digital consumerism require brands to hold a strong online presence. This also means that retail- ers have to implement strategies that support their cus- tomers’ evolving wants and needs, online and offline. Barbour and Teradata Interactive embarked on the design and construction of a Lead Nurture Program that ran over a one-month period. The campaign objective was to not only raise awareness and create demand for immediate sales activity but also to create a more long- term engagement mechanism that would lead to more

Application Case 7.7 Delivering Individualized Content and Driving Digital Engagement: How Barbour Collected More Than 49,000 New Leads in One Month with Teradata Interactive

Background

Founded in 1894, Barbour is an English heritage and lifestyle brand renowned for its waterproof outerwear—especially its classic waxed-cotton jacket. With more than 10,000 jackets ordered and hand- made each year, Barbour has held a strong position in the luxury goods industry for more than a century, building a strong relationship with fashion-conscious men and women of the British countryside. In 2000, Barbour broadened its product offering to include a full lifestyle range of everyday clothes and accesso- ries. Its major markets are the United Kingdom, the United States, and Germany; however, Barbour holds a presence in more than 40 countries worldwide, including Austria, New Zealand, and Japan. Using individualized insights derived with the services and digital marketing capabilities of Teradata Interactive, Barbour ran a one-month campaign that generated 49,700 new leads and 450,000 clicks to its website.

The Challenge: Taking Ownership of Customer Relationships

Barbour has experienced outstanding consistent growth within its lifetime, and in August 2013, it launched its first e-commerce site in a bid to gain a stronger online presence. However, being a late starter in the e-commerce world, it was a challenge for Barbour to establish itself in the saturated digi- tal arena. Having previously sold its products only through wholesalers and independent retail resellers, Barbour wanted to take ownership of the end-user relationship, whole customer journey, and perception of the brand. While the brand is iconic and highly

(Continued )

440 Part II • Predictive Analytics/Machine Learning

sales over a sustained period of time. It was clear from the start that the strong relationship Barbour enjoys with its customers was a crucial factor that set it apart from its luxury retail competitors. Teradata Interactive was keen to ensure this relationship was respected through the lead generation process.

The execution of the campaign was unique to Barbour. Typical lead generation campaigns were often executed as single registration events with a single sales promotion in mind. The data were usu- ally restricted to just email addresses and basic profile fields, generated without consideration of the regis- trant’s personal needs and imported to be used solely for generic newsletter campaigns. This strategy often missed a huge opportunity for brands when learn- ing about their prospects, often resulting in poor sales conversions. Teradata Interactive understood that the true value of lead generation is twofold. First of all, by using the registration event to gather as much informa- tion as possible, the understanding of future buying intent and its affecting factors are developed. Second, by making sure that the collated data are effectively used to deliver valuable and individualized content, relevant sales opportunities are provided to the cus- tomer when they are next in the market to buy. To make sure this strategy drove long-term sales, Teradata Interactive built a customer lifecycle program which delivered content over email and online display.

The nurture program content was integrated with display advertising and encouraged social media sharing. With Teradata Interactive’s smart tag- ging of nurture content, Barbour was able to segment audiences according to their product preferences and launch display re-targeting banners. Registrants were also invited to share content socially, which enabled Teradata Interactive to identify “social propensity” and segment users for future loyalty schemes and “Tell-a-Friend” activities. In addition to the focus of increasing Barbour’s newsletter base, Teradata conducted a data audit to analyze all of the

data collected and better understand what factors would influence user engagement behavior.

Results

The strong collaboration between Teradata and Barbour meant that over the one-month campaign period, Barbour was able to create new and inno- vative ways of communicating with its custom- ers. More than 49,700 leads were collected within the UK and DACH regions, and the lead genera- tion program showed open rates of up to 60% and click-through-rates of between 4 and 11 percent. The campaign also generated 450,000+ clicks to Barbour’s website and was so popular with fashion bloggers and national press that it was featured as a story in The Daily Mirror. Though the campaign was only a month long, a key focus was to help Barbour’s future marketing strategy. A preference center survey was implemented into the campaign design, which resulted in a 65 percent incentivized completion rate. User data included:

• Social network engagement • Device engagement • Location to nearest store • Important considerations to the customer

A deep level of insight has effectively given Barbour a huge capability to deliver personalized content and offers to its user base.

Questions for Case 7.7

1. What does Barbour do? What was the challenge Barbour was facing?

2. What was the proposed analytics solution?

3. What were the results?

Source: Teradata Case Study, “How Barbour Collected More Than 49,000 New Leads in One Month with Teradata Interactive” http://assets.teradata.com/resourceCenter/downloads/ CaseStudies/EB-8791_Interactive-Case-Study_Barbour.pdf (accessed November 2018).

Application Case 7.7 (Continued)

u SECTION 7.8 REVIEW QUESTIONS

1. What is a search engine? Why are search engines critically important for today’s businesses?

2. What is a Web crawler? What is it used for? How does it work? 3. What is “search engine optimization”? Who benefits from it? 4. What things can help Web pages rank higher in search engine results?

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 441

7.9 WEB USAGE MINING (WEB ANALYTICS)

Web usage mining (also called Web analytics) is the extraction of useful information from data generated through Web page visits and transactions. Analysis of the information collected by Web servers can help us better understand user behavior. Analysis of these data is often called clickstream analysis. By using data and text mining techniques, a company might be able to discern interesting patterns from the clickstreams. For example, it might learn that 60 percent of visitors who searched for “hotels in Maui” had searched earlier for “airfares to Maui.” Such information could be useful in determining where to place online advertisements. Clickstream analysis might also be useful for knowing when visitors access a site. For example, if a company knew that 70 percent of software downloads from its Web site occurred between 7 and 11 p.m., it could plan for better customer support and network bandwidth during those hours. Figure 7.13 shows the process of extracting knowledge from clickstream data and how the generated knowledge is used to improve the process, improve the Web site, and most importantly, increase the customer value.

Web Analytics Technologies

There are numerous tools and technologies for Web analytics in the marketplace. Because of their power to measure, collect, and analyze Internet data to better understand and optimize Web usage, the popularity of Web analytics tools is increasing. Web analytics holds the promise of revolutionizing how business is done on the Web. Web analytics is not just a tool for measuring Web traffic; it can also be used as a tool for e-business and market research and to assess and improve the effectiveness of e-commerce Web sites. Web analytics applications can also help companies measure the results of traditional print or broadcast advertising campaigns. It can help estimate how traffic to a Web site changes after the launch of a new advertising campaign. Web analytics provides infor- mation about the number of visitors to a Web site and the number of page views. It helps gauge traffic and popularity trends, which can be used for market research.

There are two main categories of Web analytics: off-site and on-site. Off-site Web analytics refers to Web measurement and analysis about you and your products that take place outside your Web site. These measurements include a Web site’s potential audience (prospect or opportunity), share of voice (visibility or word of mouth), and buzz (com- ments or opinions) that is happening on the Internet.

What is more mainstream has been on-site Web analytics. Historically, Web analyt- ics has been referred to as on-site visitor measurement. However, in recent years, this has blurred, mainly because vendors are producing tools that span both categories. On-site Web

Web Logs

Web Site Preprocess Data Collecting Merging Cleaning Structuring - Identify users - Identify sessions - Identify page views - Identify visits

Extract Knowledge Usage patterns User profiles Page profiles Visit profiles Customer value

How to better the data

How to improve the Web Site

How to increase the customer value

User/ Customer

0% 18–24 25–34 35–44 45–54 55+

5% 10% 15% 20% 25% 30% 35% 40% 45%

P er

ce nt

o f

U se

rs

FIGURE 7.13 Extraction of Knowledge from Web Usage Data.

442 Part II • Predictive Analytics/Machine Learning

analytics measure visitors’ behavior once they are on your Web site. This includes its drivers and conversions—for example, the degree to which different landing pages are associated with online purchases. On-site Web analytics measure the performance of your Web site in a commercial context. The data collected on the Web site is then compared against key performance indicators for performance and used to improve audience response on a Web site or marketing campaign. Even though Google Analytics is the most widely used on-site Web analytics service, others are provided by Yahoo! and Microsoft, and newer and better tools are emerging constantly that provide additional layers of information.

There are two technical ways to collect the data with on-site Web analytics. The first and more traditional method is the server log file analysis by which the Web server records file requests made by browsers. The second method is page tagging, which uses JavaScript embedded in the site page code to make image requests to a third-party analytics–dedicated server whenever a page is rendered by a Web browser (or when a mouse click occurs). Both collect data that can be processed to produce Web traffic reports. In addition to these two main streams, other data sources can also be added to augment Web site behavior data. These other sources can include e-mail, direct mail cam- paign data, sales and lead history, or social media–originated data.

Web Analytics Metrics

Using a variety of data sources, Web analytics programs provide access to much valuable marketing data, which can be leveraged for better insights to grow your business and bet- ter document your return on investment (ROI). The insight and intelligence gained from Web analytics can be used to effectively manage the marketing efforts of an organization and its various products or services. Web analytics programs provide nearly real-time data, which can document an organization’s marketing campaign successes or empower it to make timely adjustments to its current marketing strategies.

Whereas Web analytics provides a broad range of metrics, four categories of metrics are generally actionable and can directly impact your business objectives (The Westover Group, 2013). These categories include:

• Web site usability: How were they using my Web site? • Traffic sources: Where did they come from? • Visitor profiles: What do my visitors look like? • Conversion statistics: What does it all mean for the business?

Web Site Usability

Beginning with your Web site, let’s take a look at how well it works for your visitors. This is where you can learn how “user friendly” it really is or whether or not you are providing the right content.

1. Page views. The most basic of measurements, this one is usually presented as the “average page views per visitor.” If people come to your Web site and do not view many pages, then your Web site could have issues with its design or structure. Another explanation for low page views is a disconnect in the marketing messages that brought the visitor to the site and the content that is actually available.

2. Time on site. Similar to page views, this is a fundamental measurement of a visitor’s interaction with your Web site. Generally, the longer a person spends on your Web site, the better it is. That could mean they are carefully reviewing your content, utilizing inter- active components you have available, and building toward an informed decision to buy, respond, or take the next step you have provided. On the contrary, the time on site also needs to be examined against the number of pages viewed to make sure the visitor is not spending his or her time trying to locate content that should be more readily accessible.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 443

3. Downloads. This includes PDFs, videos, and other resources you make avail- able to your visitors. Consider how accessible these items are as well as how well they are promoted. If your Web statistics, for example, reveal that 60 percent of the individuals who watch a demo video also make a purchase, then you will want to strategize to increase viewership of that video.

4. Click map. Most analytics programs can show you the percentage of clicks each item on your Web page received. This includes clickable photos, text links in your copy, downloads, and, of course, any navigation you have on the page. Are they clicking the most important items?

5. Click paths. Although an assessment of click paths is more involved, they can quickly reveal where you might be losing visitors in a specific process. A well- designed Web site uses a combination of graphics and information architecture to encourage visitors to follow “predefined” paths through your Web site. These are not rigid pathways but intuitive steps that align with the various processes you have built into the Web site. One process might be that of “educating” a visitor who has minimum understanding of your product or service. Another might be a process of “motivating” a returning visitor to consider an upgrade or repurchase. A third process might be structured around items you market online. You will have as many process pathways in your Web site as you have target audiences, prod- ucts, and services. Each can be measured through Web analytics to determine how effective it is.

Traffic Sources

Your Web analytics program is an incredible tool for identifying where your Web traf- fic originates. Basic categories such as search engines, referral Web sites, and visits from bookmarked pages (i.e., direct) are compiled with little involvement by the marketer. With a small amount of effort, however, you can also identify Web traffic that was gener- ated by your various offline or online advertising campaigns.

1. Referral Web sites. Other Web sites that contain links that send visitors directly to your Web site are considered referral Web sites. Your analytics program will identify each referral site your traffic comes from, and a deeper analysis will help you deter- mine which referrals produce the greatest volume, the highest conversions, the most new visitors, and so on.

2. Search engines. Data in the search engine category is divided between paid search and organic (or natural) search. You can review the top keywords that gener- ated Web traffic to your site and see if they are representative of your products and services. Depending upon your business, you might want to have hundreds (or thou- sands) of keywords that draw potential customers. Even the simplest product search can have multiple variations based on how the individual phrases the search query.

3. Direct. Direct searches are attributed to two sources. Individuals who bookmark one of your Web pages in their favorites and click that link will be recorded as a direct search. Another source occurs when someone types your URL directly into her or his browser. This happens when someone retrieves your URL from a busi- ness card, brochure, print ad, radio commercial, and so on. That’s why it is a good strategy to use coded URLs.

4. Offline campaigns. If you utilize advertising options other than Web-based cam- paigns, your Web analytics program can capture performance data if you include a mechanism for sending them to your Web site. Typically, this is a dedicated URL that you include in your advertisement (i.e., “www.mycompany.com/offer50”) that delivers those visitors to a specific landing page. You now have data on how many responded to that ad by visiting your Web site.

444 Part II • Predictive Analytics/Machine Learning

5. Online campaigns. If you are running a banner ad campaign, search engine ad- vertising campaign, or even e-mail campaign, you can measure individual campaign effectiveness by simply using a dedicated URL similar to the offline campaign strategy.

Visitor Profiles

One of the ways you can leverage your Web analytics into a really powerful marketing tool is through segmentation. By blending data from different analytics reports, you will begin to see a variety of user profiles emerge.

1. Keywords. Within your analytics report, you can see what keywords visitors used in search engines to locate your Web site. If you aggregate your keywords by similar attributes, you will begin to see distinct visitor groups who are using your Web site. For example, the particular search phrase that was used can indicate how well they understand your product or its benefits. If they use words that mirror your own product or service descriptions, then they probably are already aware of your offerings from effective advertisements, brochures, and so on. If the terms are more general in nature, then your visitors are seeking a solution for a problem that has happened upon your Web site. If this second group of searchers is sizable, then you will want to ensure that your site has a strong education component to convince them they have found their answer and then move them into your sales channel.

2. Content groupings. Depending on how you group your content, you could be able to analyze sections of your Web site that correspond to specific products, ser- vices, campaigns, and other marketing tactics. If you conduct a number of trade shows and drive traffic to your Web site for specific product literature, then your Web analytics will highlight the activity in that section.

3. Geography. Analytics permits you to see where your traffic geographically origi- nates, including country, state, and city locations. This can be especially useful if you use geotargeted campaigns or want to measure your visibility across a region.

4. Time of day. Web traffic generally has peaks at the beginning of the workday, during lunch, and toward the end of the workday. It is not unusual, however, to find strong Web traffic entering your Web site up until the late evening. You can analyze these data to determine when people browse versus buy and to make decisions on what hours you should offer customer service.

5. Landing page profiles. If you structure your various advertising campaigns properly, you can drive each of your targeted groups to a different landing page, which your Web analytics will capture and measure. By combining these numbers with the demographics of your campaign media, you can know what percentage of your visitors fits each demographic.

Conversion Statistics

Each organization defines a “conversion” according to its specific marketing objectives. Some Web analytics programs use the term goal to benchmark certain Web site objec- tives, whether that be a certain number of visitors to a page, a completed registration form, or an online purchase.

1. New visitors. If you are working to increase visibility, you will want to study the trends in your new visitors data. Analytics identifies all visitors as either new or returning.

2. Returning visitors. If you are involved in loyalty programs or offer a product that has a long purchase cycle, then your returning visitors data will help you mea- sure progress in this area.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 445

3. Leads. Once a form is submitted and a thank-you page is generated, you have created a lead. Web analytics will permit you to calculate a completion rate (or aban- donment rate) by dividing the number of completed forms by the number of Web visitors that came to your page. A low completion percentage would indicate a page that needs attention.

4. Sales/conversions. Depending on the intent of your Web site, you can define a “sale” by an online purchase, a completed registration, an online submission, or any number of other Web activities. Monitoring these figures will alert you to any changes (or successes!) that occur further upstream.

5. Abandonment/exit rates. Just as important as those moving through your Web site are those who began a process and quit or came to your Web site and left after a page or two. In the first case, you’ll want to analyze where the visitor terminated the process and whether there are a number of visitors quitting at the same place. Then investigate the situation for resolution. In the latter case, a high exit rate on a Web site or a specific page generally indicates an issue with expectations. Visitors click to your Web site based on some message contained in an advertisement, a pre- sentation, and so on, and expect some continuity in that message. Make sure you are advertising a message that your Web site can reinforce and deliver.

Within each of these items are metrics that can be established for your specific organization. You can create a weekly dashboard that includes specific numbers or per- centages that will indicate where you are succeeding—or highlight a marketing challenge that should be addressed. When these metrics are evaluated consistently and used in conjunction with other available marketing data, they can lead you to a highly quantified marketing program. Figure 7.14 shows a Web analytics dashboard created with freely available Google Analytics tools.

FIGURE 7.14 Sample Web Analytics Dashboard.

446 Part II • Predictive Analytics/Machine Learning

u SECTION 7.9 REVIEW QUESTIONS

1. What are the three types of data generated through Web page visits? 2. What is clickstream analysis? What is it used for? 3. What are the main applications of Web mining? 4. What are commonly used Web analytics metrics? What is the importance of metrics?

7.10 SOCIAL ANALYTICS

Social analytics could mean different things to different people based on their world- view and field of study. For instance, the dictionary definition of social analytics refers to a philosophical perspective developed by the Danish historian and philosopher Lars- Henrik Schmidt in the 1980s. The theoretical object of the perspective is socius, a kind of “commonness” that is neither a universal account nor a communality shared by every member of a body (Schmidt, 1996). Thus, social analytics differs from traditional philoso- phy as well as sociology. It might be viewed as a perspective that attempts to articulate the contentions between philosophy and sociology.

Our definition of social analytics is somewhat different; as opposed to focusing on the “social” part (as is done in its philosophical definition), we are more interested in the “analytics” part of the term. Gartner (a very well-known global IT consultancy company) defined social analytics as “monitoring, analyzing, measuring and interpret- ing digital interactions and relationships of people, topics, ideas and content” (gartner. com/it- glossary/social-analytics/). Social analytics include mining the textual content created in social media (e.g., sentiment analysis, NLP) and analyzing socially established networks (e.g., influencer identification, profiling, prediction) for the purpose of gaining insight about existing and potential customers’ current and future behaviors, and about the likes and dislikes toward a firm’s products and services. Based on this definition and the current practices, social analytics can be classified into two different, but not nec- essarily mutually exclusive, branches: social network analysis (SNA) and social media analytics.

Social Network Analysis

A social network is a social structure composed of individuals/people (or groups of individuals or organizations) linked to one another with some type of connections/rela- tionships. The social network perspective provides a holistic approach to analyzing the structure and dynamics of social entities. The study of these structures uses SNA to iden- tify local and global patterns, locate influential entities, and examine network dynamics. Social networks and their analysis is essentially an interdisciplinary field that emerged from social psychology, sociology, statistics, and graph theory. Development and for- malization of the mathematical extent of SNA dates back to the 1950s; the development of foundational theories and methods of social networks dates back to the 1980s (Scott & Davis, 2003). SNA is now one of the major paradigms in business analytics, consumer intelligence, and contemporary sociology and is employed in a number of other social and formal sciences.

A social network is a theoretical construct useful in the social sciences to study relationships between individuals, groups, organizations, or even entire societies (social units). The term is used to describe a social structure determined by such interactions. The ties through which any given social unit connects represent the convergence of the various social contacts of that unit. In general, social networks are self-organizing, emer- gent, and complex, such that a globally coherent pattern appears from the local interac- tion of the elements (individuals and groups of individuals) that make up the system.

Following are a few typical social network types that are relevant to business activities.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 447

COMMUNICATION NETWORKS Communication studies are often considered a part of both the social sciences and the humanities, drawing heavily on fields such as sociology, psychology, anthropology, information science, biology, political science, and econom- ics. Many communications concepts describe the transfer of information from one source to another and thus can be represented as a social network. Telecommunication compa- nies are tapping into this rich information source to optimize their business practices and to improve customer relationships.

COMMUNITY NETWORKS Traditionally, community has referred to a specific geographic location, and studies of community ties had to do with who talked, associated, traded, and attended social activities with whom. Today, however, there are extended “online” communities developed through social networking tools and telecommunications de- vices. Such tools and devices continuously generate large amounts of data that companies can use to discover invaluable, actionable information.

CRIMINAL NETWORKS In criminology and urban sociology, much attention has been paid to the social networks among criminal actors. For example, studying gang murders and other illegal activities as a series of exchanges between gangs can lead to better understanding and prevention of such criminal activities. Now that we live in a highly connected world (thanks to the Internet), much of the criminal networks’ formations and their activities are being watched/pursued by security agencies using state-of-the- art Internet tools and tactics. Even though the Internet has changed the landscape for criminal networks and law enforcement agencies, the traditional social and philosophical theories still apply to a large extent.

INNOVATION NETWORKS Business studies on the diffusion of ideas and innovations in a network environment focus on the spread and use of ideas among the members of the social network. The idea is to understand why some networks are more innovative, and why some communities are early adopters of ideas and innovations (i.e., examining the impact of social network structure on influencing the spread of an innovation and innovative behavior).

Social Network Analysis Metrics

SNA, the systematic examination of social networks, views social relationships in terms of network theory consisting of nodes (representing individuals or organizations within the network) and ties/connections (which represent relationships between the individuals or organizations, such as friendship, kinship, or organizational position). These networks are often represented using social network diagrams, where nodes are represented as points and ties are represented as lines.

Application Case 7.8 provides an interesting example of multichannel social analytics.

If Tito’s Handmade Vodka had to identify a single social media metric that most accurately reflects its mission, it would be engagement. Connecting with vodka lovers in an inclusive, authentic way is some- thing Tito’s takes very seriously, and the brand’s social strategy reflects that vision.

Founded nearly two decades ago, Tito’s credits the advent of social media with playing an integral role in engaging fans and raising brand awareness. In an interview with Entrepreneur, founder Bert “Tito” Beveridge credited social media for enabling Tito’s to compete for shelf space with more established liquor

Application Case 7.8 Tito’s Vodka Establishes Brand Loyalty with an Authentic Social Strategy

(Continued )

448 Part II • Predictive Analytics/Machine Learning

brands. “Social media is a great platform for a word- of-mouth brand, because it’s not just about who has the biggest megaphone,” Beveridge told Entrepreneur.

As Tito’s has matured, the social team has remained true to the brand’s founding values and actively uses Twitter and Instagram to have one-on-one conversations and connect with brand enthusiasts. “We never viewed social media as another way to adver- tise,” said Katy Gelhausen, Web & social media coordi- nator. “We’re on social so our customers can talk to us.”

To that end, Tito’s uses Sprout Social to under- stand the industry atmosphere, develop a consistent social brand, and create a dialogue with its audi- ence. As a result, Tito’s recently organically grew its Twitter and Instagram communities by 43.5 percent and 12.6 percent, respectively, within four months.

Informing a Seasonal, Integrated Marketing Strategy

Tito’s quarterly cocktail program is a key part of the brand’s integrated marketing strategy. Each quarter, a cocktail recipe is developed and distributed through Tito’s online and offline marketing initiatives.

It is important for Tito’s to ensure that the rec- ipe is aligned with the brand’s focus as well as the larger industry direction. Therefore, Gelhausen uses

Sprout’s Brand Keywords to monitor industry trends and cocktail flavor profiles. “Sprout has been a really important tool for social monitoring. The Inbox is a nice way to keep on top of hashtags and see general trends in one stream,” she said.

The information learned is presented to Tito’s in-house mixology team and used to ensure that the same quarterly recipe is communicated to the brand’s sales team and across marketing channels. “Whether you’re drinking Tito’s at a bar, buying it from a liquor store or following us on social media you’re getting the same quarterly cocktail,” said Gelhausen.

The program ensures that, at every consumer touch point, a person receives a consistent brand experience—and that consistency is vital. In fact, according to an Infosys study on the omnichan- nel shopping experience, 34 percent of consumers attribute cross-channel consistency as a reason they spend more on a brand. Meanwhile, 39 percent cite inconsistency as reason enough to spend less.

At Tito’s, gathering industry insights starts with social monitoring on Twitter and Instagram through Sprout. But the brand’s social strategy does not stop there. Staying true to its roots, Tito’s uses the plat- form on a daily basis to authentically connect with customers.

Application Case 7.8 (Continued)

Used with permission of Sprout Social, Inc.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 449

Sprout’s Smart Inbox displays Tito’s Twitter and Instagram accounts in a single, cohesive feed. This helps Gelhausen manage inbound messages and quickly identify which require a response.

“Sprout allows us to stay on top of the conver- sations we’re having with our followers. I love how you can easily interact with content from multiple accounts in one place,” she said.

Spreading the Word on Twitter

Tito’s approach to Twitter is simple: engage in per- sonal, one-on-one conversations with fans. Dialogue is a driving force for the brand, and over the course of four months, 88 percent of Tweets sent were replies to inbound messages.

Using Twitter as an open line of communication between Tito’s and its fans resulted in a 162.2 percent increase in engagement and a 43.5 percent gain in followers. Even more impressively, Tito’s ended the quarter with 538,306 organic impressions—an 81 per- cent rise. A similar strategy is applied to Instagram, which Tito’s uses to strengthen and foster a relation- ship with fans by publishing photos and videos of new recipe ideas, brand events, and initiatives.

Capturing the Party on Instagram

On Instagram, Tito’s primarily publishes lifestyle content and encourages followers to incorporate its brand in everyday occasions. Tito’s also uses the platform to promote its cause through marketing

efforts and to tell its brand story. The team finds value in Sprout’s Instagram Profiles Report, which helps them identify what media is receiving the most engagement, analyze audience demograph- ics and growth, dive more deeply into publish- ing patterns, and quantify outbound hashtag per- formance. “Given Instagram’s new personalized feed, it’s important that we pay attention to what really does resonate,” said Gelhausen.

Using the Instagram Profiles Report, Tito’s has been able to measure the impact of its Instagram marketing strategy and revise its approach accord- ingly. By utilizing the network as another way to engage with fans, the brand has steadily grown its organic audience. In four months, @TitosVodka saw a 12.6 percent rise in followers and a 37.1 percent increase in engagement. On average, each piece of published content gained 534 interactions, and men- tions of the brand’s hashtag, #titoshandmadevodka, grew by 33 percent.

Where to from Here?

Social is an ongoing investment in time and atten- tion. Tito’s will continue the momentum the brand experienced by segmenting each quarter into its own campaign. “We’re always getting smarter with our social strategies and making sure that what we’re posting is relevant and resonates,” said Gelhausen. Using social to connect with fans in a consistent, genuine, and memorable way will remain a cor- nerstone of the brand’s digital marketing efforts.

(Continued )

450 Part II • Predictive Analytics/Machine Learning

Over the years, various metrics (or measurements) have been developed to analyze social network structures from different perspectives. These metrics are often grouped into three categories: connections, distributions, and segmentation.

Connections

The connections category of metrics groups includes the following:

Homophily: The extent to which actors form ties with similar versus dissimilar oth- ers. Similarity can be defined by gender, race, age, occupation, educational achieve- ment, status, values, or any other salient characteristic.

Multiplexity: The number of content forms contained in a tie. For example, two people who are friends and also work together would have a multiplexity of two. Multiplexity has been associated with relationship strength.

Mutuality/reciprocity: The extent to which two actors reciprocate each other’s friendship or other interaction.

Network closure: A measure of the completeness of relational triads. An individu- al’s assumption of network closure (i.e., that their friends are also friends) is called transitivity. Transitivity is an outcome of the individual or situational trait of need for cognitive closure.

Propinquity: The tendency for actors to have more ties with geographically close others.

Distributions

The following relate to the distributions category:

Bridge: An individual whose weak ties fill a structural hole, providing the only link between two individuals or clusters. It also includes the shortest route when a lon- ger one is unfeasible due to a high risk of message distortion or delivery failure.

Centrality: A group of metrics that aims to quantify the importance or influence (in a variety of senses) of a particular node (or group) within a network. Examples of common methods of measuring centrality include betweenness centrality, closeness centrality, eigenvector centrality, alpha centrality, and degree centrality.

Density: The proportion of direct ties in a network relative to the total number possible. Distance: The minimum number of ties required to connect two particular actors. Structural holes: The absence of ties between two parts of a network. Finding and

exploiting a structural hole can give an entrepreneur a competitive advantage. This concept was developed by sociologist Ronald Burt and is sometimes referred to as an alternate conception of social capital.

Using Sprout’s suite of social media management tools, Tito’s will continue to foster a community of loyalists.

Some highlights of Tito’s success follow:

• A 162 percent increase in organic engagement on Twitter.

• An 81 percent increase in organic Twitter impressions.

• A 37 percent increase in engagement on Instagram.

Questions for Case 7.8

1. How can social media analytics be used in the consumer products industry?

2. What do you think are the key challenges, poten- tial solutions, and probable results in applying social media analytics in consumer products and services firms?

Source: SproutSocial Case Study, “Tito’s Vodka Establishes Brand Loyalty with an Authentic Social Strategy.” http://sproutsocial. com/insights/case-studies/titos/ (accessed July 2016). Used with permission.

Application Case 7.8 (Continued)

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 451

Tie strength: Defined by the linear combination of time, emotional intensity, inti- macy, and reciprocity (i.e., mutuality). Strong ties are associated with homophily, propinquity, and transitivity, whereas weak ties are associated with bridges.

Segmentation

This category includes following:

Cliques and social circles: Groups are identified as cliques if every individual is directly tied to every other individual or social circles if there is less stringency of direct contact, which is imprecise, or as structurally cohesive blocks if precision is wanted.

Clustering coefficient: A measure of the likelihood that two members of a node are associates. A higher clustering coefficient indicates a greater cliquishness.

Cohesion: The degree to which actors are connected directly to each other by cohe- sive bonds. Structural cohesion refers to the minimum number of members who, if removed from a group, would disconnect the group.

Social Media Analytics

Social media refers to the enabling technologies of social interactions among people in which they create, share, and exchange information, ideas, and opinions in virtual com- munities and networks. Social media is a group of Internet-based software applications that build on the ideological and technological foundations of Web 2.0 and that allows the creation and exchange of user-generated content (Kaplan & Haenlein, 2010). Social media depends on mobile and other Web-based technologies to create highly interactive platforms for individuals and communities to share, co-create, discuss, and modify user- generated content. It introduces substantial changes to communication among organiza- tions, communities, and individuals.

Since their emergence in the early 1990s, Web-based social media technologies have seen a significant improvement in both quality and quantity. These technologies take on many different forms, including online magazines, Internet forums, Web logs, so- cial blogs, microblogging, wikis, social networks, podcasts, pictures, videos, and product/ service evaluations/ratings. By applying a set of theories in the field of media research (social presence, media richness) and social processes (self-presentation, self-disclosure), Kaplan and Haenlein (2010) created a classification scheme with six different types of so- cial media: collaborative projects (e.g., Wikipedia), blogs and microblogs (e.g., Twitter), content communities (e.g., YouTube), social networking sites (e.g., Facebook), virtual game worlds (e.g., World of Warcraft), and virtual social worlds (e.g., Second Life).

Web-based social media are different from traditional/industrial media, such as newspapers, television, and film, because they are comparatively inexpensive and ac- cessible to enable anyone (even private individuals) to publish or access/consume in- formation. Industrial media generally require significant resources to publish information because in most cases, the articles (or books) go through many revisions before being published (as was the case in the publication of this very book). The following are some of the most prevailing characteristics that help differentiate between social and industrial media (Morgan, Jones, & Hodges, 2010):

Quality: In industrial publishing—mediated by a publisher—the typical range of qual- ity is substantially narrower than in niche, unmediated markets. The main challenge posed by content in social media sites is the fact that the distribution of quality has high variance from very high-quality items to low-quality, sometimes abusive, content.

Reach: Both industrial and social media technologies provide scale and are capable of reaching a global audience. Industrial media, however, typically use a centralized framework for organization, production, and dissemination, whereas social media

452 Part II • Predictive Analytics/Machine Learning

are by their very nature more decentralized, less hierarchical, and distinguished by multiple points of production and utility.

Frequency: Compared to industrial media, updating and reposting on social media platforms is easier, faster, and cheaper, and therefore practiced more frequently, resulting in fresher content.

Accessibility: The means of production for industrial media are typically govern- ment and/or corporate (privately owned) and are costly, whereas social media tools are generally available to the public at little or no cost.

Usability: Industrial media production typically requires specialized skills and train- ing. Conversely, most social media production requires only modest reinterpretation of existing skills; in theory, anyone with access can operate the means of social media production.

Immediacy: The time lag between communications produced by industrial media can be long (weeks, months, or even years) compared to social media (which can be capable of virtually instantaneous responses).

Updatability: Industrial media, once created, cannot be altered (once a magazine article is printed and distributed, changes cannot be made to that same article), whereas social media can be altered almost instantaneously by comments or editing.

How Do People Use Social Media?

Not only are the numbers on social networking sites growing, but so is the degree to which they are engaged with the channel. Brogan and Bastone (2011) presented research results that stratify users according to how actively they use social media and tracked the evolution of these user segments over time. They listed six different engagement levels (Figure 7.15).

According to the research results, the online user community has been steadily mi- grating upward on this engagement hierarchy. The most notable change is among Inactives. Of the online population, 44 percent fell into this category in 2008. Two years later, more than half of those Inactives had jumped into social media in some form or another. “Now roughly 82% of the adult population online is in one of the upper categories,” said Bastone. “Social media has truly reached a state of mass adoption” (Brogan and Bastone, 2011).

Social media analytics refers to the systematic and scientific ways to consume the vast amount of content created by Web-based social media outlets, tools, and techniques for the betterment of an organization’s competitiveness. Social media analytics is rapidly

Collectors

Joiners

Critics

Creators

Time

Le ve

l o f

S oc

ia l M

ed ia

E ng

ag em

en t

Spectators

Inactives

FIGURE 7.15 Evolution of Social Media User Engagement.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 453

becoming a new force in organizations around the world, allowing them to reach out to and understand consumers as never before. In many companies, it is becoming the tool for integrated marketing and communications strategies.

The exponential growth of social media outlets from blogs, Facebook, and Twitter to LinkedIn and YouTube and of analytics tools that tap into these rich data sources offer organizations the chance to join a conversation with millions of customers around the globe every day. This ability is why nearly two-thirds of the 2,100 companies who participated in a recent survey by Harvard Business Review Analytic Services said they were either currently using social media channels or had social media plans in the works (Harvard Business Review, 2010). But many still say social media is an experiment, as they try to understand how to best use the different channels, gauge their effectiveness, and integrate social media into their strategy.

Measuring the Social Media Impact

For organizations, small or large, there is valuable insight hidden in all the user-generated content on social media sites. But how do you dig it out of dozens of review sites, thou- sands of blogs, millions of Facebook posts, and billions of tweets? Once you do that, how do you measure the impact of your efforts? These questions can be addressed by the analytics extension of the social media technologies. Once you decide on your goal for social media (what it is that you want to accomplish), a multitude of tools can help you get there. These analysis tools usually fall into three broad categories:

• Descriptive analytics: Uses simple statistics to identify activity characteristics and trends, such as how many followers you have, how many reviews were gener- ated on Facebook, and which channels are being used most often.

• Social network analysis: Follows the links between friends, fans, and follow- ers to identify connections of influence as well as the biggest sources of influence.

• Advanced analytics: Includes predictive analytics and text analytics that exam- ine the content in online conversations to identify themes, sentiments, and connec- tions that would not be revealed by casual surveillance.

Sophisticated tools and solutions to social media analytics use all three categories of analytics (i.e., descriptive, predictive, and prescriptive) in a somewhat progressive fashion.

Best Practices in Social Media Analytics

As an emerging tool, social media analytics is practiced by companies in a somewhat haphazard fashion. Because there are not well-established methodologies, everybody is trying to create their own by trial and error. What follows are some of the best field-tested practices for social media analytics proposed by Paine and Chaves (2012).

THINK OF MEASUREMENT AS A GUIDANCE SYSTEM, NOT A RATING SYSTEM Measure- ments are often used for punishment or rewards; they should not be. They should be about figuring out what the most effective tools and practices are, what needs to be dis- continued because it does not work, and what needs to be done more because it does work very well. A good analytics system should tell you where you need to focus. Maybe all that emphasis on Facebook does not really matter because that is not where your audience is. Maybe they are all on Twitter, or vice versa. According to Paine and Chaves (2012), channel preference will not necessarily be intuitive: “We just worked with a hotel that had virtually no activity on Twitter for one brand but lots of Twitter activity for one of their higher brands.” Without an accurate measurement tool, you would not know.

TRACK THE ELUSIVE SENTIMENT Customers want to take what they are hearing and learn- ing from online conversations and act on it. The key is to be precise in extracting and tag- ging their intentions by measuring their sentiments. As we saw earlier in this chapter, text

454 Part II • Predictive Analytics/Machine Learning

analytic tools can categorize online content, uncover linked concepts, and reveal the senti- ment in a conversation as “positive,” “negative,” or “neutral,” based on the words people use. Ideally, you would like to be able to attribute sentiment to a specific product, service, and business unit. The more precise you can be in understanding the tone and percep- tion that people express, the more actionable the information becomes because you are mitigating concerns about mixed polarity. A mixed-polarity phrase, such as “hotel in great location but bathroom was smelly,” should not be tagged as “neutral” because you have positives and negatives offsetting each other. To be actionable, these types of phrases are to be treated separately; “bathroom was smelly” is something someone can own and im- prove on. One can classify and categorize these sentiments, look at trends over time, and see significant differences in the way people speak either positively or negatively about you. Furthermore, you can compare sentiment about your brand to your competitors.

CONTINUOUSLY IMPROVE THE ACCURACY OF TEXT ANALYSIS An industry-specific text analytics package will already know the vocabulary of your business. The system will have linguistic rules built into it, but it learns over time and gets better and better. Much as you would tune a statistical model as you have more data, better parameters, or new techniques to deliver better results, you would do the same thing with the NLP that goes into sentiment analysis. You set up rules, taxonomies, categorization, and meaning of words; watch what the results look like and then go back and do it again.

LOOK AT THE RIPPLE EFFECT It is one thing to be a great hit on a high-profile site, but that is only the start. There is a difference between a great hit that just sits there and goes away versus a great hit that is tweeted, retweeted, and picked up by influential bloggers. Analysis should show you which social media activities go “viral” and which quickly go dormant—and why.

LOOK BEYOND THE BRAND One of the biggest mistakes people make is to be concerned only about their brand. To successfully analyze and act on social media, people need to understand not just what is being said about their brand but also the broader conversa- tion about the spectrum of issues surrounding their product or service, as well. Customers do not usually care about a firm’s message or its brand; they care about themselves. Therefore, you should pay attention to what they are talking about, where they are talk- ing, and where their interests are.

IDENTIFY YOUR MOST POWERFUL INFLUENCERS Organizations struggle to identify who has the most power in shaping public opinion. It turns out, their most important influ- encers are not necessarily the ones who advocate specifically for their brand; they are the ones who influence the whole realm of conversation about their topic. Organizations need to understand whether influencers are saying nice things, expressing support, or simply making observations or critiquing. What is the nature of their conversations? How is the organization’s brand being positioned relative to the competition in that space?

LOOK CLOSELY AT THE ACCURACY OF ANALYTIC TOOLS USED Until recently, computer- based automated tools were not as accurate as humans for sifting through online content. Even now, accuracy varies depending on the media. For product review sites, hotel review sites, and Twitter, the accuracy can reach anywhere between 80 and 90 percent because the context is more boxed in. When an organization starts looking at blogs and discussion fo- rums where the conversation is more wide ranging, the software can deliver 60 to 70 percent accuracy (Paine & Chaves, 2012). These figures will increase over time because the analyt- ics tools are continually upgraded with new rules and improved algorithms to reflect field experience, new products, changing market conditions, and emerging patterns of speech.

INCORPORATE SOCIAL MEDIA INTELLIGENCE INTO PLANNING Once an organization has a big-picture perspective and detailed insight, it can begin to incorporate this information into

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 455

its planning cycle. But that is easier said than done. A quick audience poll revealed that very few people currently incorporate learning from online conversations into their planning cycles (Paine & Chaves, 2012). One way to achieve this is to find time-linked associations between social media metrics and other business activities or market events. Social media is typically either organically invoked or invoked by something an organization does; therefore, if it sees a spike in activity at some point in time, it wants to know what was behind that.

u SECTION 7.10 REVIEW QUESTIONS

1. What is meant by social analytics? Why is it an important business topic? 2. What is a social network? What is the need for SNA? 3. What is social media? How does it relate to Web 2.0? 4. What is social media analytics? What are the reasons behind its increasing popularity? 5. How can you measure the impact of social media analytics?

Chapter Highlights

• Text mining is the discovery of knowledge from unstructured (mostly text-based) data sources. Because a great deal of information is in text form, text mining is one of the fastest-growing branches of the business intelligence field.

• Text mining applications are in virtually every area of business and government, including mar- keting, finance, healthcare, medicine, and home- land security.

• Text mining uses NLP to induce structure into the text collection and then uses data mining algorithms such as classification, clustering, association, and se- quence discovery to extract knowledge from it.

• Sentiment can be defined as a settled opinion re- flective of one’s feelings.

• Sentiment analysis deals with differentiating be- tween two classes, positive and negative.

• As a field of research, sentiment analysis is closely related to computational linguistics, NLP, and text mining.

• Sentiment analysis is trying to answer the ques- tion, “What do people feel about a certain topic?” by digging into opinions of many by using a vari- ety of automated tools.

• VOC is an integral part of an analytic CRM and customer experience management systems and is often powered by sentiment analysis.

• VOM is about understanding aggregate opinions and trends at the market level.

• Polarity identification in sentiment analysis is ac- complished either by using a lexicon as a refer- ence library or by using a collection of training documents.

• WordNet is a popular general-purpose lexicon created at Princeton University.

• SentiWordNet is an extension of WordNet to be used for sentiment identification.

• Speech analytics is a growing field of science that allows users to analyze and extract information from both live and recorded conversations.

• Web mining can be defined as the discovery and analysis of interesting and useful information from the Web, about the Web, and usually using Web-based tools.

• Web mining can be viewed as consisting of three areas: content mining, structure mining, and usage mining.

• Web content mining refers to the automatic ex- traction of useful information from Web pages. It can be used to enhance search results produced by search engines.

• Web structure mining refers to generating inter- esting information from the links on Web pages.

• Web structure mining can also be used to identify the members of a specific community and perhaps even the roles of the members in the community.

• Web usage mining refers to developing useful information through analyzing Web server logs, user profiles, and transaction information.

• Text and Web mining are emerging as critical components of the next generation of business intelligence tools to enable organizations to com- pete successfully.

• A search engine is a software program that searches for documents (Internet sites or files) based on the keywords (individual words, multi- word terms, or a complete sentence) users have provided that relate to the subject of their inquiry.

• SEO is the intentional activity of affecting the visibility of an e-commerce site or a Web site

456 Part II • Predictive Analytics/Machine Learning

in a search engine’s natural (unpaid or organic) search results.

• VOC is a term generally used to describe the ana- lytic process of capturing a customer’s expecta- tions, preferences, and aversions.

• Social analytics is the monitoring, analyzing, mea- suring, and interpreting of digital interactions and relationships of people, topics, ideas, and content.

• A social network is a social structure composed of individuals/people (or groups of individuals or organizations) linked to one another with some type of connections/relationships.

• Social media analytics refers to the systematic and scientific ways to consume the vast amount of content created by Web-based social media out- lets, tools, and techniques to better an organiza- tion’s competitiveness.

Key Terms

association authoritative pages classification clickstream analysis clustering corpus deception detection hubs hyperlink-induced topic search

(HITS) natural language processing (NLP) part-of-speech tagging

polarity identification polyseme search engine sentiment analysis SentiWordNet singular value decomposition (SVD) social media analytics social network spider stemming stop words term–document matrix (TDM)

text mining tokenizing trend analysis unstructured data voice of the customer (VOC) Web analytics Web content mining Web crawler Web mining Web structure mining Web usage mining WordNet

Questions for Discussion

1. Explain the relationship among data mining, text min- ing, and sentiment analysis.

2. In your own words, define text mining, and discuss its most popular applications.

3. What does it mean to induce structure into text-based data? Discuss the alternative ways of inducing structure into them.

4. What is the role of NLP in text mining? Discuss the capa- bilities and limitations of NLP in the context of text mining.

5. List and discuss three prominent application areas for text mining. What is the common theme among the three application areas you chose?

6. What is sentiment analysis? How does it relate to text mining?

7. What are the common challenges with which sentiment analysis deals?

8. What are the most popular application areas for senti- ment analysis? Why?

9. What are the main steps in carrying out sentiment analy- sis projects?

10. What are the two common methods for polarity identi- fication? Explain.

11. Discuss the differences and commonalities between text mining and Web mining.

12. In your own words, define Web mining, and discuss its importance.

13. What are the three main areas of Web mining? Discuss the differences and commonalities among these three areas.

14. What is a search engine? Why is it important for businesses? 15. What is SEO? Who benefits from it? How? 16. What is Web analytics? What are the metrics used in

Web analytics? 17. Define social analytics, social network, and social net-

work analysis. What are the relationships among them? 18. What is social media analytics? How is it done? Who

does it? What comes out of it?

Exercises

Teradata University Network (TUN) and Other Hands-on Exercises

1. Visit teradatauniversitynetwork.com. Identify cases about text mining. Describe recent developments in the field. If you cannot find enough cases at the Teradata

University Network Web site, broaden your search to other Web-based resources.

2. Go to teradatauniversitynetwork.com to locate white papers, Web seminars, and other materials related to text mining. Synthesize your findings into a short written report.

Chapter 7 • Text Mining, Sentiment Analysis, and Social Analytics 457

3. Go to teradatauniversitynetwork.com and find the case study named “eBay Analytics.” Read the case carefully and extend your understanding of it by searching the Internet for additional information, and answer the case questions.

4. Go to teradatauniversitynetwork.com and find the sentiment analysis case named “How Do We Fix an App Like That?” Read the description, and follow the direc- tions to download the data and the tool to carry out the exercise.

5. Visit teradatauniversitynetwork.com. Identify cases about Web mining. Describe recent developments in the field. If you cannot find enough cases at the Teradata University Network Web site, broaden your search to other Web-based resources.

6. Browse the Web and your library’s digital databases to identify articles that make the linkage between text/Web mining and contemporary business intelligence systems.

Team Assignments and Role-Playing Projects

1. Examine how textual data can be captured automati- cally using Web-based technologies. Once captured, what are the potential patterns that you can extract from these unstructured data sources?

2. Interview administrators at your college or executives in your organization to determine how text mining and Web mining could assist them in their work. Write a proposal describing your findings. Include a preliminary cost–benefit analysis in your report.

3. Go to your library’s online resources. Learn how to download attributes of a collection of literature (journal articles) in a specific topic. Download and process the data using a methodology similar to the one explained in Application Case 7.5.

4. Find a readily available sentiment text data set (see Technology Insights 7.2 for a list of popular data sets) and download it onto your computer. If you have an analytics tool that is capable of text mining, use that. If not, download RapidMiner (http://rapid-i.com) and install it. Also install the Text Analytics add-on for RapidMiner. Process the downloaded data using your text mining tool (i.e., convert the data into a structured form). Build models and assess the sentiment detection accuracy of several classification models (e.g., support

vector machines, decision trees, neural networks, logis- tic regression). Write a detailed report in which you explain your findings and your experiences.

5. Examine how Web-based data can be captured auto- matically using the latest technologies. Once captured, what are the potential patterns that you can extract from these content-rich, mostly unstructured data sources?

Internet Exercises

1. Find recent cases of successful text mining and Web mining applications. Try text and Web mining software vendors and consultancy firms and look for cases or success stories. Prepare a report summarizing five new case studies.

2. Go to statsoft.com. Select Downloads, and download at least three white papers on applications. Which of these applications might have used the data/text/Web mining techniques discussed in this chapter?

3. Go to sas.com. Download at least three white papers on applications. Which of these applications might have used the data/text/Web mining techniques discussed in this chapter?

4. Go to ibm.com. Download at least three white papers on applications. Which of these applications might have used the data/text/Web mining techniques discussed in this chapter?

5. Go to teradata.com. Download at least three white papers on applications. Which of these applications might have used the data/text/Web mining techniques discussed in this chapter?

6. Go to clarabridge.com. Download at least three white papers on applications. Which of these applications might have used text mining in a creative way?

7. Go to kdnuggets.com. Explore the sections on appli- cations as well as software. Find names of at least three additional packages for data mining and text mining.

8. Survey some Web mining tools and vendors. Identify some Web mining products and service providers that are not mentioned in this chapter.

9. Go to attensity.com. Download at least three white papers on Web analytics applications. Which of these applications might have used a combination of data/ text/Web mining techniques?

References

Bond, C. F., & B. M. DePaulo. (2006). “Accuracy of Deception Judgments.” Personality and Social Psychology Reports, 10(3), pp. 214–234.

Brogan, C., & J. Bastone. (2011). “Acting on Customer Intelligence from Social Media: The New Edge for Building Customer Loyalty and Your Brand.” SAS white paper.

Chun, H. W., Y. Tsuruoka, J. D. Kim, R. Shiba, N. Nagata, & T. Hishiki. (2006). “Extraction of Gene-Disease Relations from MEDLINE Using Domain Dictionaries and Machine Learning.” Proceedings of the Eleventh Pacific Symposium on Biocomputing, pp. 4–15.

Coussement, K., & D. Van Den Poel. (2008). “Improving Customer Complaint Management by Automatic Email Classification Using Linguistic Style Features as Predictors.” Decision Support Systems, 44(4), pp. 870–882.

Coussement, K., & D. Van Den Poel. (2009). “Improving Customer Attrition Prediction by Integrating Emotions from Client/ Company Interaction Emails and Evaluating Multiple Classi- fiers.” Expert Systems with Applications, 36(3), pp. 6127–6134.

Cutts, M. (2006, February 4). “Ramping Up on International Web- spam.” mattcutts.com/blog. mattcutts.com/blog/ ramping- up-on-international-webspam (accessed March 2013).

458 Part II • Predictive Analytics/Machine Learning

Delen, D., & M. Crossland. (2008). “Seeding the Survey and Analysis of Research Literature with Text Mining.” Expert Systems with Applications, 34(3), pp. 1707–1720.

Esuli, A., & F. Sebastiani. (2006, May). SentiWordNet: A Pub- licly Available Lexical Resource for Opinion Mining. Pro- ceedings of LREC, 6, pp. 417–422.

Etzioni, O. (1996). “The World Wide Web: Quagmire or Gold Mine?” Communications of the ACM, 39(11), pp. 65–68.

EUROPOL. (2007). EUROPOL Work Program 2005. statewatch.org/news/2006/apr/europol-work- programme-2005.pdf (accessed October 2008).

Feldman, R., & J. Sanger. (2007). The Text Mining Handbook: Advanced Approaches in Analyzing Unstructured Data. Boston, MA: ABS Ventures.

Fuller, C. M., D. Biros, and D. Delen. (2008). “Exploration of Feature Selection and Advanced Classification Models for High-Stakes Deception Detection.” Proceedings of the Forty-First Annual Hawaii International Conference on Sys- tem Sciences (HICSS). Big Island, HI: IEEE Press, pp. 80–99.

Ghani, R., K. Probst, Y. Liu, M. Krema, and A. Fano. (2006). “Text Mining for Product Attribute Extraction.” SIGKDD Ex- plorations, 8(1), pp. 41–48.

Goodman, A. (2005). “Search Engine Showdown: Black Hats Versus White Hats at SES. SearchEngineWatch.” searchenginewatch.com/article/2066090/Search- Engine-Showdown-Black-Hats-vs.-White-Hats-at-SES (accessed February 2013).

Han, J., & M. Kamber. (2006). Data Mining: Concepts and Techniques, 2nd ed. San Francisco, CA: Morgan Kaufmann.

Harvard Business Review. (2010). “The New Conversation: Taking Social Media from Talk to Action.” A SAS–Sponsored Research Report by Harvard Business Review Analytic Ser- vices. sas.com/resources/whitepaper/wp_23348.pdf (accessed March 2013).

Kaplan, A. M., & M. Haenlein. (2010). “Users of the World, Unite! The Challenges and Opportunities of Social Media.” Business Horizons, 53(1), pp. 59–68.

Kim, S. M., & E. Hovy. (2004, August). “Determining the Senti- ment of Opinions.” Proceedings of the Twentieth Interna- tional Conference on Computational Linguistics, p. 1367.

Kleinberg, J. (1999). “Authoritative Sources in a Hyperlinked Environment.” Journal of the ACM, 46(5), pp. 604–632.

Lin, J., & D. Demner-Fushman. (2005). “Bag of Words” Is Not Enough for Strength of Evidence Classification.” AMIA Annual Symposium Proceedings, pp. 1031–1032. pubmedcentral. nih.gov/articlerender.fcgi?artid=1560897.

Liu, B., M. Hu, & J. Cheng. (2005, May). “Opinion Observer: Analyzing and Comparing Opinions on the Web.” Proceed- ings of the Fourth International Conference on World Wide Web, pp. 342–351.

Mahgoub, H., D. Rösner, N. Ismail, and F. Torkey. (2008). “A Text Mining Technique Using Association Rules Extrac- tion.” International Journal of Computational Intelligence, 4(1), pp. 21–28.

Manning, C. D., & H. Schutze. (1999). Foundations of Statis- tical Natural Language Processing. Cambridge, MA: MIT Press.

McKnight, W. (2005, January 1). “Text Data Mining in Business Intelligence.” Information Management Magazine. information-management.com/issues/20050101/ 1016487-1.html (accessed May 22, 2009).

Mejova, Y. (2009). “Sentiment Analysis: An Overview.” Comprehensive exam paper. http://www.cs.uiowa. edu/~ymejova/publications/CompsYelenaMejova. pdf (accessed February 2013).

Miller, T. W. (2005). Data and Text Mining: A Business: Ap- plications Approach. Upper Saddle River, NJ: Prentice Hall.

Morgan, N., G. Jones, & A. Hodges. (2010). “The Complete Guide to Social Media from the Social Media Guys.” thesocialmediaguys.co.uk/wp-content/uploads/ downloads/2011/03/CompleteGuidetoSocialMedia. pdf (accessed February 2013).

Nakov, P., A. Schwartz, B. Wolf, and M. A. Hearst. (2005). “Supporting Annotation Layers for Natural Language Pro- cessing.” Proceedings of the ACL, Interactive Poster and Demonstration Sessions. Ann Arbor, MI: Association for Computational Linguistics, pp. 65–68.

Paine, K. D., & M. Chaves. (2012). “Social Media Metrics.” SAS white paper. sas.com/resources/whitepaper/ wp_19861.pdf (accessed February 2013).

Pang, B., & L. Lee. (2008). OPINION Mining and Sentiment Analysis. Hanover, MA: Now Publishers; available at http://books.google.com.

Ramage, D., D. Hall, R. Nallapati, & C. D. Manning. (2009, Au- gust). “Labeled LDA: A Supervised Topic Model for Credit Attribution in Multi-Labeled Corpora.” Proceedings of the 2009 Conference on Empirical Methods in Natural Lan- guage Processing: Volume 1, pp. 248–256.

Schmidt, L.-H. (1996). “Commonness Across Cultures.” In A. N. Balslev (ed.), Cross-Cultural Conversation: Initiation (pp. 119–132). New York: Oxford University Press.

Scott, W. R., & G. F. Davis. (2003). “Networks in and Around Organizations.” Organizations and Organizing. Upper Saddle River: NJ: Pearson Prentice Hall.

Shatkay, H., A. Höglund, S. Brady, T. Blum, P. Dönnes, and O. Kohlbacher. (2007). “SherLoc: High-Accuracy Prediction of Protein Subcellular Localization by Integrating Text and Pro- tein Sequence Data.” Bioinformatics, 23(11), pp. 1410–1415.

Snyder, B., & R. Barzilay. (2007, April). “Multiple Aspect Rank- ing Using the Good Grief Algorithm.” HLT-NAACL, pp. 300–307.

Strapparava, C., & A. Valitutti. (2004, May). “WordNet Affect: An Affective Extension of WordNet.” LREC, 4, pp. 1083–1086.

The Westover Group. (2013). “20 Key Web Analytics Metrics and How to Use Them.” http://www.thewestovergroup. com (accessed February 2013).

Thomas, M., B. Pang, & L. Lee. (2006, July). “Get Out the Vote: Determining Support or Opposition from Congressional Floor-Debate Transcripts.” In Proceedings of the 2006 Con- ference on Empirical Methods in Natural Language Pro- cessing, pp. 327–335.

Weng, S. S., & C. K. Liu. (2004). “Using Text Classification and Multiple Concepts to Answer E-Mails.” Expert Systems with Applications, 26(4), pp. 529–543.

459

P A R T

Prescriptive Analytics and Big Data

III

460

Prescriptive Analytics: Optimization and Simulation

LEARNING OBJECTIVES

■■ Understand the applications of prescriptive analytics techniques in combination with reporting and predictive analytics

■■ Understand the basic concepts of analytical decision modeling

■■ Understand the concepts of analytical models for selected decision problems, including linear pro- gramming and simulation models for decision support

■■ Describe how spreadsheets can be used for analytical modeling and solutions

■■ Explain the basic concepts of optimization and when to use them

■■ Describe how to structure a linear programming model

■■ Explain what is meant by sensitivity analysis, what-if analysis, and goal seeking

■■ Understand the concepts and applications of different types of simulation

■■ Understand potential applications of discrete event simulation

T his chapter extends the analytics applications beyond reporting and predictive analytics. It includes coverage of selected techniques that can be employed in combination with predictive models to help support decision making. We focus on techniques that can be implemented relatively easily using either spreadsheet tools or by using stand-alone software tools. Of course, there is much additional detail to be learned about management science models, but the objective of this chapter is to simply illustrate what is possible and how it has been implemented in real settings.

We present this material with a note of caution: Modeling can be a difficult topic and is as much an art as it is a science. The purpose of this chapter is not necessarily for you to master the topics of modeling and analysis. Rather, the material is geared toward gaining familiarity with the important concepts as they relate to prescriptive analytics and their use in decision making. It is important to recognize that the mod- eling we discuss here is only cursorily related to the concepts of data modeling. You should not confuse the two. We walk through some basic concepts and definitions of decision modeling. We next introduce the idea of modeling directly in spreadsheets. We then discuss the structure and application of two successful time-proven models

8 C H A P T E R

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 461

and methodologies: linear programming and discrete event simulation. As noted earlier, one could take multiple courses just in these two topics, but our goal is to give you a sense of what is possible. This chapter includes the following sections:

8.1 Opening Vignette: School District of Philadelphia Uses Prescriptive Analytics to Find Optimal Solution for Awarding Bus Route Contracts 461

8.2 Model-Based Decision Making 462 8.3 Structure of Mathematical Models for Decision Support 469 8.4 Certainty, Uncertainty, and Risk 471 8.5 Decision Modeling with Spreadsheets 473 8.6 Mathematical Programming Optimization 477 8.7 Multiple Goals, Sensitivity Analysis, What-If Analysis, and Goal Seeking 486 8.8 Decision Analysis with Decision Tables and Decision Trees 490 8.9 Introduction to Simulation 493

8.10 Visual Interactive Simulation 500

8.1 OPENING VIGNETTE: School District of Philadelphia Uses Prescriptive Analytics to Find Optimal Solution for Awarding Bus Route Contracts

BACKGROUND

Selecting the best vendors to work with is a laborious yet important task for companies and government organizations. After a vendor submits a proposal for a specific task through a bidding process, the company or organization evaluates the proposal and makes a decision on which vendor is best suited for their needs. Typically, governments are required to use a bidding process to select one or more vendors. The School District of Philadelphia was in search of private bus vendors to outsource some of their bus routes. The district owned a few school buses, but needed more to serve their student population. They wanted to use their own school buses for 30 to 40% of the routes, and outsource the rest of the routes to these private vendors. Charles Lowitz, the fiscal coor- dinator for the transportation office, was tasked with determining how to maximize the return on investment and refine the way routes were awarded to various vendors.

Historically, the process of deciding which bus vendor contracts to award given the budget and time constraints was laborious as it was done manually by hand. In addition, the different variables and factors that had to be taken into account added to the com- plexity. The vendors were evaluated based on five variables: cost, capabilities, reliance, financial stability, and business acumen. Each vendor submitted a proposal with a differ- ent price for different routes. Some vendors specified a minimum number of routes, and if that minimum wasn’t met, their cost would increase. Lowitz needed to figure out how to combine the information from each proposal to determine which bus route to award to which vendor to meet all the route requirements at the least cost for the district.

SOLUTION

Lowitz initially looked for software that he could use in conjunction with his contract model in Excel. He began using the Premium Solver Platform from Frontline Systems, Inc., which allowed him to find the most beneficial vendors for the district from a financial and operational standpoint. He created an optimization model that took into account the aforementioned variables associated with each vendor. The model included binary integer variables (yes/no) for each of the routes to be awarded to the bidders

462 Part III • Prescriptive Analytics and Big Data

who proposed to serve a specific route at a specific cost. This amounted to about 1,600 yes/no variables. The model also included constraints indicating that each route was to be awarded to one vendor, and of course, each route had to be serviced. Other con- straints specified the minimum number of routes a vendor would accept and a few other details. All such constraints can be written as equations and entered in an integer linear programming model. Such models can be formulated and solved through many soft- ware tools, but using Microsoft Excel makes it easier to understand the model. Frontline Systems’ Solver software is built into Microsoft Excel to solve smaller problems for free. A larger version can be purchased to solve larger and more complex models. That is what Lowitz used.

BENEFITS

In addition to determining how many of the vendors should be awarded contracts, the model helped develop the size of each of the contracts. The size of the contracts var- ied from one vendor getting four routes to another receiving 97 routes. Ultimately, the School District of Philadelphia was able to create a plan with an optimized number of bus company vendors using Excel instead of a manual handwritten process. By using the Premium Solver Platform analytic tools to create an optimization model with the different variables, the district saved both time and money.

u QUESTIONS FOR THE OPENING VIGNETTE

1. What decision was being made in this vignette? 2. What data (descriptive and or predictive) might one need to make the best

allocations in this scenario?

3. What other costs or constraints might you have to consider in awarding contracts for such routes?

4. Which other situations might be appropriate for applications of such models?

WHAT CAN WE LEARN FROM THIS OPENING VIGNETTE?

Most organizations face the problem of making decisions where one has to select from multiple options. Each option has a cost and capability associated with it. The goal of such models is to select the combination of options that meet all the requirements and yet optimizes the costs. Prescriptive analytics particularly apply to the problem of such decisions. And tools such as built-in or Premium Solver for Excel make it easy to apply such techniques.

Source: Based on “Optimizing Vendor Contract Awards Gets an A+ ,” http://www.solver.com/news/ optimizing-vendor-contract-awards-gets, 2016 (accessed Sept 2018).

8.2 MODEL-BASED DECISION MAKING

As the preceding vignette indicates, making decisions using some kind of analytical model is what we call prescriptive analytics. In the last several chapters we have learned the value and the process of knowing the history of what has been going on and use that information to also predict what is likely to happen. However, we go through that exer- cise to determine what we should do next. This might entail deciding which customers are likely to buy from us and making an offer or giving a price point that will maximize the likelihood that they would buy and our profit would be optimized. Conversely, it might involve being able to predict which customer is likely to go somewhere else and

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 463

making a promotion offer to retain them as a customer and optimize our value. We may need to make decisions on awarding contracts to our vendors to make sure all our needs are covered and the costs are minimized. We could be facing a situation of deciding which prospective customers should receive what promotional campaign material so that our cost of promotion is not outrageous, and we maximize the response rate while man- aging within a budget. We may be deciding how much to pay for different paid search keywords to maximize the return on investment of our advertising budget. In another setting, we may have to study the history of our customers’ arrival patterns and use that information to predict future arrival rates, and apply that to schedule an appropriate num- ber of store employees to maximize customer responses and optimize our labor costs. We could be deciding where to locate our warehouses based on our analysis and prediction of demand for our products and the supply chain costs. We could be setting daily deliv- ery routes on the basis of product volumes to be delivered at various locations and the delivery costs and vehicle availability. One can find hundreds of examples of situations where data-based decisions are valuable. Indeed, the biggest opportunity for the grow- ing analytics profession is the ability to use descriptive and predictive insights to help a decision maker make better decisions. Although there are situations where one can use experience and intuition to make decisions, it is more likely that a decision supported by a model will help a decision maker make better decisions. In addition, it also provides decision makers with justification for what they are recommending. Thus prescriptive analytics has emerged as the next frontier for analytics. It essentially involves using an analytical model to help guide a decision maker in making a decision, or automating the decision process so that a model can make recommendations or decisions. Because the focus of prescriptive analytics is on making recommendations or making decisions, some call this category of analytics decision analytics.

INFORMS publications, such as Interfaces, ORMS Today, and Analytics magazine, all include stories that illustrate successful applications of decision models in real settings. This chapter includes many examples of such prescriptive analytic applications. Applying models to real-world situations can save millions of dollars or generate millions of dollars in revenue. Christiansen et al. (2009) describe the applications of such models in shipping company operations using TurboRouter, a decision support system (DSS) for ship routing and scheduling. They claim that over the course of a 3-week period, a company used this model to better utilize its fleet, generating additional profit of $1-2 million in such a short time. We provide another example of a model application in Application Case 8.1 that illustrates a sports application.

Canadian Football League (CFL) is Canada’s equiv- alent of the U.S. National Football League (NFL). It had a challenge of organizing 81 football games for 9 teams over a period of 5 months optimally while stabilizing matching priorities for sales revenue, television ratings, and the team rest days. Other considerations include organizing games over dif- ferent time zones and the main rivalry games to be

held on major public holidays. For any league, a robust schedule is a driving force for a variety of business collaborations, such as coordinating with broadcasting channels and organizing ground ticket sales. If the schedule is not optimized, it would directly hamper the promotions thus resulting in a huge loss of revenue and bad channel ratings. CFL used to create match schedules manually and

Application Case 8.1 Canadian Football League Optimizes Game Schedule

(Continued )

464 Part III • Prescriptive Analytics and Big Data

hence had to figure out finer ways to improve their schedules, taking all the constraints into account. They had tried to work with a consultant to build a comprehensive model for scheduling, but the implementation remained a challenge. The League decided to tackle the issue with the Solver avail- able within Microsoft Excel. Some of the match- ing priorities to be balanced while optimizing the schedule were:

1. Sales Revenue—Setting a schedule with match- es and time slots to those clubs that generate more revenue.

2. Channel Ratings—Setting a schedule with games that would improve channel ratings for the broadcasting company.

3. Team Rest Days—Setting a schedule with the two teams playing against each other having enough rest days.

The league decided to improve the match sched- ules by giving the player rest days as a higher priority, followed by sales revenue and channel scores for the broadcasting company. This is mainly because the sales revenue and channel scores are a byproduct of team players’ performance on the field, which is directly related to the rest days taken by the teams.

Methodology/Solution

Initially, organizing schedules was a huge task to perform on Excel through the built-in Solver feature. Frontline systems provided a premium version for Solver which allowed the model size to grow from about 200 decisions to 8,000 decisions. The League had to even add in more industry-specific constraints such as telecasting across different time zones, double header games cannot be overlapped, and arch rival games to be scheduled on Labor Day. Added limita- tions were never simple until the Frontline Systems consultants stepped up to help CFL turn this nonlin- ear problem into a linear problem. The linear pro- gramming “engine” got the model running. Premium Solver software turned out to be of great help to get an improved schedule.

Results/Benefits

Using the optimized schedule would lead to increased revenue through higher ticket sales and higher TV scores for the broadcasting channels. This was achieved because the tool was able to support added constraints of the vendors with great ease. The optimized schedule pleased most of the league’s stakeholders. This is a repetitive process, but those match schedules were CFL’s most advanced season match schedules to date.

Questions for DisCussion

1. List three ways in which Solver-based scheduling of games could result in more revenue as com- pared to the manual scheduling.

2. In what other ways can CFL leverage the Solver software to expand and enhance their other business operations?

3. What other considerations could be important in scheduling such games?

What Can We Learn from This Application Case?

By using the Solver add-in for Excel, the CFL made better decisions in scheduling their games by taking stakeholders and industry constraints into consider- ation, leading to revenue generation and good chan- nel ratings. Thus, an optimized schedule, a purview of prescriptive analytics, derived significant value. According to the case study, the modeler, Mr Trevor Hardy, was an expert Excel user, but not an expert in modeling. However, the ease of use of Excel per- mitted him to develop a practical application of pre- scriptive analytics.

Compiled from “Canadian Football League Uses Frontline Solvers to Optimize Scheduling in 2016.” Solver, September 7 2016, www. solver.com/news/canadian-football-league-uses-frontline- solvers-optimize-scheduling-2016 (accessed September 2018); Kostuk, Kent J., and Keith A. Willoughby. “A Decision Support System for Scheduling the Canadian Football League.” Interfaces, vol. 42, no. 3, 2012, pp. 286–295; Dilkina, Bistra N., and William S. Havens. The U.S. National Football League Scheduling Problem. Intelligent Systems Lab, www.cs.cornell.edu/~bistra/papers/ NFLsched1.pdf (accessed September 2018).

Application Case 8.1 (Continued)

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 465

Prescriptive Analytics Model Examples

Modeling is a key element for prescriptive analytics. In the examples mentioned earlier in the introduction and application cases, one has to employ a mathematical model to be able to recommend a decision for any realistic problem. For example, deciding which customers (among potentially millions) will receive what offer so as to maximize the overall response value but staying within a budget is not something you can do manu- ally. Building a probability-based response maximization model with the budget as a constraint would give us the information we are seeking. Depending on the problem we are addressing, there are many classes of models, and there are often many specialized techniques for solving each one. We will learn about two different modeling methods in this chapter. Most universities have multiple courses that cover these topics under titles such as Operations Research, Management Science, Decision Support Systems, and Simulation that can help you build more expertise in these topics. Because prescriptive analytics typically involves the application of mathematical models, sometimes the term data science is more commonly associated with the application of such mathematical models. Before we learn about mathematical modeling support in prescriptive analytics, let us understand some modeling issues first.

Identification of the Problem and Environmental Analysis

No decision is made in a vacuum. It is important to analyze the scope of the domain and the forces and dynamics of the environment. A decision maker needs to identify the organizational culture and the corporate decision-making processes (e.g., who makes decisions, degree of centralization). It is entirely possible that environmental factors have created the current problem. This can formally be called environmental scanning and analysis, which is the monitoring, scanning, and interpretation of collected information. Business intelligence/business analytics (BI/BA) tools can help identify problems by scan- ning for them. The problem must be understood, and everyone involved should share the same frame of understanding because the problem will ultimately be represented by the model in one form or another. Otherwise, the model will not help the decision maker.

VARIABLE IDENTIFICATION Identification of a model’s variables (e.g., decision, result, uncontrollable) is critical, as are the relationships among the variables. Influence dia- grams, which are graphical models of mathematical models, can facilitate the identifica- tion process. A more general form of an influence diagram, a cognitive map, can help a decision maker develop a better understanding of a problem, especially of variables and their interactions.

FORECASTING (PREDICTIVE ANALYTICS) As we have noted previously, an important prerequisite of prescriptive analytics is knowing what has happened and what is likely to happen. This form of predictive analytics is essential for construction and manipulating models because when a decision is implemented, the results usually occur in the future. There is no point in running a what-if (sensitivity) analysis on the past because decisions made then have no impact on the future. Online commerce and communication has cre- ated an immense need for forecasting and an abundance of available information for performing it. These activities occur quickly, yet information about such purchases is gathered and should be analyzed to produce forecasts. Part of the analysis involves sim- ply predicting demand; however, forecasting models can use product life-cycle needs and information about the marketplace and consumers to analyze the entire situation, ideally leading to additional sales of products and services.

We describe an effective example of such forecasting and its use in decision making at Ingram Micro in Application Case 8.2.

466 Part III • Prescriptive Analytics and Big Data

Ingram Micro is the world’s largest two-tier distrib- utor of technology products. In a two-tier distribu- tion system, a company purchases products from manufacturers and sells them to retailers who in turn sell these products to the end users. For exam- ple, one can purchase a Microsoft Office 365 pack- age from Ingram rather than purchasing it directly from Microsoft. Ingram has partnerships with Best Buy, Buffalo, Google, Honeywell, Libratone, and Sharper Image. The company delivers its products to 200,000 solution providers across the world and thus has a large volume of transaction data. Ingram wanted to use insights from this data to identify cross-selling opportunities and determine prices to offer to specific customers in conjunction with product bundles. This required setting up a busi- ness intelligence center (BIC) to compile and ana- lyze the data. In setting up the BIC, Ingram faced various issues.

1. Ingram faced several issues in their data- capture process such as a lack of loss data, ensuring the accuracy of end-user information, and linking quotes to orders.

2. Ingram faced technical issues in implementing a customer relationship management (CRM) system capable enough to handle its opera- tions around the world.

3. They faced resistance to the idea of demand pricing (determining price based on demand of product).

Methodology/Solution

Ingram explored communicating directly with its customers (resellers) using e-mail and offered them discounts on the purchase of supporting technolo- gies related to the products being ordered. They identified these opportunities through segmented market-basket analysis and developed the follow- ing business intelligence applications that helped in determining optimized prices. Ingram devel- oped a new price optimization tool known as IMPRIME, which is capable of setting data-driven prices and providing data-driven negotiation guid- ance. IMPRIME sets an optimized price for each

level of the product hierarchy (i.e., customer level, vendor-customer level, customer-segment level, and vendor-customer segment level). It does so by tak- ing into account the trade-off between the demand signal and pricing at that level.

The company also developed a digital market- ing platform known as Intelligence INGRAM. This platform utilizes predictive lead scoring (PLS), which selects end users to target with specific marketing programs. PLS is their system to score predictive leads for companies that have no direct relation with end users. Intelligence INGRAM is used to run white space programs, which encourage a reseller to purchase related products by offering discounts. For example, if a reseller purchases a server from INGRAM, then INGRAM offers a discount on disk storage units as both products are required to work together. Similarly, Intelligence INGRAM is used to run growth incentive campaigns (offering cash rewards to resellers on exceeding quarterly spend goals) and cross-sell campaigns (e-mailing the end users about the products that are related to their recently purchased product).

Results/Benefits

Profit generated by using IMPRIME is measured using a lift measurement methodology. This meth- odology compares periods before and after chang- ing the prices and compares test groups versus control groups. Lift measurement is done on aver- age daily sales, gross margin, and machine margin. The use of IMPRIME led to a $757 million growth in revenue and a $18.8 million increase in gross profits.

Questions for DisCussion

1. What were the main challenges faced by Ingram Micro in developing a BIC?

2. List all the business intelligence solutions devel- oped by Ingram to optimize the prices of their products and to profile their customers.

3. What benefits did Ingram receive after using the newly developed BI applications?

Application Case 8.2 Ingram Micro Uses Business Intelligence Applications to Make Pricing Decisions

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 467

What Can We Learn from This Application Case?

By first building a BIC, a company begins to bet- ter understand its product lines, its customers, and their purchasing patterns. This insight is derived from what we call descriptive and predictive ana- lytics. Further value from this is derived through

price optimization, a purview of prescriptive analytics.

Sources: R. Mookherjee, J. Martineau, L. Xu, M. Gullo, K. Zhou, A. Hazlewood, X. Zhang, F. Griarte, & N. Li. (2016). “End-to-End Predictive Analytics and Optimization in Ingram Micro’s Two-Tier Distribution Business.” Interfaces, 46(1), 49–73; ingrammicro- commerce.com, “CUSTOMERS,” https://www.ingrammicro- commerce.com/customers/ (accessed July 2016).

Model Categories

Table 8.1 classifies some decision models into seven groups and lists several representa- tive techniques for each category. Each technique can be applied to either a static or a dynamic model, which can be constructed under assumed environments of certainty, uncertainty, or risk. To expedite model construction, we can use special decision analysis systems that have modeling languages and capabilities embedded in them. These in- clude spreadsheets, data mining systems, online analytic processing (OLAP) systems, and modeling languages that help an analyst build a model. We will introduce one of these systems later in the chapter.

MODEL MANAGEMENT Models, like data, must be managed to maintain their integrity, and thus their applicability. Such management is done with the aid of model-based management systems, which are analogous to database management systems (DBMS).

KNOWLEDGE-BASED MODELING DSS uses mostly quantitative models, whereas expert systems use qualitative, knowledge-based models in their applications. Some knowledge is necessary to construct solvable (and therefore usable) models. Many of the predictive

TABLE 8.1 Categories of Models

Category Process and Objective Representative Techniques

Optimization of problems with few alternatives

Find the best solution from a small number of alternatives

Decision tables, decision trees, analytic hierarchy process

Optimization via algorithm

Find the best solution from a large number of alternatives, using a step-by-step improvement process

Linear and other mathematical programming models, network models

Optimization via an analytic formula

Find the best solution in one step, using a formula

Some inventory models

Simulation Find a good enough solution or the best among the alternatives checked, using experimentation

Several types of simulation

Heuristics Find a good enough solution, using rules Heuristic programming, expert systems

Predictive models Predict the future for a given scenario Forecasting models, Markov analysis

Other models Solve a what-if case, using a formula Financial modeling, waiting lines

468 Part III • Prescriptive Analytics and Big Data

analytics techniques, such as classification and clustering, can be used in building knowledge-based models.

CURRENT TRENDS IN MODELING One recent trend in modeling involves the develop- ment of model libraries and solution technique libraries. Some of these codes can be run directly on the owner’s Web server for free, and others can be downloaded and run on a local computer. The availability of these codes means that powerful optimization and simulation packages are available to decision makers who may have only experienced these tools from the perspective of classroom problems. For example, the Mathematics and Computer Science Division at Argonne National Laboratory (Argonne, Illinois) main- tains the NEOS Server for Optimization at https://neos-server.org/neos/index.html. You can find links to other sites by clicking the Resources link at informs.org, the Web site of the Institute for Operations Research and the Management Sciences (INFORMS). A wealth of modeling and solution information is available from INFORMS. The Web site for one of INFORMS’ publications, OR/MS Today, at http://www.orms-today.org/ ormsmain.shtml includes links to many categories of modeling software. We will learn about some of these shortly.

There is a clear trend toward developing and using cloud-based tools and soft- ware to access and even run software to perform modeling, optimization, simulation, and so on. This has, in many ways, simplified the application of many models to real-world problems. However, to use models and solution techniques effectively, it is necessary to truly gain experience through developing and solving simple ones. This aspect is often overlooked. Organizations that have key analysts who understand how to apply models indeed apply them very effectively. This is most notably occurring in the revenue management area, which has moved from the province of airlines, hotels, and automobile rentals to retail, insurance, entertainment, and many other areas. CRM also uses models, but they are often transparent to the user. With management mod- els, the amount of data and model sizes are quite large, necessitating the use of data warehouses to supply the data and parallel computing hardware to obtain solutions in a reasonable time frame.

There is a continuing trend toward making analytics models completely transparent to the decision maker. For example, multidimensional analysis (modeling) involves data analysis in several dimensions. In multidimensional analysis (modeling), data are generally shown in a spreadsheet format, with which most decision makers are familiar. Many decision makers accustomed to slicing and dicing data cubes are now using OLAP systems that access data warehouses. Although these methods may make modeling pal- atable, they also eliminate many important and applicable model classes from consid- eration, and they eliminate some important and subtle solution interpretation aspects. Modeling involves much more than data analysis with trend lines and establishing rela- tionships with statistical methods.

There is also a trend to build a model of a model to help in its analysis. An influence diagram is a graphical representation of a model; that is, a model of a model. Some in- fluence diagram software packages are capable of generating and solving the resultant model.

u SECTION 8.2 REVIEW QUESTIONS

1. List three lessons learned from modeling. 2. List and describe the major issues in modeling. 3. What are the major types of models used in DSS? 4. Why are models not used in industry as frequently as they should or could be? 5. What are the current trends in modeling?

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 469

8.3 STRUCTURE OF MATHEMATICAL MODELS FOR DECISION SUPPORT

In the following sections, we present the topics of analytical mathematical models (e.g., mathematical, financial, and engineering). These include the components and the struc- ture of models.

The Components of Decision Support Mathematical Models

All quantitative models are typically made up of four basic components (see Figure 8.1): result (or outcome) variables, decision variables, uncontrollable variables (and/or parameters), and intermediate result variables. Mathematical relationships link these components together. In nonquantitative models, the relationships are symbolic or qualitative. The results of decisions are determined based on the decision made (i.e., the values of the decision variables), the factors that cannot be controlled by the decision maker (in the environment), and the relationships among the variables. The modeling process involves identifying the variables and relationships among them. Solving a model determines the values of these and the result variable(s).

RESULT (OUTCOME) VARIABLES Result (outcome) variables reflect the level of effectiveness of a system; that is, they indicate how well the system performs or at- tains its goal(s). These variables are outputs. Examples of result variables are shown in Table 8.2. Result variables are considered dependent variables. Intermediate result vari- ables are sometimes used in modeling to identify intermediate outcomes. In the case of a dependent variable, another event must occur first before the event described by the variable can occur. Result variables depend on the occurrence of the decision variables and the uncontrollable variables.

DECISION VARIABLES Decision variables describe alternative courses of action. The decision maker controls the decision variables. For example, for an investment problem, the amount to invest in bonds is a decision variable. In a scheduling problem, the deci- sion variables are people, times, and schedules. Other examples are listed in Table 8.2.

UNCONTROLLABLE VARIABLES, OR PARAMETERS In any decision-making situation, there are factors that affect the result variables but are not under the control of the decision maker. Either these factors can be fixed, in which case they are called uncontrollable variables, or parameters, or they can vary, in which case they are called variables. Examples of factors are the prime interest rate, a city’s building code, tax regulations, and utilities costs. Most of these factors are uncontrollable because they are in and determined by elements of the system environment in which the decision maker works. Some of

Mathematical relationships

Uncontrollable variables

Intermediate variables

Decision variables

Result variables

FIGURE 8.1 The General Structure of a Quantitative Model.

470 Part III • Prescriptive Analytics and Big Data

TABLE 8.2 Examples of Components of Models

Area Decision Variables Result Variables Uncontrollable Variables and Parameters

Financial investment

Investment alternatives and amounts

Total profit, risk Rate of return on investment (ROI) Earnings per share Liquidity level

Inflation rate Prime rate Competition

Marketing Advertising budget Where to advertise

Market share Customer satisfaction

Customer’s income Competitor’s actions

Manufacturing What and how much to produce Inventory levels Compensation programs

Total cost Quality level Employee satisfaction

Machine capacity Technology Materials prices

Accounting Use of computers Audit schedule

Data processing cost Error rate

Computer technology Tax rates Legal requirements

Transportation Shipments schedule Use of smart cards

Total transport cost Payment float time

Delivery distance Regulations

Services Staffing levels Customer satisfaction Demand for services

these variables limit the decision maker and therefore form what are called constraints of the problem.

INTERMEDIATE RESULT VARIABLES Intermediate result variables reflect intermediate outcomes in mathematical models. For example, in determining machine scheduling, spoil- age is an intermediate result variable, and total profit is the result variable (i.e., spoilage is one determinant of total profit). Another example is employee salaries. This constitutes a decision variable for management: It determines employee satisfaction (i.e., intermediate outcome), which, in turn, determines the productivity level (i.e., final result).

The Structure of Mathematical Models

The components of a quantitative model are linked by mathematical (algebraic) expressions— equations or inequalities.

A very simple financial model is

P = R - C

where P = profit, R = revenue, and C = cost. This equation describes the relationship among the variables. Another well-known financial model is the simple present-value cash flow model, where P = present value, F = a future single payment in dollars, i = interest rate (percentage), and n = number of years. With this model, it is possible to determine the present value of a payment of $100,000 to be made 5 years from today, at a 10% (0.1) interest rate, as follows:

P = 100,000>(1 + 0.1)5 = 62,092 We present more interesting and complex mathematical models in the following

sections.

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 471

u SECTION 8.3 REVIEW QUESTIONS

1. What is a decision variable? 2. List and briefly discuss the major components of a quantitative model. 3. Explain the role of intermediate result variables.

8.4 CERTAINTY, UNCERTAINTY, AND RISK

The1 decision-making process involves evaluating and comparing alternatives. During this process, it is necessary to predict the future outcome of each proposed alternative. Decision situations are often classified on the basis of what the decision maker knows (or believes) about the forecasted results. We customarily classify this knowledge into three categories (see Figure 8.2), ranging from complete knowledge to complete ignorance:

• Certainty • Uncertainty • Risk

When we develop models, any of these conditions can occur, and different kinds of models are appropriate for each case. Next, we discuss both the basic definitions of these terms and some important modeling issues for each condition.

Decision Making under Certainty

In decision making under certainty, it is assumed that complete knowledge is available so that the decision maker knows exactly what the outcome of each course of action will be (as in a deterministic environment). It may not be true that the outcomes are 100% known, nor is it necessary to really evaluate all the outcomes, but often this assumption simplifies the model and makes it tractable. The decision maker is viewed as a perfect predictor of the future because it is assumed that there is only one outcome for each al- ternative. For example, the alternative of investing in U.S. Treasury bills is one for which there is complete availability of information about the future return on investment if it is held to maturity. A situation involving decision making under certainty occurs most often with structured problems and short time horizons (up to 1 year). Certainty models are relatively easy to develop and solve, and they can yield optimal solutions. Many financial models are constructed under assumed certainty, even though the market is anything but 100% certain.

1Some parts of the original versions of these sections were adapted from Turban and Meredith (1994).

Increasing knowledge

Complete Knowledge Certainty

Total Ignorance

Uncertainty

Decreasing knowledge

Risk

FIGURE 8.2 The Zones of Decision Making.

472 Part III • Prescriptive Analytics and Big Data

Decision Making under Uncertainty

In decision making under uncertainty, the decision maker considers situations in which several outcomes are possible for each course of action. In contrast to the risk situation, in this case, the decision maker does not know, or cannot estimate, the probability of occurrence of the possible outcomes. Decision making under uncertainty is more difficult than decision making under certainty because there is insufficient information. Modeling of such situations involves assessment of the decision maker’s (or the organization’s) attitude toward risk.

Managers attempt to avoid uncertainty as much as possible, even to the point of assuming it away. Instead of dealing with uncertainty, they attempt to obtain more in- formation so that the problem can be treated under certainty (because it can be “almost” certain) or under calculated (i.e., assumed) risk. If more information is not available, the problem must be treated under a condition of uncertainty, which is less definitive than the other categories.

Decision Making under Risk (Risk Analysis)

A decision made under risk2 (also known as a probabilistic or stochastic decision-making situation) is one in which the decision maker must consider several possible outcomes for each alternative, each with a given probability of occurrence. The long-run probabilities that the given outcomes will occur are assumed to be known or can be estimated. Under these assumptions, the decision maker can assess the degree of risk associated with each alternative (called calculated risk). Most major business decisions are made under assumed risk. Risk analysis (i.e., calculated risk) is a decision-making method that ana- lyzes the risk (based on assumed known probabilities) associated with different alterna- tives. Risk analysis can be performed by calculating the expected value of each alternative and selecting the one with the best expected value. Application Case 8.3 illustrates one application to reduce uncertainty.

2Our definitions of the terms risk and uncertainty were formulated by F. H. Knight of the University of Chicago in 1933. Other, comparable definitions also are in use.

American Airlines, Inc. (AA) is one of the world’s largest airlines. Its core business is passenger trans- portation, but it has other vital ancillary functions that include full-truckload (FTL) freight shipment of maintenance equipment and in-flight shipment of passenger service items that could add up to over $1 billion in inventory at any given time. AA receives numerous bids from suppliers in response to requests for quotes (RFQs) for inventories. AA’s RFQs could total over 500 in any given year. Bid quotes vary significantly as a result of the large number of bids and resultant complex bidding process. Sometimes, a single contract bid could deviate by about 200%. As a

result of the complex process, it is common to either overpay or underpay suppliers for their services. To this end, AA wanted a should-cost model that would streamline and assess bid quotes from suppliers to choose bid quotes that were fair to both them and their suppliers.

Methodology/Solution

To determine fair cost for supplier products and ser- vices, three steps were taken:

1. Primary (e.g., interviews) and secondary (e.g., Internet) sources were scouted for base-case

Application Case 8.3 American Airlines Uses Should-Cost Modeling to Assess the Uncertainty of Bids for Shipment Routes

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 473

u SECTION 8.4 REVIEW QUESTIONS

1. Define what it means to perform decision making under assumed certainty, risk, and uncertainty.

2. How can decision-making problems under assumed certainty be handled? 3. How can decision-making problems under assumed uncertainty be handled? 4. How can decision-making problems under assumed risk be handled?

8.5 DECISION MODELING WITH SPREADSHEETS

Models can be developed and implemented in a variety of programming languages and systems. We focus primarily on spreadsheets (with their add-ins), modeling languages, and transparent data analysis tools. With their strength and flexibility, spreadsheet packages were quickly recognized as easy-to-use implementation software for the development of a wide range of applications in business, engineering, mathematics, and science. Spreadsheets include extensive statistical, forecasting, and other modeling and database management capabilities, functions, and routines. As spreadsheet packages evolved, add- ins were developed for structuring and solving specific model classes. Among the add-in packages, many were developed for DSS development. These DSS-related add-ins in- clude Solver (Frontline Systems Inc., solver.com) and What’sBest! (a version of Lindo, from Lindo Systems, Inc., lindo.com) for performing linear and nonlinear optimization; Braincel (Jurik Research Software, Inc., jurikres.com) and NeuralTools (Palisade Corp., palisade.com) for artificial neural networks; Evolver (Palisade Corp.) for genetic algo- rithms; and @RISK (Palisade Corp.) for performing simulation studies. Comparable add-ins are available for free or at a very low cost. (Conduct a Web search to find them; new ones are added to the marketplace on a regular basis.)

and range data that would inform cost vari- ables that affect an FTL bid.

2. Cost variables were chosen so that they were mutually exclusive and collectively exhaustive.

3. The DPL decision analysis software was used to model the uncertainty.

Furthermore, Extended Swanson-Megill approximation was used to model the probability distribution of the most sensitive cost variables used. This was done to account for the high variability in the bids in the initial model.

Results/Benefits

A pilot test was done on an RFQ that attracted bids from six FTL carriers. Out of the six bids presented, five were within three standard deviations from the mean, whereas one was considered an outlier. Subsequently, AA used the should-cost FTL model on more than 20 RFQs to determine what a fair and accurate cost of goods and services should be.

It is expected that this model will help in reduc- ing the risk of either overpaying or underpaying its suppliers.

Questions for DisCussion

1. Besides reducing the risk of overpaying or underpaying suppliers, what are some other benefits AA would derive from its “should-be” model?

2. Can you think of other domains besides air transportation where such a model could be used?

3. Discuss other possible methods with which AA could have solved its bid overpayment and underpayment problem.

Source: Based on Bailey, M. J., Snapp, J., Yetur, S., Stonebraker, J. S., Edwards, S. A., Davis, A., & Cox, R. (2011). Practice sum- maries: American Airlines uses should-cost modeling to assess the uncertainty of bids for its full-truckload shipment routes. Interfaces, 41(2), 194–196.

474 Part III • Prescriptive Analytics and Big Data

The spreadsheet is clearly the most popular end-user modeling tool because it incorporates many powerful financial, statistical, mathematical, and other functions. Spreadsheets can perform model solution tasks such as linear programming and regres- sion analysis. The spreadsheet has evolved into an important tool for analysis, plan- ning, and modeling (see Farasyn, Perkoz, & Van de Velde, 2008; Hurley & Balez, 2008; Ovchinnikov & Milner, 2008). Application Cases 8.4 and 8.5 describe interesting applica- tions of spreadsheet-based models in a nonprofit setting.

The Pennsylvania Adoption Exchange (PAE) was established in 1979 by the State of Pennsylvania to help county and nonprofit agencies find prospec- tive families for orphan children who had not been adopted due to age or special needs. The PAE keeps detailed records about children and preferences of families who may adopt them. The exchange looks for families for the children across all 67 counties of Pennsylvania.

The Pennsylvania Statewide Adoption and Permanency Network is responsible for finding per- manent homes for orphans. If after a few attempts the network fails to place a child with a family, they then get help from the PAE. The PAE uses an auto- mated assessment tool to match children to fami- lies. This tool gives matching recommendations by calculating a score between 0 and 100% for a child on 78 pairs of the child’s attribute values and fam- ily preferences. For some years now, the PAE has struggled to give adoption match recommendations to caseworkers for children. They are finding it diffi- cult to manage a vast database of children collected over time for all 67 counties. The basic search algo- rithm produced match recommendations that were proving unfruitful for caseworkers. As a result, the number of children who have not been adopted has increased significantly, and there is a growing urgency to find families for these orphans.

Methodology/Solution

The PAE started collecting information about the orphans and families through online surveys that include a new set of questions. These questions col- lect information about hobbies of the child, child– caseworker preferences for families, and preference of the age range of children by families. The PAE and consultants created a spreadsheet matching

tool that included additional features compared to the previously used automated tool. In this model, caseworkers can specify the weight of the attributes for selecting a family for a child. For example, if a family had a narrow set of preferences regarding gender, age, and race, then those factors can receive a higher weight. Also, caseworkers can give pref- erence about the family’s county of residence, as community relationship is an important factor for a child. Using this tool, the matching committee can compare a child and family on each attribute, thus making a more accurate match decision between a family and a child.

Results/Benefits

Since the PAE started using the new spreadsheet model for matching a family with a child, they have been able to make better matching decisions. As a result, the percentage of children getting a perma- nent home has increased.

This short case is one of the many examples of using spreadsheets as a decision support tool. By creating a simple scoring system for a family’s desire and a child’s attribute, a better matching system is produced so that fewer rejections are reported on either side.

Questions for DisCussion

1. What were the challenges faced by PAE while making adoption matching decisions?

2. What features of the new spreadsheet tool helped PAE solve their issues of matching a fam- ily with a child?

Source: Based on Slaugh, V. W., Akan, M., Kesten, O., & Unver, M. U. (2016). The Pennsylvania Adoption Exchange improves its matching process. Interfaces, 46(2), 133–154.

Application Case 8.4 Pennsylvania Adoption Exchange Uses Spreadsheet Model to Better Match Children with Families

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 475

Meals on Wheels Association of America (now Meals on Wheels America) is a not-for-profit organi- zation that delivers approximately one million meals to homes of older people in need across the United States. Metro Meals on Wheels Treasure Valley is a local branch of Meals on Wheels America operat- ing in Idaho. This branch has a team of volunteer drivers that drive their personal vehicles each day to deliver meals to 800 clients along 21 routes and cover an area of 2,745 square kilometers.

The Meals on Wheels Treasure Valley organi- zation was facing many issues. First, they were look- ing to minimize the delivery time as the cooked food was temperature sensitive and could perish easily. They wanted to deliver the cooked food within 90 minutes after a driver left for the delivery. Second, the scheduling process was very time consuming. Two employees spent much of their time develop- ing scheduled routes for delivery. A route coordina- tor determined the stops according to the number of meal recipients for a given day. After determining the stops, the coordinator made a sequence of stops that minimized the travel time of volunteers. This routing schedule was then entered into an online tool to determine turn-by-turn driving instructions for drivers. The whole process of manually deciding routes was taking a lot of extra time. Metro Meals on Wheels wanted a routing tool that could improve their delivery system and generate routing solu- tions for both one-way and round-trip directions for delivering meals. Those who drive regularly could deliver the warmers or coolers the next day. Others who drive only occasionally would need to come back to the kitchen to drop off the warmers/coolers.

Methodology/Solution

To solve the routing problem, a spreadsheet-based tool was developed. This tool had an interface to easily input information about the recipient such as his/her name, meal requirements, and delivery

address. This information needed to be filled in the spreadsheet for each stop in the route. Next, Excel’s Visual Basic for Applications functionality was used to access a developer’s networking map application programming interface (API) called MapQuest. This API was used to create a travel matrix that calcu- lated time and distance needed for delivery of the meal. This tool gave time and distance information for 5,000 location pairs a day without any cost.

When the program starts, the MapQuest API first validates the entered addresses of meal recipi- ents. Then the program uses the API to retrieve driving distance, estimated driving time, and turn- by-turn instructions for driving between all stops in the route. The tool can then find the optimal route for up to 30 stops within a feasible time limit.

Results/Benefits

As a result of using this tool, the total annual driv- ing distance decreased by 10,000 miles, while travel time was reduced by 530 hours. Metro Meals on Wheels Treasure Valley saved $5,800 in 2015, based on an estimated savings rate of $0.58 per mile (for a midsize sedan). This tool also reduced the time spent on route planning for meal deliveries. Other benefits included increased volunteer satisfaction and more retention of volunteers.

Questions for DisCussion

1. What were the challenges faced by Metro Meals on Wheels Treasure Valley related to meal delivery before adoption of the spreadsheet-based tool?

2. Explain the design of the spreadsheet-based model.

3. What are the intangible benefits of using the Excel-based model to Metro Meals on Wheels?

Source: Based on Manikas, A. S., Kroes, J. R., & Gattiker, T. F. (2016). Metro Meals on Wheels Treasure Valley employs a low- cost routing tool to improve deliveries. Interfaces, 46(2), 154–167.

Application Case 8.5 Metro Meals on Wheels Treasure Valley Uses Excel to Find Optimal Delivery Routes

Other important spreadsheet features include what-if analysis, goal seeking, data management, and programmability (i.e., macros). With a spreadsheet, it is easy to change a cell’s value and immediately see the result. Goal seeking is performed by indicating a target cell, its desired value, and a changing cell. Extensive database management can be performed with small data sets, or parts of a database can be imported for analysis (which

476 Part III • Prescriptive Analytics and Big Data

is essentially how OLAP works with multidimensional data cubes; in fact, most OLAP sys- tems have the look and feel of advanced spreadsheet software after the data are loaded). Templates, macros, and other tools enhance the productivity of building DSS.

Most spreadsheet packages provide fairly seamless integration because they read and write common file structures and easily interface with databases and other tools. Microsoft Excel is the most popular spreadsheet package. In Figure 8.3, we show a simple loan calculation model in which the boxes on the spreadsheet describe the contents of the cells, which contain formulas. A change in the interest rate in cell E7 is immediately reflected in the monthly payment in cell E13. The results can be observed and analyzed immediately. If we require a specific monthly payment, we can use goal seeking to deter- mine an appropriate interest rate or loan amount.

Static or dynamic models can be built in a spreadsheet. For example, the monthly loan calculation spreadsheet shown in Figure 8.3 is static. Although the problem affects the borrower over time, the model indicates a single month’s performance, which is repli- cated. A dynamic model, in contrast, represents behavior over time. The loan calculations in the spreadsheet shown in Figure 8.4 indicate the effect of prepayment on the principal over time. Risk analysis can be incorporated into spreadsheets by using built-in random- number generators to develop simulation models (see the next chapter).

Spreadsheet applications for models are reported regularly. We will learn how to use a spreadsheet-based optimization model in the next section.

u SECTION 8.5 REVIEW QUESTIONS

1. What is a spreadsheet? 2. What is a spreadsheet add-in? How can add-ins help in DSS creation and use? 3. Explain why a spreadsheet is so conducive to the development of DSS.

FIGURE 8.3 Excel Spreadsheet Static Model Example of a Simple Loan Calculation of Monthly Payments.

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 477

8.6 MATHEMATICAL PROGRAMMING OPTIMIZATION

Mathematical programming is a family of tools designed to help solve managerial problems in which the decision maker must allocate scarce resources among competing activities to optimize a measurable goal. For example, the distribution of machine time (the resource) among various products (the activities) is a typical allocation problem. Linear programming (LP) is the best-known technique in a family of optimization tools called mathematical programming; in LP, all relationships among the variables are linear. It is used extensively in DSS (see Application Case 8.6). LP models have many important applications in practice. These include supply chain management, product mix decisions, routing, and so on. Special forms of the models can be used for specific applications. For example, Application Case 8.6 describes a spreadsheet model that was used to create a schedule for physicians.

LP allocation problems usually display the following characteristics:

• A limited quantity of economic resources is available for allocation. • The resources are used in the production of products or services. • There are two or more ways in which the resources can be used. Each is called a

solution or a program. • Each activity (product or service) in which the resources are used yields a return in

terms of the stated goal. • The allocation is usually restricted by several limitations and requirements, called

constraints.

FIGURE 8.4 Excel Spreadsheet Dynamic Model Example of a Simple Loan Calculation of Monthly Payments and the Effects of Prepayment.

478 Part III • Prescriptive Analytics and Big Data

Regional Neonatal Associates is a nine-physician group working for the Neonatal Intensive Care Unit (NICU) at the University of Tennessee Medical Center in Knoxville, Tennessee. The group also serves two local hospitals in the Knoxville area for emergency purposes. For many years, one member of the group would schedule physicians manually. However, as his retirement approached, there was a need for a more automatic system to schedule physi- cians. The physicians wanted this system to balance their workload, as the previous schedules did not properly balance workload among them. In addi- tion, the schedule needed to ensure there would be 24-7 NICU coverage by the physicians, and if possible, accommodate individual preferences of physicians for shift types. To address this problem, the physicians contacted the faculty of Management Science at the University of Tennessee.

The problem of scheduling physicians to shifts was characterized by constraints based on work- load and lifestyle choices. The first step in solving the scheduling issue was to group shifts according to their types (day and night). The next step was determining constraints for the problem. The model needed to cover a nine-week period with nine phy- sicians, with two physicians working weekdays and one physician overnight and on weekends. In addi- tion, one physician had to be assigned exclusively for 24-7 coverage to the two local hospitals. Other obvious constraints also needed to be considered. For example, a day shift could not be assigned to a physician just after a night shift.

Methodology/Solution

The problem was formulated by creating a binary, mixed-integer optimization model. The first model divided workload equally among the nine physi- cians. But it could not assign an equal number of day and night shifts among them. This created a ques- tion of fair distribution. In addition, the physicians had differing opinions of the assigned workload. Six physicians wanted a schedule in which an equal

number of day and night shifts would be assigned to each physician in the nine-week schedule, while the others wanted a schedule based on individual preference of shifts. To satisfy requirements of both groups of physicians, a new model was formed and named the Hybrid Preference Scheduling Model (HPSM). For satisfying the equality requirement of six physicians, the model first calculated one week’s workload and divided it for nine weeks for them. This way, the work was divided equally for all six physicians. The workload for the three remaining physicians was distributed in the nine-week sched- ule according to their preference. The resulting schedule was reviewed by the physicians and they found the schedule more acceptable.

Results/Benefits

The HPSM method accommodated both the equal- ity and individual preference requirements of the physicians. In addition, the schedules from this model provided better rest times for the physicians compared to the previous manual schedules, and vacation requests could also be accommodated in the schedules. The HPSM model can solve similar scheduling problems demanding relative prefer- ences among shift types.

Techniques such as mixed-integer program- ming models can build optimal schedules and help in operations. These techniques have been used in large organizations for a long time. Now it is pos- sible to implement such prescriptive analytic models in spreadsheets and other easily available software.

Questions for DisCussion

1. What was the issue faced by the Regional Neonatal Associates group?

2. How did the HPSM model solve all of the physi- cian’s requirements?

Source: Adapted from Bowers, M. R., Noon, C. E., Wu, W., & Bass, J. K. (2016). Neonatal physician scheduling at the University of Tennessee Medical Center. Interfaces, 46(2), 168–182.

Application Case 8.6 Mixed-Integer Programming Model Helps the University of Tennessee Medical Center with Scheduling Physicians

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 479

The LP allocation model is based on the following rational economic assumptions:

• Returns from different allocations can be compared; that is, they can be measured by a common unit (e.g., dollars, utility).

• The return from any allocation is independent of other allocations. • The total return is the sum of the returns yielded by the different activities. • All data are known with certainty. • The resources are to be used in the most economical manner.

Allocation problems typically have a large number of possible solutions. Depending on the underlying assumptions, the number of solutions can be either infinite or finite. Usually, different solutions yield different rewards. Of the available solutions, at least one is the best, in the sense that the degree of goal attainment associated with it is the highest (i.e., the total reward is maximized). This is called an optimal solution, and it can be found by using a special algorithm.

Linear Programming Model

Every LP model is composed of decision variables (whose values are unknown and are searched for), an objective function (a linear mathematical function that relates the deci- sion variables to the goal, measures goal attainment, and is to be optimized), objective function coefficients (unit profit or cost coefficients indicating the contribution to the ob- jective of one unit of a decision variable), constraints (expressed in the form of linear in- equalities or equalities that limit resources and/or requirements; these relate the variables through linear relationships), capacities (which describe the upper and sometimes lower limits on the constraints and variables), and input/output (technology) coefficients (which indicate resource utilization for a decision variable).

Let us look at an example. MBI Corporation, which manufactures special-purpose computers, needs to make a decision: How many computers should it produce next month at the Boston plant? MBI is considering two types of computers: the CC@7, which requires 300 days of labor and $10,000 in materials, and the CC@8, which requires 500 days of labor and $15,000 in materials. The profit contribution of each CC@7 is $8,000, whereas that of each CC@8 is $12,000. The plant has a capacity of 200,000 working days per month, and the material budget is $8 million per month. Marketing requires that at least 100 units of the CC@7 and at least 200 units of the CC@8 be produced each month. The problem is to maximize the company’s profits by determining how many units of the CC@7 and how many units of the CC@8 should be produced each month. Note that in a real-world envi- ronment, it could possibly take months to obtain the data in the problem statement, and

TECHNOLOGY INSIGHTS 8.1 Linear Programming

LP is perhaps the best-known optimization model. It deals with the optimal allocation of re- sources among competing activities. The allocation problem is represented by the model de- scribed here.

The problem is to find the values of the decision variables X1, X2, and so on, such that the value of the result variable Z is maximized, subject to a set of linear constraints that ex- press the technology, market conditions, and other uncontrollable variables. The mathematical relationships are all linear equations and inequalities. Theoretically, any allocation problem of this type has an infinite number of possible solutions. Using special mathematical proce- dures, the LP approach applies a unique computerized search procedure that finds the best solution(s) in a matter of seconds. Furthermore, the solution approach provides automatic sensitivity analysis.

480 Part III • Prescriptive Analytics and Big Data

while gathering the data the decision maker would no doubt uncover facts about how to structure the model to be solved. Web-based tools for gathering data can help.

Modeling in LP: An Example

A standard LP model can be developed for the MBI Corporation problem just described. As discussed in Technology Insights 8.1, the LP model has three components: decision variables, result variables, and uncontrollable variables (constraints). The decision variables are as follows:

X1 = units of CC-7 to be produced X2 = units of CC-8 to be produced

The result variable is as follows:

Total profit = Z

The objective is to maximize total profit:

Z = 8,000X1 + 12,000X2 The uncontrollable variables (constraints) are as follows:

Labor constraint: 300X1 + 500X2 … 200,000 (in days) Budget constraint: 10,000X1 + 15,0 0 0X2 … 8,000,000 (in dollars) Marketing requirement for CC-7: X1 Ú 100 (in units) Marketing requirement for CC-8: X2 Ú 200 (in units)

This information is summarized in Figure 8.5. The model also has a fourth, hidden component. Every LP model has some internal

intermediate variables that are not explicitly stated. The labor and budget constraints may each have some slack in them when the left-hand side is strictly less than the right-hand side. This slack is represented internally by slack variables that indicate excess resources available. The marketing requirement constraints may each have some surplus in them when the left-hand side is strictly greater than the right-hand side. This surplus is rep- resented internally by surplus variables indicating that there is some room to adjust the right-hand sides of these constraints. These slack and surplus variables are intermediate. They can be of great value to a decision maker because LP solution methods use them in establishing sensitivity parameters for economic what-if analyses.

Decision variables Mathematical (logical)

relationships

Maximize Z (profit)

subject to constraints

Total profit 5 Z

Z 5 8,000X1 1 12,000X2

Result variables

X1 5 units of CC-7

X2 5 units of CC-8

300X1 1 500X2 # 200,000

10,000X1 1 15,000X2 # 8,000,000

X1 $ 100

X2 $ 200

Constraints (uncontrollable)

FIGURE 8.5 Mathematical Model of a Product-Mix Example.

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 481

The product-mix model has an infinite number of possible solutions. Assuming that a production plan is not restricted to whole numbers—which is a reasonable assumption in a monthly production plan—we want a solution that maximizes total profit: an optimal solution. Fortunately, Excel comes with the add-in Solver, which can readily obtain an optimal (best) solution to this problem. Although the location of Solver add-in has moved from one version of Excel to another, it is still available as a free add-in. Look for it under the Data tab and on the Analysis ribbon. If it is not there, you should be able to enable it by going to Excel’s Options Menu and selecting Add-ins.

We enter these data directly into an Excel spreadsheet, activate Solver, and identify the goal (by setting Target Cell equal to Max), decision variables (by setting By Changing Cells), and constraints (by ensuring that Total Consumed elements is less than or equal to Limit for the first two rows and is greater than or equal to Limit for the third and fourth rows). Cells C7 and D7 constitute the decision variable cells. Results in these cells will be filled after running the Solver Add-in. Target Cell is Cell E7, which is also the result vari- able, representing a product of decision variable cells and their per unit profit coefficients (in Cells C8 and D8). Note that all the numbers have been divided by 1,000 to make it easier to type (except the decision variables). Rows 9–12 describe the constraints of the problem: the constraints on labor capacity, budget, and the desired minimum production of the two products X1 and X2. Columns C and D define the coefficients of these con- straints. Column E includes the formulae that multiply the decision variables (Cells C7 and D7 ) with their respective coefficients in each row. Column F defines the right-hand side value of these constraints. Excel’s matrix multiplication capabilities (e.g., SUMPRODUCT function) can be used to develop such row and column multiplications easily.

After the model’s calculations have been set up in Excel, it is time to invoke the Solver Add-in. Clicking on the Solver Add-in (again under the Analysis group under Data Tab) opens a dialog box (window) that lets you specify the cells or ranges that define the objective function cell, decision/changing variables (cells), and the constraints. Also, in Options, we select the solution method (usually Simplex LP), and then we solve the problem. Next, we select all three reports—Answer, Sensitivity, and Limits—to obtain an optimal solution of X1 = 333.33, X2 = 200, Profit = $5,066,667, as shown in Figure 8.6. Solver produces three useful reports about the solution. Try it. Solver now also includes the ability to solve nonlinear programming problems and integer programming problems by using other solution methods available within it.

The following example was created by Professor Rick Wilson of Oklahoma State University to further illustrate the power of spreadsheet modeling for decision support.

The table in Figure 8.7 describes some hypothetical data and attributes of nine “swing states” for the 2016 election. Attributes of the nine states include their number of electoral votes, two regional descriptors (note that three states are classified as neither North nor South), and an estimated “influence function,” which relates to increased can- didate support per unit of campaign financial investment in that state.

For instance, influence function F1 shows that for every financial unit invested in that state, there will be a total of a 10-unit increase in voter support (let units stay general here), made up of an increase in young men support by 3 units, old men support by 1 unit, and young and old women each by 3 units.

The campaign has 1,050 financial units to invest in the nine states. It must invest at least 5% in each state of the total overall invested, but no more than 25% of the overall total invested can be in any one state. All 1,050 units do not have to be invested (your model must correctly deal with this).

The campaign has some other restrictions as well. From a financial investment standpoint, the West states (in total) must have campaign investments at levels that are at least 60% of the total invested in East states. In terms of people influenced, the decision to allocate financial investments to states must lead to at least 9,200 total people influenced.

482 Part III • Prescriptive Analytics and Big Data

FIGURE 8.7 Data for Election Resource Allocation Example.

FIGURE 8.6 Excel Solver Solution to the Product-Mix Example.

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 483

Overall, the total number of females influenced must be greater than or equal to the total number of males influenced. Also, at least 46% of all people influenced must be “old.”

Our task is to create an appropriate integer programming model that determines the optimal integer (i.e., whole number) allocation of financial units to states that maximizes the sum of the products of the electoral votes times units invested subject to the other aforementioned restrictions. (Thus, indirectly, this model is giving preference to states with higher numbers of electoral votes.) Note that for ease of implementation by the cam- paign staff, all decisions for allocation in the model should lead to integer values.

The three aspects of the models can be categorized based on the following ques- tions that they answer:

1. What do we control? The amount invested in advertisements across the nine states, Nevada, Colorado, Iowa, Wisconsin, Ohio, Virginia, North Carolina, Florida, and New Hampshire, which are represented by the nine decision variables, NV, CO, IA, WI OH, VA, NC, FL, and NH.

2. What do we want to achieve? We want to maximize the total number of elec- toral votes gains. We know the value of each electoral vote in each state (EV), so this amounts to EV*Investments aggregated over the nine states, that is,

Max (6NV + 9CO + 6IA + 10WI + 18OH + 13VA + 15NC + 29FL + 4NH)

3. What constrains us? Following are the constraints as given in the problem description:

a. No more than 1,050 financial units to invest into, that is, NV + CO + IA + WI + OH + VA + NC + FL + NH <= 1,050.

b. Invest at least 5% of the total in each state, that is,

NV 7 = 0.05 (NV + CO + IA + WI + OH + VA + NC + FL + NH) CO 7 = 0.05 (NV + CO + IA + WI + OH + VA + NC + FL + NH) IA 7 = 0.05 (NV + CO + IA + WI + OH + VA + NC + FL + NH) WI 7 = 0.05 (NV + CO + IA + WI + OH + VA + NC + FL + NH) OH 7 = 0.05 (NV + CO + IA + WI + OH + VA + NC + FL + NH) VA 7 = 0.05 (NV + CO + IA + WI + OH + VA + NC + FL + NH) NC 7 = 0.05 (NV + CO + IA + WI + OH + VA + NC + FL + NH) FL 7 = 0.05 (NV + CO + IA + WI + OH + VA + NC + FL + NH)

NH 7 = 0.05 (NV + CO + IA + WI + OH + VA + NC + FL + NH)

We can implement these nine constraints in a variety of ways using Excel. c. Invest no more than 25% of the total in each state.

As with (b) we need nine individual constraints again because we do not know how much of the 1,050 we will invest. We must write the constraints in “general” terms.

NV 6 = 0.25 (NV + CO + IA + WI + OH + VA + NC + FL + NH) CO 6 = 0.25 (NV + CO + IA + WI + OH + VA + NC + FL + NH) IA 6 = 0.25 (NV + CO + IA + WI + OH + VA + NC + FL + NH) WI 6 = 0.25 (NV + CO + IA + WI + OH + VA + NC + FL + NH) OH 6 = 0.25 (NV + CO + IA + WI + OH + VA + NC + FL + NH) VA 6 = 0.25 (NV + CO + IA + WI + OH + VA + NC + FL + NH) NC 6 = 0.25 (NV + CO + IA + WI + OH + VA + NC + FL + NH) FL 6 = 0.25 (NV + CO + IA + WI + OH + VA + NC + FL + NH)

NH 6 = 0.25 (NV + CO + IA + WI + OH + VA + NC + FL + NH)

484 Part III • Prescriptive Analytics and Big Data

d. Western states must have investment levels that are at least 60% of the Eastern states.

West States = NV + CO + IA + WI East States = OH + VA + NC + FL + NH

So, (NV + CO + IA + WI) 7 = 0.60 (OH + VA + NC + FL + NH). Again, we can implement this constraint in a variety of ways using Excel.

e. Influence at least 9,200 total people, that is,

(10NV + 7.5CO + 8IA + 10WI + 7.5OH + 7.5VA + 10NC + 8FL + 8 NH) 7 = 9,200

f. Influence at least as many females as males. This requires transition of influence functions.

F1 = 6 women influenced, F2 = 3.5 women F3 = 3 women influenced F1 = 4 men influenced, F2 = 4 men F3 = 5 men influenced

So, implementing females 7 = males, we get:

(6NV + 3.5CO + 3IA + 6WI + 3.5OH + 3.5VA + 6NC + 3FL + 3NH) 7 = (4NV + 4CO + 5IA + 4WI + 4OH + 4VA + 4NC + 5FL + 5NH)

As before, we can implement this in Excel in a couple of different ways. g. At least 46% of all people influenced must be old.

All people influenced were on the left-hand side of the constraint (e) . So, old people influenced would be:

(4NV + 3.5CO + 4.5IA + 4WI + 3.5OH + 3.5VA + 4NC + 4.5FL + 4.5NH)

This would be set 7 = 0.46* the left-hand side of constraint (e). (10NV + 7.5CO + 8IA + 10WI + 7.5OH + 7.5VA + 10NC + 8FL + 8NH), which would give a right-hand side of (0.46NV + 3 .45CO + 3 .68IA + 4 .6WI + 3 .45OH + 3 .45VA + 4.6NC + 3.68FL + 3 .68NH)

This is the last constraint other than to force all variables to be integers.

All told in algebraic terms, this integer programming model would have 9 decision variables and 24 constraints (one constraint for integer requirements).

Implementation

One approach would be to implement the model in strict “standard form,” or a row-column form, where all constraints are written with decision variables on the left-hand side, a number on the right-hand side. Figure 8.8 shows such an implementation and displays the solved model.

Alternatively, we could use the spreadsheet to calculate different parts of the model in a less rigid manner, as well as uniquely implementing the repetitive constraints (b) and (c), and have a much more concise (but not as transparent) spreadsheet. This is shown in Figure 8.9.

LP models (and their specializations and generalizations) can also be specified di- rectly in a number of other user-friendly modeling systems. Two of the best known are Lindo and Lingo (Lindo Systems, Inc., lindo.com; demos are available). Lindo is an

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 485

FIGURE 8.8 Model for Election Resource Allocation—Standard Version.

LP and integer programming system. Models are specified in essentially the same way that they are defined algebraically. Based on the success of Lindo, the company developed Lingo, a modeling language that includes the powerful Lindo optimizer and extensions for solving nonlinear problems. Many other modeling languages such as AMPL, AIMMS, MPL, XPRESS, and others are available.

The most common optimization models can be solved by a variety of mathematical programming methods, including the following:

• Assignment (best matching of objects) • Dynamic programming • Goal programming • Investment (maximizing rate of return) • Linear and integer programming • Network models for planning and scheduling • Nonlinear programming • Replacement (capital budgeting) • Simple inventory models (e.g., economic order quantity) • Transportation (minimize cost of shipments)

u SECTION 8.6 REVIEW QUESTIONS

1. List and explain the assumptions involved in LP. 2. List and explain the characteristics of LP. 3. Describe an allocation problem. 4. Define the product-mix problem. 5. Define the blending problem. 6. List several common optimization models.

486 Part III • Prescriptive Analytics and Big Data

8.7 MULTIPLE GOALS, SENSITIVITY ANALYSIS, WHAT-IF ANALYSIS, AND GOAL SEEKING

Many, if not most, decision situations involve juggling between competing goals and alternatives. In addition, there is significant uncertainty about the assumptions and pre- dictions being used in building a prescriptive analytics model. The following paragraphs simply recognize that these are also addressed in prescriptive analytics software and techniques. Coverage of these techniques is usually common in prescriptive analytics or operations research/management science courses.

Multiple Goals

The analysis of management decisions aims at evaluating, to the greatest possible extent, how far each alternative advances managers toward their goals. Unfortunately, manage- rial problems are seldom evaluated with a single simple goal, such as profit maximiza- tion. Today’s management systems are much more complex, and one with a single goal is rare. Instead, managers want to attain simultaneous goals, some of which may conflict. Different stakeholders have different goals. Therefore, it is often necessary to analyze

FIGURE 8.9 A Compact Formulation for Election Resource Allocation.

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 487

each alternative in light of its determination of each of several goals (see Koksalan & Zionts, 2001).

For example, consider a profit-making firm. In addition to earning money, the com- pany wants to grow, develop its products and employees, provide job security to its workers, and serve the community. Managers want to satisfy the shareholders and at the same time enjoy high salaries and expense accounts, and employees want to increase their take-home pay and benefits. When a decision is to be made—say, about an in- vestment project—some of these goals complement each other, whereas others conflict. Kearns (2004) described how the analytic hierarchy process (AHP) combined with in- teger programming, addresses multiple goals in evaluating information technology (IT) investments.

Many quantitative models of decision theory are based on comparing a single mea- sure of effectiveness, generally some form of utility to the decision maker. Therefore, it is usually necessary to transform a multiple-goal problem into a single-measure-of- effectiveness problem before comparing the effects of the solutions. This is a common method for handling multiple goals in an LP model.

Certain difficulties may arise when analyzing multiple goals:

• It is usually difficult to obtain an explicit statement of the organization’s goals. • The decision maker may change the importance assigned to specific goals over time

or for different decision scenarios. • Goals and subgoals are viewed differently at various levels of the organization and

within different departments. • Goals change in response to changes in the organization and its environment. • The relationship between alternatives and their role in determining goals may be

difficult to quantify. • Complex problems are solved by groups of decision makers, each of whom has a

personal agenda. • Participants assess the importance (priorities) of the various goals differently.

Several methods of handling multiple goals can be used when working with such situations. The most common ones are

• Utility theory • Goal programming • Expression of goals as constraints, using LP • A points system

Sensitivity Analysis

A model builder makes predictions and assumptions regarding input data, many of which deal with the assessment of uncertain futures. When the model is solved, the results de- pend on these data. Sensitivity analysis attempts to assess the impact of a change in the input data or parameters on the proposed solution (i.e., the result variable).

Sensitivity analysis is extremely important in prescriptive analytics because it al- lows flexibility and adaptation to changing conditions and to the requirements of dif- ferent decision-making situations, provides a better understanding of the model and the decision-making situation it attempts to describe, and permits the manager to input data to increase the confidence in the model. Sensitivity analysis tests relationships such as the following:

• The impact of changes in external (uncontrollable) variables and parameters on the outcome variable(s)

• The impact of changes in decision variables on the outcome variable(s)

488 Part III • Prescriptive Analytics and Big Data

• The effect of uncertainty in estimating external variables • The effects of different dependent interactions among variables • The robustness of decisions under changing conditions

Sensitivity analyses are used for:

• Revising models to eliminate too-large sensitivities • Adding details about sensitive variables or scenarios • Obtaining better estimates of sensitive external variables • Altering a real-world system to reduce actual sensitivities • Accepting and using the sensitive (and hence vulnerable) real world, leading to the

continuous and close monitoring of actual results

The two types of sensitivity analyses are automatic and trial and error.

AUTOMATIC SENSITIVITY ANALYSIS Automatic sensitivity analysis is performed in stan- dard quantitative model implementations such as LP. For example, it reports the range within which a certain input variable or parameter value (e.g., unit cost) can vary without having any significant impact on the proposed solution. Automatic sensitivity analysis is usually limited to one change at a time, and only for certain variables. However, it is powerful because of its ability to establish ranges and limits very fast (and with little or no additional computational effort). Sensitivity analysis is provided by Solver and almost all other software packages such as Lindo. Consider the MBI Corporation example intro- duced previously. Sensitivity analysis could be used to determine that if the right-hand side of the marketing constraint on CC-8 could be decreased by one unit, then the net profit would increase by $1,333.33. This is valid for the right-hand side decreasing to zero. Significant additional analysis is possible along these lines.

TRIAL-AND-ERROR SENSITIVITY ANALYSIS The impact of changes in any variable, or in several variables, can be determined through a simple trial-and-error approach. You change some input data and solve the problem again. When the changes are repeated several times, better and better solutions may be discovered. Such experimentation, which is easy to conduct when using appropriate modeling software, such as Excel, has two approaches: what-if analysis and goal seeking.

What-If Analysis

What-if analysis is structured as What will happen to the solution if an input variable, an assumption, or a parameter value is changed? Here are some examples:

• What will happen to the total inventory cost if the cost of carrying inventories in- creases by 10%?

• What will be the market share if the advertising budget increases by 5%?

With the appropriate user interface, it is easy for managers to ask a computer model these types of questions and get immediate answers. Furthermore, they can perform multiple cases and thereby change the percentage, or any other data in the question, as desired. The decision maker does all this directly, without a computer programmer.

Figure 8.10 shows a spreadsheet example of a what-if query for a cash flow prob- lem. When the user changes the cells containing the initial sales (from 100 to 120) and the sales growth rate (from 3% to 4% per quarter), the program immediately recomputes the value of the annual net profit cell (from $127 to $182). At first, initial sales were 100, growing at 3% per quarter, yielding an annual net profit of $127. Changing the initial sales cell to 120 and the sales growth rate to 4% causes the annual net profit to rise to $182. What-if analysis is common in many decision systems. Users are given the opportunity to

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 489

change their answers to some of the system’s questions, and a revised recommendation is found.

Goal Seeking

Goal seeking calculates the values of the inputs necessary to achieve a desired level of an output (goal). It represents a backward solution approach. The following are some examples of goal seeking:

• What annual R&D budget is needed for an annual growth rate of 15% by 2018? • How many nurses are needed to reduce the average waiting time of a patient in the

emergency room to less than 10 minutes?

An example of goal seeking is shown in Figure 8.11. For example, in a financial planning model in Excel, the internal rate of return (IRR) is the interest rate that produces a net present value (NPV) of zero. Given a stream of annual returns in Column E, we can compute the NPV of planned investment. By applying goal seeking, we can determine the internal rate of return where the NPV is zero. The goal to be achieved is NPV equal to zero, which determines the internal rate of return of this cash flow, including the invest- ment. We set the NPV cell to the value 0 by changing the interest rate cell. The answer is 38.77059%.

COMPUTING A BREAK-EVEN POINT BY USING GOAL SEEKING Some modeling software packages can directly compute break-even points, which is an important application of goal seeking. This involves determining the value of the decision variables (e.g., quantity to produce) that generate zero profit.

In many general applications programs, it can be difficult to conduct sensitivity analysis because the prewritten routines usually present only a limited opportunity for asking what-if questions. In a DSS, the what-if and the goal-seeking options must be easy to perform.

FIGURE 8.10 Example of a What-If Analysis Done in an Excel Worksheet.

490 Part III • Prescriptive Analytics and Big Data

u SECTION 8.7 REVIEW QUESTIONS

1. List some difficulties that may arise when analyzing multiple goals. 2. List the reasons for performing sensitivity analysis. 3. Explain why a manager might perform what-if analysis. 4. Explain why a manager might use goal seeking.

8.8 DECISION ANALYSIS WITH DECISION TABLES AND DECISION TREES

Decision situations that involve a finite and usually not too large number of alterna- tives are modeled through an approach called decision analysis (see Arsham, 2006a,b; Decision Analysis Society, decision-analysis.society.informs.org). Using this approach, the alternatives are listed in a table or a graph, with their forecasted contributions to the goal(s) and the probability of obtaining the contribution. These can be evaluated to select the best alternative.

Single-goal situations can be modeled with decision tables or decision trees. Multiple goals (criteria) can be modeled with several other techniques, described later in this chapter.

Decision Tables

Decision tables conveniently organize information and knowledge in a systematic, tabu- lar manner to prepare it for analysis. For example, say that an investment company is considering investing in one of three alternatives: bonds, stocks, or certificates of deposit (CDs). The company is interested in one goal: maximizing the yield on the investment after 1 year. If it were interested in other goals, such as safety or liquidity, the problem would be classified as one of multicriteria decision analysis (see Koksalan & Zionts, 2001).

The yield depends on the state of the economy sometime in the future (often called the state of nature), which can be in solid growth, stagnation, or inflation. Experts esti- mated the following annual yields:

FIGURE 8.11 Goal-Seeking Analysis.

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 491

• If there is solid growth in the economy, bonds will yield 12%, stocks 15%, and time deposits 6.5%.

• If stagnation prevails, bonds will yield 6%, stocks 3%, and time deposits 6.5%. • If inflation prevails, bonds will yield 3%, stocks will bring a loss of 2%, and time

deposits will yield 6.5%.

The problem is to select the one best investment alternative. These are assumed to be discrete alternatives. Combinations such as investing 50% in bonds and 50% in stocks must be treated as new alternatives.

The investment decision-making problem can be viewed as a two-person game (see Kelly, 2002). The investor makes a choice (i.e., a move), and then a state of nature occurs (i.e., makes a move). Table 8.3 shows the payoff of a mathematical model. The table includes decision variables (the alternatives), uncontrollable variables (the states of the economy; e.g., the environment), and result variables (the projected yield; e.g., out- comes). All the models in this section are structured in a spreadsheet framework.

If this were a decision-making problem under certainty, we would know what the economy would be and could easily choose the best investment. But that is not the case, so we must consider the two situations of uncertainty and risk. For uncertainty, we do not know the probabilities of each state of nature. For risk, we assume that we know the probabilities with which each state of nature will occur.

TREATING UNCERTAINTY Several methods are available for handling uncertainty. For example, the optimistic approach assumes that the best possible outcome of each alterna- tive will occur and then selects the best of the best (i.e., stocks). The pessimistic approach assumes that the worst possible outcome for each alternative will occur and selects the best of these (i.e., CDs). Another approach simply assumes that all states of nature are equally possible (see Clemen & Reilly, 2000; Goodwin & Wright, 2000; Kontoghiorghes, Rustem, & Siokos, 2002). Every approach for handling uncertainty has serious problems. Whenever possible, the analyst should attempt to gather enough information so that the problem can be treated under assumed certainty or risk.

TREATING RISK The most common method for solving this risk analysis problem is to select the alternative with the greatest expected value. Assume that experts estimate the chance of solid growth at 50%, the chance of stagnation at 30%, and the chance of inflation at 20%. The decision table is then rewritten with the known probabilities (see Table 8.3). An expected value is computed by multiplying the results (i.e., outcomes) by their respective probabilities and adding them. For example, investing in bonds yields an expected return of 12(0.5) + 6(0.3) + 3(0.2) = 8.4%.

This approach can sometimes be a dangerous strategy because the utility of each potential outcome may be different from the value. Even if there is an infinitesimal chance of a catastrophic loss, the expected value may seem reasonable, but the investor may not be willing to cover the loss. For example, suppose a financial advisor presents you with

TABLE 8.3 Investment Problem Decision Table Model

State of Nature (Uncontrollable Variables)

Alternative Solid Growth (%) Stagnation (%) Inflation (%)

Bonds 12.0 6.0 3.0

Stocks 15.0 3.0 –2.0

CDs 6.5 6.5 6.5

492 Part III • Prescriptive Analytics and Big Data

an “almost sure” investment of $1,000 that can double your money in one day, and then the advisor says, “Well, there is a 0.9999 probability that you will double your money, but unfortunately there is a 0.0001 probability that you will be liable for a $500,000 out-of- pocket loss.” The expected value of this investment is as follows:

0.9999 ($2,000 - $1,000) + .0001(-$500,000 - $1,000) = $999.90 - $50.10 = $949.80

The potential loss could be catastrophic for any investor who is not a billionaire. Depending on the investor’s ability to cover the loss, an investment has different ex- pected utilities. Remember that the investor makes the decision only once.

Decision Trees

An alternative representation of the decision table is a decision tree. A decision tree shows the relationships of the problem graphically and can handle complex situations in a compact form. However, a decision tree can be cumbersome if there are many al- ternatives or states of nature. TreeAge Pro (TreeAge Software Inc., treeage.com) and PrecisionTree (Palisade Corp., palisade.com) include powerful, intuitive, and sophisti- cated decision tree analysis systems. These vendors also provide excellent examples of decision trees used in practice. Note that the phrase decision tree has been used to de- scribe two different types of models and algorithms. In the current context, decision trees refer to scenario analysis. On the other hand, some classification algorithms in predictive analysis (see Chapters 4 and 5) are also called decision tree algorithms. The reader is ad- vised to note the difference between two different uses of the same name – decision tree.

A simplified investment case of multiple goals (a decision situation in which alter- natives are evaluated with several, sometimes conflicting, goals) is shown in Table 8.4. The three goals (criteria) are yield, safety, and liquidity. This situation is under assumed certainty; that is, only one possible consequence is projected for each alternative; the more complex cases of risk or uncertainty could be considered. Some of the results are qualitative (e.g., low, high) rather than numeric.

See Clemen and Reilly (2000), Goodwin and Wright (2000), and Decision Analysis Society (informs.org/Community/DAS) for more on decision analysis. Although doing so is quite complex, it is possible to apply mathematical programming directly to decision-making situations under risk. We discuss several other methods of treating risk later in the book. These include simulation, certainty factors, and fuzzy logic.

u SECTION 8.8 REVIEW QUESTIONS

1. What is a decision table? 2. What is a decision tree? 3. How can a decision tree be used in decision making? 4. Describe what it means to have multiple goals.

TABLE 8.4 Multiple Goals

Alternative Yield (%) Safety Liquidity

Bonds 8.4 High High

Stocks 8.0 Low High

CDs 6.5 Very high High

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 493

8.9 INTRODUCTION TO SIMULATION

In this section and the next we introduce a category of techniques that are used for support- ing decision making. Very broadly, these methods fall under the umbrella of simulation. Simulation is the appearance of reality. In decision systems, simulation is a technique for conducting experiments (e.g., what-if analyses) with a computer on a model of a manage- ment system. Strictly speaking, simulation is a descriptive rather than a prescriptive method. There is no automatic search for an optimal solution. Instead, a simulation model describes or predicts the characteristics of a given system under different conditions. When the values of the characteristics are computed, the best of several alternatives can be selected. The simulation process usually repeats an experiment many times to obtain an estimate (and a variance) of the overall effect of certain actions. For most situations, a computer simulation is appropriate, but there are some well-known manual simulations (e.g., a city police de- partment simulated its patrol car scheduling with a carnival game wheel).

Typically, real decision-making situations involve some randomness. Because many decision situations deal with semistructured or unstructured situations, reality is complex, which may not be easily represented by optimization or other models but can often be handled by simulation. Simulation is one of the most commonly used decision support methods. See Application Case 8.6 for an example. Application Case 8.7 illustrates the value of simulation in another setting where the problem complexity does not permit building a traditional optimization model.

Major Characteristics of Simulation

Simulation typically involves building a model of reality to the extent practical. Simulation models may suffer from fewer assumptions about the decision situation as compared to other prescriptive analytic models. In addition, simulation is a technique for conducting experiments. Therefore, it involves testing specific values of the decision or uncontrol- lable variables in the model and observing the impact on the output variables.

Finally, simulation is normally used only when a problem is too complex to be treated using numerical optimization techniques. Complexity in this situation means either that the problem cannot be formulated for optimization (e.g., because the assumptions do not hold), that the formulation is too large, that there are too many interactions among the variables, or that the problem is stochastic in nature (i.e., exhibits risk or uncertainty).

A steel manufacturing plant produces rolled-steel tubes for different industries across the country. They build tubes based on a customer’s require- ments and specifications. Maintaining high-quality norms and timely delivery of products are two of the foremost important criteria for this steel tub- ing plant. The plant views its manufacturing sys- tem as a sequence of operations where it unrolls steel from one reel and rolls it onto a different reel. This happens once the forming, welding, editing,

or inspecting operation is finished. The ultimate product would be a reel of rolled steel tubing that weighs about 20 tons. The reel is then shipped to the customer.

A key challenge for management is to be able to predict the appropriate delivery date for an order, and its impact on the currently planned production schedule. Given the complexity of the produc- tion process, it is not easy to develop an optimi- zation model in Excel or other software to build a

Application Case 8.7 Steel Tubing Manufacturer Uses a Simulation-Based Production Scheduling System

(Continued )

494 Part III • Prescriptive Analytics and Big Data

Advantages of Simulation

Simulation is used in decision support modeling for the following reasons:

• The theory is fairly straightforward. • A great amount of time compression can be attained, quickly giving a manager some

feel as to the long-term (1@ to 10@ year) effects of many policies. • Simulation is descriptive rather than normative. This allows the manager to pose

what-if questions. Managers can use a trial-and-error approach to problem solving and can do so faster, at less expense, more accurately, and with less risk.

• A manager can experiment to determine which decision variables and which parts of the environment are really important, and with different alternatives.

• An accurate simulation model requires an intimate knowledge of the problem, thus forcing the model builder to constantly interact with the manager. This is desirable

production schedule (see Application Case 8.1). The issue is that these tools fail to capture key planning issues such as employee schedules and qualifica- tions, material accessibility, material allocation com- plication, and random aspects of the operation.

Methodology/Solution

When traditional modeling methods do not capture the problem subtleties or complexities, a simula- tion model could perhaps be built. The predictive analysis approach uses a versatile Simio simula- tion model that takes into consideration all the operational complexity, manufacturing material matching algorithms, and deadline considerations. Also, Simio’s service offering, known as risk-based planning and scheduling (RPS), provides some user interfaces and reports simply designed for production management. This gives the client the ability to explore the impact of a new order on their production plan and schedule within about 10 minutes.

Results/Benefits

Such models provide significant visibility into the production schedule. The risk-based planning and scheduling system should be able to warn the master scheduler that a specific order has a chance of being delivered late. Changes could also be made sooner to rectify issues with an order. Success for this steel tubing manufacturer is directly tied to product qual- ity and on-time delivery. By exploitation of Simio’s

predictive RPS offering, the plant expects improved market share.

Questions for DisCussion

1. Explain the advantages of using Simio’s simula- tion model over traditional methods.

2. In what ways has the predictive analysis approach helped management achieve the goals of analyz- ing the production schedules?

3. Besides the steel manufacturing industry, in what other industries could such a modeling approach help improve quality and service?

What Can We Learn from This Application Case?

By using Simio’s simulation model, the manufacturing plant made better decisions in assessment of opera- tions, taking all of the problem issues into consider- ation. Thus, a simulation-based production scheduling system could derive higher returns and market share for the steel tubing manufacturer. Simulation is an important technique for prescriptive analytics.

Compiled from Arthur, Molly. “Simulation-Based Production Scheduling System.” www.simio.com, Simio LLC, 2014, www. simio.com/case-studies/A-Steel-Tubing-Manufacturer- Expects-More-Market-Share/A-Steel-Tubing-Manufacturer- Expects-More-Market-Share.pdf (accessed September 2018); “Risk-Based Planning and Scheduling (RPS) with Simio.” www.simio. com, Simio LLC, www.simio.com/about-simio/why-simio/ simio-RPS-risk-based-planning-and-scheduling.php (accessed September 2018).

Application Case 8.7 (Continued)

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 495

for DSS development because the developer and manager both gain a better under- standing of the problem and the potential decisions available.

• The model is built from the manager’s perspective. • The simulation model is built for one particular problem and typically cannot solve

any other problem. Thus, no generalized understanding is required of the manager; every component in the model corresponds to part of the real system.

• Simulation can handle an extremely wide variety of problem types, such as inven- tory and staffing, as well as higher-level managerial functions, such as long-range planning.

• Simulation generally can include the real complexities of problems; simplifications are not necessary. For example, simulation can use real probability distributions rather than approximate theoretical distributions.

• Simulation automatically produces many important performance measures. • Simulation is often the only DSS modeling method that can readily handle relatively

unstructured problems. • Some relatively easy-to-use simulation packages (e.g., Monte Carlo simulation) are

available. These include add-in spreadsheet packages (e.g., @RISK), influence dia- gram software, Java-based (and other Web development) packages, and the visual interactive simulation systems to be discussed shortly.

Disadvantages of Simulation

The primary disadvantages of simulation are as follows:

• An optimal solution cannot be guaranteed, but relatively good ones are generally found.

• Simulation model construction can be a slow and costly process, although newer modeling systems are easier to use than ever.

• Solutions and inferences from a simulation study are usually not transferable to other problems because the model incorporates unique problem factors.

• Simulation is sometimes so easy to explain to managers that analytic methods are often overlooked.

• Simulation software sometimes requires special skills because of the complexity of the formal solution method.

The Methodology of Simulation

Simulation involves setting up a model of a real system and conducting repetitive experi- ments on it. The methodology consists of the following steps, as shown in Figure 8.12:

1. Define the problem. We examine and classify the real-world problem, specifying why a simulation approach is appropriate. The system’s boundaries, environment, and other such aspects of problem clarification are handled here.

2. Construct the simulation model. This step involves determination of the vari- ables and their relationships, as well as data gathering. Often the process is de- scribed by using a flowchart, and then a computer program is written.

3. Test and validate the model. The simulation model must properly represent the system being studied. Testing and validation ensure this.

4. Design the experiment. When the model has been proven valid, an experiment is designed. Determining how long to run the simulation is part of this step. There are two important and conflicting objectives: accuracy and cost. It is also prudent to identify typical (e.g., mean and median cases for random variables), best-case (e.g., low-cost, high-revenue), and worst-case (e.g., high-cost, low-revenue) scenarios.

496 Part III • Prescriptive Analytics and Big Data

These help establish the ranges of the decision variables and environment in which to work and also assist in debugging the simulation model.

5. Conduct the experiment. Conducting the experiment involves issues ranging from random-number generation to result presentation.

6. Evaluate the results. The results must be interpreted. In addition to standard statistical tools, sensitivity analyses can also be used.

7. Implement the results. The implementation of simulation results involves the same issues as any other implementation. However, the chances of success are bet- ter because the manager is usually more involved with the simulation process than with other models. Higher levels of managerial involvement generally lead to higher levels of implementation success.

Banks and Gibson (2009) presented some useful advice about simulation practices. For example, they list the following seven issues as the common mistakes committed by simulation modelers. The list, though not exhaustive, provides general directions for professionals working on simulation projects.

• Focusing more on the model than on the problem • Providing point estimates • Not knowing when to stop • Reporting what the client wants to hear rather than what the model results say • Lack of understanding of statistics • Confusing cause and effect • Failure to replicate reality

In a follow-up article they provide additional guidelines. You should consult this article: analytics-magazine.org/spring-2009/205-software-solutions-the-abcs-of-simulation- practice.html.

Simulation Types

As we have seen, simulation and modeling are used when pilot studies and experiment- ing with real systems are expensive or sometimes impossible. Simulation models allow us to investigate various interesting scenarios before making any investment. In fact, in simulations, the real-world operations are mapped into the simulation model. The model consists of relationships and, consequently, equations that all together present the

Do over/Feedback

Define the problem

Construct the simulation

model

Test and validate the

model

Design and conduct the experiments

Evaluate the experiments’

results

Implement the results

Change the real-world problem

Real-world problem

FIGURE 8.12 The Process of Simulation.

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 497

real-world operations. The results of a simulation model, then, depend on the set of pa- rameters given to the model as inputs.

There are various simulation paradigms such as Monte Carlo simulation, discrete event, agent based, or system dynamics. One of the factors that determine the type of simulation technique is the level of abstraction in the problem. Discrete events and agent- based models are usually used for middle or low levels of abstraction. They usually con- sider individual elements such as people, parts, and products in the simulation models, whereas systems dynamics is more appropriate for aggregate analysis.

In the following section, we introduce the major types of simulation: probabilistic simulation, time-dependent and time-independent simulation, and visual simulation. There are many other simulation techniques such as system dynamics modeling, and agent- based modeling. As has been noted before, the goal here is to make you aware of the potential of some of these techniques as opposed to make you an expert in using them.

PROBABILISTIC SIMULATION In probabilistic simulation, one or more of the indepen- dent variables (e.g., the demand in an inventory problem) are probabilistic. They follow certain probability distributions, which can be either discrete distributions or continuous distributions:

• Discrete distributions involve a situation with a limited number of events (or vari- ables) that can take on only a finite number of values.

• Continuous distributions are situations with unlimited numbers of possible events that follow density functions, such as the normal distribution.

The two types of distributions are shown in Table 8.5.

TIME-DEPENDENT VERSUS TIME-INDEPENDENT SIMULATION Time-independent refers to a situation in which it is not important to know exactly when the event occurred. For ex- ample, we may know that the demand for a certain product is three units per day, but we do not care when during the day the item is demanded. In some situations, time may not be a factor in the simulation at all, such as in steady-state plant control design. However, in waiting-line problems applicable to e-commerce, it is important to know the precise time of arrival (to know whether the customer will have to wait). This is a time-dependent situation.

Monte Carlo Simulation

In most business decision problems, we usually employ one of the following two types of probabilistic simulations. The most common simulation method for business decision problems is the Monte Carlo simulation. This method usually begins with building a

TABLE 8.5 Discrete versus Continuous Probability Distributions

Daily Demand Discrete Probability Continuous Probability

5 0.10 Daily demand is normally distributed with a mean of 7 and a standard deviation of 1.2

6 0.15

7 0.30

8 0.25

9 0.20

498 Part III • Prescriptive Analytics and Big Data

model of the decision problem without having to consider the uncertainty of any vari- ables. Then we recognize that certain parameters or variables are uncertain or follow an assumed or estimated probability distribution. This estimation is based on analysis of past data. Then we begin running sampling experiments. Running sampling experiments consists of generating random values of uncertain parameters and then computing val- ues of the variables that are impacted by such parameters or variables. These sampling experiments essentially amount to solving the same model hundreds or thousands of times. We can then analyze the behavior of these dependent or performance variables by examining their statistical distributions. This method has been used in simulations of physical as well as business systems. A good public tutorial on the Monte Carlo simulation method is available on Palisade.com (http://www.palisade.com/risk/monte_carlo_ simulation.asp). Palisade markets a tool called @RISK, a popular spreadsheet-based Monte Carlo simulation software. Another popular software in this category is Crystal Ball, now marketed by Oracle as Oracle Crystal Ball. Of course, it is also possible to build and run Monte Carlo experiments within an Excel spreadsheet without using any add- on software such as the two just mentioned. But these tools make it more convenient to run such experiments in Excel-based models. Monte Carlo simulation models have been used in many commercial applications. Examples include Procter & Gamble using these models to determine hedging foreign-exchange risks; Lilly using the model for deciding optimal plant capacity; Abu Dhabi Water and Electricity Company using @Risk for fore- casting water demand in Abu Dhabi; and literally thousands of other actual case studies. Each of the simulation software companies’ Web sites include many such success stories.

Discrete Event Simulation

Discrete event simulation refers to building a model of a system where the interaction between different entities is studied. The simplest example of this is a shop consisting of a server and customers. By modeling the customers arriving at various rates and the server serving at various rates, we can estimate the average performance of the system, waiting time, the number of waiting customers, and so on. Such systems are viewed as collections of customers, queues, and servers. There are thousands of documented applications of discrete event simulation models in engineering, business, and so on. Tools for building discrete event simulation models have been around for a long time, but these have evolved to take advan- tage of developments in graphical capabilities for building and understanding the results of such simulation models. We will discuss this modeling method further in the next section. Application Case 8.8 gives an example of the use of such simulation in analyzing complexities of a supply chain that uses a visual simulation to be described in the next section.

Introduction

Cosan is a Brazil-based conglomerate that operates globally. One of its major activities is to grow and process sugar cane. Besides being a major source of sugar, sugar cane is now a major source of ethanol, a main ingredient in renewable energy. Because of the growing demand for renewable energy, etha- nol production has become such a major activity for

Cosan that it now operates two refineries in addi- tion to 18 production plants, and of course, mil- lions of hectares of sugar cane farms. According to recent data, it processed over 44 million tons of sugar cane, produced over 1.3 billion liters of etha- nol, and produced 3.3 million tons of sugar. As one might imagine, operations of this scale lead to com- plex supply chains. So the logistics team was asked

Application Case 8.8 Cosan Improves Its Renewable Energy Supply Chain Using Simulation

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 499

to make recommendations to the senior manage- ment to:

• Determine the optimum number of vehicles re- quired in a fleet used to transport sugar cane to processing mills to preserve capital.

• Propose how to increase the actual capacity of sugar cane received at the sugar mills.

• Identify the production bottleneck problems to solve to improve the flow of sugar cane.

Methodology/Solution

The logistics team worked with Simio software and built a complex simulation model of the Cosan supply chain as it pertains to these issues. According to a Simio brief, “Over the course of three months, newly hired engineers collected data in the field and received hands-on training and modeling assistance from Paragon Consulting of San Palo.”

To model agricultural operations to analyze the sugar cane’s postharvest journey to produc- tion mills, the model objectives included details of the fleet of road transport sugar cane crop to Unity Costa Pinto, the actual capacity of reception of cane sugar mills, bottlenecks and points for improvement in the flow of CCT (cut-load-haul) of cane sugar, and so on.

The model parameters are as follows:

Input Variables: 32 Output Variables: 39 Auxiliary Variables: 92 Variable Entities: 8 Input Tables: 19 Simulated Days: 240 (1st season) Number of Entities: 12 (10 harvester composi- tional types for transport of sugar cane)

Results/Benefits

Analyses produced by these Simio models provided a good view of the risk of operation over the 240- day period due to various uncertainties. By analyz- ing the various bottlenecks and ways to mitigate those scenarios, the company was able to make bet- ter decisions and save over $500,000 from this mod- eling effort alone.

Questions for DisCussion

1. What type of supply chain disruptions might occur in moving the sugar cane from the field to the production plants to develop sugar and ethanol?

2. What types of advanced planning and prediction might be useful in mitigating such disruptions?

What Can We Learn from This Application Case?

This short application story illustrates the value of applying simulation to a problem where it might be difficult to build an optimization model. By incorpo- rating a discrete event simulation model and visual interactive simulation (VIS), one can visualize the impact of interruptions in supply chain due to fleet failure, unexpected downtime at the plant, and so on, and come up with planned corrections.

Sources: Compiled from Wikipedia contributors, Cosan, Wikipedia, The Free Encyclopedia, https://en.wikipedia.org/w/index. php?title=Cosan&oldid=713298536 (accessed July 10, 2016); Agricultural Operations Simulation Case Study: Cosan, http:// www.simio.com/case-studies/Cosan-agricultural-logistics- simulation-software-case-study/agricultural-simulation- software-case-study-video-cosan.php (accessed July 2016); Cosan Case Study: Optimizing agricultural logistics operations, http://www.simio.com/case-studies/Cosan-agricultural- logistics-simulation-software-case-study/index.php (accessed July 2016).

u SECTION 8.9 REVIEW QUESTIONS

1. List the characteristics of simulation. 2. List the advantages and disadvantages of simulation. 3. List and describe the steps in the methodology of simulation. 4. List and describe the types of simulation.

500 Part III • Prescriptive Analytics and Big Data

8.10 VISUAL INTERACTIVE SIMULATION

We next examine methods that show a decision maker a representation of the decision- making situation in action as it runs through scenarios of the various alternatives. These powerful methods overcome some of the inadequacies of conventional methods and help build trust in the solution attained because they can be visualized directly.

Conventional Simulation Inadequacies

Simulation is a well-established, useful, descriptive, mathematics-based method for gain- ing insight into complex decision-making situations. However, simulation does not usu- ally allow decision makers to see how a solution to a complex problem evolves over (compressed) time, nor can decision makers interact with the simulation (which would be useful for training purposes and teaching). Simulation generally reports statistical re- sults at the end of a set of experiments. Decision makers are thus not an integral part of simulation development and experimentation, and their experience and judgment cannot be used directly. If the simulation results do not match the intuition or judgment of the decision maker, a confidence gap in the results can occur.

Visual Interactive Simulation

Visual interactive simulation (VIS), also known as visual interactive modeling (VIM) and visual interactive problem solving, is a simulation method that lets decision makers see what the model is doing and how it interacts with the decisions made, as they are made. This technique has been used with great success in operations analysis in many fields such as supply chain and healthcare. The user can employ his or her knowl- edge to determine and try different decision strategies while interacting with the model. Enhanced learning, about both the problem and the impact of the alternatives tested, can and does occur. Decision makers also contribute to model validation. Decision makers who use VIS generally support and trust their results.

VIS uses animated computer graphic displays to present the impact of different managerial decisions. It differs from regular graphics in that the user can adjust the decision-making process and see results of the intervention. A visual model is a graphic used as an integral part of decision making or problem solving, not just as a communica- tion device. Some people respond better than others to graphical displays, and this type of interaction can help managers learn about the decision-making situation.

VIS can represent static or dynamic systems. Static models display a visual image of the result of one decision alternative at a time. Dynamic models display systems that evolve over time, and the evolution is represented by animation. The latest visual simula- tion technology has been coupled with the concept of virtual reality, where an artificial world is created for a number of purposes, from training to entertainment to viewing data in an artificial landscape. For example, the U.S. military uses VIS systems so that ground troops can gain familiarity with terrain or a city to very quickly orient themselves. Pilots also use VIS to gain familiarity with targets by simulating attack runs. The VIS software can also include GIS coordinates.

Visual Interactive Models and DSS

VIM in DSS has been used in several operations management decisions. The method consists of priming (like priming a water pump) a visual interactive model of a plant (or company) with its current status. The model then runs rapidly on a computer, allowing managers to observe how a plant is likely to operate in the future.

Waiting-line management (queuing) is a good example of VIM. Such a DSS usu- ally computes several measures of performance for the various decision alternatives

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 501

(e.g.,  waiting time in the system). Complex waiting-line problems require simulation. VIM can display the size of the waiting line as it changes during the simulation runs and can also graphically present the answers to what-if questions regarding changes in input variables. Application Case 8.9 gives an example of a visual simulation that was used to explore the applications of radio-frequency identification (RFID) technology in develop- ing new scheduling rules in a manufacturing setting.

The VIM approach can also be used in conjunction with artificial intelligence. Integration of the two techniques adds several capabilities that range from the ability to build systems graphically to learning about the dynamics of the system. These systems, especially those developed for the military and the video-game industry, have “thinking” characters who can behave with a relatively high level of intelligence in their interactions with users.

Simulation Software

Hundreds of simulation packages are available for a variety of decision-making situations. Many run as Web-based systems. ORMS Today publishes a periodic review of simulation software. One recent review (current as of October 2018) is located at https://www.in- forms.org/ORMS-Today/Public-Articles/October-Volume-44-Number-5/Simulation- Software-Survey-Simulation-new-and-improved-reality-show (accessed November 2018). PC software packages include Analytica (Lumina Decision Systems, lumina.com) and the Excel add-ins Crystal Ball (now sold by Oracle as Oracle Crystal Ball, oracle.com) and @RISK (Palisade Corp., palisade.com). A major commercial software for discrete event sim- ulation has been Arena (sold by Rockwell Intl., arenasimulation.com). Original developers of Arena have now developed Simio (simio.com), a user-friendly VIS software. Another popular discrete event VIS software is ExtendSim (extendsim.com). SAS has a graphical analytics software package called JMP that also includes a simulation component in it.

A manufacturing services provider of complex opti- cal and electromechanical components seeks to gain efficiency in its job-shop scheduling decision because the current shop-floor operations suffer from a few issues:

• There is no system to record when the work- in-process (WIP) items actually arrive at or leave operating workstations and how long those WIPs actually stay at each workstation.

• The current system cannot monitor or keep track of the movement of each WIP in the pro- duction line in real time.

As a result, the company is facing two main issues at this production line: high backlogs and high costs of overtime to meet the demand. In addi- tion, the upstream cannot respond to unexpected incidents such as changes in demand or material shortages quickly enough and revise schedules in a cost-effective manner. The company is considering

implementing RFID on a production line. However, the company does not know if going to this major expense of adding RFID chips on production boxes, installing RFID readers throughout the production line, and of course, the systems to process this information will result in any real gains. So one question is to explore any new production sched- uling changes that may result by investing in RFID infrastructure.

Methodology

Because exploring the introduction of any new system in the physical production system can be extremely expensive or even disruptive, a discrete event simulation model was developed to exam- ine how tracking and traceability through RFID can facilitate job-shop production scheduling activities. A visibility-based scheduling (VBS) rule that utilizes the real-time traceability systems to track those WIPs, parts and components, and raw

Application Case 8.9 Improving Job-Shop Scheduling Decisions through RFID: A Simulation-Based Assessment

(Continued )

502 Part III • Prescriptive Analytics and Big Data

materials in shop-floor operations was proposed. A simulation approach was applied to examine the benefit of the VBS rule against the classical scheduling rules: the first-in-first-out and earliest due date dispatching rules. The simulation model was developed using Simio. Simio is a 3@D simu- lation modeling software package that employs an object-oriented approach to modeling and has recently been used in many areas such as facto- ries, supply chains, healthcare, airports, and ser- vice systems.

Figure 8.13 presents a screenshot of the Simio interface panel of this production line. The param- eter estimates used for the initial state in the simu- lation model include weekly demand and forecast, process flow, number of workstations, number of shop-floor operators, and operating time at each workstation. In addition, parameters of some of the input data such as RFID tagging time, informa- tion retrieving time, or system updating time are estimated from a pilot study and from the subject

matter experts. Figure 8.14 presents the process view of the simulation model where specific sim- ulation commands are implemented and coded. Figures 8.15 and 8.16 present the standard report view and pivot grid report of the simulation model. The standard report and pivot grid format provide a very quick method to find specific statistical results such as average, percent, total, maximum, or mini- mum values of variables assigned and captured as an output of the simulation model.

Results

The results of the simulation suggest that an RFID- based scheduling rule generates better performance compared to traditional scheduling rules with regard to processing time, production time, resource uti- lization, backlogs, and productivity. The company can take these productivity gains and perform cost/ benefit analyses in making the final investment decisions.

Application Case 8.9 (Continued)

FIGURE 8.13 Simio Interface View of the Simulation System. Source: Used with permission from Simio LLC.

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 503

FIGURE 8.15 Standard Report View. Source: Used with permission from Simio LLC.

FIGURE 8.14 Process View of the Simulation Model. Source: Used with permission from Simio LLC.

(Continued )

504 Part III • Prescriptive Analytics and Big Data

u SECTION 8.10 REVIEW QUESTIONS

1. Define visual simulation and compare it to conventional simulation. 2. Describe the features of VIS (i.e., VIM) that make it attractive for decision makers. 3. How can VIS be used in operations management? 4. How is an animated film like a VIS application?

Questions for DisCussion

1. In situations such as what this case depicts, what other approaches can one take to analyze invest- ment decisions?

2. How would one save time if an RFID chip can tell the exact location of a product in process?

3. Research to learn about the applications of RFID sensors in other settings. Which one do you find most interesting?

Source: Based on Chongwatpol, J., & Sharda, R. (2013). RFID-enabled track and traceability in job-shop scheduling environment. European Journal of Operational Research, 227(3), 453–463, http://dx.doi.org/10.1016/j.ejor.2013.01.009.

For information about simulation software, see the Society for Modeling and Simulation International (scs.org) and the annual software sur- veys at ORMS Today (https://www.informs.org/ ORMS-Today/).

Application Case 8.9 (Continued)

FIGURE 8.16 Pivot Grid Report from a Simio Run. Source: Used with permission from Simio LLC.

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 505

Chapter Highlights

• Models play a major role in DSS because they are used to describe real decision-making situations. There are several types of models.

• Models can be static (i.e., a single snapshot of a situation) or dynamic (i.e., multiperiod).

• Analysis is conducted under assumed certainty (which is most desirable), risk, or uncertainty (which is least desirable).

• Influence diagrams graphically show the inter- relationships of a model. They can be used to enhance the use of spreadsheet technology.

• Spreadsheets have many capabilities, includ- ing what-if analysis, goal seeking, program- ming, database management, optimization, and simulation.

• Decision tables and decision trees can model and solve simple decision-making problems.

• Mathematical programming is an important opti- mization method.

• LP is the most common mathematical program- ming method. It attempts to find an optimal allocation of limited resources under organiza- tional constraints.

• The major parts of an LP model are the objec- tive function, the decision variables, and the constraints.

• Multicriteria decision-making problems are diffi- cult but not impossible to solve.

• What-if and goal seeking are the two most com- mon methods of sensitivity analysis.

• Many DSS development tools include built-in quantitative models (e.g., financial, statistical) or can easily interface with such models.

• Simulation is a widely used DSS approach that involves experimentation with a model that rep- resents the real decision-making situation.

• Simulation can deal with more complex situations than optimization, but it does not guarantee an optimal solution.

• There are many different simulation methods. Some that are important for decision making in- clude Monte Carlo simulation and discrete event simulation.

• VIS/VIM allows a decision maker to interact di- rectly with a model and shows results in an easily understood manner.

Key Terms

certainty decision analysis decision table decision tree decision variable discrete event simulation dynamic models environmental scanning and analysis forecasting goal seeking influence diagram

intermediate result variable linear programming (LP) mathematical programming Monte Carlo simulation multidimensional analysis

(modeling) multiple goals optimal solution parameter quantitative model result (outcome) variable

risk risk analysis sensitivity analysis simulation static models uncertainty uncontrollable variable visual interactive modeling (VIM) visual interactive simulation (VIS) what-if analysis

Questions for Discussion

1. How does prescriptive analytics relate to descriptive and predictive analytics?

2. Explain the differences between static and dynamic models. How can one evolve into the other?

3. What is the difference between an optimistic approach and a pessimistic approach to decision making under assumed uncertainty?

4. Explain why solving problems under uncertainty some- times involves assuming that the problem is to be solved under conditions of risk.

5. Excel is probably the most popular spreadsheet soft- ware for PCs. Why? What can we do with this package that makes it so attractive for modeling efforts?

506 Part III • Prescriptive Analytics and Big Data

6. Explain how decision trees work. How can a complex problem be solved by using a decision tree?

7. Explain how LP can solve allocation problems. 8. What are the advantages of using a spreadsheet package to

create and solve LP models? What are the disadvantages? 9. What are the advantages of using an LP package to cre-

ate and solve LP models? What are the disadvantages? 10. What is the difference between decision analysis with

a single goal and decision analysis with multiple goals (i.e., criteria)? Explain the difficulties that may arise when analyzing multiple goals.

11. Explain how multiple goals can arise in practice. 12. Compare and contrast what-if analysis and goal seeking. 13. Describe the general process of simulation. 14. List some of the major advantages of simulation over

optimization and vice versa. 15. Many computer games can be considered visual simula-

tion. Explain why. 16. Explain why VIS is particularly helpful in implementing

recommendations derived by computers.

Exercises

Teradata University Network (TUN) and Other Hands-on Exercises

1. Explore teradatauniversitynetwork.com, and deter- mine how models are used in the BI cases and papers.

2. Create the spreadsheet models shown in Figures 8.3 and 8.4. a. What is the effect of a change in the interest rate

from 8% to 10% in the spreadsheet model shown in Figure 8.3?

b. For the original model in Figure 8.3, what interest rate is required to decrease the monthly payments by 20% ? What change in the loan amount would have the same effect?

c. In the spreadsheet shown in Figure 8.4, what is the effect of a prepayment of $200 per month? What prepayment would be necessary to pay off the loan in 25 years instead of 30 years?

3. Solve the MBI product-mix problem described in this chapter, using either Excel’s Solver or a student version of an LP solver, such as Lindo. Lindo is available from Lindo Systems, Inc., at lindo.com; others are also available— search the Web. Examine the solution (output) reports for the answers and sensitivity report. Did you get the same results as reported in this chapter? Try the sensitivity

analysis outlined in the chapter; that is, lower the right- hand side of the CC-8 marketing constraint by one unit, from 200 to 199. What happens to the solution when you solve this modified problem? Eliminate the CC-8 lower- bound constraint entirely (this can be done easily by either deleting it in Solver or setting the lower limit to zero) and re-solve the problem. What happens? Using the original formulation, try modifying the objective function coefficients and see what happens.

4. Investigate via a Web search how models and their solutions are used by the U.S. Department of Homeland Security in the “war against terrorism.” Also investigate how other governments or government agencies are using models in their missions.

5. This problem was contributed by Dr. Rick Wilson of Oklahoma State University.

The recent drought has hit farmers hard. Cows are eating candy corn!

You are interested in creating a feed plan for the next week for your cattle using the following seven non- traditional feeding products: Chocolate Lucky Charms cereal, Butterfinger bars, Milk Duds, vanilla ice cream, Cap’n Crunch cereal, candy corn (because the real corn is all dead), and Chips Ahoy cookies.

Choc Lucky Charms Butterfinger Milk Duds

Vanilla Ice Cream

Cap’n Crunch

Candy Corn

Chips Ahoy

$$/lb 2.15 7 4.25 6.35 5.25 4 6.75

Choc YES YES YES NO NO NO YES

Protein 75 80 45 65 72 26 62

TDN 12 20 18 6 11 8 12

Calcium 3 4 4.5 12 2 1 5

Their per pound cost is shown, as is the protein units per pound they contribute, the total digestible nutrients (TDN) they contribute per pound, and the calcium units per pound.

You estimate that the total amount of nontradi- tional feeding products contribute the following amount of nutrients: at least 20,000 units of protein, at least

4,025  units of TDN, at least 1,000 but no more than 1,200 units of calcium.

There are some other miscellaneous requirements as well. • The chocolate in your overall feed plan (in pounds)

cannot exceed the amount of nonchocolate pound- age. Whether a product is considered chocolate

Chapter 8 • Prescriptive Analytics: Optimization and Simulation 507

or not is shown in the table (YES = chocolate, NO = not chocolate).

• No one feeding product can make up more than 25% of the total pounds needed to create an acceptable feed mix.

• There are two cereals (Chocolate Lucky Charms and Cap’n Crunch). Combined, they can be no more than 40% (in pounds) of the total mix required to meet the mix requirements.

Determine the optimal levels of the seven prod- ucts to create your weekly feed plan that minimizes cost. Note that all amounts of products must not have frac- tional values (whole numbered pounds only).

6. This exercise was also contributed by Dr. Rick Wilson of Oklahoma State University to illustrate the modeling capabilities of Excel Solver.

National signing day for rugby recruiting sea- son 2018 has been completed. Now, as the recruiting coordinator for the San Diego State University Aztec rugby team, it is time to analyze the results and plan for 2019.

You’ve developed complex analytics and data col- lection processes and applied them for the past few recruit- ing seasons to help you develop a plan for 2019. Basically, you have divided the area in which you actively recruit rugby players into eight different regions. Each region has a per-target cost, a “star rating” (average recruit “star” rank- ing, from 0 to 5, similar to what Rivals uses for football), a yield or acceptance rate percentage (the percentage of targeted recruits who come to SDSU), and a visibility mea- sure, which represents a measure of how much publicity SDSU gets for recruiting in that region, measured per target (increased visibility will enhance future recruiting efforts).

Cost/target avg star rating

acceptance rate %

visibility per target

Region1 125 3 40 0

Region2 89 2.5 42 0

Region3 234 3.25 25 2

Region4 148 3.1 30 3

Region5 321 3.5 22 7

Region6 274 3.45 20 4

Region7 412 3.76 17 5

Region8 326 3.2 18 5.5

Your goal is to create a LINEAR mathematical model that determines the number of target recruits you should pur- sue in each region in order to have an estimated yield (expected number) of at least 25 rugby recruits for next year while minimizing cost. (Region 1 with yield of 40%: if we target 10 people, the expected number that will come is .4 * 10 = 40 .)

In determining the optimal number of targets in each region (which, not surprisingly, should be integer values), you must also satisfy the following conditions:

• No more than 20% of the total targets (not the expected number of recruits) should be from any one region.

• Each region should have at least 4% of the total tar- gets (again, not the expected number of recruits, but the number of targets).

• The average star rating of the targets must be at least equal to 3.3.

• The average visibility value of the targets must be at least equal to 3.5.

• Off on the recruiting trail you go!

7. This exercise was also contributed by Dr. Rick Wilson of Oklahoma State University.

You are the Water Resources Manager for Thirstiville, OK, and are working out the details for next year’s con- tracts with three different entities to supply water to your town. Each water source (A, B, C) provides water of differ- ent quality. The quality assessment is aggregated together in two values P1 and P2, representing a composite of con- taminants, such as THMs, HAAs, and so on. The sources each have a maximum of water that they can provide (measured in thousands of gallons), a minimum that we must purchase from them, and a per-thousand-gallon cost.

MIN MAX P1 P2 COST

Source A 400 1000 4 1 0.25

Source B 1000 2500 3.5 3 0.175

Source C 0 775 5 2.5 0.20

On the product end, you must procure water such that you can provide three distinct water products for next year (this is all being done at the aggregate “city” level). You must provide drinking water to the city, and then water to two different wholesale clients (this is com- monly done by municipalities). The table below shows requirements for these three products, and the “sales” or revenue that you get from each customer (by thousand gallons, same scale as the earlier cost).

For each of the three water products/customers, MIN is the minimum that we have to provide to each, MAX is the maximum that we can provide (it is reasonable to be provided with a targeted range of product to pro- vide to our customers), the maximum P1 and P2 weighted average for the water blended together for each quality “category” (the contaminants) per customer, and the sales price.

MIN MAX P1 P2 SALES

Drinking 1500 1700 3.75 2.25 0.35

WSale 1 250 325 No Req. 2.75 0.4

WSale 2 No limit No limit 4 2 0.425

Yes, the second wholesale customer (WSale 2) will take as much water as you can blend together for them.

Obviously, water from all three sources will need to be blended together to meet the Thirstiville custom- er requirements. There is one more requirement: for each of the three products (drinking water and the two wholesale clients), Source A and Source B both individ- ually (yes, separately) must make up at least 20% of the

508 Part III • Prescriptive Analytics and Big Data

total amount of the production of that particular water type. We do not have such a requirement for Source C.

Create an appropriate LP model that determines how to meet customer water demand for next year while maximizing profit (sales less costs). Summarize your

results (something more than telepathy—say, some sort of table of data beyond the model solution?) It must use words ( ) and indicate how much water we should promise to buy from our three sources. Integers are not required.

References

“Canadian Football League Uses Frontline Solvers to Optimize Scheduling in 2016.” www.solver.com/news/canadian- football-league-uses-frontline-solvers-optimize- scheduling-2016 (accessed September 2018).

“Risk-Based Planning and Scheduling (RPS) with Simio.” Www.simio.com, Simio LLC, www.simio.com/about- simio/why-simio/simio-RPS-risk-based-planning- and-scheduling.php (accessed September 2018).

Arsham, H. (2006a). “Modeling and Simulation Resources.” home.ubalt.edu/ntsbarsh/Business-stat/RefSim.htm (accessed November 2018).

Arsham, H. (2006b). “Decision Science Resources.” home.ubalt .edu/ntsbarsh/Business-stat/Refop.htm (accessed No- vember 2018).

Arthur, Molly. “Simulation-Based Production Scheduling System.” www.simio.com, Simio LLC, 2014, www.simio.com/ case-studies/A-Steel-Tubing-Manufacturer- Expects-More- Market-Share/A-Steel-Tubing-Manufacturer-Expects- More-Market-Share.pdf (accessed September 2018).

Bailey, M. J., J. Snapp, S. Yetur, J. S. Stonebraker, S. A. Edwards, A. Davis, & R. Cox. (2011). “Practice Summaries: American Airlines Uses Should-Cost Modeling to Assess the Uncer- tainty of Bids for Its Full-Truckload Shipment Routes.” Interfaces, 41(2), 194–196.

Banks, J., & Gibson, R. R. (2009). Seven Sins of Simulation Practice.” INFORMS Analytics, 24–27. www.analytics- magazine.org/summer-2009/193-strategic-problems- modeling-the-market-space (accessed September 2018).

Bowers, M. R., C. E. Noon, W. Wu, & J. K. Bass. (2016). “Neo- natal Physician Scheduling at the University of Tennessee Medical Center.” Interfaces, 46(2), 168–182.

Chongwatpol, J., & R. Sharda. (2013). “RFID-Enabled Track and Traceability in Job-Shop Scheduling Environment.” European Journal of Operational Research, 227(3), 453–463, http://dx.doi.org/10.1016/j.ejor.2013.01.009.

Christiansen, M., K. Fagerholt, G. Hasle, A. Minsaas, & B. Nygreen. (2009, April). “Maritime Transport Optimization: An Ocean of Opportunities.” OR/MS Today, 36(2), 26–31.

Clemen, R. T., & Reilly, T. (2000). Making Hard Decisions with Decision Tools Suite. Belmont, MA: Duxbury Press.

Dilkina, B. N., & W. S. Havens. “The U.S. National Football League Scheduling Problem. Intelligent Systems Lab,” www.cs.cornell.edu/~bistra/papers/NFLsched1.pdf (accessed September 2018).

Farasyn, I., K. Perkoz, & W. Van de Velde. (2008, July/ August). “Spreadsheet Models for Inventory Target Setting at Procter & Gamble.” Interfaces, 38(4), 241–250.

Goodwin, P., & Wright, G. (2000). Decision Analysis for Management Judgment, 2nd ed. New York: Wiley.

Hurley, W. J., & M. Balez. (2008, July/August). “A Spreadsheet Implementation of an Ammunition Requirements Planning Model for the Canadian Army.” Interfaces, 38(4), 271–280. ingrammicrocommerce.com, “ CUSTOMERS,” https:// www.ingrammicrocommerce.com/customers/ (accessed July 2016).

Kearns, G. S. (2004, January–March). “A Multi-Objective, Mul- ticriteria Approach for Evaluating IT Investments: Results from Two Case Studies.” Information Resources Manage- ment Journal, 17(1), 37–62.

Kelly, A. (2002). Decision Making Using Game Theory: An Introduction for Managers. Cambridge, UK: Cambridge University Press.

Knight, F. H. (1933). Risk, Uncertainty and Profit: With an Additional Introductory Essay Hither to Unpublished. Lon- don school of economics and political science.

Koksalan, M., & S. Zionts. (Eds.). (2001). Multiple Criteria Deci- sion Making in the New Millennium. Berlin: Springer-Verlag.

Kontoghiorghes, E. J., B. Rustem, & S. Siokos. (2002). Com- putational Methods in Decision Making, Economics, and Finance. Boston: Kluwer.

Kostuk, Kent J., and K. A. Willoughby. (2012). “A Decision Support System for Scheduling the Canadian Football League.” Interfaces, 42(3), 286–295.

Manikas, A. S., J. R. Kroes, & T. F. Gattiker. (2016). Metro Meals on Wheels Treasure Valley Employs a Low-Cost Routing Tool to Improve Deliveries. Interfaces, 46(2), 154–167.

Mookherjee, R., J. Martineau, L. Xu, M. Gullo, K. Zhou, A. Hazlewood, X. Zhang, F. Griarte, & N. Li. (2016). “End-to- End Predictive Analytics and Optimization in Ingram Micro’s Two-Tier Distribution Business.” Interfaces, 46 (1),49–73.

Ovchinnikov, A., & J. Milner. (2008, July/August). “Spread- sheet Model Helps to Assign Medical Residents at the University of Vermont’s College of Medicine.” Interfaces, 38(4), 311–323.

Simio.com. “Cosan Case Study—Optimizing Agricultural Logistics Operations.” http://www.simio.com/case-studies/ Cosan-agricultural-logistics-simulation-software- case-study/index.php (accessed September 2018).

Slaugh, V. W., M. Akan, O. Kesten, & M. U. Unver. (2016). “The Pennsylvania Adoption Exchange Improves Its Matching Process.” Interfaces, 462, 133–154.

Solver.com. “Optimizing Vendor Contract Awards Gets an A+.” solver.com/news/optimizing-vendor-contract- awards-gets (accessed September 2018).

Turban, E., & J. Meredith. (1994). Fundamentals of Manage- ment Science, 6th ed. Richard D. Irwin, Inc.

Wikipedia.com. Cosan. https://en.wikipedia.org/wiki/ Cosan (accessed November 2018).

509

Big Data, Cloud Computing, and Location Analytics: Concepts and Tools

LEARNING OBJECTIVES

■■ Learn what Big Data is and how it is changing the world of analytics

■■ Understand the motivation for and business drivers of Big Data analytics

■■ Become familiar with the wide range of enabling technologies for Big Data analytics

■■ Learn about Hadoop, MapReduce, and NoSQL as they relate to Big Data analytics

■■ Compare and contrast the complementary uses of data warehousing and Big Data technologies

■■ Become familiar with in-memory analytics and Spark applications

■■ Become familiar with select Big Data platforms and services

■■ Understand the need for and appreciate the capabilities of stream analytics

■■ Learn about the applications of stream analytics ■■ Describe the current and future use of cloud computing in business analytics

■■ Describe how geospatial and location-based analytics are assisting organizations

B ig Data, which means many things to many people, is not a new technological fad. It has become a business priority that has the potential to profoundly change the competitive landscape in today’s globally integrated economy. In addition to providing innovative solutions to enduring business challenges, Big Data and analyt- ics instigate new ways to transform processes, organizations, entire industries, and even society altogether. Yet extensive media coverage makes it hard to distinguish hype from reality. This chapter aims to provide a comprehensive coverage of Big Data, its enabling technologies, and related analytics concepts to help understand the capabilities and limi- tations of this emerging technology. The chapter starts with a definition and related con- cepts of Big Data followed by the technical details of the enabling technologies, including Hadoop, MapReduce, and NoSQL. We provide a comparative analysis between data warehousing and Big Data analytics. The last part of the chapter is dedicated to stream

9 C H A P T E R

510 Part III • Prescriptive Analytics and Big Data

analytics, which is one of the most promising value propositions of Big Data analytics. This chapter contains the following sections:

9.1 Opening Vignette: Analyzing Customer Churn in a Telecom Company Using Big Data Methods 510

9.2 Definition of Big Data 513 9.3 Fundamentals of Big Data Analytics 519 9.4 Big Data Technologies 523 9.5 Big Data and Data Warehousing 532 9.6 In-Memory Analytics and Apache SparkTM 537 9.7 Big Data and Stream Analytics 543 9.8 Big Data Vendors and Platforms 549 9.9 Cloud Computing and Business Analytics 557

9.10 Location-Based Analytics for Organizations 567

9.1 OPENING VIGNETTE: Analyzing Customer Churn in a Telecom Company Using Big Data Methods

BACKGROUND

A telecom company (named Access Telecom [AT] for privacy reasons) wanted to stem the tide of customers churning from its telecom services. Customer churn in the telecommuni- cations industry is common. However, Access Telecom was losing customers at an alarm- ing rate. Several reasons and potential solutions were attributed to this phenomenon. The management of the company realized that many cancellations involved communications between the customer service department and the customers. To this end, a task force comprising members from the customer relations office and the information technology (IT) department was assembled to explore the problem further. Their task was to explore how the problem of customer churn could be reduced based on an analysis of the cus- tomers’ communication patterns (Asamoah, Sharda, Zadeh, & Kalgotra, 2016).

BIG DATA HURDLES

Whenever a customer had a problem about issues such as their bill, plan, and call quality, they would contact the company in multiple ways. These included a call center, company Web site (contact us links), and physical service center walk-ins. Customers could cancel an account through one of these listed interactions. The company wanted to see if analyz- ing these customer interactions could yield any insights about the questions the custom- ers asked or the contact channel(s) they used before canceling their account. The data generated because of these interactions were in both text and audio. So, AT would have to combine all the data into one location. The company explored the use of traditional platforms for data management but soon found they were not versatile enough to handle advanced data analysis in the scenario where there were multiple formats of data from multiple sources (Thusoo, Shao, & Anthony, 2010).

There were two major challenges in analyzing this data: multiple data sources leading to a variety of data and also a large volume of data.

1. Data from multiple sources: Customers could connect with the company by ac- cessing their accounts on the company’s Web site, allowing AT to generate Web log information on customer activity. The Web log track allowed the company to identify if and when a customer reviewed his/her current plan, submitted a complaint, or

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 511

checked the bill online. At the customer service center, customers could also lodge a service complaint, request a plan change, or cancel the service. These activities were logged into the company’s transaction system and then the enterprise data warehouse. Last, a customer could call the customer service center on the phone and transact business just like he/she would do in person at a customer service center. Such transactions could involve a balance inquiry or an initiation of plan cancella- tion. Call logs were available in one system with a record of the reasons a customer was calling. For meaningful analysis to be performed, the individual data sets had to be converted into similar structured formats.

2. Data volume: The second challenge was the sheer quantity of data from the three sources that had to be extracted, cleaned, restructured, and analyzed. Although pre- vious data analytics projects mostly utilized a small sample set of data for analysis, AT decided to leverage the multiple variety and sources of data as well as the large volume of data recorded to generate as many insights as possible.

An analytical approach that could make use of all the channels and sources of data, although huge, would have the potential of generating rich and in-depth insights from the data to help curb the churn.

SOLUTION

Teradata Vantage’s unified Big Data architecture (previously offered as Teradata Aster) was utilized to manage and analyze the large multistructured data. We will introduce Teradata Vantage in Section 9.8. A schematic of which data was combined is shown in Figure 9.1. Based on each data source, three tables were created with each table containing the following variables: customer ID, channel of communication, date/time

Data on ASTER

TERADATA ASTER

Online Data

Store Data

Data on TERADATA

SQL-H connector Load_from_teradata

Callcenter Data

HCatalog metadata and Data on HDFS

FIGURE 9.1 Multiple Data Sources Integrated into Teradata Vantage. Source: Teradata Corp.

512 Part III • Prescriptive Analytics and Big Data

stamp, and action taken. Prior to final cancellation of a service, the action-taken vari- able could be one or more of these 11 options (simplified for this case): present a bill dispute, request for plan upgrade, request for plan downgrade, perform profile update, view account summary, access customer support, view bill, review contract, access store locator function on the Web site, access frequently asked questions section on the Web site, or browse devices. The target of the analysis focused on finding the most common path resulting in a final service cancellation. The data was sessionized to group a string of events involving a particular customer into a defined time period (5 days over all the channels of communication) as one session. Finally, Vantage’s nPath time sequence func- tion (operationalized in an SQL-MapReduce framework) was used to analyze common trends that led to a cancellation.

RESULTS

The initial results identified several routes that could lead to a request for service cancel- lation. The company determined thousands of routes that a customer may take to cancel service. A follow-up analysis was performed to identify the most frequent routes to can- cellation requests. This was termed as the Golden Path. The top 20 most occurring paths that led to a cancellation were identified in both short and long terms. A sample is shown in Figure 9.2.

This analysis helped the company identify a customer before they would cancel their service and offer incentives or at least escalate the problem resolution to a level where the customer’s path to cancellation did not materialize.

u QUESTIONS FOR THE OPENING VIGNETTE

1. What problem did customer service cancellation pose to AT’s business survival? 2. Identify and explain the technical hurdles presented by the nature and

characteristics of AT’s data.

3. What is sessionizing? Why was it necessary for AT to sessionize its data?

Callcenter:Bill Dispute

Store:Bill Dispute

Store:New Account

Store:Service Complaint

Store:Service Complaint

Callcenter:Service

Complaint

Online:Cancel Service

Callcenter:Cancel Service

Store:Cancel Service

Callcenter:Cancel Service

Callcenter:Bill

Dispute

Store:Bill Dispute

Store:Cancel Service

Callcenter: Service

Complaint

FIGURE 9.2 Top 20 Paths Visualization. Source: Teradata Corp.

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 513

4. Research other studies where customer churn models have been employed. What types of variables were used in those studies? How is this vignette different?

5. Besides Teradata Vantage, identify other popular Big Data analytics platforms that could handle the analysis described in the preceding case. (Hint: see Section 9.8.)

WHAT CAN WE LEARN FROM THIS VIGNETTE?

Not all business problems merit the use of a Big Data analytics platform. This situation presents a business case that warranted the use of a Big Data platform. The main challenge revolved around the characteristics of the data under consideration. The three different types of customer interaction data sets presented a challenge in analysis. The formats and fields of data generated in each of these systems was huge. And the volume was large as well. This made it imperative to use a platform that uses technologies to permit analysis of a large volume of data that comes in a variety of formats.

Recently, Teradata stopped marketing Aster as a separate product and has merged all of the Aster capabilities into its new offering called Teradata Vantage. Although that change somewhat impacts how the application would be developed today, it is still a ter- rific example of how a variety of data can be brought together to make business decisions.

It is also worthwhile to note that AT aligned the questions asked of the data with the organization’s business strategy. The questions also informed the type of analysis that was performed. It is important to understand that for any application of a Big Data architec- ture, the organization’s business strategy and the generation of relevant questions are key to identifying the type of analysis to perform.

Sources: D. Asamoah, R. Sharda, A. Zadeh, & P. Kalgotra. (2016). “Preparing a Big Data Analytics Professional: A Pedagogic Experience.” In DSI 2016 Conference, Austin, TX. A. Thusoo, Z. Shao, & S. Anthony. (2010). “Data Warehousing and Analytics Infrastructure at Facebook.” In Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data (p. 1013). doi: 10.1145/1807167.1807278.

9.2 DEFINITION OF BIG DATA

Using data to understand customers/clients and business operations to sustain (and fos- ter) growth and profitability is an increasingly challenging task for today’s enterprises. As more and more data becomes available in various forms and fashions, timely processing of the data with traditional means becomes impractical. Nowadays, this phenomenon is called Big Data, which is receiving substantial press coverage and drawing increasing interest from both business users and IT professionals. The result is that Big Data is be- coming an overhyped and overused marketing buzzword, leading some industry experts to argue dropping this phrase altogether.

Big Data means different things to people with different backgrounds and interests. Traditionally, the term Big Data has been used to describe the massive volumes of data analyzed by huge organizations like Google or research science projects at NASA. But for most businesses, it’s a relative term: “Big” depends on an organization’s size. The point is more about finding new value within and outside conventional data sources. Pushing the boundaries of data analytics uncovers new insights and opportunities, and “big” depends on where you start and how you proceed. Consider the popular description of Big Data: Big Data exceeds the reach of commonly used hardware environments and/or capabili- ties of software tools to capture, manage, and process it within a tolerable time span for its user population. Big Data has become a popular term to describe the exponential growth, availability, and use of information, both structured and unstructured. Much has been written on the Big Data trend and how it can serve as the basis for innovation,

514 Part III • Prescriptive Analytics and Big Data

differentiation, and growth. Because of the technology challenges in managing the large volume of data coming from multiple sources, sometimes at a rapid speed, additional new technologies have been developed to overcome the technology challenges. Use of the term Big Data is usually associated with such technologies. Because a prime use of storing such data is generating insights through analytics, sometimes the term Big Data is expanded as Big Data analytics. But the term is becoming content free in that it can mean different things to different people. Because our goal is to introduce you to the large data sets and their potential in generating insights, we will use the original term in this chapter.

Where does Big Data come from? A simple answer is “everywhere.” The sources that were ignored because of the technical limitations are now treated as gold mines. Big Data may come from Web logs, radio-frequency identification (RFID), global positioning systems (GPS), sensor networks, social networks, Internet-based text documents, Internet search indexes, detail call records, astronomy, atmospheric science, biology, genomics, nuclear physics, biochemical experiments, medical records, scientific research, military surveillance, photography archives, video archives, and large-scale e-commerce practices.

Big Data is not new. What is new is that the definition and the structure of Big Data constantly change. Companies have been storing and analyzing large volumes of data since the advent of the data warehouses in the early 1990s. Whereas terabytes used to be synonymous with Big Data warehouses, now it’s exabytes, and the rate of growth in data volume continues to escalate as organizations seek to store and analyze greater levels of transaction details, as well as Web- and machine-generated data, to gain a better under- standing of customer behavior and business drivers.

Many (academics and industry analysts/leaders alike) think that “Big Data” is a misnomer. What it says and what it means are not exactly the same. That is, Big Data is not just “big.” The sheer volume of the data is only one of many characteristics that are often associated with Big Data, including variety, velocity, veracity, variability, and value proposition, among others.

The “V”s That Define Big Data

Big Data is typically defined by three “V”s: volume, variety, velocity. In addition to these three, we see some of the leading Big Data solution providers adding other “V”s, such as veracity (IBM), variability (SAS), and value proposition.

VOLUME Volume is obviously the most common trait of Big Data. Many factors contributed to the exponential increase in data volume, such as transaction-based data stored through the years, text data constantly streaming in from social media, increasing amounts of sensor data being collected, automatically generated RFID and GPS data, and so on. In the past, excessive data volume created storage issues, both technical and financial. But with today’s advanced technologies coupled with decreasing storage costs, these issues are no longer significant; instead, other issues have emerged, including how to determine relevance amid the large volumes of data and how to create value from data that is deemed to be relevant.

As mentioned before, big is a relative term. It changes over time and is perceived differently by different organizations. With the staggering increase in data volume, even the naming of the next Big Data echelon has been a challenge. The highest mass of data that used to be called petabytes (PB) has left its place to zettabytes (ZB), which is a tril- lion gigabytes (GB) or a billion terabytes (TB). Technology Insights 9.1 provides an over- view of the size and naming of Big Data volumes.

From a short historical perspective, in 2009 the world had about 0.8 ZB of data; in 2010, it exceeded the 1 ZB mark; at the end of 2011, the number was 1.8 ZB. It is ex- pected to be 44 ZB in 2020 (Adshead, 2014). With the growth of sensors and the Internet of Things (IoT—to be introduced in the next chapter), these forecasts could all be wrong.

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 515

Though these numbers are astonishing in size, so are the challenges and opportunities that come with them.

VARIETY Data today come in all types of formats—ranging from traditional databases to hierarchical data stores created by the end users and OLAP systems to text documents, e-mail, XML, meter-collected and sensor-captured data, to video, audio, and stock ticker data. By some estimates, 80 to 85% of all organizations’ data are in some sort of unstruc- tured or semi-structured format (a format that is not suitable for traditional database sche- mas). But there is no denying its value, and hence, it must be included in the analyses to support decision making.

VELOCITY According to Gartner, velocity means both how fast data is being pro- duced and how fast the data must be processed (i.e., captured, stored, and analyzed) to meet the need or demand. RFID tags, automated sensors, GPS devices, and smart meters are driving an increasing need to deal with torrents of data in near real time. Velocity is perhaps the most overlooked characteristic of Big Data. Reacting quickly enough to deal with velocity is a challenge to most organizations. For time-sensitive environments, the opportunity cost clock of the data starts ticking the moment the data is created. As time passes, the value proposition of the data degrades and even- tually becomes worthless. Whether the subject matter is the health of a patient, the well-being of a traffic system, or the health of an investment portfolio, accessing the data and reacting faster to the circumstances will always create more advantageous outcomes.

TECHNOLOGY INSIGHTS 9.1 The Data Size Is Getting Big, Bigger, and Bigger

The measure of data size is having a hard time keeping up with new names. We all know kilobyte (KB, which is 1,000 bytes), megabyte (MB, which is 1,000,000 bytes), gigabyte (GB, which is 1,000,000,000 bytes), and terabyte (TB, which is 1,000,000,000,000 bytes). Beyond that, the names given to data sizes are relatively new to most of us. The following table shows what comes after terabyte and beyond.

Name Symbol Value

Kilobyte kB 103

Megabyte MB 106

Gigabyte GB 109

Terabyte TB 1012

Petabyte PB 1015

Exabyte EB 1018

Zettabyte ZB 1021

Yottabyte YB 1024

Brontobyte* BB 1027

Gegobyte* GeB 1030

*Not an official SI (International System of Units) name/symbol, yet.

Consider that an exabyte of data is created on the Internet each day, which equates to 250 million DVDs’ worth of information. And the idea of even larger amounts of data—a zettabyte— isn’t too far off when it comes to the amount of information traversing the Web in any one year. In fact, industry experts are already estimating that we will see 1.3 zettabytes of traffic annually

516 Part III • Prescriptive Analytics and Big Data

over the Internet by 2016—and it could jump to 2.3 zettabytes by 2020. By 2020, Internet traffic is expected to reach 300 GB per capita per year. When referring to yottabytes, some of the Big Data scientists often wonder about how much data the NSA or FBI have on people altogether. Put in terms of DVDs, a yottabyte would require 250 trillion of them. A brontobyte, which is not an official SI prefix but is apparently recognized by some people in the measurement com- munity, is a 1 followed by 27 zeros. The size of such a magnitude can be used to describe the amount of sensor data that we will get from the Internet in the next decade, if not sooner.

A gegobyte is 10 to the power of 30. With respect to where the Big Data comes from, consider the following:

• The CERN Large Hadron Collider generates 1 petabyte per second. • Sensors from a Boeing jet engine create 20 terabytes of data every hour. • Every day, 600 terabytes of new data are ingested in Facebook databases. • On YouTube, 300 hours of video are uploaded per minute, translating to 1 terabyte every

minute. • The proposed Square Kilometer Array telescope (the world’s proposed biggest telescope)

will generate an exabyte of data per day.

Sources: S. Higginbotham. (2012). “As Data Gets Bigger, What Comes after a Yottabyte?” gigaom.com/ 2012/10/30/as-data-gets-bigger-what-comes-after-a-yottabyte (accessed October 2018). Cisco. (2016). “The Zettabyte Era: Trends and Analysis.” cisco.com/c/en/us/solutions/collateral/service-provider/ visual-networking-index-vni/vni-hyperconnectivity-wp.pdf (accessed October 2018).

In the Big Data storm that we are currently witnessing, almost everyone is fixated on at-rest analytics, using optimized software and hardware systems to mine large quanti- ties of variant data sources. Although this is critically important and highly valuable, there is another class of analytics, driven from the velocity of Big Data, called “data stream analytics” or “in-motion analytics,” which is evolving fast. If done correctly, data stream analytics can be as valuable as, and in some business environments more valuable than at-rest analytics. Later in this chapter we will cover this topic in more detail.

VERACITY Veracity is a term coined by IBM that is being used as the fourth “V” to de- scribe Big Data. It refers to conformity to facts: accuracy, quality, truthfulness, or trustwor- thiness of the data. Tools and techniques are often used to handle Big Data’s veracity by transforming the data into quality and trustworthy insights.

VARIABILITY In addition to the increasing velocities and varieties of data, data flows can be highly inconsistent with periodic peaks. Is something big trending in the social media? Perhaps there is a high-profile IPO looming. Maybe swimming with pigs in the Bahamas is suddenly the must-do vacation activity. Daily, seasonal, and event-triggered peak data loads can be highly variable and thus challenging to manage—especially with social media involved.

VALUE PROPOSITION The excitement around Big Data is its value proposition. A precon- ceived notion about “Big” data is that it contains (or has a greater potential to contain) more patterns and interesting anomalies than “small” data. Thus, by analyzing large and feature- rich data, organizations can gain greater business value that they may not have otherwise. Although users can detect the patterns in small data sets using simple statistical and machine- learning methods or ad hoc query and reporting tools, Big Data means “big” analytics. Big analytics means greater insight and better decisions, something that every organization needs.

Because the exact definition of Big Data (or its successor terms) is still a matter of ongoing discussion in academic and industrial circles, it is likely that more characteristics (perhaps more “V”s) are likely to be added to this list. Regardless of what happens, the importance and value proposition of Big Data is here to stay. Figure 9.3 shows a concep- tual architecture where Big Data (at the left side of the figure) is converted to business

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 517

insight through the use of a combination of advanced analytics and delivered to a variety of different users/roles for faster/better decision making.

Another term that is being added to Big Data buzzwords is alternative data. Application Case 9.1 shows examples of multiple types of data in a number of different scenarios.

MOVE

DATA PLATFORM

Fast Data Loading & Availability

Filtering & Processing

Deep History: Online Archival

Data Mgmt. (data lake)

MANAGE ACCESS Marketing Marketing

Executives

Operational Systems

Customers Partners

Frontline Workers

Business Analysts

Data Scientists

Engineers

USERS

Applications

Business Intelligence

Data Mining

Math and Stats

Languages

ANALYTIC TOOLS & APPS

ERP

SCM

CRM

Images

Audio and Video

Machine Logs

Text

Web and Social

SOURCES

Business lntelligence Predictive Analytics

Operational Intelligence

INTEGRATED DATA WAREHOUSE

Data Discovery Fast-Fail Hypothesis Testing

Path, Graph, Time-Series Analysis Pattern Detection

INTEGRATED DISCOVERY PLATFORM

FIGURE 9.3 A High-Level Conceptual Architecture for Big Data Solutions. Source: Teradata Company.

Getting a good forecast and understanding of the situ- ation is crucial for any scenario, but it is especially important to players in the investment industry. Being able to get an early indication of how a particular retailer’s sales are doing can give an investor a leg up on whether to buy or sell that retailer’s stock even before the earnings reports are released. The prob- lem of forecasting economic activity or microclimates based on a variety of data beyond the usual retail data is a very recent phenomenon and has led to another

buzzword—“alternative data.” A major mix in this alternative data category is satellite imagery, but it also includes other data such as social media, government filings, job postings, traffic patterns, changes in park- ing lots or open spaces detected by satellite images, mobile phone usage patterns in any given location at any given time, search patterns on search engines, and so on. Facebook and other companies have invested in satellites to try to image the whole globe every day so that daily changes can be tracked at any location

Application Case 9.1 Alternative Data for Market Analysis or Forecasts

(Continued )

518 Part III • Prescriptive Analytics and Big Data

and the information can be used for forecasting. Many interesting examples of more reliable and advanced forecasts have been reported. Indeed, this activity is being led by start-up companies. Tartar (2018) cited several examples. We mentioned some in Chapter 1. Here are some of the examples identified by them and many other proponents of alternative data:

• RS Metrics monitored parking lots across the United States for various hedge funds. In 2015, based on an analysis of the parking lots, RS Met- rics predicted a strong second quarter in 2015 for JC Penney. Its clients (mostly hedge funds) profited from this advanced insight. A similar story has been reported for Wal-Mart using car counts in its parking lots to forecast sales.

• Spaceknow keeps track of changes in factory surroundings for over 6,000 Chinese factory sites. Using this data, the company has been able to provide a better idea of China’s indus- trial economic activity than what the Chinese government has been reporting.

• Telluslabs, Inc. compiles data from NASA and European satellites to build prediction models for various crops such as corn, rice, soybean, wheat, and so on. Besides the images from the satellites, they incorporate measurements of thermal infrared bands, which help measure radiating heat to predict health of the crops.

• DigitalGlobe is able to analyze the size of a forest with more accuracy because its software can count every single tree in a forest. This re- sults in a more accurate estimate because there is no need to use a representative sample.

These examples illustrate just a sample of ways data can be combined to generate new insights. Of course, there are privacy concerns in some cases. For example, Yodlee, a division of Envestnet, pro- vides personal finance tools to many large banks as well as personal financial tools to individuals. Thus,

it has access to massive information about individu- als. It has faced concerns about the privacy and security of this information, especially in light of the major breaches reported by Facebook, Cambridge Analytics, and Equifax. Although such concerns will eventually be resolved by policy makers or the mar- ket, what is clear is that new and interesting ways of combining satellite data and many other data sources are spawning a new crop of analytics com- panies. All of these organizations are working with data that meets the three V’s—variety, volume, and velocity characterizations. Some of these companies also work with another category of data—sensors. But this group of companies certainly also falls under a group of innovative and emerging applications.

Sources: C. Dillow. (2016). “What Happens When You Combine Artificial Intelligence and Satellite Imagery.” fortune.com/2016/ 03/30/facebook-ai-satellite-imagery/ (accessed October 2018). G. Ekster. (2015). “Driving Investment Performance with Alternative Data.” integrity-research.com/wp-content/uploads/2015/11/ Driving-Investment-Performance-With-Alternative-Data.pdf (accessed October 2018). B. Hope. (2015). “Provider of Personal Finance Tools Tracks Bank Cards, Sells Data to Investors.” wsj.com/ articles/provider-of-personal-finance-tools-tracks-bank-cards- sells-data-to-investors-1438914620 (accessed October 2018). C.  Shaw. (2016). “Satellite Companies Moving Markets.” quandl. com/blog/alternative-data-satellite-companies (accessed Octo- ber 2018). C. Steiner. (2009). “Sky High Tips for Crop Traders.” www.forbes.com/forbes/2009/0907/technology-software- satellites-sky-high-tips-for-crop-traders.html (accessed October 2018). M. Turner. (2015). “This Is the Future of Investing, and You Probably Can’t Afford It.” businessinsider.com/hedge-funds-are- analysing-data-to-get-an-edge-2015-8 (accessed October 2018).

Questions for DisCussion

1. What is a common thread in the examples dis- cussed in this application case?

2. Can you think of other data streams that might help give an early indication of sales at a retailer?

3. Can you think of other applications along the lines presented in this application case?

u SECTION 9.2 REVIEW QUESTIONS

1. Why is Big Data important? What has changed to put it in the center of the analytics world? 2. How do you define Big Data? Why is it difficult to define? 3. Out of the “V”s that are used to define Big Data, in your opinion, which one is the

most important? Why?

4. What do you think the future of Big Data will be like? Will it leave its popularity to something else? If so, what will it be?

Application Case 9.1 (Continued)

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 519

9.3 FUNDAMENTALS OF BIG DATA ANALYTICS

Big Data by itself, regardless of the size, type, or speed, is worthless unless business users do something with it that delivers value to their organizations. That’s where “big” analyt- ics comes into the picture. Although organizations have always run reports and dash- boards against data warehouses, most have not opened these repositories to in-depth on-demand exploration. This is partly because analysis tools are too complex for the average user but also because the repositories often do not contain all the data needed by the power user. But this is about to change (and has been changing, for some) in a dramatic fashion, thanks to the new Big Data analytics paradigm.

With the value proposition, Big Data also brought about big challenges for organi- zations. The traditional means for capturing, storing, and analyzing data are not capable of dealing with Big Data effectively and efficiently. Therefore, new breeds of technolo- gies need to be developed (or purchased/hired/outsourced) to take on the Big Data challenge. Before making such an investment, organizations should justify the means. Here are some questions that may help shed light on this situation. If any of the follow- ing statements are true, then you need to seriously consider embarking on a Big Data journey.

• You can’t process the amount of data that you want to because of the limitations posed by your current platform or environment.

• You want to involve new/contemporary data sources (e.g., social media, RFID, sen- sory, Web, GPS, textual data) into your analytics platform, but you can’t because it does not comply with the data storage schema-defined rows and columns without sacrificing fidelity or the richness of the new data.

• You need to (or want to) integrate data as quickly as possible to be current on your analysis.

• You want to work with a schema-on-demand (as opposed to predetermined schema used in relational database management systems [RDBMSs]) data storage paradigm because the nature of the new data may not be known, or there may not be enough time to determine it and develop a schema for it.

• The data is arriving so fast at your organization’s doorstep that your traditional ana- lytics platform cannot handle it.

As is the case with any other large IT investment, the success in Big Data analytics depends on a number of factors. Figure 9.4 shows a graphical depiction of the most criti- cal success factors (Watson, 2012).

The following are the most critical success factors for Big Data analytics (Watson, Sharda, & Schrader, 2012):

1. A clear business need (alignment with the vision and the strategy). Business investments ought to be made for the good of the business, not for the sake of mere technology advancements. Therefore, the main driver for Big Data analytics should be the needs of the business, at any level—strategic, tactical, and operations.

2. Strong, committed sponsorship (executive champion). It is a well-known fact that if you don’t have strong, committed executive sponsorship, it is difficult (if not impossible) to succeed. If the scope is a single or a few analytical applications, the sponsorship can be at the departmental level. However, if the target is enterprise- wide organizational transformation, which is often the case for Big Data initiatives, sponsorship needs to be at the highest levels and organization wide.

3. Alignment between the business and IT strategy. It is essential to make sure that the analytics work is always supporting the business strategy, and not the other way around. Analytics should play the enabling role in successfully executing the business strategy.

520 Part III • Prescriptive Analytics and Big Data

4. A fact-based decision-making culture. In a fact-based decision-making culture, the numbers rather than intuition, gut feeling, or supposition drive decision making. There is also a culture of experimentation to see what works and what doesn’t. To create a fact-based decision-making culture, senior management needs to:

• Recognize that some people can’t or won’t adjust • Be a vocal supporter • Stress that outdated methods must be discontinued • Ask to see what analytics went into decisions • Link incentives and compensation to desired behaviors

5. A strong data infrastructure. Data warehouses have provided the data infra- structure for analytics. This infrastructure is changing and being enhanced in the Big Data era with new technologies. Success requires marrying the old with the new for a holistic infrastructure that works synergistically.

As the size and complexity increase, the need for more efficient analytical systems is also increasing. To keep up with the computational needs of Big Data, a number of new and innovative computational techniques and platforms have been developed. These tech- niques are collectively called high-performance computing, which includes the following:

• In-memory analytics: Solves complex problems in near real time with highly accurate insights by allowing analytical computations and Big Data to be processed in-memory and distributed across a dedicated set of nodes.

• In-database analytics: Speeds time to insights and enables better data gover- nance by performing data integration and analytic functions inside the database so you won’t have to move or convert data repeatedly.

A clear business

need

Strong, committed

sponsorship

Alignment between the business and IT strategy

The right analytics

tools

Personnel with advanced

analytical skills

Keys to Success

with Big Data Analytics

A strong data infrastructure

A fact-based decision-making

culture

FIGURE 9.4 Critical Success Factors for Big Data Analytics. Source: Watson, H. (2012). The requirements for being an analytics-based organization. Business Intelligence Journal, 17(2), 42–44.

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 521

• Grid computing: Promotes efficiency, lower cost, and better performance by processing jobs in a shared, centrally managed pool of IT resources.

• Appliances: Brings together hardware and software in a physical unit that is not only fast but also scalable on an as-needed basis.

Computational requirements are just a small part of the list of challenges that Big Data impose on today’s enterprises. The following is a list of challenges that are found by business executives to have a significant impact on successful implementation of Big Data analytics. When considering Big Data projects and architecture, being mindful of these challenges will make the journey to analytics competency a less stressful one.

Data volume: The ability to capture, store, and process a huge volume of data at an acceptable speed so that the latest information is available to decision makers when they need it.

Data integration: The ability to combine data that is not similar in structure or source and to do so quickly and at a reasonable cost.

Processing capabilities: The ability to process data quickly, as it is captured. The traditional way of collecting and processing data may not work. In many situations, data needs to be analyzed as soon as it is captured to leverage the most value. (This is called stream analytics, which will be covered later in this chapter.)

Data governance: The ability to keep up with the security, privacy, ownership, and quality issues of Big Data. As the volume, variety (format and source), and velocity of data change, so should the capabilities of governance practices.

Skills availability: Big Data is being harnessed with new tools and is being looked at in different ways. There is a shortage of people (often called data scientists) with skills to do the job.

Solution cost: Because Big Data has opened up a world of possible business improvements, a great deal of experimentation and discovery is taking place to determine the patterns that matter and the insights that turn to value. To ensure a positive return on investment on a Big Data project, therefore, it is crucial to reduce the cost of the solutions used to find that value.

Though the challenges are real, so is the value proposition of Big Data analytics. Anything that you can do as a business analytics leader to help prove the value of new data sources to the business will move your organization beyond experimenting and ex- ploring Big Data into adapting and embracing it as a differentiator. There is nothing wrong with exploration, but ultimately the value comes from putting those insights into action.

Business Problems Addressed by Big Data Analytics

The top business problems addressed by Big Data overall are process efficiency and cost reduction, as well as enhancing customer experience, but different priorities emerge when it is looked at by industry. Process efficiency and cost reduction are perhaps among the top-ranked problems that can be addressed with Big Data analytics for the manu- facturing, government, energy and utilities, communications and media, transport, and healthcare sectors. Enhanced customer experience may be at the top of the list of prob- lems addressed by insurance companies and retailers. Risk management usually is at the top of the list for companies in banking and education. Here is a partial list of problems that can be addressed using Big Data analytics:

Process efficiency and cost reduction Brand management Revenue maximization, cross-selling, and up-selling Enhanced customer experience Churn identification, customer recruiting

522 Part III • Prescriptive Analytics and Big Data

Improved customer service Identifying new products and market opportunities Risk management Regulatory compliance Enhanced security capabilities

Application Case 9.2 illustrates an excellent example in the retail industry, where dispa- rate data sources are integrated into a Big Data infrastructure to understand customer journeys.

Major retail organizations such as Overstock.com invest in many marketing campaigns to grow their revenue. These may include targeted online and direct mail campaigns, advertising through various channels, growing the loyalty program by providing different customer incentives, and so on. Each of these entails significant marketing costs and yet has varying levels of ROI. A challenge for any company analyzing all these campaigns is to unify these data in one location and understand customer journeys. Which combina- tions of campaigns or interactions eventually led to customers purchasing some items, and at what level? These data sources may include Web site traffic data that are in somewhat unstructured log files; e-mail campaign performance data may come through e-mail campaign companies in semistructured format; social media data such as Facebook posts and responses are in yet different streams. Linking all of these data to company’s internal product data to assign value to a customer’s purchase to be able to compute ROI on the combinations of campaigns is another data inte- gration challenge. But combining such data sources is more practical under the Big Data framework. Then by using Path analysis capabilities that were illustrated in the opening vignette (Section 9.1), a nontechnical user can also look at various customer journeys and identify the ones that lead to most efficient sales and

a high ROI for marketing efforts. The goal is to build a long-term relationship with the customers by under- standing their search patterns, purchase behaviors, website responses, and so on. Overstock.com has been able to achieve this successfully using Teradata Vantage’s path analysis functionality but by combin- ing the very different data sources under the Big data framework.

Questions for DisCussion:

1. What are some of the different marketing cam- paigns a company might run to woo customers? What format might data about these campaigns take?

2. By visualizing the most common customer paths to sales, how would you use that information to make decisions on the future marketing campaigns?

3. What other applications of such path analysis techniques can you think of?

Compiled from: “Overstock.com Uses Teradata Path Analysis to Boost Its Customer Journey Analytics,” March 27, 2018, at www. retailitinsights.com/doc/overstock-com-uses-teradata- path-analysis-boost-customer-journey-analytics-0001 (accessed October 2018), and “Overstock.com: Revolutionizing Data and Analytics to Connect Soulfully with Their Customers,” at www. teradata .com/Resources/Videos/Overstock-com- Revolutionizing-data-and-analy (accessed October 2018).

Application Case 9.2 Overstock.com Combines Multiple Datasets to Understand Customer Journeys

This section has introduced the basics of Big Data and some potential applications. In the next section we will learn about a few terms and technologies that have emerged in Big Data space.

u SECTION 9.3 REVIEW QUESTIONS

1. What is Big Data analytics? How does it differ from regular analytics? 2. What are the critical success factors for Big Data analytics? 3. What are the big challenges that one should be mindful of when considering imple-

mentation of Big Data analytics?

4. What are the common business problems addressed by Big Data analytics?

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 523

9.4 BIG DATA TECHNOLOGIES

There are a number of technologies for processing and analyzing Big Data, but most have some common characteristics (Kelly, 2012). Namely, they take advantage of commodity hardware to enable scale-out and parallel-processing techniques; employ nonrelational data storage capabilities to process unstructured and semistructured data; and apply ad- vanced analytics and data visualization technology to Big Data to convey insights to end users. The three Big Data technologies that stand out that most believe will transform the business analytics and data management markets are MapReduce, Hadoop, and NoSQL.

MapReduce

MapReduce is a technique popularized by Google that distributes the processing of very large multistructured data files across a large cluster of machines. High performance is achieved by breaking the processing into small units of work that can be run in parallel across the hundreds, potentially thousands, of nodes in the cluster. To quote the seminal paper on MapReduce:

MapReduce is a programming model and an associated implementation for processing and generating large data sets. Programs written in this func- tional style are automatically parallelized and executed on a large cluster of commodity machines. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distrib- uted system. (Dean & Ghemawat, 2004)

The key point to note from this quote is that MapReduce is a programming model, not a programming language, that is, it is designed to be used by programmers, rather than business users. The easiest way to describe how MapReduce works is through the use of an example (see the Colored Square Counter in Figure 9.5).

The input to the MapReduce process in Figure 9.5 is a set of colored squares. The objective is to count the number of squares of each color. The programmer in this example is responsible for coding the map and reducing programs; the remainder of the process- ing is handled by the software system implementing the MapReduce programming model.

The MapReduce system first reads the input file and splits it into multiple pieces. In this example, there are two splits, but in a real-life scenario, the number of splits would typically be much higher. These splits are then processed by multiple map programs run- ning in parallel on the nodes of the cluster. The role of each map program in this case is to group the data in a split by color. The MapReduce system then takes the output from each map program and merges (shuffle/sort) the results for input to the reduce program, which calculates the sum of the number of squares of each color. In this example, only one copy of the reduce program is used, but there may be more in practice. To optimize performance, programmers can provide their own shuffle/sort program and can also deploy a combiner that combines local map output files to reduce the number of output files that have to be remotely accessed across the cluster by the shuffle/sort step.

Why Use MapReduce?

MapReduce aids organizations in processing and analyzing large volumes of multistruc- tured data. Application examples include indexing and search, graph analysis, text analy- sis, machine learning, data transformation, and so forth. These types of applications are often difficult to implement using the standard SQL employed by relational DBMSs.

The procedural nature of MapReduce makes it easily understood by skilled pro- grammers. It also has the advantage that developers do not have to be concerned with implementing parallel computing—this is handled transparently by the system. Although

524 Part III • Prescriptive Analytics and Big Data

MapReduce is designed for programmers, nonprogrammers can exploit the value of pre- built MapReduce applications and function libraries. Both commercial and open source MapReduce libraries are available that provide a wide range of analytic capabilities. Apache Mahout, for example, is an open source machine-learning library of “algorithms for clustering, classification and batch-based collaborative filtering” that are implemented using MapReduce.

Hadoop

Hadoop is an open source framework for processing, storing, and analyzing massive amounts of distributed, unstructured data. Originally created by Doug Cutting at Yahoo!, Hadoop was inspired by MapReduce, a user-defined function developed by Google in the early 2000s for indexing the Web. It was designed to handle petabytes and exabytes of data distributed over multiple nodes in parallel. Hadoop clusters run on inexpensive commodity hardware so projects can scale-out without breaking the bank. Hadoop is

Courtesy of The Apache Software Foundation.

The Goal: Determining the frequency counts of the shapes

Results: Frequency counts of the shapes

4

3

3

3

3

Shape Count

Raw Data Map Function Reduce Function

FIGURE 9.5 A Graphical Depiction of the MapReduce Process.

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 525

now a project of the Apache Software Foundation, where hundreds of contributors con- tinuously improve the core technology. Fundamental concept: Rather than banging away at one huge block of data with a single machine, Hadoop breaks up Big Data into mul- tiple parts so each part can be processed and analyzed at the same time.

How Does Hadoop Work?

A client accesses unstructured and semistructured data from sources including log files, social media feeds, and internal data stores. It breaks the data up into “parts,” which are then loaded into a file system made up of multiple nodes running on commodity hardware. The default file store in Hadoop is the Hadoop Distributed File System, or HDFS. File systems such as HDFS are adept at storing large volumes of unstructured and semistructured data as they do not require data to be organized into relational rows and columns. Each “part” is replicated multiple times and loaded into the file system so that if a node fails, another node has a copy of the data contained on the failed node. A Name Node acts as facilitator, communicating back to the client information such as which nodes are available, where in the cluster certain data resides, and which nodes have failed.

Once the data is loaded into the cluster, it is ready to be analyzed via the MapReduce framework. The client submits a “Map” job—usually a query written in Java—to one of the nodes in the cluster known as the Job Tracker. The Job Tracker refers to the Name Node to determine which data it needs to access to complete the job and where in the cluster that data is located. Once determined, the Job Tracker submits the query to the relevant nodes. Rather than bringing all the data back into a central location for process- ing, the processing occurs at each node simultaneously, or in parallel. This is an essential characteristic of Hadoop.

When each node has finished processing its given job, it stores the results. The cli- ent initiates a “Reduce” job through the Job Tracker in which results of the map phase stored locally on individual nodes are aggregated to determine the “answer” to the origi- nal query, and then are loaded onto another node in the cluster. The client accesses these results, which can then be loaded into one of a number of analytic environments for analysis. The MapReduce job has now been completed.

Once the MapReduce phase is complete, the processed data is ready for further analysis by data scientists and others with advanced data analytics skills. Data scientists can manipulate and analyze the data using any of a number of tools for any number of uses, including searching for hidden insights and patterns, or use as the foundation for building user-facing analytic applications. The data can also be modeled and transferred from Hadoop clusters into existing relational databases, data warehouses, and other tradi- tional IT systems for further analysis and/or to support transactional processing.

Hadoop Technical Components

A Hadoop “stack” is made up of a number of components, which include

Hadoop Distributed File System (HDFS): The default storage layer in any given Hadoop cluster.

Name Node: The node in a Hadoop cluster that provides the client information on where in the cluster particular data is stored and if any nodes fail.

Secondary Node: A backup to the Name Node, it periodically replicates and stores data from the Name Node should it fail.

Job Tracker: The node in a Hadoop cluster that initiates and coordinates MapReduce jobs or the processing of the data.

Slave Nodes: The grunts of any Hadoop cluster, slave nodes store data and take direction to process it from the Job Tracker.

526 Part III • Prescriptive Analytics and Big Data

In addition to these components, the Hadoop ecosystem is made up of a number of complementary subprojects. NoSQL data stores like Cassandra and HBase are also used to store the results of MapReduce jobs in Hadoop. In addition to Java, some MapReduce jobs and other Hadoop functions are written in Pig, an open source language designed specifically for Hadoop. Hive is an open source data warehouse originally developed by Facebook that allows for analytic modeling within Hadoop. Here are the most commonly referenced subprojects for Hadoop.

HIVE Hive is a Hadoop-based data warehousing–like framework originally developed by Facebook. It allows users to write queries in an SQL-like language called HiveQL, which are then converted to MapReduce. This allows SQL programmers with no MapReduce experience to use the warehouse and makes it easier to integrate with busi- ness intelligence (BI) and visualization tools such as Microstrategy, Tableau, Revolutions Analytics, and so forth.

PIG Pig is a Hadoop-based query language developed by Yahoo! It is relatively easy to learn and is adept at very deep, very long data pipelines (a limitation of SQL).

HBASE HBase is a nonrelational database that allows for low-latency, quick lookups in Hadoop. It adds transactional capabilities to Hadoop, allowing users to conduct updates, inserts, and deletes. eBay and Facebook use HBase heavily.

FLUME Flume is a framework for populating Hadoop with data. Agents are populated throughout one’s IT infrastructure—inside Web servers, application servers, and mobile devices, for example—to collect data and integrate it into Hadoop.

OOZIE Oozie is a workflow processing system that lets users define a series of jobs writ- ten in multiple languages—such as MapReduce, Pig, and Hive—and then intelligently link them to one another. Oozie allows users to specify, for example, that a particular query is only to be initiated after specified previous jobs on which it relies for data are completed.

AMBARI Ambari is a Web-based set of tools for deploying, administering, and monitoring Apache Hadoop clusters. Its development is being led by engineers from Hortonworks, which includes Ambari in its Hortonworks Data Platform.

AVRO Avro is a data serialization system that allows for encoding the schema of Hadoop files. It is adept at parsing data and performing removed procedure calls.

MAHOUT Mahout is a data mining library. It takes the most popular data mining algo- rithms for performing clustering, regression testing, and statistical modeling and imple- ments them using the MapReduce model.

SQOOP Sqoop is a connectivity tool for moving data from non-Hadoop data stores— such as relational databases and data warehouses—into Hadoop. It allows users to spec- ify the target location inside of Hadoop and instructs Sqoop to move data from Oracle, Teradata, or other relational databases to the target.

HCATALOG HCatalog is a centralized metadata management and sharing service for Apache Hadoop. It allows for a unified view of all data in Hadoop clusters and allows diverse tools, including Pig and Hive, to process any data elements without needing to know physically where in the cluster the data is stored.

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 527

Hadoop: The Pros and Cons

The main benefit of Hadoop is that it allows enterprises to process and analyze large vol- umes of unstructured and semistructured data, heretofore inaccessible to them, in a cost- and time-effective manner. Because Hadoop clusters can scale to petabytes and even exabytes of data, enterprises no longer must rely on sample data sets but can process and analyze all relevant data. Data scientists can apply an iterative approach to analysis, continually refining and testing queries to uncover previously unknown insights. It is also inexpensive to get started with Hadoop. Developers can download the Apache Hadoop distribution for free and begin experimenting with Hadoop in less than a day.

The downside to Hadoop and its myriad components is that they are immature and still developing. As with any young, raw technology, implementing and managing Hadoop clusters and performing advanced analytics on large volumes of unstructured data require significant expertise, skill, and training. Unfortunately, there is currently a dearth of Hadoop developers and data scientists available, making it impractical for many enterprises to maintain and take advantage of complex Hadoop clusters. Further, as Hadoop’s myriad components are improved on by the community and new components are created, there is, as with any immature open source technology/approach, a risk of forking. Finally, Hadoop is a batch-oriented framework, meaning it does not support real- time data processing and analysis.

The good news is that some of the brightest minds in IT are contributing to the Apache Hadoop project, and a new generation of Hadoop developers and data scien- tists is coming of age. As a result, the technology is advancing rapidly, becoming both more powerful and easier to implement and manage. An ecosystem of vendors, both Hadoop-focused start-ups like Cloudera and Hortonworks and well-worn IT stalwarts like IBM, Microsoft, Teradata, and Oracle are working to offer commercial, enterprise- ready Hadoop distributions, tools, and services to make deploying and managing the technology a practical reality for the traditional enterprise. Other bleeding edge start-ups are working to perfect NoSQL (Not Only SQL) data stores capable of delivering near– real-time insights in conjunction with Hadoop. Technology Insights 9.2 provides a few facts to clarify some misconceptions about Hadoop.

TECHNOLOGY INSIGHTS 9.2 A Few Demystifying Facts about Hadoop

Although Hadoop and related technologies have been around for more than five years now, most people still have several misconceptions about Hadoop and related technologies such as MapReduce and Hive. The following list of 10 facts intends to clarify what Hadoop is and does relative to BI, as well as in which business and technology situations Hadoop-based BI, data warehousing, and analytics can be useful (Russom, 2013).

Fact #1. Hadoop consists of multiple products. We talk about Hadoop as if it’s one monolithic software, whereas it is actually a family of open source products and technolo- gies overseen by the Apache Software Foundation (ASF). (Some Hadoop products are also available via vendor distributions; more on that later.)

The Apache Hadoop library includes (in BI priority order) HDFS, MapReduce, Hive, Hbase, Pig, Zookeeper, Flume, Sqoop, Oozie, Hue, and so on. You can combine these in various ways, but HDFS and MapReduce (perhaps with Hbase and Hive) constitute a use- ful technology stack for applications in BI, data warehousing, and analytics.

Fact #2. Hadoop is open source but available from vendors, too. Apache Hadoop’s open source software library is available from ASF at apache.org. For users desiring a more enterprise-ready package, a few vendors now offer Hadoop distributions that include additional administrative tools and technical support.

528 Part III • Prescriptive Analytics and Big Data

Fact #3. Hadoop is an ecosystem, not a single product. In addition to products from Apache, the extended Hadoop ecosystem includes a growing list of vendor products that integrate with or expand Hadoop technologies. One minute on your favorite search engine will reveal these.

Fact #4. HDFS is a file system, not a database management system (DBMS). Hadoop is primarily a distributed file system and lacks capabilities we would associate with a DBMS, such as indexing, random access to data, and support for SQL. That’s okay, because HDFS does things DBMSs cannot do.

Fact #5. Hive resembles SQL but is not standard SQL. Many of us are handcuffed to SQL because we know it well and our tools demand it. People who know SQL can quickly learn to hand-code Hive, but that doesn’t solve compatibility issues with SQL-based tools. TDWI feels that over time, Hadoop products will support standard SQL, so this issue will soon be moot.

Fact #6. Hadoop and MapReduce are related but don’t require each other. Developers at Google developed MapReduce before HDFS existed, and some variations of MapReduce work with a variety of storage technologies, including HDFS, other file systems, and some DBMSs.

Fact #7. MapReduce provides control for analytics, not analytics per se. MapReduce is a general-purpose execution engine that handles the complexities of network communica- tion, parallel programming, and fault tolerance for any kind of application that you can hand code—not just analytics.

Fact #8. Hadoop is about data diversity, not just data volume. Theoretically, HDFS can manage the storage and access of any data type as long as you can put the data in a file and copy that file into HDFS. As outrageously simplistic as that sounds, it’s largely true, and it’s exactly what brings many users to Apache HDFS.

Fact #9. Hadoop complements a DW; it’s rarely a replacement. Most organizations have designed their DW for structured, relational data, which makes it difficult to wring BI value from unstructured and semistructured data. Hadoop promises to complement DWs by handling the multistructured data types most DWs can’t.

Fact #10. Hadoop enables many types of analytics, not just Web analytics. Hadoop gets a lot of press about how Internet companies use it for analyzing Web logs and other Web data, but other use cases exist. For example, consider the Big Data coming from sensory devices, such as robotics in manufacturing, RFID in retail, or grid monitoring in utilities. Older analytic applications that need large data samples—such as customer-base segmentation, fraud detection, and risk analysis—can benefit from the additional Big Data managed by Hadoop. Likewise, Hadoop’s additional data can expand 360-degree views to create a more complete and granular view.

NoSQL

A related new style of database called NoSQL (Not Only SQL) has emerged to, like Hadoop, process large volumes of multistructured data. However, whereas Hadoop is adept at supporting large-scale, batch-style historical analysis, NoSQL databases are aimed, for the most part (though there are some important exceptions), at serving up discrete data stored among large volumes of multistructured data to end-user and au- tomated Big Data applications. This capability is sorely lacking from relational database technology, which simply can’t maintain needed application performance levels at a Big Data scale.

In some cases, NoSQL and Hadoop work in conjunction. The aforementioned HBase, for example, is a popular NoSQL database modeled after Google BigTable that is often deployed on top of HDFS, the Hadoop Distributed File System, to pro- vide low-latency, quick lookups in Hadoop. The downside of most NoSQL databases

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 529

today is that they trade ACID (atomicity, consistency, isolation, durability) compliance for performance and scalability. Many also lack mature management and monitoring tools. Both of these shortcomings are in the process of being overcome by the open source NoSQL communities and a handful of vendors that are attempting to com- mercialize the various NoSQL databases. NoSQL databases currently available include HBase, Cassandra, MongoDB, Accumulo, Riak, CouchDB, and DynamoDB, among oth- ers. Application Case 9.3 shows the use of NoSQL databases at eBay. Although the case is a few years old, we include it to give you a flavor of how multiple datasets come together. Application Case 9.4 illustrates a social media application where the Hadoop infrastructure was used to compile a corpus of messages on Twitter to understand which types of users engage in which type of support for healthcare patients seeking information about chronic mental diseases.

eBay is one of the world’s largest online market- places, enabling the buying and selling of practically anything. One of the keys to eBay’s extraordinary success is its ability to turn the enormous volumes of data it generates into useful insights that its custom- ers can glean directly from the pages they frequent. To accommodate eBay’s explosive data growth—its data centers perform billions of reads and writes each day—and due to the increasing demand to process data at blistering speeds, eBay needed a solution that did not have the typical bottlenecks, scalability issues, and transactional constraints asso- ciated with common relational database approaches. The company also needed to perform rapid analysis on a broad assortment of the structured and unstruc- tured data it captured.

The Solution: Integrated Real-Time Data and Analytics

Its Big Data requirements brought eBay to NoSQL technologies, specifically Apache Cassandra and DataStax Enterprise. Along with Cassandra and its high-velocity data capabilities, eBay was also drawn to the integrated Apache Hadoop analytics that come with DataStax Enterprise. The solution incorporates a scale-out architecture that enables eBay to deploy multiple DataStax Enterprise clus- ters across several different data centers using com- modity hardware. The end result is that eBay is now able to more cost effectively process massive

amounts of data at very high speeds, at very high velocities, and achieve far more than they were able to with the higher cost proprietary system they had been using. Currently, eBay is managing a siz- able portion of its data center needs—250TBs+ of storage—in Apache Cassandra and DataStax Enterprise clusters.

Additional technical factors that played a role in eBay’s decision to deploy DataStax Enterprise so widely include the solution’s linear scalability, high availability with no single point of failure, and out- standing write performance.

Handling Diverse Use Cases

eBay employs DataStax Enterprise for many dif- ferent use cases. The following examples illustrate some of the ways the company is able to meet its Big Data needs with the extremely fast data han- dling and analytics capabilities the solution pro- vides. Naturally, eBay experiences huge amounts of write traffic, which the Cassandra implementa- tion in DataStax Enterprise handles more efficiently than any other RDBMS or NoSQL solution. eBay currently sees 6 billion+ writes per day across mul- tiple Cassandra clusters and 5 billion+ reads (mostly offline) per day as well.

One use case supported by DataStax Enterprise involves quantifying the social data eBay displays on its product pages. The Cassandra distribution in DataStax Enterprise stores all the

Application Case 9.3 eBay’s Big Data Solution

(Continued )

530 Part III • Prescriptive Analytics and Big Data

information needed to provide counts for “like,” “own,” and “want” data on eBay product pages. It also provides the same data for the eBay “Your Favorites” page that contains all the items a user likes, owns, or wants, with Cassandra serving up the entire “Your Favorites” page. eBay provides this data through Cassandra’s scalable counters feature.

Load balancing and application availability are important aspects to this particular use case. The DataStax Enterprise solution gave eBay architects the flexibility they needed to design a system that enables any user request to go to any data center, with each data center having a single DataStax Enterprise clus- ter spanning those centers. This design feature helps balance the incoming user load and eliminates any possible threat to application downtime. In addition to the line of business data powering the Web pages its customers visit, eBay is also able to perform high- speed analysis with the ability to maintain a sepa- rate data center running Hadoop nodes of the same DataStax Enterprise ring (see Figure 9.6).

Another use case involves the Hunch (an eBay sister company) “taste graph” for eBay users and items, which provides customer recommenda- tions based on user interests. eBay’s Web site is essentially a graph between all users and the items for sale. All events (bid, buy, sell, and list) are cap- tured by eBay’s systems and stored as a graph in Cassandra. The application sees more than 200

million writes daily and holds more than 40 billion pieces of data.

eBay also uses DataStax Enterprise for many time-series use cases in which processing high- volume, real-time data is a foremost priority. These include mobile notification logging and tracking (every time eBay sends a notification to a mobile phone or device it is logged in Cassandra), fraud detection, SOA request/response payload logging, and RedLaser (another eBay sister company) server logs and analytics.

Across all of these use cases is the common requirement of uptime. eBay is acutely aware of the need to keep their business up and open for business, and DataStax Enterprise plays a key part in that through its support of high-availability clus- ters. “We have to be ready for disaster recovery all the time. It’s really great that Cassandra allows for active-active multiple data centers where we can read and write data anywhere, anytime,” says eBay architect Jay Patel.

Questions for DisCussion

1. Why did eBay need a Big Data solution?

2. What were the challenges, the proposed solu- tion, and the obtained results?

Source: DataStax. Customer case studies. datastax.com/ resources/casestudies/eBay (accessed October 2018).

DATA CENTER 1 DATA CENTER 2

Cassandra Ring

DATA CENTER 3

Analytics Nodes Running DSE Hadoop

for near real-time analytics

Topology—NTS RF–2:2:2

LB

://DNS

LB

FIGURE 9.6 eBay’s Multi–Data Center Deployment. Source: DataStax.

Application Case 9.3 (Continued)

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 531

On the Internet today, all users have the power to contribute as well as consume information. This power is used in many ways. On social network platforms such as Twitter, users are able to post information about their health condition as well as receive help on how best to manage those health conditions. Many users have wondered about the quality of information disseminated on social net- work platforms. Whereas the ability to author and disseminate health information on Twitter seems valuable to many users who use it to seek support for their disease, the authenticity of such informa- tion, especially when it originates from lay individ- uals, has been in doubt. Many users have asked, “How do I verify and trust information from non- experts about how to manage a vital issue like my health condition?”

What types of users share and discuss what type of information? Do users with a large fol- lowing discuss and share the same type of infor- mation as users with a smaller following? The number of followers of a user relate to the influ- ence of a user. Characteristics of the information are measured in terms of quality and objectivity of the Tweet posted. A team of data scientists set out to explore the relationship between the num- ber of followers a user had and the characteristics of information the user disseminated (Asamoah & Sharda, 2015).

Solution

Data was extracted from the Twitter platform using Twitter’s API. The data scientists adapted the knowledge-discovery and data management model to manage and analyze this large set of data. The model was optimized for managing and analyzing Big Data derived from a social network platform and included phases for gaining domain knowledge, developing an appropriate Big Data platform, data acquisition and storage, data clean- ing, data validation, data analysis, and results and deployment.

Technology Used

The tweets were extracted, managed, and ana- lyzed using Cloudera’s distribution of the Apache Hadoop. The Apache Hadoop framework has sev- eral subprojects that support different kinds of data management activities. For instance, the Apache Hive subproject supported the reading, writing, and managing of the large tweet data. Data analyt- ics tools such as Gephi were used for social net- work analysis and R for predictive modeling. They conducted two parallel analyses; social network analysis to understand the influence network on the platform and text mining to understand the content of tweets posted by users.

What Was Found?

As noted earlier, tweets from both influential and noninfluential users were collected and analyzed. The results showed that the quality and objectiv- ity of information disseminated by influential users was higher than that disseminated by noninfluential users. They also found that influential users con- trolled the flow of information in a network and that other users were more likely to follow their opinion on a subject. There was a clear difference between the type of information support provided by influ- ential users versus the others. Influential users dis- cussed more objective information regarding the disease management—things such as diagnoses, medications, and formal therapies. Noninfluential users provided more information about emotional support and alternative ways of coping with such diseases. Thus, a clear difference between influen- tial users and the others was evident.

From the nonexperts’ perspective, the data scientists portray how healthcare provision can be augmented by helping patients identify and use valuable resources on the Web for managing their disease condition. This work also helps identify how nonexperts can locate and filter healthcare information that may not necessarily be beneficial to the management of their health condition.

Application Case 9.4 Understanding Quality and Reliability of Healthcare Support Information on Twitter

(Continued )

532 Part III • Prescriptive Analytics and Big Data

u SECTION 9.4 REVIEW QUESTIONS

1. What are the common characteristics of emerging Big Data technologies? 2. What is MapReduce? What does it do? How does it do it? 3. What is Hadoop? How does it work? 4. What are the main Hadoop components? What functions do they perform? 5. What is NoSQL? How does it fit into the Big Data analytics picture?

9.5 BIG DATA AND DATA WAREHOUSING

There is no doubt that the emergence of Big Data has changed and will continue to change data warehousing in a significant way. Until recently, enterprise data warehouses (Chapter 3 and online supplements) were the centerpiece of all decision support tech- nologies. Now, they have to share the spotlight with the newcomer, Big Data. The ques- tion that is popping up everywhere is whether Big Data and its enabling technologies such as Hadoop will replace data warehousing and its core technology RDBMS. Are we witnessing a data warehouse versus Big Data challenge (or from the technology stand- point, Hadoop versus RDBMS)? In this section we will explain why these questions have no basis—and at least justify that such an either-or choice is not a reflection of the reality at this point in time.

In the last decade or so, we have seen significant improvement in the area of computer-based decision support systems, which can largely be credited to data warehousing and technological advancements in both software and hardware to capture, store, and analyze data. As the size of the data increased, so did the capabilities of data warehouses. Some of these data warehousing advances included massively parallel pro- cessing (moving from one or few to many parallel processors), storage area networks (easily scalable storage solutions), solid-state storage, in-database processing, in-memory processing, and columnar (column-oriented) databases, just to name a few. These ad- vancements helped keep the increasing size of data under control, while effectively serv- ing analytics needs of the decision makers. What has changed the landscape in recent years is the variety and complexity of data, which made data warehouses incapable of keeping up. It is not the volume of the data but the variety and velocity that forced the world of IT to develop a new paradigm, which we now call “Big Data.” Now that we have these two paradigms—data warehousing and Big Data—seemingly competing for the same job—turning data into actionable information—which one will prevail? Is this a fair question to ask? Or are we missing the big picture? In this section, we try to shed some light on this intriguing question.

Questions for DisCussion

1. What was the data scientists’ main concern regarding health information that is disseminated on the Twitter platform?

2. How did the data scientists ensure that nonexpert information disseminated on social media could indeed contain valuable health information?

3. Does it make sense that influential users would share more objective information whereas less

influential users could focus more on subjective information? Why?

Sources: D. Asamoah & R. Sharda. (2015). “Adapting CRISP-DM Process for Social Network Analytics: Application to Healthcare.” In AMCIS 2015 Proceedings. aisel.aisnet.org/amcis2015/ BizAnalytics/GeneralPresentations/33/ (accessed October 2018). Sarasohn-Kahn, J. (2008). The Wisdom of Patients: Health Care Meets Online Social Media. Oakland, CA: California HealthCare Foundation.

Application Case 9.4 (Continued)

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 533

As has been the case for many previous technology innovations, hype about Big Data and its enabling technologies like Hadoop and MapReduce is rampant. Nonpractitioners as well as practitioners are overwhelmed by diverse opinions. Yet oth- ers have begun to recognize that people are missing the point in claiming that Hadoop replaces relational databases and is becoming the new data warehouse. It is easy to see where these claims originate because both Hadoop and data warehouse systems can run in parallel, scale-up to enormous data volumes, and have shared-nothing architectures. At a conceptual level, one would think they are interchangeable. The reality is that they are not, and the differences between the two overwhelm the similarities. If they are not interchangeable, then how do we decide when to deploy Hadoop and when to use a data warehouse?

Use Cases for Hadoop

As we have covered earlier in this chapter, Hadoop is the result of new developments in computer and storage grid technologies. Using commodity hardware as a foundation, Hadoop provides a layer of software that spans the entire grid, turning it into a single system. Consequently, some major differentiators are obvious in this architecture:

Hadoop is the repository and refinery for raw data. Hadoop is a powerful, economical, and active archive.

Thus, Hadoop sits at both ends of the large-scale data life cycle—first when raw data is born, and finally when data is retiring, but is still occasionally needed.

1. Hadoop as the repository and refinery. As volumes of Big Data arrive from sources such as sensors, machines, social media, and clickstream interactions, the first step is to capture all the data reliably and cost effectively. When data volumes are huge, the traditional single-server strategy does not work for long. Pouring the data into HDFS gives architects much needed flexibility. Not only can they capture hundreds of terabytes in a day, but they can also adjust the Hadoop configuration up or down to meet surges and lulls in data ingestion. This is accomplished at the lowest possible cost per gigabyte due to open source economics and leveraging commodity hardware.

Because the data is stored on local storage instead of storage area networks, Hadoop data access is often much faster, and it does not clog the network with tera- bytes of data movement. Once the raw data is captured, Hadoop is used to refine it. Hadoop can act as a parallel “ETL engine on steroids,” leveraging handwritten or commercial data transformation technologies. Many of these raw data transforma- tions require the unraveling of complex freeform data into structured formats. This is particularly true with clickstreams (or Web logs) and complex sensor data formats. Consequently, a programmer needs to tease the wheat from the chaff, identifying the valuable signal in the noise.

2. Hadoop as the active archive. In a 2003 interview with ACM, Jim Gray claimed that hard disks could be treated as tape. Although it may take many more years for magnetic tape archives to be retired, today some portions of tape workloads are already being redirected to Hadoop clusters. This shift is occurring for two funda- mental reasons. First, although it may appear inexpensive to store data on tape, the true cost comes with the difficulty of retrieval. Not only is the data stored offline, requiring hours if not days to restore, but tape cartridges themselves are also prone to degradation over time, making data loss a reality and forcing companies to factor in those costs. To make matters worse, tape formats change every couple of years, requiring organizations to either perform massive data migrations to the newest tape format or risk the inability to restore data from obsolete tapes.

534 Part III • Prescriptive Analytics and Big Data

Second, it has been shown that there is value in keeping historical data online and accessible. As in the clickstream example, keeping raw data on a spinning disk for a longer duration makes it easy for companies to revisit data when the context changes and new constraints need to be applied. Searching thousands of disks with Hadoop is dramatically faster and easier than spinning through hundreds of magnetic tapes. In addition, as disk densities continue to double every 18 months, it becomes economically feasible for organizations to hold many years’ worth of raw or refined data in HDFS. Thus, the Hadoop storage grid is useful both in the preprocessing of raw data and the long-term storage of data. It’s a true “active archive” because it not only stores and protects the data, but also enables users to quickly, easily, and per- petually derive value from it.

Use Cases for Data Warehousing

After nearly 30 years of investment, refinement, and growth, the list of features available in a data warehouse is quite staggering. Built on relational database technology using schemas and integrating BI tools, the major differences in this architecture are

Data warehouse performance Integrated data that provides business value Interactive BI tools for end users

1. Data warehouse performance. Basic indexing, found in open source databases, such as MySQL or Postgres, is a standard feature used to improve query response times or enforce constraints on data. More advanced forms such as materialized views, aggregate join indexes, cube indexes, and sparse join indexes enable numer- ous performance gains in data warehouses. However, the most important perfor- mance enhancement to date is the cost-based optimizer. The optimizer examines incoming SQL and considers multiple plans for executing each query as fast as possible. It achieves this by comparing the SQL request to the database design and extensive data statistics that help identify the best combination of execution steps. In essence, the optimizer is like having a genius programmer examine every query and tune it for the best performance. Lacking an optimizer or data demographic statistics, a query that could run in minutes may take hours, even with many indexes. For this reason, database vendors are constantly adding new index types, partitioning, statis- tics, and optimizer features. For the past 30 years, every software release has been a performance release. As we will note at the end of his section, Hadoop is now gain- ing on traditional data warehouses in terms of query performance.

2. Integrating data that provides business value. At the heart of any data warehouse is the promise to answer essential business questions. Integrated data is the unique foundation required to achieve this goal. Pulling data from multiple subject areas and numerous applications into one repository is the raison d’être for data warehouses. Data model designers and Extract, Transform, and Load (ETL) architects armed with metadata, data-cleansing tools, and patience must ra- tionalize data formats, source systems, and semantic meaning of the data to make it understandable and trustworthy. This creates a common vocabulary within the corporation so that critical concepts such as “customer,” “end of month,” and “price elasticity” are uniformly measured and understood. Nowhere else in the entire IT data center is data collected, cleaned, and integrated as it is in the data warehouse.

3. Interactive BI tools. BI tools such as MicroStrategy, Tableau, IBM Cognos, and others provide business users with direct access to data warehouse insights. First, the business user can create reports and complex analysis quickly and easily using

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 535

these tools. As a result, there is a trend in many data warehouse sites toward end- user self-service. Business users can easily demand more reports than IT has staffing to provide. More important than self-service, however, is that the users become inti- mately familiar with the data. They can run a report, discover they missed a metric or filter, make an adjustment, and run their report again all within minutes. This process results in significant changes in business users’ understanding of the business and their decision-making process. First, users stop asking trivial questions and start ask- ing more complex strategic questions. Generally, the more complex and strategic the report, the more revenue and cost savings the user captures. This leads to some users becoming “power users” in a company. These individuals become wizards at teasing business value from the data and supplying valuable strategic information to the ex- ecutive staff. Every data warehouse has anywhere from 2 to 20 power users. As noted in Section 9.8, all of these BI tools have begun to embrace Hadoop to be able to scale their offerings to larger data stores.

The Gray Areas (Any One of the Two Would Do the Job)

Even though there are several areas that differentiate one from the other, there are also gray areas where the data warehouse and Hadoop cannot be clearly discerned. In these areas ei- ther tool could be the right solution—either doing an equally good or a not-so-good job on the task at hand. Choosing one over the other depends on the requirements and the prefer- ences of the organization. In many cases, Hadoop and the data warehouse work together in an information supply chain, and just as often, one tool is better for a specific workload (Awadallah & Graham, 2012). Table 9.1 illustrates the preferred platform (one versus the other, or equally likely) under a number of commonly observed requirements.

TABLE 9.1 When to Use Which Platform—Hadoop versus DW

Requirement Data Warehouse Hadoop

Low latency, interactive reports, and OLAP �

ANSI 2003 SQL compliance is required � �

Preprocessing or exploration of raw unstructured data �

Online archives alternative to tape �

High-quality cleansed and consistent data � �

100s to 1,000s of concurrent users � �

Discover unknown relationships in the data �

Parallel complex process logic � �

CPU intense analysis �

System, users, and data governance �

Many flexible programming languages running in parallel �

Unrestricted, ungoverned sandbox explorations �

Analysis of provisional data �

Extensive security and regulatory compliance � �

536 Part III • Prescriptive Analytics and Big Data

Coexistence of Hadoop and Data Warehouse

There are several possible scenarios under which using a combination of Hadoop and relational DBMS-based data warehousing technologies makes more sense. Here are some of those scenarios (White, 2012):

1. Use Hadoop for storing and archiving multistructured data. A connector to a relational DBMS can then be used to extract required data from Hadoop for analy- sis by the relational DBMS. If the relational DBMS supports MapReduce functions, these functions can be used to do the extraction. The Vantage-Hadoop adaptor, for example, uses SQL-MapReduce functions to provide fast, two-way data loading be- tween HDFS and the Vantage Database. Data loaded into the Vantage Database can then be analyzed using both SQL and MapReduce.

2. Use Hadoop for filtering, transforming, and/or consolidating multistruc- tured data. A connector such as the Vantage-Hadoop adaptor can be used to extract the results from Hadoop processing to the relational DBMS for analysis.

3. Use Hadoop to analyze large volumes of multistructured data and publish the analytical results. In this application, Hadoop serves as the analytics plat- form, but the results can be posted back to the traditional data warehousing environ- ment, a shared workgroup data store, or a common user interface.

4. Use a relational DBMS that provides MapReduce capabilities as an investi- gative computing platform. Data scientists can employ the relational DBMS (the Vantage Database system, for example) to analyze a combination of structured data and multistructured data (loaded from Hadoop) using a mixture of SQL processing and MapReduce analytic functions.

5. Use a front-end query tool to access and analyze data. Here, the data are stored in both Hadoop and the relational DBMS.

These scenarios support an environment where the Hadoop and relational DBMSs are separate from each other and connectivity software is used to exchange data between the two systems (see Figure 9.7). The direction of the industry over the next few years will likely be moving toward more tightly coupled Hadoop and relational DBMS-based data warehouse technologies—both software and hardware. Such integration provides

Raw Data Streams

Extract, Transform

Operational Systems

Sensor Data

Blogs E-mail

Web Data

Docs PDFs

Images Videos CRM SCM ERP Legacy

3rd Party

File Copy

Developer Environments Business Intelligence Tools

Extract, Transform, Load

Integrated Data Warehouse

FIGURE 9.7 Coexistence of Hadoop and Data Warehouses. Source: “Hadoop and the Data Warehouse: When to Use Which, teradata, 2012.” Used with permission from Teradata Corporation.

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 537

many benefits, including eliminating the need to install and maintain multiple systems, reducing data movement, providing a single metadata store for application develop- ment, and providing a single interface for both business users and analytical tools. The opening vignette (Section 9.1) provided an example of how data from a traditional data warehouse and two different unstructured data sets stored on Hadoop were integrated to create an analytics application to gain insight into a customer’s interactions with a company before canceling an account. As a manager, you care about the insights you can derive from the data, not whether the data is stored in a structured data warehouse or a Hadoop cluster.

u SECTION 9.5 REVIEW QUESTIONS

1. What are the challenges facing data warehousing and Big Data? Are we witnessing the end of the data warehousing era? Why or why not?

2. What are the use cases for Big Data and Hadoop? 3. What are the use cases for data warehousing and RDBMS? 4. In what scenarios can Hadoop and RDBMS coexist?

9.6 IN-MEMORY ANALYTICS AND APACHE SPARKTM

Hadoop utilizes the batch processing framework and lacks real time processing capabili- ties. In the evolution of big data computing, in-memory analytics is an emerging process- ing technique to analyse data stored in in-memory databases. Because accessing data stored in memory is much faster than the data in hard disk, in-memory processing is more efficient than the batch processing. This also allows for the analytics of streaming data in real-time.

In-memory analytics have several applications where low latency execution is required. It can help build real-time dashboards for better insights and faster deci- sion making. The real-time applications include understanding customer behaviour and engagement, forecasting stock price, optimizing airfare, predicting fraud, and several others.

The most popular tool supporting the in-memory processing is Apache SparkTM . It is a unified analytics engine that can execute both batch and streaming data. Originally developed at University of California, Berkeley in 2009, Apache SparkTM uses in-memory computation to achieve high performance on large-scale data pro- cessing. By adopting an in-memory processing approach, Apache SparkTM runs faster than the traditional Apache Hadoop. Moreover, it can be interactively used from the Java, Scala, Python, R, and SQL shells for writing data management and machine learning applications. Apache SparkTM can run on Apache Hadoop, Apache Mesos, Kubernetes, standalone, or in the cloud. Besides, it can connect to different external data sources such as HDFS, Alluxio, Apache Cassandra, Apache HBase, Apache Hive, and others.

Apache SparkTM can be used to create machine learning, fog computing, graph, streaming, and real-time analytics applications. Several big market players in the analytics sector have adopted Apache SparkTM . Examples include Uber, Pinterest, Netflix, Yahoo, and eBay. Uber uses Apache SparkTM to detect fraudulent trips at scale. Pinterest measures user engagement in real-time using Apache SparkTM. The recommendation engine of Netflix also utilizes the capabilities of Apache SparkTM. Yahoo, one of the early adopters of Apache SparkTM, has used it for creating business intelligence applications. Finally, eBay has used Apache SparkTM for data management and stream processing.

538 Part III • Prescriptive Analytics and Big Data

Architecture of Apache SparkTM

Apache SparkTM works on a master-slave framework. There is a driver program that talks to the master node, also known as cluster manager, which manages the worker nodes. The execution of the tasks takes place in the worker nodes where executors run. The entry point of the engine is called a Spark Context. It acts as a bridge of communica- tion between the application and the Spark execution environment as represented in Figure 9.8. As discussed earlier, Spark can run in different modes. In a standalone mode, it runs an application on different nodes in the cluster managed by the Spark itself. However, in a Hadoop mode, Spark uses Hadoop cluster to run jobs and leverage HDFS and MapReduce framework.

The TripAdvisor web platform contains information about hotels, restaurants, and other travel-related content. It also includes interactive travel forums that records the reviews of hotels or restaurants from the customers and managers. To improve the content on review forum, TripAdvisor decided to include tags to every travel attraction including restaurants and hotels. TripAdvisor collected reviews by send- ing out a review form to each of them, which basi- cally had general review and some “yes” or “no” type questions. The yes/no responses from the customers resulted in different tags. Using the past information, the company decided to build a logis- tic regression model to forecast the yes/no response from a future customer and predict the tags. The problem is complex because each location has its own features. Using the past customers’ experiences in the form of reviews, the textual information was used to train the model. The training model used the reviews on locations possessing the tag votes as well as the unlabeled reviews.

To create the model on the big data contain- ing millions of reviews and hundreds of tags, the company adopted the Apache SparkTM. Using the parallel processing and in-memory processing of Spark, a model was trained for each tag at each location. The data was partitioned by the loca- tion so as to minimize the communication among nodes. The entire process was implemented in an efficient manner.

Questions for DisCussion

1. How did the predictive modelling help TripAdvisor?

2. Why was Spark used?

Compiled from: Palmucci, J., “Using Apache Spark for Massively Parallel NLP,” at http://engineering.tripadvisor.com/using- apache-spark-for-massively-parallel-nlp/ (accessed October 2018) and Dalininaa, R., “Using Natural Language Processing to Analyze Customer Feedback in Hotel Reviews,” at www.datascience .com/resources/notebooks/data-science-summarize-hotel- reviews (accessed October 2018)

Application Case 9.5 Using Natural Language Processing to analyze customer feedback in TripAdvisor reviews

Application Driver Program

Spark Context

Cluster Manager

Worker Node

Executor

Worker Node

Executor

FIGURE 9.8 Apache SparkTM Architecture.

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 539

A very important component of Apache SparkTM is Resilient Distributed Dataset, com- monly known as RDD. It handles lineage, memory management, fault tolerance, and data partitioning across all nodes in a cluster. RDD provides several transformation functions like map, filter, and join that are performed on existing RDDs to create a new RDD. All transfor- mations in Spark are lazy in nature, that is, Spark does not execute these operations until any action function is performed on data. The action functions (e.g., count, reduce) print or return value after an execution. This approach is called a Lazy Evaluation. In Spark Streaming, a series of RDDs, also known as a Dstream, are utilized to process streaming data.

Getting Started with Apache SparkTM

In this section, we explain how to get started with Apache Spark on a Quick Start (QS) version of Cloudera Hadoop. It begins with downloading the latest version of Cloudera QS Virtual Machine (VM) and concludes with running your Spark query.

Hardware and Software Requirements Check • A computer with 64-bit host Operating System (Windows or Linux) with at least 12 GB

RAM for good performance • VMware Workstation Player: Download and install the latest (free) version of

the VMware Player from www.vmware.com/products/workstation-player/ workstation-player-evaluation.html

• 8 GB memory for the VM | 20 GB free disk space • 7-Zip: Extract (or unzip) the Cloudera Quick Start package using 7-Zip, available

from: www.7-zip.org/

Steps to be followed to get started with Spark on Cloudera QS VM:

1. Download Cloudera QS VM from www.cloudera.com/downloads/quickstart_ vms/5-13.html

2. Unzip it with 7-Zip. The downloaded file contains a VM machine. 3. Install VMware Workstation Player and turn it on. Now, open the Cloudera VM im-

ages through VMWare Player (Player > File > Open > full_path_of_vmx file). 4. Before turning on the VM, you must configure the memory and processer settings.

The default memory on VM will be 4 GB RAM. Click on the “Edit virtual machine setting” to change the settings. Make sure the RAM is more than 8 GB and the num- ber of processor cores is 2.

5. Turn on the machine. Cloudera has installed Hadoop and components on CentOS Linux.

6. A default user named “cloudera” with password “cloudera” is already available. 7. On the desktop of VM, open “Launch Cloudera Express.” The engine will take a few

minutes to get started.

540 Part III • Prescriptive Analytics and Big Data

8. Once started, open the web browser inside the VM. You will find an icon on the top of Cloudera Desktop.

9. Login into Cloudera Manager using username “cloudera” and password “cloudera.” 10. To use HDFS and map-reduce, we will want to start two services: HDFS and YARN

using the drop-down menu in front of them.

11. To turn on Spark, start the Spark service. 12. To run queries on Spark, we can use Python or Scala programming. Open Terminal

by right clicking on the Desktop of VM. 13. Type pyspark to enter Python shell as shown in the screenshot below. To exit the

Python shell, type exit().

Courtesy of Mozilla Firefox.

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 541

14. Type spark-shell to enter Scala Spark shell as shown in the screenshot below. To exit the Scala Spark shell, type exit.

15. From here onward, we describe steps to run a Spark streaming word count applica- tion where count of words will be calculated interactively. We run this application in Scala Spark shell. To use Spark Streaming interactively, we need to run Scala Spark shell with at least two threads. To do so, type spark-shell –master local[2] as shown in this screenshot.

a. Next, to run a streaming application, we need to import three related classes one by one as shown in this screenshot.

542 Part III • Prescriptive Analytics and Big Data

b. After importing the required classes, create a Spark Streaming Context sss with a batch duration of 10 seconds as in this screenshot.

c. Create a discretized Stream (DStream), the basic abstraction in Spark Streaming, to read text from port 1111. It is presented in this screenshot.

d. To count the occurrence of words on the stream, MapReduce codes shown in the screenshot are run. Then, count.print() command is used to print word count in the batch of 10 seconds.

e. At this point, open a new terminal and run command nc - lkv 1111 as shown in the right-hand-side terminal in this screenshot.

f. To start streaming context, run the sss.start() command in the Spark shell. This will result a connection of DStream sss with the socket (right-hand-side terminal).

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 543

g. In the final step, run sss.awaitTermination() in the Spark shell and start typing some words in the right-hand-side terminal as shown in this screenshot. After every 10 seconds, the word count pairs will be calculated in the Spark shell.

h. To stop the process, close the right-hand-side terminal and press CTRL + C in the left-hand-side terminal.

i. Because you may want to run the application again, all commands are listed here. spark-shell –master local[2] import org.apache.spark.streaming.StreamingContext import org.apache.spark.streaming.StreamingContext._ import org.apache.spark.streaming.Seconds val sss = new StreamingContext(sc,Seconds(10)) val firststream = sss.socketTextStream(“localhost”,1111) val words = firststream.flatMap(_.split(“ “)) val pairs = words.map(word = 7 (word, 1)) val count = pairs.reduceByKey(_+_) count.print() sss.start() sss.awaitTermination()

u SECTION 9.6 REVIEW QUESTIONS

1. What are some of the unique features of Spark as compared to Hadoop? 2. Give examples of companies that have adopted Apache Spark. Find new examples online. 3. Run the exercise as described in this section. What do you learn from this exercise?

9.7 BIG DATA AND STREAM ANALYTICS

Along with volume and variety, as we have seen earlier in this chapter, one of the key characteristics that defines Big Data is velocity, which refers to the speed at which the data is created and streamed into the analytics environment. Organizations are looking for new means to process streaming data as it comes in to react quickly and accurately to problems and opportunities to please their customers and to gain a competitive advan- tage. In situations where data streams in rapidly and continuously, traditional analytics approaches that work with previously accumulated data (i.e., data at rest) often either ar- rive at the wrong decisions because of using too much out-of-context data, or they arrive at the correct decisions but too late to be of any use to the organization. Therefore, it is critical for a number of business situations to analyze the data soon after it is created and/ or as soon as it is streamed into the analytics system.

The presumption that the vast majority of modern-day businesses are currently liv- ing by is that it is important and critical to record every piece of data because it might contain valuable information now or sometime in the near future. However, as long as

544 Part III • Prescriptive Analytics and Big Data

the number of data sources increases, the “store-everything” approach becomes harder and harder and, in some cases, not even feasible. In fact, despite technological advances, current total storage capacity lags far behind the digital information being generated in the world. Moreover, in the constantly changing business environment, real-time detec- tion of meaningful changes in data as well as of complex pattern variations within a given short time window are essential to come up with the actions that better fit with the new environment. These facts become the main triggers for a paradigm that we call stream analytics. The stream analytics paradigm was born as an answer to these challenges, namely, the unbounded flows of data that cannot be permanently stored to be subse- quently analyzed, in a timely and efficient manner, and complex pattern variations that need to be detected and acted on as soon as they happen.

Stream analytics (also called data-in-motion analytics and real-time data analyt- ics, among other names) is a term commonly used for the analytic process of extracting actionable information from continuously flowing/streaming data. A stream is defined as a continuous sequence of data elements (Zikopoulos et al., 2013). The data elements in a stream are often called tuples. In a relational database sense, a tuple is similar to a row of data (a record, an object, an instance). However, in the context of semistructured or un- structured data, a tuple is an abstraction that represents a package of data, which can be characterized as a set of attributes for a given object. If a tuple by itself is not sufficiently informative for analysis or a correlation—or other collective relationships among tuples are needed—then a window of data that includes a set of tuples is used. A window of data is a finite number/sequence of tuples, where the windows are continuously updated as new data become available. The size of the window is determined based on the system being analyzed. Stream analytics is becoming increasingly more popular because of two things. First, time-to-action has become an ever-decreasing value, and second, we have the technological means to capture and process the data while it is created.

Some of the most impactful applications of stream analytics were developed in the energy industry, specifically for smart grid (electric power supply chain) systems. The new smart grids are capable of not only real-time creation and processing of multiple streams of data to determine optimal power distribution to fulfill real customer needs, but also generating accurate short-term predictions aimed at covering unexpected de- mand and renewable energy generation peaks. Figure 9.9 shows a depiction of a generic use case for streaming analytics in the energy industry (a typical smart grid application). The goal is to accurately predict electricity demand and production in real time by using streaming data that is coming from smart meters, production system sensors, and meteo- rological models. The ability to predict near future consumption/production trends and detect anomalies in real time can be used to optimize supply decisions (how much to produce, what sources of production to use, and optimally adjust production capacities) as well as to adjust smart meters to regulate consumption and favorable energy pricing.

Stream Analytics versus Perpetual Analytics

The terms streaming and perpetual probably sound like the same thing to most people, and in many cases they are used synonymously. However, in the context of intelli- gent systems, there is a difference (Jonas, 2007). Streaming analytics involves applying transaction-level logic to real-time observations. The rules applied to these observa- tions take into account previous observations as long as they occurred in the prescribed window; these windows have some arbitrary size (e.g., last 5 seconds, last 10,000 obser- vations). Perpetual analytics, on the other hand, evaluates every incoming observation against all prior observations, where there is no window size. Recognizing how the new observation relates to all prior observations enables the discovery of real-time insight.

Both streaming and perpetual analytics have their pros and cons and their respec- tive places in the business analytics world. For example, sometimes transactional volumes

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 545

are high and the time-to-decision is too short, favoring nonpersistence and small win- dow sizes, which translates into using streaming analytics. However, when the mission is critical and transaction volumes can be managed in real time, then perpetual analytics is a better answer. That way, one can answer questions such as “How does what I just learned relate to what I have known?” “Does this matter?” and “Who needs to know?”

Critical Event Processing

Critical event processing is a method of capturing, tracking, and analyzing streams of data to detect events (out of normal happenings) of certain types that are worthy of the effort. Complex event processing is an application of stream analytics that combines data from multiple sources to infer events or patterns of interest either before they actually occur or as soon as they happen. The goal is to take rapid actions to prevent (or mitigate the negative effects of) these events (e.g., fraud or network intrusion) from occurring, or in the case of a short window of opportunity, take full advantage of the situation within the allowed time (based on user behavior on an e-commerce site, create promotional of- fers that they are more likely to respond to).

These critical events may be happening across the various layers of an organization such as sales leads, orders, or customer service calls. Or, more broadly, they may be news items, text messages, social media posts, stock market feeds, traffic reports, weather condi- tions, or other kinds of anomalies that may have a significant impact on the well-being of the organization. An event may also be defined generically as a “change of state,” which may be detected as a measurement exceeding a predefined threshold of time, tempera- ture, or some other value. Even though there is no denying the value proposition of critical event processing, one has to be selective in what to measure, when to measure, and how often to measure. Because of the vast amount of information available about events, which is sometimes referred to as the event cloud, there is a possibility of overdoing it, in which case as opposed to helping the organization, it may hurt the operational effectiveness.

Energy Production System (Traditional

and Renewable)

Energy Consumption System (Residential

and Commercial)

Sensor Data (Energy Production

System Status)

Meteorological Data (Wind, Light, Temperature, etc.)

Streaming Analytics (Predicting Usage, Production, and

Anomalies)

Usage Data (Smart Meters, Smart Grid Devices)

Permanent Storage Area

Data Integration and Temporary

Staging

Capacity Decisions

Pricing Decisions

FIGURE 9.9 A Use Case of Streaming Analytics in the Energy Industry.

546 Part III • Prescriptive Analytics and Big Data

Data Stream Mining

Data stream mining, as an enabling technology for stream analytics, is the process of extracting novel patterns and knowledge structures from continuous, rapid data records. As we saw in the data mining chapter (Chapter 4), traditional data mining methods require the data to be collected and organized in a proper file format, and then processed in a recursive manner to learn the underlying patterns. In contrast, a data stream is a continuous flow of an ordered sequence of instances that in many applications of data stream mining can be read/processed only once or a small number of times using limited computing and storage capabilities. Examples of data streams include sensor data, computer network traffic, phone conversations, ATM transactions, Web searches, and financial data. Data stream mining is considered a subfield of data mining, machine learning, and knowledge discovery.

In many data stream mining applications, the goal is to predict the class or value of new instances in the data stream given some knowledge about the class membership or values of previous instances in the data stream. Specialized machine-learning techniques (mostly derivative of traditional machine-learning techniques) can be used to learn this prediction task from labeled examples in an automated fashion. An example of such a prediction method was developed by Delen, Kletke, and Kim (2005), where they gradu- ally built and refined a decision tree model by using a subset of the data at a time.

Applications of Stream Analytics

Because of its power to create insight instantly, helping decision makers to be on top of events as they unfold and allowing organizations to address issues before they become problems, the use of streaming analytics is on an exponentially increasing trend. The fol- lowing are some of the application areas that have already benefited from stream analytics.

e-Commerce

Companies like Amazon and eBay (among many others) are trying to make the most out of the data that they collect while a customer is on their Web site. Every page visit, every product looked at, every search conducted, and every click made is recorded and ana- lyzed to maximize the value gained from a user’s visit. If done quickly, analysis of such a stream of data can turn browsers into buyers and buyers into shopaholics. When we visit an e-commerce Web site, even the ones where we are not a member, after a few clicks here and there we start to get very interesting product and bundle price offers. Behind the scenes, advanced analytics are crunching the real-time data coming from our clicks, and the clicks of thousands of others, to “understand” what it is that we are interested in (in some cases, even we do not know that) and make the most of that information by making creative offerings.

Telecommunications

The volume of data that come from call detail records (CDR) for telecommunications companies is astounding. Although this information has been used for billing purposes for quite some time now, there is a wealth of knowledge buried deep inside this Big Data that the telecommunications companies are just now realizing to tap. For instance, CDR data can be analyzed to prevent churn by identifying networks of callers, influencers, leaders, and followers within those networks and proactively acting on this information. As we all know, influencers and leaders have the effect of changing the perception of the followers within their network toward the service provider, either positively or nega- tively. Using social network analysis techniques, telecommunication companies are iden- tifying the leaders and influencers and their network participants to better manage their customer base. In addition to churn analysis, such information can also be used to recruit new members and maximize the value of the existing members.

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 547

Salesforce has expanded their Marketing Cloud ser- vices to include Predictive Scores and Predictive Audience features called the Marketing Cloud Predictive Journey. This addition uses real-time streaming data to enhance the customer engagement online. First, the customers are given a Predictive Score unique to them. This score is calculated from several different factors, including how long their browsing history is, if they clicked an e-mail link, if they made a purchase, how much they spent, how long ago did they make a purchase, or if they have ever responded to an e-mail or ad campaign. Once customers have a score, they are then segmented into different groups. These groups are given different marketing objectives and plans based on the predictive behaviors assigned to them. The scores and segments are updated and changed daily and give companies a better road map to target and achieve a desired response. These mar- keting solutions are more accurate and create more personalized ways companies can accommodate their customer retention methods.

Questions for DisCussion

1. Are there areas in any industry where streaming data is irrelevant?

2. Besides customer retention, what are other ben- efits of using predictive analytics?

What Can We Learn from This Case?

Through the analysis of data acquired in the here and now, companies are able to make predictions and decisions about their consumers more rap- idly. This ensures that businesses target, attract, and retain the right customers and maximize their value. Data acquired last week is not as beneficial as the data companies have today. Using relevant data makes our predictive analysis more accurate and efficient.

Sources: M. Amodio. (2015). “Salesforce Adds Predictive Analytics to Marketing Cloud. Cloud Contact Center.” www.cloudcontactcenter zone.com/topics/cloud- contact-center/articles/413611- salesforce-adds-predictive-analytics-marketing-cloud.htm (ac- cessed October 2018). J. Davis. (2015). “Salesforce Adds New Predictive Analytics to Marketing Cloud.” Information Week. www.information week.com/big-data/big-data-analytics/salesforce-adds- new-predictive-analytics-to-marketing-cloud/d/d-id/1323201 (accessed October 2018). D. Henschen. (2016). “Salesforce Reboots Wave Analytics, Preps IoT Cloud.” ZD Net. www.zdnet.com/ article/salesforce-reboots-wave-analytics-preps-iot-cloud/ (ac- cessed October 2018).

Application Case 9.6 Salesforce Is Using Streaming Data to Enhance Customer Value

Continuous streams of data that come from CDR can be combined with social media data (sentiment analysis) to assess the effectiveness of marketing campaigns. Insight gained from these data streams can be used to rapidly react to adverse effects (which may lead to loss of customers) or boost the impact of positive effects (which may lead to maximizing purchases of existing customers and recruitment of new customers) observed in these campaigns. Furthermore, the process of gaining insight from CDR can be rep- licated for data networks using Internet protocol detail records. Because most telecom- munications companies provide both of these service types, a holistic optimization of all offerings and marketing campaigns could lead to extraordinary market gains. Application Case 9.6 is an example of how Salesforce.com gets a better sense of its customers based upon an analysis of clickstreams.

Law Enforcement and Cybersecurity

Streams of Big Data provide excellent opportunities for improved crime prevention, law enforcement, and enhanced security. They offer unmatched potential when it comes to security applications that can be built in the space, such as real-time situ- ational awareness, multimodal surveillance, cyber-security detection, legal wiretapping, video surveillance, and face recognition (Zikopoulos et al., 2013). As an application of

548 Part III • Prescriptive Analytics and Big Data

information assurance, enterprises can use streaming analytics to detect and prevent net- work intrusions, cyberattacks, and malicious activities by streaming and analyzing net- work logs and other Internet activity monitoring resources.

Power Industry

Because of the increasing use of smart meters, the amount of real-time data collected by power utilities is increasing exponentially. Moving from once a month to every 15 min- utes (or more frequently), meter reading accumulates large quantities of invaluable data for power utilities. These smart meters and other sensors placed all around the power grid are sending information back to the control centers to be analyzed in real time. Such analyses help utility companies to optimize their supply chain decisions (e.g., capacity adjustments, distribution network options, real-time buying or selling) based on the up- to-the-minute consumer usage and demand patterns. In addition, utility companies can integrate weather and other natural conditions data into their analytics to optimize power generation from alternative sources (e.g., wind, solar) and to better forecast energy de- mand on different geographic granulations. Similar benefits also apply to other utilities such as water and natural gas.

Financial Services

Financial service companies are among the prime examples where analysis of Big Data streams can provide faster and better decisions, competitive advantage, and regula- tory oversight. The ability to analyze fast-paced, high volumes of trading data at very low latency across markets and countries offers a tremendous advantage to making the split-second buy/sell decisions that potentially translate into big financial gains. In addition to optimal buy/sell decisions, stream analytics can also help financial service companies in real-time trade monitoring to detect fraud and other illegal activities.

Health Sciences

Modern-era medical devices (e.g., electrocardiograms and equipment that measure blood pressure, blood oxygen level, blood sugar level, and body temperature) are capable of producing invaluable streaming diagnostic/sensory data at a very fast rate. Harnessing this data and analyzing it in real time offers benefits—the kind that we often call “life and death”—unlike any other field. In addition to helping healthcare companies become more effective and efficient (and hence more competitive and profitable), stream analyt- ics is also improving patient conditions and saving lives.

Many hospital systems all around the world are developing care infrastructures and health systems that are futuristic. These systems aim to take full advantage of what the technology has to offer, and more. Using hardware devices that generate high-resolution data at a very rapid rate, coupled with super-fast computers that can synergistically ana- lyze multiple streams of data, increases the chances of keeping patients safe by quickly detecting anomalies. These systems are meant to help human decision makers make faster and better decisions by being exposed to a multitude of information as soon as it becomes available.

Government

Governments around the world are trying to find ways to be more efficient (via optimal use of limited resources) and effective (providing the services that people need and want). As the practices for e-government become mainstream, coupled with widespread use and access to social media, very large quantities of data (both structured and un- structured) are at the disposal of government agencies. Proper and timely use of these

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 549

Big Data streams differentiates proactive and highly efficient agencies from the ones who are still using traditional methods to react to situations as they unfold. Another way in which government agencies can leverage real-time analytics capabilities is to man- age natural disasters such as snowstorms, hurricanes, tornadoes, and wildfires through a surveillance of streaming data coming from radar, sensors, and other smart detection devices. They can also use similar approaches to monitor water quality, air quality, and consumption patterns and detect anomalies before they become significant problems. Another area where government agencies use stream analytics is in traffic management in congested cities. By using the data coming from traffic flow cameras, GPS data com- ing from commercial vehicles, and traffic sensors embedded in roadways, agencies are able to change traffic light sequences and traffic flow lanes to ease the pain caused by traffic congestion problems.

u SECTION 9.7 REVIEW QUESTIONS

1. What is a stream (in the Big Data world)? 2. What are the motivations for stream analytics? 3. What is stream analytics? How does it differ from regular analytics? 4. What is critical event processing? How does it relate to stream analytics? 5. Define data stream mining. What additional challenges are posed by data stream

mining?

6. What are the most fruitful industries for stream analytics? 7. How can stream analytics be used in e-commerce? 8. In addition to what is listed in this section, can you think of other industries and/or

application areas where stream analytics can be used?

9. Compared to regular analytics, do you think stream analytics will have more (or less) use cases in the era of Big Data analytics? Why?

9.8 BIG DATA VENDORS AND PLATFORMS

The Big Data vendor landscape is developing very rapidly. As is the case with many emerging technologies, even the terms change. Many Big Data technologies or solutions providers have rechristened themselves to be AI providers. In this section, we will do a quick overview of several categories of Big Data providers. Then we will briefly describe one provider’s platform.

One way to study Big Data vendors and platforms is to go back to Chapter 1’s analytics ecosystem (depicted in Figure 1.17). If we focus on some of the outermost petals of that analyt- ics flower, we can see some categories of Big Data platform offerings. A more detailed classi- fication of Big Data/AI providers is also included in Matt Turck’s Big Data Ecosystem blog and the associated figure available at http://mattturck.com/wp-content/uploads/2018/07/ Matt_Turck_FirstMark_Big_Data_Landscape_2018_Final_reduced-768x539.png (ac- cessed October 2018). The reader is urged to check this site frequently to get updated ver- sions of his view of the Big Data ecosystem.

In terms of technology providers, one thing is certain: everyone wants a bigger share of the technology spending pie and is thus willing to offer every single piece of technology, or partner with another provider so that the customer does not consider a competitor offering. Thus, many players seem to compete with each other by adding capabilities that their partners offer or by collaborating with their partners. In addi- tion, there is always significant merger/acquisition activity. Finally, most vendors keep changing their products’ names as the platforms evolve. This makes this specific sec- tion likely to be obsolete sooner than one might think. Recognizing all these caveats,

550 Part III • Prescriptive Analytics and Big Data

one highly aggregated way to group the Big Data providers is to use the following broad categories:

• Infrastructure Services Providers • Analytics Solution Providers • Legacy BI Providers Moving to Big Data

Infrastructure Services Providers

Big Data infrastructure was initially developed by two companies coming out of initial col- laboration between Yahoo and Facebook. A number of vendors have developed their own Hadoop distributions, most based on the Apache open source distribution but with various levels of proprietary customization. Two market leaders were Cloudera (cloudera.com) and Hortonworks (hortonworks.com). Cloudera was started by Big Data experts includ- ing Hadoop creator Doug Cutting and former Facebook data scientist Jeff Hammerbacher. Hortonworks was spun out of Yahoo! These two companies have just (October 2018) an- nounced a plan to merge into one company to provide a full suite of services in Big Data. The combined company will be able to offer Big Data services and be able to compete and partner with all other major providers. This makes it perhaps the largest independent provider of Hadoop distribution that provides on-premise Hadoop infrastructure, train- ing, and support. MapR (mapr.com) offers its own Hadoop distribution that supplements HDFS with its proprietary network file system (NFS) for improved performance. Similarly, EMC was acquired by Dell to provide its own Big Data on-premise distribution. There are many other vendors that offer similar platforms with their own minor variations.

Another category of Hadoop distributors that add their own value-added services for customers are companies such as Datastax, Nutanix, VMWare, and so on. These compa- nies deliver commercially supported versions of the various flavors of NoSQL. DataStax, for example, offers a commercial version of Cassandra that includes enterprise support and services, as well as integration with Hadoop and open source enterprise search. Many other companies provide Hadoop connectors and complementary tools aimed at making it easier for developers to move data around and within Hadoop clusters.

The next category of major infrastructure providers is the large cloud providers such as Amazon Web Services, Microsoft Azure, Google Cloud, and IBM Cloud. All of these com- panies offer storage and computing services but have invested heavily to provide Big Data and AI technology offerings. For example, Amazon AWS includes Hadoop and many other Big Data/AI capabilities (e.g., Amazon Neptune). Azure is a popular cloud provider for many analytics vendors, but Azure also offers its own Machine Learning and other capabilities. IBM and Google similarly offer their cloud services, but have major data science/AI offerings available, such as IBM Watson analytics and Google Tensor Flow, AutoML, and so on.

Analytics Solution Providers

The analytics layer of the Big Data stack is also experiencing significant development. Not surprisingly, all major traditional analytics and data service providers have incor- porated Big Data analytics capabilities into their offerings. For example, Dell EMC, IBM Big Insights (now part of Watson), Microsoft Analytics, SAP’s Hanna, Oracle Big Data, and Teradata have all integrated Hadoop, Streaming, IoT, and Spark capabilities into their platforms. IBM’s BigInsights platform is based on Apache Hadoop, but includes numerous proprietary modules including the Netezza database, InfoSphere Warehouse, Cognos business intelligence tools, and SPSS data mining capabilities. It also offers IBM InfoSphere Streams, a platform designed for streaming Big Data analysis. With the suc- cess of Watson analytics brand, IBM has folded many of its analytics offerings in general and Big Data offerings in particular under the Watson label. Teradata Vantage similarly implements many of the commonly used analytics functions in the Big Data environment.

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 551

Further, as noted earlier, most of these platforms are also accessible through their own as well as public cloud providers. Rather than showing software details for all the platforms (which are quite similar anyway), we illustrate their description by using Teradata’s newest offering, Teradata Vantage, in Technology Insights 9.3.

Business Intelligence Providers Incorporating Big Data

In this group we note several of the major BI software providers who, again not surpris- ingly, have incorporated Big Data technologies into their offerings. The major names to note in this space include SAS, Microstrategy, and their peers. For example, SAS Viya claims to perform in-memory analytics on massive data. Data visualization specialist Tableau Software has added Hadoop and Next Generation Data Warehouse connectivity to its product suite. Relatively newer players such as Qlik and Spotfire also are adapting their offerings to include Big data capabilities.

Application Case 9.7 illustrates an example of a Big data project where both IBM and Teradata analytics software capabilities were used in addition to pulling data from Google and Twitter.

Infectious diseases impose a significant burden to the U.S. public health system. The rise of HIV/AIDS in the late 1970s, pandemic H1N1 flu in 2009, the H3N2 epidemic during the 2012–2013 winter season, the Ebola virus disease outbreak in 2015, and the Zika virus scare in 2016 have demonstrated the suscepti- bility of people to such contagious diseases. Virtually each year influenza outbreaks happen in various forms and result in consequences of varying impacts. The annual impact of seasonal influenza outbreaks in the United States is reported to be an average of 610,660 undiscounted life-years lost, 3.1 million hos- pitalized days, 31.4 million outpatient visits, and a total of $87.1 billion in economic burden. As a result of this growing trend, new data analytics techniques and technologies capable of detecting, tracking, map- ping, and managing such diseases have come on the scene in recent years. In particular, digital surveil- lance systems have shown promise in their capacity to discover public health-seeking patterns and trans- form these discoveries into actionable strategies.

This project demonstrated that social media can be utilized as an effective method for early detection of influenza outbreaks. We used a Big Data platform to employ Twitter data to monitor influenza activity in the United States. Our Big Data analytics meth- ods comprised temporal, spatial, and text mining. In the temporal analysis, we examined whether Twitter

data could indeed be adapted for the nowcasting of influenza outbreaks. In spatial analysis, we mapped flu outbreaks to the geospatial property of Twitter data to identify influenza hotspots. Text analytics was performed to identify popular symptoms and treatments of flu that were mentioned in tweets.

The IBM InfoSphere BigInsights platform was employed to analyze two sets of flu activity data: Twitter data were used to monitor flu outbreaks in the United States, and Cerner HealthFacts data warehouse was used to track real-world clinical encounters. A huge volume of flu-related tweets was crawled from Twitter using Twitter Streaming API and was then ingested into a Hadoop cluster. Once the data were successfully imported, the JSON Query Language (JAQL) tool was used to manip- ulate and parse semistructured JavaScript Object Notation (JSON) data. Next, Hive was used to tabu- larize the text data and segregate the information for the spatial-temporal location analysis and visualiza- tion in R. The entire data mining process was imple- mented using MapReduce functions. We used the package BigR to submit the R scripts over the data stored in HDFS. The package BigR enabled us to benefit from the parallel computation of HDFS and to perform MapReduce operations. Google’s Maps API libraries were used as a basic mapping tool to visualize the tweet locations.

Application Case 9.7 Using Social Media for Nowcasting Flu Activity

(Continued )

552 Part III • Prescriptive Analytics and Big Data

TECHNOLOGY INSIGHTS 9.3 An Illustrative Big Data Technology Platform: Teradata Vantage™

Introduction This description is adapted from content provided by Teradata, especially Sri Raghavan. Teradata Vantage is an advanced analytics platform embedded with analytic engines and functions, which can be implemented with preferred data science languages (e.g., SQL, Python, R) and tools (e.g., Teradata Studio, Teradata AppCenter, R Studio, Jupyter Notebook) on any data volume of any type by diverse analytics personas (e.g., Data Scientist, Citizen Data Scientist, Business Analyst) across multiple environments (On-Premises, Private Cloud, Public Cloud Marketplaces). There are five important conceptual pieces central to understanding Vantage: Analytics Engines and Functions, Data Storage and Access, Analytic Languages and Tools, Deployment, and Usage. Figure 9.10 illustrates the general architecture of Vantage and its interrelationships with other tools.

Analytic Engines and Functions

An analytic engine is a comprehensive framework that includes all the software components that are well integrated into a container (e.g., Docker) to deliver advanced analytics functionality that can be implemented by a well-defined set of user personas. An analytic engine’s components include:

• Advanced Analytics functions • Access points to data storage that can ingest multiple data types • Integration into visualization and analytic workflow tools • Built in management and monitoring tools • Highly scalable and performant environment with established thresholds

It is advantageous to have an analytic engine as it delivers a containerized compute envi- ronment that can be separated from data storage. Furthermore, analytic engines can be tailored for access and use by specific personas (e.g., DS, Business Analyst).

There are three analytic engines in the first release of Vantage. These are NewSQL Engine, Machine Learning Engine, and Graph Engine.

Our findings demonstrated that the integration of social media and medical records can be a valu- able supplement to the existing surveillance systems. Our results confirmed that flu-related traffic on social media is closely related with the actual flu outbreak. This has been shown by other researchers as well (St Louis & Zorlu, 2012; Broniatowski, Paul, & Dredze, 2013). We performed a time-series analysis to obtain the spatial-temporal cross- correlation between the two trends (91%) and observed that clinical flu encoun- ters lag behind online posts. In addition, our location analysis revealed several public locations from which a majority of tweets were originated. These findings can help health officials and governments to develop more accurate and timely forecasting models during outbreaks and to inform individuals about the loca- tions that they should avoid during that time period.

Questions for DisCussion

1. Why would social media be able to serve as an early predictor of flu outbreaks?

2. What other variables might help in predicting such outbreaks?

3. Why would this problem be a good problem to solve using Big Data technologies mentioned in this chapter?

Sources: A. H. Zadeh, H. M. Zolbanin, R. Sharda, & D. Delen. (2015). “Social Media for Nowcasting the Flu Activity: Spatial- Temporal and Text Analysis.” Business Analytics Congress, Pre-ICIS Conference, Fort Worth, TX. D. A. Broniatowski, M. J. Paul, & M. Dredze. (2013). “National and Local Influenza Surveillance through Twitter: An Analysis of the 2012–2013 Influenza Epidemic.” PloS One, 8(12), e83672. P. A. Moran. (1950). “Notes on Continuous Stochastic Phenomena.” Biometrika, 17–23.

Application Case 9.7 (Continued)

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 553

The NewSQL engine includes embedded analytic functions. Teradata will continue to add more functions for the high-speed analytics processing required to operationalize analytics. New functions within the NewSQL engine include:

• nPath • Sessionization • Attribution • Time series • 4D analytics • Scoring functions (e.g., Naïve Bayes, GLM, Decision Forests)

The Machine Learning engine delivers more than 120 prebuilt analytic functions for path, pat- tern, statistical, and text analytics to solve a range of business problems. Functions range from understanding sentiment to predictive part failure analysis.

The Graph engine provides a set of functions that discover relationships between people, products, and processes within a network. Graph analytics solve complex problems such as social network connections, influencer relationships, fraud detection, and threat identification.

Vantage embeds analytic engines close to the data, which eliminates the need to move data, allowing users to run their analytics against larger data sets without sampling and execute models with greater speed and frequency. This is made possible through the use of containers managed by Kubernetes, which allow businesses to easily manage and deploy new cutting-edge analytic engines, such as Spark and TensorFlow, both of which will be available in the near fu- ture. Another benefit of containers is the ability to scale out the engines.

From a user’s perspective, Vantage is a unified analytic and data framework. Under the covers, it contains a cross-engine orchestration layer that pipelines the right data and analytic re- quest to the right analytic engine across a high-speed data fabric. This enables a business analyst or data scientist, for example, to invoke analytic functions from different engines in a single ap- plication, such as Jupyter Notebook, without enduring the trouble of hopping from one analytic server or application to another. The result is a tightly integrated analytic implementation that’s not restrained by functional or data silos.

Data Storage and Access: Teradata Vantage comes with a natively embedded Teradata MPP Database. Furthermore, a high-speed data fabric (Teradata QueryGrid™ and Presto™) connects the platform to external data sources that include third-party enterprise data warehouses (e.g., Oracle), open source data platforms (e.g., Hadoop), no-SQL databases

H ig

h S

pe ed

F ab

ri c

SQL Engine

Data Storage

Analytic Engines

Analytic Languages

AppCenter

Analytic Tools

Machine Learning Engine

Teradata Data Store

Graph Engine

FIGURE 9.10 Teradata Vantage Architecture. Source: Teradata Corp.

554 Part III • Prescriptive Analytics and Big Data

Application Case 9.8 illustrates another application of Teradata Vantage where its advanced network analytics capabilities were deployed to analyze data from a large elec- tronic medical records data warehouse.

(e.g., Cassandra), and others. Data support ranges from relational, spatial, and temporal to XML, JSON, Avro, and time-series formats.

Analytic Languages and Tools: Teradata Vantage was built out of the recognition that ana- lytics professionals such as Data Scientists and Business Analysts require a diverse set of languages and tools to process large data volumes to deliver analytic insights. Vantage includes languages such as SQL, R, and Python on which analytics functions can be ex- ecuted through Teradata Studio, R Studio, and Jupyter Notebooks.

Deployment: Vantage platform provides the same analytic processing across deployment op- tions, including the Teradata Cloud and public cloud, as well as on-premises installations on Teradata hardware or commodity hardware. It is also available as a service.

Usage: Teradata Vantage is intended to be used by multiple analytic personas. The ease of SQL ensures that citizen data scientists and business analysts can implement prebuilt analytic func- tions integrated into the analytic engines. The ability to invoke Teradata-supported packages such as dplyr and teradataml ensures that Data Scientists familiar with R and Python can exe- cute analytic packages through R Studio and Jupyter notebooks, respectively, on the platform. Users who are not proficient at executing programs can invoke analytic functions codified into Apps built into Teradata AppCenter, an app building framework available in Vantage, to deliver compelling visualizations such as Sankey, Tree, Sigma diagrams, or word clouds.

Example Usage: A global retailer had a website that suboptimally delivered search results to potential buyers. With online purchases accounting for 25% of total sales, inaccurate search results negatively impacted the customer experience and the bottom line.

The retailer implemented Teradata machine learning algorithms, available in Teradata Vantage, to accumulate, parse, and classify search terms and phrases. The algorithms delivered the answers needed to identify search results that closely matched online customer needs. This led to more than $1 .3 million in incremental revenue from high-value customers, as measured by purchase volumes, over a two-month holiday period.

The Center for Health Systems Innovation at Oklahoma State University has been given a mas- sive data warehouse by Cerner Corporation, a major electronic medical records (EMRs) provider, to help develop analytic applications. The data warehouse contains EMRs on the visits of more than 50 million unique patients across U.S. hospitals (2000–2015). It is the largest and the industry’s only relational data- base that includes comprehensive records with phar- macy, laboratory, clinical events, admissions, and billing data. The database also includes more than 2.4 billion laboratory results and more than 295 million orders for nearly 4,500 drugs by name and brand. It is one of the largest compilations of de-identified, real-world, HIPAA-compliant data of its type.

The EMRs can be used to develop multiple ana- lytics applications. One application is to understand the relationships between diseases based on the information about the simultaneous diseases devel- oped in the patients. When multiple diseases are present in a patient, the condition is called comorbid- ity. The comorbidities can be different across popu- lation groups. In an application (Kalgotra, Sharda, & Croff, 2017), the authors studied health disparities in terms of comorbidities by gender.

To compare the comorbidities, a network analysis approach was applied. A network is com- prised of a defined set of items called nodes, which are linked to each other through edges. An edge represents a defined relationship between the

Application Case 9.8 Analyzing Disease Patterns from an Electronic Medical Records Data Warehouse

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 555

ICD-9 Description

0012139: Infectious and parasitic diseases

1402239: Neoplasms

2402279: Endocrine, nutritional and metabolic diseases, and immunity disorders

2802289: Diseases of the blood and blood-forming organs

2902319: Mental disorders

3202359: Diseases of the nervous system

3602389: Diseases of the sense organs

3902459: Diseases of the circulatory system

4602519: Diseases of the respiratory system

5202579: Diseases of the digestive system

5802629: Diseases of the genitourinary system

6302679: Complications of pregnancy, childbirth, and the puerperium

6802709: Diseases of the skin and subcutaneous tissue

7102739: Diseases of the musculoskeletal system and connective tissue

7402759: Congenital anomalies

7602779: Certain conditions originating in the perinatal period

8002999: Injury and poisoning

Male Comorbidity Network

Female Comorbidity Network

FIGURE 9.11 Female and Male Comorbidity Networks.

(Continued )

556 Part III • Prescriptive Analytics and Big Data

As noted earlier, our goal in this section is to highlight some of the players in Big data technology space. In addition to the vendors listed above, there are hundreds of oth- ers in the categories identified earlier as well as very specific industry applications. Rather than listing these names here, we urge you to check the latest version of the Big Data analytics ecosystem at http://mattturck.com/bigdata2018/ (accessed October 2018). Matt Turck’s updated ecosystem diagram identifies companies in each cluster.

u SECTION 9.8 REVIEW QUESTIONS

1. Identify some of the key Big Data technology vendors whose key focus is on-premise Hadoop platforms.

2. What is special about the Big Data vendor landscape? Who are the big players? 3. Search and identify the key similarity and differences between cloud providers’ ana-

lytics offerings and analytics providers’ presence on specific cloud platforms.

4. What are some of the features of a platform such as Teradata Vantage?

nodes. A very common example of network is a friendship network in which individuals are con- nected to each other if they are friends. Other com- mon networks are computer networks, Web page networks, road networks, and airport networks. To compare the comorbidities, networks of the diagnoses developed by men and women were created. The information about the diseases devel- oped by each patient in the lifetime history was used to create a comorbidity network. For the analysis, 12 million female patients and 9.9 mil- lion male patients were used. To manage such a huge data set, Teradata Aster Big Data platform was used. To extract and prepare the network data, SQL, SQL-MR, and SQL-GR frameworks supported by Aster were utilized. To visualize the networks, Aster AppCenter and Gephi were used.

Figure 9.11 presents the female and male comor- bidity networks. In these networks, nodes represent different diseases classified as the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM), aggregated at the three- digit level. Two diseases are linked based on the similarity calculated using Salton Cosine Index. The larger the size of a node, the greater the comorbid- ity of that disease. The female comorbidity network is denser than the male network. The number of nodes and edges in the female network are 899 and 14,810, respectively, whereas the number of nodes and edges in the male network are 839 and 12,498,

respectively. The visualizations present a difference between the pattern of diseases developed in male and female patients. Specifically, females have more comorbidities of mental disorders than males. On the other hand, the strength of some disease asso- ciations between lipid metabolism and chronic heart disorders is stronger in males than females. Such health disparities present questions for biological, behavioural, clinical, and policy research.

The traditional database systems would be taxed in efficiently processing such a huge data set. The Teradata Aster made the analysis of data con- taining information on millions of records fairly fast and easy. Network analysis is often suggested as one method to analyze big data sets. It helps under- stand the data in one picture. In this application, the comorbidity network explains the relationship between diseases at one place.

Questions for DisCussion

1. What could be the reasons behind the health dis- parities across gender?

2. What are the main components of a network?

3. What type of analytics was applied in this application?

Source: Kalgotra, P., Sharda, R., & Croff, J. M. (2017). Examining health disparities by gender: A multimorbidity network analysis of electronic medical record. International Journal of Medical Informatics, 108, 22–28.

Application Case 9.8 (Continued)

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 557

9.9 CLOUD COMPUTING AND BUSINESS ANALYTICS

Another emerging technology trend that business analytics users should be aware of is cloud computing. The National Institute of Standards and Technology (NIST) defines cloud computing as “a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, and services) that can be rapidly provisioned and released with minimal management effort or service-provider interaction.” Wikipedia (n.d., Cloud Computing) defines cloud comput- ing as “a style of computing in which dynamically scalable and often virtualized resources are provided over the Internet. Users need not have knowledge of, experience in, or con- trol over the technology infrastructures in the cloud that supports them.” This definition is broad and comprehensive. In some ways, cloud computing is a new name for many previous, related trends: utility computing, application service provider grid computing, on-demand computing, software as a service (SaaS), and even older, centralized com- puting with dumb terminals. But the term cloud computing originates from a reference to the Internet as a “cloud” and represents an evolution of all of the previously shared/ centralized computing trends. The Wikipedia entry also recognizes that cloud computing is a combination of several IT components as services. For example, infrastructure as a service (IaaS) refers to providing computing platforms as a service (PaaS), as well as all of the basic platform provisioning, such as management administration, security, and so on. It also includes SaaS, which includes applications to be delivered through a Web browser, whereas the data and the application programs are on some other server.

Although we do not typically look at Web-based e-mail as an example of cloud com- puting, it can be considered a basic cloud application. Typically, the e-mail application stores the data (e-mail messages) and the software (e-mail programs that let us process and manage e-mails). The e-mail provider also supplies the hardware/software and all of the basic infrastructure. As long as the Internet is available, one can access the e-mail application from anywhere in the cloud. When the application is updated by the e-mail provider (e.g., when Gmail updates its e-mail application), it becomes available to all cus- tomers. Social networking Web sites like Facebook, Twitter, and LinkedIn, are also exam- ples of cloud computing. Thus, any Web-based general application is in a way an example of a cloud application. Another example of a general cloud application is Google Docs and Spreadsheets. This application allows a user to create text documents or spreadsheets that are stored on Google’s servers and are available to the users anywhere they have ac- cess to the Internet. Again, no programs need to be installed as “the application is in the cloud.” The storage space is also “in the cloud.” Even Microsoft’s popular office applica- tions are all available in the cloud, with the user not needing to download any software.

A good general business example of cloud computing is Amazon.com’s Web services. Amazon.com has developed an impressive technology infrastructure for e-commerce as well as for BI, customer relationship management, and supply-chain management. It has built major data centers to manage its own operations. However, through Amazon.com’s cloud services, many other companies can employ these very same facilities to gain advantages of these technologies without having to make a similar investment. Like other cloud-computing services, a user can subscribe to any of the fa- cilities on a pay-as-you-go basis. This model of letting someone else own the hardware and software but making use of the facilities on a pay-per-use basis is the cornerstone of cloud computing. A number of companies offer cloud-computing services, including Salesforce.com, IBM Cloud, Microsoft Azure, Google, Adobe, and many others.

Cloud computing, like many other IT trends, has resulted in new offerings in ana- lytics. These options permit an organization to scale up its data warehouse and pay only for what it uses. The end user of a cloud-based analytics service may use one or- ganization for analysis applications that, in turn, uses another firm for the platform or

558 Part III • Prescriptive Analytics and Big Data

infrastructure. The next several paragraphs summarize the latest trends in the interface of cloud computing and BI/business analytics. A few of these statements are adapted from an early paper written by Haluk Demirkan and one of the coauthors of this book (Demirkan & Delen, 2013).

Figure 9.12 illustrates a conceptual architecture of a service-oriented decision sup- port environment, that is, a cloud-based analytics system. This figure superimposes the cloud-based services on the general analytics architecture presented in previous chapters.

In service-oriented decision support solutions, (1) operational systems, (2) data warehouses, (3) online analytic processing, and (4) end-user components can be ob- tained individually or bundled and provided to the users as service. Any or all of these services can be obtained through the cloud. Because the field of cloud computing is fast evolving and growing at a rapid pace, there is much confusion about the terminology being used by various vendors and users. The labels vary from Infrastructure, Platform, Software, Data, Information, and Analytics as a Service. In the following, we define these services. Then we summarize the current technology platforms and highlight applications of each through application cases.

Data as a Service (DaaS)

The concept of data as a service basically advocates the view that “where data lives”— the actual platform on which the data resides—doesn’t matter. Data can reside in a local computer or in a server at a server farm inside a cloud-computing environment.

Information Sources

Data Management

Information Management

Operations Management

Analytics Service

Information Service

Data Service

Data mart (…)

Replication

External Data

Other OLTP/Web

POS

Legacy

ERP

Enterprise Data Warehouse

ETL

Servers Software

Intranet Search for Content

Dashboards

OLAP

Routine Business Reporting

Metadata

Data mart (Marketing)

Data mart (Engineering)

Data mart (Finance)

Data Mining

Text Mining

Simulation

Automated Decision System

Optimization

FIGURE 9.12 Conceptual Architecture of a Cloud-Oriented Support System. Source: Based on Demirkan, H., & Delen, D. (2013, April). Leveraging the capabilities of service-oriented decision support systems: Putting analytics and Big Data in cloud.

Decision Support Systems, 55(1), 412–421.

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 559

With DaaS, any business process can access data wherever it resides. Data as a service began with the notion that data quality could happen in a centralized place, cleansing and enriching data and offering it to different systems, applications, or users, irrespective of where they were in the organization, computers, or on the network. This has now been replaced with master data management and customer data integration solutions, where the record of the customer (or product, or asset, etc.) may reside anywhere and is available as a service to any application that has the services allowing access to it. By applying a standard set of transformations to the various sources of data (for example, ensuring that gender fields containing different notation styles [e.g., M/F, Mr./Ms.] are all translated into male/female) and then enabling applications to access the data via open standards such as SQL, XQuery, and XML, service requestors can access the data regard- less of vendor or system.

With DaaS, customers can move quickly thanks to the simplicity of the data access and the fact that they don’t need extensive knowledge of the underlying data. If cus- tomers require a slightly different data structure or have location-specific requirements, the implementation is easy because the changes are minimal (agility). Second, provid- ers can build the base with the data experts and outsource the analysis or presentation layers (which allows for very cost-effective user interfaces and makes change requests at the presentation layer much more feasible), and access to the data is controlled through the data services. It tends to improve data quality because there is a single point for updates.

Software as a Service (SaaS)

This model allows consumers to use applications and software that run on distant comput- ers in the cloud infrastructure. Consumers need not worry about managing underlying cloud infrastructure and have to pay for the use of software only. All we need is a Web browser or an app on a mobile device to connect to the cloud. Gmail is an example of SaaS.

Platform as a Service (PaaS)

Using this model, companies can deploy their software and applications in the cloud so that their customers can use them. Companies don’t have to manage resources needed to manage their applications in cloud-like networks, servers, storage, or operating systems. This reduces the cost of maintaining underlying infrastructure for running their software and also saves time for setting up this infrastructure. Now, users can focus on their busi- ness rather than focusing on managing infrastructure for running their software. Examples of PaaS are Microsoft Azure, Amazon EC2, and Google App Engine.

Infrastructure as a Service (IaaS)

In this model, infrastructure resources like networks, storage, servers, and other comput- ing resources are provided to client companies. Clients can run their application and have administrative rights to use these resources but do not manage underlying infrastructure. Clients have to pay for usage of infrastructure. A good example of this is Amazon.com’s Web services. Amazon.com has developed impressive technology infrastructure that in- cludes data centers. Other companies can use Amazon.com’s cloud services on a pay- per-use-basis without having to make similar investments. Similar services are offered by all major cloud providers such as IBM, Microsoft, Google, and so on.

We should note that there is considerable confusion and overlap in the use of cloud terminology. For example, some vendors also add information as a service (IaaS), which is an extension of DaaS. Clearly, this IaaS is different from infrastructure as a service described earlier. Our goal here is to just recognize that there are varying

560 Part III • Prescriptive Analytics and Big Data

degrees of services that an organization can subscribe to in order to manage the ana- lytics applications. Figure 9.13 highlights the level of service subscriptions a client uses in each of the three major types of cloud offerings. SaaS is clearly the highest level of cloud service that a client may get. For example, in using Office 365, an organization is using the software as a service. The client is only responsible for bringing in the data. Many of the analytics as a service application fall in this category as well. Further, several analytics as a service provider may in turn use clouds such as Amazon’s AWS or Microsoft Azure to provide their services to the end users. We will see examples of such services shortly.

Essential Technologies for Cloud Computing

VIRTUALIZATION Virtualization is the creation of a virtual version of something like an operating system or server. A simple example of virtualization is the logical division of a hard drive to create two separate hard drives in a computer. Virtualization can be in all three areas of computing:

Network virtualization: It is the splitting of available bandwidth into channels, which disguises complexity of the network by dividing it into manageable parts. Then each bandwidth can be allocated to a particular server or device in real time.

Storage virtualization: It is the pooling of physical storage from multiple network storage devices into a single storage device that can be managed from a central console.

Server virtualization: It is the masking of physical servers from server users. Users don’t have to manage the actual servers or understand complicated details of server resources.

This difference in the level of virtualization directly relates to which cloud service one employs.

Application Case 9.9 illustrates an application of cloud technologies that enable a mobile application and allow for significant reduction in information miscommunication.

Networking

Infrastructure as a Service

IaaS

Storage

Servers

Virtualization

Operating System

Middleware

Runtime

Data

Application

Platform as a Service

PaaS

Networking

Storage

Servers

Virtualization

Operating System

Middleware

Runtime

Data

Application

Software as a Service

SaaS

Networking

Storage

Servers

Virtualization

Operating System

Middleware

Runtime

Data

Application

Managed by Client

Managed by Cloud Vendor

FIGURE 9.13 Technology Stack as a Service for Different Types of Cloud Offerings.

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 561

Historical communication between utilities and first responders has been by phone calls or two-way radios. Some of these are with first responders on the scene, and some with dispatch or other units of the first responder organization. When a member of the public sees an incident on the field, they usu- ally just call 911, which is routed to first respond- ers. Dispatch centers route the closest first responder to the field, who then call back to the center either on their radios or cell phones to let them know the actual status. The dispatch centers then call the inci- dent in to the appropriate utility, who then sends their own team to the field for further resolution. This also requires that the exact location be conveyed to the dispatch center from the field, and from the for- mer to utility—particularly challenging if the incident location is not at a specific address (e.g., along a free- way, across open land, etc.). The utility also needs to let the dispatch center know the status of their own crew. This information must also be relayed to the first responders on the field. Much of this process relies on information being communicated orally and then forwarded to one or more recipients, with infor- mation also flowing back and forth along the same chain. All of this can result in garbled communication and/or incomplete messages, which can eat away precious minutes or even hours in emergencies.

A major West Coast Utility, a leader in using technology to address traditional problems, deter- mined that many of these challenges can be addressed through better information sharing in a timelier manner using cloud-mobile technology. Their territory encompassed densely populated cit- ies to far-flung rural communities with intervening miles of desert, national parks, and more.

Recognizing that most first responders have a smartphone or tablet, the utility selected Connixt’s iMarq™ mobile suite to provide a simple-to-use mobile app that allows first responders to advise the utility of any incident in the field. The technol- ogy also keeps the first responders apprised of the utility’s response status with respect to the incident.

With a targeted base of over 20,000 first respond- ers spread across the entire territory, lowering barri- ers to adoption was an especially important factor. “Improving communication with groups that are outside your organization is historically difficult,” says G. Satish,

cofounder and CEO, Connixt. “For this deployment, the focus on simplicity is the key to its success.”

First responders are invited to download and self-register the app, and once the utility grants access rights, they can report incidents using their own tab- lets or smartphones. The first responder simply uses a drop-down menu to pick from a list of preconfigured incidents, taps an option to indicate if they will wait at the scene, and attach photographs with annotations— all with a few touches on their device. The utility receives notification of the incident, reviews the time and geostamped information (no more mixed-up addresses), and updates their response. This response (which may be a truck roll) is sent to the first respond- ers and maintained in the app.

The simplicity of the solution makes it easy for the first responders. They use their own phone or tablet, communicate in a way they are used to, and provide needed information simply and effectively. They can see the utility updates (such as the current status of the truck that was sent). Missed or garbled phone messages are minimized. Options such as recording voice memos, using speech-to-text and more, are also available.

Cloud technology has been particularly use- ful in this case—deployment is faster without issues related to hardware procurement, installation, and appropriate backups. Connixt’s cloud-based Mobile Extension Framework (MXF™) is architected for rapid configuration and deployment—configuration is completed in the cloud, and, once configured, the apps are ready for download and deployment. More importantly, MXF enables easy modifications to forms and processes—for example, if the utility needs to add additional options to the incident drop- down, they simply add this once in MXF. Within minutes the option is now available on the field for all users. Figure 9.14 illustrates this architecture.

There are further benefits from a system that leverages ubiquitous cloud and mobile technologies. Because all of the business logic and configurations are stored in the cloud, the solution itself can act as a stand-alone system for customers who have no back- end systems—very important in the context of small and medium businesses (SMBs). And for those with back-end systems, the connectivity is seamless through Web services and the back-end system serves as the

Application Case 9.9 Major West Coast Utility Uses Cloud-Mobile Technology to Provide Real-Time Incident Reporting

(Continued )

562 Part III • Prescriptive Analytics and Big Data

system of record. This additionally helps businesses adopt technology in a phased manner—starting with a noninvasive, standalone system with minimal inter- nal IT impact while automating field operations, and then moving toward back-end system integration.

On the other hand, the mobile apps are them- selves system agnostic—they communicate using standard Web services and the end device can be Android or iOS and smartphone or tablet. Thus, irrespective of the device used, all communication, business logic, and algorithms are standardized across platforms/devices. As native apps across all devices, iMarq leverages standard technology that is provided by the device manufacturers and the OS vendors. For example, using native maps appli- cations allows the apps to benefit from improve- ments made by the platform vendors; thus, as maps become more accurate, the end users of the mobile apps also benefit from these advances.

Finally, for successful deployments, enterprise cloud-mobile technology has to be heavily user-centric. The look and feel must be geared to user-comfort, much as users expect from any mobile app they use. Treating the business user as an app consumer meets their standard expectations of an intuitive app that immediately saves them time and effort. This approach is essential to ensuring successful adoption.

The utility now has better information from first responders, as information is directly shared

from the field (not through a dispatcher or other third party), pictures are available, and there is geo- and time stamping. Garbled phone messages are avoided. The two-way communication between util- ity and the first responder in the field is improved. Historical records of the incidents are kept.

The utility and the first responders are now more unified in their quick and complete responses to incidents, improving service to the public. By tightening ties with first responders (police and fire department personnel), the public is served with a better coordinated and superior response for inci- dents that are discovered by first responders.

Questions for DisCussion

1. How does cloud technology impact enterprise software for small and mid-size businesses?

2. What are some of the areas where businesses can use mobile technology?

3. What types of businesses are likely to be the fore- runners in adopting cloud-mobile technology?

4. What are the advantages of cloud-based enterprise software instead of the traditional on-premise model?

5. What are the likely risks of cloud versus tradi- tional on-premise applications?

Source: Used with permission from G Satish, Connixit, Inc.

2

Configure Connixt MXF for integration, business rules etc.

CUSTOMER

FIELD WORKER MANAGER

Field Users download ConnixtApp

1

Mobile Extension Framework

(connixtMXFTM)

Business Rules Webservices

Workflows

User Objects

Adaptors

Authentication

MAC- HINE NEEDS ATTEN- TION

Customer Back-end System(s)

FIGURE 9.14 Interconnections between workers and technology in a cloud analytics application.

Application Case 9.9 (Continued)

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 563

Cloud Deployment Models

Cloud services can be acquired in several ways, from building an entirely private infra- structure to sharing with others. The following three models are the most common.

Private cloud: This can also be called internal cloud or corporate cloud. It is a more secure form of cloud service than public clouds like Microsoft Azure and Amazon Web Services. It is operated solely for a single organization having a mis- sion critical workload and security concerns. It provides the same benefits as a public cloud-like service, scalability, changing computing resources on demand, and so on. Companies that have a private cloud have direct control over their data and applications. The disadvantage of having a private cloud is the cost of main- taining and managing the cloud because on-premise IT staff are responsible for managing it.

Public cloud: In this model the subscriber uses the resources offered by ser- vice providers over the Internet. The cloud infrastructure is managed by the service provider. The main advantage of this public cloud model is saving time and money in setting up hardware and software required to run their business. Examples of public clouds are Microsoft Azure, Google Cloud Platform, and Amazon AWS.

Hybrid cloud: The hybrid cloud gives businesses great flexibility by moving work- loads between private and public clouds. For example, a company can use hybrid cloud storage to store its sales and marketing data, and then use a public cloud platform like Amazon Redshift to run analytical queries to analyze its data. The main requirement is network connectivity and API (application program interface) com- patibility between the private and public cloud.

Major Cloud Platform Providers in Analytics

This section first identifies some key cloud players that provide the infrastructure for analytics as a service, as well as selected analytics functionalities. Then we also mention representative analytics-as-a-service offerings that may even run on these cloud platforms.

Amazon Elastic Beanstalk: Amazon Elastic Beanstalk is a service offered from Amazon Web Services. It can deploy, manage, and scale Web applications. It sup- ports the following programming languages: Java, Ruby, Python, PHP, and .NET on servers like Apache HTTP, Apache Tomcat, and IIS. A user has to upload the code for the application, and Elastic Beanstalk handles the deployment of the application, load balancing, and autoscaling and monitors the health of the application. So the user can focus on building Web sites, mobile applications, API backend, content management systems, SaaS, and so on, while the applications and infrastructure to manage them are taken care of by Elastic Beanstalk. A user can use Amazon Web Services or an integrated development environment like Eclipse or Visual Studio to upload their application. A user has to pay for AWS resources needed to store and run the applications.

IBM Cloud: IBM Cloud is a cloud platform that allows a user to build apps using many open source computer technologies. Users can also deploy and manage hy- brid applications using the software. With IBM Watson, whose services are available on IBM Cloud, users can now create next-generation cognitive applications that can discover, innovate, and make decisions. IBM Watson services can be used for ana- lyzing emotions and synthesizing natural-sounding speech from text. Watson uses the concept of cognitive computing to analyze text, video, and images. It supports programming languages like Java, Go, PHP, Ruby, and Python.

Microsoft Azure: Azure is a cloud platform created by Microsoft to build, deploy, and manage applications and services through a network of Microsoft data centers.

564 Part III • Prescriptive Analytics and Big Data

It serves as both PaaS and IaaS and offers many solutions such as analytics, data warehousing, remote monitoring, and predictive maintenance.

Google App Engine: Google App Engine is Google’s Cloud computing platform used for developing and hosting applications. Managed by Google’s data centers, it supports developing apps in Python, Java, Ruby, and PHP programming languages. The big query environment offers data warehouse services through the cloud.

Openshift: Openshift is Red Hat’s cloud application platform based on a PaaS model. Through this model, application developers can deploy their applications on the cloud. There are two different models available for openshift. One serves as a public PaaS and the other serves as a private PaaS. Openshift Online is Red Hat’s public PaaS that offers development, build, hosting, and deployment of ap- plications in the cloud. The private PaaS, openshift Enterprise, allows develop- ment, build, and deployment of applications on an internal server or a private cloud platform.

Analytics as a Service (AaaS)

Analytics and data-based managerial solutions—the applications that query data for use in business planning, problem solving, and decision support—are evolving rapidly and being used by almost every organization. Enterprises are being flooded with information, and getting insights from this data is a big challenge for them. Along with that, there are challenges related to data security, data quality, and compliance. AaaS is an extensible analytical platform using a cloud-based delivery model where various BI and data analyt- ics tools can help companies in better decision making and get insights from their huge amount of data. The platform covers all functionality aspects from collecting data from physical devices to data visualization. AaaS provides an agile model for reporting and analytics to businesses so they can focus on what they do best. Customers can either run their own analytical applications in the cloud or they can put their data on the cloud and receive useful insights.

AaaS combines aspects of cloud computing with Big Data analytics and empowers data scientists and analysts by allowing them to access centrally managed information data sets. They can now explore information data sets more interactively and discover richer insights more rapidly, thus erasing many of the delays that they may face while discovering data trends. For example, a provider might offer access to a remote analyt- ics platform for a fee. This allows the client to use analytics software for as long as it is needed. AaaS is a part of SaaS, PaaS, and IaaS, thus helping IT significantly reduce costs and compliance risk, while increasing productivity of users.

AaaS in the cloud has economies of scale and scope by providing many virtual analytical applications with better scalability and higher cost savings. With growing data volumes and dozens of virtual analytical applications, chances are that more of them leverage processing at different times, usage patterns, and frequencies.

Data and text mining is another very promising application of AaaS. The capabilities that a service orientation (along with cloud computing, pooled resources, and parallel processing) brings to the analytics world can also be used for large-scale optimization, highly complex multicriteria decision problems, and distributed simulation models. Next we identify selected cloud-based analytics offerings.

Representative Analytics as a Service Offerings

IBM CLOUD IBM is making all of its analytics offerings available through its cloud. IBM Cloud offers several categories of analytics and AI. For example, IBM Watson Analytics integrates most of the analytics features and capabilities that can be built and deployed through their cloud. In addition, IBM Watson Cognitive has been a major cloud-based

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 565

offering that employs text mining and deep learning at a very high level. It was intro- duced earlier in the context of text mining.

MINEMYTEXT.COM One of the areas of major growth in analytics is text mining. Text mining identifies high-level topics of documents, infers sentiments from reviews, and visualizes the document or term/concept relationships, as covered in the text mining chapter. A start-up called MineMyText.com offers these capabilities in the cloud through their Web site.

SAS VIYA SAS Institute is making its analytics software offering available on demand through the cloud. Currently, SAS Visual Statistics is only available as a cloud service and is a competitor of Tableau.

TABLEAU Tableau, a major visualization software that was introduced in the context of descriptive analytics, is also available through the cloud.

SNOWFLAKE Snowflake is a cloud-based data warehouse solution. Users can bring to- gether their data from multiple sources as one source and analyze it using Snowflake.

Illustrative Analytics Applications Employing the Cloud Infrastructure

In this section we highlight several cloud analytics applications. We present them as one section as opposed to individual Application Cases.

Using Azure IOT, Stream Analytics, and Machine Learning to Improve Mobile Health Care Services

People are increasingly using mobile applications to keep track of the amount of exer- cise they do every day and maintain their health history as well. Zion China, which is a provider of mobile healthcare services, has come up with an innovative health monitor- ing tool that gathers data about health problems such as glucose levels, blood pressure, diet, medication, and exercise of their users and help them improve their quality of life by giving them suggestions on how to manage their health and prevent or cure illness on a daily basis.

The huge volume of real-time data presented scalability and data management problems, so the company collaborated with Microsoft to take advantage of Stream Analytics, Machine Learning, IOT solution and Power BI, which also improved data security and analysis. Zion China was completely dependent on traditional BI with data being collected from various devices or cloud. Using a cloud-based analytics architecture, Zion was able to add several features, speed, and security. They added an IoT hub to the front end for better transmission of data from device to cloud. The data is first transferred from the device to a mobile application via Bluetooth and then to an IoT hub via HTTPS and AMQP. Stream Analytics helps in processing the real time gathered in the IoT hub, and generates insights and useful information, which is further streamed to an SQL database. They use Azure Machine Learning to generate predictive models on diabetes patient data and improve the analysis and prediction levels. Power BI provides simple and easy visualization of data insights achieved from analysis to the users.

Sources: “Zion China Uses Azure IoT, Stream Analytics, and Machine Learning to Evolve Its Intelligent Dia- betes Management Solution” at www.codeproject.com/Articles/1194824/Zion-China-uses-Azure-IoT- Stream-Analytics-and-M (accessed October 2018) and https://microsoft.github.io/techcasestudies/ iot/2016/12/02/IoT-ZionChina.html (accessed October 2018).

566 Part III • Prescriptive Analytics and Big Data

Gulf Air Uses Big Data to Get Deeper Customer Insight

Gulf Air is the national carrier of Bahrain. It is a major international carrier with 3,000 employees, serving 45 cities in 24 countries across three continents. Gulf Air is an indus- try leader in providing traditional Arabian hospitality to customers. To learn more about how their customers felt about their hospitality services, the airline wanted to know what their customers were saying on social media about the airline’s hospitality. The challenge was analyzing all the comments and posts from their customers, as there were hundreds of thousands of posts every day. Monitoring these posts manually would be a time-consuming and daunting task and would also be prone to human error.

Gulf Air wanted to automate this task and analyze the data to learn of the emerging market trends. Along with that, the company wanted a robust infrastructure to host such a social media monitoring solution that would be available around the clock and agile across geographical boundaries.

Gulf Air developed a sentiment analysis solution, “Arabic Sentiment Analysis,” that analyzes English and Arabic social media posts. The Arabic Sentiment Analysis tool is based on Cloudera’s distribution of Hadoop Big Data framework. It runs on Gulf Air’s private cloud environment and also uses the Red Hat JBoss Enterprise Application platform. The private cloud holds about 50 terabytes of data, and the Arabic Sentiment Analysis tool can analyze thousands of posts on social media, providing sentiment results in minutes.

Gulf Air achieved substantial cost savings by putting the “Arabic Sentiment Analysis” application on the company’s existing private cloud environment as they didn’t need to invest in setting up the infrastructure for deploying the application. “Arabic Sentiment Analysis” helps Gulf Air in deciding promotions and offers for their passengers on a timely basis and helps them stay ahead of their competitors. In case the master server fails, the airline created “ghost images” of the server that can be deployed quickly, and the image can start functioning in its place. The Big Data solution quickly and efficiently captures posts periodically and transforms them into reports, giving Gulf Air up-to-date views of any change in sentiment or shifts in demand, enabling them to respond quickly. Insights from the Big Data solution have had a positive impact on the work performed by the employees of Gulf Air.

Sources: RedHat.com. (2016). “Gulf Air Builds Private Cloud for Big Data Innovation with Red Hat Technolo- gies.” www.redhat.com/en/about/press-releases/gulf-air-builds-private-cloud-big-data-innovation-red- hat-technologies (accessed October 2018); RedHat.com. (2016). “Gulf Air’s Big Data Innovation Delivers Deeper Customer Insight.” www.redhat.com/en/success-stories (accessed October 2018); ComputerWeekly.com. (2016). “Big-Data and Open Source Cloud Technology Help Gulf Air Pin Down Customer Sentiment.” www. computerweekly.com/news/450297404/Big-data-and-open-source-cloud-technology-help-Gulf-Air- pin-down-customer-sentiment (accessed October 2018).

Chime Enhances Customer Experience Using Snowflake

Chime, a banking option, offers a Visa debit card, FDIC-insured spending and savings ac- count, and a mobile application app that makes banking easier for people. Chime wanted to learn about their customer engagement. They wanted to analyze data across their mobile, Web, and backend platforms to help enhance the user experience. However, pulling and aggregating data from multiple sources such as ad services from Facebook and Google and events from other third-party analytics tools like JSON (JavaScript Object Notation) docs, was a laborious task. They wanted a solution that could aggregate data from these multiple sources and analyze the data set. Chime needed a solution that could process JSON data sources and query them using standard SQL database tables.

Chime started using Snowflake Elastic Data Warehouse solution. Snowflake pulled data from all 14 data sources of chime, including data like JSON docs from applications.

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 567

Snowflake helped Chime analyze JSON data quickly to enhance member services and provide a more personalized banking experience to customers.

Source: Based on Snowflake.net. (n.d.). Chime delivers personalized customer experience using Chime. http://www.snowflake.net/product (accessed Oct 2018).

We are entering the “petabyte age,” and traditional data and analytics approaches are beginning to show their limits. Cloud analytics is an emerging alternative solution for large- scale data analysis. Data-oriented cloud systems include storage and computing in a distrib- uted and virtualized environment. A major advantage of these offerings is the rapid diffusion of advanced analysis tools among the users, without significant investment in technology acquisition. These solutions also come with many challenges, such as security, service level, and data governance. A number of concerns have been raised about cloud computing, in- cluding loss of control and privacy, legal liabilities, cross-border political issues, and so on. According to Cloud Security Alliance, the top three security threats in the cloud are data loss and leakage, hardware failure of equipment, and an insecure interface. All the data in the cloud is accessible by the service provider, so the service provider can unknowingly or deliberately alter the data or can pass the data to a third party for purposes of law without asking the company. Research is still limited in this area. As a result, there is ample oppor- tunity to bring analytical, computational, and conceptual modeling into the context of ser- vice science, service orientation, and cloud intelligence. Nonetheless, cloud computing is an important initiative for an analytics professional to watch as it is a fast-growing area.

u SECTION 9.9 REVIEW QUESTIONS

1. Define cloud computing. How does it relate to PaaS, SaaS, and IaaS? 2. Give examples of companies offering cloud services. 3. How does cloud computing affect BI? 4. How does DaaS change the way data is handled? 5. What are the different types of cloud platforms? 6. Why is AaaS cost-effective? 7. Name at least three major cloud service providers. 8. Give at least three examples of analytics-as-a-service providers.

9.10 LOCATION-BASED ANALYTICS FOR ORGANIZATIONS

Thus far, we have seen many examples of organizations employing analytical techniques to gain insights into their existing processes through informative reporting, predictive analytics, forecasting, and optimization techniques. In this section, we learn about a critical emerg- ing trend—incorporation of location data in analytics. Figure 9.15 gives our classification of location-based analytic applications. We first review applications that make use of static location data that is usually called geospatial data. We then examine the explosive growth of applications that take advantage of all the location data being generated by today’s devices. This section first focuses on analytics applications that are being developed by organizations to make better decisions in managing operations, targeting customers, promotions, and so forth. Then we will also explore analytics applications that are being developed to be used directly by a consumer, some of which also take advantage of the location data.

Geospatial Analytics

A consolidated view of the overall performance of an organization is usually repre- sented through the visualization tools that provide actionable information. The in- formation may include current and forecasted values of various business factors and

568 Part III • Prescriptive Analytics and Big Data

key performance indicators (KPIs). Looking at the KPIs as overall numbers via vari- ous graphs and charts can be overwhelming. There is a high risk of missing potential growth opportunities or not identifying the problematic areas. As an alternative to sim- ply viewing reports, organizations employ visual maps that are geographically mapped and based on the traditional location data, usually grouped by postal codes. These map-based visualizations have been used by organizations to view the aggregated data and get more meaningful location-based insights. The traditional location-based ana- lytic techniques using geocoding of organizational locations and consumers hamper the organizations in understanding “true location-based” impacts. Locations based on postal codes offer an aggregate view of a large geographic area. This poor granularity may not help pinpoint the growth opportunities within a region, as the location of tar- get customers can change rapidly. Thus, an organization’s promotional campaigns may not target the right customers if it is based on postal codes. To address these concerns, organizations are embracing location and spatial extensions to analytics. The addition of location components based on latitudinal and longitudinal attributes to the tradi- tional analytical techniques enables organizations to add a new dimension of “where” to their traditional business analyses, which currently answers the questions of “who,” “what,” “when,” and “how much.”

Location-based data are now readily available from geographic information systems (GIS). These are used to capture, store, analyze, and manage data linked to a location using integrated sensor technologies, global positioning systems in- stalled  in  smartphones, or through RFID deployments in the retail and healthcare industries.

By integrating information about the location with other critical business data, organizations are now creating location intelligence. Location intelligence is enabling organizations to gain critical insights and make better decisions by optimizing impor- tant processes and applications. Organizations now create interactive maps that further drill down to details about any location, offering analysts the ability to investigate new trends and correlate location-specific factors across multiple KPIs. Analysts can now pinpoint trends and patterns in revenue, sales, and profitability across geographical areas.

LOCATION-BASED ANALYTICS

GEOSPATIAL STATIC APPROACH

Examining Geographic Site Locations

Live Location Feeds; Real-Time Marketing

Promotions

GPS Navigation and Data Analysis

Historic and Current Location Demand Analysis; Predictive

Parking; Health-Social Networks

GEOSPATIAL STATIC APPROACH

LOCATION-BASED DYNAMIC APPROACH

LOCATION-BASED DYNAMIC APPROACH

ORGANIZATION ORIENTED CONSUMER ORIENTED

FIGURE 9.15 Classification of Location-Based Analytics Applications.

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 569

By incorporating demographic details into locations, retailers can determine how sales vary by population level and proximity to other competitors; they can assess the demand and efficiency of supply-chain operations. Consumer product companies can identify the specific needs of customers and customer complaint locations and easily trace them back to the products. Sales reps can better target their prospects by analyzing their geography.

A company that is the market leader in providing GIS data is ESRI (esri.com). ESRI licenses its ArcGIS software to thousands of customers including commercial, government, and the military. It would take a book or more to highlight applications of ESRI’s GIS database and software! Another company grindgis.com identifies over 60 categories of GIS applications (http://grindgis.com/blog/gis-applications-uses (accessed October 2018)). A few examples that have not been mentioned yet include the following:

Agricultural applications: By combining location, weather, soil, and crop-related data, very precise irrigation and fertilizer applications can be planned. Examples include companies such as sstsoftware.com and sensefly.com (they com- bine GIS and the latest information collected through drones, another emerging technology).

Crime analysis: Superimposition of crime data including date, time, and type of crime onto the GIS data can provide significant insights into crime patterns and police staffing.

Disease spread prediction: One of the first known examples of descriptive analyt- ics is the analysis of the cholera outbreak in London in 1854. Dr. John Snow plotted the cases of cholera on a map and was able to refute the theory that the cholera outbreak was being caused by bad air. The map helped him pinpoint the outbreak to a bad water well (TheGuardian.com, 2013). We have come a long way from needing to plot maps manually, but the idea of being able to track and then predict outbreaks of diseases, such as the flu, using GIS and other data has become a major field in itself. Application Case 9.7 gave an example of using social media data along with GIS data to pinpoint flu trends.

In addition, with location intelligence, organizations can quickly overlay weather and environmental effects and forecast the level of impact on critical business operations. With technology advancements, geospatial data is now being directly incorporated in enterprise data warehouses. Location-based in-database analytics enable organizations to perform complex calculations with increased efficiency and get a single view of all the spatially oriented data, revealing hidden trends and new opportunities. For exam- ple, Teradata’s data warehouse supports the geospatial data feature based on the SQL/ MM standard. The geospatial feature is captured as a new geometric data type called ST_GEOMETRY. It supports a large spectrum of shapes, from simple points, lines, and curves to complex polygons in representing the geographic areas. They are converting the nonspatial data of their operating business locations by incorporating the latitude and longitude coordinates. This process of geocoding is readily supported by service compa- nies like NAVTEQ and Tele Atlas, which maintain worldwide databases of addresses with geospatial features and make use of address-cleansing tools like Informatica and Trillium, which support mapping of spatial coordinates to the addresses as part of extract, trans- form, and load functions.

Organizations across a variety of business sectors are employing geospatial analyt- ics. We will review some examples next. Application Case 9.10 provides an example of how location-based information was used in making site selection decisions in expand- ing a company’s footprint. Application Case 9.11 illustrates another application that goes beyond just the location decision.

570 Part III • Prescriptive Analytics and Big Data

Great Clips, one of the world’s largest and fastest- growing hair salons, has more than 3,000 salons throughout the United States and Canada. Great Clips franchise success depends on a growth strategy that is driven by rapidly opening new stores in the right locations and markets. The company needed to ana- lyze the locations based on the requirements for a potential customer base, demographic trends, and sales impact on existing franchises in the target loca- tion. Choosing a good site is of utmost importance. The current processes took a long time to analyze a single site and a great deal of labor requiring inten- sive analyst resources to manually assess the data from multiple data sources.

With thousands of locations to analyze each year, the delay was risking the loss of prime sites to competitors and was proving expensive; Great Clips employed external contractors to cope with the delay. The company created a site-selection workflow application to evaluate the new salon site locations by using the geospatial analytical capabili- ties of Alteryx. A new site location was evaluated by its drive-time proximity and convenience for serv- ing all the existing customers of the Great Clips net- work in the area. The Alteryx-based solution also enabled evaluation of each new location based on

demographics and consumer behavior data, aligning with existing Great Clips customer profiles and the potential impact of new site revenue on the exist- ing sites. As a result of using location-based ana- lytic techniques, Great Clips was able to reduce the time to assess new locations by nearly 95%. The labor-intensive analysis was automated and devel- oped into a data collection analysis, mapping, and reporting application that could be easily used by the nontechnical real estate managers. Furthermore, it enabled the company to implement proactive pre- dictive analytics for a new franchise location, as the whole process now took just a few minutes.

Questions for DisCussion

1. How is geospatial analytics employed at Great Clips?

2. What criteria should a company consider in eval- uating sites for future locations?

3. Can you think of other applications where such geospatial data might be useful?

Source: Based on Alteryx.com. Great Clips. alteryx.com/sites/ default/files/resources/files/case-study-great-chips.pdf (accessed Sept 2018).

Application Case 9.10 Great Clips Employs Spatial Analytics to Shave Time in Location Decisions

One of the key challenges for any organization that is trying to grow its presence is deciding the loca- tion of its next store. Starbucks faces the same ques- tion. To identify new store locations, more than 700 Starbucks employees (referred to as partners) in 15 countries use an ArcGIS-based market plan- ning and BI solution called Atlas. Atlas provides partners with workflows, analysis, and store perfor- mance information so that local partners in the field can make decisions when identifying new business opportunities.

As reported in multiple sources, Atlas is employed by local decision makers to understand the population trends and demand. For example,

in China, there are over 1,200 Starbucks stores, and the company is opening a new store almost every day. Information such as trade areas, retail clusters and generators, traffic, and demographics is important in deciding the next store’s location. After analyzing a new market and neighborhood, a manager can look at specific locations by zoom- ing into an area in the city and identifying where three new office towers may be completed over the next 2 months, for example. After viewing this area on the map, a workflow window can be cre- ated that will help the manager move the new site through approval, permitting, construction, and eventually opening.

Application Case 9.11 Starbucks Exploits GIS and Analytics to Grow Worldwide

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 571

In addition to the retail transaction analysis applications highlighted here, there are many other applications of combining geographic information with other data being gen- erated by an organization. For example, network operations and communication compa- nies often generate massive amounts of data every day. The ability to analyze the data quickly with a high level of location-specific granularity can better identify the customer churn and help in formulating strategies specific to locations for increasing operational efficiency, quality of service, and revenue.

Geospatial analysis can enable communication companies to capture daily transac- tions from a network to identify the geographic areas experiencing a large number of failed connection attempts of voice, data, text, or Internet. Analytics can help determine the exact causes based on location and drill down to an individual customer to provide better customer service. You can see this in action by completing the following multime- dia exercise.

A Multimedia Exercise in Analytics Employing Geospatial Analytics

Teradata University Network includes a BSI video on the case of dropped mobile calls. Please watch the video that appears on YouTube at the following link: www. teradatauniversitynetwork.com/Library/Samples/BSI-The-Case-of-the-Dropped- Mobile-Calls (accessed October 2018).

A telecommunication company launches a new line of smartphones and faces prob- lems with dropped calls. The new rollout is in trouble, and the northeast region is the worst hit region as they compare effects of dropped calls on the profits for the geographic region. The company hires BSI to analyze the problems arising due to defects in smart- phone handsets, tower coverage, and software glitches. The entire northeast region data

By integrating weather and other local data, one can also better manage demand and supply- chain operations. Starbucks is integrating its enter- prise business systems with its GIS solutions in Web services to see the world and its business in new ways. For example, Starbucks integrates AccuWeather’s forecasted real-feel temperature data. This forecasted temperature data can help localize marketing efforts. If a really hot week in Memphis is forthcoming, Starbucks analysts can select a group of coffee houses and get detailed information on past and future weather patterns, as well as store characteristics. This knowledge can be used to design a localized promotion for Frappuccinos, for example, helping Starbucks antic- ipate what its customers will be wanting a week in advance.

Major events also have an impact on coffee houses. When 150,000 people descended on San Diego for the Pride Parade, local baristas served a lot of customers. To ensure the best possible cus- tomer experience, Starbucks used this local event

knowledge to plan staffing and inventory at loca- tions near the parade.

Questions for DisCussion

1. What type of demographics and GIS informa- tion would be relevant for deciding on a store location?

2. It has been mentioned that Starbucks encourages its customers to use its mobile app. What type of information might the company gather from the app to help it better plan operations?

3. Will the availability of free Wi-Fi at Starbucks stores provide any information to Starbucks for better analytics?

Sources: Digit.HBS.org. (2015). “Starbucks: Brewing up a Data Storm!” https://digit.hbs.org/submission/starbucks-brewing- up-a-data-storm/ (accessed October 2018); Wheeler, C. (2014). “Going Big with GIS.” www.esri.com/esri-news/arcwatch/ 0814/going-big-with-gis (accessed October 2018); Blogs. ESRI.com. “From Customers to CxOs, Starbucks Delivers World- Class Service.” (2014). https://blogs.esri.com/esri/ucinsider/ 2014/07/29/starbucks/ (accessed October 2018).

572 Part III • Prescriptive Analytics and Big Data

is divided into geographic clusters, and the company solves the problem by identifying the individual customer data. The BSI team employs geospatial analytics to identify the locations where network coverage was leading to dropped calls and suggests installing a few additional towers where unhappy customers are located.

After the video is complete, you can see how the analysis was prepared at: slideshare. net/teradata/bsi-teradata-the-case-of-the-dropped-mobile-calls (accessed October 2018).

This multimedia excursion provides an example of a combination of geospatial ana- lytics along with Big Data analytics that assist in better decision making.

Real-Time Location Intelligence

Many devices in use by consumers and professionals are constantly sending out their location information. Cars, buses, taxis, mobile phones, cameras, and personal naviga- tion devices all transmit their locations thanks to network-connected positioning tech- nologies such as GPS, Wi-Fi, and cell tower triangulation. Millions of consumers and businesses use location-enabled devices for finding nearby services, locating friends and family, navigating, tracking assets and pets, dispatching, and engaging in sports, games, and hobbies. This surge in location-enabled services has resulted in a massive database of historical and real-time streaming location information. It is, of course, scattered and not very useful by itself. The automated data collection enabled through capture of cell phones and Wi-Fi hotspot access points presents an interesting new di- mension in nonintrusive market research, data collection, and, of course, microanalysis of such massive data sets.

By analyzing and learning from these large-scale patterns of movement, it is pos- sible to identify distinct classes of behaviors in specific contexts. This approach allows a business to better understand its customer patterns and make more informed deci- sions about promotions, pricing, and so on. By applying algorithms that reduce the dimensionality of location data, one can characterize places according to the activity and movement between them. From massive amounts of high-dimensional location data, these algorithms uncover trends, meaning, and relationships to eventually produce human-understandable representations. It then becomes possible to use such data to automatically make intelligent predictions and find important matches and similarities between places and people.

Location-based analytics finds its application in consumer-oriented marketing appli- cations. Many companies are now offering platforms to analyze location trails of mobile users based on geospatial data obtained from the GPS and target tech-savvy customers with coupons on their smartphones as they pass by a retailer. This illustrates the emerg- ing trend in the retail space where companies are looking to improve efficiency of mar- keting campaigns—not just by targeting every customer based on real-time location, but by employing more sophisticated predictive analytics in real time on consumer behav- ioral profiles to find the right set of consumers for advertising campaigns.

Yet another extension of location-based analytics is to use augmented reality. In 2016, Pokémon GO became a market sensation. It is a location-sensing augmented reality- based game that encourages users to claim virtual items from select geographic locations. The user can start anywhere in a city and follow markers on the app to reach a specific item. Virtual items are visible through the app when the user points a phone’s camera toward the virtual item. The user can then claim this item. Business applications of such technologies are also emerging. For example, an app called Candybar allows businesses to place these virtual items on a map using Google Maps. The placement of this item can be fine-tuned using Google’s Street View. Once all virtual items have been configured with the information and location, the business can submit items, which are then visible to the user in real time. Candybar also provides usage analytics to the business to enable

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 573

better targeting of virtual items. The virtual reality aspect of this app improves the experi- ence of users, providing them with a “gaming” environment in real life. At the same time, it provides a powerful marketing platform for businesses to reach their customers.

As is evident from this section, location-based analytics and ensuing applications are perhaps the most important front in the near future for organizations. A common theme in this section was the use of operational or marketing data by organizations. We will next explore analytics applications that are directly targeted at users and sometimes take advantage of location information.

Analytics Applications for Consumers

The explosive growth of the apps industry for smartphone platforms (iOS, Android, Windows, and so forth) and the use of analytics are creating tremendous opportunities for developing apps where the consumers use analytics without ever realizing it. These apps differ from the previous category in that these are meant for direct use by a con- sumer, as opposed to an organization that is trying to mine a consumer’s usage/purchase data to create a profile for marketing specific products or services. Predictably, these apps are meant for enabling consumers to make better decisions by employing specific analyt- ics. We highlight two of these in the following examples.

• Waze, a social Web app that assists users in identifying a navigation path and alerts users about potential issues such as accidents, police checkpoints, speed traps, and construction, based on other users’ inputs, has become a very popular navigation app. Google acquired this app a few years ago and has enhanced it further. This app is an example of aggregating user-generated information and making it avail- able for customers.

• Many apps allow users to submit reviews and ratings for businesses, products, and so on, and then present those to the users in an aggregated form to help them make choices. These apps can also be identified as apps based on social data that are targeted at consumers where the data are generated by the consumers. One of the more popular apps in this category is Yelp. Similar apps are available all over the world.

• Another transportation-related app that uses predictive analytics, ParkPGH, has been deployed since about 2010 in Pittsburgh, Pennsylvania. Developed in collabo- ration with Carnegie Mellon University, this app includes predictive capabilities to estimate parking availability. ParkPGH directs drivers to parking lots in areas where parking is available. It calculates the number of parking spaces available in sev- eral garages in the cultural arts district of Pittsburgh. Available spaces are updated every 30 seconds, keeping the driver as close to the current availability as possible. Depending on historical demand and current events, the app is able to predict parking availability and provide information on which lots will have free space by the time the driver reaches the destination. The app’s underlying algorithm uses data on current events around the area—for example, a basketball game—to pre- dict an increase in demand for parking spaces later that day, thus saving the com- muters valuable time searching for parking spaces in the busy city. Success of this app has led to a proliferation of parking apps that work in many major cities and allow a user to book a parking space in advance, recharge the meter, even bid for a parking space, etc. Both iPhone app store and Google Play store include many such apps.

Analytics-based applications are emerging not just for fun and health, but also to enhance one’s productivity. For example, Google’s e-mail app called Gmail analyzes billions of e-mail transactions and develops automated responses for e-mails. When a

574 Part III • Prescriptive Analytics and Big Data

user receives an e-mail and reads it in her Gmail app, the app also recommends short responses for the e-mail at hand that a user can select and send to the original sender.

As is evident from these examples of consumer-centric apps, predictive analytics is beginning to enable development of software that is directly used by a consumer. We be- lieve that the growth of consumer-oriented analytic applications will continue and create many entrepreneurial opportunities for the readers of this book.

One key concern in employing these technologies is the loss of privacy. If someone can track the movement of a cell phone, the privacy of that customer is a big issue. Some of the app developers claim that they only need to gather aggregate flow information, not individually identifiable information. But many stories appear in the media that highlight violations of this general principle. Both users and developers of such apps have to be very aware of the deleterious effect of giving out private information as well as collecting such information. We discuss this issue a bit further in Chapter 14.

u SECTION 9.10 REVIEW QUESTIONS

1. How does traditional analytics make use of location-based data? 2. How can geocoded locations assist in better decision making? 3. What is the value provided by geospatial analytics? 4. Explore the use of geospatial analytics further by investigating its use across various

sectors like government census tracking, consumer marketing, and so forth.

5. Search online for other applications of consumer-oriented analytical applications. 6. How can location-based analytics help individual consumers? 7. Explore more transportation applications that may employ location-based analytics. 8. What other applications can you imagine if you were able to access cell phone loca-

tion data?

Chapter Highlights

• Big Data means different things to people with different backgrounds and interests.

• Big Data exceeds the reach of commonly used hardware environments and/or capabilities of software tools to capture, manage, and process it within a tolerable time span.

• Big Data is typically defined by three “V”s: vol- ume, variety, velocity.

• MapReduce is a technique to distribute the pro- cessing of very large multistructured data files across a large cluster of machines.

• Hadoop is an open source framework for pro- cessing, storing, and analyzing massive amounts of distributed, unstructured data.

• Hive is a Hadoop-based data warehousing– like framework originally developed by Facebook.

• Pig is a Hadoop-based query language developed by Yahoo!

• NoSQL, which stands for Not Only SQL, is a new paradigm to store and process large volumes of unstructured, semistructured, and multistructured data.

• Big Data and data warehouses are complementary (not competing) analytics technologies.

• As a relatively new area, the Big Data vendor landscape is developing very rapidly.

• Stream analytics is a term commonly used for ex- tracting actionable information from continuously flowing/streaming data sources.

• Perpetual analytics evaluates every incoming ob- servation against all prior observations.

• Critical event processing is a method of captur- ing, tracking, and analyzing streams of data to detect certain events (out of normal happenings) that are worthy of the effort.

• Data stream mining, as an enabling technology for stream analytics, is the process of extracting novel patterns and knowledge structures from continuous, rapid data records.

• Cloud computing offers the possibility of using software, hardware, platforms, and infrastructure, all on a service-subscription basis. Cloud comput- ing enables a more scalable investment on the part of a user.

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 575

• Cloud-computing–based analytic services offer organizations the latest technologies without sig- nificant up-front investment.

• Geospatial data can enhance analytics applica- tions by incorporating location information.

• Real-time location information of users can be mined to develop promotion campaigns that are targeted at a specific user in real time.

• Location information from mobile phones can be used to create profiles of user behavior and movement. Such location information can enable users to find other people with similar interests and advertisers to customize their promotions.

• Location-based analytics can also benefit consum- ers directly rather than just businesses. Mobile apps are being developed to enable such innova- tive analytics applications.

Key Terms

Big Data Big Data analytics cloud computing critical event processing data scientists data stream mining

geographic information systems (GIS)

Hadoop Hadoop Distributed File System

(HDFS) Hive

MapReduce NoSQL perpetual analytics Pig Spark stream analytics

Questions for Discussion

1. What is Big Data? Why is it important? Where does Big Data come from?

2. What do you think the future of Big Data will be? Will it lose its popularity to something else? If so, what will it be?

3. What is Big Data analytics? How does it differ from reg- ular analytics?

4. What are the critical success factors for Big Data analytics?

5. What are the big challenges that one should be mind- ful of when considering implementation of Big Data analytics?

6. What are the common business problems addressed by Big Data analytics?

7. In the era of Big Data, are we about to witness the end of data warehousing? Why?

8. What are the use cases for Big Data/Hadoop and data warehousing/RDBMS?

9. Is cloud computing “just an old wine in a new bot- tle?” How is it similar to other initiatives? How is it different?

10. What is stream analytics? How does it differ from regular analytics?

11. What are the most fruitful industries for stream analyt- ics? What is common to those industries?

12. Compared to regular analytics, do you think stream ana- lytics will have more (or less) use cases in the era of Big Data analytics? Why?

13. What are the potential benefits of using geospatial data in analytics? Give examples.

14. What types of new applications can emerge from knowing locations of users in real time? What if you also knew what they have in their shopping cart, for example?

15. How can consumers benefit from using analytics, espe- cially based on location information?

16. “Location-tracking–based profiling is powerful but also poses privacy threats.” Comment.

17. Is cloud computing “just an old wine in a new bottle?” How is it similar to other initiatives? How is it different?

18. Discuss the relationship between mobile devices and social networking.

Exercises

Teradata University Network (TUN) and Other Hands-on Exercises

1. Go to teradatauniversitynetwork.com, and search for case studies. Read cases and white papers that talk about Big Data analytics. What is the common theme in those case studies?

2. At teradatauniversitynetwork.com, find the SAS Visual Analytics white papers, case studies, and hands-on exercises. Carry out the visual analytics exercises on large data sets and prepare a report to discuss your findings.

3. At teradatauniversitynetwork.com, go to the Sports Analytics page. Find applications of Big Data in sports. Summarize your findings.

576 Part III • Prescriptive Analytics and Big Data

4. Go to teradatauniversitynetwork.com, and search for BSI Videos that talk about Big Data. Review these BSI videos, and answer the case questions related to them.

5. Go to the teradata.com and/or asterdata.com Web sites. Find at least three customer case studies on Big Data, and write a report where you discuss the com- monalities and differences of these cases.

6. Go to IBM.com. Find at least three customer case stud- ies on Big Data, and write a report where you discuss the commonalities and differences of these cases.

7. Go to claudera.com. Find at least three customer case studies on Hadoop implementation, and write a report where you discuss the commonalities and differences of these cases.

8. Go to mapr.com. Find at least three customer case studies on Hadoop implementation, and write a report where you discuss the commonalities and differences of these cases.

9. Go to hortonworks.com. Find at least three customer case studies on Hadoop implementation, and write a report in which you discuss the commonalities and dif- ferences of these cases.

10. Go to marklogic.com. Find at least three customer case studies on Hadoop implementation, and write a report where you discuss the commonalities and differ- ences of these cases.

11. Go to youtube.com. Search for videos on Big Data com- puting. Watch at least two. Summarize your findings.

12. Go to google.com/scholar, and search for articles on stream analytics. Find at least three related articles. Read and summarize your findings.

13. Enter google.com/scholar, and search for articles on data stream mining. Find at least three related articles. Read and summarize your findings.

14. Enter google.com/scholar, and search for articles that talk about Big Data versus data warehousing. Find at least five articles. Read and summarize your findings.

15. Location-tracking–based clustering provides the poten- tial for personalized services but challenges for privacy. Divide the class into two parts to argue for and against such applications.

16. Enter YouTube.com. Search for videos on cloud com- puting, and watch at least two. Summarize your findings.

17. Enter Pandora.com. Find out how you can create and share music with friends. Explore how the site analyzes user preferences.

18. Enter Humanyze.com. Review various case studies and summarize one interesting application of sensors in understanding social exchanges in organizations.

19. The objective of the exercise is to familiarize you with the capabilities of smartphones to identify human activity. The data set is available at archive.ics.uci.edu/ml/datasets/ Human+Activity+Recognition+Using+Smartphones.

It contains accelerometer and gyroscope readings on 30 subjects who had the smartphone on their waist. The data is available in a raw format and involves some data preparation efforts. Your objective is to identify and classify these readings into activities like walking, running, climb- ing, and such. More information on the data set is available on the download page. You may use clustering for initial exploration and to gain an understanding of the data. You may use tools like R to prepare and analyze this data.

References

Adapted from Alteryx.com. Great Clips. alteryx.com/sites/ default/files/resources/files/case-study-great-chips. pdf (accessed September 2018).

Adapted from Snowflake.net. (n.d.). “Chime Delivers Per- sonalized Customer Experience Using Chime.” www. snowflake.net/product (accessed September 2018).

Adshead, A. (2014). “Data Set to Grow 10-fold by 2020 as Internet of Things Takes Off.” www.computerweekly.com/ news/2240217788/Data-set-to-grow-10-fold-by-2020- as-internet-of-things-takes-off (accessed September 2018).

Altaweel, Mark. “Accessing Real-Time Satellite Imagery and Data.” GIS Lounge, 1 Aug. 2018, www.gislounge.com/ accessing-real-time-satellite-imagery/.

Amodio, M. (2015). Salesforce adds predictive analytics to Market- ing Cloud. Cloud Contact Center.cloudcontactcenterzone .com/topics/cloud-contact-center/articles/413611- salesforce-adds-predictive-analytics-marketing- cloud.htm (accessed September 2018).

Asamoah, D., & R. Sharda. (2015). Adapting CRISP-DM process for social network analytics: Application to healthcare. In AMCIS 2015 Proceedings. aisel.aisnet.org/amcis2015/

BizAnalytics/GeneralPresentations/33/ (accessed September 2018).

Asamoah, D., R. Sharda, A. Zadeh, & P. Kalgotra. (2016). “ Preparing a Big Data Analytics Professional: A Pedagogic Experience.” In DSI 2016 Conference, Austin, TX.

Awadallah, A., & D. Graham. (2012). “Hadoop and the Data Warehouse: When to Use Which.” teradata.com/white- papers/Hadoop-and-the-Data-Warehouse-When-to- Use-Which (accessed September 2018).

Blogs.ESRI.com. “From Customers to CxOs, Starbucks De- livers World-Class Service.” (2014). https://blogs.esri. com/esri/ucinsider/2014/07/29/starbucks/ (accessed September 2018).

Broniatowski, D. A., M. J. Paul, & M. Dredze. (2013). “National and Local Influenza Surveillance through Twitter: An Analysis of the 2012–2013 Influenza Epidemic.” PloS One, 8(12), e83672.

Cisco. (2016). “The Zettabyte Era: Trends and Analysis.” cisco. com/c/en/us/solutions/collateral/service-provider/ visual-networking-index-vni/vni-hyperconnectivity- wp.pdf (accessed October 2018).

Chapter 9 • Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 577

ComputerWeekly.com. (2016). “Big-Data and Open Source Cloud Technology Help Gulf Air Pin Down Customer Sentiment.” www.computerweekly.com/ news/450297404/Big-data-and-open-source-cloud- technology-help-Gulf-Air-pin-down-customer- sentiment (accessed September 2018).

CxOtoday.com. (2014). “Cloud Platform to Help Pharma Co Accelerate Growth.” www.cxotoday.com/story/ mankind-pharma-to-drive-growth-with-softlayers- cloud-platform/ (accessed September 2018).

Dalininaa, R., “Using Natural Language Processing to Ana- lyze Customer Feedback in Hotel Reviews,” awww. datascience.com/resources/notebooks/data-science- summarize-hotel-reviews (Accessed October 2018).

DataStax. “Customer Case Studies.” datastax.com/resources/ casestudies/eBay (accessed September 2018).

Davis, J. (2015). “Salesforce Adds New Predictive Analytics to Marketing Cloud. Information Week.” informationweek. com/big-data/big-data-analytics/salesforce-adds- new-predictive-analytics-to-marketing-cloud/d/d- id/1323201 (accessed September 2018).

Dean, J., & S. Ghemawat. (2004). “MapReduce: Simplified Data Processing on Large Clusters.” research.google.com/ archive/mapreduce.html (accessed September 2018).

Delen, D., M. Kletke, & J. Kim. (2005). “A Scalable Classifica- tion Algorithm for Very Large Datasets.” Journal of Infor- mation and Knowledge Management, 4(2), 83–94.

Demirkan, H., & D. Delen. (2013, April). “Leveraging the Ca- pabilities of Service-Oriented Decision Support Systems: Putting Analytics and Big Data in Cloud.” Decision Support Systems, 55(1), 412–421.

Digit.HBS.org. (2015). “Starbucks: Brewing up a Data Storm!” https://digit.hbs.org/submission/starbucks- brewing-up-a-data-storm/ (accessed September 2018).

Dillow, C. (2016). “What Happens When You Combine Ar- tificial Intelligence and Satellite Imagery.” fortune.com/ 2016/03/30/facebook-ai-satellite-imagery/ (accessed September 2018).

Ekster, G. (2015). “Driving Investment Performance with Al- ternative Data.” integrity-research.com/wp-content/ uploads/2015/11/Driving-Investment-Performance- With-Alternative-Data.pdf (accessed September 2018).

Henschen, D. (2016). “Salesforce Reboots Wave Analytics, Preps IoT Cloud.” ZD Net. zdnet.com/article/salesforce-reboots- wave-analytics-preps-iot-cloud/ (accessed September 2018).

Higginbotham, S. (2012). “As Data Gets Bigger, What Comes after a Yottabyte?” gigaom.com/2012/10/30/as-data- gets-bigger-what-comes-after-a-yottabyte (accessed September 2018).

Hope, B. (2015). “Provider of Personal Finance Tools Tracks Bank Cards Sells Data to Investors.” Wall Street Journal. wsj.com/articles/provider-of-personal-finance-tools- tracks-bank-cards-sells-data-to-investors-1438914620 (accessed September 2018).

Jonas, J. (2007). “Streaming Analytics vs. Perpetual Analyt- ics (Advantages of Windowless Thinking).” jeffjonas. typepad.com/jeff_jonas/2007/04/streaming_analy. html (accessed September 2018).

Kalgotra, P., & R. Sharda. (2016). “Rural Versus Urban Comor- bidity Networks.” Working Paper, Center for Health Sys- tems and Innovation, Oklahoma State University.

Kalgotra, P., R. Sharda, & J. M. Croff. (2017). “Examining Health Disparities by Gender: A Multimorbidity Network Analysis of Electronic Medical Record.” International Jour- nal of Medical Informatics, 108, 22–28.

Kelly, L. (2012). “Big Data: Hadoop, Business Analytics, and Beyond.” wikibon.org/wiki/v/Big_Data:_Hadoop,_ Business_Analytics_and_Beyond (accessed September 2018).

Moran, P. A. (1950). “Notes on Continuous Stochastic Phe- nomena.” Biometrika, 17–23.

“Overstock.com: Revolutionizing Data and Analytics to Con- nect Soulfully with their Customers,” at https://www. teradata.com/Resources/Videos/Overstock-com- Revolutionizing-data-and-analy (accessed October 2018).

“Overstock.com Uses Teradata Path Analysis To Boost Its Customer Journey Analytics,” March 27, 2018, at https:// www.retailitinsights.com/doc/overstock-com- uses-teradata-path-analysis-boost-customer-journey- analytics-0001 (accessed October 2018).

Palmucci, J., “Using Apache Spark for Massively Parallel NLP,” at http://engineering.tripadvisor.com/using-apache- spark-for-massively-parallel-nlp/ (accessed October 2018).

RedHat.com. (2016). “Gulf Air’s Big Data Innovation Delivers Deeper Customer Insight.” https://www.redhat.com/ en/success-stories (accessed September 2018).

RedHat.com. (2016). “Gulf Air Builds Private Cloud for Big Data Innovation with Red Hat Technologies.” https:// www.redhat.com/en/about/press-releases/gulf-air- builds-private-cloud-big-data-innovation-red-hat- technologies (accessed September 2018).

Russom, P. (2013). “Busting 10 Myths about Hadoop: the Big Data Explosion.” TDWI’s Best of Business Intelligence, 10, 45–46.

Sarasohn-Kahn, J. (2008). The Wisdom of Patients: Health Care Meets Online Social Media. Oakland, CA: California HealthCare Foundation.

Shaw, C. (2016). “Satellite Companies Moving Markets.” quandl. com/blog/alternative-data-satellite-companies (accessed September 2018).

Steiner, C. (2009). “Sky High Tips for Crop Traders” (accessed September 2018).

St Louis, C., & G. Zorlu. (2012). “Can Twitter Predict Disease Outbreaks?” BMJ, 344.

Tableau white paper. (2012). “7 Tips to Succeed with Big Data in 2013.” cdnlarge.tableausoftware.com/sites/default/ files/whitepapers/7-tips-to-succeed-with-big-data- in-2013.pdf (accessed September 2018).

578 Part III • Prescriptive Analytics and Big Data

Tartar, Andre, et al. “All the Things Satellites Can Now See From Space.” Bloomberg.com, Bloomberg, 26 July 2018, www.bloomberg.com/news/features/2018-07-26/ all-the-things-satellites-can-now-see-from-space (ac- cessed October 2018).

Thusoo, A., Z. Shao, & S. Anthony. (2010). “Data Warehousing and Analytics Infrastructure at Facebook.” In Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data (p. 1013).

Turner, M. (2015). “This Is the Future of Investing, and You Probably Can’t Afford It.” businessinsider.com/hedge- funds-are-analysing-data-to-get-an-edge-2015-8 (ac- cessed September 2018).

Watson, H. (2012). “The Requirements for Being an Analytics- Based Organization.” Business Intelligence Journal, 17(2), 42–44.

Watson, H., R. Sharda, & D. Schrader. (2012). “Big Data and How to Teach It.” Workshop at AMCIS, Seattle, WA.

Wheeler, C. (2014). “Going Big with GIS.” www.esri.com/ esri-news/arcwatch/0814/going-big-with-gis (accessed October 2018).

White, C. (2012). “MapReduce and the Data Scientist.” Tera- data Vantage White Paper. teradata.com/white-paper/ MapReduce-and-the-Data-Scientist (accessed Sep- tember 2018).

Wikipedia.com. “Petabyte.” en.wikipedia.org/wiki/Petabyte (accessed September 2018).

Zadeh, A. H., H. M. Zolbanin, R. Sharda, & D. Delen. (2015). “Social Media for Nowcasting the Flu Activity: Spatial- Temporal and Text Analysis.” Business Analytics Congress, Pre-ICIS Conference, Fort Worth, TX.

Zikopoulos, P., D. DeRoos, K. Parasuraman, T. Deutsch, D. Corrigan, & J. Giles. (2013). Harness the Power of Big Data. New York: McGraw-Hill.

“Zion China uses Azure IoT, Stream Analytics, and Machine Learning to Evolve Its Intelligent Diabe- tes Management Solution,” www.codeproject.com/ Articles/1194824/Zion-China-uses-Azure-IoT- Stream-Analytics-and-M (accessed October 2018) and https://microsoft.github.io/techcasestudies/ iot/2016/12/02/IoT-ZionChina.html (accessed October 2018).

579

P A R T

Robotics, Social Networks, AI and IoT

IV

580

Robotics: Industrial and Consumer Applications

LEARNING OBJECTIVES

■■ Discuss the general history of automation and robots

■■ Discuss the applications of robots in various industries

■■ Differentiate between industrial and consumer applications of robots

■■ Identify common components of robots ■■ Discuss impacts of robots on future jobs ■■ Identify legal issues related to robotics

C hapter 2 briefly introduced robotics, an early and practical application of con-cepts developed in AI. In this chapter, we present a number of applications of ro-bots in industrial as well as personal settings. Besides learning about the already deployed and emerging applications, we identify the general components of a robot. In the spirit of managerial considerations, we also discuss the impact of robotics on jobs as well as related legal issues. Some of the coverage is broad and impacts all other artificial intelligence (AI), so it may seem to overlap a bit with Chapter 14. But the focus in this chapter is on physical robots, not just software-driven applications of AI.

This chapter has the following sections:

10.1 Opening Vignette: Robots Provide Emotional Support to Patients and Children 581

10.2 Overview of Robotics 584 10.3 History of Robotics 584 10.4 Illustrative Applications of Robotics 586 10.5 Components of Robots 595 10.6 Various Categories of Robots 596 10.7 Autonomous Cars: Robots in Motion 597 10.8 Impact of Robots on Current and Future Jobs 600 10.9 Legal Implications of Robots and Artificial Intelligence 603

C H A P T E R

10

Chapter 10 • Robotics: Industrial and Consumer Applications 581

10.1 OPENING VIGNETTE: Robots Provide Emotional Support to Patients and Children

As discussed in this chapter, robots have impacted industrial manufacturing and other physical activities. Now, with the research and evolution of AI, robotics can straddle the social world. For example, hospitals today make an effort to give social and emotional support to patients and their families. This support is especially sensitive when offering treatment to children. Children in a hospital are in an unfamiliar environment with medi- cal instruments attached to them, and in many cases, doctors may recommend move- ment restrictions. This restriction leads to stress, anxiety, and depression in children and consequently in their family members. Hospitals try to provide childcare support special- ist or companion pet therapies to reduce the trauma. These therapies prepare children and their parents for future treatment and provide them with temporary emotional sup- port with their interactions. Due to the small number of such specialists, there is a gap between demand and supply for childcare specialists. Also, it is not possible to provide pet therapy at many centers due to the fear of allergies, dust, and bites that may cause the patient’s condition to be aggravated. To fill these gaps, the use of social robots is being explored to resolve depression and anxiety among children. A study (Jeong et al., 2015) found that the physical presence of a robot is more effective concerning emotional response as compared to a virtual machine interaction in a pediatric hospital center.

Researchers have known for a long time (e.g., Goris et al., 2010) that more than 60 percent of human communication is not verbal but rather occurs through facial expres- sions. Thus, a social robot has to be able to provide emotional communication like a child specialist. One popular robot that is providing such support is Huggable. With the help of AI, Huggable is equipped to understand facial expressions, temperament, gestures, and human cleverness. It is like a staff member added to the team of specialists who provide children some general emotional health assistance.

Huggable looks like a teddy bear having a ringed arrangement. A furry soft body provides a childish look to it and hence is perceived as a friend by the children. With its mechanical arms, Huggable can perform specific actions quickly. Rather than sporting high-tech devices, a Huggable robot is composed of an Android device whose micro- phone, speaker, and camera are in its internal sensors, and a mobile phone that acts as the central nervous system. The Android device enables the communication between the internal sensors and teleoperation interface. Its segmental arm components enable an easy replacement of sensors and hence increase its reusability. These haptic sensors along with AI enable it to process the physical touch and use it expressively.

Sensors incorporated in a Huggable transmit physical touch and pressure data to the teleoperation device or external device via an IOIO board. The Android device receives the data from the external sensors and transmits them to the motors that are attached to the body of the robot. These motors enable the movement of the robot. The capacitors are placed at various parts of the robot, known as pressure points. These pressure points enable the robot to understand the pain of a child who is unable to express it verbally but may be able to touch the robot to convey the pain. The Android device interprets the physical touch and pressure sensor data in a meaningful way and responds effectively. The Android phone enables communication between the other devices while keeping the design minimalistic. The computing power of the robot and the Android device is good enough to allow real- time communication with a child. Figure 10.1 exhibits a schematic of the Huggable robot.

Huggable has been used with children undergoing treatment at Boston Children’s Hospital. Reportedly, Aurora, a 10-year-old who had leukemia, was being treated at Dana-Farber/Boston Children’s Cancer and Blood Disorder Center. According to Aurora’s parents, “There were many activities to do at the hospital but the Huggable being there

582 Part IV • Robotics, Social Networks, AI and IoT

is great for kids.” Beatrice, another child who visits the hospital frequently due to her chronic condition, misses her classes and friends and is unable to do anything that a typi- cal child of her age would do. She was nervous and disliked the process of treatment, but during her interplay with the Huggable, she was more willing to take medicine as if it were the most natural activity to do. She recommended the robot to be a bit faster so that the next time she could play peek-a-boo correctly.

During these interactions with Huggable, children were seen hugging it, holding its hand, tickling it, giving it high-fives, and treating it as someone they need for support. Children were polite with it and used expressions such as “no, thank you” and “one sec- ond, please.” In the end, when bidding it goodbye, one child hugged the Huggable, and another wished to play with it longer.

Another benefit of such emotional support robots is in the prevention of infections. Patients may have contagious diseases, but the robots are sterilized after each use to pre- vent infection from spreading. Thus, Huggable not only provides support to children but also can be a useful tool for reducing the spread of infectious diseases.

A recently reported study by researchers at MIT’s Media Lab highlighted the differences between social robots such as Huggable and other virtual interaction technologies. A group of 54 children who were in a hospital were given three distinct social interactions: a cute regular teddy bear, a virtual persona of Huggable on a tablet, and a social robot. The bear offered a physical model but not social dialogue. A virtual version of Huggable on the tablet provided linguistic engagement, conversed with the humans in the same way, and possessed the same features as the robot but was a 3D virtual version of the Huggable robot. Both the virtual character and robot were operated by a teleoperator, and hence they perceived the interaction and responded in the same manner. Children were given one of the three inter- actions to play with based in groups according to age and gender. Necessary information was provided to the children by the care specialist, and the virtual character and robot were handled separately by these specialists just outside the room. IBM Watson’s tone Analyzer was used to attempt to identify five human emotions and five personality traits. Interactions with each of these three types of virtual agents by children were videotaped and analyzed by the researchers. The results of this experiment were quite interesting. These results showed that the children gazed more at the virtual character and the robot as compared to the bear. Touches between the children and the virtual agents were the highest with the Huggable robot followed by the virtual character Huggable and the bear. Also, the children took care of the Huggable robot and did not push or pull it. Interestingly, a few children responded to the virtual character on the tablet violently even when it made ouch sounds. The poor teddy bear was thrown and kicked around playfully. These results show that the children connected with the robotic Huggable more than the other two options.

Sensors

Actuators

Android Device

External Monitoring

Device

Sensor Data

Target Position

Reciprocal Action

Synergy Data

IOIO Board

FIGURE 10.1 A General Schematic of a Huggable Robot.

Chapter 10 • Robotics: Industrial and Consumer Applications 583

SOCIAL ROBOT FOR OLDER ADULTS: PARO

Major countries in the world will soon have population rates of people aged 65 and older exceed that of the younger population by 2050. The emotional support older adults need cannot be ignored where geographical separation and the technological divide have made it difficult for them to connect with their families. Paro, a social robot, is designed to interact with humans and is marketed as a robot used for older adults at nurs- ing homes. Paro mainly acts as pet therapy; it can also be immensely useful where pet therapy becomes inadvisable in hospitals due to the risk of infections.

Paro interprets the human touch, and it can also capture limited speech, express a limited set of vocal utterances, and move its head. Paro is not a mobile robot and resembles a seal. It was tested at two nursing homes (Broekens et al., 2009) with 23 patients. The results showed that social robots like Paro increase the social interaction. Paro not only brought smiles to patients’ faces but also some vivid, happy experiences to occupants. Even though Paro did not provide a complete response that humans do, many patients found the responses meaningful and connected emotionally to it. These robots can help break the monotonous routine of older adults and add some joy to their lives. It provides them with a feeling of being wanted and self-esteem, lowering stress and anxiety levels.

u QUESTIONS FOR THE OPENING VIGNETTE

1. What characteristics would you expect to have in a robot that provides emotional support to patients?

2. Can you think of other applications where robots such as the Huggable can play a helpful role?

3. Visit the website https://www.universal-robots.com/case-stories/aurolab/ to learn about collaborative robots. How could such robots be useful in other settings?

WHAT WE CAN LEARN FROM THIS VIGNETTE

As we have seen in various chapters throughout this book, AI is opening many inter- esting and unique applications. The stories about the Huggable and Paro introduce us to the idea of using robots for one of the most difficult aspects of work – to provide emotional support to patients, both children and adults. Combinations of technologies such as machine learning, voice synthesis, voice recognition, natural language process- ing, machine vision, automation, micromachines, and so on make it possible to com- bine these technologies to satisfy many needs. The applications can come entirely in virtual forms such as IBM Watson, which won the Jeopardy! game implementing indus- trial automation, producing self-driving cars, and even providing emotional support as noted in this opening vignette. We will see many examples of similar applications in this chapter.

Sources: J. Broekens, M. Heerink, & H. Rosendal. (2009). “Assistive Social Robots in Elderly Care: A Review.” Gerontechnology, 8, pp. 94–103. doi: 10.4017/gt.2009.08.02.002.00; S. Fallon. (2015). “A Blue Robotic Bear to Make Sick Kids Feel Less Blue.” https://www.wired.com/2015/03/blue-robotic-bear-make-sick-kids- feel-less-blue/ (accessed August 2018). Also see the YouTube video at https://youtu.be/UaRCCA2rRR0 (accessed August 2018); K. Goris et al. (2010, September). “Mechanical Design of the Huggable Robot Probo.” Robotics & Multibody Mechanics Research Group. Brussels, Belgium: Vrije Universiteit Brussel; S. Jeong et al. (2015). “A Social Robot to Mitigate Stress, Anxiety, and Pain in Hospital Pediatric Care.” Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts; S. Jeong & D. Logan. (2018, April 21–26). “Huggable: The Impact of Embodiment on Promoting Socio-emotional Interactions for Young Pediatric Surgeons.” MIT Media Lab, Cambridge, MA, CHI 2018, Montréal, Quebec, Canada.

584 Part IV • Robotics, Social Networks, AI and IoT

10.2 OVERVIEW OF ROBOTICS

Every robotics scientist has her or his own view about the definition of robot. But a common notion of robot is a machine or a physical device or software that with the cooperation of AI can accomplish a responsibility autonomously. A robot can sense and affect the environment. Applications of robotics in our day-to-day lives have been increas- ing. This evolution and use of technologies are called the fourth industrial revolution. Applications of robotics in manufacturing, health, and information technology (IT) fields in the last decade have led to rapid development in changing the future of industries. Robots are moving from just performing preselected repetitive tasks ( automation) and being unable to react to unforeseen circumstances (Ayres and Miller, 1981) to performing specialized tasks in healthcare, manufacturing, sports, financial services – virtually every industry. This capability of adaptation to new situations leads to autonomy, a sea change from previous generations of robots. Chapter 2 introduced a definition of robots and provided some applications in selected industries. In this chapter, we will supple- ment that introduction with various applications and take a slightly deeper dive into the topic.

Although our imagination of a robot may be based on the R2D2 or C3-PO from the Star Wars movies, we have experienced robots in many other ways. Factories have been using robots for a long time (see Section 10.3) for manufacturing. On the consumer side, an early application was Roomba, a robot that can clean floors on its own. Perhaps the best example of robots that we will all experience soon if not already is an autono- mous (self-driving) car. Tech Republic called the self-driving car the first robot we will all learn to trust. We will dig a bit deeper into self-driving vehicles in Section 10.7. With the growth in machine learning, especially image recognition systems, applications of robots are increasing in virtually every industry. Robots can cut sausages into the right size pieces for pizza and can automatically determine that the right number and type of peperoni pieces have been placed on a pizza before it is baked. Surgeries conducted by and with the assistance of robots are growing at a rapid pace. Section 10.4 provides many illustrative applications of robots. Then Section 10.7 discusses self-driving cars as another category of robots.

u SECTION 10.2 REVIEW QUESTIONS

1. Define robot. 2. What is the difference between automation and autonomy? 3. Give examples of robots in use. Find recent applications online and share with the class.

10.3 HISTORY OF ROBOTICS

Wikipedia includes an interesting history of robotics. Humans have been fascinated with the idea of machines serving us for a long time. The first idea of robotics was conceptual- ized in 320 BC when Aristotle, a Greek philosopher, stated, “If every tool, when ordered, or even of its own accord, could do the work that befits it, then there would be no need either of apprentices for the master workers or of slaves for the lords.” In 1495, Leonardo Da Vinci drafted strategies and images for a robot that looked like a human. Between 1700 and 1900, various automatons were created, including an excellent automation structure built by Jacques De Vaucanson, who made one clockwork duck that could flap its wings, quack, and appear to eat and digest food.

Throughout the industrial revolution, robotics was triggered by the advances in steam power and electricity. As consumer demand increased, engineers strove to devise new methods to increase production by automation and create machines that can perform

Chapter 10 • Robotics: Industrial and Consumer Applications 585

the tasks that were dangerous for a human to do. In 1893, “Steam Man,” a prototype for a humanoid robot, was proposed by Canadian professor George Moore. It was composed of steel and powered by a steam engine. It could walk autonomously at nearly nine miles per hour and could even pull relatively light loads. In 1898, Nikola Tesla exhibited a sub- marine prototype. These events led to the integration of robotics in manufacturing, space, defense, aerospace, medicine, education, and entertainment industries.

In 1913, the world’s first moving conveyor belt assembly line was started by Henry Ford. With the aid of a conveyor belt, a car could be assembled in 93 minutes. Later in 1920, the term robot was coined by Karel Capek in his play Rossum’s Universal Robots. Then a toy robot, Lilliput, was manufactured in Japan.

By the 1950s, innovators were creating machines that could handle dangerous, re- petitive tasks for defense and industrial manufacturing. Since the robots were primarily designed for heavy-duty industries, they were required to pull, lift, move, and push the same way humans did. Thus, many robots were designed like a human arm. Examples include a spray-painting gadget for a position-controlling apparatus by W. L. V. Pollard in 1938. DeVilbiss Company acquired this robot and later became a leading supplier of the robotic arms in the United States.

In the mid-1950s, the first commercial robotic arm, Planetbot, was developed, and General Motors later used it in a manufacturing plant for the production of radiators. A total of eight Planetbots were sold. According to the company, it could perform nearly 25 movements and could be reset in minutes to perform another set of operations. However, Planetbot did not achieve the desired results due to the unusual behavior of the hydraulic fuel inside it.

George Devol and Joe Engelberger designed Unimate to automate the manufactur- ing of TV picture tubes. It weighed close to 4,000 pounds and was controlled by pre- programmed commands fed on a magnetic drum. Later this was used by General Motors Corporation for production to sequence and stack hot die-cast metal components. This arm with specific upgrades became one of the famous features in assembly lines. A total of 8,500 machines was sold, and half of them went to the automotive industries. Later Unimate was modified to perform spot welding, die casting, and machine tool stacking.

In the 1960s, Ralph Mosher and his team created two remotely operated robotic arms, Handyman and Man-mate. A Handyman was a two-arm electro-hydraulic robot, and the design of the Man-mate’s arm was based on the human spine. The arms gave the robots the flexibility for artifact examination procedures. The fingers were designed in a way that they could grasp objects via a single command.

New mobile robots came into the picture. The first one, Shakey, was developed in 1963. It could move freely, avoiding obstacles in its path. A radio antenna was attached to its head. It had a vision sensor atop a central processing unit. Shakey was attached to two wheels, and its two sensors could sense obstacles. Using logic-based problem solving, it could recognize the shape of objects, move them, or go around them.

The space race started by Russia’s Sputnik and embraced by the United States led to many technology advances leading to growth of robotics. In 1976, during NASA’s mis- sion to Mars, a Viking lander was created for the atmospheric conditions of Mars. Its arms opened out and created a tube to gather samples from the Mars surface. There were some technical issues during the mission, but the scientists were able to fix them remotely.

In 1986, the first LEGO-based educational products were put on the market by Honda. In 1994, Dante II, an eight-legged walking robot built by Carnegie Mellon University, collected the volcanic gas sample from Mount Spur.

Robotics expanded exponentially as more research and money were invested. Robotic applications and research spread to Japan, Korea, and European nations. It is estimated that by 2019, there will be close to 2.6 million significant robots. Robots have applications in the fields of social support, defense, toys and entertainment, healthcare,

586 Part IV • Robotics, Social Networks, AI and IoT

food, and rescue. Many robots are now moving into next stages, going from deep-sea to interplanetary and extrasolar research. And as noted, self-driving cars will bring robots to the masses. We review several robot applications in the following sections.

u SECTION 10.3 REVIEW QUESTIONS

1. Identify some of the key milestones in the history of manufacturing that have led to the current interest in robotics.

2. How would Shakey’s capabilities compare to today’s robots? 3. How have robots helped with space missions?

10.4 ILLUSTRATIVE APPLICATIONS OF ROBOTICS

This section highlights examples of robot applications in various industries. Each of these is presented as a mini-application case, with the discussion questions presented at the end of the section.

Changing Precision Technology

A mobile production company in China, Changing Precision Technology switched to the use of robotic arms to produce parts for mobile phones. The company previously em- ployed 650 workers to operate the factory. Now, robots perform most of its operations, and the company has reduced its workforce to 60, decreasing the human workforce by 90 percent. In the future, the company intends to drop its employee count to about 20. With the robots in place, the company not only has achieved an increase in production of 250 percent but also cut the defect levels from 25 percent to a mere 5 percent.

Compiled from C. Forrest. (2015). “Chinese Factory Replaces 90% of Humans with Robots, Production Soars.” TechRepublic. https://www.techrepublic.com/article/chinese-factory-replaces-90-of-humans-with-robots- production-soars/ (accessed September 2018); J. Javelosa & K. Houser. (2017). “Production Soars for Chinese Factory Who Replaced 90% of Employees with Robots.” Future Society. https://futurism.com/2-production- soars-for-chinese-factory-who-replaced-90-of-employees-with-robots/ (accessed September 2018).

Adidas

Adidas is a worldwide leading sportswear manufacturer. Keeping trends, innovation, and customization in mind, Adidas has started to automate factories such as Speedfactory in Ansbach, Germany, and Atlanta, Georgia. A conventional supply chain from the raw ma- terials to final product takes around two months, but with automation, it takes just a few days or weeks. The implementation of robotics there was different from that of other manufacturing industries because the raw materials used in shoes manufactured by Adidas are soft textile materials. Adidas is working with the company Oechsler to implement the robotics in its supply chain. Adidas uses technologies such as additive manufacturing, ro- botic arms, and computerized knitting. At the Speedfactory, the robot that makes a part of a sneaker attaches a scannable QR code to the part. During quality check, if any part of the product turns out to be faulty, the robot that created it is thus traceable and repaired. Adidas has optimized this process, which offers the company the option to roll out a few thousands of customized shoes in the market and see how it performs and optimize the process accordingly. In the next few years, the company plans to roll out around 1 million pairs of the custom styles annually. In the long term, this strategy supports moving from manufacturing large stocks of inventory to creating the products on demand.

Compiled from “Adidas’s High-Tech Factory Brings Production Back to Germany.” (2017, January 14). The Economist. https://www.economist.com/business/2017/01/14/adidass-high-tech-factory-brings-production-back-to- germany (accessed September 2018); D. Green. (2018). “Adidas Just Opened a Futuristic New Factory – and It Will Dramatically Change How Shoes Are Sold.” Business Insider. http://www.businessinsider.com/adidas-high- tech-speedfactory-begins-production-2018-4 (accessed September 2018).

Chapter 10 • Robotics: Industrial and Consumer Applications 587

BMW Employs Collaborative Robots

The increased use of AI and automation in industries has resulted in the development of robots. Yet, human cognitive capabilities are irreplaceable. The combination of robots and humans has been achieved using collaborative robots at a BMW manufacturing unit. By doing so, the company has maximized the efficiency of its production unit and mod- ernized the work environment.

BMW’s Spartanburg, South Carolina, plant has employed 60 collaborative robots that work side by side with its human workforce. These robots, for example, furnish the interior of BMW car doors with sound and moisture insulation. This sealing pro- tects the electronic equipment that is fixed on the door and the vehicle as a whole from moisture. Previously, human workers performed this intensive task of fixing the foil with the adhesive beads by using a manual roller. With the use of cobots, a robot’s arms perform this task with precision. Cobots run on low speed and stop immedi- ately as soon as the sensors detect any obstacle in their way to maintain the safety of assembly-line workers.

At BMW’s Dingolfing factory located in Germany, a lightweight cobot is ceiling mounted in the axle transmission assembly area to pick up bevel gears. These gears can weigh up to 5.5 kilos. The cobot fits the bevel gears accurately, avoiding damage to the gear wheels.

Compiled from M. Allinson. (2017, March 4). “BMW Shows Off Its Smart Factory Technologies at Its Plants Worldwide.” BMW Press Release. Robotics and Automation. https://roboticsandautomationnews.com/2017/03/04/ bmw-shows-off-its-smart-factory-technologies-at-its-plants-worldwide/11696/ (accessed September 2018); “Innovative Human-Robot Cooperation in BMW Group Production.” (2013, October 9). https://www.press. bmwgroup.com/global/article/detail/T0209722EN/innovative-human-robot-cooperation-in-bmw-group- production?language=en (accessed September 2018).

Tega

Tega is a social bot intended to provide extended support to preschoolers by engag- ing them via storytelling and offering help with vocabulary. Like Huggable, Tega is an Android-based robot and resembles an animation character. It has an external camera and onboard speakers and is designed to run for up to six hours before needing a recharge. Tega uses Android capabilities for expressive eyes, computation abilities, and physical movements. Children’s response is fed to the Tega as a reward signal into a reinforcement learning algorithm. Tega uses a social controller, sensor processing, and motor control for moving its body and tilting and rotating left or right.

Tega is designed not only to tell stories but also to hold a conversation about the stories. With the help of an app on a tablet, Tega interacts with a child as a peer and teammate, not as an educator. Children communicate with the tablet, and Tega provides the feedback and reactions by watching the children’s emotional states. Tega also offers help with vocabulary and understands a child’s physical and emotional re- sponses, enabling it to build a relationship with the child. The tests have shown that Tega can positively impact a child’s interest in education, free thinking, and mental development. For more information, watch the video at https://www.youtube.com/ watch?v=16in922JTsw.

Compiled from E. Ackerman. (2016). IEEE Spectrum. http://spectrum.ieee.org/automaton/robotics/home- robots/tega-mit-latest-friendly-squishable-social-robot (March 5, 2017); J. K. Westlund et al. (2016). “Tega: A Social Robot.” Video Presentation. Proceedings of the Eleventh ACM/IEEE International Conference on Human Robot Interaction; H. W. Park et al. (2017). “Growing Growth Mindset with a Social Robot Peer.” Proceedings of the Twelfth ACM/IEEE International Conference on Human Robot Interaction; Personal Robots Group. (2016). https://www.youtube.com/watch?v=sF0tRCqvyT0 (accessed September 2018); Personal Robots Group, MIT Media Lab. (2016). AAAS. https://www.eurekalert.org/pub_releases/2016-03/nsf-rlc031116.php (accessed September 2018).

588 Part IV • Robotics, Social Networks, AI and IoT

San Francisco Burger Eatery

Flipping burgers is considered a low-pay, mundane task that provides many people with employment at a low salary. Such jobs are likely to disappear over time because of ro- bots. One such implementation of robotics in the food industry is at a burger restaurant in San Francisco. The burger-making machine is not a traditional robot sporting arms and legs that can move around and work as a human. Instead, it is a complete burger prep device that can work from prepping a burger for cooking and bringing together a full meal. It blends the robotic power in bringing the right taste with the help of a Michelin- star chef’s recipes and being friendly on the pocket. The restaurant has put in place two 14-foot-long machines that can make around 120 burgers per hour. Each machine has 350 sensors, 20 computers, and close to 7,000 parts.

Buns, onions, tomatoes, pickles, seasoning, and sauces are filled in transparent tubes over a conveyor belt. Once an order is placed via a mobile device, it takes close to five minutes to prepare the order. First, air pressure pushes a burger brioche roll from the transparent tube on the conveyor belt. Different components of the robot work one after the other to prepare the order, from slicing the roll in two halves, applying butter on the bun, shredding vegetables, and dropping the sauces. Also, a light specialized grip is placed on the patty to keep it intact and to bake it per the recipe. With the use of thermal sensors and an algorithm, the cooking time and temperature of the patty are determined, and once cooked, the patty is placed on the bun by a robotic arm. Workers receive a notification via an Apple watch when there is an issue with the machine regarding a mal- function on an order or the need for refills on supplies.

Compiled from “A Robot Cooks Burgers at Startup Restaurant Creator.” (2018). TechCrunch. https://techcrunch. com/video/a-robot-cooks-burgers-at-startup-restaurant-creator/ (accessed September 2018); L. Zimberoff. (2018, June 21). “A Burger Joint Where Robots Make Your Food.” https://www.wsj.com/articles/a-burger- joint-where-robots-make-your-food-1529599213 (accessed September 2018).

Spyce

Using robots to make affordable foods is demonstrated by a fast-food restaurant operat- ing in Boston that serves grain dishes and salad bowls. Spyce is a budget-friendly res- taurant founded by MIT engineering graduates. Michael Farid created the robots that can cook. This restaurant employs few people with good pay and employs robots to do much of the fast-food work.

Orders are placed at a kiosk with touch screens. Once the order is confirmed, the mechanized systems start preparing the food. Ingredients are placed in refrigerated bins that are passed via transparent tubes and are collected using a mobile device that delivers the ingredients to the requested pot. A metal plate attached to the side of the robotic pot heats the food. A temperature of about 450 degrees Fahrenheit is main- tained, and the food is tumbled for nearly two minutes and cooked. This resembles clothes being washed in a machine. Once the meal is ready, the robotic pot tilts and transfers the food to a bowl. After each cooking round, the robotic pot washes itself with a high-pressure hot water stream and then returns to its initial position, ready to cook the next meal. The customer name is also added to the bowl. The meal is then served by a human after any final changes. Spyce is also trying to put in place a robot that can cook pancakes.

Compiled from B. Coxworth. (2018, May 29). “Restaurant Keeps Its Prices Down – With a Robotic Kitchen.” New Atlas. https://newatlas.com/spyce-restaurant-robotic-kitchen/54818/ (accessed September 2018); J. Engel. (2018, May 3). “Spyce, MIT-Born Robotic Kitchen Startup, Launches Restaurant: Video.” Xconomy. https://www.xconomy.com/boston/2018/05/03/spyce-mit-born-robotic-kitchen-startup-launches- restaurant-video/ (accessed September 2018).

Chapter 10 • Robotics: Industrial and Consumer Applications 589

Mahindra & Mahindra Ltd.

As the population increases, the agricultural industry is expanding to keep up with de- mand. To keep increasing the food supply at a reasonable cost and to maintain quality, the Indian multinational firm Mahindra & Mahindra Ltd. is seeking to improve the process of harvesting tabletop grapes. The company is establishing a research and development center at Virginia Polytechnic Institute and University. It will work with other Mahindra centers situated in Finland, India, and Japan.

The grapes can be used for juice, wine, and tabletop grapes. The quality that must be maintained is vastly different for each of these. The ripeness and presentation of tabletop grapes differ from the other two uses; hence, quality control is critical. Deciding which grapes are ready to pick is a labor-intensive approach, and one must ensure the maturity, consistency, and quality of grapes. Making this decision visually requires expert training, which is not easily scalable. Using robotic harvesting instead of human pickers is being explored. Robots can achieve these goals using sensors that will keep the quality in view while speeding the process.

Compiled from L. Rosencrance. (2018, May 31). “Tabletop Grapes to Get Picked by Robots in India, with Help from Virginia Tech.” RoboticsBusinessReview. https://www.roboticsbusinessreview.com/agriculture/ tabletop-grapes-picked-robots-india-virginia-tech/ (accessed September 2018); “Tabletop Grapes to Get Picked by Robots in India.” Agtechnews.com. http://agtechnews.com/Ag-Robotics-Technology/ Tabletop-Grapes-to-Get-Picked-by-Robots-in-India.html (accessed September 2018).

Robots in the Defense Industry

For obvious reasons, the military has invested in robotic applications for a long time. Robots can replace humans in places where risk of loss of human life is too great. Robots can also reach areas where humans may not be able to go due to extreme conditions – heat, water, and so forth. Besides the recent growth of drones in military applications, several specific robots have been developed over a long time. Some are highlighted in the next sections.

MAARS MAARS (Modular Advanced Armed Robotic System) is an upgraded version of special weapons observation reconnaissance detection system (SWORDS) robots that were used by the U.S. military during the Iraq war. It is designed for reconnaissance, surveillance, and target acquisition and can have a 360@degree view. Depending on the circumstances, MAARS can drape much firepower into its tiny frame. A variety of am- munition such as tear gas, nonlethal lasers, and grenade launcher can be wrapped in it. MAARS is an army robot that can fight autonomously thus reducing risk to soldiers' lives while also protecting itself. This robot has seven types of sensors to track the heat signa- ture of an enemy during the day and night. It uses night vision cameras to monitor enemy activities during the night. On command, MAARS fires at opponents. Its other uses in- clude moving heavy loads from one place to another. It provides a range of options from nonlethal force such as warning of an attack. It can also form a two-sided communication system. The robot can also use less lethal weapons such as laughing gas, pepper spray, and smoke and start clusters to disperse crowds. The robot can be controlled from about one kilometer and is designed to increase or decrease speed, climb stairs, and walk on nonpaved roads using wheels rather than tracks.

Compiled from T. Dupont. (2015, October 15). “The MAARS Military Robot.” Prezi. https://prezi.com/ fsrlswo0qklp/the-maars-military-robot/ (accessed September 2018); Modular Advanced Armed Robotic System. (n.d.). Wikipedia. https://en.wikipedia.org/wiki/Modular_Advanced_Armed_Robotic_System (accessed September 2018); “Shipboard Autonomous Firefighting Robot – SAFFiR.” (2015, February 4). YouTube. https://www.youtube.com/watch?time_continue=252&v=K4OtS534oYU (accessed September 2018).

590 Part IV • Robotics, Social Networks, AI and IoT

SAFFIR (SHIPBOARD AUTONOMOUS FIREFIGHTING ROBOT) Fire on a ship is one of the greatest risks to shipboard life. Shipboard fires have a different and crucial set of prob- lems. Because of the confined space, there are challenges regarding smoke, gas, and limited ability to escape. Even though procedures like fire drills, onboard alarms, fire extinguishers, and other measures provide ways of dealing with fire on the sea, mod- ern technology is in place to tackle this threat in a better way. A U.S. Navy team at the Office of Naval Research has developed SAFFiR. It is a 5 foot 10 inch tall robot. It is not designed to be completely autonomous. It has a humanoid robotic structure so that it can pass through confined aisles and other nooks and corners of a ship and climb lad- ders. The robot has been designed to work with the obstacles in the passageways in a ship. SAFFiR can use protective fire gear such as fire-protective coats, suppressants, and sensors that are designed for humans. Lightweight and low-friction linear actuators improve its efficiency and control. It is equipped with several sensors: regular camera, gas, and infrared camera for night vision and in black smoke. Its body is designed not only to be fire resistant but also to throw extinguishing grenades. It can work for around half an hour without needing a charge. SAFFiR can also balance itself on an uneven surface.

Compiled from K. Drummond. (2012, March 8). “Navy’s Newest Robot Is a Mechanized Firefighter.” wired. com. https://www.wired.com/2012/03/firefight-robot/ (accessed September 2018); P. Shadbolt. (2015, February 15). “U.S. Navy Unveils Robotic Firefighter.” CNN. https://www.cnn.com/2015/02/12/tech/mci- saffir-robot/index.html (accessed September 2018); T. White. (2015, February 4). “Making Sailors ‘SAFFiR’ – Navy Unveils Firefighting Robot Prototype at Naval Tech EXPO.” America’s Navy. https://www.navy.mil/ submit/display.asp?story_id=85459 (accessed September 2018).

Pepper

Pepper is a semihumanoid robot manufactured by SoftBank Robotics that can under- stand human emotions. A screen is located on its chest. It can identify frowning, tone of voice, smiling, and user actions such as the angle of a person’s head and crossed fingers. This way Pepper can determine if a person’s mood is good or bad. Pepper can walk autonomously, recognize individuals, and can even lift their mood through its conversation.

Pepper has a height of 120 cms (about 4 feet). It has three directional wheels at- tached, enabling it to move all around the place. It can tilt its head and move its arms and fingers and is equipped with two high-definition cameras to understand the environment. Because of its anticollision functionalities, Pepper reduces unexpected collisions and can recognize humans as well as obstacles nearby. It can also remember human faces and ac- cepts smartphone and card payments. Pepper supports commands in Japanese, English, and Chinese.

Pepper is deployed in service industries as well as homes. It has several advantages for effectively communicating with customers but has also been criticized at places for incompetence or security issues. The following examples provide information on its ap- plications and drawbacks:

• Interacting with robots while shopping is changing the face of AI in commercial settings. Nestlé Japan, a leading coffee manufacturer, has employed Pepper to sell Nescafé machines to enhance customer experience. Pepper can explain the range of products Nestlé has to offer and recognize human responses using facial recogni- tion and sounds. Using a series of questions and responses to them, the robot iden- tifies a consumer’s need and can recommend the appropriate product.

• Some hotels such as Courtyard by Marriott and Mandarin Oriental are employing Pepper to increase customer satisfaction and efficiency. The hotels use Pepper to increase customer engagement, guide guests toward activities that are taking

Chapter 10 • Robotics: Industrial and Consumer Applications 591

place, and promote their reward programs. Another goal is to collect customer data and fine-tune the communication according to customer preferences. Pepper was deployed steps away from the entry at Disneyland theme park hotels, and it immediately increased customer interactions. Hotels use Pepper to converse with guests while they are checking in or out or to guide them to the spa, gym, and other amenities. It can also inform guests about campaigns and promotions and help staff members avoid the mundane task of enrolling guests in a loyalty program. Customer reactions are largely quite positive in regard to this.

• Central Electric Cooperative (CEC), an electric distribution cooperative located in Stillwater, Oklahoma, has installed Pepper to monitor outages. CEC serves more than 20,000 customers in seven counties in Oklahoma. Pepper is connected to the operations center to read information about live outages, and by connecting them to geographic information system (GIS) maps it can also inform operations about the live locations of service trucks. At CEC, Pepper is also used for conferences where attendees can know more about the company and its services. Pepper answers a range of questions regarding energy consumption. In the future, the company plans to invest more in robots to meet its requirements. See Figure 10.2 that shows Pepper participating as a team member during a prospective employee interview to provide input about CEC’s programs and so on.

• Fabio, a Pepper robot, was installed as a retail assistant at an upmarket food and wine store in England and Scotland. A week after implementing it, the store pulled the service because it was confusing customers, and they preferred the service from personal staff rather than Fabio. It provided generic answers on queries such as the shelf location of items. However, it failed to understand completely what the customer was requesting due to background noise. Fabio was provided another chance by placing it in a specific area that attracted only a few customers. Then they also complained about Fabio’s inability to move around the supermarket and direct them to a specific section. Surprisingly, the staff at the market became accustomed to Fabio rather than considering it as a competitor.

FIGURE 10.2 Pepper Robot as a Participant in a Group Meeting. Source: Central Electric Cooperative.

592 Part IV • Robotics, Social Networks, AI and IoT

• Pepper has several security concerns that were pointed out by Scandinavian research- ers. According to them, it is easy to have unauthenticated root-level access to the bot. They also found the robot to be prone to brute force attack. Pepper’s functions can be programmed using various application programming interfaces (APIs) through languages such as Python, Java, and C + +. This feature can cause it to provide access to all its sensors, making it not secure. An attacker can establish a connection and then use Pepper’s mic, camera, and other features to spy on people and their conver- sations. This is an ongoing issue for many robots and smart speakers.

Compiled from “Pepper Humanoid robot helps out at hotels in two of the nation’s most-visited destinations (2017)”. SoftBank Robotics. https://usblog.softbankrobotics.com/pepper-heads-to-hospitality-humanoid- robot-helps-out-at-hotels-in-two-of-the-nations-most-visited-destinations (accessed November 2018); R. Chirgwin. (2018, May 29). “Softbank’s ‘Pepper’ Robot Is a Security Joke.” The Register. https://www.theregister. co.uk/2018/05/29/softbank_pepper_robot_multiple_basic_security_flaws/ (accessed September 2018); A. France. (2014, December 1). “Nestlé Employs Fleet of Robots to Sell Coffee Machines in Japan.” The Guardian. https://www.theguardian.com/technology/2014/dec/01/nestle-robots-coffee-machines-japan-george- clooney-pepper-android-softbank (accessed September 2018); Jiji. (2017, November 21). “SoftBank Upgrades Humanoid Robot Pepper.” The Japan Times. https://www.japantimes.co.jp/news/2017/11/21/business/tech/ softbank-upgrades-humanoid-robot-pepper/#.W6B3qPZFzIV (accessed September 2018); C. Prasad. (2018, January 22). “Fabio, the Pepper Robot, Fired for ‘Incompetence’ at Edinburgh Store.” IBN Times. https://www. ibtimes.com/fabio-pepper-robot-fired-incompetence-edinburgh-store-2643653 (accessed September 2018).

Da Vinci Surgical System

Over the last decade, the use of robotics has emerged in surgeries. One of the most famous robotic systems used in surgery is the Da Vinci system that has performed thou- sands of surgeries. According to surgeons, Da Vinci is the most ubiquitous robot used in more units than any other robot. It is designed to perform numerous nominally invasive operations and can perform simple as well as complex and delicate surgeries. The criti- cal components of Da Vinci are the surgeon console, patient side cart, endowrist instru- ments, and vision system.

The surgeon console is where the surgeon operates the machine. It provides a high- definition, 3D image of the inside of the patient’s body. The console has master controls that a surgeon can grasp by the robotic fingers and operate on the patient. The movements are accurate and in real time, and the surgeon is entirely in control and can prevent the robotic fingers from moving by themselves. The patient side cart is the location where the patient resides during the operation. It has either three or four arms attached that the surgeon con- trols using master controls, and each arm has certain fixed pivot points around which the arms move. The third component is the endowrist instruments, which are available while performing surgery. They have a total of seven degrees of freedom, and each instrument is designed for a specific purpose. Levers can be released quickly for a change of instruments. The last component is a vision system, which has a high-definition, 3D endoscope and image-processing device that provides real-life images of the patient’s anatomy. A viewing monitor also helps the surgeon by providing a broad perspective during the process.

Patients who have surgery that used the Da Vinci system heal faster than those performed by traditional methods because the cuts by robotic arms are quite small and precise. A surgeon must undergo online and hands-on training and must perform at least five surgeries in front of a surgeon who is certified to use the Da Vinci system. This tech- nology does increase the cost of the surgery, but its ability to ease pain while increasing precision makes it the future of such procedures.

Compiled from “Da Vinci Robotic Prostatectomy – A Modern Surgery Choice!” (2018). Robotic Oncology. https://www.roboticoncology.com/da-vinci-robotic-prostatectomy/ (accessed September 2018); “The da Vinci® Surgical System.” (2015, September). Da Vinci Surgery. http://www.davincisurgery.com/da-vinci- surgery/da-vinci-surgical-system/ (accessed September 2018).

Chapter 10 • Robotics: Industrial and Consumer Applications 593

Snoo – A Robotic Crib

Snoo, a robotic, Wi-Fi-enabled crib was developed by Yves Behar, pediatrician Dr. Harvey Karp, and MIT–trained engineers. According to its designers, Snoo mimics Dr. Karp’s famous sleep strategy called the five S’s, which implies swaddled, side or stomach position, shush, swing, and suck. Snoo is an electrified crib that puts babies to sleep automatically. It recreates sensations experienced by the child during the last tri- mester of a pregnancy. Infants are at maximum ease when they hear white noise, feel movements, and are wrapped, which Snoo provides at par. Once a baby is securely attached to the bassinet, Snoo senses whether it is fussy, keeps track of its movements, and, if found, moves the crib in a womblike motion until the baby calms down. An app can be installed on Snoo’s smartphone to control its speed and white noise. Also, Snoo can be turned off after eight minutes or can continue rocking through the night. The company advertises it as the safest bed ever made with a built-in swaddling strap that ensures that the child does not move from his or her back. Snoo prevents parents from getting up several times in the night to do this themselves; hence, it gives them a sound sleep.

Compiled from S. M. Kelly. (2017, August 10). “A Robotic Crib Rocked My Baby to Sleep for Months.” CNN Tech. https://money.cnn.com/2017/08/10/technology/gadgets/snoo-review/index.html (accessed September 2018); L. Ro. (2016, October 18). “World’s First Smart Crib SNOO Will Help Put Babies to Sleep.” Curbed. https:// www.curbed.com/2016/10/18/13322582/snoo-smart-crib-yves-behar-dr-harvey-karp-happiest-baby (accessed September 2018).

MEDi

MEDi, short for Machine and Engineering Designing Intelligence, is available at six hos- pitals in Canada and one in the United States. MEDi helps reduce stress in children from painful surgeries, tests, and injections. It is two feet tall and weighs around 11 pounds. It looks like a toy. Dr. Tanya Beran proposed using MEDi after working in hospitals where she heard children exclaiming with joy at the sight of the robot. She suggests that since there is not enough pain management expertise available in such situations, technology can provide a helping hand. The robot can speak 19 languages and can easily be integrated into various cultures. Aldebaran built this robot, which calls itself NAO. It can cost $8,000 and more. Beran bought MEDi to life by adding software that could operate in hospital settings with kids. MEDi strikes up conversations with the kids during a variety of procedures. It was first programmed for flu vaccines and since has been used in other tests. MEDi can even tell story to a child. The robot helps not only children but also nurses by lowering children’s stress and relaxing them. Parents have said that when children leave the hospital, they did not speak about needles and pain but in fact left with happy memories.

Compiled from A. Bereznak. (2015, January 7). “This Robot Can Comfort Children Through Chemotherapy.” Yahoo Finance. https://finance.yahoo.com/news/this-robot-can-comfort-children-through-107365533404.html (accessed September 2018); R. McHugh & J. Rascon. (2015, May 23). “Meet MEDi, the Robot Taking Pain Out of Kids’ Hospital Visits.” NBC News. https://www.nbcnews.com/news/us-news/meet-medi-robot-taking- pain-out-kids-hospital-visits-n363191 (accessed September 2018).

Care-E Robot

Airports are growing in size and the number of people who go to them, and this has in- creased air traffic, flight cancellations, and gate switches, causing travelers to run to differ- ent boarding gates. KLM Royal Dutch Airlines is trying a new way to ease this process from problems related to security, boarding gates, and hectic travel with the use of the blue bot “Care-E Robot.” This service is scheduled to launch at international airports in New York and

594 Part IV • Robotics, Social Networks, AI and IoT

San Francisco. This robot could be found at security checkpoints and take travelers and their carry-on luggage wherever they need to go. Through its nonverbal sounds and signals, Care-E directs travelers to scan their boarding passes and, once scanned, is at their service when they are busy strolling the shops or using restrooms. Care-E also avoids collision using eight sen- sors with its “peripheral collision avoidance.” One of its best features is to relate boarding gate changes to travelers and provide them transportation to the newly assigned gate.

Care-E Robot can carry luggage weighing up to 80 pounds. It runs at a speed of 3 mph, which might be a little too slow for someone running late to catch a flight. However, early travelers who want to explore the airport can use Care-E on a free trial for two days. Implementations of robots like these have not yielded the desired results due to frequent changes in airport policies regarding batteries, but the market for such a robot is quite optimistic about its future.

Compiled from M. Kelly. (2018, July 16). “This Adorable Robot Wants to Make Air Travel Less Stressful.” The Verge. https://www.theverge.com/2018/7/16/17576334/klm-royal-dutch-airlines-robot-travel-airport (accessed September 2018); S. O’Kane. (2018, May 17). “Raden is the Second Startup to Bite the Dust After Airlines Ban Some Smart Luggage.” Circuit Breaker. https://www.theverge.com/circuitbreaker/2018/5/17/17364922/raden- smart-luggage-airline-ban-bluesmart (accessed September 2018).

AGROBOT

The combination of sweetness loaded with multiple health benefits makes strawber- ries one of the world’s most popular and consumed fruits. Close to 5 million tons of strawberries are harvested every year, an upward trend in the United States, Turkey, and Spain as top harvesters. AGROBOT, a company engaged in the business of agricultural robots, has developed a robot that can harvest strawberries at any place. Robots using 24 robotic manipulators built on a mobile platform work to identify superior quality strawberries.

Strawberries require a high degree of care because they are delicate compared to other fruits. Fruits such as apples, bananas, and mangoes ripen after being picked whereas strawberries are picked at their full maturity. Hence, harvesting strawberries has been an entirely manual process until recently. AGROBOT was developed in Spain; this robot performs automated processes except selecting the strawberries and packing them. To protect strawberries from being squeezed during picking, the robot cuts them with two razor-sharp blades and catches them in baskets lined with rubber rolls. Once full, the baskets are placed on a conveyor belt and passed to the packing station. Human opera- tors can directly select and pack the berries.

AGROBOT is operated by one man, and a maximum of two people can ride on it. Robotic arms control the coordination between blades and basket. The robot has four main components: inductive sensors, ultrasonic sensors, a collision control system, and a camera system. Camera-based sensors view each fruit and analyze it for ripeness ac- cording to its form and color; once a berry is ripe, the robot cuts it from its branches with precise movements. Each arm is fortified with two inductive sensors to stop at the end positions. The collision control system must be capable of responding to dust, tempera- ture change, vibration, and shock; hence, an ultrasonic sensor is attached to the robot to prevent the arms from touching the ground. Each wheel is equipped with ultrasonic sen- sors to determine the distance between the strawberry and the robot’s current position. These sensors also help in keeping the robot on track and preventing damage to the fruit. Signals received from the sensors are continuously transmitted to an automatic steering system to regulate the position of wheels.

Compiled from “Berry Picking at Its Best with Sensor Technology.” Pepperl+Fuchs. https://www.pepperl- fuchs.com/usa/en/27566.htm (accessed September 2018); R. Bogue. (2016). “Robots Poised to Revolutionise Agriculture.” Industrial Robot: An International Journal, 43(5), pp. 45–456; “Robots in Agriculture.” (2015, July 6). Intorobotics. https://www.intorobotics.com/35-robots-in-agriculture/ (accessed September 2018).

Chapter 10 • Robotics: Industrial and Consumer Applications 595

u SECTION 10.4 REVIEW QUESTIONS

1. Identify applications of robots in agriculture. 2. How could a social support robot such as Pepper or MEDi be useful in healthcare? 3. Based on the illustrative applications of robots in this section, build a matrix where

the rows are the robots’ capabilities and the columns are industries. What similarities and differences do you observe across these robots?

10.5 COMPONENTS OF ROBOTS

Depending on their purpose, robots are made of different components. However, all robots have some common ones, and others are tweaked according to a robot’s pur- pose. Figure  10.3 identifies the components. Common components of the robots are described next.

CPU or Controller It can be said that “the controller controls the robot.” This processes information received from sensors and directs effectors to the next action to be performed.

Power Supply Activates sensors, effectors, and every part of the robot. It converts electrical energy into mechanical energy.

Effectors Effectors are the parts that do the work. They can be hands, legs, or any body part that constitutes a robot. End effectors interact with the objects in the real world and are specific.

Sensors Sensors are measuring devices to capture quantities such as velocity, position, force, temperature, and so on. These mimic human behavior such as listening and capturing. Data captured by these are transferred to CPU.

Actuator System Actuators along with other devices that convert electrical energy into its mechanical form or work is the actuator system. Actuators can be various types of motors that perform the job.

FIGURE 10.3 General Components of a Robot.

596 Part IV • Robotics, Social Networks, AI and IoT

POWER CONTROLLER A power controller is the driving force of a robot. Most robots run on batteries, but a few are powered by a direct current (DC) electrical supply. Other factors (i.e., usage, sufficient power to drive all parts) must be kept in mind while designing robots.

SENSORS Sensors are used to direct a robot in its surrounding. Force sensors, ultra- sound sensors, distance sensors, laser scanners, and so on help robots to make decisions according to their environment. Sensors are used for robots to identify speech, vision, temperature, position, distance, touch, force, sound, and time. Vision sensors or cameras are used to build a picture of the environment and for the robot to learn about it and to differentiate between which items to choose and which to ignore. In collaborative robots, sensors are also used to prevent them from bumping into humans or other robots. This way, humans and robots can work next to each other without the fear that the robot might unintentionally harm the human. Sensors collect information and send it to the central processing unit (CPU) electronically.

EFFECTORS OR ROVER OR MANIPULATOR An effector is nothing but a body of a robot. It can also describe the devices that affect the environment, such as hands, legs, arms, bod- ies, and fingers. The CPU controls the actions of effectors. An essential function of them is to move the robot and other objects from one place to another, and their characteristics depend on the role that has been outlined. Industrial robots have end effectors that con- tribute to the robot’s work as a hand. Depending on the type of robot, end effectors can be magnets, welding torches, or vacuums.

NAVIGATION OR ACTUATOR SYSTEM Actuators are devices that define how a robot trav- els. With the help of an actuator, electrical energy converts into mechanical energy, en- abling the robot to move back, forward, left, right and to lift, drop, and perform its job. The actuator can be a hydraulic cylinder or an electric motor. The actuator system is the way that all of the robot’s components are embedded into one.

CONTROLLER/CPU This is the brain of the robot and has the AI embedded in it. The CPU allows a robot to perform its function by connecting all systems into one. It also provides commands for the robot to learn from the surrounding movement of the body or any of its actions.

u SECTION 10.5 REVIEW QUESTIONS

1. What are the common components of a robot? 2. What is the function of sensors in a robot? 3. How many different types of sensors might exist in a robot? 4. What is the function of a manipulator?

10.6 VARIOUS CATEGORIES OF ROBOTS

Robots perform a variety of functions. Depending on these, robots can be categorized into the following categories.

PRESET ROBOTS Preset robots are preprogrammed. They have been designed to perform the same task over time and can work 24 hours a day, 7 days a week without any breaks. Preset robots do not alter their behavior. Therefore, these robots have an incredibly low error rate and are suitable for wearisome work. They are frequently used in manufacturing sectors such as the mobile industry, vehicle manufacturing, material handling, and welding to save time and money. Preset robots deliver jobs in environments where it is hazardous

Chapter 10 • Robotics: Industrial and Consumer Applications 597

for humans to work. Robots move heavy objects, perform assembly tasks, paint, inspect parts, and handle chemicals. A preset robot articulates according to the operation it per- forms. It can perform a significant role in the medical field because the tasks it performs must have high efficiency at a level comparable to human beings.

COLLABORATIVE ROBOTS OR COBOTS Cobots are the robots that can collaborate with human workers, assisting them to achieve their goals. The use of cobots is trending in the market, and there is an excellent outlook for collaborative robots. According to the sur- vey by MarketsandMarkets, the cobots market in 2020 will be worth around $3.3 billion. There are various functions of collaborative robots. Depending on the usage, the collab- orative robots are used. Collaborative robots have various applications in manufacturing as well as the medical industry.

STAND-ALONE ROBOTS Stand-alone robots are the robots that have a built-in AI system and work independently without much interference from humans. These robots per- form tasks depending on the environment and adapt to changes in it. With the use of AI, a stand-alone robot learns to modify its behavior and excel in performing its assign- ment. Autonomous robots have household, military, education, and healthcare applica- tions. They can walk like a human being, avoid obstacles, and provide social-emotional support. Some of these robots are used for domestic purposes as stand-alone vacuum cleaners, such as iRobot Roomba. Stand-alone robots are also used in hospitals to deliver medications, keep track of patients who are yet to receive them, and send this informa- tion to the nurses working on that shift and other shifts without chance of any error.

REMOTE-CONTROLLED ROBOTS Even though robots can perform stand-alone tasks, they do not have human brains; hence, many tasks require human supervision. These robots can be controlled via Wi-Fi, Internet, or satellite. Humans direct remote-controlled robots to perform complicated or dangerous tasks. The military uses these robots to detonate bombs or to act as soldiers around the clock on the battlefield. In the space program research field, their scope of use is extensive. Remote-controlled cobots are also used to perform marginally invasive surgeries.

SUPPLEMENTARY ROBOTS Supplementary robots enhance the existing capabilities or replace capabilities that a human has lost or does not have. This type of robot can be directly attached to a human’s body. It connects to a user’s body and communicates with the robot’s operator directly or when the operator grips the body. The robot can be controlled by a human body, and in some cases, even by thinking of a specific action. Its applications include serving as a robotic prosthetic arm or providing precision for the surgeons. Extensive research on building prosthetic limbs is being conducted.

u SECTION 10.6 REVIEW QUESTIONS

1. Identify some key categories of robots. 2. Define and illustrate the capabilities of a cobot. 3. Distinguish between a preset robot and a stand-alone robot. Give examples of each.

10.7 AUTONOMOUS CARS: ROBOTS IN MOTION

A robot that may eventually touch most people’s lives is an autonomous (self-driving) car. Like many other technologies, self-driving cars have been at peak hype recently, but peo- ple also recognize their technical, behavioral, and regulatory challenges. Nevertheless, technology and processes are evolving to make the self-driving car a reality in the future,

598 Part IV • Robotics, Social Networks, AI and IoT

at least in specific settings if not all over the world. Early versions of self-driving cars were enabled by the radio antenna developed in 1925. In 1989, researchers at Carnegie Mellon used neural networks to control an autonomous vehicle. Since then, many technologies have come together to accelerate development of self-driving cars. These include:

• Mobile phones: With the help of low-powered computer processors and other accessories such as cameras, mobile phones have become ubiquitous. Many tech- nologies developed for phones, such as location awareness and computer vision, are finding applications in cars.

• Wireless Internet: Connectivity has become much more feasible with the rise of 4G networks and Wi-Fi. Going forward, growth in 5G will perhaps be important for self-driving cars to allow their processors to communicate with each other in real time.

• Computer centers in cars: A number of new technologies are available in today’s cars, such as rearview cameras and front and back sensors that help vehicles detect objects in the environment and alert the driver to them or even take necessary actions automatically. For example, adaptive cruise control automatically adjusts the speed of a car based upon the speed of the vehicle in front.

• Maps: Navigation maps on mobile phones or navigation systems in cars have made a driver’s job easy with regard to navigation. These maps enable an autono- mous vehicle to follow a specific path.

• Deep learning: With advances in deep learning, the ability to recognize an object is a key enabler of self-driving cars. For example, being able to distinguish a person from an object such as a tree, or whether the object is moving or stationary is critical in taking actions in a moving vehicle.

Autonomous Vehicle Development

The heart of an autonomous vehicle system is a laser rangefinder (or light detection and ranging – lidar device), which is on the vehicle’s roof. The lidar generates a 3D image of the car’s surroundings and then combines it with high-resolution world maps to pro- duce different data models for taking action to avoid obstacles and follow traffic rules. In addition, many other cameras are mounted. For example, a camera positioned near a rearview mirror detects traffic lights and takes videos. Before making any navigation deci- sions, the vehicle filters all data collected from the sensor and camera and builds a map of its surroundings and then precisely locates itself in that map using GPS. This process is called mapping and localization.

The vehicle also consists of other sensors such as the four radar devices that are on the front and back bumpers. These devices allow the vehicle to see far distances so that they can make decisions beforehand and deal with fast-moving traffic. A wheel encoder determines the vehicle’s location and maintains records of its movements. Algorithms such as neural networks, rule-based decision making, and a hybrid approach are used to determine the vehicle’s speed, direction, and position, and the collected data are used to direct the vehicle on the road to avoid obstacles.

Autonomous vehicles must rely on detailed maps of roads. Thus, before sending driverless cars on roads, engineers drive a route several times and collect data about its surroundings. When driverless vehicles are in operation, engineers compare the data ac- quired by them to the historical data.

There is an entire town built for the sole purpose of testing autonomous vehicles. It is located in Michigan. This city has no single resident, and self-driving vehicles roam the streets without the risks in the real world. This city, called Mcity, is truly a city for robotic vehicles. Mcity includes intersections, traffic signals, buildings, construction work, and

Chapter 10 • Robotics: Industrial and Consumer Applications 599

moving obstacles such as humans and bicycles similar to those in real cities. Autonomous vehicles are not only tested in this closed environment but are being used in the real world as well.

Google’s Waymo unit is one of the early pioneers of self-driving vehicles. They have been tested on California roads, but before they start to drive next to human-driven cars, companies have to test them thoroughly because one negative incident can impede their acceptance. For example, in the spring of 2018, a self-driving vehicle being tested by Uber killed a pedestrian in Tempe, Arizona. This led to the suspension of all public testing of autonomous vehicles by Uber. The technology is still in development, but it has come far enough that limited testing on public roads is safe. We might be surprised in the near future by the fact that the person in the driver’s seat of a vehicle next to you in traffic might not actually be driving it at all.

In 2016, the U.S. Department of Transportation (DOT) began to embrace driverless vehicles to speed their development. In September 2016, DOT announced the first-ever guidelines for autonomous driving. A groundbreaking announcement by the National Highway Traffic Safety Administration (NHTSA) a month later allowed for the AI system controlling Google’s self-driving vehicle to be considered a driver in response to the com- pany’s proposal to the NHTSA in November 2015.

Some states currently have specific laws that ban autonomous driving. For example, as of this writing, the state of New York does not allow any hands-free driving. Without clear regulations, testing self-driving vehicles is a challenge. Although a few states such as Arizona, California, Nevada, Florida, and Michigan currently allow autonomous vehicles on the road, California is the only one with licensing regulations at this point.

Google might be the most well known for autonomous vehicles, but it is not the only one. A handful of the most powerful companies, such as Uber and Tesla, are in the same race as well. Every major car company is working either with technology compa- nies or its own technology to develop autonomous vehicles or at least to participate in this revolution.

Issues with Self-Driving Cars

Autonomous cars have been connected to a number of issues.

• Challenges with technology: There have been several challenges with the tech- nology used in self-driven cars. Several software and mechanical hurdles are still to be overcome in order to roll out a fully autonomous car. For example, Google is still trying to update its software on an almost daily basis for its self-driven car. Several other companies are still trying to figure out the amount of authority to be trans- ferred when a human driver takes control from an automatic vehicle.

• Environmental challenges: Technology and mechanical capabilities cannot yet address many environmental factors affected by self-driving cars. For example, there are still concerns regarding their performance in bad weather. Likewise, several systems have not been tested in extreme conditions such as snow and hail. There are several tricky navigating situations on the road, such as when an animal jumps onto it.

• Regulatory challenges: All companies planning to become involved with self- driving cars need to address regulatory hurdles. There are still many unanswered questions about the regulation of autonomous driving. Several questions about lia- bility include these: What will a license involve? Will new drivers be required to get traditional licenses even if they are not drivers? What about young people, or older people with disabilities? What will be required to operate these new vehicles? Governments need to work quickly to catch up with the booming technology.

600 Part IV • Robotics, Social Networks, AI and IoT

Considering that public safety is on the line, auto regulations should be some of the strictest regulations in the modern world.

• Public trust issues: Most people do not yet believe that an autonomous car can keep them safe. Trust and consumer acceptance are the crucial factors. For example, if there is a situation when an autonomous car is being forced to choose between the life of a passenger versus that of a pedestrian, what should be done? Consumers may refuse the whole idea of driverless cars. No technology can be perfect, but the question is which company will be able to best convince its customers to entrust their lives to them.

Advances similar to those for self-driving cars are being explored in other autono- mous vehicles. For example, several companies have already launched trials of self-driving trucks. Autonomous trucks, if ever fully deployed, will have a massive disruptive effect on jobs in the transportation industry. Similarly, self-driving tractors are being tested. Finally, autonomous drones and aircrafts are also being developed. These developments will have a huge impact on future jobs while creating other new jobs in the process.

Self-driving vehicles have become part of this world of technology in spite of related technical and regulatory barriers. Autonomous vehicles are yet to achieve the knowledge capabilities of human drivers, but as the technology improves, more-reliable driving vehi- cles will become a reality. Like many technologies, the short-term impact may be cloudy, but the long-term impact is yet to be determined.

u SECTION 10.7 REVIEW QUESTIONS

1. What are some of the key technology advancements that have enabled the growth of self-driving cars?

2. Give examples of regulatory issues in self-driving cars. 3. Conduct online research to identify the latest developments in autonomous car

deployment. Give examples of positive and negative developments.

4. Which type of self-driving vehicles are likely to have the most disruptive effect on jobs, and why?

10.8 IMPACT OF ROBOTS ON CURRENT AND FUTURE JOBS

Robotics has been a boon to the manufacturing industry. Besides automation that is pos- sible with robotics, new technologies such as image recognition systems are automating jobs that used to require humans for inspection and quality control.

Various industry experts report that by 2025, up to 25 percent of current jobs will be replaced by robots or AI. Davenport and Kirby’s book Humans Need Apply: Winners and Losers in the Age of Smart Machines (2016) focuses on this topic. Of course, many other researchers, journalists, consultants, and futurists have given their own predictions. In this section, we review some related issues. These issues are relevant to AI in general and robotics in particular. Thus, Chapter 14 will also cover these issues, but we want to study these in the context of robotics in this chapter.

As a group activity, watch the following video: https://www.youtube.com/watch? v=GHc63Xgc0-8. Also watch https://www.youtube.com/watch?v=ggN8wCWSIx4 for a different view. What are your takeaways from these videos? What is the most likely scenario in your view? How can you prepare for the day when indeed humans may not need to apply for many jobs?

IBM Watson’s ability to digest vast amounts of data in the medical research literature and provide the latest information to a physician has been written about in the literature. Similar job enhancement opportunities in many other areas have been seen. Consider

Chapter 10 • Robotics: Industrial and Consumer Applications 601

this: AI-powered technologies such as narrative science and automated insights that can ingest structured data include visualizations generated by software such as Tableau and develop an initial draft of a story to narrate what the results convey. Of course, that would appear to threaten the job of a journalist or even a data scientist. In reality, this can also enhance that job by presenting an initial draft of a story. Then the storyteller can focus on more advanced and strategic issues related to that data and visualization.

The power of consistency and comprehensiveness can also be helpful in the com- pletion of jobs. For example, as noted by Meister (2017), chatbots can likely provide much of the initial human resource (HR) information to new employees. Chatbots can also be helpful in providing such information to remote employees. A chatbot is more likely than a human to provide complete and consistent information each time. Of course, this im- plies that workers whose main job is to recite such information to each new employee or serve as the first source of information may not be needed.

Hernandez (2018) identified seven job categories into which robotics in particular and AI in general will expand. She also quoted several other studies. According to a McKinsey & Co study, AI could result in 20950 million new jobs in the next 10915 years. McKinsey also predicts that 759375 million people may need to change jobs/occupations in the same time period because of robotics and AI. According to Hernandez, the follow- ing seven jobs are likely to increase:

1. AI development: This is an obvious growing area. As more companies develop products and services based on AI, the need for such developers will continue to increase. As an example, iRobot Co, which produces robotic vacuum cleaners, is shifting its hiring from hardware to software engineers as it works to develop its next generation of products that are more adaptive and AI based. Newer robot vacuums are going to be able to “see” a wall. They can also alert the owner to how long the cleanup took and the area swept.

2. Customer–robot interactions: As more companies deploy robots in these orga- nizations, acceptance of such robots by both employees and customers is uncertain. A new job category has emerged to study the interactions between a robot and its coworkers and customers and to retrain the robot or take this information into ac- count in designing the next generation. Clearly, the study of such interactions may enable the use of analytics/data science as well.

3. Robot managers: Although robots might do the bulk of their work in a spe- cific situation, humans will still need to observe them and ensure that the work is progressing as expected. Further, if any unusual conditions arise, a human worker has to be alerted and respond to the situation. This would be true in many settings where the robots are performing the bulk of tasks in areas such as manufacturing. Hernandez (2018) gives an example of Cobalt robots, which work as security guards. These robots alert a human whenever an intruder is detected or they notice anything unusual. Of course, a human robot manager is typically able to supervise many more robots than human workers because the primary role of the manager is to supervise them and respond to unusual situations.

4. Data labelers: Robots or AI algorithms learn from examples. And the more ex- amples they are given, the better their learning can be (see Chapter 5 for a longer description of this issue). For example, image recognition systems in virtually every setting (see Chapter 5 on deep learning for examples) require as many examples as possible to improve those systems’ recognition capability. This is crucial for not just facial recognition but also image applications to detect cancer from X-ray images, weather features from radar images, and so on. It requires that humans view the example images and label them as representing a specific person, feature, or class. This work is tedious and requires humans. Many companies have hired hundreds

602 Part IV • Robotics, Social Networks, AI and IoT

of human labelers to view the images and tag them appropriately. As such image applications grow, the need for labelers will also increase. These workers are also needed for continuous improvement of the robot or AI algorithms by recording false positives or newer examples.

5. Robot pilots and artists: Robots in general and drones in particular are being used to provide action shots using overhead cameras or angles that would be dif- ficult if not impossible for humans to do. Drones could also be dressed as birds or flowers and provide a unique overview as well as enhance a setting. Similarly, other robots might be dressed in unique outfits to create a cultural ambiance. Such designer/makeup artists are being hired by many companies that provide services for events such as concerts, weddings, and so forth. In addition, drone piloting has become a highly specialized skill for entertainment, commercial, and military appli- cations. These jobs will increase as the applications evolve.

6. Test drivers and quality inspectors. Autonomous cars are already becoming reality. With each such automation of vehicles, at least for the foreseeable future, there is a growing need for safety drivers who monitor each vehicle’s performance and take appropriate actions in unusual situations. Their jobs would not entail the use of remote controls as drone pilots employ but continuous watch of the vehicle’s operations and response to emergent situations. Similar jobs also exist in other ro- botic applications as the robots are trained and tested to work in specific settings.

7. AI lab scientists. This brings us to the very first category of new jobs we identified—AI coders. While their job is to develop the algorithms for robots or AI programs, a similar category of highly specialized users is also emerging—folks who are trained and employed in using these hardware and software systems for special applications. For example, physicians have to undergo additional training to be cer- tified in the use of robots in their surgeries, cardiology and urology practices, and so on. Another category of such specialists involves scientists who customize these robots and AI algorithms for their domain. For example, quite a few companies are using AI tools to identify new drug molecules to develop and test new treatment options for diseases. AI could speed such development. These scientists not only de- velop their domain expertise but also data scientists’ knowledge or at least the ability to work with data scientists to create their new applications.

Although the preceding list identifies several categories of jobs that are likely to develop or increase, millions of jobs are likely to be eliminated. For example, automation is already impacting the number of jobs in logistics. When autonomous trucks become a reality, at least some of the well-paying jobs in transportation will likely be gone. There might be disagreements on when the massive change may occur, but the long-term im- pact on jobs is certain to occur. The major issue this time is that many of the knowledge economy “white-collar” jobs are the ones that are more likely to be automated. And this change is unprecedented in history. Many social scientists, economists, and leading think- ers are worried about the upheaval that this next wave of robotic automation will cause, and they are considering various solutions. For example, the concept of universal basic income (UBI) has been proposed. UBI proponents argue that giving every citizen a mini- mal basic income will ensure that no one goes hungry despite the massive loss of jobs that is likely to occur. Others, for example, Lee (2018), have argued that providing UBI may not satisfy human beings’ need for meaningful achievements and contributions in life. Lee proposes a social investment stipend (SIS), which would recognize individuals’ contributions to society for providing support and care, community service, or education. The stipend would be paid in recognition of an individual’s service in one of these cat- egories. Lee’s book focuses on this issue and is one of the many ideas being proposed on how to plan for and address the upcoming disruption from automation. Our goal in this section is to simply alert you to these issues.

Chapter 10 • Robotics: Industrial and Consumer Applications 603

u SECTION 10.8 REVIEW QUESTIONS

1. Which jobs are most at risk of disappearing as the result of the new robotics revolution? 2. Identify at least three new categories of jobs that are likely to result in a significant

number of new employees.

3. Are the tasks undertaken by data labelers just for one time or longer lasting? 4. Research the concepts of UBI and SIS.

10.9 LEGAL IMPLICATIONS OF ROBOTS AND ARTIFICIAL INTELLIGENCE

As we noted in the previous section, the impact of AI in general and robotics in particular is far and wide and can be studied both specifically in the context of robots and more broadly for AI. Legal implications of robotics and AI are discussed in this chapter and in Chapter 14. Many legal issues are yet to be untangled as we embrace and employ AI tech- nologies, robots, and self-driving cars. This section highlights some of the key dimensions of legal impacts related to AI. The following material has been contributed by Professor Michael Schuster, assistant professor of Legal Studies at the Spears School of Business at Oklahoma State University. He is a noted expert in legal matters related to AI. He has also published extensively in this area.

Tort Liability

Self-driving cars and other systems controlled by AI represent a Pandora’s box of po- tential tort liability (where a wrongful act creates on obligation to pay damages to an- other). Imagine that a motorcyclist is injured when a self-driving car veers into his lane and they collide. This was the alleged event that led to Nilsson v. General Motors (N.D. California, 2018)—a case with the potential to address difficult questions of AI-created tort liability. The suit settled, and thus did not clarify who should pay when someone is injured by an AI-controlled system. Potential candidates for liability include program- mers of an AI system, manufacturers of a product incorporating AI, and owners of a product at the time it harmed another. Medical malpractice litigation may similarly be altered by new technologies. As doctors defer some decision making to AI, lawsuits for injurious medical care move from professional liability cases (against the doctor) to product liability (against manufacturers of AI systems). An early example of this phe- nomenon is the lawsuits over allegedly botched surgeries using the Da Vinci Surgical Systems robot.

Patents

The introduction of AI systems capable of independent or human-assisted invention raises a variety of questions about patenting these creations. Patents have traditionally been granted for novel inventions that would not have been an obvious improvement of a known technology as viewed through the eyes of an average party working in the relevant field. Accordingly, this standard has traditionally asked whether a new technol- ogy would have been obvious to a human, but as AI becomes ubiquitous, the scope of what is obvious expands. If the average person in an industry has access to an AI system capable of inventing new things, many improvements on known technologies can be- come obvious. Since these improvements would then be obvious, they would no longer be patentable. Inventing AI will thus make it harder for a human to get a patent as such technology becomes more commonplace. Moving beyond human inventions, a host of issues arise regarding AI that can independently invent. If a person does not contribute to the invention (but rather merely identifies a goal to be achieved or provides background

604 Part IV • Robotics, Social Networks, AI and IoT

data), he or she does not satisfy the statutory threshold to be an inventor (and thus, does not qualify for patent ownership). See Schuster (2018). If AI creates an invention without a human inventor, who owns the patent or should a patent be granted at all? Some assert the U.S. Constitution’s mandate that patents can be granted only to “inventors” necessarily requires a human actor, and thus, Congress cannot constitutionally allow the patenting of AI-created technologies. Additional commentators present a variety of policy positions arguing why parties such as the computer’s owner, the AI’s creator, or others should own patents for computer-created inventions. These issues are yet to be resolved.

Property

A basic tenet of U.S. law is a strong protection of property rights. These values extend to corporate entities that can own both real estate and movable property to the exclu- sion of all others while shareholders retain some allotted portion of the corporation itself. Analysts are presently addressing to what extent property rights should extend to autono- mous AI. If a robot were to engage in work for hire, might it be able to make purchases of goods or realty to further its interests? Could an eccentric octogenarian millionaire leave his entire fortune to a loyal robot housekeeper? Topics of this nature will raise a va- riety of new legal queries, including intestate passage of AI-owned property at its “death” and the standard for death in a nonbiological entity.

Taxation

Robotics and AI will replace a significant number of jobs presently undertaken by hu- mans. Commentators are divided on whether new technologies will create a number of new jobs equal to those replaced by automation. Should the scope of new jobs fall shy of those lost, it is feasible that tax-based issues will be created. A particular concern deals with federal payroll taxes whereby workers and employers pay taxes premised on wages made by the employee. If the aggregate number of workers and net pay are reduced by job automation, the payroll tax base will be reduced. Given that these taxes are important to the sustainability of various government-run safety net programs (e.g., Social Security), payroll tax shortfalls could have significant societal ramifications. In 2017, Bill Gates (Microsoft cofounder) set forth a proposal to tax robots that are used to automate existing human jobs. This new tax would theoretically supplement extant payroll taxes to ensure continued funding for government programs. Commentators are split on the advisability (or need) for such a tax. A common criticism is that tax- ing robots discourages technological advancement, which is contrary to the accepted policy of encouraging such endeavors. A satisfactory resolution to this debate is yet to be reached.

Practice of Law

Beyond what the law is or should be, AI will have substantial effects within discrete seg- ments of the practice of law. A prime example of this influence is in the area of document review—the part of a lawsuit where litigants evaluate documents provided by their op- ponents for relevance to the case. Costs associated with this process can be substantial given that some cases entail review of millions of pages by attorneys who bill hundreds of dollars per hour. Corporate clients looking to reduce costs—and law firms seeking a competitive advantage—have adopted (or intend to adopt) AI-based document review systems to minimize the number of billed hours. Similarly, some firms have adopted industry-specific technologies to create competitive advantage. At least one major law firm instituted the use of an AI-driven system to analyze strengths and weaknesses of its clients’ patent portfolios.

Chapter 10 • Robotics: Industrial and Consumer Applications 605

Constitutional Law

As the state of AI advances, it continues to move toward “human-level” intelligence. But as it becomes more “personlike,” questions arise regarding whether AI should be afforded rights commonly granted to humans. The U.S. Constitution’s First Amendment provides for freedoms of speech, assembly, and religion, but should rights of this nature extend to AI? For instance, one might argue that these rights preclude the government from dictat- ing what a robot can say (violating its right to free speech). At first blush, this proposition seems far-fetched, but perhaps it is not. On the issue, it is notable that the Supreme Court of the United States recently extended some free speech rights and religious liberties to corporate entities. Accordingly, there is some domestic precedent for affording constitu- tional protections to nonhuman actors. Further, in 2017, Saudi Arabia granted citizenship to “Sophia,” a humanoid robot created by Hong Kong’s Hanson Robotics in 2015. How this issue will resolve itself (domestically and globally) remains to be seen.

Professional Certification

There are many activities for which humans must receive certification issued by a govern- ment or professional organization prior to undertaking that act (e.g., the practice of law or medicine). As the state of AI progresses, AI will increasingly be capable of perform- ing these state-regulated endeavors independent of human engagement. With this in mind, standards must be developed to determine whether an AI technology is capable of providing satisfactory service in regulated professional fields. If an autonomous robot is capable of passing a state’s bar exam, should it be able to give legal advice without human supervision? To the extent that many professional groups require annual training to maintain competence, how will these policies apply to AI technologies? Is there value in requiring that a computer undertaking legal functions “attend” continuing legal educa- tion classes? These issues will be settled as AI begins to carry out work currently done exclusively by human professionals like doctors and lawyers.

Law Enforcement

In addition to policy choices detailing what the law is or should be, AI may influence en- forcement of the law. Rapid growth in technology will soon afford police forces access to large amounts of near real-time data and computing capacity to determine where crimes are being committed. Recognized infractions may run the gamut from common public transgres- sions (e.g., running a red light) to more private acts, such as underreporting income on a tax return. The capacity to recognize such criminal acts on a large scale raises a variety of enforcement questions. Discretion in prosecuting infractions has long been a part of law enforcement. Should this power of choice regarding issues associated with stereotype-based prosecution decisions be delegated to AI systems? Moreover, machine-based enforcement programs have consistently been met with questions of their constitutionality (e.g., using cameras to identify drivers not stopping for red lights). While these arguments have thus far proven unsuccessful, they will likely be relitigated as the practice expands. Beyond enforce- ment questions, some have raised the possibility of implementing AI in the judiciary. For instance, it has been proposed that data-based sentencing may more successfully achieve targeted goals (e.g., successful education while incarcerated or avoidance of recidivism) than arguably idiosyncratic members of the judiciary. Such a mechanism will, of course, raise po- tential transparency issues and arguments relating to granting too much power to AI systems.

Regardless of the issues that the last two sections have raised, robotics technologies and applications are evolving rapidly. As managers, you have to continue to think about how to manage these technologies while being fully aware of immediate behavioral and legal issues in implementing the technologies.

606 Part IV • Robotics, Social Networks, AI and IoT

u SECTION 10.9 REVIEW QUESTIONS

1. Identify some of the key legal issues for robotics and AI. 2. Liability for harm (tort liability) is an obvious early question for any technology. What

are some of the key challenges in identifying such liability?

3. Recent news about illegal intervention in elections has led to the discussion about who is responsible for damage control. When chatbots and automated social media systems have the ability to propagate “fake news,” who should be required to moni- tor them and prevent such action?

4. What are some of the law enforcement issues in employing AI?

Chapter Highlights

• Industrial automation brought the first wave of robots, but now the robots are becoming autono- mous and finding applications in many areas.

• Robotic applications span industries such as agri- culture, healthcare, and customer service.

• Social robots are emerging as well to provide care and emotional support to children, patients, and older adults.

• All robots include some common components: power unit, sensors, manipulator/effector, logic unit/CPU, and location sensor/GPS.

• Collaborative robots are evolving quickly, leading to a category called cobots.

• Autonomous cars are probably the first category of robots to touch most consumers.

• Self-driving cars are challenging the limits of AI innovation and legal doctrines

• Millions of jobs are at risk of being lost due to the use of robots and AI, but some new job catego- ries will emerge.

• Robots and AI are also creating new challenges in many legal dimensions.

Key Terms

automation autonomous car autonomy effector

patent robot sensor social robot

tort liability universal basic income (UBI)

Questions for Discussion

1. Based upon the current state of the art of robotics ap- plications, which industries are most likely to embrace robotics? Why?

2. Watch the following two videos: https://www. youtube.com/watch?v=GHc63Xgc0-8 and https:// www.youtube.com/watch?v=ggN8wCWSIx4 for a different view on impact of AI on future jobs. What are your takeaways from these videos? What is the more likely scenario in your view? How can you prepare for the day when humans indeed may not need to apply for many jobs?

3. There have been many books and opinion pieces writ- ten about the impact of AI on jobs and ideas for societal responses to address the issues. Two ideas were mentioned

in the chapter – UBI and SIS. What are the pros and cons of these ideas? How would these be implemented?

4. There has been much focus on job protection through tariffs and trade negotiation recently. Discuss how and why this focus may or may not address the job changes coming due to robotics and AI technologies.

5. Laws rely on incentive structures to encourage proso- cial behavior. For example, criminal law encourages compliance by punishing those who break the law. Patent law incentivizes creation of new technologies by offering inventors a period of limited monopoly dur- ing which they can exclusively use their invention. To what extent do these (and other) incentives make sense when applied to AI? How can incentive structures be

Chapter 10 • Robotics: Industrial and Consumer Applications 607

created to encourage AI devices to behave in prosocial manners?

6. To what extent do extralegal considerations come into play with regard to the above issues? Are there moral (or religious) dimensions to be considered when deter- mining whether AI should be given rights similar to those of a person? Would AI-assisted law enforcement or court action erode faith in the criminal justice system and judiciary?

7. Adopting policies that maximize the value of AI encourages future development of these technologies.

Such a course, however, is not without drawbacks. For instance, determining that a “robot tax” is not a preferred policy choice would increase the incentive to adopt a robot workforce and improve any relevant technologies. Elevating the state of robotics is a laud- able goal, but in this instance, it would come at the anticipated cost of reduced public funds. How should trade-offs such as these be evaluated? Where should encouragement of technological progress (especially regarding AI) fall in the hierarchy of government priorities?

Exercises

1. Identify applications other than those discussed in this chapter where Pepper is being used for commercial and personal purposes.

2. Go through specifications of MAARS at https://www. qinetiq-na.com/wp-content/uploads/brochure_ maars.pdf. What are the functions of MAARS?

3. Conduct online research to find at least one new robot- ics application in agriculture. Prepare a brief summary of your research: the problem addressed, technology summary, results achieved if any, and lessons learned.

4. Conduct online research to find at least one new robot- ics application in healthcare. Prepare a brief summary of your research: the problem addressed, technology sum- mary, results achieved if any, and lessons learned.

5. Conduct online research to find at least one new robotics application in customer service. Prepare a brief summary of your research: the problem addressed, technology summary, results achieved if any, and lessons learned.

6. Conduct online research to find at least one new robot- ics application in an industry of your choice. Prepare a brief summary of your research: the problem addressed, technology summary, results achieved if any, and les- sons learned.

7. Conduct research to identify the most recent develop- ments in self-driving cars.

8. Conduct research to learn and summarize any new investments and partnerships in self-driving cars.

9. Conduct research to identify any recent examples of legal issues regarding self-driving cars.

10. Conduct research to identify any other new types of jobs that would be enabled by AI and robotics beyond what was covered in the chapter.

11. Conduct research to report on the latest projections for job losses due to robotics and AI.

12. Identify case stories for each of the legal dimensions identified by Schuster (2018) in Section 10.9.

References

“A Brief History of Robotics since 1950.” Encyclopedia.com. http://www.encyclopedia.com/science/encyclopedias- almanacs-transcripts-and-maps/brief-history-robotics- 1950 (accessed September 2018).

Ackerman, E. (2016). IEEE Spectrum. http://spectrum .ieee.org/automaton/robotics/home-robots/tega- mit-latest-friendly-squishable-social-robot (March 5, 2017).

“Adidas’s High-Tech Factory Brings Production Back to Ger- many.” (2017, January 14). The Economist. https://www. economist.com/business/2017/01/14/adidass-high- tech-factory-brings-production-back-to-germany (ac- cessed September 2018).

Allinson, M. (2017, March 4). “BMW Shows Off Its Smart Factory Technologies at Its Plants Worldwide.” Robot- ics and Automation. https://roboticsandautomation news.com/2017/03/04/bmw-shows-off-its-smart- factory-technologies-at-its-plants-worldwide/11696/ (accessed September 2018).

Aoki, S., et al. (1999). “Automatic Construction Method of Tree-Structural Image Conversion Method ACTIT.” Jour- nal of the Institute of Image Information and Television Engine, 53(6), pp. 888–894 (in Japanese).

“A Robot Cooks Burgers at Startup Restaurant Creator.” (2018). Techcrunch. https://techcrunch.com/video/a- robot-cooks-burgers-at-startup-restaurant-creator/ (accessed September 2018).

Ayres, R., & S. Miller. (1981, November). “The Impacts of In- dustrial Robots.” Report CMU-RI-TR-81-7. Pittsburgh, PA: The Robotics Institute at Carnegie Mellon University.

Bereznak, A. (2015, January 7). “This Robot Can Com- fort Children Through Chemotherapy.” Yahoo Finance. https://finance.yahoo.com/news/this-robot-can- comfort-children-through-107365533404.html (ac- cessed September 2018).

“Berry Picking at Its Best with Sensor Technology.” (2018). Pepperl+Fuchs. https://www.pepperl-fuchs.com/usa/ en/27566.htm (accessed September 2018).

608 Part IV • Robotics, Social Networks, AI and IoT

Bogue, R. (2016). “Robots Poised to Revolutionise Agricul- ture.” Industrial Robot: An International Journal, 43(5), pp. 450–456

Broekens, J., M. Heerink, & H. Rosendal. (2009). “Assistive Social Robots in Elderly Care: A Review.” Gerontechnology, 8, pp. 94–103 doi: 10.4017/gt.2009.08.02.002.00.

Carlsson, B. (1998) “The Evolution of Manufacturing Technol- ogy and Its impact on Industrial Structure: An International Study.” IUI Working Paper 203. Internation Joseph A. Schum- peter Society Conference on Evolution of Technology and Market in an International Context. The Research Institute of Industrial Economics (IUI), Stockholm, May 24–28, 1988.

“Case Study Pepper, Courtyard Marriott.” SoftBank Robotics. https://www.softbankrobotics.com/us/solutions/ pepper-marriott (accessed September 2018).

Chirgwin, R. (2018, May 29). “Softbank’s ‘Pepper’ Robot Is a Security Joke.” The Register. https://www.theregister. co.uk/2018/05/29/softbank_pepper_robot_multi- ple_basic_security_flaws/ (accessed September 2018).

Coxworth, B. (2018, May 29). “Restaurant Keeps Its Prices Down – With a Robotic Kitchen.” New Atlas. https://newatlas.com/spyce-restaurant-robotic- kitchen/54818/ (accessed September 2018).

“Da Vinci Robotic Prostatectomy – A Modern Surgery Choice!” Robotic Oncology. https://www.roboticoncology.com/ da-vinci-robotic-prostatectomy/ (accessed September 2018).

Drummond, K. (2012, March 8). “Navy’s Newest Robot Is a Mechanized Firefighter.” wired.com. https://www. wired.com/2012/03/firefight-robot/ (accessed Sep- tember 2018).

Dupont, T. (2015, October 15). “The MAARS Military Robot.” Prezi. https://prezi.com/fsrlswo0qklp/the-maars-military- robot/ (accessed September 2018).

Engel, J. (2018, May 3). “Spyce, MIT-Born Robotic Kitchen Startup, Launches Restaurant: Video.” Xconomy. https:// www.xconomy.com/boston/2018/05/03/spyce-mit- born-robotic-kitchen-startup-launches-restaurant- video/ (accessed September 2018).

Fallon, S. (2015). “A Blue Robotic Bear to Make Sick Kids Feel Less Blue.” https://www.wired.com/2015/03/blue- robotic-bear-make-sick-kids-feel-less-blue/ (accessed August 2018).

Forrest, C. (2015).“Chinese Factory Replaces 90% of Humans with Robots, Production Soars.” TechRepublic. https:// www.techrepublic.com/article/chinese-factory- replaces-90-of-humans-with-robots-production- soars/ (accessed September 2018).

France, A. (2014, December 1). “Nestlé Employs Fleet of Robots to Sell Coffee Machines in Japan.” The Guardian. https:// www.theguardian.com/technology/2014/dec/01/ nestle-robots-coffee-machines-japan-george-clooney- pepper-android-softbank (accessed September 2018).

Gandhi, A. (2013, February 23). “Basics of Robotics.” Slideshare. https://www.slideshare.net/AmeyaGandhi/basics-of- robotics (accessed September 2018).

Goris, K., et al. (2010, September). “Mechanical Design of the Huggable Robot Probo.” Robotics & Multibody Mechanics Research Group. Brussels, Belgium: Vrije Universiteit Brussels.

Green, D. (2018). “Adidas Just Opened a Futuristic New Factory – and It Will Dramatically Change How Shoes Are Sold.” Business Insider. http://www.businessinsider.com/ adidas-high-tech-speedfactory-begins-production- 2018-4 (accessed September 2018).

Hernandez, D. (2018). “Seven Jobs Robots Will Create – or Expand.” The Wall Street Journal. https://www. wsj.com/articles/seven-jobs-robots-will-createor- expand-1525054021 (accessed September 2018).

History of Robots. (n.d.). Wikipedia. https://en.wikipedia. org/wiki/History_of_robots (accessed September 2018).

“Huggable Robot Befriends Girl in Hospital.” YouTube video. https://youtu.be/UaRCCA2rRR0 (accessed August 2018).

“Innovative Human-Robot Cooperation in BMW Group Production.” (2013, October 9). BMW Press Release. https://www.press.bmwgroup.com/global/article/ detail/T0209722EN/innovative-human-robot- cooperation-in-bmw-group-production?language=en (accessed September 2018).

Javelosa, J., & K. Houser. (2017). “Production Soars for Chinese Factory Who Replaced 90% of Employees with Robots.” Fu- ture Society. https://futurism.com/2-production-soars- for-chinese-factory-who-replaced-90-of-employees- with-robots/ (accessed September 2018).

Jeong, S., et al. (2015). “A Social Robot to Mitigate Stress, Anx- iety, and Pain in Hospital Pediatric Care.” Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts.

Jeong, S., et al. (2015). “Designing a Socially Assistive Robot for Pediatric Care.” Proceedings of the Fourteenth International Conference on Interaction Design and Children. ACM.

Jeong, S., & D. Logan. (2018, April 21–26).“Huggable: The Impact of Embodiment on Promoting Socio-emotional In- teractions for Young Pediatric Surgeons.” MIT Media Lab, Cambridge, MA, CHI 2018, Montréal, QC, Canada.

Jiji. (2017, November 21). “SoftBank Upgrades Humanoid Robot Pepper.” The Japan Times. https://www.japantimes. co.jp/news/2017/11/21/business/tech/softbank- upgrades-humanoid-robot-pepper/#.W6B3qPZFzIV (accessed September 2018).

Joshua, J. (2013, February 24). “The 3 Types of Robots.” Prezi. https://prezi.com/iifjw387ebum/the-3-types-of-ro- bots/ (accessed September 2018).

Kelly, M. (2018, July 16). “This Adorable Robot Wants to Make Air Travel Less Stressful.” The Verge. https://www.theverge. com/2018/7/16/17576334/klm-royal-dutch-airlines- robot-travel-airport (accessed September 2018).

Kelly, S. M. (2017, August 10). “A Robotic Crib Rocked My Baby to Sleep for Months.” CNN Tech. https://money. cnn.com/2017/08/10/technology/gadgets/snoo- review/index.html (accessed September 2018).

Chapter 10 • Robotics: Industrial and Consumer Applications 609

Lee, K. F. (2018). “The Human Promise of the AI Revolution.” The Wall Street Journal. https://www.wsj.com/articles/ the-human-promise-of-the-ai-revolution-1536935115 (accessed September 2018).

Mayank. (2012, June 18). “Basic Parts of a Robot.” max- Embedded.com. http://maxembedded.com/2012/06/ basic-parts-of-a-robot/ (accessed September 2018).

McHugh, R., & J. Rascon. (2015, May 23). “Meet MEDi, the Robot Taking Pain Out of Kids’ Hospital Visits.” NBC News. https:// www.nbcnews.com/news/us-news/meet-medi- robot-taking-pain-out-kids-hospital-visits-n363191 (accessed September 2018).

Meister, J. (2017), “The Future Of Work: How Artificial Intelligence Will Transform The Employee Experience,” https://www. forbes.com/sites/jeannemeister/2017/11/09/the- future-of-work-how-artificial-intelligence-will- transform-the-employee-experience/ (accessed No- vember 2018).

Modular Advanced Armed Robotic System. (n.d.). Wikipedia. https://en.wikipedia.org/wiki/Modular_Advanced_ Armed_Robotic_System (accessed September 2018).

Nagato, T., H. Shibuya, H. Okamoto, & T. Koezuka. (2017, July). “Machine Learning Technology Applied to Produc- tion Lines: Image Recognition System.” Fujitsu Scientific & Technical Journal, 53(4).

O’Kane, S. (2018, May 17). “Raden Is the Second Startup to Bite the Dust After Airlines Ban Some Smart Luggage.” Circuit Breaker. https://www.theverge.com/circuitbreaker/2018/5/17/ 17364922/raden-smart-luggage-airline-ban-bluesmart (accessed September 2018).

Park, H. W., et al. (2017). “Growing Growth Mindset with a Social Robot Peer.” Proceedings of the Twelfth ACM/IEEE International Conference on Human Robot Interaction.

Personal Robots Group. (2016). https://www.youtube. com/watch?v=sF0tRCqvyT0 (accessed March 5, 2017).

Personal Robots Group, MIT Media Lab. (2017). “Growing Growth Mindset with a Social Robot Peer.” Proceedings of the Twelfth ACM/IEEE International Conference on Human Robot Interaction.

Prasad, C. (2018, January 22). “Fabio, the Pepper Robot, Fired for ‘Incompetence’ at Edinburgh Store.” IBN Times. https://www.ibtimes.com/fabio-pepper-robot-fired- incompetence-edinburgh-store-2643653 (accessed September 2018).

Ro, L. (2016, October 18). “World’s First Smart Crib SNOO Will Help Put Babies to Sleep.” Curbed. https://www. curbed.com/2016/10/18/13322582/snoo-smart-crib- yves-behar-dr-harvey-karp-happiest-baby (accessed September 2018).

“Robotics Facts.” Idaho Public Television. http://idahoptv. org/sciencetrek/topics/robots/facts.cfm (accessed September 2018).

“Robots in Agriculture.” (2015, July 6). Intorobotics. https:// www.intorobotics.com/35-robots-in-agriculture/ (ac- cessed September 2018).

“Robotics: Types of Robots.” ElectronicsTeacher.com. http://www.electronicsteacher.com/robotics/type- of-robots.php (accessed September 2018).

Rosencrance, L. (2018 May 31). “Tabletop Grapes to Get Picked by Robots in India, with Help from Virginia Tech.” Robotics Business Review. https://www.robotic- sbusinessreview.com/agriculture/tabletop-grapes- picked-robots-india-virginia-tech/ (accessed Septem- ber 2018).

Schuster, W. M. (2018). “Artificial Intelligence and Patent Ownership.” Washington & Lee L. Rev., 75.

Shadbolt, P. (2015, February 15). “U.S. Navy Unveils Robotic Firefighter.” CNN. https://www.cnn.com/2015/02/12/ tech/mci-saffir-robot/index.html (accessed September 2018).

“Shipboard Autonomous Firefighting Robot – SAFFiR.” (2015, February 4). YouTube. https://www.youtube.com/ watch?time_continue=252&v=K4OtS534oYU (ac- cessed September 2018).

Simon, M. (2018, May 17). “The Wired Guide to Robots.” Wired. https://www.wired.com/story/wired-guide-to- robots/ (accessed September 2018).

“Tabletop Grapes to Get Picked by Robots in India.” Agtechnews.com. http://agtechnews.com/Ag-Robotics- Technology/Tabletop-Grapes-to-Get-Picked-by-Robots- in-India.html (accessed September 2018).

“The da Vinci® Surgical System.” (2015, September). Da Vinci Sur- gery. http://www.davincisurgery.com/da-vinci-surgery/ da-vinci-surgical-system/ (accessed September 2018).

“Types of Robots.” (2018). RoverRanch. https://prime. jsc.nasa.gov/ROV/types.html (accessed September 2018).

Westlund, J. K., J. M. Lee, J. Plummer, L. Faridia, F. Gray, J. Ber- lin, M. Quintus-Bosz, H. Harmann, R. Hess, M. Dyer, S. dos Santos, K. Adalgeirsson, S. Gordon, G. Spaulding, S. Marti- nez, M. Das, M. Archie, M. Jeong, & C. Breazeal, C. (2016). “Tega: A Social Robot.” Video Presentation. Proceedings of the Eleventh ACM/IEEE International Conference on Hu- man Robot Interaction.

White, T. (2015, February 4). “Making Sailors ‘SAFFiR’ – Navy Unveils Firefighting Robot Prototype at Naval Tech EXPO.” America’s Navy. https://www.navy.mil/ submit/display.asp?story_id=85459 (accessed Sep- tember 2018).

Zimberoff, L. (2018, June 21). “A Burger Joint Where Robots Make Your Food.” https://www.wsj.com/articles/ a-burger-joint-where-robots-make-your-food- 1529599213 (accessed September 2018).

610

11 C H A P T E R

I n this chapter, we present several topics related to group decision support and col-laboration. People work together, and groups (or teams) make many of the complex decisions in organizations. The increase in organizational decision-making complex- ity drives the need for meetings and group work. Supporting group work in which team members may be in different locations and working at different times emphasizes the important aspects of communications, computer-mediated collaboration, and workplace methodologies. Group support is a critical aspect of decision support systems (DSS). Effective computer-supported group support systems have evolved to increase gains and decrease losses in task performance and underlying processes. New tools and methodol- ogy are used to support teamwork. These include collective intelligence, crowdsourcing, and different types of AI. Finally, human–machine and machine–machine collaboration

■■ Understand the basic concepts and processes of group work, communication, and collaboration

■■ Describe how computer systems facilitate team communication and collaboration in an enterprise

■■ Explain the concepts and importance of the time/ place framework

■■ Explain the underlying principles and capabilities of groupware, such as group support systems (GSS)

■■ Understand how the Web enables collaborative computing and group support of virtual meetings

■■ Describe collective intelligence and its role in decision making

■■ Define crowdsourcing and explain how it supports decision making and problem solving

■■ Describe the role of AI in supporting collaboration, group work, and decision making

■■ Describe human–machine collaboration ■■ Explain how teams of robots work

LEARNING OBJECTIVES

Group Decision Making, Collaborative Systems, and AI Support

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 611

are increasing the power of collaboration and problem solving. All these are presented in the following sections:

11.1 Opening Vignette: Hendrick Motorsport Excels with Collaboration Teams 611

11.2 Making Decisions in Groups: Characteristics, Processes, Benefits, and Dysfunctions 613

11.3 Supporting Group Work and Team Collaboration with Computerized Systems 616

11.4 Electronic Support to Group Communication and Collaboration 619 11.5 Direct Computerized Support for Group Decision Making 623 11.6 Collective Intelligence and Collaborative Intelligence 629 11.7 Crowdsourcing as a Method for Decision Support 633 11.8 Artificial Intelligence and Swarm AI Support of Team Collaboration

and Group Decision Making 636

11.9 Human–Machine Collaboration and Teams of Robots 640

11.1 OPENING VIGNETTE: Hendrick Motorsports Excels with Collaborative Teams

Hendrick Motorsports (HMS) is a leading car racing company (with more than 500 employees) that competes in the Monster Energy NASCAR Cup Series. HMS’s major objective is to win as many races as possible each year. The company enters four race cars and their teams. HMS also builds its race cars. This includes building or rebuilding 550 car engines every year. In this kind of business, teamwork is critical because many different people with different skills and knowledge and several professional teams contribute to the success of the company.

THE OPERATIONS

HMS is engaged in car races all over the United States during the racing season (38 weeks a year). The company moves to a different racetrack every week. During the off-season time (14 weeks), the company analyzes the data obtained, and lessons learned during the latest racing seasons, and prepares for the following season. The company’s headquarters contains 19 buildings scattered over 100 acres.

THE PROBLEMS DURING THE RACING SEASON

The company needs to make quick decisions during races—some in real time, sometimes in a split second. Different team members need to participate, and they are in different locations. Communication and collaboration are critical.

Car racing is based on teamwork, drivers, engineers, planners, mechanics, and others who participate. Members must communicate and collaborate to make decisions.

The environment is too noisy to talk during a race. However, team members need to share data, graphs, and images, and chat quickly. Several decisions need to be made in real time that will help win races (e.g., how much fuel to add in the next few seconds to a car in the middle of the race). Team members must communicate and share data, including visual. It takes about 45–50 seconds for a car to complete a 2.5-mile lap at Daytona 500. During the race, top engineers need to communicate constantly with the fuelers. Last- minute data are common during the racing session.

612 Part IV • Robotics, Social Networks, AI and IoT

Any knowledge acquired in each lap can be used to improve the next one. In races, fueling decisions are critical. There are many other decisions to be made during the racing season. For example, after each race, the company needs to move a large crew with equipment and supplies from one location to the next (38 different venues). Moves need to be fast, efficient, and economical. Again, teamwork, as well as coordination, is needed.

OFF-SEASON PROBLEMS

There are 14 weeks to prepare for the next season. In addition, there is a considerable amount of data to analyze, simulate, discuss, and manipulate. For this, people need not only communication and collaboration tools but also analytics of different types.

THE SOLUTION

HMS decided to use Microsoft Teams, which is a chat-based platform, for team workspace in Microsoft Office 365. This platform is used as a communication hub for team members at the race tracks and at any other location in the organization.

Microsoft Teams stores data in different formats in its Teams workspace. Therefore, car crews, engineers, and mechanics can make split-second decisions that may help win races. This also enables computational analysis in a central place.

Microsoft Teams includes several subprograms and is easily connected to other soft- ware in Office 365. Office 365 provides several other tools that increase collaboration (e.g., SharePoint). For example, in the HSM solution, there is a working link to Excel as well as to SharePoint. Also, One Note of Teams is used to share meeting notes. Before Teams, the company used Slack (Section 11.4), but Slack did not provide enough security and functions.

Members need to share and discuss the massive amount of data accumulated during the racing season. Note that several employees have multiple skills and tasks. The solution included the creation of a collaboration hub for concurrent projects. Note that each different project may require different talents and data, depending on the project’s type. Also, the solution involves other information technology (IT) tools. For example, HMS uses Power BI dashboard to com- municate data visually. Some data can be processed as Excel-based spreadsheets.

Microsoft Teams is also available as a mobile app. Each team’s data file is available on the track at home and even under a car. So, the software package is able to respond to important situations right away.

The Results

The major results were improved productivity, smoother communication, easier collabora- tion, and reduction of the need for the time consumed in face-to-face meetings. People can chat online, seeing their partners without leaving their physical workplace. The company admits that without Teams, it would not have been able to accomplish its success. Today, Teams has everything the company needs at its fingertips.

u QUESTIONS FOR THE OPENING VIGNETTE

1. What were the major drivers for the use of Microsoft’s Teams? 2. List some discussions held during the racing season, and how they were supported

by the technology.

3. List decisions held during the off-season, and how they were supported by the technology.

4. Discuss why Microsoft Teams was selected, and explain how it supports teamwork group decision making.

5. Trace communication and collaboration within and between groups.

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 613

6. Specify the function of Microsoft Teams workspace. 7. Watch the video at youtube.com/watch?time_continue=108&v=xnFdM9IOaTE

and summarize its content.

WHAT WE CAN LEARN FROM THIS VIGNETTE

The first lesson is that many tasks today must be done by collaborating teams in order to succeed. Second, time is critical; therefore, companies must use technology to speed opera- tions and facilitate communication and collaboration in teamwork. Third, it is possible to use existing software for support, but it is better to use a major vendor that has additional products that can supplement the collaboration/communication software. Fourth, chat- ting can expedite communication, and visual technology support can be useful. Fifth, team members belong to diverse units and have diverse skills. The software brings them together. Team members should have clear goals and understand how to achieve them. Finally, collaboration can be both within and between groups.

Sources: Compiled from Ruiz-Hopper (2016) and Microsoft (2017).

11.2 MAKING DECISIONS IN GROUPS: CHARACTERISTICS, PROCESS, BENEFITS, AND DYSFUNCTIONS

Managers and other knowledge workers continuously make decisions, design products, develop policies and strategies, create software systems, and so on. Frequently they do it in groups. When people work in groups (i.e., teams), they perform group work or teamwork. Group work refers to work done by two or more people together. One aspect of group work is group decision making.

Group decision making refers to a situation in which people make decisions together. Let’s first look at the characteristics of group work.

Characteristics of Group Work

The following are some of the functions and characteristics of group work:

• Group members may be located in different places. • Group members may work at different times. • Group members may work for the same organization or different organizations. • A group can be permanent or temporary. • A group can be at one managerial level or span several levels. • A group can create synergy (leading to process and task gains) or result in conflict. • A group can generate productivity gains and/or losses. • A group’s task may have to be accomplished very quickly. • It may be impossible or too expensive for all team members to meet in one place

at the same time, especially when the meeting is called for emergency purposes. • Some of the groups’ needed data, information, or knowledge may be located in

several sources, some of which may be external to the organization. • The expertise of a group’s team members may be needed. • Groups perform many tasks; however, groups of managers and analysts frequently

concentrate on decision making or problem solving. • The decisions made by a group are easier to implement if supported by all (or at

least most) members. • Group work has many benefits and, unfortunately, some possible dysfunctions. • Group behaviors are influenced by several factors and may affect group decisions.

614 Part IV • Robotics, Social Networks, AI and IoT

Types of Decisions Made by Groups

Groups are usually involved in two major types of decision making:

1. Making a decision together. 2. Supporting activities or tasks related to the decision-making process. For example,

the group may select criteria for evaluating alternative solutions, prioritizing possible ones, and helping design strategy to implement them.

Group Decision-Making Process

The process of group decision making is similar to that of the general decision-making process described in Chapter 1 but it has more steps. Steps of the group decision-making process are illustrated in Figure 11.1.

Step 1. Prepare for meetings regarding the agenda, time, place, participants, and schedule. Step 2. Determine the topic of the meeting (e.g., problem definition). Step 3. Select participants for the meeting. Step 4. Select criteria for evaluating the alternatives and the selected solution. Step 5. Generate alternative ideas (brainstorm). Step 6. Organize the ideas generated into similar groups. Step 7. Evaluate the ideas, discuss, and brainstorm.

FIGURE 11.1 The Process of Group Decision Making.

Preparation, schedule, agenda

Select participants

Define the problem

Select evaluation criteria

Idea generation, alternative solution

Organize submitted ideas

Idea evaluation, discussion

Select or find idea or shortlist of ideas

Make a choice, recommendations

Plan implementation

Implement solutions

1

2

3

4

5

6

7

8

9

10

11

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 615

Step 8. Select a short list (finalists). Step 9. Select a recommended solution. Step 10. Plan implementation of the solution. Step 11. Implement the solution.

The process is shown as sequential, but as shown in Figure 11.1, some loops are possible. Also, if no solution is found, the process may start again.

GROUP DECISION FACTS When a group is going through the steps shown in Figure 11.1, the following is usually true:

• The decisions made need to be implemented. • Group members are typically of equal or nearly equal status. • The outcome of a meeting depends partly on the knowledge, opinions, and judg-

ments of its participants and the support they give to the outcome. • The outcome of a meeting depends on the composition of the group and on the

decision-making process it uses. • Group members settle differences of opinions either by the ranking person present

or through negotiations or arbitration. • The members of a group can be in one place, meeting face-to-face, or they can be

a virtual team, in which case they are in different places meeting electronically. They can also meet at different times.

Benefits and Limitations of Group Work

Some people endure meetings (the most common form of group work) as a necessity; oth- ers find meetings to be a waste of time. Many things can go wrong in a meeting. Participants may not clearly understand its purpose, may lack focus, or may have hidden agendas. Many participants may be afraid to speak up, or a few may dominate the discussions. Misunder- standings occur because of different interpretations of language, gestures, or expression. Technology Insight 11.1 provides a list of factors that can hinder the effectiveness of a manually managed meeting. Besides being challenging, teamwork is also expensive. A meeting of several top managers or executives can cost thousands of dollars.

Group work may have potential benefits (process gains) or drawbacks (process losses). Process gains are the benefits of working in groups. The unfortunate dysfunc- tions that may occur when people work in groups are called process losses. Examples of each are listed in Technology Insight 11.1.

TECHNOLOGY INSIGHT 11.1 Benefits and Dysfunctions of Working in Groups

The following are the possible major benefits and dysfunctions of group works.

Benefits of Working in Groups (Process Gains) Dysfunctions of Face-to-Face Group Process

(Process Losses)

• It provides learning. Groups are better than individuals at understanding problems. They can teach each other.

• Social pressures of conformity may result in groupthink (i.e., people begin to think alike and not tolerate new ideas; they yield to conformance pressure).

• People readily take ownership of problems and their solutions.

• It is a time-consuming, slow process. • Some relevant information could be missing.

• Group members have their egos embedded in the final decision, so they are committed it.

• A meeting can lack coordination, have a poor agenda, or be poorly planned.

616 Part IV • Robotics, Social Networks, AI and IoT

Benefits of Working in Groups (Process Gains) Dysfunctions of Face-to-Face Group Process

(Process Losses)

• Groups are better than individuals at catching errors.

• A meeting may be dominated by time, topic, opinion of one or a few individuals, or fear of contributing because of the possibility of conflicts.

• A group has more information and knowledge than any one member does. Members can combine their knowledge to create new knowledge. More and more creative alternatives for problem solving can be generated, and better solutions can be derived (e.g., through brainstorming).

• Some group members can tend to influence the agenda while some try to rely on others to do most of the work (free riding). The group may ignore good solutions, have poorly defined goals, or be composed of the wrong participants.

• A group may produce synergy during problem solving, therefore the effectiveness and/or quality of group work can be greater than the sum of what individual members produce.

• Some members may be afraid to speak up. • The group may be unable to reach consensus. • The group may lack focus.

• Working in a group may stimulate the creativity of the participants and the process.

• There can be a tendency to produce poor- quality compromises.

• Working together could allow a group to have better and more precise communication.

• There is often nonproductive time (e.g., socializing, preparing, waiting for latecomers).

• Risk propensity is balanced. Groups moderate high-risk takers and encourage conservatives.

• There can be a tendency to repeat what has already been said (because of failure to remember or process).

• Meeting costs can be high (e.g., travel, participation time spent).

• There can be incomplete or inappropriate use of information.

• There can be too much information (i.e., information overload).

• There can be incomplete or incorrect task analysis.

• There can be inappropriate or incomplete representation in the group.

• There can be attention or concentration blockage.

u SECTION 11.2 REVIEW QUESTIONS

1. Define group work. 2. List five characteristics of group work. 3. Describe the steps of group decision making. 4. List the major activities that occur in group work. 5. List and discuss five benefits of group work. 6. List and discuss five dysfunctions of group-made decisions.

11.3 SUPPORTING GROUP WORK AND TEAM COLLABORATION WITH COMPUTERIZED SYSTEMS

When people work in teams, especially when the members are in different locations and may work at different times, they need to communicate, collaborate, and access a diverse set of information sources in multiple formats. This makes meetings, especially virtual ones, complex with an increased chance for process losses. Therefore, it is important to follow certain processes and procedures for conducting meetings.

Group work may require different levels of coordination. Sometimes a group oper- ates at the individual work level with members making individual efforts that require

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 617

no coordination. As with a team of sprinters representing a country participating in a 100-meter dash, group productivity is simply the best of the individual results. At other times, group members may interact in coordination. At this level, as with a team in a relay race, the work requires careful coordination between otherwise independent individual efforts. Sometimes a team may operate at the concerted work level. As in a rowing race, teams working at this level must make a continuous concerted effort to be successful. Different mechanisms support group work at different levels of coordination.

Most organizations, small and large, use some computer-based communication and collaboration methods and tools to support people working in teams or groups. From e-mails to mobile phones and Short Message Service (SMS), as well as conferencing tech- nologies, such tools are an indispensable part of today’s work life. We next highlight some related technologies and applications.

Overview of Group Support Systems (GSS)

For groups to collaborate effectively, appropriate communication methods and technolo- gies are needed. We refer to these technologies as group support systems (GSS). The Internet and its derivatives (i.e., intranets, Internet of Things [IoT], and extranets) are the infrastructures on which much communication and collaboration occurs. The Web supports intra- and inter-organizational collaborative decision making.

Computers have been used for several decades to facilitate group work and decision making. Lately, collaborative tools have received more attention due to their increased capabilities and ability to save time and money (e.g., on travel cost) and to expedite deci- sion making. Computerized tools can be classified according to time and place categories.

Time/Place Framework

The tools used to support collaboration, groups, and the effectiveness of collaborative com- puting technology depend on the location of the group members and on the time that shared information is sent and received. DeSanctis and Gallupe (1987) proposed a framework for classifying IT communication support technologies. In this framework, communication is divided into four cells, which are shown with representative computerized support technolo- gies in Figure 11.2. The four cells are organized along two dimensions—time and place.

When information is sent and received almost simultaneously, the communication is in synchronous (real-time) mode. Telephones, instant messaging (IM), and face-to-face meet- ings are examples of synchronous communication. Asynchronous communication occurs when the receiver gets (or views) the information, such as an e-mail, at a different time than it was sent. The senders and the receivers can be in the same place or in different places.

As shown in Figure 11.2, time and place combinations can be viewed as a four-cell matrix, or framework. The four cells of the framework are as follows:

• Same time/same place. Participants meet face-to-face, as in a traditional meeting, or decisions are made in a specially equipped decision room. This is still an impor- tant way to meet even when Web-based support is used because it is sometimes critical for participants to leave their regular workplace to eliminate distractions.

• Same time/different place. Participants are in different places, but they com- municate at the same time (e.g., with videoconferencing or IM).

• Different time/same place. People work in shifts. One shift leaves information for the next shift.

• Different time/(any place) different place (any place). Participants are in different places, and they send and receive information at different times. This occurs when team members are traveling, have conflicting schedules, or work in different time zones.

618 Part IV • Robotics, Social Networks, AI and IoT

Groups and group work in organizations are proliferating. Consequently, groupware continues to evolve to support effective group work, mostly for communication and col- laboration (Section 11.4).

Group Collaboration for Decision Support

In addition to making decisions, groups also support decision-making subprocesses such as brainstorming. Collaboration technology is known to be the driving force for productivity increase and boosting people and organizational performance. Groups collaborate to make decisions in several ways. For example, groups provide assistance for the steps in Figure 11.1. Groups can help to identify problems, to assist in choosing criteria for selecting solutions, generating solutions (e.g., brainstorming), evaluating alternatives, and assisting in the selection of the best solution and implementing it. The group can be involved in one step or in several steps. In addition, it can collect the necessary data.

Many technologies can be used for collaboration; several of them are computerized and are described in several sections in this chapter.

Studies indicate that adopting collaboration technologies increases productiv- ity: for example, visual collaborative solutions increase employees’ satisfaction and productivity.

COMPUTERIZED TOOLS AND PLATFORMS We divide the computerized support into two parts. In Section 11.4, we present the major support of generic activities in com- munication and collaboration. Note that hundreds, maybe thousands, of commercial products are available to support communication and collaboration. We cover only a sample here.

FIGURE 11.2 The Time/Place Framework.

Same Time

• Instant Messaging • Chatting, decision room • Web-based GSS • Multimedia presentation system • Whiteboard • Document sharing • Workspace

• GSS in a decision room • Web-based GSS • Workflow management system • Document sharing • E-mail, V-mail • Videoconferencing playback

• Web-based GSS • Virtual whiteboard • Document sharing • Videoconferencing • Audio-conferencing • Computer conferencing • E-mail, V-mail • Virtual workspace

• Web-based GSS • Virtual whiteboard • Document sharing • E-mail, V-mail • Workflow management system • Computer conferencing with memory • videoconferencing playback • Voice memo

Different Time

Same Place

Different Place

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 619

Section 11.5 covers direct support of decision making, both to the entire process and to the major steps in the process. Note that some products, such as Microsoft Teams, which is cited in the opening vignette, support both generic activities and those in the decision-making process.

u SECTION 11.3 REVIEW QUESTIONS

1. Why do companies use computers to support group work? 2. What is GSS? 3. Describe the components of the time/place framework. 4. Describe the importance of collaboration for decision making.

11.4 ELECTRONIC SUPPORT FOR GROUP COMMUNICATION AND COLLABORATION

A large number of tools and methods are available to facilitate group work, e-collaboration, and communication. The following sections present only some tools that support the process. Our attention here is on indirect support to decision making. In Section 11.5, we cover direct support.

Groupware for Group Collaboration

Many computerized tools have been developed to provide group support. These tools are called groupware because their primary objective is to support group work indirectly as described in this section. Some e-mail programs, chat rooms, IM, and teleconferences provide indirect support.

Groupware provides a mechanism for team members to share opinions, data, infor- mation, knowledge, and other resources. Different computing technologies support group work in different ways depending on the task and size of the group, the security required, and other factors.

CATEGORIES OF GROUPWARE PRODUCTS AND FEATURES Many groupware products to enhance the collaboration of a small and large number of people are available on the Inter- net or intranets. A prime example is Microsoft’s Teams (opening vignette). The features of groupware products that support commutation, collaboration, and coordination are listed in Table 11.1. What follows are brief definitions of some of those features.

Synchronous versus Asynchronous Products

The products and features described in Table 11.1 may be synchronous or asynchronous. Web conferencing and IM, as well as voice-over IP (VoIP), are associated with the syn- chronous mode. Methods associated with asynchronous modes include e-mail and online workspaces where participants can collaborate while working at different times. Google Drive (drive.google.com) and Microsoft SharePoint (http://office.microsoft.com/en-us/ SharePoint/collaboration-software-SharePoint-FX103479517.aspx) allow users to set up online workspaces for storing, sharing, and working collaboratively on different types of documents. Similar products are Google Cloud Platform and Citrix Workspace Cloud.

Companies such as Dropbox.com provide an easy way to share documents. Similar systems, such as photo sharing (e.g., Instagram, WhatsApp, Facebook), are evolving for consumer home use.

620 Part IV • Robotics, Social Networks, AI and IoT

TABLE 11.1 Groupware Products and Features

General (Can Be Either Synchronous or Asynchronous)

• Built-in e-mail, messaging system • Browser interface • Joint Web page creation • Active hyperlink sharing • File sharing (graphics, video, audio, or other) • Built-in search functions (by topic or keyword) • Workflow tools • Corporate portals for communication, collaboration, and search • Shared screens • Electronic decision rooms • Peer-to-peer networks

Synchronous (Same Time)

• IM • Videoconferences, multimedia conferences • Audioconferences • Shared whiteboard, smart whiteboard • Instant videos • Brainstorming • Polling (voting) and other decision support (activities such as consensus building, scheduling)

• Chats with people • Chats with bots

Asynchronous (Different Times)

• Virtual workspaces • Tweets • Ability to receive/send e-mail, SMS • Ability to receive notification alerts via e-mail or SMS • Ability to collapse/expand discussion threads • Message sorting (by date, author, or read/unread) • Auto responders • Chat session logs • Electronic bulletin boards, discussion groups • Blogs and wikis

• Collaborative planning and/or design tools

Groupware products are either stand-alone, supporting one task (such as videoconfer- encing), or integrated, including several tools. In general, groupware technology products are fairly inexpensive and can easily be incorporated into existing information systems.

Virtual Meeting Systems

The advancement of Web-based systems opens the door for improved electronically sup- ported virtual meetings with the virtual team members in different locations, even in different countries. Online meetings and presentation tools are provided by tools such as webex, GoToMeeting.com, Skype.com, and many others. These systems feature Web

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 621

seminars (popularly called Webinars), screen sharing, audioconferencing, videoconferenc- ing, polling, question–answer sessions, and so on. Microsoft Office 365 includes a built-in virtual meeting capability. Even smartphones now have sufficient interaction capabilities to allow live meetings through applications such as FaceTime.

COLLABORATIVE WORKFLOW Collaborative workflow refers to software products that address project-oriented and collaborative processes. They are administered centrally yet are capable of being accessed and used by workers from different departments and from different physical locations. The goal of collaborative workflow tools is to empower knowl- edge workers. The focus of an enterprise solution for collaborative workflow is on allowing workers to communicate, negotiate, and collaborate within an integrated environment. Some leading vendors of collaborative workflow applications are FileNet and Action Tech- nologies. Collaborative workflow is related to but different than collaborative workspace.

DIGITAL COLLABORATIVE WORKSPACE: PHYSICAL AND VIRTUAL A collaborative work- space is where people can work together from any location at the same or at a different time. Originally, it was a physical conference room that teams used for conducting meet- ings. It was expanded to be a shared workspace, also known as “coworking space.” Some of these are in companies; others are offered for rent. Different computerized technologies are available to support group work in a physical structure. For 12 benefits of collaborative workspace, see Pena (2017).

A virtual collaboration workspace is an environment equipped with digital support by which group members who are in different locations can share information and col- laborate. A simple example is Google Drive, which enables sharing spreadsheets.

Collaborative workspace enables tech-savvy employees to access systems and tools from any device they need. People can work together in a secure way from anywhere. The digital workspace increases team productivity and innovation. It empowers employees and unlocks innovation. It allows workers to reach other people for collaborative work. For details and other collaboration technologies, see de Lares Norris (2018).

Example

PricewaterhouseCoopers (PwC) built an ideation war room in its Paris office as a large, immersive collaboration facility to support customer meetings.

MAJOR VENDORS OF VIRTUAL WORKSPACE Products by five major vendors follow:

• Google Cloud Platform is deployed on the “cloud,” so it is offered as a platform-as-a- service (PaaS). Google is also known for its Flexible Workspace product.

• Citrix Workspace Cloud is also deployed on the “cloud.” Citrix is known for its GoToMeeting collaboration tool. Citrix Workspace Cloud users can manage secure digital workplaces on Google Cloud.

• Microsoft Workspace is part of Office 365. • Cisco’s Webex, a popular collaboration package including Meeting. • Slack workspace is a very popular workspace.

ESSENTIALS OF SLACK Slack workspace is a digital space on which teammates share, communicate, and collaborate on work. It can be in one organization, or large organiza- tions may have multiple interconnected Slack spaces.

Each workspace includes several topical channels. These can be organized as pub- lic, private, or shared. The remaining components of Slack are messages, searches, and notifications. There are four groups of people involved with Slack: workspace owners,

622 Part IV • Robotics, Social Networks, AI and IoT

workspace administrators, members, and guests. For a Slack Guide, see get.slack.help/ hc/en-us/articles/115004071768-What-is-Slack-.

Slack has many key features and can deliver secure virtual apps to almost any device.

Collaborative Networks and Hubs

Traditionally, collaboration has taken place among supply chain members, frequently those that were close to each other (e.g., a manufacturer and its distributor or a distributor and a retailer). Even if more partners were involved, the focus was on the optimization of information and product flow between existing nodes in the traditional supply chain. Advanced methods, such as collaborative planning, forecasting, and replenishment (CPFR), do not change this basic structure.

Traditional collaboration results in a vertically integrated supply chain. However, Web technologies can fundamentally change the shape of the supply chain, the number of play- ers in it, and their individual roles. In a collaborative network, partners at any point in the network can interact with each other, bypassing what are traditional partners. Interaction may occur among several manufacturers or distributors as well as with new players, such as software agents that act as aggregators.

Collaborative Hubs

The purpose of a collaborative hub is to be a center point for group collaboration. Collaborative hub platforms need to enable participants’ interactions to unfold in

various forms online.

Example: Surface Hub for Business by Microsoft

This product connects individuals wherever they are and whenever they want to use a digital whiteboard and integrating software and apps. It helps to create a collaboration workplace where multiple devices are connected wirelessly to create a powerful work environment.

Social Collaboration

Social collaboration refers to collaboration conducted within and between socially ori- ented groups. It is a process of group interactions and information/knowledge sharing while attempting to attain common goals. Social collaboration is usually done on social media sites, and it is enabled by the Internet, IoT, and diversified social collaboration software. Social collaboration groups and schemes can take many different shapes. For images, conduct a Google search for “images of social collaboration.”

COLLABORATION IN SOCIAL NETWORKS Business-related collaboration is most evidenced on Facebook and LinkedIn. However, Instagram, Pinterest, and Twitter support collabora- tion as well.

• Facebook. Facebook’s Workspace facebook.com/workspace is used by hundreds of thousands of companies utilizing its features, such as “groups,” to support team members. For example, 80 percent of Starbucks store managers use this software.

• LinkedIn. LinkedIn provides several collaboration tools to its members. For exam- ple, LinkedIn Lookup provides several tools. Also, LinkedIn is a Microsoft company and it provides some integrated tools. The creation of subgroups of interest is a useful facilitator.

SOCIAL COLLABORATION SOFTWARE FOR TEAMS In addition to the generic collabora- tion software that can be used by two people and by teams, there are software platforms specifically for forming teams and supporting their activities. A few popular examples

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 623

according to collaboration-software.financesonline.com/c/social-collaboration- software/ are Wrike, Ryver, Azendoo, Zimbra social platform, Samepage, Zoho, Asana, Jive, Chatter, and Social Tables. For viewing the best social collaboration software by category, see technologyadvice.com/social-collaboration-software/.

Sample of Popular Collaboration Software

As noted earlier, there are hundreds or maybe thousands of communication and collabora- tion software products. Furthermore, their capabilities are ever changing. Given that our major interest is decision-making support, we provide only a small sample of these tools. We use the classification and example of Time Doctor, using the 2018 list (see Digneo, 2018).

• Communication tools: Yammer (social collaboration), Slack, Skype, Google Hangouts, GoToMeeting

• Design tools: InVision, Mural, Red Pen, Logo Maker • Documentation tools: Office Online, Google Docs, Zoho • File-sharing tools: Google Drive, Dropbox, Box • Project management tools: Asana, Podio, Trello, WorkflowMax, Kanban Tool, • Software tools: GitHub, Usersnap,Workflow tools: Integrity, BP Logix

OTHER TOOLS THAT SUPPORT COLLABORATION AND/OR COMMUNICATION

Notejoy (makes collaborative notes for team). Kahootz (brings stakeholders together to form communities of interest). Nowbridge (offers team connectivity, ability to see participants). Walkabout Workplace (is a 3D virtual office for remote teams). RealtimeBoard (is a enterprise visual collaboration). Quora (is a popular place for posting questions to the crowd). Pinterest (provides an e-commerce workspace that allows collection of text and images on selected topics). IBM connection closed (offers a comprehensive communication and collaboration tool set). Skedda (schedules space for coworking) Zinc (is a social collaboration tool) Scribblar (is an online collaboration room for virtual brainstorming) Collokia (is a machine learning platform for workflow) For additional tools, see Steward (2017).

u SECTION 11.4 REVIEW QUESTIONS

1. Define groupware. 2. List the major groupware tools and divide them into synchronous and asynchronous types. 3. Identify specific tools for Web conferencing and their capabilities. 4. Describe collaborative workflow. 5. What is collaborative workspace? What are its benefits? 6. Describe social collaboration.

11.5 DIRECT COMPUTERIZED SUPPORT FOR GROUP DECISION MAKING

Decisions are made frequently at meetings, some of which are called in order to make a one-time specific decision. For example, directors are elected at shareholder meetings, orga- nizations allocate budgets in meetings, cities decide which candidates to hire for their top

624 Part IV • Robotics, Social Networks, AI and IoT

positions, and the U.S. federal government meets periodically to set the short-term interest rate. Some of these decisions are complex; others can be controversial, as in resource alloca- tion by a city government. Process dysfunctions can be significantly large in such situations; therefore, computerized support has often been suggested to mitigate these controversies. These computer-based support systems have appeared in the literature under different names, including group decision support systems (GDSS), group support systems (GSS), computer- supported collaborative work (CSCW), and electronic meeting systems (EMS). These systems are the subject of this section. In addition to supporting entire processes, there are tools that support one or several activities in the group decision-making process (e.g., brainstorming).

Group Decision Support Systems (GDSS)

During the 1980s, researchers realized that computerized support to managerial decision making needed to be expanded to groups, because major organizational decisions are made by groups, such as executive committees and special task forces. The result was the creation of the group decision support systems methodology.

A group decision support system (GDSS) is an interactive computer-based sys- tem that facilitates the solution of semistructured or unstructured problems by a group of decision makers. The goals of GDSS are to improve the productivity of decision-making meetings by speeding up the decision-making process and/or to increase the quality of the resulting decisions.

MAJOR CHARACTERISTICS AND CAPABILITIES OF A GDSS GDSS characteristics follow:

• It supports the process of group decision makers mainly by providing automation of subprocesses (e.g., brainstorming) and using information technology tools.

• It is a specially designed information system, not merely a configuration of already existing system components. It can be designed to address one type of problem or make a variety of group-level organizational decisions.

• It encourages generation of ideas, resolution of conflicts, and freedom of expres- sion. It contains built-in mechanisms that discourage development of negative group behaviors, such as destructive conflict, miscommunication, and groupthink.

The first generation of GDSS was designed to support face-to-face meetings in a decision room. Today, support is provided mostly over the Web to virtual teams. A group can meet at the same time or at different times. GDSS is especially useful when controver- sial decisions have to be made (e.g., resource allocation, determining which individuals to lay off). GDSS applications require a facilitator for one physical place or a coordinator or leader for online virtual meetings.

GDSS can improve the decision-making process in various ways. For one, GDSS gen- erally provides structure to the meeting planning process, which keeps a group meeting on track, although some applications permit the group to use unstructured techniques and methods for idea generation. In addition, GDSS offers rapid and easy access to external and stored information needed for decision making. It also supports parallel processing of information and idea generation by participants and allows asynchronous computer discus- sion. GDSS makes possible larger group meetings that would otherwise be unmanageable; having a larger group means that more complete information, knowledge, and skills can be represented in the meeting. Finally, voting can be anonymous with instant results, and all information that passes through the system can be recorded for future analysis (producing organizational memory).

Over time, it became clear that supporting teams needed to be broader than GDSS has beed supported in a decision room. Furthermore, it became clear that what was really

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 625

needed was support for virtual teams, both in different place/same time and different place/different time situations. Also, it became clear that teams needed indirect support in most decision-making cases (e.g., help in searching for information or in collaboration) rather than direct support for the decision-making process. Although GDSS expanded to virtual team support, it was unable to meet all the other needs. In addition, the traditional GDSS was designed to deal with contradictory decisions when conflicts were likely to arise. Thus, a new generation of GDSS that supports collaboration work was needed. As we will see later, products such as Stormboard provide those needs.

Characteristics of GDSS

There are two options for deploying GDSS technology: (1) in a special-purpose decision room and (2) as Internet-based groupware with client programs running wherever the group members are.

DECISION ROOMS The earliest GDSS was installed in expensive, customized, special- purpose facilities called decision rooms (or electronic meeting rooms) that had PCs and a large public screen at the front of each room. The original idea was that only executives and high-level managers would use the expensive facility. The software in an electronic meeting room usually ran over a local area network (LAN), and these rooms were fairly plush in their furnishings. Electronic meeting rooms were structured in different shapes and sizes. A common design was a room equipped with 12 to 30 networked PCs, usually recessed into the desktop (for better participant viewing). A server PC was attached to a large screen projection system and connected to the network to display the work at indi- vidual workstations and aggregated information from the facilitator’s workstation. Breakout rooms equipped with PCs connected to the server, in which small subgroups could consult, were sometimes located adjacent to the decision room. The output from the subgroups was able to be displayed on the large public screen. A few companies offered such rooms for a daily rent. Only a few upgraded rooms are still available today, usually for high rent.

INTERNET-BASED GROUPWARE Since the late 1990s, the most common approach to GSS and GDSS delivery has been to use an Internet-based groupware that allows group mem- bers to work from any location at any time (e.g., WebEx, GoToMeeting, Adobe Connect, IBM Connections, Microsoft Teams). This groupware often includes audio conferencing and videoconferencing. The availability of relatively inexpensive groupware, as described in Section 11.4, combined with the power and low cost of computers and the availability of mobile devices, makes this type of system very attractive.

Supporting the Entire Decision-Making Process

The process that was illustrated in Figure 11.1 can be supported by a variety of software products. In this section, we provide an example of one product, Stormboard, that sup- ports several aspects of that process.

Example: Stormboard

Stormboard stormboard.com provides support for different brainstorming and group decision-making configurations. The following is the product’s sequence of activities:

1. Define the problem and the users’ objectives (what they are hoping to achieve). 2. Brainstorm ideas (to be discussed later). 3. Organize the ideas in groups of similar flavor, look for patterns, and select only

viable ideas.

626 Part IV • Robotics, Social Networks, AI and IoT

4. Collaborate, refine concepts, and evaluate (using criteria) the meeting’s objectives. 5. The software enables users to prioritize proposed ideas by focusing on the selec-

tion criteria. It lets all participants express their thinking and directs the team to be cohesive.

6. It presents a short list of superior ideas. 7. The software suggests the best idea and recommends implementation. 8. It plans the project implementation. 9. It manages the project.

10. It periodically reviews progress.

For a video, see youtube.com/watch?v=0buRzu4rhJs.

COMPREHENSIVE GROUPWARE TOOLS INCLUDING THINKTHANK Although many capabili- ties that enable group decision support are embedded in common software tools for office productivity such as Microsoft Office 365, it is instructive to learn about specific software that illustrates some of groupware’s unique capabilities. MeetingRoom was one of the first comprehensive, same time/same place electronic meeting packages. Its follow-up product, GroupSystems OnLine, offered similar capabilities, and it ran in asynchronous mode (any- time/anyplace) over the Web (MeetingRoom ran only over a LAN). GroupSystems’ latest product is ThinkTank, a suite of tools that facilitate the various group decision-making activities. For example, it shortens cycle time for brainstorming. ThinkTank improves the collaboration of face-to-face or virtual teams through customizable processes toward the groups’ goals faster and more effectively than in previous product generations. ThinkTank offers the following:

• It can provide efficient participation, workflow, prioritization, and decision analysis. • Its anonymous brainstorming for ideas and comment generation is an ideal way to

capture the participants’ creativity and experience. • The product’s enhanced Web 2.0 user interface ensures that participants do not

need special training to join, so they can focus 100 percent on solving problems and making decisions.

• With ThinkTank, all of the knowledge shared by participants is captured and saved in documents and spreadsheets, automatically converted to the meeting minutes, and made available to all participants at the end of the session.

Examples: ThinkTank Use (thinktank.net/case-study)

The following are two examples of ThinkTank’s use.

• It enables transformational collaboration between supply chain partners. Their meet- ing was supported by collective intelligence tools and procedures. Partners agreed on how to cut costs, speed processes, and improve efficiencies. In the past, there had been no progress on these issues.

• The University of Nebraska and the American College of Cardiology collaborated using ThinkTank tools and procedures to rethink how electronic health records could be reorganized to help medical consultants save time. Patients’ appointment times were shortened by 5 to 8 minutes. Other improvements also were achieved. Both patient care and large monetary savings were achieved.

OTHER DECISION-MAKING SUPPORT The following is a list of other types of support pro- vided by intelligent systems:

• Using knowledge systems and a product called Expert Choice Software for dealing with multiple-criteria group decision making.

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 627

• A mediating group decision-making method for infrastructure asset management was proposed by Yoon et al. (2017).

• For a group decision-making support system in logistics and supply chain manage- ment, see Yazdani et al. (2017).

Brainstorming for Idea Generation and Problem Solving

A major activity in group decision making is idea generation. Brainstorming is a process for generating creative ideas. It involves freewheeling group discussions and spontaneous contribution of ideas for solving problems and making strategy and resource allocation. Contributors’ ideas are discussed by the members. An attempt is made to generate as many ideas as possible, no matter how bizarre they look. Generated ideas are discussed and evaluated by the group. There is evidence that groups not only generate more ideas but also better ones (McMahon et al., 2016). Manually managed brainstorming has some of the limitations of group work described in Section 11.2. Therefore, computer support is frequently recommended.

COMPUTER-SUPPORTED BRAINSTORMING Computer programs can support the various brainstorming activities. The support is usually for online brainstorming, synchronously or asynchronously. Hopefully, electronic brainstorming eliminates many of the process dysfunctions cited in Section 11.2 and helps in the generation of many new ideas. Brain- storming software can stand alone or be a part of a general group support package. The major features of software packages follow:

• Creation of a large number of ideas. • Large group participation. • Real-time updates. • Information color coding. • Collaborative editing. • Design of brainstorming sessions. • Idea sharing. • People participation. • Idea mapping (e.g., create mind maps). • Text, video, documents, etc. posting. • Remote brainstorming. • Creation of an electronic archive. • Reduction of social loafing.

The major limitations of electronic software support are increased cognitive load, fear of using new technology, and need for technical assistance.

COMPANIES THAT PROVIDE ONLINE BRAINSTORMING SERVICES AND SUPPORT FOR GROUP WORK Some companies and the services and support they provide follow:

• eZ Talks Meetings. Cloud-based tool for brainstorming and idea sharing. • Bubbl.us. Visual thinking machine that provides a graphical representation of

ideas and concepts, helps in idea generation, and shows where ideas and thoughts overlap (visually, in colors).

• Mindomo. Tool for real-time collaboration that offers integrated chat capability. • Mural. Tool that enables collecting and sorting of ideas in rich media files. It is

designed as a Pinboard that invites participants. • iMindQ. Cloud-based service that enables creating mind maps and basic diagrams.

628 Part IV • Robotics, Social Networks, AI and IoT

For an evaluation of 28 online brainstorming tools, see blog.lucidmeetings.com/ blog/25-tools-for-online-brainstorming-and-decision-making-in-meetings/.

ARTIFICIAL INTELLIGENCE SUPPORTS BRAINSTORMING In Chapter 12, we will introduce the use of bots. Some software allows users to create and post a bot (or avatar) that rep- resents people in order to communicate anonymously. Artificial intelligence (AI) can also be used for pattern recognition and identifying ideas that are similar to each other. AI is also used in crowdsourcing (Section 11.7), which is used extensively for idea generation and voting.

Group Support Systems

A group support system (GSS), which was discussed earlier, is any combination of hardware and software that enhances group work. GSS is a generic term that includes all forms of communication and collaborative computing. It evolved after information technology researchers recognized that technology could be developed to support many activities that normally occur at face-to-face meetings when they occur in virtual meetings (e.g., idea generation, consensus building, anonymous ranking). Also, a focus was made on collaboration rather than on minimizing conflicts.

A complete GSS is considered a specially designed information system software, but today, its special capabilities have been embedded in standard IT productivity tools. For example, Microsoft Office 365 includes Microsoft Teams (opening vignette). It also includes the tools for Web conferences. Also, many commercial products have been developed to support only one or two aspects of teamwork (e.g., videoconferencing, idea generation, screen sharing, wikis).

HOW GSS IMPROVES GROUP WORK The goal of GSS is to provide support to participants in improving the productivity and effectiveness of meetings by streamlining and speed- ing up the decision-making process and/or by improving the quality of the results. GSS attempts to increase process and task gains and decrease process and task losses. Overall, GSS has been successful in doing just that. Improvement is achieved by providing support to group members for the generation and exchange of ideas, opinions, and preferences. Specific features such as the ability of participants in a group to work simultaneously on a task (e.g., idea generation or voting) and anonymity produce improvements. The following are some specific GSS support activities:

• Supporting parallel processing of information and idea generation (brainstorming). • Enabling the participation of larger groups with more complete information, knowl-

edge, and skills. • Permitting the group to use structured or unstructured techniques and methods. • Offering rapid, easy access to external information. • Allowing parallel computer discussions. • Helping participants frame the big picture. • Providing anonymity, which allows shy people to contribute to the meeting (i.e.,

to get up and do what needs to be done). • Providing measures that help prevent aggressive individuals from controlling a

meeting. • Providing multiple ways to participate in instant anonymous voting. • Providing structure for the planning process to keep the group on track. • Enabling several users to interact simultaneously (i.e., conferencing). • Recording all information presented at a meeting (i.e., providing organizational

memory).

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 629

For GSS success stories, look for sample cases at vendors’ Web sites. As you will see in many of these cases, collaborative computing led to dramatic process improvements and cost savings.

Note that only some of these capabilities are provided in a single package from one vendor.

u SECTION 11.5 REVIEW QUESTIONS

1. Define GDSS and list the limitations of the initial GSS software. 2. List the benefits of GDSS. 3. List process gains made by GDSS. 4. Define decision room. 5. Describe Web-based GSS. 6. Describe how GDSS supports brainstorming and idea generation.

11.6 COLLECTIVE INTELLIGENCE AND COLLABORATIVE INTELLIGENCE

Groups or teams are created for several purposes. Our book concentrates on support for decision making. This section deals with the collective intelligence and collaborative intel- ligence of groups.

Definitions and Benefits

Collective intelligence (CI) refers to the total intelligence of a group. It is also refers to as the wisdom of the crowd. People in a group are using their skills and knowledge for solving problems and providing new insights and ideas. The major benefits are the ability to solve com- plex problems and/or design new products and services that result from innovations. A major research center on collective intelligence (CI) is the MIT Center for Collective Intelligence (CCI) (cci.mit.edu). A major study aspect of CCI is how people and computers can work together so that teams can be more innovative than any individual, group, or computer can be alone. CI appears in several disciplines ranging from sociology to political science. Our interest here is in CI as it relates to computerized decision making. We cover CI here and in Section 11.7 where we present the topic of crowdsourcing. In Section 11.8, we present swarm intelligence, which is also an application of CI. For the benefits of CI, see 50Minutes.com (2017).

TYPES OF COLLECTIVE INTELLIGENCE One way to categorize CI is to divide it into three major areas of applications: cognition, cooperation, and coordination. Each of these can be further divided. For an overview, see collective intelligence on Wikipedia. Our inter- est is in applications by which the group synergy helps in problem solving and decision making. People contribute their experience and knowledge, and the group interactions and the computerized support help in making better decisions.

Thomas W. Malone, the founder and director of CCI at MIT, considers CI as a broad umbrella. He views collective intelligence as “groups of individuals act- ing collectively in ways that seem intelligent.” The CCI work, known as the Edge, is available at the Edge video (31:45 minutes) available at edge.org/conversation/ thomas_w__malone-collective-intelligence.

Computerized Support to Collective Intelligence

Collective intelligence can be supported by many of the tools and platforms described in Sections 11.4 and 11.5. In addition, the Internet, intranet, and the IoT (Chapter 13) play a major role in facilitating CI by enabling people to share knowledge and ideas.

630 Part IV • Robotics, Social Networks, AI and IoT

Example 1: The Carnegie University Foundation Supports Network Collaboration

The Carnegie Foundation was looking for ways to have people work together collab- oratively in order to accelerate improvements and to share data and learning across its networks of people. The solution is an online workspace called the Carnegie Hub, which serves as an access point to resources and enables engagement in group work and collaboration.

The Hub uses several software products, some of which were described in Section 11.4, such as Google Drive, creating a collaborative workspace. The major aspects of the Carnegie Collection Intelligence project follow:

1. Content is shared in one place (the “cloud”) for everyone to view, edit, or contribute even at the same time.

2. All data and knowledge are stored in one location on the Web. Discovery is easy. 3. Asynchronous conversations using discussion boards are easy; all notes are publicly

displayed, documented, and stored. 4. These aspects facilitate social collaboration, commitment to problem solving, and

peer support. The Carnegie University faculty is now a community of practice, using collective intelligence to plan, create, and solve problems together. For details, see Thorn and Huang (2014).

Example 2: How Governments Tap IoT for Collective Intelligence

According to Bridgwater (2018), governments are using IoT to support decision making and policy creation. Governments are trying to collect information and knowledge from people and increasingly do so via IoT. Bridgwater cites the government of the United Arab Emirates that uses IoT to enhance public decision making. The IoT systems collect ideas and aspirations of the citizens. The collective intelligence platform allows the targeting of narrowly defined groups. Real estate plans are subjected to the opinion of residents in the vicinity of proposed developments. The country’s project of smart cities is combined with CI (Chapter 13). In addition to IoT, there are activities in CI and networks as shown in Application Case 11.1.

Introduction

Water management is one of the most important chal- lenges for many communities. In general, the demand for water is growing while the supply could shrink (e.g., due to pollution). Managing water requires the involvement of numerous stakeholders ranging from consumers and suppliers to local governments and sanitation experts. The stakeholders must work together. The objective is to have responsible water use and water preservation. The accounting office of PwC published report 150CO47, “Collaboration: Preserving

Water Through Partnership That Works” available at pwc.com/hu/hu/kiadvanyok/assets/pdf/pwc_ water_collaboration.pdf. It describes the problem and its benefits and risks. The report shares the differ- ent stakeholders’ perspectives, identifies the success factors of collaboration, and weighs the trade-offs for evaluating alternative solutions for the water manage- ment issue. An interesting framework for a solution is the collaborative modeling developed at Oregon State University in collaboration with Indiana University- Purdue University.

Application Case 11.1 Collaborative Modeling for Optimal Water Management: The Oregon State University Project

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 631

The Challenge

Planning and managing water conservation activities are not simple tasks. The idea is to develop a user- friendly tool that will enable all stakeholders to par- ticipate in these activities. It is necessary to involve the stakeholder communities in using scientifically developed guidelines for designing water conserva- tion practices. Here are some of the requirements of the desired tool:

• The tool needs to be interactive and human guided and operated.

• It needs to be Web-based and user friendly. • Both individuals and groups should be able to

use it. • It should enable users to view and evaluate

solution designs based on both quantitative and qualitative criteria.

The Solution: WRESTORE

Watershed Restoration Using Spatio-Temporal Optimi- zation (WRESTORE) is a Web-based tool that meets the preceding requirements. It is based on AI and ana- lytical optimization algorithms. The algorithms process dynamic simulation models and allow users to spatially optimize the location of new water conservations. In addition to using the dynamic simulation models, users are able to include their own personal subjective views and qualitative criteria. WRESTORE generates alterna- tive practices that users can discuss and evaluate.

Incorporation of human preferences to com- puter solutions makes the solutions more accept- able. The AI part of the project includes machine learning and crowdsourcing (Section 11.7) to solicit

information from the crowd. The reason for the par- ticipative collaboration is that water is an essential resource and should not be only centrally controlled. The AI technologies “democratize” water manage- ment while harnessing the power of people and com- puters to solve difficult water management problems.

The machine-learning algorithms learn from what people are doing. Human feedback helps AI to iden- tify best solutions and strategies. Thus, humans and machines are combined to solve problems together.

The Results

WRESTORE developers are experimenting with the technology in several places and so far have achieved full collaboration from participating stakeholders. Initial results indicate the creation by WRESTORE of innova- tive ideas for developing water resources and distribu- tion methods that save significant amounts of water.

Questions for Case 11.1 1. Crowdsourcing is used to find information from a

crowd. Why is it needed in this case? (see Section 11.7 if you are not familiar with crowdsourcing).

2. How does WRESTORE act as a CI tool?

3. Debate centralized control versus participative col- laboration. Cite the pros and cons of each.

4. Why it is difficult to manage water resources?

5. How can an optimization/simulation/AI model support group work in this case?

Sources: Compiled from Basco-Carrera et al. (2017), KTVZ.com (Channel 21, Oregon, March 21, 2018), and Babbar-Sebens et al. (2015).

How Collective Intelligence May Change Work and Life

For several decades, researchers studied the relationship of CI and work. For example, Doug Engebert, a pioneer in CI, describes how people work together in response to a shared challenge and how they can leverage their collective memory, perception, planning, reasoning, and so on into powerful knowledge. Since Engebert’s pioneering work, the impact of technology is increasing organizations’ CI and building collaborative communi- ties of knowledge. In summary, CI attempts to augment human intelligence to solve busi- ness and social problems. This basically means that CI allows more people to have more engagement and involvement in organizational decision making. At MIT’s CCI, research is done on how people and computers can work together to improve work (see also Sec- tion 11.9). MIT’s CCI focuses on the role of networks, including the Internet, intranets, and IoT. Researchers there found that organizations’ structures tend to be flatter, and more

632 Part IV • Robotics, Social Networks, AI and IoT

decisions are delegated to teams. All this results in decentralized workplaces. For further discussion on MIT’s CCI, see MIT’s blog of April 3, 2016, at executive.mit.edu/blog/will- collective-intelligence-change-the-way-we-work/. For a comprehensive view on how CI can change the entire world, see Mulgan (2017).

A major thrust in CI is the collaboration efforts within a group, as described next.

Collaborative Intelligence

Placing people in groups and expecting them to collaborate with the help of technology may be wishful thinking. Management and behavioral researchers study the issue of how to make people collaborate in groups.

Called by some collaborative intelligence, Coleman (2011) stipulates that group col- laboration has the following 10 components: (1) willingness to share, (2) knowing how to share, (3) being willing to collaborate, (4) knowing what to share, (5) knowing how to build trust, (6) understanding team dynamics, (7) using correct hubs for networking, (8) mentoring and coaching properly, (9) being open to new ideas, and (10) using com- puterized tools and technology. A similar list is provided at thebalancecareers.com/ collaboration-skills-with-examples-2059686.

Computerized tools and technologies are critical enablers of communication, col- laboration, and people’s understanding of each other.

How to Create Business Value from Collaboration: The IBM Study

Groups and team members provide ideas and insights. To excel, organizations must utilize people’s knowledge, some of which is created by collective intelligence. One way to do this is provided by a study of collective intelligence conducted by the IBM Institute for Business Value. The study is available (free) at www-935.ibm.com/services/us/gbs/ thoughtleadership/ibv-collective-intelligence.html. There is also a free executive sum- mary. The study presents three major points:

1. CI can enhance organizational outcomes by correctly tapping the knowledge and experience of working groups (including customers, partners, and employees).

2. It is crucial to target and motivate the appropriate participants. 3. CI needs to address the issue of participants’ resistance to change. All in all, IBM

concludes that “Collective intelligence is a powerful resource for creating value using the experience and insights of vast numbers of people around the world.”

Access the untapped knowledge of your networks, IBM. (www-935.ibm.com/ services/us/gbs/thoughtleadership/ibv-collective-intelligence.html)

An offshoot of CI is crowdsourcing, the topic of the next section (11.7).

u SECTION 11.6 REVIEW QUESTIONS

1. What is collective intelligence (CI)? 2. List the major benefits of CI. 3. How is CI supported by computers? 4. How can CI change work and life? 5. How can CI impact organization structure and decision making? 6. The Carnegie case described how standard collaboration tools create a collective intel-

ligence infrastructure. The WRESTORE case described a modeling analytical framework that enables stakeholders to collaborate. What are the similarities and differences between the two cases?

7. Describe collaborative intelligence. 8. How do you create business value from collective intelligence?

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 633

11.7 CROWDSOURCING AS A METHOD FOR DECISION SUPPORT

Crowdsourcing refers to outsourcing tasks to a large group of people (crowd). One of the major reasons for doing so is the potential for the wisdom of a crowd to improve decision making and assist in solving difficult problems; see Power (2014). Therefore, crowdsourc- ing can be viewed as a method of collective intelligence. This section is divided into three parts: The essentials of crowdsourcing, crowdsourcing as a decision support mechanism, and implementing crowdsourcing for problem solving.

The Essentials of Crowdsourcing

Crowdsourcing has several definitions because it is used for several purposes in a number of fields. For a tutorial on crowdsourcing and examples, view the video (14:51 min.) at youtube.com/watch?v=lXhydxSSNOY. Crowdsourcing means that an organization is outsourcing or farming out work for several reasons: Necessary skills may not be available internally, speed of execution is needed, problems are too complex to solve, or special innovation is needed.

SOME EXAMPLES

• Since 2005, Doritos Inc. has run a “Crash the Super Bowl” contest for creating a 30-second video for the Super Bowl. The company has given $7 million in prizes in the last 10 years for commercials composed by the public.

• Airbnb is using user-submitted videos (15 seconds each) that describe travel sites. • Dell’s Idea Storm (ideastorm.com) enables customers to vote on features of Idea

Storm the customers prefer, including new ones. Dell is using a technically oriented crowd, such as the Linux (linux.org) community. The crowd submits ideas and sometimes members of the community vote on them.

• Procter & Gamble’s researchers post their problems at innocentive.com and ninesigma.com, offering cash rewards to problem solvers. It uses other crowdsourc- ing service providers such as yourencore.com.

• The LEGO company has a platform called LEGO Ideas through which users can submit ideas for new LEGO sets and vote on submitted ideas by the crowd. Accepted ideas generate royalties to those who proposed them if the ideas are commercialized.

• PepsiCo solicits ideas regarding new potato chip flavors for the company’s Lay’s brand. Over the years, the company has received over 14 million suggestions. The estimated contribution to sales increase is 8 percent.

• Cities in Canada are creating real-time electronic city maps to inform cyclists about high-risk areas to make the streets safer. Users can mark the maps when they expe- rience a collision, bike theft, road hazard, and so on. For details, see Keith (2018).

• U.S. intelligence agencies have been using ordinary people (crowds) to predict world events ranging from the results of elections to the direction of prices.

• Hershey crowdsourced potential solutions of how to ship chocolate in warm climates. For how this was done, see Dignan (2016). The winning prize was $25,000.

These examples illustrate some of the benefits of crowdsourcing, such as wide exposure to expertise, increased performance and speed, and improved problem-solving and innova- tion capabilities. These examples also illustrate the variety of applications.

MAJOR TYPES OF CROWDSOURCING Howe (2008), a crowdsourcing pioneer, divided the crowdsourcing applications into the following types (or models):

1. Collective intelligence (or wisdom). People in crowds are solving problems and providing new insights and ideas leading to product, process, or service innovations.

2. Crowd creation. People are creating various types of content and sharing it with others (for pay or free). The created content may be used for problem

634 Part IV • Robotics, Social Networks, AI and IoT

Another way to classify crowdsourcing is by the type of work it does. Some examples with a crowdsourcing vendor for each follow:

• Logo design—Design Bill • Problem solving—InnoCentive, NineSigma, IdeaConnection • Business innovation—Chardix • Brand names—Name This • Product and manufacturing design—Pronto ERP • Data cleansing—Amazon Mechanical Turk • Software testing—uTest • Trend watching—TrendWatching • Images—Flickr Creative Commons

For a compressive list of crowdsourcing, collective intelligence, and related compa- nies, see boardofinnovation.com.

THE PROCESS OF CROWDSOURCING The process of crowdsourcing differs from applica- tion to application, depending on the nature of the specific problem to be solved and the method used. However, the following steps exist in most enterprise crowdsourcing applications, even though the details of the execution may differ. The process is illustrated in Figure 11.3.

1. Identify the problem and the task(s) to be outsourced. 2. Select the target crowd (if not an open call). 3. Broadcast the task to the crowd (or to an unidentified crowd in an open call). 4. Engage the crowd in accomplishing the task (e.g., idea generation, problem

solving). 5. Collect user-generated content. 6. Have the quality of submitted material evaluated by the management that initiated

the request, by experts, or by a crowd. 7. Select the best solution (or a short list). 8. Compensate the crowd (e.g., the winning proposal). 9. Implement the solution.

Note that we show the process as sequential, but there could be loops returning to previ- ous steps.

Crowdsourcing for Problem-Solving and Decision Support

Although there are many potential activities in crowdsourcing, major ones are support- ing the managerial decision-making process and/or providing a solution to a problem. A complicated problem that is difficult for one decision maker or a small group to solve may be solved by a crowd, which can generate a large number of ideas for solving a

solving, advertising, or knowledge accumulation. Content creation can also be done by splitting large tasks into small segments (e.g., contributing content to create Wikipedia).

3. Crowd voting. People are giving their opinions and ratings on ideas, products, or services, as well as evaluating and filtering information presented to them. An example is voting in American Idol competitions.

4. Crowd support and funding. People are contributing and supporting endeavors for social or business causes, such as offering donations, and micro-financing new ventures.

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 635

problem. However, inappropriate use of crowdsourcing could generate negative results (e.g., see Grant, 2015). On how to avoid the potential pitfalls of crowdsourcing, see Bhandari et al., 2018.

THE ROLE OF CROWDSOURCING IN DECISION MAKING Crowds can provide ideas in a col- laborative or a competitive mode. However, the crowd’s role may differ at different stages of the decision-making process. We may use a crowd to decide how to respond to a com- petitor’s act or to help us decide whether a proposed design is useful. Chiu et al. (2014) adopted Herbert Simon’s decision-making process model to outline the potential roles of a crowd. Simon’s model includes three major phases before implementation: intelligence (information gathering and sharing for the purpose of problem solving or opportunity exploitation, problem identification, and determination of the problem’s importance), design (generating ideas and alternative solutions), and choice (evaluating the generated alternatives and then recommending or selecting the best course of action). Crowdsourc- ing can provide different types of support to this managerial decision-making process. Most of the applications are in the design phase (e.g., idea generation and co-creation) and in the choice phase (voting). In some cases, support can be provided in all phases of the process.

Implementing Crowdsourcing for Problem Solving

While using an open call to the public can be done fairly easily by the problem owner, people who need to solve difficult problems usually like to reach experts for solving problems (solvers). For a company to obtain assistance in finding such experts, especially externally, it can use a third-party vendor. Such vendors have hundreds of thousands or even millions of preregistered solvers. Then, the vendor can do the job as illustrated in Application Case 11.2.

FIGURE 11.3 The Crowdsourcing Process.

Problem owner

Problem

Preparation, specific task(s) to outsource

Crowd membersCrowd Selection

Ideas, solutions submitted

Broadcasting task

Crowd perform work

Idea evaluation

Recommended solution

Activities

Components

636 Part IV • Robotics, Social Networks, AI and IoT

GlaxoSmithKline (GSK) is a UK-based global phar- maceutical/healthcare company, with over 100,000 employees. The company strives on innovations. However, despite its mega size and global presence, it has problems that it needs outside expertise to solve.

The Problem

The company researched a potentially disruptive technology that promised cure to difficult diseases. The company wanted to discover which disease to use as a test bed for the potential innovative treatments. It was necessary to make sure that the selection will cover a disease where every aspect of the new treatment is checked. Despite its large size, GSK wanted some outside expertise to sup- port and check the in-house research efforts.

The Solution

GSK decided to crowdsource the problem solution to experts, using InnoCentive Corp. (Innocentive. com). InnoCentive is a US-based global crowdsourc- ing company. The company receives challenges from client companies like GSK. These challenges are posted for solvers to see with the potential rewards, in InnoCentive’s Challenge Center. Solvers that think they want to participate follow instructions and may

sign an agreement. The solutions submitted are eval- uated, and awards are provided to the winners.

The GSK Situation

In total, 397 solvers engaged in this challenge, even the reward was minimal ($5000). The solvers resided in several countries. The solvers submitted 66 pro- posed solutions. The entire process lasted 75 days.

The Results

The winning solution proposed a new area that was not considered by GSK teams. The proposer was a Bulgarian who based his idea on a Mexican publi- cation. Several other winning proposals contributed useful ideas. Also, the process enabled collaboration between the GSK team and the winning researchers.

Questions for Case 11.2 1. Why did GSK decide to crowdsource?

2. Why did the company use InnoCentive?

3. Comment on the global nature of the case.

4. What lessons did you learn from this case?

5. Why do you think a small $5000 reward is sufficient?

Sources: Compiled from InnoCentive Inc. Case Study GlaxoSmithKline. Waltham, MA., GSK Corporate Information (gsk. com) and InnoCentive.com/our-solvers/.

Application Case 11.2 How InnoCentive Helped GSK Solve a Difficult Problem

CROWDSOURCING FOR MARKETING More than 1 million customers are registered at Crowd Tap, the company that provides a platform named Suzy that enables marketers to conduct crowdsourcing studies.

u SECTION 11.7 REVIEW QUESTIONS

1. Define crowdsourcing. 2. Describe the crowdsourcing process. 3. List the major benefits of the technology. 4. List some areas for which crowdsourcing is suitable. 5. Why may you need a vendor to crowdsource the problem-solving process?

11.8 ARTIFICIAL INTELLIGENCE AND SWARM AI SUPPORT OF TEAM COLLABORATION AND GROUP DECISION MAKING

AI, as seen in Chapter 2, is a diversified field. Its technologies can be used to support group decision making and team collaboration.

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 637

AI Support of Group Decision Making

A major objective of AI is to automate decision making and/or to support its process. This objective holds also for decisions made by groups. However, we cannot automate a decision made by a group. All we can do is to support some of the steps in a group’s decision-making process.

A logical place to start is Figure 11.1. We can examine the different steps of the pro- cess and see where AI can be used.

1. Meeting preparation. AI is used to find a convenient time for meetings to take place. AI can assist in scheduling meetings so that all can participate.

2. Problem identification. AI technologies are used for pattern recognition that can identify areas that need attention. AI can be used in other types of analysis to identify potential or difficult to pinpoint problems.

3. Idea generation. AI is known for its quest for creativity. Team members can increase their creativity when they use AI for support.

4. Idea organization. Natural language processing (NLP) can be used to sort ideas and organize them for improved evaluation.

5. Group interaction and collaboration. AI can facilitate communication and collabo- ration among group members. This activity is critical in the process of arriving at a consensus. Also, Swarm AI (see the end of this section) is designed to increase interactions among group members so their combined wisdom is elevated.

6. Predictions. AI supports predictions that are required to assess the impact of the ideas generated regarding performance and/or impacts in the future. Machine learn- ing, deep learning, and Swarm AI are useful tools in this area.

7. Multinational groups. Collaboration among people located in different countries is on the rise. AI enables group interaction of people who speak different languages, in real time.

8. Bots are useful in supporting meetings. Group members may consult Alexa and other bots. Chatbots can provide answers to queries in real time.

9. Other advisors. IBM Watson can provide useful advice during meetings, supple- menting knowledge provided by participants and by Alexa.

Example

In 2018, Amazon.com was looking for a site for its second headquarters. A robot named Aiera from Wells Fargo Securities used deep learning to predict that the winning site would be Boston (Yurieff, 2018a). (When this chapter was written, the decision had not been made.)

For an academic approach on how to improve group decision making by AI, see Xia (2017).

AI Support of Team Collaboration

Organizations today are looking for ways to increase and improve collaboration with employees, business partners, and customers. To gain insight into how AI may impact collaboration, Cisco Systems sponsored a global survey, AI Meets Collaboration (Morar HPI, 2017), regarding the impact of AI, including the use of virtual assistants in the work space. The major findings of this survey are:

1. Virtual assistants increase employees’ productivity, creativity, and job satisfaction. Bots also enable employees to focus on high-value tasks.

2. Bots are accepted as part of workers’ teams.

638 Part IV • Robotics, Social Networks, AI and IoT

3. Bots improve conference calls. They also can take meetings notes and schedule meetings.

4. AI can use facial recognition to sign in eligible people to meetings. 5. Personal characteristics are likely to influence how people feel about AI in the

workplace. 6. Employees in general like to have AI in their teams. 7. Security is a major concern when AI, such as virtual assistants, is used in teams. 8. The major AI tools that are most useful are NLP and voice response; AI can also

summarize the key topics of meetings and understand participants’ needs. AI can be aware of organizational goals and workers’ skills and can make suggestions accordingly.

For how virtual meetings are supported with AI by Cisco Systems in their leading products, see Technology Insight 11.2.

TECHNOLOGY INSIGHT 11.2 How Cisco Improves Collaboration with AI

Cisco Systems is well known for its collaboration products such as Spark and Webex. The first step in introducing AI was to acquire MindMeld’s AI platform for use in Cisco’s collaboration products. The project’s objective was to improve the conversational interferences for any application or device so users could better understand the context of conversations. MindMeld uses machine learning to improve the accuracy of voice and text communication. To do so, it uses NLP and five varieties of machine learning. Cisco is also integrating IBM Watson into its enterprise collabora- tion solutions. As you may recall from Chapter 6, Watson is a powerful advisor. AI collaboration tools can increase efficiency, speed idea generation, and improve the quality of decisions made by groups. The improved Cisco’s technology will be used in conference rooms and everywhere else. One of the major AI projects is the assistant to Spark.

Monica, a Digital Assistant to the Spark Collaboration Platform Monica is trained to answer users’ queries by employing machine learning. Furthermore, users can use Monicait to interact with the Spark collaboration platform using natural language com- mands. It is an enterprise assistant similar to Alexa and Google Assistant (Chapter 12). Cisco’s Monica is the world’s first enterprise-ready voice assistant specifically designed to support meetings. The bot has deep-domain conversational AI that adds cognitive capabilities to the Spark platform.

Monica can assist users in several of the steps of Figure 11.1, such as:

• Organize meetings. • Provide information to participants before and during meetings. • Navigate and control Spark’s devices. • Help organizers find a meeting room and reserve it. • Help share screens and bring up a whiteboard. • Take meeting notes and organize them.

In the near future, Monica will know about participants’ internal and external activities and will schedule meetings using this information. Additional functions to support more steps of the pro- cess in Figure 11.1 will be added in the future.

For more about the assistant, see youtube.com/watch?v=8OcFSEbR_6k (5:10 minutes).

Note: Cisco Spark will become Webex Teams with more AI functionalities. In addition, Webex meetings will include videoconferencing for collaboration and other supports to meetings.

Sources: Compiled from Goecke (2017), Finnegan (2018), and Goldstein (2017).

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 639

Swarm Intelligence and Swarm AI

The term swarm intelligence refers to the collective behavior of decentralized, self- organized systems, natural or artificial (per Wikipedia). Such systems consist of things (e.g., ants, people) interacting with each other and their environment. A swarm’s actions are not centrally controlled, but they lead to intelligent behavior. In nature, there are many examples (e.g., ant colonies, fish schools) of such behaviors.

Natural groups were observed to amplify their group intelligence by forming swarms. Social creatures, including people, can improve the performance of their individual mem- bers when working together as a unified system. In contrast with animals and other species whose interactions among group members are natural, people need technology to exhibit swarm intelligence. This concept is used in studies and implementation of AI and robotics. The major applications are in the area of predictions.

Example

A study at Oxford University (United Kingdom) involved predicting the results of all 50 English Premier League soccer games over five weeks. A group of independent judges scored 55 percent accuracy when working alone. However, when predicting using an AI swarm, their prediction success increased to 72 percent (an improvement of 31 percent). Similar improvement was recorded in several other studies.

In addition to improved prediction accuracy, studies show that using swarm AI results in more ethical decisions than that of individuals (Reese, 2016).

SWARM AI TECHNOLOGY Swarm AI (or AI swarm) provides the algorithms for the inter- connections among people creating the human swarm. These connections enable the knowledge, intuition, experience, and wisdom of individuals to merge into single improved swarm intelligence. Results of swarm intelligence can be seen in the TED presentation (15:58 min.) at youtube.com/watch?v=Eu-RyZt_Uas. Swarm AI is used by several third- party companies (e.g., Unanimous.aI, as illustrated in Application Case 11.3.

XPRIZE is a nonprofit organization that allocates prizes via competitions to promote innovations that have the potential to change the world for the bet- ter. The main channel for designing prizes that solve humanity’s grandest challenges is called Visioneer- ing. It attempts to harness the power of the global crowd to develop solutions to important challenges. The organization’s major event is an annual summit meeting where prizes are designed and proposals are evaluated. The experts at XPRIZE develop concepts and turn them into incentivized competitions. Prizes are donated by leading corporations.

For example, in 2018, IBM Watson donated a $5 million prize called “AI approaches and collabora- tion.” The competition had 142 registered teams, and 62 were left in round 2 in June 2018. The teams are

invited to create their own goals and solutions to a grand challenge.

The Problem

Every year, there is a meeting of 250 members of “Visioneers Summit Ideation” where top experts (entrepreneurs, politicians, scientists, etc.), partici- pate to discover and prioritize topics for the XPRIZE agenda.

Finding the top global problems can be a very complex challenge due to a large number of vari- ables. In just a few days, top experts need to use their collective wisdom to agree on the next year’s XPRIZE top challenges. The method used to support the group’s decision is a critical success factor.

(Continued )

Application Case 11.3 XPRIZE Optimizes Visioneering

640 Part IV • Robotics, Social Networks, AI and IoT

The Solution

In the 2017 annual meeting for determining what challenge to use for 2018, the organization used the swarm AI platform (from Unanimous AI). Several small groups (swarms) moderated by AI algorithms were created to discover challenging topics. The mis- sion was to explore ideas and agree on preferred solutions. The objective was to use the talents and brainpower of the participants.

In other words, the objective was to use the thinking together feature of swarm AI to generate each group’s synergy with the AI algorithms acting as moderators. This way, smarter decisions were generated by the groups than its individual par- ticipants. The different groups examined six pre- selected topics: energy and infrastructure, learning human potential, space and new frontiers, plant and environment, civil society, and health and well-being. The groups brainstormed the issues. Then, each participant created a customized evalu- ation table. The tables were combined and ana- lyzed by algorithms.

Application Case 11.3 (Continued)

The Swarm AI replaced traditional voting meth- ods by optimizing the detailed contribution of each participant.

The Results

Use of swarm AI did the following:

• Supported the generation of optimized answers and enabled fast buy-in from the participants.

• Enabled all participants to contribute. • Provided a better voting system than in previ-

ous years.

Questions for Case 11.3 1. Why is the group discussion in this case complex?

2. Why is getting a consensus when top experts are involved more difficult than when non-experts are involved?

3. What was the contribution of swarm AI?

4. Compare simple voting to swarm AI voting.

Sources: Compiled from Unanimous AI (2018), xprize.org, and xprize.org/about.

SWARM AI FOR PREDICTIONS Swarm AI was used by Unanimous AI for making predic- tions in difficult-to-assess situations. Examples are:

• Predicting Super Bowl #52 number of points scored (used for spread waging). • Predicting winners in the regular NFL season. • Predicting the top four finishers of the 2017 Kentucky Derby. • Predicting the top recipients of the Oscars in 2018.

u SECTION 11.8 REVIEW QUESTIONS

1. Relate the use of AI to the activities in Figure 11.1. 2. Discuss the different ways that AI can facilitate group collaboration. 3. How can AI support group evaluation of ideas? 4. How can AI facilitate idea generation? 5. What is the analogy of swarm AI to swarms of living species? 6. How is swarm AI used to improve group work and to initiate group predictions?

11.9 HUMAN–MACHINE COLLABORATION AND TEAMS OF ROBOTS

Since the beginning of the Industrial Revolution, people and machines have worked together. Until the late 1900s, the collaboration was in manufacturing. But since then, due to advanced technology and changes in the nature of work, human–machine collabora- tion has spread to many other areas, including performing mental and cognitive work

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 641

and collaborating on managerial and executive work. According to Nizri (2017), human and AI collaboration will shape the future of work (see also Chapter 14).

Humans and machines can collaborate in many ways, depending on the tasks they perform. The collaboration with robots in the manufacturing scenario is an extension of the older model in which humans and robots collaborated with humans controlling and monitoring production and robots doing physical work that requires speed, power, accuracy, or nonstop attention. Robots are also doing work in hazardous environments. In general, robots complement human capabilities. An example is Amazon’s distribution centers where over 50,000 mobile robots do a variety of tasks, mostly in hauling materials and helping to fulfill customer orders. The robotic technology enables fully collaborative solutions. For details, watch the video at Kuka kuka.com/en-us/technologies/human- robot-collaboration. Kuka’s system allows the execution of complex jobs that can be done cost effectively.

Another collaborative human-robotic system is called YuMi. To see this sys- tem (from ABB Robotics) at work, watch the 4:38 min. video at youtube.com/ watch?v=2KfXY2SvlmQ. Notice that the robot has two arms.

Human–Machine Collaboration in Cognitive Jobs

Advancement in AI enables the automation of nonmanual activities. While some intelligent systems are fully automated (see automated decision making in Chapter 2 and chatbots in Chapter 12), there are many more examples of human–machine collaboration in cognitive jobs (e.g., in marketing and finance). An example is in investment decisions. A human asks the computer for advice concerning investments, and after receiving the advice, can ask more questions, changing some of the input. The difference from the past is that today the computers (machines) can provide much more accurate suggestions, by using machine learning and deep learning. Another collaboration example involves medical diagnoses of complex situations. For example, IBM Watson provides medical advice, which permits doctors and nurses to significantly improve their jobs. Actually, the entire field of machines advising humans is reaching new heights. For more on the increasing collaborative power of AI, see Carter (2017).

TOP MANAGEMENT JOBS A major task of managers is decision making, which has become one area of human–machine collaboration. Use of AI and analytics has improved decision making considerably, as illustrated throughout this book. For an overview, see Wladawsky-Berger (2017).

McKinsey & Company and MIT are two major players in researching the topic of col- laboration between managers and machines. For example, Dewhurst and Wilmott (2014) report on its increased use of man-machine collaboration, using deep learning. A Hong Kong company even appointed a decision-making algorithm to its board of directors. Com- panies are using crowdsourcing advice to support complex problem solving, as illustrated in Section 11.7.

Robots as Coworkers: Opportunities and Challenges

Sometime in the future, walking and talking humanoid robots will socialize with humans during breaks from work. Someday, robots will become cognitive coworkers and help people be more productive (as long as people do not talk too much with the robots).

According to Tobe (2015), a study at a BMW factory found that human–robot col- laboration could be more productive than either humans or robots working by themselves. Also, the study found that collaboration reduced idle time by 85 percent. This is because people and machines capitalize on the strengths of each (Marr, 2017).

642 Part IV • Robotics, Social Networks, AI and IoT

The following challenges must be considered:

• Designing a human–machine team that capitalizes on the strength of each partner. • Exchanging information between humans and robots. • Preparing company employees in all departments for the collaboration (Marr,

2017). • Changing business processes to accommodate human–robot collaboration (Moran,

2018). • Ensuring the safety of robots and employees that work together.

TECHNOLOGIES THAT SUPPORT ROBOTS AS COWORKERS Yurieff (2018b) lists the following examples of facilitating or considering robots as coworkers.

1. Virtual reality can be used as a powerful training tool (e.g., for safety). 2. A robot is working with an ad agency in Japan to generate ideas. 3. A robot can be your boss. 4. Robots are coworkers in providing parts out of bins in assembly lines and can check

quality together with humans. 5. AI tools measure blood flow and volume of the cardiac muscles in seconds (instead

of minutes when done completely by a radiologist). This information facilitates the decisions made by radiologists.

BLENDING HUMANS AND AI TO BEST SERVE CUSTOMERS Genesys Corp. commissioned Forrester Research Company to conduct a global study in 2017 to find how companies are using AI to improve customer service. The study, titled “Artificial Intelligence with the Human Touch,” is available at no charge from genesys.com/resources/artificial- intelligence-with-the-human-touch. A related video is available at youtube.com/ watch?v=NP2qqwGTNPk.

The study revealed the following:

1. “AI is already transforming enterprises by increasing worker efficiency and produc- tivity, delivering better customer experiences and uncovering new revenue streams” (from the Executive Summary).

2. A major objective of man–machine collaboration is to improve the satisfaction of both customers and companies’ agents rather than reduce cost.

3. Human agents’ ability to connect emotionally with customers for the increased satisfaction of themselves and customers is superior to that of service provided by AI.

4. By blending the strengths of humans and AI, companies achieve better customer service satisfaction of customers (71 percent) and agents (69 percent).

Note that AI excels in the support of marketing and advertising as illustrated in Chapter 2. See also Loten (2018) for the use of AI to support customer relationship management (CRM) and of crowdsourcing and collective intelligence to support marketing.

COLLABORATIVE ROBOTS (CO-BOTS) Collaborative robots (co-bots) are designed to work with people, assisting in executing various tasks. These robots are not very smart, but their low cost and high usability make them popular. For details, see Tobe (2015).

Teams of collaborating Robots

One of the future directions in robotics is creating teams of robots that are designed to do complex work. Robot teams are common in manufacturing where they serve each other or join a robot group in simple assembly jobs. An interesting example is the use of a team of robots in preparation to land on Mars.

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 643

Example: Teams of Robots to Explore Mars

Before people land on Mars, scientists need to know more about the “Red Planet.” The idea was to use teams of robots. The German Research Centers for Artificial Intelligence (DFKI) conducted simulation experiments in the desert of Utah. The details of this simulation are described by Staff Writers (2016). The process is illustrated in a 4:54 min. video at youtube. com/watch?v=pvKIzldni68/ showing robots’ collaboration. For more information, see robotik.dfki-bremen.de/en/research/projects/ft-utah.html.

DFKI is not the only entity that plans to explore the surface of Mars. NASA plans to send swarms of robot bees with flapping wings called Marsbees that will operate in a group to explore the land and air of the Red Planet. The reason for the flapping wings structure is to enable low-energy flights (like bumblebees). Each robot is the size of a bee. Part of a wireless communication network, Marsbees will together create networks of sensors. Information will be delivered to a mobile base (see Figure 11.4, showing one robot) that will be the main communication center and a recharging station for the Marsbees. For more information, see Kang (2018).

Getting robots to work together is being researched at MIT. They use their per- ception system to sense the environment, and then they communicate their findings to each other and coordinate their work. For example, a robot can open a door for another robot. Read about how this is done and watch a video at ft.com/video/ ea2d4877-f3fb-403d-84a8-a4d2d4018c5e.

Example

Alibaba.com is using teams of robots in its smart warehouses where robots do 70 percent of the work. This is shown in a video at youtube.com/watch?v=FBl4Y55V2Z4.

Social collaboration of robots is being investigated by watching the behavior of swarms of ants and other species to learn how to design robots to work in teams. Watch the TED presentation at youtube.com/watch?v=ULKyXnQ9xWA on how to design a robot collaboration.

Having robots collaborate involves several issues such as making sure they do not col- lide with each other. This is a part of the safety issue regarding robotics. Finally, you can build your own team of robots with LEGO’s Mindstorms. For details, see Hughes and Hughes (2013).

FIGURE 11.4 Team of Robots Prepares to Go to Mars. Source: C. Kang.

644 Part IV • Robotics, Social Networks, AI and IoT

Chapter Highlights

• Groupware refers to software products that pro- vide collaborative support to groups (including conducting meetings).

• Groupware can support decision-making and problem solving directly or indirectly by improv- ing communication between team members.

• People collaborate in their work (called group work). Groupware (i.e., collaborative computing software) supports group work.

• Group members may be in the same organiza- tion or in different organizations in the same or in different locations and may work at the same or different times.

• The time/place framework is a convenient way to describe the communication and collaboration pat- terns and support of group work. Different tech- nologies can support different time/place settings.

• Working in groups can result in many benefits, including improved decision making, increased productivity and speed, and cost reductions.

• Communication can be synchronous (i.e., same time) or asynchronous (i.e., sent and received at different times).

• The Internet, intranets, and IoT support virtual meetings and decision making through collabora- tive tools and access to data analysis, information, and knowledge.

• Groupware for direct support typically contains capabilities for brainstorming, conferencing, scheduling group meetings; planning; resolving conflicts; videoconferencing; sharing electronic documents; voting; formulating policy; and ana- lyzing enterprise data.

• A GDSS is any combination of hardware and software that facilitates decision-making meet- ings. It provides direct support in face-to-face settings and in virtual meetings, attempting to increase process gains, and reducing process losses of group works.

• Collective intelligence is based on the premise that the combined wisdom of several collabo- rating people is greater than that of individuals working separately.

• Each of the several configurations of collective intelligence can be supported differently by technology.

• Several collaboration platforms, such as Micro- soft Teams and Slack, can facilitate collective intelligence.

• Idea generation and brainstorming are key activ- ities in group work for decision making. Several collaboration software and AI programs are sup- porting these activities.

• Crowdsourcing is a process of outsourcing work to a crowd. Doing so can improve problem solving, idea generation, and other innovative activities.

• Crowdsourcing can be used to make predictions by groups of people, including crowds. Results have shown better predictions, especially when communication is used among the predictors than when no communication was enabled.

• One method of communication in crowdsourc- ing is based on swarm intelligence. A technol- ogy known as swarm AI has had significant success.

• AI can support many activities in group deci- sion making.

• Human–machine collaboration can be a major method of work in the future.

• Machines that once supported manufacturing work are used now also in support of cogni- tive, including managerial, work.

• For people and machines to work in teams, it is necessary to make special preparations.

• Robots may work in exclusive teams. They do so in manufacturing and possibly in other activities (e.g., explore Mars) as they become more intelligent.

u SECTION 11.9 REVIEW QUESTIONS

1. Why is there an increase in human–machine collaboration? 2. List some benefits of such collaboration. 3. Describe how collaborating robotics can be used in manufacturing. 4. Discuss the use of teams of robots. 5. What will do robots on Mars?

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 645

Exercises

1. Go to realtimeboard.com. How can the site support idea creation and brainstorming?

2. Investigate how researchers are trying to develop collab- orative computer systems that portray or display nonver- bal communication factors (e.g., images).

3. For each of the software packages Skype Business and WebEx, check the trade literature and the Web for details and explain how each includes computerized collabora- tive support system capabilities.

4. Compare Simon’s four-phase decision-making model to the steps in using GDSS.

5. A major claim in favor of wikis is that they can replace e-mail, eliminating its disadvantages (e.g., spam). Go to socialtext.com and review such claims. Find other sup- porters of switching to wikis. Then find counter argu- ments and conduct a debate on the topic.

6. Search the Internet to identify sites that describe methods for making meetings more effective and efficient.

7. Enter MIT Center for CI and review some of its recent activities. Write a report.

8. Debate the issue of the quality of crowdsourc- ing results. Start by viewing youtube.com/ watch?v=JJHAHQmiI3c.

9. Find information about Yammer (a Microsoft company). Why is it considered a social collaboration tool? Why is it popular? Write a report.

10. Enter Dropbox.com and find its collaboration tools. Write a summary.

11. Read Pena (2017). Examine the 12 benefits of collabora- tion. Which are related to social collaboration?

12. Compare Microsoft’s Universal Translator to Google’s Translator. Concentrate on face-to-face conversation in real time.

13. Write a report on the issue of whether crowdsourcing produces superior decisions. Use Quora for help. Find other sources.

14. Investigate the status of IBM Connections Cloud. Exam- ine all the collaboration and communication features. How does the product improve productivity? Write a report.

Key Terms

Questions for Discussion

1. Explain why it is useful to describe group work in terms of the time/place framework.

2. Describe the kinds of support that groupware can pro- vide to decision makers.

3. Explain why most groupware is deployed today over the Web.

4. Explain in what ways physical meetings can be ineffi- cient. Explain how technology can make meetings more effective.

5. Explain how GDSS can increase some benefits of col- laboration and decision making in groups and eliminate or reduce some losses.

6. The initial term for group support system (GSS) was group decision support system (GDSS). Why was the

word decision dropped? Does this make sense? Why, or why not?

7. Discuss why Microsoft SharePoint is considered a work- space. What kind of collaboration does it support?

8. Reese (2017) claims that swarm AI can be used instead of polls for market research. Discuss the advantages of swarm AI. In what circumstances would you prefer each method? (Read “Polls vs. Swarms” at Unanimous AI.)

9. What is a collaborative robot? What is an uncollabora- tive one?

10. Discuss the ways in which social collaboration can improve work in a digital workplace.

11. Provide an example of using analytics to improve deci- sion making in sport.

asynchronous brainstorming collective intelligence collaborative workspace crowdsourcing decision room

group decision making group decision support

system (GDSS) group support system

(GSS) groupthink

groupware group work idea generation online workspace process gain process loss

swarm intelligence synchronous (real-time) virtual meeting virtual team

646 Part IV • Robotics, Social Networks, AI and IoT

15. Compare Microsoft Teams to Spark Teams. Write a report.

16. Enter crowdtap.com and read Kurzer (2018) paper. Explain how the platforms work. Relate the material

about crowdsourcing and collective intelligence. Write a report.

17. Go to technologyreview.com and look at the May 8, 2017, video (17:42 min.) “Next Generation Human- Machine Collaboration.” Write a report.

References

Babbar-Sebens, M., et al. “A Web-Based Software Tool for Partici- patory Optimization of Conservation Practices in Watersheds.” Environmental Modelling & Software, 69, 111–127, July 2015.

Basco-Carrera, L., et al. “Collaborative Modelling for Informed Decision Making and Inclusive Water Development.” Water Resources Management, 31:9, July 2017.

Bhandari, R., et al. “How to Avoid the Pitfalls of IT Crowd- sourcing to Boost Speed, Find Talent, and Reduce Costs.” McKinsey & Company, June 2018.

Bridgwater, A. “Governments to Tap IoT for ‘Collective Intel- ligence.’” Internet of Business, January 2, 2018.

Carter, R. “The Growing Power of Artificial Intelligence in Workplace Collaboration.” UC Today, June 28, 2017.

Chiu, C-M., T. P. Liang, and E. Turban. “What Can Crowdsourc- ing Do for Decision Support?” Decision Support Systems, September 2014.

Coleman, D. “10 Components of Collaborative Intelligence.” CMS Wire, November 21, 2011.

de Lares Norris, M. A. “Collaboration Technology Is the Driv- ing Force for Productivity and Businesses Need to Embrace It . . . Now.” IT ProPortal, January 4, 2018.

DeSanctis, G., and R. B. Gallupe. “A Foundation for the Study of Group Decision Support Systems.” Management Science, 33:5, 1987.

Dewhurst, M., and P. Willmott. “Manager and Machine: The New Leadership Equation.” McKinsey & Company, September 2014.

Dignan, L. “A Sweet Idea: Hershey Crowdsourcing for Summer Chocolate Shipping Concepts.” ZDNet, January 14, 2016.

Digneo, C. “49 Online Collaboration Tools to Help Your Team Be More Productive.” Time Doctor, 2018. biz30.timedoctor.com/online-collaboration-tools/ (accessed July 2018).

50Minutes.com. The Benefits of Collective Intelligence: Make the Most of Your Team’s Skills. Brussels, Belgium: 50Minutes.com (Lemaitre Publishing), 2017.

Finnegan, M. “Cisco Shakes Up Collaboration Efforts; Morphs Spark into Webex.” Computer World, May 2, 2018.

Goecke, J. “Meet Cisco Spark Assistant, Your Virtual Assistant for Meetings.” Cisco Blogs, November 2, 2017.

Goldstein, P. “How Can AI Improve Collaboration Technol- ogy?” Biztech Magazine, June 5, 2017.

Grant, R. P. “Why Crowdsourcing Your Decision-Making Could Land You in Trouble.” The Guardian, March 10, 2015.

Howe, J. Crowdsourcing: Why the Power of the Crowd is Driv- ing the Future of Business. New York: Crown Business, 2008.

Hughes, C., et al. Build Your Own Teams of Robots with LEGO® Mindstorms® NXT and Bluetooth®. New York, NY: McGraw-Hill/Tab Electronic, 2013.

Kang, C-K. “Marsbee—Swarm of Flapping Wing Flyers for Enhanced Mars Exploration.” NASA.gov, March 30, 2018.

Keith, E. “Here’s How a New Crowd-Sourced Map Is Making Canadian Streets Safer for Cyclists.” Narcity.com, June 2018.

Kurzer, R. “Meet Suzy: The New Crowd Intelligence Platform with the Cute Name.” MarTech Today, March 27, 2018.

Loten, A. “The Morning Download: AI-Enabled Sales Tools Spotlight Data Needs.” The Wall Street Journal, March 27, 2018.

Marr, B. “Are You Ready to Meet Your Intelligent Robotic Co-Worker?” Forbes.com, September 8, 2017.

McMahon, K., et al. “Beyond Idea Generation: The Power of Groups in Developing Ideas.” Creativity Research Journal, 28, 2016.

Microsoft. “Hendrick Motorsports Uses Microsoft Teams to Win Productivity Race.” Customers.Microsoft.com, April 27, 2017.

Moran, C. “How Should Your Company Prepare for Robot Coworkers?” Fast Company, February 13, 2018.

Morar HPI. “A Global Survey Reveals Employee Perception of Advanced Technologies and Virtual Assistants in the Workplace.” Cisco.com, October 2017.

Mulgan, G. Big Mind: How Collective Intelligence Can Change Our World. Princeton, NJ: Princeton University Press, 2017.

Nizri, G. “Shaping the Future of Work: A Collaboration of Humans and AI.” Forbes.com, August 17, 2017.

Pena, S. “12 Benefits of a Collaborative Workspace.” Cre- ator, June 14, 2017. wework.com/creator/start-your- business/12-benefits-of-a-collaborative-workspace/ (accessed July 2018).

Power, B. “Improve Decision-Making with Help from the Crowd.” Harvard Business Review, April 8, 2014.

Reese, H. “New Research Shows That Swarm AI Makes More Ethical Decisions Than Individuals.” Tech Republic, June 8, 2016.

Ruiz-Hopper, M. “Hendrick Motorsports Gains Competitive Advantage on the Race Track.” Microsoft.com, September 26, 2016.

Staff Writers. “Scientists Simulate a Space Mission in Mars- Analogue Utah Desert.” Mars Daily, October 19, 2016.

Stewart, C. “The 18 Best Tools for Online Collaboration.” Cre- ative Blog, March 7, 2017.

Chapter 11 • Group Decision Making, Collaborative Systems, and AI Support 647

Thorn, C., and J. Huang. “How Carnegie Is Using Technology to Enable Collaboration in Networks.” Carnegie Founda- tion Blog, September 9, 2014.

Tobe, F. “Why Co-Bots Will Be a Huge Innovation and Growth Driver for Robotics Industry.” IEEE Spectrum, December 30, 2015.

Unanimous AI. “XPRIZE Uses Swarm AI Technology to Optimize Visioneers Summit Ideation.” Unanimous AI, 2018. UAI_ case_study_xprize_0601_0601.pdf (accessed July 2018).

Wladawsky-Berger, I. “Building an Effective Human-AI Deci- sion System.” The Wall Street Journal, December 1, 2017.

Xia, L. “Improving Group Decision-Making by Artificial Intelli- gence.” In C. Sierra, Editor, Proceedings of the Twenty-Sixth

International Joint Conference on Artificial Intelligence, IJCAI, 2017.

Yazdani, M., et al. “A Group Decision Making Support System in Logistics and Supply Chain Management.” Expert Systems with Applications, 88, December 1, 2017.

Yoon, Y., et al. “Preference Clustering-Based Mediating Group Decision-Making (PCM-GDM) Method for Infrastructure Asset Management.” Expert Systems with Applications, 83, October 15, 2017.

Yurieff, K. “Robot Predicts Boston Will Win Amazon HQ2.” CNN Tech, March 13, 2018a.

Yurieff, K. “Robot Co-Workers? 7 Cool Technologies Changing the Way We Work.” CNN Tech, May 4, 2018b.

648

C H A P T E R

12 Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants, and Robo Advisors

LEARNING OBJECTIVES

■■ Describe recommendation systems ■■ Describe expert systems ■■ Describe chatbots ■■ Understand the drivers and capabilities of chatbots and their use

■■ Describe virtual personal assistants and their benefits

■■ Describe the use of chatbots as advisors ■■ Discuss the major issues related to the implementation of chatbots

Advancement in artificial intelligence (AI) technologies and especially natural lan-guage processing (NLP), machine and deep learning and knowledge systems, coupled with the increased quality and functionalities of other intelligent systems, and mobile devices and their apps, have driven the development of chatbots (bots) for inexpensive and fast execution of many tasks related to communication, collaboration, and information retrieval. The use of chatbots in business is increasing rapidly, partly be- cause of their fit with mobile systems and devices. As a matter of fact, sending messages is probably the major activity in the mobile world.

In the last two to three years, many thousands of bots have been placed into ser- vice worldwide by both organizations (private and public) and individuals. Many people refer to these phenomena as the chatbot revolution. Chatbots today are much more so- phisticated than those of the past. They are extensively used, for example, in marketing; customer, government, and financial services; healthcare; and in manufacturing. Chatbots make communication more personal than faceless computers and excel in data gathering. Chatbots can stand alone or be parts of other knowledge systems.

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 649

We divide the applications in this chapter into four categories: expert systems, chat- bots for communication and collaboration, virtual personal assistants (native products, such as Alexa), and chatbots that are used as professional advisors. Some implementation topics of intelligent systems are described last.

This chapter has the following sections:

12.1 Opening Vignette: Sephora Excels with Chatbots 649 12.2 Expert Systems and Recommenders 650 12.3 Concepts, Drivers, and Benefits of Chatbots 660 12.4 Enterprise Chatbots 664 12.5 Virtual Personal Assistants 672 12.6 Chatbots as Professional Advisors (Robo Advisors) 676 12.7 Implementation Issues 680

12.1 OPENING VIGNETTE: Sephora Excels with Chatbots

THE PROBLEM

Sephora is a French-based cosmetics/beauty products company doing business globally. It has its own stores and sells its goods in cosmetic and department stores. In addition, Sephora sells online on Amazon and on its online store. The company sells hundreds of brands, including many of its own. It operates in a very competitive market where cus- tomer care and advertising are critical. Sephora sells some products for men, but most beauty products are targeted to women.

THE SOLUTION

Sephora’s first use of chatbots occurred through messaging services. The purpose of the first bot was to search for information for the company’s resources such as videos, images, tips, and so on. This bot operates in a question-and-answer (Q&A) mode. It rec- ommends relevant content based on customers’ interests. The company aims to appeal to young customers messaging on Kik.

Sephora researchers found that customers conversing with the Kikbot were engaged deeply in the dialog. Then the bot encouraged them to explore new products. Sephora’s newer bot called Reservation Assistant was placed on Facebook Messenger. It enables customers to book or reschedule makeover appointments.

Another Sephora bot delivered on Kik is Shade-Matching. It matches lips colors to photos (face and lips) uploaded by users and recommends the best match to them. The bot also lets users try on photos of recommended colors, using Sephora Virtual Artist that runs on Facebook Messenger. Bots are deployed as mobile apps. If users like the recom- mendation, they are directed to the company’s Web store to buy the products. Users can upload photos taken with selfies so that the program can do the matching. Over 4 million visitors tried 90 million shades in the first year of Virtual Artist’s operation.

The Q&A collection of the knowledge base was built by connecting it with store experts. Knowledge acquisition techniques (Chapter 2) were used for this purpose. The company’s bots use NLPs that were trained to understand the typical vocabulary of users.

THE RESULTS

The company’s customers loved the bots. In addition, Sephora learned the importance of providing assistance and guidance to users who are motivated to return (at a reasonable cost!), happier, and more engaged.

650 Part IV • Robotics, Social Networks, AI and IoT

Sephora’s bot asks users questions to find their tastes and preferences. Then it acts like a recommendation system (Section 12.2), offering products. Kik and Messenger users can purchase items without leaving the messaging service.

Finally, the company has improved the bots’ knowledge over time and plans new bots for additional tasks.

Note: Sephora was selected by Fast Company Magazine, March/April 2018, as one of the “World’s Most Innovative Companies.” Sephora is known for its digital transformation and innovation (Rayome, 2018). Also, Sephora’s bots are considered among the top marketing chatbots (Quoc, 2017).

Sources: Compiled from Arthur (2016), Rayome (2018), and Taylor (2016), theverge.com/2017/3/16/14946086/ sephora-virtual-assistant-ios-app-update-ar-makeup/, and sephora.com/.

u QUESTIONS FOR THE OPENING VIGNETTE

1. List and discuss the benefits of bots to the company. 2. List and discuss the benefits of bots to customers. 3. Why were the bots deployed via Messenger and Kik? 4. What would happen to Sephora if competitors use a similar approach?

WHAT WE CAN LEARN FROM THIS VIGNETTE

In the highly competitive world of retail beauty products, customer care and marketing are critical. Using only live employees can be very expensive. In addition, customers are shopping 24/7, and physical stores are open during limited hours and days. In addi- tion, there are large combinations of certain beauty products (e.g., many shades/colors) available. Sephora decided to use chatbots on Facebook Messenger and Kik to engage its customers. Chatbots, the subject of this chapter, are available 24/7 at a lower cost and are delivered via mobile devices. Bots deliver information to customers consistently and quickly direct customers to easy online shopping. Sephora placed its chatbots on messag- ing services. The logic was that people like to chat with friends on messaging services, and they may also like to chat with businesses.

In addition to several services to customers, using chatbots helps Sephora learn about customers. This type of chatbot is the most common type for customer care and market- ing. In this chapter, we cover several other types of knowledge systems, including the pioneering expert systems, recommenders, virtual personal assistants offered by several large technology companies, and robo advisors.

12.2 EXPERT SYSTEMS AND RECOMMENDERS

In Chapter 2 we introduced the reader to the concept of autonomous decision systems. An expert system is a category of autonomous decision systems and are considered the earliest applications of AI. Expert systems use started in research institutions in the early and mid- 1960s (e.g., Stanford University, IBM) and was adopted commercially during the 1980s.

Basic Concepts of Expert Systems (ES)

The following are the major concepts related to ES technology.

DEFINITIONS There are several definitions of expert systems. Our working definition is that an expert system is a computer-based system that emulates decision making and/or problem solving of human experts. These decisions and problems are in complex areas

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 651

that require expertise to solve. The basic objective is to enable nonexperts to make deci- sions and solve problems that usually require expertise. This activity is usually performed in narrowly defined domains (e.g., making small loans, providing tax advice, analyzing reasons for machine failure). Classical ES use “what-if-then” rules for their reasoning.

EXPERTS An expert is a person who has the special knowledge, judgment, experience, and skills to provide sound advice and solve complex problems in a narrowly defined area. It is an expert’s job to provide the knowledge about how to perform a task so that a nonexpert will be able to do the same task assisted by ES. An expert knows which facts are important and understands and explains the dependent relationships among those facts. In diagnosing a problem with an automobile’s electrical system, for example, an expert car mechanic knows that a broken fan belt can be the cause for the battery to discharge.

There is no standard definition of expert, but decision performance and the level of knowledge a person has are typical criteria used to determine whether a particular person is an expert as related to ES. Typically, experts must be able to solve a problem and achieve a performance level that is significantly better than average. An expert at one time or in one region may not be an expert in another time or region. For example, a legal expert in New York may not be one in Beijing, China. A medical student may be an expert compared to the general public but not in making a diagnosis or performing sur- gery. Note that experts have expertise that can help solve problems and explain certain obscure phenomena only within a specific domain. Typically, human experts are capable of doing the following:

• Recognizing and formulating a problem. • Solving a problem quickly and correctly. • Explaining a solution. • Learning from experience. • Restructuring knowledge. • Breaking rules (i.e., going outside the general norms) if necessary. • Determining relevance and associations.

Can a machine help a nonexpert perform like an expert? Can a machine make autonomous decisions that experts make? Let us see. But first, we need to explore what expertise is.

EXPERTISE An expertise is the extensive, task-specific knowledge that experts possess. The level of expertise determines the success of a decision made by an expert. Expertise is often acquired through training, learning, and experience in practice. It includes ex- plicit knowledge, such as theories learned from a textbook or a classroom and implicit knowledge gained from experience. The following is a list of possible knowledge types used in ES applications:

• Theories about the problem domain. • Rules and procedures regarding the general problem domain. • Heuristics about what to do in a given problem situation. • Global strategies for solving of problems amenable to expert systems. • Meta knowledge (i.e., knowledge about knowledge). • Facts about the problem area.

These types of knowledge enable experts to make better and faster decisions than nonexperts.

652 Part IV • Robotics, Social Networks, AI and IoT

Expertise often includes the following characteristics:

• It is usually associated with a high degree of intelligence, but it is not always as- sociated with the smartest person.

• It is usually associated with a vast quantity of knowledge. • It is based on learning from past successes and mistakes. • It is based on knowledge that is well stored, organized, and quickly retrievable

from an expert who has excellent recall of patterns from previous experiences.

Characteristics and Benefits of ES

ES were used during the period 1980 to 2010 by hundreds of companies worldwide. However, since 2011, their use has declined rapidly, mostly due to the emergence of bet- ter knowledge systems, three types of which are described in this chapter. It is important, however, to understand the major characteristics and benefits of expert systems since many of them evolved evidenced newer knowledge systems.

The major objective of ES is the transfer of expertise to a machine. The expertise will be used by nonexperts. A typical example is a diagnosis. For example, many of us can use self-diagnosis to find (and correct) problems in our computers. Even more than that, computers can find and correct problems by themselves. One field in which such ability is practiced is medicine, as described in the following example:

Example: Are You Crazy?

A Web-based ES was developed in Korea for people to self-check their mental health sta- tus. Anyone in the world can access it and get a free evaluation. The knowledge for the system was collected from a survey of 3,235 Korean immigrants. The results of the survey were analyzed and then reviewed by experts via focus group discussions. For more infor- mation, see Bae (2013).

BENEFITS OF ES Depending on the mission and structure of ES, the following are their capabilities and potential benefits:

• Perform routine tasks (e.g., diagnosis, candidate screening, credit analysis) that require expertise much faster than humans.

• Reduce the cost of operations. • Improve consistency and quality of work (e.g., reduce human errors). • Speed up decision making and make consistent decisions. • May motivate employees to increase productivity. • Preserve scarce expertise of retiring employees. • Help transfer and reuse knowledge. • Reduce employee training cost by using self-training. • Solve complex problems without experts and solve them faster. • See things that even experts sometimes miss. • Combine expertise of several experts. • Centralize decision making (e.g., by using the “cloud”). • Facilitate knowledge sharing.

These benefits can provide a significant competitive advantage to companies that use ES. Indeed, some companies have saved considerable amounts of money using them.

Despite these benefits, the use of ES is on the decline. The reasons for this and the related limitations are discussed later in this section.

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 653

Typical Areas for ES Applications

ES have been applied commercially in a number of areas, including the following:

• Finance. Finance ES include analysis of investments, credit, and financial reports; evaluation of insurance and performance; tax planning; fraud prevention; and finan- cial planning.

• Data processing. Data processing ES include system planning, equipment selec- tion, equipment maintenance, vendor evaluation, and network management.

• Marketing. Marketing ES include customer relationship management, market research and analysis, product planning, and market planning. Also, presale advice is provided for prospects.

• Human resources. Examples of human resource ES are planning, performance evaluation, staff scheduling, pension management, regulatory advising, and design of questionnaires.

• Manufacturing. Manufacturing ES include production planning, complex prod- uct configuration, quality management, product design, plant site selection, and equipment maintenance and repair (including diagnosis).

• Homeland security. These ES include terrorist threat assessment and terrorist finance detection.

• Business process automation. ES have been developed for desk automation, call center management, and regulation enforcement.

• Healthcare management. ES have been developed for bioinformatics and other healthcare management issues.

• Regulatory and compliance requirements. Regulations can be complex. ES are using a stepwise process to ensure compliance.

• Web site design. A good Web site design requires paying attention to many vari- ables and ensures that performance is up to standard. ES can lead to a proper design process.

Now that you are familiar with the basic concepts of ES, it is time to look at the internal structure of ES and how their goals are achieved.

Structure and Process of ES

As you may recall from Section 2.5 and Figure 2.5, the process of knowledge extraction and its use is divided into two distinct parts. In ES we refer to these as the development environment and the consultation environment (see Figure 12.1). An ES builder builds the necessary ES components and loads the knowledge base with appropriate repre- sentation of expert knowledge in the development environment. A nonexpert uses the consultation environment to obtain advice and solve problems using the expert knowledge embedded into the system. These two environments are usually separated.

MAJOR COMPONENTS OF ES The major components in typical expert systems include:

• Knowledge acquisition. Mostly from human experts, is usually obtained by knowledge engineers. This knowledge, which may derive from several sources, is integrated, validated, and verified.

• Knowledge base. This is a knowledge repository. The knowledge is divided into knowledge about the domain and knowledge about problem solving and solu- tion procedures. Also, the input data provided by the users may be stored in the knowledge base.

• Knowledge representation. This is frequently organized as business rules (also known as production rules).

654 Part IV • Robotics, Social Networks, AI and IoT

These major components of ES generate useful solutions in many areas. Remember that these areas need to be well structured and in fairly narrow domains. Less common is a justifier/explanation subsystem that shows users of rule-based systems the chains of rules used to arrive at conclusions. Also, least common is a knowledge refining subsystem that helped to improve knowledge (e.g., rules) when new knowledge is added.

A major provider of expert systems technologies was Exsys Inc. While the company is no longer active in this business, its Web site (Exsys.com) is. It contains tutorials and a large number of cases related to its major software product, Exsys Corvid. Application Case 12.1 is one example.

Human Expert(s)

Other Knowledge

Sources

Knowledge Base(s)

(Long Term)

Information Gathering

Knowledge Elicitation

De ve

lop me

nt

En vir

on me

nt

Co ns

ult at

ion

En vir

on me

nt

Knowledge Engineer

Knowledge Rules

Inference Engine

Explanation Facility

Knowledge Refinement

Blackboard (Workspace)

Inferencing Rules

Rules Firing

Refined Rules

Working Memory

(Short Term)

External Data Sources

Data/InformationFacts

User User Interface

Facts

Questions/ Answers

FIGURE 12.1 General Architecture of Expert Systems.

• Inference engine. Also known as the control structure or the rule interpreter, this is the “brain” of ES. It provides the reasoning capability, namely the ability to answer users’ questions, provide recommendations for solutions, generate predic- tions, and conduct other relevant tasks. The engine manipulates the rules by either forward chaining or backward chaining. In 1990s ES started to use other inference methods.

• User interface. This component allows user inference engine interactions. In clas- sical ES, this was done in writing or by using menus. In today’s knowledge sys- tems, it is done by natural languages and voice.

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 655

Terrorist attacks using chemical, biological, or radio- logical (CBR) agents are of great concern due to their potential for leading to large loss of life. The United States and other nations have spent billions of dollars on plans and protocols to defend against acts of terrorism that could involve CBR. However, CBR covers a wide range of input agents with many specific organisms that could be used in multiple ways. Timely response to such attacks requires rapid identification of the input agents involved. This can be a difficult process involving different methods and instruments.

The U.S. Environmental Protection Agency (EPA) along with Dr. Lawrence H. Keith, president of Instant Reference Sources Inc. and other consultants, have incorporated their knowledge, experience, and expertise as well as information in publicly available EPA documents to develop the CBR Advisor using Exsys Inc.’s Corvid software.

One of the most important parts of the CBR Advisor is providing advice in logical step-by- step procedures to determine the identity of a toxic agent when little or no information is avail- able, which is typical at the beginning of a terror- ist attack. The system helps response staff proceed according to a well-established action plan even in such a highly stressful environment. The sys- tem’s dual screens present three levels of informa- tion: (1) a top/executive level with brief answers, (2) an educational level with in-depth information, and (3) a research level with links to other docu- ments, slide shows, forms, and Internet sites. CBR Advisor’s content includes:

Restricted content includes CBR agents and methods for analyzing them. The CBR Advisor can be used for incident response and/or training. It has two different menus, one for emergency response and another, longer menu for training. It is a restricted software program and is not publicly available.

Questions for Case 12.1

1. How can the CBR Advisor assist in making quick decisions?

2. What characteristics of the CBR Advisor make it an expert system?

3. What could be other situations in which similar expert systems can be employed?

Expert systems are also used in high-pressure situa- tions in which human decision makers often need to take split-second actions involving both subjective as well as objective knowledge in responding to emer- gency situations.

Sources: www.exsys.com “Identification of Chemical, Biological and Radiological Agents” http://www.exsyssoftware.com/CaseStudy Selector/casestudies.html. April 2018. (Publicly available informa- tion.) Used with permission.

Application Case 12.1 ES Aid in Identification of Chemical, Biological, and Radiological Agents

Why the Classical Type of ES Is Disappearing

The large benefits described earlier drove the implementation of many ES worldwide. However, like many other technologies, the classical ES have been replaced by better sys- tems. Let us first look at some of the limitations of ES that contributed to its declining use.

1. The acquisition of knowledge from human experts has proven to be very expensive due to the shortage of good knowledge engineers as well as the possible need to interview several experts for one application.

2. Any acquired knowledge needed to be updated frequently at a high cost. 3. The rule-based foundation was frequently not robust and not too reliable or flexible

and could have too many exceptions to the rules. Improved knowledge systems use

• How to classify threat warnings. • How to conduct an initial threat evaluation. • What immediate response actions to take. • How to perform site characterization. • How to evaluate the initial site and safe entry

to it. • Where and how to best collect samples. • How to package and ship samples for analysis.

656 Part IV • Robotics, Social Networks, AI and IoT

data-driven and statistical approaches to make the inferences with better success. In addition, case-based reasoning could work better only if a sufficient number of similar cases were available. So, usually it cannot support ES.

4. The rule-based user-interface needed to be supplemented (e.g., by voice communi- cation, image maps). This could make ES too cumbersome.

5. The reasoning capability of rule-based technology is limited compared to use of newer mechanisms such as those used in machine learning.

NEW GENERATION OF EXPERT SYSTEMS Instead of using the old knowledge acquisi- tion and representation system, newer ES based on machine learning algorithms and other AI technologies are deployed to create better systems. An example is provided in Application Case 12.2.

VisiRule is an older ES company that remodeled its business over time. VisiRule (of the United Kingdom) provides easy-to-use diagramming tools to facilitate the construction of ES. Diagramming allows easier extraction and use of knowledge in expert systems.

The process of building the knowledge base can be seen on the left side of Figure 12.2. On the left-hand side, you can see the hybrid creation. Using a decision tree, the domain experts can cre- ate additional rules directly from relevant data (e.g.,

historical). In addition, rules can be created by machine learning (lower left side).

The right-hand side (upper corner) illustrates the hybrid delivery (consultation). Using interac- tive questions and answers the system can gener- ate advice. In addition, rules can be used to process data remotely and update the data repository. Note that the dual delivery option is based on machine learning’s ability to discover hidden patterns in data that can be used to form predictive decision models.

Application Case 12.2 VisiRule

Domain Expert draws rules as

decision tree using VisiRule Author

Expert Systems deployed using

interactive questionnaire

Rules are created from data using Machine Learning

Rules are used to process data remotely and

update database

Human Expert Interactive

Machine Learning

Data- Driven

Hybrid Creation

Hybrid DeliveryVisiRule

FIGURE 12.2 The Process of Recommendation Systems.

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 657

Three major AI types of applications that overcome the earlier discussed limitations of RS are chatbots, virtual personal assistants, and robo advisors, which are presented next in this chapter. Other AI technologies that perform similar activities are presented in Chapters 4 to 9. Most notable is IBM Watson (Chapter 6); some of its advising capabilities are similar to those of ES but are much superior.

Another similar AI technology, the recommendation system, is presented next. Its newer variations use machine learning and IBM Watson Analytics.

Recommendation Systems

A heavily used knowledge system for recommending one-to-one targeted products or services is the recommendation system, also known as recommender system or recommendation engine. Such a system tries to predict the importance (rating or pref- erence) that a user will attach to a product or service. Once the rating is known, a vendor knows users’ tastes and preferences and can match and recommend a prod- uct or service to the user. For comprehensive coverage, see Aggarwal (2016). For a comprehensive tutorial and case study, see analyticsvidhya.com/blood/2015/10/ recommendation-engines/.

Recommendation systems are very common and are used in many areas. Top appli- cations include movies, music, and books. However, there are also systems for travel, res- taurants, insurance, and online dating. The recommendations are typically given in rank order. Online recommendations are preferred by many people over regular searches, which are less personalized, slower, and sometimes less accurate.

BENEFITS OF RECOMMENDATION SYSTEMS Using these systems may result in substan- tial benefits both to buyers and sellers (see Makadia, 2018).

VisiRule also provides chatbots for improving the interactive part of the process and supplies an interactive map. According to the company’s Web site visirule.co.uk/, the major benefits of the product are:

All-in-all, VisiRule provides a comprehensive AI-based expert system.

Source: Courtesy of VisiRule Corp. UK. Used with permission.

Questions for Case 12.2

1. Which of the limitations of early ES have been solved by the VisiRule system?

2. Compare Figures 12.2 and 12.1. What are the dif- ferences between the creation (Fig. 12.2) and the development (Fig. 12.1) subsystems?

3. Compare Figures 12.2 and 12.1. What are the dif- ferences between the delivery (Fig. 12.2) and the consultation (Fig. 12.1) subsystems?

4. Identify all AI technologies and list their contri- bution to the VisiRule system.

5. List some benefits of this ES to users.

• It is code-free; no programming is needed. • The diagrams are drawn by human experts or

induced automatically from data. • It contains self-assessment tools with report

generation and document production. • The generated knowledge can be easily ex-

ecuted as XML code. • It provides explanation and justification. • The interactive expert advice attracts new

customers. • It can be used for training and advising em-

ployees. • Companies can easily access the corporate

knowledge repository. • The charts to use VisiRule authoring tools

are created with ease using flowcharting and decision trees.

• The charts allow creation of models that can be immediately executed and validated.

658 Part IV • Robotics, Social Networks, AI and IoT

Benefits to customers are:

According to ir.netflix.com, Netflix is (Spring 2018 data) the world’s leading Internet television network with more than 118 million members in over 190 countries enjoying more than 150 million hours of

TV shows and movies per day, including original series, documentaries, and feature films. Members can view unlimited shows without commercials for a monthly fee.

Application Case 12.3 Netflix Recommender: A Critical Success Factor

• Personalization. They receive recommendations that are very close to fulfilling what they like or need. This depends, of course, on the quality of the method used.

• Discovery. They may receive recommendations for products that they did not even know existed but were what they really need.

• Customer satisfaction. With repeated recommendations tends to increase. • Reports. Some recommenders provide reports and others provide explanations

about the selected products. • Increased dialog with sellers. Because recommendations may come with expla-

nations, buyers may want more interactions with the sellers.

Benefits to sellers are:

• Higher conversion rate. With personalized product recommendations, buyers tend to buy more.

• Increased cross-sell. Recommendation systems can suggest additional products. Amazon.com, for example, shows other products that “people bought together with the product you ordered.”

• Increased customer loyalty. As benefits to customers increase, their loyalty to the seller increases.

• Enabling of mass customization. This provides more information on potential customized orders.

Several methods are (or were) used for building recommendation systems. Two classic methods are collaborative filtering and content-based filtering.

COLLABORATIVE FILTERING This method builds a model that summarizes the past be- havior of shoppers, how they surf the Internet, what they were looking for, what they have purchased, and how much they like (rate) the products. Furthermore, collaborative filtering considers what shoppers with similar profiles bought and how they rated their purchases. From this, the method uses AI algorithms to predict the preference of both old and new customers. Then, the computer program makes a recommendation.

CONTENT-BASED FILTERING This technique allows vendors to identify preferences by the attributes of the product(s) that customers have bought or intend to buy. Knowing these preferences, the vendor recommends to customers products with similar attributes. For instance, the system may recommend a text-mining book to a customer who has shown interest in data mining, or action movies after a consumer has rented one in this category.

Each of these types has advantages and limitations (see example at en.wikipedia.org/ wiki/Recommender_system). Sometimes the two are combined into a unified method.

Several other filtering methods exist. Examples include rule-based filtering and activity-based filtering. Newer methods include machine learning and other AI technolo- gies, as illustrated in Application Case 12.3.

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 659

The Challenges

Netflix has several million titles and now produces its own shows. The large titles inventory often creates a problem for customers who have difficulty determin- ing which offerings they want to watch. An additional challenge is that Netflix expanded its business from the United States and Canada to 190 other countries. Netflix operates in a very competitive environment in which large players such as Apple, Amazon.com, and Google operate. Netflix was looking for a way to distinguish itself from the competition by making useful recommendations to its customers.

The Original Recommendation Engine

Netflix originally was solely a mail-order business for DVDs. At that time, it encountered inventory prob- lems due to its customers’ difficulties in determining which DVDs to rent. The solution was to develop a recommendation engine (called Cinematch) that told subscribers which titles they probably would like. Cinematch used data mining tools to sift through a database of billions of film ratings and customers’ rental histories. Using proprietary algorithms, it recom- mended rentals to customers. The recommendation was accomplished by comparing an individual’s likes, dislikes, and preferences against those of people with similar tastes, using a variant of collaborative filtering. Cinematch was like the geeky clerk at a small movie store who sets aside titles he knows you will like and suggests them to you when you visit the store.

To improve Cinematch’s accuracy, Netflix began a contest in October 2016, offering $1 million to the first person or team that will write a program that would increase Cinematch’s prediction accuracy by at least 10 percent. The company understood that this would take quite some time; therefore, it offered a $50,000 Progress Prize each year in which the contest was conducted. After more than two years of com- petition, the grand prize went to Bellkor’s Pragmatic Chaos team, a combination of two runner-up teams.

To learn how the movie recommendation algo- rithms work, see quora.com/How-does-the-Netflix- movie-recommendation-algorithm-work/.

The New Era

As time passed, Netflix moved to the streaming business and then to Internet TV. Also, the spread of cloud technology enabled improvement in the

recommendation system. The new system stopped making recommendations based on what people have seen in the past. Instead, it is using Amazon’s cloud to mimic the human brain in order to find what people really like in their favorite movies and shows. The system is based on AI and its technology of deep learning. The company can now visualize Big Data and draw insights for the recommenda- tions. The analysis is also used in creating the com- pany’s productions. Another major change dealt with the transformation to the global arena. In the past, recommendations had been based on information collected in the country (or region) where users live. The recommendations were based on what other people in the same country enjoyed. This approach did not work well in the global environment due to cultural, political, and social differences. The modi- fied system considers what people who live in many countries view and their viewing habits and likes.

Implementation of the new system was dif- ficult, especially when a new country or region was added. Recommendations were initially made without knowing much about the new customers. It took 70 engineers and a year of work to modify the recommendation system. For details, see Popper (2016).

The Results

As a result of implementing its recommender sys- tem, Netflix has seen very fast growth in sales and membership. The benefits include the following:

• Effective recommendations. Many Netflix members select their movies based on recom- mendations tailored to their individual tastes.

• Customer satisfaction. More than 90 per- cent of Netflix members say they are so satisfied with the Netflix service that they recommend it to family members and friends.

• Finance. The number of Netflix members has grown from 10 million in 2008 to 118 million in 2018. Its sales and profits are climbing steadily. In spring 2018, Netflix stock sold for over $400 per share compared with $140 a year earlier.

Sources: Based on Popper (2016), Arora (2016), and StartUp (2016).

(Continued )

660 Part IV • Robotics, Social Networks, AI and IoT

Questions for Case 12.3

1. Why is the recommender system useful? (Relate it to one-to-one targeted marketing.)

2. Explain how recommendations are generated.

3. Amazon disclosed its recommendation algo- rithms to the public but Netflix did not. Why?

4. Research the research activities that attempt to “mimic the human brain.”

5. Explain the changes due to the globalization of the company.

Application Case 12.3 (Continued)

u SECTION 12.2 REVIEW QUESTIONS

1. Define expert systems. 2. What is the major objective of ES? 3. Describe experts. 4. What is expertise? 5. List some areas especially amenable to ES. 6. List the major components of ES and describe each briefly. 7. Why is ES usage on the decline? 8. Define recommendation systems and describe their operations and benefits. 9. How do recommendation systems relate to AI?

12.3 CONCEPTS, DRIVERS, AND BENEFITS OF CHATBOTS

The world is now infested with chatbots. According to 2017 data (Knight, 2017c), 60 percent of millennials have already used chatbots and 53 percent of those who have not used them are interested in doing so. Millennials are not the only generation using chat- bots, although they may use them more than others. What chatbots are and what they do is the subject of this section.

What Is a Chatbot?

Short for chat robot, a chatbot, also known as a “bot” or “robo,” is a computerized service that enables easy conversations between humans and humanlike computerized robots or image characters, sometimes over the Internet. The conversations can be in writing, and more and more are by voice and images. The conversations frequently involve short questions and answers and are executed in a natural language. More intel- ligent chatbots are equipped with NLPs, so the computer can understand unstructured dialog. Interactions also can occur by taking or uploading images (e.g., as is done by Samsung Bixby on the Samsung S8 and 8). Some companies experiment with learning chatbots, which gain more knowledge with their accumulated experience. The ability of the computer to converse with a human is provided by a knowledge system (e.g., rule- based) and a natural language understanding capability. The service is often available on messaging services such as Facebook Messenger or WeChat, and on Twitter.

Chatbot Evolution

Chatbots originated decades ago. They were simple ES that enabled machines to answer questions posted by users. The first known such machine was Eliza (en.wikipedia. org/wiki/ELIZA). Eliza and similar machines were developed to work in Q&A mode.

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 661

The machine evaluated each question, usually to be found in a bank of FAQs, and gener- ated an answer matched to each question. Obviously, if the question was not in the FAQ collection, the machine provided irrelevant answers. In addition, because the power of the natural language understanding was limited, some questions were misunderstood and the answers were at times at best entertaining. Therefore, many companies opted to use live chats, some with inexpensive labor, organized as call centers around the globe. For more about Eliza’s current generation, and how to build it, see search.cpan.org/dist/ Chatbot-Eliza/Chatbot/Eliza.pm/. Chatbot use and reputation are rapidly increasing globally.

Example

Sophia is a chatbot created in Hong Kong and was awarded citizenship by Saudi Arabia in October 2017. Because she is not a Muslim, she is not wearing a hijab. She can answer many questions. For details, see newsweek.com/Saudi-arabia-robot-sophia-muslim-694152/.

TYPES OF BOTS Bots can be classified by their capabilities; three classes follow:

1. Regular bots. These are essentially conversational intelligent agents (Chapter 2). They can do simple, usually repetitive, tasks for their owners, such as showing their bank’s debits, helping them to purchase goods online, and to sell or buy stocks online.

2. Chatbots. In this category, we include more capable bots, for example, those that can stimulate conversations with people. This chapter deals mainly with chatbots.

3. Intelligent bots. These have a knowledge base that is improving with experience. That is, these bots can learn, for example, a customer’s preferences (e.g., like Alexa and some robo advisors).

A major limitation of the older types of bots was that updating their knowledge base was both slow and expensive. They were developed for specific narrow domains and/or specific users. It took many years to improve the supporting technology. NLP has become better and better. Knowledge bases are updated today in the “cloud” in a central location; the knowledge is shared by many users so the cost per user is reduced.

The stored knowledge is matched with questions asked by users. The answers by the machines have improved dramatically. Since 2000, we have seen more and more capable AI machines for Q&A dialogs. Around 2010, conversational AI machines were named chatbots and later were developed into virtual personal assistants, championed by Amazon’s Alexa.

DRIVERS OF CHATBOTS The major drivers are:

• Developers are creating powerful tools to build chatbots quickly and inexpensively with useful functionalities.

• The quality of chatbots is improving, so conversations are getting more useful to users.

• Demand for chatbots is growing due to their potential cost reduction and improved customer service and marketing services, which are provided 24/7.

• Use of chatbots allows rapid growth without the need to hire and train many cus- tomer service employees.

• Using chatbots, companies can utilize the messaging systems and related apps that are the darlings for consumers, especially younger ones.

662 Part IV • Robotics, Social Networks, AI and IoT

Components of Chatbots and the Process of Their Use

The major components of chatbots are:

Messenger, Webpage,

Mobile

Platforms

Voice, texting, video, VR

Mode of communication Question,

Order, Menu

Men– Machine Interface

NLP

Robot Chatbot

Natural Language Generation

Cloud services

Analytics Data Knowledge

Response

FIGURE 12.3 The Process of Chatting with Chatbots.

• A person (client). • A computer, avatar, or robot (the AI machine). • A knowledge base that can be embedded in the machine or available and con-

nected to the “cloud.” • A human-computer interface that provides the dialog for written or voice modes. • An NLP that enables the machine to understand natural language.

Advanced chatbots can also understand human gestures, cues, and voice variations.

PERSON-MACHINE INTERACTION PROCESS The components just listed provide the framework for people-bot conversation. Figure 12.3 shows the conversation process.

• A person (left side of the figure) needs to find some information, or need some help. • The person asks a related question from the bot by voice, texting, and so on. • NLP translates the question to machine language. • The chatbot transfers the question to cloud services. • The cloud contains a knowledge base, business logic, and analytics (if appropri-

ate) to craft a response to the question. • The response is transferred to a natural language generation program and then to

the person who asked the question in the preferred mode of dialog.

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 663

Drivers and Benefits

Chatbot use is driven by the following forces and benefits:

• The need to cut costs. • The increasing capabilities of AI, especially NLP and voice technologies. • The ability to converse in different languages (via machine translation). • The increased quality and capability of captured knowledge. • The push of devices by vendors (e.g., virtual personal assistants such as Alexa

from Amazon and Google Assistant from Alphabet). • Its use for providing superb and economic customer service and conducting mar-

ket research. • Its use for text and image recognition. • Its use to facilitate shopping. • Its support of decision making.

Chatbots and similar AI machines have been improved over time. Chatbots are beneficial to both users and organizations. For example, several hospitals employ robot reception- ists to direct patients to their place of treatment. Zora Robotics created a robot named Nao to act as a chatting companion for people who are sick or elderly. The bot acts, for example, as a form of therapy for those suffering from dementia.

Note: For some limitations of chatbots, see Section 12.7.

Representative Chatbots from Around the World

For a chatbot directory of the more than 1,250 bots in 53 countries as of April 2018, see chatbots.org/ and at botlist.co/bots/. Examples of chatbots and what they can do from chatbot.org/ are provided here:

• RoboCoke. This is a party and music recommendation bot created for Coca-Cola in Hungary.

• Kip. This shopping helper is available on Slack (a messaging platform). Tell Kip what you want to buy, and Kip will find it and even buy it for you.

• Walnut. This chatbot can discover skills relevant to you and help you learn them. It analyzes a large set of data points to discover the skills.

• Ride sharing by Taxi Bot. If you are not sure whether Uber, Lyft, Grab, or Comfort DelGro is the cheapest service, you can ask this bot. In addition, you can get current promo codes.

• ShopiiBot. When you send a picture of a product to this bot, it will find similar ones in seconds. Alternatively, tell ShopiiBot what kind of product you are looking for at what price, and it will find the best one for you.

• Concerning desired trips. It can answer questions regarding events, restaurants, and attractions in major destinations.

• BO.T. The first Bolivian chatbot, it talks to you (in Spanish) and answers your questions about Bolivia, its culture, geography, society, and more.

• Hazie. She is your digital assistant that aims to close the gap between you and your next career move. Job seekers can converse directly with Hazie just as they do with a job placement agent or friends.

• Green Card. This Visabot product helps users to properly file requests for Green Cards in the United States.

• Zoom. Zoom.ai (botlist.co/bots/369-zoomai), an automated virtual assistant, is for everyone in the workplace.

• Akita. This chatbot (botlist.co/bots/1314-akita) can connect you to businesses in your area.

664 Part IV • Robotics, Social Networks, AI and IoT

As you can see, chatbots can be used for many different tasks. Morgan (2017) classifies bots into the following categories: education, banking, insurance, retail, travel, health- care, and customer experience.

MAJOR CATEGORIES OF CHATBOTS’ APPLICATIONS Chatbots are used today for many purposes and in many industries and countries. We divide the applications into the fol- lowing categories:

• Chatbots for enterprise activities, including communication, collaboration, cus- tomer service, and sales (such as in the opening vignette). These are described in Section 12.4.

• Chatbots that act as personal assistants. These are presented in Section 12.5. • Chatbots that act as advisors, mostly on finance-related topics (Section 12.6).

For a discussion of these categories, see Ferron (2017).

u SECTION 12.3 REVIEW QUESTIONS

1. Define chatbots and describe their use. 2. List the major components of chatbots. 3. What are the major drivers of chatbot technology? 4. How do chatbots work? 5. Why are chatbots considered AI machines?

12.4 ENTERPRISE CHATBOTS

Chatbots play a major role in enterprises, both in external and internal applications. Some believe that chatbots can fundamentally change the way that business is done.

The Interest of Enterprises in Chatbots

The benefits of chatbots to enterprises are increasing rapidly, making dialog less expen- sive and more consistent. Chatbots can interact with customers and business partners more efficiently, are available anytime, and can be reached from anywhere. Businesses are clearly paying attention to the chatbot revolution. According to Beaver (2016), busi- nesses should look at enterprise bots for the following reasons:

• “AI has reached a stage in which chatbots can have increasingly engaging and human conversations, allowing businesses to leverage the inexpensive and wide- reaching technology to engage with more consumers.

• Chatbots are particularly well suited for mobile-perhaps more so than apps. Messaging is at the heart of the mobile experience, as the rapid adoption of chat app demonstrates.

• The chatbot ecosystem is already robust, encompassing many different third-party chat bots, native bots, distribution channels, and enabling technology companies.

• Chatbots could be lucrative for messaging apps and the developers who build bots for these platforms, similar to how app stores have developed into moneymaking ecosystems.”

A study conducted in 2016 found that 80 percent of businesses want chatbots by 2020 businessinsider.com/80-of-businesses-want-chatbots-by-2020-2016-12. For more opportunities in marketing, see Knight (2017a).

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 665

Enterprise Chatbots: Marketing and Customer Experience

As we saw in the opening vignette to this chapter and will see in in the several ex- amples later in this chapter, chatbots are very useful in providing marketing and customer service (e.g., Mah, 2016), obtaining sales leads, persuading customers to buy products and services, providing critical information to potential buyers, opti- mizing advertising campaigns (e.g., a bot named Baroj; see Radu, 2016), and much more. Customers want to do business on the app they are already in. For this rea- son, many bots are on Facebook Messenger, Snapchat, WhatsApp, Kik, and WeChat. Using voice and texting, it is possible to provide personalization as well as superb customer experience. Chatbots can enable vendors to improve personal relationships with customers.

In addition to the marketing areas, plenty of chatbots are in areas such as finan- cial (e.g., banks) and HRM services as well as production and operation management for communication, collaboration, and other external and internal enterprise business processes. In general, enterprises use chatbots on messaging platforms to run mar- keting campaigns (e.g., see the opening vignette) and to provide superb customer experience.

IMPROVING THE CUSTOMER EXPERIENCE Enterprise chatbots create improved customer experience by providing a conversation platform for quick and 24/7 contact with en- terprises. When customers benefit from the system, they are more inclined to buy and promote a specific brand. Chatbots can also supplement humans in providing improved customer experience.

EXAMPLES OF ENTERPRISE CHATBOTS Schlicht (2016) provides a beginner’s guide to chatbots. He presents the following hypothetical example about today’s shopping at Nordstrom (a large department store) versus the use of chatbots.

If you wanted to buy shoes from Nordstrom online, you would go to their Web site, look around until you find the shoes you wanted, and then you would purchase them. If Nordstrom makes a bot, which I am sure they will, you would simply be able to message Nordstrom on Facebook. It would ask you what you are looking for and you would simply . . . tell it.

Instead of browsing a Web site, you will have a conversation with the Nordstrom bot, mirroring the type of experience you would get when you go into the retail store.

Three additional examples follow:

Example 1: LinkedIn

LinkedIn is introducing chatbots that conduct tasks such as comparing the calendars of people participating in meetings and suggesting meeting times and places. For details, see CBS News (2016).

Example 2: Mastercard

Mastercard has two bots based on massaging platforms, one bot for banks and another bot for merchants.

666 Part IV • Robotics, Social Networks, AI and IoT

Example 3: Coca-Cola

Customers worldwide can chat with Coca-Cola bots via Facebook Messenger. The bots make users feel good with conversations that are increasingly becoming personalized. The bots collect customers’ data, including their interests, problems, local dialect, and at- titudes and then can target advertisements tailored to each user.

A 5-min. video about Facebook is available at cnbc.com/2016/04/13/ why-facebook-is-going-all-in-on-chatbots.html. It provides a Q&A session with David Marcus describing Facebook’s increasing interest in chatbots.

WHY USE MESSAGING SERVICES? So far, we have noted that enterprises are using mes- saging services such as Facebook Messenger, WeChat, Kik, Skype, and WhatsApp. The reason is that in 2017, more than 2.6 billion people were chatting on messaging services. Messaging is becoming the most widespread digital behavior. WeChat of China was the first to commercialize its service by offering “chat with business” capabilities as illustrated in Application Case 12.4.

FACEBOOK’S CHATBOTS Following the example of WeChat, Facebook launched users’ conversations with businesses’s chatbots on a large scale on Messenger, suggesting that users could message a business just the way they would message a friend. The service allows businesses to conduct text exchanges with users. In addition, the bots have a

WeChat is a very large comprehensive messaging service in China and other countries with about 1 billion members in early 2018. It pioneered the use of bots in 2013 (see mp.weixin.qq.com). Users can use the chatbot for activities such as the following:

Griffiths (2016) has provided information con- cerning a Chinese online fashion flash sales com- pany, Meici. The company used its WeChat account to gather information related to sales. Each time new users followed Meici’s account, a welcome mes- sage instructed them on how to trigger resources. WeChat is available in English and other languages worldwide due to its usefulness. Facebook installed similar capabilities in 2015.

Questions for Case 12.4

1. Find some recent activities that WeChat does.

2. What makes this chatbot so unique?

3. Compare the bot of WeChat to bots offered by Facebook.

Application Case 12.4 WeChat’s Super Chatbot

• Conduct market research. • Get information and recommendations on

products and services. • Launch a start-up on WeChat (you can make

your own bot on WeChat for this purpose).

• Hail a taxi. • Order food to be delivered. • Buy movie tickets and other items. • Customize and order a pair of Nikes. • Send an order to the nearest Starbucks. • Track your daily fitness progress. • Shop Burberry’s latest collection. • Book doctor appointments. • Pay your water bill. • Host a business conference call. • Send voice messages, emoticons, and snap-

shots to friends. • Send voice messages to communicate with

businesses. • Communicate and engage with customers. • Provide a framework for teamwork and col-

laboration.

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 667

learning ability that enables them to accurately analyze people’s input and provide cor- rect responses. Overall, as of early 2018, there were more than 30,000 company bots on Facebook Messenger. Some companies use Messenger bots to recognize faces in pictures, suggesting recipients for targeted ads. According to Guynn (2016), Facebook allows soft- ware developers access to its tools that build its personal assistant called “M,” which com- bines AI with a human touch for tasks such as ordering food or sending flowers. Using the M tools, developers can build applications for Messenger that can have an increased understanding of requests made in natural languages. A major benefit of these bots for Facebook is their collection of data and creation of profiles of users.

The following is another example of how the use of chatbots is facilitating customer service and marketing (Application Case 12.5).

Vera Gold Mark is a real estate developer of luxury high rises in Punjab, India.

The Problem

Vera Gold Mark (VGM) is active in a very competi- tive market. As a developer of luxury apartments, which are usually expensive, it must try to attract many potential buyers and thus needs as many sales leads as possible at a reasonable cost. Chatting live with potential customers can be expensive since it requires very knowledgeable and courteous agents available 24/7. VGM has a large inventory of units that must be sold as soon as possible.

The Solution

VGM decided to use chatbots to supplement or replace expensive manual live chats. These work in the fol- lowing ways. Buyers may click on the “chat with the robot” button on the company’s Facebook page, and receive any information they need. The chat helps VGM promote its available products. When they click, users are able to chat and get information about pric- ing, delivery dates, construction sites, and much more for VGM projects. Users can also tweet. The chatbots provide answers about the projects. Facebook pro- vides VGM access to potential buyers’ profiles (with users’ permission), which VGM sales teams can use to refine sales strategies. The system is available 24/7. Voice communication is coming soon (2018).

The Results

VGM is now viewed in a very positive way and is considered to be very professional. VGM is getting good reviews for its customer service. The builder is considered more honest and unbiased because it provides written answers and promises to cus- tomers. Salespeople at VGM get an increased num- ber of sales leads, and because they know more about prospective customers, they can better align them with units (optimal fit). The system is also able to attract international buyers without increas- ing cost. Because the system is available 24/7, global buyers can easily evaluate VGM’s available condominiums.

The chatbot is also used as a teaching tool for new employees. At the time that this case was writ- ten, no financial data were available.

The technology is available to other build- ers from Kenyt Technologies of India kenyt.com, which provides the smart real estate chatbot.

Sources: Based on Garg (2017) and facebook.com/ veragoldmark/ (accessed April 2018).

Questions for Case 12.5

1. List the benefits to VGM.

2. List the benefits to buyers.

3. What is the role of Kenyt Technologies?

Application Case 12.5 How Vera Gold Mark Uses Chatbots to Increase Sales

Chatbots Magazine provides a three-part overview on the use of chatbots for re- tail and e-commerce. For details, see chatbotsmagazine.com/chatbots-for-retail-and- e-commerce-part-three-c112a89c0b48.

668 Part IV • Robotics, Social Networks, AI and IoT

Enterprise Chatbots: Financial Services

The second area in which enterprise bots are active is financial services. Here we briefly discuss their use in banking. In Section 12.6, we present the robo financial advisors for investment.

BANKING A 2017 survey (Morgan, 2017) found that most people in the United States will bank via chatbots by 2019. Chatbots can use predictive analytics and cognitive messaging to perform tasks such as making payments. They can inform customers about personalized deals. Banks’ credit cards can be advertised via chatbots on Facebook Messenger. It seems that customers prefer to deal with chatbots rather than with salespeople who can be pushy.

Examples

POSB of Singapore has an AI-driven bot on Facebook Messenger. The bot was created with the help of Kasisto, Inc. of the United States. Using actual Q&A sessions, it took IT workers 11,000 hours to create the bot. Its knowledge base was tested and verified. The bot can learn to improve its performance. Known as POSB digi-bank virtual assistant, the service is accessed via Messenger. Customers save time rather than waiting for human customer service. In the future, the service will be available on other messaging plat- forms. For details, see Nur (2017).

A similar application in Singapore is used by Citi Bank (by Citi Group). It can an- swer FAQs about people’s accounts in a natural language (English). The bank is adding progressively more capabilities to its bot.

A generic banking bot is Verbal Access (from North Side Co.) that provides recom- mendations for banking services (see Hunt, 2017).

Enterprise Chatbots: Service Industries

Chatbots are used extensively in many services. We provide several examples in the fol- lowing sections.

HEALTHCARE Chatbots are extremely active in the healthcare area, helping millions of people worldwide (Larson, 2016). Here are a few examples:

• Robot receptionists direct patients to departments in hospitals. (Similar services are available at airports, hotels, universities, government offices, and private and other public organizations.)

• Several chatbots are chatty companions for people who are elderly and sick (e.g., Zora Robotics).

• Chatbots are used in telemedicine; patients converse with doctors and healthcare professionals who are in different locations. For example, the Chinese company Baidu developed the Melody chatbot for this purpose.

• Chatbots can connect patients quickly and easily with information they need. • Important services in the healthcare field are currently provided by IBM Watson

(Chapter 6).

For more on bots for healthcare, see the end of Section 12.6.

EDUCATION Chatbot tutors are used in several countries to teach subjects ranging from English (in Korea) to mathematics (in Russia). One thing is certain: The chatbot treats all students equally. Students like the chatbots in online education as well. Machine transla- tion of languages will enable students to take online classes in languages other than their own. Finally, chatbots can be used as private tutors.

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 669

GOVERNMENT According to Lacheca (2017), chatbots are spreading in government as a new dialog tool for use by the public. The most popular use is in providing access to government information and answering government-related questions.

TRAVEL AND HOSPITALITY Chatbots are working as tour guides in several countries (e.g., Norway). They are not only cheaper (or free) but also may know more than some human guides. Chatbots work as guides in several hotels in Japan. In hotels, they act as concierges, providing information and personalized recommendations (e.g., about res- taurants). Chatbots can arrange reservations for hotel rooms, meals, and events. In busy hotels, there is frequently a wait for human concierges; chatbots are available on smart- phones all the time. As with other computer services, the chatbots are fast, inexpensive, easy to reach, and always nice. They give excellent customer experience.

An example of external travel service is given in Application Case 12.6.

Chatbot Platforms

CHATBOTS INSIDE ENTERPRISES So far we have seen chatbots that are working in the external side of enterprises, mostly in customer care and marketing (e.g., the opening vignette). However, companies lately have started to use chatbots to automate tasks for supporting internal communication, collaboration, and business processes. According to

Background

The air travel business is very competitive, espe- cially in Europe. There is a clear trend for younger customers to use wireless devices as well as social media sites and chatting. Customers like to com- municate with travel businesses by using their pre- ferred technology via their preferred platforms. Most popular is Facebook Messenger, where over 1.2 bil- lion people chat, many times via their smartphones. These users today interact not only among them- selves but also with the business world.

Messaging platforms such as Messenger, WhatsApp, and WeChat are becoming the norm for this customer group. Vendors are building smart apps for the messaging platforms including bots.

Transavia’s Bot

Learning from other companies, Transavia decided to create a bot on Facebook Messenger. To do so, it hired the IT consultant Cognizant Digital Business unit, called Mirabean, which specializes in conversa- tion interfaces, especially via bots. Transavia’s activi- ties business processes, marketing, and customer

care were combined with Mirabean’s technological experience to enable a quick deployment of the bot in weeks. It now enables real-time dialog with customers. The first application is Transavia Flight Search, which provides flight information as well as the ability to buy tickets. The system is now inte- grated with business processes that facilitate other transactions via the bot. Giving customers their digi- tal tool of choice enables Transavia to increase mar- ket share and to drive growth.

Note that KLM, the owner of Transavia, was the first European airline that implemented a similar chatbot on Facebook Messenger in 2016.

Sources: Compiled from Cognizant (2017) and transavia.com.

Questions for Case 12.6

1. What drives consumer preference for mobile devices and chat?

2. Why was the bot placed on Facebook Messenger?

3. What were the benefits of using Cognizant?

4. What is the advantage of buying a ticket from a bot rather than from an online store?

Application Case 12.6 Transavia Airlines Uses Bots for Communication and Customer Care Delivery

670 Part IV • Robotics, Social Networks, AI and IoT

Hunt (2017), “Enterprise and internal chatbots are revolutionizing the way companies do business.” Chatbots in enterprises can do many tasks and support decision-making activi- ties. For examples, see Newlands (2017a). Chatbots can cut costs, increase productivity, assist working groups, and foster relationships with business partners. Representative examples of chatbot tasks are:

• Help with project management. • Handle data entry. • Conduct scheduling. • Streamline payments with partners. • Advise on authorization of funds. • Monitor work and workers. • Analyze internal Big Data. • Find discounted and less expensive products. • Simplify interactions. • Facilitate data-driven strategy. • Use machine learning.

Facilitate and manage personal finance.

Given the large number of bots, it is not surprising that many developers started to offer tools and platforms to assist in building chatbots as discussed in Technology Insights 12.1.

TECHNOLOGY INSIGHTS 12.1 Chatbots’ Platform Providers

Several companies provide platforms for building enterprise chatbots. The companies can con- struct chatbots fairly easily using these tools for their entry into popular messaging platforms or for their Web sites. Some of the tools have machine-learning capability to ensure that the bots learn with every interaction. According to Hunt (2017), these are some popular vendors:

1. ChattyPeople. This chatbot builder assists in creating bots requiring minimal program- ming skills. It simply allows a business to link its social media pages to its ChattyPeople account. The created bot can: • Arrange for payments to or from social media contacts. • Use major payment providers such as Apple Pay and PayPal. • Recognize variations in keywords. • Support messaging.

2. Kudi. This financial helper allows people to make payments to vendors directly from their messaging apps, specifically, Messenger, Skype, and Telegram and through an Internet browser. Using the bot, users can: • Pay bills. • Set bill payment reminders. • Transfer money by sending text messages.

The bot is safe and it protects users’ privacy. Vendors can easily install it for use. 3. Twyla. This chatbot building platform is for improving existing customer care and of-

fering live chats. It acts as a messaging platform for customers who prefer to use chatting. The major objective is to free humans in HR departments from routine tasks.

The most popular platforms are:

• IBM Watson. This package uses a neural network of 1 billion words for excellent understanding of natural languages (e.g., English, Japanese). Watson provides free devel- opment tools, such as Java SDK, Node SDK, Pyton SDK, and iOS SDK.

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 671

For additional information about chatbot platforms for building enterprise chatbots, see entrepreneur.com/article/289788.

INDUSTRY-SPECIFIC BOTS As we have seen, bots can be specialists (e.g., for investment ad- vice, customer service) or industry-specific experts (e.g., banking, airlines). An interesting bot for the waste industry is Alto (from Bio Hi Tech Global), which enables users to communicate intelligently with industrial equipment. This helps owners of the equipment make decisions that improve performance levels, smooth maintenance routines, and facilitate communication.

Knowledge for Enterprise Chatbots

Knowledge for chatbots depends on their tasks. Most marketing and customer care bots require proprietary knowledge, which is usually generated and maintained in-house. This knowledge is similar to that of ES; in many cases, enterprise chatbots operate very simi- larly to ES except that the interface occurs in a natural language and frequently by voice. For example, the knowledge of Sephora’s bot (opening vignette) is specific to that com- pany and its products and is organized in a Q&A format.

On the other hand, chatbots that are used within the enterprise (e.g., to train em- ployees or to provide advice on security or compliance with government regulations) may not be company specific. A company can buy this knowledge and modify it to fit local situations and its specific needs (as is done in ES; e.g., see Exsys Inc.). Newer chat- bots use machine learning to extract knowledge from data.

PERSONAL ASSISTANTS IN THE ENTERPRISE Enterprise chatbots can also be virtual per- sonal assistants as will be described in Section 12.5. For example, these bots can answer work-related queries and help in increasing employees’ decision-making capabilities and productivity.

u SECTION 12.4 REVIEW QUESTIONS

1. Describe some marketing bots. 2. What can bots do for financial services? 3. How can bots assist shoppers? 4. List some benefits of enterprise chatbots. 5. Describe the sources of knowledge for enterprise chatbots.

• Microsoft’s Bot Framework. Similar to IBM, Microsoft offers a variety of tools translat- able into 30 languages. It is an open source. The system has three parts, Bot Connector, Developer Portal, and Bot Directory and is interconnected with Microsoft Language Understanding Intelligent Service (LUIS) that understands users’ intent. The system also includes active learning technology. A simplified tool is AZURE; see Section 12.7 and Afaq (2017). For a comparative table of 25 chatbots platforms, see Davydova (2017). For a list of other platforms, see Ismail (2017).

Sources: Compiled from Hunt (2017) and Davydova (2017).

DisCussion Questions

1. What is the difference between a regular enterprise bot and a platform?

2. Discuss the benefits of ChattyPeople.

3. Discuss the need for Kudi.

4. Discuss the reasons for consumers to prefer messaging platforms.

672 Part IV • Robotics, Social Networks, AI and IoT

12.5 VIRTUAL PERSONAL ASSISTANTS

In the previous section, we introduced enterprise chatbots that can be used to conduct conversations. In marketing and sales, they can facilitate customer relationship manage- ment (CRM, execute searches for customers, provide information, and execute many specific tasks in organizations for their customers and employees. For comprehensive coverage, including research issues, see Costa et al. (2018)).

An emerging type of chatbot is designed as a virtual personal assistant for both individuals and organizations. Known as a virtual personal assistant (VPA), this software agent helps people improve their work, assist in decision making, and fa- cilitate their lifestyle. VPAs are basically extensions of intelligent software agents that interact with people. VPAs are chatbots whose major objective is to help people better perform certain tasks. At this time, millions of people are using Siri with their Apple products, Google Assistant, and Amazon’s Alexa. The assistants’ knowledge bases are usually universal, and they are maintained centrally in the “cloud,” which makes them economical for a large number of users. Users can get assistance and advice from their virtual assistants anytime. In this section, we provide some interesting applications. The first set of applications involves virtual personal assistants, notably Amazon’s Alexa and Apple’s Siri and Google Assistant. O’Brien (2016) provides a discussion of what personal assistant chatbots can do for business. The second set (presented in Section 12.6) is about computer programs that act mostly as advisors on specific topics (mostly investments).

Assistant for Information Search

A major task of virtual personal assistants is to help users conduct a search by voice for information. Without the assistant, users need to surf the Internet to find information and many times abandon the search. In business situations, users can call a live cus- tomer service agent for assistance. This may be an expensive service for the vendors. Delegating the search to a machine may save sellers considerable money and make customers happy by not having to wait for the service. For example, Lenovo uses the noHold assistant in its Single Point of Search service to help customers find answers to their questions.

If You Were Mark Zuckerberg, Facebook CEO

While Siri and Alexa were in development, Zuckerberg decided to develop his own personal assistant to help him run his home and his work as the CEO of Facebook. He viewed this assistant as Jarvis from Iron Man. Zuckerberg trained the bot to recognize his voice and understand basic commands related to home appliances. The assistant can rec- ognize the faces of visitors and monitor the movement of Zuckerberg’s young daughter. For details, see Ulanoff (2016).

The essentials of this assistant can be seen in a 2:13 min. video at youtube.com/ watch?v=vvimBPJ3XGQ and one (5:01 min.) at youtube.com/watch?v=vPoT2vdVkVc, with the narration by Morgan Freeman. Today, similar assistants are available for a mini- mal fee or even for free. The most well-known such assistant is Amazon’s Alexa.

Amazon’s Alexa and Echo

Of the several virtual personal assistants, the one considered the best in 2018 was Alexa. She was developed by Amazon to compete with Apple’s Siri and is a superior prod- uct. (See Figure 12.4.) Alexa works with a smart speaker, such as Amazon’s Echo (to be described later).

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 673

Amazon’s Alexa is a cloud-based virtual personal voice assistant that can do many things such as:

FIGURE 12.4 Amazon’s Echo and Alexa. Source: McClatchy-Tribune/Tribune Content Agency LLC/ Alamy Stock Photo

• Answer questions in several domains. • Control smartphone operations with voice commands. • Provide real-time weather and traffic updates. • Control smart home appliances and other devices by using itself as a home auto-

mation hub. • Make to-do lists. • Arrange music in Playbox. • Set alarms. • Play audio books. • Control home automation devices, as well as home appliances (e.g., a microwave). • Analyze shopping lists. • Control a car’s devices. • Deliver proactive notification. • Shop for its user. • Make phone calls and send text messages.

Alexa has the ability to recognize different voices, so it can provide personalized responses. Also, she uses a mix of speech and touch to deliver news, hail an Uber, and play games. As time passes, her capabilities and skill grow. For more capabilities, which are ever-increasing, see Johnson (2017). For what Alexa can hear and remember and how she learns, see Oremus (2018).

Watch the 3:55 min. video of how Alexa works at youtube.com/ watch?v=jCtfRdqPlbw. For more tasks, see cnet.com/pictures/what-can- amazon-echo-and-alexa-do-pictures/, Mangalindan (2017), and tomsguide.com/ us/pictures-story/1012-alexa-tricks-and-easter-eggs.html.

674 Part IV • Robotics, Social Networks, AI and IoT

ALEXA’S SKILLS In addition to the standard (native) capabilities listed, people can use Alexa apps (referred to as Skills) to download customized capabilities to Alexa (via your smartphone). Skills are intended to teach Alexa something new.

The following are examples of Alexa’s Skills (Apps):

• Call Uber and find the cost of a ride. • Order a pizza. • Order take-out meals. • Obtain financial advice. • Start a person’s Hyundai Genesis car from inside her or his house (Korosec, 2016).

These skills are provided by third-party vendors; they are required to activate invo- cation commands. There are tens of thousands of them.

For example, a person can say, “Alexa, call Uber to pick me up at my office at 4:30 p.m.” For more on Amazon’s Alexa, see Kelly (2018); for its benefits, see Reisinger (2016).

Alexa is equipped with NLP user interface, so it can be activated by providing a voice command. This is done by combining the Alexa software with Amazon’s intelligent speaker, Echo.

ALEXA’S VOICE INTERFACE AND SPEAKERS Amazon has a family of three speakers (or voice communication devices for Alexa: Echo, Dot, and Tag. Alexa can be accessed by a Fire TV line and some non-Amazon devices. For the relationship between Alexa and Echo, see Gikas (2016).

AMAZON’S ECHO Echo is a hands-free intelligent (or smart) wireless speaker that is controlled by voice. It is the hardware companion of Alexa (a software product), so the two operate hand in hand. Echo is always on, always listening. When Echo hears a question, command, or request, it sends the audio to Alexa and from there up to the cloud. Amazon’s servers match responses to the questions, delivering them to Alexa as “responses to questions” in a split second. Amazon’s Alexa/Echo is now available in some Ford vehicles.

Amazon Echo Dot Amazon Echo Dot is the “little brother” of Echo. It offers full Alexa functionality but has only one very small speaker. It can be linked to any existing speaker systems to provide an Echo-like experience.

Amazon Echo Tap Amazon Echo Tap is another “little brother” of Echo that can be used on the go. It is completely wireless and portable and can be charged via a charg- ing dock.

Both Dot and Tap are less expensive than Echo, but they offer fewer functional- ities and lower quality. However, people who already have good home speakers can use Dot with them. For a discussion about the three speakers, see Trusted Review at trustedreviews.com/news/amazon-echo-show-vs-echo-2948302.

Note: Non-Amazon speakers for Alexa are available now (e.g., Eufy Genie, from third-party vendors); some are inexpensive.

Note: Alexa was smart enough earlier to admit that she did not know an answer, but today, she will make references to third-party sources for an answer she cannot make. For details and examples, see uk.finance.yahoo.com/news/alexa-recommend- third-party-skills-192700876.html.

ALEXA FOR THE ENTERPRISE While the initial use of Alexa was for individual consumers, her use for business has increased. WeWork Corp. developed a platform for helping com- panies to integrate an Alexa skill in meeting rooms, for example. For details, see Crook (2017), and yahoo.com/news/destiny-2-alexa-skills-let-140946575.html/.

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 675

Apple’s Siri

Siri (short for Speech Interpretation and Recognition Interface) is an intelligent virtual personal assistant and knowledge navigator. It is a part of Apple’s several operating systems. It can answer questions, make recommendations, and perform some actions by delegating requests to a set of Web services in the “cloud.” The software can adapt itself to the user’s individual language, search preferences with continuing use, and return per- sonalized results. Siri is available for free to iPhone and iPad users.

Siri can be integrated into Apple’s Siri Remote. Using CarPlay, Siri is available in some auto brands where it can be controlled by iPhone (5 and higher). Siri 2 is the 2017–2018 model.

VIV In 2016, Dag Kittlaus, the creator of Siri, introduced Viv, “an intelligent Interface for everything.” Viv is expected to be the next generation of intelligent virtual interac- tions (for details, see Matney, 2016). In contrast with other assistants, Viv is open to all developers (third-party ecosystem products). Viv is now a Samsung Company. In 2017, Samsung launched its own personal assistant for the Galaxy S8.

Google Assistant

Competition regarding virtual personal assistants is increasing with the improved ca- pabilities of Google Assistant, which was developed as a competitor to Siri to fit Android smartphones. An interesting demonstration of it is available at youtube.com/ watch?v=WTMbF0qYWVs; some advanced capabilities are illustrated in the video at youtube.com/watch?v=17rY2ogJQQs. For details, see Kelly (2016). The product im- proved dramatically in 2018 as shown in CES 2018 Conference.

Other Personal Assistants

Several other companies have virtual personal assistants. For example, Microsoft Cortana is well known. In September 2016, Microsoft combined Cortana and Bing (see Hachman, 2016). Alexa and Cortana now work together. Note that it is estimated that by the year 2022, voice-enabled personal assistants will reach 55 percent of all U.S. households. For this and the future of personal assistants, see Perez (2017).

Competition Among Large Tech Companies

Apple and Google have provided their personal assistants to hundreds of million users of their mobile devices. Microsoft has equipped over 250 million PCs with its personal assistant. Amazon’s Alexa/Echo sells many more assistants than others. The competition is on voice-controlled chatbots. Their competitors view them as “the biggest thing since the iPhone.”

Knowledge for Virtual Personal Assistants

As indicated earlier, the knowledge for virtual personal assistants is kept in the “cloud.” The reason is that the assistants are commodities, available to millions of users, and need to provide dynamic, updated information (e.g., weather conditions, news, stock prices). When the knowledge base is centralized, its maintenance is performed in one place. This is in contrast with the knowledge of many enterprise bots, for which updating is decen- tralized. Thus, Siri on an iPhone will always be updated for its general knowledge by AAPL. Knowledge for the skills of Alexa has to be maintained locally or by the third-party vendors that create them.

676 Part IV • Robotics, Social Networks, AI and IoT

u SECTION 12.5 REVIEW QUESTIONS

1. Describe an intelligent virtual personal assistant. 2. Describe the capabilities of Amazon’s Alexa. 3. Relate Amazon’s Alexa to Echo. 4. Describe Echo Dot and Tap. 5. Describe Apple’s Siri Google’s Assistant. 6. How is the knowledge of personal assistants maintained? 7. Explain the relationship between virtual personal assistants and chatbots.

12.6 CHATBOTS AS PROFESSIONAL ADVISORS (ROBO ADVISORS)

The personal assistants described in Section 12.5 can provide much information and ru- dimentary advice. A special category of virtual personal assistants is designed to provide personalized professional advice in specific domains. A major area for their activities is investment and portfolio management where robo advisors operate.

Robo Financial Advisors

It is known that the vast majority of “buy” and “sell” decisions of stock trading on the major exchanges, especially by financial institutions, are made by computers. However, computers can also manage an individual’s accounts in a personalized way.

According to an A. T. Kearney’s survey (reported by Regan, 2015), robo advisors are defined as online providers that offer automated, low-cost, personalized investment advisory services, usually through mobile platforms. These robo advisors use algorithms that allocate, deploy, rebalance, and trade investment products. Once enrolled for the robo service, individuals enter their investment objectives and preferences. Then, using advanced AI algorithms, the robo will offer alternative personalized investments for in- dividuals to choose from funds or exchange-traded funds [ETFs]. By conducting a dialog with the robo advisor, an AI program will refine the investment portfolio. This is all done digitally without having to talk to a live person. For details, see Keppel (2016).

Evolution of Financial Robo Advisors

The pioneering emergence of Betterment Inc. in 2010 (described later) was followed by several other companies (Future Advisor and Hedgeable in 2010 and Personal Capital, Wealthfront, and SigFig in 2011 and 2012). Other well-known companies (Schwab Intelligent Portfolios, Acorns, Vanguard RAS, and Ally) joined the crowd in 2014 and 2015. In 2016 and 2017, the brokerage houses of E*Trade and TD Ameritrade joined, as did Fidelity and Merrill Edge. There is no question that robo advisors are game-changing phenomena for the wealth management business, even though their performance so far has not been much different from that of traditional, manual, and financial services.

Robo advising companies try to cut costs by using ETFs, whose commission fees are significantly lower than that of mutual funds. Annual fees vary as does the minimum amount of required assets. Premium services are more expensive since they offer the op- portunity to consult human experts (advisors 2.0), which are described next.

Robo Advisors 2.0: Adding the Human Touch

As robo advisors matured, it became clear that sometimes they could not do an effective job by themselves. Therefore, in late 2016, several of the fully automated advisors started to add what they call the human touch (e.g., see Eule, 2017; Huang, 2017). Companies

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 677

are adding a human option, or partner with another company. For example, UBS Wealth Management Americas has partnered with pure robo advisor SigFig.

Robo advisors with human additions vary in expertise. For example, Betterment (Plus and Permission options), Schwab Intelligent Advisory, and Vanguard Personal Advisor Service use certified financial planners (CFPs); other companies offer less exper- tise. For details, see Huang (2017).

Application Case 12.7 describes how Betterment has added the human touch.

QUALITY OF ADVICE PROVIDED BY ROBO ADVISORS You may wonder how good the advice from robo advisors is. The answer is that it depends on their knowledge, the type of investments involved, the inference engine of the AI machine, and so on. However, remember that the robots are not biased and are consistent. They may prove to be even

As the pioneer of financial robo advisors in 2010, Betterment created an automated platform for wealth management. Since then, it has played a leading role in a growing industry. In 2017, the company controlled more than $9 billion in assets, yielding over an 11 percent return to its 200,000 members. Like other robo advisors, Betterment appeals to investors who do not want to manage their portfolio by themselves or pay the 2 to 3 percent annual fee charged by human advisors.

The company advertises the following benefits:

Premium Service—Adding the Human Touch

Like Amazon.com and Expedia, which started as pure online companies and later added physical commerce, in 2017 Betterment added what it calls a human touch; its Plus service is offered to custom- ers with assets of over $100,000 who are willing to pay an annual fee of 0.4 percent for this service. Using it, customers can interact with human advi- sors in addition to the automated bot. An even bet- ter service is the company’s Premium level, which requires $250,000 in assets and charges 0.5 percent in fees.

While the quality of the automated service is getting better with added knowledge (machine learning), complex situations that require human intervention still remain. This is where the Plus and Premium services enter the picture. Several com- petitors also have added the human touch to their offering.

Sources: Compiled from O’Shea (2017), Eule (2017), and betterment.com (accessed April 2018).

Questions for Case 12.7

1. What are Betterment’s benefits to investors?

2. Compare Betterment to its major competitors (see Eule, 2017).

3. What are the benefits of adding the human touch (i.e., compared to pure automation and only human service)?

4. Find some new information about Betterment. Write a report.

Application Case 12.7 Betterment, the Pioneer of Financial Robo Advisors

• Provides unlimited professional expert advice (by the bot) anytime and anywhere.

• Provides advice from bots that contain the knowledge of human investment advisors.

• Assists investors in making decisions of how much to invest.

• Helps investors figure out how much risk to take.

• Helps in lowering investment-related tax. • Provides actionable answers to questions. • Advises on college savings. • Helps plan for retirement. • Assists in mortgage management (e.g., refi-

nance). • Provides personalized service via the use of

investors’ goal-based analysis.

Betterment has no account minimum (compet- itors require up to $100,000).

Each investor’s portfolio is automatically adjusted to market conditions to meet his or her goals. All portfolios are built and managed by AI algorithms.

678 Part IV • Robotics, Social Networks, AI and IoT

better than humans at one of the most important aspects in investment advising: know how to legally minimize the related tax. This implies that institutional-grade tax-loss harvesting is now within the reach of all investors. By contrast, some people believe that it is difficult to replace investment brokers with robots. De Aenlle (2018) believes that humans are still dominating advisory services (see the example of Nordea Bank by Pohjanpalo, 2017).

For a list of the best robo advisors, see Eule (2017), O’Shea (2016), and investorjunkie.com/35919/roboadvisors. For comprehensive coverage of robo advi- sors in finance and investment, including the major companies in the advisory industry, see McClellan (2016).

An emerging commercial robo advisor is being developed at Cornell University under the name Gsphere. In addition, robo advisors appear in countries other than the United States (e.g., Marvelstone Capital in Singapore).

FINANCIAL INSTITUTIONS AND THEIR COMPETITION Several large financial institutions and banks have reacted to robo advisors by creating their own or partnering with them. It is difficult to assess the winners and losers in this competition because there are no suf- ficient long-term data. So far it seems that customers like robo advisors, basically because they cost as little as 10 percent of full-service human advisors. For a discussion and data, see Marino (2016). Note that some observers point to the danger of using robo advisors in a declining stock market due to their use of ETFs.

Managing Mutual Funds Using AI

Many institutions and some individual investors buy stocks using AI algorithms. Some people prefer to buy a mutual fund that picks its holding with AI. EquBot is such a fund (its symbol is AIEQ). Its 2017 performance was above average.

The AI algorithms used by EquBot can process 1 million pieces of data each day. They follow 6,000 companies. For details, see Ell (2018).

Other Professional Advisors

In addition to investment advisors, there are several other types of robo advisors ranging from travel to medicine to legal areas.

The following are examples of noninvestment advisors:

• Computer operations. To cut costs, major computer vendors (hardware and soft- ware) try to provide users with self-guides to solve encountered problems. If users cannot get help from the guides, they can contact live customer service agents. This service may not be available in real time, which can upset customers. Live agents are expensive, especially when provided 24/7. Therefore, companies are using in- teractive virtual advisors (or assistants).

As an example, Lenovo Computers use a generic bot called noHold’s AI to provide assistance to customers as a single point of help for conducting a search.

• Travel. Several companies provide advice on planning future national and inter- national trips.

For example, Utrip (utrip.com) helps plan European trips. Based on their stated objectives, travelers get recommendations for what to visit in certain destina- tions. The service is different from others in that it customizes trips.

• Medical and health advisors. A large number of health and medical care advi- sors operate in many countries. An example is Ad a Health of Germany. Founded in late 2017 as a chatbot, it assists people in activities such as deciphering their ail- ments and can connect patients to live physicians. This can be the future of health in adding bot-based patient-doctor collaboration.

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 679

A list of the top useful chatbots as of 2017 is provided by TalKing (2017). It includes:

• Health Tap acts like a medical doctor by providing a solution to common symptoms provided by patients.

• YourMd is similar to Health Tap. • Florence is a personal nurse available on Facebook Messenger.

Other bots include OneStopHealth, HealthBot, GYANT, Buoy, Bouylon, and Mewhat.

• Bots are acting as companions (e.g., Endurance for dementia patients). In Japan, bots that look and feel like dogs are very popular companions for elderly peo- ple. Several bots are designed to increase patient engagement. For example, Lovett (2018) reports that a bot for patient engagement increased patients’ response rate to a flu shot campaign by 30 percent. Finally, the classic pioneering bot, ELIZA, acted as a very naïve psychologist.

• Shopping advisors (shopbots). Shopbots can act as shopping advisors. An example is Shop Advisor (see shopadvisor.com/our-platform. It is a comprehen- sive platform that includes three components to help companies attract customers. The platform is a self-learning system that improves its operation over time. Its components are:

1. Product intelligence, which processes complex and diverse product data. It includes a competitive analysis.

2. Context intelligence, which collects and catalogs contextual data points about marketing facilities and inventories in different locations.

3. Shopper intelligence, which studies consumers’ actions related to different mag- azines, mobile apps, and Web sites.

There are thousands of other shopping advisors. Sephora (opening vignette) has several of them. There are chatbots for Mercedes cars and for top department stores such as Nordstrom, Saks, and DFS. The use of shopping chatbots is increasing rapidly due to the use of mobile shopping and mobile chatting on social networks. Marketers, as we stated earlier, can collect customer data and deliver targeted ads and customer service to specific customers.

Another trend that facilitates online shopping with the assistance of bots is the increase in the number of virtual personal shopping assistants. Users only have to tell Alexa by voice, for example, to buy something for them. Better than that, they can use their smartphones from anywhere to tell Alexa to go shopping. Ordering via voice directly from vendors (e.g., delivery of pizzas) is becoming popular. In addition to chatbots that operate by sellers, there are bots for providing advice on what and where to buy.

Example: Smart Assistant Shopping Bots

Shopping bots ask a few questions to understand what a customer needs and prefers. Then they recommend the best match for the customer. This makes customers feel they are receiving personalized service. The assistance simplifies the customer’s decision- making process. Smart assistants also offer advice on issues of concern to customers via Q&A conversations. For a guided test, go to a demo at smartassistant.com/advicebots. Note that these bots are essentially recommendation systems and that users need to ask for advice whereas other recommendation systems (e.g., that of Amazon.com) provide advice even when users do not ask for it.

680 Part IV • Robotics, Social Networks, AI and IoT

A well-known global shopping assistant in the area of fashion is Alibaba’s Fashion AI. It helps customers who shop in stores. When shoppers enter a fitting room, the AI Fashion Consultant goes into action. For details of how this is done, see Sun (2017).

Another type of shopping advisor works as a virtual personal advisor to shoppers. This type was developed from traditional e-commerce intelligent agents, such as bizrate. com and pricegrabber.com.

IBM Watson

Probably the most knowledgeable virtual advisor is IBM Watson (see Chapter 6). Some examples of its use follow:

• Macy’s developed a service, Macy’s On Call, to help customers navigate its physi- cal stores while they shop. Using location-based software, the app knows where they are in the store. By using smartphones, customers can ask questions regarding products and services in the stores and then receive a customized response from the chatbot.

• Watson can help physicians make a diagnosis (or verify one) quickly and suggest the best treatment. Watson’s Medical Advisor can analyze images very fast and look for things that physicians may miss. Watson already is used extensively in India where there is a large shortage of doctors.

• Deep Thunder provides accurate weather-forecasting service. • Hilton Hotels are using Watson-based “Connie Robot” in their front desks. Connie

did a superb job in experiments, and its service is improving.

Clark (2016) reports that 1 billion people will use Watson by 2018. This is in part be- cause IBM Watson is coming to smartphones as an advisor. For more, see Noyes (2016).

u SECTION 12.6 REVIEW QUESTIONS

1. Define robo advisor. 2. Explain how robo advisors work for investments. 3. Discuss some of the shortcomings of robo advisors for investments. 4. Explain the people-machine collaboration in robo advising. 5. Describe IBM Watson as an advisor.

12.7 IMPLEMENTATION ISSUES

Several implementation issues are unique to chatbots and personal assistants. Examples of representative systems are described next.

Technology Issues

Many chatbots, including virtual personal assistants, have imperfect (but improving) voice recognition. There is no good feedback system yet for voice recognition systems to tell users, in real time, how well it understands them. In addition, voice recogni- tion systems may not know when to do a current task and need to ask for human intervention.

Chatbots that are internal to organizations need to be connected to an NLP system. This may be a problem, but a bigger one may exist when chatbots are connected to the Internet, due to security and connectivity difficulties.

Some chatbots need to be multilingual. Therefore, they need to be connected to a machine language translator.

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 681

Disadvantages and Limitations of Bots

The following are points (which were observed at the time this book was written dur- ing 2017 and 2018) regarding bots’ disadvantages and limitations; some will disappear with time:

• Some bots provide inferior performance, at least during their initiation, making users frustrated.

• Some bots do not properly represent their brand. Poor design may result in poor representation.

• The quality of AI-based bots depends on the use of complex algorithms that are expensive to build and use.

• Some bots are not convenient to use. • Some bots operate in an inconsistent manner. • Enterprise chatbots pose great security and integration challenges.

For methods to eliminate some of the disadvantages and limitations, see Kaya 2017.

VIRTUAL ASSISTANTS UNDER ATTACK Cortana, Siri, Alexa, and Google Assistant are under attack by people who are enraged at machines in general, or just like to make fun of them. In some cases, the bots’ administrators try to compose a response to the attacks; in other cases, some machines provide senseless responses to the senseless attacks.

Quality of Chatbots

While the quality of most systems is not perfect, it is improving over time. However, the quality of those that retrieve information for users and are properly programmed can do a perfect job. Generally speaking, the more a company invests in acquiring or leasing a chatbot, the better its accuracy will be. In addition, bots that serve a large number of people, such as Alexa and Google Assistant, exhibit an increasing level of accuracy.

QUALITY OF ROBO ADVISORS Given the short time since the emergence of robo advi- sors for financial services, it is difficult to assess the quality of their advice. Backend Benchmarking publishes a quarterly report (theroboreport.com) regarding robo advisor companies. Some reports are free. According to this service, Schwab’s Intelligent Portfolio Robot was the top performer in 2017. However, note that portfolio performance needs to be measured for the long run (e.g., 5 to 10 years).

A major issue when engaging bots is the potential loss of human touch. It is needed to build trust and answer complex questions so customers can understand bots’ answers. Also, bots cannot bring empathy or a sense of friendship. According to Knight (2017b), there is a solution to this. First, bots should perform only tasks that they are suited to do. Second, they should provide a visible benefit to the customer. Finally, because the bots face customers, the interactions must be fully planned to make sure the customers are happy.

In addition, note that robo advisors provide personalized advice. For information as to which robo may be best for you based on your objectives, see Eule (2017), who also provides a scorecard for the leading companies in the field. Finally, Gilani (2016) provides a guide for robo advisors as well as their possible dangers.

MICROSOFT’S TAY Tay was a Twitter-based chatbot that failed and was discontinued by Microsoft. It collected information from the Internet, but Microsoft had not given the bot the knowledge of how to deal with some inappropriate material used on the Internet (e.g., trolls, fake news). Therefore, Tay’s output was useless and frequently offended its users. As a result, Microsoft discontinued the service of Tay.

682 Part IV • Robotics, Social Networks, AI and IoT

Setting Up Alexa’s Smart Home System

Alexa is useful in controlling smart homes. Crist (2017) proposed a six-step process for how to use Alexa in smart homes:

1. Get a speaker (e.g., Echo). 2. Think about the location of the speaker. 3. Set up the smart home devices. 4. Sync related gadgets with Alexa. 5. Set up group and scene. 6. Fine-tune during the process.

These steps are demonstrated at cnet.com/uk/how-to/how-to-get-started-with- an-alexa-smart-home/.

Constructing Bots

Earlier, we presented some companies that provide development platforms for chatbots. In addition, several companies can build bots for users, so they can also build a simple bot by themselves. A step-by-step guide with the tools used is provided by Ignat (2017). The bot was constructed on Facebook Messenger. Another guide for creating a Facebook Messenger bot is provided by Newlands (2017b), who suggested the following steps:

1. Give it a unique name. 2. Give customers guides on how to build a bot and how to converse with it. 3. Experiment in making a natural conversation flow. 4. Make the bot sound smart, but use simple terminology. 5. Do not deploy all features at the same time. 6. Optimize and maintain the bot to constantly improve its performance.

There are several free sources for building chatbots. Most of them include “how-to” instructions. Several messaging services (e.g., Facebook Messenger, Telegraph) provide both chatbot platforms as well as their own chatbots. For a 2017 list of enterprise chatbot platforms and their capabilities, see entrepreneur.com/article/296504.

USING MICROSOFT’S AZURE BOT SERVICE Azure is a comprehensive but not a very com- plex bot builder. Its Bot Service provides five templates for quick and easy creation of bots. According to docs.microsoft.com/en-us/bot-framework/azure-bot-service- overview/, any of the templates shown in Table 12.1 can be used.

For a detailed tutorial for creating bots, see “Create a Bot with Azure Bot Service” at docs.microsoft.com/en-us/bot-framework/azure-bot-service-overview/.

TABLE 12.1 Azure’s Templates

Template Description

Basic Creates a bot that uses dialogues to respond to user input.

Form Creates a bot that collects input from users via a guided conversation that is created using Form Flow.

Language understanding

Creates a bot that uses natural language models (LUIS) to understand user intent.

Proactive Creates a bot that uses Azure Functions to alert users of events.

Question & Answer Creates a bot that uses a knowledge base to answer users’ questions.

Note: Microsoft also provides a bot framework on which bots can be constructed (similar to that of Facebook Messenger). For Microsoft’s Bot and a tutorial, see Afaq (2017).

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 683

Chapter Highlights

• Chatbots can save organizations money, provide a 24/7 link with customers and/or business part- ners, and are consistent in what they say.

• An expert system was the first commercially ap- plied AI product.

• ES transfer knowledge from experts to machines so the machines can have the expertise needed for problem solving.

• Classical ES use business rules to represent knowledge and generate answers to users’ ques- tions from it.

• The major components of ES are knowledge acquisition, knowledge representation, knowl- edge base, user interface, and interface engine. Additional components may include an expla- nation subsystem and a knowledge-refining system.

• ES help retain scarce knowledge in organizations. • New types of knowledge systems are superior to

classical ES, making ES disappear. • We distinguish three major types of chatbots:

enterprise, virtual personal assistants, and robo advisors.

• A relatively new application of knowledge sys- tems is the virtual personal assistant. Major examples of such assistants are Amazon’s Alexa, Apple’s Siri, and Google’s Assistant.

• Knowledge for virtual personal assistants is cen- trally maintained in the “cloud” and it is usually disseminated via a Q&A dialog.

• Personal assistants can receive voice commands that they can execute.

• Personal assistants can provide personalized ad- vice to their owners.

• Special breeds of assistants are personal advisors, such as robo advisors, that provide personalized advice to investors.

• Recommenders today use several AI technologies to provide personalized recommendations about products and services.

• People can communicate with chatbots via writ- ten messages, voice, and images.

• Chatbots contain a knowledge base and a natural language interface.

• Chatbots are used primarily for information search, communication and collaboration, and rendering advice in limited, specific domains.

• Chatbots can facilitate online shopping by pro- viding information and customer service.

• Chatbots work very well with messaging systems (e.g., Facebook Messenger, WeChat).

• Enterprise chatbots serve customers of all types and can work with business partners. They can also serve organizational employees.

• Virtual personal assistants (VPAs) are designed to work with individuals and can be customized for them.

• VPAs are created as “native” products for the masses.

• A well-known VPA is Amazon’s Alexa that is ac- cessed via a smart speaker called Echo (or other smart speakers).

• VPAs are available from several vendors. Well known are Amazon’s Alexa, Apple’s Siri, and Google’s Assistant.

• VPAs can specialize in specific domains and work as investment advisors.

• Robo advisors provide personalized online in- vestment advice at a much lower cost than human advisors. So far, the quality seems to be comparable.

• Robo advisors can be combined with human ad- visors to handle special cases.

Key Terms

Alexa chatbot Echo

expert systems Google’s Assistant recommendation systems

robo advisors Siri virtual personal assistant (VPA)

684 Part IV • Robotics, Social Networks, AI and IoT

Questions for Discussion

1. Some people say that chatbots are inferior for chatting. Others disagree. Discuss.

2. Discuss the financial benefits of chatbots. 3. Discuss how IBM Watson will reach 1 billion people by

2018 and what the implications of that are. 4. Discuss the limitation of chatbots and how to overcome

them. 5. Discuss what made ES popular for almost 30 years

before their decline. 6. Summarize the difficulties in knowledge acquisition

from experts (also consult Chapter 2). 7. Compare the ES knowledge-refining system with knowl-

edge improvement in machine learning. 8. Discuss the difference of enterprises’ use of chatbots

internally and externally.

9. Some people say that without a virtual personal assis- tant, a home cannot be smart. Why?

10. Compare Facebook Messenger virtual assistant project M with that of competitors.

11. Examine Alexa’s skill in ordering drinks from Starbucks. 12. Discuss the advantages of robo advisors over human

advisors. What are the disadvantages? 13. Explain how marketers can reach more customers with

bots. 14. Are robo advisors the future of finance? Debate; start

with Demmissie (2017). 15. Research the potential impact of chatbots on work and

write a summary.

Exercises

1. Compare the chatbots of Facebook and WeChat. Which has more functionalities?

2. Enter nuance.com and find information about Dragon Medical Advisor. Describe its benefits. Write a report.

3. Enter shopadvisor.com/our-platform and review the platform’s components. Examine the product’s capabili- ties and compare them with those of two other shopping advisors.

4. Enter chatbots.org/ and join a forum of your interest. Also explore research issues of your interest. Write a report.

5. There is intense competition between all major tech com- panies regarding their virtual personal assistants. New innovations and capabilities appear daily. Research the status of these assistants for Amazon, Apple, Microsoft, Google, and Samsung. Write a report.

6. Some people believe that chatbots will change how peo- ple interact with the Internet and browse online. Prepare a report regarding this.

7. Explain why is Amazon’s Echo needed to work with Alexa? Read howtogeek.com/253719/do-i-need-an- amazon-echo-to-use-alexa/. Write a report.

8. Find out how Simon Property Group is using chatbots across over 200 shopping malls. Write about the benefits to different types of users and to the company.

9. Read recent information about enterprise bots. Write a report.

10. Enter gravityinvestments.com/digital-advice-platform- demo. Would you invest in this project? Research and write a report.

11. Enter visirule.co.uk and find all products it has for expert systems. List them and write a short report.

12. Research the role of chatbots in helping patients with dementia.

13. Find information on the Baidu’s Melody chatbot and how it works with Baidu Doctor.

14. Pose a question related to a chatbot on quora.com. Summarize the answers received in a report.

15. Nina is an intelligent chatbot from Nuance Communication Inc. that works for Alexa Internet of Things (IoT), smart homes, and more. Find information and write a report about Nina’s capabilities and benefits.

16. Microsoft partners with the government of Singapore to develop chatbots for e-services. Find out how this is done.

17. Study the Tommy Hilfiger Facebook Messenger bot. Find out how it is (and was) used in the company’s mar- keting campaigns.

18. Two comprehensive building tools for chatbots are Botsify and Personality Forge (personalityforge.com). Compare the tools. Write a report.

19. Find information about the Alibaba-backed robo advi- sor Youyu by Yunfeng’s Investment. What is unique about this service? Start by visiting http://www. international-adviser.com/news/1035281/alibaba- backed-retail-robo-adviser-youyu-launches-honk- kong/.

20. Enter exsys.com. Select three case studies and explain why they were successful.

21. It is time now to build your own bot. Consult with your instructor about which software to use. Have several bots constructed in your class and compare their capabilities. Use Microsoft’s Azure if you have some programming experience.

Chapter 12 • Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants 685

References

Afaq, O. “Developing a Chatbot Using Microsoft’s Bot Frame- work, LUIS and Node.js (Part 1).” Smashing Magazine, May 30, 2017. smashingmagazine.com/2017/05/ chatbot-microsoft-bot-framework-luis-nodejs-part1/ (accessed April 2018).

Aggarwal, C. Recommended Systems: The Textbook. [eText- book]. New York, NY: Springer, 2016.

Arora, S. “Recommendation Engines: How Amazon and Netf- lix Are Winning the Personalization Battle.” Martech Advi- sor, June 28, 2016.

Arthur, R. “Sephora Launches Chatbot on Messaging App Kik.” Forbes, March 30, 2016.

Bae, J. “Development and Application of a Web-Based Ex- pert System Using Artificial Intelligence for Management of Mental Health by Korean Emigrants.” Journal of Korean Academy of Nursing, April 2013.

Beaver, L. “Chatbots Explained: Why Businesses Should Be Paying Attention to the Chatbot Revolution.” Business In- sider, March 4, 2016.

CBS News. “LinkedIn Adding New Training Features, News Feeds and ‘Bots.’” CBS News, September 22, 2016. cbsnews.com/news/linkedin-adding-new-training- features-news-feeds-and-bots (accessed April 2018).

Clark, D. “IBM: A Billion People to Use Watson by 2018.” The Wall Street Journal, October 26, 2016.

Cognizant. “Bot Brings Transavia Airlines Closer to Customers.” Cognizant Services, 2017. https://www.cognizant.com/ content/dam/Cognizant_Dotcom/landing-page- resources/transavia-case-study.pdf (accessed April 2018).

Costa, A., et al. (eds.) Personal Assistants: Emerging Computa- tional Technologies (Intelligent Systems Reference Library). New York, NY: Springer, 2018.

Crist, R. “How to Get Started with an Alexa Smart Home.” CNET, July 5, 2017. cnet.com/how-to/how-to-get-started-with- an-alexa-smart-home/ (accessed April 2018).

Crook, J. “WeWork Has Big Plans for Alexa for Business.” TechCrunch, November 30, 2017.

Davydova, O. “25 Chatbot Platforms: A Comparative Table.” Chatbots Journal, May 11, 2017.

De Aenlle, C. “A.I. Has Arrived in Investing. Humans Are Still Dominating.” The New York Times, January 12, 2018.

Demmissie, L. “Robo Advisors: The Future of Finance.” The Ticker Tape, March 13, 2017.

Ell, K. “ETFs Powered by Artificial Intelligence Are Getting Smarter, Says Fund Co-Founder.” CNBC News, January 23, 2018.

Eule, A. “Rating the Robo-Advisors.” Barron’s, July 29, 2017. Ferron, E. “Mobile 101: What Are Bots, Chatbots and Vir-

tual Assistants?” New Atlas, February 16, 2017. newatlas. com/what-is-bot-chatbot-guide/47965/ (accessed April 2018).

Garg, N. “Case Study: How Kenyt Real Estate Chatbot Is Gen- erating Leads.” Medium, June 22, 2017.

Gikas, M. “What the Amazon Echo and Alexa Do Best.” Con- sumer Reports, July 29, 2016. consumerreports.org/ wireless-speakers/what-amazon-echo-and-alexa-do- best (accessed April 2018).

Gilani, S. “Your Perfectly Diversified Portfolio Could Be in Danger—Here’s Why.” Money Morning @ Wall Street, De- cember 6, 2016.

Griffiths, T. “Using Chatbots to Improve CRM Data: A WeChat Case Study.” Half a World, November 16, 2016.

Guynn, J. “Zuckerberg’s Facebook Messenger Launches ‘Chat Bots’ Platform.” USA Today, April 12, 2016.

Hachman, M. “Microsoft Combines Cortana and Bing with Mi- crosoft Research to Accelerate New Features.” PCWorld, September 29, 2016.

Huang, N. “Robo Advisers Get the Human Touch.” Kiplinger’s Personal Finance, September 2017.

Hunt, M. “Enterprise Chatbots and the Conversational Com- merce Revolutionizing Business.” Entrepreneur, July 3, 2017.

Ignat, A. “Iggy—A Chatbot UX Case Study.” Chatbot’s Life, Au- gust 9, 2017. chatbotslife.com/iggy-a-chatbot-ux-case- study-b5ac0379029c/ (accessed April 2018).

Ismail, K. “Top 14 Chatbot Building Platforms of 2014.” CMS Wire, December 19, 2017.

Johnson, K. “Everything Amazon’s Alexa Learned to Do in 2017.” Venturebeat.com, December 29, 2017.

Kaya, E. Bot Business 101: How to Start, Run & Grow Bot/ AI Business. Kindle Edition. Seattle, WA: Amazon Digital Services, 2017.

Kelly, H. “Amazon wants Alexa everywhere.” CNN Tech, Sep- tember 22, 2018.

Kelly, H. “Battle of the Smart Speakers: Google Home vs. Amazon Echo.” CNN Tech, May 20, 2016. money.cnn. com/2016/05/20/technology/google-home-amazon- echo/index.html?iid=EL (accessed April 2018).

Keppel, D. Best Robo-Advisor: Ultimate Automatic Wealth Man- agement. North Charleston, SC: Create Space Pub., 2016.

Knight, K. “Expert: Bots May Be a Marketers New Best Friend.” BizReport, December 7, 2017a.

Knight, K. “Expert: How to Engage Chatbots Without Losing the Human Touch.” BizReport, February 13, 2017b.

Knight, K. “Report: Over Half of Millennials Have or Will Use Bots.” Biz Report, February 24, 2017c.

Korosec, K. “Start Your Car from Inside Your Home Using Amazon’s Alexa.” Fortune.com, August 18, 2016.

Lacheca, D. “Conversational AI Creates New Dialogues for Government.” eGovInnovation, October 24, 2017.

Larson, S. “Baidu Is Bringing AI Chatbots to Healthcare.” CNNTech, October 11, 2016.

Lovett, L. “Chatbot Campaign for Flu Shots Bolsters Patient Response Rate by 30%.” Healthcareitnews.com, January 24, 2018.

Mah, P. “The State of Chatbots in Marketing.” CMOInnovation, November 4, 2016.

686 Part IV • Robotics, Social Networks, AI and IoT

Makadia, M. “Benefits for Recommendation Engines to the Ecommerce Sector.” Business 2 Community, January 7, 2018.

Mangalindan, J. P. “RBC: Amazon Has a Potential Mega-Hit on Its Hand.” Yahoo! Finance, April 25, 2017.

Marino, J. “Big Banks Are Fighting Robo-Advisors Head On.” CNBC News, June 26, 2016.

Matney, L. “Siri-Creator Shows Off First Public Demo of Viv, ‘The Intelligent Interface for Everything.’” Tech Crunch, May 9, 2016. techcrunch.com/2016/05/09/siri-creator- shows-off-first-public-demo-of-viv-the-intelligent- interface-for-everything (accessed April 2018).

McClellan, J. “What the Evolving Robo Advisory Industry Of- fers.” AAII Journal, October 2016.

Morgan, B. “How Chatbots Improve Customer Experience in Every Industry: An Infograph.” Forbes, June 8, 2017.

Newlands, M. “How to Create a Facebook Messenger Chatbot for Free Without Coding,” Entrepreneur, March 14, 2017a.

Newlands, M. “10 Ways Enterprise Chatbots Empower CEOs.” MSN.com, August 9, 2017b. msn.com/en-us/ money/smallbusiness/10-ways-enterprise-chatbots- empower-ceos/ar-AApMgU8 (accessed April 2018).

Noyes, K. “Watson’s the Name, Data’s the Game.” PCWorld, October 7, 2016.

Nur, N. “Singapore’s POSB Launches AI-Driven Chatbot on Facebook Messenger.” MIS Asia, January 19, 2017.

O’Brien, M. “What Can Chatbots Do for Ecommerce?” ClickZ. com, April 11, 2016.

Oremus, W. “When Will Alexa Know Everything?” Slate.com, April 6, 2018.

O’Shea, A. “Best Robo-Advisors: 2016 Top Picks.” NerdWallet, March 14, 2016.

O’Shea, A. “Betterment Review 2017.” NerdWallet, January 31, 2017.

Perez, S. “Voice-Enabled Smart Speakers to Reach 55% of U.S. Households by 2022, Says Report.” Tech Crunch, Novem- ber 8, 2017.

Pohjanpalo, K. “Investment Bankers Are Hard to Replace with Robots, Nordea Says.” Bloomberg, November 27, 2017.

Popper, B. “How Netflix Completely Revamped Recommen- dations for Its New Global Audience.” The Verge, February 17, 2016. theverge.com/2016/2/17/11030200/netflix- new-recommendation-system-global-regional (ac- cessed April 2018).

Quoc, M. “10 Ecommerce Brands Succeeding with Chatbots.” A Better Lemonade Stand, October 23, 2017.

Radu, M. “How to Pay Less for Advertising? Use Baro—An Ad Robot for Campaigns Optimization.” 150sec.com, August 18, 2016.

Rayome, A. “How Sephora Is Leveraging AR and AI to Transform Retail and Help Customers Buy Cosmetics.” TechRepublic, February, 2018, “If This Model Is Right.” Bloomberg Business, June 18, 2015.

Reisinger, D. “10 Reasons to Buy the Amazon Echo Virtual Personal Assistant.” Slide Show. eWeek, February 9, 2016.

Schlicht, M. “The Complete Beginner’s Guide to Chatbots.” Chatbots Magazine, April 20, 2016. chatbotsmagazine. com/the-complete-beginner-s-guide-to-chatbots- 8280b7b906ca (accessed April 2018).

StartUp. “How Netflix Uses Big Data.” Medium.com, January 12, 2018. medium.com/swlh/how-netflix-uses-big- data-20b5419c1edf (accessed April 2018).

Sun, Y. “Alibaba’s AI Fashion Consultant Helps Achieve Record-Setting Sales.” MITTechnology Review, November 13, 2017.

TalKing. “Top Useful Chatbots for Health.” Chatbots Maga- zine, February 7, 2017.

Taylor, S. “Very Human Lessons from Three Brands That Use Chatbots to Talk to Customers.” Fast Company, October 21, 2016. fastcompany.com/3064845/human-lessons- from-brands-using-chatbots (accessed April 2018).

Ulanoff, L. “Mark Zuckerberg’s AI Is Already Making Him Toast.” Mashable, July 22, 2016.