BIG DATA ANALYTICS

sri169025

sharda_dss10e_pp_ch13_NEW.pptx

Home >Computer Science homework help >BIG DATA ANALYTICS

Chapter 13:

Big Data Analytics

Business Intelligence and Analytics: Systems for Decision Support

(10th Edition)

Business Intelligence and Analytics: Systems for Decision Support

(10th Edition)

13-‹#›

Learning Objectives

Learn what Big Data is and how it is changing the world of analytics

Understand the motivation for and business drivers of Big Data analytics

Become familiar with the wide range of enabling technologies for Big Data analytics

Learn about Hadoop, MapReduce, and NoSQL as they relate to Big Data analytics

Understand the role of and capabilities/ skills for data scientist as a new analytics profession

(Continued…)

13-‹#›

Learning Objectives

Compare and contrast the complementary uses of data warehousing and Big Data

Become familiar with the vendors of Big Data tools and services

Understand the need for and appreciate the capabilities of stream analytics

Learn about the applications of stream analytics

13-‹#›

Opening Vignette…

Big Data Meets Big Science at CERN

Situation

Problem

Solution

Results

Answer & discuss the case questions.

13-‹#›

Questions for the Opening Vignette

What is CERN, and why is it important to the world of science?

How does the Large Hadron Collider work? What does it produce?

What is the essence of the data challenge at CERN? How significant is it?

What was the solution? How were the Big Data challenges addressed with this solution?

What were the results? Do you think the current solution is sufficient?

13-‹#›

Big Data - Definition and Concepts

Big [volume] Data is not new!

Big Data means different things to people with different backgrounds and interests

Traditionally, “Big Data” = massive volumes of data

E.g., volume of data at CERN, NASA, Google, …

Where does the Big Data come from?

Everywhere! Web logs, RFID, GPS systems, sensor networks, social networks, Internet-based text documents, Internet search indexes, detail call records, astronomy, atmospheric science, biology, genomics, nuclear physics, biochemical experiments, medical records, scientific research, military surveillance, multimedia archives, …

13-‹#›

Technology Insights 6.1 The Data Size Is Getting Big, Bigger…

Hadron Collider - 1 PB/sec

Boeing jet - 20 TB/hr

Facebook - 500 TB/day.

YouTube – 1 TB/4 min.

The proposed Square Kilometer Array telescope (the world’s proposed biggest telescope) – 1 EB/day

Names for Big Data Sizes

13-‹#›

Big Data - Definition and Concepts

Big Data is a misnomer!

Big Data is more than just “big”

The Vs that define Big Data

Volume

Variety

Velocity

Veracity

Variability

Value

…

13-‹#›

A High-level Conceptual Architecture for Big Data Solutions

(by AsterData / Teradata)

13-‹#›

Application Case 13.1

BigData Analytics Helps Luxottica Improvement its Marketing Effectiveness

Questions for Discussion

What does “big data” mean to Luxottica?

What were their main challenges?

What were the proposed solution and the obtained results?

13-‹#›

Fundamentals of Big Data Analytics

Big Data by itself, regardless of the size, type, or speed, is worthless

Big Data + “big” analytics = value

With the value proposition, Big Data also brought about big challenges

Effectively and efficiently capturing, storing, and analyzing Big Data

New breed of technologies needed (developed (or purchased or hired or outsourced …)

13-‹#›

Big Data Considerations

You can’t process the amount of data that you want to because of the limitations of your current platform.

You can’t include new/contemporary data sources (e.g., social media, RFID, Sensory, Web, GPS, textual data) because it does not comply with the data schema.

You need to (or want to) integrate data as quickly as possible to be current on your analysis.

You want to work with a schema-on-demand data storage paradigm because the variety of data types.

The data is arriving so fast at your organization’s doorstep that your analytics platform cannot handle it.

…

13-‹#›

Critical Success Factors for Big Data Analytics

A clear business need (alignment with the vision and the strategy)

Strong, committed sponsorship (executive champion)

Alignment between the business and IT strategy

A fact-based decision-making culture

A strong data infrastructure

The right analytics tools

Right people with right skills

13-‹#›

Critical Success Factors for Big Data Analytics

13-‹#›

Enablers of Big Data Analytics

In-memory analytics

Storing and processing the complete data set in RAM

In-database analytics

Placing analytic procedures close to where data is stored

Grid computing & MPP

Use of many machines and processors in parallel (MPP- massively parallel processing)

Appliances

Combining hardware, software and storage in a single unit for performance and scalability

13-‹#›

Challenges of Big Data Analytics

Data volume

The ability to capture, store, and process the huge volume of data in a timely manner

Data integration

The ability to combine data quickly/cost effectively

Processing capabilities

The ability to process the data quickly, as it is captured (i.e., stream analytics)

Data governance (… security, privacy, access)

Skill availability (… data scientist)

Solution cost (ROI)

13-‹#›

Business Problems Addressed by Big Data Analytics

Process efficiency and cost reduction

Brand management

Revenue maximization, cross-selling/up-selling

Enhanced customer experience

Churn identification, customer recruiting

Improved customer service

Identifying new products and market opportunities

Risk management

Regulatory compliance

Enhanced security capabilities

…

13-‹#›

Application Case 13.2

Top 5 Investment Bank Achieves Single Source of the Truth

Questions for Discussion

How can Big Data benefit large-scale trading banks?

How did MarkLogic infrastructure help ease the leveraging of Big Data?

What were the challenges, the proposed solution, and the obtained results?

13-‹#›

Application Case 13.2

Moving from many old systems to a unified new system

13-‹#›

Big Data Technologies

MapReduce …

Hadoop …

Hive

Pig

Hbase

Flume

Oozie

Ambari

Avro

Mahout, Sqoop, Hcatalog, ….

13-‹#›

Big Data Technologies - MapReduce

MapReduce distributes the processing of very large multi-structured data files across a large cluster of ordinary machines/processors

Goal - achieving high performance with “simple” computers

Developed and popularized by Google

Good at processing and analyzing large volumes of multi-structured data in a timely manner

Example tasks: indexing the Web for seearch, graph analysis, text analysis, machine learning, …

13-‹#›

Big Data Technologies - MapReduce

How does

MapReduce

work?

13-‹#›

Big Data Technologies - Hadoop

Hadoop is an open source framework for storing and analyzing massive amounts of distributed, unstructured data

Originally created by Doug Cutting at Yahoo!

Hadoop clusters run on inexpensive commodity hardware so projects can scale-out inexpensively

Hadoop is now part of Apache Software Foundation

Open source - hundreds of contributors continuously improve the core technology

MapReduce + Hadoop = Big Data core technology

13-‹#›

Big Data Technologies - Hadoop

How Does Hadoop Work?

Access unstructured and semi-structured data (e.g., log files, social media feeds, other data sources)

Break the data up into “parts,” which are then loaded into a file system made up of multiple nodes running on commodity hardware using HDFS

Each “part” is replicated multiple times and loaded into the file system for replication and failsafe processing

A node acts as the Facilitator and another as Job Tracker

Jobs are distributed to the clients, and once completed the results are collected and aggregated using MapReduce

13-‹#›

Big Data Technologies - Hadoop

Hadoop Technical Components

Hadoop Distributed File System (HDFS)

Name Node (primary facilitator)

Secondary Node (backup to Name Node)

Job Tracker

Slave Nodes (the grunts of any Hadoop cluster)

Additionally, Hadoop ecosystem is made-up of a number of complementary sub-projects: NoSQL (Cassandra, Hbase), DW (Hive), …

NoSQL = not only SQL

13-‹#›

Big Data Technologies Hadoop - Demystifying Facts

Hadoop consists of multiple products

Hadoop is open source but available from vendors, too

Hadoop is an ecosystem, not a single product

HDFS is a file system, not a DBMS

Hive resembles SQL but is not standard SQL

Hadoop and MapReduce are related but not the same

MapReduce provides control for analytics, not analytics

Hadoop is about data diversity, not just data volume.

Hadoop complements a DW; it’s rarely a replacement.

Hadoop enables many types of analytics, not just Web analytics.

13-‹#›

Application Case 13.3

eBay’s

Big Data

Solution

Questions for Discussion

Why did eBay need a Big Data solution?

What were the challenges, the proposed solution, and the obtained results?

EBay’s Multi Data-Center Deployment

13-‹#›

Data Scientist

“The Sexiest Job of the 21st Century”

Thomas H. Davenport and D. J. Patil

Harvard Business Review, October 2012

Data Scientist = Big Data guru

One with skills to investigate Big Data

Very high salaries, very high expectations

Where do Data Scientist come from?

M.S./Ph.D. in MIS, CS, IE,… and/or Analytics

There is not a specific degree program for DS!

PE, PML, … DSP (Data Sceice Professional)

13-‹#›

Skills That Define a Data Scientist

13-‹#›

A Typical Job Post for Data Scientist

13-‹#›

Application Case 13.4

Big Data and Analytics in Politics

Questions for Discussion

What is the role of analytics and Big Data in modern day politics?

Do you think Big Data analytics could change the outcome of an election?

What do you think are the challenges, the potential solution, and the probable results of the use of Big Data analytics in politics?

13-‹#›

Application Case 13.4 Big Data and Analytics in Politics

13-‹#›

Big Data And Data Warehousing

What is the impact of Big Data on DW?

Big Data and RDBMS do not go nicely together

Will Hadoop replace data warehousing/RDBMS?

Use Cases for Hadoop

Hadoop as the repository and refinery

Hadoop as the active archive

Use Cases for Data Warehousing

Data warehouse performance

Integrating data that provides business value

Interactive BI tools

13-‹#›

Hadoop versus Data Warehouse When to Use Which Platform

13-‹#›

Coexistence of Hadoop and DW

Use Hadoop for storing and archiving multi-structured data

Use Hadoop for filtering, transforming, and/or consolidating multi-structured data

Use Hadoop to analyze large volumes of multi-structured data and publish the analytical results

Use a relational DBMS that provides MapReduce capabilities as an investigative computing platform

Use a front-end query tool to access and analyze data

13-‹#›

Coexistence of Hadoop and DW

Source: Teradata

13-‹#›

Big Data Vendors

Big Data vendor landscape is developing very rapidly

A representative list would include

Cloudera - cloudera.com

MapR – mapr.com

Hortonworks - hortonworks.com

Also, IBM (Netezza, InfoSphere), Oracle (Exadata, Exalogic), Microsoft, Amazon, Google, …

Software,

Hardware,

Service, …

13-‹#›

Top 10 Big Data Vendors with Primary Focus on Hadoop

13-‹#›

Application Case 13.5

Dublin City Council Is Leveraging Big Data to Reduce Traffic Congestion

Questions for Discussion

Is there a strong case to make for large cities to use Big Data Analytics and related information technologies? Identify and discuss examples of what can be done with analytics beyond what is portrayed in this application case.

How can a big data analytics help ease the traffic problem in large cities?

What were the challenges Dublin City was facing; what were the proposed solution, initial results, and future plans?

13-‹#›

Technology Insights 13.4 How to Succeed with Big Data

Simplify

Coexist

Visualize

Empower

Integrate

Govern

Evangelize

13-‹#›

Application Case 13.6

Creditreform Boosts Credit Rating Quality with Big Data Visual Analytics

Questions for Discussion

How did Creditreform boost credit rating quality with Big Data and visual analytics?

What were the challenges, proposed solution, and initial results?

13-‹#›

Big Data And Stream Analytics

Data-in-motion analytics and real-time data analytics

One of the Vs in Big Data = Velocity

Analytic process of extracting actionable information from continuously flowing/streaming data

Why Stream Analytics?

It may not be feasible to store the data

It may loose its value if not processed immediately

Stream Analytics Versus Perpetual Analytics

Critical Event Processing?

13-‹#›

Stream Analytics A Use Case in Energy Industry

13-‹#›

Stream Analytics Applications

e-Commerce

Telecommunication

Law Enforcement and Cyber Security

Power Industry

Financial Services

Health Services

Government

13-‹#›

Application Case 13.7

Turning Machine-Generated Streaming Data into Valuable Business Insights

Questions for Discussion

Why is stream analytics becoming more popular?

How did the telecommunication company in this case use stream analytics for better business outcomes? What additional benefits can you foresee?

What were the challenges, proposed solution, and initial results?

13-‹#›

End of the Chapter

Questions, comments

13-‹#›

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher. Printed in the United States of America.

13-‹#›

Math and StatsDataMiningBusinessIntelligenceApplicationsLanguagesMarketing

ANALYTIC TOOLS & APPSUSERS

DISCOVERY PLATFORMINTEGRATED DATA WAREHOUSEDATAPLATFORM ACCESSMANAGEMOVE

UNIFIED DATA ARCHITECTURE

System Conceptual View

MarketingExecutivesOperationalSystemsFrontlineWorkersCustomersPartnersEngineersDataScientistsBusinessAnalysts

EVENT PROCESSING

ERPERPSCMCRMImagesAudio and VideoMachine LogsTextWeb and Social

BIG DATA SOURCES

ERP

Keys to Success

with Big Data

Analytics

A Clear

business need

Strong,

committed

sponsorship

Alignment

between the

business and IT

strategy

A fact-based

decision-making

culture

A strong data

infrastructure

The right

analytics tools

Personnel with

advanced

analytical skills

BeforeAfter

Before it was difficult to identify financial

exposure across many systems (separate

copies of derivatives trade store)

After it was possible to analyze all contracts in

single database (MarkLogic Server eliminates

the need for 20 database copies)

Raw DataMap FunctionReduce Function

Curiosity and

Creativity

Internet and Social

Media/Social Networking

Technologies

Programming,

Scripting and Hacking

Data Access and

Management

(both traditional and

new data systems)

Domain Expertise,

Problem Definition and

Decision Modeling

Communication and

Interpersonal

DATA

SCIENTIST

$0$10$20$30$40$50$60$70

Sensor Data

(Energy Production

System Status)

Meteorological Data

(Wind, Light,

Temperature, etc.)

Usage Data

(Smart Meters,

Smart Grid Devises)

Permanent

Storage Area

Streaming Analytics

(Predicting Usage,

Production and

Anomalies)

Energy Production System

(Traditional and Renewable)

Energy Consumption System

(Residential and Commercial)

Data Integration

and Temporary

Staging

Capacity Decisions

Pricing Decisions