Discusssion

varunydd2

Sharda_11e_full_accessible_ppt_03.pptx

Home >Computer Science homework help >Discusssion

Analytics, Data Science and A I: Systems for Decision Support

Eleventh Edition

Chapter 3

Nature of Data, Statistical Modeling and Visualization

Slide in this Presentation Contain Hyperlinks. JAWS users should be able to get a list of links by using INSERT+F77

If this PowerPoint presentation contains mathematical equations, you may need to check that your computer has the following installed:

1) Math Type Plugin

2) Math Player (free versions available)

3) NVDA Reader (free versions available)

Learning Objectives (1 of 2)

3.1 Understand the nature of data as it relates to business intelligence (B I) and analytics

3.2 Learn the methods used to make real-world data analytics ready

3.3 Describe statistical modeling and its relationship to business analytics

3.4 Learn about descriptive and inferential statistics

3.5 Define business reporting, and understand its historical evolution

Slide 2 is list of textbook LO numbers and statements

Learning Objectives (2 of 2)

3.6 Understand the importance of data/information visualization

3.7 Learn different types of visualization techniques

3.8 Appreciate the value that visual analytics brings to business analytics

3.9 Know the capabilities and limitations of dashboards

Slide 2 is list of textbook LO numbers and statements

Opening Vignette

Attracts and Engages a New Generation of Radio Consumers with Data-Driven Marketing

What does Sirius X M do? In what type of market does it conduct its business?

What were the challenges? Comment on both technology and data-related challenges.

What were the proposed solutions?

How did they implement the proposed solutions? Did they face any implementation challenges?

What were the results and benefits? Were they worth the effort/investment?

The Nature of Data (1 of 2)

Data: a collection of facts

usually obtained as the result of experiences, observations, or experiments

Data may consist of numbers, words, images, …

Data is the lowest level of abstraction (from which information and knowledge are derived)

Data is the source for information and knowledge

Data quality and data integrity  critical to analytics

The Nature of Data (2 of 2)

Metrics for Analytics ready Data

Data source reliability

Data content accuracy

Data accessibility

Data security and data privacy

Data richness

Data consistency

Data currency/data timeliness

Data granularity

Data validity and data relevancy

A Simple Taxonomy of Data (1 of 2)

Data (datum—singular form of data): facts

Structured data

Targeted for computers to process

Numeric versus nominal

Unstructured/textual data

Targeted for humans to process/digest

Semi-structured data?

X M L, H T M L, Log files, etc.

Data taxonomy…

A Simple Taxonomy of Data (2 of 2)

Application Case 3.1

Verizon Answers the Call for Innovation: The Nation’s Largest Network Provider uses Advanced Analytics to Bring the Future to its Customers

Questions for Discussion:

What was the challenge Verizon was facing?

What was the data-driven solution proposed for Verizon’s business units?

What were the results?

The Art and Science of Data Preprocessing (1 of 2)

The real-world data is dirty, misaligned, overly complex, and inaccurate

Not ready for analytics!

Readying the data for analytics is needed

Data preprocessing

Data consolidation

Data cleaning

Data transformation

Data reduction

Art – it develops and improves with experience

The Art and Science of Data Preprocessing (2 of 2)

Data reduction

Variables

Dimensional reduction

Variable selection

2. Cases/samples

Sampling

Balancing / stratification

Data Preprocessing Tasks and Methods

Table 3.1 A Summary of Data Preprocessing Tasks and Potential Methods.

Main Task	Subtasks	Popular Methods
Data consolidation	Access and collect the data Select and filter the data Integrate and unify the data	SQL queries, software agents, Web services. Domain expertise, SQL queries, statistical tests. SQL queries, domain expertise, ontology-driven data mapping.
Data cleaning	Handle missing values in the data	Fill in missing values (imputations) with most appropriate values (mean, median, min/max, mode, etc.); recode the missing values with a constant such as “ML”; remove the record of the missing value; do nothing.
Blank	Identify and reduce noise in the data	Identify the outliers in data with simple statistical techniques (such as averages and standard deviations) or with cluster analysis; once identified, either remove the outliers or smooth them by using binning, regression, or simple averages.
Blank	Find and eliminate erroneous data	Identify the erroneous values in data (other than outliers), such as odd values, inconsistent class labels, odd distributions; once identified, use domain expertise to correct the values or remove the records holding the erroneous values.
Data transformation	Normalize the data	Reduce the range of values in each numerically valued variable to a standard range (e.g., 0 to 1 or −1 to +1) by using a variety of normalization or scaling techniques.
Blank	Discretize or aggregate the data	If needed, convert the numeric variables into discrete representations using range- or frequency-based binning techniques; for categorical variables, reduce the number of values by applying proper concept hierarchies.
Blank	Construct new attributes	Derive new and more informative variables from the existing ones using a wide range of mathematical functions (as simple as addition and multiplication or as complex as a hybrid combination of log transformations).
Data reduction	Reduce number of attributes	Use principal component analysis, independent component analysis, chi-square testing, correlation analysis, and decision tree induction.
Blank	Reduce number of records	Perform random sampling, stratified sampling, expert-knowledge-driven purposeful sampling.
Blank	Balance skewed data	Oversample the less represented or undersample the more represented classes.

Application Case 3.2 (1 of 4)

Improving Student Retention with Data-Driven Analytics

Questions for Discussion:

What is student attrition, and why is it an important problem in higher education?

What were the traditional methods to deal with the attrition problem?

List and discuss the data-related challenges within context of this case study.

What was the proposed solution? And, what were the results?

Application Case 3.2 (2 of 4)

Improving Student Retention with Data-Driven Analytics

Student retention

Freshmen class

Why it is important?

What are the common techniques to deal with student attrition?

Analytics versus theoretical approaches to student retention problem

Application Case 3.2 (3 of 4)

Improving Student Retention with Data-Driven Analytics

Data imbalance problem

Application Case 3.2 (4 of 4)

Improving Student Retention with Data-Driven Analytics

Results…

Statistical Modeling for Business Analytics (1 of 2)

Statistical Modeling for Business Analytics (2 of 2)

Statistics

A collection of mathematical techniques to characterize and interpret data

Descriptive Statistics

Describing the data (as it is)

Inferential statistics

Drawing inferences about the population based on a sample data

Descriptive statistics for descriptive analytics

Descriptive Statistics Measures of Centrality Tendency (1 of 2)

Arithmetic mean

Median

The number in the middle

Mode

The most frequent observation

Descriptive Statistics Measures of Dispersion (1 of 2)

Dispersion

Degree of variation in a given variable

Range

Max - Min

Variance Standard Deviation

Mean Absolute Deviation (M A D)

Average absolute deviation from the mean

Descriptive Statistics Measures of Dispersion (2 of 2)

Quartiles

Box-and-Whiskers Plot

a.k.a. box-plot

Versatile / informative

Descriptive Statistics Measures of Centrality Tendency (2 of 2)

Histogram - frequency chart

Skewness

Measure of asymmetry

Kurtosis

Peak/tall/skinny nature of the distribution

Relationship Between Dispersion and Shape Properties

Technology Insights 3.1 – Descriptive Statistics in Excel

Technology Insights 3.1 – Descriptive Statistics in Excel Creating box-plot in Microsoft Excel

Application Case 3.3

Town of Cary Uses Analytics to Analyze Data from Sensors, Assess Demand, and Detect Problems

Questions for Discussion:

What were the challenges the Town of Cary was facing?

What was the proposed solution?

What were the results?

What other problems and data analytics solutions do you foresee for towns like Cary?

Regression Modeling for Inferential Statistics

Regression

A part of inferential statistics

The most widely known and used analytics technique in statistics

Used to characterize relationship between explanatory (input) and response (output) variable

It can be used for

Hypothesis testing (explanation)

Forecasting (prediction)

Regression Modeling (1 of 3)

Correlation versus Regression

What is the difference (or relationship)?

Simple Regression versus Multiple Regression

Base on number of input variables

How do we develop linear regression models?

Scatter plots (visualization—for simple regression)

Ordinary least squares method

A line that minimizes squared of the errors

Regression Modeling (2 of 3)

Regression Modeling (3 of 3)

x: input, y: output

Simple Linear Regression

Multiple Linear Regression

The meaning of Beta ( ) coefficients

Sign (+ or −) and magnitude

Process of Developing a Regression Model

How do we know if the model is good enough?

R2 (R-Square)

p Values

Error measures (for prediction problems)

M S E, M A D, R M S E

Regression Modeling Assumptions

Linearity

Independence

Normality (Normal Distribution)

Constant Variance

Multicollinearity

What happens if the assumptions do NOT hold?

What do we do then?

Logistic Regression Modeling (1 of 2)

A very popular statistics-based classification algorithm

Employs supervised learning

Developed in 1940s

The difference between Linear Regression and Logistic Regression

In Logistic Regression Output/Target variable is a binomial (binary classification) variable (as supposed to numeric variable)

Logistic Regression Modeling (2 of 2)

Application Case 3.4 (1 of 4)

Predicting N C A A Bowl Game Outcomes

Application Case 3.4 (2 of 4)

Predicting N C A A Bowl Game Outcomes

The analytics process to develop prediction models (both regression and classification type) for N C A A Bowl Game outcomes

Application Case 3.4 (3 of 4)

Predicting N C A A Bowl Game Outcomes

Prediction Results

Classification

Regression

Application Case 3.4 (4 of 4)

Predicting N C A A Bowl Game Outcomes

Questions for Discussion:

What are the foreseeable challenges in predicting sporting event outcomes (e.g., college bowl games)?

How did the researchers formulate/design the prediction problem (i.e., what were the inputs and output, and what was the representation of a single sample—row of data)?

How successful were the prediction results? What else can they do to improve the accuracy?

Time Series Forecasting

Is it different than Simple Linear Regression? How?

Business Reporting Definitions and Concepts

Report = Information  Decision

Report?

Any communication artifact prepared to convey specific information

A report can fulfill many functions

To ensure proper departmental functioning

To provide information

To provide the results of an analysis

To persuade others to act

To create an organizational memory…

What is a Business Report?

A written document that contains information regarding business matters.

Purpose: to improve managerial decisions

Source: data from inside and outside the organization (via the use of E T L)

Format: text + tables + graphs/charts

Distribution: in-print, email, portal/intranet

Data acquisition  Information generation  Decision making  Process management

Business Reporting

Types of Business Reports

Metric Management Reports

Help manage business performance through metrics (S L A s for externals; K P I s for internals)

Can be used as part of Six Sigma and/or T Q M

Dashboard-Type Reports

Graphical presentation of several performance indicators in a single page using dials/gauges

Balanced Scorecard–Type Reports

Include financial, customer, business process, and learning & growth indicators

Application Case 3.5

Flood of Paper Ends at F E M A

Questions for Discussion:

What is F E M A, and what does it do?

What are the main challenges that F E M A faces?

How did F E M A improve its inefficient reporting practices?

Data Visualization

“The use of visual representations to explore, make sense of, and communicate data.”

Data visualization vs. Information visualization

Information = aggregation, summarization, and contextualization of data

Related to information graphics, scientific visualization, and statistical graphics

Often includes charts, graphs, illustrations, …

A Brief History of Data Visualization

Data visualization can date back to the second century A D

Most developments have occurred in the last two and a half centuries

Until recently it was not recognized as a discipline

Today’s most popular visual forms date back a few centuries

The First Pie Chart Created by William Playfair in 1801

William Playfair is widely credited as the inventor of the modern chart, having created the first line and pie charts.

Decimation of Napoleon’s Army During the 1812 Russian Campaign

By Charles Joseph Minard

Arguably the most popular multi-dimensional chart

Application Case 3.6

Macfarlan Smith Improves Operational Performance Insight with Tableau Online

Questions for Discussion:

What were the data and reporting related challenges Macfarlan Smith facing?

What was the solution and the obtained results and/or benefits?

Which Chart or Graph Should You Use?

Figure 3.21 A Taxonomy of Charts and Graphs.

Source: Adapted from Abela, A. (2008). Advanced Presentations by Design: Creating Communication That Drives Action. New York: Wiley.

An Example Gapminder Chart: Wealth and Health of Nations

Figure 3.22 A Gapminder Chart That Shows the Wealth and Health of Nations.

Source: gapminder.org.

The Emergence of Data Visualization And Visual Analytics (1 of 2)

Figure 3.23 Magic Quadrant for Business Intelligence and Analytics Platforms.

Magic Quadrant for Business Intelligence and Analytics Platforms (Source: Gartner.com)

Many data visualization companies are in the 4th quadrant

There is a move towards visualization

Source: Used with permission from Gartner Inc.

The Emergence of Data Visualization And Visual Analytics (2 of 2)

Emergence of new companies

Tableau, Spotfire, QlikView, …

Increased focus by the big players

MicroStrategy improved Visual Insight

S A P launched Visual Intelligence

S A S launched Visual Analytics

Microsoft bolstered PowerPivot with Power View

I B M launched Cognos Insight

Oracle acquired Endeca

Visual Analytics

A recently coined term

Information visualization + predictive analytics

Information visualization

Descriptive, backward focused

“what happened” “what is happening”

Predictive analytics

Predictive, future focused

“what will happen” “why will it happen”

There is a strong move toward visual analytics

Visual Analytics by S A S Institute (1 of 2)

Figure 3.25 An Overview of S A S Visual Analytics Architecture.

S A S Visual Analytics Architecture

Big data + In memory + Massively parallel processing + ..

Visual Analytics by S A S Institute (2 of 2)

At teradatauniversitynetwork.com, you can learn more about S A S V A, experiment with the tool

Figure 3.26 A Screenshot from S A S Visual Analytics.

Technology Insight 3.3 - Telling Great Stories with Data and Visualization

Figure 3.24 A Storyline Visualization in Tableau Software.

Source: Used with permission from Tableau Software, Inc.

Performance Dashboards (1 of 4)

Performance dashboards are commonly used in B P M software suites and B I platforms

Dashboards provide visual displays of important information that is consolidated and arranged on a single screen so that information can be digested at a single glance and easily drilled in and further explored

Performance Dashboards (2 of 4)

Figure 3.27 A Sample Executive Dashboard.

Source: A Sample Executive Dashboard from Dundas Data Visualization, Inc., www.dundas.com, reprinted with permission.

Application Case 3.7

Dallas Cowboys Score Big with Tableau and Teknion

Questions for Discussion:

How did the Dallas Cowboys use information visualization?

What were the challenge, the proposed solution, and the obtained results?

Performance Dashboards (3 of 4)

Dashboard design

The fundamental challenge of dashboard design is to display all the required information on a single screen, clearly and without distraction, in a manner that can be assimilated quickly

Three layer of information

Monitoring

Analysis

Management

Performance Dashboards (4 of 4)

What to look for in a dashboard

Use of visual components to highlight data and exceptions that require action.

Transparent to the user, meaning that they require minimal training and are extremely easy to use

Combine data from a variety of systems into a single, summarized, unified view of the business

Enable drill-down or drill-through to underlying data sources or reports

Present a dynamic, real-world view with timely data

Require little coding to implement, deploy, and maintain

Best Practices in Dashboard Design

Benchmark K P I s with Industry Standards

Wrap the Metrics with Contextual Metadata

Validate the Design by a Usability Specialist

Prioritize and Rank Alerts and Exceptions

Enrich Dashboard with Business-User Comments

Present Information in Three Different Levels

Pick the Right Visual Constructs

Provide for Guided Analytics

Application Case 3.8

Visual analytics helps energy supplier Make better connections

Questions for Discussion:

Why do you think energy supply companies are among the prime users of information visualization tools?

How did Electrabel use information visualization for the single version of the truth?

What were their challenges, the proposed solution, and the obtained results?

End of Chapter 3

Questions / Comments

This work is protected by United States copyright laws and is provided solely for the use of instructors in teaching their courses and assessing student learning. Dissemination or sale of any part of this work (including on the World Wide Web) will destroy the integrity of the work and is not permitted. The work and materials from it should never be made available to students except by instructors using the accompanying text in their classes. All recipients of this work are expected to abide by these restrictions and to honor the intended pedagogical purposes and the needs of other instructors who rely on these materials.

...

xxx

+++

()

(1)

skewnesss

()

kurtosisK

==-

0112233

...

yxxxx

bbbbb

=+++++

()

.MsftOfcThm_Text1_Fill { fill:#000000; } .MsftOfcThm_MainDark1_Stroke { stroke:#000000; }