Data Mining

Dr. Oner Celepcikay

ITS 632

ITS 632 - Data Mining

Week 2 - Introduction to Data Mining

v Dr. Oner Celepcikay @ocelepcikay

v Students (please respond to Discussion Board)

v Name / Major / Department / Position

v Data Mining Background

v Programming (esp. Python) Background

v Expectations from this course

v Anything else you want to share

Introductions ITS 632

Dr. Oner Celepcikay @ocelepcikay

v I teach

v Data Science @ Rice University

v Data Management @ Rice University

v Data Mining & Machine Learning @ UHD

v Big Data Analytics @ UH Sugar Land

v Data Mining @ Texas A&M Commerce

v CyberSecurity & Data Mining @ UCumberlands

Introductions ITS 632

Tentative Schedule ITS 632

ITS 632 Tentative Schedule

v Why Python

v What is Data Mining v Data Mining Real World Applications v What is Data Mining

v Machine Learning Methods

Outline ITS 632

v Going over Syllabus

v Recommended Book

v Install Anaconda v Questions about the course and the Logistics??

About the Course ITS 632

v Easy to learn, easy to teach

vEmerging Jobs Report by LinkedIn: The most prominent skills in 2018 (to be hired)

Why Python ITS 632

6. Python

7. Software Development

8. Analytics 9. Cloud Computing 10. Retail

1. Management

2. Sales

3. Communication 4. Marketing 5. Start-Ups

Why Python ITS 632

Why Python ITS 632

Data Mining – Increasingly Popular v Question: Driving forces behind popularity of Data

Analytics & Data Mining?

ITS 632

ITS 632

v Enormous amount of Data Collection

v Web data, e-commerce, Customer history v Purchases at department / grocery stores (RFIDs)

v Bank/Credit Card transactions

v Competitive Pressure is Strong (i.e. Netflix) v Provide better, customized services

v Collect Now, Analyze Later – cheap storage!

What are the Driving forces behind popularity of Data Analytics & Data Mining?

Data Revolution ITS 632

• Lots of data is being collected and warehoused – Web data, e-commerce – Financial transactions, bank/credit

transactions – Online trading and purchasing – Social Network

Data Revolution ITS 632

• Google processes 20 PB a day (2008)

• Facebook has 60 TB of daily logs

• eBay has 6.5 PB of user data + 50 TB/day (5/2009)

• 1000 genomes project: 200 TB

Data Revolution ITS 632

ITS 632

Big Data ITS 632

v Big Data is any data that is expensive to manage and hard to extract value from v Volume

v The size of the data v Velocity

v The latency of data processing relative to the growing demand for interactivity

v Variety and Complexity vthe diversity of sources, formats, quality, structures.

Big Data ITS 632

Definition of Data Mining ITS 632

• Theories and techniques from many fields and disciplines are

used to investigate and analyze a large amount of data to help

decision makers in many industries such as science,

engineering, economics, politics, finance, and education – Computer Science

• Pattern recognition, visualization, data warehousing, High performance computing, Databases, AI

– Mathematics • Mathematical Modeling

– Statistics • Statistical and Stochastic modeling, Probability.

Definition of Data Mining ITS 632

Definition of Data Mining ITS 632

Definition of Data Mining ITS 632

Data Scientist ITS 632

v Data scientists are the key to realizing the

opportunities presented by big data.

v They bring structure to it, find compelling patterns

in it, and advise executives on the implications for products, processes, and decisions

Data Mining Concentrations

ITS 632

v Mathematics and Applied Mathematics

v Applied Statistics/Data Analysis

v Solid Programming Skills (R, Python, Julia, SQL)

v Data Mining

v Data Base Storage and Management

v Machine Learning and discovery

Data Mining Applications I ITS 632

ITS 632

“My daughter got this in the mail!” he said. “She’s still in high school, and you’re sending her coupons for baby clothes and cribs? Are you trying to encourage her to get pregnant?”

v What do you think happened?

v How could be used as an competitive advantage?

v Another example: Lexus Car Sales Marketing

Data Mining Applications I

ITS 632 Data Mining Applications 1

ITS 632

http://www.nytimes.com/2015/01/20/science/facebook-knows- you-better-than-anyone-else.html

Data Mining Applications 2 – Social Media

ITS 632

v The survey rated each subject on 5 personality traits: v openness, v conscientiousness, v extraversion,

v Given enough data, the algorithm was better able to

predict a person’s personality traits than humans.

v 10 likes to beat a work colleague, 70 to beat a

roommate, 150 to beat a parent or sibling, and 300 to

beat a spouse.

v agreeableness v neuroticism.

Data Mining Applications 2 – Social Media

Talk on Data Mining on Social Media Data

https://youtu.be/hgWie9dnssU

ITS 632

Curly Fries and Being Smart??

• The Obama campaigns in 2008 and 2012 are credited

for their successful use of social media and data mining.

• Micro-targeting in 2012

– Micro-profiles built from multiple sources accessed by aps, real-

time updating data based on door-to-door visits, focused media

buys, e-mails and Facebook messages highly targeted.

– 1 million people installed the Obama FB app that gave access to

info on “friends”.

31

ITS 632

Data Mining Applications 3 – Elections

32

ITS 632

Data Mining Applications 3 – Elections

33

ITS 632

v How About 2016 Election??

v Controversy around FB, Data sharing & Cambridge

Analytica

Data Mining Applications 3 – Elections

34

ITS 632

He told the Observer:

“We exploited Facebook to

harvest millions of people’s

profiles. And built models to

exploit what we knew about

them and target their inner

demons. That was the basis the

entire company was built on.”

https://www.theguardian.com/news/2018/mar/17/cambridge-analytica-facebook-

influence-us-election

Data Mining Applications 3 – Elections

Data in Science

ITS 632

v Data collected and stored at

enormous speeds v remote sensors on a satellite

v telescopes scanning the skies

v microarrays generating gene expression

data

v scientific simulations generating

terabytes of data

v Traditional techniques infeasible for raw data

Mining Large Data Sets – Data Revolution v There is often

information “hidden”

in the data that is

not readily evident

v Human analysts may take weeks to discover useful information

v Much of the data is never analyzed at all

ITS 632

ITS 632 Data Science Apps 4 – Recommender System

v In 2009, Netflix awarded a $1 million prize to anyone

who could come up with a better movie recommending

model. (improve by %10)

vProvided contestants 100 million ratings from 480,000

customers of 18,000 movies.

vWinners would be determined by which model best

predicted 2.8 million ratings that they were NOT given

(a bit more complex than this

v Recommender systems identify recommendations autonomously for individual users based on past purchases and searches, and on other users' behavior.

v Examples:

ITS 632 Data Science Apps 4 – Recommender System

ITS 632 Data Science Apps 5 – Prediction (Regression) v Predict a value of a given continuous valued variable

based on the values of other variables, assuming a linear or nonlinear model of dependency.

v Greatly studied in statistics, neural network fields. v Examples:

vPredicting sales amounts of new product based on advertising expenditure.

vPredicting wind velocities as a function of temperature, humidity, air pressure, etc.

vTime series prediction of stock market indices.

ITS 632 Data Science Apps 6 – Anomaly Detection

v Detect significant deviations from normal behavior

v Applications: v Credit Card Fraud Detection v Network Intrusion Detection

What is Data Mining? v Non-trivial extraction of implicit, previously unknown

and potentially useful (---) information from data v Exploration & analysis, by automatic means, of large

quantities of data to discover meaningful patterns

ITS 632

v Prediction Methods v Use variables to predict unknown or future values

v e.g. house prices, medical studies, dep./ind. Variables

v Description Methods v Find human-interpretable patterns that describe the data.

v E.g.: correlation, clusters, trajectories, anomalies

Data Mining Tasks ITS 632

What is Next? ITS 632

Data Mining Methods (Tasks)... ITS 632

v Classification [Predictive]

v Clustering [Descriptive] v Association Rule Discovery [Descriptive]

v Sequential Pattern Discovery [Descriptive] v Regression [Predictive] v Deviation Detection [Predictive]

v Recommender Systems [Predictive]

Data Science Methods I - Classification ITS 632

v Given a collection of records (training set)

- Each record contains a set of attributes, one of the

attributes is the class.

v Find a model for class attribute as a function of the values of other attributes.

v A test set is used to estimate the accuracy of the model.

Goal: previously unseen records (test set) should be assigned a class as accurately as possible.

Classification Example ITS 632

Tid Refund Marital Status

Taxable Income Cheat

1 Yes Single 125K No

2 No Married 100K No

3 No Single 70K No

4 Yes Married 120K No

5 No Divorced 95K Yes

6 No Married 60K No

7 Yes Divorced 220K No

8 No Single 85K Yes

9 No Married 75K No

10 No Single 90K Yes 10

ca teg

or ica

l

ca teg

or ica

l

co nti

nu ou

s

cla ss

Refund Marital Status

Taxable Income Cheat

No Single 75K ?

Yes Married 50K ?

No Married 150K ?

Yes Divorced 90K ?

No Single 40K ?

No Married 80K ? 10 Test

Set

Training Set Model

Learn Classifier

Classification – Application II: Direct Marketing

ITS 632

Approach:

v Use the data for a similar product introduced before.

v We know which customers decided to buy and which did not.

This {buy, don’t buy} decision forms the class attribute.

v Demographic, lifestyle, & related information about customers.

v Type of business, where they stay, how much they earn, etc.

v Use this info as input attributes to learn a classifier model.

Goal: Reduce cost of mailing by targeting a set of consumers likely to buy a new cell-phone product.

Jupyter Notebooks ITS 632

v For this week, make sure you installed Anaconda

properly and can launch Jupyter Notebook

successfully

v Next weeks we will work on Jupyter and Python

v 1st programming assignment would be on Jupyter

Notebooks and given soon