PYTHON
LITTLE
14_MachineLearning.pdf
Goal Explore the types of machine learning
Use scikit-learn with popular datasets to perform machine learning
Examine datasets bundled with Scikit-learn
Learn about Steps in a typical data science study
Use Matplotlib to visualize and explore data
Divide a dataset into training, test(, and validation) sets
Case study: classification with K-Nearest neighbors and digits dataset
2
Machine Learning
Machine Learning
One of the most exciting and promising subfields of artificial
intelligence
You’ll see how to quickly solve challenging and intriguing
problems that beginners and most experienced programmers
probably would not have attempted just a few years ago.
Big, complex topic
Our goal is a friendly, hands-on introduction to a simple
machine-learning technique
3
What Is Machine Learning?
Can we really make our machines (computers) learn?
“Secret sauce” is data, and lots of it
Rather than programming expertise into our applications,
we program them to learn from data
Build working machine-learning models then use them to
make remarkably accurate predictions
4
Quick Overview
Watch this introductory video:
• https://www.youtube.com/watch?v=ukzFI9rgwfU
5
Prediction Prediction - wouldn’t it be fantastic if we could
improve weather forecasting to save lives, minimize injuries and property
damage
improve cancer diagnoses and treatment regimens to save lives
improve business forecasts to maximize profits and secure people’s jobs
detect fraudulent credit-card purchases and insurance claims
predict what prices houses are likely to sell for, anticipate revenue of
new productions and services
predict the best strategies for coaches and players to use to win more
games and championships
All of these kinds of predictions are happening today with machine learning.
6
Popular Machine Learning Applications
Top 10 Applications of Machine Learning
https://www.youtube.com/watch?v=ahRcGObyEZo
7
Libraries for Machine Learning Python libraries that used in Machine Learning are:
https://www.geeksforgeeks.org/best-python-libraries-for-machine-learning/
Keras
PyTorch
Pandas
Matplotlib
Anaconda offers these packages as part of its free distribution
We’ll use the popular scikit-learn machine learning library in this lesson
Numpy
Scipy
Scikit-learn
TensorFlow
Theano
8
Scikit-learn
can be used for data-mining and data-analysis, which makes it a great tool
starting out with Machine Learning (ML)
is one of the most popular ML libraries for classical ML algorithms
is built on top of two basic Python libraries, viz., NumPy and SciPy
supports most of the supervised and unsupervised learning algorithms
TensorFlow
is a very popular open-source library for high performance numerical
computation developed by the Google Brain team in Google.
is widely used in the field of deep learning research and application.
Can train and run deep neural networks that can be used to develop several
AI applications.
https://scikit-learn.org/stable/
https://www.tensorflow.org
9
SciKit-Learn vs. TensorFlow runs on a single CPU
processor
Is for traditional ML
Is a higher-level library
Is built on top of TensorFlow
Likes a horse
runs on multiple processors
including GPU
is more for Deep Learning
is a low-level library
starts where SciKit-Learn stops
Likes a chariot
10
Scikit-learn
Scikit-learn, also called sklearn
Conveniently packages the most effective machine-learning
algorithms as estimators
Each is encapsulated, so you don’t see the details and heavy
mathematics of how these algorithms work - like you drive a car
without knowing the intricate details of how engine, brake,
transmission, steering system work
With sklearn and a small amount of Python code, you can create powerful models quickly for analyzing data,
extracting insights from the data and making predictions
11
Scikit-learn
Use sklearn to train each model on a subset of your data,
then test each model on the rest of your data to see how
well the model works
Once the models are trained, you’ll put them to work
making predictions based on data they have not seen
Sklearn has tools that automate training and testing models
Your computer now takes on characteristics of intelligence
12
How to choose estimators
Difficult to know in advance which model(s) will perform
best on data
A popular approach is to run many models and pick the best one(s)
Sklearn makes it easy for you to try them all. It take s a few lines of
code to create and use each model
How to evaluate which model performs the best?
The models report their performance so you can compare the
results and pick the model(s) with the best performance
The more data you train with, the greater chance your model is
accurately to predict 13
Types of Machine Learning
Three basic types of Machine Learning
Supervised learning
Works with labeled data
Falls into two categories - classification and regression
Unsupervised learning
Works with unlabeled data
For example: recommendation systems on amazon
Reinforcement learning
Learn the logic of problem through feedback and improvement
14
Predict a continuous output, such as temperature output in the weather forecasting
Predict discrete classes (categories) Binary classification uses two classes Multi-classification uses more than two classes, such as 10 classes
15
16
17
Supervised Machine Learning
Falls into two categories
Classification and Regression
train machine-learning models on datasets that consist of rows and columns
Each row represents a data sample
Each column represents a feature of that sample (for example, sound: bark, meow)
In supervised machine learning
Each sample has an associated label called a target (like “dog” or “cat”)
This is the value you’re trying to predict for new data that you present to your
models
18
Datasets “Toy” datasets
small number of samples with a limited number of features
Real-world datasets (with many richly features)
containing tens of thousands of samples or more
In the world of big data, datasets commonly have millions and billions of
samples, or even more
An enormous number of free and open datasets available
Libraries like sklearn package up popular datasets for you to experiment with, and
provide mechanisms for loading datasets from various repositories such as openml
We’ll work with a free dataset, using a simplest machine learning technique
19
Datasets Bundled with Scikit-Learn
Sklearn
packages up popular datasets and provide mechanisms for loading datasets from
various repositories, such as 20,000+ datasets at https://www.openml.org/
20
Unsupervised Machine Learning Unsupervised machine learning with clustering algorithms
Helps develop predictive models/patterns in dataset without pre-
existing labels, based on both input and output data
Use dimensionality reduction (e.g. with sklearn’s TSNE estimator) to
compress the dataset’s many features down to two or few for
visualization. This enables us to see how nicely the data “cluster up”.
We can build this clustering model with just a few lines of code without
having to understand the inner workings of clustering algorithms
This is the beautify of object-based programming.
For example, build powerful deep learning models using the open source Keras library.
K-Means clustering: specify the desired # of clusters. Find similarity in unlabeled data.
Assigning labels to unlabeled data so supervised learning estimators can process it 21
Supervised vs Unsupervised
Goal: determine data patterns/grouping
For example:
For example: uses two classes, such as “spam” or “not spam” in an email classification application
For example:
uses more than two classes
uses many classes in Digits datasets that bundled with sklearn to classify handwritten with the k-Nearest neighbor algorithm
Predict continuous output:
Iris Dataset: bundled with sklearn
Predict discrete classes:
22
Multiple linear Regression uses multiple numerical features to make more sophisticated prediction than a single feature. Use LinearRegression function In Sklearn.linear_model
Special case of Multiple linear Regression with a single feature.
Big Data and Big Computer Processing Power
Computing power is exploding, and computing cost dramatically declines. This enables
us to think differently about the solution approaches. We now can program computers
to learn from data, and lots of it. It’s now all about predicting from data.
“I’m drowning in data and I don’t know what to do with it.”
People used to say:
With machine learning, we now say, “Flood me with big data so I can use machine- learning technology and powerful computing capabilities to extract insights and make predictions from it.”
23
Steps in a Typical Data Science Study
The steps of a typical machine-learning case study includes
loading the dataset
There are important steps cleaning your data before using it for machine learning
exploring the data with pandas and visualizations
transforming your data (converting non-numeric data to numeric data
because sklearn requires numeric data)
splitting the data for training and testing
creating the model
training and testing the model
tuning the model and evaluating its accuracy
making predictions on live data that the model hasn’t seen before 24
Case study with digits dataset
25
Case Study with digits dataset Environment configuration
Matplotlib (installed with Anaconda)
Scikit-learn (installed with Anaconda)
Jupyter Notebook (installed with Anaconda)
Case Study:
To process mail efficiently, postal service computers must be able to
recognize scanned handwritten letters and digits. Powerful libraries like
sklearn enable such machine learning problems manageable for a novice
We’ll look at classification in supervised machine learning, with k-Nearest
Neighbors, and the Digits Dataset which is bundled with sklearn:
Handwritten digits dataset https://scikit-learn.org/stable/datasets
26
Case Study Use Digits dataset bundled with scikit-learn
contains 8-by-8 pixel images representing 1797 hand-written digits (0 through 9)
This dataset was produced in early 1990s. At today’s high definition camera and scanner
resolutions, such images can be captured at much higher resolutions
Our goal is to predict which digit an image represents
Multi-classification problem — 10 classes since there are 10 possible digits
each class refers to a digit (0 through 9)
Train a classification model using labeled data, we know in advance each digit’s class
In this case study, we’ll use one of the simplest machine-learning classification
algorithms, k-nearest neighbors (k-NN), to recognize handwritten digits
27
Note: the term “class in this case means “category”, not the Python concept of a class
K-Nearest Neighbors Algorithm (k-NN)
Predict a sample’s class by looking at
the k training samples nearest in
"distance" to the sample
Filled dots represent four distinct
classes—A (blue), B (green), C (red)
and D (purple)
Class with the most “votes” wins
Use odd k value avoids ties — there’s never an equal number of votes
For example, x and y are close to their neighbors while z’s choice is not clear. So based
on 2 red vote and 1 green votes, z belongs to the red class
How KNN algorithms works: https://www.youtube.com/watch?v=UqYde-LULfs 28
Hyperparameters and Hyperparameter Tuning
In machine learning, a model implements a machine learning algorithm
In sklearn, models are called estimators
Two parameter types in machine learning:
Those the estimator calculates as it learns from the data provided
Those we specify in advance when creating the sklearn estimator object that
represents the model – is called hyperparameters
In the k-nearest neighbors algorithm, k is a hyperparameter, by default k=5
For simplicity, we use sklearn’s default hyperparameter value in this case study
In real-world machine-learning studies, you want to experiment with different values of k to
produce the best possible models for your studies. This process is called hyperparameter
tuning. Sklearn also has automated hyperparameter tuning capability
29
Steps in a Typical Data Science Study
Follow the steps of a typical machine-learning case study:
loading the dataset (cleaning your data before using it for machine learning)
exploring the data with pandas and visualizations
transforming your data (converting non-numeric data to numeric data because
scikit-learn requires numeric data)
splitting the data for training and testing
creating the model
training and testing the model
tuning the model and evaluating its accuracy
making predictions on live data that the model hasn’t seen before
You’ll see each step only requires at most a few lines of python code
30
Step 1: loading the data set Loading the Dataset with the load_digits Function
The Load_digits() function from sklearn.datasets module returns a scikit-
learn bunch object containing digit samples and metadata
A bunch is a subclass of dictionary with additional attributes for
interacting with the dataset
from sklearn.datasets import load_digits
digits = load_digits()
from sklearn.datasets import load_digits
digits = load_digits()
31
Displaying Digits Dataset's Description
Digits dataset is a subset of the UCI (University of California Irvine) ML hand-
written digits dataset
Original dataset: 5620 samples
3823 for training and 1797 for testing
Digits dataset bundled with sklearn contains only 1797 testing samples
A Bunch’s DESCR attribute contains a description of the dataset
According to the description, each sample has 64 features (Number of Attributes) that
represent an 8-by-8 image with pixel values in the range 0–16 (Attribute Information)
This dataset has no missing values (Missing Attribute Values)
64 features may seem like a lot
Real world datasets can have hundreds, thousands or even millions of features
Processing datasets like these can require enormous computing capabilities
32
from sklearn.datasets import load_digits
digits = load_digits()
print(digits.DESCR)
from sklearn.datasets import load_digits
digits = load_digits()
print(digits.DESCR)
33 https://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits
Checking the Sample and Target Sizes
Bunch object’s data and target attributes are NumPy arrays:
The data array contains the 1797 samples (digit images), each with 64
features having values in the range 0-16, representing pixel intensities
Pixel intensities in grayscale shades from white (0) to black (16)
target array contains the images’ labels, that is the classes indicating
which digit each image represents. The array is called target because
when making predictions, you’re aiming to “hit the target” values
# The number of target values (1797) matches the number of samples
# The number of samples (1797), and features per sample (64)
34
Display the target values of every 100th sample:
digits.target[::100] # target values of every 100th sampledigits.target[::100] # target values of every 100th sample
array([0, 4, 1, 7, 4, 8, 2, 2, 4, 4, 1, 9, 7, 3, 2, 1, 2, 5]) array([0, 4, 1, 7, 4, 8, 2, 2, 4, 4, 1, 9, 7, 3, 2, 1, 2, 5])
Output:
A Bunch object’s target attributes are NumPy array containing the dataset’s labels
35
A Sample Digit Image
Images are two-dimensional— a width and a height in pixels
The Bunch object returned by load_digits contains an images attribute
An array in which each element is a two-dimensional 8-by-8 array
representing a digit image’s pixel intensities
Scikit-learn stores the intensity values as NumPy type float64
This 8-by-8 array digits.images[13] corresponds to 1-by-64 array digits.data[13]
A 2D array representing the sample image at index 13
36
Preparing the Data for Use with sklearn
Sklearn’s machine learning algorithms require samples to be stored in a 2D array
of floating-point values (or 2D array-like collection, such as a list of lists or a
pandas DataFrame)
Each row represents one sample
Each column in a given row represents one feature for that sample
For your convenience, the load_digits function returns the preprocessed data
ready for machine learning
The Digits dataset is numerical, so load_digits simply flattens each image’s 2D array
into a 1D array. For example, the 8 by 8 array digits.image[13] shown in snippet [8]
corresponds to the 1 by 64 array digits.data[13] below in snippet [9]
37
Visualization of digits.images[13]
The image represented by this 2D array
In this 1D array, the first 8 elements are the 2D array's row 0, the next 8 elements are the 2D array’s row 1, and so on
A Bunch object’s data attribute is NumPy array containing the dataset’s samples
38
Step 2: visualizing the data Always familiarize with your data — this is called data exploration
Let's visualize the Digits dataset’s first 24 images with Matplotlib
Color map plt.cm.gray_r is for grayscale with 0 for white
figure, axes = plt.subplots(nrows=4, ncols=6, figsize=(6, 4))
for item in zip(axes.ravel(), digits.images, digits.target):
axes, image, target = item
axes.imshow(image, cmap=plt.cm.gray_r)
axes.set_xticks([]) #remove x-axis tick marks
axes.set_yticks([]) #remove y-axis tick marks
axes.set_title(target)
plt.tight_layout()
figure, axes = plt.subplots(nrows=4, ncols=6, figsize=(6, 4))
for item in zip(axes.ravel(), digits.images, digits.target):
axes, image, target = item
axes.imshow(image, cmap=plt.cm.gray_r)
axes.set_xticks([]) #remove x-axis tick marks
axes.set_yticks([]) #remove y-axis tick marks
axes.set_title(target)
plt.tight_layout()
Return a contiguous flattened array.Join tuples together
39
Output:
Creating the Diagram In [12]: plot function subplots creates a 6 by 4 inch Figure - specified by the figsize(6, 4) keyword argument, containing
24 subplots arranged in 4 rows and 6 columns. Each subplot has its own Axes object, which we’ll use to display one digit image. plt.subplots returns the Axes objects in a 2D NumPy array.
Each iteration of the loop: Use a for statement with the build-in zip function to iterate in parallel through the 24 Axes objects, the first 24 images
in digits.images, and the first 24 values in digits.target Ravel(): creates a 1D view of a multidimensional array Buildin function zip(): enables you to iterate over multiple iterables of data at the same time. It receives as arguments any number of iterables and returns an iterator that produces tuples containing the elements at the same index in each. (argument with the fewest elements determines how many tuples zip returns) Unpacks one tuple from the zipped item into three variables representing the Axes object, image and target value Call Axes object’s imshow method to display one image. The keyword argument cmap=plt.cm.gray_r determines the
colors displayed in the image. This particular color map displays the image’s pixels in grayscale, in the range of 0-16 For Matplotlib’s color map names see https://matplotlib.org/examples/color/colormaps_reference.html. Axes object’s set_xticks and set_yticks methods with empty list indicates the x- and y-axes should not have tick marks. Axes object’s set_title method displays the target value above the image – the actual value that the image represents
plt.tight_layout(): remove the extra whitespace at the Figure’s top, right, bottom, and left, so the rows and columns of digits images can fill more of the Figure. 40
Pay attention to the variations among the images of the 3s in the first, third, and fourth rows It seems difficult to recognize different handwritten digits
41
Output:
Step 3: Splitting the data Splitting the Data for Training and Testing
Typically train a machine-learning model with a subset of a dataset (training set)
Typically the more data you have for training, the better you can train the model
Set aside a portion of your data for testing, so you can evaluate a model’s
performance using data the model has not yet seen (testing set)
Function train_test_split() (from the sklearn.model_selection module)
shuffles the data to randomize it, then splits the samples in the data array and the
target values in the target array into training and testing sets
This ensures the training and testing sets have similar characteristics
After confirming a well performed model, use it to make predictions for new data it
hasn’t seen from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
digits.data, digits.target, random_state=11, test_size=0.20)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
digits.data, digits.target, random_state=11, test_size=0.20)
42
Function train_test_split() returns a tuple of four elements, in which the first two are the samples
split into training and testing sets, and the last two are the corresponding target values split into
training and testing sets. By convention, uppercase X is used to represent the samples, and
lowercase y is used to represent the target values.
random_state helps to seed a random number generator for reproducibility. Here it helps to
confirm your results by working with the same randomly selected data. 11 is the seed value we
selected arbitrarily. When you run the code in the future with the same seed value, train_test_split
will select the same data for the training set and the same data for the testing set.
test_size=0.20 means 20% of the data is for testing while 80% is inferred for training By default, train_test_split reserves 75% of the data for training and 25% for testing
To specify different splits, set train_test_split keyword arguments test_size or train_size. Use floating-point
values from 0.0 through 1.0 to specify the percentage, or integer values to set the precise # of samples. If
one of these keyword arguments specified, the other is inferred.
43
Training and Testing Set Sizes
Note:
Convention:
Uppercase X represents samples
Lowercase y represents target values
Scikit-learn bundled classification datasets have balanced classes
Samples are divided evenly among the classes
Unbalanced classes could lead to incorrect results
44
Step 4: Creating the model In scikit-learn, models are called estimators
KNeighborsClassifier estimator (module sklearn.neighbors) implements
the k-nearest neighbors algorithm
Create the KNeighborsClassifier estimator object:
To create an estimator (model), you simply create an object. The internal
details of how this object implements the k-nearest neighbors algorithms are
hidden in the object. You just simply call its methods. This is the essence of
Python object-based programming 45
https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html
Step 5: Training the model Training the Model with the KNeighborsClassifier Object’s fit() method
Load sample training set (X_train) and target training set (y_train) into the estimator:
The fit method returns the estimator. So IPython displays its string representation (includes the estimator’s default settings)
By default, parameters for class sklearn.neighbors.KNeighborsClassifier(n_neighbors=5, weights='uniform',
algorithm='auto', leaf_size=30, p=2, metric='minkowski', metric_params=None, n_jobs=None)
n_neighbors value corresponds to k in the KNN algorithm. By default a KNeighborsClassifier looks at the 5 nearest
neighbors to make its predictions
The KNeighborsClassifier’s fit() method just loads the data into the estimator - knn has no initial learning process. The
estimator is said to be lazy because its work is performed only when you use it for predictions
For most other sklearn estimators, the fit() method loads the data into the estimator then uses that data to perform
complex calculations behind the scenes that learn from the data and train the model. Lots of models have significant
training phases that can take minutes, hours, days or months (- deep learning)
High-performance GPUs and TPUs can significantly reduce model training time 46
https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html
Step 6: Predicting Now that we’ve loaded the data into the KNeighborsClassifier, we can use it
with the test samples to make predictions
Predicting Digit Classes with the KNeighborsClassifier’s predict() method
With X_test as an argument
Returns an array containing the predicted class of each test image:
predicted digits vs. expected digits for the first 20 test samples Mismatch at index 18
47
Step 7: Evaluating the model Locate all incorrect predictions for the entire test set:
The list comprehension uses zip to create tuples containing the corresponding elements in predicted and expected.
We include a tuple in the result only if its p (predicted value) and e (expected value) differ – that is the predicted value incorrect.
In this example, the estimator incorrectly predicted only 5 of the 360 test samples.
48
Step 7: Evaluating the model (II)
Evaluate the k-NN classification estimator’s accuracy
Estimator Method score
Each estimator has a score method that returns an indication of how well the
estimator performances for the test data you pass as argument
The kNeighborsClassifier’s with its default k (that is, n_neighbors=5) achieved 98.61% accuracy
For classification estimators, this score method returns the prediction accuracy for the test data
49
Unsupervised Machine Learning
Brief overview of unsupervised machine learning https://www.youtube.com/watch?v=IUn8k5zSI6g
50
Summary Explore the types of machine learning
Use scikit-learn with popular datasets to perform machine learning
Examine datasets bundled with Scikit-learn
Learn about Steps in a typical data science study
Use Matplotlib to visualize and explore data
Divide a dataset into training, test(, and validation) sets
Case study: classification with K-Nearest neighbors and digits dataset
51
Final Project Part II
Final Project Part II available in myleoOnline
Due date: Apr. 22 Thu. by Midnight 11:59pm
Note:
Be sure to reserve time in next week (Week 15: 4/19-4/25) to
review for Final Exam.
Week 16 (4/26-4/30) is Final Exam week.
