Machine learning

CSI436/536 homework 5 Machine Learning

Requirement:

· You need to submit your homework as a .py or an .ipynb file.

· You need to test your code to make sure it runs with no bugs. Code that cannot run will lose all points.

Logistic Regression

Consider the objective function in logistic regression problem 𝑙(𝜃) =

𝑚

∑ (𝑦𝑖 log𝜎(𝜃𝑇𝑥𝑖) + (1 − 𝑦𝑖 )log(1 − 𝜎(𝜃𝑇𝑥𝑖))),

𝑖=1

where 𝜎(𝑧) = (1 + 𝑒−𝑧)−1 is the logistic function.

The given hwX.txt and hwY.txt contain the inputs 𝑥𝑖 ∈ 𝑅2 and outputs 𝑦𝑖 ∈ {0,1} respectively for a binary classification problem, with one training example per row.

1. (30 points) Implement the gradient descent method for optimizing 𝑙(𝜃), and apply it to fit a logistic regression model to the data. Initialize gradient descent method with 𝜃 = 0 (the vector of all zeros). What are the coefficients 𝜃 resulting from your fit? (Remember to include the intercept term.)

2. (40 points) Implement Newton's method to maximize 𝑙(𝜃), and compare the overall running time and number of iterations needed to converge to the same precision.

3. (10 points) Plot the training data (your axes should correspond to the two coordinates of the inputs, and you should use a different symbol for each point plotted to indicate whether that example had label 1 or 0).

4. (20 points) Also plot on the same figure the decision boundary fit by logistic regression. (i.e., this should be a straight line showing the boundary separating the region where 𝜎(𝜃𝑇𝑥) > 0.5 from where𝜎(𝜃𝑇𝑥) < 0.5.)