linear regression and R program

GR5205 Handout 3 Linear regression Due: Friday, Oct. 7 2016

Homework 3

1. Consider the function f(x1,x2) = (x1 −1)2 + (x2 −2)2 + (x3 −1)2 + x1x2x3. Calculate the directional derivative of this function at x = [1, 1, 1]T in the direction [1, 1, 1]T .

2. (Direction of maximum ascent) One of the important questions that is often asked in the area of optimization is about the direction of maximum ascent. Consider a function f : Rp → R. We look at the function in a neighborhood of xo and ask ourselves which direction we should pick in a neighborhood of f to increase the function value the most (this is informal definition of the maximum ascent problem). More formally, let α ∈ Rp satisfy the property αT α = 1. The question can be asked in the following way: for which value of α is Dfα(xo) maximized.

(a) Explain the connection between the informal and formal definitions of the “direc- tion of maximum ascent problem”.

(b) In order to find the direction of maximum ascent we should first prove another re- sult, known as Cauchy-Shwartz inequality. Suppose that we have two vectors x ∈ Rp and y ∈ Rp. Cauchy-Schwartz inequality claims that (xT y)2 ≤ (xT x)(yT y). To prove this claim note that for every γ ∈ R we have

(x−γy)T (x−γy) ≥ 0.

Think how we can set γ to get the Cauchy-Schwartz inequality.

(c) Use Cauchy-Schwartz inequality to obtain the direction of maximum ascent.

3. Consider the function f : R2 → R.Assume that all the second order partial derivatives of this function exist and are continuous. We define the function g : R → R in the following way:

g(t) = f(xo + tα),

where xo and α are two vectors in R 2.

(a) Calculate g′(to) in terms of the gradient of f.

(b) (optional) calculate g′′(0) in terms of ∂ 2f

∂x21 , ∂

2f ∂x22

, and ∂ 2f

∂x1∂x2 . Can you connect this

problem to the discussion we had in class about the Hessian matrix?

4. Consider the linear regression model. In many cases the errors on each sample have different levels of importance for us. We will see good examples later in the course. Let ei = (yi −xTi β̂). Then, instead of minimizing

∑n i=1 e

2 i , we would like minimize

n∑ i=1

wie 2 i ,

where wis denote fixed weights. Write a matrix form of this cost. You can use a diagonal matrix W with diagonal elements (w1,w2, . . . ,wn). Now minimize this cost function and find its minimizer β̂.

1

5. (Data clean-up) Download the CPI dataset I have uploaded on the course website. This dataset gives you the consumer-product-index as a function of year and month. Answer these questions about this dataset.

Hint: Look up the following commands and see if they can help you in this problem: lm, abline, summary, is.na, which,

(a) Read the CPI dataset in R. You may use ‘read.csv’ for this purpose. What is the type of the data that you get?

(b) Since the value of CPI was not available for some years, you see some ‘na’ instead of the actual CPI values. Construct two vectors. One for CPIs excluding all the cells that have ‘na’ and one for the corresponding years. Call them year.month.vec and cpi.vec.ym. For this purpose you may use ‘is.na’ command in R. What is the size of these two vectors.

(c) Use the plot command to plot cpi.vec.ym in terms of year.month.vec. Why do you see multiple points for each year? Construct two more vectors ‘year.vec’ and ‘cpi.vec’ in the following way. For each year i you take the average of CPI in all the months of that year to get the corresponding CPI. Then put the average CPI value in ”cpi.vec” and the corresponding year in ”year. vec”. Again plot cpi.vec as a function of year.vec.

(d) Suppose that we want to model the inflation rate and see how much inflation we should expect per year (this is the minimum amount of raise you should expect in your salaries in the future. Do you see why?). What type of model do you suggest for this problem? Explain your model very clearly and then find the parameters of your model? Plot the predictions of your model in the same figure that you have your data points. We will explain how you can quantitatively evaluate the accuracy of your fit later, but for the moment based on the graph you have, do you think the fit is good? Which years seem to deviate from your predictions more?

(e) Suppose that the salary of a person is 100K per year. Given your model, what is the minimum amount of raise he/she should expect after 1 year of working for the company? How about after 10 years.

(f) Great depression started in US around 1929 and lasted for 10 years. I encourage you to read a little bit about the great depression. Can you see any sign of the great depression in the CPI graph? Also, great recession happened in 2009 in US. Do you see any sign of that in CPI?

(g) Now suppose that I claim, because of great depression we should not use the data points of the years 1929 to 1940 for model fitting. They are in some sense outliers and do not represent the usual behavior of the inflation. Then use the same model you used in part (d), but this time only fit it to the CPI data that are collected after 1940. How much difference do you see in your predictions? Again suppose that the salary of a person is 100K. Based on your new predictions, what is the minimum raise he/she should expect per year?

2