I want to solve questions 1 3 and 6

HW Assignment 2 (Mathematical Data Science Math 3670/5670)

1. Concerning linear regression, i.e. the situation where we know that E(Y |X) = XT β, for some β ∈ IRd, we proved in class that the value of β can be found by minimizing the EPE for the problem, namely E((Y − XT β)2), and this minimization gives rise to

β = (E(XXT ))−1E(XY ). (1)

Prove this formula directly from the fact that

E(Y |X) = XT β. (2)

In particular, (a) multiply both sides of (2) by X (from the left), then (b) take the expectation w.r.t. X on both sides (i.e. you will get E(XE(Y |X)) = E(XXT β), and (c) show that this implies formula (1).

2. Let X = (X1,X2) T be a random vector. This problem computes the expected squared length E(‖X‖2) =

E(X21 + X 2 2 ) under different assumptions on the random variables X1,X2.

(a) Suppose X1 is a uniform random variable on [0, 1], and X2 is Gaussian random variable with mean zero and variance one, and X1,X2 are independent. Write an expression for the pdf pX (x1,x2) = pX1,X2 (x1,x2) and compute E(‖X‖2). (b) Suppose X1 is a uniform random variable on [0, 1] and X2 is a uniform random variable on [X1,X1 + 1]. Note that X1,X2 are not independent. Write expressions for the pdf pX1 (x1) and the conditional pdf pX2|X1 (x2|x1) and compute E(‖X‖

2).

3. Suppose X is a random vector in IRd and Y is a random variable with mean E(Y ) = µ and variance V ar(Y )] = σ2. Suppose that X and Y are independent. Find a simple expression for the regression function f : IRd → IR that minimizes the EPE E((Y − f(X))2). Rigorously justify your answer, but also give an intuitive explanation.

4. Suppose X is uniformly distributed on [−1, 1], and Z is a Gaussian random variable with mean 0 and variance 1. Assume X,Z are independent and define Y = eX + ZX. Compute the regression function E(Y |X = x).

5. Suppose X is uniformly distributed on [0, 2] and Z is uniformly distributed on [0, 1] and X,Z are independent. Let

Y =

{ 1, if Z ≤ X, 0, if Z > X.

Compute the regression function E(Y |X = x). Remark: Y is not a continuous random variable, but you can use E(Y |X = x) = (1)P(Z ≤ X|X = x) + (0)P(Z > X|X = x). Note that we wish to predict an output that only takes values 0 or 1. Does your regression function only take values 0 or 1? Recall that E(Y |X = x) arose by solving a minimization problem with the squared loss function. Does this give you any intuition about the suitability of standard squared loss regression when outputs are discrete-valued?

6. Let G1 and G2 be independent Gaussian random variables with mean zero and variance one (i.e. standard normal r.v.s). Define the random vector X = (X1,X2)

T by X1 = G1 and X2 = G1 + G2. Define Y = X1 + (X2)

2 + Z, where Z is a uniform random variable on [−1, 1] that is independent of both G1 and G2. Find the best linear model for predicting the output Y from the input X. In other words, find the vector β = (β1,β2)

T that minimizes E((Y −〈X,β〉)2). (Note: Thus, we know that the true regression function is not linear, but we are still attempting to approximate it by a linear function.)

1